Exploration of a problem interpreting binary test results

Copyright 2015 Allen Downey
MIT License

```python
from __future__ import print_function, division

import thinkbayes2

from sympy import symbols
```

`p` is the prevalence of a condition
`s` is the sensititivity of the test

The false positive rate is known to be either `t1` (with probability `q`) or `t2` (with probability `1-q`)

```python
p, q, s, t1, t2 = symbols('p q s t1 t2')
```

I'll use `a` through `h` for each of the 8 possible conditions.

```python
a, b, c, d, e, f, g, h = symbols('a b c d e f g h')
```

And here are the probabilities of the conditions.

```python
a =   q   *   p   *   s
b =   q   *   p   * (1-s)
c =   q   * (1-p) *   t1
d =   q   * (1-p) * (1-t1)
e = (1-q) *   p   *   s
f = (1-q) *   p   * (1-s)
g = (1-q) * (1-p) *   t2
h = (1-q) * (1-p) * (1-t2)
```

```python
pmf1 = thinkbayes2.Pmf()
pmf1['sick'] = p*s
pmf1['notsick'] = (1-p)*t1
pmf1
```

    Pmf({'notsick': t1*(-p + 1), 'sick': p*s})

```python
nc1 = pmf1.Normalize()
nc1.simplify()
```

    p*s - t1*(p - 1)

```python
pmf2 = thinkbayes2.Pmf()
pmf2['sick'] = p*s
pmf2['notsick'] = (1-p)*t2
pmf2
```

    Pmf({'notsick': t2*(-p + 1), 'sick': p*s})

```python
nc2 = pmf2.Normalize()
nc2.simplify()
```

    p*s - t2*(p - 1)

```python
pmf_t = thinkbayes2.Pmf({t1:q, t2:1-q})
pmf_t[t1] *= nc1
pmf_t[t2] *= nc2
pmf_t.Normalize()
```

    1.0*q*(p*s + t1*(-p + 1)) + (-1.0*q + 1.0)*(p*s + t2*(-p + 1))

```python
pmf_t.Mean().simplify()
```

    1.0*(q*t1*(p*s - t1*(p - 1)) - t2*(q - 1)*(p*s - t2*(p - 1)))/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))

```python
d1 = dict(q=0.5, p=0.1, s=0.5, t1=0.2, t2=0.8)
pmf_t.Mean().evalf(subs=d1)
```

    0.662000000000000

```python
d2 = dict(q=0.75, p=0.1, s=0.5, t1=0.4, t2=0.8)
pmf_t.Mean().evalf(subs=d2)
```

    0.554000000000000

```python
pmf_t[t1].evalf(subs=d2)
```

    0.615000000000000

```python
x = pmf_t[t1] * pmf1['sick'] + pmf_t[t2] * pmf2['sick']
x.simplify()
```

    1.0*p*s/(q*(p*s - t1*(p - 1)) - (q - 1)*(p*s - t2*(p - 1)))

```python
x.evalf(subs=d1)
```

    0.100000000000000

```python
x.evalf(subs=d2)
```

    0.100000000000000

```python

```

```python
t = q * t1 + (1-q) * t2

pmf = thinkbayes2.Pmf()
pmf['sick'] = p*s
pmf['notsick'] = (1-p)*t
pmf
```

    Pmf({'notsick': (-p + 1)*(q*t1 + t2*(-q + 1)), 'sick': p*s})

```python
pmf.Normalize()
```

    p*s + (-p + 1)*(q*t1 + t2*(-q + 1))

```python
pmf['sick'].simplify()
```

    1.0*p*s/(p*s - (p - 1)*(q*t1 - t2*(q - 1)))

```python
pmf['sick'].evalf(subs=d1)
```

    0.100000000000000

```python
pmf['sick'].evalf(subs=d2)
```

    0.100000000000000

```python

```

```python
gold = thinkbayes2.Pmf()
gold['0 sick t1'] = q * (1-p)**2 * t1**2
gold['1 sick t1'] = q * 2*p*(1-p) * s * t1
gold['2 sick t1'] = q * p**2 * s**2
gold['0 sick t2'] = (1-q) * (1-p)**2 * t2**2
gold['1 sick t2'] = (1-q) * 2*p*(1-p) * s * t2
gold['2 sick t2'] = (1-q) * p**2 * s**2
gold.Normalize()
```

    p**2*q*s**2 + p**2*s**2*(-q + 1) + 2*p*q*s*t1*(-p + 1) + p*s*t2*(-p + 1)*(-2*q + 2) + q*t1**2*(-p + 1)**2 + t2**2*(-p + 1)**2*(-q + 1)

```python
p0 = gold['0 sick t1'] + gold['0 sick t2']
p0.evalf(subs=d1)
```

    0.852895633323010

```python
p0.evalf(subs=d2)
```

    0.826831935836675

```python

```

```python
t = q * t1 + (1-q) * t2
```

```python
collapsed = thinkbayes2.Pmf()
collapsed['0 sick'] = (1-p)**2 * t**2
collapsed['1 sick'] = 2*p*(1-p) * s * t
collapsed['2 sick'] = p**2 * s**2
```

```python
collapsed.Normalize()
```

    p**2*s**2 + 2*p*s*(-p + 1)*(q*t1 + t2*(-q + 1)) + (-p + 1)**2*(q*t1 + t2*(-q + 1))**2

```python
collapsed['0 sick'].evalf(subs=d1)
```

    0.810000000000000

```python
collapsed['0 sick'].evalf(subs=d2)
```

    0.810000000000000

```python

```

```python
pmf1 = thinkbayes2.Pmf()
pmf1['0 sick'] = (1-p)**2 * t1**2
pmf1['1 sick'] = 2*p*(1-p) * s * t1
pmf1['2 sick'] = p**2 * s**2
nc1 = pmf1.Normalize()
```

```python
pmf2 = thinkbayes2.Pmf()
pmf2['0 sick'] = (1-p)**2 * t2**2
pmf2['1 sick'] = 2*p*(1-p) * s * t2
pmf2['2 sick'] = p**2 * s**2
nc2 = pmf2.Normalize()
```

```python
pmf_t = thinkbayes2.Pmf({t1:q, t2:1-q})
pmf_t[t1] *= nc1
pmf_t[t2] *= nc2
pmf_t.Normalize()
```

    1.0*q*(p**2*s**2 + 2*p*s*t1*(-p + 1) + t1**2*(-p + 1)**2) + (-1.0*q + 1.0)*(p**2*s**2 + 2*p*s*t2*(-p + 1) + t2**2*(-p + 1)**2)

```python
x = pmf_t[t1] * pmf1['0 sick'] + pmf_t[t2] * pmf2['0 sick']
x.simplify()
```

    1.0*(p - 1)**2*(q*t1**2 + t2**2*(-q + 1))/(q*(p**2*s**2 - 2*p*s*t1*(p - 1) + t1**2*(p - 1)**2) - (q - 1)*(p**2*s**2 - 2*p*s*t2*(p - 1) + t2**2*(p - 1)**2))

```python
x.evalf(subs=d1), p0.evalf(subs=d1)
```

    (0.852895633323010, 0.852895633323010)

```python
x.evalf(subs=d2), p0.evalf(subs=d2)
```

    (0.826831935836675, 0.826831935836675)

`pmf_t` represents the distribution of `t`

```python
pmf_t = thinkbayes2.Pmf({t1:q, t2:1-q})
pmf_t.Mean().simplify()
```

    1.0*q*t1 - 1.0*t2*(q - 1)

I'll consider two sets of parameters, `d1` and `d2`, which have the same mean value of `t`.

```python
d1 = dict(q=0.5, p=0.1, s=0.5, t1=0.2, t2=0.8)
pmf_t.Mean().evalf(subs=d1)
```

    0.500000000000000

```python
d2 = dict(q=0.75, p=0.1, s=0.5, t1=0.4, t2=0.8)
pmf_t.Mean().evalf(subs=d2)
```

    0.500000000000000

`prob` takes two numbers that represent odds in favor and returns the corresponding probability.

```python
def prob(yes, no):
    return yes / (yes + no)
```

Scenario A

In the first scenario, there are two kinds of people in the world, or two kinds of tests, so there are four outcomes that yield positive tests: two true positives (a and d) and two false positives (c and g).

We can compute the probability of a true positive given a positive test:

```python
res = prob(a+e, c+g)
res.simplify()
```

    p*s/(-p*q*t1 + p*q*t2 + p*s - p*t2 + q*t1 - q*t2 + t2)

In this scenario, the two parameter sets yield the same answer:

```python
res.evalf(subs=d1)
```

    0.100000000000000

```python
res.evalf(subs=d2)
```

    0.100000000000000

Scenario B

Now suppose instead of two kinds of people, or two kinds of tests, the distribution of `t` represents our uncertainty about `t`.  That is, we are only considering one test, and we think the false positive rate is the same for everyone, but we don't know what it is.

In this scenario, we need to think about the sampling process that brings patients to see doctors.  There are three possibilities:

B1. Only patients who test positive see a doctor.

B2. All patients see a doctor with equal probability, regardless of test results and regardless of whether they are sick or not.

B3.  Patients are more or less likely to see a doctor, depending on the test results and whether they are sick or not.

Scenario B1

If patients only see a doctor after testing positive, the doctor doesn't learn anything about `t` just because a patient tests positive.  In that case, the doctor should compute the conditional probabilities:

*  `p1` is the probability the patient is sick given a positive test and `t1`
*  `p2` is the probability the patient is sick given a positive test and `t2`

We can compute `p1` and `p2`, form `pmf_p`, and compute its mean:

```python
p1 = prob(a, c)
p1.simplify()
```

    p*s/(p*s - t1*(p - 1))

```python
p1.evalf(subs=d1)
```

    0.217391304347826

```python
p2 = prob(e, g)
p2.simplify()
```

    p*s/(p*s - t2*(p - 1))

```python
p2.evalf(subs=d1)
```

    0.0649350649350649

```python
pmf_p = thinkbayes2.Pmf([p1, p2])
```

```python
pmf_p.Mean().simplify()
```

    0.5*p*s*(2*p*s - t1*(p - 1) - t2*(p - 1))/((p*s - t1*(p - 1))*(p*s - t2*(p - 1)))

```python
pmf_p.Mean().evalf(subs=d1)
```

    0.141163184641446

```python
p1.evalf(subs=d2), p2.evalf(subs=d2), pmf_p.Mean().evalf(subs=d2)
```

    (0.121951219512195, 0.0649350649350649, 0.0934431422236300)

Scenario B2

If all patients see a doctor, the doctor can learn about `t` based on the number of positive and negative tests.

* The likelihood of a positive test given `t1` is `(a+c)/q`

* The likelihood of a positive test given `t2` is `(e+g)/(1-q)`

`update` takes a `pmf` and updates it with these likelihoods

```python
def update(pmf):
    post = pmf.Copy()
    post[p1] *= (a + c) / q
    post[p2] *= (e + g) / (1-q)
    post.Normalize()
    return post
```

`post` is what we should believe about `p` after seeing one patient with a positive test:

```python
post = update(pmf_p)
post[p1].simplify()
```

    (-0.5*p*s + 0.5*t1*(p - 1))/(-1.0*p*s + 0.5*t1*(p - 1) + 0.5*t2*(p - 1))

```python
post.Mean().simplify()
```

    -1.0*p*s/(-1.0*p*s + 0.5*t1*(p - 1) + 0.5*t2*(p - 1))

When `q` is 0.5, the posterior mean is `p`:

```python
post.Mean().evalf(subs=d1)
```

    0.100000000000000

But other distributions of `t` yield different values.

```python
post.Mean().evalf(subs=d2)
```

    0.0847457627118644

Let's see what we get after seeing two patients

```python
post2 = update(post)
```

```python
post2.Mean().simplify()
```

    1.0*p*s*(2*p*s - t1*(p - 1) - t2*(p - 1))/((p*s - t1*(p - 1))**2 + (p*s - t2*(p - 1))**2)

Positive tests are more likely under `t2` than `t1`, so each positive test makes it more likely that `t=t2`.  So the expected value of `p` converges on `p2`.

```python
post2.Mean().evalf(subs=d1)
```

    0.0774233508826262

```python
post2.Mean().evalf(subs=d2)
```

    0.0775295663600526

```python
post3 = update(post2)
post3.Mean().evalf(subs=d1)
```

    0.0688926818860678

```python
post3.Mean().evalf(subs=d2)
```

    0.0724135699794844

```python

```


## 2D Linear operator with one parameter

Along the same lines as the previous example, we take a two-dimensional linear operator corresponding to a two-dimensional system with a single parameter. As previously explained, we set our problem up as follows:

It is easy to verify that $\mathcal{L}_\bar{x}^\phi$ is linear and continuous. A suitable solution for the operator is

\begin{align*}
u(\bar{x}) &= x_1 x_2 - x_2^2 \\
f(\bar{x}) &= \phi x_1 x_2 - \phi x_2^2 + x_2 - 2 \end{align*}

Now, we use the same Gaussian priors as before, except that they are now defined on a two-dimensional space.

\begin{align*}
u(\bar{x}) &\sim \mathcal{GP}(0, k_{uu}(\bar{x},\bar{x}',\theta)) \\
f(\bar{x}) &\sim \mathcal{GP}(0, k_{ff}(\bar{x},\bar{x}',\theta,\phi)) \\
y_u &= u(X) + \epsilon_u; \; \epsilon_u \sim \mathcal{N}(0, \sigma_u^2I) \\
y_f &= f(X) + \epsilon_f; \; \epsilon_f \sim \mathcal{N}(0, \sigma_f^2I) \end{align*}

The parameter estimation problem for the linear operator described above is to estimate $\phi$, given $\{x, y_u, y_f\}$. Notice that we evaluate $u, f$ on the same set of points because it makes sense from a physics point of view.

#### Simulate data

We use $\phi = 2$ and try to estimate it.

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
import time
```

```python
def get_simulated_data(n, phi):
    x = np.random.rand(n,2)
    y_u = np.multiply(x[:,0], x[:,1]) - x[:,1]**2
    y_f = phi*y_u + x[:,1] - 2
    return (x, y_u, y_f)
```

#### Evaluate kernels

The implementation of the kernels is straightforward as done earlier. Consequently, the corresponding code has been omitted from the report.

1. $k_{uu}(\bar{x}_i, \bar{x}_j; \theta) =  \theta e^{(-l_1 (x_{i,1}-x_{j,1})^2 -l_2 (x_{i,2}-x_{j,2})^2)}$
2. $k_{ff}(\bar{x}_i,\bar{x}_j;\theta,\phi) = \mathcal{L}_{\bar{x}_i}^\phi \mathcal{L}_{\bar{x}_j}^\phi k_{uu}(\bar{x}_i, \bar{x}_j; \theta) \\
= \mathcal{L}_{\bar{x}_i}^\phi \left( \phi k_{uu} + \frac{\partial}{\partial x_{j,1}}k_{uu} + \frac{\partial^2}{\partial^2 x_{j,2}}k_{uu}\right) \\
= \phi^2 k_{uu} + \phi \frac{\partial}{\partial x_{j,1}}k_{uu} + \phi \frac{\partial}{\partial x_{i,1}}k_{uu} \\
 \quad + \phi \frac{\partial^2}{\partial^2 x_{j,2}}k_{uu} + \frac{\partial}{\partial x_{i,1}}\frac{\partial}{\partial x_{j,1}}k_{uu} + \phi \frac{\partial^2}{\partial^2 x_{i,2}}k_{uu} \\
 \quad + \frac{\partial}{\partial x_{i,1}}\frac{\partial^2}{\partial^2 x_{j,2}}k_{uu} + \frac{\partial^2}{\partial^2 x_{i,2}}\frac{\partial}{\partial x_{j,1}}k_{uu} + \frac{\partial^2}{\partial^2 x_{i,2}}\frac{\partial^2}{\partial^2 x_{j,2}}k_{uu}$
3. $k_{fu}(\bar{x}_i,\bar{x}_j;\theta,\phi) = \mathcal{L}_{\bar{x}_i}^\phi k_{uu}(\bar{x}_i, \bar{x}_j; \theta) \\
= \phi k_{uu} + \frac{\partial}{\partial x_{i,1}}k_{uu} + \frac{\partial^2}{\partial x_{i,2}^2}k_{uu}$

```python
xi1, xj1, xi2, xj2, theta, l1, l2, phi = sp.symbols('xi1 xj1 xi2 xj2 theta l1 l2 phi')
# kuu_sym = theta*sp.exp(-((xi1 - xj1)**2)/(2*l1) - ((xi2 - xj2)**2)/(2*l2))
kuu_sym = theta*sp.exp(-l1*((xi1 - xj1)**2) - l2*((xi2 - xj2)**2))
kuu_fn = sp.lambdify((xi1, xj1, xi2, xj2, theta, l1, l2), kuu_sym, "numpy")
def kuu(x, theta, l1, l2):
    k = np.zeros((x.shape[0], x.shape[0]))
    for i in range(x.shape[0]):
        for j in range(x.shape[0]):
            k[i,j] = kuu_fn(x[i,0], x[j,0], x[i,1], x[j,1], theta, l1, l2)
    return k
```

```python
kff_sym = phi**2*kuu_sym \
            + phi*sp.diff(kuu_sym, xj1) \
            + phi*sp.diff(kuu_sym, xj2, xj2) \
            + phi*sp.diff(kuu_sym, xi1) \
            + sp.diff(kuu_sym, xj1, xi1) \
            + sp.diff(kuu_sym, xj2, xj2, xi1) \
            + phi*sp.diff(kuu_sym, xi2, xi2) \
            + sp.diff(kuu_sym, xj1, xi2, xi2) \
            + sp.diff(kuu_sym, xj2, xj2, xi2, xi2)
kff_fn = sp.lambdify((xi1, xj1, xi2, xj2, theta, l1, l2, phi), kff_sym, "numpy")
def kff(x, theta, l1, l2, phi):
    k = np.zeros((x.shape[0], x.shape[0]))
    for i in range(x.shape[0]):
        for j in range(x.shape[0]):
            k[i,j] = kff_fn(x[i,0], x[j,0], x[i,1], x[j,1], theta, l1, l2, phi)
    return k
```

```python
kfu_sym = phi*kuu_sym + sp.diff(kuu_sym, xi1) + sp.diff(kuu_sym, xi2, xi2)
kfu_fn = sp.lambdify((xi1, xj1, xi2, xj2, theta, l1, l2, phi), kfu_sym, "numpy")
def kfu(x, theta, l1, l2, phi):
    k = np.zeros((x.shape[0], x.shape[0]))
    for i in range(x.shape[0]):
        for j in range(x.shape[0]):
            k[i,j] = kfu_fn(x[i,0], x[j,0], x[i,1], x[j,1], theta, l1, l2, phi)
    return k
```

```python
def kuf(x, theta, l1, l2, phi):
    return kfu(x, theta, l1, l2, phi).T
```

#### NLML

$K = \begin{bmatrix}
k_{uu}(X_u, X_u; \theta) + \sigma_u^2I & k_{uf}(X_u, X_f; \theta, \phi) \\
k_{fu}(X_f, X_u; \theta, \phi) & k_{ff}(X_f, X_f; \theta, \phi) + \sigma_f^2I
\end{bmatrix}$

For simplicity, assume $\sigma_u = \sigma_f$.

$\mathcal{NLML} = \frac{1}{2} \left[ log|K| + y^TK^{-1}y + Nlog(2\pi) \right]$

where $y = \begin{bmatrix}
y_u \\
y_f
\end{bmatrix}$

```python
def nlml(params, x, y1, y2, s):
    params = np.exp(params)
    K = np.block([
        [
            kuu(x, params[0], params[1], params[2]) + s*np.identity(x.shape[0]),
            kuf(x, params[0], params[1], params[2], params[3])
        ],
        [
            kfu(x, params[0], params[1], params[2], params[3]),
            kff(x, params[0], params[1], params[2], params[3]) + s*np.identity(x.shape[0])
        ]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) \
               + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
(x, yu, yf) = get_simulated_data(10, 2)
nlml((1, 1, 1, 0.69), x, yu, yf, 1e-6)
```

    10.124377463652147

#### Optimize hyperparameters

```python
nlml_wp = lambda params: nlml(params, x, yu, yf, 1e-7)
m = minimize(nlml_wp, np.random.rand(4), method="Nelder-Mead")
```

```python
m
```

     final_simplex: (array([[14.07131834, -9.9997044 , -7.54723392,  0.69562933],
           [14.07131834, -9.9997044 , -7.54723392,  0.69562933],
           [14.07131834, -9.9997044 , -7.54723392,  0.69562933],
           [14.07131834, -9.9997044 , -7.54723392,  0.69562933],
           [14.07131834, -9.9997044 , -7.54723392,  0.69562933]]), array([-87.1620591, -87.1620591, -87.1620591, -87.1620591, -87.1620591]))
               fun: -87.16205910024283
           message: 'Optimization terminated successfully.'
              nfev: 663
               nit: 297
            status: 0
           success: True
                 x: array([14.07131834, -9.9997044 , -7.54723392,  0.69562933])

```python
np.exp(m.x)
```

    array([1.29150445e+06, 4.54133519e-05, 5.27567407e-04, 2.00497047e+00])

The optimized hyperparameters are:

|Parameter | Value |
|----------|-------|
| $\theta$ | 1.29150445e+06 |
| $l_1$    | 4.54133519e-05 |
| $l_2$    | 5.27567407e-04 |
| $\phi$   | 2.00497047 |

#### Analysis

To investigate error properties for this model, we run simulations from 5 to 25 datapoints with 5 samples of random data in each case which is 100 samples. The absolute error in the parameter estimate, $|\phi_{est} - \phi_{true}|$ is plotted in the left figure below. The execution time is plotted to the right; it includes the time taken to simulate the data and the time taken by the optimization routine.

```python
n_range = np.arange(5, 25)
plot_data = np.zeros((5, n_range.size, 4))
for j in range(plot_data.shape[0]):
    for i in range(plot_data.shape[1]):
        start_time = time.time()
        (x, y1, y2) = get_simulated_data(n_range[i], 2)
        nlml_wp = lambda params: nlml(params, x, y1, y2, 1e-7)
        m = minimize(nlml_wp, np.random.rand(4), method="Nelder-Mead")
        end_time = time.time()
        plot_data[j,i,:] = np.array([m.nfev, m.nit, np.exp(m.x[3]), end_time - start_time])
```

```python
from matplotlib.ticker import MaxNLocator

f, (ax1, ax2) = plt.subplots(ncols=2, nrows=1, figsize=(12,4))

for i in range(plot_data.shape[0]):
    ax1.plot(n_range[2:], abs(plot_data[i,2:,2] -2), "ob")
ax1.plot(n_range[2:], np.amin(abs(np.amin(plot_data, axis=2) - 2), axis=0)[2:])
ax1.xaxis.set_major_locator(MaxNLocator(integer=True))
ax1.set(xlabel = "Number of data points", ylabel = "Absolute error")
ax1.set_title("(A) Error in estimate of the parameter")
ax1.axhline(0.005, color='red', linestyle='--')

for i in range(plot_data.shape[0]):
    ax2.plot(n_range[2:], plot_data[i,2:,3], "ob")
ax2.set(xlabel = "Number of data points", ylabel = "Execution time")
ax2.set_title("(B) Execution time benchmark")
ax2.xaxis.set_major_locator(MaxNLocator(integer=True))
```

```python
plt.show()
```


# Methods for Identifying Out-of-Trend Data in Analysis of Stability Measurements–Part I: Regression Control Chart

# Acronym

* OOT : out-of-trend
* OOS : out-of-specification
* SL : Shewhart limits
* UPL : upper prediction limit
* LPL : lower prediction limit
* SI : Shewhart interval
* CI : confidence interval (信賴區間)
  * The confidence interval is the range in which a parameter (e.g., the expected value) occurs with a certain probability.
* PI : prediction interval (預測區間)
  * The prediction interval is the range in which a future observation is found with some specified probability.
* TI : tolerance interval (容忍區間)
  * The tolerance range contains a specified part of future observations (content) at a certain probability (confidence).

# **Control Chart**

* Control charts can be separated into two phases:
  **Phase I** : parameters of the distribution (typically mean and standard deviation) of the reference data are estimated.
  **Phase II** : certain statistics (e.g., sample mean, range, etc.) of the observed sample are plotted and examined.
  * The distribution parameters obtained in **Phase I** are used to construct lower and upper control limits (i.e., lower and upper quantiles) for the statistic (i.e., the individual value) and added to the chart.

* The interval between the control limits of the individuals control chart contains the allowable value of the new data of the observed process with a certain probability (typically, this probability is 0.9973), when the process is in control. This approach will be referred to as the **Shewhart method** in this paper.
    * For the authors’ study, the Shewhart method is valid only if the sample size in the reference set of **Phase I** is large enough, then the 0.9973 probability for the $\pm 3\sigma$ range is reached as a limit.

* The typical application requires at least 25 samples in the reference set.
  * In the current situation, the Shewhart method is not considered reliable because the reference data set is **small**.

* The observed sample contains only a single datapoint, thus the statistic to be plotted is the individual value (individuals control chart).
  * In the case where the sample size is small, one should consider the uncertainty of expected value and variance.

* The width of the valid control limits in the authors’ typical case (**small reference set**) are $$\pm ts\sqrt{1+\frac{1}{n}} \text{ ,}$$ which is the prediction interval.

* If a point is found to be OOT, one may choose from two routes, depending on whether data analysis is **in real time or not**:
  * in real time:
    If the non-OOT nature is justified (by remeasurement), the stability study is continued, neglecting the point. If the OOT nature is justified, a corrective action follows the detection, and thus, the original process is ended.
  * not in real time:
    The analysis of data is performed not in real time but upon collecting more data. The original process is continued, and the point that is later found to be **OOT cannot be re-measured and should be removed from the data set**. If a point is found to be non-OOT, the point is added to the historical data set, new control limits are calculated, and a new control chart is made for the next observed data.

# Assumptions in this paper

* The 95% confidence level.
* The reference data are non-OOT.
* For the authors’ data, the hypothesis of homogeneity of variances (變異數同質性) is accepted.

# **Regression Control Chart**

* In the regression control chart method, data are compared within a batch.

* Three kinds of statistical intervals are used: confidence, prediction, and tolerance.

* The “parameters” of the chart are a **regression line** (i.e., an expected value changing with time) constructed by the previous data from **Phase I** and the **variance of residuals**, and both are considered as known in the Shewhart method.

* Suggest fitting a regression line to historical data of the observed batch and extrapolating the line to the time point of the new data.
  **Control limits** are calculated for the expected value by $\text{expected value} ± ks$ (called the Shewhart limits (SL)) at the new time point.
  * The $k$ is taken from a table of normal distribution at a desired significance level.
  * The residual standard error ($s$) is calculated either by only the regression line of the observed batch, or by regression lines of historical batches. In the latter case, a common slope is assumed, and different intercepts are allowed for the historical batches.

* The prediction interval with $t$-distribution is to be used if one asks about a single new observation.

* The method requires points in the observed batch as reference data (assume that these reference data are non-OOT) to construct the first regression line.
Question : How many points from the beginning of the observed batch should be considered as the reference data set ?

* The prediction interval

  $$\hat{Y}-t_{\alpha/2}s_{y^*-\hat{Y}}<y^*<\hat{Y}+t_{\alpha/2}s_{y^*-\hat{Y}}  \quad\quad \text{(Eq. 1)}$$

  where $y^*$ is the new measured value at the new time point $x^*$,
      $\hat{Y}$ is the predicted value of the measured variable $y$ at the new time point $x^*$,
      $t_{\alpha/2}$ is the onesided upper critical $t$ value at $α/2$ one-sided level (with $n-2$ degrees of freedom),
      $n$ is the number of points used to construct the regression line, and
      $s_{y^*-\hat{Y}}$ is the sample standard deviation calculated by

  $$s_{y^*-\hat{Y}}=s_r\sqrt{1+\frac{1}{n}+\frac{(x^*-\bar{x})^2}{\sum_{j=1}^{k}(x_j-\bar{x})^2}} \quad\quad \text{(Eq. 3)}$$
          where $s_r$ is the [residual standard deviation](https://www.investopedia.com/terms/r/residual-standard-deviation.asp) and
             $\bar{x}$ is the mean of the $x_j$ values (time points) of data
(without $x^*$) used to estimate the regression line.

The right side of (Eq. 1) is the upper prediction limit (UPL), while the left side is the lower prediction limit (LPL). **If the inequality (Eq. 1) is satisfied, that is the $y^*$ is within the limits, the data is accepted, otherwise it is OOT.**

  * The residual standard error may be calculated using earlier batches as well, not only from the recent one in study. The use of earlier batches may follow several different routes. Some authors used parallel lines (i.e., the common slope for all historical batches). **The authors of this article are not convinced about the usefulness of this assumption, however.**

**If common slope is forced for historical batches, the fit is deteriorated(惡化的). This situation would lead to greater residual error than would be achieved using the best linear fits (letting different slopes).**
On the other hand, **if separate lines are fitted to historical batches, the residual standard errors obtained may be pooled.** The pooled error has higher degrees of freedom than that obtained from a single batch, thus, it is a better estimate to the variance, leading to better estimated control limits.

  * If every group has the **same number of data**, the pooled sample variance $s_{p(res)}^{2}$ is the arithmetic average of the individual sample variances. Hence, regression lines were fit to each historical batch and residual standard errors ($s_i$) for the $i$-th historical batch were calculated. The pooling is justified only if the variation of data of different batches has the same variance. For the authors’ data,
the hypothesis of homogeneity of variances (變異數同質性) is accepted.

  * For calculation of the prediction interval with pooled residual
standard error, in Equation (Eq. 3), is substituted for $s_r$. The degrees
of freedom of the pooled sample variance $s_{p(res)}^{2}$ is $p(n-2)$, thus a different $t_{α/2}$ should be taken from $t$-table when calculating the control limits.

* Shewhart limits:
  $$\text{SL} = \hat{y^*} \pm z_{\alpha/2}\sigma_{r} \quad\quad \text{(Eq. 5)}$$

  where $z_{\alpha/2}$ is the critical value of standard normal distribution at two-sided $\alpha/2$ level, and
      $\sigma_{r}$ is the square root of variance of residuals, equal to $s_{r}$.
  Note : One may use the square root of the pooled standard deviation as a substitute of the $\sigma_{r}$.

* Confidence limits:
  $$\text{CL} = \hat{y^*} \pm t_{\alpha/2} s_{r}/\sqrt{n} \quad\quad \text{(Eq. 6)}$$

  where $t_{\alpha/2}$ is the onesided upper critical $t$ value at $α/2$ one-sided level (with $n-2$ degrees of freedom).
    * For the calculations with pooled standard deviation, $s_{r}$ is substituted with $s_{p(res)}$ in (Eq. 6) and degrees of freedom of $s_{p(res)}^{2}$ is used to obtain the $t$-score.

* Tolerance limits:
  $$\text{TL} = \hat{y^*} \pm k_1 s_{r} \quad\quad \text{(Eq. 7)}$$

  where
  $$k_1 =  \sqrt{\frac{\nu \chi_{P,1}^{2}(\frac{1}{n^{'}})}{\chi_{1-\gamma, \nu}^{2}}} \quad\quad \text{(Eq. 8)}$$

  where $\nu$ is the degrees of freedom of $s_{r}^{2}$ ($n-2$ actually),
      $\chi_{1-\gamma, \nu}^{2}$ is the critical value of the chi-square distribution at a one-sided $1-\gamma$ level with $\nu$ degrees of freedom,
      $\chi_{P,1}^{2}(\delta)$ is the critical value of non-central-chi-square distribution at one-sided $P$ level with $\nu$ degrees of freedom, and
      $\delta$ is the argument in the noncentral-chi-square function, actually, $\delta = 1/n^{'}$ with

  \begin{equation}
  \begin{split}
    n^{'} &= \frac{n\sum(x_i-\bar{x})^2}{\sum(x_i-\bar{x})^2+n(x^{*}-\bar{x})^2} \quad\quad \text{(Eq. 9)} \\ \\ & = \frac{s_{r}^2}{s_{\hat{Y}}^{2}} \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \text{(Eq. 10)}
     \\ \\ & = \frac{s_{r}^2}{s_{r}\sqrt{\frac{1}{n}+\frac{(x^*-\bar{x})^2}{\sum_{j=1}^{k} (x_j-\bar{x})^2}}} \quad\quad\quad\quad \text{(Eq. 11)}
  \end{split}
  \end{equation}

    * For the calculations using pooled standard deviation, $s_{r}$ is substituted with $s_{p(res)}$ in (Eq. 10) and (Eq. 11) whenever a new effective number ($n^{'}$) of observations is to be obtained. The proper substitution is performed for degrees of freedom as well in (Eq. 8).

# References

  [1] [Pooled Variance](https://www.statisticshowto.com/pooled-variance/), Stephanie, February 25, 2020.
  [2] [10.5: Standard Error and Pooled Variance](https://stats.libretexts.org/Bookshelves/Applied_Statistics/Book%3A_An_Introduction_to_Psychological_Statistics_(Foster_et_al.)/10%3A__Independent_Samples/10.05%3A_Standard_Error_and_Pooled_Variance), Garett C. Foster, May 2, 2021.
  [3] [What is a pooled variance?](https://blogs.sas.com/content/iml/2020/06/29/pooled-variance.html), Rick Wicklin, June 29, 2020.


```python
%matplotlib inline

from __future__ import print_function, division

import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as pl
import sys
from scipy.misc import logsumexp
```

```python
def logtrapz(lnf, dx):
    return np.log(dx/2.) + logsumexp([logsumexp(lnf[:-1]), logsumexp(lnf[1:])])
```

```python
from scipy.special import erf

ndp = 10

#hs = [0.001, 0.01, 0.1, 1., 10., 20., 50.]
h = 10.

sigmas = [0.1, 1., 5., 10.]

C = 100.
hvals = np.linspace(0., C, 5000)

kldivs = []
#for h in hs:
for sigma in sigmas:
    #sigma = 1.
    d = h + sigma*np.random.randn(10)

    loglike = np.zeros((len(hvals),))
    for dval in d:
        loglike = loglike - (0.5*(dval-hvals)**2/sigma**2) - 0.5*np.log(2.*np.pi*sigma**2)

    logprior = -np.log(C)
    logevd = logtrapz(loglike+logprior, hvals[1]-hvals[0])
    logpost = loglike+logprior-logevd

    #ev = 0.5*(erf(0.5*np.sqrt(2.)*d/sigma) + erf(0.5*np.sqrt(2.)*(C-d)/sigma))

    kldivergence = np.sum(np.exp(logpost)*(logpost-logprior))
    kldivs.append(kldivergence)

#print(logevd, file=sys.stdout)
#print(kldivergence, file=sys.stdout)
```

```python
#pl.semilogx(hs, kldivs)
pl.plot(sigmas, kldivs)
```

```python
from sympy import *
```

```python
x, y, z = symbols('x y z')
integrate(exp(-(x-y)**2/(2*z**2))/sqrt(2*pi*z**2), (y, 0, 100))
```

    z*erf(sqrt(2)*x/(2*z))/(2*sqrt(z**2)) + z*erf(sqrt(2)*(-x + 100)/(2*z))/(2*sqrt(z**2))

```python

```


# Using Pòlya-Gamma Auxiliary Variables for Binary Classification

## Overview

In this notebook, we'll demonstrate how to use Pòlya-Gamma auxiliary variables to do efficient inference for Gaussian Process binary classification as in reference [1].
We will also use natural gradient descent, as described in more detail in the [Natural gradient descent](./Natural_Gradient_Descent.ipynb) tutorial.

[1] Florian Wenzel, Theo Galy-Fajou, Christan Donner, Marius Kloft, Manfred Opper. [Efficient Gaussian process classification using Pòlya-Gamma data augmentation](https://arxiv.org/abs/1802.06383). Proceedings of the AAAI Conference on Artificial Intelligence. 2019.

## Pòlya-Gamma Augmentation

When a Gaussian Process prior is paired with a Gaussian likelihood inference can be done exactly with a simple closed form expression.
Unfortunately this attractive feature does not carry over to non-conjugate likelihoods like the Bernoulli likelihood that arises in the context of binary classification with a logistic link function.
Sampling-based stochastic variational inference offers a general strategy for dealing with non-conjugate likelihoods; see the [corresponding tutorial](./Non_Gaussian_Likelihoods.ipynb).

Another possible strategy is to introduce additional latent variables that restore conjugacy.
This is the strategy we follow here.
In particular we are going to introduce a Pòlya-Gamma auxiliary variable for each data point in our training dataset.
The [Polya-Gamma](https://arxiv.org/abs/1205.0310) distribution $\rm{PG}$ is a univariate distribution with support on the positive real line.
In our context it is interesting because if $\omega_i$ is distributed according to $\rm{PG}(1,0)$ then the logistic likelihood $\sigma(\cdot)$ for data point $(x_i, y_i)$ can be represented as

\begin{align}
\sigma(y_i f_i) = \frac{1}{1 + \exp(-y_i f_i)} = \tfrac{1}{2} \mathbb{E}_{\omega_i \sim \rm{PG}(1,0)} \left[ \exp \left(\tfrac{1}{2} y_i f_i - \tfrac{\omega_i}{2} f_i^2 \right) \right]
\end{align}

where $y_i \in \{-1, 1\}$ is the binary label of data point $i$
and $f_i$ is the Gaussian Process prior evaluated at input $x_i$.
The crucial point here is that $f_i$ appears quadratically in the exponential within the expectation.
In other words, conditioned on $\omega_i$, we can integrate out $f_i$ exactly, just as if we were doing regression with a Gaussian likelihood. For more details please see the original reference.

## Setup

```python
import tqdm
import math
import torch
import gpytorch
from matplotlib import pyplot as plt

# Make plots inline
%matplotlib inline
```

For this example notebook, we'll create a simple artificial dataset.

```python
import os
from math import floor

# this is for running the notebook in our testing framework
smoke_test = ('CI' in os.environ)

N = 100
X = torch.linspace(-1., 1., N)
probs = (torch.sin(X * math.pi).add(1.).div(2.))
y = torch.distributions.Bernoulli(probs=probs).sample()
X = X.unsqueeze(-1)

train_n = int(floor(0.8 * N))
indices = torch.randperm(N)
train_x = X[indices[:train_n]].contiguous()
train_y = y[indices[:train_n]].contiguous()

test_x = X[indices[train_n:]].contiguous()
test_y = y[indices[train_n:]].contiguous()

if torch.cuda.is_available():
    train_x, train_y, test_x, test_y = train_x.cuda(), train_y.cuda(), test_x.cuda(), test_y.cuda()
```

Let's plot our artificial dataset.
Note that here the binary labels are 0/1-valued; we will need to be careful to translate between this representation and the -1/1 representation that is most natural in the context of Pòlya-Gamma augementation.

```python
plt.plot(train_x.squeeze(-1), train_y, 'o')
```

The following steps create the dataloader objects. See the [SVGP regression notebook](./SVGP_Regression_CUDA.ipynb) for details.

```python
from torch.utils.data import TensorDataset, DataLoader

train_dataset = TensorDataset(train_x, train_y)
train_loader = DataLoader(train_dataset, batch_size=100000, shuffle=False)

test_dataset = TensorDataset(test_x, test_y)
test_loader = DataLoader(test_dataset, batch_size=1024, shuffle=False)
```

## Variational Inference with PG Auxiliaries

We define a Bernoulli likelihood that leverages Pòlya-Gamma augmentation.
It turns out that we can derive closed form updates for the Pòlya-Gamma auxiliary variables. To deal with the Gaussian Process we introduce inducing points and inducing locations.
In particular we will need to learn a variational covariance matrix and a variational mean vector that control the inducing points. (See the discussion in the [SVGP tutorial](Approximate_GP_Objective_Functions.ipynb) for more details.)
We will use natural gradient updates to deal with these two variational parameters; this will allow us to take large steps, thus yielding fast convergence.

```python
class PGLikelihood(gpytorch.likelihoods._OneDimensionalLikelihood):
    # this method effectively computes the expected log likelihood
    # contribution to Eqn (10) in Reference [1].
    def expected_log_prob(self, target, input, *args, **kwargs):
        mean, variance = input.mean, input.variance
        # Compute the expectation E[f_i^2]
        raw_second_moment = variance + mean.pow(2)

        # Translate targets to be -1, 1
        target = target.to(mean.dtype).mul(2.).sub(1.)

        # We detach the following variable since we do not want
        # to differentiate through the closed-form PG update.
        c = raw_second_moment.detach().sqrt()
        # Compute mean of PG auxiliary variable omega: 0.5 * Expectation[omega]
        # See Eqn (11) and Appendix A2 and A3 in Reference [1] for details.
        half_omega = 0.25 * torch.tanh(0.5 * c) / c

        # Expected log likelihood
        res = 0.5 * target * mean - half_omega * raw_second_moment
        # Sum over data points in mini-batch
        res = res.sum(dim=-1)

        return res

    # define the likelihood
    def forward(self, function_samples):
        return torch.distributions.Bernoulli(logits=function_samples)

    # define the marginal likelihood using Gauss Hermite quadrature
    def marginal(self, function_dist):
        prob_lambda = lambda function_samples: self.forward(function_samples).probs
        probs = self.quadrature(prob_lambda, function_dist)
        return torch.distributions.Bernoulli(probs=probs)

# define the actual GP model (kernels, inducing points, etc.)
class GPModel(gpytorch.models.ApproximateGP):
    def __init__(self, inducing_points):
        variational_distribution = gpytorch.variational.NaturalVariationalDistribution(inducing_points.size(0))
        variational_strategy = gpytorch.variational.VariationalStrategy(
            self, inducing_points, variational_distribution, learn_inducing_locations=True
        )
        super(GPModel, self).__init__(variational_strategy)
        self.mean_module = gpytorch.means.ZeroMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# we initialize our model with M = 30 inducing points
M = 30
inducing_points = torch.linspace(-2., 2., M, dtype=train_x.dtype, device=train_x.device).unsqueeze(-1)
model = GPModel(inducing_points=inducing_points)
model.covar_module.base_kernel.initialize(lengthscale=0.2)
likelihood = PGLikelihood()

if torch.cuda.is_available():
    model = model.cuda()
    likelihood = likelihood.cuda()
```

### Setup optimizers

We will use a `NGD` (Natural Gradient Descent) optimizer to deal with the inducing point covariance matrix and corresponding mean vector, while we will use the `Adam` optimizer for all other parameters (the kernel hyperparmaeters as well as the inducing point locations).
Note that we use a pretty large learning rate for the `NGD` optimizer.

```python
variational_ngd_optimizer = gpytorch.optim.NGD(model.variational_parameters(), num_data=train_y.size(0), lr=0.1)

hyperparameter_optimizer = torch.optim.Adam([
    {'params': model.hyperparameters()},
    {'params': likelihood.parameters()},
], lr=0.01)
```

### Define training loop

```python
model.train()
likelihood.train()
mll = gpytorch.mlls.VariationalELBO(likelihood, model, num_data=train_y.size(0))

num_epochs = 1 if smoke_test else 100
epochs_iter = tqdm.notebook.tqdm(range(num_epochs), desc="Epoch")
for i in epochs_iter:
    minibatch_iter = tqdm.notebook.tqdm(train_loader, desc="Minibatch", leave=False)

    for x_batch, y_batch in minibatch_iter:
        ### Perform NGD step to optimize variational parameters
        variational_ngd_optimizer.zero_grad()
        hyperparameter_optimizer.zero_grad()

        output = model(x_batch)
        loss = -mll(output, y_batch)
        minibatch_iter.set_postfix(loss=loss.item())
        loss.backward()
        variational_ngd_optimizer.step()
        hyperparameter_optimizer.step()
```

    HBox(children=(FloatProgress(value=0.0, description='Epoch', style=ProgressStyle(description_width='initial'))…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

    HBox(children=(FloatProgress(value=0.0, description='Minibatch', max=1.0, style=ProgressStyle(description_widt…

### Visualization and Evaluation

```python
# push training data points through model
train_mean_f = model(train_x).loc.data.cpu()
# plot training data with y being -1/1 valued
plt.plot(train_x.squeeze(-1), train_y.mul(2.).sub(1.), 'o')
# plot mean gaussian process posterior mean evaluated at training data
plt.plot(train_x.squeeze(-1).cpu(), train_mean_f.cpu(), 'x')
```

As expected the Gaussian Process posterior mean (plotted in orange) gives confident predictions in the regions
where the correct label is unambiguous (e.g. for x ~ 0.5) and gives unconfident predictions in regions where
the correct label is ambiguous (e.g. x ~ 0.0).

We compute the negative log likelihood (NLL) and classification accuracy on the held-out test data.

```python
model.eval()
likelihood.eval()
with torch.no_grad():
    nlls = -likelihood.log_marginal(test_y, model(test_x))
    acc = (likelihood(model(test_x)).probs.gt(0.5) == test_y.bool()).float().mean()
print('Test NLL: {:.4f}'.format(nlls.mean()))
print('Test Acc: {:.4f}'.format(acc.mean()))
```

    Test NLL: 0.3481
    Test Acc: 0.9000


```python
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

    Bad key "text.kerning_factor" on line 4 in
    C:\Users\sensio\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    https://github.com/matplotlib/matplotlib/blob/v3.1.3/matplotlibrc.template
    or from the matplotlib source distribution

# Basic use

We want to solve the following PDE:

\begin{equation}
    \frac{\partial \phi}{\partial t} + u \frac{\partial \phi}{\partial x} = 0
\end{equation}

The independent variables (i.e, $x$ and $t$) are used as input values for the NN, and the solution (i.e. $\phi$) is the output. In order to find the solution, at each step the NN outputs are derived w.r.t the inputs. Then, a loss function that matches the PDE is built and the weights are updated accordingly. If the loss function goes to zero, we can assume that our NN is indeed the solution to our PDE. We will try to find a general solution for different values of $u$, so it will be set also as an input.

```python
# imports
import numpy as np
import matplotlib.pyplot as plt
import nangs
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

nangs.__version__, torch.__version__
```

    ('0.1.2', '1.5.0')

## Define your PDE

We provide the class `PDE` with the default functionality to solve the PDE. First, create a new class inheriting from the `PDE` class and overload the `computePDELoss` function to return one loss function for each PDE in your system. You must initialize this class with your inputs and outputs names.

```python
from nangs import PDE

class Adv1d(PDE):
    def computePDELoss(self, inputs, outputs):

        # compute gradients
        grads = self.computeGrads(outputs, inputs)

        # compute loss
        dpdx, dpdt = grads[:, 0], grads[:, 1]
        u = inputs[:,2]
        return {'pde': dpdt + u*dpdx}

pde = Adv1d(inputs=('x', 't', 'u'), outputs='p')

pde.inputs, pde.outputs
```

    (('x', 't', 'u'), ('p',))

## Define your data

To solve the PDE we first need a set of points to evaluate it, we will use this points as the dataset for training de NN.

```python
# define the mesh

x = np.linspace(0,1,20)
t = np.linspace(0,1,30)
u = np.linspace(-1,1,10)
```

The class `Mesh` will combine all the posible values on the previous arrays to build a mesh. You must pass the data as a dict, providing one (unique) name for each variable matching the names of the inputs defined in the PDE.

```python
from nangs import Mesh

mesh = Mesh({'x': x, 't': t, 'u': u})
mesh.vars
```

    ('x', 't', 'u')

In order to train a NN we need a `Dataset`. A `Mesh` has in fact a `Dataset` that you can iterate over.

```python
len(mesh.dataset)
```

    6000

```python
mesh.dataset[:3]
```

    tensor([[ 0.0000,  0.0000, -1.0000],
            [ 0.0000,  0.0000, -0.7778],
            [ 0.0000,  0.0000, -0.5556]])

You can also specify the device to cache the data

```python
mesh = Mesh({'x': x, 't': t, 'u': u}, device=device)
mesh.dataset[:3]
```

    tensor([[ 0.0000,  0.0000, -1.0000],
            [ 0.0000,  0.0000, -0.7778],
            [ 0.0000,  0.0000, -0.5556]], device='cuda:0')

Finally, add the mesh to your PDE.

```python
pde.set_mesh(mesh)
```

```python
mesh.vars
```

    ('x', 't', 'u')

```python
pde.inputs
```

    ('x', 't', 'u')

## Boundary Conditions

In order to find a non-trivial solution to our PDE we need to specify a set of boundary conditions. Nangs provide different classes for common boundary conditions.

```python
# initial condition (t = 0)

t0 = np.array([0])
_x, _t, _u = np.meshgrid(x, t0, u)
p0 = np.sin(2*np.pi*(_x- _u*_t))
p0.shape
```

    (1, 20, 10)

```python
plt.plot(x, p0[0,:,0])
plt.grid(True)
plt.xlabel('x')
plt.ylabel('$p_0$')
plt.show()
```

A boundary condition has its own `Mesh` and logic to compute values and gradients. In a `Dirichlet` boundary conditions we are fixing some values, so we need the inputs and also the target outputs (in this case, our initial condition). If you are including some free-parameters as NN inputs, make sure you specify an initial condition for each one of them (here we use the same for every value of $u$).

```python
from nangs import Dirichlet

initial_condition = Dirichlet({'x': x, 't': t0, 'u': u}, {'p': p0.reshape(-1)}, device=device, name="initial")
initial_condition.vars
```

    [('x', 't', 'u'), ('p',)]

```python
initial_condition.dataset[:5]
```

    (tensor([[ 0.0000,  0.0000, -1.0000],
             [ 0.0000,  0.0000, -0.7778],
             [ 0.0000,  0.0000, -0.5556],
             [ 0.0000,  0.0000, -0.3333],
             [ 0.0000,  0.0000, -0.1111]], device='cuda:0'),
     tensor([[0.],
             [0.],
             [0.],
             [0.],
             [0.]], device='cuda:0'))

```python
pde.add_boco(initial_condition)
```

We use a periodic boundary condition at $x=0$ and $x=1$. During training we will enforce this values to match.

```python
# boundary conditions (peridic conditions at x = 0 and x = 1)

xb0 = np.array([0])
mb0 = np.meshgrid(xb0, t)
mb0 = np.stack(mb0, -1).reshape(-1, 2)
mb0.shape
```

    (30, 2)

```python
from nangs import Periodic

x1 = np.array([0.])
x2 = np.array([1.])
periodic = Periodic({'x': x1, 't': t, 'u': u}, {'x': x2, 't': t, 'u': u}, device=device)
periodic.vars
```

    [('x', 't', 'u'), ('x', 't', 'u')]

```python
periodic.dataset[:5]
```

    (tensor([[ 0.0000,  0.0000, -1.0000],
             [ 0.0000,  0.0000, -0.7778],
             [ 0.0000,  0.0000, -0.5556],
             [ 0.0000,  0.0000, -0.3333],
             [ 0.0000,  0.0000, -0.1111]], device='cuda:0'),
     tensor([[ 1.0000,  0.0000, -1.0000],
             [ 1.0000,  0.0000, -0.7778],
             [ 1.0000,  0.0000, -0.5556],
             [ 1.0000,  0.0000, -0.3333],
             [ 1.0000,  0.0000, -0.1111]], device='cuda:0'))

```python
pde.add_boco(periodic)
```

## Define your solution

We provide a basic `MLP` class to impement a Multilayer Perceptron as solution approxiamtion to the PDE, but you can always define you own NN (just be sure to match the number of inputs and outputs).

```python
from nangs import MLP

mlp = MLP(inputs=len(pde.inputs), outputs=len(pde.outputs), layers=3, neurons=100)

mlp
```

    MLP(
      (mlp): Sequential(
        (0): Linear(in_features=3, out_features=100, bias=True)
        (1): Sequential(
          (0): Sine()
          (1): Linear(in_features=100, out_features=100, bias=True)
        )
        (2): Sequential(
          (0): Sine()
          (1): Linear(in_features=100, out_features=100, bias=True)
        )
        (3): Sequential(
          (0): Sine()
          (1): Linear(in_features=100, out_features=1, bias=True)
        )
      )
    )

```python
assert mlp(torch.randn(32, len(pde.inputs))).shape == torch.randn(32, len(pde.outputs)).shape
```

## Solve the PDE

To solve the PDE, you must first call de `compile` function to specify the NN, optimizer and optionally the loss function (we use MSE by default) or a learning rate scheduler.

```python
EPOCHS = 40

optimizer = torch.optim.Adam(mlp.parameters())
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, pct_start=0.1, total_steps=EPOCHS)

pde.compile(mlp.to(device), optimizer, scheduler)
```

To solve the PDE, call the `solve` function with a number of epochs and batch size for the optimization process. It returns a hitory with all the tracked metrics.

```python
BATCH_SIZE = 256

%time hist = pde.solve(EPOCHS, BATCH_SIZE)
```

## Evaluate your solution

Finally, you can evaluate your solution

```python
from matplotlib import animation, rc
rc('animation', html='html5')

def update_plot(i, x, u, t, p):
    ax.clear()
    pe = np.sin(2.*np.pi*(x-u*t[i]))
    ax.plot(x, pe, label=f"exact (u = {U})")
    ax.plot(x, p[i], '.k', label="solution")
    ax.set_xlabel("x", fontsize=14)
    ax.set_ylabel("p", fontsize=14, rotation=np.pi/2)
    ax.legend(loc="upper right")
    ax.grid(True)
    ax.set_xlim([0, 1])
    ax.set_ylim([-1.2, 1.2])
    l2 = np.sqrt(np.sum((p[i]-pe)**2))
    ax.set_title(f"t = {t[i]:.3f} (L2 = {l2:.5f})")
    return ax
```

```python
#t = np.linspace(0,1,10)
U = -0.5

x = np.linspace(0,1,30)
t = np.linspace(0,1,20)
u = np.array([U])
eval_mesh = Mesh({'x': x, 't': t, 'u': u}, device=device)

p = pde.eval(eval_mesh)
p = p.cpu().numpy().reshape(len(t), -1)

fig = plt.figure()
ax = plt.subplot(111)
anim = animation.FuncAnimation(fig, update_plot, frames=len(t), fargs=(x, U, t, p), interval=300)
plt.close()
```

```python
anim
```


# 径向基函数网络/RBF网络

## 简介

RBF网络时具有隐藏层的前向网络，隐藏层以RBF作为激活函数。不同于一般的BP网络，RBF网络具有局部逼近的特性。并且Poggio和Girosi已经证明RBF网络是连续函数的最佳逼近。

BP网络和RBF网络的区别见下图

## 符号定义
|符号|含义|
|:-:|:-:|
|$\bm{x}$|输入向量|
|$\bm{c}$|中心坐标向量，每一个RBF函数均有自己的中心坐标向量|
|$\phi$|高斯核函数|
|$\varphi$|RBF函数|
|$\sigma$|标准化参数|
|$\bm{\hat{y}}$|模型输出|
|$\bm{y}$|真实输出|
|$\bm{W}$|隐藏层到输出层权重矩阵|
|$n$|输入维度|
|$k$|隐藏层神经元数量|
|$o$|输出维度|
|$\eta$|学习率|

## RBF网络的正向计算

### 定义RBF函数

$$
\begin{equation}
    \phi(\bm{x}, \bm{c}) = \phi(||\bm{x}-\bm{c}||)
\end{equation}
$$
上式中$c$为中心坐标，$\phi$为高斯核函数

* 以高斯核函数和欧氏距离定义的RBF函数
$$
\begin{equation}
    \varphi(\bm{x}) = exp(-\frac{1}{2}\sum_{i=1}^{n}\frac{(x_i-c_i)^2}{\sigma_i^2})
\end{equation}
$$

上式中，输入维度为$n$，$c_i$和$\sigma_i$分别为第$i$个维度的中心坐标和标准化参数

### 从输入层到隐藏层

假设隐藏层有k个神经元，则对于每一个神经元有
$$
\begin{equation}
    \varphi_i(\bm{x}) = exp(-\frac{1}{2}\sum_{j=1}^{n}\frac{(x_j-c_{ij})^2}{\sigma_{ij}^2}), \ i = 1, 2, 3, \cdots, k
\end{equation}
$$

若第i个神经元的标准化参数均相同，则可以得到
$$
\begin{equation}
    \begin{split}
    \varphi_i(\bm{x})
    &= exp(-\frac{1}{2\sigma_{i}^2}\sum_{j=1}^{n}(x_j-c_{ij})^2), \\
    &= exp(-\frac{1}{2\sigma_{i}^2}(\bm{x}-\bm{c_i})^T(\bm{x}-\bm{c_i})) \\
    &= exp(-\beta_i(\bm{x}-\bm{c_i})^T(\bm{x}-\bm{c_i}))
    \end{split}
    \ , i = 1, 2, 3, \cdots, k
\end{equation}
$$

$$
\begin{equation}
    \beta_i = \frac{1}{2\sigma_{i}^2}\ , i = 1, 2, 3, \cdots, k
\end{equation}
$$

### 从隐藏层到输出层
假设输出层有o个输出单元，则对于输出层有
$$
\begin{equation}
    \begin{split}
        \bm{y}
        &= \bm{W}\bm{\varphi} \\
        &=
        \left[
            \begin{array}{cc}
                \bm{w_1} \\
                \bm{w_2} \\
                \vdots \\
                \bm{w_o} \\
            \end{array}
        \right]
        \left[
            \begin{array}{cc}
                \varphi_1 \\
                \varphi_2 \\
                \vdots \\
                \varphi_k
            \end{array}
        \right]
    \end{split}
\end{equation}
$$

## RBF网络的学习算法

RBF网络待学习参数如下：
1. 高斯RBF函数的中心坐标$\bm{c}$和参数$\beta$
2. 隐藏层到输出层的权重$\bm{W}$

### 自组织选取中心学习算法
1. 无监督学习过程：求解高斯RBF函数相关参数
2. 有监督学习过程：求解隐藏层到输出层的权重

对于第一步，可以使用聚类相关方法来进行。即对于k个RBF单元，通过对输入样本聚类为k类来获得相应的中心坐标以及散度参数。

在完成第一步后，第二步为确定隐藏层到输出层的权重。这一步为线性映射，因此可以使用各种线性优化算法来求得最优解，例如最小二乘法等。

### 监督学习
对于RBF网络，其可以使用类似于一般BP算法中的梯度下降法对网络参数进行求解。在本文件中主要是实现这一优化方法。

## RBF网络的监督学习

* 定义损失
$$
\begin{equation}
    \mathcal{L(\bm{c}, \bm{\beta}, \bm{W})} = \frac{1}{M}\sum_{i=1}^M||\bm{\bar{y}}_i - \bm{y}_i||^2_2
\end{equation}
$$
其中$\bm{\hat{y}}_i$为第$i$个样本的输出，$\bm{y}_i$为第$i$个样本的标签，$M$为训练集的样本总数

* 定义优化函数
$$
\begin{equation}
    \min\limits_{\bm{c}, \bm{\beta}, \bm{W}}{\mathcal{L}(\bm{c}, \bm{\beta}, \bm{W})}=\frac{1}{M}\sum_{i=1}^M||\bm{\hat{y}}_i - \bm{y}_i||^2_2
\end{equation}
$$
* 误差反向传播(单样本)
    * $\bm{W}$
    $$
    \begin{equation}
        \begin{split}
            \frac{\partial \mathcal{L}}{\partial \bm{W}}
            &= \frac{\partial \mathcal{L}}{\partial \bm{\hat{y}}} \frac{\partial \bm{\hat{y}}}{\partial \bm{W}} \\
            &= 2(\bm{\hat{y}}-\bm{y})\bm{\varphi}^T
        \end{split}
    \end{equation}
    $$
    * $\bm{\varphi}$
    $$
    \begin{equation}
        \begin{split}
            \frac{\partial \mathcal{L}}{\partial \bm{\varphi}}
            &= \frac{\partial \mathcal{L}}{\partial \bm{\hat{y}}} \frac{\partial \bm{\hat{y}}}{\partial \bm{\varphi}} \\
            &= 2(\bm{\hat{y}}-\bm{y})^T \bm{W}
        \end{split}
    \end{equation}
    $$
    * $\bm{c}$
    对于每一个隐藏单元的中心参数$\bm{c_i}$有
    $$
    \begin{equation}
        \begin{split}
            \frac{\partial \mathcal{L}}{\partial \bm{c_i}}
            &= \frac{\partial \mathcal{L}}{\partial \varphi_i} \frac{\partial \varphi_i}{\partial \bm{c_i}} \\
            &= (\sum_{j=1}^o2(\hat{y}_j-y_j)\bm{W}_{ji})(\varphi_i)(2\beta_i(\bm{x}-\bm{c_i})^T) \\
            &= 4\varphi_i\beta_i(\bm{x}-{\bm{c_i}})^T\sum_{j=1}^o(\hat{y}_j-y_j)\bm{W}_{ji}
        \end{split}
    \end{equation}
    $$

    * $\bm{\beta}$
    对于每一个隐藏单元的参数$\beta_i$有
    $$
    \begin{equation}
        \begin{split}
            \frac{\partial \mathcal{L}}{\partial \bm{\beta_i}}
            &= \frac{\partial \mathcal{L}}{\partial \bm{\varphi_i}} \frac{\partial \bm{\varphi_i}}{\partial \bm{\beta_i}} \\
            &= (\sum_{j=1}^o2(\hat{y}_j-y_j)\bm{W}_{ji})(\varphi_i)((\bm{x}-\bm{c_i})^T(\bm{x}-\bm{c_i})) \\
            &= 2\varphi_i(\bm{x}-\bm{c_i})^T(\bm{x}-\bm{c_i})\sum_{j=1}^o(\hat{y}_j-y_j)\bm{W}_{ji}
        \end{split}
    \end{equation}
    $$

* 更新参数
    * $\bm{W}$
    $$
    \begin{equation}
        \bm{W}(t+1) = \bm{W}(t) - \eta\frac{\partial \mathcal{L}}{\partial \bm{W}(t)}
    \end{equation}
    $$
    * $\bm{c}$
    对于每一个隐藏单元的中心参数$\bm{c_i}$有
    $$
    \begin{equation}
        \bm{c_i}(t+1) = \bm{c_{i}}(t) - \eta (\frac{\partial \mathcal{L}}{\partial \bm{c_i}(t)})^T
    \end{equation}
    $$
    * $\bm{\beta}$
    对于每一个隐藏单元的参数$\bm{\beta_i}$有
    $$
    \begin{equation}
        \beta_i(t+1) = \beta_{i}(t) - \eta \frac{\partial \mathcal{L}}{\partial \beta_i(t)}
    \end{equation}
    $$

```python
import numpy as np
import matplotlib.pyplot as plt
from IPython import display
import seaborn as sns
import imageio
import os
import random
from sklearn.datasets import make_blobs, make_moons

# seed
random.seed(1024)
np.random.seed(1024)
%matplotlib inline
```

```python
class RadialBasisFunction(object):

    def __init__(self, input_dims, hidden_dims, output_dims):

        self.input_dims = input_dims
        self.hidden_dims = hidden_dims
        self.output_dims = output_dims

        # 模型权重
        random.seed(1024)
        np.random.seed(1024)

        # center parmas
        # hidden_dims, input_dims
        self.center_params = np.random.randn(hidden_dims, input_dims)
        # normalize params
        # hidden_dims,
        self.norm_params = np.random.random(hidden_dims)
        # output weights
        self.output_weights = np.random.randn(output_dims, hidden_dims+1)

        # 中间结果
        self.input_vector = None
        self.hidden_vector = None
        self.output_vector = None

        # 梯度结果
        self.grad_center_params = None
        self.grad_norm_params = None
        self.grad_hidden_output_vector = None
        self.grad_output_weights = None

    def forward(self, input_vector):

        """
        input_vector: 输入向量
        """
        assert np.shape(input_vector) == (self.input_dims, 1)
        self.input_vector = np.array(input_vector)

        # 输入到隐藏层输出
        self._input2hidden()

        # 隐藏层到输出层
        self._hidden2output()

        return self.output_vector

    def fit(self, X, Y, lr, iteration=10000, draw_fig=False, draw_type="contour sheet"):
        """
        训练函数
        X: 训练样本输入list
        Y: 训练样本标签list
        lr: 学习率
        iteration: 迭代次数
        draw_type: 绘图类型
        """

        random.seed(1024)
        np.random.seed(1024)

        # 总训练样本数
        samples_num = len(X)

        print_frequency = 1
        fig_count = 1

        plt.figure(figsize=(20, 10))
        for i in range(iteration):

            # 随机选择训练样本
            sample_index = np.random.randint(samples_num)

            # 输入向量
            train_input_vector = X[sample_index]
            # 输出向量
            train_output_vector = Y[sample_index]

            # 正向计算
            self.forward(train_input_vector)

            # 误差反向传播
            absolute_error = self.output_vector - train_output_vector
            # 隐藏层到输出层的权重矩阵梯度
            self.grad_output_weights = 2 * np.matmul(absolute_error, self.hidden_vector.T)
            # 隐藏层输出梯度
            self.grad_hidden_output_vector = 2 * np.matmul(absolute_error.T, self.output_weights).T

            # 对于每一个隐藏层神经元的梯度
            self.grad_center_params = np.zeros((self.hidden_dims, self.input_dims))
            self.grad_norm_params = np.zeros(self.hidden_dims)
            for j in range(self.hidden_dims):
                temp_vector = self.input_vector - self.center_params[j, :].reshape(-1, 1)

                self.grad_center_params[j, :] = 4 * self.hidden_vector[j] * self.norm_params[j] * temp_vector.reshape(-1) * self.grad_hidden_output_vector[j]
                self.grad_norm_params[j] = 2 * self.norm_params[j] * self.hidden_vector[j] * np.matmul(temp_vector.T, temp_vector) * self.grad_hidden_output_vector[j]

            # 参数更新
            self.output_weights -= lr * self.grad_output_weights
            self.center_params -= lr * self.grad_center_params

            # norm_mask = self.norm_params - lr * self.grad_norm_params
            self.norm_params -= lr * self.grad_norm_params

            if i % print_frequency == 0:
                self.val_func(X, Y, i, iteration, fig_count, draw_fig, draw_type, show_fig=False)
                fig_count += 1
                print_frequency = int(pow(1.1, fig_count))

        self.val_func(X, Y, iteration, iteration, fig_count, draw_fig, draw_type, show_fig=True)

    def val_func(self, X, Y, cur_iter, total_iter, fig_count, draw_fig=False, draw_type="contour sheet", show_fig=False):
        train_loss = 0
        pred_list = list()
        for s in range(len(X)):
            pred_y = self.forward(X[s])
            pred_list.append(pred_y)
            train_loss += np.sum((pred_y-Y[s])**2)
        train_loss /= len(X)
        print("iter[{}/{}], train_loss:{:.5f}".format(cur_iter, total_iter, train_loss))

        # draw
        if draw_fig:
            self.draw(X, Y, fig_count, cur_iter, pred_list=pred_list, draw_type=draw_type, if_show=show_fig)

    def draw(self, train_x_list, train_y_list, fig_count, cur_iter, pred_list=None, draw_type = "contour sheet", if_show=False, contour_resolution=100):
        """
        draw_type: 绘图类型；若为contour sheet则绘制等值线，否则根据输入数据绘制拟合曲线
        """
        if draw_type == "contour sheet":
            # 绘制等值线图
            x_list = np.linspace(np.min(train_x_list[:, 0]), np.max(train_x_list[:, 0]), contour_resolution)
            y_list = np.linspace(np.min(train_x_list[:, 1]), np.max(train_x_list[:, 1]), contour_resolution)

            show_x1, show_x2 = np.meshgrid(x_list, y_list)

            result_matrix = np.zeros((len(x_list), len(y_list)))
            for x_coor in range(len(x_list)):
                for y_coor in range(len(y_list)):
                    result_matrix[x_coor, y_coor] = self.forward([[x_list[x_coor]], [y_list[y_coor]]])

            plt.cla()
            plt.clf()
            plt.subplot(1, 2, 1)
            plt.title("train samples", fontsize=20)
            plt.xticks(fontsize=20)
            plt.yticks(fontsize=20)
            scatter = plt.scatter(train_x_list[:, 0], train_x_list[:, 1], c=train_y_list.reshape(-1))
            plt.legend(*scatter.legend_elements(), fontsize=15)
            plt.subplot(1, 2, 2)
            plt.title("contour sheet & iter:{}".format(cur_iter), fontsize=20)
            plt.xticks(fontsize=20)
            plt.yticks(fontsize=20)
            c = plt.contour(show_x1, show_x2, result_matrix, 8, colors="black")
            plt.contourf(show_x1, show_x2, result_matrix, 8)
            plt.clabel(c, inline=1, fontsize=20)

            plt.savefig("./images/temp/{}.png".format(fig_count), dpi=100)
            if if_show:
                plt.show()
        else:
            # 绘制回归图
            # 目前仅支持输入为1维，输出为1维的绘图
            plt.cla()
            plt.clf()
            assert np.shape(train_x)[1] == 1
            plt.title("regression & iter:{}".format(cur_iter), fontsize=20)

            # 绘制RBF函数
            for r in range(self.hidden_dims):
                show_x_list = list()
                show_y_list = list()
                center_point = self.center_params[r][0]
                normal_value = self.norm_params[r]
                input_list = np.linspace(np.min(train_x_list), np.max(train_x_list), 200)
                for _x in input_list:
                    temp_value = _x - center_point
                    temp_y = np.exp(-normal_value * temp_value**2) * self.output_weights[0, r]

                    show_x_list.append(_x)
                    show_y_list.append(temp_y)

                plt.plot(show_x_list, show_y_list, label="RBF output", c="#96ce54", linewidth=2)

            # 绘制数据和拟合曲线
            train_x_list = np.reshape(train_x_list, (-1))
            train_y_list = np.reshape(train_y_list, (-1))
            plt.plot(train_x_list.reshape(-1), train_y_list.reshape(-1), label="true value", c="#1685a9", linewidth=5)
            if pred_list is not None:
                pred_list = np.array(pred_list)
                plt.plot(train_x_list.reshape(-1), pred_list.reshape(-1), label="pred value", c="#fa8c35", linewidth=5)

            plt.xticks(fontsize=20)
            plt.yticks(fontsize=20)
            handles, labels = plt.gca().get_legend_handles_labels()
            by_label = dict(zip(labels, handles))
            plt.legend(by_label.values(), by_label.keys(), fontsize=15, loc=1)
            plt.savefig("./images/temp/{}.png".format(fig_count), dpi=100)
            if if_show:
                plt.show()

    def _input2hidden(self):
        """
        从输入向量到隐藏层输出
        """
        self.hidden_vector = np.ones((self.hidden_dims+1, 1))
        for i in range(self.hidden_dims):
            # 计算RBF输出
            temp_vector = self.input_vector - self.center_params[i, :].reshape(-1, 1)
            self.hidden_vector[i] = np.exp(-self.norm_params[i]*np.matmul(temp_vector.T, temp_vector))[0]

    def _hidden2output(self):
        """
        从隐藏层输出到输出层输出
        """
        self.output_vector = np.matmul(self.output_weights, self.hidden_vector)

```

```python
# 数据
class DataCreator(object):

    def __init__(self) -> None:
        super().__init__()

    @staticmethod
    def get_and_data():
        x_list = [[0, 0], [0, 1], [1, 0], [1, 1]]
        y_list = [-1, -1, -1, 1]
        return x_list, y_list

    @staticmethod
    def get_or_data():
        x_list = [[0, 0], [0, 1], [1, 0], [1, 1]]
        y_list = [-1, 1, 1, 1]
        return x_list, y_list

    @staticmethod
    def get_nor_data(nor_index):
        x_list = [[0, 0], [0, 1], [1, 0], [1, 1]]
        y_list = list()
        for i in range(len(x_list)):
            if x_list[i][nor_index]:
                y_list.append(-1)
            else:
                y_list.append(1)

        return x_list, y_list

    @staticmethod
    def get_xor_data():
        x_list = [[0, 0], [0, 1], [1, 0], [1, 1]]
        y_list = [-1, 1, 1, -1]
        return x_list, y_list

    @staticmethod
    def get_moon_data(sample_num, noise=0.05):
        x_list, y_list = make_moons(sample_num, noise=noise, random_state=1024)
        y_list = y_list * 2 - 1
        return x_list, y_list

    @staticmethod
    def get_blob_data(sample_num, std=0.3):
        x_list, y_list = make_blobs(n_samples=sample_num, centers=2,
                                    cluster_std=std, random_state=1024, center_box=(-2, 2))
        y_list = y_list * 2 - 1
        return x_list, y_list
```

```python
# 测试

# 输入一维
# 隐藏层3个RBF函数
# 输出二维

input_dims = 1
hidden_dims = 3
output_dims = 2

model = RadialBasisFunction(input_dims, hidden_dims, output_dims)

# 测试输入
test_input = [[1]]
# 测试真实输出
test_output = [[1], [2]]

print(model.forward(test_input))

# 模型训练
model.fit([test_input], [test_output], lr=0.01, iteration=200)

print(model.forward(test_input))
```

    [[2.71612115]
     [2.49176832]]
    iter[0/200], train_loss:2.74691
    iter[1/200], train_loss:2.36720
    iter[2/200], train_loss:2.03946
    iter[3/200], train_loss:1.75661
    iter[4/200], train_loss:1.51254
    iter[5/200], train_loss:1.30199
    iter[6/200], train_loss:1.12043
    iter[8/200], train_loss:0.82904
    iter[10/200], train_loss:0.61283
    iter[12/200], train_loss:0.45261
    iter[14/200], train_loss:0.33404
    iter[15/200], train_loss:0.28690
    iter[18/200], train_loss:0.18162
    iter[21/200], train_loss:0.11486
    iter[24/200], train_loss:0.07258
    iter[28/200], train_loss:0.03933
    iter[30/200], train_loss:0.02894
    iter[35/200], train_loss:0.01343
    iter[36/200], train_loss:0.01152
    iter[42/200], train_loss:0.00458
    iter[49/200], train_loss:0.00156
    iter[56/200], train_loss:0.00053
    iter[64/200], train_loss:0.00016
    iter[72/200], train_loss:0.00005
    iter[80/200], train_loss:0.00001
    iter[88/200], train_loss:0.00000
    iter[91/200], train_loss:0.00000
    iter[98/200], train_loss:0.00000
    iter[105/200], train_loss:0.00000
    iter[119/200], train_loss:0.00000
    iter[133/200], train_loss:0.00000
    iter[147/200], train_loss:0.00000
    iter[161/200], train_loss:0.00000
    iter[175/200], train_loss:0.00000
    iter[196/200], train_loss:0.00000
    iter[200/200], train_loss:0.00000
    [[1.00000034]
     [2.00000023]]

    <Figure size 1440x720 with 0 Axes>

### 线性可分样本簇

```python
# 数据生成器
data_creator = DataCreator()
# 获取训练数据
train_x_all, train_y_all = data_creator.get_blob_data(600, std=0.2)
train_x_range = [np.min(train_x_all), np.max(train_x_all)]
# 定义模型
# 输入维度：2
# 输出维度：1
# 隐藏层维度：20
input_dims = 2
hidden_dims = 10
output_dims = 1

model = RadialBasisFunction(input_dims, hidden_dims, output_dims)

plt.figure(figsize=(20, 10))
model.draw(train_x_all, train_y_all, 0, 0, if_show=True)

# 模型训练
pred_recorder = model.fit(np.expand_dims(train_x_all, axis=2), np.expand_dims(train_y_all, axis=(1, 2)), lr=0.0005, iteration=20000, draw_fig=True)

# 保存gif
with imageio.get_writer("./images/{}.gif".format("RBF_regression"), mode="I", fps=5) as Writer:
    fig_num = len(os.listdir("./images/temp"))
    for ind in range(fig_num):
        image = imageio.imread("./images/temp/{}.png".format(ind))
        if ind != fig_num-1:
            os.remove("./images/temp/{}.png".format(ind))
        Writer.append_data(image)
    for i in range(10):
        Writer.append_data(image)
    os.remove("./images/temp/{}.png".format(fig_num-1))
```

#### 训练过程

### 回归

```python
# 对sin(x)cos(x)进行回归

# 定义数据
train_x = np.linspace(-3, 3, 1000).reshape(-1, 1, 1)
train_y = np.sin(train_x) * np.cos(train_x)

# 定义模型
input_dims = 1
hidden_dims = 10
output_dims = 1

model = RadialBasisFunction(input_dims, hidden_dims, output_dims)

pred_list = list()
for s in range(len(train_x)):
    pred_y = model.forward(train_x[s])
    pred_list.append(pred_y)

plt.figure(figsize=(20, 10))
model.draw(train_x, train_y, 0, 0, pred_list=pred_list, draw_type="regression", if_show=True)

# 模型训练
pred_recorder = model.fit(train_x, train_y, lr=0.0005, iteration=60000, draw_fig=True, draw_type="regression")

# 保存gif
with imageio.get_writer("./images/{}.gif".format("RBF_regression"), mode="I", fps=5) as Writer:
    fig_num = len(os.listdir("./images/temp"))
    for ind in range(fig_num):
        image = imageio.imread("./images/temp/{}.png".format(ind))
        if ind != fig_num-1:
            os.remove("./images/temp/{}.png".format(ind))
        Writer.append_data(image)
    for i in range(10):Writer.append_data(image)
    os.remove("./images/temp/{}.png".format(fig_num-1))
```

#### 训练过程


# Harmonic vibration in a 1D crystal

```
import numpy as np
import matplotlib.pyplot as plt
```

Properties like thermal conductivity, thermal capacity, $\ldots$ are related to the lattice parameters (size a unit cell, number and type of atoms). Due to thermal fluctuations, atoms get collective motions in the lattice called *phonons*. *Phonons* are basically described as *waves*, and can be characterize through a quadruplet $(\omega, \vec{q})$, containing the angular frequency and the angular wave number.

Experimentalist estimate the lattice parameter from X-ray crystallography. One measure terahertz wave packets velocities to get information related to *phonons*. A basic way to understand the principle of the measurement, it's to imagine a piezoelectric crystal with a controlled frequency, generating wave packet in a length-known sample. By measuring the signal on an oscilloscope, you get the delay of the echoes, and therefor the *velocity*.

*graphs adapted from Maznev, A. A. et al. Propagation of THz acoustic wave packets in GaN at room temperature. Appl. Phys. Lett. 112, 061903 (2018).*

Your goal is to understand the main features of phonon propagation using a simple 1D atomic chain. This project looks for :

* defining the properties related to *phonons* on the simplest 1D system ;
* complexifying the lattice in a perfect periodic crystal ;
* understanding the effect of the impurities in the lattice.

## I. Basic definitions

Consider an infinite one-dimensional chain of $N$ identical atoms with these assumptions :

1. there is interaction only in between contact neighbors,
2. the potential is harmonic with force constant noted $k$,
3. atoms are spaced with equal distance noted $a$.

***Q. 1*** Using the second law of Newton, give the ordinary differential equation which describe the motion of atoms $i \in [1,N]$

***R. 1***

***Q. 2*** Searching solutions in plane-waves form :

$$
\begin{equation} x_i(t) = x_0 e^{j(\omega t - qa i)} \end{equation}
$$
Find the dispersion solution relating the angular frequency to the angular wave number. Plot it.

***R. 2***

```
#Plots
```

The *first Brillouin* is the first period, in between $\left[-\frac{\pi}{2} , \frac{\pi}{2} \right]$. One might define two particular positions :

* the *Brillouin zone center*, defined by $q = 0$
* the *Brillouin* zone boundary, defined by $q_\pm = \pm \frac{\pi}{a}$

***Q. 3*** Explain physically why the frequency $\omega \to 0$ when $q \to 0$.

***R. 3***

***Q. 4*** Represents where one has to read the phase and group velocity on plot ***R.2***.

***R. 4***

## II. Two atoms in the unit cell

A unit cell of size $a$ contains $2$ atoms of different type. Each atom is characterize by a doublet $(t,n)$, where $t$ is the atom-type label and nnnnnnnnn the cell label.

***Q. 5*** As in I., write down the linear differential equation of motions and search the solutions as plane waves. You will note $\frac{u_j}{\sqrt{m_j}}$ the amplitude of the plane wave of atom-type $j$

***R. 5***

***Q. 6*** From the solution ***Q. 5***, write the system of solutions as an eigenproblem with the form :

$$
D \cdot U = \omega^2 U
$$
where $U = \begin{pmatrix} u_1 \\ u_2 \end{pmatrix}$

***R. 6***

***Q. 7*** Find the two eigenvalues $\omega_\pm^2$ and the corresponding eigenvectors $U_\pm$ analytically.

***R. 7***

***Q. 8*** Write a code to find the eigenvalues numerically. The code shall take as input :

* masses $m_1$, $m_2$ ;
* constant $k$ ;
* periodicity $a$ .

It should return $\omega_\pm = f(q)$. Plot it.

```
#R.8
```

***Q. 9*** From the dispersion relation, plot the *phase* and *group velocity*.

```
#R.9
```

***Q. 10*** One of the branch is called the *acoustic branch* and the other branch is called *optical branch*. Using your knowledge of the acoustic electromagnetic waves, justify the terminology. You can eventually plot the eigenvalues in the direct space to get an answer.

***R. 10***

```
#R.10
```

## III. Extension at a system of $N$ atoms in a unit cell

Now, you get the main features of the physics of phonons. Let's extend our infinite system with $N$ atoms in a $a$-length cell. The system is characterized by :

* $\xi_{i}$ the average position of atom $i$ in the unit cell ;
* $k_{i,j}$ the force constant between atom $i$ & $j$ in a unit cell ;
* $m_i$ the mass of atom $i$ in a unit cell .

***Q. 11*** As in II., write down the linear differential equation of motions and search the solutions as plane waves with the form :

$$
x_{p,i} = \frac{u_p}{\sqrt{m_p}} e^{j(\omega t - q(ia + \xi_i))}
$$
***R. 11***

***Q. 12*** Write the values in the $D$ matrix for all the doublet $(i,j)$.

***R. 12***

***Q. 13*** Write a code which take as input :

* a unit cell size $a$ ;
* the range and the number of point $q$ to solve ;
* an array of average position $\xi$ with size $N$ in $a$ unit ;
* an array of constant forces $k$ with size $N-1$ ;
* an array of masses with size $N-1$.

Add conditions on the input data to be successful. Return the complete matrix $D$. You shall write functions to build the matrix depending on a value of $q$.

```
#R. 13
```

***Q. 14*** Write an inversion code which take a vector $U$ of shape $(N,)$ and a matrix $D$ of shape $(N, N)$. If the problem is invertible, it returns the eigenvalues.

```
#R. 14
```

***Q. 15*** Apply the inversion problem recursively for different value of $q$ and reconstruct the dispersion relation. Take a condition as equivalent solution in II. to benchmark your code.

```
#R. 15
```

***Q. 16*** Describe the physics of an infinite periodical problem (effect of a unit box of size $a$, effect of the number of atoms and their masses, effect of the value of $k$, $\ldots$) .

```
#R. 16
```

## IV. Open problem

Real crystal are not perfectly periodic. Impurities are presents in the unit cell and break the translation invariant. You shall write a code which take at least :

* take the number of atoms $N$ in the **chain** with constant mass $m$ and constant force constant $k$ ;
* take the number of impurities $N_i$ in the **chain** with their site location, their mass, and their force constant.

*Help : You cannot solve exactly as previously because of the broken periodicity. The code in absence of impurities must reproduce the previous results on a perfect crystal.*

```
# python
```

*inspired from* :

*\[1\] SMNO, M2.*

*\[2\] Neil. W. Ashcroft & N.David Mermin, Solid state physics, Chp. 22 Classical Theory of the Harmonic Crystals,*

*\[3\] Maznev, A. A. et al. Propagation of THz acoustic wave packets in GaN at room temperature. Appl. Phys. Lett. 112, 061903 (2018)*