# Thay thế t-test BẰNG ước lượng Bayesian
BS Vương Kiến Thanh

## Đặt vấn đề

Có rất nhiều quy trình suy luận thống kê dùng để so sánh sự khác nhau của 2 nhóm.

Trong thế giới thực, sự khác nhau luôn luôn kèm với noise từ đo lường hoặc do hiện tượng phân phối ngẫu nhiên, và dĩ nhiên những kết luận rút ra được từ data quan sát đôi khi không phù hợp.

Do đó, ta cần một model thống kê.

Phương pháp thông dụng nhất là dùng một kiểm định giả thuyết, bao gồm thiết lập giả thuyết vô hiệu, chọn test phù hợp, tính ra số statistic, xem nó có lớn hơn một ngưỡng cụ thể (hay pvalue <0.05)

Tuy nhiên, để thực hiện những test này không dễ, và kết quả có thể bị diễn đạt sai. Ngoài ra, nhiều người sử dụng những lựa chọn test thống kê thông dụng (nhưng không phù hợp) hơn là dựa vào câu hỏi nghiên cứu.

Một cách tiếp cận hiệu quả và cho nhiều thông tin hơn là ước lượng Bayesian.

Thay vì kiểm định 2 nhóm có khác nhau không, ta ước lượng mức độ của sự khác nhau. Hơn nữa, ta có thể ước lượng được tính bất định của sự khác nhau ấy.

## Ví dụ: Đánh giá hiệu quả của thuốc

Đây là ví dụ ảo. Ví dụ nghiên cứu hiệu quả của thuốc làm tăng IQ của trẻ em.
Nghiên cứu sẽ thu thập IQ của từng cá nhân trong nhóm dùng thuốc và nhóm dùng placebo.

```python
import arviz as az
import matplotlib.pyplot as plt

import pandas as pd

import jax.numpy as np
from jax.random import PRNGKey

import numpyro
import numpyro.distributions as dist
from numpyro.infer import MCMC, NUTS

az.style.use("arviz-darkgrid")
%matplotlib inline
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')
```

```python
y1 = np.array([101,100,102,104,102,97,105,105,98,101,100,123,105,103,100,95,102,106,
        109,102,82,102,100,102,102,101,102,102,103,103,97,97,103,101,97,104,
        96,103,124,101,101,100,101,101,104,100,101])
y2 = np.array([99,101,100,101,102,100,97,101,104,101,102,102,100,105,88,101,100,
           104,100,100,100,101,102,103,97,101,101,100,101,99,101,100,100,
           101,100,99,101,100,102,99,100,99])

y = pd.concat([pd.DataFrame({'value':y1, 'group':'drug'}),
           pd.DataFrame({'value':y2, 'group':'placebo'}) ])
y
```

      value
      group

      0
      101
      drug

      1
      100
      drug

      2
      102
      drug

      3
      104
      drug

      4
      102
      drug

      ...
      ...
      ...

      37
      100
      placebo

      38
      102
      placebo

      39
      99
      placebo

      40
      100
      placebo

      41
      99
      placebo

89 rows × 2 columns

```python
y.hist('value', by='group', figsize=(12, 4));
```

### Bước 1: Xác định likelihood
Phân phối IQ sẽ tuân theo phân phối student-t. Phân phối student-t tăng sức mạnh của phân tích, vì nó ít nhạy cảm hơn với các outliers.

Phân phối student-t có 3 parameters: trung bình **mu**, độ lệch chuẩn **sigma**, độ tự do **nu** (tương đương với loc, scale, df).

Ta sẽ áp dụng likelihood này cho 2 nhóm `drug` và `placebo`.

### Bước 2: Xác định priors cho từng parameters.
- **mu ~ Normal(mean_data, std_data $\times$ 2)**
Vì giả thuyết vô hiệu là 2 nhóm như nhau, nên ta dùng trung bình và độ lệch chuẩn của toàn bộ data. std nhân 2 để cho thấy mức độ thông tin còn ít và không ưu thế một nhóm nào trong 2 nhóm.
- **sigma ~ Uniform(0,10)**
Giới hạn lại độ biến thiên của IQ.
- **nu ~ Exponential(1/30)**
Phân phối exponential này có trung bình là 30, là ngưỡng mà phân phối student-t có hình dạng gần giống normal.

```python
mu_m = y.value.mean()
mu_s = y.value.std() * 2
sigma_low = 1
sigma_high = 10
```

Ta có thể viết model như sau:

$$
\begin{align}
y^{(treat)}_i &\sim T(\nu, \mu_1, \sigma_1)  && \quad \text{[likelihood group1]}\\
y^{(placebo)}_i &\sim T(\nu, \mu_2, \sigma_2)  && \quad \text{[likelihood group2]}\\
\mu_1 &\sim N(\bar{x}, 2s)  && \quad \text{[prior mu1]}\\
\mu_2 &\sim N(\bar{x}, 2s)  && \quad \text{[prior mu2]}\\
\sigma_1 &\sim \text{Uniform}(1, 10)  && \quad \text{[prior sigma1]}\\
\sigma_2 &\sim \text{Uniform}(1, 10)  && \quad \text{[prior sigma2]}\\
\nu &\sim \text{Exponential}(^1/_{30})  && \quad \text{[prior nu]}
\end{align}
$$

### Bước 3: Xác định các thông số so sánh cần ước lượng
Để đánh giá sự khác nhau của 2 nhóm, ta cần xác định hiệu của trung bình và độ lệch chuẩn của 2 nhóm.
Nó được khai báo thông qua hàm `deterministic` .
Ta cũng có thể đánh giá sự khác nhau bằng *kích thước hiệu ứng* (**effect size**). Con số này có thể khó diễn giải, nhưng nó là hiệu trung bình được hiệu chỉnh bởi độ lệch chuẩn gộp của 2 nhóm (**pooled std**).

$$
\begin{align}
&\text{difference of means} &&= \mu_1 - \mu_2  \\
&\text{difference of stds} &&= \sigma_1 - \sigma_2  \\
&\text{effect size} &&= \frac{\mu_1 - \mu_2}{\sqrt{(\sigma_1^2 + \sigma_2^2)/2}}
\end{align}
$$

>Effect size is a quantitative measure of the magnitude of the experimenter effect. The larger the effect size the stronger the relationship between two variables.
>Cohen suggested that d = 0.2 be considered a 'small' effect size, 0.5 represents a 'medium' effect size and 0.8 a 'large' effect size. This means that if two groups' means don't differ by 0.2 standard deviations or more, the difference is trivial, even if it is statistically significant.
>
>| Relative size | Effect size |
|-|-|
|Small| <0.2|
|Medium | 0.5|
|Large | > 0.8|

## Numpyro là một probabilistic programming language. Với numpyro, ta có thể tạo ra một Bayesian model một cách dễ dàng.

```python
def model(y1,y2):
    # priors
    group1_mean = numpyro.sample('group1_mean', dist.Normal(mu_m, mu_s))
    group2_mean = numpyro.sample('group2_mean', dist.Normal(mu_m, mu_s))
    group1_std = numpyro.sample('group1_std', dist.Uniform(sigma_low, sigma_high))
    group2_std = numpyro.sample('group2_std', dist.Uniform(sigma_low, sigma_high))
    nu = numpyro.sample('nu', dist.Exponential(1/30))
    # likelihoods
    numpyro.sample('group1', dist.StudentT(nu, group1_mean, group1_std), obs=y1)
    numpyro.sample('group2', dist.StudentT(nu, group2_mean, group2_std), obs=y2)
    # estimate
    diff_of_means = numpyro.deterministic('difference of means', group1_mean - group2_mean)
    diff_of_stds = numpyro.deterministic('difference of stds', group1_std - group2_std)
    effect_size = numpyro.deterministic(
        'effect size',
        diff_of_means / np.sqrt((group1_std**2 + group2_std**2) / 2)
    )
```

### Bước 4: Tính posterior thông qua Markov Chain Monte Carlo (MCMC)
MCMC là một thuật toán lấy mẫu posterior, hay còn gọi là **sampler**. Trong MCMC có rất nhiều sampler, từ đơn giản đến phức tạp: Metropolis Hasting, Gibb, Hamiltonian Monte Carlo (HMC), No U Turn Sampler (NUTS), ....
**NUTS** là một sampler mạnh và phổ biến, được dùng trong ví dụ này.

Các bạn có thể tham khảo demo các samplers trong link này:
https://chi-feng.github.io/mcmc-demo/app.html

Ở đây tôi chọn NUTS, loại bỏ 500 mẫu đầu tiên, lấy 2000 mẫu, chạy 4 lần như vậy (dùng 1 CPU core, 4 tác vụ lần lượt).

```python
tracer = MCMC(NUTS(model), num_warmup=500, num_samples=2000, num_chains=4, chain_method="sequential")
```

Bắt đầu thu thập posterior.

```python
tracer.run(PRNGKey(0), y1, y2)
```

    sample: 100%|██████████| 2500/2500 [00:16<00:00, 156.14it/s, 7 steps of size 5.07e-01. acc. prob=0.91]
    sample: 100%|██████████| 2500/2500 [00:05<00:00, 449.91it/s, 7 steps of size 4.78e-01. acc. prob=0.91]
    sample: 100%|██████████| 2500/2500 [00:04<00:00, 529.14it/s, 3 steps of size 5.21e-01. acc. prob=0.92]
    sample: 100%|██████████| 2500/2500 [00:05<00:00, 497.37it/s, 7 steps of size 5.50e-01. acc. prob=0.90]

### Bước 5: Hiển thị kết quả parameters trong posterior sau khi update (có kèm các thông số deterministic).

```python
tracer.print_summary(exclude_deterministic=False)
```

                               mean       std    median      5.0%     95.0%     n_eff     r_hat
      difference of means      1.02      0.45      1.02      0.26      1.74   8779.88      1.00
       difference of stds      0.93      0.45      0.90      0.23      1.67   7721.10      1.00
              effect size      0.61      0.28      0.60      0.15      1.08   8267.93      1.00
              group1_mean    101.55      0.39    101.56    100.92    102.17   8633.62      1.00
               group1_std      2.12      0.44      2.09      1.41      2.82   6849.61      1.00
              group2_mean    100.54      0.23    100.54    100.17    100.92   8775.77      1.00
               group2_std      1.19      0.15      1.16      1.00      1.40   8132.38      1.00
                       nu      1.97      0.50      1.90      1.19      2.72   6019.32      1.00

    Number of divergences: 0

### Vẽ phân phối của từng param

```python
az.plot_posterior(tracer, var_names=['group1_mean','group2_mean', 'group1_std', 'group2_std', 'nu'],
                  color='#87ceeb', figsize=(12,6));
```

### Và phân phối của các thông số so sánh.
`ref_val` sẽ giúp ta biết được mật độ xác suất của thông số.

```python
az.plot_posterior(tracer, var_names=['difference of means','difference of stds', 'effect size'],
                  ref_val=0,
                  color='#87ceeb', figsize=(10,3));
```

# Kết luận:
Nhìn vào hình, ta kết luận có sự khác nhau có ý nghĩa giữa 2 nhóm.

Với hiệu trung bình, xác suất hiệu trung bình lớn hơn 0 là 98.6%, có nghĩa là trung bình 2 nhóm là khác nhau có ý nghĩa thống kê. Tương tự với hiệu độ lệch chuẩn và kích thước hiệu ứng.

Có thể nói là, thuốc này giúp tăng IQ, cũng như làm tăng độ biến thiên của nó. Có nghĩa là vẫn có cá thể không nhận được lợi ích từ thuốc.

### Bước 6: Kiểm tra độ hội tụ (convergence)
Bởi vì ta dùng 4 chains để chạy MCMC, ta có thể kiểm tra độ hội tụ của kết quả MCMC.
Và 4 chains này cho kết quả gần giống nhau, nên ta kết luận model đã hội tụ.

```python
az.plot_forest(tracer, var_names=['group1_mean',
                                'group2_mean'],
               figsize=(5,2));
```

```python
az.plot_forest(tracer, var_names=['group1_std',
                               'group2_std',
                               'nu'],
               figsize=(5,3));
```

References¶

1. Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine. 1999;130(12):995-1004. doi:10.7326/0003-4819-130-12-199906150-00008.
2. Johnson D. The insignificance of statistical significance testing. Journal of Wildlife Management. 1999;63(3):763-772.
3. Kruschke JK. Bayesian estimation supersedes the t test. J Exp Psychol Gen. 2013;142(2):573-603. doi:10.1037/a0029146.

```python

```


# Exercise Sheet 3

```python
# Import necessary libraries
import time
import numpy as np
import matplotlib.pyplot as plt
from sklearn import metrics
from sklearn.datasets import fetch_openml # MNIST data
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.utils import check_random_state
%matplotlib inline
```

## [1] MNIST - regression
The aim of this exercise is to get some experience with logistic regression and regularisation. We
shall use the implementations in sklearn for this exercise.
* Using the SGDClassifer and the LogisticRegression, build classifers for the MNIST dataset.
* Find some handwritten digits which your classifers are not capable of classifying correctly. How do they differ from correctly classifed digits?
* Scan over hyper-parameters (i.e. l1, l2 regularisation, different optimisers) and compare the
performance.
* Compare your results to results you would have obtained by pure random guessing.

### Solution

```python
# Code adapted from the "A high-bias, low-variance introduction to Machine Learning for physicists" review by Mehta et al.

### load MNIST data from https://www.openml.org/d/554
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)
```

Please have a look at the *excellent* [documentation of sklearn](https://scikit-learn.org/stable/modules/linear_model.html). There you will also find detailed information on the different solvers.

```python
train_size = 10000
test_size = 2000

# shuffle data
random_state = check_random_state(0)
permutation = random_state.permutation(X.shape[0])
X = X[permutation]
y = y[permutation]

# pick training and test data sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=train_size, test_size=test_size)

# scale data to have zero mean and unit variance [required by regressor]
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # fit AND transform
X_test = scaler.transform(X_test)
```

```python
def log_classify(reg_strength=1e4, multi_class='multinomial', penalty='l2', solver='sag', tol=1e-2):
    t0 = time.time()
    # apply logistic regressor with 'sag' solver, C is the inverse regularization strength
    clf = linear_model.LogisticRegression(C=reg_strength, multi_class=multi_class, penalty=penalty, solver=solver, tol=tol)
    # fit data
    clf.fit(X_train, y_train)
    # percentage of nonzero weights
    sparsity = np.mean(clf.coef_ == 0) * 100
    # compute accuracy
    score = clf.score(X_test, y_test)

    #display run time
    run_time = time.time() - t0
    print('Example run in %.3f s' % run_time)
    print("Sparsity: %.2f%%" % sparsity)
    print("Testscore: %.4f" % score)
    return clf
```

```python
log_model = log_classify()
```

    Example run in 16.041 s
    Sparsity: 13.52%
    Testscore: 0.9010

```python
# SGDClassifier

def SGD_classify():
    t0 = time.time()
    clf = linear_model.SGDClassifier(loss='log', max_iter=5, tol=1e-5)
    clf.fit(X_train, y_train)
    score = clf.score(X_test, y_test)

    #display run time
    run_time = time.time() - t0
    print('Example run in %.3f s' % run_time)
    print("Testscore: %.4f" % score)
    return clf
```

```python
SGD_model = SGD_classify()
```

    Example run in 1.394 s
    Testscore: 0.8910

    /Users/danielk/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:603: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
      ConvergenceWarning)

```python
# Find examples that were not classified correctly:

# We use a boolean mask to get those which were not classified correctly -> see 'SupplementaryMaterial3.ipynb'
mask = (log_model.predict(X_test) != y_test)
X_false = X_test[mask]

#plot a few next to each other
plt.figure(figsize=(10, 5))
for i in range(10):
    l2_plot = plt.subplot(2, 5, i + 1)
    l2_plot.imshow(X_false[i].reshape(28, 28), interpolation='nearest',
                   cmap=plt.cm.Greys)
    l2_plot.set_xticks(())
    l2_plot.set_yticks(())
#plt.subplots_adjust(wspace=-0.8,hspace=0, left=0,right=0.5)
```

```python
# Compare to correctly classified ones

# We use a boolean mask to get those which were classified correctly
mask = (log_model.predict(X_test) == y_test)
X_true = X_test[mask]

#plot a few next to each other
plt.figure(figsize=(10, 5))
for i in range(10):
    l2_plot = plt.subplot(2, 5, i + 1)
    l2_plot.imshow(X_true[i].reshape(28, 28), interpolation='nearest',
                   cmap=plt.cm.Greys)
    l2_plot.set_xticks(())
    l2_plot.set_yticks(())
```

We see that those numbers with a lot of straight lines are easier to classify than the ones with loops. The performance on the number 8 seems to be particularly bad. Let's quantify this:

```python
# again we use boolean masking to extract the number of wrongly predited digits.

plt.figure()
for i in range(4):
    prediction_per_class = log_model.predict(X_test)[y_test==str(i)]
    unique, counts = np.unique(prediction_per_class, return_counts=True)
    plt.subplot(2, 2, i+1)
    plt.bar(unique[unique!=str(i)], 100*counts[unique!=str(i)]/len(prediction_per_class))
    plt.title('Misclassification of ' + str(i) + ' in Percent')
    plt.xlabel('Classified as')
plt.tight_layout()

plt.figure()
for i in range(4,8):
    prediction_per_class = log_model.predict(X_test)[y_test==str(i)]
    unique, counts = np.unique(prediction_per_class, return_counts=True)
    plt.subplot(2, 2, i-3)
    plt.bar(unique[unique!=str(i)], 100*counts[unique!=str(i)]/len(prediction_per_class))
    plt.title('Misclassification of ' + str(i) + ' in Percent')
    plt.xlabel('Classified as')
plt.tight_layout()

plt.figure()
for i in range(8,10):
    prediction_per_class = log_model.predict(X_test)[y_test==str(i)]
    unique, counts = np.unique(prediction_per_class, return_counts=True)
    plt.subplot(2, 2, i-7)
    plt.bar(unique[unique!=str(i)], 100*counts[unique!=str(i)]/len(prediction_per_class))
    plt.title('Misclassification of ' + str(i) + ' in Percent')
    plt.xlabel('Classified as')
plt.tight_layout()
```

We see that 9,8 and 5 are very problematic. 1 is the easiest to classify (intuitively obvious).

Now try L1 regularization

```python
# we use the saga solver because it supports l1 regularization
log_classify(reg_strength=1e4, multi_class='multinomial', penalty='l1', solver='saga', tol=1e-2)
```

    Example run in 44.153 s
    Sparsity: 13.52%
    Testscore: 0.9015

    LogisticRegression(C=10000.0, class_weight=None, dual=False,
              fit_intercept=True, intercept_scaling=1, max_iter=100,
              multi_class='multinomial', n_jobs=None, penalty='l1',
              random_state=None, solver='saga', tol=0.01, verbose=0,
              warm_start=False)

We see that there is no significant difference. Let's try other choices of solvers. Supported are (‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’)

```python
log_classify(reg_strength=1e4, multi_class='ovr', penalty='l2', solver='liblinear', tol=1e-2)
```

    Example run in 137.512 s
    Sparsity: 13.52%
    Testscore: 0.8525

    LogisticRegression(C=10000.0, class_weight=None, dual=False,
              fit_intercept=True, intercept_scaling=1, max_iter=100,
              multi_class='ovr', n_jobs=None, penalty='l2', random_state=None,
              solver='liblinear', tol=0.01, verbose=0, warm_start=False)

Runtime is significantly longer. The result is also significantly worse.

```python
log_classify(reg_strength=1e4, multi_class='multinomial', penalty='l2', solver='newton-cg', tol=1e-2)
```

    Example run in 37.230 s
    Sparsity: 13.52%
    Testscore: 0.8565

    LogisticRegression(C=10000.0, class_weight=None, dual=False,
              fit_intercept=True, intercept_scaling=1, max_iter=100,
              multi_class='multinomial', n_jobs=None, penalty='l2',
              random_state=None, solver='newton-cg', tol=0.01, verbose=0,
              warm_start=False)

Result is comparatively poor.

```python
log_classify(reg_strength=1e4, multi_class='multinomial', penalty='l2', solver='lbfgs', tol=1e-2)
```

    Example run in 3.667 s
    Sparsity: 13.52%
    Testscore: 0.8625

    /Users/danielk/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:758: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
      "of iterations.", ConvergenceWarning)

    LogisticRegression(C=10000.0, class_weight=None, dual=False,
              fit_intercept=True, intercept_scaling=1, max_iter=100,
              multi_class='multinomial', n_jobs=None, penalty='l2',
              random_state=None, solver='lbfgs', tol=0.01, verbose=0,
              warm_start=False)

This is quite fast but gives a poor result.

You are invited to also explore what happens if we change the tolerance and regularization strength. Looping over this should be quite straightforward given the function ``log_classify()``.

Of course we see that all of the results are much better than random guessing, which would have a 0.1 success rate.

## [2] 1D Ising - Regression
In the lecture we have shown different performances of linear regression on predicting the energy in
the 1D Ising model depending on our choice of regularisation. The aim of this exercise is to analyse
this performance a bit further.
* Instead of the built-in scoring function, plot the error $E_{out}$ for the choice of parameters used in
the lecture.
* How does the performance change when changing the number of samples?

### Solution

We copy the code from the lecture:

```python
# Wrap the code from the lecture in to a function 'ising()' because we want to adjust the number of samples later.
def ising(n_samples, L):
    # create 10000 random Ising states
    states = np.random.choice([-1, 1], size=(10000, L))

    def ising_energies(states,L):
        """
        This function calculates the energies of the states in the nn Ising Hamiltonian
        """
        J = np.zeros((L,L),)
        for i in range(L):
            J[i,(i+1)%L] -= 1.0
        # compute energies
        E = np.einsum('...i,ij,...j->...',states,J,states)

        return E
    # calculate Ising energies
    energies=ising_energies(states,L)

    # reshape Ising states into RL samples: S_iS_j --> X_p
    states = np.einsum('...i,...j->...ij', states, states)
    shape = states.shape
    states = states.reshape((shape[0],shape[1]*shape[2]))
    # build final data set
    Data = [states,energies]

    # define train and test data sets
    X_train = Data[0][:n_samples]
    Y_train = Data[1][:n_samples] #+ np.random.normal(0,4.0,size=X_train.shape[0])
    X_test = Data[0][n_samples:3*n_samples//2]
    Y_test = Data[1][n_samples:3*n_samples//2] #+ np.random.normal(0,4.0,size=X_test.shape[0])

    # set up Lasso and Ridge Regression models
    leastsq=linear_model.LinearRegression()
    ridge=linear_model.Ridge()
    lasso = linear_model.Lasso()

    # define error lists
    test_errors_leastsq = []

    test_errors_ridge = []

    test_errors_lasso = []

    # set regularisations trength values
    lmbdas = np.logspace(-4, 5, 10)

    #Initialize coeffficients for ridge regression and Lasso
    coefs_leastsq = []
    coefs_ridge = []
    coefs_lasso = []

    for lmbda in lmbdas:

        ### ordinary least squares
        leastsq.fit(X_train, Y_train) # fit model
        coefs_leastsq.append(leastsq.coef_) # store weights
        # use the coefficient of determination R^2 as the performance of prediction.
        #train_errors_leastsq.append(leastsq.score(X_train, Y_train))
        #test_errors_leastsq.append(leastsq.score(X_test,Y_test))

        # use E_out as a measure of performance
        test_errors_leastsq.append(metrics.mean_squared_error(leastsq.predict(X_test),Y_test))

        ### apply Ridge regression
        ridge.set_params(alpha = lmbda) # set regularisation parameter
        ridge.fit(X_train, Y_train) # fit model
        coefs_ridge.append(ridge.coef_) # store weights
        # use the coefficient of determination R^2 as the performance of prediction.
        #train_errors_ridge.append(ridge.score(X_train, Y_train))
        #test_errors_ridge.append(ridge.score(X_test,Y_test))

        # use E_out as a measure of performance
        test_errors_ridge.append(metrics.mean_squared_error(ridge.predict(X_test),Y_test))

        ### apply Ridge regression
        lasso.set_params(alpha = lmbda) # set regularisation parameter
        lasso.fit(X_train, Y_train) # fit model
        coefs_lasso.append(lasso.coef_) # store weights
        # use the coefficient of determination R^2 as the performance of prediction.
        #train_errors_lasso.append(lasso.score(X_train, Y_train))
        #test_errors_lasso.append(lasso.score(X_test,Y_test))

        # use E_out as a measure of performance
        test_errors_lasso.append(metrics.mean_squared_error(lasso.predict(X_test),Y_test))
    return [test_errors_leastsq, test_errors_ridge, test_errors_lasso]

# Plotting routine for visualizing Eout vs. regularization strength
def plot_ising_error(plot_title, errors):
    plt.figure()
    lmbdas = np.logspace(-4, 5, 10)
    plt.scatter(np.log(lmbdas), errors[0], label = 'leastsq', marker = 'x')
    plt.scatter(np.log(lmbdas), errors[1], label = 'ridge')
    plt.scatter(np.log(lmbdas), errors[2], label = 'lasso', marker = '+')
    plt.legend()
    plt.ylabel('$E_{out}$')
    plt.title(plot_title)
    plt.xlabel('log($\lambda$) [regularization strength]')
```

```python
plot_ising_error('100 samples', ising(100, 40))
plot_ising_error('500 samples', ising(500, 40))
plot_ising_error('1000 samples', ising(1000, 40))
```

The best possible $E_{out}$ decreases drastically as a function of the number of samples for all methods. For ridge regression, the regularization strength should be sufficiently small (depending on the number of samples). For LASSO, there is an optimal value for the regularization strength, as the error does not behave monotonically. For the 500 samples it performs best among all methods, given that $\lambda\sim 10^{-5}$.

## [3] Perceptron - XOR
* Show that a single perceptron does not have the capability to realise the XOR function.
* $\star$ How can this be avoided using multiple perceptrons?

### Solution

The XOR function is defined by $$\begin{pmatrix}0&0\\0&1\\1&0\\1&1\end{pmatrix}\to \begin{pmatrix}0\\1\\1\\0\end{pmatrix}$$
Hence for a single perceptron with weights $w_i$ and bias $b$ we would have to satisfy
$$
\begin{align}
0\cdot w_1+0\cdot w_2+b&\leq 0\\
0\cdot w_1+1\cdot w_2+b&> 0\\
1\cdot w_1+0\cdot w_2+b&> 0\\
1\cdot w_1+1\cdot w_2+b&\leq 0
\end{align}
$$
which is easily seen to be self-contradictory.

NOTE: A perceptron cannot realize any linearly inseperable problem.

## [4] Shannon information content
* For independent random variables ($P(X,Y) = P(X)P(Y)$) show that Shannon entropy is
additive, i.e. $H(X,Y) = H(X) + H(Y)$.
* $\star$ This is slightly off-topic but interesting nevertheless. Utilising information content (e.g. by
making it explicit on your decision tree), think about a solution for the following problem. You
are given 12 balls, all equal in weight except for one that is either heavier or lighter. You are
given a two-pan balance to use. In each use of the balance you may put any number of 12 balls
on the left pan, and the same number on the right pan. Your task is to design a strategy to
determine which is the odd ball and whether it is heavier or lighter than the others in as few
uses of the balance as possible.

### Solution

$$
\begin{align}
H(X,Y)&\stackrel{\text{def}}{=}E\left[-\log\left(P(X,Y)\right)\right]
=E\left[-\log\left(P(X)P(Y)\right)\right]\\
&=E\left[-\log\left(P(X)\right)\right]+E\left[-\log\left(P(Y)\right)\right]
=H(X)+H(Y)
\end{align}
$$

The ball weighing problem has $24$ different outcomes ($12\cdot2$, which ball & heavier vs lighter). So the total entropy in base $2$ is $H=\log_2(24)\approx 4.6$. We need to gain slightly more than $1.5$ bit per weighing and we can do it in three steps. This seems feasible and suggests working modulo $3$ in the first step.

We split the balls into three sets of four $(A,B,C)$. We weigh set $A$ vs set $B$. There are three possible outcomes. $A>B$, $B>A$, $A=B$. This is a good first step because for any of the three results the information gain is $\approx 1.6$ bits.

Note that if we had split into two sets, the information gain in the first step would only be $\Delta H=1 \text{bit}$

Let's explore option $A=B$: We know the odd ball is in set $C$ now. We label the balls in this set $1234$. From either set A or B take three reference balls R. If $123=R$ we know that $4$ is the odd ball and a third weighing against a reference ball gives us the heavier/lighter info. If $123>R$ we know that $4$ is normal and either one of $123$ is heavier. We weigh $1\text{vs.}2$ to see which ball is the heavier one. $123<R$ works the same.

If on the other hand we had determined $A>B$ in the beginning, we know that either there is a heavy ball in A or a light ball in B. Label $A=(1234)$, $B=(5678)$. We take a single reference ball $R$ from set $C$. We weigh $125\text{vs.}36R$. If $125=36R$ we know that either $4$ is light or either one of $7,8$ is heavy. So we weigh $7\text{vs.}8$ to determine the last piece of information needed.

If $A>B$ and in the second step we get $125<36R$, the either $6$ is heavy or either one of $1,2$ is light. We weigh $1\text{vs.}2$ to obtain the last piece of information.
$A<B$ works the same as $A>B$


```python
from sympy import *
init_printing()
import numpy as np
from collections import OrderedDict
import matplotlib.pyplot as plt
%matplotlib inline
```

The variance of the bare force diverges in the Monte Carlo integral.

```python
r = Symbol('r',positive=True)
V = 1/r
F = - diff(V, r)
```

```python
F
```

```python
epos = np.array([[0.0, 1.0, 0.0],
                 [0.2, 0.3, 0.0]])
npos = np.array([[0.0, 0.0, .0]])
```

```python
def compute_bare_force(npos, epos, F, r):
    forces = np.zeros_like(npos)
    for ion_idx,ion_pos in enumerate(npos):
        for elec in epos:
            dr = elec - ion_pos
            dr2 = np.dot(dr,dr)
            dr_norm = np.sqrt(dr2)
            #print dr_norm
            dr_hat = dr/dr_norm
            force_component = float(F.subs(r, dr_norm))
            #print dr_norm, 1.0/dr_norm, force_component
            forces[ion_idx] += force_component*dr_hat
            #print ''
    return forces
```

```python
forces = compute_bare_force(npos, epos, F, r)
print forces
```

    [[ 4.26692459  7.40038688  0.        ]]

One solution is to smooth the contribution inside some cutoff radius $R_c$.
See Chiesa, Ceperley, and Zhang (PRL 94, 036404 (2005))

```python
M = Symbol('M', integer=True)
k = Symbol('k', integer=True,positive=True)
a = IndexedBase('a',(M,))
Rc = Symbol('R_c', positive=True)
```

The goal is to fit a constant function with a polynomial that is missing the constant term (only powers of $r$ with $k\gt 0$)

```python
# Equation 4 in the paper
fbar = Sum(a[k]*r**k, (k,1,M))
fbar
```

```python
# Integrate individual terms
integrate(r**k,(r,0,Rc)).doit()
```

```python
# Still need to get to equations 6-8 from equations 2-4
```

```python
m = Symbol('m')
c = IndexedBase('c',(M,))
j = Symbol('j',integer=True)
```

Matrix and vector that define the coefficients, equation 8 in the paper:

```python
Skj = Rc**(m+k+j+1)/(m+k+j+1)
hj = Rc**(j+1)/(j+1)
hj
```

```python
grfits = OrderedDict()
```

```python
Mval = 4
mval = 2
S = np.zeros((Mval,Mval))
h = np.zeros(Mval)
Rcval = 0.4
for kval in range(Mval):
    for jval in range(Mval):
        S[kval,jval] = Skj.subs({M:Mval,m:mval,k:kval+1,j:jval+1,Rc:Rcval})
for jval in range(Mval):
    h[jval] = hj.subs({j:jval+1, Rc:Rcval})
print S
print h
```

    [[  2.04800000e-03   6.82666667e-04   2.34057143e-04   8.19200000e-05]
     [  6.82666667e-04   2.34057143e-04   8.19200000e-05   2.91271111e-05]
     [  2.34057143e-04   8.19200000e-05   2.91271111e-05   1.04857600e-05]
     [  8.19200000e-05   2.91271111e-05   1.04857600e-05   3.81300364e-06]]
    [ 0.08        0.02133333  0.0064      0.002048  ]

```python
ck = np.linalg.solve(S,h)
ck
```

    array([   4375.00000011,  -44296.87500127,  147656.25000466,
           -161132.81250545])

```python
# Dump in C++ format for inclusion in the unit test
print '// for m_exp=%d, N_basis=%d, Rcut=%f'%(mval, Mval, Rcval)
print 'coeff[%d] = {'%len(ck),
print ','.join(['%g'%ck1 for ck1 in ck]),
#for c in ck:
#    print '%g,'%c,
print '};'
```

    // for m_exp=2, N_basis=4, Rcut=0.400000
    coeff[4] = { 4375,-44296.9,147656,-161133 };

```python
np.dot(np.linalg.inv(S),h)
```

    array([   4375.00000011,  -44296.87500127,  147656.25000465,
           -161132.81250544])

```python
gr = Sum(c[k]*r**(k+m),(k,1,M))
gr
```

```python

```

```python
gr2 = gr.subs({M:Mval, m:mval}).doit()

cc = c.subs(M,Mval)
print 'gr2 = ',gr2
for kval in range(Mval):
    print kval, c[kval+1],ck[kval]
    gr2 = gr2.subs(cc[kval+1],ck[kval])
    print kval,gr2
gr2
grfits[Mval] = gr2
```

    gr2 =  r**6*c[4] + r**5*c[3] + r**4*c[2] + r**3*c[1]
    0 c[1] 4375.00000011
    0 r**6*c[4] + r**5*c[3] + r**4*c[2] + 4375.00000010899*r**3
    1 c[2] -44296.8750013
    1 r**6*c[4] + r**5*c[3] - 44296.8750012704*r**4 + 4375.00000010899*r**3
    2 c[3] 147656.250005
    2 r**6*c[4] + 147656.250004658*r**5 - 44296.8750012704*r**4 + 4375.00000010899*r**3
    3 c[4] -161132.812505
    3 -161132.812505447*r**6 + 147656.250004658*r**5 - 44296.8750012704*r**4 + 4375.00000010899*r**3

```python
def compute_smoothed_force(npos, epos, F, r):
    forces = np.zeros_like(npos)
    for ion_idx,ion_pos in enumerate(npos):
        for elec in epos:
            dr = elec - ion_pos
            dr2 = np.dot(dr,dr)
            dr_norm = np.sqrt(dr2)
            #print dr_norm
            dr_hat = dr/dr_norm
            #print 'dr_norm',dr_norm
            if dr_norm < Rcval:
                force_component = float(gr2.subs(r, dr_norm)/dr2)
            else:
                force_component = float(F.subs(r, dr_norm))
            #print dr_norm, 1.0/dr_norm, force_component
            forces[ion_idx] += force_component*dr_hat
            #print ''
    return forces
```

```python
print 'bare =',compute_bare_force(npos, epos, F, r)
forces = compute_smoothed_force(npos, epos, F, r)
print 'smoothed =',forces
```

    bare = [[ 4.26692459  7.40038688  0.        ]]
    smoothed = [[  9.24061106  14.86091659   0.        ]]

```python
xss = OrderedDict()
yss = OrderedDict()
xs = []
ys = []
step = Rcval/50
for i in range(50):
    rval = i*step + step
    xs.append(rval)
    ys.append(exp(-rval**2))
#xss[0] = xs
#yss[0] = ys
for Mval,gr2 in grfits.iteritems():
    xs = []
    ys = []
    for i in range(50):
        rval = i*step + step
        #print rval, gr2.subs(r,rval)
        xs.append(rval)
        #ys.append(exp(-rval**2)*gr2.subs(r,rval))
        ys.append(gr2.subs(r,rval))
    xss[Mval] = xs
    yss[Mval] = ys
```

```python

for Mval in xss.keys()[0:1]:
    plt.plot(xss[Mval],yss[Mval])
plt.show()
```

```python
fx = lambda rval: exp(-rval**2)
print 0,mpmath.quad(fx, [0,Rcval])
for Mval, gr2 in grfits.iteritems():
    #fx = lambda rval : exp(-rval**2)*gr2.subs(r, rval)
    fx = lambda rval : gr2.subs(r,rval)
    ival = mpmath.quad(fx, [0,Rcval])
    #ival = integrate(exp(-r**2)*gr2/r**2,(r,0,Rcval))
    print Mval,ival
```

    0 0.379652839700475
    4 0.365714285715105

```python

```

```python

```


# 2021-09-29 Handling stiffness

## Last time

* A and L stability
* transient advection
* transient heat equation

## Today

* Stability for advection-diffusion
* Cost scaling

```julia
using Plots
using LinearAlgebra
using SparseArrays
using Zygote

function vander(x, k=nothing)
    if k === nothing
        k = length(x)
    end
    V = ones(length(x), k)
    for j = 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function fdstencil(source, target, k)
    "kth derivative stencil from source to target"
    x = source .- target
    V = vander(x)
    rhs = zero(x)'
    rhs[k+1] = factorial(k)
    rhs / V
end

function newton(residual, jacobian, u0; maxits=20)
    u = u0
    uhist = [copy(u)]
    normhist = []
    for k in 1:maxits
        f = residual(u)
        push!(normhist, norm(f))
        J = jacobian(u)
        delta_u = - J \ f
        u += delta_u
        push!(uhist, copy(u))
    end
    uhist, normhist
end

function plot_stability(Rz, title; xlims=(-2, 2), ylims=(-2, 2))
    x = LinRange(xlims[1], xlims[2], 100)
    y = LinRange(ylims[1], ylims[2], 100)
    heatmap(x, y, (x, y) -> abs(Rz(x + 1im*y)), c=:bwr, clims=(0, 2), aspect_ratio=:equal, title=title)
end
Rz_theta(z, theta) = (1 + (1 - theta)*z) / (1 - theta*z)
plot_stability(z -> Rz_theta(z, .5), "Midpoint method")
```

# A $\theta$ solver

```julia
function ode_theta_linear(A, u0; forcing=zero, tfinal=1, h=0.1, theta=.5)
    u = copy(u0)
    t = 0.
    thist = [t]
    uhist = [u0]
    while t < tfinal
        tnext = min(t+h, tfinal)
        h = tnext - t
        rhs = (I + h*(1-theta)*A) * u .+ h*forcing(t+h*theta)
        u = (I - h*theta*A) \ rhs
        t = tnext
        push!(thist, t)
        push!(uhist, u)
    end
    thist, hcat(uhist...)
end
```

    ode_theta_linear (generic function with 1 method)

```julia
# Test on oscillator
A = [0 1; -1 0]
thist, uhist = ode_theta_linear(A, [0., 1], h=.1, theta=.5, tfinal=20)
scatter(thist, uhist')
plot!([cos, sin])
```

# Stiff decay to cosine

$$\dot u = -k(u - \cos t) = -k u + k \cos t$$

```julia
k = 10000
thist, uhist = ode_theta_linear(-k, [.2], forcing=t -> k*cos(t), tfinal=5, h=.1, theta=1)
scatter(thist, uhist[1,:])
plot!(cos)
```

# Advection-diffusion as linear ODE

\begin{align}
(-\kappa u_x + w u)_x &= 0 & Pe_L &= \frac{L \lvert w \rvert}{\kappa}
\end{align}

```julia
function advdiff_matrix(n; kappa=1, wind=1, upwind=0.)
    dx = 2 / n
    rows = [1]
    cols = [1]
    vals = [0.]
    wrap(j) = (j + n - 1) % n + 1
    for i in 1:n
        append!(rows, [i, i, i])
        append!(cols, wrap.(i-1:i+1))
        diffuse = [-1, 2, -1] * kappa / dx^2
        advect_upwind = [-1, 1, 0] * wind / dx
        advect_center = [-1, 0, 1] * wind / 2dx
        stencil = -diffuse - upwind * advect_upwind - (1 - upwind) * advect_center
        append!(vals, stencil)
    end
    sparse(rows, cols, vals)
end
advdiff_matrix(5, kappa=.1)
```

    5×5 SparseMatrixCSC{Float64, Int64} with 15 stored entries:
     -1.25   -0.625    ⋅       ⋅      1.875
      1.875  -1.25   -0.625    ⋅       ⋅
       ⋅      1.875  -1.25   -0.625    ⋅
       ⋅       ⋅      1.875  -1.25   -0.625
     -0.625    ⋅       ⋅      1.875  -1.25

```julia
n = 50
A = advdiff_matrix(n, kappa=1/30, upwind=0)
x = LinRange(-1, 1, n+1)[1:end-1]
u0 = exp.(-16 * x .^ 2)
thist, uhist = ode_theta_linear(A, u0, h=.04, theta=.5)
plot(x, uhist[:, 1:5])
```

# Explain the mechanism and obstacles to fixing

```julia
A = advdiff_matrix(50, kappa=.02, upwind=0)
thist, uhist = ode_theta_linear(A, u0, h=.04, theta=.5, tfinal=1.);
plot(x, uhist[:, 1:5])
```

```julia
A = advdiff_matrix(50, kappa=0, upwind=0)
thist, uhist = ode_theta_linear(A, u0, h=.04, theta=1, tfinal=1.)
plot(x, uhist[:, 1:5])
```

# Explain the mechanism

```julia
A = advdiff_matrix(50, kappa=.02, upwind=0)
thist, uhist = ode_theta_linear(A, u0, h=.04, theta=.5, tfinal=1.);
plot(x, uhist[:, 1:5])
```

```julia
n = 50
A = advdiff_matrix(n, kappa=0, upwind=0)
x = LinRange(-1, 1, n)
u0 = exp.(-16 * x .^ 2)
thist, uhist = ode_theta_linear(A, u0, h=.01, theta=1, tfinal=1.)
plot(x, uhist[:, 1:20])
```

# Stability classes and the $\theta$ method

## Definition: $A$-stability
A method is $A$-stable if the stability region
$$ \{ z : |R(z)| \le 1 \} $$
contains the entire left half plane $$ \Re[z] \le 0 .$$
This means that the method can take arbitrarily large time steps without becoming unstable (diverging) for any problem that is indeed physically stable.

## Definition: $L$-stability
A time integrator with stability function $R(z)$ is $L$-stable if
$$ \lim_{z\to\infty} R(z) = 0 .$$
For the $\theta$ method, we have
$$ \lim_{z\to \infty} \frac{1 + (1-\theta)z}{1 - \theta z} = \frac{1-\theta}{\theta} . $$
Evidently only $\theta=1$ is $L$-stable.

# A test problem

```julia
function initial_three(x)
    a = max(0, 1 - 5 * abs(x + .6))
    b = abs(x) < .2
    c = exp(-100 * (x - .6)^2)
    a + b + c
end
plot(initial_three, xlims=(-1, 1))
```

```julia
n = 100
A = advdiff_matrix(n, kappa=1, upwind=1)
x = LinRange(-1, 1, n)
u0 = initial_three.(x)
thist, uhist = ode_theta_linear(A, u0, h=.05, theta=.8)
plot(x, uhist[:, 1:5], legend=:none)
```

* Small and large $Pe_L$, Upwind vs centered, vary $\theta$

# Spectrum of operators

```julia
n = 20
kappa = 0.01
h = .1
theta = 0
A = advdiff_matrix(n, kappa=kappa, upwind=0)
plot_stability(z -> Rz_theta(z, theta), "\$\\theta=$theta, h=$h\$")
ev = eigvals(Matrix(h*A))
scatter!(real(ev), imag(ev))
```

# Stability limits: CFL principle for advection

> A stable method must be capable of propagating information at least as fast as the continuous system it discretizes.

* Courant, Friedrichs, Levy (1928)
* Discrete information moves at most $\Delta x$ in time $\Delta t$ (with explicit integrator)
* Physical information moves distance $|w| \Delta t$
* "CFL number" or "Courant number"
$$ \textrm{CFL} = \frac{\Delta t |w|}{\Delta x} \le 1 $$
* What does this mean for implicit methods?

```julia
theta=0
n = 20
dx = 2 / n
cfl = .5
h = cfl * dx / 1

plot_stability(z -> Rz_theta(z, theta), "\$\\theta=$theta, h=$h, \\mathrm{CFL}=$cfl\$")
ev = eigvals(Matrix(h*advdiff_matrix(n, kappa=0, upwind=1)))
scatter!(real(ev), imag(ev))
```

# Stability limits: explicit diffusion

* The diffusion operator has stencil
$$ \frac{\kappa}{(\Delta x)^2} \begin{bmatrix} 1 & -2 & 1 \end{bmatrix} $$
* Apply this to test function $e^{i\theta x}$ to get the symbol
\begin{align}
L_{\Delta x} e^{i\theta x}
&=
\frac{\kappa}{(\Delta x)^2} \Big[ e^{-i\theta\Delta x} - 2 + e^{i\theta\Delta x} \Big] e^{i\theta x} \\
&= \frac{2\kappa}{(\Delta x)^2} \underbrace{\Big[ \cos (\theta \Delta x) - 1 \Big]}_{\min = -2} e^{i\theta x}
\end{align}
* Conclude $\frac{-4 \kappa}{(\Delta x)^2} \le \lambda \le 0$, need $-2 \le h\lambda \le 0$ for Euler

```julia
theta=0
n = 20
dx = 2 / n
kappa = 1
lambda_min = -4 * kappa / dx^2
h = -2 / lambda_min

plot_stability(z -> Rz_theta(z, theta), "\$\\theta=$theta, h=$h\$")
ev = eigvals(Matrix(h*advdiff_matrix(n, kappa=kappa, wind=0)))
scatter!(real(ev), imag(ev))
```

# Stiffness

> Stiff equations are problems for which explicit methods don't work. (Hairer and Wanner, 2002)

* "stiff" dates to Curtiss and Hirschfelder (1952)

```julia
k = 5
thist, uhist = ode_theta_linear(-k, [.2], forcing=t -> k*cos(t), tfinal=5, h=.5, theta=1)
scatter(thist, uhist[1,:])
plot!(cos)
```

```julia
function mms_error(h; theta=.5, k=10)
    u0 = [.2]
    thist, uhist = ode_theta_linear(-k, u0, forcing=t -> k*cos(t), tfinal=3, h=h, theta=theta)
    T = thist[end]
    u_exact = (u0 .- k^2/(k^2+1)) * exp(-k*T) .+ k*(sin(T) + k*cos(T))/(k^2 + 1)
    uhist[1,end] .- u_exact
end
```

```julia
hs = .5 .^ (1:8)
errors = mms_error.(hs, theta=0, k=10)
plot(hs, norm.(errors), marker=:auto, xscale=:log10, yscale=:log10)
plot!(hs, hs, label="\$h\$", legend=:topleft)
plot!(hs, hs.^2, label="\$h^2\$")
```

## Discuss: is advection-diffusion stiff?

```julia
theta=0
n = 20
dx = 2 / n
kappa = .05
lambda_min = -4 * kappa / dx^2
cfl = 1
h = min(-2 / lambda_min, cfl * dx)

plot_stability(z -> Rz_theta(z, theta),
    "\$\\theta=$theta, h=$h, Pe_h = $(h/kappa)\$")
ev = eigvals(Matrix(h*advdiff_matrix(n, kappa=kappa, wind=1)))
scatter!(real(ev), imag(ev))
```

# Cost scaling

* Spatial discretization with error $O((\Delta x)^p)$
* Time discretization with error $O((\Delta t)^q)$


## The GIBBON (General-purpose Information-Based Bayesian OptimisatioN) acquisition function

A particularly intuitive and empirically effective class of acquisition functions has arisen based on information theory. Information-theoretic Bayesian Optimisation (BO) seeks to reduce uncertainty in the location of high-performing areas of the search space, as measured in terms of differential entropy. BoTorch already supports information-theoretic BO through an implementation of the Max-value Entropy Search (MES) acquisition function [1] (see the [Max-Value Entropy tutorial](./max_value_entropy) for details), which makes evaluations that reduce uncertainty in the maximum value attained by the objective function. However, in order to support batch and multi-fidelity BO, our implementation of MES employs numerical integrations and fantasy observations  (i. e., we generate one point each time and when we try to generate the 𝑖-th point of a batch, we condition the models on the 𝑖−1 points generated prior to this). Unfortunately,  Each of these calculations can can add significantly to the computational overhead incurred by BO.

In this notebook, we provide an information-theoretic acquisition function for tasks where objective function query costs are not large enough to overshadow significant optimisation overheads known as General-purpose Information-Based Bayesian OptimisatioN (GIBBON) [2]. In this tutorial, we present a very high-level overview of GIBBON and demonstrate its use within BoTorch.

### Calculating GIBBON

Following a principled information-theoretic construction, the GIBBON acquisition function measures the utility of evaluating a candidate batch of $B$ points $\{\textbf{x}\}_{i=1}^B$ as
\begin{align}
    \alpha_{\text{GIBBON}}(\{\textbf{x}\}_{i=1}^B)
    &= \frac{1}{2}\log |C| + \sum_{i=1}^B  \hat{\alpha}_{\text{MES}}(\textbf{x}_i)
\end{align}
where $|C|$ is the determinant of the $B\times B$ correlation matrix between the batch elements and $\hat{\alpha}_{\text{MES}}$ is an analytical approximation of the standard (non-batch) MES acquisition function. The GIBBON acquisition function forms a lower bound on the exact (but intractable) batch MES function and is consequently referred to as the `qLowerBoundMaxValueEntropy` in BoTorch. Crucially, GIBBON can be computed in closed-form and so incurs substantially lower overheads than batch MES via fantasies.

### Interpretating GIBBON
Note that the above decomposition of GIBBON has two terms and each has a helpful intuitive justification. In particular, the first term encourages diversity within the batch (achieving high values for points with low predictive correlation), whereas the second term ensures that evaluations are targeted in areas of the search space that provide large amounts of information about the maximum value attained by the objective function.

<br>
__References__

[1] [Wang, Z., Jegelka, S., _Max-value Entropy Search for Efficient Bayesian Optimization._ arXiv:1703.01968v3, 2018](https://arxiv.org/abs/1703.01968)

[2] [Moss, M., et al., _GIBBON: General-purpose Information-Based Bayesian Optimisation._  arXiv:2102.03324, 2020](https://arxiv.org/abs/2102.03324)

```
import os
SMOKE_TEST = os.environ.get("SMOKE_TEST")
```

### 1. Setting up a toy model
We will fit a standard SingleTaskGP model on noisy observations of the synthetic 2D SixHumpCamel function.

```
import math
import torch

from botorch.test_functions import SixHumpCamel
from botorch.fit import fit_gpytorch_model
from botorch.models import SingleTaskGP
from botorch.utils.transforms import standardize, normalize
from gpytorch.mlls import ExactMarginalLogLikelihood

torch.manual_seed(123456)

bounds = torch.tensor(SixHumpCamel._bounds).T
bounds_norm = torch.tensor([[0.,0.],[1.,1.]])
train_X = bounds[0] + (bounds[1] - bounds[0]) * torch.rand(5, 2)
train_Y = SixHumpCamel(negate=True)(train_X).unsqueeze(-1)

train_X = normalize(train_X, bounds=bounds)
train_Y = standardize(train_Y + 0.05 * torch.randn_like(train_Y))

model = SingleTaskGP(train_X, train_Y)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_model(mll, max_retries=10);
```

### 2. Defining the GIBBON acquisition function

GIBBON is implemented in BoTorch as `qLowerBoundMaxValueEntropy` and supports pending points through its `X_pending` argument. Required arguments for the constructor are `model` and `candidate_set` (the discretized candidate points in the design space that will be used to draw max value samples). There are also other optional parameters, such as number of max value samples. Just like in our implementation of MES, two different sampling algorithms are supported for the max value samples: discretized Thompson sampling and Gumbel sampling (the default choice).

```
from botorch.acquisition.max_value_entropy_search import qLowerBoundMaxValueEntropy

candidate_set_size = 1000 if not SMOKE_TEST else 5
candidate_set = torch.rand(candidate_set_size, bounds_norm.size(1), device=bounds.device, dtype=bounds.dtype)
qGIBBON = qLowerBoundMaxValueEntropy(model, candidate_set)

```

### 3. Optimizing the GIBBON acquisition function to get the next candidate points

In order to obtain the next candidate point(s) to query, we need to optimize the acquisition function over the design space. For $q=1$ case, we can simply call the `optimize_acqf` function in the library. For $q>1$,  we greedily build batches using sequential optimization.

```
from botorch.optim import optimize_acqf

NUM_RESTARTS = 10 if not SMOKE_TEST else 2
RAW_SAMPLES = 512 if not SMOKE_TEST else 4

# for q = 1
candidates, acq_value = optimize_acqf(
    acq_function=qGIBBON,
    bounds=bounds,
    q=1,
    num_restarts=NUM_RESTARTS,
    raw_samples=RAW_SAMPLES,
)
candidates, acq_value
```

    (tensor([[ 0.1199, -0.0158]]), tensor(0.0085))

```
from botorch.optim import optimize_acqf

# for q = 2, sequential optimsiation
candidates, acq_value = optimize_acqf(
    acq_function=qGIBBON,
    bounds=bounds,
    q=2,
    num_restarts=NUM_RESTARTS,
    raw_samples=RAW_SAMPLES,
    sequential=True,
)
candidates, acq_value
```

    (tensor([[ 0.1194, -0.0160],
             [ 1.4241,  0.4417]]),
     tensor([0.0085, 0.0104]))

### 4. Comparing GIBBON with other acquisition functions

We now perform an illustrative comparison between GIBBON and the other low-cost acquisition functions implemented in BoTorch. We plot points chosen by each of the acquisition functions, each acquisition function's surface.

#### Sequential BO (q=1)

Firstly, we investigate GIBBON in the purely sequential case, comparing agaisnt MES, Expected Improvement (EI) and Probability of Improvement (PI). We see that GIBBON provides a very high-quality approximation of MES, choosing essentially the same location.

```
from botorch.acquisition import ExpectedImprovement, ProbabilityOfImprovement, qMaxValueEntropy
import matplotlib.pyplot as plt

%matplotlib inline

# prep different acqusition functions
acqs = {}
candidate_set = torch.rand(10000, bounds.size(1), device=bounds.device, dtype=bounds.dtype)
acqs['GIBBON'] = qLowerBoundMaxValueEntropy(model, candidate_set)
acqs['MES'] = qMaxValueEntropy(model, candidate_set)
acqs['EI'] = ExpectedImprovement(model,best_f=train_Y.max())
acqs['PI'] = ProbabilityOfImprovement(model,best_f=train_Y.max())

# prep grid to evaluate acq functions
n = 100 if not SMOKE_TEST else 2
xv, yv = torch.meshgrid([torch.linspace(0, 1, n), torch.linspace(0, 1, n)])
test_x = torch.stack([xv.reshape(n*n, 1), yv.reshape(n*n, 1)], -1)

# eval and maximise acq functions
evals = {}
candidates = {}
for acq in acqs.keys():
    evals[acq] = acqs[acq](test_x).detach().reshape(n, n)
    candidates[acq],_ = optimize_acqf(
        acq_function=acqs[acq],
        bounds=bounds_norm,
        q=1,
        num_restarts=5,
        raw_samples=100
    )

# plot acqusition function values and chosen points
fig, (ax1, ax2, ax3, ax4) = plt.subplots(nrows=1, ncols=4, sharex=True, sharey=True, figsize=(10,5))
ax1.contourf(xv,yv,evals['GIBBON'],levels=20)
ax1.scatter(candidates['GIBBON'][:,0],candidates['GIBBON'][:,1],marker="X",c="r")
ax1.set_title("GIBBON")
ax2.contourf(xv,yv,evals['MES'],levels=20)
ax2.scatter(candidates['MES'][:,0],candidates['MES'][:,1],marker="X",c="r")
ax2.set_title("MES")
ax3.contourf(xv,yv,evals['EI'],levels=20)
ax3.scatter(candidates['EI'][:,0],candidates['EI'][:,1],marker="X",c="r")
ax3.set_title("EI")
ax4.contourf(xv,yv,evals['PI'],levels=20)
ax4.scatter(candidates['PI'][:,0],candidates['PI'][:,1],marker="X",c="r")
ax4.set_title("PI")
fig.text(0.5, -0.1, 'x_1', ha='center')
fig.text(-0.1, 0.5, 'x_2', va='center')
```

#### Batch BO (q=3)

For the batch BO case, GIBBON selects similar points to MES but with an order-of-magnitude lower computational overhead, i.e perfoming information-theoretic BO at the cost of much simpler acqusition functions like EI and PI. We stress that this gap in computational overhead between GIBBON and MES grows substantially as the optimisation progresses (see [2]).

```
from botorch.acquisition import qNoisyExpectedImprovement, qProbabilityOfImprovement
from time import time

# prep different acqusition functions
acqs = {}
candidate_set = torch.rand(10000, bounds.size(1), device=bounds.device, dtype=bounds.dtype)
acqs['GIBBON'] = qLowerBoundMaxValueEntropy(model, candidate_set)
acqs['MES'] = qMaxValueEntropy(model, candidate_set)
acqs['EI'] = qNoisyExpectedImprovement(model, train_X)
acqs['PI'] = qProbabilityOfImprovement(model, best_f=train_Y.max())

# prep grid to evaluate acq functions
n = 100 if not SMOKE_TEST else 2
xv, yv = torch.meshgrid([torch.linspace(0, 1, n), torch.linspace(0, 1, n)])
test_x = torch.stack([xv.reshape(n*n, 1), yv.reshape(n*n, 1)], -1)

# eval and maximise acq functions
evals = {}
candidates = {}
times = {}
for acq in acqs.keys():
    evals[acq] = acqs[acq](test_x).detach().reshape(n, n)
    t_0 = time()
    candidates[acq],_ = optimize_acqf(
        acq_function=acqs[acq],
        bounds=bounds_norm,
        q=3,
        num_restarts=5,
        raw_samples=100,
        sequential=True
    )
    times[acq] = time() - t_0

# plot acqusition function values and chosen points
fig, (ax1, ax2, ax3, ax4) = plt.subplots(nrows=1, ncols=4, sharex=True, sharey=True, figsize=(10,5))
ax1.contourf(xv,yv,evals['GIBBON'],levels=20)
ax1.scatter(candidates['GIBBON'][:,0],candidates['GIBBON'][:,1],marker="X",c="r")
ax1.set_title("GIBBON")
ax2.contourf(xv,yv,evals['MES'],levels=20)
ax2.scatter(candidates['MES'][:,0],candidates['MES'][:,1],marker="X",c="r")
ax2.set_title("MES")
ax3.contourf(xv,yv,evals['EI'],levels=20)
ax3.scatter(candidates['EI'][:,0],candidates['EI'][:,1],marker="X",c="r")
ax3.set_title("EI")
ax4.contourf(xv,yv,evals['PI'],levels=20)
ax4.scatter(candidates['PI'][:,0],candidates['PI'][:,1],marker="X",c="r")
ax4.set_title("PI")
fig.text(0.5, -0.1, 'x_1', ha='center')
fig.text(-0.1, 0.5, 'x_2', va='center')

# plot computational overheads
plt.figure()
heights = [times[acq] for acq in acqs.keys()]
plt.bar(acqs.keys(), heights)
plt.ylabel("Computation Time")
plt.xlabel("Acquisition Function")
```


# Implementing a basic neural network
By Brett Naul (UC Berkeley)

In this exercise we'll implement and train a simple single-layer neural network classifier using `numpy`. First, let's create some example data: we'll sample points from along interlocking spirals like we saw in the [TensorFlow playground](http://playground.tensorflow.org/#dataset=spiral).

*Based on Stanford CS 231n "Neural Network Case Study" exercise.*

```python
# Imports / plotting configuration
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context('poster')
plt.rcParams['image.interpolation'] = 'nearest'  # hard classification boundaries
plt.rcParams['image.cmap'] = 'viridis'
np.random.seed(13)
```

```python
# Generate spiral sample data
def spiral_data(N, K=3, sigma=0.1):
    X = np.zeros((N * K, 2))
    y = np.zeros(N * K, dtype='int')

    for j in range(K):
        ix = range(N * j, N * (j + 1))
        r = np.linspace(0.0, 1, N)  # radius
        theta = 2 * np.pi * j / K + np.linspace(0, 3 * np.pi, N) + np.random.randn(N) * sigma
        X[ix] = np.c_[r * np.sin(theta), r * np.cos(theta)]
        y[ix] = j

    return X, y
```

```python
N = 100
K = 3
X, y = spiral_data(N, K, 0.1)

# Visualize the generated data
fig = plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap='viridis')
plt.xlim([-1, 1])
plt.ylim([-1, 1]);
```

## Setup
Our neural network will take an $\mathbf{x} = (x_1, x_2)$ vector as input and output a $K$-dimensional vector $\mathbf{p}=(p_1,\dots,p_K)$ of class probabilities. For simplicity we'll focus on a single choice of activation function, the ReLU function $f(x) = \max(x, 0)$.

We'll follow the `scikit-learn` model API and construct a simple model with `fit` and `predict` methods. The exercises below will step you through the necessary steps to implement a fully-functional neural network classifier.

```python
from sklearn.base import BaseEstimator, ClassifierMixin

class SingleLayerReLU(BaseEstimator, ClassifierMixin):
    """Skeleton code for single-layer multi-class neural network classifier w/ ReLU activation.
    NOTE: Whenever you change the code below, you need to re-run this cell AND re-initialize
    your model (`model = SingleLayerNet(...)`) in order to update your specific `model` object.
    """
    def __init__(self, hidden_size, num_classes, sigma_init=0.01):
        """Initialize weights with Gaussian noise scaled by `sigma_init` and
        biases with zeros.
        """
        self.hidden_size = hidden_size
        self.num_classes = num_classes
        self.W1 = sigma_init * np.random.randn(hidden_size, 2)
        self.W2 = sigma_init * np.random.randn(num_classes, hidden_size)
        self.b1 = np.zeros(hidden_size)
        self.b2 = np.zeros(num_classes)

    def loss(self, y, P):
        """Compute total softmax loss.
        Inputs:  y -> (N,) array of true (integer) labels
                 p -> (N, K) array of predicted probabilities
        Outputs: L -> total loss value
        """
        return -np.sum(np.log(P[range(len(P)), y]))

    def dloss(self, X, y):
        """Compute gradient of softmax loss with respect to network weights.
        Inputs:  X -> (N, 2) array of network inputs
                 y -> (N,) array of true labels
        Outputs: dW1 -> (hidden_size, 2) array of weight derivatives
                 dW2 -> (hidden_size, 2) array of weight derivatives
                 db1 -> (hidden_size,) array of bias derivatives
                 db2 -> (hidden_size,) array of bias derivatives
        """
        H = np.maximum(0, X @ self.W1.T + self.b1)  # ReLU activation
        Z = H @ self.W2.T + self.b2
        P = np.exp(Z) / np.sum(np.exp(Z), axis=1, keepdims=True)

        dZ = P
        dZ[range(len(X)), y] -= 1

        dW2 = (H.T @ dZ).T
        db2 = np.sum(dZ, axis=0)

        dH = dZ @ self.W2
        dH[H <= 0] = 0  # backprop ReLU  activation

        dW1 = (X.T @ dH).T
        db1 = np.sum(dH, axis=0)

        return (dW1, dW2, db1, db2)

    def predict_proba(self, X):
        """Compute forward pass for all input values.
        Inputs:  X -> (N, 2) array of network inputs
        Outputs: P -> (N, K) array of class probabilities
        """
        H = np.maximum(0, X @ self.W1.T + self.b1)  # ReLU activation
        Z = H @ self.W2.T + self.b2
        P = np.exp(Z) / np.sum(np.exp(Z), axis=1, keepdims=True)

        return P

    def predict(self, X):
        """Compute most likely class labels for all input values.
        Inputs:  X -> (N, 2) array of network inputs
        Outputs: P -> (N, K) array of class probabilities
        """
        P = self.predict_proba(X)
        return np.argmax(P, 1)

    def fit(self, X, y, step_size=3e-3, n_iter=10000):
        """Optimize model parameters W1, W2, b1, b2 via gradient descent.
        Inputs:  X -> (N, 2) array of network inputs
                 y -> (N,) array of true labels
                 step_size -> gradient descent step size
                 n_iter -> number of gradient descent steps to perform
        Outputs: losses -> (n_iter,) array of loss values after each step
        """
        losses = np.zeros(n_iter + 1)
        for i in range(0, n_iter + 1):
            dW1, dW2, db1, db2 = self.dloss(X, y)
            self.W1 -= step_size * dW1
            self.W2 -= step_size * dW2
            self.b1 -= step_size * db1
            self.b2 -= step_size * db2

            P = self.predict_proba(X)
            losses[i] = self.loss(y, P)

            if i % 1000 == 0:
                print("Iteration {}: loss={}".format(i, losses[i]))
        return losses
```

## Part 1: Forward pass
The "forward pass" step (i.e., computing the output for a given input) can be written as
$$
\begin{align}
\mathbf{h} &= f(W_1 \mathbf{x} + \mathbf{b_1}) \\
\mathbf{z} &= W_2 \mathbf{h} + \mathbf{b_2} \\
\mathbf{p} &= \operatorname{softmax}(\mathbf{z})
\end{align}
$$
where $f(\cdot)$ is an activation function, $W_1$ and $W_2$ are weight matrices, and $\mathbf{b}_1$ and $\mathbf{b}_2$ are bias vectors. The softmax function is given by
$$
\operatorname{softmax}(\mathbf{z}) = \exp\left(\mathbf{z}\right) / \left(\sum_{k=1}^K \exp\left( z_k\right)\right)
$$
and the ReLU activation is
$$
f(x) = \max(x, 0).
$$

### 1a) Implement `predict` function
Your function should loop over the input points (i.e., the rows of $X$) and compute the estimated class probabilities for each. The model parameters $W_1, W_2, b_1, b_2$ are treated as fixed.

### 1b) Examine predictions
The function `visualize_predictions` below can be used to inspect the decision boundaries for a given model, dataset, and class labels. Initialize a network and see what the classifications look like. Try varying the number of hidden units and see how the decision boundary shapes change.

```python
def visualize_predictions(model, X, y, step=0.02):
    x_min, x_max = X[:, 0].min(), X[:, 0].max()
    y_min, y_max = X[:, 1].min(), X[:, 1].max()
    xx, yy = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=40)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.title("Accuracy: {}%".format(100 * np.mean(y == model.predict(X))))

model = SingleLayerReLU(100, K)
visualize_predictions(model, X, y)
```

## Part 2: Training
Obviously generating classifications at random isn't the way to go. Now we'll use gradient descent to iteratively improve the weights of our network and hopefully improve our classification accuracy.

If $y$ is some true label and $\mathbf{p}$ are the estimated class probabilities from our network, then our loss function for a single training example can be written as
$$L(y, \mathbf{p}) = -\log p_y,$$
that is, the negative log of the predicted probability for the true class: 0 if our predicted probability was 1, and $+\infty$ if it was 0.
In order to perform gradient descent, we'll need the derivatives of the loss with respect to all of our model parameters*. We can compute these derivatives via the backpropagation (effectively just the chain rule)**:
$$
\begin{aligned}
\frac{\partial L}{\partial z_k}   &= p_k - \text{\{1 if $y=k$, 0 otherwise\}} \\
\frac{\partial L}{\partial W_2}          &= \mathbf{h} \left(\frac{\partial L}{\partial \mathbf{z}}\right)^T \\
\frac{\partial L}{\partial \mathbf{b}_2} &= \frac{\partial L}{\partial \mathbf{z}}\\
\frac{\partial L}{\partial \mathbf{h}}   &= W_2 \frac{\partial L}{\partial \mathbf{z}} \circ f'(\mathbf{h}) \\
\frac{\partial L}{\partial W_1}          &= \mathbf{x} \left(\frac{\partial L}{\partial \mathbf{h}}\right)^T\\
\frac{\partial L}{\partial \mathbf{b}_1} &= \frac{\partial L}{\partial \mathbf{h}}\\
\end{aligned}
$$
Even for very deep or complicated networks, the gradient of the loss function can always be decomposed in this way, one connection at a time.

The actual quantity we wish to minimize is the sum of this loss value over all the samples in our data set. The gradient of this total loss is just the sum of the gradient for each sample.

### 2a) Implement `fit` function
Our gradient descent training consists of the following steps:
- Compute loss gradient with respect to model parameters using `dloss`. Note: for the sake of efficiency, this function returns the sum of the gradient of over all samples, so you should pass in the entire dataset.
- Update model parameters as $W_i \leftarrow W_i - \alpha\frac{\partial{L}}{\partial W_i}$ and $\mathbf{b}_i \leftarrow \mathbf{b}_i - \alpha\frac{\partial{L}}{\partial \mathbf{b}_i}$, where $\alpha$ is some chosen step size.
- Repeat for `num_iter` iterations.

Implement these steps in the `fit` function above. Check that the loss function is decreasing, either by occasionaly printing the loss value or by returning the losses from each step and plotting the trajectory.

### 2b) Examine predictions for fitted model
Train a model and examine the predictions using `visualize_predictions`. Repeat for a few combinations hidden layer sizes, gradient descent step sizes, and numbers of training iterations. What's the best accuracy you can achieve? How does the loss curve change with the step size?

```python
model = SingleLayerReLU(100, K)
losses = model.fit(X, y, step_size=2e-3, n_iter=20000)
plt.plot(losses, '-')
plt.xlabel('Iteration')
plt.ylabel('Total loss');
plt.figure()
visualize_predictions(model, X, y)
```

### 2c) (optional) Numerically check the gradient implementation in `dloss`
This is an extremely important step when coding derivatives by hand: check the gradients returned by `dloss` by perturbing a single weight value by $\epsilon$ and comparing the resulting change in loss to what you'd expect from the gradient.

\* This is a bit of an abuse of notation since we're taking some derivatives with respect to vectors/matrices; this just means a vector/matrix of derivatives with respect to each element.
\*\* For an excrutiating amount of detail about backprop, see e.g. [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/chap2.html).

```python
dW1, dW2, db1, db2 = model.dloss(X, y)
h = 1e-6
i = 0; j = 0
l0 = model.loss(y, model.predict_proba(X))
model.W1[i, j] += h
lh = model.loss(y, model.predict_proba(X))
print("Derivative: {}; Numerical esitmate: {}".format(dW1[i, j], (lh - l0) / h))
```

    Derivative: -0.03241138676097138; Numerical esitmate: -0.03241135537734863

## Part 3: Compare with `scikit-learn`
The `scikit-learn` package does contain a minimal implementation of a neural network classifier in `scikit-learn.neural_networks.MLPClassifier`. This classifier is nowhere near as powerful or fully-featured as the neural networks provided by other packages such as `tensorflow`, `theano`, etc., but it still can represent complicated non-linear relationships within small datasets.

### 3a) Implement a simple neural network classifier using `scikit-learn`
What parameter values give a network comparable to the one we implemented above? Note that the results may not be equivalent since `scikit-learn` version could reach a different set of (locally) optimal parameters.

Hint: try using `solver='lbfgs'` (a more complicated optimization method) for better results.

### 3b) Experiment with different activations/network sizes
How do the decision boundaries change for different models?

### 3c) Experiment with different numbers of classes and noise levels
Go back to where we initialized the data and change the parameters `K` and `sigma`. Can a single- or multi-layer perceptron still represent more complicated patterns?

```python
from sklearn.neural_network import MLPClassifier

X, y = spiral_data(100, 5, 0.1)
single_layer_model = MLPClassifier((8,), activation='identity', solver='lbfgs')
single_layer_model.fit(X, y)
visualize_predictions(single_layer_model, X, y)
```

```python
multi_layer_model = MLPClassifier((100, 100), alpha=0.5, activation='relu', solver='lbfgs')
multi_layer_model.fit(X, y)
visualize_predictions(multi_layer_model, X, y)
```

## Part 4: Visualize neuron activations (optional)
Interpreting neural network classifiers can be quite difficult, especially when the network contains many layers. One way of attempting to make sense of the model is by visualizing the activations of individual neurons: by plotting the activation of a neuron over the entire input space, we can see what patterns the neurons are learning to represent.

Try below to generate plots that show the activation of an individual neuron. `scikit-learn` doesn't make this immediately available, but we can make it work: try setting the last layer's weights (in `coefs_[-1]`) to zero for everything except a single unit, and setting that unit's weights to 1 (so that the activation just passes straight through). You can re-use most of the code from `visualize_predictions` to generate a contour map of activation strengths.

```python
from copy import deepcopy

X, y = spiral_data(100, 2, 0.1)
multi_layer_model = MLPClassifier((64, 8), alpha=0.5, activation='relu', solver='lbfgs')
multi_layer_model.fit(X, y)
visualize_predictions(multi_layer_model, X, y)

def visualize_activations(model, unit, X, y, step=0.02):
    """Visualize activations of ith neuron of last layer."""
    model = deepcopy(model)
    model.coefs_[-1][:unit] = 0  # zero out other units
    model.coefs_[-1][unit] = 1  # just want the activation function
    model.coefs_[-1][(unit + 1):] = 0  # zero out other units

    x_min, x_max = X[:, 0].min(), X[:, 0].max()
    y_min, y_max = X[:, 1].min(), X[:, 1].max()
    xx, yy = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step))
    Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 0]
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=40)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())

fig, ax = plt.subplots(4, 2, figsize=(14, 20))
for i in range(len(multi_layer_model.coefs_[-1])):
    plt.sca(ax.ravel()[i])
    visualize_activations(multi_layer_model, i, X, y)
```


Notebook version 1.0, 31 Aug 2021. Written by Otto Salmenkivi / CSC - IT Center for Science Ltd. otto.salmenkivi@gmail.com

Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php

Tested on Kvasi, running QLM version 1.2.1: https://research.csc.fi/-/kvasi
***
# Quantum algorithm for T-Bill expected value
Based on Woener and Egger, 2018: https://arxiv.org/abs/1806.06893

In this notebook we implement the calculation of the expectation value of a US Treasury bill. The zero coupon T-bill has a face value $V_F$ and is discounted at an interest rate $r$.  We want to calculate the value at a later time step given that there is a $p$ probability that the interest is constant, and $1-p$ probability that the interest rises to $r+\delta r$. Therefore the T-bill has a value of
<div class='math'>
            \begin{equation}
            V = \dfrac{(1-p)V_F}{1+r+\delta r} + \dfrac{pV_F}{1+r} = (1-p)V_{\text{low}} + pV_\text{high}.
            \end{equation}

We set $p = 30\%$. The values for $V$ need to be mapped to qubits. In this simplified demonstration in order to use only one qubit for the distribution of $V$, we map $ V_\text{low} = 0\: \$ $ and $ V_\text{high} = 1\: \$ $. These correspond to the states $|0\rangle$ and $|1\rangle$, respectively.

We use Quantum Amplitude Estimation to calculate the value of the T-bill. QAE is build from four components:
- initial superposition of evaluation qubits with Hadamard gates
- an $A$ gate that encodes the distribution of the possible outcomes for $V$ with corresponding probabilities and the objective function
- a $Q$ gate based on $A$ and the number of evaluation qubits
-  an inverse Quantum Fourier Transform.

In our simple case the $A$ gate is defined as $A =R_y(\theta)$, where $\theta = 2\sin^{-1}(\sqrt{p})$, i.e. it's merely a rotation around the y-axis parametrized by the probability $p$. Therefore it acts as $A|0\rangle = \sqrt{1-p}|0\rangle + \sqrt{p}|1\rangle$, and respresents both the uncertainty distribution and the objective of our algorithm. Another gate needed is the amplitude estimation operator $Q$. Again, for us it simplifies to $Q = AZA^{\dagger}Z = R_y(2\theta)$. Powers of this gate are controllably applied $m$ times to the objective qubit, where $m$ is the number of evaluation qubits. These qubits are responsible for the accuracy and resolution of the estimate. The controlled gates are of form $ Q^{2^j} = R_y(2^{j+1}\theta)$, where $j \in \{0,\ldots,m-1\}$.

We begin by importing the libraries needed.

```python
from qat.lang.AQASM import Program, QRoutine, H, RY, SWAP
from qat.lang.AQASM.qftarith import QFT
import numpy as np
```

In the following we define two functions that return `QRoutine` objects. First applies a Hadamard gate to each evaluation qubit and the second builds the controlled $Q$ gates.

```python
def hadamard_all(nbqbits):
    rout = QRoutine()
    for i in range(nbqbits):
        rout.apply(H,i)
    return rout
```

```python
# Function for creating the Q^2^j gates
def Q_gates(m):
    rout = QRoutine()
    wires = rout.new_wires(m)
    single_wire = rout.new_wires(1)
    for i in range(m):
        angle = 2**(i+1)*theta_p % (4*np.pi)  # the modulo operator % is not necessary. RY repeats every 4*pi
        rout.apply(RY(angle).ctrl(), wires[i], single_wire)
    return rout
```

The probalitity $p$ and number of evaluation qubits $m$ can be changed below. The circuit is build based on those values.

```python
p = 0.3                            # Probability
theta_p = 2*np.arcsin(np.sqrt(p))  # And the related angle for Q gates
m = 7                              # Number of evaluation qubits

# initiation
prog = Program()
m_qbits = prog.qalloc(m)
single_qbit = prog.qalloc(1)

# Hadamard to all evaluation qubits
prog.apply(hadamard_all(m),m_qbits)

# The A gate
prog.apply(RY(theta_p),single_qbit)

# Applying Q gates
prog.apply(Q_gates(m), m_qbits , single_qbit)

# Inverse Quantum Fourier Transform
prog.apply(QFT(m).dag(),m_qbits)

circuit = prog.to_circ()
%qatdisplay circuit
```

The circuit for the algorithm is now ready. Let's run it on the simulator and print the initial results. The useful information is stored in the evaluation qubits, so we ignore the last qubit in the measurement.

```python
# Running the circuit
from qat.qpus import LinAlg
nbshots = 0
job = circuit.to_job(qubits = m_qbits, nbshots = nbshots)
result = LinAlg().submit(job)

for sample in result:
    print(f'State: {sample.state},  amplitude: {sample.amplitude},  probability: {sample.probability}')

# Visualizing the initial measurement results
import matplotlib.pyplot as plt
plt.figure()
states = [str(sample.state) for sample in result]
probs = [sample.probability for sample in result]
plt.bar(states,probs)
plt.ylabel('Probability')
plt.xticks(rotation = 60)
plt.draw()
```

The measurement results correspond to integers $y \in \{0,\ldots,M-1\}$, where $M = m^2$. These a mapped to the estimator $\tilde{p} = \sin^2(y\pi/M) \in [0,1]$. The estimator converges to $p$ in $O(M^{-1})$, which is quadratically faster than classical Monte Carlo methods and at the root of the proposed computational speed-up.

We do the post-processing below. Since the results contain only measured states, we create a new array and fill in the missing zero amplitudes. It is also noteworthy that multiple states correspond to the same estimator value due to the periodic nature of the $sin$ function.

```python
# Creating an array of zeros for all state probabilities and replacing the non-zero values for measured probabilities
probs = np.zeros(2**m, dtype=float)
for sample in result:
    state_decimal = sample.state.int
    probs[state_decimal] = sample.probability

# The mapping used between measured states and corresponding estimate for the probability p
p_tilde = [np.round(np.sin(i*np.pi/(2**m))**2,3) for i in range(2**(m-1)+1)]
print(f'Possible values for the approximation: \n {p_tilde}')
print(f'{len(p_tilde)} possibilities in total.')
# Aggregating the data from different states that correspond to sama probability bins
freqs =[]
freqs.append(probs[0])
i = 1
while i < 2**m/2:
    # print(f'These states correspond to the same bin: {i} and {2**m-i}')
    freqs.append(probs[i] + probs[2**m-i])
    i += 1
freqs.append(probs[2**(m-1)])

# finding p with max frequency
max_freq = p_tilde[np.argmax(freqs)]
```

That's all the computation and calculation we need to do. Let's finish the notebook by printing the circuit again and by visualizing the final results.

```python
print(f'Probability was set to {p} and {m} evaluation qubits were used.')
print('The circuit built for the estimation on T-bill value using QAA:')
%qatdisplay circuit

print(f'The highest peak is at p = {max_freq}.')

print('The final results after calculating the mapping.\n The red line represents the target value.')
plt.figure()
plt.bar(p_tilde, freqs, width = 0.01, color = 'royalblue')
plt.ylim(0,1)
#plt.xlim(0.1,0.5)
plt.axvline(x=0.3, color = 'r', linestyle = '--', linewidth = 3 )
plt.xlabel(r'$\tilde{p}$',fontsize = '12')
plt.ylabel(' Todennäköisyys ',fontsize = '12')
plt.tick_params(axis='both', which='major', labelsize='11')
textstr = '\n'.join((f'$m=${m}',f'$M=${2**m}', r'$\tilde{p} \approx %.3f$' % (max_freq, )))
plt.text(0.4,0.75, textstr, fontsize = '13')
filename = f'velkakujaava_m{m}'
#plt.savefig(filename, dpi = 300)
plt.plot()
```

A clear peak close to the set probability $p$ should be visible. Resolution and accuracy grows with evaluation qubits. Since we had originally mapped our two possible states $|0\rangle$ and $|1\rangle$ to values $0 \$ $ and $1 \$$ respectively, we can trivially calculate the expected value. For $p = 0.3$ and $m = 5$, QAA produces a approximate value of $\tilde{p} = 0.309$. Therefore the expectation value
    \begin{equation}
        V = (1 - 3.09)\cdot 0 \$ + 0.309 \cdot 1 \$ = 0.309 \$ \,.
    \end{equation}

This was a simple demonstration of the Quantum Amplitude Estimation algorithm applied to calculating the expactation value of a US Treasury bill, meant to show the key parts of the method. Next step could be to implement a more comprehensive distribution of possible outcomes, which would mean a non-trivial objective operator acting on an additional qubit as well as more complex $A$ and $Q$ gates. That's a story for another time.

```python

```


# One Step Methods
Given the first order equation
\begin{equation}
y^{'} = 2y-0.1y^2,
\end{equation}
with the initial condition
\begin{equation}
y(0) = 1.
\end{equation}

Aproximate the solution of the inital value problem for the interval $0 \leq t \leq 4$ using $N=8$.
Using the:
1. Euler method:
$$ w_{i+1}=w_i+hf(t_i,w_i);$$

2. Midpoint method:
$$ w_{i+1}=w_i+hf(t_i+\frac{h}{2},w_i+\frac{h}{2}k1),$$
where $k1=f(t_i,w_i)$;

3. 4th Order Runge Kutta method:
$$ w_{i+1}=w_i+\frac{h}{6}(k1+2k2+2k3+k4),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+\frac{h}{2},w_i+\frac{h}{2}k1),$$
$$k3=f(t_i+\frac{h}{2},w_i+\frac{h}{2}k2),$$
$$k4=f(t_i+h,w_i+hk3);$$

4. Imposter-Butler method:
$$ w_{i+1}=w_i+\frac{h}{4}(k1+2k2),$$
where $$k1=f(t_i,w_i),$$
$$k2=f(t_i+h,w_i+hk1).$$

What is wrong with the Imposter-Butler method?

### DECLARING LIBRARIES

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

### Outputting Table

```python
class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)
```

## Setting up the discrete time axis
$$ h=\frac{4-0}{N}=0.5 $$
$$t_i=t_0+ih$$
$$t_i=0+0.5i$$
for $i=0,1,2,...,8.$

```python
N=4
x_end=2.0
h=((x_end-1)/N)

INITIALCONDITION=1
time=np.zeros(N+1)

table = ListTable()
table.append(['time', 'u', 'w'])
```

### Defining the function
From the initial value problem the function
$$ f(t,y)= 2y-0.1y^2. $$

```python
def myfun(w,u,x):
    return -2*x*w+u*u
```

## NUMERICAL SOLUTION

```python
time[0]=1
```

## Euler Method
$$w_0=1$$
$$ w_{i+1}=w_i+h(2w_i-0.1w_i^2)$$
$$N=0,1,2,...7$$

```python
w=np.zeros(N+1)
u=np.zeros(N+1)
u[0]=0.5
w[0]=-0.5
i=0
table.append([time[i], u[i], w[i]])

for i in range (0,N):
    # Euler Method
    w[i+1]=w[i]+h*myfun(w[i],u[i],time[i])
    u[i+1]=u[i]+h*w[i]
    time[i+1]=time[i]+h
    table.append([time[i+1], u[i+1], w[i+1]])
```

## Plotting

```python
fig = plt.figure(figsize=(8,4))
plt.plot(time,w,'o:',color='red',label='w')
plt.plot(time,u,'o:',color='blue',label='u')
plt.legend(loc=4)
plt.title('Numerical Solutions of initial value problem.')
```

```python
table
```

time	u	w

0.0	0.0	0.0

1.25	0.375	-0.1875

1.5	0.328125	-0.03515625

1.75	0.3193359375	0.01812744140625

2.0	0.3238677978515625	0.027759790420532227

1.0	0.5	-0.5

1.25	0.375	-0.1875

1.5	0.328125	-0.03515625

1.75	0.3193359375	0.01812744140625

2.0	0.3238677978515625	0.027759790420532227

```python

```

.ipynb_checkpoints/Euler Method 2nd Order Equation-checkpoint.ipynb


# State space modeling: Local Linear Trends

This notebook describes how to extend the Statsmodels statespace classes to create and estimate a custom model. Here we develop a local linear trend model.

The Local Linear Trend model has the form (see Durbin and Koopman 2012, Chapter 3.2 for all notation and details):

$$
\begin{align}
y_t & = \mu_t + \varepsilon_t \qquad & \varepsilon_t \sim
    N(0, \sigma_\varepsilon^2) \\
\mu_{t+1} & = \mu_t + \nu_t + \xi_t & \xi_t \sim N(0, \sigma_\xi^2) \\
\nu_{t+1} & = \nu_t + \zeta_t & \zeta_t \sim N(0, \sigma_\zeta^2)
\end{align}
$$

It is easy to see that this can be cast into state space form as:

$$
\begin{align}
y_t & = \begin{pmatrix} 1 & 0 \end{pmatrix} \begin{pmatrix} \mu_t \\ \nu_t \end{pmatrix} + \varepsilon_t \\
\begin{pmatrix} \mu_{t+1} \\ \nu_{t+1} \end{pmatrix} & = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \begin{pmatrix} \mu_t \\ \nu_t \end{pmatrix} + \begin{pmatrix} \xi_t \\ \zeta_t \end{pmatrix}
\end{align}
$$

Notice that much of the state space representation is composed of known values; in fact the only parts in which parameters to be estimated appear are in the variance / covariance matrices:

$$
\begin{align}
H_t & = \begin{bmatrix} \sigma_\varepsilon^2 \end{bmatrix} \\
Q_t & = \begin{bmatrix} \sigma_\xi^2 & 0 \\ 0 & \sigma_\zeta^2 \end{bmatrix}
\end{align}
$$

```python
%matplotlib inline

import numpy as np
import pandas as pd
from scipy.stats import norm
import statsmodels.api as sm
import matplotlib.pyplot as plt
```

To take advantage of the existing infrastructure, including Kalman filtering and maximum likelihood estimation, we create a new class which extends from `statsmodels.tsa.statespace.MLEModel`. There are a number of things that must be specified:

1. **k_states**, **k_posdef**: These two parameters must be provided to the base classes in initialization. The inform the statespace model about the size of, respectively, the state vector, above $\begin{pmatrix} \mu_t & \nu_t \end{pmatrix}'$, and   the state error vector, above $\begin{pmatrix} \xi_t & \zeta_t \end{pmatrix}'$. Note that the dimension of the endogenous vector does not have to be specified, since it can be inferred from the `endog` array.
2. **update**: The method `update`, with argument `params`, must be specified (it is used when `fit()` is called to calculate the MLE). It takes the parameters and fills them into the appropriate state space matrices. For example, below, the `params` vector contains variance parameters $\begin{pmatrix} \sigma_\varepsilon^2 & \sigma_\xi^2 & \sigma_\zeta^2\end{pmatrix}$, and the `update` method must place them in the observation and state covariance matrices. More generally, the parameter vector might be mapped into many different places in all of the statespace matrices.
3. **statespace matrices**: by default, all state space matrices (`obs_intercept, design, obs_cov, state_intercept, transition, selection, state_cov`) are set to zeros. Values that are fixed (like the ones in the design and transition matrices here) can be set in initialization, whereas values that vary with the parameters should be set in the `update` method. Note that it is easy to forget to set the selection matrix, which is often just the identity matrix (as it is here), but not setting it will lead to a very different model (one where there is not a stochastic component to the transition equation).
4. **start params**: start parameters must be set, even if it is just a vector of zeros, although often good start parameters can be found from the data. Maximum likelihood estimation by gradient methods (as employed here) can be sensitive to the starting parameters, so it is important to select good ones if possible. Here it does not matter too much (although as variances, they should't be set zero).
5. **initialization**: in addition to defined state space matrices, all state space models must be initialized with the mean and variance for the initial distribution of the state vector. If the distribution is known, `initialize_known(initial_state, initial_state_cov)` can be called, or if the model is stationary (e.g. an ARMA model), `initialize_stationary` can be used. Otherwise, `initialize_approximate_diffuse` is a reasonable generic initialization (exact diffuse initialization is not yet available). Since the local linear trend model is not stationary (it is composed of random walks) and since the distribution is not generally known, we use `initialize_approximate_diffuse` below.

The above are the minimum necessary for a successful model. There are also a number of things that do not have to be set, but which may be helpful or important for some applications:

1. **transform / untransform**: when `fit` is called, the optimizer in the background will use gradient methods to select the parameters that maximize the likelihood function. By default it uses unbounded optimization, which means that it may select any parameter value. In many cases, that is not the desired behavior; variances, for example, cannot be negative. To get around this, the `transform` method takes the unconstrained vector of parameters provided by the optimizer and returns a constrained vector of parameters used in likelihood evaluation. `untransform` provides the reverse operation.
2. **param_names**: this internal method can be used to set names for the estimated parameters so that e.g. the summary provides meaningful names. If not present, parameters are named `param0`, `param1`, etc.

```python
"""
Univariate Local Linear Trend Model
"""
class LocalLinearTrend(sm.tsa.statespace.MLEModel):
    def __init__(self, endog):
        # Model order
        k_states = k_posdef = 2

        # Initialize the statespace
        super(LocalLinearTrend, self).__init__(
            endog, k_states=k_states, k_posdef=k_posdef,
            initialization='approximate_diffuse',
            loglikelihood_burn=k_states
        )

        # Initialize the matrices
        self.ssm['design'] = np.array([1, 0])
        self.ssm['transition'] = np.array([[1, 1],
                                       [0, 1]])
        self.ssm['selection'] = np.eye(k_states)

        # Cache some indices
        self._state_cov_idx = ('state_cov',) + np.diag_indices(k_posdef)

    @property
    def param_names(self):
        return ['sigma2.measurement', 'sigma2.level', 'sigma2.trend']

    @property
    def start_params(self):
        return [np.std(self.endog)]*3

    def transform_params(self, unconstrained):
        return unconstrained**2

    def untransform_params(self, constrained):
        return constrained**0.5

    def update(self, params, *args, **kwargs):
        params = super(LocalLinearTrend, self).update(params, *args, **kwargs)

        # Observation covariance
        self.ssm['obs_cov',0,0] = params[0]

        # State covariance
        self.ssm[self._state_cov_idx] = params[1:]
```

Using this simple model, we can estimate the parameters from a local linear trend model. The following example is from Commandeur and Koopman (2007), section 3.4., modeling motor vehicle fatalities in Finland.

```python
import requests
from io import BytesIO
from zipfile import ZipFile

# Download the dataset
ck = requests.get('http://staff.feweb.vu.nl/koopman/projects/ckbook/OxCodeAll.zip').content
zipped = ZipFile(BytesIO(ck))
df = pd.read_table(
    BytesIO(zipped.read('OxCodeIntroStateSpaceBook/Chapter_2/NorwayFinland.txt')),
    skiprows=1, header=None, sep='\s+', engine='python',
    names=['date','nf', 'ff']
)
```

Since we defined the local linear trend model as extending from `MLEModel`, the `fit()` method is immediately available, just as in other Statsmodels maximum likelihood classes. Similarly, the returned results class supports many of the same post-estimation results, like the `summary` method.

```python
# Load Dataset
df.index = pd.date_range(start='%d-01-01' % df.date[0], end='%d-01-01' % df.iloc[-1, 0], freq='AS')

# Log transform
df['lff'] = np.log(df['ff'])

# Setup the model
mod = LocalLinearTrend(df['lff'])

# Fit it using MLE (recall that we are fitting the three variance parameters)
res = mod.fit()
print(res.summary())
```

Finally, we can do post-estimation prediction and forecasting. Notice that the end period can be specified as a date.

```python
# Perform prediction and forecasting
predict = res.get_prediction()
forecast = res.get_forecast('2014')
```

```python
fig, ax = plt.subplots(figsize=(10,4))

# Plot the results
df['lff'].plot(ax=ax, style='k.', label='Observations')
predict.predicted_mean.plot(ax=ax, label='One-step-ahead Prediction')
predict_ci = predict.conf_int(alpha=0.05)
predict_index = np.arange(len(predict_ci))
ax.fill_between(predict_index[2:], predict_ci.iloc[2:, 0], predict_ci.iloc[2:, 1], alpha=0.1)

forecast.predicted_mean.plot(ax=ax, style='r', label='Forecast')
forecast_ci = forecast.conf_int()
forecast_index = np.arange(len(predict_ci), len(predict_ci) + len(forecast_ci))
ax.fill_between(forecast_index, forecast_ci.iloc[:, 0], forecast_ci.iloc[:, 1], alpha=0.1)

# Cleanup the image
ax.set_ylim((4, 8));
legend = ax.legend(loc='lower left');
```

### References

    Commandeur, Jacques J. F., and Siem Jan Koopman. 2007.
    An Introduction to State Space Time Series Analysis.
    Oxford ; New York: Oxford University Press.

    Durbin, James, and Siem Jan Koopman. 2012.
    Time Series Analysis by State Space Methods: Second Edition.
    Oxford University Press.