```python
## Based on code at
#  - https://scikit-optimize.github.io/stable/auto_examples/bayesian-optimization.html
## Also see:
#  - https://machinelearningmastery.com/what-is-bayesian-optimization/
#  - https://distill.pub/2020/bayesian-optimization/
```

```python
%matplotlib inline
```

```python
## Install as needed
!pip install scikit-optimize
```

    Collecting scikit-optimize
      Downloading scikit_optimize-0.8.1-py2.py3-none-any.whl (101 kB)
         |████████████████████████████████| 101 kB 46 kB/s ta 0:00:01
    Requirement already satisfied: joblib>=0.11 in /Users/biplavs/opt/anaconda3/lib/python3.8/site-packages (from scikit-optimize) (0.16.0)
    Requirement already satisfied: numpy>=1.13.3 in /Users/biplavs/opt/anaconda3/lib/python3.8/site-packages (from scikit-optimize) (1.20.1)
    Requirement already satisfied: scipy>=0.19.1 in /Users/biplavs/opt/anaconda3/lib/python3.8/site-packages (from scikit-optimize) (1.6.0)
    Requirement already satisfied: scikit-learn>=0.20.0 in /Users/biplavs/opt/anaconda3/lib/python3.8/site-packages (from scikit-optimize) (0.24.1)
    Collecting pyaml>=16.9
      Downloading pyaml-20.4.0-py2.py3-none-any.whl (17 kB)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/biplavs/opt/anaconda3/lib/python3.8/site-packages (from scikit-learn>=0.20.0->scikit-optimize) (2.1.0)
    Requirement already satisfied: PyYAML in /Users/biplavs/opt/anaconda3/lib/python3.8/site-packages (from pyaml>=16.9->scikit-optimize) (5.3.1)
    Installing collected packages: pyaml, scikit-optimize
    Successfully installed pyaml-20.4.0 scikit-optimize-0.8.1

# Bayesian optimization with `skopt`

Gilles Louppe, Manoj Kumar July 2016.
Reformatted by Holger Nahrstaedt 2020

.. currentmodule:: skopt

## Problem statement

We are interested in solving

\begin{align}x^* = arg \min_x f(x)\end{align}

under the constraints that

- $f$ is a black box for which no closed form is known
  (nor its gradients);
- $f$ is expensive to evaluate;
- and evaluations of $y = f(x)$ may be noisy.

**Disclaimer.** If you do not have these constraints, then there
is certainly a better optimization algorithm than Bayesian optimization.

This example uses :class:`plots.plot_gaussian_process` which is available
since version 0.8.

## Bayesian optimization loop

For $t=1:T$:

1. Given observations $(x_i, y_i=f(x_i))$ for $i=1:t$, build a
   probabilistic model for the objective $f$. Integrate out all
   possible true functions, using Gaussian process regression.

2. optimize a cheap acquisition/utility function $u$ based on the
   posterior distribution for sampling the next point.
   $x_{t+1} = arg \min_x u(x)$
   Exploit uncertainty to balance exploration against exploitation.

3. Sample the next observation $y_{t+1}$ at $x_{t+1}$.

## Acquisition functions

Acquisition functions $u(x)$ specify which sample $x$: should be
tried next:

- Expected improvement (default):
  $-EI(x) = -\mathbb{E} [f(x) - f(x_t^+)]$
- Lower confidence bound: $LCB(x) = \mu_{GP}(x) + \kappa \sigma_{GP}(x)$
- Probability of improvement: $-PI(x) = -P(f(x) \geq f(x_t^+) + \kappa)$

where $x_t^+$ is the best point observed so far.

In most cases, acquisition functions provide knobs (e.g., $\kappa$) for
controlling the exploration-exploitation trade-off.
- Search in regions where $\mu_{GP}(x)$ is high (exploitation)
- Probe regions where uncertainty $\sigma_{GP}(x)$ is high (exploration)

```python
print(__doc__)

import numpy as np
np.random.seed(237)
import matplotlib.pyplot as plt
from skopt.plots import plot_gaussian_process
```

    Automatically created module for IPython interactive environment

## Toy example

Let assume the following noisy function $f$:

```python
noise_level = 0.1

def f(x, noise_level=noise_level):
    return np.sin(5 * x[0]) \
           + np.random.randn() * noise_level

```

**Note.** In `skopt`, functions $f$ are assumed to take as input a 1D
vector $x$: represented as an array-like and to return a scalar
$f(x)$:.

```python
# Plot f(x) + contours
x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = [f(x_i, noise_level=0.0) for x_i in x]
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],
                         [fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),
         alpha=.2, fc="r", ec="None")
plt.legend()
plt.grid()
plt.show()
```

Bayesian optimization based on gaussian process regression is implemented in
:class:`gp_minimize` and can be carried out as follows:

```python
from skopt import gp_minimize

res = gp_minimize(f,                  # the function to minimize
                  [(-2.0, 2.0)],      # the bounds on each dimension of x
                  acq_func="EI",      # the acquisition function
                  n_calls=15,         # the number of evaluations of f
                  n_random_starts=5,  # the number of random initialization points
                  noise=0.1**2,       # the noise level (optional)
                  random_state=1234)   # the random seed
```

Accordingly, the approximated minimum is found to be:

```python
"x^*=%.4f, f(x^*)=%.4f" % (res.x[0], res.fun)
```

    'x^*=1.0019, f(x^*)=-1.1122'

For further inspection of the results, attributes of the `res` named tuple
provide the following information:

- `x` [float]: location of the minimum.
- `fun` [float]: function value at the minimum.
- `models`: surrogate models used for each iteration.
- `x_iters` [array]:
  location of function evaluation for each iteration.
- `func_vals` [array]: function value for each iteration.
- `space` [Space]: the optimization space.
- `specs` [dict]: parameters passed to the function.

```python
print(res)
```

              fun: -1.1121897630505762
        func_vals: array([-0.1016967 ,  0.21955974,  0.94017496, -0.67312655,  0.93181798,
           -1.11218976, -0.92480335, -0.36793422,  0.95832219, -0.84072408,
            0.48650594, -0.60384929, -0.90983982, -0.96329364, -0.6149741 ])
           models: [GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5) + WhiteKernel(noise_level=0.01),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775)]
     random_state: RandomState(MT19937) at 0x7FBCF3821140
            space: Space([Real(low=-2.0, high=2.0, prior='uniform', transform='normalize')])
            specs: {'args': {'func': <function f at 0x7fbcf25c8790>, 'dimensions': Space([Real(low=-2.0, high=2.0, prior='uniform', transform='normalize')]), 'base_estimator': GaussianProcessRegressor(kernel=1**2 * Matern(length_scale=1, nu=2.5),
                             n_restarts_optimizer=2, noise=0.010000000000000002,
                             normalize_y=True, random_state=822569775), 'n_calls': 15, 'n_random_starts': 5, 'n_initial_points': 10, 'initial_point_generator': 'random', 'acq_func': 'EI', 'acq_optimizer': 'auto', 'x0': None, 'y0': None, 'random_state': RandomState(MT19937) at 0x7FBCF3821140, 'verbose': False, 'callback': None, 'n_points': 10000, 'n_restarts_optimizer': 5, 'xi': 0.01, 'kappa': 1.96, 'n_jobs': 1, 'model_queue_size': None}, 'function': 'base_minimize'}
                x: [1.0019199033882362]
          x_iters: [[-0.009345334109402526], [1.2713537644662787], [0.4484475787090836], [1.0854396754496047], [1.4426790855107496], [1.0019199033882362], [0.9084971918979265], [-0.5502630233409587], [-0.9450498977253534], [-0.3097851559933513], [-2.0], [2.0], [0.9756453653364003], [-1.5141721803352448], [-1.381403768524824]]

Together these attributes can be used to visually inspect the results of the
minimization, such as the convergence trace or the acquisition function at
the last iteration:

```python
from skopt.plots import plot_convergence
plot_convergence(res);
```

Let us now visually examine

1. The approximation of the fit gp model to the original function.
2. The acquisition values that determine the next point to be queried.

```python
plt.rcParams["figure.figsize"] = (8, 14)

def f_wo_noise(x):
    return f(x, noise_level=0)
```

Plot the 5 iterations following the 5 random points

```python
for n_iter in range(5):
    # Plot true function.
    plt.subplot(5, 2, 2*n_iter+1)

    if n_iter == 0:
        show_legend = True
    else:
        show_legend = False

    ax = plotgaussian_process(res, n_calls=n_iter,
                               objective=f_wo_noise,
                               noise_level=noise_level,
                               show_legend=show_legend, show_title=False,
                               show_next_point=False, show_acq_func=False)
    ax.set_ylabel("")
    ax.set_xlabel("")
    # Plot EI(x)
    plt.subplot(5, 2, 2*n_iter+2)
    ax = plot_gaussian_process(res, n_calls=n_iter,
                               show_legend=show_legend, show_title=False,
                               show_mu=False, show_acq_func=True,
                               show_observations=False,
                               show_next_point=True)
    ax.set_ylabel("")
    ax.set_xlabel("")

plt.show()
```

The first column shows the following:

1. The true function.
2. The approximation to the original function by the gaussian process model
3. How sure the GP is about the function.

The second column shows the acquisition function values after every
surrogate model is fit. It is possible that we do not choose the global
minimum but a local minimum depending on the minimizer used to minimize
the acquisition function.

At the points closer to the points previously evaluated at, the variance
dips to zero.

Finally, as we increase the number of points, the GP model approaches
the actual function. The final few points are clustered around the minimum
because the GP does not gain anything more by further exploration:

```python
plt.rcParams["figure.figsize"] = (6, 4)

# Plot f(x) + contours
_ = plot_gaussian_process(res, objective=f_wo_noise,
                          noise_level=noise_level)

plt.show()
```

## Larger Example

```python
def f(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
           + np.random.randn() * noise_level
```

```python
# Plot f(x) + contours
x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = [f(x_i, noise_level=0.0) for x_i in x]
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],
                         [fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),
         alpha=.2, fc="r", ec="None")
plt.legend()
plt.grid()
plt.show()
```

```python
res = gp_minimize(f,                  # the function to minimize
                  [(-2.0, 2.0)],      # the bounds on each dimension of x
                  acq_func="EI",      # the acquisition function
                  n_calls=15,         # the number of evaluations of f
                  n_random_starts=5,  # the number of random initialization points
                  noise=0.1**2,       # the noise level (optional)
                  random_state=1234)   # the random seed
```

```python
"x^*=%.4f, f(x^*)=%.4f" % (res.x[0], res.fun)
```

    'x^*=-0.3818, f(x^*)=-0.8385'

```python
plot_convergence(res);
```

```python
for n_iter in range(5):
    # Plot true function.
    plt.subplot(5, 2, 2*n_iter+1)

    if n_iter == 0:
        show_legend = True
    else:
        show_legend = False

    ax = plot_gaussian_process(res, n_calls=n_iter,
                               objective=f_wo_noise,
                               noise_level=noise_level,
                               show_legend=show_legend, show_title=False,
                               show_next_point=False, show_acq_func=False)
    ax.set_ylabel("")
    ax.set_xlabel("")
    # Plot EI(x)
    plt.subplot(5, 2, 2*n_iter+2)
    ax = plot_gaussian_process(res, n_calls=n_iter,
                               show_legend=show_legend, show_title=False,
                               show_mu=False, show_acq_func=True,
                               show_observations=False,
                               show_next_point=True)
    ax.set_ylabel("")
    ax.set_xlabel("")

plt.show()
```

```python
plt.rcParams["figure.figsize"] = (6, 4)

# Plot f(x) + contours
_ = plot_gaussian_process(res, objective=f_wo_noise,
                          noise_level=noise_level)

plt.show()
```

```python

```


## What is missing is that the GAussian kernel is not positive definite.
## Read reference 3

# Determinantal Point Processes (DPP) for physicist

This is a brief introduction to DPP. The best references are:
1. https://arxiv.org/abs/1207.6083
2. https://homes.cs.washington.edu/~taskar/pubs/dpp_tut.pdf
3. https://arxiv.org/pdf/1205.4818.pdf (for Gaussian DPP sections 3.3 and 3.4)

The main idea is that DPP are an elegant way to include repulsion between elements. The only thing you need is to specify the **similarity symmetric matrix $L$** where $L$ is positive semidefinite, i.e. **all its eigenvalues are non-negative**. A related object is is the **marginal kernel $K$**. The relation between the two is:

\begin{equation}
L=K(I-K)^{-1},\quad K=L(L+I)^{-1}
\end{equation}

In terms of the spectral decompisition the relation above simply means that:

\begin{equation}
L=\sum_{i=1}^N \lambda_i v_i v_i^T,\quad K=\sum_{i=1}^N \frac{\lambda_i}{1+\lambda_i} v_i v_i^T
\end{equation}

The similarity matrix $L$ has size $N \times N$. For example, if there are $N$ locations on a line the matrix $L$ gives the similarity between any to pair of locations. To achieve particle repulsion we assume that $L_{i,i}=1$ and $L_{i,j}$ decays with the cartesian distance between locations $i$ and $j$. A common choice is to use $L_{i,j}=v_i \cdot v_j$ where $v_i, v_j$ are the vectors corresponding to elements $i,j$ respectively. This means that $L=B^T B$ matrix where $B^T$ is tall and $B$ is wide. In this case $L$ is known as a Grahm matrix and for any vector $z$ we have $z^T L z= z^T B^T B z = (Bz)^T (Bz) = |Bz|^2 \ge 0$, i.e. $L$ is positive definite.

Other choices are possible but you need to make sure that the resulting matrix is positive definite.
Another choice is a gaussian kernel: $L_{i,j}=\exp[-\gamma d(i,j)^2]$. This matrix is positive semi-definite.

The fundamental properties of a DPP are:
1. The probability of selecting any subset $Y$ of particles is $P(Y) = \frac{det(L_Y)}{det(L+I)}$ where $L_Y$ is the similarity matrix restricted to the rows and columns corresponding to the few particles selected and $det(L+I)$ is just the normalization constant. Therefore the probability of each selecting the subset $Y=(i_1,i_3,i_7)$ is $P(Y)\propto det(L_Y)$ where $L_Y$ is a $3 \times 3$ matrix.
2. The normalization constant is $\sum_Y det(L_Y) = det(L+I)$. For example if there are only two locations then there are only 4 possibility: zero particle, one particle at 1, one particle at 2, two particles at 1 and 2. Then we have
\begin{equation}
\sum_Y det(L_Y)= 1 + L_{1,1} + L_{2,2} + \left(L_{1,1}L_{2,2} - L_{1,2}L_{2,1}\right) = (L_{1,1}+1)(L_{2,2}+1)- L_{1,2}L_{2,1} = det(L+I).
\end{equation} Note that we have adopted the convention that the determinant of the empy matrix is $1$, i.e. $det(L_{Y=\text{empty}})=1$.
3. Sometimes we are interested in the probability of inclusion, i.e. we are interested in the probability that particles $A=(i_1,i_2)$ belong to the subset $Y$. This probability is $P(A\subseteq Y)\propto det(K_A)$ where $K_A$ is the marginal kernel restricted to the rows and columns corresponding to the subset $A$.
4. DPP encodes repulsion because the probability of selecting the particle at location $i$ is $P(i\subseteq Y) = K_{i,i}$ and the probability of selecting two particles at locations $i,j$ is $P(i,j\subseteq Y) = L_{i,i}L_{j,j}-L_{i,j}L_{j,i} = P(i)P(j) - L_{i,j}^2 \le P(i)P(j)$, i.e. the particle repeal each other.

**Note that DPP are probability distribution over all possible subsets of $N$ elements. Obviously the number of such subsets scales exponentially in $N$ but, amazingly, the normalization constant can be computed in $\mathcal{O}(N^3)$ operations (any determinant can be computed by LU-factorization).**

## Sampling from DPP intuition

A very nice property of DPP is that an efficient sampling algorithm exists.

We can use physics to provide the intuition. Suppose that you consider a system of **non-interacting fermions** with Hamiltonian $H$. In this scenario $H_{i,i}$ describes the external potential at site $i$ and $H_{i,j}$ describes the hopping between site $i$ and $j$ (the hopping is long range but decays with distance). Now you diagonalize $H$ to obtain its eigenvalues and eigenvectors $\lambda_i,v_i$.
In physics any configuration (i.e. wave-function) of $n$ **non-interacting fermions** can be obtained by filling $n$ single-particle eigenstates and by **anti-symmetrize**. The total energy of the configuration is $E=\sum \lambda_i$ and the probability of that configuration is $p\propto exp[-\beta E] = exp[-\beta \sum_i \lambda_i ] = \prod_i f(\lambda_i)$ where $f(\lambda)\equiv \exp[-\beta \lambda]$.

The idea is the same for DPP with the difference that $H=-K$ and that the function $f(\lambda)$ is not an exponentail but it is linear, i.e. $f(\lambda)\equiv\lambda$. This is necessary since in the DPP the probability is proportional to the determinant which is just the product of the eigenvalues.

So if we want to sample the location of particles according to a DPP process we should:
1. select each eigenstate of $K$ with probability equal to its eigenvalues (note that the eigenvalues are $\in (0,1)$ b/c $K$ is positive definite).
2. sample the location of the particles according to that anti-symmetrized wave-function.

In the math literature, this approach is based on the fact that **DPPs are Mixtures of Elementary DPPs**. In step (1) we select the mixture (of eigenstate) and in step (2) we sample from that mixture without worrying about the values of the selected eigenvalues.
Step (1) is trivial. Step (2) on the other hand is still difficul.

For example, let us say, that you have selected the eigenstates $v_1,v_5,v_7$. Now you have to sample the location x,y,z of these three particles correspong to the wave-function
\begin{equation}
\psi(x,y,z)= \frac{1}{\sqrt 3} det \begin{bmatrix}v_1(x) & v_1(y) & v_1(z) \\v_5(x) & v_5(y) & v_5(z) \\ v_7(x) & v_7(y) & v_7(z) \end{bmatrix}
\end{equation}
This is difficult because we do NOT want to work with the full wave-function. If there are 100 locations on the line to choose from the three-particles wave-function above has $100^3=10^6$ distinct values. The brute force approach would be to use the three selected eigenstates to compute the $10^6$ entries for the matrix $M_{i,j,k}$ and then select one entry proportionally to its value. The selected entry would specify, at once, the locations of the three particles. **This, all at one approach, scales badly with the number $n$ of particles** (the total wave-function has $N^n$ values). We would much rather **sample one particle at the time and then update the wave-function to take into account the previous choices**.

Again, the intuition comes from physics. Having observed one particle at location $x_0$ we know that no other particles can be at location $x_0$ (because the total wave-function vanishes every time two corrdinates are intentical). Therefore we need to take the wave-function and project it on the subspace perpendicular to $\psi(x_0)$. We can now repeat the process, i.e. sample another particle and so on.

Mathematically, this approach relyes on:
1. DPP are closed under conditioning (i.e. projecting a fermionic wave-function on a subspace gives another fermionin wave-function)
2. the order of the observation and projection does not matter, i.e. sequentially sample $x_0,x_1,x_2$ is exactly as likely as sampling $x_1,x_0,x_2$ or any other permutations.

## Sampling from DPP algorithm

Recalling that:
1. $H=-K$
2. the probability of selecting each eigenstate is proportional to its eigenvalue
3. $K=L(L+I)^{-1}$

Then the algorithm for smapling from the DPP is:
0. Define the similarity matrix $L$ and perform spectral decomposition $L=V D V^T$ where $D=\text{diag}(\lambda_1,\lambda_2,...,\lambda_N)$ and $V$ contains as columns the eigenvectors of $L$, i.e. $L V_\alpha = \lambda_\alpha V_\alpha$. Note that $L,V$ are $N \times N$.
2. select each $v_\alpha$ with probability $p_\alpha=\frac{\lambda_\alpha}{1+\lambda_\alpha}$
3. At this point you have selected a particular set of eigenvectors $V_J = \{v_\alpha : \alpha \in J\}$. Now you have to sample from the elementary DPP with marginal kernel $K^J = V_J V_J^T = \sum_{\alpha \in J} V_\alpha V_\alpha^T$ where $V_J$ is a matrix of size $|J|\times N$, i.e. $V_J$ is tall matrix and $V_J^T$ is wide matrix
4. Select a location $P(i\subset Y) = K^J_{i,i} = <i| \left(\sum_{\alpha \in J} v_\alpha v_\alpha^T\right)|i> = \sum_{\alpha \in J} v_\alpha(i)^2$
4. Project the wave-function on the subspace orthogonal to $e_i$ and compute an orto-normal basis in that space. Go back to step 3.

See figure below for the schematic of the algorithm

## k-DPP

Suppose we want exactly $k$ particles. The probability of k-DPP is

\begin{equation}
P_L^k(Y)=\frac{P_L(Y)}{\sum_{|Y'|=k}P_L(Y')}=\frac{det(L_Y)}{\sum_{|Y'|=k}det(L_{Y'})}
\end{equation}
Note that the normalization constant is now much more difficult to compute, i.e. it is no longer the determinant of $det(L+I)$.
However, it is clear that,

\begin{equation}
\sum_{|Y'|=2} det(L_{Y'}) = \lambda_1\lambda_2 + \lambda_1 \lambda_3 + \dots + \lambda_{N-1}\lambda_N
\end{equation}
i.e. the sum of all the distinct polynomials of degree $2$ that can be generated using $\Lambda=\{\lambda_1,\dots,\lambda_N\}$. More generally:

\begin{equation}
Z_k = \sum_{|Y'|=k} det(L_{Y'}) = e_k(\lambda_1,\dots,\lambda_N)
\end{equation}
where $e_k$ is the elementary symmetric polynomial.
Likely the symmetric polynomials can be computed recursively:

See figure below for the recursion algorithm

Once the symmetric polynomials are known we can sample the k eigenvectors according to the algorithm below:

```python
import sys
import numpy as np
from sklearn.metrics.pairwise import rbf_kernel
import matplotlib.pylab as plt
%matplotlib inline

class dpp():
    def __init__(self):
        self.lenght = None
        self.D  = None
        self.V  = None
        self.M  = None #It is important that the matrix is positive definite
        self.E  = None

    def compute_symmetric_poly(self,k: int) -> None:
        N=self.lenght
        self.E=np.zeros((N+1,k+1)) # i.e. E[N,k] is IN bound
        self.E[:,0]=1
        for l in range(1,k+1):
            for n in range(1,N+1):
                self.E[n,l]=self.E[n-1,l]+self.D[n-1]*self.E[n-1,l-1]

    def select_k_eigenvectors(self,k: int) -> list:

        # preparation
        N=self.lenght
        n=N
        l=k
        selected = []

        # Compute the symmetric polynomial if I don;t have them already
        if(self.E is None):
            self.compute_symmetric_poly(k)
        elif(k != self.E.shape[1]):
            self.compute_symmetric_poly(k)

        while l > 0:
            marg = self.D[n-1]*self.E[n-1,l-1]/self.E[n,l]
            if(np.random.uniform() < marg):
                selected.append(n-1) # the -1 is just an indexing problems
                l = l - 1
            n = n - 1

        #print("selected eigenvectors",selected)
        return selected

    def sample(self,k=None,verbose=False) -> list:
        """
        Sample the DPP and returns the locations of the DPP particles.
        If k is provided exactly k locations are returned
        """

        # PHASE 1: select the eigenvectors
        if(k==None):
            # Select each eigenvectors with probability lambda/(1+lambda)
            prob = np.divide(self.D,1+self.D)
            x = np.random.rand(prob.size)
            v = np.argwhere(x<prob)
            k = len(v)
            V = self.V[:, v]
            V = V.reshape(self.lenght,-1)
        elif(k==self.lenght):
            #select all the eigenvectors
            k=self.lenght
            V=self.V
        elif(k>self.lenght):
            # error: Cannot sample more than self.lenght DPP particles
            print("Error: k>self.lenght")
            sys.exit()
        else:
            # select exactly k eigevectors
            selected=self.select_k_eigenvectors(k)
            k = len(selected)
            V = self.V[:, selected]
            V = V.reshape(self.lenght,-1)

        if(verbose):
            print("I will select "+str(k)+" points:")

        # PHASE 2: Select iteratively a location
        #ground_set, rem_set = np.arange(N), np.full(N, True)
        locations = list(np.arange(0,self.lenght))
        chosen=[]

        for n in range(0,k):
            # compute probabilities for each item
            # N, k = V.shape
            prob=np.linalg.norm(V,ord=None,axis=1)**2/(k-n) #this is probability of choosing each location
            prob=prob.flatten()

            # select one location from the available_one
            loc = np.random.choice(locations,size=1, p=prob.flatten()).item()
            chosen.append(loc)
            if(verbose):
                print("add location",loc)

            if(len(chosen)==k):
                return chosen
            else:

                # V is a ortonormal basis.
                # Now I need to find a new ortonormal basis for the subspace of V perpendicular to e_loc
                # The constraint that the subspace spanned by V is hortogonal to e_loc is:
                # (c1 V_1 + c2 V_2 + c_k V_k) cdot e_loc = 0
                # This is equivalent to:
                # c1 V_1(loc) + c2 V_2(loc) + c_k V_k(loc) = 0
                # This subspace has dimension k-1 since the coefficients c1,...,ck are now constrained.
                # I need to find a new ortonormal basis for this subspace.
                #
                # The procedure is as follow:
                # e_loc = c1* V_1 + c2* V_2 + ... + ck* V_k
                # The coeff of this decomposition are already available (read off the entry of the vectors):
                # cn* = <e_loc,V_n> = V_n(loc)
                # Build k-1 vectors in the subspace of V perpendicular to e_loc by doing:
                # U_n = V_n - V_1 (cn*/c1*) for n=2,k
                # Note that: <U_n | e_loc> = cn* - c1* (cn*/c1*) = 0
                # Feed these k-1 vectors into a Gram–Schmidt procedure to obtain a hortonormal basis
                # The procedure above works only is c1* is different from zero.
                # For this reason I save myself some problem by choosing V1 as the vector with the largest coeff.
                # Recall that:
                # N, k = V.shape

                cstar=V[loc,:] #for each one of the k eigenvectors I have a cstar
                j  = np.argmax(cstar)
                Vj = V[:,j] # select the special vector
                cj = Vj[loc]
                cstar = np.delete(cstar,j)
                V = np.delete(V,j,axis=1)

                # Next I do a vectorized form for:
                # U[0:N,0:k-1] = V[0:N,0:k-1] - Vj[0:N] (cstar[0:k-1]/cj)
                V=V-np.matmul(Vj.reshape(-1,1),(cstar/cj).reshape(1,-1))

                # Gram–Schmidt procedure to find a basis out of the representative vectors
                basis = []
                for l in range(k-n-1):
                    v=V[:,l]
                    w = v - np.sum( np.dot(v,b)*b  for b in basis )
                    basis.append(w/np.linalg.norm(w))
                V=np.array(basis).T
                if(verbose):
                    print("len(basis)",len(basis))

class dpp_line(dpp):
    def __init__(self, lenght=10, sigma2=1, similarity='soft', bc='PBC'):
        super().__init__()
        self.lenght = lenght
        data = np.zeros([lenght,1])
        data[:,0]=np.arange(0,lenght)
        self.M= np.zeros((self.lenght,self.lenght))
        gamma = 1.0/(2.0*sigma2)

        # Compute the matrix with distance-square
        for n1 in range(0,self.lenght):
            self.M[n1,n1]= 1.0
            for n2 in range(n1+1,self.lenght):
                if(bc == 'OBC'):
                    dx = data[n1,0]-data[n2,0]
                elif(bc == 'PBC'):
                    dx =(data[n1,0]-data[n2,0]+self.lenght ) % self.lenght
                    dx = min(dx, self.lenght - dx)
                else:
                    print("ERROR: Unknown bc")
                    sys.exit()

                d2=dx*dx

                if(similarity == 'soft'):
                    tmp = np.exp(-gamma*d2)
                elif(similarity == 'hard'):
                    tmp = 0.5*float(d2<sigma2)
                else:
                    print("ERROR: Unknown similarity")
                    sys.exit()

                self.M[n1,n2]= tmp
                self.M[n2,n1]= tmp

        # Diagonalize matrix
        self.D, self.V  = np.linalg.eigh(self.M) # Use this because matrix is symmetric
        for l in self.D:
            if(l<-1E-10):
                print("ERROR: Matrix is NOT positive definite")
                print(self.D)
                print(self.M)
                sys.exit()

    def sample(self,k=None,verbose=False):
        chosen = np.array(super().sample(k,verbose))
        return chosen

class dpp_plane(dpp):
    def __init__(self, lx=10, ly=10, sigma2=1, similarity='soft', bc='PBC'):
        super().__init__()

        data = np.zeros((lx,ly,2))
        for ix in range(lx):
            data[ix,:,0]=ix
        for iy in range(ly):
            data[:,iy,1]=iy
        data=data.reshape(-1,2)
        self.data=data
        self.lenght = lx*ly
        self.lx = lx
        self.ly = ly
        self.M= np.zeros((self.lenght,self.lenght))
        gamma = 1.0/(2.0*sigma2)

        # Compute the matrix with distance-square
        for n1 in range(0,self.lenght):
            self.M[n1,n1]= 1.0
            for n2 in range(n1+1,self.lenght):
                if(bc == 'OBC'):
                    dx = data[n1,0]-data[n2,0]
                    dy = data[n1,1]-data[n2,1]
                elif(bc == 'PBC'):
                    dx =(data[n1,0]-data[n2,0]+self.lx ) % self.lx
                    dx = min(dx, self.lx - dx)
                    dy =(data[n1,1]-data[n2,1]+self.ly ) % self.ly
                    dy = min(dy, self.ly - dy)
                else:
                    print("ERROR: Unknown bc")
                    sys.exit()

                d2=dx*dx+dy*dy

                if(similarity == 'soft'):
                    tmp = np.exp(-gamma*d2)
                elif(similarity == 'hard'):
                    tmp = 0.5*float(d2<sigma2)
                else:
                    print("ERROR: Unknown similarity")
                    sys.exit()

                self.M[n1,n2]= tmp
                self.M[n2,n1]= tmp

        # Diagonalize matrix
        self.D, self.V  = np.linalg.eigh(self.M) # Use this because matrix is symmetric
        for l in self.D:
            if(l<-1E-10):
                print("ERROR: Matrix is NOT positive definite")
                print(self.D)
                print(self.M)
                sys.exit()

    def sample(self,k=None,verbose=False):
        chosen_list = super().sample(k,verbose)
        xy_pair=np.array(self.data[chosen_list])
        return xy_pair

class random():
    def __init__(self):
        self.lenght = None

    def sample(self,k=None) -> list:
        """
        Sample k random locations without replacement
        A list of locations is returned
        """
        chosen=np.random.choice(range(self.lenght),size=k,p=None).tolist()
        return chosen

class random_line(random):
    def __init__(self, lenght=10):
        super().__init__()
        self.lenght = lenght
    def sample(self,k=5):
        chosen = np.array(super().sample(k))
        return chosen

class random_plane(random):
    def __init__(self, lx=10, ly=10):
        super().__init__()
        data = np.zeros((lx,ly,2))
        for ix in range(lx):
            data[ix,:,0]=ix
        for iy in range(ly):
            data[:,iy,1]=iy
        data=data.reshape(-1,2)
        self.data=data
        self.lenght = lx*ly

    def sample(self,k=10):
        chosen_list = super().sample(k)
        xy_pair=np.array(self.data[chosen_list])
        return xy_pair
```

```python
dpp=dpp_line(lenght=20,sigma2=2, similarity='soft', bc='PBC')
xy_dpp=dpp.sample(verbose=False,k=12)
print(xy_dpp)
```

    [17  6  1 15  2 14  0  8 12 10 19  4]

```python
lx = 10
ly = 10
dpp=dpp_plane(lx=lx,ly=ly,sigma2=1, similarity='hard',bc='OBC')

```

```python
xy_dpp=dpp.sample(verbose=False,k=12)
n_points_dpp = xy_dpp.shape[0]
random2=random_plane(lx,ly)
xy_random=random2.sample(n_points_dpp)
n_points_random = xy_random.shape[0]

f, (ax1, ax2) = plt.subplots(1, 2, sharey=True, subplot_kw = {'aspect' : 1.0})
ax1.scatter(xy_dpp[:,0],xy_dpp[:,1])
ax1.set_title('DPP')
ax2.scatter(xy_random[:,0],xy_random[:,1])
ax2.set_title('random without replacement')
print("N_points: dpp vs random",n_points_dpp,n_points_random)
```

```python

```

```python

```


# The diffusion constant

In this notebook we'll measure the diffusion constant by looking at the mean squared displacement of every atom in an Argon gas. That is, we compute the mean squared displacement from
\begin{align}
    \langle r^2(t) \rangle
    = \frac{1}{N} \sum_{i = 1}^{N} \left(
        \mathbf{r}_i(t) - \mathbf{r}_i(t_0)
    \right)^2,
\end{align}
where $t_0$ is the time at the start of the simulation. Computing the displacement at every time step can prove hard due to periodic boundary conditions. Luckily, Lammps can compute the quantity for us!

```python
import os
import sys
import re

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.linear_model

sns.set(color_codes=True)
```

```python
%%writefile scripts/diffusion.in

# 3d Lennard-Jones gas
units lj
dimension 3
timestep 0.005
# Periodic boundiaries
boundary p p p
atom_style atomic

variable sigma equal 3.405
variable b equal 15
variable reduced_density equal 4/((${b}/${sigma})^3)

lattice fcc ${reduced_density}
region simbox block 0 10 0 10 0 10
create_box 1 simbox
create_atoms 1 box

mass 1 1.0
velocity all create ${T} 87287

pair_style lj/cut 3.0
pair_coeff 1 1 1.0 1.0 3.0

fix 1 all nvt temp ${T} ${T} 3.0
#fix 1 all nve

thermo 10
run 5000

reset_timestep 0
variable mytime equal dt*step
compute msd all msd
thermo_style custom step temp time v_mytime press ke pe etotal c_msd[4]

log dat/diffusion_${T}.log

run 10000
```

    Overwriting scripts/diffusion.in

```python
%%writefile scripts/argon_diffusion.in

export OMP_NUM_THREADS=4

for T in $(seq 0.01 0.1 0.9) $(seq 1 21); do
    mpirun -np 4 lmp -var T $T -in scripts/diffusion.in
done
```

    Overwriting scripts/argon_diffusion.in

```python
file_list = list(filter(lambda x: x.startswith("diffusion"), os.listdir("dat")))

pattern = r"diffusion_(.+).log"
temperature_list = [float(re.search(pattern, filename).group(1)) for filename in file_list]

file_list = list(map(lambda x: os.path.join("dat", x), file_list))

temperature_list, file_list = zip(*sorted(zip(temperature_list, file_list)))
```

```python
fig = plt.figure(figsize=(14, 10))

log_df_list = []

for T, filename in zip(temperature_list, file_list):
    log_df = pd.read_csv(filename, sep=r"\s+", skiprows=3, skipfooter=27, engine="python")
    log_df_list.append(log_df)
    plt.plot(
        log_df["v_mytime"],
        log_df["c_msd[4]"],
        label=fr"$T = {T:.2f}$"
    )

plt.xlabel(r"$t$")
plt.ylabel(r"$\langle r^2(t)\rangle$")
plt.legend(loc="best")
plt.title(r"Mean squared displacement as a function of time")
plt.show()
```

In this plot we see the mean squared displacement plotted against time. When $t$ is larger than some threshold $t_t$ we see that the expected linear relationship
\begin{align}
    \langle r^2(t) \rangle = 6 D(T) t,
\end{align}
is achieved. However, when $t$ is close to the initial time $t_0$ we get _anomlous diffusion_, that is, the mean squared displacement is not a linear function of time. As long as $t < t_t$ the mean squared displacement experiences _superdiffusion_, that is,
\begin{align}
    \langle r^2 (t) \rangle \propto t^{\alpha},
\end{align}
where $\alpha > 1$. As it is hard to determine _exactly_ where $t_t$ is located, we will look at $t \gg t_t$ for the linear diffusion and $t \ll t_t$ for the superdiffusion.

```python
t_t = 20 # We define this to be the transition from anomalous to linear diffusion
```

```python
D_list = []
alpha_list = []

for T, log_df in zip(temperature_list, log_df_list):
    linear_df = log_df[log_df["v_mytime"] > t_t * 2]
    time = np.array(linear_df["v_mytime"])
    msd = np.array(linear_df["c_msd[4]"])

    clf = sklearn.linear_model.LinearRegression().fit(
        time[:, np.newaxis], msd[:, np.newaxis]
    )
    D_list.append(clf.coef_[0, 0] / 6)

    super_df = log_df[(log_df["v_mytime"] < t_t / 2) & (log_df["v_mytime"] > 0)]
    time = np.log(super_df["v_mytime"])
    msd = np.log(super_df["c_msd[4]"])

    clf = sklearn.linear_model.LinearRegression().fit(
        time[:, np.newaxis], msd[:, np.newaxis]
    )
    alpha_list.append(clf.coef_[0, 0])
```

```python
fig = plt.figure(figsize=(14, 10))

plt.plot(temperature_list, D_list)
plt.title(r"Plot of the diffusion constant as a function of temperature")
plt.xlabel(r"$T$")
plt.ylabel(r"$D(t)$")
plt.show()
```

In this figure we can see how the diffusion constant follows a linear relationship with the temperature $T$ after the anomalous diffusion period.


# Welcome to QC Hackathon 2021!

Quantum Computation is an up and coming field that uses quantum gates to perform operations which exploit properties of a quantum state like superposition, and entanglement to perform computations. The aim is to solve computation problems in executable time that might not be solvable using its classical analogue. In this hackathon, we will be introducing you to challenges of different difficulty levels.

A good introduction to all of these concepts can be found in the [qiskit textbook](https://qiskit.org/textbook/preface.html). We will be using qiskit in this hackathon.

# Challenge Tier - I
For this first set of challenges, we encourage you to read [Section 1.4](https://qiskit.org/textbook/ch-states/single-qubit-gates.html) and [Section 2.2](https://qiskit.org/textbook/ch-gates/multiple-qubits-entangled-states.html) to get familiar with the basics of quantum computing.

```python
#before we start, run this cell to install qiskit
!pip install qiskit
!pip install pylatexenc
```

## Task 1
$$\newcommand{\ket}[1]{\left|{#1}\right\rangle}$$
$$\newcommand{\bra}[1]{\left\langle{#1}\right|}$$

An $H$ gate, or a Hadamard gate, is a single qubit gate which performs the following operations:

$\begin{align}
  H\ket{0} &= \dfrac{\ket{0}+\ket{1}}{\sqrt{2}}\\
  H\ket{1} &= \dfrac{\ket{0}-\ket{1}}{\sqrt{2}}
\end{align}$

A $Z$ gate, similarly, performs the following operations:

$\begin{align}
  Z\ket{0} &= \ket{0}\\
  Z\ket{1} &= -\ket{1}
\end{align}$

In this challenge, you will be given a single bit unitary operation that can either be an $H$ gate or a $Z$ gate. You can apply the unitary operation **at most twice**. Using what you learnt from [Section 1.4](https://qiskit.org/textbook/ch-states/single-qubit-gates.html), distinguish whether the given operation is an H gate or a Z gate.

**Output:** 0 if it is the $Z$ gate, 1 if it is the $H$ gate.

```python
from qiskit import *

def A1_solve(Unitary, qc):
  ##########################################
  #enter your code here
  #To apply the unitary to the qubit, use syntax: Unitary(0)

  Unitary(0)
  qc.z(0)
  Unitary(0)

  #Expected Output: {'0': 1024} if Z gate; and
  #                 {'1': 1024} if H gate

  ###########################################
  qc.measure(0,0)
  print(qc.draw())

  backend = Aer.get_backend('qasm_simulator')
  job = execute(qc, backend)
  counts = job.result().get_counts()

  return counts

import random
qr = QuantumRegister(1)
cr = ClassicalRegister(1)
A1_qc = QuantumCircuit(qr, cr)
if __name__ == "__main__":
  if random.uniform(0,1) < 0.5:
    Unitary = A1_qc.z
  else:
    Unitary = A1_qc.h

  _output = A1_solve(Unitary, A1_qc)
  print(_output)
```

             ┌───┐┌───┐┌───┐┌─┐
    q2171_0: ┤ Z ├┤ Z ├┤ Z ├┤M├
             └───┘└───┘└───┘└╥┘
      c11: 1/════════════════╩═
                             0
    {'0': 1024}

## Task 2
Any quantum state can be written in the form of a superposition of the $\ket{0}$ and $\ket{1}$ states in the dirac notation. So any single qubit quantum state, $\ket{\psi}$, can be written as:
$\begin{equation}\ket{\psi} = \alpha \ket{0} + \beta \ket{1} \end{equation}$ where the squares of the coefficients of $\ket{0}$ and $\ket{1}$ are equal to the respective probability of the output on measuring $\ket{\psi}$.

Similarly, two qubit states can be described by: $\displaystyle \begin{equation}\ket{\psi} = \sum_{j, k\in\{0,1\}} \alpha_{jk}\ket{j k} \end{equation}$

Using simple quantum gates like the ones you have learnt in [Section 1.4](https://qiskit.org/textbook/ch-states/single-qubit-gates.html) and [Section 2.2](https://qiskit.org/textbook/ch-gates/multiple-qubits-entangled-states.html), prepare the superposition state described by:
$\begin{align} \ket{\psi} = \dfrac{1}{2}\ket{01} + \dfrac{1}{2}\ket{10} + \dfrac{1}{\sqrt{2}}\ket{00} \end{align}$

```python
from qiskit import *
import math

def A2_solve():

  #enter your code here, specify the qubits in QuantumCircuit
  qc = QuantumCircuit(2, 2)
  qc.u(math.pi/3, 0, 0, 1)
  qc.x(1)
  qc.cu(2 * math.asin(1 / 3 ** 0.5), 0, 0, 0, 1, 0)
  qc.x(1)
  qc.measure([0, 1],[0, 1])
  print(qc.draw())

  backend = Aer.get_backend('qasm_simulator')
  job = execute(qc, backend)
  counts = job.result().get_counts()

  print(counts)
  return

if __name__ == "__main__":
  A2_solve()

```

                            ┌────────────────┐     ┌─┐
    q_0: ───────────────────┤ U(1.231,0,0,0) ├─────┤M├───
         ┌────────────┐┌───┐└───────┬────────┘┌───┐└╥┘┌─┐
    q_1: ┤ U(π/3,0,0) ├┤ X ├────────■─────────┤ X ├─╫─┤M├
         └────────────┘└───┘                  └───┘ ║ └╥┘
    c: 2/═══════════════════════════════════════════╩══╩═
                                                    0  1
    {'00': 528, '01': 251, '10': 245}

## Task 3

A full subtractor is a combinational circuit that performs subtraction of two bits in classical computation. A thorough explanation and implementation of the circuit can be found here: [Full Subtractor Circuit in digital logic](https://www.geeksforgeeks.org/full-subtractor-in-digital-logic/). Throughout this challenge you may have realised that the quantum gates are very similar to classical gates, and the elementary quantum gates can be used in place of their classical analogues in a quantum circuit.

Your task is to implement a full subtractor circuit using only quantum gates.  You can take help from the resource mentioned above, and try to replace the classical gates with their quantum analogues. In more precise terms, you must reproduce the results of the truth table that can be found in the link above.

**Input:** Three values: $A$, $B$, $B_{in}$; that correspond to the truth table.

**Output:** Two values: $D$, $B_{out}$ (You must measure two qubits in your circuit)

There is a bonus point for using the minimum number of qubits.

```python
from qiskit import *

def A3_solve(inputBits):
  no_of_qubits_used = 5  #Enter the number of qubits you require
  c = ClassicalRegister(2, "c")
#   A = QuantumRegister(1, "q_A")
#   B = QuantumRegister(1, "q_B")
#   B_in = QuantumRegister(1, "q_B_in")
  q = QuantumRegister(no_of_qubits_used, "q")
  qc = QuantumCircuit(q, c)
  for i in range(0, len(inputBits)):
    if inputBits[i] == 1:
      qc.x(i)
  #Enter code from here:

  # D = (A XOR B) XOR B_in

  qc.cx(1, 3)
  qc.cx(0, 3)
  # q_3 has (A XOR B)

  qc.cx(2, 4)
  qc.cx(3, 4)
  # q_4 has D
  qc.measure(4, 0)
  qc.barrier()

  # initialize q_4 to 0
  qc.x(4).c_if(c, 1)
  qc.x(4).c_if(c, 3)

  # B_out = B_in (A XOR B)' + A'B

  qc.x(3)
  qc.ccx(2, 3, 4)
  # q_4 has B_in (A XOR B)'

  # initialize q_3 to 0
  qc.measure(3, 1)
  qc.x(3).c_if(c, 2)
  qc.x(3).c_if(c, 3)

  qc.x(0)
  qc.ccx(0, 1, 3)
  # q_3 has A'B

  # initialize q_2 to 0
  qc.measure(2, 1)
  qc.x(2).c_if(c, 2)
  qc.x(2).c_if(c, 3)

  qc.cx(3, 2)
  qc.cx(4, 2)
  qc.ccx(3, 4, 2)
  #q_2 has B_out

  qc.measure(2, 1)

  ################### Tests #####################################
  print(qc.draw())
  backend = Aer.get_backend('qasm_simulator')
  job = execute(qc, backend)
  counts = job.result().get_counts()
  outputList = list(counts.keys())
  if len(outputList) != 1 or len(outputList[0][0]) > 2:
    print('Error: There are too many outputs.')
    return
  output = outputList[0]
  if len(output) != 2:
    print('Error: There are too few outputs. Please make sure you are *measuring* the appropriate qubits in your circuit. Use qc.measure() method to measure the qubits.')
    return

  return output[1], output[0]

if __name__ == "__main__":
  A = 1
  B = 0
  B_in = 1
  _D, _Bout = A3_solve([A, B, B_in])
  print(_D, _Bout)
  #You need to check whether these are consistent with the truth table yourself.
```

         ┌───┐                        ░  ┌───┐                                   »
    q_0: ┤ X ├───────■────────────────░──┤ X ├────────────────────────────────■──»
         └───┘       │                ░  └───┘                                │  »
    q_1: ───────■────┼────────────────░───────────────────────────────────────■──»
         ┌───┐  │    │                ░                                       │  »
    q_2: ┤ X ├──┼────┼────■───────────░─────────────────■─────────────────────┼──»
         └───┘┌─┴─┐┌─┴─┐  │           ░  ┌───┐          │  ┌─┐ ┌───┐  ┌───┐ ┌─┴─┐»
    q_3: ─────┤ X ├┤ X ├──┼────■──────░──┤ X ├──────────■──┤M├─┤ X ├──┤ X ├─┤ X ├»
              └───┘└───┘┌─┴─┐┌─┴─┐┌─┐ ░  ├───┤  ┌───┐ ┌─┴─┐└╥┘ └─╥─┘  └─╥─┘ └───┘»
    q_4: ───────────────┤ X ├┤ X ├┤M├─░──┤ X ├──┤ X ├─┤ X ├─╫────╫──────╫────────»
                        └───┘└───┘└╥┘ ░  └─╥─┘  └─╥─┘ └───┘ ║    ║      ║        »
                                   ║    ┌──╨──┐┌──╨──┐      ║ ┌──╨──┐┌──╨──┐     »
    c: 2/══════════════════════════╩════╡ = 1 ╞╡ = 3 ╞══════╩═╡ = 2 ╞╡ = 3 ╞═════»
                                   0    └─────┘└─────┘      1 └─────┘└─────┘     »
    «
    «q_0: ───────────────────────────────────
    «
    «q_1: ───────────────────────────────────
    «     ┌─┐ ┌───┐  ┌───┐ ┌───┐┌───┐┌───┐┌─┐
    «q_2: ┤M├─┤ X ├──┤ X ├─┤ X ├┤ X ├┤ X ├┤M├
    «     └╥┘ └─╥─┘  └─╥─┘ └─┬─┘└─┬─┘└─┬─┘└╥┘
    «q_3: ─╫────╫──────╫─────■────┼────■───╫─
    «      ║    ║      ║          │    │   ║
    «q_4: ─╫────╫──────╫──────────■────■───╫─
    «      ║ ┌──╨──┐┌──╨──┐                ║
    «c: 2/═╩═╡ = 2 ╞╡ = 3 ╞════════════════╩═
    «      1 └─────┘└─────┘                1
    0 0

## Additional information

**Created by:** Yashee Sinha


# Non Linear Regression Analysis

Estimated time needed: **20** minutes

## Objectives

After completing this lab you will be able to:

*   Differentiate between linear and non-linear regression
*   Use non-linear regression model in Python

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression since linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example in python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Although linear regression can do a great job at modeling some datasets, it cannot be used for all datasets. First recall how linear regression, models a dataset. It models the linear relationship between a dependent variable y and the independent variables x. It has a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

Non-linear regression is a method to model the non-linear relationship between the independent variables $x$ and the dependent variable $y$. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).  For example:

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and so on. For example: $$ y = \log(x)$$

We can have a function that's even more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying the logarithmic map from the input $x$ to the output $y$. It is one of the simplest form of **log()**: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be a polynomial representation of the $x$ values. In general form it would be written as\
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2021-06-16 17:09:21 URL:https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta\_1(X-\beta\_2)}}$$

$\beta\_1$: Controls the curve's steepness,

$\beta\_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, \*popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.451715, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

Can you calculate what is the accuracy of our model?

```python
# write your code here

```

Click here for the solution

```python
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

### Thank you for completing this lab!

## Author

Saeed Aghabozorgi

### Other Contributors

Joseph Santarcangelo

## Change Log

|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |
|---|---|---|---|
| 2020-11-03  | 2.1 | Lakshmi |  Made changes in URL |
| 2020-08-27  | 2.0  | Lavanya  |  Moved lab to course repo in GitLab |
|   |   |   |   |
|   |   |   |   |

##
 © IBM Corporation 2020. All rights reserved.


# 10 Ordinary differential equations – Integrators
More about integrators to solve ODEs.

* Euler's Rule (and it's shortcomings)
* Runge-Kutta methods

## Euler's Rule

From the last lecture: Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{h}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + h \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

Euler's rule cannot obtain high precision without very small $h$ but this leads to higher round-off error accumulation.

### Euler's Rule applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
y^{(0)}_1 &= y^{(0)}_0 + h y^{(1)}_0 = x_0 + v_0 h\\
y^{(1)}_1 &= y^{(1)}_0 + h \left(-\frac{k}{m}y^{(0)}_0\right) = v_0 + \frac{-k x_0}{m} h
\end{align}

Compare to the equations of movement under a constant force (*ballistic motion*):

$$
x = x_0 + v_0 h + \frac{1}{2} a h^2, \quad v = v_0 + a h
$$

In Euler, the position does not even contain the $h^2$ term, i.e., the acceleration only contributes *indirectly* via the velocity of the *next* step. (Remember: Euler's error is $\mathcal{O}(h^2)$!)

### Euler's Rule: Application to 3 oscillators (numerical)
`git pull` new resources.

Put the notebook `10-ODE_integrators-students.ipynb` *and* the module `integrators.py` into the same directory, for instance a work directory `~/PHY494/10_ODEs`.

Open [integrators.py](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/10_ODEs/integrators.py) to have a look at it: it contains code to integrate Newton's equations of motions for different forces and with different integrators (based on the work done in the last lecture).

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators_solution as integrators
```

```python
t_max = 20
t_harm, y_harm = integrators.integrate_newton(t_max=t_max)
t_anharm, y_anharm = integrators.integrate_newton(t_max=t_max,
                                                  force=integrators.F_anharmonic)
t_power, y_power = integrators.integrate_newton(t_max=t_max,
                                                force=integrators.F_power)
```

```python
plt.plot(t_harm, y_harm[:, 0], label=r"harmonic $\frac{k}{2} x^2$")
plt.plot(t_anharm, y_anharm[:, 0], label=r"anharmonic $\frac{k}{2} x^2(1 - \frac{2}{3} \alpha x)$")
plt.plot(t_power, y_power[:, 0], label=r"power $\frac{k}{6} x^6$")
plt.legend(loc="best")
```

#### Phase-space portrait
Plot position against momentum ($x, p$):

```python
m = 1
x = y_harm[:, 0]
p = m * y_harm[:, 1]
ax = plt.subplot(1,1,1)
ax.plot(x, p)
ax.set_aspect(1)
```

```python
t, y = integrators.integrate_newton(t_max=40,
                                    force=integrators.F_harmonic,
                                    h=0.01)
t2, y2 = integrators.integrate_newton(t_max=40, h=0.001,
                                      force=integrators.F_harmonic)
```

```python
ax = plt.subplot(1,1,1)
ax.plot(y[:, 0], m * y[:, 1], 'b-', label=r"$h=0.01$")
ax.plot(y2[:, 0], m * y2[:, 1], 'r--', label="$h=0.001$")
ax.set_xlabel("position $x$")
ax.set_ylabel("momentum $p$")
ax.set_aspect(1)
ax.legend(loc="best")
```

#### Energy
Energy is conserved: stringent test for the integrator!

$$
E = T_\text{kin} + U\\
\frac{dE}{dt} = 0
$$
with $T_\text{kin} = \frac{1}{2} m v^2$.

Calculate and plot the three energy terms:

```python
KE = 0.5 * m * y_harm[:, 1]**2
PE = integrators.U_harmonic(y_harm[:, 0])
energy = KE + PE
plt.plot(t_harm, KE, 'r-', t_harm, PE, 'b-', t_harm, energy, 'k--')
```

Harmonic oscillator with larger timestep $h=0.01$

```python
m = 1
t, y = integrators.integrate_newton(t_max=500, force=integrators.F_harmonic, h=0.01, mass=m)
x, v = y.T
KE = 1/2 * m * v**2
PE = integrators.U_harmonic(x)
energy = KE + PE
```

```python
ax = plt.subplot(1,1,1)
ax.plot(t, KE, 'r-', t, PE, 'b-', t, energy, 'k--')
ax.set_xlim(0, 20)
ax.set_ylim(0, 0.7)
ax.set_xlabel("time")
ax.set_ylabel("energy")
ax.set_title("Euler: harmonic oscillator")
```

Note how energy conservation becomes worse when $h$ is increased.

For longer time scales, the energy just keeps on increasing — clearly, we are not solving the equations of motions in a physically meaningful way.

```python
ax.set_ylim(0, 80)
ax.set_xlim(0,500)
ax.figure
```

##### Quantify energy drift
Measure the relative departure from energy conservation

$$
\epsilon_E(t) = \log_{10}\left|\frac{E(t) - E(t=0)}{E(t=0)}\right|
$$

(By using the decadic logarithm, $-\epsilon_E$ is a direct measure of the precision.)

###### Harmonic oscillator
Analyze energy drift for the last harmonic oscillator example ($h=0.01$):

```python
energy_precision = np.log10(energy/energy[0] - 1)
plt.plot(t, energy_precision)
plt.ylabel("log(relative error in $E$)")
plt.xlabel("time")
```

As we already know, with such a large time step, energy conservation is very bad with the *Euler* integrator: The relative (!) error quickly becomes 100% (after $t=200$).

##### Code re-use: energy analysis as a function
The module `integrators` contains the function `integrators.analyze_energies(t, y, U)` to automate the task of analysing the energy. You *must* provide the appropriate potential energy function `U`.

##### Problem: Make `analyze_energies()` work and analyze the power law oscillator
The code is *incomplete*.
1. Open `integrators.py` in your editor.
2. Complete the `kinetic_energy()` function (remember: $T_\text{kin} = \frac{1}{2} m v^2$)

Then `reload()` the module:

```python
# when working on the integrator module you need to reload() whenever you make changes
from importlib import reload   # for Python 3
reload(integrators);
```

and try it out for power law oscillator with a large time step $h = 0.01$. Can you find a value of $h$ that will give you energy precision to 4th decimal over 40 time units?

###### Energy stability analysis of the 6-th power oscillator
$x^6$ potential with $h=0.01$:

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.01)
```

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

Trying 100 times smaller $h$ (takes 100 times longer!!)

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.0001)
integrators.analyze_energies(t, y, integrators.U_power)
```

*Reducing $h$ further does not seem to help and takes order of magnitude longer to run*. **We need a better integrator!**

## Runge-Kutta methods
Explicit RK methods propagate with the general form

$$
y_{n+1} = y_n + h \sum_{i=1}^s b_i k_i, \quad t_{n+1} = t_n + h
$$

where $s$ is the number of *stages* and the coefficients $a_{ij}$, $b_i$ and $c_i$ define the method.

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + c_2 h, y_n + h(a_{21} k_1))\\
k_3 &= f(t_n + c_3 h, y_n + h(a_{31} k_1 + a_{32} k_2))\\
 \vdots& \\
k_s &= f(t_n + c_s h, y_n + h(a_{s1} k_1 + a_{s2} k_2 + \cdots + a_{s,s-1} k_{s-1})).
\end{align}

Basic idea: improve estimate for the slope by using weighted averages of slopes over the interval.

### RK4
General purpose ODE solver.

$$
y_{n+1} = y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big), \quad t_{n+1} = t_n + h
$$

and

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

`rk4` can obtain $\mathcal{O}(h^4)$ precision.

### RK2
`rk2` is simpler to derive and understand than `rk4` (which we will use more often).

Start with the exact solution

$$
\frac{dy}{dt} = f(t, y) \quad\text{and}\quad y(t) = \int f(t, y)dt\\
y_{n+1} = y_n + \int_{t_n}^{t_{n+1}} f(t, y)dt
$$

Expand $f(t, y)$ around *midpoint* $t_{n+1/2} = t + \frac{h}{2}$ of integration interval

\begin{gather}
f(t, y) \approx f(t_{n+1/2}, y_{n+1/2}) + (t - t_{n+1/2})\,  f^{(1)}(t_{n+1/2}) +
   \mathcal{O}(h^2)
\end{gather}

Integrate the *expansion* (Taylor series around midpoint):
\begin{align}
\int_{t_n}^{t_{n+1}} f(t, y) \, dt &\approx h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)\\
y_{n+1} &= y_n + h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)
\end{align}

I.e., use **slope at midpoint**! (The $\mathcal{O}(h^2)$ term in the integral is zero because $[(h/2)^2 - (h/2)^2] = 0$.)

But we don't have $y_{n+1/2}$ so we use Euler's rule to estimate it to $\mathcal{O}(h^2)$:

$$
y_{n+1/2} \approx y_n + \frac{1}{2} h\, f(t_n, y_n)
$$

and insert into $y_{n+1} = y_n + h\, f(t_{n+1/2}, y_{n+1/2})$ (so it is still $\mathcal{O}(h^3)$ accurate).

Altogether: **RK2** algorithm

$$
\mathbf{y}_{n+1} = \mathbf{y}_n + h \mathbf{k}_2, \quad t_{n+1} = t_n + h
$$

and

\begin{align}
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

with $\mathcal{O}(h^3)$ accuracy.

#### RK2 applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
  y^{(0)}_{1} &=
    y^{(0)} _{0} + hf^{(0)} \Big(\frac{h}{2}, y^{(0)}_{0} + hk_{1} \Big)\\
    &= x_{0} + h\left[v_{0} + \frac{h}{2} \left(-\frac{k}{m} x_{0}\right)  \right]\\
    &= x_0 + v_0 h + \frac{1}{2} \left(-\frac{k}{m} x_{0}\right) h^2\\
  y^{(1)}_{1} &= y^{(1)}_{0} +
                h f^{(1)}\left(\frac{h}{2},
                     \mathbf{y}_{0} + \frac{h}{2} \mathbf{f}(0, \mathbf{y}_{0})\right) \\
             &= v_{0} + \frac{h}{m} \left[-k\left(x_{0} - \frac{h}{2} k x_{0}\right)\right]
\end{align}

Note that `rk2` contains a $h^2$ term in the position update.

#### RK2 applied in practice
Let's implement `rk2`

\begin{align}
\mathbf{y}_{n+1} &= \mathbf{y}_n + h \mathbf{k}_2\\
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

in our `integrators.py`.

```python
from importlib import reload
reload(integrators);
```

##### RK2 for the power potential

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk2, h=0.01)
```

###### phase-space portrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

#### RK4 applied in practice
Implement `rk4` in `integrator.py`:

\begin{align}
y_{n+1} &= y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big)\\
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

```python
reload(integrators)
```

    <module 'integrators_solution' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2018/PHY494-resources/10_ODEs/integrators_solution.py'>

and test the 6th power potential again. What level of energy precision can you obtain over 40 time units?

##### RK4 for the 6th power potential
Note that we can easily run `rk4` with a $h=0.01$!

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.01)
```

###### phase-space protrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

You should be able to get 8 - 11 decimals precision from `rk4` with a smaller $h$.

##### High precision RK4 for the 6th power potential
Try a 10 times smaller step $h=0.001$

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.001)
integrators.analyze_energies(t, y, integrators.U_power)
```

This looks like stable integration with an error of not worse than $10^{-12}$: *Looking very good!* (Especially when you compare it to *Euler* with $h=0.001$!)

It looks as if the main source of error comes from the force evaluation because the error spikes when the potential energy is dominant.


# Adams Predictor Corrector
The Adams Predictor corrector method is used to side step the issue of the Adams-Moulton method being implicit. It can also be used to estimate error when the solution is unknown.

```python
from IPython.display import HTML
HTML('')
```

## Intial Value Problem

The differential equation
\begin{equation}y^{'}=t-y, \ \ (0 \leq t \leq 2) \end{equation}
with the initial condition
\begin{equation}y(0)=1\end{equation}
will be used to illustrate the method.

## Python Libraries

```python
import numpy as np
import math
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

### Defining the function
\begin{equation} f(t,y)=t-y.$$

```python
def myfun_ty(t,y):
    return t-y
```

## Discrete Interval
Defining the step size  $h$  from the interval range  $a\leq t \leq b$  and number of steps  $N$
\begin{equation}h=\frac{b−a}{N}.\end{equation}

This gives the discrete time steps,
\begin{equation}t_i=t_0+ih,\end{equation}
where  $t0=a.$

Here the interval is  $0≤t≤2$  and number of step  4
\begin{equation}h=\frac{2−0}{4}=0.5.\end{equation}

This gives the discrete time steps,
\begin{equation}t_i=0+i0.5,\end{equation}
for  $i=0,1,⋯,4.$

```python
# Start and end of interval
b=2
a=0
# Step size
N=4
h=(b-a)/(N)
t=np.arange(a,b+h,h)
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.xlim((0,2))
plt.title('Illustration of discrete time points for h=%s'%(h))
```

## Exact Solution
The initial value problem has the exact solution
\begin{equation}y(t)=2e^{-t}+t-1.\end{equation}
The figure below plots the exact solution.

```python
IC=1 # Intial condtion
y=(IC+1)*np.exp(-t)+t-1
fig = plt.figure(figsize=(6,4))
plt.plot(t,y,'o-',color='black')
plt.title('Exact Solution ')
plt.xlabel('time')
```

```python
# Initial Condition
w=np.zeros(N+1)
#np.zeros(N+1)
w[0]=IC
```

## 2-step Adams Bashforth (Predictor)

The 2-step Adams Bashforth difference equation is
\begin{equation}w^{0}_{i+1} = w_{i} + \frac{h}{2}(3f(t_i,w_i)-f(t_{i-1},w_{i-1})) \end{equation}
\begin{equation}w^{0}_{i+1} = w_{i} + \frac{h}{2}(3(t_i-w_i)-(t_{i-1}-w_{i-1}))\end{equation}
## 1-step Adams Moulton (Corrector)
\begin{equation}w^{1}_{i+1} = w_{i} + \frac{h}{2}(f(t_{i+1},w^{0}_{i+1})+f(t_{i},w_{i})) \end{equation}

\begin{equation} w^{1}_{i+1} = w_{i} + \frac{h}{2}((t_{i+1}-w^0_{i+1})+(t_{i}-w_{i})) \end{equation}

For $i=0$ the system of difference equation is:
\begin{equation}w^{0}_{1} = w_{0} + \frac{h}{2}(3(t_0-w_0)-(t_{-1}-w_{-1})) \end{equation}
\begin{equation}w_{1} = w_{0} + \frac{h}{2}((t_{1}-w^{0}_{1})+(t_{0}-w_{0})). \end{equation}
this is not solvable as  $w_{-1}$  is unknown.

For $i=1$ the difference equation is:
\begin{equation}w^{0}_{2} = w_{1} + \frac{h}{2}(3(t_1-w_1)-(t_{0}-w_{0})) \end{equation}
\begin{equation}w_{2} = w_{1} + \frac{h}{2}((t_{2}-w^{0}_{2})+(t_{1}-w_{1})). \end{equation}
this is not solvable as  $w_{1}$   is unknown. $w_1$ can be  approximated using a one step method. Here, as the exact solution is known,
\begin{equation}w_1=2e^{-t_1}+t_1-1.$$

```python
### Initial conditions
w=np.zeros(len(t))
w0=np.zeros(len(t))
w[0]=IC
w[1]=y[1]
w0[0]=IC
w0[1]=y[1]
```

### Loop

```python
for k in range (1,N):
    w0[k+1]=(w[k]+h/2.0*(3*myfun_ty(t[k],w[k])-myfun_ty(t[k-1],w[k-1])))
    w[k+1]=(w[k]+h/2.0*(myfun_ty(t[k+1],w0[k+1])+myfun_ty(t[k],w[k])))
```

### Plotting solution

```python
def plotting(t,w,w0,y):
    fig = plt.figure(figsize=(10,4))
    plt.plot(t,y, 'o-',color='black',label='Exact')
    plt.plot(t,w0,'v:',color='blue',label='Adams-Bashforth Predictor')
    plt.plot(t,w,'^:',color='red',label='Adams-Moulton Corrector')
    plt.xlabel('time')
    plt.legend()
    plt.show
```

The plot below shows the exact solution (black) and the Adams Predictor Corrector approximation (red) of the intial value problem

```python
plotting(t,w,w0,y)
```

## Local Error
The error for the 1 step Adams Moulton is:
\begin{equation}\frac{y_{n+1}-y_{n}}{h}=\frac{1}{2}[f(t_{n+1},w_{n+1})+f(t_{n-1},w_{n-1})] +\frac{-h^2}{12}y^{(3)}(\eta),\end{equation}
where $\eta \in [t_{n-1},t_{n+1}]$.
The error for the 2 step Adams Bashforth is:
\begin{equation}\frac{y_{n+1}-y_{n}}{h}=\frac{1}{2}[3f(t_{n},w_{n})-f(t_{n},w_{n})] +\frac{5h^2}{12}y^{(3)}(\xi),\end{equation}
where $\xi \in [t_{n-1},t_{n+1}]$.

Rearranging the equations gives
\begin{equation}\frac{w_{n+1}-w_{n+1}^0}{h}= h^2[\frac{y^{(3)}(\eta)}{12}+\frac{5y^{(3)}(\xi)}{12}]\approx h^2[\frac{6}{12}y^{(3)}(\xi)].\end{equation}
Making the assumption that
\begin{equation}y^{(3)}(\xi)\approx \frac{12}{6h^2}\frac{w_{n+1}-w_{n+1}^0}{h},\end{equation}
\begin{equation}y^{(3)}(\xi)\approx \frac{12}{6h^2}\frac{w_{n+1}-w_{n+1}^0}{h}.\end{equation}

```python

d = {'time t_i': t, 'Adams Predictor w0': w0,
     'Adams Corrector':w,'Exact (y)':y,'|w-y|':np.round(np.abs(y-w),5),'|w0-w|':np.round(np.abs(w0-w),5),'estimate LTE':np.round(1/(6*h)*abs(w0-w),5)}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Adams Predictor w0
      Adams Corrector
      Exact (y)
      |w-y|
      |w0-w|
      estimate LTE

      0
      0.0
      1.000000
      1.000000
      1.000000
      0.00000
      0.00000
      0.00000

      1
      0.5
      0.713061
      0.713061
      0.713061
      0.00000
      0.00000
      0.00000

      2
      1.0
      0.803265
      0.708980
      0.735759
      0.02678
      0.09429
      0.03143

      3
      1.5
      0.980510
      0.911607
      0.946260
      0.03465
      0.06890
      0.02297

      4
      2.0
      1.280147
      1.238669
      1.270671
      0.03200
      0.04148
      0.01383

```python

```

Chapter 04 - Multistep Methods/405_Adams Predictor Corrector Example.ipynb


[](https://colab.research.google.com/github/comp-hci-lab/BDSI_2021_ML/blob/master/9%20-%20Deep%20Learning.ipynb)

## BDSI ML 2021

# 9 - Deep Learning

Tutorial [optional]: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

We will implement a (shallow) neural network i.e. logistic regression in pytorch. Recall the loss function for logistic regression with $y \in \{-1, +1\}$ is

\begin{equation}
J(\bar\theta) = \frac{1}{N} \sum_{i=1}^{N} \ln(1+e^{-y^{(i)}\bar{\theta}\cdot \bar{x}^{(i)}})
\end{equation}

**Note:** If you wish to use GPU, please select from the menu "Runtime - Change runtime type" and set "Hardware accelerator" to "GPU".

```python
#@title Run this cell to download preprocessed data (features + labels). { display-mode: "form" }
!pip install -U wget
!rm -rf preprocessed
!mkdir preprocessed

import wget
wget.download('https://github.com/comp-hci-lab/BDSI_2021_ML/raw/master/preprocessed/data.npz', 'preprocessed/data.npz')

!mkdir -p checkpoint
```

```python
import numpy as np
import pandas as pd
from tqdm import tqdm
import matplotlib.pyplot as plt
from sklearn import metrics, exceptions
import os, random
```

```python
#@title Run this cell to run preprocessing. { display-mode: "form" }

with np.load('preprocessed/data.npz') as f:
    X = f['X']
    y = f['y']
    feature_names = f['feature_names']

def impute_missing_values(X):
    """
    For each feature column, impute missing values  (np.nan) with the
    population mean for that feature.

    Args:
        X: np.array, shape (N, d). X could contain missing values
    Returns:
        X: np.array, shape (N, d). X does not contain any missing values
    """
    from sklearn.impute import SimpleImputer
    return SimpleImputer().fit_transform(X)

def normalize_feature_matrix(X):
    """
    For each feature column, normalize all values to range [0, 1].

    Args:
        X: np.array, shape (N, d).
    Returns:
        X: np.array, shape (N, d). Values are normalized per column.
    """
    from sklearn.preprocessing import MinMaxScaler
    return MinMaxScaler().fit_transform(X)

X = impute_missing_values(X)
X = normalize_feature_matrix(X)
```

```python
print('Data Shapes:', X.shape, y.shape)
```

```python
N, d = X.shape
```

```python
# hyperparameters and other random stuff...
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('using device:', device)

torch.random.manual_seed(0)
np.random.seed(0)
random.seed(0)

batch_size = 64
n_epochs = 500
learning_rate = 1e-3
```

## (A) Data loading & batching

- Reference: **Section 1** of https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#loading-and-normalizing-cifar10
- Documentation: `Dataset` and `DataLoader` classes at https://pytorch.org/docs/stable/data.html

TODOs:
1. Create a subclass of `Dataset` named `SimpleDataset`, which is a wrapper for our dataset.
     - The subclass should override `__len__`, that provides the size of the dataset, and `__getitem__`, supporting integer indexing in range from `0` to `len(self)` exclusive.
     - This can be used to create a "virtual dataset" that dynamically loads data from the disk, and is extremely useful if we have BIG data that don't fit in the memory.
     - For our case, just store the data in the initializer and query the corresponding element in `__getitem__`.
     - `__getitem__` should return the data as a `torch.Tensor` with approapriate `dtype`.
2. Create a `SimpleDataset` instance for each of train/val/test sets.
3. Use the dataset to create a `DataLoader`.
    - The data loader provides an "iterator" that allows us to iterate through the dataset in batches, and also handles shuffling at the end of each epoch.
    - Usage example:
    ```
    for i, (X, y) in enumerate(data_loader):
        # i is the current batch number
        # X and y are the features and labels for a batch of examples
    ```

```python
# Split data into train (70%), validation (15%) and test (15%)
from sklearn.model_selection import train_test_split
Xtr, X__, ytr, y__ = train_test_split(X,   y,   train_size=0.7, stratify=y,   random_state=0)
Xva, Xte, yva, yte = train_test_split(X__, y__, test_size=0.5, stratify=y__, random_state=0)
```

```python
import torch
from torch.utils.data import Dataset, DataLoader

class SimpleDataset(Dataset):
    def __init__(self, X, y):
        pass

    def __getitem__(self, idx):
        pass

    def __len__(self):
        pass
```

```python
#@title TEST CASE. { display-mode: "form" }
d_ = SimpleDataset(
    np.array([[1,2,3],[4,5,6],[7,8,9]]),
    np.array([0,1,0]),
)
assert len(d_) == 3
assert (d_[0][0] == torch.tensor([1,2,3]).float()).all()
assert (d_[0][1] == torch.tensor(0).float()).all()
del d_
```

```python
# Define datasets and data loaders for training, validation, and test set
batch_size = 64

# Shuffle for train set
tr = SimpleDataset(Xtr, ytr)
tr_loader = DataLoader(tr, batch_size=batch_size, shuffle=True)

# Do not shuffle for validation set
va =
va_loader =

# Do not shuffle for test set
te =
te_loader =
```

## (B) Architecture definition

- Reference: **Section 2** of https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#define-a-convolutional-neural-network
- Documentation: `nn.Module` at https://pytorch.org/docs/stable/nn.html

TODOs:
1. Create a single layer neural network with sigmoid activation.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class Net(nn.Module):
    def __init__(self, input_size):
        super().__init__()
        self.fc =

    def forward(self, x):
        z =
        return z
```

```python
#@title TEST CASE { display-mode: "form" }
m_ = Net(4)
x_ = torch.tensor([[1,2,3,4], [0,1,2,3]]).float()
out = m_(x_)
del m_, x_
```

```python
# Create our logistic regression model
model = Net(d)
```

```python
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print('Number of learnable parameters:', count_parameters(model))
```

#@title How many learnable parameters does our model have? Check your answer below. { display-mode: "form" }
- answer =

## (C) Define training loop

- Reference: **Section 3** of https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network
- Reference: https://pytorch.org/docs/stable/optim.html

TODOs:
1. Write the `_train_epoch` function, which trains the model for one epoch (one entire pass through the trainig set).
    - Take a batch of examples
    - Do forward pass
    - Do backward pass
    - Update model parameters
    - Repeat

```python
# Define loss function and optimizer
model = Net(d)
model = model.to(device)
criterion = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
```

```python
def _train_epoch(data_loader, model, criterion, optimizer):
    """
    Train the `model` for one epoch of data from `data_loader`
    Use `optimizer` to optimize the specified `criterion`
    """
    model.train()
    for i, (X, y) in enumerate(data_loader):
        # optionally, move data to the appropriate device (GPU or CPU)
        ...

        # clear parameter gradients
        ...

        # forward + backward + optimize
        ...
```

```python
#@title TEST CASE ("compile mode" only) { display-mode: "form" }
model = Net(d)
model = model.to(device)
criterion = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
_train_epoch(tr_loader, model, criterion, optimizer)
```

## (D) Train the network

- Here, we monitor network training progress by printing/plotting the loss and AUROC for training and validation respectively.
- The loss function determines which linear model exactly we train. For example, `nn.BCELoss` with `torch.sigmoid` activation in `Net` is equivalent to _logistic regression_.

```python
#@title Run this cell to define helper functions for evaluation and checkpointing... { display-mode: "form" }
def _evaluate_epoch(tr_loader, va_loader, model, criterion):
    model.eval()
    with torch.no_grad():
        # Evaluate on train
        y_true, y_score = [], []
        running_loss = []
        for X, y in tr_loader:
            X, y = X.to(device), y.to(device)
            output = model(X)
            y_true.append(y.cpu().numpy())
            y_score.append(output.cpu().numpy())
            running_loss.append(criterion(output, y).item())

        y_true, y_score = np.concatenate(y_true), np.concatenate(y_score)
        train_loss = np.mean(running_loss)
        train_score = metrics.roc_auc_score(y_true, y_score)
        print('tr loss', train_loss, 'tr AUROC', train_score)

        # Evaluate on validation
        y_true, y_score = [], []
        running_loss = []
        for X, y in va_loader:
            X, y = X.to(device), y.to(device)
            with torch.no_grad():
                output = model(X)
                y_true.append(y.cpu().numpy())
                y_score.append(output.cpu().numpy())
                running_loss.append(criterion(output, y).item())

        y_true, y_score = np.concatenate(y_true), np.concatenate(y_score)
        val_loss = np.mean(running_loss)
        val_score = metrics.roc_auc_score(y_true, y_score)
        print('va loss', val_loss, 'va AUROC', val_score)
    return train_loss, val_loss, train_score, val_score

def save_checkpoint(model, epoch, checkpoint_dir):
    state = {
        'epoch': epoch,
        'state_dict': model.state_dict(),
    }

    filename = os.path.join(checkpoint_dir, 'epoch={}.checkpoint.pth.tar'.format(epoch))
    torch.save(state, filename)
```

```python
#@title Re-initialize model, loss function and optimizer { display-mode: "form" }
model = None #@param [ "None", "Net(d)" ] {type:"raw"}
model = model.to(device)  # Move model to the appropriate device
criterion = None #@param [ "None", "nn.BCELoss()", "nn.MSELoss", "nn.L1Loss" ] {type:"raw"}
optimizer = None #@param [ "None", "torch.optim.Adam(model.parameters(), lr=learning_rate)", "torch.optim.SGD(model.parameters(), lr=learning_rate)"] {type:"raw"}
```

```python
outputs = []
print('Epoch', 0)
out = _evaluate_epoch(tr_loader, va_loader, model, criterion)
outputs.append(out)

for epoch in range(0, n_epochs):
    print('Epoch', epoch+1)
    # Train model
    _train_epoch(tr_loader, model, criterion, optimizer)

    # Evaluate model
    out = _evaluate_epoch(tr_loader, va_loader, model, criterion)
    outputs.append(out)

    # Save model parameters
    save_checkpoint(model, epoch+1, 'checkpoint/')
```

```python
#@title Plot loss and AUROC scores for training and validation { display-mode: "form" }
train_losses, val_losses, train_scores, val_scores = zip(*outputs)

fig, ax = plt.subplots(figsize=(5,5))
plt.plot(range(n_epochs + 1), train_scores, '-', label='Train')
plt.plot(range(n_epochs + 1), val_scores, '-', label='Validation')
plt.xlabel('epoch')
plt.ylabel('AUROC')
plt.legend()
plt.savefig('auroc.png', dpi=300)

fig, ax = plt.subplots(figsize=(5,5))
plt.plot(range(n_epochs + 1), train_losses, '-', label='Train')
plt.plot(range(n_epochs + 1), val_losses, '-', label='Validation')
plt.xlabel('epoch')
plt.ylabel('Loss (binary cross entropy)')
plt.legend()
plt.savefig('loss.png', dpi=300)
```

## (E*) Final evaluation
- Get final model performance by evaluating on the test set
- Inspect the model parameters

```python
# TODO
```