## The max value entropy search acquisition function

Max-value entropy search (MES) acquisition function quantifies the information gain about the maximum of a black-box function by observing this black-box function $f$ at the candidate set $\{\textbf{x}\}$ (see [1, 2]). BoTorch provides implementations of the MES acquisition function and its multi-fidelity (MF) version with support for trace observations. In this tutorial, we explain at a high level how the MES acquisition function works, its implementation in BoTorch and how to use the MES acquisition function to query the next point in the optimization process.

### 1. MES acquisition function for $q=1$ with noisy observation
For illustrative purposes, we focus in this section on the non-q-batch-mode case ($q=1$). We also assume that the evaluation of the black-box function is noisy. Let us first introduce some notation:
+ $f^* = \max_\mathcal{X} (f(\textbf{x}))$, the maximum of the black-box function $f(\textbf{x})$ in the design space $\mathcal{X}$
+ $y = f(\textbf{x}) + \epsilon, \epsilon \sim N(0, \sigma^2_\epsilon)$, the noisy observation at the design point $\textbf{x}$
+ $h(Y) = \mathbb{E}_Y[-\log(p(y))] = -\int_\mathcal{Y} p(y)\log p(y) dy$, the differential entropy of random variable $Y$ with support $\mathcal{Y}$: the larger is $h(Y)$, the larger is the uncertainty of $Y$.
+ $v(\mathcal{D}) = -\mathbb{E}_D[h(F^*\mid\mathcal{D})]$, the value of data set $\mathcal{D}$, where $F^*$ denotes the function maximum (a random variable in our context of our model).

The Max-value Entropy Search (MES) acquisition function at $\textbf{x}$ after observing $\mathcal{D}_t$ can be written as
\begin{align}
    \alpha_{\text{MES}}(\textbf{x})
    &= v(\mathcal{D}_t\cup \{(\textbf{x}, y)\}) - v(\mathcal{D}_t) \\
    &= - \mathbb{E}_Y[h(F^* \mid \mathcal{D}_t\cup \{(\textbf{x}, Y)\})] + h(F^*\mid\mathcal{D}_t) \\
    &= - \mathbb{E}_Y[h(F^* \mid Y)] + h(F^*) \\
    &= I(F^*; Y) \\
    &= I(Y; F^*) \quad \text{(symmetry)} \\
    &= - \mathbb{E}_{F^*}[h(Y \mid F^*)] + h(Y) \\
\end{align}
, which is the mutual information of random variables
$F^*\mid \mathcal{D}_t$ and $Y \mid \textbf{x}, \mathcal{D}_t$.
Here $F^*$ follows the max value distribution conditioned on $\mathcal{D}_t$, and $Y$ follows the GP posterior distribution with noise at $\textbf{x}$ after observing $\mathcal{D}_t$.

Rewrite the above formula as
\begin{align}
    \alpha_{\text{MES}}(\textbf{x}) &= - H_1 + H_0, \\
    H_0 &= h(Y) = \log \left(\sqrt{2\pi e (\sigma_f^2 + \sigma_\epsilon^2)}\right) \\
    H_1 &= \mathbb{E}_{F^*}[h(Y \mid F^*)] \\
        &\simeq \frac{1}{\left|\mathcal{F}_*\right|} \Sigma_{\mathcal{F}_*} h(Y\mid f^*))
\end{align}
, where $\mathcal{F}_*$ are the max value samples drawn from the posterior after observing $\mathcal{D}_t$. Without noise, $p(y \mid f^*) = p(f \mid f \leq f^*)$ is a truncated normal distribution with an analytic expression for its entropy. With noise, $Y\mid F\leq f^*$ is not a truncated normal distribution anymore. The question is then how to compute $h(Y\mid f^*)$ or equivalently $p(y\mid f \leq f^*)$?

Using Bayes' theorem,
\begin{align}
    p(y\mid f \leq f^*) = \frac{P(f \leq f^* \mid y) p(y)}{P(f \leq f^* )}
\end{align}
, where
+ $p(y)$ is the posterior probability density function (PDF) with observation noise.
+ $P(f \leq f^*)$ is the posterior cummulative distribution function (CDF) without observation noise, given any $f^*$.

We also know from the GP predictive distribution
\begin{align}
    \begin{bmatrix}
        y \\ f
    \end{bmatrix}
    \sim \mathcal{N} \left(
    \begin{bmatrix}
        \mu \\ \mu
    \end{bmatrix} ,
    \begin{bmatrix}
        \sigma_f^2 + \sigma_\epsilon^2 & \sigma_f^2 \\
        \sigma_f^2 & \sigma_f^2
    \end{bmatrix}
    \right).
\end{align}
So
\begin{align}
    f \mid y \sim \mathcal{N} (u, s^2)
\end{align}
, where
\begin{align}
    u   &= \frac{\sigma_f^2(y-\mu)}{\sigma_f^2 + \sigma_\epsilon^2} + \mu \\
    s^2 &= \sigma_f^2 - \frac{(\sigma_f^2)^2}{\sigma_f^2 + \sigma_\epsilon^2}
        = \frac{\sigma_f^2\sigma_\epsilon^2}{\sigma_f^2 + \sigma_\epsilon^2}
\end{align}
Thus, $P(f \leq f^* \mid y)$ is the CDF of above Gaussian.

Finally, given $f^*$, we have
\begin{align}
    h(Y \mid f^*)
    &= -\int_\mathcal{Y} p(y \mid f^*)\log(p(y \mid f^*)) dy\\
    &= -\int_\mathcal{Y} Zp(y)\log(Zp(y)) dy \\
    &\simeq -\frac{1}{\left|\mathcal{Y}\right|} \Sigma_{\mathcal{Y}} Z\log(Zp(y)), \\
    Z &= \frac{P(f \leq f^* \mid y)}{P(f \leq f^* )}
\end{align}
, where $Z$ is the ratio of two CDFs and $\mathcal{Y}$ is the samples drawn from the posterior distribution with noisy observation. The above formulation for noisy MES is inspired from the MF-MES formulation proposed by Takeno _et. al_ [1], which is essentially the same as what is outlined above.

Putting all together,
\begin{align}
    \alpha_{\text{MES}}(\textbf{x})
    &= H_0 - H_1 \\
    &\simeq H_0 - H_1^{MC}\\
    &= \log \left(\sqrt{2\pi e (\sigma_f^2 + \sigma_\epsilon^2)}\right) + \frac{1}{\left|\mathcal{F}^*\right|} \Sigma_{\mathcal{F}^*} \frac{1}{\left|\mathcal{Y}\right|} \Sigma_{\mathcal{Y}} (Z\log Z + Z\log p(y))
\end{align}

The next design point to query is chosen as the point that maximizes this aquisition function, _i. e._,
\begin{align}
    \textbf{x}_{\text{next}} = \max_{\textbf{x} \in \mathcal{X}} \alpha_{\text{MES}}(\textbf{x})
\end{align}

The implementation in Botorch basically follows the above formulation for both non-MF and MF cases. One difference is that, in order to reduce the variance of the MC estimator for $H_1$, we apply also regression adjustment to get an estimation of $H_1$,
\begin{align}
    \widehat{H}_1 &= H_1^{MC} - \beta (H_0^{MC} - H_0)
\end{align}
, where
\begin{align}
    H_0^{MC} &= - \frac{1}{\left|\mathcal{Y}\right|} \Sigma_{\mathcal{Y}} \log p(y) \\
    \beta &= \frac{Cov(h_1, h_0)}{\sqrt{Var(h_1)Var(h_0)}} \\
    h_0 &= -\log p(y) \\
    h_1 &= -Z\log(Zp(y)) \\
\end{align}
This turns out to reduce the variance of the acquisition value by a significant factor, especially when the acquisition value is small, hence making the algorithm numerically more stable.

For the case of $q > 1$, joint optimization becomes difficult, since the q-batch-mode MES acquisiton function becomes not tractable due to the multivariate normal CDF functions in $Z$. Instead, the MES acquisition optimization is solved sequentially and using fantasies, _i. e._, we generate one point each time and when we try to generate the $i$-th point, we condition the models on the $i-1$ points generated prior to this (using the $i-1$ points as fantasies).

<br>
__References__

[1] [Takeno, S., et al., _Multi-fidelity Bayesian Optimization with Max-value Entropy Search._  arXiv:1901.08275v1, 2019](https://arxiv.org/abs/1901.08275)

[2] [Wang, Z., Jegelka, S., _Max-value Entropy Search for Efficient Bayesian Optimization._ arXiv:1703.01968v3, 2018](https://arxiv.org/abs/1703.01968)

### 2. Setting up a toy model
We will fit a standard SingleTaskGP model on noisy observations of the synthetic 2D Branin function on the hypercube $[-5,10]\times [0, 15]$.

```python
import math
import torch

from botorch.test_functions import Branin
from botorch.fit import fit_gpytorch_model
from botorch.models import SingleTaskGP
from botorch.utils.transforms import standardize, normalize
from gpytorch.mlls import ExactMarginalLogLikelihood

torch.manual_seed(7)

bounds = torch.tensor(Branin._bounds).T
train_X = bounds[0] + (bounds[1] - bounds[0]) * torch.rand(10, 2)
train_Y = Branin(negate=True)(train_X).unsqueeze(-1)

train_X = normalize(train_X, bounds=bounds)
train_Y = standardize(train_Y + 0.05 * torch.randn_like(train_Y))

model = SingleTaskGP(train_X, train_Y)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_model(mll);
```

### 3. Defining the MES acquisition function

The `qMaxValueEntropy` acquisition function is a subclass of `MCAcquisitionFunction` and supports pending points `X_pending`. Required arguments for the constructor are `model` and `candidate_set` (the discretized candidate points in the design space that will be used to draw max value samples). There are also other optional parameters, such as number of max value samples $\mathcal{F^*}$, number of $\mathcal{Y}$ samples and number of fantasies (in case of $q>1$). Two different sampling algorithms are supported for the max value samples: the discretized Thompson sampling and the Gumbel sampling introduced in [2]. Gumbel sampling is the default choice in the acquisition function.

```python
from botorch.acquisition.max_value_entropy_search import qMaxValueEntropy

candidate_set = torch.rand(1000, bounds.size(1), device=bounds.device, dtype=bounds.dtype)
candidate_set = bounds[0] + (bounds[1] - bounds[0]) * candidate_set
qMES = qMaxValueEntropy(model, candidate_set)
```

### 4. Optimizing the MES acquisition function to get the next candidate points
In order to obtain the next candidate point(s) to query, we need to optimize the acquisition function over the design space. For $q=1$ case, we can simply call the `optimize_acqf` function in the library. At $q>1$, due to the intractability of the aquisition function in this case, we need to use either sequential or cyclic optimization (multiple cycles of sequential optimization).

```python
from botorch.optim import optimize_acqf

# for q = 1
candidates, acq_value = optimize_acqf(
    acq_function=qMES,
    bounds=bounds,
    q=1,
    num_restarts=10,
    raw_samples=512,
)
candidates, acq_value
```

    (tensor([[1.5350, 0.0758]]), tensor(0.0121))

```python
# for q = 2, sequential optimization
candidates_q2, acq_value_q2 = optimize_acqf(
    acq_function=qMES,
    bounds=bounds,
    q=2,
    num_restarts=10,
    raw_samples=512,
    sequential=True,
)
candidates_q2, acq_value_q2
```

    (tensor([[-0.3238,  0.6565],
             [ 1.5349,  0.0748]]), tensor([0.0135, 0.0065]))

```python
from botorch.optim import optimize_acqf_cyclic

# for q = 2, cyclic optimization
candidates_q2_cyclic, acq_value_q2_cyclic = optimize_acqf_cyclic(
    acq_function=qMES,
    bounds=bounds,
    q=2,
    num_restarts=10,
    raw_samples=512,
    cyclic_options={"maxiter": 2}
)
candidates_q2_cyclic, acq_value_q2_cyclic
```

    (tensor([[-0.3236,  0.6563],
             [ 1.5326,  0.0732]]), tensor([0.0101, 0.0064]))

The use of the `qMultiFidelityMaxValueEntropy` acquisition function is very similar to `qMaxValueEntropy`, but requires additional optional arguments related to the fidelity and cost models. We will provide more details on the MF-MES acquisition function in a separate tutorial.


# References

1. Joseph C. Kolecki, (September 2002) An Introduction to Tensors for Students of Physics and Engineering National Aeronautics and Space Administration, Glenn Research Center, Cleveland, Ohio 44135
2. Daniel A Fleisch - A student's guide to vectors and tensors (2012, Cambridge University Press)
3. [H.K._Dass]_Advanced_Engineering_Mathematics(z-lib.org) (21st Revised Edition)
4. K.A. Stroud, Dexter J. Booth, Advanced Engineering Mathematics

## Imports

```python
import numpy as np
import math
import sympy
import sympy as sp
from sympy import Eq, IndexedBase, symbols, Idx, Indexed, Sum, S, N
from sympy.functions.special.tensor_functions import KroneckerDelta
from sympy.vector import Vector, CoordSys3D, AxisOrienter, BodyOrienter, Del, curl, divergence, gradient, is_conservative, is_solenoidal, scalar_potential, Point, scalar_potential_difference, Del, express, matrix_to_vector
import matplotlib.pyplot as plt
from sympy.physics.vector import ReferenceFrame
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import display
from IPython.display import display_latex
from sympy import latex
from sympy import Array, Matrix, transpose, zeros, diff, Function, Derivative, cos, sin, sqrt, solve, linsolve, acos, atan, asin

from sympy import symbols
from sympy.plotting import plot
```

## Setup

```python
sympy.init_printing()
np.set_printoptions(precision=3)
```

```python
a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z = sympy.symbols('a:z')
a_1, a_2, a_3, b_1, b_2, b_3, c_1, c_2, c_3, d_1, d_2, d_3 = symbols('a_1 a_2 a_3 b_1 b_2 b_3 c_1 c_2 c_3 d_1 d_2 d_3')
u_1, u_2, u_3, v_1, v_2, v_3, h_1, h_2, h_3 = symbols('u_1 u_2 u_3 v_1 v_2 v_3 h_1 h_2 h_3')
alpha, beta, gamma, theta, phi, rho, omega = sympy.symbols('alpha beta gamma theta phi rho omega')
Alpha, Beta, Gamma, Theta, Phi, Rho, Omega = sympy.symbols('Alpha Beta Gamma Theta Phi Rho Omega')
```

## $LaTeX$ Cheat Sheet

### Symbols

$\mathbb{R}$
$\bar{R}$
$\frac{\partial^2 R}{\partial t}$
$\overline{R}$
$\vec{R}$
$\hat{x}$
$\widehat{x}$
$\therefore$
$\because$
$\Rightarrow$
$\rightarrow$
$\implies$
$\iff$
$\leftrightarrow$
$\Leftrightarrow$
$\equiv$
$\subset$
$\supset$
$\wedge$
$\parallel$
$\oplus$
$\ominus$
$\otimes$
$\times$
$\top$
$\bot$
$\forall$
$\exists$
$\exists!$

## Functions

```python
# Usage: display_equation('u_x', x)
def display_equation(idx, symObj):
    if(isinstance(idx, str)):
        eqn = '\\[' + idx + ' = ' + latex(symObj) + '\\]'
        display_latex(eqn, raw=True)
    else:
        eqn = '\\[' + latex(idx) + ' = ' + latex(symObj) + '\\]'
        display_latex(eqn, raw=True)
    return
```

```python
# Usage: display_full_latex('u_x')
def display_full_latex(idx):
    if(isinstance(idx, str)):
        eqn = '\\[' + idx + '\\]'
        display_latex(eqn, raw=True)
    else:
        eqn = '\\[' + latex(idx) + '\\]'
        display_latex(eqn, raw=True)
    return
```

```python
def vplot2d(O, a, origin_label='O', tip_label='P'):
    dx = a[0][0] - O[0][0]
    dy = a[1][0] - O[1][0]

    head_length = 0.3
    vec_ab = [dx,dy]

    vec_ab_magnitude = math.sqrt(dx**2+dy**2)

    dx = dx / vec_ab_magnitude
    dy = dy / vec_ab_magnitude

    vec_ab_magnitude = vec_ab_magnitude - head_length

    ax = plt.axes()
    ax.set_aspect('equal', 'box')
    #ax.axis([-range, range, -range, range]) Sub for xlim and ylim

    ax.arrow(O[0][0], O[1][0], vec_ab_magnitude*dx, vec_ab_magnitude*dy, head_width=0.1, head_length=head_length, fc='lightblue', ec='black')
    plt.scatter(O[0][0],O[1][0],color='black')
    plt.scatter(a[0][0],a[1][0],color='black')

    ax.annotate(origin_label, (O[0][0]-0.4,O[1][0]),fontsize=14)
    ax.annotate(tip_label, (a[0][0]+0.3,a[1][0]),fontsize=14)
    return
```

```python
def rotation(theta):
    R = zeros(3, 3)
    R[0,0] = cos(theta)
    R[0,1] = -sin(theta)
    R[0,2] = 0
    R[1,0] = sin(theta)
    R[1,1] = cos(theta)
    R[1,2] = 0
    R[2,0] = 0
    R[2,1] = 0
    R[2,2] = 1
    return R
    return R
```

```python
def inRadians(theta):
    return (sympy.pi / 180) * theta

def inDegrees(theta):
    return theta * (180 / sympy.pi)
```

### Coordinate Transformation

```python
x_prime, y_prime, z_prime = sympy.symbols('x\' y\' z\'')
l_1, l_2, l_3, m_1, m_2, m_3, n_1, n_2, n_3 = sympy.symbols('l_1 l_2 l_3 m_1, m_2 m_3 n_1 n_2 n_3')
x, y, z = sympy.symbols('x y z')
```

```python
x_prime, y_prime, z_prime = sp.symbols('x\' y\' z\'')
l_1, l_2, l_3, m_1, m_2, m_3, n_1, n_2, n_3 = sp.symbols('l_1 l_2 l_3 m_1, m_2 m_3 n_1 n_2 n_3')
x, y, z = sp.symbols('x y z')

# Equation 1
x_prime = l_1 * x + m_1 * y + n_1 * z
y_prime = l_2 * x + m_2 * y + n_2 * z
z_prime = l_3 * x + m_3 * y + n_3 * z

display_equation('x\'', x_prime)
display_equation('y\'', y_prime)
display_equation('z\'', z_prime)
```

\[x' = l_{1} x + m_{1} y + n_{1} z\]

\[y' = l_{2} x + m_{2} y + n_{2} z\]

\[z' = l_{3} x + m_{3} y + n_{3} z\]

```python
# Redefining the primes so equation 1 doesn't get substituted in
x_prime, y_prime, z_prime = sp.symbols('x\' y\' z\'', Integer=True)

x = l_1 * x_prime + m_1 * y_prime + n_1 * z_prime
y = l_2 * x_prime + m_2 * y_prime + n_2 * z_prime
z = l_3 * x_prime + m_3 * y_prime + n_3 * z_prime

display_equation('x\'', x)
display_equation('y\'', y)
display_equation('z\'', z)
```

\[x' = l_{1} x' + m_{1} y' + n_{1} z'\]

\[y' = l_{2} x' + m_{2} y' + n_{2} z'\]

\[z' = l_{3} x' + m_{3} y' + n_{3} z'\]

|  | $x$ | $y$ | $z$ |
|---|---|---|---|
| $x'$ | $l_1$ | $m_1$ | $n_1$ |
| $y'$ | $l_2$ | $m_2$ | $n_2$ |
| $z'$ | $l_3$ | $m_3$ | $n_3$ |

### OR

$\bar{x_1} = l_{11}x_1 + l_{21}x_2 + l_{31}x_3$
$\bar{x_2} = l_{13}x_1 + l_{22}x_2 + l_{32}x_3$
$\bar{x_3} = l_{13}x_1 + l_{23}x_2 + l_{33}x_3$

$x_1 = l_{11}\bar{x_1} + l_{12}\bar{x_2} + l_{13}\bar{x_3}$
$x_2 = l_{21}\bar{x_1} + l_{22}\bar{x_2} + l_{23}\bar{x_3}$
$x_3 = l_{31}\bar{x_1} + l_{32}\bar{x_2} + l_{33}\bar{x_3}$

### With column by row indexing

|  | $x_1$ | $x_2$ | $x_3$ |
|---|---|---|---|
| $\bar{x_1}$ | $l_{11}$ | $l_{21}$ | $l_{31}$ |
| $\bar{x_2}$ | $l_{12}$ | $l_{22}$ | $l_{32}$ |
| $\bar{x_3}$ | $l_{13}$ | $l_{23}$ | $l_{33}$ |

$\bar{x_j} = l_{ji}x_i$
$x_i = l_{ji}\bar{x_j}$
Where
$i$ = column
$j$ = row

#### With row by column indexing (More common matrix indexing, used here)

|  | $\bar{x_1}$ | $\bar{x_2}$ | $\bar{x_3}$ |
|---|---|---|---|
| $x_1$ | $l_{11}$ | $l_{12}$ | $l_{13}$ |
| $x_2$ | $l_{21}$ | $l_{22}$ | $l_{23}$ |
| $x_3$ | $l_{31}$ | $l_{32}$ | $l_{33}$ |

$x_i = l_{ij}\bar{x_j}$
$\bar{x_j} = l_{ij}x_i$
Where
$i$ = row
$j$ = column

#### Relation between the Direction Cosines of Three Mutually Perpendicular Straight Lines

1. The dot product between any two is 0.
2. The dot product of any vector on itself is 1.

In Einstein summation notation

```python
i, k = sympy.symbols('i k')
display_equation('I_{ij}I_{kj}', sp.Piecewise((1, sp.Eq(i, k)), (0, sp.Ne(i, k))))
display_equation('\delta_{kj}', sp.Piecewise((1, sp.Eq(i, k)), (0, sp.Ne(i, k))))
display_full_latex('\\therefore')
display_equation('I_{ij}I_{kj}', KroneckerDelta(i, k))
```

\[I_{ij}I_{kj} = \begin{cases} 1 & \text{for}\: i = k \\0 & \text{otherwise} \end{cases}\]

\[\delta_{kj} = \begin{cases} 1 & \text{for}\: i = k \\0 & \text{otherwise} \end{cases}\]

\[\therefore\]

\[I_{ij}I_{kj} = \delta_{i k}\]

```python
i, j, k = sp.symbols('i j k')
l = sp.IndexedBase('l')
sp.x_bar_j = l[i, j] * x[i]
display_equation('\\bar{x_j}', x_bar_j)
f = x_bar_j * l[j, k]
display_equation('l_{jk}\\bar{x_j}', f)
f = f.subs(l[i, j] * l[j, k], KroneckerDelta(i, k))
display_equation('l_{jk}\\bar{x_j}', f)
f = f.subs(i, k)
display_equation('l_{jk}\\bar{x_j}', f)
```

$\implies x_k = l_{jk}\bar{x_j}$

Note: $I_{ij}I_{kj}$ is actually the dot product of the ith row by the kth row, while $I_{ij}I_{jk}$ is the product of the jth row & the jth column. In other words,
* $I_{ij}I_{kj}$ is the dot product of two $\parallel$ vectors, divided by the product of the norms of the vectors.
* $I_{ij}I_{jk}$ is the dot product of two $\bot$ vectors, divided by the product of the norms of the vectors.

$v_i = l_iv \tag{1a}$
$\bar{v_j} = v\bar{l_j} \tag{1b}$

$\bar{l_j} = l_{ij}l_i \tag{2a}$
$l_i = l_{ij}\bar{l_j} \tag{2b}$

```python
i, j, v_i, v_j, v_bar_i, v_bar_j, l_bar_i, l_bar_j = sp.symbols('i j v_i v_j \\bar{v_i} \\bar{v_j} \\bar{l_i} \\bar{l_j}')
l_i, l_j = sp.symbols('l_i l_j')
v, l = sp.symbols('v l')
l_ij = sp.symbols('l_{ij}')

f = v * l_bar_j # From eq 1b
display_equation('\\bar{v_j}', f)
print('Substitute eq 2a')
f = f.subs(l_bar_j, l_ij * l_i) # From eq 2a
display_equation('\\bar{v_j}', f)
print('Substitute eq 1a')
f = f.subs(v, v_i / l_i) # From eq 1a
display_equation('\\bar{v_j}', f)
```

### Rank of a Tensor

| Tensor | Symbol | Rank |
|---|---|---|
| Scalar | $A$ | zero |
| Contravariant Tensor | $B^i$ | 1 |
| Covariant Tensor | $C_k$ | 1 |
| Covariant Tensor | $D_{ij}$ | 2 |
| Mixed Tensor | $E^{il}_{jkl}$ | 3 |

In an n-dimensional space, the number of components of a tensor of rank $r$ is $n^r$.

**First Order**
$a_i = l_{ij}\bar{a}_j$
$\bar{a}_p = l_{ip}a_i$

**Second Order**
$\bar{a}_{pq} = l_{ip}l_{jp}a_{ij}$

**Any Order**
$\bar{a}_{pqrs}... = l_{ip}l_{jp}l_{kr}l_{ls}...a_{ijkl}$

#### Indexed variables

```python
Delta = symbols('\\Delta')
n, m = symbols('n m', integer=True)
i = Idx('i', (1, n))
j = Idx('j', (1, n))
k = Idx('k', (0, n))
A = IndexedBase('A', shape=(m, n))
a = IndexedBase('a', shape=(m, n))
e = Eq(a[i,j] *  A[k,j], Delta * KroneckerDelta(i, k))
display(e)
print('Using k = i')
display(e.subs(k, i))
```

```python
a, b, c, d, e, f = symbols('a:f')
u1, u2, u3, v1, v2, v3 = symbols('u_1 u_2 u_3 v_1 v_2 v_3')
E = ReferenceFrame('E')
u = u1 * E.x + u2 * E.y + u3 * E.z
v = v1 * E.x + v2 * E.y + v3 * E.z
```

#### Dot Product

```python
u & v
```

#### Cross Product

```python
u ^ v
```

#### The Three Scalar Invariants of a Second Order Tensor

```python
n = 3
i = Idx('i', (1, n))
j = Idx('j', (1, n))
p = Idx('p', (1, n))
q = Idx('q', (1, n))
a = IndexedBase('a', shape=(n, n))
```

1. $a_{ii}$

```python
Sum(a[i,i], (i, 1, n)).doit()
```

2. $\frac{1}{2} (a_{ii}a_{jj} - a_{ij}a_{ji})$

```python
S('1/2') * Sum(a[i,i] * a[i,i] - a[i,j] * a[j,i] , (i, 1, n), (j, 1, n)).doit()
```

3. $|a_{ij}|$

#### Singular Tensor
$|a_{ij}| = 0$

#### Tensor Product

```python
from sympy.abc import x,y,z,t
n = 3
a = IndexedBase('a', shape=(n))
b = IndexedBase('b', shape=(n))
```

```python
from sympy.physics.quantum import TensorProduct
n = 2
A = zeros(1, n)
B = zeros(1, n)
#A = Array([a[1], a[2]])
#B = Array([b[1], b[2]])
#display(A.rank())
#display(B.rank())
#tensorproduct(A, B).rank()

for i in range(n):
    A[i] = a[i + 1]
    B[i] = b[i + 1]

display(A.T)
display(B.T)
TensorProduct(A.T, B.T)
```

```python
P = np.array([[2, 4],[5, 6]])
B = np.array([[1, 2],[3, 5]])
C = np.array([[1, 2],[3, 5]])
```

```python
3*P - 2*P
```

```python
A = np.array([[2, 3, 1],[4, 2, 1], [3, 1, 1]])
B = np.array([2, 1, 3]).T
A@B
```

```python
E = CoordSys3D('E')
U = u_1 * E.i + u_2 * E.j + u_3 * E.k
V = v_1 * E.i + v_2 * E.j + v_3 * E.k
H = h_1 * E.i + h_2 * E.j + h_3 * E.k
```

```python
U.outer(V)
```

```python
R = dyad & H
R.simplify()
```

```python
V & H
```

```python
U * (V & H)
```

```python
u, v, w = sympy.symbols('u v w')
#E.x = Function('x')(u, v, w)
#E.y = Function('y')(u, v, w)
#E.z = Function('z')(u, v, w)
x, y, z = [E.x, E.y, E.z]
r = x * E.i + y * E.j + z * E.k
#r = E.x * E.i + E.y * E.j + E.z * E.k
r
```

```python
u, v = [x, y] # Assumed
#u, v = [x, x] # Assumed
#u, v = [y, x] # Assumed All three to prove a point from [2]
#u = Function('u')(x, y, z)
#v = Function('v')(x, y, z)
#w = Function('w')(x, y, z)
```

```python
delop = Del()
a = delop(u)
a
```

```python
diff(r, u)
```

```python
res = diff(r, u).doit() & delop(u)
res
```

```python
res.doit()
```

```python
res = diff(r, u).doit() & delop(v)
res
```

```python
res.doit()
```

```python

```

```python

```

```python
x, y, xd, xdd, yd, ydd = sympy.symbols('x y \dot{x} \ddot{x} \dot{y} \ddot{y}')
x1, x2, xp1, xp2 = sympy.symbols('x_1 x_2 x^{\prime}_1 x^{\prime}_2')
Ap1, Ap2 = sympy.symbols('A^{\prime1} A^{\prime2}')
r, theta = sympy.symbols('r theta')
```

```python
x1 = x
x2 = y
xp1 = r
xp2 = theta
x = r*cos(theta)
y = r*sin(theta)
```

```python
eq = Eq(r * sin(theta), x)
solve(eq, theta)
```

```python

```

```python

```

```python

```

```python
E = CoordSys3D('E')
Ap = sympy.symbols('A^\prime')
EA = E.orient_new_axis('E_A', theta, E.k)
EAp = EA.orient_new_axis('EA^\prime', alpha, EA.k)
A = EA.i + EA.j + EA.k
A = express(A, E)
Ap = EAp.i + EAp.j + EAp.k
Ap = express(Ap, E)
EA.rotation_matrix(EAp)

#EA.position_wrt(E)
#EA.to_matrix(E)
#EA.origin.express_coordinates(E)
#type(EA.origin)
#A.__dir__()

#EA.rotation_matrix(E)
```

```python
N = CoordSys3D('N')
A = 5 * N.i + 3 * N.j
ang = inRadians(150)
Rot = rotation(ang)
res = Rot @ A.to_matrix(N)
res = res.evalf(2)
matrix_to_vector(res, N)
```

```python
E = CoordSys3D(E)
etra1, etra2, eco1, eco2 = sympy.symbols('\vec{E}^1 \vec{E}^2 \vec{E}_1 \vec{E}_2')
A, Atra1, Atra2, Ax, Ay = sympy.symbols('\vec{A} A^1 A^2 A_x A_y')

eco1 = E.i + 3 * E.j
eco2 = 4 * E.i
A = 7 * E.i + 2 * E.j
eqns = [Atra1 * eco1.components[E.i] + Atra2 * eco2.components[E.i] - A.components[E.i], Atra1 * eco1.components[E.j] + Atra2 * 0 - A.components[E.j] ]
m = sympy.linear_eq_to_matrix(eqns, [Atra1, Atra2])
res = linsolve(eqns, Atra1, Atra2)
(Atra1, Atra2) = next(iter(res))
angle = acos((eco1 & eco2) / (eco1.magnitude() * eco2.magnitude()))
angle = inDegrees(angle)
angle = angle.evalf()
t = 90 - angle
etra1mag = 1/(eco1.magnitude() * cos(inRadians(t))).evalf()
etra2mag = 1/(eco2.magnitude() * cos(inRadians(t))).evalf()
etra1 = etra1mag * E.j
etra2 = etra2mag * cos(t) * E.i + etra2mag * sin(t) * E.j
A_angle = atan(A.components[E.j] / A.components[E.i])
eco1A_ang = acos((eco1 & A) / (eco1.magnitude() * A.magnitude()))
eco2A_ang = acos((eco2 & A) / (eco2.magnitude() * A.magnitude()))
l1 = A.magnitude() * cos(eco1A_ang)
Acon1 = l1/(cos(t) * etra1mag)
Acon1 = Acon1.evalf()
l2 = A.magnitude() * cos(eco2A_ang)
Acon2 = l2/(cos(t) * etra2mag).evalf()
```

```python
Atra1, Atra2
```

```python
m
```

```python

t
```

```python
etra1mag
```

```python
etra2mag
```

```python
#l = A.magnitude() * cos(A_angle)
A_angle.evalf()
```

```python
inDegrees(e1A_ang).evalf()
```

```python
Acon1
```

```python
Acon2
```

```python
l1.evalf(), l2
```

```python
etra2
```

```python
A & eco1, A & eco2
```

```python
A & etra1, A & etra2
```

```python

```

```python

```

```python
x, y = symbols('x y')
u = [4, 8, 16, 32]
v = [-16, -12, -8, -4, 0, 4, 8]

curves = None

for i in range(len(u) - 1):
    pu = plot(u[i]/x, show=False, xlim=(0, 4), ylim=(0, 20)) #label=str(u[i]), legend=True

    if (curves == None):
        curves = pu
    else:
        curves.append(pu[0])

for i in range(len(v) - 1):
    pv = plot(x**2 - v[i], show=False, xlim=(0, 4), ylim=(0, 20)) #label=str(v[i]), legend=True

    if (curves == None):
        curves = pv
    else:
        curves.append(pv[0])

curves.show()
```

```python

```


# Lecture 29

## Law of Large Numbers, Central Limit
----

## Law of Large Numbers

Let $X_1, X_2, \dots $ be i.i.d. random variables all with with mean $\mu$ and variance $\sigma^2$.

Let the *sample mean* be denoted by $\bar{X_n} = \frac{1}{n}\sum_{i=1}^{n}X_j$, where $n$ is the sample size.

*What can we say about $\bar{X_n}$ when $n$ gets large?*

### (Strong) Law of Large Numbers: $\bar{X_n} \rightarrow \mu$ as $n \rightarrow \infty$ with probability 1.0

* this means that the sample mean $\bar{X_n}$ converges (point-wise) to the true mean $\mu$ when $n$ gets large
* hence it allows us to make an approximation of the true mean without having to actually carry out an infinite amount of experiments

#### Example

\begin{align}
  X_j &\sim Bern(p) \text{, then } \frac{X_1 + \dots + X_n}{n} \rightarrow p \text{ with probability 1.0}
\end{align}

### (Weak) Law of Large Numbers: $\text{for any } c > 0 \text{, } P( \lvert \bar{X_n} - \mu > c \rvert ) \rightarrow 0 \text{ as } n \rightarrow \infty$

* $c$ could be some small number, say, 0.001
* if $n$ is large, it is extremely likely that the sample mean is close to true mean

#### Proof

\begin{align}
  P( \lvert \bar{X_n} - \mu \rvert > c) &\leq \frac{Var(\bar{X_n})}{c^2} \quad \text{Chebyshev's Inequality} \\
  \\
  \text{but } \frac{Var(\bar{X_n})}{c^2} &= \frac{ \frac{1}{n^2} n \, \sigma^2 }{c^2} \\
  &= \frac{\sigma^2}{n \, c^2} \\
  &= 0 \text{ as } n \rightarrow \infty \quad \blacksquare
\end{align}

## Central Limit Theorem

Another way to think about the Law of Large Numbers is to see that

\begin{align}
  \displaystyle{\lim_{n \to \infty}} \, \bar{X_n} - \mu &= 0 \\
\end{align}

However, this is only thinking point-wise.

* *But what is the distribution of $\bar{X_n}$?*
* _What does that distribution look like?_
* _That might be going to 0, but how fast is it doing so?_

One way to study the distribution of $\bar{X_n}$ is to multiply $(\bar{X_n} - \mu)$ by some variable that itself goes to $\infty$.

* if that product goes to $\infty$, then we know that the variable we have chosen is dominating over $(\bar{X_n} - \mu)$
* and if that product instead still approaches 0, then that also tells us something significant about $(\bar{X_n} - \mu)$

Consider:

\begin{align}
  n^{?} \, (\bar{X_n} - \mu)
\end{align}

We could learn more about the distribution of $\bar{X_n}$ by selecting some power of $n > 0$, and thinking about what happens.

### The Central Limit Theorem

\begin{align}
  n^{1/2} \, \frac{(\bar{X_n} - \mu)}{\sigma} \rightarrow \mathcal{N}(0,1) \quad \text{in distribution} \\
\end{align}

* $X$ may be discrete or it may be continuous
* we do need to assume that $X$ has a finite variance
* the CDF of $X$ will converge to CDF $\Phi$
* CLT is the justification for using normal approximations
* $n^{1/2}$ is _just right_; that is what allows for a non-degenerate distribution that actually converges into something beautiful and useful (it doesn't just go to 0 or blow up to $\infty$)

#### Proof: in terms of sum

* we claim that the convolution $\sum_{j=1}^{n} X_j$ is approximately normal with mean $n \mu$
* we can standardize this by centering $\sum_{j=1}^{n} X_j - n \mu$; this makes mean $\mu=0$
* the variance of the convolution is $n \sigma^2$ (c.f. property [7] of covariance)
* we complete standardization by dividing by the standard deviation $\sqrt{n} \sigma$

Here is the standardized version:
\begin{align}
  &\quad \frac{\sum_{j=1}^{n} X_j - n\mu}{\sqrt{n} \, \sigma} \rightarrow \mathcal{N}(0,1)
\end{align}

* Assume that MGF $M(t)$ of $X_j$ exists
* Assume wlog that $\mu=0$ and $\sigma=1$, since we could have alternatively standardized the r.v. separately and declared $\frac{1}{\sqrt{n}} \sum_{j=1}^{n} \frac{(X_j - \mu)}{\sigma}$

Let $S_n = \sum_{j=1}^{n} X_j$, show $M \left[ \frac{S_n}{\sqrt{n}} \right] \rightarrow M \left[ \mathcal{N}(0,1) \right]$.

Here are some quick facts about Moment Generating Functions to keep in mind as we go along our proof:

1. $M(t) = \mathbb{E}(e^{tX_1})$
2. $M(0) = 1$
3. $M\prime(0) = \mu = 0$
4. $M\prime(0) = 1$

\begin{align}
  M\left[ S_n \right] &= \mathbb{E}( e^{t \, S_n / \sqrt{n} } ) \\
  &= \mathbb{E}(e^{t \, X_1/\sqrt{n}}) \dots \mathbb{E}(e^{t \, X_n/\sqrt{n}})  &\quad \text{independence, not correlated} \\
  &= \left( M\left[ \frac{t}{\sqrt{n}} \right] \right)^n &\quad \text{but these are just the MGF} \\
  &= \displaystyle{\lim_{n \to \infty}} n \, log  M\left( \frac{t}{\sqrt{n}} \right) \\
  &= \displaystyle{\lim_{n \to \infty}} \frac{log  M\left( \frac{t}{\sqrt{n}} \right)}{\frac{1}{n}} &\quad \text{let } y = \frac{1}{\sqrt{n}} \text{ and } y \text{ is real} \\
  &= \displaystyle{\lim_{y \to 0}} \frac{log M(y\,t)}{y^2} &\quad \text{and using l'Hopital's Rule} \\
  &= \displaystyle{\lim_{y \to 0}} \frac{t \, M\prime(yt)}{2y \, M(yt)} &\quad \text{and applying l'Hopital's Rule again} \\
  &= \frac{t}{2} \, \displaystyle{\lim_{y \to 0}} \frac{M\prime(yt)}{y} &\quad \text{and applying l'Hopital's Rule yet again} \\
  &= \frac{t^2}{2} \, \displaystyle{\lim_{y \to 0}} \frac{M\prime\prime(yt)}{1} \\
  &= \frac{t^2}{2} \\
\end{align}

But $\frac{t^2}{2}$ is the log of $e^{t^2 / 2}$, and that is the MGF of $\mathcal{N}(0,1)$. QED.

### Example: Basic Normal Approximation of a Binomial

Let $X \sim Bin(n,p)$, and think of $X = \sum_{j=1}^{n} X_j, \, X \sim Bern(p)$ i.i.d.

By the Central Limit Theorem, we can approximate $X$ with a Normal distribution if $n$ is large enough, and if we standardize $X$ first.

\begin{align}
  P(a \leq X \leq b) &= P\left( \frac{a - np}{\sqrt{npq}} \leq \frac{X - np}{\sqrt{npq}} \leq \frac{b - np}{\sqrt{npq}} \right) \\
  &\approx \Phi\left( \frac{b - np}{\sqrt{npq}} \right) - \Phi\left( \frac{a - np}{\sqrt{npq}} \right)
\end{align}

Contrast the above **Normal** approximation of a binomial with a **Poisson** approximation. With a Poisson approximation of a binomial, we assumed:

* $n$ is very large, i.e., $n \rightarrow \infty$
* $p$ is very small
* let $\lambda = np$, i.e. $p$ is fixed

But in the case of a Normal approximation, while we do wish $n$ to be large, it is best if $p$ is close to $\frac{1}{2}$. _Why?_

Remember that the Normal distribution in a CLR is symmetric about $\mu = 0$. If $p$ is too far away from the mean, then the distribution will get very skewed. That is bad. If $n$ is really, really, really large, then the CLR would still work no matter what $p$ might be, but you will need to be careful when $n$ is not that large.

### Continuity Correction

Now in the above example, we are approximating a discrete distribution using a continuous one.

What would we do if we instead started with something like $P(X=a)$, where $a$ is some integer?

\begin{align}
  P(X=a) &= P(a - \epsilon \leq X \leq a + \epsilon)
\end{align}

... where $\epsilon$ is a very small value that allows us to look at a _range centered at $a$_ instead of a single value $a$. Now we can continue using the Normal approximation.


# Computational Astrophysics
## Partial Differential Equations. 07
## Elliptic PDEs

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the techniques used to solve the Poisson equation.

`A. Garcia. Numerical Methods for Physics. (1999). Chapter 6 - 7 `

---

##  Elliptic Partial Differential Equations

The simplest elliptic partial differential equation is the linear Poisson equation.
For the Newtonian gravitational potential $\Phi$, this is
\begin{equation}
\nabla^2 \Phi =  4\pi G \rho\,\,.
\end{equation}

Consider a spherically symmetric mass
distribution. The Laplacian
$\nabla^2$ reduces to

\begin{equation}
\begin{aligned}
\nabla^2 [\cdot] &= \frac{1}{r^2} \frac{\partial}{\partial r}\left(r^2 \frac{\partial}{\partial r} [\cdot] \right)\,\,,\\
&= \frac{2}{r}\frac{\partial}{\partial r}[\cdot] + \frac{\partial^2}{\partial r^2} [\cdot]\,\,.
\end{aligned}
\end{equation}

---

### ODE Method

Since the PDE reduces to just one variable, $r$, we write it as a second-order ODE

\begin{equation}
\frac{d^2 \Phi}{d r^2}  + \frac{2}{r}\frac{d \Phi}{d r}  = 4 \pi G\rho\,\,.
\end{equation}

This equation is reduced to the system

\begin{equation}
\begin{aligned}
\frac{d \Phi}{dr} &= z\,\,,\\
\frac{dz}{dr}  + \frac{2}{r} z &= 4 \pi G \rho\,\,.
\end{aligned}
\end{equation}

In order to solve inside the matter distribution, we will use
inner and outer boundary conditions,

\begin{equation}
\begin{aligned}
\left . \frac{d \Phi}{dr} \right| _{r=0} & = 0\,\,,\\
\Phi(R_\mathrm{surface}) &= - \frac{GM(R_\mathrm{surface})}{R_\mathrm{surface}}\,\,.
\end{aligned}
\end{equation}

---

In practice, one sets $\Phi(r=0) = 0$ and integrates out. Then, the
result is corrected by adding a constant term to obtain the outer
boundary condition, which is the analytic value for the
gravitational potential outside a spherical mass distribution.

Remember that this added constant doesn't affect the field value
because of gauge invariance!

---

### Matrix Method

In the matrix method, we discretize Poisson Equation
with centered differences. Then, for an interior (non-boundary) grid point $i$ we have

\begin{equation}
\begin{aligned}
\frac{\partial \Phi }{\partial r} \bigg|_i &\approx \frac{1}{2\Delta r} \left(\Phi_{i+1} - \Phi_{i-1}\right)\,\,,\\
\frac{\partial^2 \Phi }{\partial r^2}  \bigg|_i &\approx \frac{1}{(\Delta r)^2} \left(\Phi_{i+1} - 2 \Phi_i + \Phi_{i-1} \right)\,\,.
\end{aligned}
\end{equation}

Then, Poisson equation is

\begin{equation}
\begin{aligned}
\frac{1}{r_i} \frac{\left(\Phi_{i+1} - \Phi_{i-1}\right)}{\Delta r}  + \frac{\left(\Phi_{i+1} - 2 \Phi_i + \Phi_{i-1} \right)}{(\Delta r)^2}  &= 4\pi G \rho_i\,\,.
\end{aligned}
\end{equation}

---
**Inner boundary condition**

Because of the $1/r$ term in the Laplacian, it is clear that the behavior
 at $r=0$ must be regularized.

It is possible to use an expansion or we can
stagger the grid so that there is no point exactly at $r=0$, for example
by moving the entire grid over by $0.5 \Delta r$.

At the inner boundary, we have the condition $\frac{\partial \Phi}{ \partial r} = 0$, so we assume

\begin{equation}
\Phi_{-1} = \Phi_{0}\,\,.
\end{equation}

---

Hence, Poisson equation can be written as linear system

\begin{equation}
J \mathbf{\Phi} = \mathbf{b}\,\,,
\label{eq:pde_poisson2}
\end{equation}

where $\Phi$ = $(\Phi_0, \cdots , \Phi_{n-1})^T$ (for a grid with $n$
points labeled $0$ to $n-1$) and  $\mathbf{b} = 4\pi G
(\rho_0, \cdots , \rho_{n-1})^T$.

The matrix $J$ has tri-diagonal form and can be explicitely given as

1. $i=j=0$:
\begin{equation}
J_{00} = - \frac{1}{(\Delta r)^2} - \frac{1}{r_0 \Delta r}\,\,,
\end{equation}

2. $i=j$:
\begin{equation}
J_{ij} = \frac{-2}{(\Delta r)^2}\,\,,
\end{equation}

3. $i+1=j$:
\begin{equation}
J_{ij} = \frac{1}{(\Delta r)^2} + \frac{1}{r_i \Delta r}\,\,,
\end{equation}

4. $i-1=j$:
\begin{equation}
J_{ij} = \frac{1}{(\Delta r)^2} - \frac{1}{r_i \Delta r}\,\,.
\end{equation}

After finding the solution to the linear system, it is neccesary to correct for the analytic outer boundary value

\begin{equation}
\Phi(R_\text{surface}) = - \frac{G M}{R_\text{surface}}.
\end{equation}

```python

```


# pendulum with spring

```python
import sympy
sympy.init_printing()

from IPython.display import display

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

import sys
sys.path.insert(0, './code')

from gauss_legendre import gauss_legendre
from evaluate_functional import evaluate_functional
```

```python
# generalized configuration coordinates
r = sympy.Symbol('r')
φ = sympy.Symbol('φ')

# parameters
from sympy.abc import m, k, l, g
params = {m: 0.1, k: 3.0, l: 0.5, g: 9.81}
```

```python
# generalized velocities
rdot = sympy.Symbol('\dot{r}')
φdot = sympy.Symbol('\dot{φ}')

# kinetic energy
T = sympy.Rational(1,2) * m * (rdot**2 + ((l+r)*φdot)**2)

# conjugate momenta
p_r = sympy.Symbol('p_r')
p_φ = sympy.Symbol('p_φ')

# Legendre transformation
lt_r = sympy.solve(T.diff(rdot) - p_r, rdot)[0]
lt_φ = sympy.solve(T.diff(φdot) - p_φ, φdot)[0]

# potential energy
V = m * g * (-(l+r) * sympy.cos(φ)) + sympy.Rational(1,2) * k * r**2

# Hamiltonian
H = T.subs([(rdot, lt_r), (φdot, lt_φ)]) + V
H = H.simplify()
display(H)
```

```python
# state vector
x = [φ, r, p_φ, p_r]

# initial conditions
r0 = (params[m] * params[g]) / params[k]
x_0 = np.array([0.0, r0, 0.1, 0.0])
```

```python
dH = sympy.Matrix([H.diff(x_i) for x_i in x])
display(dH)
```

$$\left[\begin{matrix}g l m \sin{\left (φ \right )} + g m r \sin{\left (φ \right )}\\- g m \cos{\left (φ \right )} + k r - \frac{p_{φ}^{2}}{m \left(l + r\right)^{3}}\\\frac{p_{φ}}{m \left(l + r\right)^{2}}\\\frac{p_{r}}{m}\end{matrix}\right]$$

```python
# structure matrix
J = sympy.Matrix([[0, 0, 1, 0],
                  [0, 0, 0, 1],
                  [-1, 0, 0, 0],
                  [0, -1, 0, 0]])
xdot = J @ dH
display(xdot)

# final time, time step, order of collocation polynomials
t_f = 50.0
h = 1e-2
s = 2
```

$$\left[\begin{matrix}\frac{p_{φ}}{m \left(l + r\right)^{2}}\\\frac{p_{r}}{m}\\- g l m \sin{\left (φ \right )} - g m r \sin{\left (φ \right )}\\g m \cos{\left (φ \right )} - k r + \frac{p_{φ}^{2}}{m \left(l + r\right)^{3}}\end{matrix}\right]$$

```python
%time time, solution = gauss_legendre(x, xdot, x_0, t_f, h, s, tol=1e-12, params=params)
```

    CPU times: user 1.55 s, sys: 8.17 ms, total: 1.56 s
    Wall time: 1.56 s

```python
fig, ax = plt.subplots(nrows=2, dpi=200)
fig.tight_layout(pad=1.5)
ax[0].plot(time, solution[:, 0]);
ax[1].plot(time, solution[:, 1]);
```

```python
# width of the band in which the error of energy conservation oscillates
energy = evaluate_functional(x, H, solution, params=params)
energy.max() - energy.min()
```

```python
K = int(len(time) / 2)
fig, ax = plt.subplots(dpi=200, figsize=(8,4))
fig.tight_layout(pad=1.5)
ax.set_title(r"total energy $H \: (\mathrm{J})$")
ax.set_xlabel(r"time $t \: (\mathrm{s})$")
#ax.set_ylabel("Hamiltonian")
ax.plot(time[K:2*K], energy[K:2*K]);

font = {'family' : 'Calibri',
        'weight' : 'normal',
        'size'   : 12}
matplotlib.rc('font', **font)
matplotlib.rc('text', usetex=True)

#fig.savefig("discrete_energy_conservation.pdf")
```


# Logistic regression
I think it most simple to first demonstrate what type of problem that we are interested in. So here lets compare displacement (disp) for US made and non-US made cars.

Let's first assign mtcars to a new dataframe and then identify the US (1) vs. non_US (0) under the column label "origin."

```R
cars <- mtcars
non_US <- c(1:3, 8:14, 18:21, 26:28, 30:32)
origin <- rep(1, nrow(cars))
origin[non_US] <- 0
cars$origin <- origin
head(cars)
```

	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb	origin

Mazda RX4	21.0 	6    	160  	110  	3.90 	2.620	16.46	0    	1    	4    	4    	0

Mazda RX4 Wag	21.0 	6    	160  	110  	3.90 	2.875	17.02	0    	1    	4    	4    	0

Datsun 710	22.8 	4    	108  	 93  	3.85 	2.320	18.61	1    	1    	4    	1    	0

Hornet 4 Drive	21.4 	6    	258  	110  	3.08 	3.215	19.44	1    	0    	3    	1    	1

Hornet Sportabout	18.7 	8    	360  	175  	3.15 	3.440	17.02	0    	0    	3    	2    	1

Valiant	18.1 	6    	225  	105  	2.76 	3.460	20.22	1    	0    	3    	1    	1

Now that we have an origin column in our dataframe, let's plot it compared to engine displacement (disp):

```R
par(pin=c(3,3))
plot(cars$disp, cars$origin)
```

You can see from the figure that there is an obvious bias to the left for non_US (origin = 0) cars versus US (origin = 1). So clearly, there is a relationship between origin and displacement---US cars have larger engine displacements.

The question that we may then ask is, "Given the engine displacement of a car, is that car likely to be made in the US or not?" Logistic regression is one approach to dealing with this classification problem.

## Definition: Binary logistic regression
For a binary categorical (this would be the dependant variable) versus a continuous (independent) variable we can describe this simple system with the following equations:
$$\begin{align}
t =& \beta_0 + \beta_1 x \\
Pr(y|x) = p(x) =& \frac{1}{1 + e^{-t}}
\end{align}$$

The function t is called the linear predictor function which includes an intercept and coefficient for the independent variable x. The probability of a positive classification y given x is Pr(y|x). The logit function is the natural logarithm of the odds ratio of the outcome:

$$\begin{align}
logit (p) =& \ln \left(\frac{p(x)}{1-p(x)}\right) = \beta_0 + \beta_1 x
\end{align}$$

To determine if the value of a given variable should be classified as being of y or not y, the following holds:
$$y =\left\{
                \begin{array}{ll}
                  1 \mbox{    if:    } \beta_0 + \beta_1 x + \epsilon > 0\\
                  0 \mbox{    else:}
                \end{array}
              \right. $$

Unlike linear regression, we cannot fit this equation to our data with a simple least-squares fit. Logistic regression requires the use of a different approach called maximum likelihood estimation (MLE). The computational overhead of finding the best fitting solution means that this approach only became viable with the ubiquity of computers.

Let's plot this function and play with the parameters.

```R
#Definition of the logistic function
x <- seq(-20,20, length=100)
par(mfrow=c(2,2))

beta0 <- 0
beta1 <- 1
pr <- 1/(1+exp(- (beta1*x + beta0) ))
plot(x,pr, xlim=c(-20,20), type="l", xlab='Cont. var.', main='beta(0,1)')

beta0 <- -5
beta1 <- 1
t <- beta0 + beta1*x
pr <- 1/(1+exp(- (t) ))
plot(x,pr, xlim=c(-20,20), type="l", xlab='Cont. var.', main='beta(-5,1)')

beta0 <- 0
beta1 <- 0.2
t <- beta0 + beta1*x
pr <- 1/(1+exp(- (t) ))
plot(x,pr, xlim=c(-20,20), type="l", xlab='Cont. var.', main='beta(0,0.2)')

beta0 <- -5
beta1 <- 0.2
t <- beta0 + beta1*x
pr <- 1/(1+exp(- (t) ))
plot(x,pr, xlim=c(-20,20), type="l", xlab='Cont. var.', main='beta(-5,0.2)')
```

Note that since the linear predictor function (t) is inside an exponential term, the offset resulting from the \beta_0 term is scaled non-linearly with the other terms.

Going back to our data for car origin versus engine displacement, we can use the general linear model to calculate the \beta parameters:

```R
model <- glm(origin ~ disp, family=binomial(link='logit'), data=cars)
summary(model)
```

    Call:
    glm(formula = origin ~ disp, family = binomial(link = "logit"),
        data = cars)

    Deviance Residuals:
         Min        1Q    Median        3Q       Max
    -1.51098  -0.19093  -0.07129   0.24559   1.90633

    Coefficients:
                Estimate Std. Error z value Pr(>|z|)
    (Intercept) -8.73478    3.52743  -2.476   0.0133 *
    disp         0.03153    0.01234   2.556   0.0106 *
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    (Dispersion parameter for binomial family taken to be 1)

        Null deviance: 42.340  on 31  degrees of freedom
    Residual deviance: 14.342  on 30  degrees of freedom
    AIC: 18.342

    Number of Fisher Scoring iterations: 7

From the model coefficients, we can plot the probability density function against the real values:

```R
x <- seq(50,500, length=100)
beta0 <- -8.73
beta1 <- 0.0315
t <- beta0 + beta1*x
pr <- 1/(1+exp(- (t) ))
par(pin=c(3,3))
plot(x,pr, type="l", xlab="disp")
points(cars$disp,cars$origin,col="red")
```

## Odds ratio
In order to interpret the coefficients given by the summary of the general linear model, we need to extract them from the logit equation. Here we get an expression that tells us "for every one unit increase in x, the odds multiply by e^{\beta_1}":

$$\begin{align}
odds =& e^{\beta_1}
\end{align}$$

```R
or = exp(model$coefficient[2]) #We can extract the \beta_1 from the model directly
print(or)
```

        disp
    1.032036

We can then interpret this as, "for every increase in engine displacement by 1 cu-in, the likely odds that the car has a US origin increases by 3%."

## Multiple logistic regression
There is no restriction on the number of terms present in the linear predictor function that we used in establishing the model. In the general case we may write the function as:

$$\begin{align}
t =& \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots \\
=& \sum_{i=1}^m \beta_i x_i
\end{align}$$

Note:- It is generally recommended that all predictors used in a model have at least ten corresponding events, although this is a theoretical argument. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0267-3

## Classification based on a predictor
The logistic regression model may then be used as a binary classifier for a general set of predictor variables. So given a set of data to "train" on, we can calculate the coefficients and use the predictor function to determine if a case should be classified as yes/no.

We can do this for the existing data by spliting it into a training and query set and calculating the model using only the training set. We can then use the predictor function to decide which way a case from the query set should be classified.

```R
#Take a random sample of the data
set.seed(42) #This fixes the random number generator so that you get the same "random" sample every time
trainingIndex <- sample(1:nrow(cars), 0.8*nrow(cars))
trainingData <- cars[trainingIndex,]
queryData <- cars[-trainingIndex,]
print(queryData)

logmod <- glm(origin ~ disp, family=binomial(link='logit'), data=trainingData) #Create the model
print(logmod) #\beta_0 + \beta_1 x
dispP <- predict(logmod, queryData)
print(dispP)
```

                    mpg cyl  disp  hp drat    wt  qsec vs am gear carb origin
    Datsun 710     22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1      0
    Valiant        18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1      1
    Merc 450SE     16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3      0
    Merc 450SL     17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3      0
    Merc 450SLC    15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3      0
    Fiat X1-9      27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1      0
    Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4      1

    Call:  glm(formula = origin ~ disp, family = binomial(link = "logit"),
        data = trainingData)

    Coefficients:
    (Intercept)         disp
      -10.43294      0.03948

    Degrees of Freedom: 24 Total (i.e. Null);  23 Residual
    Null Deviance:	    33.65
    Residual Deviance: 5.976 	AIC: 9.976
        Datsun 710        Valiant     Merc 450SE     Merc 450SL    Merc 450SLC
         -6.169078      -1.549898       0.455695       0.455695       0.455695
         Fiat X1-9 Ford Pantera L
         -7.314003       3.424604

From the model of the training data, we can use the \beta coefficients in the linear predictor function (t) to determine whether a classification of US or non-US is more likely.

Recall that if the t value is greater than zero, then the probability of positive classification is greater than 0.5. Reciprocally, if the t value is less than zero, we can negatively classify the input.

Finally, we can compare the results of origin with predicted origin in a table.

```R
dispP[dispP>0] <- 1
dispP[dispP<0] <- 0

queryData['predicted'] <- dispP
print(queryData)

xtabs( ~ predicted + origin, queryData) #Contingency table or confusion matrix
```

                    mpg cyl  disp  hp drat    wt  qsec vs am gear carb origin
    Datsun 710     22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1      0
    Valiant        18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1      1
    Merc 450SE     16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3      0
    Merc 450SL     17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3      0
    Merc 450SLC    15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3      0
    Fiat X1-9      27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1      0
    Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4      1
                   predicted
    Datsun 710             0
    Valiant                0
    Merc 450SE             1
    Merc 450SL             1
    Merc 450SLC            1
    Fiat X1-9              0
    Ford Pantera L         1

             origin
    predicted 0 1
            0 2 1
            1 3 1

## Contingency tables
We can look at this table in terms of a "contingency table" or in the general classification case "confusion matrix." Here the "truth" of car origin is represented by the columns, where zero is non-US and unity is US. The "predicted" classification is presented by row.

This resembles the false positive (type I) false negative (type II) table for determining the validity of a statistical hypothesis.

For a perfect model the diagonal would constitute the sum of all cases and the off-diagonal would sum to zero. In our case the classifier is fairly poor: Of the two US cars in the sample, one was classified as US (true positive) and one as non-US (false negative). Of the five non-US cars, two were classified as non-US (true negative) and three as US (false positive).

### Precision & recall
Precision is the ratio of true positives to true positives plus true negatives. Recall is the true positives to true positives plus false positives.

$$\begin{align}
P = \frac{tp}{tp + fp} \\
R = \frac{tp}{tp + fn}
\end{align}$$

In the case of predicting origin based on engine displacement alone:

$$\begin{align}
P = \frac{1}{1 + 3}= 1/3 \\
R = \frac{1}{1 + 1} = 1/2
\end{align}$$

### F_1 score
The F_1 score is a measure of a test's accuracy that considers both precision and recall.

$$\begin{align}
F_1 =& 2 \frac{P R}{P+R}
\end{align}$$

Calculated for the prior case:

$$\begin{align}
F_1 =& 2 \frac{1/6}{5/6} \\
=& 1/5
\end{align}$$

## Expanding the model
Let's train a new model that includes the cars weight to determine if our contingency table scores improve.

```R
logmod <- glm(origin ~ disp + wt, family=binomial(link='logit'), data=trainingData) #Create the model
print(logmod) #\beta_0 + \beta_1 x
dispP <- predict(logmod, queryData)
dispP[dispP>0] <- 1
dispP[dispP<0] <- 0

queryData['predicted'] <- dispP
print(queryData)

xtabs( ~ predicted + origin, queryData)
```

    Call:  glm(formula = origin ~ disp + wt, family = binomial(link = "logit"),
        data = trainingData)

    Coefficients:
    (Intercept)         disp           wt
       -3.75075      0.06992     -4.49680

    Degrees of Freedom: 24 Total (i.e. Null);  22 Residual
    Null Deviance:	    33.65
    Residual Deviance: 5.259 	AIC: 11.26
                    mpg cyl  disp  hp drat    wt  qsec vs am gear carb origin
    Datsun 710     22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1      0
    Valiant        18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1      1
    Merc 450SE     16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3      0
    Merc 450SL     17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3      0
    Merc 450SLC    15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3      0
    Fiat X1-9      27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1      0
    Ford Pantera L 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4      1
                   predicted
    Datsun 710             0
    Valiant                0
    Merc 450SE             0
    Merc 450SL             0
    Merc 450SLC            0
    Fiat X1-9              0
    Ford Pantera L         1

             origin
    predicted 0 1
            0 5 1
            1 0 1

From the contingency table, you can see that the inclusion of weight improved the classification accuracy significantly with only one error (the Valiant). The F_1 score is then 1/3.

```R

```


- title: Data clustering basics with k-means algorithm
- author: Oscar Vargas Torres
- date: 2018-07-09
- category: Clustering
- tags: k-means

# Data Clustering

This post is based on .

Data clustering is a process of assigning a set of records into subsets, called clusters, such that records in the same cluster are similar and records in different clusters are quite distinct.

A typical clustering process involves the following five steps:

1. pattern representation;
2. dissimilarity measure definition;
3. clustering;
4. data abstraction;
5. assesment of output

In this interactive session, we will be reviewing the *k-means* algorithm.

## The *k-means* Algorithm

The *k-means* algorithm is the most popular and the simplest partitional clustering algorithm. It has many variations. This exercise will review the standard algorithm and several implementations (possibly for different variations).

### Description of the Algorithm

Let $X = \left\{ \mathbf{x}_0, \mathbf{x}_1, \cdots, \mathbf{x}_{n - 1} \right\}$ be a numeric dataset containing $n$ records and $k$ be an integer in $\{1, 2, \cdots, n\}$. The *k-means* algorithm tries to divide the dataset into $k$ clusters $C_0$, $C_1$, $\cdots$, and $C_{k - 1}$ by minimizing the following objective function:
$$E = \sum_{i = 0}^{k - 1}\sum_{\mathbf{x} \in C_i} D(\mathbf{x}, \boldsymbol{\mu}_i),$$
where $D(\cdot, \cdot)$ is a distance measure and $\boldsymbol{\mu}_i$ is the mean of cluster $C_i$, i.e.,
$$\boldsymbol{\mu}_i = \frac{1}{\lvert C_i \rvert} \sum_{\mathbf{x} \in C_i}\mathbf{x}$$

Let $\gamma_i$ be the cluster membership of record $\mathbf{x}_i$ for $i = 0$, $1$, $\cdots$, $n - 1$. That is, $\gamma_i = j$ if $\mathbf{x}_i$ belongs to cluster $C_j$. Then the objective function can be rewritten as:
$$E = \sum_{i = 0}^{n - 1}D(\mathbf{x}_i, \boldsymbol{\mu}_{\gamma_i})$$

To minimize the objective function, the *k-means* algorithm employs an iterative process. At the beginning, the *k-means* algorithm selects $k$ random records from the dataset $X$ as initial cluster centers.

Suppose $\boldsymbol{\mu}_0^{(0)}$, $\boldsymbol{\mu}_1^{(0)}$, $\cdots$, and $\boldsymbol{\mu}_{k - 1}^{(0)}$ are the initial cluster centers. Based on these cluster centers, the *k-means* algorithm updates the cluster memberships $\gamma_0^{(0)}$, $\gamma_1^{(0)}$, $\cdots$, $\gamma_{n - 1}^{(0)}$ as follows:
\begin{equation}
\label{membership-update}
\gamma_i^{(0)} = \underset{0 \leq j \leq k - 1}{\operatorname{argmin}} D(\mathbf{x}_i, \boldsymbol{\mu}_j^{(0)})
\end{equation}
where $\operatorname{argmin}$ is the argument that minimizes the distance. That is, $\gamma_i^{(0)}$ is set to the index of the cluster to which $\mathbf{x}_i$ has the smallest distance.

Based on the cluster memberships $\gamma_0^{(0)}$, $\gamma_1^{(0)}$, $\cdots$, $\gamma_{n - 1}^{(0)}$, the *k-means* algorithm updates the cluster centers as follows:
\begin{equation}
\label{center-update}
\boldsymbol{\boldsymbol{\mu}}_j^{(1)} = \frac{1}{\left| \left\{ i \middle| \gamma_i^{(0)} = j\right\} \right|} \sum_{i = 0, \gamma_i^{(0)} = j}^{n - 1}\mathbf{x}_i, \qquad\text{$j = 0$, $1$, $\cdots$, $k - 1$}
\end{equation}

Then the *k-means* algorithm repeats updating the cluster memberships based on equations \ref{membership-update} and \ref{center-update}.

## Reviewing the math above with some examples

The above can be understood completely if we understand the little pieces, and try to make sense of ideas behind the math. So, don't be scared by the symbols. Let's review some notation first.

**$\lvert C_{i} \rvert$ in this context, means the *cardinality* of the set $C_{i}$**, that is, the size of the set. In mathematics, the cardinality of a set is a measure of the "number of elements of the set".

```python
S_1 = set([1, 2, 3, 4, 5]) # or { 1, 2, 3, 4, 5}
len(S_1)
```

    5

$\mathbf{x} \in C_1$ means, the vector (notice the bold font face) $\mathbf{x}$ *belongs to* (or *is in*) the set $C_1$. For example,

$$\begin{bmatrix} 1 \\ 2 \end{bmatrix} \in C_1$$

means, the vector with two components $\left[1, 2\right]^T$ belongs to the *cluster* $C_1$

### Centroid definition

```python
# Numpy ndarrays are often used in python to represent vectors.
import numpy as np
x = np.array([1, 2])
# Notice we are informally using a 2-dimensional numpy array
# to represent all the elements of the Set C_1
C_1 = np.array([[1, 2],
                [2, 5],
                [4, 7]])

x in C_1
```

    True

$$\sum_{\mathbf{x} \in C_1}\mathbf{x}$$
means: the sum (*component-wise*) of all the vectors that belong to the first cluster $C_1$.

```python
# Use help(np.sum) or np.sum? (then press Tab key) to get more information about np.sum
np.sum(C_1, 0)  # or C_1.sum(0)
```

    array([ 7, 14])

$$\boldsymbol{\mu}_1 = \frac{1}{\lvert C_1 \rvert} \sum_{\mathbf{x} \in C_1}\mathbf{x}$$

The *centroid* (or cluster center) of the first cluster $C_1$ is computed with with a scalar multiplication. For our example:

```python
1/len(C_1)
```

    0.3333333333333333

```python
1 /len(C_1) * C_1.sum(0)
```

    array([2.33333333, 4.66666667])

In other words, the *centroid* formula is simply computing the *arithmetic mean* (component-wise) of the all the vectors of a given cluster:

 * First component:
 $$\frac{1 + 2 + 4}{3} = 2.33333333$$
 * Second component:
 $$\frac{2 + 5 + 7}{3} = 4.66666667$$

### Objective function

#### First form of the objective function

Let's review:
$$E = \sum_{i = 0}^{k - 1}\sum_{\mathbf{x} \in C_i} D(\mathbf{x}, \boldsymbol{\mu}_i)$$

The inner summation,
$$\sum_{\mathbf{x} \in C_i} D(\mathbf{x}, \boldsymbol{\mu}_i)$$
says something like: compute the sum of all the distances for the *i*-th cluster. Notice the requirement in the summation index: $\mathbf{x} \in C_i$ (*only* consider those vectors that belong to cluster $C_i$).

The outer summation,
$$E = \sum_{i = 0}^{k - 1} \text{sum of distances for the $i$-th cluster}$$
simply says, that you have to accumulate the inner computation for all the clusters.

#### Second form of the objective function and cluster membership notation

The second form of the objective function, uses more compact notation:

$$E = \sum_{i = 0}^{n - 1}D(\mathbf{x}_i, \boldsymbol{\mu}_{\gamma_i})$$

It says: *compute the sum of the distances between a vector and it's corresponding cluster center*, for all the vectors in the dataset. Notice that $D(\mathbf{x}_i, \boldsymbol{\mu}_{\gamma_i})$ implies that we have a way to compute the cluster membership function: $\gamma_i$.

### Cluster membership update

Let's review \ref{membership-update}:

$$\gamma_i^{(0)} = \underset{0 \leq j \leq k - 1}{\operatorname{argmin}} D(\mathbf{x}_i, \boldsymbol{\mu}_j^{(0)})$$

Text above says:
> $\gamma_{i}^0$ is set to the index of the cluster to which $\mathbf{x}_{i}$ has the smallest distance.

Notice that we are assuming an **iterative process**. The superscript introduced here $^{(0)}$ refers to the number of iteration: the cluster membership update $0$.

The notation
$$\underset{0 \leq j \leq k - 1}{\operatorname{argmin}} \text{distance to $j$-th cluster}$$
means that we have to choose the cluster **index** $j$ (that could be any of the integer values from $0$ to $k - 1$) that minimizes the distance to current record $\mathbf{x}_i$.

### Cluster center update

Now, let's review cluster center update formula \ref{center-update} notation:

$$\boldsymbol{\boldsymbol{\mu}}_j^{(1)} = \frac{1}{\left| \left\{ i \middle| \gamma_i^{(0)} = j\right\} \right|} \sum_{i = 0, \gamma_i^{(0)} = j}^{n - 1}\mathbf{x}_i, \qquad\text{$j = 0$, $1$, $\cdots$, $k - 1$}$$

We are again using the *membership function* $\gamma_i$ to simplify notation. The outer index $j$, says that we have to update the cluster center computation for each one of the $k$ clusters.

For a specific value of $j$ (we are *re*-computing the center for $j$-th cluster), we simply have to use the cluster center formula, but we have to consider only those records that belong to this cluster:
$$\sum_{i = 0, \gamma_i^{(0)} = j}^{n - 1}\mathbf{x}_i$$
Notice the requirement in the summation index: sum every record, as long as $\gamma_{i} = j$ (as long as record belongs to $j$-th cluster).

$\left| \left\{ i \middle| \gamma_i^{(0)} = j\right\} \right|$ is simply the number of elements of $j$-th cluster. It says: the cardinality of the set of all indexes $i$, *such that* $i$-th record belongs to cluster $j$.

## Exercises

Use [this dataset](data/600points.csv).

### Visualization of dataset for exercises

```python
%matplotlib inline
```

```python
import pandas as pd

input_file = "data/600points.csv"

# Input file has labeled records. 'label' is one of 1, 2 or 3.
labeled_df = pd.read_csv(input_file,
                         header=None,
                         names=['id', 'x', 'y', 'label'])
labeled_df.dtypes
```

    id         int64
    x        float64
    y        float64
    label      int64
    dtype: object

```python
# To visualize "named" colors see:
# https://matplotlib.org/1.4.3/examples/color/named_colors.html
label_to_color = {1: 'red', 2: 'deepskyblue', 3: 'green'}
colors = labeled_df.loc[:, 'label'].map(label_to_color)
labeled_df.plot.scatter('x', 'y', grid=True, c=colors);
```

```python
# Unlabeled data (prior to "clustering").
df = labeled_df.loc[:,['x', 'y']]
df.plot.scatter('x', 'y', grid=True);
```

### To practice your Python

1. Use the kmeans implementation from scikit-learn to run the algorithm for k = 3 clusters and a max of 100 iterations. You can install that using pip (`$ pip install scikit-learn`) or anaconda, (`$ conda install scikit-learn`).
2. Use python's [docopt](https://github.com/docopt/docopt) or `argparse` (available in the standard library) to parse the following command line options:

    ```
    Allowed options:
      --help                produce help message
      --datafile arg        the data file
      --k arg (=3)          number of clusters
      --maxiter arg (=100)  maximum number of iterations
    ```

3. Implement the algorithm by yourself and compare with the results obtained with scikit-learn.
4. Use [numba](https://numba.pydata.org/) to try to improve the performance of your implementation.
5. Use [matplotlib](https://matplotlib.org/) (or another library of your preference) to visualize data and your results. Fine tune your plot so that the assesment of the result is easier to understand.

### To practice with .NET (C# or F#)

1. Use the *k-means* implementation from version 0.2 of [ML.NET](https://github.com/dotnet/machinelearning) as a reference implementation for comparison (correctness of results, performance, etc).
2. Use [.NET's docopt port](https://github.com/docopt/docopt.net) to parse command line options for your program (see analogous section for Python).
3. Implement your own version of k-means using your preferred .NET language (C#, F#, VB.NET).
4. Visualize your results with [XPlot](https://github.com/fslaborg/XPlot), [FSharp.Charting](https://github.com/fslaborg/FSharp.Charting), [Live-Charts](https://github.com/Live-Charts/Live-Charts) or any other good quality library of your preference.

### To practice with GPUs and Python/R/C

1. Take a look at [kmcuda](https://github.com/src-d/kmcuda). See what the possibilities are for your favorite programming language.
2. Visualize your results with the best library for your language of choice.

### To practice with GPUs or Multicore CPUs and Haskell

1. Take a look at [Accelerate](http://www.acceleratehs.org/) and the different supported backends (e.g. Multicore CPUs and GPUs).
2. Study and explain the [*k-means* sample](https://github.com/AccelerateHS/accelerate-examples/blob/master/examples/kmeans/Kmeans.hs)
3. Compare the implementations and performance.
4. Visualize using [`matplotlib` bindings for Haskell](https://github.com/abarbu/matplotlib-haskell).

### To practice with Scala and Spark

1. Try to use the existing Scala implementations:

    * With [Spark's MLLib](https://github.com/apache/spark/tree/master/mllib/src/main/scala/org/apache/spark/mllib/clustering)
    * With [Spire](https://github.com/non/spire/blob/master/examples/src/main/scala/spire/example/kmeans.scala)
2. Study and explain.
3. Hard: study if it is possible to improve performance using [sbt-javacpp](https://github.com/bytedeco/sbt-javacpp) bindings for CUDA and previous implementation code/algorithms.
4. Visualize using [Vegas](https://github.com/vegas-viz/Vegas) or any other good quality library of your preference.


# Setup

This notebook has been created using Julia v1.5. Run the cells below to setup the project.

```julia
import Pkg
Pkg.activate(".")
Pkg.instantiate()
```

```julia
using LinearAlgebra

import Jacobi
using PolynomialBases
using SummationByPartsOperators

import PyPlot; plt = PyPlot
using LaTeXStrings, PyCall
using Latexify
import SymPy; sp = SymPy

ticker = pyimport("matplotlib.ticker")

# line cycler adapted to colourblind people
cycler = pyimport("cycler")
cycler.cycler
colours = ["#E69F00", "#56B4E9", "#009E73", "#0072B2", "#D55E00", "#CC79A7", "#F0E442"]
linestyles = ["-", "--", "-.", ":", "-", "--", "-."]
markers = ["4", "2", "3", "1", "+", "x", "."]
colourblind_cycler = (cycler.cycler(color=colours) + cycler.cycler(linestyle=linestyles))
plt.rc("axes", prop_cycle=colourblind_cycler)

plt.rc("text", usetex=true)
plt.rc("text.latex", preamble="\\usepackage{newpxtext}\\usepackage{newpxmath}\\usepackage{commath}\\usepackage{mathtools}")
plt.rc("font", family="serif", size=18.)
plt.rc("savefig", dpi=200)
plt.rc("legend", fontsize="medium", fancybox=true, framealpha=0.5)

isdir("../figures") || mkdir("../figures")
```

# Computation of the Butcher Coefficients

```julia
SummationByPartsOperators.mass_matrix(D::LegendreDerivativeOperator) = Diagonal(D.Δx * D.basis.weights)

function butcher_coefficients(M, D, x, tL, tR)
    Δ = tR'*x - tL'*x

    # grid oscillations
    kernel = nullspace(M \ D' * M)
    if size(kernel, 2) > 1
        throw(ArgumentError("The derivative operator D must be nullspace consistent!"))
    end
    osc = vec(kernel)

    # solve the linear system -> least norm solution
    A = qr(D, Val(true)) \ (I - (osc * osc' * M) / (osc'*M*osc))
    # and set the initial values to zero
    for j in 1:size(A,2)
        A[:,j] .-= tL' * A[:,j]
    end
    A ./= Δ

    A, diag(M) ./ Δ, (x .- tL' * x) ./ Δ
end

function butcher_coefficients(basis::NodalBasis)
    M = Diagonal(basis.weights)
    D = basis.D
    x = basis.nodes
    tL = basis.interp_left
    tR = basis.interp_right

    butcher_coefficients(M, D, x, tL, tR)
end

function butcher_coefficients(D_op::SummationByPartsOperators.AbstractDerivativeOperator)
    M = mass_matrix(D_op)
    D = Matrix(D_op)
    x = grid(D_op)
    tL = zero(x); tL[1] = 1
    tR = zero(x); tR[end] = 1

    butcher_coefficients(M, D, x, tL, tR)
end
```

# Lobatto Legendre Nodes: Lobatto IIIA Methods

```julia
D_op = legendre_derivative_operator(0., 1., 2)
A, b, c = butcher_coefficients(D_op)
display(A)
# the analytical values
[0 0; 1/2 1/2]
```

```julia
D_op = legendre_derivative_operator(0., 1., 3)
A, b, c = butcher_coefficients(D_op)
display(A)
# the analytical values
[0 0 0; 5/24 1/3 -1/24; 1/6 2/3 1/6]
```

```julia
D_op = legendre_derivative_operator(0., 1., 4)
A, b, c = butcher_coefficients(D_op)
display(A)
# the analytical values
sqrt5 = sqrt(5)
[0 0 0 0;
    (11+sqrt5)/120 (25-sqrt5)/120 (25-13sqrt5)/120 (-1+sqrt5)/120;
    (11-sqrt5)/120 (25+13sqrt5)/120 (25+sqrt5)/120 (-1-sqrt5)/120;
    1/12 5/12 5/12 1/12
]
```

```julia

```

# Gauss/Radau Legendre Nodes: None of the Classical RK Schemes

```julia
basis = GaussLegendre(1)
A, b, c = butcher_coefficients(basis)
A
```

```julia
basis = GaussLegendre(2)
A, b, c = butcher_coefficients(basis)
display(A)
display(b)
display(c)

display(A - [-12+10*sqrt(15) -48+16*sqrt(15) -48+10*sqrt(15);
        45+10*sqrt(15) 16*sqrt(15) -45+10*sqrt(15);
        48+10*sqrt(15) 48+16*sqrt(15) 12+10*sqrt(15)] / (72 * sqrt(15))) # A
display(b - [5, 8, 5] / 18) # b
display(c - [5 - sqrt(15), 5, 5 + sqrt(15)] / 10) # c
```

```julia
# folowwing the derivation using this example

let T = sp.symbols("T", real=true)
    basis = GaussLegendre(2, sp.Sym)
#     basis.D * 2/T / (sqrt(sp.Sym(15))/3T) |> latexify |> println
#     basis.nodes |> latexify |> println
#     basis.weights |> latexify |> println
    tL = sp.simplify(basis.interp_left)
#     tL * 6 |> println
    tR = sp.simplify(basis.interp_right)
#     tR * 6 |> println

    sqrt3 = sqrt(sp.Sym(3))
    τ = sp.simplify.((basis.nodes .+ 1) .* T / 2)
#     τ |> latexify |> println
#     τ = [T/2 - T/2sqrt3, T/2, T/2 + T/2sqrt3]
    D = sqrt(sp.Sym(15))/(3T) * [-3 4 -1; -1 0 1; 1 -4 3]
    M = Diagonal(T/18 * [5, 8, 5])
#     sp.simplify.(D * τ.^0)
#     k = 2; sp.simplify.(D * τ.^k - k * τ.^(k-1))
#     k = 2; sp.simplify(dot(basis.interp_left, τ.^k))
#     k = 2; sp.simplify(dot(basis.interp_right, τ.^k))
#     sp.simplify.(M * D + D' * M - (tR * tR' - tL * tL'))
#     M \ (D' * M) * 30T/sqrt(sp.Sym(15))
#     sp.simplify.((3 .* τ.^2 .- 1) * (-5 / (3*T^2/4 - 1))) # highest resolvable Legendre polynomial
    osc = [4, -5, 4]
#     osc' * M * D
#     osc' * M
    F = sp.Sym.(I - osc * osc' * M / dot(osc, M, osc))
    F * 18

    sqrt15 = sqrt(sp.Sym(15))
    a1, a2, b1, b2 = sp.symbols("a_1, a_2, b_1, b_2", real=true)
    basis_V0 = [a1, a2, -(4+sqrt15)*a1 + 2*(5+sqrt15)*a2/5]
    basis_V1 = [b1, b2, -b1 + 2*b2]
    J = 3T / (24*sqrt15) * [2 4*(sqrt15-3) -2; 5 4sqrt15 -5; 2 4*(sqrt15+3) -2]
    sp.simplify.(tL' * J)
    sp.simplify.(J * D * 6)
    sp.simplify.(J * D * basis_V0)
    sp.simplify(D * J * 2)
    sp.simplify.(D * J * basis_V1)

    sp.simplify.(J * F / (T/72sqrt15))
    M
    τ * 10/T
end
```

```julia
q = Jacobi.Quadrature(Jacobi.GRJM, 2)
M = Diagonal(Jacobi.qweights(q))
D = Jacobi.qdiff(q)
x = Jacobi.qzeros(q)
tL = Jacobi.interp_mat([-one(eltype(x))], x) |> vec
tR = Jacobi.interp_mat([+one(eltype(x))], x) |> vec
A, b, c = butcher_coefficients(M, D, x, tL, tR)
A
```

```julia

```

# The FD SBP Operators used in the Numerical Tests

```julia
N = 3
order = 2
D_op = derivative_operator(MattssonNordström2004(), 1, order, 0//1, 1//1, N)
A_, b, c = butcher_coefficients(D_op)
A = rationalize.(A_, tol=1.0e-5)
display(A)
display(b)
display(c)
latexify(A) |> println
latexify(b) |> println
latexify(c) |> println
```

```julia
N = 9
order = 2
D_op = derivative_operator(MattssonNordström2004(), 1, order, 0//1, 1//1, N)
A_, b, c = butcher_coefficients(D_op)
A = rationalize.(A_, tol=1.0e-5)
display(A)
display(b)
display(c)
latexify(A) |> println
latexify(b) |> println
latexify(c) |> println
```

```julia
N = 9
order = 4
D_op = derivative_operator(MattssonNordström2004(), 1, order, 0//1, 1//1, N)
A_, b, c = butcher_coefficients(D_op)
A = rationalize.(A_, tol=1.0e-5)
display(A)
display(b)
display(c)
latexify(A) |> println
latexify(b) |> println
latexify(c) |> println
```

```julia

```

```julia

```

# Numerical Tests Using FD SBP Operators

```julia
function compute_error_nonstiff(λ, N, order)
    D_op = derivative_operator(MattssonNordström2004(), 1, order, 0., 1., N)

    M = mass_matrix(D_op)
    D = Matrix(D_op)
    t = SummationByPartsOperators.grid(D_op)
    Δ = last(t) - first(t)

    # grid oscillations
    kernel = nullspace(M \ D' * M)
    if size(kernel, 2) != 1
        throw(ArgumentError("The derivative operator D must be nullspace consistent!"))
    end
    osc = vec(kernel)

    # solve the linear system -> least norm solution
    A = qr(D, Val(true)) \ (I - (osc * osc' * M) / (osc'*M*osc))
    # and set the initial values to zero
    for j in 1:size(A,2)
        A[:,j] .-= A[1,j]
    end

    # compute solution
    u0 = fill(1., size(D,2))
    u = (I - λ*A) \ u0

    t, u, abs(last(u) - exp(λ*(last(t)-first(t))))
end

function compute_error_stiff(λ, N, order)
    D_op = derivative_operator(MattssonNordström2004(), 1, order, 0., 1., N)

    M = mass_matrix(D_op)
    D = Matrix(D_op)
    t = SummationByPartsOperators.grid(D_op)
    Δ = last(t) - first(t)

    # grid oscillations
    kernel = nullspace(M \ D' * M)
    if size(kernel, 2) != 1
        throw(ArgumentError("The derivative operator D must be nullspace consistent!"))
    end
    osc = vec(kernel)

    # solve the linear system -> least norm solution
    A = qr(D, Val(true)) \ (I - (osc * osc' * M) / (osc'*M*osc))
    # and set the initial values to zero
    for j in 1:size(A,2)
        A[:,j] .-= A[1,j]
    end

    # compute solution
    u0 = fill(1., size(D,2))
    u = (I - λ*A) \ (u0 - A * (@. (λ+1) * exp(-t)))

    t, u, abs(last(u) - exp(-(last(t)-first(t))))
end

function convergence_orders(Ns, errors)
    @assert length(Ns) == length(errors)

    orders = zero(errors)
    orders[1] = NaN
    for i in 2:length(errors)
        orders[i] = - log(errors[i-1] / errors[i]) / log(Ns[i-1] / Ns[i])
    end
    orders
end

#t, u, e = compute_error_stiff(-1000.0, 100, 2); println(e); Plots.plot(t, u); Plots.plot!(t, exp.(-t))
```

## Non-stiff Problem

```julia
Ns = 20:10:130

plt.close("all")
fig, ax = plt.subplots(1, 1)

for (idx,order) in enumerate(2:2:8)
    errors = zeros(length(Ns))
    for (idx,N) in enumerate(Ns)
        _, _, errors[idx] = compute_error_nonstiff(-1.0, N, order)
    end
    plt.plot(Ns, errors, label="Interior order $order", marker=markers[idx], ms=12, mew=1.5, lw=2)
    println(convergence_orders(Ns, errors))

    order_color = "silver"
    if order == 2
        plt.plot([Ns[1],Ns[end]], 7.e-5.*[1, (Ns[1]/Ns[end])^2], "-", color=order_color)
        plt.annotate("Order 2", (Ns[1], 6.e-6), color=order_color)
    elseif order == 4
        plt.plot([Ns[1],Ns[end]], 5.e-8.*[1, (Ns[1]/Ns[end])^4], "-", color=order_color)
        plt.annotate("Order 4", (Ns[1], 2.e-9), color=order_color)
    elseif order == 6
        plt.plot([Ns[1],Ns[end]], 1.e-10.*[1, (Ns[1]/Ns[end])^6], "-", color=order_color)
        plt.annotate("Order 6", (Ns[1], 1.5e-10), color=order_color)
    elseif order == 8
        plt.plot([Ns[1],60], 7.e-13.*[1, (Ns[1]/60)^8], "-", color=order_color)
        plt.annotate("Order 8", (Ns[1], 7.e-15), color=order_color)
    end
end
sleep(0.1)

plt.xlabel(L"N")
plt.xscale("log")
plt.ylabel("Error")
plt.yscale("log")
ax.xaxis.set_minor_formatter(ticker.NullFormatter())
ax.set_xticks([20, 30, 40, 60, 80, 100, 130], [])
ax.get_xaxis().set_major_formatter(ticker.ScalarFormatter())
plt.legend(loc="upper left", bbox_to_anchor=(1,1))
plt.savefig("../figures/convergence_scalar_nonstiff.pdf", bbox_inches="tight")
```

## Stiff Problem

```julia
Ns = 20:20:200

plt.close("all")
fig, ax = plt.subplots(1, 1)

for (idx,order) in enumerate(2:2:8)
    errors = zeros(length(Ns))
    for (idx,N) in enumerate(Ns)
        _, _, errors[idx] = compute_error_stiff(-1000.0, N, order)
    end
    plt.plot(Ns, errors, label="Interior order $order", marker=markers[idx], ms=12, mew=1.5, lw=2)
    println(convergence_orders(Ns, errors))

    order_color = "silver"
    if order == 2
        plt.plot([Ns[1],80], 2.e-5.*[1, (Ns[1]/80)^1], "-", color=order_color)
        plt.annotate("Order 1", (30, 1.5e-5), color=order_color)
        plt.plot([80,Ns[end]], 5.e-6.*[1, (80/Ns[end])^2], "-", color=order_color)
        plt.annotate("Order 2", (80, 6.e-6), color=order_color)
    elseif order == 4
        plt.plot([Ns[1],Ns[end]], 2.5e-6.*[1, (Ns[1]/Ns[end])^2.3], "-", color=order_color)
        plt.annotate("Order 2.3", (Ns[1], 3.e-6), color=order_color)
    elseif order == 6
        plt.plot([Ns[1],Ns[end]], 5.e-8.*[1, (Ns[1]/Ns[end])^3], "-", color=order_color)
        plt.annotate("Order 3", (Ns[1], 6.e-8), color=order_color)
    elseif order == 8
        plt.plot([Ns[1],Ns[end]], 7.e-9.*[1, (Ns[1]/Ns[end])^4.4], "-", color=order_color)
        plt.annotate("Order 4.4", (Ns[1], 2.5e-10), color=order_color)
    end
end
sleep(0.1)

plt.xlabel(L"N")
plt.xscale("log")
plt.ylabel("Error")
plt.yscale("log")
ax.xaxis.set_minor_formatter(ticker.NullFormatter())
ax.set_xticks([20, 30, 40, 60, 80, 100, 140, 200], [])
ax.get_xaxis().set_major_formatter(ticker.ScalarFormatter())
plt.legend(loc="upper left", bbox_to_anchor=(1,1))
plt.savefig("../figures/convergence_scalar_stiff.pdf", bbox_inches="tight")
```

```julia

```


# Problem Sheet 3 Question 1

The general form of the population growth differential equation
\begin{equation} y^{'}=t-y, \ \ (0 \leq t \leq 4) \end{equation}
with the initial condition
\begin{equation}y(0)=1,\end{equation}
Has the exact soulation. \begin{equation} y= 2e^{-t}+t-1\end{equation}

#### Setting up Libraries

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

## Defining the function
\begin{equation}f(t,y)=t-y$$

```python
def myfun_ty(t,y):
    return t-y
```

## Initial Setup
Defining the step size $h$ from the interval range $a\leq t \leq b$ and number of steps $N$
\begin{equation}h=\frac{b-a}{h}.\end{equation}
This gives the discrete time steps,
\begin{equation}t_{i}=t_0+ih,\end{equation}
where $t_0=a$.

```python
# Start and end of interval
b=4
a=0
# Step size
N=8
h=(b-a)/(N)
t=np.arange(a,b+h,h)
```

## Setting up the initial conditions of the equation
\begin{equation}w_0=IC\end{equation}

```python
# Initial Condition
IC=1
w=np.zeros(N+1)
y=(IC+1)*np.exp(-t)+t-1#np.zeros(N+1)
w[0]=IC
```

## 2nd Order Runge Kutta (Mid-method)
\begin{equation}k_1=f(t,y),\end{equation}
\begin{equation}k_2=f(t+\frac{h}{2},y+\frac{h}{2}k_2),\end{equation}
\begin{equation}w_{i+1}=w_{i}+h(k_2).\end{equation}

```python
for k in range (0,N):
    k1=myfun_ty(t[k],w[k])
    k2=myfun_ty(t[k]+h/2,w[k]+h/2*k1)
    w[k+1]=w[k]+h*(k2)
```

## Plotting Results

```python
fig = plt.figure(figsize=(10,4))
# --- left hand plot
ax = fig.add_subplot(1,3,1)
plt.plot(t,w, '--',color='blue')
#ax.legend(loc='best')
plt.title('Numerical Solution h=%s'%(h))

ax = fig.add_subplot(1,3,2)
plt.plot(t,y,color='black')
plt.title('Exact Solution ')

ax = fig.add_subplot(1,3,3)
plt.plot(t,y-w, 'o',color='red')
plt.title('Error')
# --- title, explanatory text and save
fig.suptitle(r"$y'=t-y,   y(0)=%s$"%(IC), fontsize=20)
plt.tight_layout()
plt.subplots_adjust(top=0.85)
```

```python

```

```python

```

Chapter 03 - Runge Kutta/3_Problem_Sheet/302a_Problem_Sheet.ipynb


```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

\begin{equation}
\begin{array}{ll}
\min_{x_1,x_2} & -x_1-x_2 \\
\text{s. a. }  & 50x_1+24x_2\leq 2400 \\
               & 30x_1+33x_2\leq 2100 \\
               & -x_1\leq -45 \\
               & -x_2\leq -5,
\end{array}
\end{equation}

```python

```

```python
def x2_v1(x1):
    return (2400 - 50*x1)/24
def x2_v2(x1):
    return (2100 - 30*x1)/33
```

```python
x1 = np.linspace(43, 47)
```

```python
x1 = np.linspace(0, 100)
```

```python
plt.plot(x1, x2_v1(x1), 'magenta', label = "Eq1") #desigualdad 1
plt.plot(x1, x2_v2(x1), label = "Eq2") # desigualdad 2
#plt.plot([45, 45], [0, 25], 'k')
#plt.plot([43, 47], [5, 5], 'r')
#plt.xlim(xmin = 44, xmax = 46)
#plt.ylim(ymin = 4.5, ymax = 6.5)
#plt.scatter([45], [6.25], color = 'red')
#plt.fill_between(np.array([45, 45.6]), x2_v1(np.array([45, 45.6])),
#                5*np.ones(2), alpha = .2, color = 'orange')

plt.hlines(5, 0, 100, label = "Eq4")
plt.vlines(45, -100, 100, color = "r", label = "Eq3")
plt.legend()
plt.xlabel("$x_1$", fontsize = 18)
plt.ylabel("$x_2$", fontsize = 18)
plt.show()
```

```python
plt.plot(x1, x2_v1(x1), 'magenta', label = "Eq1") #desigualdad 1
plt.plot(x1, x2_v2(x1), label = "Eq2") # desigualdad 2
plt.hlines(5, 0, 100, label = "Eq4")
plt.vlines(45, -100, 100, color = "r", label = "Eq3")
plt.legend()
plt.xlabel("$x_1$", fontsize = 18)
plt.ylabel("$x_2$", fontsize = 18)
plt.xlim(xmin = 44, xmax = 70)
plt.ylim(ymin = 2, ymax = 48)
plt.show()
```

```python
plt.plot(x1, x2_v1(x1), 'magenta', label = "Eq1") #desigualdad 1
plt.plot(x1, x2_v2(x1), label = "Eq2") # desigualdad 2
plt.hlines(5, 0, 100, label = "Eq4")
plt.vlines(45, -100, 100, color = "r", label = "Eq3")
plt.scatter([45], [6.25], color = 'k', s = 105)
#plt.fill_between(np.array([45, 45.6]), x2_v1(np.array([45, 45.6])),
#                5*np.ones(2), alpha = .2, color = 'orange')
plt.legend()
plt.xlabel("$x_1$", fontsize = 18)
plt.ylabel("$x_2$", fontsize = 18)
plt.xlim(xmin = 44, xmax = 47)
plt.ylim(ymin = 2, ymax = 10)
plt.grid()
plt.show()
```

```python
import scipy.optimize as opt
```

```python
c = np.array([-1, -1])              ## Coeficientes de la función objetivo
A = np.array([[50, 24], [30, 33]])
b = np.array([2400, 2100])
x1_bound = (45, None)
x2_bound = (5, None)
```

```python
c
```

    array([-1, -1])

```python
A
```

    array([[50, 24],
           [30, 33]])

```python
B = np.array([[2, 1, 0],[0, 1, -2],[1, 0, 4]]); B
```

    array([[ 2,  1,  0],
           [ 0,  1, -2],
           [ 1,  0,  4]])

```python
opt.linprog?
```

```python
res = opt.linprog(c, A_ub = A, b_ub = b,
                  bounds=(x1_bound, x2_bound), options={'disp':True})
```

    Optimization terminated successfully.
             Current function value: -51.250000
             Iterations: 4

```python
res
```

         fun: -51.25
     message: 'Optimization terminated successfully.'
         nit: 4
       slack: array([  0.  , 543.75,   0.  ,   1.25])
      status: 0
     success: True
           x: array([45.  ,  6.25])

```python
res.x
```

    array([45.  ,  6.25])

## Actividad

```python

```


# Partikkel skytes i tyngdefeltet

En partikkel skytes ut i tyngdefeltet med fart $v_0=|\vec{v}(0)|$ i en retning som danner en vinkel $\alpha$ med horisontalen (z-aksen). Vi antar at det kun er tyngdekraften som virker, så Newton's annen lov gir oss at

$$
\begin{equation}
m \vec{a} = -mg \mathbf{k},
\end{equation}
$$

der $m$ er massen til partikkelen. Vi kan se bort ifra massen siden vi kan dele begge sider på $m$.

Vi kan integrere Newton's annen low to ganger og finne posisjonen $\vec{r}(t)=x \mathbf{i} + z \mathbf{k}$ for partikkelen. Bruker først at $\vec{a}=d\vec{v}/dt$ slik at vi får

$$
\begin{equation}
\frac{d \vec{v}}{dt} = -g\mathbf{k}.
\end{equation}
$$

Integrerer opp og får

$$
\begin{equation}
\vec{v}(t) - \vec{v}(0) = -g t \mathbf{k}.
\end{equation}
$$

Bruker så at $\vec{v} = d \vec{r} / dt$ og integrerer på samme måte fra

$$
\begin{equation}
\frac{d \vec{r}}{dt} = \vec{v}(0) - g t \mathbf{k}.
\end{equation}
$$

Ender opp med at

$$
\begin{equation}
\vec{r}(t) - \vec{r}(0) = \vec{v}(0) t - \frac{1}{2}g t^2 \mathbf{k},
\end{equation}
$$

der vi kan ta bort $\vec{r}(0)$ siden vi regner at utskytningen starter i origo.

Vektoren $\vec{v}(0)$ kan dekomponeres langs $x$ og $y$-akser ved å bruke vinkelen $\alpha$. Får at

$$
\begin{equation}
\vec{v}(0) = v_0 \cos(\alpha) \mathbf{i} + v_0 \sin(\alpha) \mathbf{k},
\end{equation}
$$

og dermed har vi funnet posisjonsvektoren $\vec{r}$ med komponenter

$$
\begin{align}
  x &= v_0 \cos(\alpha) t,\\
  z &= v_0 \sin(\alpha) t - \frac{1}{2}gt^2.
\end{align}
$$

slik at posisjonsvektoren blir

$$
\begin{align}
\vec{r}(t) &= x(t) \mathbf{i} + z(t) \mathbf{k}, \notag\\
           &= v_0 \cos(\alpha) t \mathbf{i} + (v_0 \sin(\alpha) t - \frac{1}{2}gt^2) \mathbf{k}.
\end{align}
$$

Hastighetsvektoren kan også samles sammen til

$$
\begin{equation}
\label{eq:v}
\vec{v}(t) = v_0 \cos(\alpha) \mathbf{i} + (v_0 \sin(\alpha) - gt) \mathbf{k}.
\end{equation}
$$

Vi skal nå modellere dette med et interaktivt plot som viser partikkelbanen og vektoren. Til dette trenger vi å gjøre et par observasjoner.

 1. Hva er domenet til partikkelen?
   1. Hvor høyt skytes partikkelen?
   2. Hvor lang tid tar det før den kommer ned igjen?

Finner svar på A ved å se på når hastigheten i z-retningen er lik null (fra $z$-komponenten til $\vec{v}$ over)

$$
\begin{align*}
v_0 \sin(\alpha) - g t_0 &= 0 \\
t_0 &= \frac{v_0 \sin(\alpha)}{g}.
\end{align*}
$$

Høyden ved $t_0$ er gitt ved $z(t_0)$, som vi kaller $H$

$$
z(t_0) = H = \frac{(v_0 \sin(\alpha))^2}{2 g}
$$

Så domenet for $z$ må være minst $[0, H]$ for å få med hele partikkelbanen.

Tiden det tar for partikkelen å komme ned igjen finner vi ved å løse $z(t) = 0$

$$
\begin{align*}
v_0 \sin(\alpha) t - \frac{1}{2}gt^2 &= 0 \\
t\left(v_0 \sin(\alpha) - \frac{1}{2}g t \right) &= 0
\end{align*}
$$

som løses enkelt til

$$
t = 0 \lor t = \frac{2 v_0 \sin(\alpha)}{g}
$$

Vi kaller tiden da partikkelen igjen treffer bakken ($z=0$) for $T=2 v_0 \sin(\alpha)/g$. Vi innser at simuleringen må gå minst i $t \in [0, T]$.

Vi finner domenenet for $x$-aksen ved å sette sluttiden inn i uttrykket for $x$

$$
x\left(T\right) = v_0 \cos(\alpha) T.
$$

Så $x$-domenet blir $[0, v_0 \cos(\alpha) T]$.

Da er vi klare til å starte plottingen!

Vi starter med å importere verktøyene vi trenger. Vi vil lage et interaktivt plot som viser partikkelen langs en bane. Da kan vi bruke [ipywidgets](https://ipywidgets.readthedocs.io/en/latest/).

```python
from ipywidgets import interact
import matplotlib.pyplot as plt
import numpy as np
```

```python
%matplotlib inline

g = 9.81
v0 = 10
a = np.pi/4
T = 2*v0*np.sin(a)/g
N = 100 # Bruker interval med N punkter
t0 = np.linspace(0, T*1.1, N)
```

```python
# Beregner hele partikkelbanen først
x0 = v0*np.cos(a)*t0
z0 = v0*np.sin(a)*t0 - 0.5*g*t0**2

def partikkelbane(t):
    # plot hele partikkelbanen
    plt.figure()
    plt.plot(x0, z0, 'b')

    # plot partikkelen
    x = v0*np.cos(a)*t
    z = v0*np.sin(a)*t - 0.5*g*t**2
    plt.plot(x, z, 'ok')

    # plot en hastighetsvektor ved t0 og t
    plt.arrow(0, 0, v0*np.cos(a)*T/4, v0*np.sin(a)*T/4, head_width=v0*T/100)
    plt.arrow(x, z, v0*np.cos(a)*T/4, (v0*np.sin(a)-g*t)*T/4, head_width=v0*T/100)

    # posisjonsvektor
    plt.arrow(0, 0, x, z, length_includes_head=True)

    plt.ylim(z0.min(), 2*z0.max())

    plt.text(0.25*v0*np.cos(a), 0.3*v0*np.sin(a), r'$\vec{v}(0)$')
    plt.text(x+0.25*v0*np.cos(a), z+0.2*(v0*np.sin(a)-g*t), r'$\vec{v}(t)$')
    plt.text(0.5*x, 0.4*z, r'$\vec{r}(t)$')

    plt.show()

interact(partikkelbane, t=(0, T, T/20))
```

    interactive(children=(FloatSlider(value=0.7208020195581524, description='t', max=1.4416040391163047, step=0.07…

    <function __main__.partikkelbane(t)>

Vi kan nå beregne buelengden, eller strekningen partikkelen tilbakelegger, ved å integrere numerisk

$$
\begin{equation}
L = \int_{0}^{t_m}|\vec{v}(t)|dt
\end{equation}
$$

Numerisk integrasjon kan gjøres i `numpy` ved å bruke trapes-metoden [trapz](https://docs.scipy.org/doc/numpy/reference/generated/numpy.trapz.html). Den krever som input funksjonsverdiene $|\vec{v}(t_i)|$ og punktene der funksjonsverdiene er samplet, her $t_i=i T/(N-1)$ for $i=0, 1, \ldots, N-1$ og vi har brukt $N=100$ punkter i koden over.

```python
t0 = np.linspace(0, T, N)
x0 = v0*np.cos(a)*t0
z0 = v0*np.sin(a)*t0 - 0.5*g*t0**2
L = np.trapz(np.sqrt((v0*np.cos(a))**2 + (v0*np.sin(a) - g*t0)**2), t0)
print('Buelengden er %2.3f'%(L))
```

    Buelengden er 11.700

# Sympy

Vi skal nå gjøre samme implemetering ved å benytte [sympy](https://sympy.org) for eksakt symbolsk manipulering av vektorer. Til det skal vi bruke `sympy` sin `vector`-modul som inneholder en klasse for et koordinatsystem `CoordSys3D`.

```python
import sympy as sp
from sympy.vector import CoordSys3D
```

Vi antar nå først at både `v0,a` og `t` er variable

```python
g = 9.81
v0, a, t = sp.symbols('v0,a,t')
N = CoordSys3D('N')

x = v0*sp.cos(a)*t
z = v0*sp.sin(a)*t - 0.5*g*t**2

# Definer posisjonsvektoren
r = x*N.i + z*N.k

# Deriver mhp t for å finne hastigheten
v = sp.diff(r, t)

print(r)
print(v)
```

    (t*v0*cos(a))*N.i + (-4.905*t**2 + t*v0*sin(a))*N.k
    (v0*cos(a))*N.i + (-9.81*t + v0*sin(a))*N.k

```python
# Legg parametre i dictionary
v0 = 10
a = np.pi/4
d = {'v0': v0, 'a': a}
T = 2*v0*np.sin(a)/g

# Redefiner variable med disse parameterne
x = x.subs(d)
z = z.subs(d)
r = r.subs(d)
v = v.subs(d)

# Gjør x og z kallbare funksjoner for numpy-arrays
x = sp.lambdify(t, x)
z = sp.lambdify(t, z)

# Evaluer x og z for hele tidsdomenet
t0 = np.linspace(0, T*1.1, 100)

x0 = x(t0)
z0 = z(t0)
vx0 = np.float(v.subs(t, 0).dot(N.i))
vz0 = np.float(v.subs(t, 0).dot(N.k))
```

```python
%matplotlib inline

def partikkelbane(t):
    # plot hele partikkelbanen
    plt.figure()
    plt.plot(x0, z0, 'b')

    # plot partikkelen
    plt.plot(x(t), z(t), 'ok')

    # plot en hastighetsvektor ved t0 og t
    vx = np.float(v.subs('t', t).dot(N.i))
    vz = np.float(v.subs('t', t).dot(N.k))
    scale = T/4
    plt.arrow(0, 0, vx0*scale, vz0*scale, head_width=v0*T/100)
    plt.arrow(x(t), z(t), vx*scale, vz*scale, head_width=v0*T/100)

    # plot posisjonsvektor
    plt.arrow(0, 0, x(t), z(t), head_width=v0*T/100, length_includes_head=True)

    plt.ylim(z0.min(), 2*z0.max())

    plt.text(0.25*vx0, 0.3*vz0, r'$\vec{v}(0)$')
    plt.text(x(t)+0.25*vx0, z(t)+0.2*vz, r'$\vec{v}(t)$')
    plt.text(0.5*x(t), 0.4*z(t), r'$\vec{r}(t)$')

    plt.show()

interact(partikkelbane, t=(0, T*1.1, T*1.1/20))
```

    interactive(children=(FloatSlider(value=0.7928822215139677, description='t', max=1.5857644430279354, step=0.07…

    <function __main__.partikkelbane(t)>

```python
Ls = sp.Integral(sp.sqrt(v.dot(v)), (t, 0, T)).evalf()
print('Buelengde = %2.4f'%(Ls))
```

    Buelengde = 11.7002


# Quantum teleportation

We may not be able to clone arbitrary quantum states, but we can do something arguably cooler - we can teleport them! If Alice and Bob share an entangled pair of qubits in the state
\begin{equation}
 |\Psi\rangle = \frac{1}{\sqrt{2}} \left( |00\rangle + |11 \rangle \right),
\end{equation}
then they can use it to transfer the state of a third qubit to Bob.

**Important!** Teleportation does not mean that the physical qubit held by Alice is being shipped to Bob. Rather, the _state_ of Alice's qubit will be 'transferred' to the qubit held by Bob following the protocol.

```python
import numpy as np

from qiskit import QuantumRegister, ClassicalRegister
from qiskit import QuantumCircuit
from qiskit import execute, BasicAer

import qiskit.tools.visualization as qvis

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
```

First we'll set up the qubit registers and the circuit.

```python
# Set up the quantum circuit; Alice holds qubits 0 and 1, and Bob has qubit 2
# We need quantum register with 3 qubits, and classical register with 2 qubits
alice = QuantumRegister(2, 'a')
bob = QuantumRegister(1, 'b')
measurement_0 = ClassicalRegister(1, 'm0')
measurement_1 = ClassicalRegister(1, 'm1')

teleportation = QuantumCircuit(alice, bob, measurement_0, measurement_1)
```

Next we'll give Alice a random initial state to teleport. We'll do so by sending randomly chosen angles as parameters to a general unitary rotation gate.

```python
# Prepare a random single-qubit state that we would like to send
random_angles = np.random.rand(3)
teleportation.u3(*random_angles, alice[0])

teleportation.draw()
```

Let's see on the Bloch sphere where this state is...

```python
# To plot the Bloch vectors we will have to extract the state from the
# statevector simulator.
backend = BasicAer.get_backend('statevector_simulator') # the device to run on
result = execute(teleportation, backend).result()
initial_psi  = result.get_statevector(teleportation)

qvis.plot_bloch_multivector(initial_psi)
```

Note that qubit 1 and qubit 2 are still in state $|0\rangle$; our next step is to prepare the shared entangled state.

```python
# Prepare an entangled state for Alice and Bob to share
teleportation.h(alice[1])
teleportation.cx(alice[1], bob[0])

teleportation.draw()
```

Now that everything is set up, to teleport the qubit, Alice is going to entangle the qubit she wants to send with the qubit she holds from the entangled pair.

```python
# Now entangle Alice's first qubit with the first qubit of shared Bell state
teleportation.cx(alice[0], alice[1])
teleportation.h(alice[0])

teleportation.draw()
```

Next she will measure both her qubits, giving her two classical bits.

```python
# Execute circuit and get 2 measurement outcomes
teleportation.measure(alice[0], measurement_0[0])
teleportation.measure(alice[1], measurement_1[0])

teleportation.draw()
```

Alice phones Bob and tells him what bits she measured. Based on the outcomes, Bob will apply some Pauli gates to his half of the entangled state:
\begin{eqnarray}
 00 &\rightarrow& I \\
 01 &\rightarrow& X \\
 10 &\rightarrow& Z \\
 11 &\rightarrow& ZX
\end{eqnarray}
In the circuit these can be expressed as controlled gates using the classical outcomes as controls rather than normal qubits.

```python
# Based on outcomes, apply X, Z, or XZ
teleportation.x(bob[0]).c_if(measurement_1, 1)
teleportation.z(bob[0]).c_if(measurement_0, 1)

teleportation.draw()
```

After Bob applies his Paulis, his qubit will be in exactly the same state as the one Alice prepared! Let's plot the Bloch vectors to make sure.

```python
# Execute the full circuit
result = execute(teleportation, backend).result()
final_psi  = result.get_statevector(teleportation)
```

```python
qvis.plot_bloch_multivector(initial_psi)
```

```python
qvis.plot_bloch_multivector(final_psi)
```

Qubit 2, Bob's qubit, matches the initial state of Alice's qubit 0. Note that after the protocol, Alice's two qubits will be sitting in whatever states correspond to her classical measurement outcomes.

**Exercise**: starting from the state
\begin{equation}
 \left( \alpha |0\rangle + \beta |1 \rangle \right) \otimes \frac{1}{\sqrt{2}} \left( |00\rangle + |11 \rangle \right),
\end{equation}
work through the teleportation protocol to see how Alice's state gets transferred to Bob. You'll have to expand everything, then perform a basis change on Alice's two qubits to shift them into the Bell basis. After, Alice performs the CNOT and H before measuring; this changes back to the computational basis so we can interpret the measurements more easily. You'll see, after this transformation, each of the computational basis states will be attached to a slightly different version of the original state in Bob's qubit, which can easily be undone with the Pauli operators.


```python
import numpy as np
import sympy as sy
```

```python
s,z = sy.symbols('s,z', real=False)
Td,N,h = sy.symbols('Td, N,  h')
```

```python
Fpd = 1 + Td*s/(1+Td*s/N)
Fpdd = Fpd.subs(s, (z-1)/(z*h)).subs(N,20).subs(h,0.2)
(Fnum, Fden) = sy.fraction(sy.together(Fpdd))
Bf = sy.poly(sy.expand(Fnum), z)
print Bf
Bd = sy.poly(z-0.8, z)
print Bd
coeffs = Bf.all_coeffs()
print coeffs
```

    Poly((5.25*Td + 1.0)*z - 5.25*Td, z, domain='RR[Td]')
    Poly(1.0*z - 0.8, z, domain='RR')
    [5.25*Td + 1.0, -5.25*Td]

```python
eq = coeffs[1]/coeffs[0] + 0.8
print eq
sol = sy.solve(eq, Td)
print sol
```

    -5.25*Td/(5.25*Td + 1.0) + 0.8
    [0.761904761904762]

```python
Tdn = sy.N(sol[0])
5.25*Tdn/(5.25*Tdn+1)
```

    0.800000000000000

```python

```