```
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gaussian_kde, beta, ortho_group
from scipy.interpolate import interp1d, CubicSpline, lagrange
from scipy.sparse.linalg import cg

%matplotlib inline
```

# $Ax = b$: conjugate gradient

For each linear problem, one can construct the new set of equivalent problems changing bases in column and row spaces:

\begin{equation}
  Ax = b \Longleftrightarrow \left(V^{T}AV\right) \left(V^{-1}x\right) = \left(V^{T}b\right).
\end{equation}

If one then applies an iterative method that solves a linear problem, different distributions over $V$ will produce in different distribution over $x^{n}$. The results for conjugate gradients are given below.

## Non-orthogonal transformation

If matrix $V$ is not orthogonal, the distribution over $x^{n}$ is nontrivial.

```
np.random.seed(42)
D = 50
A = np.random.randn(D, D)
A = (A + A.T)/2
l, v = np.linalg.eigh(A)
A = (A + 3*np.eye(D)*np.min(l))
l, v = np.linalg.eigh(A)
sign = np.sign(l[0])
A = sign*A
b = np.random.randn(D)
x_exact = np.linalg.inv(A) @ b
x_0 = np.ones(D)
```

```
np.random.seed(100)
N_samples = 5000
E = np.empty((2, N_samples))
for j, N_it in zip(range(2), [10, 30]):
  for i in range(N_samples):
    V = np.eye(D) + 0.1*np.random.randn(D, D)
    V_ = np.linalg.inv(V)
    B = V.T @ A @ V
    d = V.T @ b
    y_0 = V_ @ x_0
    s = cg(B, d, y_0, maxiter = N_it)
    e = np.linalg.norm(V @ s[0] - x_exact)
    E[j, i] = e

  E[j] = E[j] - np.mean(E[j])
```

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for j, N_it, L, R in zip(range(2), [10, 30], [-1.5, -0.025], [1.5, 0.025]):
  p = gaussian_kde(E[j])
  coords = np.linspace(L, R, 500)

  ax[j].plot(coords, p(coords), color='black')
  ax[j].set_xlabel(r'$\left\||e\right\||_2 - \overline{\left\||e\right\||_2}$', size='x-large')
  ax[j].set_title(f'N = {N_it}')
  ax[j].set_ylabel('probability', size='x-large');
```

## Orthogonal transformation

The conjugate gradient method is invariant under orthogonal transformations, so the resulting distribution is highly concentrated and of no use.

```
np.random.seed(42)
D = 50
A = np.random.randn(D, D)
A = (A + A.T)/2
l, v = np.linalg.eigh(A)
A = (A + 3*np.eye(D)*np.min(l))
l, v = np.linalg.eigh(A)
sign = np.sign(l[0])
A = sign*A
b = np.random.randn(D)
x_exact = np.linalg.inv(A) @ b
x_0 = np.ones(D)
```

```
np.random.seed(100)
N_samples = 5000
E = np.empty((2, N_samples))
for j, N_it in zip(range(2), [10, 30]):
  for i in range(N_samples):
    V = ortho_group.rvs(D)
    V_ = V.T
    B = V.T @ A @ V
    d = V.T @ b
    y_0 = V_ @ x_0
    s = cg(B, d, y_0, maxiter = N_it)
    e = np.linalg.norm(V @ s[0] - x_exact)
    E[j, i] = e

  E[j] = E[j] - np.mean(E[j])
```

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for j, N_it, L, R in zip(range(2), [10, 30], [-1.5, -0.025], [1.5, 0.025]):
  p = gaussian_kde(E[j])
  coords = np.linspace(np.min(E[j]), np.max(E[j]), 500)

  ax[j].plot(coords, p(coords), color='black')
  ax[j].set_xlabel(r'$\left\||e\right\||_2 - \overline{\left\||e\right\||_2}$', size='x-large')
  ax[j].set_title(f'N = {N_it}')
  ax[j].set_ylabel('probability', size='x-large');
```

Do not miss the scale. One can see the invariance of the CG under the orthogonal group.

# Integration: trapezoidal rule

Consider the real continuous function $f:\left[a, b\right]\rightarrow \mathbb{R}$, and the set of point $x_i = a + i\Delta x,~i=\overline{0, N}$, with $x_0 = a$ and $x_{N} = b$. Then the integral of a function $f$ can be approximated by a trapezoidal rule

\begin{equation}
 \int\limits_{a}^{b} f(x) dx \simeq \Delta x \left(\sum_{k=1}^{N-1} f\left(x_k\right) + \frac{f(a) + f(b)}{2}\right) \equiv \text{TZ}\left[f|N, a, b\right]
\end{equation}

The true value of the integral is a real number that does not depend on any particular parametrization, i.e. it is invariant under the change of variable $x$. Let us introduce a $N$-parametric family of bijective smooth coordinate transformations $g_{\eta}: \left[a, b\right]\rightarrow \left[a, b\right]$. If we change the variable $x = g_{\eta}(y)$, the resulting integral remains the same

\begin{equation}
 \int\limits_{a}^{b} f(x) dx = \int\limits_{g^{-1}_{\eta}(a)}^{g^{-1}_{\eta}(b)} f(g_{\eta}(y))\left|g^{'}_{\eta}(y)\right| dy.
\end{equation}

However, the trapezoidal rule will, in general, give different answers for different functions $g_{\xi}$. So, if one introduces a distribution over $\xi$, the resulting distribution over approximations to the integral will be nontrivial.

As an example of a concrete reparametrization, we consider a family $N\left(y| x_0, \xi\right)$ that is widely used to cluster point near the inner layer. First, we introduce the Ericson function

\begin{equation}
 E\left(y|\xi\right) = \frac{e^{\xi y} - 1}{e^{\xi} - 1},~\xi>0,~y\in\left[0, 1\right].
\end{equation}

If one uses the uniform grid for $y$, this function assures clustering of the nodes for $x = E\left(y|\xi\right)$ near the left edge of the interval. Below one can see a few examples for three different $\xi$ -- the variable that defines the thickness of the layer.

```
y = np.linspace(0, 1, 20) # to show the uniform grid
z = np.linspace(0, 1, 100) # to draw a function
Ericson = lambda y, xi: (np.exp(xi*y) - 1)/(np.exp(xi) - 1)

fig, ax = plt.subplots(1, 3, figsize=(21, 5), sharey=True)
ax[0].set_ylabel(r'$x$', size='x-large')

for a, xi in zip(ax, [1, 2.5, 4]):
  a.plot(y, y*0, '.', color='black')
  a.plot(z, Ericson(z, xi), color='black')
  a.plot(y*0, Ericson(y, xi), '.', color='black')
  a.set_xlabel(r'$y$', size='x-large')
  a.set_title(rf'$\xi = {xi}$', size='x-large')

```

Using the Ericson function, one can construct a function that clusters nodes near the inner point of the interval

\begin{equation}
 N\left(y| x_0, \xi\right) = \begin{cases}
 x_0\left(1 - E\left(\left.1 - \frac{y}{x_0}\right|\xi\right)\right),\text{if } y\in\left[0, x_0\right];\\
 x_0 + \left(1 - x_0\right)E\left(\left.\frac{y - x_0}{1 - x_0}\right|\xi\right),\text{if } y\in\left[x_0, 1\right].
 \end{cases}
\end{equation}

Examples of three different $x_0$ are given below.

```
y = np.linspace(0, 1, 20) # to show the uniform grid
z = np.linspace(0, 1, 100) # to draw a function
Inner_Ericson = lambda y, x0, xi=2.5: x0*(y<x0)*(1 - Ericson(1-y/x0, xi)) + (y>=x0)*(x0 + (1 - x0)*Ericson((y-x0)/(1-x0), xi))

fig, ax = plt.subplots(1, 3, figsize=(21, 5), sharey=True)
ax[0].set_ylabel(r'$x$', size='x-large')

for a, x0 in zip(ax, [0.2, 0.6, 0.8]):
  a.plot(y, y*0, '.', color='black')
  a.plot(z, Inner_Ericson(z, x0), color='black')
  a.plot(y*0, Inner_Ericson(y, x0), '.', color='black')
  a.plot(y, x0+0*y, dashes=[4, 2, 4, 2], color='black')
  a.set_xlabel(r'$y$', size='x-large')
  a.set_title(rf'$x_0 = {x0}$, $\xi = {2.5}$', size='x-large')
```

To cluster nodes near the point $c$ on the interval $\left[a, b\right]$ one can use $N\left(a + (b-a)y\left|(c-a)\big/(b-a), \xi\right.\right)$.

## Distribution over the scale

Let the integrand $f(x)$ be the probability density function of the normal distribution

\begin{equation}
  f(x) = \sqrt{\frac{20}{\pi}} \exp\left(-40\left(x-\frac{1}{2}\right)^2\right).
\end{equation}

```
z = np.linspace(0, 1, 100)
f = lambda x: np.sqrt(40/np.pi)*np.exp(-40*(x-.5)**2)

fig, ax = plt.subplots(1, 1, figsize=(7, 5))
ax.plot(z, f(z), color='black');
```

Since the variance is small, the value of the integral is approximately $1$. To construct a set of equivalent representations we find the derivative of $N_{\xi}(y|x_0)$:

\begin{equation}
 \partial_{y}N\left(y| x_0, \xi\right) = \begin{cases}
 E^{'}\left(\left.1 - \frac{y}{x_0}\right|\xi\right),\text{if } y\in\left[0, x_0\right];\\
 E^{'}\left(\left.\frac{y - x_0}{1 - x_0}\right|\xi\right),\text{if } y\in\left[x_0, 1\right].
 \end{cases},
\end{equation}

where the derivative of the Ericson function is

\begin{equation}
 E^{'}(y|\xi) = \xi\frac{e^{\xi y}}{e^{\xi} - 1}.
\end{equation}

Since $N_{\xi}(y|x_0)$ is smooth and one-to-one, we know that it does not change the value of the integral. Hence, one can put a distribution over $\xi$, and it will be purely subjective in the sense that the $\xi$ does not make a difference for the continuous problem. We set $x_0 = 0.5$, and as a distribution over $\xi$ we choose the Gamma distribution with probability density function of the form

\begin{equation}
 p(\xi|k, \theta) = \frac{\xi^{k-1}e^{-\frac{\xi}{\theta}}}{\Gamma(k)\theta^k}.
\end{equation}

Below one can see a kernel-density estimation (rbf) to the probability density of the value of the integral computed by the trapezoidal rule for $\theta = 4$, $k = 2$ and $\theta = 4$, $k = 4$.

```
Ericson = lambda y, xi: (np.exp(xi*y) - 1)/(np.exp(xi) - 1)
d_Ericson = lambda y, xi: xi*np.exp(xi*y)/(np.exp(xi)-1)

Inner_Ericson = lambda y, xi, x0=0.5: x0*(y<x0)*(1 - Ericson(1-y/x0, xi)) + (y>=x0)*(x0 + (1 - x0)*Ericson((y-x0)/(1-x0), xi))
d_Inner_Ericson = lambda y, xi, x0=0.5: (y<x0)*d_Ericson(1-y/x0, xi) + (y>=x0)*d_Ericson((y-x0)/(1-x0), xi)

f = lambda x: np.sqrt(40/np.pi)*np.exp(-40*(x-.5)**2)
rep_f = lambda y, xi: abs(d_Inner_Ericson(y, xi))*f(Inner_Ericson(y, xi))
```

```
N_samples = 1000

fig, ax = plt.subplots(1, 2, figsize=(14, 5))
np.random.seed(42)

for scale, shape, a in zip([4, 4], [2, 4], ax):
  xi = np.random.gamma(shape, scale, N_samples)
  y = np.linspace(0, 1, 15)

  Integrals = np.empty(N_samples)
  for i in range(N_samples):
    Integrals[i] = np.trapz(rep_f(y, xi[i]), y)

  p = gaussian_kde(Integrals)
  coords = np.linspace(0, 1.5, 500)

  a.plot(coords, p(coords), color='black')
  a.set_title(rf'$\theta = {scale}$, $k = {shape}$', size='x-large')
```

The left tail of the distribution for $\theta = 4$, $k = 4$ corresponds to the too enthusiastic clustering. By construction, the distribution should converge to $\delta$-function when one increase the number of points. Below we can see a three estimated pdf for $N = 15, 30, 60$ and $\theta = 4$, $k = 4$.

```
N_samples = 1000

fig, ax = plt.subplots(1, 3, figsize=(21, 5))
scale, shape = 4, 4
np.random.seed(42)
xi = np.random.gamma(shape, scale, N_samples)

for N, a, L, R in zip([15, 30, 60], ax, [0, 0.75, 0.97], [1.5, 1.25, 1.03]):
  y = np.linspace(0, 1, N)

  Integrals = np.empty(N_samples)
  for i in range(N_samples):
    Integrals[i] = np.trapz(rep_f(y, xi[i]), y)

  p = gaussian_kde(Integrals)
  coords = np.linspace(L, R, 1000)

  a.plot(coords, p(coords), color='black')
  a.set_title(rf'$ N = {N}$', size='x-large')
```

The trend is evident. The distribution tends to the delta-function as expected.

## Distribution over the position

Let the integrand be

\begin{equation}
  f(x) = \left(x - \frac{1}{2}\right)\sqrt{\frac{20}{\pi}} \exp\left(-40\left(x-\frac{1}{2}\right)^2\right).
\end{equation}

```
z = np.linspace(0, 1, 100)
f = lambda x: (x-.5)*np.sqrt(40/np.pi)*np.exp(-40*(x-.5)**2)

fig, ax = plt.subplots(1, 1, figsize=(7, 5))
ax.plot(z, f(z), color='black');
```

The value of the integral is approximately zero. We use Beta distribution to draw values of $x_0$. Probability density function is
\begin{equation}
  p(x_0|\alpha, \beta) = \frac{x^{\alpha-1} (1-x)^{\beta-1}}{B(\alpha, \beta)}.
\end{equation}

Below we provide a three empirical pdf for $\left(\alpha, \beta\right) = (2, 5)$, $\left(\alpha, \beta\right) = (1/2, 1/2)$, $\left(\alpha, \beta\right) = (5, 2)$ with $\xi = 10$ in each case.

```
Ericson = lambda y, xi: (np.exp(xi*y) - 1)/(np.exp(xi) - 1)
d_Ericson = lambda y, xi: xi*np.exp(xi*y)/(np.exp(xi)-1)

Inner_Ericson = lambda y, x0, xi=10: x0*(y<x0)*(1 - Ericson(1-y/x0, xi)) + (y>=x0)*(x0 + (1 - x0)*Ericson((y-x0)/(1-x0), xi))
d_Inner_Ericson = lambda y, x0, xi=10: (y<x0)*d_Ericson(1-y/x0, xi) + (y>=x0)*d_Ericson((y-x0)/(1-x0), xi)

f = lambda x: (x-.5)*np.sqrt(40/np.pi)*np.exp(-40*(x-.5)**2)
rep_f = lambda y, x0: abs(d_Inner_Ericson(y, x0))*f(Inner_Ericson(y, x0))
```

```
N_samples = 5000
fig, ax = plt.subplots(1, 3, figsize=(21, 5))
np.random.seed(113)

for alpha, Beta, a in zip([2, .5, 5], [5, .5, 2], ax):
  np.random.seed(42)
  x0 = beta.rvs(alpha, Beta, size=N_samples)
  y = np.linspace(0, 1, 10)

  Integrals = np.empty(N_samples)
  for i in range(N_samples):
    Integrals[i] = np.trapz(rep_f(y, x0[i]), y)

  p = gaussian_kde(Integrals)
  coords = np.linspace(-0.2, 0.2, 500)

  a.plot(coords, p(coords), color='black')
  a.set_title(rf'$\alpha = {alpha}$, $\beta = {Beta}$', size='x-large')
```

The left (right) graph corresponds to the situations when grid points are clustered in the region where the function is negative (positive) and rarefied in all other regions. As a result, the value of the integral is negative (positive) with high probability. The middle graph exemplifies the symmetric situation and can be considered as an average of the left and right figures.

The example of the concentration of measure is given below.

```
N_samples = 5000

fig, ax = plt.subplots(1, 3, figsize=(21, 5))
alpha, Beta = 2, 5
np.random.seed(42)
x0 = beta.rvs(alpha, Beta, size=N_samples)

for N, a, L, R in zip([10, 15, 30], ax, [-.2, -.05, -.005,], [.2, .05, .005,]):
  y = np.linspace(0, 1, N)

  Integrals = np.empty(N_samples)
  for i in range(N_samples):
    Integrals[i] = np.trapz(rep_f(y, x0[i]), y)

  p = gaussian_kde(Integrals)
  coords = np.linspace(L, R, 1000)

  a.plot(coords, p(coords), color='black')
  a.set_title(rf'$N = {N}, \alpha = {alpha}$, $\beta = {Beta}$', size='x-large')
```

Again, the concentration of measure is evident. Even though we started from an apparently biased assumption about the node distribution, the measure will eventually even out.

### Supplementary

This part contains a code that reproduces two figures we used in the article in the section devoted to trapezoidal rule.

```
Ericson = lambda y, xi: (np.exp(xi*y) - 1)/(np.exp(xi) - 1)
Inner_Ericson = lambda y, x0, xi=10: x0*(y<x0)*(1 - Ericson(1-y/x0, xi)) + (y>=x0)*(x0 + (1 - x0)*Ericson((y-x0)/(1-x0), xi))
```

```
y = np.linspace(0, 1, 20) # to show the uniform grid
z = np.linspace(0, 1, 100) # to draw a function
Inner_Ericson = lambda y, x0, xi=2.5: x0*(y<x0)*(1 - Ericson(1-y/x0, xi)) + (y>=x0)*(x0 + (1 - x0)*Ericson((y-x0)/(1-x0), xi))

fig, ax = plt.subplots(1, 1, figsize=(7, 5), sharey=True)
ax.set_ylabel('transformed grid', size='x-large')
x0 = 0.6
ax.plot(y, y*0, '.', color='black')
ax.plot(z, Inner_Ericson(z, x0), color='black', label=r'$N\left(y|x_0,\xi\right)$')
ax.plot(y*0, Inner_Ericson(y, x0), '.', color='black')
ax.plot(y, x0+0*y, dashes=[4, 2, 4, 2], color='black')
ax.set_xlabel('original uniform grid', size='x-large')
ax.set_title(rf'$x_0 = {x0}$, $\xi = {2.5}$', size='x-large')
ax.legend(fontsize='x-large');
plt.savefig('grid_transformation.pdf', format='pdf')
```

```
Ericson = lambda y, xi: (np.exp(xi*y) - 1)/(np.exp(xi) - 1)
d_Ericson = lambda y, xi: xi*np.exp(xi*y)/(np.exp(xi)-1)

Inner_Ericson = lambda y, x0, xi=10: x0*(y<x0)*(1 - Ericson(1-y/x0, xi)) + (y>=x0)*(x0 + (1 - x0)*Ericson((y-x0)/(1-x0), xi))
d_Inner_Ericson = lambda y, x0, xi=10: (y<x0)*d_Ericson(1-y/x0, xi) + (y>=x0)*d_Ericson((y-x0)/(1-x0), xi)

f = lambda x: (x-.5)*np.sqrt(40/np.pi)*np.exp(-40*(x-.5)**2)
rep_f = lambda y, x0: abs(d_Inner_Ericson(y, x0))*f(Inner_Ericson(y, x0))
```

```
N_samples = 5000
fig, ax = plt.subplots(1, 1, figsize=(7, 5))

alpha, Beta = .5, .5
np.random.seed(42)
x0 = beta.rvs(alpha, Beta, size=N_samples)

y = np.linspace(0, 1, 10)

Integrals = np.empty(N_samples)
for i in range(N_samples):
  Integrals[i] = np.trapz(rep_f(y, x0[i]), y)

p = gaussian_kde(Integrals)
coords = np.linspace(-0.2, 0.2, 500)

ax.plot(coords, p(coords), color='black', label='10 points')

x0 = beta.rvs(alpha, Beta, size=N_samples)
y = np.linspace(0, 1, 20)

Integrals = np.empty(N_samples)
for i in range(N_samples):
  Integrals[i] = np.trapz(rep_f(y, x0[i]), y)

p = gaussian_kde(Integrals)
coords = np.linspace(-0.2, 0.2, 500)

ax.plot(coords, p(coords), dashes=[4, 2, 4, 2], color='black', label='20 points')

ax.set_xlabel('value of the integral', fontsize='x-large')
ax.set_ylabel('probability density', size='x-large')
ax.set_title(rf'$\alpha = {alpha}$, $\beta = {Beta}$', size='x-large')
ax.legend(fontsize='x-large');
plt.savefig('trapezoidal_rule_distribution.pdf', format='pdf')
```

# Leading eigenvalue by power iteration

Consider the eigenproblem

\begin{equation}
 Ax = \lambda x.
\end{equation}

To find an eigenvalue $\lambda_{\max}$ with the largest absolut value, one can apply a simple scheme known as a power iteration

\begin{equation}
 x^{n+1} = \frac{Ax^n}{\left\|Ax^{n}\right\|_2},~\lambda_{\max}^{n+1} = \frac{\left(x^{n+1}, A x^{n+1}\right)}{\left(A x^{n+1}, A x^{n+1}\right)}
\end{equation}

The convergence of the algorithm is well understood. In case the subspace corresponding to the eigenvalue $\lambda_{\max}$ of the largest absolute value consists of a single eigenvector (no other eigenvectors or generalized eigenvectors with the same eigenvalue) and the initial vector is not orthogonal to that subspace, one has $x^{n} \rightarrow \mu,~\lambda^{n} \rightarrow \lambda_{\max}: A\mu = \lambda_{\max} \mu$.

Suppose one perform a similarity transformation with some non-degenerate matrix $V$, i.e.

\begin{equation}
 VAV^{-1} Vx = \lambda V x,~\widetilde{A} \equiv VAV^{-1},~\widetilde{x} \equiv Vx.
\end{equation}

In the new bases power iteration will look like exactly like the iterations for the original matrix, but with $\widetilde{x}$, $\widetilde{A}$ in place of $x$ and $A$. Since the similarity transformation does not change the spectrum $\widetilde{\lambda}^{n}_{\max} \rightarrow \lambda_{\max}$. However for the finite $n$ in general $\lambda^{n}_{\max} \neq \widetilde{\lambda}^{n}_{\max}$.

To demonstrate this, we introduce the distribution over $V$
\begin{equation}
 V =
 \begin{pmatrix}
 a & 0\\
 0 & 1
 \end{pmatrix}
 \begin{pmatrix}
 \cos\phi & -\sin\phi \\
 \sin\phi & \cos\phi
 \end{pmatrix},~p(\phi) = \text{Ind}\left[\phi \in \left[0, \frac{\pi}{2}\right]\right].
\end{equation}
and choose the matrix and the initial vector as follows
\begin{equation}
 A = v v^T + (1 + \epsilon)u u^T,
 v = \frac{1}{\sqrt{2}}\begin{pmatrix}
 1\\
 -1
 \end{pmatrix},
 u = \frac{1}{\sqrt{2}}\begin{pmatrix}
 1\\
 1
 \end{pmatrix},
 x_0 = \begin{pmatrix}
 1\\
 2\\
 \end{pmatrix}.
\end{equation}

In the simulation below we use $a = 1/2$, $\epsilon = 10^{-2}$.

```
def power_iteration(A, x0):
  r = A @ x0
  s = np.linalg.norm(r)
  return r / s

def sample_l_max(A, x0, N_it=40):
  phi = np.pi*np.random.rand()/2
  B = np.array([[.5, 0], [0, 1]])
  V = B @ np.array([[np.cos(phi), -np.sin(phi)], [np.sin(phi), np.cos(phi)]])
  V_ = np.linalg.inv(V)
  wave_x0 = V @ x0
  wave_A = V @ A @ V_

  for _ in range(N_it):
    wave_x0 = power_iteration(wave_A, wave_x0)

  wave_l_max = np.dot(wave_x0, wave_A @ wave_x0) / np.dot(wave_x0, wave_x0)
  return wave_l_max
```

```
u = np.array([1, -1])/np.sqrt(2)
v = np.array([1, 1])/np.sqrt(2)
epsilon = 1e-2
A = np.outer(u, u) + (1 + epsilon)*np.outer(v, v)
#A = np.array([[1, 1], [0, 1 + 1e-4]])
x0 = np.array([1, 2])
l_exact = 1 + epsilon
N_samples = 5000
#np.random.seed(43)
delta_L = np.empty((3, N_samples))

for j, N_it in enumerate([10, 40, 160]):
  for i in range(N_samples):
    delta_L[j, i] = (l_exact - sample_l_max(A, x0, N_it = N_it))/epsilon
```

```
fig, ax = plt.subplots(1, 3, figsize=(21, 5))

for d_L, a, N, L, R in zip(delta_L, ax, [10, 40, 400], [-0.5, -0.25, -0.125], [0.5, 0.25, 0.125]):

  p = gaussian_kde(d_L)
  coords = np.linspace(L, R, 1000)

  a.plot(coords, p(coords), color='black')
  a.set_title(rf'$N = {N}$', size='x-large')
  a.set_xlabel(r'$\left(\lambda_{exact} - \lambda^{N}\right)/\epsilon$', size='x-large')
  a.set_xticks(np.linspace(L, R, 6))

ax[0].set_ylabel('probability density', fontsize='x-large');
```

```
fig, ax = plt.subplots(1, 1, figsize=(7, 5))
p1 = gaussian_kde(delta_L[0])
p2 = gaussian_kde(delta_L[2])
coords = np.linspace(-0.2, 0.3, 1000)
ax.plot(coords, p1(coords), color='black', label='N = 10 iterations')
ax.plot(coords, p2(coords), dashes=[4, 2, 4, 2], color='black', label='N = 400 iterations')
ax.set_xlabel(r'$\left(\lambda_{exact} - \lambda_{N}\right)/\epsilon$', size='x-large')
ax.legend(fontsize='x-large')
ax.set_ylabel('probability density', fontsize='x-large');
plt.savefig('power.pdf', format='pdf')
```

Again, we see that measure is concentrated as the approximation becomes better.

# Approximation/ Interpolation/ Derivatives

Consider the problem of iterpolation from data
\begin{equation}
    \left\{f(x_i)\right\}_{i=\overline{1, K}}
\end{equation}
One solution of this simple interpolation problem is to use the Lagrange polynomial
\begin{equation}
    \hat{f}(x) = \sum_{i=1}^{K}f(x_i)p_{i}(x),~p_{i}(x) = \prod_{k\neq i}\frac{x-x_{k}}{x_{i}-x_{k}}.
\end{equation}
Now, suppose we knew the values of a function at all points of the interval of interest, i.e. $f(x)$. In this case, the trivial transformation with arbitrary invertible function $g_{\xi}^{-1}\left(g_{\xi}\left(f(x)\right)\right)$ changes nothing. In particular, the original data remain intact. On the other hand, this transformation leads to the modification of that reads
\begin{equation}
    \hat{f}_{\xi}(x) \equiv g_{\xi}^{-1}\left(\sum_{i=1}^{K}g_{\xi}\left(f\left(x_i\right)\right)p_{i}(x)\right)\neq\hat{f}(x).
\end{equation}

The same pair of transormations $g_{\xi}, g_{\xi}^{-1}$ can be applied for any other interpolation, f.e., cubic splines. Below we consider examples with
\begin{equation}
\begin{split}
&g_{\xi}(x) = \exp(-\xi x),~g_{\xi}^{-1} = -\log(x)\big/\xi,\\
&p(\xi) = \text{Ind}\left[\xi \in \left[-1, 3\right]\right];
\end{split}
\end{equation}
for cubic splines and lagrange interpolation.

## Cubic spline

```
f = lambda x: 1 + np.cos(np.pi*x)
g = lambda x, xi: np.exp(-xi*x)
g_ = lambda x, xi: -np.log(x)/xi

df = lambda x: -np.pi*np.sin(np.pi*x)
dg_ = lambda x, xi: -1/(xi*x)
```

```
z = np.linspace(-1, 1, 100)
x = [np.linspace(-1, 1, 6), np.linspace(-1, 1, 12)]
np.random.seed(23)

N_samples = 50
int_samples = np.empty((2, N_samples, 100))

for k in range(2):
  for i in range(N_samples):
    xi_ = 4*np.random.uniform() - 1
    j = lambda x, xi=xi_: g(f(x), xi=xi)
    y_ = j(x[k])
    u_ = CubicSpline(x[k], y_)
    int_samples[k, i] = g_(u_(z), xi=xi_)
```

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for k in range(2):
  for int_f in int_samples[k]:
    ax[k].plot(z, int_f, color='black', alpha=0.2)
  ax[k].set_xlabel('$x$', size='x-large')
  ax[k].set_ylabel('$f(x)$', size='x-large')
  ax[k].plot(x[k], f(x[k]), 'o', color='red')
  ax[k].set_title(f'N = {len(x[k])}')
```

```
z = np.linspace(-1, 1, 100)
x = [np.linspace(-1, 1, 6), np.linspace(-1, 1, 12)]
np.random.seed(33)

N_samples = 50
int_samples = np.empty((2, N_samples, 100))

for k in range(2):
  for i in range(N_samples):
    xi_ = 4*np.random.uniform() - 1
    j = lambda x, xi=xi_: g(f(x), xi=xi)
    y_ = j(x[k])
    u_ = CubicSpline(x[k], y_)
    du_ = u_.derivative()
    int_samples[k, i] = dg_(u_(z), xi=xi_)*du_(z)
```

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for k in range(2):
  for int_f in int_samples[k]:
    ax[k].plot(z, int_f, color='black', alpha=0.2)
  ax[k].set_xlabel('$x$', size='x-large')
  ax[k].set_ylabel(r"$\partial f(x)$", size='x-large')
  ax[k].plot(x[k], df(x[k]), 'o', color='red')
  ax[k].set_title(f'N = {len(x[k])}')
```

## Lagrange polynomial

```
z = np.linspace(-1, 1, 100)
x = [np.linspace(-1, 1, 6), np.linspace(-1, 1, 12)]
np.random.seed(23)

N_samples = 50
int_samples = np.empty((2, N_samples, 100))

for k in range(2):
  for i in range(N_samples):
    xi_ = 4*np.random.uniform() - 1
    j = lambda x, xi=xi_: g(f(x), xi=xi)
    y_ = j(x[k])
    u_ = lagrange(x[k], y_)
    int_samples[k, i] = g_(u_(z), xi=xi_)
```

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for k in range(2):
  for int_f in int_samples[k]:
    ax[k].plot(z, int_f, color='black', alpha=0.2)
  ax[k].set_xlabel('$x$', size='x-large')
  ax[k].set_ylabel('$f(x)$', size='x-large')
  ax[k].plot(x[k], f(x[k]), 'o', color='red')
  ax[k].set_title(f'N = {len(x[k])}')
```

```
z = np.linspace(-1, 1, 100)
x = [np.linspace(-1, 1, 6), np.linspace(-1, 1, 12)]
np.random.seed(33)

N_samples = 50
int_samples = np.empty((2, N_samples, 100))

for k in range(2):
  for i in range(N_samples):
    xi_ = 4*np.random.uniform() - 1
    j = lambda x, xi=xi_: g(f(x), xi=xi)
    y_ = j(x[k])
    u_ = lagrange(x[k], y_)
    du_ = u_.deriv()
    int_samples[k, i] = dg_(u_(z), xi=xi_)*du_(z)
```

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for k in range(2):
  for int_f in int_samples[k]:
    ax[k].plot(z, int_f, color='black', alpha=0.2)
  ax[k].set_xlabel('$x$', size='x-large')
  ax[k].set_ylabel(r"$\partial f(x)$", size='x-large')
  ax[k].plot(x[k], df(x[k]), 'o', color='red')
  ax[k].set_title(f'N = {len(x[k])}')
```

```
int_samples = np.empty((2, N_samples, 100))
x = np.linspace(-1, 1, 6)

for i in range(N_samples):
  xi_ = 4*np.random.uniform() - 1
  j = lambda x, xi=xi_: g(f(x), xi=xi)
  y_ = j(x)
  u_ = lagrange(x, y_)
  du_ = u_.deriv()
  int_samples[0, i] = g_(u_(z), xi=xi_)
  int_samples[1, i] = dg_(u_(z), xi=xi_)*du_(z)
```

```
fig, ax = plt.subplots(1, 1, figsize=(7, 5))

for c in int_samples[1]:
  ax.plot(z, c, color='black', alpha=0.2)
ax.set_xlabel('$x$', size='x-large')
ax.set_ylabel(r"$\partial f(x)$", size='x-large')
ax.plot(x, df(x), 'o', color='black')
plt.savefig('df_int.pdf', format='pdf');
```

```
fig, ax = plt.subplots(1, 1, figsize=(7, 5))

for c in int_samples[0]:
  ax.plot(z, c, color='black', alpha=0.2)
ax.set_xlabel('$x$', size='x-large')
ax.set_ylabel('$f(x)$', size='x-large')
ax.plot(x, f(x), 'o', color='black')
plt.savefig('f_int.pdf', format='pdf');
```

# Nonlinear algebraic equations: Newton's method

The widely known technique for the solution of nonlinear algebraic equations of the form $g(x) = 0$ is Newton's method. For the scalar equation, the single-step reads

\begin{equation}
 x^{n+1} = x^{n} - \frac{g(x^{n})}{g^{'}(x^{n})}.
\end{equation}

It is always possible to reparametrise the initial equation using any function $f_{\xi}(x)$ (1-parametric family with different functions corresponding to different $\xi$) that is defined over the range of $g$, $f_{\xi}(x) = 0 \Leftrightarrow x = 0$ and $\forall x f^{'}(x) \neq 0$. Under this condition any root of the original equation $g(x) = 0$ is also a root of the modified equation $f_{\xi}(g(x)) = 0$. So it is quiet obvious that any distribution over $\xi$ does not affect the exact solution. Newertheless, it does affect the solution after the finite number of iterations.

As an example we consider the function $g(x) = (x-2)(x+3)$.

```
Newton = lambda g, dg, x0: x0 - g(x0)/dg(x0)

g = lambda x: (x-2)*(x+3)
dg = lambda x: 2*x + 1

f = lambda y, xi: (np.exp(xi*y) - 1)/(np.exp(xi) - 1)
df = lambda y, xi: xi*np.exp(xi*y)/(np.exp(xi)-1)
```

```
N_samples = 5000
X = np.empty((2, N_samples))
N_it = [20, 50]
np.random.seed(101)

for i in range(2):
  for j in range(N_samples):
    xi = np.random.randn()*0.1
    l = lambda x, xi = xi: f(g(x), xi)
    dl = lambda x, xi = xi: df(g(x), xi)*dg(x)

    x1 = 0
    for _ in range(N_it[i]):
      x1 = Newton(l, dl, x1)

    X[i, j] = x1
```

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in double_scalars
      """Entry point for launching an IPython kernel.
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:10: RuntimeWarning: invalid value encountered in double_scalars
      # Remove the CWD from sys.path while we load stuff.
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:3: RuntimeWarning: overflow encountered in double_scalars
      This is separate from the ipykernel package so we can avoid doing imports until
    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: RuntimeWarning: overflow encountered in double_scalars
      """Entry point for launching an IPython kernel.

```
fig, ax = plt.subplots(1, 2, figsize=(14, 5))

for xx, a, N in zip(X, ax, N_it):

  p = gaussian_kde(xx[abs(xx)<10])
  coords = np.linspace(-5, 5, 1000)

  a.plot(coords, p(coords), color='black')
  a.set_title(rf'${N}$ iterations', size='x-large')
  a.set_xlabel(r'$x_{N}$', size='x-large')
  #a.set_xticks(np.linspace(np.min(xx), np.max(xx), 4))
  a.set_ylabel('$p(x)$', size='x-large')
  #a.ticklabel_format(style='sci', scilimits=(0, 0), useOffset=False, useMathText=True)
```

```
fig, a = plt.subplots(1, 1, figsize=(7, 5))

p = gaussian_kde(X[1][abs(X[1])<10])
coords = np.linspace(-5, 5, 1000)

a.plot(coords, p(coords), color='black')
a.set_xlabel(r'$x^{50}$', size='x-large');
plt.savefig('newton.pdf', format='pdf')
```

Six examples of the hidden representation.ipynb


# Bayes by Backprop
An implementation of the algorithm described in https://arxiv.org/abs/1505.05424.
This notebook accompanies the article at https://www.nitarshan.com/bayes-by-backprop.

```python
%matplotlib inline
import math
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from tensorboardX import SummaryWriter
from torchvision import datasets, transforms
from torchvision.utils import make_grid
from tqdm import tqdm, trange
from copy import deepcopy

writer = SummaryWriter()
sns.set()
sns.set_style("dark")
sns.set_palette("muted")
sns.set_color_codes("muted")
```

```python
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
LOADER_KWARGS = {'num_workers': 1, 'pin_memory': True} if torch.cuda.is_available() else {}
print(torch.cuda.is_available())
```

## Data Preparation

```python
BATCH_SIZE = 100
TEST_BATCH_SIZE = 5

train_loader = torch.utils.data.DataLoader(
    datasets.FashionMNIST(
        './fmnist', train=True, download=True,
        transform=transforms.ToTensor()),
    batch_size=BATCH_SIZE, shuffle=True, **LOADER_KWARGS)
test_loader = torch.utils.data.DataLoader(
    datasets.FashionMNIST(
        './fmnist', train=False, download=True,
        transform=transforms.ToTensor()),
    batch_size=TEST_BATCH_SIZE, shuffle=False, **LOADER_KWARGS)

# train_loader = torch.utils.data.DataLoader(
#     datasets.CIFAR10(
#         '../gitignored/data', train=True, download=True,
#         transform=transforms.ToTensor()),
#     batch_size=BATCH_SIZE, shuffle=True, **LOADER_KWARGS)
# test_loader = torch.utils.data.DataLoader(
#     datasets.CIFAR10(
#         '../gitignored/data', train=False, download=True,
#         transform=transforms.ToTensor()),
#     batch_size=TEST_BATCH_SIZE, shuffle=False, **LOADER_KWARGS)

TRAIN_SIZE = len(train_loader.dataset)
TEST_SIZE = len(test_loader.dataset)
NUM_BATCHES = len(train_loader)
NUM_TEST_BATCHES = len(test_loader)

CLASSES = 10
TRAIN_EPOCHS = 20
SAMPLES = 2
TEST_SAMPLES = 10

assert (TRAIN_SIZE % BATCH_SIZE) == 0
assert (TEST_SIZE % TEST_BATCH_SIZE) == 0
```

## Modelling
$$\underline{\text{Reparameterized Gaussian}}$$
$$\begin{aligned}
\theta &= (\mu, \rho)\\
\sigma &= \ln{(1+e^\rho)}\\
\mathcal{N}(x\vert \mu, \sigma) &= \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\\
\ln{\mathcal{N}(x\vert \mu, \sigma)} &= -\ln{\sqrt{2\pi}} -\ln{\sigma} -\frac{(x-\mu)^2}{2\sigma^2}\\
P(\mathbf{w}) &= \prod_j{\mathcal{N}(\mathbf{w}_j \vert 0, \sigma^2)}\\
\ln{P(\mathbf{w})} &= \sum_j{\ln{\mathcal{N}(\mathbf{w}_j \vert 0, \sigma^2)}}\\
\end{aligned}$$

```python
class Gaussian(object):
    def __init__(self, mu, rho):
        super().__init__()
        self.mu = mu
        self.rho = rho
        self.normal = torch.distributions.Normal(0,1)
        self.mask = torch.ones_like(self.mu)

    @property
    def sigma(self):
        return torch.log1p(torch.exp(self.rho))

    def sample(self):
        epsilon = self.normal.sample(self.rho.size()).to(DEVICE)
        return self.mu + (self.sigma * epsilon) * self.mask.to(self.mu.device)

    def log_prob(self, input):
        return (-math.log(math.sqrt(2 * math.pi))
                - torch.log(self.sigma)
                - ((input - self.mu) ** 2) / (2 * self.sigma ** 2)).sum()
```

$$\underline{\text{Scale Mixture Gaussian}}$$
$$\begin{align}
P(\mathbf{w}) &= \prod_j{\pi \mathcal{N}(\mathbf{w}_j \vert 0, \sigma_1^2) + (1-\pi) \mathcal{N}(\mathbf{w}_j \vert 0, \sigma_2^2)}\\
\ln{P(\mathbf{w})} &= \sum_j{\ln{(\pi \mathcal{N}(\mathbf{w}_j \vert 0, \sigma_1^2) + (1-\pi) \mathcal{N}(\mathbf{w}_j \vert 0, \sigma_2^2))}}\\
\end{align}$$

```python
class ScaleMixtureGaussian(object):
    def __init__(self, pi, sigma1, sigma2):
        super().__init__()
        self.pi = pi
        self.sigma1 = sigma1
        self.sigma2 = sigma2
        self.gaussian1 = torch.distributions.Normal(0,sigma1)
        self.gaussian2 = torch.distributions.Normal(0,sigma2)

    def log_prob(self, input):
        prob1 = torch.exp(self.gaussian1.log_prob(input))
        prob2 = torch.exp(self.gaussian2.log_prob(input))
        return (torch.log(self.pi * prob1 + (1-self.pi) * prob2)).sum()
```

$$\pi = \frac{1}{2}$$
$$-\ln{\sigma_1} = 0$$
$$-\ln{\sigma_2} = 6$$

```python
PI = 0.5
SIGMA_1 = torch.cuda.FloatTensor([math.exp(-0)])
SIGMA_2 = torch.cuda.FloatTensor([math.exp(-6)])

# def visualize_scale_mixture_components():
#     def show_lines():
#         pass
#     mix = ScaleMixtureGaussian(PI, SIGMA_1, SIGMA_2)
#     normal_1 = torch.distributions.Normal(0, SIGMA_1)
#     normal_2 = torch.distributions.Normal(0, SIGMA_2)
#     x_points = np.linspace(-5,5,10000)
#     d1 = np.array([torch.exp(normal_1.log_prob(float(c))) for c in x_points])
#     d2 = np.array([torch.exp(normal_2.log_prob(float(c))) for c in x_points])
#     d3 = np.array([torch.exp(mix.log_prob(float(c))) for c in x_points])
#     plt.subplots(1,3,figsize=(14,4))
#     plt.subplot(1,3,1)
#     plt.plot(x_points,d2,color="g")
#     plt.plot(x_points,d3,color="r")
#     plt.plot(x_points,d1,color="b")
#     plt.legend(["sigma2", "mix", "sigma1"])
#     plt.ylim(0,0.5)
#     plt.subplot(1,3,2)
#     plt.plot(x_points,d1,color="b")
#     plt.plot(x_points,d2,color="g")
#     plt.plot(x_points,d3,color="r")
#     plt.legend(["sigma1", "sigma2", "mix"])
#     plt.ylim(0,160)
#     plt.subplot(1,3,3)
#     plt.plot(x_points,d2,color="g")
#     plt.plot(x_points,d3,color="r")
#     plt.plot(x_points,d1,color="b")
#     plt.legend(["sigma2", "mix", "sigma1"])
#     plt.ylim(0,80)

# visualize_scale_mixture_components()
```

```python
class BayesianLinear(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        # Weight parameters
        self.weight_mu = nn.Parameter(torch.Tensor(out_features, in_features).uniform_(-0.2, 0.2))
        self.weight_rho = nn.Parameter(torch.Tensor(out_features, in_features).uniform_(-5,-4))
        self.weight = Gaussian(self.weight_mu, self.weight_rho)
        # Bias parameters
        self.bias_mu = nn.Parameter(torch.Tensor(out_features).uniform_(-0.2, 0.2))
        self.bias_rho = nn.Parameter(torch.Tensor(out_features).uniform_(-5,-4))
        self.bias = Gaussian(self.bias_mu, self.bias_rho)
        # Prior distributions
        self.weight_prior = ScaleMixtureGaussian(PI, SIGMA_1, SIGMA_2)
        self.bias_prior = ScaleMixtureGaussian(PI, SIGMA_1, SIGMA_2)
        self.log_prior = 0
        self.log_variational_posterior = 0

    def forward(self, input, sample=False, calculate_log_probs=False):
        if self.training or sample:
            weight = self.weight.sample()
            bias = self.bias.sample()
        else:
            weight = self.weight.mu
            bias = self.bias.mu
        if self.training or calculate_log_probs:
            self.log_prior = self.weight_prior.log_prob(weight) + self.bias_prior.log_prob(bias)
            self.log_variational_posterior = self.weight.log_prob(weight) + self.bias.log_prob(bias)
        else:
            self.log_prior, self.log_variational_posterior = 0, 0

        return F.linear(input, weight, bias)
```

```python
class BayesianNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = BayesianLinear(28*28, 1200)
#         self.l1 = BayesianLinear(3*32*32, 1200)
        self.l2 = BayesianLinear(1200, 1200)
        self.l3 = BayesianLinear(1200, 10)

    def forward(self, x, sample=False):
        x = x.view(-1, 28*28)
#         x = x.view(-1, 3*32*32)
        x = F.relu(self.l1(x, sample))
        x = F.relu(self.l2(x, sample))
        x = F.log_softmax(self.l3(x, sample), dim=1)
        return x

    def log_prior(self):
        return self.l1.log_prior \
               + self.l2.log_prior \
               + self.l3.log_prior

    def log_variational_posterior(self):
        return self.l1.log_variational_posterior \
               + self.l2.log_variational_posterior \
               + self.l3.log_variational_posterior

    def sample_elbo(self, input, target, samples=SAMPLES):
        outputs = torch.zeros(samples, BATCH_SIZE, CLASSES).to(DEVICE)
        log_priors = torch.zeros(samples).to(DEVICE)
        log_variational_posteriors = torch.zeros(samples).to(DEVICE)
        for i in range(samples):
            outputs[i] = self(input, sample=True)
            log_priors[i] = self.log_prior()
            log_variational_posteriors[i] = self.log_variational_posterior()
        log_prior = log_priors.mean()
        log_variational_posterior = log_variational_posteriors.mean()
        negative_log_likelihood = F.nll_loss(outputs.mean(0), target, size_average=False)
        loss = (log_variational_posterior - log_prior)/NUM_BATCHES + negative_log_likelihood
        return loss, log_prior, log_variational_posterior, negative_log_likelihood

net = BayesianNetwork().to(DEVICE)
```

## Training

```python
def write_weight_histograms(epoch):
    writer.add_histogram('histogram/w1_mu', net.l1.weight_mu,epoch)
    writer.add_histogram('histogram/w1_rho', net.l1.weight_rho,epoch)
    writer.add_histogram('histogram/w2_mu', net.l2.weight_mu,epoch)
    writer.add_histogram('histogram/w2_rho', net.l2.weight_rho,epoch)
    writer.add_histogram('histogram/w3_mu', net.l3.weight_mu,epoch)
    writer.add_histogram('histogram/w3_rho', net.l3.weight_rho,epoch)
    writer.add_histogram('histogram/b1_mu', net.l1.bias_mu,epoch)
    writer.add_histogram('histogram/b1_rho', net.l1.bias_rho,epoch)
    writer.add_histogram('histogram/b2_mu', net.l2.bias_mu,epoch)
    writer.add_histogram('histogram/b2_rho', net.l2.bias_rho,epoch)
    writer.add_histogram('histogram/b3_mu', net.l3.bias_mu,epoch)
    writer.add_histogram('histogram/b3_rho', net.l3.bias_rho,epoch)

def write_loss_scalars(epoch, batch_idx, loss, log_prior, log_variational_posterior, negative_log_likelihood):
    writer.add_scalar('logs/loss', loss, epoch*NUM_BATCHES+batch_idx)
    writer.add_scalar('logs/complexity_cost', log_variational_posterior-log_prior, epoch*NUM_BATCHES+batch_idx)
    writer.add_scalar('logs/log_prior', log_prior, epoch*NUM_BATCHES+batch_idx)
    writer.add_scalar('logs/log_variational_posterior', log_variational_posterior, epoch*NUM_BATCHES+batch_idx)
    writer.add_scalar('logs/negative_log_likelihood', negative_log_likelihood, epoch*NUM_BATCHES+batch_idx)
```

```python
def train(net, optimizer, epoch):
    net.train()
    if epoch == 0: # write initial distributions
        write_weight_histograms(epoch)
    for batch_idx, (data, target) in enumerate(tqdm(train_loader)):
        data, target = data.to(DEVICE), target.to(DEVICE)
        net.zero_grad()
        loss, log_prior, log_variational_posterior, negative_log_likelihood = net.sample_elbo(data, target)
        loss.backward()
        optimizer.step()
        write_loss_scalars(epoch, batch_idx, loss, log_prior, log_variational_posterior, negative_log_likelihood)
    write_weight_histograms(epoch+1)
```

```python
optimizer = optim.Adam(net.parameters())
for epoch in range(TRAIN_EPOCHS):
    train(net, optimizer, epoch)
```

```python

backup_net = deepcopy(net)
```

```python
net = deepcopy(backup_net)
```

```python
# for name, module in net.named_modules():
#     if name in ['l1', 'l2']:
#         breakpoint()
```

```python
all_scores = []
for name, module in net.named_modules():
    if name in ['l1', 'l2']:
#         scores = - module.weight.sigma
        scores = torch.abs(module.weight.mu) / module.weight.sigma
        all_scores.append(scores.flatten())
```

```python
all_scores = torch.cat([x for x in all_scores])
```

```python
# all_scores
```

```python
# torch.histc(all_scores, 10)
```

```python
threshold, _ = torch.topk(all_scores, int(len(all_scores)*0.25), sorted=True)
```

```python
acceptable_score = threshold[-1]
```

```python
acceptable_score
```

```python
for name, module in net.named_modules():
    if name in ['l1', 'l2']:
        mask = (torch.abs(module.weight.mu) / module.weight.sigma > acceptable_score)
#         mask = (- module.weight.sigma) > acceptable_score
        module.weight.mu = mask * module.weight.mu
        module.weight.mask = mask

        print(mask.sum().float() / torch.numel(mask))
```

## Evaluation

### Model Ensemble

```python
def test_ensemble():
    net.eval()
    correct = 0
    corrects = np.zeros(TEST_SAMPLES+1, dtype=int)
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(DEVICE), target.to(DEVICE)
            outputs = torch.zeros(TEST_SAMPLES+1, TEST_BATCH_SIZE, CLASSES).to(DEVICE)
            for i in range(TEST_SAMPLES):
                outputs[i] = net(data, sample=True)
            outputs[TEST_SAMPLES] = net(data, sample=False)
            output = outputs.mean(0)
            preds = preds = outputs.max(2, keepdim=True)[1]
            pred = output.max(1, keepdim=True)[1] # index of max log-probability
            corrects += preds.eq(target.view_as(pred)).sum(dim=1).squeeze().cpu().numpy()
            correct += pred.eq(target.view_as(pred)).sum().item()
    for index, num in enumerate(corrects):
        if index < TEST_SAMPLES:
            print('Component {} Accuracy: {}/{}'.format(index, num, TEST_SIZE))
        else:
            print('Posterior Mean Accuracy: {}/{}'.format(num, TEST_SIZE))
    print('Ensemble Accuracy: {}/{}'.format(correct, TEST_SIZE))

test_ensemble()
```

### Model Uncertainty

#### In-Domain Uncertainty

```python
def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1,2,0)), interpolation='nearest')
```

```python
fmnist_sample = iter(test_loader).next()
fmnist_sample[0] = fmnist_sample[0].to(DEVICE)
print(fmnist_sample[1])
sns.set_style("dark")
show(make_grid(fmnist_sample[0].cpu()))
```

```python
net.eval()
fmnist_outputs = net(fmnist_sample[0], True).max(1, keepdim=True)[1].detach().cpu().numpy()
for _ in range(99):
    fmnist_outputs = np.append(fmnist_outputs, net(fmnist_sample[0], True).max(1, keepdim=True)[1].detach().cpu().numpy(), axis=1)
sns.set_style("darkgrid")
plt.subplots(5,1,figsize=(10,4))
for i in range(5):
    plt.subplot(5,1,i+1)
    plt.ylim(0,100)
    plt.xlabel("Categories")
    plt.xticks(range(10), ["Top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle Boot"])
    plt.ylabel("Count")
    plt.yticks(range(50,101,50))
    plt.hist(fmnist_outputs[i], np.arange(-0.5, 10, 1))
```

#### Out-of-Domain Uncertainty

```python
mnist_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../gitignored/data', train=False, download=True, transform=transforms.ToTensor()), batch_size=5, shuffle=False)

# mnist_loader = torch.utils.data.DataLoader(
#     datasets.SVHN('../gitignored/data', split='test', download=True, transform=transforms.ToTensor()), batch_size=5, shuffle=False)
```

```python
from sklearn import metrics
import numpy as np

def calculate_auroc(correct, predictions):
    fpr, tpr, thresholds = metrics.roc_curve(correct, predictions)
    auroc = metrics.auc(fpr, tpr)
    return auroc

def calculate_aupr(correct, predictions):
    aupr = metrics.average_precision_score(correct, predictions)
    return aupr
```

```python
ood_labels = []
ood_scores = []

with torch.no_grad():
    for data, _ in test_loader:
        data = data.to(DEVICE)
        out = net(data, True)
        probs = F.softmax(out, 1)

        max_probs, _  = probs.max(1)
        max_probs = max_probs.detach().cpu().numpy()
        ood_labels.append(np.ones_like(max_probs))
        ood_scores.append(max_probs)

    for data, _ in mnist_loader:
        data = data.to(DEVICE)
        out = net(data, True)
        probs = F.softmax(out, 1)

        max_probs, _  = probs.max(1)
        max_probs = max_probs.detach().cpu().numpy()
        ood_labels.append(np.zeros_like(max_probs))
        ood_scores.append(max_probs)

ood_labels = np.concatenate(ood_labels)
ood_scores = np.concatenate(ood_scores)
```

```python
ood_scores[10000:].mean()
```

```python
print(calculate_auroc(ood_labels, ood_scores))
print(calculate_aupr(ood_labels, ood_scores))
```

```python
# 0.7348483900000001
# 0.7189197489131008

# 0.6768591982944069
# 0.495362640026969

# 0.657779435
# 0.6424149416360756
```

```python
mnist_sample = iter(mnist_loader).next()
mnist_sample[0] = mnist_sample[0].to(DEVICE)
print(mnist_sample[1])
sns.set_style("dark")
show(make_grid(mnist_sample[0].cpu()))
```

```python
net.eval()
mnist_outputs = net(mnist_sample[0], True).max(1, keepdim=True)[1].detach().cpu().numpy()
for _ in range(99):
    mnist_outputs = np.append(mnist_outputs, net(mnist_sample[0], True).max(1, keepdim=True)[1].detach().cpu().numpy(), axis=1)

sns.set_style("darkgrid")
plt.subplots(5,1,figsize=(10,4))
for i in range(5):
    plt.subplot(5,1,i+1)
    plt.ylim(0,100)
    plt.xlabel("Categories")
    plt.xticks(range(10), ["Top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle Boot"])
    plt.ylabel("Count")
    plt.yticks(range(50,101,50))
    plt.hist(mnist_outputs[i], np.arange(-0.5, 10, 1))
```

```python
%load_ext watermark
%watermark --updated --datename --python --machine --watermark -p torch,numpy,matplotlib,tensorboardX,torchvision,seaborn
```

notebooks/Weight Uncertainty in Neural Networks.ipynb


# Videos and Exercises for Session 11: Regression and Regularization

In this combined teaching module and exercise set, you will learn about linear regression models in a machine learning perspective. We will see how overfitting can arise and how we can tackle it with a modification of the linear regression model.

The structure of this notebook is as follows:
1. Linear Regression Mechanics
2. Overfitting and Underfitting in Linear Regression
    - Exploring Overfitting in Linear Regression
    - A Cure for Overfitting in Linear Regression
3. Modelling Houseprices (Exercise)

## Packages
First, we need to import our standard stuff. Notice that we are not interested in seeing the convergence warning in scikit-learn, so we suppress them for now.

```python
import warnings
from sklearn.exceptions import ConvergenceWarning
warnings.filterwarnings(action='ignore', category=ConvergenceWarning)

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

%matplotlib inline
```

# Part 1: Linear Regression Mechanics
## Implementing and evaluating the gradient decent

Normally we use OLS to estimate linear regression models, but this is only way of solving the problem of minimizing the least squares problem (that minimizes the sum of squared errors). In the video below we show how to implement gradient descent below and compare it along with other approximate solutions to OLS.

You may find PML pp. 310-312, 319-324 useful as background reading.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('pl3ep6qRaZw', width=640, height=360)
```

Before continuing, we want to provide some brief intuition for how taking steps in the opposite direction of the gradient of a loss function helps us find the weights that minimize the loss. In the video, encountered the following derivation of how weights are updated with gradient descent in a regression type problem:

\begin{align}\frac{\partial SSE}{\partial\hat{\textbf{w}}} & =-\textbf{X}^{T}\textbf{e}\qquad\text{(the gradient)}\\
\Delta\hat{\textbf{w}} & =-\eta\cdot\frac{\partial SSE}{\partial\hat{\textbf{w}}}\qquad\text{(gradient descent)}\\
 & =\eta\cdot\textbf{X}^{T}\textbf{e}\\
 & =\eta\cdot\textbf{X}^{T}(\textbf{y}-\textbf{X}\hat{\textbf{w}})
\end{align}

You may ask: Why do we take steps in the opposite direction of the gradient? Consider the illustration of a one-dimensional problem below. Here $w$ is our weight, and $J(w)$ is the loss function that we want to minimize by choosing an appropriate weight. In the example, we started out by guessing a too _high_ value for the weight. At this point, the loss function is _increasing_ in the size of the weight. So what do we do? We take a step in the opposite direction of the gradient and _decrease_ the size of the weight. If the gradient is steep, we take a big step (we know that we are probably relatively far away from the solution), and if the gradient is relatively flat, we take a small step (we might be close to the solution).

We continue straight to an exercise where you are to implement a new estimator that we code up from scratch. We solve the numerical optimization using the gradient decent algorithm. This will be very similar to what we just saw in the video, but we will pay a bit more attention to each step in the process.

Using our algorithm, we will fit it to some data, and compare our own solution to the standard solution from `sklearn`

> **Ex. 11.1.0**: Import the dataset `tips` from the `seaborn`.

*Hint*: use the `load_dataset` method in seaborn

```python
### BEGIN SOLUTION
tips = sns.load_dataset("tips")
### END SOLUTION
```

> **Ex. 11.1.1**: Convert non-numeric variables to dummy variables for each category (remember to leave one column out for each catagorical variable, so you have a reference). Restructure the data so we get a dataset `y` containing the variable tip, and a dataset `X` containing the
features.

> *Hint*: You might want to use the `get_dummies` method in pandas, with the `drop_first = True` parameter.

```python
### BEGIN SOLUTION
tips_num = pd.get_dummies(tips, drop_first=True)

X = tips_num.drop('tip', axis = 1)
y = tips_num['tip']
### END SOLUTION
```

> **Ex. 11.1.2**: Divide the features and target into test and train data. Make the split 50 pct. of each. The split data should be called `X_train`, `X_test`, `y_train`, `y_test`.

> *Hint*: You may use `train_test_split` in `sklearn.model_selection`.

```python
### BEGIN SOLUTION
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=.5, random_state=161193)
### END SOLUTION
```

> **Ex. 11.1.3**: Normalize your features by converting to zero mean and one std. deviation.

> *Hint*: Take a look at `StandardScaler` in `sklearn.preprocessing`. If in doubt about which distribution to scale, you may read [this post](https://stats.stackexchange.com/questions/174823/how-to-apply-standardization-normalization-to-train-and-testset-if-prediction-i).

```python
### BEGIN SOLUTION
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
### END SOLUTION
```

> **Ex. 11.1.4**: Make a function called `compute_error` to compute the prediction errors given input target `y_`, input features `X_` and input weights `w_`. You should use matrix multiplication.
>
> *Hint:* You can use the net-input fct. from yesterday.

```python
### BEGIN SOLUTION
def net_input(X_, w_):
    ''' Computes the matrix product between X and w. Note that
    X is assumed not to contain a bias/intercept column.'''
    return np.dot(X_, w_[1:]) + w_[0]   # We have to add w_[0] separately because this is the constant term. We could also have added a constant term (columns of 1's to X_ and multipliced it to all of w_)

def compute_error(y_, X_, w_):
    return y_ - net_input(X_, w_)
### END SOLUTION
```

> **Ex. 11.1.5**: Make a function to update the weights given input target `y_`, input features `X_` and input weights `w_` as well as learning rate, $\eta$, i.e. greek `eta`. You should use matrix multiplication.

```python
# INCLUDED IN ASSIGNMENT 2
```

> **Ex. 11.1.6**: Use the code below to initialize weights `w` at zero given feature set `X`. Notice how we include an extra weight that includes the bias term. Set the learning rate `eta` to 0.001. Make a loop with 50 iterations where you iteratively apply your weight updating function.

>```python
w = np.zeros(1+X_train.shape[1])
```

```python
# INCLUDED IN ASSIGNMENT 2
```

> **Ex. 11.1.7**: Make a function to compute the mean squared error. Alter the loop so it makes 100 iterations and computes the MSE for test and train after each iteration, plot these in one figure.

> Hint: You can use the following code to check that your model works:
>```python
from sklearn.linear_model import LinearRegression
reg = LinearRegression()
reg.fit(X_train, y_train)
assert((w[1:] - reg.coef_).sum() < 0.01)
```

```python
### BEGIN SOLUTION
def MSE(y_,X_,w_):
    return (compute_error(y_, X_, w_)**2).mean()

w = np.zeros(1+X_train.shape[1])

MSE_train = [MSE(y_train, X_train, w)]
MSE_test = [MSE(y_test, X_test, w)]

for i in range(100):
    w = update_weights(y_train, X_train, w)
    MSE_train.append(MSE(y_train, X_train, w))
    MSE_test.append(MSE(y_test, X_test, w))
### END SOLUTION

pd.Series(MSE_train).plot()
pd.Series(MSE_test).plot()
```

The following bonus exercises are for those who have completed all other exercises until now and have a deep motivation for learning more.

> **Ex. 11.1.8 (BONUS)**: Implement your linear regression model as a class.

> A solution is found on p. 320 in Python for Machine Learning.

# Part 2: Overfitting and Underfitting in Linear Regression

## Exploring Overfitting in Linear Regression
How does overfitting manifest itself in linear regression? In the video below we simulate what happens as make a better and better taylor approximation, i.e. we estimate a polynomial of higher and higher order. Two issues arise simultaneously - one is related to the number of parameters and the to the size of the parameters.

You may find PML pp. 334-339 useful as background reading.

```python
YouTubeVideo('HbeTpK-2oeU', width=640, height=360)
```

## A Cure for Overfitting in Linear Regression

How do we fix the two issues of excessively large weights/coefficients and too many spurious solutions? The video below provides a solution by directly incorporating these issues into the optimization problem.

You may find PML pp. 73-76, 123-136, 332-334 useful as background reading.

```python
YouTubeVideo('r6a8WFm9jAI', width=640, height=360)
```

Above we tackled overfitting, but what about ***underfitting***? The video below shows how to address underfitting and also zooms in on some important details about regularization.

You may find PML pp. 73-76, 123-136, 332-334 useful as background reading.

```python
YouTubeVideo('IWBtYT1KI_Q', width=640, height=360)
```

> **Ex. 11.2.1 (BONUS)**: Is it possible to add a penalty to our linear model above and solve this Lasso model with gradient descent? Is there a simple fix?
>
> *Hint:* Gradient descent essentially relies on a differentiable loss function (read more [here](https://stats.stackexchange.com/questions/177800/why-proximal-gradient-descent-instead-of-plain-subgradient-methods-for-lasso))

```python
### BEGIN SOLUTION

ANSWER: No, we cannot exactly solve for the Lasso with gradient descent. However, we can make an approximate solution which is pretty close and quite intuitive - see good explanation here: https://stats.stackexchange.com/questions/177800/why-proximal-gradient-descent-instead-of-plain-subgradient-methods-for-lasso.

### END SOLUTION
```

# Part 3: Modelling Houseprices
In this example, we will try to predict houseprices using a lot of variable (or features as they are called in Machine Learning). We are going to work with Kaggle's dataset on house prices, see information [here](https://www.kaggle.com/c/house-prices-advanced-regression-techniques). Kaggle is an organization that hosts competitions in building predictive models.

> **Ex. 11.3.0:** Load the california housing data with scikit-learn using the code below. Now:
> 1. Inspect *cal_house*. How are the data stored?
> 2. Create a pandas DataFrame called *X*, using `data`. Name the columns using `feature_names`.
> 3. Crate a pandas Series called *y* using `target`.
> 4. Make a train test split of equal size.

```python
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

cal_house = fetch_california_housing()

### BEGIN SOLUTION
X = pd.DataFrame(data=cal_house['data'],
                 columns=cal_house['feature_names'])\
             .iloc[:,:-2]
y = cal_house['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.5, random_state=1)

X_train.describe()
### END SOLUTION
```

> **Ex.11.3.1**: Generate interactions between all features to third degree (make sure you **exclude** the bias/intercept term). How many variables are there? Will OLS fail? After making interactions, rescale the features to have zero mean, unit std. deviation. Should you use the distribution of the training data to rescale the test data?

> *Hint 1*: Try importing `PolynomialFeatures` from `sklearn.preprocessing`

> *Hint 2*: If in doubt about which distribution to scale, you may read [this post](https://stats.stackexchange.com/questions/174823/how-to-apply-standardization-normalization-to-train-and-testset-if-prediction-i).

```python
# INCLUDED IN ASSIGNMENT 2
```

> **Ex.11.3.2**: Estimate the Lasso model on the rescaled train data set, using values of $\lambda$ in the range from $10^{-4}$ to $10^4$. For each $\lambda$  calculate and save the Root Mean Squared Error (RMSE) for the rescaled test and train data. Take a look at the fitted coefficients for different sizes of $\lambda$. What happens when $\lambda$ increases? Why?

> *Hint 1*: use `logspace` in numpy to create the range.

> *Hint 2*: read about the `coef_` feature [here](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso).

```python
# INCLUDED IN ASSIGNMENT 2
```

> **Ex.11.3.3**: Make a plot with on the x-axis and the RMSE measures on the y-axis. What happens to RMSE for train and test data as $\lambda$ increases? The x-axis should be log scaled. Which one are we interested in minimizing?

> Bonus: Can you find the lambda that gives the lowest MSE-test score?

```python
# INCLUDED IN ASSIGNMENT 2
```


Importando bibliotecas para apresentar os dados

```python
import sympy
sympy.init_printing()
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

Definindo variáveis para usar na função

```python
t, s = sympy.symbols('t, s')
m = sympy.symbols('m', real=True, positive=True)
c = sympy.symbols('c', real=True, positive=True)
k = sympy.symbols('k', real=True, positive=True)
f0 = sympy.symbols('f0', real=True, positive=True)
```

Funções para calcular as transforamadas

```python
def invL(F):
    return sympy.inverse_laplace_transform(F, s, t)
```

```python
def L(f):
    return sympy.laplace_transform(f, t, s, noconds=True)
```

Definindo as expressões para o cálculo

```python
y = sympy.Function('y')
u = sympy.Function('u')
y = m * y(t).diff(t).diff(t) + c * y(t).diff(t) + k * y(t)
y
```

```python
Y = L(y) - f0/s
Y.subs({c: 2, f0: 3, k:1, m: 1})
```

```python
y = invL(Y)
y
```

```python
Ys =  sympy.Function('Ys')
Ys = f0/ (s * (m * s**2 + c * s + k))
Ys
```

```python
invL(Ys)
```

```python
p = sympy.plot(Ys.subs({m: 2, c: 3, k: 3, f0: 2}), Ys.subs({m: 2, c: 3, k: 3, f0: 2}),
               xlim=(0, 4), ylim=(0, 3), show=False)
p[1].line_color = 'red'
p.show()
```

```python
p = sympy.plot(invL(Ys).subs({m: 2, c: 3, k: 3, f0: 2}), invL(Ys).subs({m: 2, c: 3, k: 3, f0: 2}),
               xlim=(0, 4), ylim=(0, 3), show=False)
p[1].line_color = 'red'
p.show()
```

# Não consegui colocar as duas linhas no mesmo gráfico
obs: aparentemente esssas biblioteca não trabalha bem com derivadas (Computacionalmente pesado para fazer na Transformada inversa)


```python
# numpy is the 'Numerical Python' package
import numpy as np

# Numpy's methods for pseudorandom number generation
import numpy.random as rnd

# scipy is the 'Scientific Python' package
# We'll use the stats package to get some p.d.f.s & c.d.f.s
from scipy import stats

import matplotlib.pyplot as plt

%config InlineBackend.figure_format = 'retina'
```

## Part 1: Random number generation

1) Simulate $R=10^5$ $\mathsf{Pareto}(10,100)$ random variables, where our definition of $\mathsf{Pareto}(\alpha, \lambda)$ has the p.d.f.

$$ f(x) = \begin{cases}
    \frac{\alpha}{\lambda} \bigl[ 1 + \frac{x}{\lambda} \bigr]^{-(\alpha+1)} & \text{if } x > 0 \\
    0 & \text{otherwise.}
\end{cases} $$

Plot a histogram of these points against this p.d.f., like in the lectures.

```python
rng = rnd.default_rng(1)
R = 10**5
α = 10.0
λ = 100.0
paretos = λ * rng.pareto(α, R)
```

```python
x = np.linspace(0, np.max(paretos), 500)
pdf = (α/λ)*(1+x/λ)**(-(α+1))

plt.plot(x, pdf, 'r')
plt.hist(paretos, bins=100, density=True);
```

2) Simulate $R = 10^5$ $\mathsf{LogNormal}(3,0.1)$ random variables, where our definition of $\mathsf{LogNormal}(\mu, \sigma^2)$ has the p.d.f.

$$ f(x) = \begin{cases}
    \frac{1}{x \sigma \sqrt{2 \pi} } \exp\bigl\{ -\frac{(\log(x) - \mu)^2 }{2 \sigma^2} \bigr\} & \text{if } x > 0 \\
    0 & \text{otherwise.}
\end{cases} $$

Plot the histogram of your samples against p.d.f. above.

```python
R = 10**5
μ = 3
σ = np.sqrt(0.1)
lognormals = rng.lognormal(μ, σ, R)
```

```python
x = np.linspace(1e-5, 100, 500)
pdf = 1/(x*σ*np.sqrt(2*np.pi)) * np.exp(-(np.log(x)-μ)**2 / (2*σ**2))

plt.plot(x, pdf, 'r')
plt.hist(lognormals, bins=100, density=True);
```

3) Find $\overline{\mu}$, $\overline{\sigma}$ such that $X \sim \mathsf{LogNormal}(\overline{\mu}, \overline{\sigma}^2)$ has $\mathbb{E}[X] = m$ and $\mathbb{V}[X] = v$.

Simulate $R = 10^5$ lognormal random variables from the $\mathsf{LogNormal}(\overline{\mu}, \overline{\sigma}^2)$ where $m = 150$ and $v = 500$.

Plot a histogram of the simulated points, and overlay a vertical line at $x = m = 150$ to check that the simulated points are near the desired mean.

__For style points__: Find $\overline{\mu}$, $\overline{\sigma}$ using the `solve` function from the `sympy` 'symbolic python' package.

```python
import sympy as sym

μ, σ, m, v = sym.symbols('μ σ m v')

mean = sym.exp(μ + σ**2 / 2)
var = (sym.exp(σ**2) - 1) * sym.exp(2*μ + σ**2)

sym.solve((m - mean, v - var), (μ, σ))
```

    [(log(m) - log(1 + v/m**2)/2, -sqrt(log(1 + v/m**2))),
     (log(m) - log(1 + v/m**2)/2, sqrt(log(1 + v/m**2)))]

```python
m = 150
v = 500

μBar = np.log(m) - np.log(1 + v/m**2)/2
σBar = np.sqrt(np.log(1 + v/m**2))
R = 10**5
```

```python
lognormals = rng.lognormal(μBar, σBar, R)
```

```python
print("Sample mean:", lognormals.mean())
print("Theoretical mean:", m)
assert np.isclose(lognormals.mean(), m, rtol=0.01)
```

    Sample mean: 149.92195398260404
    Theoretical mean: 150

```python
print("Sample var:", lognormals.var())
print("Theoretical var:", v)
assert np.isclose(lognormals.var(), v, rtol=0.01)
```

    Sample var: 497.6989282181114
    Theoretical var: 500

```python
x = np.linspace(50, 250, 500)
pdf = 1/(x*σBar*np.sqrt(2*np.pi)) * np.exp(-(np.log(x)-μBar)**2 / (2*σBar**2))

plt.plot(x, pdf, 'r')
plt.hist(lognormals, bins=100, density=True);
plt.gca().axvline(m, c='black');
```

## Part 2: Crude Monte Carlo

1) We model an insurer's risk reserve process $R_t$ as

$$ R(t) = u + p t - \sum_{i=1}^{N_t} U_i $$

where $u \ge 0$, $p > 0$, $N_t$ is a Poisson process with intensity $\lambda$ and $U_i \overset{\mathrm{i.i.d.}}{\sim} \mathsf{Gamma}(r, m)$.

The only possible times when the insurer's reserve can become negative is at the times $T_1, T_2, \dots$ when the claims arrive. Define

$$ R_i = R(T_i) , \quad i = 1, 2, \dots,$$

which is reserve calculated at the time when the $i$-th claim arrives (and including this claim).

Estimate

$$\mathbb{P}(\text{Ruin before } n \text{-th arrival}) = \mathbb{P}(\min\{R_1,\dots,R_n\} < 0)$$

using crude Monte Carlo.

N.B. With $T_0 \equiv 0$, then we have $T_i - T_{i-1} \overset{\mathrm{i.i.d.}}{\sim} \mathsf{Exponential}(\lambda)$ for $i=1,2,\dots$

Constants:

```python
u = 1
p = 0.3
λ = 0.5
n = 50
r = 1
m = 0.5

R = 10**6
```

```python
%%time

T = np.zeros(R)
claims = np.zeros(R)
R_n = np.zeros(R)

alive = np.full(R, True)
overshoot = np.full(R, np.nan)
ruinTime = np.full(R, np.nan)

for i in range(1, n+1):
    T += rng.exponential(1/λ, size=R)
    claims += rng.gamma(r, m, size=R)
    R_n = u + p * T - claims

    justRuined = alive & (R_n < 0)
    alive[justRuined] = False
    overshoot[justRuined] = R_n[justRuined]
    ruinTime[justRuined] = i

ℓHat = 1 - alive.mean()
σHat = alive.std()
widthCI = 1.96 * σHat / np.sqrt(R)
print(f"CMC estimate:\t {ℓHat} (+/- {widthCI})")
print(f"CMC low bound:\t {np.maximum(ℓHat-widthCI, 0)}")
print(f"CMC upp bound:\t {ℓHat+widthCI}")
```

    CMC estimate:	 0.554385 (+/- 0.0009741856041077799)
    CMC low bound:	 0.5534108143958922
    CMC upp bound:	 0.5553591856041078
    Wall time: 3.42 s

Conditioned on a bankruptcy event, what is the expected value of the reserve process (it will be negative)?

```python
ests = overshoot[~alive]
ℓHat = ests.mean()
σHat = ests.std()
widthCI = 1.96 * σHat / np.sqrt(R)
print(f"CMC estimate:\t {ℓHat} (+/- {widthCI})")
print(f"CMC low bound:\t {ℓHat-widthCI}")
print(f"CMC upp bound:\t {ℓHat+widthCI}")
```

    CMC estimate:	 -0.499587266700193 (+/- 0.0009820346402831667)
    CMC low bound:	 -0.5005693013404762
    CMC upp bound:	 -0.4986052320599098

Conditioned on a bankruptcy event, when does it occur? Plot a histogram of <!-- the distribution of the bankruptcy time ($\in \mathbb{R}$), and another histogram of -->
the claim number which causes bankruptcy ($\in \mathbb{N}$).

```python
# This is the easiest solution
#plt.hist(ruinTime[~alive], n);

# But this is a better looking version
plt.hist(ruinTime[~alive], np.arange(51)+0.5, rwidth=0.85);
```

## Part 3: Importance sampling

Consider estimating $\mathbb{P}(\max\{X_1, X_2\} > 5)$ where $X_i \overset{\mathrm{i.i.d.}}{\sim} \mathsf{Exponential}(1)$.

Use importance sampling, where each $X_i \overset{\mathrm{i.i.d.}}{\sim} \mathsf{Exponential}(\lambda)$ for some $\lambda \in (0, 1)$.

Do this where $\lambda$ takes values in a grid of points between 0 and 1. Plot the IS estimates and the IS variances, like in the lecture slides (complete with horizontal line for the true value, and the vertical line at the minimal variance $\lambda$).

```python
λs = np.linspace(0.01, 1, 25)
R = 10**6

ellHats = np.zeros_like(λs)
sigmaHats = np.zeros_like(λs)

rng = rnd.default_rng(1)

for i, λ in enumerate(λs):

    # Simulating the index value at T
    exps = rng.exponential(1/λ, size=(R, 2))
    maxs = np.max(exps, axis=1)

    # Calculate likelihood ratios
    lrNumer = stats.expon.pdf(exps, scale=1)
    lrDenom = stats.expon.pdf(exps, scale=1/λ)
    lrs = np.prod(lrNumer / lrDenom, axis=1)

    # Calculating the MC estimate and CIs
    ests = lrs * (maxs > 5)
    ellHats[i] = ests.mean()
    sigmaHats[i] = ests.std()
```

```python
plt.plot(λs, ellHats)
plt.axhline(1 - (1-np.exp(-5))**2, ls="dashed", c="k");
```

```python
# This is actually plotting the standard deviations instead of variances.
plt.plot(λs, sigmaHats)
plt.axvline(λs[np.argmin(sigmaHats)], ls="dashed", c="k");
```

```python
# This is the actual variance plot.
plt.plot(λs, sigmaHats**2)
plt.axvline(λs[np.argmin(sigmaHats)], ls="dashed", c="k");
```

Redo the financial example from the lecture (option pricing with the sum of correlated lognormals) but apply any importance sampling you like. Show the price estimate with a 95% confidence interval, but make sure your choice of importance distribution gives you a smaller confidence interval then crude Monte Carlo.

Here is a crude Monte Carlo estimate. Some of the problem specification constants may be changed (compare to the lecture), use these values.

```python
# Problem constants
n = 2
r = 0.05
T = 1
K = 4
ρ = -0.5
σ2 = 1/10
R = 10**6

# Mean vector and covariance matrix
μ = np.arange(1, n+1) / 10
Σ = σ2 * ( (1-ρ) * np.eye(n) + ρ * np.ones(n) )

# Simulating the index value at T
rng = rnd.default_rng(1)
normals = rng.multivariate_normal(μ, Σ, size=R)
Xs = np.exp(normals)
Ss = Xs.sum(axis=1)

# Calculating the MC estimate and CIs
ests = np.exp(-r*T) * np.maximum(Ss - K, 0)
ellHatCMC = ests.mean()
sigmaHatCMC = ests.std()
widthCICMC = 1.96 * sigmaHatCMC / np.sqrt(R)

print(f"CMC option price: {ellHatCMC} (+- {widthCICMC}))")
print(f"Fraction of simulated options which payed out: {np.mean(Ss - K > 0)}")
```

    CMC option price: 0.0004718514314568297 (+- 2.9660800832771642e-05))
    Fraction of simulated options which payed out: 0.001936

N.B. To calculate pdf of this original distribution at the points in some array `x` use `stats.multivariate_normal.pdf(x, μ, Σ)`

```python
σ2Dash = 10 * σ2

ΣDash = σ2Dash * ( (1-ρ) * np.eye(n) + ρ * np.ones(n) )

# Simulating the index value at T
rng = rnd.default_rng(1)
normals = rng.multivariate_normal(μ, ΣDash, size=R)
Xs = np.exp(normals)
Ss = Xs.sum(axis=1)

# Calculate likelihood ratios
lrNumer = stats.multivariate_normal.pdf(normals, μ, Σ)
lrDenom = stats.multivariate_normal.pdf(normals, μ, ΣDash)
lrs = lrNumer / lrDenom

# Calculating the MC estimate and CIs
ests = lrs * np.exp(-r*T) * np.maximum(Ss - K, 0)
ellHat = ests.mean()
sigmaHat = ests.std()
widthCI = 1.96 * sigmaHat / np.sqrt(R)

print(f"IS option price: {ellHat} (+- {widthCI}))")
print(f"Fraction of simulated options which payed out: {np.mean(Ss - K > 0)}")
```

    IS option price: 0.00046363739368650243 (+- 3.333208983109775e-06))
    Fraction of simulated options which payed out: 0.321168


# Field Integration Techniques

Many analysis techniques for vector fields require solving an initial value problem for an arbitrary set of seed points and evaluating such solutions at a chosen resolution. Kamodo makes it easy to generate fieldline solutions by providing a function decorator that wraps scipy's powerful `solve_ivp` function. Each family of solutions is represented by a single function of a complex parameter. We illustrate the flexibility of this approach in the example below.

```python
# initialize
from plotly.offline import iplot, plot, init_notebook_mode
# init_notebook_mode(connected = True)

from kamodo import Kamodo, event, pointlike, kamodofy, solve
import numpy as np
import pandas as pd
```

## Dipole field model

We use the following dipole field model that can accept (m,) and (1,m), and (n,m) arrays.

```python
def Bdip(rvec):
    """Need math to work in a variety of arg shapes"""
    muvec = Bdip.muvec
    r = np.linalg.norm(rvec, axis = 1)
    r[r==0] = np.nan

    try:
        rhat = rvec/r
    except:
        rhat = (rvec.T/r).T

    try:
        result = 3*np.dot(rhat, muvec.T)
    except:
        result = 3*np.dot(rhat.T, muvec.T).T

    result = (rhat.T*result).T

    try:
        result = result - muvec
    except:
        result = (result - muvec.T).T

    try:
        result = result/r**3
    except:
        result = (result.T/r**3).T

    return result

# set dipole moment
Bdip.muvec = np.array([0, 0, -1])

# pointlike enforces dimensionality
Bdip = pointlike(Bdip, '(n,m)->(n,m)', [np.float], squeeze = 0)
```

```python
kamodo = Kamodo()
```

```python
kamodo['Bvec'] = Bdip # register the dipole field
kamodo
```

\begin{equation}\vec{B}{\left(\vec{r} \right)} = \lambda{\left(\vec{r} \right)}\end{equation}

## Normalization

Instead of solving the initial value problem on the original field, we will be solving on the normalized field. This will mean that the integral path is the same as the arclength, allowing us to control the visual fidelity of the resulting field.

Create a normalization function to be applied to our field

```python
@kamodofy(equation = "$$\\hat{n}(\\vec{y}) = \\vec{y}/\\sqrt{\\vec{y} \\cdot \\vec{y}} $$")
@pointlike(signature = '(m,n)->(m,n)', squeeze = 0)
def normalized(yvec):
    r = np.linalg.norm(yvec, axis = 1)
    r[r==0] = np.nan

    try:
        return yvec/r
    except:
        return (yvec.T/r).T

kamodo['nhat'] = normalized
```

Create a normalized field

```python
kamodo['bhat'] = "nhat(Bvec)"
kamodo
```

\begin{equation}\vec{B}{\left(\vec{r} \right)} = \lambda{\left(\vec{r} \right)}\end{equation}$$\hat{n}(\vec{y}) = \vec{y}/\sqrt{\vec{y} \cdot \vec{y}} $$\begin{equation}\hat{b}{\left(\vec{r} \right)} = \hat{n}{\left(\vec{B}{\left(\vec{r} \right)} \right)}\end{equation}

## Solving the initial value problem

Generate a set of seed points for integration

```python
x0 = np.linspace(-np.pi,np.pi,6)
y0 = np.linspace(-np.pi,np.pi,6)
z0 = 1

seeds = np.array(np.column_stack([c.ravel() for c in np.meshgrid(x0,y0,z0)]))
```

Create a stopping boundary for field line integrator

```python
@event
def boundary(s, rvec):
    r = np.linalg.norm(rvec)

    if np.isnan(r):
        result = 0
    else:
        result = r - 1
    return result

```

Solve the initial value problem for the normalized field

```python
kamodo['svec'] = solve(kamodo.bhat, # the field to be solved
                       seeds, # the initial positions
                       's', # the name of the integration parameter
                       (0,30), # the span to integrate over
                       npoints = 60, # the number of points to evaluate the solution
                       events = boundary, # stop at the boundary
                      )
kamodo
```

\begin{equation}\vec{B}{\left(\vec{r} \right)} = \lambda{\left(\vec{r} \right)}\end{equation}$$\hat{n}(\vec{y}) = \vec{y}/\sqrt{\vec{y} \cdot \vec{y}} $$\begin{equation}\hat{b}{\left(\vec{r} \right)} = \hat{n}{\left(\vec{B}{\left(\vec{r} \right)} \right)}\end{equation}\begin{equation}\vec{s}{\left(s \right)} = \lambda{\left(s \right)}\end{equation}

The solver returns a family of solutions, represented as a single function of a complex array, $\vec{s}(s)$ where
$s$ is a complex array.

## Evaluating the Solutions
On evaluation, $\vec{s}(s)$ returns a pandas dataframe.

```python
kamodo.svec().head()
```

      0
      1
      2

      seed
      integral

      0.0
      -6.610169
      -0.347554
      -0.347554
      -0.924068

      -6.101695
      -0.615865
      -0.615865
      -1.261579

      -5.593220
      -0.922720
      -0.922720
      -1.525495

      -5.084746
      -1.256963
      -1.256963
      -1.713145

      -4.576271
      -1.608411
      -1.608411
      -1.822192

When using the default argument above, the solution evaluates at a resolution of npoints/span, stopping at the boundary.

## Complex parameterization
Kamodo represents the family of solutions to the initial value problem as a single function of a complex array.

The floor of the real part of the input parameter corresponds to the original seed array:

```python
kamodo.svec([0,1,2]).values
```

    array([[-3.14159265, -3.14159265,  1.        ],
           [-1.88495559, -3.14159265,  1.        ],
           [-0.62831853, -3.14159265,  1.        ]])

compare with original seeds:

```python
seeds[[0,1,2]]
```

    array([[-3.14159265, -3.14159265,  1.        ],
           [-1.88495559, -3.14159265,  1.        ],
           [-0.62831853, -3.14159265,  1.        ]])

The imaginary part denotes the integral along the corresponding solution. Here, we can choose evaluation points that were not in the original solution. Parameters outside the original span will be extrapolated.

```python
kamodo.svec([-6j, -5j, 0, 5j, 6j, 4 + 4j, 4 -5.777j])
```

      0
      1
      2

      seed
      integral

      0.0
      -6.000
      -0.674467
      -0.674467
      -1.320541

      -5.000
      -1.314574
      -1.314574
      -1.737228

      0.000
      -3.141593
      -3.141593
      1.000000

      5.000
      -0.120683
      -0.120683
      0.491788

      6.000
      0.115024
      0.115024
      -0.404619

      4.0
      4.000
      0.094188
      -0.156979
      0.481816

      -5.777
      0.234822
      -0.391371
      -0.826931

## Plotting Fieldlines

We can quickly generate plots for all fieldlines at the default resolution by calling plot with the name of the fieldlines solution.

```python
import plotly.io as pio
```

```python
fig = kamodo.plot('svec')
pio.write_image(fig, 'images/fieldlines.svg')
```

To show the direction of the field at each point, we can evaluate $\hat{B}(\vec{s}(s))$

```python
fig = kamodo.plot('svec',
                  Bhat = dict(rvec = kamodo.svec()))
pio.write_image(fig,'images/fieldlines_vectors.svg')
```

## Integration totals

To compute the total integral for each fieldline individually, we need a function to subtract the integration results at the endpoints.

```python
def integral(fieldline):
    endpoints = fieldline.reset_index().integral.iloc[[0,-1]]
    return endpoints.values[-1] - endpoints.values[0]
```

```python
totals = []
for seed, fieldline in kamodo.svec().groupby(level = 'seed'):
    totals.append(integral(fieldline))

totals[:5]
```

    [10.677966101694915,
     8.64406779661017,
     7.627118644067796,
     7.627118644067796,
     8.64406779661017]

Alternatively, we can use pandas' aggregation methods to apply our function on each fieldline.

```python
pd.DataFrame.groupby.a
```

```python
kamodo.svec().groupby(level='seed').aggregate(integral)
```

      0
      1
      2

      seed

      0.0
      10.677966
      10.677966
      10.677966

      1.0
      8.644068
      8.644068
      8.644068

      2.0
      7.627119
      7.627119
      7.627119

      3.0
      7.627119
      7.627119
      7.627119

      4.0
      8.644068
      8.644068
      8.644068

      5.0
      10.677966
      10.677966
      10.677966

      6.0
      8.644068
      8.644068
      8.644068

      7.0
      6.610169
      6.610169
      6.610169

      8.0
      5.084746
      5.084746
      5.084746

      9.0
      5.084746
      5.084746
      5.084746

      10.0
      6.610169
      6.610169
      6.610169

      11.0
      8.644068
      8.644068
      8.644068

      12.0
      7.627119
      7.627119
      7.627119

      13.0
      5.084746
      5.084746
      5.084746

      14.0
      5.593220
      5.593220
      5.593220

      15.0
      5.593220
      5.593220
      5.593220

      16.0
      5.084746
      5.084746
      5.084746

      17.0
      7.627119
      7.627119
      7.627119

      18.0
      7.627119
      7.627119
      7.627119

      19.0
      5.084746
      5.084746
      5.084746

      20.0
      5.593220
      5.593220
      5.593220

      21.0
      5.593220
      5.593220
      5.593220

      22.0
      5.084746
      5.084746
      5.084746

      23.0
      7.627119
      7.627119
      7.627119

      24.0
      8.644068
      8.644068
      8.644068

      25.0
      6.610169
      6.610169
      6.610169

      26.0
      5.084746
      5.084746
      5.084746

      27.0
      5.084746
      5.084746
      5.084746

      28.0
      6.610169
      6.610169
      6.610169

      29.0
      8.644068
      8.644068
      8.644068

      30.0
      10.677966
      10.677966
      10.677966

      31.0
      8.644068
      8.644068
      8.644068

      32.0
      7.627119
      7.627119
      7.627119

      33.0
      7.627119
      7.627119
      7.627119

      34.0
      8.644068
      8.644068
      8.644068

      35.0
      10.677966
      10.677966
      10.677966


# How to size your bets - The Kelly Rule

he Kelly rule has a long history in gambling because it provides guidance on how much to stake on each of an (infinite) sequence of bets with varying (but favorable) odds to maximize terminal wealth. It was published as A New Interpretation of the Information Rate in 1956 by John Kelly who was a colleague of Claude Shannon's at Bell Labs. He was intrigued by bets placed on candidates at the new quiz show The $64,000 Question, where a viewer on the west coast used the three-hour delay to obtain insider information about the winners.

Kelly drew a connection to Shannon's information theory to solve for the bet that is optimal for long-term capital growth when the odds are favorable, but uncertainty remains. His rule maximizes logarithmic wealth as a function of the odds of success of each game, and includes implicit bankruptcy protection since log(0) is negative infinity so that a Kelly gambler would naturally avoid losing everything.

## Imports

```python
import warnings
warnings.filterwarnings('ignore')
```

```python
%matplotlib inline
from pathlib import Path

import numpy as np
from numpy.linalg import inv
from numpy.random import dirichlet
import pandas as pd

from sympy import symbols, solve, log, diff
from scipy.optimize import minimize_scalar, newton, minimize
from scipy.integrate import quad
from scipy.stats import norm

import matplotlib.pyplot as plt
import seaborn as sns
```

```python
sns.set_style('whitegrid')
np.random.seed(42)
```

```python
DATA_STORE = Path('..', 'data', 'assets.h5')
```

## The optimal size of a bet

Kelly began by analyzing games with a binary win-lose outcome. The key variables are:
- b: The odds define the amount won for a \\$1 bet. Odds = 5/1 implies a \\$5 gain if the bet wins, plus recovery of the \\$1 capital.
- p: The probability defines the likelihood of a favorable outcome.
- f: The share of the current capital to bet.
- V: The value of the capital as a result of betting.

The Kelly rule aims to maximize the value's growth rate, G, of infinitely-repeated bets (see Chapter 5 for background).
$$G=\lim_{N\rightarrow\infty}=\frac{1}{N}\log\frac{V_N}{V_0}$$

We can maximize the rate of growth G by maximizing G with respect to f, as illustrated using sympy as follows:

```python
share, odds, probability = symbols('share odds probability')
Value = probability * log(1 + odds * share) + (1 - probability) * log(1 - share)
solve(diff(Value, share), share)
```

    [(odds*probability + probability - 1)/odds]

```python
f, p = symbols('f p')
y = p * log(1 + f) + (1 - p) * log(1 - f)
solve(diff(y, f), f)
```

    [2*p - 1]

## Get S&P 500 Data

```python
with pd.HDFStore(DATA_STORE) as store:
    sp500 = store['sp500/stooq'].close
```

### Compute Returns & Standard Deviation

```python
annual_returns = sp500.resample('A').last().pct_change().dropna().to_frame('sp500')
```

```python
return_params = annual_returns.sp500.rolling(25).agg(['mean', 'std']).dropna()
```

```python
return_ci = (return_params[['mean']]
                .assign(lower=return_params['mean'].sub(return_params['std'].mul(2)))
                .assign(upper=return_params['mean'].add(return_params['std'].mul(2))))
```

```python
return_ci.plot(lw=2, figsize=(14, 8))
plt.tight_layout()
sns.despine();
```

### Kelly Rule for a Single Asset - Index Returns

In a financial market context, both outcomes and alternatives are more complex, but the Kelly rule logic does still apply. It was made popular by Ed Thorp, who first applied it profitably to gambling (described in Beat the Dealer) and later started the successful hedge fund Princeton/Newport Partners.

With continuous outcomes, the growth rate of capital is defined by an integrate over the probability distribution of the different returns that can be optimized numerically.
We can solve this expression (see book) for the optimal f* using the `scipy.optimize` module:

```python
def norm_integral(f, mean, std):
    val, er = quad(lambda s: np.log(1 + f * s) * norm.pdf(s, mean, std),
                               mean - 3 * std,
                               mean + 3 * std)
    return -val
```

```python
def norm_dev_integral(f, mean, std):
    val, er = quad(lambda s: (s / (1 + f * s)) * norm.pdf(s, mean, std), m-3*std, mean+3*std)
    return val
```

```python
def get_kelly_share(data):
    solution = minimize_scalar(norm_integral,
                        args=(data['mean'], data['std']),
                        bounds=[0, 2],
                        method='bounded')
    return solution.x
```

```python
annual_returns['f'] = return_params.apply(get_kelly_share, axis=1)
```

```python
return_params.plot(subplots=True, lw=2, figsize=(14, 8));
```

```python
annual_returns.tail()
```

      sp500
      f

      Date

      2015-12-31
      -0.007266
      1.999996

      2016-12-31
      0.095350
      1.999996

      2017-12-31
      0.194200
      1.999996

      2018-12-31
      -0.062373
      1.999996

      2019-12-31
      0.288781
      1.999996

### Performance Evaluation

```python
(annual_returns[['sp500']]
 .assign(kelly=annual_returns.sp500.mul(annual_returns.f.shift()))
 .dropna()
 .loc['1900':]
 .add(1)
 .cumprod()
 .sub(1)
 .plot(lw=2));

```

```python
annual_returns.f.describe()
```

    count    45.000000
    mean      1.979025
    std       0.062282
    min       1.708206
    25%       1.999996
    50%       1.999996
    75%       1.999996
    max       1.999996
    Name: f, dtype: float64

```python
return_ci.head()
```

      mean
      lower
      upper

      Date

      1975-12-31
      0.076574
      -0.289442
      0.442591

      1976-12-31
      0.077649
      -0.289600
      0.444897

      1977-12-31
      0.068336
      -0.306402
      0.443074

      1978-12-31
      0.071410
      -0.299973
      0.442794

      1979-12-31
      0.058325
      -0.278930
      0.395581

### Compute Kelly Fraction

```python
m = .058
s = .216
```

```python
# Option 1: minimize the expectation integral
sol = minimize_scalar(norm_integral, args=(m, s), bounds=[0., 2.], method='bounded')
print('Optimal Kelly fraction: {:.4f}'.format(sol.x))
```

    Optimal Kelly fraction: 1.1974

```python
# Option 2: take the derivative of the expectation and make it null
x0 = newton(norm_dev_integral, .1, args=(m, s))
print('Optimal Kelly fraction: {:.4f}'.format(x0))
```

    Optimal Kelly fraction: 1.1974

## Kelly Rule for Multiple Assets

We will use an example with various equities. [E. Chan (2008)](https://www.amazon.com/Quantitative-Trading-Build-Algorithmic-Business/dp/0470284889) illustrates how to arrive at a multi-asset application of the Kelly Rule, and that the result is equivalent to the (potentially levered) maximum Sharpe ratio portfolio from the mean-variance optimization.

The computation involves the dot product of the precision matrix, which is the inverse of the covariance matrix, and the return matrix:

```python
with pd.HDFStore(DATA_STORE) as store:
    sp500_stocks = store['sp500/stocks'].index
    prices = store['quandl/wiki/prices'].adj_close.unstack('ticker').filter(sp500_stocks)
```

```python
prices.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 14277 entries, 1962-01-02 to 2018-03-27
    Columns: 482 entries, MMM to ZTS
    dtypes: float64(482)
    memory usage: 52.6 MB

```python
monthly_returns = prices.loc['1988':'2017'].resample('M').last().pct_change().dropna(how='all').dropna(axis=1)
stocks = monthly_returns.columns
monthly_returns.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 359 entries, 1988-02-29 to 2017-12-31
    Freq: M
    Columns: 207 entries, MMM to XRX
    dtypes: float64(207)
    memory usage: 583.4 KB

### Compute Precision Matrix

```python
cov = monthly_returns.cov()
precision_matrix = pd.DataFrame(inv(cov), index=stocks, columns=stocks)
```

```python
kelly_allocation = monthly_returns.mean().dot(precision_matrix)
```

```python
kelly_allocation.describe()
```

    count    207.000000
    mean       0.237375
    std        3.470213
    min      -11.586454
    25%       -1.750238
    50%        0.093455
    75%        2.505289
    max        9.255888
    dtype: float64

```python
kelly_allocation.sum()
```

    49.136694419591635

### Largest Portfolio Allocation

The plot shows the tickers that receive an allocation weight > 5x their value:

```python
kelly_allocation[kelly_allocation.abs()>5].sort_values(ascending=False).plot.barh(figsize=(8, 10))
plt.yticks(fontsize=12)
sns.despine()
plt.tight_layout();
```

### Performance vs SP500

The Kelly rule does really well. But it has also been computed from historical data..

```python
ax = monthly_returns.loc['2010':].mul(kelly_allocation.div(kelly_allocation.sum())).sum(1).to_frame('Kelly').add(1).cumprod().sub(1).plot(figsize=(14,4));
sp500.filter(monthly_returns.loc['2010':].index).pct_change().add(1).cumprod().sub(1).to_frame('SP500').plot(ax=ax, legend=True)
plt.tight_layout()
sns.despine();
```


```python
import sympy as sp
sp.init_printing()
```

```python
a, b, c, w, x, y, z = sp.symbols("a b c w x y z")
```

a few amateurish experiments based on the paper "Teaching Quaternions is not Complex" by J. McDonald (http://dx.doi.org/10.1111/j.1467-8659.2010.01756.x)

```python
A = sp.Matrix([[a, 0], [0, a]])
A
```

```python
B = sp.Matrix([[b, 0], [0, b]])
B
```

```python
A + B
```

```python
A - B
```

```python
A * B
```

```python
# TODO: division?
```

```python
A**2
```

```python
A.inv()
```

```python
A.det()
```

```python
re = sp.Matrix([[1, 0], [0, 1]])
im = sp.Matrix([[0, -1], [1, 0]])
```

```python
im ** 2
```

```python
a * re + b * im
```

```python
theta = sp.symbols("theta")
theta
```

```python
n = sp.cos(theta) * re + sp.sin(theta) * im
n
```

```python
n * sp.Matrix([x, y])
```

```python
n * (x * re + y * im)
```

quaternions ...

Note: quaternions could also be represented as complex 2x2 matrices, see https://gitlab.com/certik/ijk/blob/master/mul.py.

```python
qw = sp.eye(4)
qw
```

```python
qi = sp.zeros(4, 4)
qi[0, 3] = qi[2, 1] = 1
qi[1, 2] = qi[3, 0] = -1
qi
```

```python
qj = sp.zeros(4, 4)
qj[0, 2] = qj[1, 3] = 1
qj[2, 0] = qj[3, 1] = -1
qj
```

```python
qk = sp.zeros(4, 4)
qk[1, 0] = qk[2, 3] = 1
qk[0, 1] = qk[3, 2] = -1
qk
```

```python
qi ** 2
```

```python
qj ** 2
```

```python
qk ** 2
```

```python
qi * qj == qk
```

    True

```python
qj * qk == qi
```

    True

```python
qk * qi == qj
```

    True

```python
qi**2 == qj**2 == qk**2 == -1 * qw
```

    True

```python
q = w * qw + x * qi + y * qj + z * qk
q
```

```python
q_conj = w * qw - x * qi - y * qj - z * qk
q_conj
```

```python
q.transpose() == q_conj
```

    True

  To the extent possible under law,
  the person who associated CC0
  with this work has waived all copyright and related or neighboring
  rights to this work.