```python
# Required to load webpages
from IPython.display import IFrame
```

[Table of contents](../toc.ipynb)

# SciPy

* Scipy extends numpy with powerful modules in
  * optimization,
  * interpolation,
  * linear algebra,
  * fourier transformation,
  * signal processing,
  * image processing,
  * file input output, and many more.
* Please find here the scipy reference for a complete feature list [https://docs.scipy.org/doc/scipy/reference/](https://docs.scipy.org/doc/scipy/reference/).

We will take a look at some features of scipy in the latter. Please explore the rich content of this package later on.

## Optimization

* Scipy's optimization module provides many optimization methods like least squares, gradient methods, BFGS, global optimization, and many more.
* Please find a detailed tutorial here [https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html).
* Next, we will apply one of the optimization algorithms in a simple example.

A common function to test optimization algorithms is the Rosenbrock function for $N$ variables:

$f(\boldsymbol{x}) = \sum\limits_{i=2}^N 100 \left(x_{i+1} - x_i^2\right)^2 + \left(1 - x_i^2 \right)^2$.

The optimum is at $x_i=1$, where $f(\boldsymbol{x})=0$

```python
import numpy as np
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm
%matplotlib inline
```

```python
def rosen(x):
    """The Rosenbrock function"""
    return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1]**2.0)**2.0)
```

We need to generate some data in a mesh grid.

```python
X = np.arange(-2, 2, 0.2)
Y = np.arange(-2, 2, 0.2)
X, Y = np.meshgrid(X, Y)

data = np.vstack([X.reshape(X.size), Y.reshape(Y.size)])
```

Let's evaluate the Rosenbrock function at the grid points.

```python
Z = rosen(data)
```

And we will plot the function in a 3D plot.

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.plot_surface(X, Y, Z.reshape(X.shape), cmap='bwr')
ax.view_init(40, 230)
```

Now, let us check that the true minimum value is at (1, 1).

```python
rosen(np.array([1, 1]))
```

    0.0

Finally, we will call scipy optimize and find the minimum with Nelder Mead algorithm.

```python
from scipy.optimize import minimize

x0 = np.array([1.3, 0.7])
res = minimize(rosen, x0, method='nelder-mead',
               options={'xatol': 1e-8, 'disp': True})

print(res.x)
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 72
             Function evaluations: 142
    [1. 1.]

Many more optimization examples are to find in scipy optimize tutorial [https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html).

```python
IFrame(src='https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html',
       width=1000, height=600)
```

## Interpolation

* Interpolation of data is very often required, for instance to replace NaNs or to fill missing values in data records.
* Scipy comes with
  * 1D interpolation,
  * multivariate data interpolation
  * spline, and
  * radial basis function interpolation.
* Please find here the link to interpolation tutorials [https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html](https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html).

```python
from scipy.interpolate import interp1d

x = np.linspace(10, 20, 15)
y = np.sin(x) + np.cos(x**2 / 10)

f = interp1d(x, y, kind="linear")
f1 = interp1d(x, y, kind="cubic")
```

```python
x_fine = np.linspace(10, 20, 200)

plt.plot(x, y, 'ko',
         x_fine, f(x_fine), 'b--',
         x_fine, f1(x_fine), 'r--')
plt.legend(["Data", "Linear", "Cubic"])
plt.show()
```

## Signal processing

The signal processing module is very powerful and we will have a look at its tutorial [https://docs.scipy.org/doc/scipy/reference/tutorial/signal.html](https://docs.scipy.org/doc/scipy/reference/tutorial/signal.html) for a quick overview.

```python
IFrame(src='https://docs.scipy.org/doc/scipy/reference/tutorial/signal.html',
       width=1000, height=600)
```

## Linear algebra

* In addition to numpy, scipy has its own linear algebra module.
* It offers more functionality than numpy's linear algebra module and is based on BLAS/LAPACK support, which makes it faster.
* The respective tutorial is here located [https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html](https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html).

```python
IFrame(src='https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html',
       width=1000, height=600)
```

### Total least squares as linear algebra application

We will now implement a total least squares estimator [[Markovsky2007]](../references.bib) with help of scipy's singular value decomposition (svd). The total least squares estimator provides a solution for the errors in variables problem, where model inputs and outputs are corrupted by noise.

The model becomes
$A X \approx B$, where $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{m \times d}$ are input and output data, and $X$ is the unknown parameter vector.

More specifically, the total least squares regression becomes
$\widehat{A}X = \widehat{B}$, $\widehat{A} := A + \Delta A$, $\widehat{B} := B + \Delta B$.

The estimator can be written as pseudo code as follows.

$C = [A B] = U \Sigma V^\top$, where $U \Sigma V^\top$ is the svd of $C$.

$V:= \left[\begin{align}V_{11} &V_{12} \\
V_{21} & V_{22}\end{align}\right]$,

$\widehat{X} = -V_{12} V_{22}^{-1}$.

In Python, the implementation could be like this function.

```python
from scipy import linalg

def tls(A, B):
    m, n = A.shape
    C = np.hstack((A, B))
    U, S, V = linalg.svd(C)
    V12 = V.T[0:n, n:]
    V22 = V.T[n:, n:]
    X = -V12 / V22
    return X
```

Now we create some data where input and output are appended with noise.

```python
A = np.random.rand(100, 2)
X = np.array([[3], [-7]])
B = A @ X

A += np.random.randn(100, 2) * 0.1
B += np.random.randn(100, 1) * 0.1
```

The total least squares solution becomes

```python
tls(A, B)
```

    array([[ 2.84623341],
           [-6.99453565]])

And this solution is closer to correct value $X = [3 , -7]^\top$ than ordinary least squares.

```python
linalg.solve((A.T @ A), (A.T @ B))
```

    array([[ 2.36782022],
           [-6.38173796]])

Finally, next function shows a "self" written least squares estimator, which uses QR decomposition and back substitution. This implementation is numerically robust in contrast to normal equations

$A ^\top A X = A^\top B$.

Please find more explanation in [[Golub2013]](../references.bib) and in section 3.11 of [[Burg2012]](../references.bib).

```python
def ls(A, B):
    Q, R = linalg.qr(A, mode="economic")
    z = Q.T @ B

    return linalg.solve_triangular(R, z)
```

```python
ls(A, B)
```

    array([[ 2.36782022],
           [-6.38173796]])

## Integration

* Scipy's integration can be used for general equations as well as for ordinary differential equations.
* The integration tutorial is linked here [https://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html](https://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html).

### Solving a differential equation

Here, we want to use an ode solver to simulate the differential equation (ode)

$y'' + y' + 4 y = 0$.

To evaluate this second order ode, we need to convert it into a set of first order ode. The trick is to use this substitution: $x_0 = y$, $x_1 = y'$, which yields

$\begin{align}
x'_0 &= x_1 \\
x'_1 &= -4 x_0 - x_1
\end{align}$

The implementation in Python becomes.

```python
def equation(t, x):
    return [x[1], -4 * x[0] - x[1]]
```

```python
from scipy.integrate import solve_ivp
```

```python
time_span = [0, 20]
init = [1, 0]
time = np.arange(0, 20, 0.01)
sol = solve_ivp(equation, time_span, init, t_eval=time)
```

```python
plt.plot(time, sol.y[0, :])
plt.plot(time, sol.y[1, :])
plt.legend(["$y$", "$y'$"])
plt.xlabel("Time")
plt.show()
```


```julia
using Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()

using LinearAlgebra, Symbolics, DifferentialEquations, JLD2
```

      Activating environment at `~/Research/symbolics_double_pendulum/Project.toml`
    ┌ Info: Precompiling Symbolics [0c5d862f-8b57-4792-8d23-62f2024744c7]
    └ @ Base loading.jl:1317
    ┌ Info: Precompiling DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
    └ @ Base loading.jl:1317

```julia
struct DoublePendulum{T}
	m1::T
	m2::T
	l1::T
	l2::T
end

n = 2 # number of states
model = DoublePendulum(1.0, 1.0, 1.0, 1.0) # model
```

    DoublePendulum{Float64}(1.0, 1.0, 1.0, 1.0)

```julia
# kinematics
function kinematics_1(model::DoublePendulum, q)
	θ1, θ2 = q

	[0.5 * model.l1 * sin(θ1);
	 -0.5 * model.l1 * cos(θ1)]
end

function kinematics_2(model::DoublePendulum, q)
	θ1, θ2 = q

	[model.l1 * sin(θ1) + 0.5 * model.l2 * sin(θ1 + θ2);
	 -model.l1 * cos(θ1) - 0.5 * model.l2 * cos(θ1 + θ2)]
end
```

    kinematics_2 (generic function with 1 method)

```julia
# fast kinematics functions
@variables q[1:n]
@variables q̇[1:n]

k1 = kinematics_1(model, q)
k2 = kinematics_2(model, q)

k1_exp = Symbolics.build_function(k1, q)
k2_exp = Symbolics.build_function(k2, q)

k1_func = eval(k1_exp[1])
k2_func = eval(k2_exp[1])
```

    #7 (generic function with 1 method)

```julia
# kinematics Jacobians
j1 = Symbolics.jacobian(k1, q, simplify = true)
j2 = Symbolics.jacobian(k2, q, simplify = true)

j1_exp = Symbolics.build_function(j1, q)
j2_exp = Symbolics.build_function(j2, q)

j1_func = eval(j1_exp[1])
j2_func = eval(j2_exp[1])
```

    #11 (generic function with 1 method)

```julia
# Lagrangian
function lagrangian(model, q, q̇)
	L = 0.0

	# mass 1
	v1 = j1_func(q) * q̇
	L += 0.5 * model.m1 * transpose(v1) * v1 		# kinetic energy
	L -= model.m1 * 9.81 * k1_func(q)[2]            # potential energy

	# mass 2
	v2 = j2_func(q) * q̇
	L += 0.5 * model.m2 * transpose(v2) * v2
	L -= model.m2 * 9.81 * k2_func(q)[2]

	return L
end

# fast Lagrangian
L = lagrangian(model, q, q̇)

#
dLq = Symbolics.gradient(L, q, simplify = true)
dLq̇ = Symbolics.gradient(L, q̇, simplify = true)
ddL = Symbolics.hessian(L, [q; q̇], simplify = true)

# mass matrix
M = ddL[n .+ (1:n), n .+ (1:n)]
M = simplify.(M)

# dynamics bias
C = ddL[n .+ (1:n), 1:n] * q̇ - dLq
C = simplify.(C)
```

\begin{equation}
\left[
\begin{array}{c}
q\dot{_2} \left( \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) \left(  - 0.25 q\dot{_1} \sin\left( q{_1} + q{_2} \right) - 0.25 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) + \left( \sin\left( q{_1} \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \right) \left( 0.25 q\dot{_1} \cos\left( q{_1} + q{_2} \right) + 0.25 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) + \left( 0.5 \cos\left( q{_1} \right) + 0.25 \cos\left( q{_1} + q{_2} \right) \right) \left(  - 0.5 q\dot{_1} \sin\left( q{_1} + q{_2} \right) - 0.5 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) + \left( 0.5 \sin\left( q{_1} \right) + 0.25 \sin\left( q{_1} + q{_2} \right) \right) \left( 0.5 q\dot{_1} \cos\left( q{_1} + q{_2} \right) + 0.5 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) + 0.25 \cos\left( q{_1} + q{_2} \right) \left( q\dot{_1} \left( \sin\left( q{_1} \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \right) + 0.5 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \left( 0.5 q\dot{_1} \left( \sin\left( q{_1} \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \right) + 0.25 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) - 0.25 \sin\left( q{_1} + q{_2} \right) \left( q\dot{_1} \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) + 0.5 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) - 0.5 \sin\left( q{_1} + q{_2} \right) \left( 0.5 q\dot{_1} \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) + 0.25 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) \right) + 14.715 \sin\left( q{_1} \right) + 4.905 \sin\left( q{_1} + q{_2} \right) \\
q\dot{_2} \left( 2 \cos\left( q{_1} + q{_2} \right) \left(  - 0.125 q\dot{_1} \sin\left( q{_1} + q{_2} \right) + 0.25 q\dot{_1} \left( \sin\left( q{_1} \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \right) \right) - 0.25 \sin\left( q{_1} + q{_2} \right) \left( q\dot{_1} \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) + 0.5 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \left( 0.25 q\dot{_1} \cos\left( q{_1} + q{_2} \right) + 0.25 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) + 0.25 \sin\left( q{_1} + q{_2} \right) \left( 0.5 q\dot{_1} \cos\left( q{_1} + q{_2} \right) + 0.5 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) - 0.5 \sin\left( q{_1} + q{_2} \right) \left( 0.5 q\dot{_1} \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) + 0.25 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) \right) + 4.905 \sin\left( q{_1} + q{_2} \right) - \left( q\dot{_1} \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) + 0.5 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) \left(  - 0.25 q\dot{_1} \sin\left( q{_1} + q{_2} \right) - 0.25 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) - \left( q\dot{_1} \left( \sin\left( q{_1} \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \right) + 0.5 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) \left( 0.25 q\dot{_1} \cos\left( q{_1} + q{_2} \right) + 0.25 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) - \left( 0.5 q\dot{_1} \cos\left( q{_1} + q{_2} \right) + 0.5 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) \left( 0.5 q\dot{_1} \left( \sin\left( q{_1} \right) + 0.5 \sin\left( q{_1} + q{_2} \right) \right) + 0.25 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) - \left(  - 0.5 q\dot{_1} \sin\left( q{_1} + q{_2} \right) - 0.5 q\dot{_2} \sin\left( q{_1} + q{_2} \right) \right) \left( 0.5 q\dot{_1} \left( \cos\left( q{_1} \right) + 0.5 \cos\left( q{_1} + q{_2} \right) \right) + 0.25 q\dot{_2} \cos\left( q{_1} + q{_2} \right) \right) \\
\end{array}
\right]
\end{equation}

```julia
# dynamics
# ẋ = [q̇; M \ (-1.0 * C)]
ẋ = [q̇; M \ (-0.5 * q̇ -1.0 * C)] # joint friction
# ẋ = [q̇; M \ (-1.0 * (q - [π / 10; 0.0]) -1.0 * C)] # spring

ẋ = simplify.(ẋ)

ẋ_exp = Symbolics.build_function(ẋ, q, q̇)
dynamics = eval(ẋ_exp[1])
```

    #22 (generic function with 1 method)

```julia
# save dynamics function
path = joinpath(pwd(), "dynamics.jld2")
# @save path ẋ_exp
# @load path ẋ_exp
```

    "/home/taylor/Research/symbolics_double_pendulum/dynamics.jld2"

```julia
# DifferentialEquations.jl
function dynamics!(ẋ, x, p, t)
	ẋ .= dynamics(view(x, 1:n), view(x, n .+ (1:n)))
end

# simulate
x0 = [0.5 * π; 0.0; 0.0; 0.0]
tspan = (0.0, 10.0)
dt = 0.01
prob = ODEProblem(dynamics!, x0, tspan);
```

```julia
sol = solve(prob, Tsit5(), adaptive = false, dt = dt);
```

```julia
# MeshCat.jl
include(joinpath(pwd(), "visuals.jl"))
vis = Visualizer()
render(vis)
```

    ┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
    │ http://127.0.0.1:8704
    └ @ MeshCat /home/taylor/.julia/packages/MeshCat/GlCMx/src/visualizer.jl:73

<div style="height: 500px; width: 100%; overflow-x: auto; overflow-y: hidden; resize: both">

</div>

```julia
visualize_double_pendulum!(vis, model, sol.u, Δt = dt)
```

```julia

```

```julia

```


# Korteweg–De Vries equation

* Physical space
\begin{align}
u_t + u_{xxx}u +6uu_x = 0
\end{align}
* Spectral space $\hat{u} = \mathscr{F}\{u\}$
\begin{align}
\hat{u_t} = i k_x^3 \hat{u} - 6\mathscr{F} \{ \mathscr{F}^{-1} \{ u \} \mathscr{F}^{-1} \{ ik_x \hat{u}\} \}
\end{align}

# Imports

```python
import numpy as np
import matplotlib.pyplot as plt
from rkstiff.grids import construct_x_kx_rfft
from rkstiff.etd34 import ETD34
from rkstiff.etd4 import ETD4
%matplotlib inline
```

# Construct grids

```python
# uniform grid spacing, field assumed to be real-valued -> construct_x_kx_rfft
N = 512
a, b = -150, 150
x,kx = construct_x_kx_rfft(N,a,b)
```

# Linear and nonlinear functions for spectral kdV equation

```python
L = 1j*kx**3
def NLfunc(uf):
    u = np.fft.irfft(uf)
    ux = np.fft.irfft(1j*kx*uf)
    return -6*np.fft.rfft(u*ux)
```

# Initial values set to a combination of kdV solitons

```python
A0 = np.array([0.6, 0.5, 0.4, 0.3, 0.2]).reshape(1,5)
x0 = np.array([-120, -90, -60, -30, 0]).reshape(1,5)
u0 = 0.5*A0**2/(np.cosh(A0*(x.reshape(N,1)-x0)/2)**2)
u0 = np.sum(u0,axis=1)
u0FFT = np.fft.rfft(u0)

plt.plot(x,u0)
```

# Initialize constant-step ETD4 and adaptive-step ETD34 solvers

```python
solverCS = ETD4(linop=L,NLfunc=NLfunc) # Try ETD5 for a 5th order method
solverAS = ETD34(linop=L,NLfunc=NLfunc,epsilon=1e-4)
```

# Propagate from time t0 to tf

```python
# store_data -> propagated field stored in solver.u at times solver.t
# store_freq -> propagated field values stored on every store_freq step (default is every step)
uFFT_CS = solverCS.evolve(u0FFT,t0=0,tf=600,h=0.1,store_data=True,store_freq=100)
uFFT_AS = solverAS.evolve(u0FFT,t0=0,tf=600,h_init=0.1,store_data=True,store_freq=25)
```

# Graph propagation

```python
T = [solverCS.t,solverAS.t]
U_FFT = [solverCS.u,solverAS.u]
fig = plt.figure(figsize=(16,12))
titles = ['ETD4 : constant-step','ETD34 : adaptive-step']
for i in range(len(T)):
    ax = fig.add_subplot(1,2,i+1,projection='3d')
    ax.w_xaxis.set_pane_color((0,0,0,0))
    ax.w_yaxis.set_pane_color((0,0,0,0))
    ax.w_zaxis.set_pane_color((0,0,0,0))
    t = np.array(T[i])
    for j,snapshot_fft in enumerate(U_FFT[i]):
        snapshot = np.fft.irfft(snapshot_fft)
        ax.plot(x,t[j]*np.ones_like(x),snapshot,color='black')
    ax.set_xlim([x[0],x[-1]])
    ax.set_ylim([t[0],t[-1]])
    ax.grid(False)
    ax.axis(False)
    ax.view_init(75,-94)
    ax.set_title(titles[i])
plt.tight_layout()
```


### Helpful resources
1. http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/slides/lec19.pdf
2. http://willwolf.io/2018/10/20/thorough-introduction-to-boltzmann-machines/

### Probabilistic model
A Boltzmann machine is a (fully observable) graphical model whose nodes are binary random variables, i.e., $x_i = \{0, 1\}$ for all $i=1, \dots, n$. A bias $b_i$ determines how likely it is that $x_i=1$ and a weight $w_{ij}$ determines how likely it is that $x_i$ and $x_j$ take the same value.
#### Model assumption
A Boltzmann machine assumes that the joint probability distribution $p(x) = p(x_1, \dots, x_n)$ can be modeled as
\begin{align}
p(x) = \frac{e^{H(x)}}{Z},
\end{align}
where $H$ is the happiness function defined as
\begin{align}
H(x) = \sum_{\substack{i=1 \\ j < i}}^n w_{ij} x_i x_j + \sum_i b_i,
\end{align}
and $Z$ is a normalization constant.

### Learning a Boltzmann machine
Given a dataset $\left\{X^{(j)}\right\}_{j \in [m]}$, we can learn the weights and biases of the model by the principle of maximum likelihood. In contrast to the naive Bayes model, there is no closed-form solution for the parameters, however, we can update the parameters using gradient descent.

#### Log-likelihood function
The log-likelihood function $l$ is given as
\begin{align}
l = \left[ \frac{1}{m} \sum_{k=1}^m H(X^{(k)} \right] - \log Z,
\end{align}
and it turns out that
\begin{align}
    \frac{\partial l}{\partial w_{rs}} &= \frac{1}{m} \sum_{k=1}^m X^{(k)}_r X^{(k)}_s - \sum_{x} p(x) x_r x_s, \\
    \frac{\partial l}{\partial b_{r}} &= \frac{1}{m} \sum_{k=1}^m X^{(k)}_r- \sum_{x} p(x) x_r,
\end{align}
where $\sum_x$ is the sum of all possible configurations of the Boltzmann machine.

#### Learning algorithm
We first initialize the weights and biases randomly and can then update them using gradient descent, i.e.,
\begin{align}
    w^{n+1}_{rs} &\gets w^{n}_{rs} + \frac{\partial l}{\partial w^{n}_{rs}}, \\
    b^{n+1}_{r} &\gets b^{n}_{r} + \frac{\partial l}{\partial b^{n}_{r}}.
\end{align}
Note that the sum over all configurations of the Boltzmann machine has $2^n$ terms making exact gradient descent steps infeasible. Hence, we approximate $\sum_{x} p(x) x_r x_s = \mathbb{E}_{\text{model}} [x_r x_s]$ and $\sum_{x} p(x) x_r = \mathbb{E}_{\text{model}} [x_r]$ using a Markov chain Monte Carlo method. For big datasets, we could approximate $\frac{1}{m} \sum_{k=1}^m X^{(k)}_r X^{(k)}_s = \mathbb{E}_{\text{data}} [x_r x_s]$ as well as $\frac{1}{m} \sum_{k=1}^m X^{(k)}_r = \mathbb{E}_{\text{data}} [x_r]$ using mini-batches.

### Sampling
For sampling we use a Markov chain Monte Carlo method called Gibbs sampling. The reason for doing so is that Gibbs sampling is based on the conditional probability distributions
\begin{align}
\text{Pr}\left(x_i = 1 \mid x_{-i}\right),
\end{align}
which can be easily computed with our model assumption; note that $x_{-i} = \{x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n\}$. It turns out that
\begin{align}
\text{Pr}\left(x_i = 1 \mid x_{-i}\right) = \sigma \left( \sum_{j \neq i} w_{ij} x_j + b_i \right),
\end{align}
where $\sigma$ is the logistic function
\begin{align}
\sigma(x) = \frac{1}{1 + \exp ( -x )}.
\end{align}
As usual with Markov chain Monte Carlo methods, Gibbs sampling generates a Markov chain of samples. Generally, samples from the beginining of the chain might not represent the true probability distribution well, and therefore we discard the first $n_{\text{burn-in}}$ samples.

```python
import numpy as np
```

```python
class BoltzmannMachine:
    def __init__(self, init_weights, init_biases, configs):
        self.weights = init_weights
        self.biases = init_biases
        self.configs = configs

    @staticmethod
    def _logistic_function(x):
        return 1 / (1 + np.exp(-x))

    def _happiness_function(self, x):
        happiness = 0

        for i in range(self.weights.shape[0]):
            for j in range(i):
                happiness += self.weights[i, j] * x[i] * x[j]

        for i, _ in enumerate(self.biases):
            happiness += self.biases[i] * x[i]

        return happiness

    def likelihood(self, x, compute_log_likelihood=True):
        exponential_happiness = [np.exp(self._happiness_function(x_i)) for x_i in x]

        Z = sum([np.exp(self._happiness_function(config)) for config in self.configs])

        if compute_log_likelihood == True:
            return sum([np.log(exp_happ)  - np.log(Z) for exp_happ in exponential_happiness])
        else:
            return reduce(np.multiply, [exp_happ / Z for exp_happ in exponential_happiness])

    def gibbs_sampling(self, n_samples, n_burn_in=1000):
        samples = []
        samples.append([0 for _ in self.biases])

        def _gibbs_step(sample, i):
            z = sum([self.weights[i, j] * sample[j] for j in range(len(sample)) if j != i]) + self.biases[i]
            p = self._logistic_function(z)
            return np.random.binomial(n=1, p=p)

        for _ in range(n_samples + n_burn_in):
            sample = list(samples[-1])
            for i, _ in enumerate(sample):
                sample[i] = _gibbs_step(sample=sample, i=i)

            samples.append(sample)

        return np.array([sample for i, sample in enumerate(samples[n_burn_in:])])
```

Next, we define two functions to update the weights of the Boltzmann machine. The first function computes $\mathbb{E}_{\text{model}} [x_r x_s]$ and $\mathbb{E}_{\text{model}} [x_r]$ exactly, whereas the second function approximates those quantities using Gibbs sampling.

```python
def gradient_descent(data, model, learning_rate):
    model_distribution = [(np.array(config), model.likelihood(np.array([config]), compute_log_likelihood=False)) \
                           for config in model.configs]
    ### weights
    for i in range(model.weights.shape[0]):
        for j in range(i):
            w_positive = (data[:, i] * data[:, j]).mean()

            w_negative = sum([config[i] * config[j] * likelihood for config, likelihood in model_distribution])

            model.weights[i, j] += learning_rate * (w_positive - w_negative)

    ### biases
    for i in range(model.biases.shape[0]):
        b_positive = data[:, i].mean()

        b_negative = sum([config[i] * likelihood for config, likelihood in model_distribution])

        model.biases[i] += learning_rate * (b_positive - b_negative)
```

```python
def approximate_gradient_descent_using_gibbs_sampling(data, model, learning_rate, n_samples):
    gibbs_samples = model.gibbs_sampling(n_samples)

    ### weights
    for i in range(model.weights.shape[0]):
        for j in range(i):
            w_positive = (data[:, i] * data[:, j]).mean()

            w_negative = (gibbs_samples[:, i] * gibbs_samples[:, j]).mean()

            model.weights[i, j] += learning_rate * (w_positive - w_negative)

    ### biases
    for i in range(model.biases.shape[0]):
        b_positive = data[:, i].mean()

        b_negative = (gibbs_samples[:, i]).mean()

        model.biases[i] += learning_rate * (b_positive - b_negative)
```

```python
from itertools import product
from functools import reduce
from collections import defaultdict

import matplotlib.pyplot as plt
%matplotlib inline

n_units = 3
n_points = 100
n_steps = 1000
learning_rate = 0.1
p = [.8, .1, .5]

### [[0, 0, 0], [0, 0, 1], [0, 1, 0], ..., [1, 1, 0], [1, 1, 1]]
all_configs = list(product([0, 1], repeat=n_units))

biases = np.random.randn(n_units)
### we initialize weights as a n_units by n_units, but actually just use the elements below the main diagonal
weights = np.random.randn(n_units, n_units)

boltzmann_machine = BoltzmannMachine(weights, biases, all_configs)
```

We are producing some toy data in 3 dimensions.

```python
data =  np.random.binomial(n=1, p=p, size=(n_points, n_units))

data_distribution = np.zeros((len(all_configs),))

for x in data:
    for j, config in enumerate(all_configs):
        if np.sum(x == config) == n_units:
            data_distribution[j] += 1 / n_points
```

We are now training the Boltzmann machine using $n_{\text{steps}}$ steps of gradient descent.

```python
for i in range(n_steps):
    gradient_descent(data, boltzmann_machine, learning_rate)

    if i % 100 == 0:
        log_likelihood = boltzmann_machine.likelihood(data)
        print(f'Epoch: {i:2} | Log-likelihood: {log_likelihood}')
```

    Epoch:  0 | Log-likelihood: -288.0336365242908
    Epoch: 100 | Log-likelihood: -154.3247149791821
    Epoch: 200 | Log-likelihood: -152.98770974403578
    Epoch: 300 | Log-likelihood: -152.66784399515788
    Epoch: 400 | Log-likelihood: -152.4571187752501
    Epoch: 500 | Log-likelihood: -152.30205110294634
    Epoch: 600 | Log-likelihood: -152.1863146368242
    Epoch: 700 | Log-likelihood: -152.09938160789244
    Epoch: 800 | Log-likelihood: -152.03375796159676
    Epoch: 900 | Log-likelihood: -151.98401084176203

We can not print the probability of each configuration learned by the Boltzmann machine.

```python
model_distribution = [(np.array(config),
                       boltzmann_machine.likelihood(np.array([config]),
                       compute_log_likelihood=False))
                      for config in boltzmann_machine.configs]

for i, _ in enumerate(boltzmann_machine.configs):
    print('Configuration:', all_configs[i],
          'learned distribution: {0:.3f}'.format(model_distribution[i][1]),
          'data distribution: {0:.3f}'.format(data_distribution[i]))
```

    Configuration: (0, 0, 0) learned distribution: 0.089 data distribution: 0.100
    Configuration: (0, 0, 1) learned distribution: 0.044 data distribution: 0.040
    Configuration: (0, 1, 0) learned distribution: 0.025 data distribution: 0.020
    Configuration: (0, 1, 1) learned distribution: 0.011 data distribution: 0.010
    Configuration: (1, 0, 0) learned distribution: 0.353 data distribution: 0.350
    Configuration: (1, 0, 1) learned distribution: 0.381 data distribution: 0.380
    Configuration: (1, 1, 0) learned distribution: 0.050 data distribution: 0.050
    Configuration: (1, 1, 1) learned distribution: 0.047 data distribution: 0.050

Lastly, we want to produce samples using Gibbs sampling and compare the sample distribution to the learned distribution/data distribution.

```python
n_samples = 10000
samples = boltzmann_machine.gibbs_sampling(n_samples=10000)

sampling_distribution = np.zeros((len(all_configs),))

for sample in samples:
    for j, config in enumerate(all_configs):
        if np.sum(sample == config) == n_units:
            sampling_distribution[j] += 1 / n_samples
```

```python
for i, _ in enumerate(boltzmann_machine.configs):
    print('Configuration:', all_configs[i],
          'sampling distribution: {0:.3f}'.format(sampling_distribution[i]))
```

    Configuration: (0, 0, 0) sampling distribution: 0.083
    Configuration: (0, 0, 1) sampling distribution: 0.038
    Configuration: (0, 1, 0) sampling distribution: 0.027
    Configuration: (0, 1, 1) sampling distribution: 0.011
    Configuration: (1, 0, 0) sampling distribution: 0.344
    Configuration: (1, 0, 1) sampling distribution: 0.372
    Configuration: (1, 1, 0) sampling distribution: 0.065
    Configuration: (1, 1, 1) sampling distribution: 0.059

Let us now train a model using the approximate gradient descent method (and the same data).

```python
n_units = 3
n_points = 100
n_steps = 1000
learning_rate = 0.1
p = np.random.uniform(0, 1, size=(n_units,))

all_configs = list(product([0, 1], repeat=n_units))

biases = np.random.randn(n_units)
weights = np.random.randn(n_units, n_units)

boltzmann_machine = BoltzmannMachine(weights, biases, all_configs)
```

```python
n_samples = 100

for i in range(n_steps):
    approximate_gradient_descent_using_gibbs_sampling(data, boltzmann_machine, learning_rate, n_samples)

    if i % 100 == 0:
        log_likelihood = boltzmann_machine.likelihood(data)
        print(f'Epoch: {i:2} | Log-likelihood: {log_likelihood}')
```

    Epoch:  0 | Log-likelihood: -258.1925192118695
    Epoch: 100 | Log-likelihood: -153.3524918808026
    Epoch: 200 | Log-likelihood: -153.7383024413453
    Epoch: 300 | Log-likelihood: -153.6838978949054
    Epoch: 400 | Log-likelihood: -154.05059012518075
    Epoch: 500 | Log-likelihood: -153.9162049494429
    Epoch: 600 | Log-likelihood: -154.15105041690703
    Epoch: 700 | Log-likelihood: -154.0103952779528
    Epoch: 800 | Log-likelihood: -154.17221830910893
    Epoch: 900 | Log-likelihood: -154.22467600409564

```python
model_distribution = [(np.array(config),
                       boltzmann_machine.likelihood(np.array([config]),
                       compute_log_likelihood=False))
                      for config in boltzmann_machine.configs]

for i, _ in enumerate(boltzmann_machine.configs):
    print('Configuration:', all_configs[i],
          'learned distribution: {0:.3f}'.format(model_distribution[i][1]),
          'data distribution: {0:.3f}'.format(data_distribution[i]))
```

    Configuration: (0, 0, 0) learned distribution: 0.055 data distribution: 0.100
    Configuration: (0, 0, 1) learned distribution: 0.025 data distribution: 0.040
    Configuration: (0, 1, 0) learned distribution: 0.013 data distribution: 0.020
    Configuration: (0, 1, 1) learned distribution: 0.005 data distribution: 0.010
    Configuration: (1, 0, 0) learned distribution: 0.380 data distribution: 0.350
    Configuration: (1, 0, 1) learned distribution: 0.429 data distribution: 0.380
    Configuration: (1, 1, 0) learned distribution: 0.046 data distribution: 0.050
    Configuration: (1, 1, 1) learned distribution: 0.047 data distribution: 0.050


```python
from sympy import *
x, y, z, t = symbols('x y z t')
```

## Complex numbers

Ever since Newton, the word “number” has been used to refer to one
of the following types of math objects: the naturals $\mathbb{N}$, the integers
$\mathbb{Z}$, the rationals $\mathbb{Q}$, and the real numbers $\mathbb{R}$. Each set of numbers is
associated with a different class of equations. The natural numbers
$\mathbb{N}$ appear as solutions of the equation $m + n = x$, where $m$ and $n$ are
natural numbers (denoted $m, n \in \mathbb{N}$). The integers $\mathbb{Z}$ are the solutions
to equations of the form $x + m = n$, where $m, n \in \mathbb{N}$. The rational
numbers $\mathbb{Q}$ are necessary to solve for $x$ in $mx = n$, with $m, n \in \mathbb{Z}$.
The solutions to $x^2 = 2$ are irrational (so $\not\in \mathbb{Q}$) so we need an even
larger set that contains *all* possible numbers: real set of numbers $\mathbb{R}$.
A pattern emerges where more complicated equations require the
invention of new types of numbers.

Consider the quadratic equation $x^2 = -1$. There are no real solutions
to this equation, but we can define an imaginary number $i = \sqrt{-1}$
(denoted `I` in `SymPy`) that satisfies this equation:

```python
I*I
```

$\displaystyle -1$

```python
solve( x**2 + 1 , x)
```

    [-I, I]

The solutions are $x = i$ and $x = -i$, and indeed we can verify that
$i^2 + 1 = 0$ and $(-i)^2 + 1 = 0$ since $i^2 = -1$.

The complex numbers $\mathbb{C}$ are defined as $\{ a+bi \,|\, a,b \in \mathbb{R} \}$. Complex numbers
contain a real part and an imaginary part:

```python
z = 4 + 3*I
z
```

$\displaystyle 4 + 3 i$

```python
re(z)
```

$\displaystyle 4$

```python
im(z)
```

$\displaystyle 3$

The *polar* representation of a complex number is $z\!\equiv\!|z|\angle\theta\!\equiv \!|z|e^{i\theta}$.
For a complex number $z=a+bi$,
the quantity $|z|=\sqrt{a^2+b^2}$ is known as the absolute value of $z$,
and $\theta$ is its *phase* or its *argument*:

```python
Abs(z)
```

$\displaystyle 5$

```python
arg(z)
```

$\displaystyle \operatorname{atan}{\left(\frac{3}{4} \right)}$

The complex conjugate of $z = a + bi$ is the number $\bar{z} = a - bi$:

```python
conjugate( z )
```

$\displaystyle 4 - 3 i$

Complex conjugation is important for computing the absolute value
of $z$ $\left(|z|\equiv\sqrt{z\bar{z}}\right)$ and for division by $z$ $\left(\frac{1}{z}\equiv\frac{\bar{z}}{|z|^2}\right)$.

### Euler's formula

[Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula) shows an important relation between the exponential
function $e^x$ and the trigonometric functions $sin(x)$ and $cos(x)$:

$$e^{ix} = \cos x + i \sin x.$$

To obtain this result in `SymPy`, you must specify that the number $x$ is
real and also tell `expand` that you're interested in complex expansions:

```python
x = symbols('x', real=True)
exp(I*x).expand(complex=True)
```

$\displaystyle i \sin{\left(x \right)} + \cos{\left(x \right)}$

```python
re( exp(I*x) )
```

$\displaystyle \cos{\left(x \right)}$

```python
im( exp(I*x) )
```

$\displaystyle \sin{\left(x \right)}$

Basically, $\cos(x)$ is the real part of $e^{ix}$, and $\sin(x)$ is the imaginary
part of $e^{ix}$. Whaaat? I know it's weird, but weird things are bound
to happen when you input imaginary numbers to functions.

Euler's formula is often used to rewrite the functions `sin` and `cos` in
terms of complex exponentials. For example,

```python
(cos(x)).rewrite(exp)
```

$\displaystyle \frac{e^{i x}}{2} + \frac{e^{- i x}}{2}$

Compare this expression with the definition of hyperbolic cosine.


# Hypothesis testing for two binomials distributions

Hypothesis testing for a [binomial proportion](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval) is a well-studied problem and most statistical libraries have built-in [solutions](https://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_confint.html) for generating confidence intervals (CIs) and p-values. Testing the *difference* between two binomial proportions is a similar statistical problem, but there are fewer available methods. In my experience, most statistical handbooks recommend using normal approximations or contingency tables when there are two sets of binary outcomes to be compared. Unfortunately these approaches have drawbacks. Normal approximations based on asymptotic theory have noticeably poor coverage when sample sizes are small or the probability of an event is close to zero or one. [Fisher's Exact test](https://en.wikipedia.org/wiki/Fisher%27s_exact_test) is undoubtedly a powerful contingency table method for obtaining p-values under the null of no difference. However it is hard to use for power calculations, and when the outcome space is large it requires Monte Carlo approximations. The idea of using an *exact* distribution of the difference in two binomial proportions is not a new one (see [here](https://www.jstor.org/stable/2533740?seq=1) or [here](https://arxiv.org/pdf/1903.03327.pdf)). "Exact" testing refers to using actual quantiles of the test statistic's distribution, rather than approximations, under the null. Using exact inference techniques for the difference between binomial distributions is almost never used, and, as far as I know, not supported in languages like `python`.

In this post I want to provide researchers with the tools necessary to carry out hypothesis testing for the difference between two binomial distributions. I will also highlight the main statistical challenge inherent in the inference of the problem: when the null proportion is not known before the sampling. By the end of the post, I will provide:

1. Functions to calculate the PMF and CDF for the distribution of a difference between binomials
2. Functions to generate quantiles for this distribution at any nominal level
3. A proof that the type-I errors are unknowably conservative when the null proportion needs to be estimated.

Note that if the null proportion is known before testing, then the type-I errors will be conservative, but *knowably* so, since the difference between the targeted and actual levels will be determined by the discrete nature of the CDF. However, when the null proportion itself needs to be estimated, I will show that the type-I errors will be more conservative than what is suggested by the CDF derived from the empirical null proportion.

## (1) Review: Exact inference for a single binomial distribution

This first section will review the binomial distribution, establish notation, and show that it has exact and knowably-conservative type-I errors for hypothesis testing. The binomial distribution, $y \sim B(n,\pi)$, is parameterized by the number of trials ($n$) and the probability of a positive outcome ($\pi$), where $E(y)=n\pi$. As there are many ways to have $k$ successes, the PMF of this discrete distribution uses the [binomial coefficient](https://en.wikipedia.org/wiki/Binomial_coefficient) to capture these *combinations*:

$$
\begin{align*}
f_{\pi,n}(k) &= P(y = k) = \begin{pmatrix} n \\ k \end{pmatrix} \pi^k (1-\pi)^{1-k} \\
F_{\pi,n}(k) &= P(y \leq k) = \sum_{j=0}^k f_{\pi,n}(j)
\end{align*}
$$

The binomial distribution can represent a number of useful processes we may want to carry out inference on such as the frequency of credit defaults, the sensitivity of a test, or the precision of a classier. All of these processes  count the number of times a binary event happens out of a total number of trials. Hypothesis testing for a binomial random variable is therefore to reject (or not) some claim about the underlying proportion $\pi$, since this is the parameter that defines the distribution ($n$ obviously being known). For example if we wanted to test whether a medical test had a sensitivity of greater than 90% we would want to reject the following null:

$$
\begin{align*}
H_0:& \pi \leq 0.9 \\
H_A:& \pi > 0.9
\end{align*}
$$

The test statistic $\hat\pi=y/n$ is the obvious choice to estimate this quantity since it is the  minimum-variance unbiased estimator (MVUE) of $\pi$. A two-sided hypothesis test partitions the space of observable test statistic values into a rejection and acceptance region. A hypothesis test that of the $1-\alpha$ level should have an acceptance region with the following properties:

$$
\begin{align*}
P(\hat\pi \in [l, u]| H_0) \geq 1-\alpha \hspace{1mm},
\end{align*}
$$

Where $\alpha$ is the type-I error rate. In other words, the probability that the parameter of interest falls outside this interval occurs *at most* $\alpha$% of the time, **when the null is true**. To be clear, the interval $[l,u]$ is not a confidence interval,[[^1]] it is the acceptance region. This requirement to be conservative means the choice of quantile needs to be similarly conservative with respect to the CDF. A one-sided hypothesis test is equivalent to setting either $u$ or $l$ as equal to $\pm \infty$.

For example, consider the distribution of 20 fair coin flips where the probability that 13 or fewer heads will occur is 94%, whereas 14 or fewer heads occurring is 98%. If one needed to pick an upper bound (assume a one-sided test) so that the number of times the null of $\pi\leq 0.5$ was rejected was *at most* 5%, then one would have to the interval $[0,14]$, even though the type-I error rate would actually only be 2% rather than the desired 5%.

$$
\begin{align}
k_{\alpha}^* &= F_{\pi,n}^{-1}(\alpha) = \begin{cases} \inf_k \hspace{2mm} \{ k: F_{\pi,n}(k) \geq \alpha \} &\text{if}& \alpha \geq 0.5  \\ \sup_k \{ k: F_{\pi,n}(k) \leq \alpha \} + 1 &\text{if}& \alpha < 0.5  \end{cases} \label{eq:thresh_binom}
\end{align}
$$

Equation \eqref{eq:thresh_binom} shows how to conservatively invert the CDF if the binomial distribution so that intervals formed by $[k_{\alpha}^*, k_{1-\alpha}^*]$ obtain create a acceptance region with a type-I error or at most $2\alpha$.[[^2]] But since the test statistic is $y/n$ and not $y$, it is easy enough to see how the upper and lower bounds are to be constructed:

$$
\begin{align*}
l, u &= \Bigg( \frac{k_{\alpha/2}}{n}, \frac{k_{1-\alpha/2}}{n} \Bigg)
\end{align*}
$$

Note that $(l,u)$ are implicitly functions of $\pi$ and $n$. The rejection region (the space where the null hypothesis is rejected) is also easily defined:

$$
\begin{align*}
\text{Reject } H_0 &= \big\{ \hat\pi < n^{-1}k_{\alpha/2}^*, \hspace{2mm} \hat\pi > n^{-1}k_{1-\alpha/2} \big\}
\end{align*}
$$

Where $\pi$ is the proportion under the null. The first code block below defines the function `qbinom` which implements \eqref{eq:thresh_binom} and shows whether the type-I errors are conservative and if the discrepancy between the predicted and actual type-I error rates can be made for the 90% level.

[^1]: This definition of a confidence interval has been in use since Jerzy Neyman's definition of it in a classic 1937 [paper](https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.1937.0005).

[^2]: You may be curious why I am adding 1 to the lower bound critical value but not the upper bound. The reason for this is simple, but a bit technical. Recall that $F_{0.5,20}(5)=0.02$ and $F_{0.5,20}(6)=0.06$, so that $k_{0.05}^{*}=5$. But if $P(y\leq 5)=0.02$, then $P(y \geq 6)=0.98$, because the distribution is discrete. In other words, the bound we want for $l$ is 6 and not 5, since $P(y\leq 5) \neq P(y \geq 5)$.

```python
import warnings
import numpy as np
import pandas as pd
import plotnine
from plotnine import *
from scipy.stats import norm, binom

# Function to calculate t*_alpha
def qbinom(alpha, n, pi):
    assert (alpha >= 0) & (alpha <= 1)
    kstar = int(binom.ppf(q=alpha,n=n,p=pi)) # Use first quantile estimate
    Fk = binom.cdf(k=kstar,n=n,p=pi) # Calculate p-value
    check = bool(np.where(alpha>0.5, Fk>=alpha, Fk<=alpha))
    while not check:
        kstar = int(np.where(alpha>0.5, kstar+1, kstar-1))
        Fk = binom.cdf(k=kstar,n=n,p=pi)
        check = bool(np.where(alpha>0.5, Fk>=alpha, Fk<=alpha))
    return kstar

np.random.seed(1234)
nsim = 250000
alpha = 0.1
pi_seq = np.round(np.arange(0.01,1,0.01),2)
n_seq = 2**np.arange(3,8,1)
decimals = 12 # Getting float point issues otherwise

holder = []
for pi in pi_seq:
    for n in n_seq:
        # Establish the lower/upper quantile bounds
        ku = qbinom(alpha=1-alpha/2, n=n, pi=pi)
        kl = 1 + qbinom(alpha=alpha/2, n=n, pi=pi)
        # Draw data under the null
        khat = np.random.binomial(n, pi, size=nsim)
        phat = khat / n
        # Calculate the CIs under the null
        l = phat - (ku/n - pi)
        u = phat + (pi-kl/n)
        coverage = np.mean((pi >= np.round(l,decimals)) & (pi <= np.round(u,decimals)))
        # Calculate the expected coverage
        t1_u = 1-binom.cdf(k=ku,n=n,p=pi)
        t1_l = binom.cdf(k=kl-1,n=n,p=pi)
        t1e = t1_u + t1_l
        tmp = pd.DataFrame({'pi':pi,'n':n,'coverage':coverage,'expected':t1e},index=[0])
        holder.append(tmp)
sim_exact = pd.concat(holder).reset_index(None,True).assign(n=lambda x: pd.Categorical(x.n, x.n.unique()),actual=lambda x: 1-x.coverage)
# Make plots
plotnine.options.figure_size = (5, 4)
gg_cov_exact = (ggplot(sim_exact, aes(x='pi',y='coverage',color='n'))+
               theme_bw() + geom_line() +
               labs(x='π',y='Coverage') +
               ggtitle('Figure 1A: Acceptance rate for exact binomial CIs\n90% level') +
               theme(legend_position=(0.5,0.25),legend_direction='horizontal') +
               scale_y_continuous(limits=[0.85,1.0],breaks=list(np.arange(0.85,1.01,0.05))) +
               geom_hline(yintercept=0.90,linetype='--') +
               scale_color_brewer(name='Sample size', palette='Reds'))
print(gg_cov_exact)
plotnine.options.figure_size = (5, 4)
gg_t1_exp = (ggplot(sim_exact, aes(x='expected',y='actual',color='n'))+
               theme_bw() + geom_point() +
               labs(x='Predicted type-I',y='Actual type-I') +
               ggtitle('Figure 1B: Type-I error is knowably conservative') +
               scale_color_brewer(name='Sample size', palette='Reds') +
             geom_abline(slope=1,intercept=0,linetype='--') +
             theme(legend_position=(0.75,0.35)))
print(gg_t1_exp)
```

Figure 1A demonstrates how we can obtain a conservative type-I error for the binomial distribution. Keep in mind that in the case where $\pi$ is close to 0 or 1, and $n$ is small, the type-I error may amount to 0% at the 90% level! Though we will  discuss power calculations for these tests, it is clear that no sample size will be sufficient to reject the null in this case. For example if $n=8$ and $\pi=0.7$, then $F(7)=0.94$ and $F(8)=1.0$, so the null will never be rejected a value of $\hat \pi > u$. Ironically "exact" tests can be quite inexact!  Yet it is arguable whether hypothesis testing should occur for such small sample sizes anyways. Figure 1B shows the conservatism of the type-I error can be perfectly predicted. Such knowable discrepancies are useful when performing power calculations under some alternative $\pi$.

## (2) Defining the distribution for two binomials

Suppose there are two binomial distributions $y_1 \sim B(\pi_1, n_1)$ and $y_2 \sim B(\pi_2, n_2)$, then one may want to ask whether the difference between their proportions is zero $\Delta_\pi=\pi_1 - \pi_2=0$? There are all sorts of processes where a researcher would want to know whether two binomial distributions differed. For example one could compare the proportion of negative side-effects between two drug trial arms, or the proportion of economic quarters spent in recession between two countries. In most real-word cases, the hypothesis of interest is whether $\pi_1=\pi_2$, with variations in $n_1$ and $n_2$ occurring naturally from differences in the observation process.[[^3]] There may be situations where $\pi_i$ is already known for one or both groups and one simply wants to examine the likelihood of such data being generated under this null hypothesis. Because the realizations of $y_1$ and $y_2$ are independent, one can calculate the PMF by taking the outer product of the respective PMFs.  More formally, we can define the PMF and CDF as follows:

$$
\begin{align*}
f_{\Delta}(k_1, k_2;\mathbf{n}, \mathbf{\pi}) &=  f_{\pi_1,n_1}(k_1) \cdot f_{\pi_2,n_2}(k_2) \\
F_{\Delta}(k_1, k_2; \mathbf{n}, \mathbf{\pi}) &= P(y_1\leq k_1, y_1\leq k_2 ) \\
&= \sum_{j_1=0}^{k_1} \sum_{j_2=0}^{k_2} f_\Delta(j_1, j_2;\mathbf{n}, \mathbf{\pi}) \\
&= F_{\pi_1,n_1}(j_1) \cdot F_{\pi_2,n_2}(j_2), \hspace{5mm} \text{by independence}
\end{align*}
$$

In the code block below we will define a class `dist_binom2` which can calculate the PMF and CDF of any $(y_1,y_2)$ pair as well as pre-calculating these values for the support of the distribution.

[^3]: In the case of comparing the economic performance between two countries, the number of years that GDP has been recorded will likely differ.

```python
class dist_binom2():
    def __init__(self, n1, pi1, n2, pi2):
        check1 = (n1 >= 0) & (n2 >= 0) & (isinstance(n1,int)) & (isinstance(n1,int))
        assert check1
        check2 = (pi1 >= 0) & (pi1 <= 1) & (pi2 >= 0) & (pi2 <= 1)
        assert check2
        r1, r2 = range(n1+1), range(n2+1)
        self.n1, self.pi1, self.n2, self.pi2 = n1, pi1, n2, pi2
        pmf1 = np.array([binom.pmf(k=k,n=n1,p=pi1) for k in r1])
        pmf2 = np.array([binom.pmf(k=k,n=n2,p=pi2) for k in r2])
        cdf1 = np.array([binom.cdf(k=k,n=n1,p=pi1) for k in r1])
        cdf2 = np.array([binom.cdf(k=k,n=n2,p=pi2) for k in r2])
        mat_pmf = pd.DataFrame(np.outer(pmf2, pmf1)).rename_axis('v2').reset_index()
        mat_cdf = pd.DataFrame(np.outer(cdf2, cdf1)).rename_axis('v2').reset_index()
        self.mat_dist = mat_pmf.melt('v2',None,'v1','pmf').merge(mat_cdf.melt('v2',None,'v1','cdf'),'left',['v2','v1'])
        self.mat_dist[['v1','v2']] = self.mat_dist[['v1','v2']].astype(int)
        self.mat_dist = self.mat_dist.sort_values('pmf').reset_index(None,True)
        idx_dist = self.mat_dist.groupby(['pmf']).pmf.sum().rename_axis('idx').reset_index()
        idx_dist = idx_dist.assign(pval=lambda x: x.pmf.cumsum()).drop(columns='pmf').rename(columns={'idx':'pmf'})
        self.mat_dist = self.mat_dist.merge(idx_dist,'left','pmf')

        self.mat_dist = self.mat_dist.assign(pval=lambda x: x.pmf.cumsum())

    def cdf(self,y1,y2):
        assert (y1>=0) & (y1<=self.n1) & (y2>=0) & (y2<=self.n2)
        return binom.cdf(k=y1,n=self.n1,p=self.pi1) * binom.cdf(k=y2,n=self.n2,p=self.pi2)

    def pmf(self,y1,y2):
        assert (y1>=0) & (y1<=self.n1) & (y2>=0) & (y2<=self.n2)
        return binom.pmf(k=y1,n=self.n1,p=self.pi1) * binom.pmf(k=y2,n=self.n2,p=self.pi2)

n1, n2 = 10, 15
pi1, pi2 = 0.5, 0.5
dist = dist_binom2(n1=n1, pi1=pi1, n2=n2, pi2=pi2)
mat_dist = dist.mat_dist.copy().drop(columns='pval')
# mat_dist = mat_dist.assign(pval=2*np.min(np.c_[mat_dist.cdf,1-mat_dist.cdf],1)).assign(pval=lambda x: np.where(x.pval==0,1e-10,x.pval))
mat_dist = mat_dist.melt(['v1','v2'],None,'tt')
plotnine.options.figure_size = (8, 4)
gg_outer = (ggplot(mat_dist,aes(x='v1',y='v2',fill='-np.log10(value)')) + theme_bw() +
            geom_tile() + guides(color=False) + labs(x='k1',y='k2') +
            facet_wrap('~tt',labeller=labeller(tt={'cdf':'CDF','pmf':'PMF','pval':'P-value'})) +
            ggtitle('Figure 2A: PMF and CDF for n1=10, n2=15, and π=0.5') +
            scale_x_continuous(breaks=list(range(n1+1))) +
            scale_y_continuous(breaks=list(range(n2+1))) +
            scale_fill_continuous(name='-log10(p)'))
gg_outer
```

### CHECK THAT FISHER AND EXACT ARE EQUALLY CONSERVATIVE

```python
n1, n2 = 10, 15
pi1, pi2 = 0.5, 0.5
dist = dist_binom2(n1=n1, pi1=pi1, n2=n2, pi2=pi2)

# Do a simulation
from scipy.stats import fisher_exact as fisher_exact_sp
from pyranges.statistics import fisher_exact as fisher_exact_py
nsim = 15000
np.random.seed(1234)
y1, y2 = np.random.binomial(n1, pi1, nsim), np.random.binomial(n2, pi2, nsim)
df_sim = pd.DataFrame({'y1':y1,'f1':n1-y1, 'y2':y2, 'f2':n2-y2})
pval_fisher = fisher_exact_py(df_sim.y1, df_sim.f1, df_sim.y2, df_sim.f2).P
df_sim = df_sim.rename(columns={'y1':'v1','y2':'v2'}).drop(columns=['f1','f2'])
pval_exact = df_sim.merge(dist.mat_dist.drop(columns=['pmf','cdf'])).pval
p_seq = np.arange(0.01,1,0.01)
t1_fisher = [np.mean(pval_fisher <= p) for p in p_seq]
t1_exact = [np.mean(pval_exact <= p) for p in p_seq]
df_t1_comp = pd.DataFrame({'p':p_seq,'fisher':t1_fisher, 'exact':t1_exact}).melt('p',None,'tt')
plotnine.options.figure_size = (8, 4)
gg_t1_comp = (ggplot(df_t1_comp, aes(x='p',y='value',color='tt')) + theme_bw() + geom_point() +
              facet_wrap('~tt',labeller=labeller(tt={'exact':'Exact Binomial','fisher':"Fisher's Test"})) +
              geom_abline(slope=1,intercept=0,color='black',linetype='--') +
              guides(color=False) + labs(x='P-value',y='Null rejection (%)') +
              ggtitle('Figure X: '))
gg_t1_comp
```

```python

```

### VARY PI2 AND COMPARE POWER

```python
n1, n2, pi1 = 10, 15, 0.5
pi2_null = 0.5
dist_null = dist_binom2(n1=n1, pi1=pi1, n2=n2, pi2=pi2_null)

holder = []
for p in p_seq:
    tmp_dist = dist_binom2(n1=n1, pi1=p, n2=n2, pi2=p)
    holder.append(tmp_dist.mat_dist.assign(p=p))
dist_emp = pd.concat(holder).reset_index(None,True)

pi2_seq = np.round(np.arange(pi2_null,0.24,-0.01),2)
pi2_seq = np.round(np.arange(pi2_null,0.76,0.01),2)
nsim = 15000
np.random.seed(nsim)
```

```python
di_cn = {'P':'fisher_both','PLeft':'fisher_left', 'PRight':'fisher_right'}
holder = []
for pi2 in pi2_seq:
    y1, y2 = np.random.binomial(n1, pi1, nsim), np.random.binomial(n2, pi2, nsim)
    pi_hat = (y1+y2)/(n1+n2)
    df_sim = pd.DataFrame({'v1':y1,'f1':n1-y1, 'v2':y2, 'f2':n2-y2, 'p':pi_hat})
    pval_fisher = fisher_exact_py(df_sim.v1, df_sim.f1, df_sim.v2, df_sim.f2).P
    pval_exact_hat = df_sim.merge(dist_emp,'left',['v1','v2','p']).pval
    pval_exact_null = df_sim.merge(dist_null.mat_dist,'left',['v1','v2']).pval
    tmp = pd.DataFrame({'pi2':pi2, 'fisher':pval_fisher, 'exact_null':pval_exact_null, 'exact_hat':pval_exact_hat})
    holder.append(tmp)
res_sim = pd.concat(holder).melt('pi2',None,'tt')
res_power = res_sim.groupby(['pi2','tt']).apply(lambda x: np.mean(x.value<0.05)).reset_index().rename(columns={0:'power'})

plotnine.options.figure_size = (6, 4.5)
gg_power = (ggplot(res_power,aes(x='pi2',y='power',color='tt')) + theme_bw() +
            geom_path(size=2) + geom_hline(yintercept=0.05, linetype='--') +
            ggtitle('Figure X: Comparison of power by pi2'))
gg_power
```

```python

```

Figure 2A shows that shows the PMF centered on the mean of the two binomials and is symmetric (since $\pi_1=\pi_2=0.5$). The multivariate CDF does have the "normal" properties we expect from a univariate CDF. For example, the sum of the densities for $F_\Delta(k_1,k_2)\approx 0.05$ does not equal 0.05!

Next let's examine how different combinations of $\pi_1$ and $\pi_2$ shift the PMF and the corresponding "rejection" rejection at the 90% level (i.e. 10% type-I error).

```python
n1, n2 = 10, 15
pi1_seq = [0.25, 0.5, 0.75] #[0.2, 0.4, 0.6, 0.8]
pi2_seq = pi1_seq

holder = []
for pi1 in pi1_seq:
    for pi2 in pi2_seq:
        dist = dist_binom2(n1=n1, pi1=pi1, n2=n2, pi2=pi2)
        dist.mat_dist = dist.mat_dist.assign(reject=lambda x: np.where(x.cdf<0.05,'x',np.where(x.cdf>0.95,'y','')))
        holder.append(dist.mat_dist.assign(pi1=pi1,pi2=pi2))
        del dist
df = pd.concat(holder)
df = df.assign(reject2=lambda x: np.where(x.reject=='',0,1),
               reject_l=lambda x: np.where(x.reject=='x',1,0),
               reject_u=lambda x: np.where(x.reject=='y',1,0))
plotnine.options.figure_size = (10, 8)
gg_comp = (ggplot(df, aes(x='v1',y='v2',fill='-np.log10(pmf)')) + theme_bw() +
          geom_tile() + facet_grid('pi1~pi2',labeller=label_both)) #geom_text(aes(label='reject'),color='white')
gg_comp
```

```python

```

As in Section 1, we will want to define a conservative rejection region to bound the type-I error rates.

$$
\begin{align}
(k_{1}^*(\alpha), k_{2}^*(\alpha)) &= \begin{cases} \{ (k_1, k_2): F_{\Delta}(k_1, k_2) \geq \alpha \} \text{ if } \alpha \geq 0.5 \\ \{ (k_1, k_2): F_{\Delta}(k_1, k_2) \leq \alpha \} \text{ if } \alpha < 0.5 \end{cases} \label{eq:thresh_binom}
\end{align}
$$

In other words, the rejection for two binomials is the set of pairs of successes that occur less than $1-\alpha$ percent of the time under the null of the distribution on the CDF. At this point you may be wondering why we haven't defined a new random variable, $d=y_1-y_2$, and used this as the basis for inference under the null. The problem is that hypothesis testing for a such a distribution is not identifiable because multiple configurations of $(\pi_1,\pi_2)$ can lead to the same moments of the distribution.

## (3) Hypothesis testing with $\pi_1$ and $\pi_2$ are known

When the proportions (and trial size) or both groups 1 and 2 are known, then exact CIs can be constructed similar to those of the single-group case in section (1). The population parameter of interest is now $\pi_1 - \pi_2=\Delta_\pi\in [-1,1]$, with the MVUE statistic being,

$$
\begin{align*}
\hat \Delta_\pi=\begin{cases}
y_1/n_1-y_2/n_2 &\text{if } \Delta_\pi\neq 0 \\
(y_1+y_2)/(n_1+n_2) &\text{if } \Delta_\pi= 0
\end{cases}
\end{align*}
$$

1. If $\pi_1=\pi_2$, this is easy
2. If $n_1=n_2$, ...
3. Problem of identifiability? Can we use simulataneous CIs for normal?
<!-- https://projecteuclid.org/download/pdf_1/euclid.aoms/1177728912 -->

```python
np.random.seed(1234)
nsim = 100000
alpha = 0.10
pi1_seq = [0.4] # np.round(np.arange(0.1,1,0.1),2)
pi2_seq = [0.6]
n1_seq = [50] # [50, 100, 150]
n2_seq = [100] # n1_seq
nperm = len(pi_seq)*len(n1_seq)*len(n2_seq)
holder = []
count = 0
for pi1 in pi1_seq:
    for pi2 in pi2_seq:
        for n1 in n1_seq:
            for n2 in n2_seq:
                # Generate the data and statistics
                n_tot = (n1+n2)
                y1 = np.random.binomial(n1, pi1, nsim)
                y2 = np.random.binomial(n2, pi2, nsim)
                delta = y1 - y2
                if pi1 == pi2:
                    Dpi_hat = (y1+y2)/n_tot
                else:
                    Dpi_hat = y1/n1 - y2/n2

```

```python

```

```python
# Generate quantile bounds/CIs
Dpi_0 = np.round((pi1-pi2),2)
dist_dbinom = binom_diff(n1, pi1, n2, pi2)
dl, du = dist_dbinom.qdf(alpha/2), dist_dbinom.qdf(1-alpha/2)
l = phat - (du/n - pi)
u = phat + (pi-kl/n)
coverage = np.mean((pi >= np.round(l,decimals)) & (pi <= np.round(u,decimals)))
# Calculate the expected coverage
t1_u = 1-binom.cdf(k=ku,n=n,p=pi)
t1_l = binom.cdf(k=kl-1,n=n,p=pi)
t1e = t1_u + t1_l
tmp = pd.DataFrame({'pi':pi,'n':n,'coverage':coverage,'expected':t1e},index=[0])
```

    array([-48, -51, -38, ..., -40, -32, -36])

```python
# type1 = np.zeros([nsim, 2])
# bounds = type1.copy().astype(int)
# stime = time()
# for ii in range(nsim):
#     dist_ii = binom_diff(pi1=pi_hat[ii],n1=n1,pi2=pi_hat[ii],n2=n2)
#     lb_ii, ub_ii = (dist_ii.qdf(alpha/2), dist_ii.qdf(1-alpha/2))
#     type1[ii] = [dist_ii.cdf(lb_ii), 1-dist_ii.cdf(ub_ii)]
#     lb_ii += 1  # see explaination above for this
#     bounds[ii] = [lb_ii, ub_ii]
# cov_lb = np.mean(delta_hat >= bounds[:,0])
# cov_ub = np.mean(delta_hat <= bounds[:,1])
# err_lb, err_ub = np.mean(type1[:,0]), np.mean(type1[:,1])
# tmp = pd.DataFrame({'cov_lb':cov_lb, 'cov_ub':cov_ub,
#                     'err_lb':err_lb, 'err_ub':err_ub,
#                     'n1':n1, 'n2':n2, 'pi':pi},index=[0])
# print(tmp)
# holder.append(tmp)
# count += 1
# nleft, nsec = nperm - count, time() - stime
# print('pi: %0.2f, n1=%i, n2=%i, took %i seconds (%i of %i)' % (pi, n1, n2, nsec, count, nperm))
# df_res = pd.concat(holder).reset_index(None,True)
# #df_res.to_csv('df_res.csv',index=False)
# # df_res = df_res.assign(t1_lb=lambda x: 1-x.cov_lb, t1_ub=lambda x: 1-x.cov_ub)
# # df_res = df_res.assign(type1=lambda x: x.t1_lb+x.t1_ub, etype1=lambda x: x.err_lb+x.err_ub)
# # df_res_long = df_res.melt(['n1','n2','pi'],['type1','etype1'],'tt')

# # plotnine.options.figure_size = (7, 6.5)
# # gg_cov_delta = (ggplot(df_res_long,aes(x='pi',y='value',color='tt')) + theme_bw() +
# #                 geom_point() + facet_wrap('~n1+n2',ncol=3,labeller=label_both) +
# #                 geom_hline(yintercept=0.1,color='black') + labs(x='π',y='Type-I error') +
# #                 ggtitle('Figure 3: Type-I error rates for 90% CI') +
# #                 scale_x_continuous(breaks=list(np.sort(df_res_long.pi.unique()))) +
# #                 scale_y_continuous(limits=[0,0.12],breaks=list(np.arange(0,0.14,0.02))) +
# #                 theme(panel_grid_minor=element_blank()) +
# #                 scale_color_discrete(name='Type-I',labels=['Expected','Actual']))
# # gg_cov_delta
```

Figure 3 shows that when $n_1=n_2$, then the expected type-I error rate lines up with the actual one. First let me explain how the expected type-I error rate was calculated. Recall the previous example where $F_\delta(-63)=0.038$ and $F_\delta(-38)=0.962$, then setting the non-rejection region as $-62 \leq \delta \leq -38$ will get an actual type-I error rate of $0.038+(1-0.962)=0.0762$, which though smaller than the 10% level we are targetting, is at least knowable. However, in practice we don't know the true distribution of $F_\delta$ because we have to estimated $\pi$ (only $n_1+n_2$ is known).

```python
np.random.seed(seed)
alpha = 0.10
n1, n2, pi = 150, 150, 0.5
nsim, n_tot = 5000, n1+n2
y1 = np.random.binomial(n1, pi, nsim)
y2 = np.random.binomial(n2, pi, nsim)
dist_gt = binom_diff(pi1=pi,n1=n1,pi2=pi,n2=n2)
pi_hat = (y1+y2)/n_tot
delta_hat = (y1 - y2)
type1 = np.zeros([nsim, 2])
bounds = type1.copy().astype(int)
for ii in range(nsim):
    dist_ii = binom_diff(pi1=pi_hat[ii],n1=n1,pi2=pi_hat[ii],n2=n2)
    lb_ii, ub_ii = (dist_ii.qdf(alpha/2), dist_ii.qdf(1-alpha/2))
    type1[ii] = [dist_ii.cdf(lb_ii), 1-dist_ii.cdf(ub_ii)]
    lb_ii += 1
    bounds[ii] = [lb_ii, ub_ii]

print(type1.mean(0))
print((np.mean(delta_hat < bounds[:,0]), np.mean(delta_hat > bounds[:,1])))

dist_ii = binom_diff(pi1=0.35,n1=n1,pi2=0.35,n2=n2)
(dist_ii.qdf(alpha/2), dist_ii.qdf(1-alpha/2))

lb_gt, ub_gt = (dist_gt.qdf(alpha/2), dist_gt.qdf(1-alpha/2))
t1_lb_gt, t1_ub_gt = (dist_gt.cdf(lb_gt), 1-dist_gt.cdf(ub_gt))
lb_gt += 1
dat_bounds_gt = pd.DataFrame({'bound':['lb','ub'], 'gt':[lb_gt, ub_gt],'pval':[t1_lb_gt,t1_ub_gt]})
dat_bounds = pd.DataFrame(bounds,columns=['lb','ub']).assign(phat=pi_hat,idx=range(len(pi_hat)))
dat_bounds = dat_bounds.melt(['idx','phat'],None,'bound')
dat_cov = pd.DataFrame({'lb':delta_hat < bounds[:,0], 'ub':delta_hat > bounds[:,1],'pi':pi_hat})
dat_cov['qq'] = pd.cut(dat_cov.pi,[0]+dat_cov.pi.quantile(np.arange(0.2,1.01,0.2)).to_list())
print(dat_cov.groupby('qq')[['lb','ub']].mean().sum(1))

# Expected type-I by pi_hat
dat_et1 = pd.DataFrame({'t1':type1.sum(1),'phat':pi_hat})
plotnine.options.figure_size = (5,4)
(ggplot(dat_et1, aes(x='phat',y='t1')) + theme_bw() + geom_point())

plotnine.options.figure_size = (7,3.5)
gg_tmp = (ggplot(dat_bounds,aes(y='phat',x='value',color='bound')) + theme_bw() +
          facet_wrap('~bound',scales='free_x') + geom_point() +
         theme(subplots_adjust={'wspace': 0.1}) +
         geom_vline(aes(xintercept='gt'),data=dat_bounds_gt))
gg_tmp

(ggplot(pd.DataFrame({'d':delta_hat,'pi':pi_hat}), aes(x='pi',y='d')) + geom_point() +
 theme_bw())
```


```python
from sympy.physics.mechanics import ReferenceFrame,Point,dynamicsymbols
from sympy.physics.mechanics import Point
from sympy import latex,pprint,symbols, init_printing
import numpy as np
init_printing() # Para visualizar símbolos
```

## 1.1 Marcos de referencia y sistemas de coordenadas

```python
# Definimos marcos de referencia con sympy.physics.mechanics
a=ReferenceFrame('A')
b=ReferenceFrame('B')
c=ReferenceFrame('C')
```

```python
# Cada marco de referencia automaticamente define un sistema de coordenadas.
# Observe que la notación en sympy es x,y,z en vez de 1,2,3.
# Podemos verificar la definición de los vectores unitarios con el producto cruz (pag 2.)
c.z.cross(c.x)
```

## 1.2 Variables de movimiento

```python
#Para el ejemplo de la atracción de parque (Figura 3) en vez de definir los marcos de referencia
# por separado podemos utilizar cada marco de referencia para definir el siguiente y vincularlos
# a través de una rotación relativa, definida por las variables de movimiento (q1,q2,q3).

# Definimos primero el marco A (base de la atracción)
a=ReferenceFrame('A')
# Ahora los symbolos para las variables de movimiento
q1,q2=symbols('q1,q2')
# Definimos b rotando q1 respecto a a.x
b=a.orientnew('B','Axis',(q1,a.x))
# Definimos c rotando q2 respecto a b.z
c=b.orientnew('C','Axis',(q2,b.z))

# De esta manera quedan definidas las coordenadas como propone el ejemplo. Esto lo podemos verificar
# facilmente:
print(a.x==b.x) #True
print(c.z==c.z) #True
print(a.x==b.y) #False

```

    True
    True
    False

## 1.3 Derivadas de vectores

```python
# Implemente el robot SCARA de la Figura 4:
# Defina los símbolos para las variables de movimiento
q1,q2,q3=symbols('q1,q2,q3')
# Defina los marcos de referencia para cada parte A B C A
a=ReferenceFrame('A')
# Defina b rotando q1 respecto a a.z
b=a.orientnew('B','Axis',(q1,a.z))
# Defina c rotando q2 respecto a b.z
c=b.orientnew('C','Axis',(q2,b.z))
# No necesita definir D, ya que tiene la misma orientación de C

# Luega defina cada punto usando los sistemas de coordenadas
# Primero el Origen O
O=Point('O')
# El Punto P en la base del hombro esta a una distancia l1 en el eje a.z
l1,l2,l3=symbols('l1,l2,l3')
P=O.locatenew('P',l1*a.z)
Q=P.locatenew('Q',l2*b.x)
R=Q.locatenew('R',l3*c.x)
S=R.locatenew('S',q3*c.z)

```

```python
#Ahora encuentre la posición del punto O al punto S
v=S.pos_from(O)
v
```

```python
# Y la posición del punto Q al punto S
w=S.pos_from(Q)
w
```

```python
# Si se calcula la derivada en el marco de referencia B
w.diff(q1,b)
```

```python
# Si se calcula la derivada en el marco de referencia A
w.diff(q1,a)
```

## 1.4 Derivadas parciales

```python
# Ejemplo modelo simplificado de una pierna

# Defina los símbolos para las variables de movimiento
q1,q2,q3,q4=symbols('q1,q2,q3,q4')

# Se definen marcos de referencia para cada parte A B C D
a=ReferenceFrame('A')

# Aqui un comentario para aclarar esta seccion:
# Segun el ejemplo del libro q1 es rotacion en a1(ax) y luego q2 es rotacion en y.
# La figura 5 puede no ser muy clara por si sola hasta que no se revisa el eje intermedio e (figura 6) y se entiende
# que las rotaciones q1 y q2  son flexión/extensión (q1) y abducción/adducción (q2) de la cadera.
# Teniendo en cuenta esto, se define también el marco de referencia intermedio "E" para mayor claridad.
# Aunque se podría realizar directamente la definición del marco b asi:
# b=a.orientnew('B','Body',(q1,q2,0),'XYZ')

# Defina e rotando q1 en a.x
e=a.orientnew('E','Axis',(q1,a.x))
# Defina b rotando q2 en e.y
b=e.orientnew('B','Axis',(q2,e.y))
# Defina c rotando q3 en b.z
c=b.orientnew('C','Axis',(q3,b.x))
# Defina D rotando q4 en c.x
d=c.orientnew('D','Axis',(q4,-c.x))

# Luega defina cada punto usando los sistemas de coordenadas
# Primero el Origen O en la pelvis
O=Point('O')
l1,l2,l3,l4,l5,l6,l7,l8=symbols('l1,l2,l3,l4,l5,l6,l7,l8') # variables de distancia
origen_b=O.locatenew('P',-l1*a.x+l2*a.y-l3*b.x-l4*b.y)
rodilla=origen_b.locatenew('Q',-l5*b.z)
origen_c=rodilla.locatenew('R',-l6*c.z)
origen_d=origen_c.locatenew('S',-l7*d.z-l8*d.y)

# Ahora calcule los vectores
u=origen_d.pos_from(rodilla) # rodilla hasta punta del pie
v=origen_c.pos_from(origen_b) # cadera hasta tobillo
w=rodilla.pos_from(O) # pelvis hasta rodilla
```

```python
# Tabla 1. Cambio en los vectores en los marcos de referencia
tables=dict()
for frame in [a,b,c,d]:
    tbl=np.zeros((4,3),dtype=bool)
    for j,vec in enumerate([u,v,w]):
        for i,coord in enumerate([q1,q2,q3,q4]):
            tbl[i,j]=not(vec.diff(coord,frame)==0)
    tables[frame.name]=tbl
```

```python
# Confirmando los resultados de la tabla 1.
tables
```

    {'A': array([[ True,  True,  True],
            [ True,  True,  True],
            [ True,  True, False],
            [ True, False, False]]),
     'B': array([[False, False,  True],
            [False, False,  True],
            [ True,  True, False],
            [ True, False, False]]),
     'C': array([[False, False,  True],
            [False, False,  True],
            [False,  True,  True],
            [ True, False, False]]),
     'D': array([[False, False,  True],
            [False, False,  True],
            [False,  True,  True],
            [ True,  True,  True]])}

```python
# Puede utilizar el metodo express para expresar cualquier vector en el
# marco de referencia deseado y verificar si contiene terminos qi.
w.express(b)
```

```python
# Tambien puede utilizar el metodo express para establecer las relaciones
# entre vectores unitarios de la figura 6.
d.y.express(c)
```

```python
# Y por supuesto calcular derivadas parciales de estos vectores
d.y.diff(q4,c).simplify()
```

```python
# Puede verificar todas las derivadas parciales del ejemplo
b.x.diff(q2,a).simplify()
#b.y.diff(q2,a).simplify()
#b.z.diff(q2,a).simplify()
#a.x.diff(q1,b).simplify()
#a.y.diff(q1,b).simplify()
#a.z.diff(q1,b).simplify()

```

## 1.5 Derivada total

```python
# Se utiliza el modelo simplificado de una pierna con una simplificación
# adicional: espesor 0.

# Se definen las variables de movimiento.
# Como se quiere encontrar la derivada total (derivada respecto al tiempo), en este caso
# se utilizan simbolos dinamicos para las variables de movimiento. (q1(t), q2(t), etc.)
q1,q2,q3,q4=dynamicsymbols('q1,q2,q3,q4')

# Los marcos se mantienen igual, pero podemos simplificar la definción y eliminar
# el marco intermedio E, al usar b orientado con rotaciones sucesivas ('Body').
# Se definen marcos de referencia para cada parte A B C D
a=ReferenceFrame('A')
# Defina b rotando sucesivamente en x (ax) y luego en y (ey).
b=a.orientnew('B','Body',(q1,q2,0),'XYZ')
# Defina c rotando q3 en b.z
c=b.orientnew('C','Axis',(q3,b.x))
# Defina D rotando q4 en c.x
d=c.orientnew('D','Axis',(q4,-c.x))

# Luega defina cada punto usando los sistemas de coordenadas
# Primero el Origen O en la pelvis
O=Point('O')
la,lb,lc,ld=symbols('la,lb,lc,ld') # variables de distancia
origen_b=O.locatenew('P',-la*a.x)
rodilla=origen_b.locatenew('Q',-lb*b.z)
origen_c=rodilla.locatenew('R',-lc*c.z)
origen_d=origen_c.locatenew('S',-ld*d.y)

# Ahora calcule los vectores
u=origen_d.pos_from(rodilla) # rodilla hasta punta del pie
v=origen_c.pos_from(origen_b) # cadera hasta tobillo
w=rodilla.pos_from(O) # pelvis hasta rodilla
```

```python
# Puede verificar los vectores
u # rodilla hasta punta del pie
# v # cadera hasta tobillo
# w # pelvis hasta rodilla
```

```python
# Puede calcular las derivadas totales
u.dt(d) # derivada du/dt en el marco D
#v.dt(d).express(c) # derivada dv/dt en el marco D expresadas en coordenadas c
#w.dt(a) # derivada dw/dt en el marco A
```

## 1.6 Matrices de rotacion

```python
# Defina un marco de referencia A
a=ReferenceFrame('A')
# Defina el símbolo theta para el angulo de rotación
theta=symbols('theta')
# Defina un marco de referencia B aplicando una rotación en ax.
b=a.orientnew('B','Axis',(theta,a.x))
# Puede expresar vectores definidos con componentes en b. Por ejemplo:
b1,b2,b3=symbols('b1,b2,b3')
vec1=b1*b.x+b2*b.y+b3*b.z # Vector vec1 definido por componentes b1 en bx, b2 en by, b3, en bz.

# Observe que pasa cuando expresa el vector en el sistema de coordenadas de A
vec1.express(a)
#vec1.express(b)
```

## 1.7 Cosenos directores

```python
# Defina un marco de referencia A
a=ReferenceFrame('A')
# Defina el símbolo theta para el ángulo de rotación
theta=symbols('theta')
# Ahora defina un marco de referencia B con con diferentes rotaciones.
b=a.orientnew('B','Axis',(theta,a.x))
print('en x:')
pprint(a.dcm(b))
b=a.orientnew('B','Axis',(theta,a.y))
print('en y:')
pprint(a.dcm(b))
b=a.orientnew('B','Axis',(theta,a.z))
print('en z:')
pprint(a.dcm(b))
```

    en x:
    ⎡1    0        0   ⎤
    ⎢                  ⎥
    ⎢0  cos(θ)  -sin(θ)⎥
    ⎢                  ⎥
    ⎣0  sin(θ)  cos(θ) ⎦
    en y:
    ⎡cos(θ)   0  sin(θ)⎤
    ⎢                  ⎥
    ⎢   0     1    0   ⎥
    ⎢                  ⎥
    ⎣-sin(θ)  0  cos(θ)⎦
    en z:
    ⎡cos(θ)  -sin(θ)  0⎤
    ⎢                  ⎥
    ⎢sin(θ)  cos(θ)   0⎥
    ⎢                  ⎥
    ⎣  0        0     1⎦

## 1.8 Diádicas de vectores

```python
# Puede construir diádicas utilizando el método outer (producto diádico)
from sympy.physics.mechanics import outer,dot
A=outer(a.x,a.x)+outer(a.y,a.y)+outer(a.z,a.z)
A # Diádica A
```

```python
#Calule el producto A.b
A.dot(b1*b.x+b2*b.y+b3*b.z)
```


Recursion

## Review: Running time and big-$\mathcal{O}$: Recursive Algorithms for Computing Powers

As example of the use of linear recursion, we consider the problem of raising a number $x$ to an arbitrary nonnegative integer, $n$. That is, we wish to compute the power function, defined as $power(x,n) = xn$. (We use the name “power” for this discussion, to differentiate from the built-in Python function _pow_ that provides such functionality.) We will consider two different recursive formulations for the problem that lead to algorithms with very different performance.
A trivial recursive definition follows from the fact that $x^n = x \cdot x^{n−1}$ for $n > 0$.

\begin{equation}
power(x,n) ==\left\{
                \begin{array}{ll}
                        1 \ if \ n = 0\\
                        x \ power(x, n − 1) \ otherwise.
                \end{array}
              \right.
\end{equation}

This definition leads to a recursive algorithm shown below:

```python
def basicpower(x, n):
    '''Compute the value x**n for integer n.'''
    if n == 0:
        return 1
    else:
        return x  * power(x, n-1)
```

A recursive call to this version of power(x,n) runs in $\mathcal{O}(n)$ time.

```python
def power(x, n):
    '''Compute the value x**n for integer n.'''
    if n == 0:
        return 1
    else:
        partial = power(x, n // 2)
        result = partial * partial
    if n % 2 == 1:
        result *= x
    return result
```

To analyze the running time of the revised algorithm, we observe that the exponent in each recursive call of function power(x,n) is at most half of the preceding exponent. As we saw with the analysis of binary search, the number of times that we can divide $n$ in half before getting to one or less is $\mathcal{O}(log n)$. Therefore, our new formulation of the power function results in $\mathcal{O}(logn)$ recursive calls. Each individual activation of the function uses $\mathcal{O}(1)$ operations (excluding the recursive calls), and so the total number of operations for computing power(x,n) is O(logn). This is a significant improvement over the original $\mathcal{O}(n)$-time algorithm.

**Exercise:** Test the two functions with a timer and see the difference. Identify the base case and stopping conditions.

# Exercise: Fibonacci

The sequence, in which each number is the sum of the two preceding numbers is known as the Fibonacci sequence: $1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, \cdots$ (each number is the sum of the previous two, e.g., $144=55 + 89$).

As you covered Fibonacci in Semester 1, this should be familiar. The twist is we are going to look at a variety of ways of implementing it and the complexity of execution for each.

A direct implementation of the description above would lead to pseudo code like this:
function: fibonacci(n)
---------------------
INPUT n, a natural number
OUTPUT the n'th Fibonacci number

   if n <= 1 then
        return n
   else
        fibonacci(n-1) + fibonacci(n-2)
   endif

Two recursive solutions for a Fibonacci function are coded in Python below. The pseudo code version is given a function name bad_fibonacci.

1. Run the two solutions with timers. Which one has a faster running time and why is one considered bad and the other good? What is the big-$\mathcal{O}$ complexity for bad_fibonacci?
2. Is good_fibonacci tail recursive? Explain your answer. What is the big-$\mathcal{O}$ complexity?
3. Implement a tail recursive solution. **Hint:** use two "accumulators", one (say, **acc2**) for the previous value in the sequence ($n-1$ if you're computing the $n$-th number) and one (say, **acc1**) for the value before that in the sequence ($n-2$ if you're computing the $n$-th element of the sequence). Thus the function becomes **fibo_tail(n,acc1,acc2)** and in the algorithm you have a recursive call to **fibo_tail(n-1,acc2,acc2+acc1)**?: you decrease $n$ until the stopping condition ($n=1$) and accumulate in the two other parameters. **Hint 2:** If you follow this, the initial call will be **fibo_tail(n,1,1)**?.
4. Implement an iterative version of fibonacci using a for or while loop. What is the big-$\mathcal{O}$ complexity?
5. Write another iterative version of Fibonacci using the concept we (briefly) introduced in the lecture under "dynamic programming": use an array $my\_array$ of size $n$ starting with $my\_array[0]=1$ and $my\_array[1]=1$ and for each $i>1$  set $my\_array[i]=my\_array[i-1]+my\_array[i-2]$.

```python
def bad_fibonacci(n):
  """Return the nth Fibonacci number."""
  if n <= 1:
    return n
  else:
    return bad_fibonacci(n-2) + bad_fibonacci(n-1)

def good_fibonacci(n):
  """Return pair of Fibonacci numbers, F(n) and F(n-1)."""
  if n <= 1:
    return (n,0)
  else:
    (a, b) = good_fibonacci(n-1)
    return (a+b, a)
```

## Fibonacci Solutions

1. timing

```python
%%time

bad_fibonacci(20)
```

    CPU times: user 4.48 ms, sys: 442 µs, total: 4.92 ms
    Wall time: 4.54 ms

    6765

```python
%%time

good_fibonacci(20)
```

    CPU times: user 17 µs, sys: 6 µs, total: 23 µs
    Wall time: 29.1 µs

    (6765, 4181)

2. It is not tail recursive becuase there is something to do after returning the values (adding them together). We can use 2 accumulators to avoid this.

```python
def fibo_tail(n, acc2, acc1):
    # Base case
    if n ==1:
        # Return accumulator
        return acc2
    else:
        # Accumulators are updated at each recursive call.
        return fibo_tail(n-1, acc1, acc2+acc1)
```

```python
%%time

fibo_tail(20,1,1)
```

    CPU times: user 13 µs, sys: 1 µs, total: 14 µs
    Wall time: 20 µs

    6765

Example using a helper to hide the accumulator parameters.

```python
def fib_rectail(n):
    def fib_helper(n, acc2, acc1):
        return fib_helper(n-1, acc1, acc2+acc1) if n > 0 else acc1
    return fib_helper(n-1, 0, 1)
```

```python
fib_rectail(20)
```

    6765

In each non-base case we make two recursive calls to the fibonacci function! Computing the $n^{th}$ Fibonacci number in this way requires an exponential number of calls to the function.

Why?

Let $c_n$ denote the number of calls performed in the execution of bad_fibonacci(n). How will the sequence progress for $c_0$ to $c_8$?

$
\begin{align}
c_0 & = 1 \\
c_1 & = 1 \\
c_2 & = 1+c_0+c_1=1+1+1=3 \\
c_3 & = 1+c_1+c_2=1+1+3=5 \\
c_4 & = 1+c_2+c_3=1+3+5=9 \\
c_5 & = 1+c_3 +c_4 = 1+5+9 = 15 \\
c_6 & = 1+c_4 +c_5 = 1+9+15 = 25\\
c_7 & = 1+c_5 +c_6 = 1+15+25 = 41\\
c_8 & = 1+c_6 +c_7 = 1+25+41 = 67
\end{align}
$

Examining this sequence we see by induction that the number of calls more than doubles for each two consecutive indices. Thus, $c_n > 2^{n/2}$, which is $\mathcal{O}(2^n)$ or $\mathcal{O}(c^n)$.

The good_fibonacci(n) is $\mathcal{O}(n)$ because other than the recursive call the function runs in constant time.

```python
# With for loop
def fibonacci_for(n):
    fib = 1
    fib1, fib2 = 1, 1
    if n > 2:
        for i in range(2, n):
            fib = fib1 + fib2
            fib1, fib2 = fib2, fib
    return fib

# With while loop
def fibonacci_while(n):
    fib = 1
    fib1, fib2 = 1, 1
    i = 2
    while i < n:
        fib = fib1 + fib2
        fib1, fib2 = fib2, fib
        i+=1
    return fib
```

We can see here that in both iterative solutions, the loops will run $n-2$ times. Not taking into account constants, we can see that the Big-$\mathcal{O}$ complexity will be $\mathcal{O}(n)$.

```python
def dynamic_fibonacci(n):
    # First two elements of series are placed in index 0 and 1 of our array.
    my_array = [1, 1]
    # We add the elements at the previous two indexes to get the element of the next index.
    for i in range(2, n):
        my_array.append(my_array[i-1] + my_array[i-2])
    return my_array[n-1]
```

```python
%%time
dynamic_fibonacci(20)
```

    CPU times: user 16 µs, sys: 1 µs, total: 17 µs
    Wall time: 23.1 µs

    6765

# Example: An English Imperial Ruler

Graduations on a twelve inch imperial scale ruler. This is an example to show you that there is more to recursion than Fibonacci!

```python
def draw_line(tick_length, tick_label=''):
  """Draw one line with given tick length (followed by optional label)."""
  line = '-' * tick_length
  if tick_label:
    line += ' ' + tick_label
  print(line)

def draw_interval(center_length):
  """Draw tick interval based upon a central tick length."""
  if center_length > 0:                   # stop when length drops to 0
    draw_interval(center_length - 1)      # recursively draw top ticks
    draw_line(center_length)              # draw center tick
    draw_interval(center_length - 1)      # recursively draw bottom ticks

def draw_ruler(num_inches, major_length):
  """Draw English ruler with given number of inches and major tick length."""
  draw_line(major_length, '0')            # draw inch 0 line
  for j in range(1, 1 + num_inches):
    draw_interval(major_length - 1)       # draw interior ticks for inch
    draw_line(major_length, str(j))       # draw inch j line and label

draw_ruler(3,4)
```

    ---- 0
    -
    --
    -
    ---
    -
    --
    -
    ---- 1
    -
    --
    -
    ---
    -
    --
    -
    ---- 2
    -
    --
    -
    ---
    -
    --
    -
    ---- 3

# Example: Binary Search

Binary search is a classic recursive algorithm to find a target value within a sorted sequence. This is among the most important of computer algorithms, and it is the reason that we so often store data in sorted order.

e.g. For the sorted sequence below stored in a Python list with indexes above:

0	1	 2	3	4	 5	6	7	 8	9	10	 11

2	4	 5	6	8	 9	15	16	 17	22	30	 31

If the sequence was unsorted a simple solution is a _sequential search algorithm_: use a loop to examine every element. You either reach the end of the list or find the target.

It is linear complexity, running in $\mathcal{O}(n)$ time as worst case it inspects every element in the sequence.

A sorted sequence allows a much faster approach. Think about how you would accomplish this task by hand: divide in two, and choose the middle digit as a candidate to compare to the target. Everything to the left of the candidate is lower than it and everything to the right is higher. Compare the target to the candidate and discard the left if the candidate is lower and the right if it is higher. Then repeat your _binary search_ algorithm. This is much more efficient, running in $\mathcal{O}($log$n$) time.

Review the iterative and recursive binary search algorithm implementations below.

```python
def binary_search_iterative(data, target):
  """Return True if target is found in the given Python list."""
  low = 0
  high = len(data)-1
  while low <= high:
    mid = (low + high) // 2
    if target == data[mid]:         # found a match
      return True
    elif target < data[mid]:
      high = mid - 1                # only consider values left of mid
    else:
      low = mid + 1                 # only consider values right of mid
  return False                      # loop ended without success
```

```python
def binary_search(data, target, low, high):
  """Return True if target is found in indicated portion of a Python list.

  The search only considers the portion from data[low] to data[high] inclusive.
  """
  if low > high:
    return False                    # interval is empty; no match
  else:
    mid = (low + high) // 2
    if target == data[mid]:         # found a match
      return True
    elif target < data[mid]:
      # recur on the portion left of the middle
      return binary_search(data, target, low, mid - 1)
    else:
      # recur on the portion right of the middle
      return binary_search(data, target, mid + 1, high)
```

When a function makes two recursive calls, we say that it uses binary recursion. Clearly the binary_search above is a binary recursion. Drawing the English ruler and the bad fibonacci function are also examples of binary recursion.

```python
data=[2,4,5,6,8,9,15,16,17,22,30,31]

rec_ans = binary_search(data,5,0,len(data))
iter_ans = binary_search_iterative(data,100)

print(rec_ans,iter_ans)
```

    True False

# Example: Directory Size
File system traversal is a classic example of using recursion to navigate a data structure (the file system). We will look at an example of finding the total file size of the current directory and any sub-directories. Remember that directories can be nested arbitrarily deeply within other directories and you can have 0 or more sub-directories in a directory. Python allows us to use the **os** module to write code that will execute on any OS.  Run it on your machine and compare it to other students running a diffent OS at your table.

```python
import os

def disk_usage(path):
  """Return the number of bytes used by a file/folder and any descendents."""
  total = os.path.getsize(path)                  # account for direct usage
  if os.path.isdir(path):                        # if this is a directory,
    for filename in os.listdir(path):            # then for each child:
      childpath = os.path.join(path, filename)   # compose full path to child
      total += disk_usage(childpath)             # add child's usage to total

  print ('{0:<7}'.format(total), path)           # descriptive output (optional)
  return total

disk_usage('.')
```


```python
from Model_GPR import Model,GPR,gamma,normal,RBF_2
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import animation, cm
import pandas as pd
```

## Generate Data

```python
import numpy as np
np.random.seed(404)
noise = 0.25
x, y = np.arange(-5, 5, 0.3), np.arange(-5, 5, 0.3)
xx, yy = np.meshgrid(x, y)
X_2D = np.column_stack((xx.reshape(-1),yy.reshape(-1)))
X_obs = np.random.uniform(-4, 4, (100, 2))
Y_obs = np.sin(np.linalg.norm(X_obs, axis=1))-np.tanh(0.5*np.linalg.norm(X_obs,axis=1)) + \
             noise * np.random.randn(len(X_obs))
```

```python
fig_1 ,(ax1,ax2) = plt.subplots(1,2,figsize=(10,4))
ax1.scatter(X_obs[:,0],Y_obs, c=Y_obs, cmap=cm.jet)
ax1.set_xlabel('X1')
ax1.set_ylabel('Y')
ax2.scatter(X_obs[:,1],Y_obs,c=Y_obs,cmap=cm.jet)
ax2.set_xlabel('X2');
```

```python
from MLE import MLE,plot_MLE_posterior_dist
from scipy.optimize import minimize
from numpy.linalg import cholesky, det
from scipy.linalg import solve_triangular
```

## Without parameter optimization

```python
fig = plt.figure(figsize=(7,7))
plot_MLE_posterior_dist(xx,yy,X_obs,Y_obs,X_2D,[1.0,1.0,0.3],"Before parameter optimization: l={} sigma={} noise={}".format(1.0,1.0,0.3))
```

## Let's start with an MLE of the parameters

We ne to define an objective function, which is the negative loglikelihood of the data

\begin{equation}
 -l(y|\theta)= \frac{1}{2}(y-\mu)^{T}K_{y}(\theta)^{-1}(y-\mu)+ \frac{m}{2}\log{2\pi}+ \frac{1}{2}\log{|K_{y}(\theta)|}.
\end{equation}

a more stable version following http://www.gaussianprocess.org/gpml/chapters/RW.pdf (algorithm 2.1) is

\begin{align}
    L &= \text{cholesky}(K_y(\theta))\\
    \alpha &= L^{-T}(L^{-1}y)\\
    -l(y|\theta) &=\frac{1}{2}y^{T}\alpha + \sum_{i}^{m}\log{L_{ii}} + \frac{m}{2}\log{2\pi}
\end{align}

which holds for a zero mean Gaussian process.

```python
res_MLE = minimize(MLE(X_obs, Y_obs),[1.0,1.0,0.3],
               bounds=((1e-5,None),(1e-5,None),(1e-5,None)),
               method='L-BFGS-B')
res_MLE
```

          fun: 40.60040642011458
     hess_inv: <3x3 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 0.00406146, -0.00175433, -0.0138165 ])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 72
          nit: 14
       status: 0
      success: True
            x: array([2.18534535, 1.35116544, 0.23744747])

```python
fig = plt.figure(figsize=(7,7))
plot_MLE_posterior_dist(xx,yy,X_obs,Y_obs,X_2D,res_MLE.x,"After parameter optimization (MLE): l={} sigma={} noise={}".format(np.round(res_MLE.x[0],2),np.round(res_MLE.x[1],2),np.round(res_MLE.x[2],2)))
```

## Define Model

We use three gamma priors for the unkown hyperparameters $\ell$, $\sigma$ and noise level $\sigma_{\text{noise}}$

\begin{align}
    \ell \sim \Gamma(2.0,3.0)\\
    \sigma  \sim \Gamma(2.0,3.0)\\
    \sigma\_{\text{noise}}\sim \Gamma(2.0,3.0)
\end{align}

and define the model chronologically. This means we start by defining the prior assumptions for the hyperparameters and noise level. After doing this we choose a covariance function - in this example we use the RBF Kernel to model the covariance between the input points. We can print a summary of the defined model (optional).
The last step is to use the MAP function of the model to find the optimized parameters. The initial guess was arbitrairly set to 1,1,0.3.

```python
test_model = GPR("Test_1")
with test_model as model_2:
    model_2.add_prior("ℓ",gamma(2,3))
    model_2.add_prior("σ",gamma(2,3))
    model_2.add_prior("noise", gamma(2,3))
    model_2.add_cov(RBF_2)
    model_2.summarize()
    res = model_2.MAP(X_obs,Y_obs,theta=[1, 1,0.3])
print(res)
```

    model:Test_1
     self.prior :{'ℓ': <Model_GPR.gamma object at 0x0000027074A4C5F8>, 'σ': <Model_GPR.gamma object at 0x0000027074A4CD30>, 'noise': <Model_GPR.gamma object at 0x0000027074A4CBE0>}
          fun: 45.25451798676848
     hess_inv: <3x3 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 1.69109171e-04,  2.77111667e-05, -4.49063009e-04])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 60
          nit: 11
       status: 0
      success: True
            x: array([2.03257791, 1.17100866, 0.23610388])

## The posterior distribution with optimized parameters

```python
model_2.posterior_dist(X_2D,X_obs,Y_obs)
fig_2 = plt.figure(figsize=(7,7))
model_2.plot_post_dist(xx,yy,X_obs,Y_obs,"After parameter optimization (MAP): l={} sigma={} noise={}".format(np.round(res.x[0],2),np.round(res.x[1],2),np.round(res.x[2],2)))
```

## The variation of the parameters while optimization

```python
model_2.theta_df.plot(subplots=True,layout=(3,1),figsize=(10,10), color='dodgerblue');
```

## Average trainings loss for 10 function evaluations

```python
fig_3,ax = plt.subplots(figsize=(10,7))
pd.DataFrame(model_2.training_loss, columns=['logp']).rolling(10).mean()[10:].plot(color='dodgerblue', ax=ax)
ax.set_yscale('log')
```

## Second approach with pymc3

```python
import pymc3 as pm
import theano
import theano.tensor as tt
```

We define the same model as before. Again we start by defining the prior distributions for the kernel hyperparameter. After doing this we select a covariance function. In contrast to above, we first need to define a Gaussian process with the selected covariance function. Note that the covariance function itself stores the prior assumptions as random variables. Then we define a prior distribution for the noise level and finally construct the loglikelihood of the modell. Finally we can use the find_MAP method to optimize the parameters.

```python
with pm.Model() as model:
    ℓ = pm.Gamma("ℓ", alpha=2.0, beta=3.0)
    σ = pm.Gamma("σ", alpha=2.0,beta=3.0)

    cov = σ ** 2 * pm.gp.cov.ExpQuad(2, ℓ)
    gp = pm.gp.Marginal(cov_func=cov)

    σ_noise = pm.Gamma("σ_noise", alpha=2.0,beta=3.0)
    y_ = gp.marginal_likelihood("y", X=X_obs, y=Y_obs, noise=σ_noise)

    mp = pm.find_MAP(method='BFGS')
```

  <progress value='15' class='' max='15' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [15/15 00:00<00:00 logp = -45.256, ||grad|| = 0.72597]

We can now check if the pymc3 optimized hyperparameter lead to any significant differences to the optimized solution with the help of our model.

```python
np.array([mp['ℓ'], mp['σ'], mp['σ_noise']])-model_2.theta
```

    array([-6.03719906e-06, -4.55456266e-06,  7.65385917e-08])

## MCMC parameter sampling

```python
with model:
    trace = pm.sample(1000, chains=2)
```

    C:\ProgramData\Anaconda3\lib\site-packages\pymc3\sampling.py:468: FutureWarning: In an upcoming release, pm.sample will return an `arviz.InferenceData` object instead of a `MultiTrace` by default. You can pass return_inferencedata=True or return_inferencedata=False to be safe and silence this warning.
      FutureWarning,
    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [σ_noise, σ, ℓ]

  <progress value='4000' class='' max='4000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [4000/4000 00:26<00:00 Sampling 2 chains, 0 divergences]

    Sampling 2 chains for 1_000 tune and 1_000 draw iterations (2_000 + 2_000 draws total) took 41 seconds.

```python
import arviz as az
```

```python
with model:
    az.plot_trace(trace, var_names=["ℓ", "σ", "σ_noise"]);
    display(az.summary(trace, round_to=2))
```

```python

```