```julia
using JuMP, Cbc
```

# MS-E2121 Exercise session 10
### 10.5: Cover separation problem

Consider the knapsack set from the lecture material (Lecture 10):

$$X = \{ x \in \{0,1\}^7: 11x_1 + 6x_2 + 6x_3 + 5x_4 + 5x_5 + 4x_6 + x_7 \le 19\}$$

and a solution $\bar{x} = (0, 2/3, 0, 1, 1, 1, 1)$ to its LP relaxation. Find a cover inequality cutting out the fractional solution $\bar{x}$.

We could list all covers and their corresponding inequalities, but the easier way to find a violated cover inequality is to formulate *the separation problem*.

We know that the cover inequalities are of the form $\sum_{j \in C} x_j \le |C|-1$, which can be rearranged as $\sum_{j \in C} (1-x_j) \ge 1$.

The idea of the separation problem is as follows: find a violated cover inequality such that $C$ is a cover. This leads us to the separation problem (as a combinatorial optimization problem)

$$\begin{align}
\min_{C \subseteq N} &\sum_{j\in C} (1-\bar{x}_j)\\
\text{s.t.} &\sum_{j\in C}a_{j} > b,
\end{align}$$

or the equivalent IP form
$$\begin{align}
\min_{z} &\sum_{j\in N} z_j(1-\bar{x}_j)\\
\text{s.t.} &\sum_{j\in N}z_ja_{j} > b\\
z \ &\text{binary},
\end{align}$$

where $z_j=1$ if $j \in C$. We can see that if the optimal objective value is less than one, the inequality $\sum_{j \in C} (1-\bar{x}_j) \ge 1$ is violated.

```julia
xbar = [0, 2/3, 0, 1, 1, 1, 1]
a = [11,6,6,5,5,4,1]
b = 19
N = 1:7

model = Model(Cbc.Optimizer)

@variable(model, z[N], Bin)
@objective(model, Min, sum(z[j]*(1-xbar[j]) for j in N))
# JuMP doesn't accept strict inequality constraints
# The item weights are integers, making the following equivalent to a'z > b since z is integer
@constraint(model, sum(a[j]*z[j] for j in N) >= b+1)
optimize!(model)
println("Objective value $(objective_value(model)) with z=$(value.(z).data)")
```

    Objective value 0.33333333333333337 with z=[0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0]
    Welcome to the CBC MILP Solver
    Version: 2.10.5
    Build Date: Jan  1 1970

    command line - Cbc_C_Interface -solve -quit (default strategy 1)
    Continuous objective value is 0.277778 - 0.00 seconds
    Cgl0004I processed model has 1 rows, 3 columns (3 integer (3 of which binary)) and 3 elements
    Cutoff increment increased from 1e-05 to 0.333233
    Cbc0038I Initial state - 0 integers unsatisfied sum - 0
    Cbc0038I Solution found of 0.333333
    Cbc0038I Before mini branch and bound, 3 integers at bound fixed and 0 continuous
    Cbc0038I Mini branch and bound did not improve solution (0.00 seconds)
    Cbc0038I After 0.00 seconds - Feasibility pump exiting with objective of 0.333333 - took 0.00 seconds
    Cbc0012I Integer solution of 0.33333333 found by feasibility pump after 0 iterations and 0 nodes (0.00 seconds)
    Cbc0001I Search completed - best objective 0.3333333333333334, took 0 iterations and 0 nodes (0.00 seconds)
    Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
    Cuts at root node changed objective from 0.333333 to 0.333333
    Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)

    Result - Optimal solution found

    Objective value:                0.33333333
    Enumerated nodes:               0
    Total iterations:               0
    Time (CPU seconds):             0.00
    Time (Wallclock seconds):       0.00

    Total time (CPU seconds):       0.00   (Wallclock seconds):       0.00

The solution tells us that the most violated cover inequality is that corresponding to $C = \{2,4,5,6,7\}$.

$$ \sum_{j \in \{2,4,5,6,7\}} \bar{x}_j \le 4, $$

which becomes $ 4.667 \le 4$. The total weight of the item selection $C$ is 21, which exceeds the knapsack capacity of 19, making $C$ a cover.

### 10.4: Solving Problem 9.3 with Gomory cuts

Consider the following integer programming problem $IP$:

$$\begin{matrix}
\text{max} &x_{1} &+&2x_{2} & \\
\text{s.t.}&-3x_{1} &+&4x_{2} &\le 4 \\
&3x_{1} &+&2x_{2} &\le 11 \\
&2x_{1} &-&x_{2} &\le 5 \\
&x_{1}, &x_{2} & \text{integer} &\\
\end{matrix}$$

Solve the problem by adding Gomory cuts to the LP relaxation until you find an integer solution. Below you'll find the initial LP relaxation and its solution.

```julia

model = Model(Cbc.Optimizer)
@variable(model, x[1:2]>=0)
@objective(model, Max, x[1]+x[2])
@constraint(model, -3*x[1]+4*x[2] <= 4)
@constraint(model, 3*x[1]+2*x[2] <= 11)
@constraint(model, 2*x[1]-x[2] <= 5)
optimize!(model)

```

    Presolve 3 (0) rows, 2 (0) columns and 6 (0) elements
    0  Obj -0 Dual inf 1.9999998 (2)
    2  Obj 4.5
    Optimal - objective value 4.5
    Optimal objective 4.5 - 2 iterations time 0.002

```julia

println("Optimal value $(objective_value(model))")
println("with x = $(value.(model[:x]))")

```

    Optimal value 4.5
    with x = [2.0, 2.5]

```julia
#%
# This function takes a vector and returns a vector where all elements that are sufficiently close to the nearest integer are rounded to that integer value
function round_ints(nums, ϵ=1E-9)
    # abs(round(x)-x) gives the absolute difference between the number and the nearest integer
    # If that difference is greater than epsilon, the value is not integer
    # The Julia syntax f ? a : b is a one-line equivalent to
    #     if f
    #         a
    #     else
    #         b
    #     end
    return [(abs(round(x)-x) > ϵ ? x : round(x)) for x in nums]
end

# This function takes the constraints in standard form and a basis index vector B_ind
# It prints the Gomory cuts corresponding to each fractional variable in the basic solution
function get_Gomory_cut(A, b, B_ind)
    B = A[:,B_ind] # Basis matrix B
    invB = inv(B)  # Inverse of B
    x_B = invB*b   # Basic solution corresponding to B

    # Get all rows with a fractional solution x
    ϵ = 1E-9
    i_frac = findall(x -> abs(round(x)-x) > ϵ, x_B)
    for i in i_frac

        # Get the row i of invB*A and the i-th component of the basic solution (which is always fractional in this loop)
        # Also round the row coefficients to the nearest integer if their values are practically integer, this helps with some numerical issues
        a_i = round_ints((invB*A)[i,:], ϵ)
        a_i0 = x_B[i]

        # Get the coefficients of the Gomory cut
        f_i = a_i.-floor.(a_i)
        f_i0 = a_i0-floor(a_i0)

        # The Gomory cut as presented in the lectures
        println("Gomory cut corresponding to fractional x$(B_ind[i]): $(f_i)x >= $(f_i0)")

        # Form and print the Gomory cut with original variables
        β = round_ints(invB[i,:]', ϵ) # The i-th row of invB
        q = β.-floor.(β)
        qA = round_ints(q*A, ϵ)
        qb = round_ints(q*b, ϵ)
        println("With original variables: $(floor.(qA))x <= $(floor.(qb)[1])")
    end
end
#%
```

    get_Gomory_cut (generic function with 1 method)

```julia
#%
A = [-3 4 1 0 0; 3 2 0 1 0; 2 -1 0 0 1]
b = [4; 11; 5]
B_ind = [1; 2; 5]
get_Gomory_cut(A, b, B_ind)
#%
```

    Gomory cut corresponding to fractional x2: [0.0, 0.0, 0.16666666666666666, 0.16666666666666666, 0.0]x >= 0.5
    With original variables: [0.0 1.0 0.0 0.0 0.0]x <= 2.0
    Gomory cut corresponding to fractional x5: [0.0, 0.0, 0.3888888888888889, 0.7222222222222223, 0.0]x >= 0.5000000000000009
    With original variables: [1.0 3.0 0.0 0.0 0.0]x <= 9.0

```julia
#%
model = Model(Cbc.Optimizer)
@variable(model, x[1:2]>=0)
@objective(model, Max, x[1]+x[2])
@constraint(model, -3*x[1]+4*x[2] <= 4)
@constraint(model, 3*x[1]+2*x[2] <= 11)
@constraint(model, 2*x[1]-x[2] <= 5)
@constraint(model, x[2] <= 2)
@constraint(model, x[1]+3*x[2] <= 9)
optimize!(model)
#%
```

    Presolve 4 (-1) rows, 2 (0) columns and 8 (-1) elements
    0  Obj -0 Dual inf 1.999998 (2)
    1  Obj 4.3333333
    Optimal - objective value 4.3333333
    After Postsolve, objective 4.3333333, infeasibilities - dual 0 (0), primal 0 (0)
    Optimal objective 4.333333333 - 1 iterations time 0.002, Presolve 0.00

```julia
#%
println("Optimal value $(objective_value(model))")
println("with x = $(value.(model[:x]))")
#%
```

    Optimal value 4.333333333333333
    with x = [2.333333333333333, 2.0]

```julia
#%
A = [-3 4 1 0 0 0 0; 3 2 0 1 0 0 0; 2 -1 0 0 1 0 0; 0 1 0 0 0 1 0; 1 3 0 0 0 0 1]
b = [4; 11; 5; 2; 9]
B_ind = [1; 2; 3; 5; 7]
get_Gomory_cut(A, b, B_ind)
#%
```

    Gomory cut corresponding to fractional x1: [0.0, 0.0, 0.0, 0.3333333333333333, 0.0, 0.33333333333333337, 0.0]x >= 0.33333333333333304
    With original variables: [1.0 1.0 0.0 0.0 0.0 0.0 0.0]x <= 4.0
    Gomory cut corresponding to fractional x5: [0.0, 0.0, 0.0, 0.33333333333333326, 0.0, 0.3333333333333339, 0.0]x >= 0.3333333333333339
    With original variables: [1.0 1.0 0.0 0.0 0.0 0.0 0.0]x <= 4.0
    Gomory cut corresponding to fractional x7: [0.0, 0.0, 0.0, 0.6666666666666667, 0.0, 0.666666666666667, 0.0]x >= 0.6666666666666679
    With original variables: [2.0 2.0 0.0 0.0 0.0 0.0 0.0]x <= 8.0

```julia
#%
model = Model(Cbc.Optimizer)
@variable(model, x[1:2]>=0)
@objective(model, Max, x[1]+x[2])
@constraint(model, -3*x[1]+4*x[2] <= 4)
@constraint(model, 3*x[1]+2*x[2] <= 11)
@constraint(model, 2*x[1]-x[2] <= 5)
@constraint(model, x[2] <= 2)
@constraint(model, x[1]+3*x[2] <= 9)
@constraint(model, x[1]+x[2] <= 4)
@constraint(model, 2*x[1]+2*x[2] <= 8)
optimize!(model)
#%
```

    Presolve 6 (-2) rows, 2 (0) columns and 12 (-2) elements
    0  Obj -0 Dual inf 1.999998 (2)
    1  Obj 4
    Optimal - objective value 4
    After Postsolve, objective 4, infeasibilities - dual 0 (0), primal 0 (0)
    Optimal objective 4 - 1 iterations time 0.002, Presolve 0.00

```julia
#%
println("Optimal value $(objective_value(model))")
println("with x = $(value.(model[:x]))")
#%
```

    Optimal value 4.0
    with x = [2.0, 2.0]

```julia
#%
# Polyhedra is a convenient package for creating and plotting polyhedra
# (surely for other things as well, that just happens to be the only thing I've used it for)
using Polyhedra, Plots

# This function is just for this exercise here, could surely be made more general, maybe for next year's course...
function plot_sol_9_3(fs)
    # Add nonnegativity constraints and convert the intersection of halfspaces into a polyhedron p
    fs_nonnegative = fs ∩ HalfSpace([-1, 0], 0) ∩ HalfSpace([0, -1], 0)
    p = polyhedron(fs_nonnegative)

    # Plot the polyhedron with some opacity so that we can see the coordinate grid
    plot(p, opacity = 0.5, ratio=:equal, label="feasible region", legend = true)

    # Plot the objective function at four conveniently chosen values
    plot!([0, 4], [2.5, 0.5], linestyle=:dash, label="x1+2x2=5")
    plot!([0, 4], [3, 1], linestyle=:dash, label="x1+2x2=6")
    plot!([0, 4], [3.167, 1.167], linestyle=:dash, label="x1+2x2=6.33")
    plot!([0, 4], [3.5, 1.5], linestyle=:dash, label="x1+2x2=7")

    # Check which integer points of the original formulation remain feasible in this formulation
    integer_points = [0 0; 1 0; 2 0; 0 1; 1 1; 2 1; 3 1; 2 2] # These are the points that were originally feasible
    feasible_rows = [] # This will eventually include the row indices of feasible points in the above matrix
    for i in 1:size(integer_points)[1]
        if integer_points[i,:] in fs_nonnegative
            push!(feasible_rows, i)
        end
    end

    # Plot the feasible points and tweak the axes
    scatter!(integer_points[feasible_rows,1], integer_points[feasible_rows,2], label="Feasible integer points", xlim = (-0.2,4.5), ylim = (-0.2,3.5))
end;
#%
```

```julia
#%
# Plotting the feasible region using the function we wrote for this
fs_orig = HalfSpace([-3, 4], 4) ∩ HalfSpace([3, 2], 11) ∩ HalfSpace([2, -1], 5)
plot_sol_9_3(fs_orig)
#%
```

```julia
#%
fs_1 = fs_orig ∩ HalfSpace([1, 3], 9) ∩ HalfSpace([0, 1], 2)
plot_sol_9_3(fs_1)
#%
```

```julia
#%
fs_2 = fs_1 ∩ HalfSpace([1, 1], 4) ∩ HalfSpace([2, 2], 8)
plot_sol_9_3(fs_2)
#%
```

```julia

```


# Custom Embedded Runge-Kutta Method

```python
!pip install fastrk matplotlib
```

    Requirement already satisfied: fastrk in c:\users\stasb\pycharmprojects\fastrk (0.0.5)
    Requirement already satisfied: matplotlib in c:\users\stasb\appdata\roaming\python\python38\site-packages (3.5.0)
    Requirement already satisfied: numba>=0.51.2 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from fastrk) (0.54.1)
    Requirement already satisfied: numpy>=1.19.2 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from fastrk) (1.20.3)
    Requirement already satisfied: sympy>=1.8 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from fastrk) (1.9)
    Requirement already satisfied: scipy>=1.2 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from fastrk) (1.7.2)
    Requirement already satisfied: packaging>=20.0 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from matplotlib) (21.3)
    Requirement already satisfied: fonttools>=4.22.0 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from matplotlib) (4.28.1)
    Requirement already satisfied: python-dateutil>=2.7 in c:\users\stasb\appdata\roaming\python\python38\site-packages\python_dateutil-2.8.1-py3.8.egg (from matplotlib) (2.8.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\stasb\appdata\roaming\python\python38\site-packages\kiwisolver-1.3.1-py3.8-win-amd64.egg (from matplotlib) (1.3.1)
    Requirement already satisfied: pyparsing>=2.2.1 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from matplotlib) (3.0.6)
    Requirement already satisfied: cycler>=0.10 in c:\users\stasb\appdata\roaming\python\python38\site-packages\cycler-0.10.0-py3.8.egg (from matplotlib) (0.10.0)
    Requirement already satisfied: pillow>=6.2.0 in c:\users\stasb\appdata\roaming\python\python38\site-packages\pillow-8.0.1-py3.8-win-amd64.egg (from matplotlib) (8.0.1)
    Requirement already satisfied: setuptools-scm>=4 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from matplotlib) (6.3.2)
    Requirement already satisfied: six in c:\users\stasb\appdata\roaming\python\python38\site-packages\six-1.15.0-py3.8.egg (from cycler>=0.10->matplotlib) (1.15.0)
    Requirement already satisfied: llvmlite<0.38,>=0.37.0rc1 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from numba>=0.51.2->fastrk) (0.37.0)
    Requirement already satisfied: setuptools in c:\users\stasb\anaconda3\envs\p38\lib\site-packages (from numba>=0.51.2->fastrk) (58.0.4)
    Requirement already satisfied: tomli>=1.0.0 in c:\users\stasb\appdata\roaming\python\python38\site-packages (from setuptools-scm>=4->matplotlib) (1.2.2)
    Requirement already satisfied: mpmath>=0.19 in c:\users\stasb\appdata\roaming\python\python38\site-packages\mpmath-1.1.0-py3.8.egg (from sympy>=1.8->fastrk) (1.1.0)

```python
import math
import numpy as np
import matplotlib.pyplot as plt
from cycler import cycler
from numba import jit, cfunc
from fastrk import (ButcherTable, RKCodeGen,
                    EventsCodeGen, default_jitkwargs)
import bt_456_custom

```

### Create python file with Butcher Table elements
- matrix A
- vectors b_main, b_subs, c
- and tuple of orders

```python
!type bt_456_custom.py
```

    '''Butcher Table for RK4(5)6 from Prince and Dormand.
    Original matlab code at:
    https://github.com/USNavalResearchLaboratory/TrackerComponentLibrary/blob/master/Mathematical_Functions/Differential_Equations/RungeKStep.m
    '''

    A = [
         [0, 0, 0, 0, 0, 0],
         [1 / 4, 0, 0, 0, 0, 0],
         [3 / 32, 9 / 32, 0, 0, 0, 0],
         [1932 / 2197, -7200 / 2197, 7296 / 2197, 0, 0, 0],
         [439 / 216, -8, 3680 / 513, -845 / 4104, 0, 0],
         [-8 / 27, 2, -3544 / 2565, 1859 / 4104, -11 / 40, 0]
        ]

    c = [0, 1 / 4, 3 / 8, 12 / 13, 1, 1 / 2]

    b_main = [25 / 216, 0,  1408 / 2565,   2197 / 4104,  -1 / 5,      0]

    b_subs = [16 / 135, 0, 6656 / 12825, 28561 / 56430, -9 / 50, 2 / 55]

    order = (4, 5)

```python
BT456 = ButcherTable("456", bt_456_custom)

# rk_456.py will be created and imported here
rk_module = RKCodeGen(BT456, autonomous=False).save_and_import()
rk_prop = rk_module.rk_prop
```

```python
# using of cfunc decorator instead of jit decorator allows you to compile separately
# ODE right part and rk_prop with no recompilation required on next run
@cfunc('f8[:](f8, f8[:])')
def eq(t, s):
    '''
    Mathematical pendulum equation set.

    Parameters
    ----------

    t: scalar
        time

    s: np.array
        state vector [theta, omega]

    Return
    ------

    ds: np.array
        ds/dt - time derivative of state vector
    '''
    theta, omega = s
    return np.array([omega, -math.sin(theta)])
```

```python
N = 100
ws = np.linspace(0, 2, N+2)[1:-1]

data = []
for i in range(N):
    s0 = np.array([0., ws[i]])
    trj = rk_prop(eq, s0, 0, 2*np.pi, np.inf, 1e-12, 1e-12)
    data.append(trj)
```

```python
colors = plt.cm.twilight(np.linspace(0, 1, N))
plt.rc('axes', prop_cycle=cycler(color=colors))

plt.figure(dpi=200)
for i in range(N):
    plt.plot(data[i][:, 1], data[i][:, 2])
plt.xlabel(r'$\theta$')
plt.ylabel(r'$\omega$')
plt.title('phase portrait near center fixed point')
#plt.show()
```

## Calculate oscillation periods

```python
def event_sin_theta(t, s):
    return math.sin(s[0])

events = [event_sin_theta]

values = np.array([0.])
terminals = np.array([True])
directions = np.array([1])
counts = np.array([-1])
accurates = np.array([True])

call_event = EventsCodeGen(events).save_and_import()
rk_prop_ev = rk_module.rk_prop_ev
```

```python
trjs = []
evs = []
for i in range(N):
    s0 = np.array([0., ws[i]])
    trj, evarr = rk_prop_ev(eq, s0, 0, 1000, np.inf, 1e-12, 1e-12,
                            values, terminals, directions, counts, accurates,
                            call_event, 1e-12, 1e-12, 100)
    trjs.append(trj)
    evs.append(evarr)
```

```python
plt.figure(dpi=200)
for i in range(N):
    plt.plot(trjs[i][:, 1], trjs[i][:, 2])
plt.xlabel(r'$\theta$')
plt.ylabel(r'$\omega$')
plt.title('phase portrait near center fixed point')
```

```python
periods = np.array(evs)[:, 0, 2]
plt.plot(ws, periods, '-k')
plt.xlabel(r'$\omega_0$')
plt.ylabel(r'period')
plt.title('mathematical pendulum oscillation periods\nfor $\omega_0 \in (0, 2)$');
```

```python

```


```python
import numpy as np
from scipy.integrate import odeint
%matplotlib inline
import matplotlib.pyplot as plt
```

## Polymer Cross-linkg reaction

 HPAM react with chromium ion as below:
 \begin{equation}
 a[Cr]+b[Py]\to c[Cr-Py]
 \end{equation}

Which is an irreversible reaction and the reaction rates are given by:

\begin{equation}
   {{R}_{Cr}}=-{{k}_{f}}\cdot {{{c}_{Cr}}^{1.22}}\cdot {{{c}_{Py}}^{0.53}}\cdot {{{c}_{H}}^{-0.9}}
\end{equation}
\begin{equation}
  {{R}_{Py}}=-{{k}_{f}}\cdot {{{c}_{Cr}}^{1.22}}\cdot {{{c}_{Py}}^{0.53}}\cdot {{{c}_{H}}^{-0.9}}
\end{equation}
\begin{equation}
  {{R}_{Cr-Py}}={{k}_{f}}\cdot {{{c}_{Cr}}^{1.22}}\cdot {{{c}_{Py}}^{0.53}}\cdot {{{c}_{H}}^{-0.9}}
\end{equation}

Where Hydrogen ion (H+) concentration is calculated from measured PH value using (G. Paul, 2006):
 \begin{equation}
{{c}_{H}}={{10}^{-pH}}
\end{equation}

### Input Parameters
Temperature 20 ℃:${{k}_{f}}=1.13\times {{10}^{-3}}\text{  }{{h}^{-1}}{{(mol/L)}^{0.15}}$

Temperature 35 ℃:${{k}_{f}}=3.16\times {{10}^{-3}}\text{  }{{h}^{-1}}{{(mol/L)}^{0.15}}$

Constant pH=4.9

Initial Concentration: ${{c}_{Cr}}=1\text{ , }{{c}_{Py}}=60$

```python
kf_20=1.13e-3
kf_35=3.16e-3
pH=4.9
cCr_0=1
cPy_0=3
cCrPy_0=0
```

```python
def Reaction_rate(C,t,k,pH):
    cCr=C[0]
    cPy=C[1]
    cCrPy=C[2]
    cH=10**(-pH)

    dCr_dt=-k*(cCr**1.22)*(cPy**0.53)*(cH**-0.9)
    dPy_dt=-k*(cCr**1.22)*(cPy**0.53)*(cH**-0.9)
    dCrPy_dt=k*(cCr**1.22)*(cPy**0.53)*(cH**-0.9)
    return [dCr_dt,dPy_dt,dCrPy_dt]
```

```python
t=np.linspace(0,0.5,100)
C0=[cCr_0,cPy_0,cCrPy_0]
C=odeint(Reaction_rate,C0,t,args=(kf_20, pH))

plt.plot(t,C[:,0],'r-',linewidth=2)
plt.plot(t,C[:,1],'b-',linewidth=2)
plt.plot(t,C[:,2],'g-',linewidth=2)

plt.title('Cr+Py->Cr-Py\nk_f=%s  pH=%s'%(kf_20,pH))
plt.xlabel('Time (hours)',fontsize=12)
plt.ylabel('Concentration (mol/L)',fontsize=12)
plt.legend(['Cr','Polymer','Cross-linked CrPy'],loc='best')
plt.tick_params(axis='both', which='major', labelsize=12)
plt.savefig('Cr-Po.png', dpi=300)
```

```python

```


```python
import numpy as np
import scipy as sp
import scipy.stats
import matplotlib.pyplot as plt
import math as mt
import scipy.special
import seaborn as sns
plt.style.use('fivethirtyeight')
from statsmodels.graphics.tsaplots import plot_acf
import pandas as pd
```

Define some color in this cell.
$$
\require{color}
\definecolor{red}{RGB}{240,5,5}
\definecolor{blue}{RGB}{5,5,240}
\definecolor{green}{RGB}{4,240,5}
\definecolor{black}{RGB}{0,0,0}
\definecolor{dsb}{RGB}{72, 61, 139}
\definecolor{Maroon}{RGB}{128,0,0}
$$

#  Linear Regression Model

##  Normal-Gamma Conjugacy

The common matrix form of linear regression is
$$
y =    X \beta+u
$$
where $u \sim N( {0}, \sigma^2 I)$.

The covariance matrix of disturbance term $u$ is
$$
\begin{aligned}
&\operatorname{var}(u) \equiv\left[\begin{array}{cccc}
\operatorname{var}\left(u_{1}\right) & \operatorname{cov}\left(u_{1}, u_{2}\right) & \ldots & \operatorname{cov}\left(u_{1}, u_{N}\right) \\
\operatorname{cov}\left(u_{1}, u_{2}\right) & \operatorname{var}\left(u_{2}\right) & \ldots & . \\
\cdot & \operatorname{cov}\left(u_{2}, u_{3}\right) & \ldots & . \\
\cdot & \cdot & \ldots & \operatorname{cov}\left(u_{N-1}, u_{N}\right) \\
\operatorname{cov}\left(u_{1}, u_{N}\right) & . & \ldots & \operatorname{var}\left(u_{N}\right)
\end{array}\right]=\left[\begin{array}{cccc}
h^{-1} & 0 & \ldots & 0 \\
0 & h^{-1} & \ldots & . \\
. & . & \ldots & . \\
. & . & \cdots & 0 \\
0 & . & . & h^{-1}
\end{array}\right]
\end{aligned}
$$

For easier mathematical manipulation, we ususally use $h = 1/\sigma^2$, which is the **precision**. The diagonal form of covariance matrix reprensents two assumptions: **no serial correlation** and **homoscedasticity**.

Also with the assumption that $X$ is exogenous, we can construct the joint probability density
$$
P(y, X \mid \beta, h)=P(y \mid X, \beta, h) P(X)
$$

However because $X$ does not depend $\beta$ and $h$, we narrow our interest onto
$$
P(y \mid X, \beta, h)
$$

Recall that multivariabe normal distribution takes the form

$$
f(X)=(2 \pi)^{-N / 2}|\Sigma|^{-1 / 2} \exp \left(-\frac{1}{2}(X-\mu)^T \Sigma^{-1}(X-\mu)\right) \text { for } \sigma>0
$$

where $|\Sigma|$ is the determinant of covariance matrix, $\Sigma^{-1}$ is the inverse of covariance matrix.

In the linear regression context, the determinat of covariance matrix is
$$
|\text{var}(u)|^{-1/2}=\left(\prod_{i=1}^Nh^{-1}\right)^{-1/2} = (h^{-N})^{-1/2}=h^{N/2}
$$

The inverse matrix of covariance matrix
$$
\text{var}(u)^{-1} = (h^{-1}I)^{-1}=hI
$$

##  Likelihood Function

With all previous preparation, the the likelihood function $P(y \mid X, \beta, h)$ simplified to
$$
P(y \mid  X, \beta, h)=(2 \pi)^{-\frac{N}{2}} h^{\frac{N}{2}} \exp \left[-\frac{h}{2}(y-\mathrm{X} \beta)^T(y-\mathrm{X} \beta)\right]
$$
However more mathematical manipulation needed in order to turn it into the conjugate form.

Use the fact that $y-X\hat{\beta}=\hat{u}$, we rewrite

$$
(y-\mathrm{X} \beta)^T(y-\mathrm{X} \beta) = \left(\hat{u}-\mathrm{X}\left(\beta-\widehat{\beta}\right)\right)^T\left(\hat{u}-\mathrm{X}\left(\beta-\widehat{\beta}\right)\right)
$$
Expand it

$$
(y-\mathrm{X} \beta)^T(y-\mathrm{X} \beta) =\hat{u}^T \hat{u}-\hat{u}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)-\left(\hat{u}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)\right)^T+\left(\beta-\widehat{\beta}\right)^T \mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)
$$

Use the fact $\hat{u}^TX = 0$, expression reduces to
$$
(y-\mathrm{X} \beta)^T(y-\mathrm{X} \beta)=\hat{u}^T \hat{u}+\left(\beta-\widehat{\beta}\right)^T \mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)
$$

Plug in back to likelihood function
$$
P(y \mid  X, \beta, h)=(2 \pi)^{-\frac{N}{2}} h^{\frac{N}{2}} \exp \left[-\frac{h}{2}\left(\hat{u}^T \hat{u}+\left(\beta-\widehat{\beta}\right)^T \mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)\right)\right]
$$

Seperate the exponential part
$$
p(y \mid \mathrm{X}, \beta, h)=(2 \pi)^{-\frac{N}{2}} h^{\frac{N}{2}} \exp \left[-\frac{h}{2}\left(\beta-\widehat{\beta}\right)^T\mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)\right] \exp \left[-\frac{h}{2}\hat{u}^T \hat{u}\right]
$$

This form of likelihood function suggest a **natural conjugate prior** which has the same function form as likelihood and also yields a posterior within the same class of distribution.

##  Prior

Our prior should be a joint distribution $P(\beta, h)$, however in order to transform into NG distribution, it proves convenient to write
$$
P(\beta, h)=P(\beta \mid h) P(h)
$$

And it is the **Normal-Gamma distribution** we are talking about, the right-hand side has the follow distribution
$$
\begin{gathered}
\beta \mid h \sim N\left(\mu, h^{-1} V\right) \\
h \sim \operatorname{Gamma}(m, v)
\end{gathered}
$$
The advantage of this distribution is that $\beta$ is on real number's domain and $h$ is on postive number's domain.

Recall the function form of NG distribution.
$$
f(X, h)=(2 \pi)^{-\frac{N}{2}}(h)^{-\frac{N}{2}}|\Sigma|^{-1 / 2} \exp \left(-\frac{h}{2}(X-\mu)^T \Sigma^{-1}(X-\mu)\right) \frac{1}{\left(\frac{2 m}{v}\right)^{v / 2}} \frac{1}{\Gamma\left(\frac{v}{2}\right)} h^{\frac{v-2}{2}} \exp \left[-\frac{h v}{2 m}\right]
$$

Seperate it for $\beta|h$ and $h$ for their priors
$$
\begin{gathered}
P(\beta \mid h)=(2 \pi)^{-\frac{k}{2}} h^{\frac{k}{2}}|V|^{-\frac{1}{2}} \exp \left[-\frac{h}{2}(\beta-\mu)^{\prime} V^{-1}(\beta-\mu)\right] \\
P(h)=\frac{1}{\left(\frac{2 m}{v}\right)^{v / 2}} \frac{1}{\Gamma\left(\frac{v}{2}\right)} h^{\frac{v-2}{2}} \exp \left[-\frac{h v}{2 m}\right]
\end{gathered}
$$
where $k$ is the number of parameters in $\beta$.

##  Posterior

The posterior is formulated by Bayes' Theorem

$$
P(\beta, h \mid y, X)\propto P(y \mid  X, \beta, h)P(\beta\mid h) P(h)
$$

$$
P(\beta, h \mid Y, \mathrm{X}) \propto \ h^{\frac{k}{2}} \exp \left[-\frac{h}{2}\left[\left(\beta-\widehat{\beta}\right)^T\mathrm{X}^{\prime} \mathrm{X}\left(\beta-\widehat{\beta}\right)+(\beta-\mu)^T V^{-1}(\beta-\mu)\right]\right]  h^{\frac{N+v-2}{2}} \exp \left[-\frac{h}{2}\left(\hat{u}^T \hat{u}+\frac{v}{m}\right)\right]
$$

$$
\begin{align}
P(\beta, h \mid Y, \mathrm{X}) &\propto(2 \pi)^{-\frac{N}{2}} h^{\frac{N}{2}} \exp \left[-\frac{h}{2}\left(\beta-\widehat{\beta}\right)^T\mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)\right] \exp \left[-\frac{h}{2}\hat{u}^T \hat{u}\right]\\
&\times(2 \pi)^{-\frac{k}{2}} h^{\frac{k}{2}}|V|^{-\frac{1}{2}}\exp \left[-\frac{h}{2}(\beta-\mu)^{\prime} V^{-1}(\beta-\mu)\right]\frac{1}{\left(\frac{2 m}{v}\right)^{v / 2}} \frac{1}{\Gamma\left(\frac{v}{2}\right)} h^{\frac{v-2}{2}} \exp \left[-\frac{h v}{2 m}\right]
\end{align}
$$

First ignore the constant parts such as $2\pi$ and gamma function, then combine $h$

$$
\begin{align}
P(\beta, h \mid Y, \mathrm{X}) &\propto h^{\frac{k}{2}} \exp \left[-\frac{h}{2}\left(\beta-\widehat{\beta}\right)^T\mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)\right] \exp \left[-\frac{h}{2}\hat{u}^T \hat{u}\right]\\
&\times \exp \left[-\frac{h}{2}(\beta-\mu)^{\prime} V^{-1}(\beta-\mu)\right] h^{\frac{N+v-2}{2}} \exp \left[-\frac{h v}{2 m}\right]
\end{align}
$$

Join exponential terms
$$
\begin{align}
P(\beta, h \mid Y, \mathrm{X}) &\propto h^{\frac{k}{2}} \exp \left[-\frac{h}{2}\left(\beta-\widehat{\beta}\right)^T\mathrm{X}^T \mathrm{X}\left(\beta-\widehat{\beta}\right)-\frac{h}{2}(\beta-\mu)^{\prime} V^{-1}(\beta-\mu)\right] h^{\frac{N+v-2}{2}} \exp \left[-\frac{h}{2}\hat{u}^T \hat{u}-\frac{h v}{2 m}\right]\\
&\propto\ h^{\frac{k}{2}} \exp \left[-\frac{h}{2}\left[\left(\beta-\widehat{\beta}\right)^T\mathrm{X}^{\prime} \mathrm{X}\left(\beta-\widehat{\beta}\right)+(\beta-\mu)^T V^{-1}(\beta-\mu)\right]\right]  h^{\frac{N+v-2}{2}} \exp \left[-\frac{h}{2}\left(\hat{u}^T \hat{u}+\frac{v}{m}\right)\right]
\end{align}
$$

```python

```

##  Normal-Inverse Gamma Conjugacy

Recall multivariate normal distribution density function

$$
f(X)=(2 \pi)^{-\frac{N}{2}}|\Sigma|^{-\frac{1}{2}} \exp \left(-\frac{1}{2}(X-\mu)^T \Sigma^{-1}(X-\mu)\right) \text { for } \sigma>0
$$

We assume likelihood ${y} \sim {N}\left({ {X}\beta}, \sigma^{2} {I}\right)$ is exactly same, just switched the denotions

$$
p\left({y} \mid {X}, {\beta}, \sigma^{2}\right)=(2 \pi)^{-\frac{N}{2}}\left|\sigma^{2} {I}\right|^{-\frac{1}{2}} \exp \left(-\frac{1}{2}\left({y}-{ {X}\beta}\right)^{T}\left(\sigma^{2} {I}\right)^{-1}\left({y}-{ {X}\beta}\right)\right)
$$

Use determinants rules
$$
\left|\sigma^{2} {I}\right|=\sigma^{2N}\\
\left(\sigma^{2} {I}\right)^{-1}=\sigma^{-2}I
$$

    Likelihood simplifed to
$$
p\left({y} \mid {X}, {\beta}, \sigma^{2}\right)=\left(2 \pi \sigma^{2}\right)^{-N / 2} \exp \left(-\frac{1}{2 \sigma^{2}}\left({y}- {X}{\beta}\right)^{T}\left( {y}- {X}{\beta}\right)\right)
$$

Prior is usually decomposed $p\left({\beta}, \sigma^{2}\right)=p\left({\beta} \mid \sigma^{2}\right) p\left(\sigma^{2}\right)$ where ${\beta} \mid \sigma^{2} \sim {N}\left({0}, \sigma^{2} {\Lambda}^{-1}\right)$ and $\sigma^{2} \sim \operatorname{InvGamma}\left(\alpha, \beta\right)$

$$
p\left( {\beta} \mid \sigma^{2}\right)=(2 \pi)^{-\frac{k}{2}}\left|\sigma^{2} {\Lambda}^{-1}\right|^{-\frac{1}{2}} \exp \left(-\frac{1}{2}\left({\beta}-{\mu}\right)^T\left(\sigma^2 {\Lambda}^{-1}\right)^{-1}\left({\beta}-{\mu}\right)\right)
$$

Use determinants rules
$$
|\sigma^2\Lambda^{-1}|^{-\frac{1}{2}}=(\sigma^{2k}|\Lambda^{-1}|)^{-\frac{1}{2}}=(\sigma^{2k}|\Lambda|^{-1})^{-\frac{1}{2}}=\sigma^{-k}|\Lambda|^{\frac{1}{2}}\\
$$
then
$$
(2 \pi)^{-\frac{k}{2}}\left|\sigma^{2} {\Lambda}^{-1}\right|^{-\frac{1}{2}}=(2\pi\sigma^2)^{-\frac{k}{2}}|\Lambda|^\frac{1}{2}
$$

Prior $P\left( {\beta} \mid \sigma^{2}\right)$ simplified to
$$
p\left( {\beta} \mid \sigma^{2}\right)=\left(2 \pi \sigma^{2}\right)^{-\frac{k}{2}}\left|{\Lambda}\right|^{\frac{1}{2}} \exp \left(-\frac{1}{2 \sigma^{2}}\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)\right)
$$

Recall the inverse gamma distribution is

$$
f(x ; \alpha, \beta)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}(1 / x)^{\alpha+1} \exp (-\beta / x)
$$

follow the function form, prior $P(\sigma^2)$ is
$$
P\left(\sigma^{2}\right)=\frac{\beta^{\alpha}}{\Gamma\left(\alpha\right)}\left(\sigma^{2}\right)^{-\left(\alpha+1\right)} \exp \left(-\beta / \sigma^{2}\right)
$$

##  Kernel Decomposition

To move forward, there will be horrible amount of linear algebraic manipulation, the first is kernel decomposition

$$
\begin{aligned}
( {y}- {X}  {\beta})^{T}( {y}- {X}  {\beta})=&(\overbrace{ {y}- {X} \hat{ {\beta}}}^{A}+\overbrace{{X} \hat{{\beta}}- {X}  {\beta}}^{B})^{T} (\overbrace{{y}- {X} \hat{ {\beta}}}^{A}+\overbrace{ {X} \hat{ {\beta}}- {X}  {\beta}}^{B}) \\
=& A^{T} A+B^{T} B+ A^{T} B+B^TA\\
=& A^{T} A+B^{T} B+2 A^{T} B \\
&\text{(We use the fact }A^TB=B^TA \text{, because both terms are scalar, transposition is itself)}\\
=&A^{T} A+B^{T} B-2\overbrace{( {y}- {X} \hat{ {\beta}})^{T}( {X} \hat{ {\beta}}- {X}  {\beta})}^{0} \\
=&({y}-{X} \hat{{\beta}})^{T}({y}-{X} \hat{{\beta}})+(\hat{\beta}^TX^T-\beta^TX^T)(X\hat{\beta}-X\beta) \\
=&({y}-{X} \hat{{\beta}})^{T}({y}-{X} \hat{{\beta}})+(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})
\end{aligned}
$$

Kernel decomposition result is
$$
( {y}- {X}  {\beta})^{T}( {y}- {X}  {\beta})=({y}-{X} \hat{{\beta}})^{T}({y}-{X} \hat{{\beta}})+(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})
$$

The quadratic form $(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})$ is created be combined with the kernal of prior $P\left( {\beta} \mid \sigma^{2}\right)$ that is $\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)$

Joining exponential terms in likehood and prior will end up as an addition of them
$$
(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})+\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)
$$

Expand both terms, again the $2 \hat{ {\beta}}^T  {X}^{T}  {X}  {\beta}$ and $2  {\mu}^{T}  {\Lambda}  {\beta}$ exist because cross terms are scalars.
\begin{align}
(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})+\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)&= {\beta}^{T}  {X}^{T}  {X}  {\beta}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}-2 \hat{ {\beta}}^T  {X}^{T}  {X}  {\beta}+ {\beta}^{T}  {\Lambda}  {\beta}+ {\mu}^{T}  {\Lambda}  {\mu}-2  {\mu}^{T}  {\Lambda}  {\beta}\\
&=\color{red}{\beta}^{T}  {X}^{T}  {X}  {\beta}\color{black} + \color{red}{\beta}^{T}  {\Lambda}  {\beta} \color{black}-\color{blue}2 \hat{ {\beta}}^T  {X}^{T}  {X}  {\beta}-\color{blue}2  {\mu}^{T}  {\Lambda}  {\beta}+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}\\
&=\color{red}\beta^T\underbrace{(X^TX+\Lambda)}_{M}\beta\color{black}-\color{blue}2\underbrace{(\hat{\beta}^TX^TX+\mu^T\Lambda)}_{m^T}\beta+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}\\
&=\underbrace{\color{red}\beta^TM\beta\color{black}-\color{blue}2m^T\beta}_{\text{needs completing the square}}+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}\\
\end{align}

Details of completing the square
$$
\begin{align}
\color{red}\beta^TM\beta\color{black}-\color{blue}2m^T\beta\color{black}+\overbrace{\color{green}m^TM^{-1}m-m^TM^{-1}m}^{0}
&=\overbrace{\color{red}\beta^TM\beta\color{black}-\color{blue}m^T\beta\color{black}-\color{blue}\beta^Tm\color{black}+\color{green}m^TM^{-1}m}^{\text{complete the square}}-\color{green}m^TM^{-1}m\\
&=\color{Maroon}(\beta^TM-m^T)(\beta-M^{-1}m)\color{black}-\color{green}m^TM^{-1}m\\
&=\color{Maroon}(\beta^T-m^TM^{-1})M(\beta-M^{-1}m)\color{black}-\color{green}m^TM^{-1}m\\
&=\color{Maroon}(\beta-M^{-1}m)^TM(\beta-M^{-1}m)\color{black}-\color{green}m^TM^{-1}m
\end{align}
$$

Last step above made use of the fact $(M^{-1})^T=M^{-1}$.

So this is the new expression we are working on
$$
(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})+\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)=\color{Maroon}\underbrace{\color{Maroon}(\beta-M^{-1}m)^TM(\beta-M^{-1}m)}_{\text{Posterior normal kernal}}\color{black}-\color{green}m^TM^{-1}m+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}
$$

So far we have achieved the kernal for the normal distribution part in posterior. And the rest will be shuffle into the inverse-gamma part.

Define the mean in the posterior kernel
$$
\begin{align}
\mu_{N} = M^{-1}m &= M^{-1}(X^TX\hat{\beta}+\Lambda^T\mu)\\
&=M^{-1}(X^TX(X^TX)^{-1}X^Xy+\Lambda^T\mu)\\
&=M^{-1}\underbrace{(X^Ty+\Lambda^T\mu)}_{m}
\end{align}
$$

So this is the new expression we are working on
$$
(\hat{{\beta}}-{\beta})^{T} {X}^{T} {X}(\hat{{\beta}}-{\beta})+\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)=\color{Maroon}\underbrace{\color{Maroon}(\beta-\mu_N)^TM(\beta-\mu_N)}_{\text{Posterior normal kernal}}\color{black}-\color{green}m^TM^{-1}m+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}
$$

$$
({y}-{X} {\beta})^{T}({y}-{X} {\beta})+\left({\beta}-{\mu}\right)^{T} {\Lambda}\left({\beta}-{\mu}\right)=({y}-{X} \hat{\beta})^{T}({y}-{X} \hat{\beta})+\color{Maroon}\underbrace{\color{Maroon}(\beta-\mu_N)^TM(\beta-\mu_N)}_{\text{Posterior normal kernal}}\color{black}-\color{green}m^TM^{-1}m+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}
$$

Simplify each term on the right hand side

$$
({y}-{X} \hat{\beta})^{T}({y}-{X} \hat{\beta})+\color{Maroon}\underbrace{\color{Maroon}(\beta-\mu_N)^TM(\beta-\mu_N)}_{\text{Posterior normal kernal}}\color{black}-\color{green}m^TM^{-1}m+\color{black} {\mu}^{T}  {\Lambda}  {\mu}+\hat{ {\beta}}^T  {X}^{T}  {X} \hat{ {\beta}}
$$

$$
\begin{aligned}
( {y}- {X} \hat{ {\beta}})^{T}( {y}- {X} \hat{ {\beta}})&= {y}^{T}  {y}+\hat{ {\beta}}^{T}  {X}^{T}  {X} \hat{ {\beta}} -2  {y}^{T}  {X} \hat{ {\beta}} \\
&= {y}^{T}  {y}+\hat{ {\beta}}^{T}  {X}^{T}  {X} (\overbrace{X^TX)^{-1}X^Ty}^{\hat{\beta}} -2  {y}^{T}  {X} \hat{ {\beta}}  \\
&= {y}^{T}  {y}+\hat{ {\beta}}^{T}  X^Ty -2  {y}^{T}  {X} \hat{ {\beta}}  \\
&= {y}^{T}  {y}+(\overbrace{(X^TX)^{-1}X^Ty}^{\hat{\beta}})^{T}  X^Ty -2  {y}^{T}  {X} \hat{ {\beta}}  \\
&= y^Ty+y^TX(X^TX)^{-1}X^Ty - 2y^TX\hat{\beta}\\
&=  {y}^{T}  {y}+{y}^{T}  {X} \hat{ {\beta}}-2{y}^{T}  {X} \hat{ {\beta}}\\
&={y}^{T}  {y}-{y}^{T}  {X} \hat{ {\beta}}\\
\end{aligned}
$$

$$
m^TM^{-1}m = (X^Ty+\Lambda^T\mu)^TM^{-1}(X^Ty+\Lambda^T\mu)
$$

```python

```

```python

```

The purpose is to decompose the kernel of likelihood into two quadratic forms.

$$
\begin{gathered}
P(\beta \mid h)=(2 \pi)^{-\frac{k}{2}} h^{\frac{k}{2}}|V|^{-\frac{1}{2}} \exp \left[-\frac{h}{2}(\beta-\mu)^{\prime} V^{-1}(\beta-\mu)\right] \\
P(h)=\frac{1}{\left(\frac{2 m}{v}\right)^{v / 2}} \frac{1}{\Gamma\left(\frac{v}{2}\right)} h^{\frac{v-2}{2}} \exp \left[-\frac{h v}{2 m}\right]
\end{gathered}
$$

```python

```

Chapter 7 - Revisit Linear Regression With MCMC.ipynb


$$\newcommand{\mean}{\mathrm{mean}}
\newcommand{\std}{\mathrm{std}}
\newcommand{\wt}{\mathrm{wt}}
\newcommand{\like}{\mathrm{like}}
\newcommand{\pred}{\mathrm{pred}}
\newcommand{\test}{\mathrm{test}}
\newcommand{\intr}{\mathrm{intr}}
\newcommand{\train}{\mathrm{train}}
\newcommand{\true}{\mathrm{true}}
\newcommand{\set}[1]{\left\{ #1 \right\}}
\newcommand{\ev}[1]{\left. #1 \right|}
\newcommand{\bra}[1]{\left\langle #1 \right|}
\newcommand{\ket}[1]{\left| #1 \right\rangle}
\newcommand{\braket}[1]{\left\langle #1 \right\rangle}
\newcommand{\bbraket}[1]{\left\llangle #1 \right\rrangle}$$

# Underlying Mathematics
$$\newcommand{\eq}[1]{Eq.\ (\ref{#1})}
\newcommand{\fig}[2]{Fig.\ \ref{#1}#2}
\newcommand{\diag}{{\rm diag}}
\newcommand{\cov}{{\rm cov}}
\newcommand{\data}{{\rm data}}
\newcommand{\real}{{\rm real}}
\newcommand{\die}{\mathrm{die}}
\newcommand{\live}{\mathrm{live}}
\newcommand{\Lap}{\mathrm{Lap}}
\newcommand{\sym}{\{ \mathrm{sym~factor} \}}
\newcommand{\const}{{\rm const}}
\newcommand{\true}{\mathrm{true}}
\newcommand{\set}[1]{\left\{ #1 \right\}}
\newcommand{\ev}[1]{\left. #1 \right|}
\newcommand{\bra}[1]{\left\langle #1 \right|}
\newcommand{\ket}[1]{\left| #1 \right\rangle}
\newcommand{\braket}[1]{\left\langle #1 \right\rangle}
\newcommand{\bbraket}[1]{\left\llangle #1 \right\rrangle}
\newcommand{\mat}[2]{\left( \begin{array}{#1} #2 \end{array} \right)}
\newcommand{\bracemat}[2]{\left\{ \begin{array}{#1} #2 \end{array} \right\}}
\newcommand{\comment}[1]{{\color{red}[#1]}}
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\mean}{\mathrm{mean}}
\newcommand{\std}{\mathrm{std}}
\newcommand{\wt}{\mathrm{wt}}
\newcommand{\var}{\mathrm{var}}
\newcommand{\pred}{\mathrm{pred}}
\newcommand{\test}{\mathrm{test}}
\newcommand{\intr}{\mathrm{intr}}
\newcommand{\train}{\mathrm{train}}
\newcommand{\model}{\mathrm{model}}
\newcommand{\true}{\mathrm{true}}$$

## Notation

Each MAVE dataset is represented a set of $N$ observations, $\set{(\vec{x}_n,y_n)}_{n=1}^N$, where each observation consists of a sequence $\vec{x}_n$ and a measurement $y_n$. Here, $y_n$ can be either a continuous real-valued number, or a categorical variable representing the bin in which the $n$th sequence was found. Note that, in this representation the same sequence $\vec{x}$ can be observed multiple times, potentially with different values for $y$ due to experimental noise. Datasets with real-valued measurements $y$ are analyzed using GE regression, while datasets with categorical $y$ values are analyzed using MPA regression. Sets of measurements are denoted $\set{y_n}$, etc., and the mean and standard deviations of sets of numbers are denoted by the functions $\mean(\set{\cdot})$ and $\std(\set{\cdot})$

In the ``mavenn.Model`` constructor, ``L`` controls sequence length, ``alphabet`` controls the set of characters, ``Y`` controls the number of $y$-bins (MPA regression only), and ``regression_type`` controls whether GE or MPA regression is used.

## Genotype-phenotype (G-P) maps

We assume that all sequences have the same length $L$, and that at each of the $L$ positions in each sequence there is one of $C$ possible characters ($C = 4$ for DNA and RNA; $C = 20$ for protein). In general, MAVE-NN represents sequences using a vector of binary features $\vec{x}$. We adopt the following notation for the individual features in $\vec{x}$:

$$\begin{eqnarray}
x_{l:c} &=& \left\{
\begin{array}{cl}
    1 & \mathrm{if~character~}c\mathrm{~occurs~at~position~}l~~~ \\
    0 & \mathrm{otherwise},
\end{array} \right.
\end{eqnarray}$$

where $l = 1, 2, \ldots, L$ indexes positions within the sequence, and $c$ indexes the $C$ distinct characters.

We assume that the latent phenotype is given by a function $\phi (\vec{x}; \vec{\theta} )$ that depends on a vector of G-P map parameters $\vec{\theta}$. MAVE-NN currently supports four types of G-P maps, all of which can be inferred using either GE regression or MPA regression.

In the ``mavenn.Model`` constructor, ``gpmap_type`` controls which type of G-P map is used.

### Additive G-P maps
Additive G-P maps assume that each position in $\vec{x}$ contributes independently to the latent phenotype $\phi$, and is comptued using,

$$\begin{eqnarray}
\phi(\vec{x};\vec{\theta})_\text{additive} &=& \theta_0 + \sum_{l=0}^{L-1} \sum_c \theta_{l:c}x_{l:c}.
\end{eqnarray}$$

To use an additive G-P map, set ``gpmap_type='additive'`` in the ``mavenn.Model`` constructor.

### Neighbor G-P maps

Neighbor G-P maps assume that each neighboring pair of positions in $\vec{x}$ contributes to the latent phenotype $\phi$, and is comptued using,

$$\begin{eqnarray}
\phi_{\text{neighbor}}(\vec{x};\vec{\theta}) &=& \theta_0 + \sum_{l=0}^{L-1} \sum_c \theta_{l:c}x_{l:c} + \sum_{l=0}^{L-2} \sum_{c,c^\prime} \theta_{l:c,l+1:c'} x_{l:c}x_{l+1:c^\prime}, \nonumber
\end{eqnarray}$$

To use an neighbor G-P map, set ``gpmap_type='neighbor'`` in the ``mavenn.Model`` constructor.

### Pairwise G-P maps

Pairwise G-P maps assume that every pair of positions in $\vec{x}$ contributes to the latent phenotype $\phi$, and is comptued using,

$$\begin{eqnarray}
\phi_{\text{pairwise}} (\vec{x};\vec{\theta})  &=&   \theta_0 + \sum_{l=0}^{L-1} \sum_c \theta_{l:c}x_{l:c} + \sum_{l=0}^{L-2} \sum_{l^\prime=l+1}^{L-1} \sum_{c,c^\prime}  \theta_{l:c,l^\prime:c^\prime} x_{l:c} x_{l^\prime:c^\prime}. \nonumber
\end{eqnarray}$$

To use a pairwise G-P map, set ``gpmap_type='pairwise'`` in the ``mavenn.Model`` constructor.

### Black box G-P maps

Black box G-P maps are represented as dense feed-forward neural networks, i.e. as multi-layer perceptrons (MLPs). To use a black box G-P map, set ``gpmap_type='black'`` in the ``mavenn.Model`` constructor. Note that one can set the number of nodes used in each layer of the ML. For instance, to create an MLP with three hidden layers of size 100, 30, and 10, set ``gpmap_kwargs={'hidden_layer_sizes':[100,30,10]}`` in the ``mavenn.Model`` constructor.

## MPA measurement processes

In MPA regression, MAVE-NN directly models the measurement process $p(y|\phi)$. At present, MAVE-NN only supports MPA regression for discrete values of $y$, which are indexed using nonnegative integers. MAVE-NN takes two forms of input for MPA regression. One is a set of (non-unique) sequence-measurement pairs $\set{(\vec{x}_n, y_n)}_{n=0}^{N-1}$, where $N$ is the total number of independent measurements and $y_n \in  \{0,1,\ldots,Y-1\}$, where $Y$ is the total number of bins. The other is a set of (unique) sequence-count-vector pairs $\set{(\vec{x}_m, \vec{c}_m)}_{m=0}^{M-1}$, where $M$ is the total number of unique sequences in the data set, and $\vec{c}_m = (c_{m0}, c_{m1}, \ldots, c_{m(Y-1)})$ lists, for each value of $y$, the number of times $c_{my}$ that the sequence $\vec{x}_m$ was observed in bin $y$. MPA measurement processes are computed internally using the latter representation via
\begin{eqnarray}
p(y|\phi) &=& \frac{w_y(\phi)}{\sum_{y'} w_{y'}(\phi)} \\
w_y(\phi) &=& \exp \left[ a_y + \sum_{k=0}^{K-1} b_{yk} \tanh(c_{yk} \phi + d_{yk}) \right]~~~~~~
\end{eqnarray}
where $K$ is the number of hidden nodes per value of $y$. The trainable parameters of this measurement process are thus $\vec{\eta} = \set{a_y, b_{yk}, c_{yk}, d_{yk}}$.

## GE nonlinearities

GE models assume that each measurement $y$ of a sequence $\vec{x}$ is a nonlinear function $g(\phi)$ plus some noise. In MAVE-NN, this nonlinearity is represented as a sum of $\tanh$ sigmoids, i.e.,

$$\begin{equation}
g(\phi;\vec{\alpha}) = a + \sum_{k=0}^{K-1} b_k \tanh(c_k \phi + d_k)
\end{equation}$$

where $K$ specifies the number of "hidden nodes" contributing to the sum, and $\vec{\alpha} = \set{a, b_k, c_k, d_k}$ are trainable parameters. By default, MAVE-NN constrains $g(\phi;\vec{\alpha})$ to be monotonic in $\phi$ by requiring all $b_k \ge 0$ and $c_k \ge 0$, but this constraint can be relaxed.

In the ``mavenn.Model`` constructor, ``ge_nonlinearity_hidden_nodes`` controls the value for $K$, while  ``ge_nonlinearity_monotonic`` controls the monotonicity constraint on $g(\phi;\vec{\alpha})$.

## GE noise models

MAVE-NN supports three types of GE noise models: Gaussian, Cauchy, and skewed-t. These are specified using the ``ge_noise_model`` keyword argument in the ``mavenn.Model`` constructor.

### Gaussian nose models

The Gaussian noise model, corresponding to ``ge_noise_model='Gaussian'``, is given by

$$\begin{eqnarray}
    p_\mathrm{gauss}(y|\hat{y};s) = \frac{1}{\sqrt{2 \pi s^2}} \exp\left[-\frac{(y - \hat{y})^2}{2 s^2}\right],
\end{eqnarray}$$

where $s$ denotes the standard deviation. Importantly, MAVE-NN allows this noise model to be heteroskedastic by representing $s$ as an exponentiated polynomial in $\hat{y}$, i.e.,

$$\begin{equation}
s(\hat{y}) = \exp \left[ \sum_{k=0}^K a_k \hat{y}^k \right],
\end{equation}$$

where $K$ is the order of the polynomial and $\set{a_k}$ are trainable parameters. The user has the option to set $K$, and using $K=0$ renders this noise model homoskedastic. Quantiles are computed using $y_q = \hat{y} + s\, \sqrt{2}\, \mathrm{erf}^{-1}(2 q -1) $ for user-specified values of $q \in [0,1]$.

### Cauchy noise models

The Cauchy noise model, corresponding to ``ge_noise_model='Cauchy'``, is given by

$$\begin{eqnarray}
    p_\mathrm{cauchy}(y|\hat{y};s) = \left[ \pi s \left( 1 + \frac{(y-\hat{y})^2}{s^2} \right) \right]^{-1}
\end{eqnarray}$$

where the scale parameter $s$ is an exponentiated $K$'th order polynomial in $\hat{y}$. Quantiles are computed using $y_q = \hat{y} + s\, \mathrm{tan}[\pi(q-\frac{1}{2})]$.

### Skewed-t noise models

The skewed-t noise model, corresponding to ``ge_noise_model='SkewedT'``, is of the form described by Jones & Faddy (2003), and is given by

$$\begin{equation}
     p_\mathrm{skewt}(y|\hat{y};s,a,b) = s^{-1} f(t; a,b),
\end{equation}$$

where

$$\begin{equation}
     t = t^* + \frac{y-\hat{y}}{s}. ~~~t^* = \frac{(a-b)\sqrt{a+b}}{\sqrt{2a+1}\sqrt{2b+1}},
\end{equation}$$

and

$$\begin{eqnarray}
 f(t;a,b) &=&  \frac{2^{1-a-b}}{\sqrt{a+b}}\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}  \left[1 + \frac{t}{\sqrt{a+b+t^2}} \right]^{a+\frac{1}{2}} \left[ 1 - \frac{t}{\sqrt{a+b+t^2}} \right]^{b+\frac{1}{2}}.
 \end{eqnarray}$$

Note that the $t$ statistic here is an affine function of $y$ chosen so that the distribution's mode (corresponding to $t^*$) is positioned at $\hat{y}$. The three parameters of this model, $\set{s, a, b}$, are each represented using $K$-th order exponentiated polynomial with trainable coefficients. Quantiles are computed using

$$\begin{equation}
y_q = \hat{y}  + (t_q-t^*)s,
\end{equation}$$

where

$$\begin{equation}
t_q = \frac{(2x_q-1)\sqrt{a+b}}{\sqrt{1 - (2x_q-1)^2}},~~~x_q = I^{-1}_{q}(a,b),
 \end{equation}$$

and $I^{-1}$ denotes the inverse of the regularized incomplete Beta function $I_x(a,b)$.

## Gauge modes and diffemorphic modes

These G-P maps also have non-identifiable degrees of freedom that must be "fixed", i.e. pinned down, before the values of individual parameters can be meaningfully interpreted. These degrees of freedom come in two flavors: gauge modes and diffeomorphic modes. Gauge modes are changes to $\vec{\theta}$ that do not alter the values of the latent phenotype $\phi$. Diffeomorphic modes (Kinney & Atwal, 2014; Atwal & Kinney, 2016) are changes to $\vec{\theta}$ that do alter $\phi$, but do so in ways that can be undone by transformations of the measurement process $p(y|\phi)$ along corresponding "dual modes". As shown in (Kinney & Atwal, 2014), the diffeomorphic modes of linear G-P maps like those considered here will in general correspond to affine transformations of $\phi$ (though there are exceptions at special values of $\vec{\theta}$). MAVE-NN fixes both gauge modes and diffeomorphic modes before providing parameter values or other access to model internals to the user, e.g. when ``model.get_theta()`` is called.

### Fixing diffeomorphic modes

Diffeomorphic modes of G-P maps are fit by transforming G-P map parameters $\vec{\theta}$ via

$$\begin{equation}
    \theta_0 \to \theta_0 - a
\end{equation}$$

and

$$\begin{eqnarray}
\vec{\theta} \to \vec{\theta}/b
\end{eqnarray}$$

where $a= \mean(\set{\phi_n})$ and $b = \std(\set{\phi_n})$ are the mean and standard deviation of $\phi$ values computed on the training data. This produces a corresponding change in latent phenotype values $\phi \to (\phi - a)/b$. To avoid altering likelihood contributions $p(y|x)$, MAVE-NN further transforms the measurement process $p(y|\phi)$ along dual modes in order to compensate. In GE regression this is done by adjusting the GE nonlinearity while keeping the noise model $p(y|\hat{y})$ fixed,

$$\begin{eqnarray}
g(\phi) \to g(a + b \phi),
\end{eqnarray}$$

while in MPA regression MAVE-NN adjusts the full measurement process,

$$\begin{eqnarray}
p(y | \phi) \to p(y | a + b \phi).
\end{eqnarray}$$

### Fixing the gauge

Gauge modes are fit using a "hierarchical gauge", in which lower-order terms in $\phi(\vec{x}; \vec{\theta})$ account for the highest possible fraction of variance in $\phi$. Such gauges further require a probability distribution on sequence space, which is used to compute these variances. MAVE-NN assumes that such distributions factorize by position, and can thus be represented by a probability matrix with elements $p_{l:c}$, representing the probability of character $c$ at position $l$. MAVE-NN then transforms G-P map parameters via,

$$\begin{eqnarray}
    \theta_0 &\to & \theta_0
             + \sum_{l} \sum_{c'} \theta_{l:c'}\,p_{l:c'}
             + \sum_{l} \sum_{l'>l} \sum_{c,c'}\theta_{l:c,l':c'}\,p_{l:c}\,p_{l':c'}, \nonumber
\end{eqnarray}$$

$$\begin{eqnarray}
    \theta_{l:c}&\to &{\theta}_{l:c}
                -\sum_{c'}\theta_{l:c'}\,p_{l:c} \nonumber \\
                & &  +\sum_{l'>l} \sum_{c'} \theta_{l:c,l':c'}\,p_{l':c'}
                +\sum_{l'<l} \sum_{c'} \theta_{l':c',l:c}\,p_{l':c'} \nonumber \\
                & &  -\sum_{l'>l} \sum_{c',c''}  \theta_{l:c',l':c''}\,p_{l:c'}\,p_{l':c''},
                -\sum_{l'<l} \sum_{c',c''}  \theta_{l':c'',l:c'}\,p_{l:c'}\,p_{l':c''},
\end{eqnarray}$$

and

$$\begin{eqnarray}
    \theta_{l:c,l':c'} & \to & \theta_{l:c,l':c'}
    -\sum_{c''} \theta_{l:c'',l':c'}\,p_{l:c''}
    -\sum_{c''} \theta_{l:c,l':c''}\,p_{l':c''}
    +\sum_{c'',c'''}\theta_{l:c'',l':c'''}\,p_{l:c''}\,p_{l':c'''}.\nonumber
\end{eqnarray}$$

The gauge-fixing transformations for the both the additive and neighbor models follow the same rules, but with $\theta_{l:c,l':c'} = 0$ for all $l,l'$ (additive model) or whenever $l' \neq l+1$ (neighbor model).

## Metrics

### Loss function

Let $\vec{\theta}$ denote the parameters of G-P map, and $\vec{\eta}$ denote the parameters of the measurement process. MAVE-NN optimizations these parameters using stochastic gradient descent (SGD) on a loss function given by

$$\begin{equation}
    \mathcal{L} = \mathcal{L}_\var + \mathcal{L}_\mathrm{reg}
\end{equation}$$

where $\mathcal{L}_\like$ is the negative log likelihood of the model, given by

$$\begin{eqnarray}
    \mathcal{L}_\like[\vec{\theta},\vec{\eta}] =
        -\sum_{n=0}^{N-1} \log \left[ p(y_n|\phi_n;\vec{\eta})\right],~\phi_n = \phi(\vec{x}_n;\vec{\theta}),
\end{eqnarray}$$

and $\mathcal{L}_\mathrm{reg}$ provides for regularization of the model parameters.

In the context of GE regression, we can write $\vec{\eta} = (\vec{\alpha},\vec{\beta})$ where $\vec{\alpha}$ represents the parameters of the GE nonlinearity $g(\phi)$, and $\vec{\beta}$ denotes the parameters of the noise model $p(y|\hat{y})$. The likelihood contributions from each observation $n$ then becomes $p(y_n|\phi_n;\vec{\eta}) = p(y_n|\hat{y}_n;\vec{\beta})$ where $\hat{y}_n = g(\phi_n;\vec{\alpha})$. In the context of MPA regression with a dataset of the form $\set{(\vec{x}_m, \vec{c}_m)}_{m=1}^M$, the loss function simplifies to

$$\begin{eqnarray}
    \mathcal{L}_\like[\vec{\theta},\vec{\eta}] &=& \sum_{m=0}^{M-1} \sum_{y=0}^{Y-1} c_{my} \log [p(y|\phi_m;\vec{\eta})],
\end{eqnarray}$$

where $\phi_m = \phi(x_m;\vec{\theta})$. For the regularization term, MAVE-NN uses an $L_2$ penalty of the form

$$\begin{equation}
    \mathcal{L}_\mathrm{reg}[\vec{\theta},\vec{\eta}] = \lambda_{\theta}|\vec{\theta}|^2 + \lambda_{\eta}|\vec{\eta}|^2,
\end{equation}$$

where $\lambda_{\theta}$ and $\lambda_{\eta}$ respectively control the strength of regularization for the G-P map and measurement process parameters.

### Variational information

It is sometimes useful to consider a quantity we call the "variational information", $I_\var[y;\phi]$, which is an affine transformation of $\mathcal{L}_\like$:

$$\begin{equation}
    I_\var[y;\phi] = H[y] - \frac{\log_2(e)}{N}\mathcal{L}_\like.
\end{equation}$$

Here $H[y]$ is the entropy of the data $\set{y_n}$, which is estimated using the $k$'th nearest neighbor (kNN) estimator from the NPEET package (Ver Steeg, 2014). Noting that this quantity can also be written as $I_\var[y;\phi] = H[y] - \mean(\set{Q_n})$ where $Q_n = -\log_2 p(y_n|\phi_n)$, we estimate the associated uncertainty using

$$\begin{equation}
    \delta I_\var[y;\phi] = \sqrt{\delta H[y]^2 + \var(\set{Q_n})/N}.
\end{equation} $$

Variational information quantifies the mutual information between $\phi$ and $y$ assuming that the inferred measurement process $p(y|\phi)$ is correct. We therefore propose $I_\var \pm \delta I_\var$ as a diagnostic for model fitting and performance, as it is directly comparable to the predictive information of the model, $I_\pred[y;\phi]$, and can be computed during model training without any costly entropy estimation steps.

It is worth noting that $I_\var[y;\phi]$ provides an approximate lower bound to $I_\pred[y;\phi]$. In the $N_\test \to \infty$ limit,

$$\begin{eqnarray}
I_\pred[y;\phi] &=& H[y] - H[y|\phi] \\
&=& H[y] + \braket{ \log_2 p_\true(y|\phi) }_\true  \\
&=& H[y] + \braket{ \log_2 p_\model(y|\phi)}_\true + \braket{ \log_2 \frac{p_\true(y|\phi)}{p_\model(y|\phi)} }_\true  \\
& = & H[y] - \frac{\log_2(e)}{N_\test} \mathcal{L}_\like + D_{KL}(p_\true || p_\model) \\
&=& I_\var[y;\phi] + D_\mathrm{KL}(p_\true || p_\model) \\
&\geq & I_\var[y;\phi],
\end{eqnarray}$$

where $D_{KL}$ denotes the Kullback-Leibler divergence and $\braket{\cdot}_\true$ indicates averaging over $p_\true(y,\phi)$. When $N_\test$ is finite this lower bound is only approximate, of course, but the uncertainties in these information quantities are often quite small due to the large size of typical MAVE datasets. Moreover the difference between these quantities,

$$\begin{equation} I_\pred[y;\phi] - I_\var[y;\phi] \approx D_\mathrm{KL}(p_\true || p_\model) \end{equation},$$

quantifies how much $p_\model$ deviates from $p_\true$, and can thus be used as a diagnostic for how accurate the assumed form of the measurement process is.

### Predictive information
The predictive information $I_\pred[y;\phi]$, when computed on data not used for training, provides a measure of how strongly a G-P map predicts experimental measurements irrespective of the corresponding measurement process. To estimate this quantity we use k'th nearest neighbor (kNN) estimators of entropy and mutual information adapted from the NPEET Python package (VanSteeg, 2014). Here, the user has the option of adjusting $k$, which controls a variance/bias tradeoff. When $y$ is discrete (MPA regression), $I_\pred[y;\phi]$ is computed using the classic kNN entropy estimator (Vasicek, 1976; Kraskov et al., 2004) via the decomposition $I_\pred[y;\phi] = H[\phi] - \sum_y p(y) H_y[\phi]$, where $H_y[\phi]$ denotes the entropy of $p(\phi|y)$ for fixed $y$. When $y$ is continuous (GE regression), $I_\pred[y;\phi]$ is estimated using the kNN-based KSG algorithm (Kraskov et al, 2004). This approach optionally supports a local nonuniformity correction of (Gao et al., 2014), which is important when $y$ and $\phi$ exhibit strong dependencies, but which also requires substantially more time to compute. We emphasize that both estimates of $I_\pred[y;\phi]$ make no use of the measurement process $p(y|\phi)$ inferred by MAVE-NN.

### Uncertainties in kNN estimates

MAVE-NN quantifies uncertainties in $H[y]$ and $I[y;\phi]$ using multiple random samples of half the data. Let $\mathcal{D}_{100\%}$ denote a full dataset, and let $\mathcal{D}_{50\%,r}$ denote a 50\% subsample (indexed by $r$) of this dataset. Given an estimator $E(\cdot)$ of either entropy or mutual information, as well as the number of subsamples $R$ to use, the uncertainty in $E(\mathcal{D}_{100\%})$ is estimated as

$$\begin{equation}
    \delta E(\mathcal{D}_{100\%}) = \frac{1}{\sqrt{2}} \std \left[\set{E(\mathcal{D}_{50\%,r})}_{r=0}^{R-1} \right].
\end{equation}$$

By default MAVE-NN uses $R=25$. We note that computing such uncertainty estimates substantially increase computation time, as $E(\cdot)$ needs to be evaluated $R+1$ times instead of just once. We also note that boostrap resampling is not advisable in this context, as it systematically underestimates $H[y]$ and overestimates $I[y;z]$ (data not shown).

## References

1. Jones M, Faddy M (2003). A skew extension of the t-distribution, with applications. [J Roy Stat Soc B Met.  65(1):159-174.](https://dx.doi.org/10.1111/1467-9868.00378)

1. Kinney J, Atwal G (2014). Parametric Inference in the Large Data Limit Using Maximally Informative Models. [Neural Comput 26(4):637-653.](https://dx.doi.org/10.1162/neco_a_00568)

1. Atwal G, Kinney J (2016). Learning Quantitative Sequence–Function Relationships from Massively Parallel Experiments. [J Stat Phys. 162(5):1203-1243.](https://dx.doi.org/10.1007/s10955-015-1398-3)

1. Ver Steeg G (2014). Non-Parametric Entropy Estimation Toolbox (NPEET). https://github.com/gregversteeg/NPEET

1. Kraskov A, Stögbauer H, Grassberger P (2004). Estimating mutual information. [Phys Rev E. 69(6):066138.]( https://dx.doi.org/10.1103/physreve.69.066138)

1. Gao S, Steeg G, Galstyan A (2015). Efficient Estimation of Mutual Information for Strongly Dependent Variables. [PMLR 38:277-286](http://proceedings.mlr.press/v38/gao15.html)


Euler Problem 51
================

By replacing the 1st digit of the 2-digit number \*3, it turns out that six of the nine possible values: 13, 23, 43, 53, 73, and 83, are all prime.

By replacing the 3rd and 4th digits of 56\*\*3 with the same digit, this 5-digit number is the first example having seven primes among the ten generated numbers, yielding the family: 56003, 56113, 56333, 56443, 56663, 56773, and 56993. Consequently 56003, being the first member of this family, is the smallest prime with this property.

Find the smallest prime which, by replacing part of the number (not necessarily adjacent digits) with the same digit, is part of an eight prime value family.

```python
from sympy import sieve
from itertools import combinations, product

primes = set(sieve.primerange(10**5, 10**6))
powers = [10, 100, 1000, 10000, 100000]
smallest_prime = 999999

for i in [1,2,3,4]:
    for j in range(i):
        pattern = 111110 - powers[i] - powers[j]
        for a in range(10):
            for b in range(10):
                for c in 1, 3, 7, 9:
                    current_prime = 0
                    strikes = 0
                    N = a*powers[i] + b*powers[j] + c
                    for k in range(10):
                        if N > smallest_prime:
                            break
                        if not (N in primes):
                            strikes += 1
                            if strikes > 2:
                                break
                        elif current_prime == 0:
                            current_prime = N
                        N += pattern
                    else:
                        print(current_prime)
                        smallest_prime = min(smallest_prime, current_prime)
```

    121313

**Explanation:** We search for examples with up to six digits. The number of asterisks must be a multiple of 3,
since otherwise at least two of the numbers obtained by replacing the asterisks with a digit will be divisible by 3.
For example, the pattern 56\*\*3 generates the 10 numbers {56003, 56113, 56223, ..., 56993}, and three of these numbers
are divisible by 3 (56223, 56553, and 56883). Clearly the last digit in the pattern cannot be an asterisk, since a prime
greater than 5 can end only in 1, 3, 7, or 9.

In the code, `pattern` is a six-digit number with three 0s and three 1s, such as `110100`. The ones indicate the positions of the asterisks, and the zeros indicate positions where other digits `a`, `b`, `c` can be placed. For example, if `pattern = 110100`, `a = 5`, `b = 6`, and `c = 7`, then the candidate primes are 005067, 115167, 225267, ..., 995967. We search for examples of this sort in which 8 out of the 10 candidates are prime.

Euler 051 - Prime digit replacements.ipynb


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Stratified Sphere Radar Cross Section
***

Referring to Section 7.4.1.5, Mie gives the exact solution for scattering from a sphere.  The solution is composed of vector wave functions defined in a spherical coordinate system.  The terms of the Mie series are obtained from boundary value techniques.  Therefore, the Mie formulation may be employed regardless of the composition of the sphere.  To calculate the radar cross section of a sphere, use the Mie formulation along with far field approximations to give (Equations 7.42 and 7.43)

\begin{align}
    S_1(\theta_o) &= \sum\limits_{n=1}^\infty(j)^{n+1}\Big[A_n\frac{P_n^1(\cos\theta_o)}{\sin\theta_o} - jB_n\frac{d}{d\theta_o}P_n^1(\cos\theta_o)\Big], \\ \nonumber \\
    S_2(\theta_o) &= \sum\limits_{n=1}^\infty(j)^{n+1}\Big[A_n\frac{d}{d\theta_o}\frac{P_n^1(\cos\theta_o)}{\sin\theta_o} - jB_nP_n^1(\cos\theta_o)\Big],
\end{align}

where $P_n^1$ is the associated Legendre polynomial and may be calculated using the SciPy implementation ***scipy.special.lpmn(m, n, z)***.  $S_1(\theta_o)$ and $S_2(\theta_o)$ are the complex far-field scattered radiation values for the $\hat{\theta}$ and $\hat{\phi}$ directions.  The radar cross section for the $\hat{\theta}$ and $\hat{\phi}$ polarization states is then found to be (Equations 7.44 and 7.45)

\begin{align}\label{eq:rcs_mie}
    \sigma_\theta &= \frac{4\pi}{k_0^2}S_1(\theta_o)\cos^2(\phi_0) \hspace{0.5in} \text{(m^2)}, \\ \nonumber \\
    \sigma_\phi &= \frac{4\pi}{k_0^2}S_2(\theta_o)\sin^2(\phi_0) \hspace{0.5in} \text{(m}^2\text{)}.
\end{align}

For the $N$-layer concentric sphere, use the Mie coefficients of the following form (Equations 7.46 and 7.47)

\begin{align}\label{eq:mie_coefficients_layered}
    A_n = -(j)^n &\frac{2n+1}{n(n+1)}\frac{k_0a_0J_n(k_0a_0) + jZ_n(k_0a_0)(k_0a_0J_n^\prime(k_0a_0)}{k_0a_0H_n(k_0a_0) + jZ_n(k_0a_0)(k_0a_0H_n^\prime(k_0a_0)},  \\ \nonumber \\
    B_n = (j)^n &\frac{2n+1}{n(n+1)}\frac{k_0a_0J_n(k_0a_0) + jY_n(k_0a_0)(k_0a_0J_n^\prime(k_0a_0)}{k_0a_0H_n(k_0a_0) + jY_n(k_0a_0)(k_0a_0H_n^\prime(k_0a_0)}.
\end{align}

Ruck et al. showed that the modal surface impedance and admittance can be derived using an iterative technique similar to the method used for transmission lines.  To begin, the impedance at the interface between the core and the first layer, $Z_n^N$, is determined independently.  Then, the impedance at the second interface, $Z_n^{N-1}$, is determined from $Z_n^N$.  This process continues until the impedance at the outermost surface, $Z_n^0$, is found.  Then $Z_n(k_0a_0) = j(Z_n^0/\eta)$.  Following the same process for the admittance, $Y_n(k_0a_0)$ may also be calculated.
The impedance and admittance are used in the Mie coefficients of (7.46) for the scattering radiation calculation in (7.42).  Finally, the radar cross section is obtained from (7.44).

***

Begin by getting the library path

```python
import lib_path
```

Set the operating frequency (Hz), the radii (m), the relative permeabilities, the relative permittivities, the number of modes and the flag for perfectly conducting core

```python
frequency = 1e9

radius = [1.0, 1.25]

mu_r = [1.0, 1.0]

eps_r = [1.0, 4.0]

number_of_modes = 60

pec = True
```

Size the ordered arrays

```python
from numpy import ones

nr = len(radius)

mu = ones(nr + 1)

eps = ones(nr + 1)

ra = ones(nr)
```

Set up the parameters in the correct order

```python
i = 0

for r in radius:

    ra[nr - 1 - i] = float(r)

    i += 1

i = 0

for m, e in zip(mu_r, eps_r):

    mu[nr - i] = float(m)

    eps[nr - i] = float(e)

    i += 1
```

Set the observation angles (radians) using the `linspace` routine from `scipy`

```python
from numpy import linspace

from scipy.constants import pi

observation_angle = linspace(0, pi, 721)
```

Calculate the coefficients for the sphere

```python
from Libs.rcs.stratified_sphere import coefficients

An, Bn = coefficients(frequency, eps, mu, ra, number_of_modes, pec)
```

Calculate the radar cross section (m^2) for the stratified sphere

```python
from Libs.rcs.stratified_sphere import radar_cross_section

from scipy import array

et = array([radar_cross_section(frequency, oa, 0, An, Bn) for oa in observation_angle])

ep = array([radar_cross_section(frequency, oa, 90, An, Bn) for oa in observation_angle])
```

Display the radar cross section (dBsm) for the stratified sphere using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

from numpy import log10, degrees

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(degrees(observation_angle), 20.0 * log10(abs(ep[:, 1])), '', label='TE')

plt.plot(degrees(observation_angle), 20.0 * log10(abs(et[:, 0])), '--', label='TM')

# Set the plot title and labels

plt.title('RCS vs Bistatic Angle', size=14)

plt.ylabel('RCS (dBsm)', size=12)

plt.xlabel('Observation Angle (deg)', size=12)

plt.ylim(min(20.0 * log10(abs(et[:,0]))) - 3, max(20.0 * log10(abs(et[:,0]))) + 3)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the legend

plt.legend(loc='upper left', prop={'size': 10})
```

```python

```


##  Deriving Schwarschild Solution to Einstein's equation

We play and rederive equations from General theory of relativity (chapter 18, by Dirac) where they describe a sperically symetric solution to Einstein's equation in vaccum.

This example calculates the Ricci tensor from the metric and does this
on the example of Schwarzschild solution.
If you want to derive this by hand, follow the wiki page here:
http://en.wikipedia.org/wiki/Deriving_the_Schwarzschild_solution
Also read the above wiki and follow the references from there if
something is not clear, like what the Ricci tensor is, etc.

```python
from sympy import (exp, Symbol, sin, Rational, Derivative, dsolve, Function,
                  Matrix, Eq, pprint, Pow, classify_ode, solve)
```

```python
def grad(f, X):
    a = []
    for x in X:
        a.append(f.diff(x))
    return a

def d(m, x):
    return grad(m[0, 0], x)

class MT(object):
    def __init__(self, m):
        self.gdd = m
        self.guu = m.inv()

    def __str__(self):
        return "g_dd =\n" + str(self.gdd)

    def dd(self, i, j):
        return self.gdd[i, j]

    def uu(self, i, j):
        return self.guu[i, j]

class G(object):
    def __init__(self, g, x):
        self.g = g
        self.x = x

    def udd(self, i, k, l):
        g = self.g
        x = self.x
        r = 0
        for m in [0, 1, 2, 3]:
            r += g.uu(i, m)/2 * (g.dd(m, k).diff(x[l]) + g.dd(m, l).diff(x[k])
                    - g.dd(k, l).diff(x[m]))
        return r

class Riemann(object):
    def __init__(self, G, x):
        self.G = G
        self.x = x

    def uddd(self, rho, sigma, mu, nu):
        G = self.G
        x = self.x
        r = G.udd(rho, nu, sigma).diff(x[mu]) - G.udd(rho, mu, sigma).diff(x[nu])
        for lam in [0, 1, 2, 3]:
            r += G.udd(rho, mu, lam)*G.udd(lam, nu, sigma) \
                - G.udd(rho, nu, lam)*G.udd(lam, mu, sigma)
        return r

class Ricci(object):
    def __init__(self, R, x):
        self.R = R
        self.x = x
        self.g = R.G.g

    def dd(self, mu, nu):
        R = self.R
        x = self.x
        r = 0
        for lam in [0, 1, 2, 3]:
            r += R.uddd(lam, mu, lam, nu)
        return r

    def ud(self, mu, nu):
        r = 0
        for lam in [0, 1, 2, 3]:
            r += self.g.uu(mu, lam)*self.dd(lam, nu)
        return r.expand()

def curvature(Rmn):
    return Rmn.ud(0, 0) + Rmn.ud(1, 1) + Rmn.ud(2, 2) + Rmn.ud(3, 3)

nu = Function("nu")
lam = Function("lambda")

t = Symbol("t")
r = Symbol("r")
theta = Symbol(r"theta")
phi = Symbol(r"phi")

# general, spherically symmetric metric
gdd = Matrix((
    (-exp(nu(r)), 0, 0, 0),
    (0, exp(lam(r)), 0, 0),
    (0, 0, r**2, 0),
    (0, 0, 0, r**2*sin(theta)**2)
))
g = MT(gdd)
X = (t, r, theta, phi)
Gamma = G(g, X)
Rmn = Ricci(Riemann(Gamma, X), X)

def pprint_Gamma_udd(i, k, l):
    pprint(Eq(Symbol('Gamma^%i_%i%i' % (i, k, l)), Gamma.udd(i, k, l)))

def pprint_Rmn_dd(i, j):
    pprint(Eq(Symbol('R_%i%i' % (i, j)), Rmn.dd(i, j)))

# from Differential Equations example
def eq1():
    r = Symbol("r")
    e = Rmn.dd(0, 0)
    e = e.subs(nu(r), -lam(r))
    pprint(dsolve(e, lam(r)))

def eq2():
    r = Symbol("r")
    e = Rmn.dd(1, 1)
    C = Symbol("CC")
    e = e.subs(nu(r), -lam(r))
    pprint(dsolve(e, lam(r)))

def eq3():
    r = Symbol("r")
    e = Rmn.dd(2, 2)
    e = e.subs(nu(r), -lam(r))
    pprint(dsolve(e, lam(r)))

def eq4():
    r = Symbol("r")
    e = Rmn.dd(3, 3)
    e = e.subs(nu(r), -lam(r))
    pprint(dsolve(e, lam(r)))
    pprint(dsolve(e, lam(r), 'best'))

def main():

    print("Initial metric:")
    pprint(gdd)
    print("-"*40)
    print("Christoffel symbols:")
    pprint_Gamma_udd(0, 1, 0)
    pprint_Gamma_udd(0, 0, 1)
    print()
    pprint_Gamma_udd(1, 0, 0)
    pprint_Gamma_udd(1, 1, 1)
    pprint_Gamma_udd(1, 2, 2)
    pprint_Gamma_udd(1, 3, 3)
    print()
    pprint_Gamma_udd(2, 2, 1)
    pprint_Gamma_udd(2, 1, 2)
    pprint_Gamma_udd(2, 3, 3)
    print()
    pprint_Gamma_udd(3, 2, 3)
    pprint_Gamma_udd(3, 3, 2)
    pprint_Gamma_udd(3, 1, 3)
    pprint_Gamma_udd(3, 3, 1)
    print("-"*40)
    print("Ricci tensor:")
    pprint_Rmn_dd(0, 0)
    e = Rmn.dd(1, 1)
    pprint_Rmn_dd(1, 1)
    pprint_Rmn_dd(2, 2)
    pprint_Rmn_dd(3, 3)
    print("-"*40)
    print("Solve Einstein's equations:")
    e = e.subs(nu(r), -lam(r)).doit()
    l = dsolve(e, lam(r))
    pprint(l)
    lamsol = solve(l, lam(r))[0]
    metric = gdd.subs(lam(r), lamsol).subs(nu(r), -lamsol)  # .combine()
    print("metric:")
    pprint(metric)

if __name__ == "__main__":
    main()
```

    Initial metric:
    ⎡  ν(r)                       ⎤
    ⎢-ℯ        0    0       0     ⎥
    ⎢                             ⎥
    ⎢         λ(r)                ⎥
    ⎢  0     ℯ      0       0     ⎥
    ⎢                             ⎥
    ⎢                2            ⎥
    ⎢  0       0    r       0     ⎥
    ⎢                             ⎥
    ⎢                    2    2   ⎥
    ⎣  0       0    0   r ⋅sin (θ)⎦
    ----------------------------------------
    Christoffel symbols:
           d
           ──(ν(r))
           dr
    Γ⁰₁₀ = ────────
              2
           d
           ──(ν(r))
           dr
    Γ⁰₀₁ = ────────
              2

            -λ(r)  ν(r) d
           ℯ     ⋅ℯ    ⋅──(ν(r))
                        dr
    Γ¹₀₀ = ─────────────────────
                     2
           d
           ──(λ(r))
           dr
    Γ¹₁₁ = ────────
              2
               -λ(r)
    Γ¹₂₂ = -r⋅ℯ
               -λ(r)    2
    Γ¹₃₃ = -r⋅ℯ     ⋅sin (θ)

           1
    Γ²₂₁ = ─
           r
           1
    Γ²₁₂ = ─
           r
    Γ²₃₃ = -sin(θ)⋅cos(θ)

           cos(θ)
    Γ³₂₃ = ──────
           sin(θ)
           cos(θ)
    Γ³₃₂ = ──────
           sin(θ)
           1
    Γ³₁₃ = ─
           r
           1
    Γ³₃₁ = ─
           r
    ----------------------------------------
    Ricci tensor:

                                                                    2    -λ(r)  ν(
             -λ(r)  ν(r) d        d           -λ(r)  ν(r) ⎛d       ⎞    ℯ     ⋅ℯ
            ℯ     ⋅ℯ    ⋅──(λ(r))⋅──(ν(r))   ℯ     ⋅ℯ    ⋅⎜──(ν(r))⎟
                         dr       dr                      ⎝dr      ⎠
    R₀₀ = - ────────────────────────────── + ──────────────────────── + ──────────
                          4                             4

         2
    r)  d
      ⋅───(ν(r))    -λ(r)  ν(r) d
         2         ℯ     ⋅ℯ    ⋅──(ν(r))
       dr                       dr
    ──────────── + ─────────────────────
    2                        r
                                              2
                                        2    d
          d        d          ⎛d       ⎞    ───(ν(r))   d
          ──(λ(r))⋅──(ν(r))   ⎜──(ν(r))⎟      2         ──(λ(r))
          dr       dr         ⎝dr      ⎠    dr          dr
    R₁₁ = ───────────────── - ─────────── - ───────── + ────────
                  4                4            2          r
             -λ(r) d             -λ(r) d
          r⋅ℯ     ⋅──(λ(r))   r⋅ℯ     ⋅──(ν(r))
                   dr                  dr              -λ(r)
    R₂₂ = ───────────────── - ───────────────── + 1 - ℯ
                  2                   2
             -λ(r)    2    d             -λ(r)    2    d
          r⋅ℯ     ⋅sin (θ)⋅──(λ(r))   r⋅ℯ     ⋅sin (θ)⋅──(ν(r))
                           dr                          dr            2       -λ(r)
    R₃₃ = ───────────────────────── - ───────────────────────── + sin (θ) - ℯ
                      2                           2

        2
    ⋅sin (θ)

    ----------------------------------------
    Solve Einstein's equations:
              ⎛    r    ⎞
    λ(r) = log⎜─────────⎟
              ⎝C₁ + C₂⋅r⎠
    metric:
    ⎡-(C₁ + C₂⋅r)                            ⎤
    ⎢─────────────      0      0       0     ⎥
    ⎢      r                                 ⎥
    ⎢                                        ⎥
    ⎢                   r                    ⎥
    ⎢      0        ─────────  0       0     ⎥
    ⎢               C₁ + C₂⋅r                ⎥
    ⎢                                        ⎥
    ⎢                           2            ⎥
    ⎢      0            0      r       0     ⎥
    ⎢                                        ⎥
    ⎢                               2    2   ⎥
    ⎣      0            0      0   r ⋅sin (θ)⎦

```python

```

sympy/Intermediate - Schwarschild Solution to Einstein's equation.ipynb


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Auto-Correlation Function

[Correlation](https://en.wikipedia.org/wiki/Correlation_and_dependence) is a statistical measure for the dependencies between random processes or between the samples of one random process. The [auto-correlation function](https://en.wikipedia.org/wiki/Autocorrelation) (ACF) characterizes the temporal dependencies within one random signal $x[k]$. It is an important measure for the analysis of signals in communications engineering, source coding and system identification.

### Definition

For a continuous-amplitude real-valued random signal $x[k]$ the ACF is defined by the [second-order ensemble avarage](ensemble_averages.ipynb#Second-Order-Ensemble-Averages) of the signal at two different time-instants $k_1$ and $k_2$

\begin{equation}
\varphi_{xx}[k_1, k_2] = E\{ x[k_1] \cdot x[k_2] \}
\end{equation}

Under the assumption of [wide-sense stationarity](stationary_ergodic.ipynb#Wide-Sense-Stationary-Random-Processes) (WSS) the ACF does only depend on the difference $\kappa = k_1 - k_2$ between the considered sample indexes

\begin{equation}
\varphi_{xx}[\kappa] = E\{x[k] \cdot x[k-\kappa] \}
\end{equation}

where $\kappa$ is commonly chosen as sample index instead of $k$ in order to indicate that it denotes a shift/lag. The ACF quantifies the similarity of a signal with a shifted version of itself. It has high values for high similarity and low values for low similarity.

If the process is additionally [wide-sense ergodic](stationary_ergodic.ipynb#Wide-Sense-Ergodic-Random-Processes), the ACF can be computed by averaging along one sample function

\begin{equation}
\varphi_{xx}[\kappa] = \lim_{K \to \infty} \frac{1}{2K + 1} \sum_{k=-K}^{K} x[k] \cdot x[k-\kappa]
\end{equation}

Note that the normalization on the left side of the sum is discarded in some definitions of the ACF. Above summation resembles strongly the definition of the [discrete convolution](https://en.wikipedia.org/wiki/Convolution#Discrete_convolution). For a random signal $x_N[k] = \text{rect}_N[k] \cdot x[k]$ of finite length $N$ and by exploiting the properties of a wide-sense ergodic random process one yields

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{N} \sum_{k=0}^{N-1} x_N[k] \cdot x_N[k-\kappa] = \frac{1}{N} \; x_N[k] * x_N[-k]
\end{equation}

where the ACF $\varphi_{xx}[\kappa] = 0$ for $|\kappa| > N-1$. Hence, the ACF can be computed by (fast) convolution of the random signal with a time reversed version of itself.

Note in numerical implementations (e.g. MATLAB, Python), the computed ACF is stored in a vector of length $2 N - 1$. The positive indexes $0, 1, \dots, 2 N - 1$ of this vector cannot be directly interpreted as $\kappa$. The indexes of the vector have to be shifted by $N-1$ for a proper interpretation.

### Properties

The following properties of the ACF can be deduced from its definition

1. The ACF $\varphi_{xx}[\kappa]$ has a maximum for $\kappa = 0$. It is given as

    $$ \varphi_{xx}[0] = E\{x^2[k]\} = \sigma_x^2 + \mu_x^2 $$

    This is due to the fact that the signal is equal to itself for $\kappa = 0$. Please note that for periodic random signals more than one maximum will be present.

2. The ACF is a function with even symmetry

    $$ \varphi_{xx}[\kappa] = \varphi_{xx}[-\kappa] $$

3. For typical random signals, the ACF approaches the limiting value

    $$ \lim_{|\kappa| \to \infty} \varphi_{xx}[\kappa] = \mu_x^2 $$

    The similarity of a typical random signal is often low for large lags $\kappa$.

A random signal $x[k]$ is said to be *uncorrelated* if

\begin{equation}
\varphi_{xx}[\kappa] = \mu_x^2 + \sigma_x^2 \cdot \delta[\kappa]
\end{equation}

and as correlated if this condition is not met. The samples of a signal which is uncorrelated show no dependencies between each other in a statistical sense.

### Example

The following example estimates and plots the ACF of a short recorded speech signal.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.io import wavfile

K = 30  # upper/lower limit for lag in ACF

# read audio file
fs, x = wavfile.read('../data/speech_8k.wav')
x = np.asarray(x, dtype=float)/2**15

# compute and truncate ACF
acf = 1/len(x) * np.correlate(x, x, mode='full')
acf = acf[(len(x)-1)-(K-1):(len(x)-1)+K]
kappa = np.arange(-(K-1), K)

# plot ACF
fig = plt.figure(figsize = (10, 8))

plt.stem(kappa, acf)
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')
plt.axis([-K, K, 1.1*min(acf), 1.1*max(acf)]);
plt.grid()
```

**Exercise**

* Does the ACF fulfill the properties stated above?
* The plot shows only a small part of the ACF. Increase the range `K` for $-K \leq \kappa < K$ and check if the last property is fulfilled for the speech signal.
* Is the speech signal uncorrelated or correlated?

It can be observed that the plotted ACF has its maximum value at $\kappa = 0$ and that it shows an even symmetry. The third property can be confirmed by increasing `K` in above example. The speech signal is correlated since $\varphi_{xx}[\kappa] \neq 0$ for $\kappa \neq 0$.

## Auto-Covariance Function

The [auto-covariance function](https://en.wikipedia.org/wiki/Covariance) is defined as the ACF of a zero-mean random signal. For a random signal $x[k]$ with linear mean $\mu_x \neq 0$ it is given as

\begin{equation}
\psi_{xx}[\kappa] = \varphi_{xx}[\kappa] - \mu_x^2
\end{equation}

## Cross-Correlation Function

The cross-correlation function (CCF) is a measure of similarity that two random signals $x[k]$ and $y[k]$ show in a statistical sense.

### Definition

For a continuous-amplitude real-valued random signal $x[k]$ the CCF is defined by the [second-order ensemble avarage](ensemble_averages.ipynb#Second-Order-Ensemble-Averages) of the two signals $x[k]$ and $y[k]$ at two different time-instants $k_x$ and $k_y$

\begin{equation}
\varphi_{xy}[k_x, k_y] = E\{ x[k_x] \cdot y[k_y] \}
\end{equation}

Under the assumption of [wide-sense stationarity](stationary_ergodic.ipynb#Wide-Sense-Stationary-Random-Processes) (WSS) the CCF does only depend on the difference $\kappa = k_x - k_y$ between the considered sample indexes

\begin{equation}
\varphi_{xy}[\kappa] = E\{x[k] \cdot y[k - \kappa] \} = E\{x[k + \kappa] \cdot y[k] \}
\end{equation}

The cross-correlation function (CCF) is a measure of similarity that two random signals $x[k]$ and $y[k - \kappa]$ have with respect to the shift $\kappa \in \mathbb{Z}$. If $x[k]$ and $y[k]$ are wide-sense ergodic processes, the CCF can be computed by averaging along one sample function

\begin{equation}
\varphi_{xy}[\kappa] = \lim_{K \to \infty} \frac{1}{2K + 1} \sum_{k=-K}^{K} x[k] \cdot y[k-\kappa]
\end{equation}

For random signals $x_N[k] = \text{rect}_N[k] \cdot x[k]$ and $y_M[k] = \text{rect}_M[k] \cdot y[k]$ of finite lengths $N$ and $M$ one yields

\begin{equation}
\varphi_{xy}[\kappa] = \frac{1}{N} \sum_{k=0}^{N-1} x_N[k] \cdot y_M[k-\kappa] = \frac{1}{N} \; x_N[k] * y_M[-k]
\end{equation}

where the CCF $\varphi_{xy}[\kappa] = 0$ for $\kappa < -(M-1)$ and $\kappa > N-1$. The CCF can be computed by (fast) convolution of one random signal with a time reversed version of the other random signal. Note in numerical implementations (e.g. MATLAB, Python), the computed CCF is stored in a vector of length $N + M - 1$. The positive indexes $0, 1, \dots, N + M - 1$ of this vector cannot be directly interpreted as $\kappa$. The indexes of the vector have to be shifted by $M-1$ for a proper interpretation.

Above results hold also for the CCF $\varphi_{yx}[\kappa]$ when exchanging $x[k]$ with $y[k]$ and $N$ with $M$.

### Properties

1. The CCF exhibits the following symmetry when exchanging the two random signals

    $$ \varphi_{xy}[\kappa] = \varphi_{yx}[-\kappa] $$

2. The CCF is constant for [*uncorrelated*](https://en.wikipedia.org/wiki/Uncorrelated_random_variables) random signals

    $$ \varphi_{xy}[\kappa] = \mu_x \cdot \mu_y $$

3. Typical random processes are uncorrelated for $|\kappa| \to \infty$

    $$ \lim_{|\kappa| \to \infty} \varphi_{xy}[\kappa] = \mu_x \cdot \mu_y$$

### Example

The following example estimates and plots the CCF of two random signals $x[k]$ and $y[k]$ of finite lengths $N$ and $M = 2 N$.

```python
N = 1024  # length of random signals

# generate two uncorrelated random signals
np.random.seed(2)
x = 2 + np.random.normal(size=N)
y = 1 + np.random.normal(size=2*N)

# compute CCF
ccf = 1/len(x) * np.correlate(x, y, mode='full')
kappa = np.arange(-(N-1), 2*N)

# print mean values of signals
print('Mean of signal x[k]: %f' %np.mean(x))
print('Mean of signal y[k]: %f' %np.mean(y))

# plot CCF
plt.figure(figsize = (10, 8))
plt.stem(kappa, ccf)
plt.title('Estimated cross-correlation function')
plt.ylabel(r'$\hat{\varphi}_{xy}[\kappa]$')
plt.xlabel(r'$\kappa$')
plt.axis([kappa[0], kappa[-1], 0, 1.1*max(ccf)]);
plt.grid()
```

**Exercise**

* Can the signals be assumed to be correlated or uncorrelated when considering the CCF in the range $0 < \kappa < N$ only?
* Why does the CCF of two finite length signals have this overall trapezoid like shape?
* What would be its theoretic value for signals of infinte length?

The CCF is approximately constant for $0 < \kappa < N$. Consequently, the random signals $x[k]$ and $y[k]$ can be assumed to be uncorrelated. The trapezoidal shape of the CCF results from the truncation of the random signals to a finite number of samples. This truncation can be modeled as a multiplication of the infinite length signals by a rectangular signal $\text{rect}_N[k]$ and $\text{rect}_M[k]$, respectively. Interpreting the CCF as convolution and having in mind that the convolution of two rectangular signals results in a signal of trapezoidal shape explains the shape of the CCF shown above. Its theoretic value is constant with a value given by the multiplication of the two linear means of the random signals $\varphi_{xy}[\kappa] = \mu_x \cdot \mu_y$.

## Cross-Covariance Function

The [cross-covariance function](https://en.wikipedia.org/wiki/Covariance) is defined as the CCF of two zero-mean random signals. For two random signals $x[k]$ and $y[k]$ with linear means $\mu_x \neq 0$ and $\mu_y \neq 0$ it is given as

\begin{equation}
\psi_{xy}[\kappa] = \varphi_{xy}[\kappa] - \mu_x \mu_y
\end{equation}

**Exercise**

* What would the plot for $\psi_{xy}[\kappa]$ look like for above example under the assumption of signals of infinite length?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.