# Simon's Algorithm

## Problem statement:

Given: a function $f$ acting on bit strings $f:\{0,1\}^n \rightarrow \{0,1\}^n$ and a promise that $f(x)=f(x \oplus s)$ for all $x$ (addition mod 2). The goal is to use Simon's algorithm to find the unknown string $s$.

### Example:

For example, if $n = 3$, then the following function is an example of a function that satisfies the required and just mentioned property:

|$x$|   $$f(x)$$|
|---|------|
|000|	101|
|001|	010|
|010|	000|
|011|	110|
|100|	000|
|101|	110|
|110|	101|
|111|	010|

Given $f$ is a two-to-one function i.e. it maps exactly two inputs to every unique output, and we find 2 values of input $x$ that have the same output, $f(x_1)=f(x_2)$ then it is guaranteed that $x_1 \oplus x_2 = s$

For example, the input strings $011$ and $101$ are both mapped by $f$ to the same output string $110$. If we XOR $011$ and $101$ we obtain $s$, that is:

$$011 \oplus 101 = 110$$

so for this example $s = 110$

## Problem hardness

To solve this classically, you need to find two different inputs $x$ and $y$ for which $f(x)=f(y)$. Given $f$ is a blackbox, we can discover something about $f$ (or what it does) only when, for two different inputs, we obtain the same output. In any case, we would need to guess $ \Omega ({\sqrt {2^{n}}})$ different inputs before being likely to find a pair on which $f$ takes the same output.

## Simon's Algorithm

The high-level idea behind Simon's algorithm is to "probe" a quantum circuit "enough times" to find $n-1$ (linearly independent) n-bit strings, that is

$$ y_{1},y_{2},\dots ,y_{n-1}\in \{0,1\}^{n} $$

such that the following equations are satisfied

$$ \begin{aligned}y_{1}\cdot s&=0\\y_{2}\cdot s&=0\\&\,\,\,\vdots \\y_{n-1}\cdot s&=0\end{aligned}$$

where $ y_{i}\cdot s$ is the modulo-2 dot product; that is, $ y_{i}\cdot s=y_{i1}s_{1}\oplus y_{i2}s_{2}\oplus \dots \oplus y_{in}s_{n} $

So, this linear system contains $n-1$ linear equations in $n$ unknowns (i.e. the bits of $s$, and the goal is to solve it to obtain $s$, and $s$ is fixed for a given function $f$.

### Simon's quantum circuit

The quantum circuit below is the implementation (and visualization) of the quantum part of Simon's algorithm.

The circuit acts on $2n$ qubits (where $n$ is the length of the bit string in question (i.e., $n=3$ for our example). Apply a Hadamard gate to the first $n$ qubits, then apply $U_f$ - which is an oracle (or "black box"), which knows how to compute $f$ , then apply a Hadamard gate to the first $n$ qubits.

For more details on Simon's algorithm refer to [Wikipedia](<https://en.wikipedia.org/wiki/Simon%27s_problem#Simon's_algorithm>)

## IMPORTS and SETUP

```python
from braket.circuits import Circuit
from braket.devices import LocalSimulator
import numpy as np
import time
import matplotlib.pyplot as plt
%matplotlib inline

# Sets the device to run the circuit on
device = LocalSimulator()

```

### Circuit Definition

1. We start by applying Hadamard gates to the first $n$-qubits.

2. We apply the oracle/black box ($U_f$ gate) to map to $f(x)$. Here we build it dynamically based on our chosen value of $s$. (You can experiment by setting $s$ to other values and different lengths).

3. We apply Hadamard Gate to the first $n$-qubits.

```python
s = '110'
n = len(str(s))

circ = Circuit()

## Apply Hadamard gates to first n qubits before querying the oracle
circ.h(range(n))

# Generic Oracle to implement function f(x) with chosen value of s.
# Apply cnot only for bits of s that are 1
for i in range(n):
    if int(s[i]) == 1:
        for j in range(n):
            circ.cnot(i, n + j)

# Apply Hadamard gates to the first n qbits
circ.h(range(n))

print(circ)
```

    T  : |0| 1 | 2 | 3 |4|5|

    q0 : -H-C---C---C---H---
            |   |   |
    q1 : -H-|---|-C-|-C-C-H-
            |   | | | | |
    q2 : -H-|-H-|-|-|-|-|---
            |   | | | | |
    q3 : ---X---|-X-|-|-|---
                |   | | |
    q4 : -------X---|-X-|---
                    |   |
    q5 : -----------X---X---

    T  : |0| 1 | 2 | 3 |4|5|

## Now run the circuit

```python
task = device.run(circ, shots=2*n-1)
```

### Analyze the results

The measurements are performed on all $2n$ qubits. We are only interested in the first $n$ qubits. So we count the unique results of first $n$ qubits and plott the results

```python
result = task.result()
print(f"measurement_counts: {result.measurement_counts}\n")

# Simulate partial measurement by seperating out first n bits
answer_plot = {}
for measresult in result.measurement_counts.keys():
    measresult_input = measresult[:len(str(s))]
    if measresult_input in answer_plot:
        answer_plot[measresult_input] += result.measurement_counts[measresult]
    else:
        answer_plot[measresult_input] = result.measurement_counts[measresult]

print(f"measurement_of_input_registers: {answer_plot}\n")
plt.bar(answer_plot.keys(), answer_plot.values())
plt.show()

```

Using these results, we can recover the value of $s$

```python
y1,y2 = answer_plot.keys()

s_calc = int(y1)^int(y2)

print(f"Bitwise XOR / Dot product of the 2 results y1 and y2 retrieves our unknown string 's' --> {s_calc}\n")
```

    Bitwise XOR / Dot product of the 2 results y1 and y2 retrieves our unknown string 's' --> 110

In Real Qunatum devices and larger bitstrings, we might get multiple answers that satisfy the equations:
$$ \begin{aligned}y_{1}\cdot s&=0\\y_{2}\cdot s&=0\\&\,\,\,\vdots \\y_{n-1}\cdot s&=0\end{aligned}$$

Once we have $n$ linear equations, we proceed to post-processing with computations that can be done on classical computers using Gaussian elimination.

The steps are as below. ( Reference : https://github.com/nallg00d/simonqiskit/blob/master/simon_algorithm.ipynb )

```python
from braket.aws import AwsDevice, AwsQuantumTask
# Please enter the S3 bucket you created during onboarding in the code below
my_bucket = f"amazon-braket-251940851769" # the name of the bucket
my_prefix = "simons-output" # the name of the folder in the bucket
s3_folder = (my_bucket, my_prefix)

# set up device
rigetti = AwsDevice("arn:aws:braket:::device/qpu/rigetti/Aspen-8")

# run circuit with a polling time of 5 days
rigetti_task = rigetti.run(circ, s3_folder, shots=10*n-1, poll_timeout_seconds=5*24*60*60)

# get id and status of submitted task
rigetti_task_id = rigetti_task.id
rigetti_status = rigetti_task.state()
# print('ID of task:', rigetti_task_id)
print('Status of task:', rigetti_status)
```

    Status of task: CREATED

```python
status = rigetti_task.state()
print('Status of (reconstructed) task:', status)
```

    Status of (reconstructed) task: COMPLETED

```python
result = rigetti_task.result()
print(f"measurement_counts: {result.measurement_counts}\n")

# Simulate partial measurement by seperating out first n bits
answer_plot = {}
for measresult in result.measurement_counts.keys():
    measresult_input = measresult[:len(str(s))]
    if measresult_input in answer_plot:
        answer_plot[measresult_input] += result.measurement_counts[measresult]
    else:
        answer_plot[measresult_input] = result.measurement_counts[measresult]

print(f"measurement_of_input_registers: {answer_plot}\n")
plt.bar(answer_plot.keys(), answer_plot.values())
plt.show()

```

```python
# Post-processing step
# Constructing the system of linear equations Y s = 0

lAnswer = [ (k,v) for k,v in answer_plot.items() if k != "0"*n  ] #excluding the trivial all-zero

#Sort the basis by their probabilities
lAnswer.sort(key = lambda x: x[1], reverse=True)

Y = []
for k, v in lAnswer:
    Y.append( [ int(c) for c in k ] )

#import tools from sympy
from sympy import Matrix, pprint, MatrixSymbol, expand, mod_inverse

Y = Matrix(Y)
print(Y)

#Perform Gaussian elimination on Y
Y_transformed = Y.rref(iszerofunc=lambda x: x % 2==0) # linear algebra on GF(2)

#to convert rational and negatives in rref of linear algebra on GF(2)
def mod(x,modulus):
    numer, denom = x.as_numer_denom()
    return numer*mod_inverse(denom,modulus) % modulus

Y_new = Y_transformed[0].applyfunc(lambda x: mod(x,2)) #must takecare of negatives and fractional values
print(Y_new)

print("The hidden bistring s[ 0 ], s[ 1 ]....s[",n-1,"] is the one satisfying the following system of linear equations:")
rows, cols = Y_new.shape
for r in range(rows):
    Yr = [ "s[ "+str(i)+" ]" for i, v in enumerate(list(Y_new[r,:])) if v == 1 ]
    if len(Yr) > 0:
        tStr = " + ".join(Yr)
        print(tStr, "= 0")
```

    Matrix([[0, 1, 0], [1, 1, 0], [1, 0, 0], [1, 0, 1], [0, 1, 1], [1, 1, 1]])
    Matrix([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 0, 0], [0, 0, 0], [0, 0, 0]])
    The hidden bistring s[ 0 ], s[ 1 ]....s[ 2 ] is the one satisfying the following system of linear equations:
    s[ 0 ] = 0
    s[ 1 ] = 0
    s[ 2 ] = 0

As seen above, the system of linear equations is satisfied by the unknown string $s$.


## bayestestimation basis

The bayestestimation module uses a hierachical Bayesian model to estimate the posterior distributions of two samples, the parameters of these samples can be approximated by simulation, as can the difference in the paramters.

#### Sections

- Specifying the hierachial model
- Estimating the posterior distribution
- Estimating the Bayes Factor

#### Specifying the hierachial model

The module largely follows the Bayesian-estimation-supercedes-the-t-test (BEST) implementation as specified by Kruschke ([link](https://pdfs.semanticscholar.org/dea6/0927efbd1f284b4132eae3461ea7ce0fb62a.pdf)).

Let $Y_A$ and $Y_B$ represent samples of continuous data from populations $A$ and $B$.  The distributions of $Y_A$ and $Y_B$ can be specified using the following hierachial model:

\begin{equation}
\begin{aligned}
Y_A &\sim \textrm{T}(\nu, \mu_A, \sigma_A)
\\
Y_B &\sim \textrm{T}(\nu, \mu_B, \sigma_B)
\\
\mu_A, \mu_B &\sim \textrm{N}(\mu, 2s)
\\
\sigma_A, \sigma_B &\sim \textrm{Inv-Gamma}(\alpha, \beta)
\\
\nu &\sim \textrm{Exp(+1)}(\phi)
\end{aligned}
\end{equation}

Where $\mu$, $s$, $\alpha$, $\beta$ and $\phi$ are constants. $\textrm{Exp(+1)}$ represents an exponential distribution shifted by +1.

Following Krusckhe, the default value for $\phi$ is 1/30.  Also following Kruschke, the default values for $\mu$ and $s$ are the sample mean of the combined samples of $Y_A$ and $Y_B$ ($\bar{Y}$), and the combined sample standard deviation of $Y_A$ and $Y_B$, respectively.

Deviating from Kruscke, the prior distributions of $\sigma_A$ and $\sigma_B$ are modelled using an inverse-gamma distribution.  The weakly-informative default values $\alpha$ and $\beta$ are set to 0.001.

#### Estimating the posterior distribution

Estimation of the posterior distributions of $\mu_A$, $\mu_B$, $\sigma_A$, $\sigma_B$ is carried out using [pystan's](https://pystan.readthedocs.io/en/latest/index.html) MCMC sampling.

The parameter $\mu_B - \mu_A$ can easily be estimated using the draws from the posteriors of $\mu_A$ and $\mu_B$.

#### Estimating the Bayes Factor

(watch this space)

```python

```


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.5.1 (Python 3.6.9-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

```python
#init_printing?
```

### Parameters and two Gaussians

```python
rc, a, a1, a2 = symbols('rc, a, a1 a2', positive=True, real=True)
```

```python
u1=x**2*exp(-a1*x**2)
u2=x**2*exp(-a2*x**2)
gp=x**4*exp(-a*x**2)
wc=-1/x
wb=exp(-4*rc**2/x**2) - 1
u1, u2, gp, wc, wb
```

### Radial wavefunction $u$
We don't solve the radial equation for $R(r)$, but rather for $u(r) = r \cdot R(r)$.
So, if $R$ is a s-function, $u$ is a p-function, and if $R$ is a p-function, $u$ is a d-function, etc.

$S = \int r^2dr R^2 = \int dr\, u^2$

$T = \int r^2dr R_1 \hat{T} R_2 = \int dr\, u_1\, \frac{d^2}{dr^2} \, u_2 $

### Normalization constant

```python
N=integrate(u1*u1, (x, 0, oo))
N
```

```python
1/sqrt(N)
```

```python
printing.sstrrepr(1/sqrt(N))
```

### Coulomb

```python
W=integrate(gp*wc, (x, 0, oo))
W
```

```python
Ws=W.simplify()
Ws
```

```python
printing.sstrrepr(Ws)
```

    '-1/(2*a**2)'

### Soft-box

```python
W=integrate(gp*wb, (x, 0, oo))
W
```

```python
Wf=W.factor()
Wf
```

```python
Ws=W.simplify()
Ws
```

```python
Ws.expand()
```

```python
printing.sstrrepr(Ws.expand())
```

    '-3*sqrt(pi)*rc*sinh(4*sqrt(a)*rc)/(2*a**2) + 3*sqrt(pi)*rc*cosh(4*sqrt(a)*rc)/(2*a**2) - 2*sqrt(pi)*rc**2*sinh(4*sqrt(a)*rc)/a**(3/2) + 2*sqrt(pi)*rc**2*cosh(4*sqrt(a)*rc)/a**(3/2) - 3*sqrt(pi)*sinh(4*sqrt(a)*rc)/(8*a**(5/2)) + 3*sqrt(pi)*cosh(4*sqrt(a)*rc)/(8*a**(5/2)) - 3*sqrt(pi)/(8*a**(5/2))'

```python
(cosh(a)-sinh(a)).simplify()
```

```python

```


```python
import sympy as sm
import sympy.physics.mechanics as me
me.init_vprinting()
```

```python
m, g, kt, kl, l = sm.symbols('m, g, k_t, k_l, l')
q1, q2, q3 = me.dynamicsymbols('q1, q2, q3')
u1, u2, u3 = me.dynamicsymbols('u1, u2, u3')

N = me.ReferenceFrame('N')
A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')

A.orient_axis(N, q1, N.z)
B.orient_axis(A, q2, A.x)

A.set_ang_vel(N, u1*N.z)
B.set_ang_vel(A, u2*A.x)

O = me.Point('O')
Ao = me.Point('A_O')
Bo = me.Point('B_O')

Ao.set_pos(O, l/2*A.x)
Bo.set_pos(O, l*A.x)

O.set_vel(N, 0)
Ao.v2pt_theory(O, N, A)
Bo.v2pt_theory(O, N, A)

Ao.vel(N), Bo.vel(N), A.ang_vel_in(N), B.ang_vel_in(N)
```

```python
Q = me.Point('Q')

Q.set_pos(Bo, q3*B.y)

Q.set_vel(B, u3*B.y)

Q.v1pt_theory(Bo, N, B)
```

```python
Ao.acc(N)
```

```python
Bo.acc(N)
```

```python
Q.acc(N)
```

```python
qd_repl = {q1.diff(): u1, q2.diff(): u2, q3.diff(): u3}
qd_repl
```

```python
Q.set_acc(N, Q.acc(N).xreplace(qd_repl))
Q.acc(N)
```

```python
A.ang_acc_in(N)
```

```python
B.ang_acc_in(N)
```

```python
v_Ao_1 = Ao.vel(N).diff(u1, N)
v_Ao_2 = Ao.vel(N).diff(u2, N)
v_Ao_3 = Ao.vel(N).diff(u3, N)

v_Ao_1, v_Ao_2, v_Ao_3
```

```python
v_Bo_1 = Bo.vel(N).diff(u1, N)
v_Bo_2 = Bo.vel(N).diff(u2, N)
v_Bo_3 = Bo.vel(N).diff(u3, N)

v_Bo_1, v_Bo_2, v_Bo_3
```

```python
v_Q_1 = Q.vel(N).diff(u1, N)
v_Q_2 = Q.vel(N).diff(u2, N)
v_Q_3 = Q.vel(N).diff(u3, N)

v_Q_1, v_Q_2, v_Q_3
```

```python
w_A_1 = A.ang_vel_in(N).diff(u1, N)
w_A_2 = A.ang_vel_in(N).diff(u2, N)
w_A_3 = A.ang_vel_in(N).diff(u3, N)

w_A_1, w_A_2, w_A_3
```

```python
w_B_1 = B.ang_vel_in(N).diff(u1, N)
w_B_2 = B.ang_vel_in(N).diff(u2, N)
w_B_3 = B.ang_vel_in(N).diff(u3, N)

w_B_1, w_B_2, w_B_3
```

```python
I = m*l**2/12
I_A_Ao = I*me.outer(A.y, A.y) + I*me.outer(A.z, A.z)
I_B_Bo = I*me.outer(B.x, B.x) + I*me.outer(B.z, B.z)

I_A_Ao, I_B_Bo
```

```python
F = me.dynamicsymbols('F')

R_Ao = m*g*N.x
R_Bo = m*g*N.x - F*B.y
R_Q = m/4*g*N.x + F*B.y

R_Ao, R_Bo, R_Q
```

```python
T_A = -kt*q1*N.z + kt*q2*B.x
T_B = -kt*q2*B.x

T_A, T_B
```

```python
Rs_Ao = -m*Ao.acc(N)
Rs_Bo = -m*Bo.acc(N)
Rs_Q = -m/4*Q.acc(N)
```

```python
Ts_A = -(me.dot(A.ang_acc_in(N), I_A_Ao) +
         me.dot(me.cross(A.ang_vel_in(N), I_A_Ao), A.ang_vel_in(N)))
Ts_A
```

```python
Ts_B = -(me.dot(B.ang_acc_in(N), I_B_Bo) +
         me.dot(me.cross(B.ang_vel_in(N), I_B_Bo), B.ang_vel_in(N)))
Ts_B
```

```python
me.find_dynamicsymbols(Ts_A, reference_frame=N)
```

```python
me.find_dynamicsymbols(Ts_B, reference_frame=N)
```

```python
Fr = sm.Matrix([
    v_Ao_1.dot(R_Ao) + v_Bo_1.dot(R_Bo) + v_Q_1.dot(R_Q) +
    w_A_1.dot(T_A) + w_B_1.dot(T_B),
    v_Ao_2.dot(R_Ao) + v_Bo_2.dot(R_Bo) + v_Q_2.dot(R_Q) +
    w_A_2.dot(T_A) + w_B_2.dot(T_B),
    v_Ao_3.dot(R_Ao) + v_Bo_3.dot(R_Bo) + v_Q_3.dot(R_Q) +
    w_A_3.dot(T_A) + w_B_3.dot(T_B),
])
Fr
```

```python
Frs = sm.Matrix([
    v_Ao_1.dot(Rs_Ao) + v_Bo_1.dot(Rs_Bo) + v_Q_1.dot(Rs_Q) +
    w_A_1.dot(Ts_A) + w_B_1.dot(Ts_B),
    v_Ao_2.dot(Rs_Ao) + v_Bo_2.dot(Rs_Bo) + v_Q_2.dot(Rs_Q) +
    w_A_2.dot(Ts_A) + w_B_2.dot(Ts_B),
    v_Ao_3.dot(Rs_Ao) + v_Bo_3.dot(Rs_Bo) + v_Q_3.dot(Rs_Q) +
    w_A_3.dot(Ts_A) + w_B_3.dot(Ts_B),
])
Frs
```

```python
kanes_eqs = Fr + Frs
kanes_eqs
```

```python
u = sm.Matrix([u1, u2, u3])
t = me.dynamicsymbols._t
ud = u.diff(t)
ud
```

```python
Md = Frs.jacobian(ud)
Md
```

```python
ud_zero_repl = {u1.diff(): 0, u2.diff(): 0, u3.diff(): 0}
ud_zero_repl
```

```python
gd = Frs.xreplace(ud_zero_repl) + Fr
gd
```

```python
res = sm.cse(Md.LUsolve(-gd))
```

```python
res[0]
```

```python
res[1]
```

```python

```


# Introduction to LAPM
LAPM is a python package for the analysis of linear autonomous pool (compartmental) models. It can be used to obtain a large set of different system-level diagnostics of compartmental models. This can be done either in symbolic or numeric form. In this notebook, we will introduce the basics of the package.

We assume that you have already installed the package following the instructions provided in the download page. After the package is installed, you can import it following the commands:

```python
from sympy import *
from LAPM import *
from LAPM.linear_autonomous_pool_model import LinearAutonomousPoolModel
```

In the second line above, we imported also the linear_autonomous_pool_model module which contains most of the functions required for the examples in this notebook.
We will create now a simple two-compartment model with the following syntax

```python
lambda_1, lambda_2, alpha, u_1, u_2 = symbols('lambda_1 lambda_2 alpha u_1 u_2', positive=True)
A = Matrix([[     -lambda_1,        0],
            [alpha*lambda_1, -lambda_2]])
u = Matrix(2, 1, [u_1, u_2])
```

Notice that we created a symblic version of the model with no assignment of values to the parameters. This is useful because we can make computations in symbolic form only, or we can assign parameter values later.
With the compartmental matrix and the input vector created above, we can now create a compartmental model as

```python
M=LinearAutonomousPoolModel(u, A)
```

## Symbolic computations
We can use now the set of functions available in LAPM with this model. For example, we can compute the mean system age as

```python
M.A_expected_value
```

    (alpha*u_1/lambda_2 + u_2/lambda_2)/(lambda_2*(alpha*u_1/lambda_2 + u_2/lambda_2 + u_1/lambda_1)) + u_1*(alpha/lambda_2 + 1/lambda_1)/(lambda_1*(alpha*u_1/lambda_2 + u_2/lambda_2 + u_1/lambda_1))

For an output easier to read on the screen

```python
pprint(M.A_expected_value)
```

         α⋅u₁   u₂               ⎛α    1 ⎞
         ──── + ──            u₁⋅⎜── + ──⎟
          λ₂    λ₂               ⎝λ₂   λ₁⎠
    ─────────────────── + ───────────────────
       ⎛α⋅u₁   u₂   u₁⎞      ⎛α⋅u₁   u₂   u₁⎞
    λ₂⋅⎜──── + ── + ──⎟   λ₁⋅⎜──── + ── + ──⎟
       ⎝ λ₂    λ₂   λ₁⎠      ⎝ λ₂    λ₂   λ₁⎠

Latex output can be obtained as

```python
print(latex(M.A_expected_value))
```

    \frac{\frac{\alpha u_{1}}{\lambda_{2}} + \frac{u_{2}}{\lambda_{2}}}{\lambda_{2} \left(\frac{\alpha u_{1}}{\lambda_{2}} + \frac{u_{2}}{\lambda_{2}} + \frac{u_{1}}{\lambda_{1}}\right)} + \frac{u_{1} \left(\frac{\alpha}{\lambda_{2}} + \frac{1}{\lambda_{1}}\right)}{\lambda_{1} \left(\frac{\alpha u_{1}}{\lambda_{2}} + \frac{u_{2}}{\lambda_{2}} + \frac{u_{1}}{\lambda_{1}}\right)}

You can copy and paste this Latex ouput to a markdown or latex document
$$
\frac{\frac{\alpha u_{1}}{\lambda_{2}} + \frac{u_{2}}{\lambda_{2}}}{\lambda_{2} \left(\frac{\alpha u_{1}}{\lambda_{2}} + \frac{u_{2}}{\lambda_{2}} + \frac{u_{1}}{\lambda_{1}}\right)} + \frac{u_{1} \left(\frac{\alpha}{\lambda_{2}} + \frac{1}{\lambda_{1}}\right)}{\lambda_{1} \left(\frac{\alpha u_{1}}{\lambda_{2}} + \frac{u_{2}}{\lambda_{2}} + \frac{u_{1}}{\lambda_{1}}\right)}
$$

## Numerical calculations
In most cases, we actually want to perform numerical computations of system-level diagnostics. In this case, we need first to assign values to the elements of the compartmental system. For instance, we can use the subs function to assign numerical values to existing matrices and vectors:

```python
u1=u.subs({u_1: 2, u_2: 4})
A1=A.subs({lambda_1: 0.8, lambda_2: 0.01, alpha: 0.13})
```

We take now these numerical arguments and create a new compartmental system and compute the mean system age as above

```python
M1=LinearAutonomousPoolModel(u1, A1)
M1.A_expected_value
```

    99.4997082847141

Other useful system diagnostics are:

```python
M1.A_standard_deviation # standard deviation of the system age distribution
```

    99.99249461290282

```python
M1.A_quantile(0.5) # Median (50% quantile) of the system age distribution
```

    68.80680584095865

```python
M1.T_expected_value #Mean transit time
```

    71.4166666666667

```python
M1.T_standard_deviation # standard deviation of the transit time distribution
```

    95.45435936730298

```python
M1.T_quantile(0.5) # Median (50% quantile) of the transit time distribution
```

    35.14291412333978

```python
M1.a_expected_value # Mean age vector of individual pools
```

    Matrix([
    [            1.25],
    [100.076291079812]])

```python
M1.a_quantile(0.5) # Median age of individual pools
```

    array([ 0.86643398, 69.39194502])

```python
M1.T_laplace
```

    0.232/(s + 0.8) + 0.00666666666666667/(s + 0.01) + 0.000346666666666667/((s + 0.01)*(s + 0.8))

```python
M1.A_laplace
```

    0.00406067677946324/(s + 0.8) + 0.0099416569428238/(s + 0.01) + 6.06767794632439e-6/((s + 0.01)*(s + 0.8))

```python
M1.r_compartments # release flux of individual compartments
```

    Matrix([
    [1.74],
    [4.26]])

```python
M1.r_total # Total release flux
```

    6.00000000000000

```python
M1.entropy_per_jump
```

    -0.32626427406199*log(2) + 0.489396411092985*log(3) + 2.21020273307626

```python
M1.entropy_per_cycle
```

    -0.666666666666667*log(2) + 1.0*log(3) + 2.95715004792764

```python
M1.entropy_rate
```

    -0.0865384615384615*log(2) + 0.129807692307692*log(3) + 0.383860823529068

```python

```


# Ewald summation

This notebook provides a detailed look at classical electrostatics, demonstrating problems related to convergence and how to overcome those issues.  The techniques discussed are ubiquitous in periodic simulations, both classical and quantum.

```python
%matplotlib notebook
from mpl_toolkits.mplot3d import Axes3D
import matplotlib
import numpy as np
import functools
import scipy.special
import itertools
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from ipywidgets import interact, interactive, fixed, FloatSlider, widgets
```

## Classical electrostatics

Molecular simulations of periodic systems and crystals are an extremely important area of study.  To demonstrate the difficulties, and some popular methods to overcome them, we'll investigate a simple system of unit charges.  Central to our discussion is the potential $\phi$ at a point $\mathbf{R}$ generated by a Gaussian charge density centered at $\mathbf{R}_0$ with exponent $\alpha$

\begin{equation}
    q G(\alpha, \mathbf{R}_0, \mathbf{R}) \equiv q \left(\frac{\alpha}{\sqrt{\pi}}\right)^3
        e^{-\alpha^2(\mathbf{R}-\mathbf{R}_0)^2},
\end{equation}

which can be obtained as the solution of Poisson's equation:

\begin{equation}
    \nabla^2 \phi(\alpha, \mathbf{R}) = -\frac{q G(\alpha, \mathbf{0}, \mathbf{R})}{\epsilon_0}.
\end{equation}

The above involves the permittivity of free space $\epsilon_0$ and we have assumed the Gaussian is placed at the origin.  The Laplacian in spherical coordinates is

\begin{align}
    \nabla^2 & = \frac{1}{r^2}\frac{\partial}{\partial r}\left(r^2\frac{\partial}{\partial r}\right)
    +\frac{1}{r^2 \sin^2\phi} \frac{\partial^2}{\partial \theta^2}
    +\frac{1}{r^2 \sin \theta}\frac{\partial}{\partial \phi}\left(\sin \phi \frac{\partial}{\partial \phi}\right),
\end{align}

but only the leading term will survive when applied to a spherically symmetric density.  Recognizing that $\frac{1}{r^2}\frac{\partial}{\partial r}\left(r^2\frac{\partial}{\partial r}\right)f = \frac{1}{r}\frac{\partial^2}{\partial r^2}(rf)$ Poisson's equation for a Gaussian potential is

\begin{align}
    \frac{1}{r}\frac{\partial^2}{\partial r^2}\left[r \phi(\alpha, \mathbf{R})\right]
        &= -\frac{q G(\alpha, \mathbf{0}, \mathbf{R})}{\epsilon_0} \\
    \frac{\partial^2}{\partial r^2}\left[r \phi(\alpha, \mathbf{R})\right]
        &= -\frac{r q G(\alpha, \mathbf{0}, \mathbf{R})}{\epsilon_0} \\
    \frac{\partial}{\partial r}\left[r \phi(\alpha, \mathbf{R})\right]
        &= -\int_r^\infty \frac{r q G(\alpha, \mathbf{0}, \mathbf{R})}{\epsilon_0}\mathrm{d}r
            = \frac{q G(\alpha, \mathbf{0}, \mathbf{R})}{2\alpha^2\epsilon_0} \\
    r \phi(\alpha, \mathbf{R})
        &= \int_0^r \frac{q G(\alpha, \mathbf{0}, \mathbf{R})}{2\alpha^2\epsilon_0}\mathrm{d}r \\
    \phi(\alpha, \mathbf{R})
        &= \frac{q \mathrm{Erf}(\alpha r)}{4 \pi \epsilon_0 r},\\
\end{align}

which introduces the definition of the error function:
\begin{equation}
    \mathrm{Erf}(z) \equiv \frac{2}{\sqrt{\pi}}\int_0^z{-t^2}\mathrm{d}t
\end{equation}

and the scalar distance $r=|\mathbf{R}-\mathbf{R}_0|$.
Recognizing that the infinite exponent limit of a Gaussian is simply a delta function, we get a very familiar result

\begin{equation}
    \lim_{\alpha\rightarrow\infty}\frac{q \mathrm{Erf}(\alpha r)}{4 \pi \epsilon_0r} = \frac{q}{4 \pi \epsilon_0 r}
\end{equation}

*i.e.* the potential at a distance $r$ from a point charge $q$ is $\phi(r) = \frac{q}{4 \pi \epsilon_0 r}$.  Placing a test charge $Q$ that same distance away then gets us to Coulomb's law:

\begin{equation}
    U = Q\phi(r) = \frac{Qq}{4 \pi \epsilon_0 r}.
\end{equation}

If we want to embed a unit cell in an infinite number of replica ("image") unit cells in all directions, we just need to sum over all interactions within the primary unit cell, and over all primary-image cell interactions.  The unit cell is described by a set of vectors, which for the orthorhombic case with lengths $a,b,c$ are simply

$$
    \mathbf{a} = \left(\begin{eqnarray} &a&0&0\\ &0&b&0\\ &0&0&c\\ \end{eqnarray}\right).
$$

With this notation, and labeling atoms $\{i,j\}$ possessing charges $\{q_i,q_j\}$, image cells can be compactly denoted by a vector $\mathbf{n} = \{n_1 \mathbf{a}_1, n_2 \mathbf{a}_2, n_3 \mathbf{a}_3\}$, where $n_1, n_2$ and $n_3$ are integers:

\begin{equation}
    U = \frac{1}{8 \pi \epsilon_0}\sum_{\mathbf{n}'}\sum_{i,j}\frac{q_i q_j}
            {\left|\mathbf{r}_i - \mathbf{r}_j + \mathbf{n}\right|}
      = \frac{1}{8 \pi \epsilon_0}\sum_{\mathbf{n}'}\sum_{i,j}\frac{q_i q_j}{r_{ij}}
\end{equation}

The prime on the summation of over image cells is a reminder that $\mathbf{n}=\mathbf{0}$ terms where $i=j$ should be omited and we have introduced the shorthand notation $r_{ij}=\left|\mathbf{r}_i - \mathbf{r}_j + \mathbf{n}\right|$.  Implementing electrostatics by brute force summation over many lattice vectors is not a good idea; to see why we'll take a brief diversion.

## Conditional convergence
To get a feel for conditional convergence consider the classic example of the alternating harmonic series:

\begin{equation}
    \sum_{n=0}^\infty \frac{(-1)^{n+1}}{n} = \left(1 - \frac{1}{2} + \frac{1}{3}
        - \frac{1}{4} + \frac{1}{5} - \frac{1}{6} + \frac{1}{7} - \frac{1}{8} + \ldots\right) = \ln(2).
\end{equation}

Summing the terms in ascending order yields the result show.  However, we can also reorder the terms in parentheses before summing

\begin{equation}
    \underbrace{1 - \frac{1}{2}}_{\frac{1}{2}} -
    \underbrace{\frac{1}{4}}_{\frac{1}{4} } +
    \underbrace{\frac{1}{3} - \frac{1}{6} }_{\frac{1}{6}} -
    \underbrace{\frac{1}{8}}_{\frac{1}{8} } +
    \underbrace{\frac{1}{5} - \frac{1}{10} }_{\frac{1}{10}} -
    \underbrace{\frac{1}{12}}_{\frac{1}{12} } +
    \underbrace{\frac{1}{7} - \frac{1}{14} }_{\frac{1}{14}} -
    \underbrace{\frac{1}{16}}_{\frac{1}{16} } +
    \ldots,
\end{equation}

which yields exactly half of what we got above:

\begin{equation}
    \frac{1}{2}\left(1 - \frac{1}{2} + \frac{1}{3} - \frac{1}{4} + \frac{1}{5} - \frac{1}{6}
        + \frac{1}{7} - \frac{1}{8} + \ldots\right) = \frac{1}{2} \ln(2).
\end{equation}

There exists some specific permutation of the summands that will generate any given result when the sum is conditionally convergent.  A sum over some series $a_n$ is conditionally convergent if $\sum_{n=1}^\infty |a_n| = \infty$.  Returning to electrostatics, we introduce the integral analog of the potential expression above, avoiding the singularity by neglecting the infinitesimal point $\mathbf{r}=\mathbf{0}$:
$$\int_{r>0} \frac{1}{|\mathbf{r}|} \mathrm{d}^3\mathbf{r} = 4\pi \int_{\delta}^\infty \frac{r^2}{r} \mathrm{d}r = \infty$$
The factor of $r^2$ comes from the conversion to polar coordinates.  We find that

\begin{equation}
    \int_{\delta}^\infty \frac{1}{r^{n-2}} \mathrm{d}r =
    \begin{cases}
    \frac{r_c^{3-n}}{n-3}, & \text{if}\ n>3 \\
    \infty, & \text{otherwise}
    \end{cases}
\end{equation}

so, by analogy to the integrals, the Coulomb summation is conditionally convergent in three dimensional space.  We don't arrive at an *absolutely* convergent series until we deal with an operator that decays as $\frac{1}{r^4}$ in three dimensions.  Not only is the Coulombic summation conditionally convergent, its convergence is very slow w.r.t. increasing cutoffs due to the slow decay of the operator.

The purpose of this notebook is to demonstrate a clever trick to decompose the slow, conditionally convergent summation into two rapid, absolutely convergent terms.  We use kJ/mol for energies, nm for length and atomic units for charge throughout.

```python
# Setup up a cubic simulation box
atoms_per_dim = 5
natoms = atoms_per_dim**3
epsilon0 = 5.72765E-4 # (e^2 mol) / (kJ nm)
coulomb = 1/(4*np.pi*epsilon0)
boxlength = 0.8 #nm
# Initial coordinates; regular lattice
spacing = np.linspace(-boxlength/2.0, boxlength/2.0, atoms_per_dim, endpoint=False)
spacing += boxlength/(2.0*atoms_per_dim)
# Center the array
coords = np.array(list(itertools.product(spacing, spacing, spacing)))
# Move the atoms very slightly away from their regular grid positions
np.random.seed(0)
coords += 0.2*(np.random.rand(natoms,3)-0.5)
coords[coords>boxlength/2.0] = boxlength/2.0
coords[coords<-boxlength/2.0] = -boxlength/2.0
charges = np.zeros(len(coords))
if 1:
    # Generate a large dipole
    charges[:natoms//2] = 1.0
    charges[(natoms+1)//2:] = -1.0
else:
    # Generate a more homogeneous distribution
    charges[1:natoms:2] = -1.0
    charges[0:natoms-1:2] = 1.0
poscrd = coords[charges > 0]
negcrd = coords[charges < 0]
neucrd = coords[charges == 0]
print("Sum of charges", np.sum(charges))
# Visualize the coordinates and charges in a rotatable 3D plot.  Atoms are colored according to their charge.
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(poscrd[:,0], poscrd[:,1], poscrd[:,2], 'bo')
ax.scatter(negcrd[:,0], negcrd[:,1], negcrd[:,2], 'ro')
ax.scatter(neucrd[:,0], neucrd[:,1], neucrd[:,2], 'ko')
ax.set_xlim(-boxlength/2.0, boxlength/2.0)
ax.set_ylim(-boxlength/2.0, boxlength/2.0)
ax.set_zlim(-boxlength/2.0, boxlength/2.0)
ax.set_xticks([])
ax.set_yticks([])
ax.set_zticks([])
plt.show()
```

```python
# Define the lattice energy using brute force summation, and evaluate over a range of lattice cutoffs.

def coulomb_energy(qij, R):
    """ Evaluate the conventional Coulomb energy, given products of charges and
        distances between the centers bearing those charges. """
    return np.sum(qij/R)

def lattice_sum(charges, coords, nboxes, efun):
    """ Compute the Coulomb energy of a system of charges, by considering interactions
        between the primary box and all image boxes that are within nboxes of the
        primary cell. The energy is evaluated by the provided function, efun. """
    natoms = coords.shape[0]
    boxlist = range(-nboxes,nboxes+1)

    # Build some lists to describe interactions
    # products of charges between two boxes
    chargeij = np.prod(list(itertools.product(charges, charges)), axis=1)
    dRij = np.repeat(coords, natoms, axis=0) - np.tile(coords, (natoms,1))
    energy = 0.0
    truncated_energy = 0.0
    for nvecs in itertools.product(boxlist, boxlist, boxlist):
        nvecs = np.array(nvecs)
        shiftvec = nvecs*boxlength
        absvecs = np.abs(nvecs)
        if nvecs.any():
            # This is an image box
            shiftedRij = np.linalg.norm(dRij+shiftvec, axis=1)
            eterm = 0.5*efun(qij=chargeij, R=shiftedRij)
        else:
            # Handle the 'home' box
            eterm = 0.0
            for i in range(natoms):
                for j in range(i):
                    dR = np.linalg.norm(coords[i]-coords[j])
                    eterm += efun(qij=charges[i]*charges[j], R=dR)
        eterm *= coulomb
        if np.linalg.norm(absvecs) <= nboxes:
            truncated_energy += eterm
        energy += eterm
    return energy,truncated_energy

vals = []
for n in range(7):
    vals.append(lattice_sum(charges, coords, n, efun=coulomb_energy))
vals = np.array(vals)

# Plot the convergence of the energy w.r.t. cutoff using the two truncation schemes
fig = plt.figure()
ax = fig.add_subplot(111)
xvals = range(1,len(vals))
ax.plot(xvals, vals[1:,0], 'bo', label ='Cubic')
ax.plot(xvals, vals[1:,1], 'ro', label = 'Spherical')
ax.set_xticks(xvals)
ax.set_xlabel('Number of image box layers')
ax.set_ylabel('Energy (kJ/mol)')
ax.legend(loc='lower right')
plt.show()

# Check the reference values
assert np.allclose(vals,
[[ 361515.2359571,   361515.2359571 ],
 [ 528282.46725449,  557057.25818972],
 [ 534335.79047581,  536496.90616012],
 [ 535962.70789396,  536005.76078745],
 [ 536633.65606731,  537475.71261986],
 [ 536973.60561882,  537518.58787992],
 [ 537169.21808044,  537527.68088663]])
```

The above code sets up a cell and performs the summation in two different ways 1) adding a complete layer of image boxes, which corresponds to a cubic macroscopic crystal and 2) adding image boxes in such a way that the macroscopic system is roughly spherical.  The convergence is both slow and conditional; for this contrived example of a large net dipole the energies for the two summation schemes converge to slightly different results.  When the second layer of vectors is added, we are introducing interaction up to three times the box length and the results are still not fully converged!

To speed up the convergence, Ewald summation introduces Gaussian functions at the atomic positions, with the opposite sign to a given atom's charge, to screen the point charges, making them short range.  To compensate, a Gaussian with the same sign as the charge is added in, but is handled separately.  From the above arguments, this affects the potential by splitting the Coulomb operator into short- and long-range components:

\begin{align}
    \frac{1}{r} &= \frac{1-\mathrm{Erf}(\alpha r)}{r} + \frac{\mathrm{Erf}(\alpha r)}{r}\\
                &= \frac{\mathrm{Erfc}(\alpha r)}{r} + \frac{\mathrm{Erf}(\alpha r)}{r}
\end{align}

In the plot below we show the density (left) where point charges and their screening Gaussians are shown in blue; the resulting $\mathrm{Erfc}$ potential is shown in the right panel.  Increasing the attenuation parameter $\alpha$, which is inversely proportional to the width of the Gaussians, screens this short range potential more effectively, but also increases the overall contribution of the long-range potential (red) that results from the compensating Gaussians.  The rapid decay of the short-range term means we can evaluate it in the standard real-space treatment using pairwise loops with short cutoffs. The red part is still long-range so will still decay slowly; however it is now free from the singularity that plagues the full (black) potential, and we can develop a treatment for it.

```python
# Define some point charges to demonstrate the density and potential
positions = [1, 3, 5]
qvals  = [1, -1, 1]

def plot_gaussians(gausplot, erfplot, alpha=3):
    """ Generate a plot that shows point charges in a 1D coordinates system, whose
        density is represented by blue delta functions, with blue screening Gaussians
        introduced to attenuate these point charges.  The Gaussian is also subtracted
        out, which is shown in red, to make the method exact. The potential corresponding
        to the regular Coulomb operator (black) is also shown, decomposed into its short
        range (blue) component and its long range (red) term. """
    prefac = alpha/np.sqrt(np.pi) # 1D Gaussian normalization
    def plot_gaussian(axis, xvals, origin, exponent, prefac, col):
        axis.plot(xvals, prefac*np.exp(-np.square(exponent*(xvals-origin))), color=col)
    npts = 150
    xvals = np.linspace(0.001, 6, npts)
    gausplot.cla()
    erfplot.cla()
    for p,c in zip(positions, qvals):
        plot_gaussian(gausplot, xvals, p, alpha,  c*prefac, 'r')
        plot_gaussian(gausplot, xvals, p, alpha, -c*prefac, 'b')
        gausplot.plot([p, p], [0, c], color='b')
    gausplot.plot([0, 40], [0, 0], color='k', linestyle='-', linewidth=2)
    erfvals = scipy.special.erf(alpha*xvals)
    inv = 1/xvals
    erfplot.set_title('Potential')
    erfplot.plot(xvals, erfvals*inv, color='r', label='Erf(alpha R) / R')
    erfplot.plot(xvals, (1-erfvals)*inv, color='b', label='Erfc(alpha R) / R')
    erfplot.plot(xvals, inv, '--', color='k', label='1 / R')
    erfplot.set_xlabel('r (nm)')
    erfplot.set_ylabel('$\\phi(r)$')
    erfplot.legend()
    gausplot.set_title('Density')
    gausplot.set_xlim(0, 6)
    gausplot.set_ylim(-5,5)
    gausplot.set_ylabel('$\\rho(r)$')
    gausplot.set_xlabel('Position')
    erfplot.set_xlim(0, 1.5)
    erfplot.set_ylim(0, 10)
    fig.canvas.draw()

# Draw the plots with a default alpha value.  Run the box below
# to get a slider to vary alpha and update the plots in real time.
fig, (gausplot, erfplot) = plt.subplots(1,2,figsize=(8,4))
plot_gaussians(gausplot, erfplot)
```

```python
# Evaluate this cell, then play with the sliders to see the effect of
# varying alpha on the long and short range densities and potentials.
interact(plot_gaussians, gausplot=fixed(gausplot), erfplot=fixed(erfplot),
         alpha=widgets.FloatSlider(min=0,max=9,step=.2,value=3),);
```

Looking at the form of the long-range density in the left panel of the above plot, it is a periodic, smoothly oscillating function so any solution is likely to involve trigonometric functions somehow.  Returning to the discussion at the top of the notebook, we remember that the potential comes from solving the Poisson equation:

\begin{equation}
    \nabla^2 \phi(\alpha, \mathbf{R}) = -\frac{q G(\alpha, \mathbf{0}, \mathbf{R})}{\epsilon_0}.
\end{equation}

Combining these considerations, it's easy to imagine that the trigonometric functions will be introduced via complex exponentials; because $\frac{\partial}{\partial x}e^{-imx} = -im e^{-imx}$, introducing an exponential will effectively turn the derivative operator in the Poisson equation into a constant, making it easy to handle.  As a reminder, the connection between complex exponentials and trigonometric functions is provided by Euler's formula: $e^{ix} = \mathrm{cos}(x)+i\mathrm{sin}(x)$.  The transformation that will introduce the complex exponential is the Fourier transform.

First, we normalize the coordinates into the range $[0,1]$, appropriate for integration, by introducing the reciprocal lattice vectors $\mathbf{a}^*$, which are defined in terms of the unit cell vectors by the orthonormality relation $\mathbf{a}_i^*\cdot\mathbf{a}_j=\delta_{ij}$ and for the orthorhombic unit cell are

$$ \mathbf{a}^* = \left(\begin{eqnarray} &\frac1a&0&0\\ &0&\frac1b&0\\ &0&0&\frac1c\\ \end{eqnarray}\right). $$

The resulting reciprocal space coordinate system has orthogonal axes, even for non-orthorhombic crystals.  Analogous to $\mathbf{n}$ indexing summation over image unit cells, we use $\mathbf{m}$ to index summation over *reciprocal* space lattice vectors, $\mathbf{m} = \{m_1 \mathbf{a}_1^*, m_2 \mathbf{a}_2^*, m_3 \mathbf{a}_3^*\}$; because we're evaluating the energy using these vectors, we call the long range energy the *reciprocal* energy, $U_\mathbf{rec}$.

The long-range density $\rho^L(\mathbf{r})$ and potential $\phi^L(\mathbf{r})$ have Fourier transforms defined by

\begin{align}
    \tilde\rho^L(\mathbf{m}) &= \int_V \rho^L(\mathbf{r}) e^{-2\pi i\mathbf{m}\cdot\mathbf{r}} \mathrm{d}^3\mathbf{r}\\
    \tilde\phi^L(\mathbf{m}) &= \int_V \phi^L(\mathbf{r}) e^{-2\pi i\mathbf{m}\cdot\mathbf{r}} \mathrm{d}^3\mathbf{r}
\end{align}

and the inverse transforms are defined by summation over reciprocal lattice vectors $\mathbf{m}$

\begin{align}
    \rho^L(\mathbf{r}) &= \frac1V \sum_{\mathbf{m}}\tilde\rho^L(\mathbf{m}) e^{2\pi i\mathbf{r}\cdot\mathbf{m}}\\
    \phi^L(\mathbf{r}) &= \frac1V \sum_{\mathbf{m}}\tilde\phi^L(\mathbf{m}) e^{2\pi i\mathbf{r}\cdot\mathbf{m}}
\end{align}

Inserting $\tilde\rho^L(\mathbf{m})$ into Poisson's equation yields the Fourier space expression

\begin{equation}
    4\pi^2m^2 \tilde\phi^L(\mathbf{m}) = \frac{\tilde\rho^L(\mathbf{m})}{\epsilon_0}
\end{equation}

whose solution is trivial if we neglect the $\mathbf{m}=\mathbf{0}$ term, which is zero for neutral unit cells with $\sum_i q_i = 0$. We define the scalar $m=|\mathbf{m}|$.  The reciprocal space long-range density is obtained from the Fourier transform:

\begin{align}
    \tilde\rho(\mathbf{m}) &= \int_V \sum_\mathbf{n}\sum_j q_j
        G(\alpha, \mathbf{r}_j, \mathbf{r+\mathbf{n}})
        e^{-2\pi i \mathbf{r}\cdot\mathbf{m}}\mathrm{d}\mathbf{r}^3 \\
     &=\sum_\mathbf{n}\sum_j q_j  \int_\mathrm{all\ space}
         G(\alpha, \mathbf{r}_j, \mathbf{\mathbf{w}})
         e^{-2\pi i \mathbf{(w-n)}\cdot\mathbf{m}}\mathrm{d}\mathbf{w}^3 \\
     &=\sum_\mathbf{n}\sum_j q_j  \int_\mathrm{all\ space}
         G(\alpha, \mathbf{r}_j, \mathbf{\mathbf{w}})
         e^{-2\pi i \mathbf{w}\cdot\mathbf{m}}\mathrm{d}\mathbf{w}^3 \\
     &= \sum_j q_j e^{-\frac{m^2\pi^2}{\alpha^2}}
         e^{-2\pi i \mathbf{r}_j\cdot\mathbf{m}}.
\end{align}

In developing this expression we first change variables $\mathbf{w}=\mathbf{r+n}$ before taking advantage of the fact that $e^{2\pi i \mathbf{n}\cdot\mathbf{m}} = 1$. The integral on the third line is the definition of the Fourier transform of a 3D Gaussian, which is evaluated for each dimension using the standard result $\int e^{a^2x^2}e^{-2\pi i m x}\mathrm{d}x = \frac{\sqrt{\pi}}{a}e^{-\frac{\pi^2m^2}{a^2}}$.  With the Fourier space expression for the density in hand, we can now get the Fourier space potential from the simplified Poisson equation:

\begin{equation}
    \tilde\phi(\mathbf{m}) = \frac{1}{4\pi^2\epsilon_0}
    \frac{e^{-\frac{m^2\pi^2}{\alpha^2}}}{m^2} \sum_j q_j  e^{-2\pi i \mathbf{r}_j\cdot\mathbf{m}}
\end{equation}

This is inverse Fourier transformed to obtain the real space potential at some point $\mathbf{r}$:

\begin{equation}
    \phi(\mathbf{r}) = \frac{1}{4\pi^2\epsilon_0 V}
        \sum_{\mathbf{m}\ne\mathbf{0}}\frac{e^{-\frac{m^2\pi^2}{\alpha^2}}}{m^2}
        \sum_j q_j  e^{-2\pi i \mathbf{r}_j\cdot\mathbf{m}}
        e^{2\pi i \mathbf{r}\cdot\mathbf{m}}
\end{equation}

And finally, as above, we evaluate the potential at each atom, multiply by the charge of that atom, and divide by 2 to avoid double-counting:

\begin{align}
  U_\mathrm{rec} &= \frac{1}{8\pi^2\epsilon_0 V}
      \sum_{\mathbf{m}\ne\mathbf{0}}\frac{e^{-\frac{m^2\pi^2}{\alpha^2}}}{m^2}
      \sum_{i} q_i e^{2\pi i \mathbf{r}_i\cdot\mathbf{m}}
      \sum_{j} q_j  e^{-2\pi i \mathbf{r}_j\cdot\mathbf{m}} \\
   &= \frac{1}{8\pi^2\epsilon_0 V}\sum_{\mathbf{m}\ne\mathbf{0}}
       \frac{e^{-\frac{m^2\pi^2}{\alpha^2}}}{m^2}
       S(\mathbf{m})
       S(-\mathbf{m})
\end{align}

On the surface, this doesn't seem like a simplification.  However, the presence of the factor $\frac{e^{-\frac{m^2\pi^2}{\alpha^2}}}{m^2}$ leads to a rapidly convergent summation involving a limited number of reciprocal lattice vectors $\mathbf{m}$.  The quantity $S(\mathbf{m})$ is known as a structure factor.  The only remaining issue to deal with is the fact that the real space term neglects the $i=j$ term in the primary unit cell summation, but the reciprocal space energy contains this term as a result of the reciprocal space potential containing contributions from all atoms.  The spurious term that we need to remove is the interaction of the probe charge with its own contribution to the reciprocal space potential, which comes from its screening Gaussian:

\begin{split}
    U_\mathrm{self} &= -\sum_i \lim_{r\rightarrow0} \frac{q_i \mathrm{Erf}(\alpha r)q_i}{4 \pi \epsilon_0 r} \\
   &= -\frac{1}{4 \pi \epsilon_0}\frac{\alpha}{\sqrt\pi}\sum_i q_i^2
\end{split}

In the code below, we investigate the effect of introducing the screening to the short-range *direct space* (as opposed to reciprocal space, used to evaluate the long-range term) energy:

\begin{equation}
    U = \frac{1}{2}\sum_{\mathbf{n}'}\sum_{i,j}
        \frac{q_i q_j\mathrm{Erfc}(\alpha\left|\mathbf{r}_i - \mathbf{r}_j + \mathbf{n}\right|)}
                                {\left|\mathbf{r}_i - \mathbf{r}_j + \mathbf{n}\right|}.
\end{equation}

After choosing an attenuation parameter $\alpha=4 \mathrm{nm}^{-1}$ we repeat the lattice summation shown above:

```python
def ewald_dir_energy(alpha, maxn):
    """ An energy function, to be passed to the lattice sum code, that
        evaluated the direct contribution to the Ewald energy. """
    # Define the direct space energy as the attenuated direct Ewald energy.
    def ewald_dir_interaction(qij, R, alpha):
        """ Evaluate the energy of pairs of charges, separated by R,
            using the attenuated Coulomb operator. """
        return np.sum(qij*scipy.special.erfc(alpha*R)/R)
    enerfunc = functools.partial(ewald_dir_interaction, alpha=alpha)
    return lattice_sum(charges, coords, n, efun=enerfunc)

vals = np.array([ewald_dir_energy(alpha=4, maxn=n) for n in range(4)])

# Plot the convergence of the lattice sum, with respect to number of
# image cells included, using cubic and spherical layering schemes.
fig = plt.figure()
ax = fig.add_subplot(111)
xvals = range(1,len(vals))
ax.plot(xvals, vals[1:,0], 'bo', label ='Cubic')
ax.plot(xvals, vals[1:,1], 'ro', label = 'Spherical')
ax.set_xticks(xvals)
ax.set_xlabel('Number of image box layers')
ax.set_ylabel('Energy (kJ/mol)')
ax.legend()

plt.show()
```

Now both summation schemes converge rapidly to the same results, showing that the direct space part of the lattice sum is well-behaved.  Implementing the reciprocal and self terms, we find that the overall Ewald energy is also well-behaved:

```python
def ewald_rec_energy(alpha, maxm):
    """ Evaluate the long range (reciprocal space) Ewald energy using an
        attenuation parameter of alpha, and using up to maxm wavevectors. """
    mdim = 2*maxm+1
    twopii = np.pi*2j
    pi2_a2 = np.pi**2/alpha**2 if alpha != 0.0 else 1e50
    miter = [ 0 ]
    for m in range(1,maxm+1):
        miter.extend([m,-m])
    miter = np.array(miter)
    mvals = miter / boxlength
    xvecs = np.exp(twopii*np.outer(coords[:,0], mvals))
    yvecs = np.exp(twopii*np.outer(coords[:,1], mvals))
    zvecs = np.exp(twopii*np.outer(coords[:,2], mvals))
    Sm = np.einsum('n,nx,ny,nz->xyz', charges, xvecs, yvecs, zvecs)
    m = np.array(list(itertools.product(mvals, mvals, mvals)))
    m2 = np.einsum('nx,nx->n', m, m).reshape(mdim,mdim,mdim)
    prefac = np.exp(-pi2_a2*m2)
    m2[0,0,0] = 1e50 # avoid division by 0
    prefac /= m2
    V = boxlength**3
    return coulomb*np.einsum('xyz,xyz', prefac, np.square(Sm.real) + np.square(Sm.imag)) / (2*np.pi*V)

def ewald_self_energy(alpha):
    """ Evaluate the Ewald self interaction energy. """
    return -coulomb*(alpha/np.sqrt(np.pi))*np.sum(np.square(charges))

def ewald_energy(alpha, maxn, maxm):
    """ Evaluate all components of the Ewald sum, to get the total energy. """
    Udir = ewald_dir_energy(alpha, maxn)[0]
    Urec = ewald_rec_energy(alpha, maxm)
    Uslf = ewald_self_energy(alpha)
    U = Udir+Urec+Uslf
    return Udir,Urec,Uslf,U

# A list of alpha, maxn, maxm to loop over
params = np.array([
    [0, 4, 2],
    [1, 3, 2],
    [2, 2, 2],
    [3, 2, 2],
    [4, 1, 3],
    [5, 1, 3],
    [6, 1, 4],
])
params = np.array([
    [0, 4, 2],
    [1, 3, 3],
    [2, 2, 4],
    [3, 2, 4],
    [4, 1, 5],
    [5, 1, 5],
    [6, 1, 6],
])

# Generate Ewald energies, varying the attentuation paramter alpha,
# as well as the real and recpiprocal space cutoffs needed to reach
# convergence for each alpha value.
Uvals = np.array([ewald_energy(*p) for p in params])

# Plot the total energy, and its components, as a function of the
# attenuation parameter alpha.
fig = plt.figure()
ax = fig.add_subplot(111)
alphavals = params[:,0]
ax.plot(alphavals, Uvals[:,0], '--r', label = 'Udir')
ax.plot(alphavals, Uvals[:,1], '--b', label = 'Urec')
ax.plot(alphavals, Uvals[:,2], '--g', label = 'Uslf')
ax.plot(alphavals, Uvals[:,3], 'k', label = 'U')
ax.set_xticks(alphavals)
ax.set_xlabel('Alpha (1/nm)')
ax.set_ylabel('Energy (kJ/mol)')
ax.legend()

plt.show()

# Confirm that all high alpha values have the same total energy
assert np.allclose(Uvals[2,3], Uvals[3:,3])
```

For almost any nonzero screening parameter, the total energy is the same, although its composition in terms of reciprocal, direct and self terms changes; this useful fact is very handy when testing the correctness of an Ewald implementation.  For small $\alpha$ more $\mathbf{n}$ vectors are needed, as the short-range operator is not well attenuated, while a large $\alpha$ demands more $\mathbf{m}$ vectors to evaluate the reciprocal space term; this is evident from the values chosen in the `params` array above.  Because cutoffs can be used for the direct space term, it scales as $\mathcal{O}(N)$.  Being attenuated, the direct space term can be evaluated within the primary unit cell, using the minimum image convention.

Constructing the stucture factor $S(\mathbf{m})$ involves a summation over $N$ atoms as well as $\mathbf{m}$; for a given $\alpha$ value the latter scales with the volume of the box and is therefore proportional to $N$.  Overall, the Ewald reciprocal energy therefore scales as $\mathcal{O}(N^2)$, although it can be shown that optimally tuning $\alpha$ to minimize computational cost yields a $\mathcal{O}(N^{{3/2}})$ algorithm.  A clever trick introduced by Darden, York and Pedersen in 1993 reduces this cost further, as we will now demonstrate.

The key insight is that a discrete Fourier transform can be performed using a fast recursion algorithm if the atoms are arranged on a uniform mesh, hence the name particle mesh Ewald (PME).  A method that only works for selected configurations would be useless, so PME uses spline interpolation to effectively coerce the atoms to predefined, regularly spaced locations.  Consider some function $f$ we've tabulated at every integer.  If we want either $f(12)$ or $f(13)$, we're in luck and can just look up the tabulated values.  However, $f(12.5)$ would come from $\frac{f(12)+f(13)}{2}$, while $f(12.1)$ would be best approximated with two points by $0.9f(12) + 0.1f(13)$. This simple order 2 example can be generalized to order $n$, with the coefficients of each pre-tabulated data point provided by an order $n$ *spline* function.  The cardinal basis spline (B-spline) used in PME is defined for interpolation using the nearerst $n$ points by the recursion

\begin{equation}
    M_n(u) = \frac{u}{n-1}M_{n-1}(u) + \frac{n-u}{n-1} M_{n-1}(u-1)
\end{equation}

where $u$ is the fractional distance from the nearest grid point to the left of the point of interest.  Using this method to discretize the density to a regular lattice the fast Fourier transform (FFT) can then be used to perform the discrete Fourier transform step, and scales as $\mathcal{O}(N log(N))$ instead of $\mathcal{O}(N^2)$; the same is true of the inverse tranformation step.  The PME algorithm is otherwise similar to conventional Ewald method, as demonstrated below:

```python
def PME_rec_energy(alpha, maxm, splineorder):
    """ Evaluate the particle mesh Ewald long range (reciprocal space)
        energy via discretization onto a regular cubic grid. """

    def build_spline(order, u):
        """ Construct a B spline of given order, from a seed value u """
        def one_pass_bspline(array, val, n):
            div = 1.0 / float(n - 1)
            array[n-1] = div * val * array[n-2]
            for j in range(1, n-1):
                array[n-j-1] = div * ((val+j) * array[n-j-2] + (n-j-val) * array[n-j-1])
            array[0] *= div * (1 - val)
            return array

        array = np.zeros(order)
        array[0] = 1.0 - u
        array[1] = u
        for m_n in range(1, order-1):
            array = one_pass_bspline(array, u, m_n+2)
        return array

    def dftmod(splineorder, nfft):
        """ Compute the norm of a B spline in Fourier space. """
        ftarr = np.zeros(nfft)
        # modulus is invariant to position, so just make a spline with 0
        ftarr[:splineorder] = build_spline(splineorder, 0.0)
        ftarr = np.fft.fft(ftarr)
        return np.square(ftarr.real) + np.square(ftarr.imag)

    mdim = 2*maxm+1
    # A convenient array to allow iteration over the grid from any given
    # starting point,  correctly wrapping around the boundaries.
    griditer = np.array([ [ ((p+n)%mdim) for p in range(splineorder) ] for n in range(mdim) ])
    # Define a simple iterator array to bring m into the range -0.5 < m <= 0.5
    shiftm = np.array([ k - mdim if k >= (mdim+1)/2 else k for k in range(mdim) ])

    splines = np.zeros((natoms, 3, splineorder))
    gridstart = np.zeros((natoms, 3), dtype=int)
    # Define coordinates in fractional coordinates
    reduced_coords = coords/boxlength + 0.5
    for atom,coord in enumerate(reduced_coords):
        # Define gridpoint to the left in each dimension, small shift
        # to handle atoms located on the rightmost grid point
        gridstart[atom,:] = list(map(int, mdim*coord-1e-8))
        wvals = mdim*coord - gridstart[atom]
        for xyz in range(3):
            splines[atom,xyz,:] = build_spline(splineorder, wvals[xyz])

    # Form the influence function, including spline normalization terms.  Note
    # that this is independent of particle positions and can therefore be formed
    # at the start of the simulation and cached for reuse in each time step.
    pi2_a2 = np.pi**2/alpha**2 if alpha != 0.0 else 1e50
    mvals = shiftm / float(boxlength)
    splinemod = 1/dftmod(splineorder, mdim)
    splinemods = np.einsum('x,y,z->xyz', splinemod, splinemod, splinemod)
    m = np.array(list(itertools.product(mvals, mvals, mvals)))
    m2 = np.einsum('nx,nx->n', m, m).reshape(mdim,mdim,mdim)
    gF = np.multiply(splinemods, np.exp(-pi2_a2*m2))
    m2[0,0,0] = 1e50 # avoid division by 0
    gF /= m2

    # Spread the charges onto the charge grid, using spline coefficients at O(N) cost
    Qgrid = np.zeros((mdim, mdim, mdim))
    for atom in range(natoms):
        q = charges[atom]
        for nx,xgrid in enumerate(griditer[gridstart[atom,0]]):
            xspline = splines[atom,0,nx]
            for ny,ygrid in enumerate(griditer[gridstart[atom,1]]):
                yspline = splines[atom,1,ny]
                for nz,zgrid in enumerate(griditer[gridstart[atom,2]]):
                    zspline = splines[atom,2,nz]
                    Qgrid[xgrid,ygrid,zgrid] += q*xspline*yspline*zspline

    # Forward FFT Q to form S(m), at O(Nlog(N)) cost
    Sm = np.fft.fftn(Qgrid)

    # Convolution of gF with S(m), at O(N) cost
    conv = np.multiply(gF, Sm)

    # Inverse FFT (S(m)*gF) real potential grid, at O(Nlog(N)) cost
    phigrid = mdim**3*np.fft.ifftn(conv).real

    # Probe the potential grid using the splines, to get the energy at O(N) cost
    e = 0.0
    for atom in range(natoms):
        q = charges[atom]
        for nx,xgrid in enumerate(griditer[gridstart[atom,0]]):
            xspline = splines[atom,0,nx]
            for ny,ygrid in enumerate(griditer[gridstart[atom,1]]):
                yspline = splines[atom,1,ny]
                for nz,zgrid in enumerate(griditer[gridstart[atom,2]]):
                    zspline = splines[atom,2,nz]
                    e += q*xspline*yspline*zspline*phigrid[xgrid,ygrid,zgrid]

    return e*coulomb / (2*np.pi*boxlength**3)

def PME_energy(alpha, maxn, maxm, splineorder):
    """ Compute the total PME energy by evaluating all components. """
    Udir = ewald_dir_energy(alpha, maxn)[0]
    Urec = PME_rec_energy(alpha, maxm, splineorder)
    Uslf = ewald_self_energy(alpha)
    U = Udir+Urec+Uslf
    return Udir,Urec,Uslf,U

# Repeat the same scan performed above for Ewald, using PME with 4th order spline.
efunc = functools.partial(PME_energy, splineorder=5)
Uvals = np.array([efunc(*p) for p in params])

# Plot the PME energy and its components as a function of attenuation parameter.
fig = plt.figure()
ax = fig.add_subplot(111)
alphavals = params[:,0]
ax.plot(alphavals, Uvals[:,0], '--r', label = 'Udir')
ax.plot(alphavals, Uvals[:,1], '--b', label = 'Urec')
ax.plot(alphavals, Uvals[:,2], '--g', label = 'Uslf')
ax.plot(alphavals, Uvals[:,3], 'k', label = 'U')
ax.set_xticks(alphavals)
ax.set_xlabel('Alpha (1/nm)')
ax.set_ylabel('Energy (kJ/mol)')
ax.legend()

plt.show()

# Confirm that all high alpha values have the same total energy
assert np.allclose(Uvals[2,3], Uvals[3:,3], atol=5)
```

The PME energy performs similarly to regular Ewald summation, with significantly better asymptotic scaling.  Forces may be evaluated easily by replacing the splines used to probe the potential grid with their derivatives, which are defined by a very simple recursion relation, as outlined by Darden and co-workers.

In developing the absolutely convergent sums above, we seem to have magically removed the conditional convergence.  The reason for this is somewhat complicated and beyond the scope of this discussion.  The conditional convergence results from the mutual interaction of the image cells' dipole moments.  By thinking of the periodic system not as an infinite system, but a large macroscopic crystal with a given shape (we considered cubes and spheres in the lattice sum code above) is is possible to derive a "correction" to the Ewald energy that describes the conditionally (and therefore crystal shape) dependent part of the lattice sum.  Such corrections have been derived for various crystal shapes, and they all depend on the cell dipole.  In dynamic simulations under periodic boundary condisions, a charge particle exiting the top of a unit cell will reappear at the bottom, possibly flipping the dipole of the cell.  If the extra shape-dependent term were considered in this circumstance, the potential energy would be discontinuous, so these corrections are usually neglected.

References
======

1. M. P. Allen and D. J. Tildesley, Computer Simulation of Liquids, Oxford University Press, New York 1987.
2. D. Frenkel and B. Smit, Understanding Molecular Simulation: From Algorithms to Applications, Academic Press, San Diego 1996.
3. U. Essmann, L. Perera, M. L. Berkowitz, T. A. Darden, H. Lee, and L. G. Pedersen, J. Chem. Phys. 103, 8577 (1995).
4. C. Sagui, L. G. Pedersen, and T. A. Darden, J. Chem. Phys. 120, 73 (2004).


```python
import cmath
import itertools
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
```

```python
vecs = [cmath.exp(2 * cmath.pi * i / 6 * 1j) for i in range(6)]
```

```python
vs = [v1 + v2 for v1, v2 in itertools.combinations_with_replacement(vecs, 2)]
vsr = [v.real for v in vs]
vsi = [v.imag for v in vs]

plt.axes().set_aspect('equal')
plt.scatter(vsr, vsi, alpha=0.3, color='red')
plt.show()
```

```python
vs2 = [sum(s) for s in itertools.combinations_with_replacement(vecs, 3)]
vs2r = [v.real for v in vs2]
vs2i = [v.imag for v in vs2]

plt.axes().set_aspect('equal')
plt.scatter(vs2r, vs2i, alpha=0.3, color='red')
plt.show()
```

```python
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True, positive=True)
f, g, h = symbols('f g h', cls=Function)
```

```python
f = binomial(k-1, 2) * (Rational(2, 3))**(k-3) * (Rational(1, 3))**3
f
```

$\displaystyle \frac{\left(\frac{2}{3}\right)^{k - 3} {\binom{k - 1}{2}}}{27}$

```python
f = simplify(f)
f
```

$\displaystyle 2^{k - 4} \cdot 3^{- k} \left(k - 2\right) \left(k - 1\right)$

```python
g = simplify(f.subs([(k, k+1)]) / f)
g
```

$\displaystyle \frac{2 k}{3 \left(k - 2\right)}$

```python
for i in range(3, 12):
    print(f'{i}: {f.subs([(k, i)]).evalf()}')

for i in range(3, 12):
    display(f.subs([(k, i)]))
```

    3: 0.0370370370370370
    4: 0.0740740740740741
    5: 0.0987654320987654
    6: 0.109739368998628
    7: 0.109739368998628
    8: 0.102423411065386
    9: 0.0910430320581212
    10: 0.0780368846212468
    11: 0.0650307371843723

$\displaystyle \frac{1}{27}$

$\displaystyle \frac{2}{27}$

$\displaystyle \frac{8}{81}$

$\displaystyle \frac{80}{729}$

$\displaystyle \frac{80}{729}$

$\displaystyle \frac{224}{2187}$

$\displaystyle \frac{1792}{19683}$

$\displaystyle \frac{512}{6561}$

$\displaystyle \frac{1280}{19683}$

```python
f = (ff(4, 3) * ff(n-4, k-3) * binomial(k-1, 2)) / ff(n, k)
f
```

$\displaystyle \frac{24 {\left(n - 4\right)}_{\left(k - 3\right)} {\binom{k - 1}{2}}}{{\left(n\right)}_{k}}$

```python
f = simplify(f)
f
```

$\displaystyle - \frac{12 \left(k - 2\right) \left(k - 1\right) \left(k - n\right)}{n \left(n - 3\right) \left(n - 2\right) \left(n - 1\right)}$

```python
g = simplify(f.subs([(k, k+1)]) / f)
g
```

$\displaystyle \frac{k \left(k - n + 1\right)}{\left(k - 2\right) \left(k - n\right)}$

```python
h = g - 1
solve(h, k)
```

    [2*n/3]

```python
f.subs([(k, 3)])
```

$\displaystyle - \frac{24 \left(3 - n\right)}{n \left(n - 3\right) \left(n - 2\right) \left(n - 1\right)}$

```python
f.subs([(k, n-1)])
```

$\displaystyle \frac{12}{n \left(n - 1\right)}$

```python
simplify(f.subs([(k, n-1)]) - f.subs([(k, 3)]))
```

$\displaystyle \frac{12 \left(n - 4\right)}{n \left(n - 2\right) \left(n - 1\right)}$

```python

```

```python

```


# *Ab* *initio* molecular dynamics of the vibrational motion of HF
We are going to construct what is often referred to as an *ab* *initio* potential energy surface of the diatomic
molecule hydrogen fluoride.  That is, we are going to use various electronic structure theories (Hartree-Fock theory (RHF), 2nd-order perturbation theory (MP2), and Coupled Cluster theory with single and double substitutions (CCSD)) to compute the electronic energy at different geometries of a simple diatomic molecule.  The same basis set (correlation consistent polarized triple-zeta, cc-pVTZ) will be used for all calculations.  We will use Psi4numpy to facilitate the electronic structure calculations, and then the interpolation capabilities of scipy to simplify the evalution of the potential energy at separations for which we did not explicitly evaluate the electronic energy.  We will also use scipy to differentiate the interpolated potential energy surface to obtain the forces acting on the atoms at different separations.

We will start by importing the necessary libraries:

```python
import numpy as np
import psi4
from matplotlib import pyplot as plt
from scipy.interpolate import InterpolatedUnivariateSpline
```

```python
### template for the z-matrix
mol_tmpl = """H
F 1 **R**"""
### array of bond-lengths in anstromgs
r_array = np.array([0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0, 2.1, 2.2, 2.3])
### array for different instances of the HF molecule
molecules =[]
### array for the different RHF energies for different HF bond-lengths
HF_E_array = []
### array for the different MP2 energies for different HF bond-lengths
MP2_E_array = []
### array for the different CCSD energies for different HF bond-lengths
CCSD_E_array = []

### loop over the different bond-lengths, create different instances
### of HF molecule
for r in r_array:
    molecule = psi4.geometry(mol_tmpl.replace("**R**", str(r)))
    molecules.append(molecule)

### loop over instances of molecules, compute the RHF, MP2, and CCSD
### energies and store them in their respective arrays
for mol in molecules:
    energy = psi4.energy("SCF/cc-pVTZ", molecule=mol)
    HF_E_array.append(energy)
    energy = psi4.energy("MP2/cc-pVTZ", molecule=mol)
    MP2_E_array.append(energy)
    energy = psi4.energy("CCSD/cc-pVTZ",molecule=mol)
    CCSD_E_array.append(energy)

### Plot the 3 different PES
plt.plot(r_array,HF_E_array,'r*', label='RHF')
plt.plot(r_array,MP2_E_array,'g*', label='MP2')
plt.plot(r_array,CCSD_E_array,'b*', label='CCSD')
```

Now that you have the raw data, we will interpolate this data using cubic splines.  This will permit us to
estimate the potential energy at any arbitrary separation between 0.5 and 2.3 Angstroms (roughly
1 and 4.3 a.u.) with fairly high confidence, and will also allow us to estimate the force
(the negative of the derivative of the PES with respect to separation)
at any separation between 1.0 and 4.3 a.u. since the derivative of cubic splines are readily available.

```python
### use cubic spline interpolation
order = 3

### get separation vector in atomic units
r_array_au = 1.89*r_array

### spline for RHF Energy
RHF_E_Spline = InterpolatedUnivariateSpline(r_array_au, HF_E_array, k=order)

### spline for MP2 Energy
MP2_E_Spline = InterpolatedUnivariateSpline(r_array_au, MP2_E_array, k=order)

### spline for CCSD Energy
CCSD_E_Spline = InterpolatedUnivariateSpline(r_array_au, CCSD_E_array, k=order)

### form a much finer grid
r_fine = np.linspace(0.5/0.529,2.3/0.529,200)

### compute the interpolated/extrapolated values for RHF Energy on this grid
RHF_E_fine = RHF_E_Spline(r_fine)

### compute the interpolated/extrapolated values for RHF Energy on this grid
MP2_E_fine = MP2_E_Spline(r_fine)

### compute the interpolated/extrapolated values for RHF Energy on this grid
CCSD_E_fine = CCSD_E_Spline(r_fine)

### plot the interpolated data
plt.plot(r_fine, RHF_E_fine, 'red', r_array_au, HF_E_array, 'r*')
plt.plot(r_fine, MP2_E_fine, 'green', r_array_au, MP2_E_array, 'g*')
plt.plot(r_fine, CCSD_E_fine, 'blue', r_array_au, CCSD_E_array, 'b*')

plt.show()
```

We can estimate the equilibrium bond length by finding the separation at which the potential is minimum; note this would also be the position that the force goes to zero:
\begin{equation}
\frac{d}{dr} V(r_{eq}) = -F(r_{eq}) = 0.
\end{equation}

First we will compute the forces at each level of theory:

```python
### take the derivative of the potential to get the negative of the force from RHF
RHF_Force = RHF_E_Spline.derivative()

### negative of the force from MP2
MP2_Force = MP2_E_Spline.derivative()

### negative of the force from CCSD
CCSD_Force = CCSD_E_Spline.derivative()

### let's plot the forces for each level of theory!

### plot the forces... note we need to multiply by -1 since the spline
### derivative gave us the negative of the force!
plt.plot(r_fine, -1*RHF_Force(r_fine), 'red')
plt.plot(r_fine, -1*MP2_Force(r_fine), 'green')
plt.plot(r_fine, -1*CCSD_Force(r_fine), 'blue')
plt.show()
```

Next we will find where the minimum of the potential energy surfaces are and use that
to find the equilibrium bond length:

```python
### Find Equilibrium Bond-Lengths for each level of theory
RHF_Req_idx = np.argmin(RHF_E_fine)
MP2_Req_idx = np.argmin(MP2_E_fine)
CCSD_Req_idx = np.argmin(CCSD_E_fine)

### find the value of the separation corresponding to that index
RHF_Req = r_fine[RHF_Req_idx]
MP2_Req = r_fine[MP2_Req_idx]
CCSD_Req = r_fine[CCSD_Req_idx]

### print equilibrium bond-lengths at each level of theory!
print(" Equilibrium bond length at RHF/cc-pVDZ level is ",RHF_Req, "atomic units")
print(" Equilibrium bond length at MP2/cc-pVDZ level is ",MP2_Req, "atomic units")
print(" Equilibrium bond lengthat CCSD/cc-pVDZ level is ",CCSD_Req, "atomic units")

```

     Equilibrium bond length at RHF/cc-pVDZ level is  1.6975235344966797 atomic units
     Equilibrium bond length at MP2/cc-pVDZ level is  1.7317209867864842 atomic units
     Equilibrium bond lengthat CCSD/cc-pVDZ level is  1.7317209867864842 atomic units

At this point, take a moment to compare your equilibrium bond length with other teams who have used different levels of theory to compute their potential energy surfaces.  Which equilibrium bond length should be most trustworthy?

You might have learned that the Harmonic Oscillator potential, which is a reasonable model for the vibrational motion of diatomic atomcs near their equilibrium bond length, is given by
\begin{equation}
V(r) = \frac{1}{2} k r^2
\end{equation}
and that the vibrational frequency of the molecule within the Harmonic oscillator model is given by
\begin{equation}
\nu = \frac{1}{2\pi}\sqrt{\frac{k}{\mu}}
\end{equation}
where $\mu$ is the reduced mass of the molecule and $k$ is known as the force constant.
We can estimate the force constant as
\begin{equation}
k = \frac{d^2}{dr^2} V(r_{eq}).
\end{equation}

Let's go ahead and get the force constants at each level of theory
and estimate the potential energy within the Harmonic approximation!

```python
### get second derivative of potential energy curve... recall that we fit a spline to
### to the first derivative already and called that spline function X_Force, where
### X is either RHF, MP2, or CCSD

RHF_Curvature = RHF_Force.derivative()
MP2_Curvature = MP2_Force.derivative()
CCSD_Curvature = CCSD_Force.derivative()

### evaluate the second derivative at r_eq to get k
RHF_k = RHF_Curvature(RHF_Req)
MP2_k = MP2_Curvature(MP2_Req)
CCSD_k = CCSD_Curvature(CCSD_Req)

### Print force constants for each level of theory!
print("Hartree-Fock force constant is ",RHF_k," atomic units")
print("MP2 force constant is ",MP2_k," atomic units")
print("CCSD force constant is ",CCSD_k," atomic units")

### define harmonic potential for each level of theory
RHF_Harm_Pot = RHF_k*(r_fine-RHF_Req)**2 + RHF_E_Spline(RHF_Req)
MP2_Harm_Pot = MP2_k*(r_fine-MP2_Req)**2 + MP2_E_Spline(MP2_Req)
CCSD_Harm_Pot = CCSD_k*(r_fine-CCSD_Req)**2 + CCSD_E_Spline(CCSD_Req)

### plot!
plt.plot(r_fine, RHF_Harm_Pot, 'red')
plt.plot(r_fine, MP2_Harm_Pot, 'green')
plt.plot(r_fine, CCSD_Harm_Pot, 'blue')
plt.show()
```

And we can actually estimate the fundamental vibrational frequency of the molecule
within this model using the force constant and the reduced mass of the molecule.

### Question 1:  What is the reduced mass of the HF molecule in atomic units?

### Question 2:  Use your spline fit to the PES of HF to estimate the vibrational frequency of HF.  Express your number in atomic units and also convert to a common spectroscopic unit system of your choosing (wavenumbers, nm, microns, Hertz, THz are all acceptable choices).

```python
#HE = InterpolatedUnivariateSpline(r_fine, harm_pot, k=order)
#HF = -HE.derivative()
### define reduced mass of HF as m_H * m_H /(m_F + m_H) where mass is in atomic units (electron mass = 1)
m_F = 34883.
m_H = 1836.
mu = (m_F * m_H)/(m_F + m_H)

### compute the fundamental frequency at each level of theory
RHF_nu = 1/(np.pi*2) * np.sqrt(RHF_k/mu)
MP2_nu = 1/(np.pi*2) * np.sqrt(MP2_k/mu)
CCSD_nu = 1/(np.pi*2) * np.sqrt(CCSD_k/mu)

### print the values in atomic units!
print("Vibrational frequency of HF at the RHF/cc-pVDZ level is ",RHF_nu," atomic units")
print("Vibrational frequency of HF at the MP2/cc-pVDZ level is ",MP2_nu," atomic units")
print("Vibrational frequency of HF at the CCSD/cc-pVDZ level is ",CCSD_nu," atomic units")

```

    1744.1974999319154
    Vibrational frequency of HF at the RHF/cc-pVDZ level is  0.0032340020089343204  atomic units
    Vibrational frequency of HF at the MP2/cc-pVDZ level is  0.0030541643010280816  atomic units
    Vibrational frequency of HF at the CCSD/cc-pVDZ level is  0.003047094936694507  atomic units

### Question 3:  How does these vibrational frequencies compare to the experimental vibrational frequency of HF?

Next, we want to actually simulate the dynamics of the HF molecule on this _ab_ _initio_ potential energy surface.  To do so, we need to solve Newton's equations of motion subject to some initial condition for the position (separation) and momentum (in a relative sense) of the particles.  Newton's equations can be written
\begin{equation}
F(r) = \mu \frac{d^2}{dr^2}
\end{equation}
where $\mu$ is the reduced mass in atomic units and $F(r)$ is the Force vs separation in atomic units that was determined previously.

### Question 4:  What will be the accelation of the bond stretch when H is separated by F by 3 atomic units?  You can express your acceleration in atomic units, also.

If the acceleration, position, and velocity of the bond stretch coordinate are known at some instant in
time $t_i$, then the position and velocity can be estimated at some later time $t_{i+1} = t_i + \Delta t$:
\begin{equation}
r(t_i + \Delta t) = r(t_i) + v(t_i)\Delta t + \frac{1}{2}a(t_i)\Delta t^2
\end{equation}
and
\begin{equation}
v(t_i + \Delta t) = v(t_i) + \frac{1}{2} \left(a(t_i) + a(t_i + \Delta t)  \right) \Delta t.
\end{equation}
This prescription for updating the velocities and positions is known as the Velocity-Verlet algorithm.
Note that we need to perform 2 force evaluations per Velocity-Verlet iteration: one corresponding
to position $r(t_i)$ to update the position, and then a second time at the updated position $r(t_i + \Delta t)$
to complete the velocity update.

```python
def Velocity_Verlet(r_curr, v_curr, mu, f_interp, dt):
    ### get acceleration at current time
    a_curr = -1*f_interp(r_curr)/mu

    ### use current acceleration and velocity to update position
    r_fut = r_curr + v_curr * dt + 0.5 * a_curr * dt**2

    ### use r_fut to get future acceleration a_fut
    a_fut = -1*f_interp(r_fut)/mu
    ### use current and future acceleration to get future velocity v_fut
    v_fut = v_curr + 0.5*(a_curr + a_fut) * dt

    result = [r_fut, v_fut]

    return result
```

To be able to define the very first update, an initial position and velocity must be specified.  Typically, these are chosen at random from a sensible range of values.

### Question 5: What makes a "sensible range of values" for position and velocity?

In this case, we will initialize the position to be a random number between 1.0 and 4.0; for the velocity, we will use the fact that we can estimate the expectation value of kinetic energy for a very similar system (the Harmonic oscillator) in the ground state as follows:
\begin{equation}
\langle T \rangle = \frac{1}{2} E_g,
\end{equation}
where $E_g$ is the ground state of the Harmonic oscillator (this is making use of the Virial theorem).  We can easily
find the ground state energy in the Harmonic oscillator approximation of $HF$ using our frequency calculation described above as
\begin{equation}
E_g = \frac{1}{2} h \nu,
\end{equation}
which implies the kinetic energy expectation value is
\begin{equation}
\langle T \rangle = \frac{h}{8 \pi} \sqrt{\frac{k}{\mu}}.
\end{equation}
Since we can say classically that the kinetic energy is given by $T = \frac{1}{2}\mu v^2$, we can estimate the velocity of the bond stretch as follows:
\begin{equation}
v = \sqrt{\frac{2 \langle T \rangle}{\mu}} = \sqrt{ \frac{\hbar \sqrt{\frac{k}{\mu}}}{2\mu}}
\end{equation}
where we have simplified using the fact that $\hbar = \frac{h}{2\pi}$ ($\hbar$ has the value 1 in the atomic unit system we are using up to this point!).  We will assume that a reasonable
range of velocities spans plus or minus 3 times this "ground-state" velocity.

```python
### define "ground-state" velocity for each level of theory
v_RHF = np.sqrt( np.sqrt(RHF_k/mu)/(2*mu))
v_MP2 = np.sqrt( np.sqrt(MP2_k/mu)/(2*mu))
v_CCSD = np.sqrt( np.sqrt(CCSD_k/mu)/(2*mu))

### get random position and velocity for RHF HF within a reasonable range
r_init = np.random.uniform(0.75*RHF_Req,2*RHF_Req)
v_init = np.random.uniform(-2*v_RHF,2*v_RHF)

### print initial position and velocity
print("Initial separation is ",r_init, "atomic units")
print("Initial velocity is   ",v_init, "atomic units")

### get initial force on the particle based on its separation
RHF_F_init = -1*RHF_Force(r_init)
print("Initial Force is ", RHF_F_init, "atomic units")

```

    Initial separation is  2.472236235287161 atomic units
    Initial velocity is    -2.727188785244146e-05 atomic units
    Initial Force is  -0.1563776103880313 atomic units

Now that we have our initial conditions chosen, our force as a function of separation known, and our Velocity Verlet function completed, we are ready to run our simulations!

```python
### how many updates do you want to perform?
N_updates = 200000

### establish time-step for integration to be 0.02 atomic units... this is about 0.0005 femtoseconds
### so total time is 200000*0.02 atomic units of time which is ~9.6e-13 s, or 960 fs
dt = 0.02

### Now use r_init and v_init and run velocity verlet update N_updates times, plot results
### these arrays will store the time, the position vs time, and the velocity vs time
r_vs_t = np.zeros(N_updates)
v_vs_t = np.zeros(N_updates)
t_array = np.zeros(N_updates)

### harmonic results
hr_vs_t = np.zeros(N_updates)
hv_vs_t = np.zeros(N_updates)

### first entry is the intial position and velocity
r_vs_t[0] = r_init
v_vs_t[0] = v_init

hr_vs_t[0] = r_init
hv_vs_t[0] = v_init

### first Velocity Verlet update
result_array = Velocity_Verlet(r_init, v_init, mu, RHF_Force, dt)
#hresult_array = Velocity_Verlet(r_init, v_init, mu, HF, dt)

print(result_array)
#print(hresult_array)

### do the update N_update-1 more times
for i in range(1,N_updates):
    tmp = Velocity_Verlet(result_array[0], result_array[1], mu, RHF_Force, dt)
    result_array = tmp
    t_array[i] = dt*i
    r_vs_t[i] = result_array[0]
    v_vs_t[i] = result_array[1]
    #tmp = Velocity_Verlet(hresult_array[0], hresult_array[1], mu, HF, dt)
    #hr_vs_t[i] = hresult_array[0]
    #hv_vs_t[i] = hresult_array[1]

### Plot the trajectory of bondlength vs time:
#plt.plot(t_array, r_vs_t, 'red', t_array, hr_vs_t, 'blue')
plt.plot(t_array, r_vs_t)
plt.show()

### plot the phase space trajectory of position vs momentum
#plt.plot(mu*v_vs_t, r_vs_t, 'blue', mu*hv_vs_t, hr_vs_t, 'purple')
#plt.show()

```

```python

```


# EXERCICE 2:

>On considère l'équation : $$ f(x)=x^3+x^2-3x-3=0$$
>1) Dessiner la courbe de $f$ dans l'intervalle $[-2;2]$, puis trouver des intervalles convenables pour appliquer la méthode de >bisection.
>
>2) Utiliser un programme `dicho.m` qui permettra de trouver les solutions de l'équation $f(x)=0$ avec $eps=0.001$.
>
>3) Modifier le programme `dicho.m` pour qu'il donne le nombre d'itération nécessaire pour avoir la solution à $eps$ près.

## La version python de `dicho`

```python
def dicho(a,b,f,tol):
    m=(a+b)/2
    err=abs(b-a)
    while err>tol:
        if f(m)==0 :
            break
        elif f(a)*f(m)<0 :
            b=m
        else :
            a=m
        m=(a+b)/2
        err=abs(b-a)
    return m

```

```python
f=lambda x: x**3+x**2-3*x-3
x3=dicho(-2,-1.5,f,10)
x10=dicho(-2,-1.5,f,10)
x20=dicho(-2,-1.5,f,20)
print(x3,x10,x20)
#print(x3)
```

    -1.75 -1.75 -1.75

```python
print('f(x10)={},f(x20)={}'.format(f(x10),f(x20)))
```

    f(x10)=-0.046875,f(x20)=-0.046875

### Version tant que qui renvoi la solution avec le nombre d'itération

```python
def dichow (a ,b,f,eps):

    g,d = min (a,b),max(a,b)
    gau,dor = f(g),f(d)
    n = 0
    while d-g > eps :
        n += 1
        m = ( g + d )/2
        valm = f ( m )
        if gau * valm < 0 :
            d = m
            dor = valm
        else :
            g = m
            gau = valm

    return (g+d)/2,n
```

```python
dichow(-2,-1.5,f,0.0001)
```

```python
import numpy as np
np.floor(2.3)
np.ceil(2.3)
```

    3.0

```python
import numpy as np
a=-1.5
b=-0.5
p=8 # précision à 10 ^-8 prés
dichow(a,b,f,10**(-p))#à 10^-8
```

    (-0.5000000037252903, 27)

```python
interval=[[-2,-1.5],[-1.5,-0.5],[1.5,2]]
for u in interval:
    print(dichow(u[0],u[1],f,10**(-3)))
################################################
print(dichow(-2,-1.5,f,10**(-3)))
print(dichow(-1.5,-0.5,f,10**(-3)))
print(dichow(1.5,2,f,10**(-3)))
```

    (-1.73193359375, 9)
    (-0.50048828125, 10)
    (1.73193359375, 9)
    (-1.73193359375, 9)
    (-0.50048828125, 10)
    (1.73193359375, 9)

## Nombre d'étiration

Le nombre d'itération N nécéssaire à la méthode de la dichotomie pour trouver $\alpha$ à $10^{-p}$ prés

$$N \geq E\left(\dfrac{\ln(b-a)+p\ln(10)}{\ln(2)}\right)+1$$

```python
def itteration(a,b,k):
    return int((np.log(b-a)+k*np.log(10)/np.log(2)))+1
```

```python
itteration(-2,-1.5,5)
```

    16

```python
#alpha 1
dichow(-2,-1.5,f,10**(-5))
#à 10^-8
```

    (-1.7320518493652344, 16)

```python
#alpha 2
dichow(-1.51,-0.5,f,10**(-5))
#à 10^-8
```

    (-1.0000028610229492, 17)

## Méthode graphique pour la recherche de la solution

```python
import numpy as np
import matplotlib.pyplot as plt

f=lambda x:x**3+x**2-3*x-3

t=np.linspace(1.7,1.75,100)
plt.subplot(2,2,1)
plt.plot(t,f(t),'r')
plt.grid(True)
#################
t=np.linspace(1.2,1.4,100)
plt.subplot(2,2,2)
plt.plot(t,f(t),'r')
plt.grid(True)
#################
t=np.linspace(1.5,1.8,100)
plt.subplot(2,2,3)
plt.plot(t,f(t),'r')
plt.grid(True)
#################
t=np.linspace(1.365,1.366,100)
plt.subplot(2,2,4)
plt.plot(t,f(t),'r')
plt.grid(True)

```

```python
def f ( x ):
    return x **3+ x**2- 3*x - 3
x=np.linspace ( -2 ,2)
y=f(x)
plt.title("solution dans [-2,2]");
plt.xlabel("x")
plt.ylabel("f(x)")
plt.grid(True)
plt.plot (x,y)
plt.axhline()
plt.show()
```

# EXERCICE 3:
>On considère l'équation : $$ f(x)=x^3+4x^2-10=0$$
>1) Dessiner la courbe de $f$ dans l'intervalle $[1;2]$, puis trouver une valeur approché de la solution à 0.001 près (En >utilisant la fonction zoom du graphique).
>
>2) Vérifier que cette équation est équivalente aux 3 schéma du point fixe suivant:
>
>a) $g_1(x)=\frac{1}{2}\sqrt{10-x^3}=x$.
>
>b) $g_2(x)=\sqrt{\displaystyle\frac{10}{4+x}}=x$.
>
>c) $g_3(x)=x-x^3-4x^2+10=x$.
>
>3) Ecrire puis appliquer  une fonction Matlab que l'on appellera  `pointfixe.m` aux trois schémas précédents en prenant >`x0=1.5;eps=0.001`.
>
>4) Quelle est la fonction (parmi $g_1, g_2$ et $g_3$) qui donne la convergence la plus rapide?
>

## Calcul symbolique de la dérivée

```python
from sympy import *
import numpy as np
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
g1=lambda x: 1/2*sqrt(10-x**3)
```

```python
dg1=lambdify(x,diff(g1(x),x,1))
dg1(1.365)
```

### => Le dérive est < 1, donc la fonction converge localement

## Etude de la convergences de g2 et g3

```python
def pt_fixe(f,x_0,e,n):
    a=x_0
    for i in range(n):
        x=a
        a=f(x)
        err=abs(a-x)
        print(a)
        if err<e:
            return a,i
    return "Divergence :( "
```

```python
import numpy as np
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
g1=lambda x: 1/2*sqrt(10-x**3)
g2=lambda x: sqrt(10/(x+4))
g3=lambda x: x-x**3-4*x**2+10
```

```python
pt_fixe(g1,1.5,0.001,50)
```

```python
pt_fixe(g2,1.5,0.001,50)
```

```python
pt_fixe(g3,1.5,0.001,30)
```

### => G2 a convergé plus rapidements avec un nombre d'itérations = 3

# EXERCICE 4:
>1) On considère la fonction $f(x)=e^{-x}-x$. Ecrire puis appliquer  une fonction Matlab que l'on appellera `newton.m` à la >fonction $f$, en prenant $x0=0$, $eps=0.001$ .
>
>2) Montrer que l'équation $f(x)=0$ admet une solution unique que l'on peut obtenir en utilisant la méthode de Newton à partir >de $x0=1$.

### Solution Python

```python
def newtonn(f,fprime,a,eps):
    x1=a
    x2=x1-f(x1)/fprime(x1)
    while abs(x1-x2)>eps :
        x1=x2
        x2=x1-f(x1)/fprime(x1)
    return x2
```

```python
import math
from sympy import *
from sympy.abc import x

f=lambda x: math.exp(-x)-x
fprime=lambda x: -math.exp(-x)-1
newtonn(f,fprime,0,0.001)
```

```python

```


### Encoding of outputs
* Encoding of output can impact performance.
* Common encoding methods for classification:
    - Integer label for each class
    - One-hot enconding (binary vectors with one indicator for each class)
    - Binary (or other base) encoding
* Similar choices can be made for regression and other problems

### Rate of Learning

* Back-propagation algorithm provides an approximation to the trajectory in the weight space computed by the method of steepest descent
*  Small $\eta$: smoother trajectory, slower learning
*  Too Large $\eta$: large changes in the synaptic weights, may become unstable (oscillatory)

*  *Include a momentum term*: tries to increase rate of learning while avoiding instability
* Recall: $\Delta w_{ij}(n) = \eta \delta_j(n) y_i(n) = \eta \frac{\partial E(n)}{\partial v_j(n)} y_i(n)$
*  *The Generalized Delta rule (Delta rule with momentum):* $\Delta w_{ij}(n) = \alpha \Delta w_{ij}(n-1) + \eta \delta_j(n) y_i(n)$
*  Can write the generalized delta rule as a time series (index $t$):
\begin{eqnarray}
\Delta w_{ij}(n) &=& -\eta \sum_{t=0}^n \alpha^{(n-t)}\delta_j(t)y_i(t)\\
&=& -\eta \sum_{t=0}^n \alpha^{(n-t)} \frac{\partial E(t)}{\partial w_{ij}(t)}
\end{eqnarray}

#### Observations & Comments:
*  $\Delta w_{ij}(n)$ is the sum of exponentially weighted time series.  For it to converge, $0 \le |\alpha| < 1$, If $\alpha = 0$, then you are operating without momentum
*  Inclusion of momentum accelerate descent in steady downhill directions
*  Inclusion of momentum has a stabilizing effect in directions that oscillate in sign
*  Momentum may prevent termination/convergence in a shallow minimum
*  The learning rate *can* be connection dependent, $\eta_{ij}$, can even set it to zero for some connections

*See Haykin's Neural Network text for more reading on this and the following.  It was the reference for these notes and provides a nice description.

### Online vs. Batch Update

* Thus far we have focused on Online learning. *What is meant by online learning?*
* You can also do:  Batch update or (Stochastic) Mini-Batch Update

### A Sampling of Heuristics/Methods that may help Back-Propagation Algorithm Perform Better

* Maximizing Information Content
     * Provide training samples that provide the largest information content
    * Use example with largest training error
    * Use example radically different than the ones before
    * Emphasizing scheme, present more difficult patterns to the network, difficulty is determined by error
    * Problems with emphasizing scheme:
        * Distribution of samples in an epoch is distorted
        * Outliers or mislabeled samples can cause major problems

*  Activation Function: MLPs may learn better with activation functions that are antisymmetric rather than nonsymmetric.
\begin{equation}
\phi(-v) = -\phi(v)
\end{equation}

* Target values need to be in the range of the sigmoid.
* It is good if the input variables are uncorrelated
* It is good if variances are approximately equal

### Strategy when training a network for the first time

* This is a good read.  Includes suggestions that I often give students and agree with:  http://karpathy.github.io/2019/04/25/recipe/
* Most commonly overlooked in my experience are:
    - Be sure you know and understand your data.  Make use of any knowledge you have or gain about the problem. For example, if you realize you have an imbalanced data set, you may want to implement a method to address this such as upsampling/data augmentation for the smaller class, weighting the examples from each class accordingly, or another approach.
    - Overfit your network with a small subset of the data.  This helps to make sure your code is working, alert you to any bugs or issues in the data.

### Network Pruning Techniques

*  *Network Growing:* Start with a small MLP and add to it when unable to meet design specifications
*  *Network Pruning:* Start with a large MLP and prune it by eliminating weights (driving them to zero)

*  *Regularization:* Just as we applied regularization during regression, we can apply regularization to our NN training: $R(w) = E_s(w) + \lambda E_c(w)$ where $E_s$ is the error measure, $E_c$ is the regularization penalty term, and $\lambda$ is a regularization "trade-off" parameter
     * $\lambda = 0$: Training based only on error and training data
     * $\lambda \rightarrow \infty$: Training samples are unreliable, Minimize complexity

#### Weight Decay
\begin{equation}
E_c(w) = \left\| \mathbf{w} \right\|^2 = \sum_i w_i^2
\end{equation}
* Drives some weights to be small.
* Weights with little or no influence are excess/unnecessary weights.  The goal: encourage unnecessary weights to go to zero and thereby improve generalization
 or at least have the smallest values that can solve the problem.

* *So, how would you include weight decay in your Back-Propagation Training Algorithm?*

### Ill-conditioned Data

* Consider the following two systems of linear equations:
\begin{eqnarray}
 x_1 + 2x_2 = 2\\
 x_1 + 2.0001x_2 = 2
 \end{eqnarray}

 \begin{eqnarray}
 x_1 + 2x_2 = 2\\
 x_1 + 2.0001x_2 = 2.0001
 \end{eqnarray}

* *What are the solutions to both of these linear systems?*

* Ill-conditioned problems mean that small changes in values result in a large change in the solution.  This is related to *adversarial examples* in deep learning architectures. See: https://blog.openai.com/adversarial-example-research/  Adversarial examples are also related to the Curse of Dimensionality.

* Ill-conditioned data poses problems for MLPs as well.  Ill-conditioning affects the speed and accuracy of back-propagation.
* The *condition number* is the ratio of the largest and smallest eigenvalues of the Hessian of the error function.
* *What is the Hessian of the least squared error function for the simple linear example shown above?*
* *What are the condition numbers for the following data sets:*

\begin{eqnarray}
\mathbf{X} = [-1.5, 1; -0.5, 1; 0.5, 1; 1.5, 1]\\
\mathbf{Y} = [1.5; 2.5; 3.5; 4.5]
\end{eqnarray}

\begin{eqnarray}
\mathbf{X} = [0, 1; 1, 1; 2, 1; 3, 1]\\
\mathbf{Y} = [3; 4; 5; 6]
\end{eqnarray}

\begin{eqnarray}
\mathbf{X} = [10, 1; 11, 1; 12, 1; 13, 1]\\
\mathbf{Y} = [13; 14; 15; 16]
\end{eqnarray}

* The Hessian matrix describes the local curvature of the error surface. Ill conditioned data have eccentric error contours (where as well conditioned data will have circular or nearly circular error contours).

* It is also good to have similar variances across input values and have node-specific learning weights.  Consider a linear network.  Suppose you have one input value with an average of $\approx 3000$, a bias fixed to 1 and a target value around $\approx 30$. Then, the weight for the input must be small to map 3000 to 30 (.001). If we had a $10\%$ error, then we would end up needing a very small learning rate to avoid instability.  However, if we had such a small learning rate and our bias was also wrong, then it would take extremely long to train.

* We also don't want to saturate our activation functions as this leads to very slow learning.  So, initialize weights appropriately and normalize input data.

### Some helpful visualizations to check out:

* https://playground.tensorflow.org
* https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
* http://experiments.mostafa.io/public/ffbpann/
* https://google-developers.appspot.com/machine-learning/crash-course/backprop-scroll/

```python

```


# Linear Regression - with single variable

## Intro

I first tried coding linear regression algorithm being taught by Luis Serrano. Luis produces youtube videos on data-science subjects with easy-to-understand visualizations. In his video [Linear Regression: A friendly introduction](https://www.youtube.com/watch?v=wYPUhge9w5c) he uses the following approach :

**Note:**

The dataset we're using contains salary of some people and the number of year of experience.

We are trying to predict the salary given the number of year of experience.

So the number of year of experience is the independent variable and the salary is the dependent variable.

The x-axis is related to the number of year of experience.

The y-axis is related to the salary.

y-intercept is the point that satisfy x = 0, in other words the point of the line that intersects the y-axis

Increasing y-intercept means translating the line up, and decreasing y-intercept means translating the line down

## Implementation

Run the following cell to import all needed modules, you must have opened this document on Google Colab before doing so:

```
import pandas as pd
from sympy.geometry import Point, Line
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
import progressbar
```

I used the component Line from the module sympy.geometry. To create a Line I need to specify two Points. The line is also characterized by 3 coefficients (a, b and c) that match the following equation :

In my appoach I am dealing with a line equation of this sort :

So I translated the first equation to match my equation requirement :

Run the following cell. It contains the functions that will be used in the program:

```
def drawAll(X, Y, line):
    """ plot the points from the dataset and draw the actual Line """
    coefs = line.coefficients
    x = np.linspace(X.min(),X.max())
    y = (-coefs[0] * x - coefs[2]) / coefs[1]
    plt.plot(x, y)
    plt.scatter(X, Y, color = 'red')
    plt.show()

def transformLine(point, line, x_median, learning_rate):
    """ According to the random point, update the Line """
    # We take the median of the x values for better results for the calculations of the horizontal distances

    # Creation of the vertical line passing through the new point
    ymin = line.points[0] if line.direction.y > 0 else line.points[1]
    ymax = line.points[1] if line.direction.y > 0 else line.points[0]
    vertical_line = Line(Point(point.x,ymin.y), Point(point.x,ymax.y))
    # Find the intersection with our line (to calculate the vertical distance)
    I = line.intersection(vertical_line)
    vertical_distance = point.y - I[0].y
    horizontal_distance = point.x - x_median

    coefs = line.coefficients

    a = coefs[0]
    b = coefs[1]
    c = coefs[2]

    # Calculation of the points which constitute the new line
    # Reminder: we add (learning_rate * vertical_distance * horizontal_distance) to the slope and we add (learning_rate * vertical_distance) to y-intercept
    # The equation now looks like :
    # y = - (a/b)*x + (learning_rate * vertical_distance * horizontal_distance) * x - (c/b) + learning_rate * vertical_distance

    # We keep the same scope of the line so the min value of x and the max value of x don't change

    x_min = line.points[0].x
    y_min = - (a/b)*x_min + (learning_rate * vertical_distance * horizontal_distance * x_min) - (c/b) + learning_rate * vertical_distance

    x_max = line.points[1].x
    y_max = - (a/b)*x_max + (learning_rate * vertical_distance * horizontal_distance * x_max) - (c/b) + learning_rate * vertical_distance

    newLine = Line(Point(x_min, y_min), Point(x_max, y_max))
    return newLine

def predict(X, line):
    """ I use my model (the equation of the line) to predict new values """
    prediction = []
    coefs = line.coefficients
    a = coefs[0]
    b = coefs[1]
    c = coefs[2]
    for x in X.values:
        y = - (a/b)*x - (c/b)
        prediction.append(y)
    return prediction
```

Run the following cell to launch the linear regression program:

```
# Set the learning rate and the number of iterations
learning_rate = 0.01
nb_epochs = 1000

# Read the data
dataset = pd.read_csv("https://raw.githubusercontent.com/davy-datascience/ml_algorithms/master/LinearRegression/Approach-1/dataset/Salary_Data.csv")

# Separate the dataset into a training set and a test set
train, test = train_test_split(dataset, test_size = 0.2)

# Separation independent variable X - dependent variable y for the train set & the test set
X_train = train.YearsExperience
y_train = train.Salary

X_test = test.YearsExperience
y_test = test.Salary

# Looking for 1st line equation
# The line must have the same scope than the scatter plots from the dataset
# I decided to build the line choosing the point that has the max x-value and the point that has the min x-value

# Find the point with the maximum value of x in the dataset
idx_max = X_train.idxmax()
x_max = Point(X_train.loc[idx_max], y_train.loc[idx_max])

# Find the point with the minimum value of x in the dataset
idx_min = X_train.idxmin()
x_min = Point(X_train.loc[idx_min], y_train.loc[idx_min])

# Build the line with the 2 points
line = Line(x_min, x_max)
drawAll(X_train, y_train, line)

# Iterate choosing a random point and moving the line with the function transformLine
for i in progressbar.progressbar(range(nb_epochs)):
    sample = train.sample()
    point = Point(sample.YearsExperience, sample.Salary)
    line = transformLine(point, line, X_train.median(), learning_rate)
    #drawAll(X_train, y_train, line)    # Uncomment this line to see the line at each iteration

drawAll(X_train, y_train, line)

# Predict the test set with my model and see
y_pred = predict(X_test, line)
print("MAE (Mean Absolute Error) is used to evaluate the model accuracy")
print("MAE for my model: {}".format(mean_absolute_error(y_pred, y_test)))

# Predict the test set with the sklearn algorithm
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train.to_frame(), y_train)
y_pred2 = regressor.predict(X_test.to_frame())
print("MAE for the algorithm of the sklearn module: {}".format(mean_absolute_error(y_pred2, y_test)))
```