```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sy
import simtk.unit as unit
from simtk import openmm as mm
from simtk.openmm import app
import skopt as skopt
from tqdm import tqdm
```

# A Lennard-Jones Fluid

## The Lennard-Jones potential

The Lennard-Jones (LJ) potential between two particles is defined by the following equation, where $x$ is the distance between the particles, and $\sigma$ and $\epsilon$ are two parameters of the potential:

\begin{equation}
V(x) = 4 \epsilon \left[ \left( \frac{\sigma}{x} \right)^{12} - \left( \frac{\sigma}{x} \right)^6 \right]
\end{equation}

Lets see the shape of this function:

```python
def LJ (x, sigma, epsilon):

    t = sigma/x
    t6 = t**6
    t12 = t6**2

    return 4.0*epsilon*(t12-t6)
```

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

xlim_figure = [0.01, 6.0]
ylim_figure = [-2.0, 10.0]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom
plt.plot(x, LJ(x, sigma, epsilon))
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

The way the LJ potential is built, the $\sigma$ and $\epsilon$ parameters have a straightforward interpretation. The cut with $y=0$ is located in $x=\sigma$:

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

xlim_figure = [0.01, 6.0]
ylim_figure = [-2.0, 10.0]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom
plt.plot(x, LJ(x, sigma, epsilon))
plt.hlines(0, xlim_figure[0], xlim_figure[1], linestyles='dotted', color='gray')
plt.vlines(sigma._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='red')
plt.text(sigma._value+0.02*xlim_figure[1], 0.7*ylim_figure[1], '$\sigma$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

And $\epsilon$ is the depth of the minimum measured from $y=0$:

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

xlim_figure = [0.01, 6.0]
ylim_figure = [-2.0, 10.0]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom
plt.plot(x, LJ(x, sigma, epsilon))
plt.hlines(0, xlim_figure[0], xlim_figure[1], linestyles='dotted', color='gray')
plt.hlines(-epsilon._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='red')
plt.annotate(text='', xy=(1.0,0.0), xytext=(1.0,-epsilon._value), arrowprops=dict(arrowstyle='<->'))
plt.text(1.0+0.02*xlim_figure[1], -0.7*epsilon._value, '$\epsilon$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

Notice that the LJ potential has physical meaning when $\epsilon>0$ and $\sigma>0$ only. Actually, the potential vanishes whether $\epsilon=0$ or $\sigma=0$.

### The Lennard Jones minimum and the size of the particles

The LJ potential has a single minimum located in $x_{min}$. Lets equal to $0$ the first derivative of the potential to find the value of $x_{min}$:

```python
x, sigma, epsilon = sy.symbols('x sigma epsilon', real=True, positive=True)
V = 4.0*epsilon*((sigma/x)**12-(sigma/x)**6)
gradV = sy.diff(V,x)
roots=sy.solve(gradV, x)
x_min = roots[0]
```

```python
x_min
```

$\displaystyle 1.12246204830937 \sigma$

The minimum is then located in:

\begin{equation}
x_{min} = 2^{1/6} \sigma
\end{equation}

where the potential takes the value:

\begin{equation}
V(x_{min}) = -\epsilon
\end{equation}

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.4, x_min._value+0.4]
ylim_figure = [y_min._value-0.1, y_min._value+0.5]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

This way two particles in the equilibrium position will be placed at a $2^{1/6} \sigma$ distance. The potential is thereby modeling two "soft spheres" atracting each other very lightly. Their radii, given that both particles are equal, are equal to $r$:

\begin{equation}
r = \frac{1}{2} x_{min} = 2^{-5/6} \sigma
\end{equation}

And we say these spheres are "soft" because their volume is not limited by a hard-wall potential, they can penetrate each other suffering a not infinite repulsive force.

### Time period of the small harmonic oscillations around the minimum

If we want to perform a molecular simulation of this two particles we should wonder how big the integrator timestep must be. To answer this question we can study the harmonic approximation around the minimum. Lets calculate the time period, $\tau$, of a small harmonic oscillation around the minimum:

```python
x, sigma, epsilon = sy.symbols('x sigma epsilon', real=True, positive=True)
V = 4.0*epsilon*((sigma/x)**12-(sigma/x)**6)
gradV = sy.diff(V,x)
grad2V = sy.diff(V,x,x)

x_min = sy.solve(gradV,x)[0]
k_harm = grad2V.subs(x, x_min)
```

```python
k_harm
```

$\displaystyle \frac{57.1464378708551 \epsilon}{\sigma^{2}}$

The harmonic constant of the second degree Taylor polynomial of the LJ potential at $x=x_{min}$ is then:

\begin{equation}
k_{harm} = 36·2^{2/3} \frac{\epsilon}{\sigma^2}
\end{equation}

The oscillation period of a particle with $m$ mass in an harmonic potential defined by $\frac{1}{2} k x²$ is:

\begin{equation}
\tau = 2 \pi \sqrt{ \frac{m}{k}}
\end{equation}

As such, the period of the small harmonic oscillations around the LJ minimum of particle with $m$ mass is:

\begin{equation}
\tau = 2 \pi \sqrt{ \frac{m}{k_{harm}}} = \frac{\pi}{3·2^{1/3}} \sqrt{\frac{m\sigma^2}{\epsilon}}
\end{equation}

With the mass and parameters taking values of amus, angstroms and kilocalories per mole, the time period is in the order of:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/k)

print(tau)
```

    0.5746513694274475 ps

But, is this characteristic time a good threshold for a LJ potential? If the oscillations around the minimum are not small enough, the harmonic potential of the second degree term of the taylor expansion is easily overcome by the sharp left branch of the LJ potential:

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.2, x_min._value+0.2]
ylim_figure = [y_min._value-0.1, y_min._value+0.6]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.plot(x, 0.5*k*(x-x_min)**2+y_min)
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

Let's imagine the following situation. Let a particle be in the harmonic potential at temperature of 300K. Will the particle be more constrained in space than in the well of the LJ potential? Will the particle feel the harmonic potential softer or sharper than the LJ? Lets make some numbers to evaluate if the oscillation time period of the harmonic approximation can be a good time threshold for the integration timestep of a molecular dynamics of the LJ potential.

The standard deviation of an harmonic oscillation with the shape $\frac{1}{2}k x^2$ in contact with a stochastic thermal bath can be computed as:

\begin{equation}
\beta = \frac{1}{k_{\rm B} T}
\end{equation}

\begin{equation}
Z_x = \int_{-\infty}^{\infty} {\rm e}^{- \beta \frac{1}{2}k x^2} = \sqrt{\frac{2 \pi}{\beta k}}
\end{equation}

\begin{equation}
\left< x \right> = \frac{1}{Z_x} \int_{-\infty}^{\infty} x {\rm e}^{-\beta \frac{1}{2}k x^2} = 0
\end{equation}

\begin{equation}
\left< x^2 \right> = \frac{1}{Z_x} \int_{-\infty}^{\infty} x^{2} {\rm e}^{-\beta \frac{1}{2}k x^2} = \frac{1}{Z_x}  \sqrt{\frac{2 \pi}{\beta³ k^3}} = \frac{1}{\beta k}
\end{equation}

\begin{equation}
{\rm std} = \left( \left< x^2 \right> -\left< x \right>^2 \right)^{1/2} = \sqrt{ \frac{k_{\rm B}T}{k} }
\end{equation}

This way, in the case of the harmonic potential obtained as the second degree term of the Taylor expansion around the LJ minimum:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole
temperature = 300 * unit.kelvin
kB = unit.BOLTZMANN_CONSTANT_kB * unit.AVOGADRO_CONSTANT_NA

k = 36 * 2**(2/3) * epsilon/sigma**2
std = np.sqrt(kB*temperature/k)

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.4, x_min._value+0.4]
ylim_figure = [y_min._value-0.1, y_min._value+0.6]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.plot(x, 0.5*k*(x-x_min)**2+y_min)
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.axvspan(x_min._value - std._value, x_min._value + std._value, alpha=0.2, color='red')
plt.annotate(text='', xy=(x_min._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             xytext=(x_min._value-std._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             arrowprops=dict(arrowstyle='<->'))
plt.text(x_min._value-0.6*std._value, y_min._value - 0.4*(y_min._value-ylim_figure[0]), '$std$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

The harmonic potential is too soft as approximation. Its oscillation time used as threshold to choose the integration timestep can yield to numeric problems. Let's try with a stiffer potential, let's double the harmonic constant:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole
temperature = 300 * unit.kelvin
kB = unit.BOLTZMANN_CONSTANT_kB * unit.AVOGADRO_CONSTANT_NA

k = 36 * 2**(2/3) * epsilon/sigma**2
std = np.sqrt(kB*temperature/k)

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.4, x_min._value+0.4]
ylim_figure = [y_min._value-0.1, y_min._value+0.6]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.plot(x, 0.5*k*(x-x_min)**2+y_min)
plt.plot(x, k*(x-x_min)**2+y_min, label='2k_{harm}')
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.axvspan(x_min._value - std._value, x_min._value + std._value, alpha=0.2, color='red')
plt.annotate(text='', xy=(x_min._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             xytext=(x_min._value-std._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             arrowprops=dict(arrowstyle='<->'))
plt.text(x_min._value-0.6*std._value, y_min._value - 0.4*(y_min._value-ylim_figure[0]), '$std$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

Lets take then, as reference, an harmonic potential with constant equal to $2k_{harm}$ could be a better idea. Lets compute then the new time threshold to choose the integration timestep:

\begin{equation}
\tau = 2 \pi \sqrt{ \frac{m}{2k_{harm}}} = \frac{\pi}{3·2^{5/6}} \sqrt{\frac{m\sigma^2}{\epsilon}}
\end{equation}

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/(2*k))

print(tau)
```

    0.4063398801402841 ps

It is an accepted rule of thumb that the integration timestep must be as large as $\tau / 10$, being $\tau$ the oscillation time period of the fastest possible vibration mode. So finally, in this case the integration time step should not be longer than:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/(2*k))

print(tau/10.0)
```

    0.04063398801402841 ps

For the case of a LJ potential modeling an Argon fluid:

```python
mass = 39.9 * 2 * unit.amu # reduced mass of a pair of atoms
sigma = 3.4 * unit.angstrom
epsilon = 0.238 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/(2*k))

print(tau/10.0)
```

    0.17888187339607967 ps

## Two Lennard-Jones atoms in vacuum

### Ar-Ar

```python
mass = 39.948 * unit.amu
sigma = 3.404 * unit.angstroms
epsilon = 0.238 * unit.kilocalories_per_mole
charge = 0.0 * unit.elementary_charge
```

```python
system = mm.System()

non_bonded_force = mm.NonbondedForce()
non_bonded_force.setNonbondedMethod(mm.NonbondedForce.NoCutoff)

# First Ar atom
system.addParticle(mass)
non_bonded_force.addParticle(charge, sigma, epsilon)

# Second Ar atom
system.addParticle(mass)
non_bonded_force.addParticle(charge, sigma, epsilon)

system.addForce(non_bonded_force)

integrator = mm.VerletIntegrator(2*unit.femtoseconds)
platform = mm.Platform.getPlatformByName('CUDA')

context = mm.Context(system, integrator, platform)
```

```python
positions = np.zeros([2,3], float) * unit.angstroms

x = np.linspace(1.0, 8.0, 200, endpoint=True) * unit.angstroms
V = [] * unit.kilocalories_per_mole

for xi in x:
    positions[1,0] = xi
    context.setPositions(positions)
    state = context.getState(getEnergy=True)
    potential_energy = state.getPotentialEnergy()
    V.append(potential_energy)
```

Position of minimum:

\begin{equation}
x_{min} = 2^{1/6} \sigma
\end{equation}

```python
x_min = 2**(1/6)*sigma
```

```python
x_min
```

    Quantity(value=3.8208608124451056, unit=angstrom)

```python
k = 36 * 2**(2/3) * epsilon/sigma**2
reduced_mass = (mass**2)/(2*mass)

tau = 2*np.pi * np.sqrt(reduced_mass/k)

print(tau)
```

    1.267135457848527 ps

```python
V._value = np.array(V._value)

xlim_figure = [0.01, 8.0]
ylim_figure = [-2.0, 10.0]

plt.plot(x, V, linewidth=2)
plt.plot(x, LJ(x, sigma, epsilon), linestyle='--', color='red')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

```python
xlim_figure = [3.0, 5.0]
ylim_figure = [-0.4, 0.2]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom

plt.plot(x, LJ(x, sigma, epsilon))
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')

plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

```python
initial_distance = x_min + 0.05 * unit.angstroms
simulation_time = 4*tau
saving_time = 0.01*tau
integration_timestep = 0.01*tau

saving_steps = int(saving_time/integration_timestep)
n_savings = int(simulation_time/saving_time)

integrator = context.getIntegrator()
integrator.setStepSize(integration_timestep)

trajectory = []*unit.angstroms
times = []*unit.picoseconds

positions = np.zeros([2,3], float) * unit.angstroms
positions[1,0] = initial_distance
context.setPositions(positions)

velocities = np.zeros([2,3], float) * unit.angstroms/unit.picoseconds
context.setVelocities(velocities)

context.setTime(0.0*unit.picoseconds)

state = context.getState(getPositions=True)
positions = state.getPositions(asNumpy=True)
time = state.getTime()
distance = positions[1,0] - positions[0,0]
trajectory.append(distance)
times.append(time)

for _ in range(n_savings):

    integrator.step(saving_steps)
    state = context.getState(getPositions=True)
    positions = state.getPositions(asNumpy=True)
    time = state.getTime()
    distance = positions[1,0] - positions[0,0]
    trajectory.append(distance)
    times.append(time)

times._value = np.array(times._value)
trajectory._value = np.array(trajectory._value)

plt.plot(times, trajectory)
plt.show()
```

- Try to place the initial distance between the pair of atoms further than the simulated above.

### Ar-Xe molecular system

```python
mass_Ar = 39.948 * unit.amu
sigma_Ar = 3.404 * unit.angstroms
epsilon_Ar = 0.238 * unit.kilocalories_per_mole
charge_Ar = 0.0 * unit.elementary_charge

mass_Xe = 131.293 * unit.amu
sigma_Xe = 3.961 * unit.angstroms
epsilon_Xe = 0.459 * unit.kilocalories_per_mole
charge_Xe = 0.0 * unit.elementary_charge
```

```python
system = mm.System()

non_bonded_force = mm.NonbondedForce()
non_bonded_force.setNonbondedMethod(mm.NonbondedForce.NoCutoff)

# Ar atom
system.addParticle(mass_Ar)
non_bonded_force.addParticle(charge_Ar, sigma_Ar, epsilon_Ar)

# Xe atom
system.addParticle(mass_Xe)
non_bonded_force.addParticle(charge_Xe, sigma_Xe, epsilon_Xe)

system.addForce(non_bonded_force)

integrator = mm.VerletIntegrator(2*unit.femtoseconds)
platform = mm.Platform.getPlatformByName('CUDA')

context = mm.Context(system, integrator, platform)
```

```python
positions = np.zeros([2,3], float) * unit.angstroms

x = np.linspace(1.0, 8.0, 200, endpoint=True) * unit.angstroms
V = [] * unit.kilocalories_per_mole

for xi in x:

    positions[1,0] = xi
    context.setPositions(positions)
    state = context.getState(getEnergy=True)
    potential_energy = state.getPotentialEnergy()
    V.append(potential_energy)
```

```python
sigma = (sigma_Ar+sigma_Xe)/2.0
epsilon = np.sqrt(epsilon_Ar*epsilon_Xe)

V._value = np.array(V._value)

xlim_figure = [0.01, 8.0]
ylim_figure = [-2.0, 10.0]

plt.plot(x, V, linewidth=2)
plt.plot(x, LJ(x, sigma, epsilon), linestyle='--', color='red')
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

## Liquid Argon model

```python
n_particles = 1000
reduced_density = 0.75
mass = 39.948 * unit.amu
sigma = 3.404 * unit.angstroms
epsilon = 0.238 * unit.kilocalories_per_mole
charge = 0.0 * unit.elementary_charge

temperature = 300.0 * unit.kelvin

integration_timestep = 2.0 * unit.femtoseconds
collisions_rate = 1.0 / unit.picoseconds

equilibration_time = 1.0 * unit.nanoseconds

production_time = 5.0 * unit.nanoseconds
saving_time = 50.0 * unit.picoseconds
```

The Van der Waals radius of Argon is 1.88 angstroms.

```python
radius = 2.0**(-5/6) * sigma
print(radius)
```

```python
volume_particles = n_particles * sigma**3
volume = volume_particles/reduced_density
l_box = volume**(1/3)
```

```python
system = mm.System()
```

```python
v1 = np.zeros(3) * unit.angstroms
v2 = np.zeros(3) * unit.angstroms
v3 = np.zeros(3) * unit.angstroms

v1[0] = l_box
v2[1] = l_box
v3[2] = l_box

system.setDefaultPeriodicBoxVectors(v1, v2, v3)
```

```python
non_bonded_force = mm.NonbondedForce()
non_bonded_force.setNonbondedMethod(mm.NonbondedForce.CutoffPeriodic)
non_bonded_force.setCutoffDistance(3.0*sigma)
non_bonded_force.setUseSwitchingFunction(True)
non_bonded_force.setSwitchingDistance(2.0*sigma)
non_bonded_force.setUseDispersionCorrection(True)
```

```python
for _ in range(n_particles):
    system.addParticle(mass)
    non_bonded_force.addParticle(charge, sigma, epsilon)
```

```python
_ = system.addForce(non_bonded_force)
```

```python
space = skopt.Space([[0.0, l_box._value], [0.0, l_box._value], [0.0, l_box._value]])
```

```python
integrator = mm.LangevinIntegrator(temperature, collisions_rate, integration_timestep)
platform = mm.Platform.getPlatformByName('CUDA')
context = mm.Context(system, integrator, platform)
```

```python
grid_generator = skopt.sampler.Grid(use_full_layout=False)
initial_positions = grid_generator.generate(space.dimensions, n_particles)
initial_positions = np.array(initial_positions)*unit.angstroms
```

```python
context.setPositions(initial_positions)
context.setVelocitiesToTemperature(temperature)
```

```python
state=context.getState(getEnergy=True)
print("Before minimization: {}".format(state.getPotentialEnergy()))
mm.LocalEnergyMinimizer_minimize(context)
state=context.getState(getEnergy=True)
print("After minimization: {}".format(state.getPotentialEnergy()))
```

```python
equilibration_n_steps = int(equilibration_time/integration_timestep)
integrator.step(equilibration_n_steps)
context.setTime(0.0*unit.picoseconds)
```

```python
production_n_steps = int(production_time/integration_timestep)
saving_n_steps = int(saving_time/integration_timestep)
n_saving_periods = int(production_n_steps/saving_n_steps)

time = np.zeros([n_saving_periods]) * unit.nanoseconds
trajectory = np.zeros([n_saving_periods, n_particles, 3]) * unit.angstroms
potential_energy = np.zeros([n_saving_periods]) * unit.kilocalories_per_mole

for ii in tqdm(range(n_saving_periods)):
    integrator.step(saving_n_steps)
    state = context.getState(getPositions=True, getEnergy=True)
    time[ii] = state.getTime()
    trajectory[ii,:,:] = state.getPositions(asNumpy=True)
    potential_energy = state.getPotentialEnergy()
```

```python
trajectory_mem = trajectory.size * trajectory.itemsize * unit.bytes
print('Trajectory size: {} GB'.format(trajectory_mem._value/(1024*1024)))
```

```python
l_box
```

```python
trajectory.max()
```

```python
- Calcular difusion de las partículas
- Calcular RDF
- Energy
```


# Integrating Over Parameters, Tractability, MLE, MAP, Bayesian, and Probability Theory

## Probability Theory Basics

**Rule I**: $\displaystyle p(a)=\int p(a,b)\,db$

**Rule II**: $\displaystyle p(a\mid b)=\frac{p(a,b)}{p(b)}$

**Bayes Rule**: $\displaystyle p(a\mid b)=\frac{p(b\mid a)\cdot p(a)}{p(b)}$

## Maximum Likelihood Estimation (MLE)

Method for estimating the parameter(s) of a distribution (?).

Given is a set of points $\mathcal{D}=\{y_n\}_{n=1}^N$.
We _choose to model_ the points with a normal distribution $\mathcal{N}(\mu,\sigma)$ so we can assess the probability of a new point $y$ as $p(y;\mu,\sigma)\sim\mathcal{N}(\mu,\sigma)$.
The goal is to be able to compute $p(y;\mu,\sigma)$ for any $y$.
To be able to do that we need to estimate $\mu$ and $\sigma$ and that is what we use MLE for.

The data is all information we have, so we want $p(\mathcal{D};\mu,\sigma)$ to be high.
We assume the data is i.i.d. given the parameters $\mu$ and $\sigma$.
We can therefore write

$$p(\mathcal{D};\mu,\sigma)=\prod_{n=1}^Np(y_n;\mu,\sigma)$$

What we actually look for are the parameters.
We are interested in the particular set of parameters for which $p(\mathcal{D};\mu,\sigma)$ is maximized.
Let $\mu^\star$ and $\sigma^\star$ denote this maximizer.
We search for
\begin{align}
\mu^\star=&\arg\max_\mu p(\mathcal{D};\mu,\sigma)\\
=&\arg\max_\mu\prod_{n=1}^Np(y_n;\mu,\sigma)\\
=&\arg\max_\mu\log\prod_{n=1}^Np(y_n;\mu,\sigma)\\
=&\arg\max_\mu\sum_{n=1}^N\log p(y_n;\mu,\sigma)
\end{align}

We can apply a $\log$ to the product because it will not change the maximizer.

The same formulation can be written down for $\sigma^\star$ to find the optimal parameter here.

In order to _actually_ get a value for $\mu^\star$ we replace $p(y_n;\mu,\sigma)$ with the definition of our model.
Above it is a normal distribution, so it would be TODO.

Depending on the model there may be a closed-form solution for computing maximizer (i.e., the parameterization for which the data is the most likely) or one can restort to gradient descent to find it.

TODO: apply this to linear regression for example. How can we get $p(y;w)$ there?

## Maximum Aposteriori (MAP)

Method for estimating the parameter(s) of a distribution (?).
Just like MLE, except the probabilistic approach is a bit different.

Previously, with MLE, we modeled $p(\mathcal{D};\mu,\sigma)$, that is, the probability of the data given the parameters.
We then looked for the parameters for which this probability is the highest.
Now we model the probability $p(\mu\mid\mathcal{D})$, that is, the probability of the parameters given the data.
Note that $\sigma$ is also a parameter, it's left out for readability.

Again, analogous to MLE, we seek for the argument that maximizes the probability.
Mathematically speaking, that is

$$\mu^\star=\arg\max_\mu p(\mu\mid\mathcal{D})\,.$$

Following Bayes rule,

$$p(\mu\mid\mathcal{D})=\frac{p(\mathcal{D}\mid\mu)\cdot p(\mu)}{p(\mathcal{D})}\,.$$

We can plug the rewritten version of $p(\mu\mid\mathcal{D})$ into the optimization objective from above:

\begin{align}
\mu^\star=&\arg\max_\mu p(\mu\mid\mathcal{D})\\
=&\arg\max_\mu\frac{p(\mathcal{D}\mid\mu)\cdot p(\mu)}{p(\mathcal{D})}\\
=&\arg\max_\mu p(\mathcal{D}\mid\mu)\cdot p(\mu)
\end{align}

Note that the denominator $p(\mathcal{D})$ can be dropped because it does not depend on $\mu$.

Just like we did with MLE, we first expand $p(\mathcal{D}\mid\mu)$ into a product over all data samples available to us, and second, apply a $\log$ to ease the optimization, without affecting the position of the maximum (the best parameter configuration).

\begin{align}
\mu^\star=&\arg\max_\mu p(\mathcal{D}\mid\mu)\cdot p(\mu)\\
=&\arg\max_\mu\prod_{n=1}^Np(y_n\mid\mu)\cdot p(\mu)\\
=&\arg\max_\mu\log\left(\prod_{n=1}^N p(y_n\mid\mu)\cdot p(\mu)\right)\\
=&\arg\max_\mu\sum_{n=1}^N\log p(y_n\mid\mu)+\log p(\mu)\\
\end{align}

This is very similar to MLE, with the only difference that there is a new component in the optimization problem that we need to consider.
It is the probability of the parameters, $p(\mu)$, also known as the **prior**.

This is where we have to _explicitly_ make an assumption, for example, by saying our parameter is normally distributed. This is in contrast to MLE where this assumption was made implicitly. (?)

TODO: plug in normal distribution definition and show how it resolves to MSE loss with L2 regularization.

## Bayesian Modeling

Instead of just estimating a single value per parameter (like above just a scalar for the mean of our normal distribution), we now **model the parameters as random variables** themselves.

What was previously $\mu$ is now $\mu$ but stands for a random variable, which is defined by a parameterized distribution we choose.

**Terminology**

* Parameters: $\mu$
* Data: $\mathcal{D}$
* Prior: $p(\mu)$
* Posterior: generally _after having seen the data_, i.e., $p(\cdot\mid\mathcal{D})$, for example the updated parameters $p(\mu\mid\mathcal{D})$
* Likelihood: model of the data given the parameters, i.e., $p(y\mid\mu,\mathcal{D})=p(y\mid\mu)$
* Posterior predictive distribution: $p(y\mid\mathcal{D})$
* Evidence: probability of the data $p(\mathcal{D})$
* Marginal: simplified form in which a random variable was integrated away, e.g., $p(y)$ where the $\mu$ was integrated over
* Marginal likelihood: ?

**Posterior** is always after having seen the data.

What we are interested in is modeling the probability of a new (test) data point $y$ given all the information we have, namely the data $\mathcal{D}$. It is called the **posterior predictive distribution**.

\begin{align}
p(y\mid\mathcal{D})=&\int p(y\mid\mu,\mathcal{D})\cdot p(\mu\mid\mathcal{D})\,d\mu\\
=&\int p(y\mid\mu)\cdot p(\mu\mid\mathcal{D})\,d\mu
\end{align}

We constructed the integral using Rule I.
The choice of $\mu$ is an arbitrary one, one could have integrated over anything, but it is a _helpful_ one. In the second equality we remove $\mathcal{D}$ because the data is i.i.d. given the parameter. Having the parameter plus the data does not provide additional information.

For example: We know the mean of a normal distribution. Gathering additional samples from it will not provide us with additional information.

There are not two components in the integral.
The first is the **likelihood** $p(y\mid\mu,\mathcal{D})$ and something we can compute, because it is our model. It is *easy* because it was our model choice, how to model $y$s.

For example: $p(y\mid\mu)=\mathcal{N}(\mu,\sigma=1)$; $\sigma=1$ for brevity.

As for the second part, the so called **posterior distribution of the parameters** $p(\mu\mid\mathcal{D})$, we can again apply Bayes rule as done in MAP above:

$$p(\mu\mid\mathcal{D})=\frac{p(\mathcal{D}\mid\mu)\cdot p(\mu)}{p(\mathcal{D})}\,.$$

This time around we cannot just discard $p(\mathcal{D})$, because we are not looking for the $\arg\max_\mu$ but the actual probability $p(\mu\mid\mathcal{D})$.

There are two difficulties: One is the numerator, one the denominator.

**Numerator**. Because the data is i.i.d. given the parameters we can break it up into a product of the likelihoods of the data samples. Likelihood, because the part that we condition on is unknown, not the one in the front (where it would be a probability).

$$p(\mathcal{D}\mid\mu)\cdot p(\mu)=\prod_{i=1}^Np(y_i\mid\mu)\cdot p(\mu)$$

The two distributions which we multiply with each other can easily (once multiplied) into a new distribution which is hard to handle / unknown. In some cases, though, for example, for two normal distributions we know what the product is (also normal).

More generally, if the two distributions (likelihood and prior) are **conjugate** to each other, the resulting posterior has the same as the distribution as the prior. Examples are:
* normal and normal -> normal,
* bernoulli and beta -> beta,
* multinoulli and dirichlet --> dirichlet,
* ...

Example.

Our prior is $$p(\mu)=\operatorname{Beta}(1,1)=\frac{\mu^{\alpha-1}(1-\mu)^{\beta-1}}{B(\alpha,\beta)}$$

with ${\displaystyle \mathrm {B} (\alpha ,\beta )={\frac {\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha +\beta )}}}$.

The likelihood is $$p(y\mid\mu)=\operatorname{Ber}(\mu)=\mu^{y}(1-\mu)^{1-y}$$

This gives the numerator

$$\mu^{y}(1-\mu)^{1-y}\cdot\frac{\mu^{\alpha-1}(1-\mu)^{\beta-1}}{B(\alpha,\beta)}\,.$$

For a data point $y=1$ it is

\begin{align}
&\mu\cdot\frac{\mu^{\alpha-1}(1-\mu)^{\beta-1}}{B(\alpha,\beta)}\\
=&\frac{1}{B(\alpha,\beta)}\cdot\mu\cdot\mu^{\alpha-1}(1-\mu)^{\beta-1}\\
=&\frac{1}{B(\alpha,\beta)}\cdot\mu^{\underbrace{(\alpha+1)}_{\text{new }\alpha}-1}(1-\mu)^{\beta-1}\\
\end{align}

It is apparent how the multiplication with Bernoulli did not change the distribution type.
We still have a Beta distribution and its parameter $\alpha$ has changed slightly to turn into $\alpha+1$.
Had our data point been $y=0$, would $\alpha$ have stayed the same whereas $\beta$ would have been incremented.

**Denominator**. $p(\mathcal{D})$ is just a constant. So the part that actually determines the _distribution part_ of $p(\mu\mid\mathcal{D})$ is the numerator.

In the following, $\mathcal{D}=\{y\}$, i.e., there is only a single data point. Taking the numerator for the example from above and plugging it back into the fraction we get

\begin{align}
p(\mu\mid\mathcal{D})=&\frac{\mu^{y}(1-\mu)^{1-y}\cdot\frac{\mu^{\alpha-1}(1-\mu)^{\beta-1}}{B(\alpha,\beta)}}{p(\mathcal{D})}\\
=&\underbrace{\frac{1}{p(\mathcal{D})\cdot B(\alpha,\beta)}}_{\text{numbers}}\cdot\underbrace{\mu^{y}(1-\mu)^{1-y}\cdot\mu^{\alpha-1}(1-\mu)^{\beta-1}}_{\text{depends on }\mu}\\
=&\underbrace{\frac{1}{p(\mathcal{D})\cdot B(\alpha,\beta)}}_{\text{numbers}}\cdot\underbrace{\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}}_{\text{depends on }\mu}
\end{align}

The additional information comes from the fact that the distribution has to integrate to 1. That helps us determine the unknown component $p(\mathcal{D})$.

\begin{align}
\int p(\mu\mid\mathcal{D})\,d\mu=&1\\
\int \frac{1}{p(\mathcal{D})\cdot B(\alpha,\beta)}\cdot\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu=&1\\
\frac{1}{p(\mathcal{D})\cdot B(\alpha,\beta)}\cdot\int\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu=&1\\
\int\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu=&\underbrace{p(\mathcal{D})}_\text{unknown}\cdot B(\alpha,\beta)\\
\int\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu=&B(\alpha+y,\beta+1-y)
\end{align}

The last equality holds because we know that the left side is an unnormalized $\operatorname{Beta}(\alpha+y,\beta+1-y)$ distribution PDF. So we define the right-hand side to be its corresponding normalization constant, which is $B(\alpha+y,\beta+1-y)$. And $p(\mathcal{D})$ is just a part of it.

$$p(\mathcal{D})=\frac{B(\alpha+y,\beta+1-y)}{B(\alpha,\beta)}$$

---

Now we jump back to the posterior predictive distribution, where we aim to determine the probability of a test point $y^*$ given the data. The above paragraphs helped determine the $p(\mu\mid\mathcal{D})$ part of it, and the likelihood is given by $p(y\mid\mu)=\operatorname{Ber}(\mu)=\mu^{y}(1-\mu)^{1-y}$

\begin{align}
p(y^*\mid\mathcal{D})=&\int p(y^*\mid\mu)\cdot p(\mu\mid\mathcal{D})\,d\mu=\mathbb{E}_{p(\mu\mid\mathcal{D})}\left[p(y^*\mid\mu)\right]\\
=&\int p(y^*\mid\mu)\cdot\frac{1}{p(\mathcal{D})\cdot B(\alpha,\beta)}\cdot\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu\\
=&\int p(y^*\mid\mu)\cdot\frac{1}{B(\alpha+y,\beta+1-y)}\cdot\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu\\
=&\int \mu^{y^*}(1-\mu)^{1-y^*}\cdot\frac{1}{B(\alpha+y,\beta+1-y)}\cdot\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu\\
=&\frac{1}{B(\alpha+y,\beta+1-y)}\cdot\int \mu^{y^*}(1-\mu)^{1-y^*}\cdot\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu\\
=&\frac{1}{B(\alpha+y,\beta+1-y)}\cdot\int \mu^{y^*}(1-\mu)^{1-y^*}\cdot\mu^{(\alpha+y)-1}(1-\mu)^{(\beta+1-y)-1}\,d\mu\\
=&\frac{1}{B(\alpha+y,\beta+1-y)}\cdot\int \mu^{(\alpha+y+y^*)-1}(1-\mu)^{(\beta+1-y+1-y^*)-1}\,d\mu\\
=&\frac{1}{B(\alpha+y,\beta+1-y)}\cdot B(\alpha+y+y^*,\beta+1-y+1-y^*)\\
=&\frac{B(\alpha+y+y^*,\beta+1-y+1-y^*)}{B(\alpha+y,\beta+1-y)}\\
\end{align}

\begin{align}
p(y^*\mid\mathcal{D})=&\begin{cases}
\underbrace{\frac{B(\alpha+y,\beta-y+2)}{B(\alpha+y,\beta+1-y)}}_{1-\mu?} & \text{if }y^*=0\\
\underbrace{\frac{B(\alpha+y+1,\beta+1-y)}{B(\alpha+y,\beta+1-y)}}_{\mu?} & \text{if }y^*=1
\end{cases}\\
=&\operatorname{Ber}\left(\frac{B(\alpha+y+1,\beta+1-y)}{B(\alpha+y,\beta+1-y)}\right)
\end{align}

Note: Here $y$ is the data that we learned from (a single point).

Here we would have almost given up. What we hope here is (as **successfully** validated in the following) that

\begin{align}
\frac{B(\alpha+y,\beta-y+2)}{B(\alpha+y,\beta-y+1)}+\frac{B(\alpha+y+1,\beta-y+1)}{B(\alpha+y,\beta-y+1)}\overset{?}{=}&1\\\\
\frac{B(\alpha+y,\beta-y+2)+B(\alpha+y+1,\beta-y+1)}{B(\alpha+y,\beta-y+1)}\overset{?}{=}&1\\
\end{align}

With the definition of $B$

$${\displaystyle \mathrm {B} (\alpha ,\beta )={\frac {\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha +\beta )}}}$$

we get the fraction

\begin{align}
\frac{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+2)}{\Gamma(\alpha+y+\beta-y+2)}+\frac{\Gamma(\alpha+y+1)\Gamma(\beta-y+1)}{\Gamma(\alpha+y+1+\beta-y+1)}}{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+1)}{\Gamma(\alpha+y+\beta-y+1)}}\overset{?}{=}&1\\\\
\frac{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+2)}{\Gamma(\alpha+\beta+2)}+\frac{\Gamma(\alpha+y+1)\Gamma(\beta-y+1)}{\Gamma(\alpha+\beta+2)}}{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+1)}{\Gamma(\alpha+\beta+1)}}\overset{?}{=}&1\\\\
\frac{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+2)+\Gamma(\alpha+y+1)\Gamma(\beta-y+1)}{\Gamma(\alpha+\beta+2)}}{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+1)}{\Gamma(\alpha+\beta+1)}}\overset{?}{=}&1\\
\end{align}

With the definition of ${\displaystyle \Gamma (n)=(n-1)!}$ we continue to pursue our goals at 00:43 am...

$$\Gamma (n+1)=\Gamma(n)\cdot n=n!$$

\begin{align}
\frac{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+2)+\Gamma(\alpha+y+1)\Gamma(\beta-y+1)}{\Gamma(\alpha+\beta+2)}}{\frac{\Gamma(\alpha+y)\Gamma(\beta-y+1)}{\Gamma(\alpha+\beta+1)}}\overset{?}{=}&1\\\\
\frac{
    (\Gamma(\alpha+y)
    \Gamma(\beta-y+2)+\Gamma(\alpha+y+1)\Gamma(\beta-y+1))\cdot\Gamma(\alpha+\beta+1)}
    {\Gamma(\alpha+\beta+2)\cdot\Gamma(\alpha+y)\cdot\Gamma(\beta-y+1)}\overset{?}{=}&1\\\\
\frac{
    (\Gamma(\alpha+y)
    \Gamma(\beta-y+2)+\Gamma(\alpha+y+1)\Gamma(\beta-y+1))}
    {(\alpha+\beta+1)\cdot\Gamma(\alpha+y)\cdot\Gamma(\beta-y+1)}\overset{?}{=}&1\\\\
\frac{
    (
    \Gamma(\beta-y+2)+(\alpha+y)\Gamma(\beta-y+1))}
    {(\alpha+\beta+1)\cdot\Gamma(\beta-y+1)}\overset{?}{=}&1\\\\
\frac{\beta-y+1+\alpha+y}{\alpha+\beta+1}\overset{?}{=}&1\\\\
\frac{\beta+1+\alpha}{\alpha+\beta+1}\overset{!}{=}&1\\\\
\end{align}

---

What we did above is a special case. It's an example for a conjugate prior (bernoulli likelihood and beta prior) in which a closed-form solution was available. In cases where there is no closed-form solution, one can solve

$$p(y^*\mid\mathcal{D})=\int p(y^*\mid\mu)\cdot p(\mu\mid\mathcal{D})\,d\mu=\mathbb{E}_{p(\mu\mid\mathcal{D})}\left[p(y^*\mid\mu)\right]$$

by sampling from $p(\mu\mid\mathcal{D})$ to approximate the expectation (**Monte Carlo**). So here we would approximate $p(y^*\mid\mu)$.

Alternatively, one can (probably?) approximate $p(\mathcal{D}\mid\mu)$ in

$$p(\mu\mid\mathcal{D})=\frac{p(\mathcal{D}\mid\mu)\cdot p(\mu)}{p(\mathcal{D})}\,.$$

**Variational inference** (probably?) fits into this picture where one says we define $q$ to approximate $p(y^*\mid\mathcal{D})$ by minimizing the KL between the two.


# First exploration of finite-difference approximations to derivatives

Note: Prof. Brown's [differentiation notebook](https://github.com/cu-numcomp/numcomp-class/blob/master/Differentiation.ipynb) for CSCI-3656 is quite good, and covers [algorithmic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation) using JAX.  [](https://colab.research.google.com/github/cu-numcomp/numcomp-class/blob/master/Differentiation.ipynb)

- Task 1: try the simple forward differences formula to approximate the derivative of $f(x)=\sin(x)$. The formula is:
$$f'(x) \approx \frac{ f(x+h)-f(x) }{h}$$
Try this for a range of stepsizes $h$. Can we make the approximation (aka "truncation") error arbitrarily small?

```
import numpy as np
import matplotlib.pyplot as plt

# Approximate the derivative of sin(x)
f   = lambda x : np.sin(x)
trueDerivative = lambda x : np.cos(x)

def twoPointForwardDifferences( f, x, h ):
  """ The simplest, plain vanilla finite difference approx"""
  return ( f(x+h) - f(x) )/h

# Pick some point x at which we want the derivative
x   = 2.
df      = trueDerivative(x)
twoPointF= lambda h : twoPointForwardDifferences( f, x, h)

# Pick stepsizes h
hList   = np.logspace(-1,-12,30)

# Plot
plt.loglog( hList, np.abs( df - twoPointF(hList) ), 'o-', label="2 Pt Forward Diff" );
plt.xlabel("h");
plt.ylabel("Error");
```

## Getting fancier: adding in other methods
- Task 2: try other finite difference methods (e.g, 3 point forward differences; 3 point centered differences; 5 point centered differences)

```
import numpy as np
import matplotlib.pyplot as plt

# Approximate the derivative of sin(x)
f   = lambda x : np.sin(x)
trueDerivative = lambda x : np.cos(x)

def twoPointForwardDifferences( f, x, h ):
  """ The simplest, plain vanilla finite difference approx"""
  return ( f(x+h) - f(x) )/h
def threePointForwardDifferences( f, x, h):
  """Eq. (4.4) in the Burden and Faires book"""
  return (-3*f(x)+4*f(x+h)-f(x+2*h))/(2*h)
def threePointCenteredDifferences( f, x, h ):
  """ Book calls it 3 points, since it sort of includes f(x) """
  return ( f(x+h) - f(x-h) )/(2*h)
def fivePointCenteredDifferences( f, x, h ):
  """ Eq. (4.6) in Burden and Faires book """
  return ( f(x-2*h) - 8*f(x-h) + 8*f(x+h) - f(x+2*h) )/(12*h)

# Pick some point x at which we want the derivative
x   = 2.
df      = trueDerivative(x)
twoPointF= lambda h : twoPointForwardDifferences( f, x, h)
threePointC= lambda h : threePointCenteredDifferences( f, x, h)
threePointF= lambda h : threePointForwardDifferences( f, x, h)
fivePointC= lambda h : fivePointCenteredDifferences( f, x, h)

# Pick stepsizes h
hList   = np.logspace(-1,-12,20)

# Plot
plt.figure(figsize=(10,8))
plt.loglog( hList, np.abs( df - twoPointF(hList) ), 'o-', label="2 Pt Forward Diff" );
plt.loglog( hList, np.abs( df - threePointC(hList) ), 'o-', label="3 Pt Centered Diff" );
plt.loglog( hList, np.abs( df - threePointF(hList) ), 'o-', label="3 Pt Forward Diff" );
plt.loglog( hList, np.abs( df - fivePointC(hList) ), 'o-', label="5 Pt Centered Diff" )
plt.legend();
plt.xlabel("h");
plt.ylabel("Error");
```

## Can we estimate how low the error can be?
It's a trade off between truncation error (low $h$ is good), and roundoff error (low $h$ is bad)
- Task 3: can we predict how low each method gets? Assume error looks like
$$\epsilon/h + h^k$$ where $k$ is the order, and use `sympy` to minimize this (e.g., differentiate it and set it equal to 0).  You may want to declare $\epsilon$ as a symbol like:  `e = sym.symbols('\epsilon',positive=True)`

```
import sympy as sym
from sympy import init_printing
init_printing()

h = sym.symbols('h')
e = sym.symbols('\epsilon',positive=True)
g = e/h + h
d = sym.diff( g,  h)
d
h0 = list( sym.solveset( d, h, domain=sym.S.Reals) )[0]
print("Order 1, (optimal h, value-at-optimal)")
h0, g.subs(h,h0)
```

```
g = e/h + h**2
d = sym.diff( g,  h)
h0 = list( sym.solveset( d, h, domain=sym.S.Reals) )[0]
print("Order 2")
h0, g.subs(h,h0)
```

```
g = e/h + h**3
d = sym.diff( g,  h)
h0 = list( sym.solveset( d, h, domain=sym.S.Reals) )[0]
print("Order 3")
h0, g.subs(h,h0)
```

```
g = e/h + h**4
d = sym.diff( g,  h)
h0 = list( sym.solveset( d, h, domain=sym.S.Reals) )[0]
print("Order 4")
h0, g.subs(h,h0)
```

### And put these in the plot
- 2 pt forward diff is $O(h)$
- 3 pt forward diff is $O(h^2)$
- 3 pt centered diff is $O(h^2)$
- 5 pt centered diff is $O(h^4)$

```
# So add to the plot
plt.figure(figsize=(10,8))
plt.loglog( hList, np.abs( df - twoPointF(hList) ), 'o-', label="2 Pt Forward Diff" );
plt.loglog( hList, np.abs( df - threePointC(hList) ), 'o-', label="3 Pt Centered Diff" );
plt.loglog( hList, np.abs( df - threePointF(hList) ), 'o-', label="3 Pt Forward Diff" );
plt.loglog( hList, np.abs( df - fivePointC(hList) ), 'o-', label="5 Pt Centered Diff" )
plt.legend();
plt.xlabel("h");
plt.ylabel("Error");

# And can we predict the values of h which gives us the smallest amount?
eps = np.finfo(float).eps
eps**(1/2)
eps**(1/4)
plt.vlines( eps**(1/2), *plt.gca().get_ylim() ,linestyles='--',colors="C0");
plt.vlines( eps**(1/3), *plt.gca().get_ylim() ,linestyles='--',colors="C1");
plt.vlines( eps**(1/5), *plt.gca().get_ylim() ,linestyles='--',colors="C3");

plt.hlines( eps**(1/2), *plt.gca().get_xlim() ,linestyles='--',colors="C0");
plt.hlines( eps**(2/3), *plt.gca().get_xlim() ,linestyles='--',colors="C1");
plt.hlines( eps**(4/5), *plt.gca().get_xlim() ,linestyles='--',colors="C3");
```


```python
def f(x):
    return x**3 -3*x**2 + x
```

```python
from scipy.misc import derivative

print(derivative(f, 0, dx=1e-6))
print(derivative(f, 1, dx=1e-6))
```

    1.000000000001
    -2.000000000002

```python
def relu(x):
    return np.where(x>0, x, 0)

xx = np.linspace(-2, 2, 100)
plt.plot(xx, relu(xx))
plt.plot(0, 0, marker="o", ms=10)
plt.title("ReLu")
plt.xlabel("$x$")
plt.ylabel("ReLU(x)")
plt.show()
```

```python
import sympy
sympy.init_printing(use_latex='mathjax')

```

```python
x = sympy.symbols('x')
x
```

$\displaystyle x$

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
f = x*sympy.exp(x)
f
```

$\displaystyle x e^{x}$

```python
sympy.diff(f)
```

$\displaystyle x e^{x} + e^{x}$

```python
sympy.simplify(sympy.diff(f))
```

$\displaystyle \left(x + 1\right) e^{x}$

```python
x, y = sympy.symbols('x y')
f = x**2 + 4*x*y + 4*y**2
f
```

$\displaystyle x^{2} + 4 x y + 4 y^{2}$

```python
sympy.diff(f, x)
```

$\displaystyle 2 x + 4 y$

```python
sympy.diff(f, y)
```

$\displaystyle 4 x + 8 y$

```python
x, mu, sigma = sympy.symbols('x mu sigma')
f = sympy.exp((x-mu)**2/sigma**2)
f
```

$\displaystyle e^{\frac{\left(- \mu + x\right)^{2}}{\sigma^{2}}}$

```python
sympy.diff(f, x)
```

$\displaystyle \frac{\left(- 2 \mu + 2 x\right) e^{\frac{\left(- \mu + x\right)^{2}}{\sigma^{2}}}}{\sigma^{2}}$

```python
sympy.simplify(sympy.diff(f,x))
```

$\displaystyle \frac{2 \left(- \mu + x\right) e^{\frac{\left(\mu - x\right)^{2}}{\sigma^{2}}}}{\sigma^{2}}$

```python
sympy.diff(f, x, x)
```

$\displaystyle \frac{2 \left(1 + \frac{2 \left(\mu - x\right)^{2}}{\sigma^{2}}\right) e^{\frac{\left(\mu - x\right)^{2}}{\sigma^{2}}}}{\sigma^{2}}$

```python
f = x**3 - 1
```

```python
sympy.diff(f, x)
```

$\displaystyle 3 x^{2}$

```python
k = sympy.symbols('k')
f = sympy.log(x**2- 3*k)
sympy.simplify(sympy.diff(f, x))
```

$\displaystyle \frac{2 x}{- 3 k + x^{2}}$

```python
a,b = sympy.symbols('a b')
```

```python
f = sympy.exp(a*(x**b))
f
```

$\displaystyle e^{a x^{b}}$

```python
sympy.simplify(sympy.diff(f, x))
```

$\displaystyle a b x^{b - 1} e^{a x^{b}}$

```python
y = sympy.symbols('y')
```

```python
f = sympy.exp(x**2 + 2*(y**2))

fx = sympy.simplify(sympy.diff(f, x))
fy = sympy.simplify(sympy.diff(f, y))
fxx = sympy.simplify(sympy.diff(f, x, x))
fxy = sympy.simplify(sympy.diff(f, x, y))
fyx = sympy.simplify(sympy.diff(f, y, x))
fyy = sympy.simplify(sympy.diff(f, y, y))

```

    2*x*exp(x**2 + 2*y**2) 4*y*exp(x**2 + 2*y**2) (4*x**2 + 2)*exp(x**2 + 2*y**2) 8*x*y*exp(x**2 + 2*y**2) 8*x*y*exp(x**2 + 2*y**2) (16*y**2 + 4)*exp(x**2 + 2*y**2)

```python
fx, fy, fxx, fxy, fyx, fyy
```

$\displaystyle \left( 2 x e^{x^{2} + 2 y^{2}}, \  4 y e^{x^{2} + 2 y^{2}}, \  \left(4 x^{2} + 2\right) e^{x^{2} + 2 y^{2}}, \  8 x y e^{x^{2} + 2 y^{2}}, \  8 x y e^{x^{2} + 2 y^{2}}, \  \left(16 y^{2} + 4\right) e^{x^{2} + 2 y^{2}}\right)$

```python
x = sympy.symbols('x')
f = x*sympy.exp(x) + sympy.exp(x)
f
```

$\displaystyle x e^{x} + e^{x}$

```python
sympy.integrate(f)
```

$\displaystyle x e^{x}$

```python
x, y = sympy.symbols('x y')
f = 2*x + y
f
```

$\displaystyle 2 x + y$

```python
sympy.integrate(f, x)
```

$\displaystyle x^{2} + x y$

```python
f = 3*(x**2)
sympy.integrate(f)
```

$\displaystyle x^{3}$

```python
f = 3*(x**2) - 6*x + 1
```

```python
sympy.integrate(f)
```

$\displaystyle x^{3} - 3 x^{2} + x$

```python
f = 2 + 6*x + 4*(sympy.exp(x)) + 5*(x**-1)
```

```python
f
```

$\displaystyle 6 x + 4 e^{x} + 2 + \frac{5}{x}$

```python
sympy.integrate(f)
```

$\displaystyle 3 x^{2} + 2 x + 4 e^{x} + 5 \log{\left(x \right)}$

```python
f = 2*x/x**2 -1
```

```python
sympy.integrate(f)
```

$\displaystyle - x + 2 \log{\left(x \right)}$

```python
g = 1+(x*y)
g
```

$\displaystyle x y + 1$

```python
sympy.integrate(g, x)
```

$\displaystyle \frac{x^{2} y}{2} + x$

```python
g = x*y*(sympy.exp(x**2 + y**2))
```

```python
sympy.integrate(g, x)
```

$\displaystyle \frac{y e^{x^{2} + y^{2}}}{2}$

```python
f = x**3 - 3*x**2 + x + 6
```

```python
F = sympy.integrate(f)
```

```python
(F.subs(x, 2) - F.subs(x, 0)).evalf()
```

$\displaystyle 10.0$

```python
import scipy as sp
def f(x):
    return x**3 - 3*x**2 + x + 6

sp.integrate.quad(f, 0, 2)

```

$\displaystyle \left( 10.0, \  1.1102230246251565e-13\right)$

```python
import scipy as sp
def f(x, y):
    return np.exp(-x * y) / y**2

sp.integrate.dblquad(f, 1, np.inf, lambda x:0, lambda x:np.inf)
```

$\displaystyle \left( 0.4999999999999961, \  1.068453874338024e-08\right)$

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
_x = np.arange(12) / 2 + 2
_y = np.arange(12) / 2
X, Y = np.meshgrid(_x, _y)
x, y = X.ravel(), Y.ravel()
z = x * x - 10 * x + y + 50
z0 = np.zeros_like(z)
ax.bar3d(x, y, z0, 0.48, 0.48, z)
ax.set_xlim(0, 10)
ax.set_ylim(-2, 10)
ax.set_zlim(0, 50)
ax.set_xlabel("x")
ax.set_ylabel("y")
plt.title("f(x, y)")
plt.show()

```

```python
from mpl_toolkits.mplot3d import Axes3D
```

```python
#연습문제 4.3.6
def f(x, y):
    return (1+x*y)
sp.integrate.dblquad(f, -1, 1, lambda x:-1, lambda x:1 )
```

$\displaystyle \left( 4.0, \  4.440892098500626e-14\right)$

```python

```


## 10. Resolver Ecuaciones

[Playlist de Ciencia de Datos en castellano](https://www.youtube.com/playlist?list=PLjyvn6Y1kpbEmRY4-ELeRA80ZywV7Xd67)
[](https://www.youtube.com/watch?v=c40z75JnT44&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy "Python Data Science")

Las ecuaciones son fundamentales en Ciencia de Datos. Permiten convertir datos en información procesable mediante el desarrollo de expresiones matemáticas que describen sistemas físicos. Algunas expresiones matemáticas son simples y pueden calcularse secuencialmente, por ejemplo:

$x=1 \quad y=x^2+2x-4$

La solución es $x=1$, $y=1+2-4=-1$.

Considera el caso donde $x$ también depende de $y$.

$x=y \quad y=x^2+2x-4$

Hay dos soluciones que se calculan a partir de la fórmula cuadrática $y=\frac{-b\pm\sqrt{b^2-4ac}}{2a}$

$0=y^2+(2y-y)-4$,

$\quad y^2+y-4 = 0$,    con $a=1$, $b=1$ y $c=-4$.

$y = \frac{-1 \pm \sqrt{17}}{2} = {1.56,-2.56}$

Hay dos métodos para resolver este problema. El primero es una **solución numérica**, donde la computadora usa métodos de prueba y error para hallar la solución. Se utilizan métodos numéricos cuando hay un gran número de ecuaciones y no hay solución analítica. El segundo método es una **solución simbólica** que genera una solución exacta.

### Solución Numérica

Los problemas complejos y a gran escala utilizan una solución numérica, por ejemplo con `fsolve` o `gekko`. Se requiere una función que devuelva el error residual de la ecuación. Este residual es $f(y)=y^2+y-4$, que no es igual a cero cuando el valor de $y$ no es la solución correcta. Un valor inicial de `1` o `-2` da un resultado diferente porque se empieza cerca de uno u otro.

#### Solución con Scipy fsolve

```python
from scipy.optimize import fsolve
def f(y):
    return y**2+y-4
z = fsolve(f,1); print(z)
z = fsolve(f,-2); print(z)
```

**Solución con Python Gekko**

```python
from gekko import GEKKO
m = GEKKO(remote=False)
#Declaración de la variable y. y=1
y = m.Var(1)
#Declaración de la ecuación
m.Equation(y**2+y-4==0)
#Resolución
m.solve(disp=False)
print(y.value)
#Valor inicial y=-2
y.value = -2
m.solve(disp=False); print(y.value)
```

### Resolver 2 Ecuaciones

Es similar tener una o dos ecuaciones

$y=x^2+2x-4$

$x=y$

La función devuelve el error residual de cada ecuación como una lista.

Se requieren dos valores iniciales.

Este mismo método también se aplica para más ecuaciones. Los algoritmos que resuelven ecuaciones (solvers) pueden solucionar problemas con miles o millones de variables!!

**Solución con Scipy `fsolve`**

```python
from scipy.optimize import fsolve
def f(z):
    x,y = z
    return [x-y,y-x**2-2*x+4]
z = fsolve(f,[1,1]); print(z)
z = fsolve(f,[-2,-2]); print(z)
```

**Solución con Python Gekko**

```python
m = GEKKO(remote=False)
x=m.Var(); y = m.Var(1);
m.Equations([y==x**2+2*x-4, x==y])
m.solve(disp=False)
print(x.value, y.value)

x.value=-2; y.value=-2
m.solve(disp=False)
print(x.value, y.value)
```

### Resolver 3 Ecuaciones

$x^2+y^2+z^2=1$

$x-2y+3z=0.5$

$x+y+z=0$

Resuelve el problema con 3 variables y 3 ecuaciones.

**Solución con Scipy `fsolve`**

```python

```

**Solución con Python Gekko**

```python

```

### Solución Simbólica

Es posible expresar simbólicamente la solución analítica de problemas simples. Una librería con símbolos matemáticos en Python es `sympy`. La función `display` está disponible para imprimir la ecuación en Jupyter notebook. Se requiere importar `from IPython.display import display`.

```python
from IPython.display import display
import sympy as sym
x = sym.Symbol('x')
y = sym.Symbol('y')
ans = sym.nonlinsolve([x-y, y-x**2-2*x+4], [x,y])
display(ans)
```

### Resolver Simbólicamente 3 Ecuaciones

$x\,y\,z=0$

$x\,y=0$

$x+5\,y+z$

Resuelve el problema con 3 variables y 3 ecuaciones de forma simbólica. El problema está subespecificado, por lo que una de las variables aparecerá en la solución; es decir, hay un set infinito de soluciones.

```python

```

### Ecuaciones Lineales

Puedes resolver ecuaciones lineales en Python. Existen métodos eficientes como `x = np.linalg.solve(A,b)` para resolver ecuaciones de tipo $A x = b$ con la matriz $A$ y los vectores $x$ y $b$.

$A = \begin{bmatrix}3 & 2\\ 1 & 2 \end{bmatrix} \quad b = \begin{bmatrix}1 \\ 0 \end{bmatrix}$

```python
import numpy as np
A = np.array([[3,2],[1,2]])
b = np.array([1,0])

x = np.linalg.solve(A,b)
print(x)
```

La solución simbólica para este set de ecuaciones lineales está disponible con la función `sympy` `linsolve`. Si el problema es lineal, se prefiere usar `linsolve` porque es más eficiente que `nonlinsolve`, pero puede resolver los dos.

```python
import sympy as sym
x, y = sym.symbols('x y')
ans = sym.linsolve([3*x + 2*y - 1, x + 2*y], (x, y))
sym.pprint(ans)
```

### Optimización

Cuando hay más variables que ecuaciones, el problema está subespecificado y no puede hallarse solución con una función como `fsolve` (para problemas lineales y no lineales) o `linalg.solve` (solo para problemas lineales). Se requiere información adicional para guiar la selección de las variables extra.

Una función objetivo $J(x)$ es una manera de especificar el problema para que exista solución única. El objetivo es minimizar  $x_1 x_4 \left(x_1 + x_2 + x_3\right) + x_3$. Las dos ecuaciones que guían la selección de dos variables son las restricciones de desigualdad $\left(x_1 x_2 x_3 x_4 \ge 25\right)$ y de igualdad $\left(x_1^2 + x_2^2 + x_3^2 + x_4^2 = 40\right)$. Las cuatro variables deben estar entre `1` (límite inferior) y `5` (límite superior).

$\quad \min x_1 x_4 \left(x_1 + x_2 + x_3\right) + x_3$

$\quad \mathrm{s.t.:} \quad x_1 x_2 x_3 x_4 \ge 25$

$\quad x_1^2 + x_2^2 + x_3^2 + x_4^2 = 40$

$\quad 1\le x_1, x_2, x_3, x_4 \le 5$

con valores iniciales:

$\quad x_0 = (1,5,5,1)$

Información adicional sobre optimización encontrarás en el [Curso de Diseño y Optimización en inglés](https://apmonitor.com/me575) y en el [Libro de Diseño y Optimización en inglés](https://apmonitor.com/me575/index.php/Main/BookChapters).

Referencia:
Optimization Methods for Engineering Design, Parkinson, A.R., Balling, R., and J.D. Hedengren, Second Edition, Brigham Young University, 2018.

El primer método de resolución es con `scipy.optimize.minimize`. Las funciones en esta librería trabajan bien con problemas de tamaño moderado y modelos de caja negra, donde una función objetivo está disponible a través de la llamada a una función (function call).

```python
import numpy as np
from scipy.optimize import minimize

def objective(x):
    return x[0]*x[3]*(x[0]+x[1]+x[2])+x[2]

def constraint1(x):
    return x[0]*x[1]*x[2]*x[3]-25.0

def constraint2(x):
    sum_eq = 40.0
    for i in range(4):
        sum_eq = sum_eq - x[i]**2
    return sum_eq

# condiciones iniciales
n = 4
x0 = np.zeros(n)
x0[0] = 1.0
x0[1] = 5.0
x0[2] = 5.0
x0[3] = 1.0

# optimización
b = (1.0,5.0)
bnds = (b, b, b, b)
con1 = {'type': 'ineq', 'fun': constraint1}
con2 = {'type': 'eq', 'fun': constraint2}
cons = ([con1,con2])
solution = minimize(objective,x0,method='SLSQP',\
                    bounds=bnds,constraints=cons)
x = solution.x

# mostrar objetivo final
print('Final Objective: ' + str(objective(x)))

# imprimir solución
print('Solution')
print('x1 = ' + str(x[0]))
print('x2 = ' + str(x[1]))
print('x3 = ' + str(x[2]))
print('x4 = ' + str(x[3]))
```

### Optimización con Gekko

[Python Gekko](https://gekko.readthedocs.io/en/latest/) también resuelve problemas de optimización. Usa diferenciación automática y algoritmos (solvers) basados en gradientes como `APOPT` o `IPOPT` para hallar una solución. Este método de solución es mejor para problemas a gran escala. [Tutoriales Adicionales sobre Gekko en inglés](https://apmonitor.com/wiki/index.php/Main/GekkoPythonOptimization) muestran cómo resolver otro tipo de problemas de optimización.

```python
from gekko import GEKKO
m = GEKKO(remote=False)

# Inicializar Variables
x1,x2,x3,x4 = [m.Var(lb=1, ub=5) for i in range(4)]

# Condiciones Iniciales
x1.value = 1
x2.value = 5
x3.value = 5
x4.value = 1

# Ecuaciones
m.Equation(x1*x2*x3*x4>=25)
m.Equation(x1**2+x2**2+x3**2+x4**2==40)

# Objetivo
m.Obj(x1*x4*(x1+x2+x3)+x3)

# Resolver
m.solve(disp=False)

# Objectivo Final
print('Final Objective: ' + str(m.options.objfcnval))

# Imprimir soluciones
print('Solution')
print('x1: ' + str(x1.value))
print('x2: ' + str(x2.value))
print('x3: ' + str(x3.value))
print('x4: ' + str(x4.value))
```

### Actividad con el TCLab

### Recolección de Datos

Enciende el calentador 1 al 100% y graba $T_1$ cada 10 segundos por 3 minutos. Los datos deben incluir un total de 19 puntos para cada sensor de temperatura y el tiempo, iniciando en cero. Toma nota de los puntos de temperatura en 0, 90 y 180 segundos.

```python

```

### Ecuaciones Lineales

Se requieren tres puntos para especificar un polinomio de grado 2 de la forma $y =a_0 + a_1 \; x + a_2 \; x^2$. Crea una regresión cuadrática para $T_2$ usando solo el primer, medio y último punto. Supón que los puntos para $T_2$ son los siguientes:

| Tiempo (sec) | Temperatura (°C)  |
|------|------|
| 0    | 23.0 |
| 90    | 33.0 |
| 180    | 43.0 |

Resuelve la regresión lineal como un set de tres ecuaciones que se derivan conectando los tres puntos a la ecuación polinómica. Crea tres ecuaciones que consideren $y=T_2$ y $x=tiempo$.

$\quad a_0 + a_1 \; 0 + a_2 \; 0^2 = 23.0$

$\quad a_0 + a_1 \; 90 + a_2 \; 90^2 = 33.0$

$\quad a_0 + a_1 \; 180 + a_2 \; 180^2 = 43.0$

En forma de matriz, el set de ecuaciones lineales es:

$\quad \begin{bmatrix}1 & 0 & 0 \\ 1 & 90 & 90^2 \\ 1 & 180 & 180^2 \end{bmatrix}\begin{bmatrix}a_0\\a_1\\a_2\end{bmatrix} = \begin{bmatrix}23.0\\33.0\\43.0\end{bmatrix}$

Resuelve el set de ecuaciones para hallar los parámetros cuadráticos $a_0$, $a_1$ y $a_2$ con los datos recolectados al inicio de la actividad con el TCLab. Dibuja el ajuste cuadrático con los datos para asegurar que la curva pasa por los tres puntos indicados.

```python

```

### Ecuaciones No Lineales

Ajusta los datos de $T_1$ a una correlación no lineal usando solo tres puntos.

$\quad T_1 = a + b \exp{(c \, tiempo)}$

Se requieren únicamente tres puntos para especificar un modelo con tres parámetros. Cuando hay más del mínimo número de puntos requerido, usualmente se ejecuta una regresión de mínimos cuadrados para minimizar el cuadrado del error entre los valores medidos y predecidos. Para este ejercicio, usa solo tres puntos (primero, medio y último) de los datos $T_1$. Supón que los puntos para $T_1$ son los siguientes:

| Tiempo (sec) | Temperatura (°C)  |
|------|------|
| 0    | 22.0 |
| 90    | 42.0 |
| 180    | 52.0 |

Resuelve para hallar los tres parámetros a partir de las tres ecuaciones que intersecan a los datos requeridos.

$\quad 22.0 = a + b \exp{(c \, 0)}$

$\quad 42.0 = a + b \exp{(c \, 90.3)}$

$\quad 52.0 = a + b \exp{(c \, 180.5)}$

Resuelve este set de ecuaciones para los parámetros desconocidos $a$, $b$ y $c$ con los datos recolectados al inicio de esta actividad. Utiliza como valores iniciales $a=100$, $b=-100$ y $c=-0.01$. Grafica el ajuste no lineal con los datos para asegurar que este pasa por los tres puntos especificados. Añade las etiquetas necesarias en el gráfico.

```python

```


---
# Section 2.1: Vector and Matrix Norms
---

## Motivation

When solving $Ax = b$ numerically, we get a solution $\hat{x}$ that satisfies

$$
\hat{A} \hat{x} = \hat{b}.
$$

We hope that $\hat{A} \approx A$ and $\hat{b} \approx b$.

It is for this reason that we need a way to measure the distance between both matrices and vectors.

---

## Vector norms

A (vector) **norm** is a function

$$
\|\cdot\| : \mathbb{R}^n \rightarrow \mathbb{R}
$$

that satisfies

1. $\|x\| > 0$ if $x \neq 0$ and $\|0\| = 0$
2. $\|\alpha x\| = |\alpha| \|x\|$
3. $\|x + y\| \leq \|x\| + \|y\| \qquad$ (triangle inequality)

The **distance** between vectors $x$ and $y$ can then be measured by

$$
\mathrm{dist}(x,y) = \| x - y \|.
$$

---

## The $p$-norm

$$
\|x\|_p = \left( \sum_{i=1}^n \left|x_i\right|^p \right)^{1/p}, \quad 1 \leq p \leq \infty.
$$

Some examples are:

$$
\begin{array}{llclll}
\|x\|_1 &=& \displaystyle{\sum_{i=1}^n |x_i|} &=& \texttt{norm(x, 1)} \qquad &\text{(Manhattan norm)} \\
\|x\|_2 &=& \displaystyle{\sqrt{\sum_{i=1}^n |x_i|^2}} &=& \texttt{norm(x)} \qquad &\text{(Euclidean norm)} \\
\|x\|_\infty &=& \displaystyle{\max_{i=1,\ldots,n} |x_i|} &=& \texttt{norm(x, Inf)} \qquad &\text{(max-norm)} \\
\end{array}
$$

The **unit $p$-norm ball** is the set of vectors having $p$-norm at most $1$:

$$
B_p = \big\{ x \in \mathbb{R}^n : \lVert x \rVert_p \leq 1 \big\}.
$$

---

## `norm` of a vector

```julia
using LinearAlgebra
```

```julia
?norm
```

```julia
x = Float64[1, -2, 1]
```

```julia
y = Float64[0.9, -1.9, 1.1]
```

```julia
# The 1-norm of x
norm(x, 1)
```

```julia
# The 0-norm counts the number of nonzero entries in x
norm([1, 0, 0, 0, 1], 0)
```

```julia
# The Manhattan distance between x and y
norm(x - y, 1)
```

```julia
x
```

```julia
# The Euclidean norm of x
norm(x, 2)
```

```julia
sqrt(1 + 4 + 1)
```

```julia
# The default norm is the Euclidean norm
norm(x)
```

```julia
# The Euclidean distance between x and y
norm(x - y)
```

```julia
x
```

```julia
# The max-norm of x
norm(x, Inf)
```

```julia
# The max-norm distance between x and y
norm(x - y, Inf)
```

---

## The $A$-norm

$$
\|x\|_A = \sqrt{x^T A x}
$$

where $A$ is an $n \times n$ positive definite matrix.

For example, when $A = I$, we have the usual Euclidean norm:

$$
\|x\|_I = \|x\|_2.
$$

---

## The resistance distance on a social network

A practical example of the $A$-norm is the **resistance distance** between individuals on a **social network**.

A social network can be represented as a **graph** where individuals are **nodes** which are connected by an **edge** if they are friends.

This graph can be represented using an **adjacency matrix** $A = [a_{ij}]$ where $a_{ij} = 1$ if $i$ and $j$ are friends, otherwise $a_{ij} = 0$; you cannot be friends with yourself, so $a_{ii} = 0$.

The **Laplacian matrix** $L = [l_{ij}]$ of the graph has $l_{ii} = \deg(i)$ (i.e., the number of friends of $i$), and $l_{ij} = -a_{ij}$ for $i \neq j$. That is,

$$
L = \mathrm{Diag}(Ae) - A,
$$

where $e$ is the vector of all ones, and $\mathrm{Diag}(Ae)$ is the diagonal matrix with the vector $Ae$ on its diagonal.

The **resistance distance** between $i$ and $j$ is then given by

$$
\mathrm{dist}(i,j) = \|e_i - e_j\|_B,  \qquad \text{where $B = (L + ee^T)^{-1}$}.
$$

Here $e_i$ and $e_j$ are the $i^\mathrm{th}$ and $j^\mathrm{th}$ columns of the identity matrix. (Note: it can be shown that $L + ee^T$ is positive definite, which implies that $(L + ee^T)^{-1}$ is positive definite.)

```julia
n = 6
A = Symmetric(rand(n, n))
A = round.(A)
A = A - diagm(diag(A))  # Make the diagonal zero
```

```julia
e = ones(n)
L = diagm(A*e) - A
```

```julia
L + e*e'
```

```julia
isposdef(L + e*e')
```

```julia
B = inv(L + e*e')
```

```julia
cholesky(Symmetric(B))
```

```julia
eigvals(Symmetric(B))
```

```julia
isposdef(Symmetric(B))
```

```julia
# Define the resistance norm
resnorm(x) = sqrt(x'*B*x)[1]

# Form the matrix of distances between all nodes
I = diagm(ones(n))
D = Float64[resnorm(I[:,i] - I[:,j]) for i = 1:n, j = 1:n]
```

```julia
D[3,4]
```

```julia
D[3,6]
```

```julia
D[3,4] < D[3,6]
```

```julia
using GraphPlot, LightGraphs
```

```julia
g = Graph(A)
gplot(g, nodelabel=1:n)
```

```julia
D[3,2]
```

```julia
D[3,6]
```

---

## The Euclidean norm is a norm

To prove that the **Euclidean norm** is indeed a norm, we need to show it satisfies the **triangle inequality**:

$$
\|x + y\|_2 \leq \|x\|_2 + \|y\|_2.
$$

We will prove this using the following fundamental result.

> ### Theorem: (Cauchy-Schwarz Inequality)
>
> $$
\left|x^Ty\right| \leq \|x\|_2 \|y\|_2, \qquad \forall x, y \in \mathbb{R}^n.
$$

If $\theta \in [0, \pi]$ is the angle between the nonzero vectors $x$ and $y$, then

$$
\cos \theta = \frac{x^T y}{\|x\|_2 \|y\|_2}.
$$

### Proof of the triangle inequality.

Let $x, y \in \mathbb{R}^n$. Then

$$
\begin{align}
\|x + y\|_2^2
&= (x+y)^T(x+y) \\
&= x^Tx + x^Ty + y^Tx + y^Ty \\
&= \|x\|_2^2 + 2x^Ty + \|y\|_2^2 \\
&\leq \|x\|_2^2 + 2\|x\|_2\|y\|_2 + \|y\|_2^2 \qquad \text{(Cauchy-Schwarz inequality)} \\
&= \big(\|x\|_2 + \|y\|_2\big)^2. \\
\end{align}
$$

Taking the square root of both sides, we obtain $\|x + y\|_2 \leq \|x\|_2 + \|y\|_2$. $\blacksquare$

---

Now let's see a proof of the Cauchy-Schwarz inequality.

### Proof of Cauchy-Schwarz.

Let $t \in \mathbb{R}$. Then

$$
\begin{align}
0 \leq \|x + ty\|_2^2
&= (x + ty)^T(x + ty) \\
&= x^T x + 2tx^Ty + t^2y^Ty \\
&= \|x\|_2^2 + \big(2x^Ty\big) t + \|y\|_2^2 t^2  \\
&= c + bt + at^2,
\end{align}
$$

where

$$
a = \|y\|_2^2, \qquad
b = 2x^Ty, \qquad
c = \|x\|_2^2.
$$

$\therefore$ $at^2 + bt + c \geq 0$, $\forall t \in \mathbb{R}$.

This implies that the equation

$$
at^2 + bt + c = 0
$$

either has no solution or exactly one solution. By the **quadratic formula**,

$$
t = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a},
$$

we must have $b^2 - 4ac \leq 0$.

Thus,

$$
\big(2x^Ty\big)^2 - 4 \|y\|_2^2 \|x\|_2^2 \leq 0,
$$

which simplifies to

$$
\left|x^Ty\right| \leq \|x\|_2 \|y\|_2. \qquad \blacksquare
$$

---

## Matrix norms

A **matrix norm** is a function

$$
\|\cdot\| : \mathbb{R}^{n \times n} \rightarrow \mathbb{R}
$$

that satisfies

1. $\|A\| > 0$ if $A \neq 0$ and $\|0\| = 0$
2. $\|\alpha A\| = |\alpha| \|A\|$
3. $\|A + B\| \leq \|A\| + \|B\|$
4. $\|AB\| \leq \|A\|\|B\| \qquad$ (submultiplicativity)

The **distance** between matrices $A$ and $B$ can then be measured by

$$
\mathrm{dist}(A, B) = \| A - B \|.
$$

---

### The Frobenius norm

$$
\|A\|_F = \sqrt{\sum_{i=1}^n \sum_{j=1}^n |a_{ij}|^2} = \texttt{norm(A)}
$$

```julia
A = [1 2; 3 4.0]
```

```julia
norm(A)
```

```julia
sqrt(30)
```

---

### Exercise

Compute $\lVert I \rVert_F$, where $I$ is the $n \times n$ identity matrix.

$$
\|I\|_F = \sqrt{n}.
$$

---

```julia
I = diagm(ones(4))
```

```julia
norm(I)
```

---

## Induced Matrix Norms

Let $\|\cdot\| : \mathbb{R}^n \rightarrow \mathbb{R}$ be a vector norm.

The **induced matrix norm** (a.k.a. the **operator norm**) is defined as

$$
\|A\| = \max_{x \neq 0} \frac{\|Ax\|}{\|x\|} = \max_{\|x\| = 1} \|Ax\|.
$$

The induced matrix norm is a matrix norm.

---

## Exercise

Let $\|\cdot\|$ be an induced matrix norm. Compute $\|I\|$.

Let $x$ be a nonzero vector. Then

$$
\frac{\|Ix\|}{\|x\|} = \frac{\|x\|}{\|x\|} = 1.
$$

Therefore, $\|I\| = 1$.

---

## Examples

$$
\begin{array}{llclll}
\|A\|_p &=& \displaystyle{\max_{x \neq 0} \frac{\|Ax\|_p}{\|x\|_p}} &=& \texttt{opnorm(A, p)} \qquad &\text{($p$-norm)} \\
\|A\|_1 &=& \displaystyle{\max_{1 \leq j \leq n} \sum_{i=1}^n |a_{ij}|} &=& \texttt{opnorm(A, 1)} \qquad &\text{(max-column-sum)} \\
\|A\|_2 &=& \displaystyle{\sqrt{\lambda_{\max}\left(A^TA\right)}} &=& \texttt{opnorm(A)} \qquad &\text{(spectral norm)} \\
\|A\|_\infty &=& \displaystyle{\max_{1 \leq i \leq n} \sum_{j=1}^n |a_{ij}|} &=& \texttt{opnorm(A, Inf)} \qquad &\text{(max-row-sum)} \\
\end{array}
$$

**Note:** $\lambda_{\max}\left(A^TA\right)$ is the **largest eigenvalue** of the symmetric matrix $A^TA$; $\sqrt{\lambda_{\max}\left(A^TA\right)}$ is the **largest singular value** of the matrix $A$.

However, the Frobenius norm is **not** an induced matrix norm.

```julia
A = [1 3; -2 0.0]

sqrt(maximum(eigvals(A'*A)))
```

```julia
maximum(svdvals(A))
```

```julia
eigvals(A'*A)
```

```julia
svdvals(A)
```

---

## `opnorm` of a matrix

```julia
A = [1 2 3; 4 5 6; 7 8 9.0]
```

```julia
# max-column-sum
opnorm(A, 1)
```

```julia
# max-row-sum
opnorm(A, Inf)
```

```julia
# spectral norm
opnorm(A)
```

```julia
# The spectral norm of A is the
# maximum singular value of A
svdvals(A)
```

```julia
# The singular values of A are the square root of the
# eigenvalues of A'*A
λ = max.(eigvals(A'*A),0)
sqrt.(λ)
```

---

## Induced Matrix Norm Inequality

> ### Theorem: (Induced Matrix Norm Inequality)
>
> Let $\|\cdot\| : \mathbb{R}^n \rightarrow \mathbb{R}$ be a vector norm. Then the corresponding induced matrix norm satisfies
>
> $$\|Ax\| \leq \|A\|\|x\|,\qquad \text{for all $A \in \mathbb{R}^{n \times n}$ and $x \in \mathbb{R}^n$.}$$

### Proof:

Let $x \in \mathbb{R}^n$. If $x = 0$, then $\|Ax\| \le \|A\|\|x\|$ clearly holds since both sides of the inequality would equal zero. Now suppose that $x \ne 0$. Then,

$$
\frac{\|Ax\|}{\|x\|} \le \max_{y \ne 0} \frac{\|Ay\|}{\|y\|} = \|A\|.
$$

Multiplying both sides by $\|x\|$, we have $\|Ax\| \le \|A\|\|x\|$. $\blacksquare$

---

Section 2.1 - Vector and Matrix Norms.ipynb


Euler Problem 7
===============

By listing the first six prime numbers: 2, 3, 5, 7, 11, and 13, we can see that the 6th prime is 13.

What is the 10,001st prime number?

```python
MAX = 200000
index = 10001
sieve = [True] * MAX
prime_count = 0
for p in range(2, MAX):
    if sieve[p]:
        prime_count += 1
        if prime_count == index:
            print(p)
            break
        for n in range(p*p, MAX, p):
            sieve[n] = False
```

    104743

The Sympy module makes this problem even easier.

```python
from sympy import prime
print(prime(index))
```

    104743

```python

```


# Implement a physically correct distribution

In order to understand lubrication better, we simulate thin layers of lubricant on a metallic surface, solvated in water.
Different structures of lubricant films are created by varying parameters like their concentration and the charge of the surface.
The lubricant is somewhat solvable in water, thus parts of the film will diffuse into the bulk water.
Lubricant molecules are charged, and their distribution is roughly exponential.

So, let's first generate a simple exponential distribution:

```python
import numpy as np
import scipy.constants as sc
import matplotlib.pyplot as plt

from generate_structure import generate_structure
from generate_structure import plot_dist
from generate_structure import get_histogram
from generate_structure import exponential

np.random.seed(74)
```

```python
box = np.array([50, 50, 100])
struc = generate_structure(distribution=exponential, box=box, atom_count=1000)
histx, histy, histz = get_histogram(struc, box=box)
plot_dist(histz, 'z', reference_distribution=exponential)
```

This seems to work reasonably well, but does not describe a physical system - units and the slope of the exponential are rather abitrary.

Now, to incorporate a precise physical description of the distribution, we describe the concentration of our ion species, $c_{Na^+}$, by the solution to the Poisson-Boltzmann equation:

$
\begin{align}
\rho_{Na^+}(z) &= \rho_{Na^+}(\infty) e^{-e \Psi(z)/k_B T}\\
\Psi(z) &= \frac{2k_B T}{e} \log\Big(\frac{1 + \gamma e^{-\kappa z}}{1- \gamma e^{-\kappa z}}\Big)
        \approx \frac{4k_B T}{e} \gamma e^{-\kappa z} \\
\gamma &= \tanh(\frac{e\Psi(0)}{4k_B T})\\
\kappa &= 1/\lambda_D\\
\lambda_D &= \Big(\frac{\epsilon \epsilon_0 k_B T}{\sum_{i} \rho_i(\infty) e^2 z_i^2} \Big)^\frac{1}{2} [m^{-1}]
\end{align}
$

With:
* $z$: Distance from the double layer
* $\Psi(0)$: Potential at the surface
* $\Psi(z)$: Potential in the solution
* $k_B$: Boltzmann Constant
* $T$: Temperature [Kelvin]
* $e$: Elemental Charge (or Euler's constant when exponentiated)
* $\gamma$: Term from Gouy-Chapmann theory
    * $\gamma \rightarrow 1$ for high potentials
    * $\Psi(z) \approx \Psi_0 e^{-\kappa z}$ for low potentials $\Psi(0) \approx 0$
* $\lambda_D$: Debye Length (34.0 nm for NaCl, 10^-4 M, 25°C)
* $\rho_{Na^+}$: Concentration of Natrium ions
* $\rho_{Na^+}(\infty)$: Bulk Natrium concentration (at infinity, where the solution is homogeneous)
* $\epsilon$: Permittivity of the solution
* $\epsilon_0$: Electric constant aka Vacuum permittivity
* $z_i$: Charge of species i

Let's translate these equations into python code.

## Debye length
We start with the expression for the Debye length, which describes the thickness of the ionic double layer, or the distance at which the surface charge is mostly screened off.

```python
def debye(rho_bulk, charge, permittivity=79, temperature=298.15):
    """Calculate the Debye length.
    The Dybe length indicates at which distance a charge will be screened off.

    Arguments:
    rho_bulk: dictionary of the bulk number densites for each ionic species [1/m^3]
    charge: dictionary of the charge of each ionic species [1]
    permittivity: capacitance of the ionic solution [1]
    temperature: Temperature of the solution [K]

    Returns:
    float: the Debye length [m], should be around 10^-19

    Example: the Debye length of 10^-4 M salt water is 30.4 nm.
    >>> density = sc.Avogadro * 1000 * 10**-4
    >>> rho = {'Na': density, 'Cl': density}
    >>> charge = {'Na': 1, 'Cl': -1}
    >>> deb = debye(rho_bulk=rho, charge=charge) * 10**9
    >>> deb - 30.4 < 0.5
    True
    """
    # The numerator of the expression in the square root
    # e_r * e_0 * k_B * T
    numerator = permittivity * sc.epsilon_0 * sc.Boltzmann * temperature

    # The divisor of the expression in the square root
    # \sum_i rho_i e^2 z_i^2
    divisor = 0
    for key in rho_bulk.keys():
        divisor += rho_bulk[key] * sc.elementary_charge ** 2 * charge[key] ** 2

    # The complete square root
    return np.sqrt(numerator / divisor)

# Factor of 1000 is due to 1/m^3, would be 1 for 1/l
default_density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': default_density, 'Cl': default_density}
charge = {'Na': 1, 'Cl': -1}
deb = debye(rho_bulk=rho, charge=charge) * 10**9
print('Debye Length of 10^-4 M saltwater: {} nm (Target: 30.4 nm)'.format(round(deb, 2)))

density = np.logspace(-6, 0, 50) * sc.Avogadro * 1000

debyes = [debye(rho_bulk={'Na': d, 'Cl': d}, charge=charge) * 10**9 for d in density]
plt.xlabel('Density [1/m^3]')
plt.ylabel('Debye length at 25° [nm]')
plt.semilogx(density, debyes, marker='.')
plt.show()
```

The debye length depends on the concentration of ions in solution, at low concentrations it becomes large. We can reproduce literature debye lengths with our function, so everything looks good.

## Gamma Function

Next we calculate the gamma function $\gamma = \tanh(\frac{e\Psi(0)}{4k_B T})$

```python
def gamma(surface_potential, temperature):
    """Calculate term from Gouy-Chapmann theory."""
    product = sc.elementary_charge * surface_potential / (4 * sc.Stefan_Boltzmann * temperature)
    return np.tanh(product)

x = np.linspace(12, 16, 40)
gammas = [gamma(10 ** i, 300) for i in x]
plt.xlabel('Potential')
plt.ylabel('Gamma at 300K')
plt.plot(x, gammas, marker='o')
plt.show()
```

Which looks as expected, but we have no values to compare it against.

## Potential

We plug these two functions, into the expression for the potential

$\Psi(z) = \frac{2k_B T}{e} \log\Big(\frac{1 + \gamma e^{-\kappa z}}{1- \gamma e^{-\kappa z}}\Big)
        \approx \frac{4k_B T}{e} \gamma e^{-\kappa z}$

```python
def potential(location, rho_bulk, charge, surface_potential, temperature=300, permittivity=80):
    """The potential near a charged surface in an ionic solution.
    Arguments:
    location: z-distance from the surface [m]
    temperature: Temperature of the soultion [Kelvin]
    gamma: term from Gouy-Chapmann theory
    kappa: the inverse of the debye length
    charge: dictionary of the charge of each ionic species
    permittivity: capacitance of the ionic solution []
    temperature: Temperature of the solution [Kelvin]

    Returns:
    psi: Electrostatic potential [V]
    """
    prefactor = 2 * sc.Stefan_Boltzmann * temperature / sc.elementary_charge

    gamma_value = gamma(surface_potential=surface_potential, temperature=temperature)
    debye_value = debye(rho_bulk=rho_bulk, charge=charge, permittivity=permittivity, temperature=temperature)
    kappa = 1/debye_value

    numerator = 1 + gamma_value * np.exp(-kappa * location)
    divisor = 1 - gamma_value * np.exp(-kappa * location)

    psi = prefactor * np.log(numerator / divisor)

    return psi
```

```python
z = np.linspace(0, 2*10**-7, 10000)
density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': density, 'Cl':density}
charge = {'Na': 1, 'Cl': -1}
pot_0 = 100
psi = [potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]
plt.xlabel('z [nm]')
plt.ylabel('Potential [V]')
plt.plot(z*10**9, psi, marker='')
plt.show()
```

As expected, this gives us an exponentially decaying potential.

Unfortunately, the differences in our potential calculation already strain python's numerical precision:

```python
z = np.linspace(0, 3*10**-10, 10000)
psi = [potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]
plt.xlabel('z [nm]')
plt.ylabel('Potential [V]')
plt.plot(z*10**9, psi, marker='')
plt.show()
```

These steps in the potential will lead to major errors in the density, so we want to remove them.

To get a smooth potential, we use the decimal package with increased float precision.

```python
from decimal import *
getcontext().prec = 30

def precise_potential(location, rho_bulk, charge, surface_potential, temperature=300, permittivity=80):
    """The potential near a charged surface in an ionic solution.

    The decimal package is used for increased precision.

    Arguments:
    location: z-distance from the surface [m]
    temperature: Temperature of the soultion [Kelvin]
    gamma: term from Gouy-Chapmann theory
    kappa: the inverse of the debye length
    charge: dictionary of the charge of each ionic species
    permittivity: capacitance of the ionic solution []
    temperature: Temperature of the solution [Kelvin]

    Returns:
    psi: Electrostatic potential [V]
    """
    prefactor = Decimal(2 * sc.Stefan_Boltzmann * temperature / sc.elementary_charge)

    debye_value =  debye(rho_bulk=rho_bulk, charge=charge, permittivity=permittivity, temperature=temperature)
    kappa = 1/debye_value

    gamma_value =  Decimal(gamma(surface_potential=surface_potential, temperature=temperature))

    exponential = Decimal(np.exp(-kappa * location))
    numerator = Decimal(1) + gamma_value * exponential
    divisor =   Decimal(1) - gamma_value * exponential

    psi = prefactor * (numerator / divisor).ln()
    psi = float(psi)

    return psi
```

```python
z = np.linspace(0, 2*10**-7, 10000)
density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': density, 'Cl':density}
charge = {'Na': 1, 'Cl': -1}
pot_0 = 100
psi = [precise_potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]
plt.xlabel('z [nm]')
plt.ylabel('Potential [V]')
plt.plot(z*10**9, psi, marker='')
plt.show()

z = np.linspace(0, 3*10**-10, 10000)
psi = [precise_potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]
plt.xlabel('z [nm]')
plt.ylabel('Potential [V]')
plt.plot(z*10**9, psi, marker='')
plt.show()
```

We can see that the steps in the potential are gone, everything is smooth now.

## Charge density

Now we obtain the charge density $\rho$ from the potential $\Psi$ via

$\rho_{Na^+}(z) = \rho_{Na^+}(\infty) e^{-e \Psi(z)/k_B T}$

```python
def charge_density(location, rho_bulk, charge, surface_potential, temperature=300, permittivity=80, species='Na'):
    """The potential near a charged surface in an ionic solution.
    Arguments:
    location: z-distance from the surface [m]
    temperature: Temperature of the soultion [Kelvin]
    gamma: term from Gouy-Chapmann theory
    kappa: the inverse of the debye length
    charge: dictionary of the charge of each ionic species
    permittivity: capacitance of the ionic solution []
    temperature: Temperature of the solution [Kelvin]

    Returns:
    rho: number density of salt ions
    """
    potential_value = precise_potential(location, rho_bulk, charge, surface_potential, temperature, permittivity)

    # Save the species' densities
    rho = rho_bulk[species] * np.exp(-1 * charge[species] * sc.elementary_charge * potential_value / (sc.Boltzmann * temperature))

    return rho
```

```python
z = np.linspace(0, 100*10**-9, 2000)
density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': density, 'Cl':density}
charge = {'Na': 1, 'Cl': -1}
pot_0 = 0.05  # Breaks if > 1

psi = [precise_potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]

rho_na = np.array([charge_density(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0, species='Na') for loc in z])
rho_cl = np.array([charge_density(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0, species='Cl') for loc in z])

deb = debye(rho_bulk=rho, charge=charge) * 10**9

fig, ax1 = plt.subplots(figsize=[15,5])
ax1.set_xlabel('z [nm]')
ax1.plot(z*10**9, psi, marker='', color='red', label='Potential')
ax1.set_ylabel('Potential')
ax1.axvline(x=deb, label='Debye Length', color='orange')

ax2 = ax1.twinx()
ax2.plot(z*10**9, [density]*len(z), label='Bulk concentration', color='grey')
ax2.plot(z*10**9, rho_na, marker='', color='green', label='Na+ ions')
ax2.plot(z*10**9, rho_cl, marker='', color='blue', label='Cl- ions')
ax2.set_ylabel('Density')

#fig.legend(loc='center')
ax2.legend(loc='best', fontsize=15)
ax1.legend(loc='upper center', fontsize=15)
fig.tight_layout()
plt.show()
```

The charge density behaves as expected, it interpolates between low (high) concentration and the bulk concentration within the first few debye lengths.

## Sampling
Now let's see if we can just plug our new distribution in our existing framework.

First, we need to convert the physical distribution to the format we were using so far:

```python
def wrap_distribution(x, species):
    """Wrapper for na+ ions."""
    density = sc.Avogadro * 1000 * 10**-4
    rho = {'Na': density, 'Cl':density}
    charge = {'Na': 1, 'Cl': -1}
    pot_0 = 0.05  # Breaks if > 1

    def call_distri(loc):
        distri = charge_density(location=loc, rho_bulk=rho,
                                charge=charge, surface_potential=pot_0, species=species)
        return float(distri)

    if not np.isscalar(x):
        y = []
        for i in range(0, len(x)):
            val = call_distri(x[i])

            # Normalize to be 1 at x=0
            val /= call_distri(0)
            # Scale distribution to have values in [0, 0.1] for ease of sampling
            val /= 10
            y += [val]
        return np.array(y)

    # If we have only a point estimate
    val = call_distri(x)
    # Normalize to be 1 at x=0
    val /= call_distri(0)
    # Scale distribution to have values in [0, 0.1] for ease of sampling
    val /= 10
    return val

def cl_distribution(x):
    return wrap_distribution(x, species='Cl')

def na_distribution(x):
    return wrap_distribution(x, species='Na')
```

```python
x = 50 * 10**-9
z = 100 * 10**-9
box = np.array([x, x, z])
struc = generate_structure(distribution=na_distribution, box=box, atom_count=10000)
histx, histy, histz = get_histogram(struc, box=box, n_bins=101)
plot_dist(histz, 'z', reference_distribution=na_distribution)
```

```python
x = 50 * 10**-9
z = 100 * 10**-9
box = np.array([x, x, z])
struc = generate_structure(distribution=cl_distribution, box=box, atom_count=10000)
histx, histy, histz = get_histogram(struc, box=box, n_bins=101)
plot_dist(histz, 'z', reference_distribution=cl_distribution)
```

## Write to file
To visualize our structure, we export it to the .xyz file format, which is basically

```
ATOM_NUMBER
Optional comment
atom_type x y z
atom_type x y z
```

Avogadro expects x, y, z to be in units of $10^{-9}~m$, so we convert our salt "solution" to this unit.

```python
from generate_structure import concat_names_structs
from generate_structure import export_named_struc

cl_struc = generate_structure(distribution=cl_distribution, box=box, atom_count=100)
na_struc = generate_structure(distribution=na_distribution, box=box, atom_count=100)

concat_list = concat_names_structs(struc_list=[cl_struc, na_struc], name_list=['Cl', 'Na'])
rescaled_list = []
for line in concat_list:
    name, x, y, z = line
    x = float(x) * 10**9
    y = float(y) * 10**9
    z = float(z) * 10**9
    rescaled_list += [[name, x, y, z]]

rescaled_list = np.array(rescaled_list)
export_named_struc(rescaled_list)
```

```python
from IPython.display import Image
Image(filename='distributed_atom_structure.png')
```


# 15 Partial Differential Equations — 1

## Solving Laplace's or Poisson's equation

**Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

Solutions depend on the **boundary conditions**:

* the *value of the potential* on the *boundary* or
* the *electric field* (i.e. the derivative of the potential, $\mathbf{E} = -\nabla\Phi$ *normal to the surface* ($\mathbf{n}\cdot\mathbf{E}$), which directly follows from the charge distribution).

### Example: 2D Laplace equation
$$
\frac{\partial^2 \Phi(x,y)}{\partial x^2} + \frac{\partial^2 \Phi(x,y)}{\partial y^2} = 0
$$
("elliptic PDE")

Boundary conditions:
* square area surrounded by wires
* three wires at ground (0 V), one wire at 100 V

## Finite difference algorithm for Poisson's equation
Discretize space on a lattice (2D) and solve for $\Phi$ on each lattice site.

Taylor-expansion of the four neighbors of $\Phi(x, y)$:

\begin{align}
\Phi(x \pm \Delta x, y) &= \Phi(x, y) \pm \Phi_x \Delta x + \frac{1}{2} \Phi_{xx} \Delta x^2 + \dots\\
\Phi(x, y \pm \Delta y) &= \Phi(x, y) \pm \Phi_y \Delta x + \frac{1}{2} \Phi_{yy} \Delta x^2 + \dots\\
\end{align}

Add equations in pairs: odd terms cancel, and **central difference approximation** for 2nd order partial derivatives (to $\mathcal{O}(\Delta^4)$):

\begin{align}
\Phi_{xx}(x,y) = \frac{\partial^2 \Phi}{\partial x^2} & \approx
  \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) - 2\Phi(x,y)}{\Delta x^2} \\
\Phi_{yy}(x,y) = \frac{\partial^2 \Phi}{\partial y^2} &\approx
  \frac{\Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) - 2\Phi(x,y)}{\Delta y^2}
\end{align}

Take $x$ and $y$ grids of equal spacing $\Delta$: Discretized Poisson equation

$$
\begin{split}
\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) &+ \\
   +\, \Phi(x,y-\Delta y) - 4\Phi(x,y) &= -4\pi\rho(x,y)\,\Delta^2
   \end{split}
$$

Defines a system of $N_x \times N_y$ simultaneous algebraic equations for $\Phi_{ij}$ to be solved.

Can be solved directly via matrix approaches (and then is the best solution) but can be unwieldy for large grids.

Alternatively: **iterative solution**:

$$
\begin{split}
4\Phi(x,y) &= \Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\\
 &+ \Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) + 4\pi\rho(x,y)\,\Delta^2
\end{split}
$$

Or written for lattice sites $(i, j)$ where

$$
x = x_0 + i\Delta\quad\text{and}\quad y = y_0 + j\Delta, \quad 0 \leq i,j < N_\text{max}
$$

$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

* Converged solution at $(i, j)$ will be the average potential from the four neighbor sites + charge density contribution.
* *Not a direct solution*: iterate and hope for convergence.

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

### Solution via relaxation (Gauss-Seidel)

Solve the box-wire problem on a lattice: The wire at $x=0$ (the $y$-axis) is at 100 V, the other three sides of the box are grounded (0 V).

Note: $\rho=0$ inside the box.

Note for Jupyter notebook use:
* For interactive 3D plots, select
  ```
  %matplotlib widget
  ```
  or if the above fails, try
  ```
  %matplotlib notebook
  ```
* For standard inline figures (e.g. for exporting the notebook to LaTeX/PDF or html) use
  ```
  %matplotlib inline
  ```

Enable a matplotlib-Jupyter integration that works for you (try `conda install ipympl` or `pip install ipympl` first to get `%matplotlib widget` working).

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
# for interactive work

%matplotlib widget
#%matplotlib inline
#%matplotlib notebook
```

```python
# for plotting/saving
%matplotlib inline
```

#### Wire on a box: Solution of Laplace's equation with the Gauss-Seidel algorithm

```python
Nmax = 100
Max_iter = 70
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

Nx, Ny = Phi.shape

for n_iter in range(Max_iter):
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
```

#### Visualization of the potential

```python
# plot Phi(x,y)
x = np.arange(Phi.shape[0])
y = np.arange(Phi.shape[1])
X, Y = np.meshgrid(x, y)

Z = Phi[X, Y]
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')

ax.view_init(elev=40, azim=-65)
```

Nicer plot (use this code for other projects):

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
surf = ax.plot_surface(X, Y, Z, cmap=plt.cm.coolwarm, alpha=0.6)
cset = ax.contourf(X, Y, Z, zdir='z', offset=-50, cmap=plt.cm.coolwarm)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')
ax.set_zlim(-50, 100)
ax.view_init(elev=40, azim=-65)

cb = fig.colorbar(surf, shrink=0.5, aspect=5)
cb.set_label(r"potential $\Phi$ (V)")
```

(Note that the calculation above is is *not converged* ... see next lecture.)