# Sandbox/demo to go along with the MD assignment and lectures

## PharmSci 175/275
Author: David Mobley

## MD for a simple Lennard-Jones system

### Here, let's play with a very simple model system
In the MD assignment, we will be looking at a very simple polymer model. But here, let's backtrack to simple Lennard-Jones spheres (as used in the energy minimization assignment) and look at what happens if we run a simple dynamics calculation on a couple of particles subject to this potential, which will allow us to get a feel for how the integrator works.

Remember, for a Lennard-Jones system, the dimensionless form of our potential is
\begin{equation}
U^* = \sum \limits_{i} 4\left( r_{ij}^{-12} - r_{ij}^{-6}\right)
\end{equation}

which we can easily graph to get a sense of what will happen.

### We graph the potential for a pair of particles:

```python
#Get pylab ready for plotting in this notebook - "magic" command specific for iPython notebooks
%pylab inline
#Import numpy
import numpy as np

#Generate array of distances to graph at
r = np.arange( 0,3, 0.01)

#Calculate U
U = 4.*(r**(-12.) - r**(-6.))

#Graph
plot( r, U, 'b-')
#Label x and y axes
xlabel('r')
ylabel('U')
#Adjust y limits not to be auto-scaled; since this goes to infinity at zero, the graph will not be useful unless we truncate
ylim(-2,2)

```

## Now, let's run some dynamics on a pair of particles subject to this potential

Here I've written a modified `mdlib.f90` (updated from that from the energy minimization and MD assignments) called `md_sandbox.f90`. Compile it as usual (with `f2py`) into `md_sandbox` (e.g. `f2py -c -m md_sandbox md_sandbox.f90` or similar) so that you can import it below.

The difference between this and `mdlib` that you will use in your MD assignment is that I've removed the bonds between atoms, so that we have simple Lennard-Jones particles rather than LJ polymers.

### First, we set up our system:

```python
#Let's define the variables we'll need
dt = 0.001
Cut = 2.5
L = 10 #Let's just put these in a fairly big box so we have room
M = 1 #Here this doesn't actually do anything, but I didn't remove it from the functions

#Import our library
from md_sandbox import *

#Choose N for number of particles
N = 2

#Allocate position array - initially just zeros
Pos = np.zeros((N,3), float)

#In this case, place two LJ particles at specified initial positions - chosen so they are relatively near each other
#If you had more than two particles you'd need to adjust this
Pos[0,:] = np.array([0,0,0])
Pos[1,:] = np.array([1.2,0,0])
#Might be worth experimenting to see what happens if we randomly place the particles instead

#Assign initial velocities - in this case I'll start off with them stationary and see what happens
Vel = np.array([[0,0,0],[0,0,0]], float)
#You could tweak the initial velocities to see what happens under different conditions
```

## Before we do any dynamics, notice that this is really a 1D system, so let's write a function to compute r, which we will store later

```python
def get_r(Pos):
    """Calculate r, the distance between particles, for a position array containing just two particles. Return it."""

    #Get displacement
    dist = Pos[1,:] - Pos[0,:]
    #Calculate distance and return
    return np.sqrt( np.dot( dist, dist))
```

## OK, now let's start doing some dynamics

We're going to want to store the distance between the two particles as a function of time, so we can graph it. We also might want to look at the population of each distance as a function of time and see how that compares with the energy landscape. Though, right now that's a little premature. Let's just start off by taking a few timesteps and see how the distance changes.

```python
#Define storage for positions at each time so we can track them
max_steps = 5000 #Maximum number of steps we will take - so we know how many positions we might store
Pos_t = np.zeros(( N,3,max_steps), float)

#Store initial positions
Pos_t[:,:,0] = Pos

#Make up initial forces
Forces = np.zeros((N,3), float)

#Kick things off by calculating energy and forces
energy, Forces = calcenergyforces( Pos, M, L, Cut, Forces )

#Take a timestep
Pos, Vel, Accel, KEnergy, PEnergy = vvintegrate( Pos, Vel, Forces, M, L, Cut, dt )

#Store new positions
Pos_t[:,:,1] = Pos

#Print original and current distance
for i in range(0,2):
    print(get_r(Pos_t[:,:,i]))
```

## What should the long-time behavior of this system be?

Before going on to the step below, think for a minute about what motion these particles should exhibit on long timescales and what it would look like if you've graphed it.

Once you've done so, write a `for` loop to run over max_steps and at each step, update the energy and forces, take a timestep, and store the new positions.

```python
for i in range(max_steps):
   #Your code goes here
    energy, Forces = calcenergyforces( Pos, M, L, Cut, Forces )
    Pos, Vel, Accel, KEnergy, PEnergy = vvintegrate( Pos, Vel, Forces, M, L, Cut, dt )
    Pos_t[:,:,i] = Pos

```

## Once you've done that, use this code to graph r versus time for your particles

```python
#Find x axis (time values)
t = dt*np.arange(0,max_steps)
#Find y axis (r values)
r_vs_t = []
for i in range(max_steps):
    r=get_r(Pos_t[:,:,i])
    r_vs_t.append(r)

r_vs_t = np.array(r_vs_t)

#Plot
figure()
plot(t, r_vs_t)

```

## Additional exercises

Some things you might want to try on the above 1D system:
- Change the amount of time the data is graphed for; see if you can make it oscillate repeatedly
- Consider how you could adjust the total energy of the system
- What if you change the total energy (increasing it or reducing it). Can you make it look more like a harmonic oscillator? Less like a harmonic oscillator? Why or why not?
- Can you find solutions which are not oscillatory? How?
- For the ambitious: Consider extending this example to three or more particles and graph just some of the distances involved. What kind of solutions can you find?

# Now let's do MD on a molecule!

So that was some very simple MD on a 1D Lennard-Jones system. Now let's head towards the opposite extreme and do our first molecular system.

Here, we'll draw on OpenMM, a molecular mechanics toolkit we'll see again later in the class, for running some simple energy minimizations and molecular dynamics. And we'll use the new `openforcefield` tools for assigning a force field for a molecule, because they make this very simple (and you have them already installed).

## First, we generate a molecule

```python
# What SMILES string for the guest? Should be isomeric SMILES
mol_smiles = 'OC(CC1CCCC1)=O'

# Import stuff
from openeye.oechem import *
import oenotebook as oenb
from openeye.oeomega import * # conformer generation
from openeye.oequacpac import * #for partial charge assignment

# Create empty OEMol
mol = OEMol()
# Convert SMILES
OESmilesToMol(mol, mol_smiles)
# Draw
oenb.draw_mol(mol)
```

## Now we generate a 3D structure of our compound and assign partial charges

```python
#initialize omega for conformer generation
omega = OEOmega()
omega.SetMaxConfs(100) #Generate up to 100 conformers since we'll use for docking
omega.SetIncludeInput(False)
omega.SetStrictStereo(True) #Refuse to generate conformers if stereochemistry not provided

#Initialize charge generation
chargeEngine = OEAM1BCCCharges()

# Set to use a simple neutral pH model
OESetNeutralpHModel(mol)

# Generate conformers with Omega; keep only best conformer
status = omega(mol)
if not status:
    print("Error generating conformers for %s." % (guest_smiles))

# Assign AM1-BCC charges
OEAssignCharges(mol, chargeEngine)

# Write out PDB of molecule
ofile = oemolostream('mymolecule.pdb')
OEWriteMolecule(ofile, mol)
ofile.close()
```

## Now we apply a force field to assign parameters to the system

This uses the `openforcefield` toolkit and its `smirnoff99Frosst` forcefield (a new GAFF-like forcefield for small molecules).

```python
from openforcefield.typing.engines.smirnoff import *

ff = ForceField('forcefield/smirnoff99Frosst.offxml')
topology = generateTopologyFromOEMol(mol)
system = ff.createSystem(topology, [mol])

from openforcefield.utils import get_data_filename, extractPositionsFromOEMol, generateTopologyFromOEMol
positions = extractPositionsFromOEMol(mol)
```

## Next we'll start off by doing an energy minimization

We first prepare the system, but then the actual energy minimization is very simple. I believe OpenMM is using an L-BFGS minimizer.

```python
# Even though we're just going to minimize, we still have to set up an integrator, since a Simulation needs one
integrator = openmm.VerletIntegrator(2.0*unit.femtoseconds)
# Prep the Simulation using the parameterized system, the integrator, and the topology
simulation = app.Simulation(topology, system, integrator)
# Copy in the positions
simulation.context.setPositions( positions)

# Get initial state and energy; print
state = simulation.context.getState(getEnergy = True)
energy = state.getPotentialEnergy() / unit.kilocalories_per_mole
print("Energy before minimization (kcal/mol): %.2g" % energy)

# Minimize, get final state and energy and print
simulation.minimizeEnergy()
state = simulation.context.getState(getEnergy=True, getPositions=True)
energy = state.getPotentialEnergy() / unit.kilocalories_per_mole
print("Energy after minimization (kcal/mol): %.2g" % energy)
newpositions = state.getPositions()
```

    Energy before minimization (kcal/mol): 95
    Energy after minimization (kcal/mol): 61

## Now we do some bookkeeping to run MD

Most of this is just setup, file bookkeeping, temperature selection, etc. The key part which actually runs it is the second-to-last block of code which says `simulation.step(1000)` which runs 1000 steps of dynamics. The timestep for this is set near the top, and is 2 femtoseconds per timestep, so the total time is 2000 fs, or 2 ps.

Simulation snapshots are stored to a NetCDF file for later visualization, every 100 frames (so we'll end up with 10 of them). You can adjust these settings if you like.

```python
# Set up NetCDF reporter for storing trajectory; prep for Langevin dynamics
from mdtraj.reporters import NetCDFReporter
integrator = openmm.LangevinIntegrator(300*unit.kelvin, 1./unit.picosecond, 2.*unit.femtoseconds)

# Prep Simulation
simulation = app.Simulation(topology, system, integrator)
# Copy in minimized positions
simulation.context.setPositions(newpositions)

# Initialize velocities to correct temperature
simulation.context.setVelocitiesToTemperature(300*unit.kelvin)
# Set up to write trajectory file to NetCDF file in data directory every 100 frames
netcdf_reporter = NetCDFReporter(os.path.join('.', 'trajectory.nc'), 100) #Store every 100 frames
# Initialize reporters, including a CSV file to store certain stats every 100 frames
simulation.reporters.append(netcdf_reporter)
simulation.reporters.append(app.StateDataReporter(os.path.join('.', 'data.csv'), 100, step=True, potentialEnergy=True, temperature=True, density=True))

# Run the simulation and print start info; store timing
print("Starting simulation")
start = time.clock()
simulation.step(1000) #1000 steps of dynamics
end = time.clock()

# Print elapsed time info, finalize trajectory file
print("Elapsed time %.2f seconds" % (end-start))
netcdf_reporter.close()
print("Done!")
```

    Starting simulation
    Elapsed time 0.15 seconds
    Done!

## Now we can visualize the results with `nglview`

If you don't have `nglview`, you can also load the PDB file, `mymolecule.pdb`, with a standard viewer like VMD or Chimera, and then load the NetCDF trajectory file (`trajectory.nc`) in afterwards.

```python
# Load stored trajectory using MDTraj; the trajectory doesn't contain chemistry info so we also load a PDB
import mdtraj
import nglview
traj= mdtraj.load(os.path.join('.', 'trajectory.nc'), top=os.path.join('.', 'mymolecule.pdb'))

# View the trajectory
view = nglview.show_mdtraj(traj)
view
```

Failed to display Jupyter Widget of type NGLWidget.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

## Exercises

- Try your own molecule. What about one which has the potential for internal hydrogen bonding?
- Run the dynamics longer
- What is different if you lower the temperature? Raise the temperature?


# Python-in-the-lab: packages and tools

```python
import os
import numpy as np
import matplotlib.pylab as plt
# %matplotlib inline
```

## NumPy

[NumPy](www.numpy.org) is the fundamental package for scientific computing with Python. It contains among other things:

* a powerful N-dimensional array object
* sophisticated (broadcasting) functions
* tools for integrating C/C++ and Fortran code
* useful linear algebra, Fourier transform, and random number capabilities

Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.

* [A tutorial](https://docs.scipy.org/doc/numpy-dev/user/quickstart.html)
* [The Numpy reference](https://docs.scipy.org/doc/numpy/reference/)
* [Numpy I/O](https://docs.scipy.org/doc/numpy/reference/routines.io.html)
* [Numpy Memmap: write on disk](https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html)
* [Numpy masked arrays](https://docs.scipy.org/doc/numpy/reference/routines.ma.html)

```python
# Masked arrays
import numpy.ma as ma

# The file below contains a noise signal, made of a sequence of spikes (called Barkhausen jumps)
# The first line is the total time of the signal
# Note: the signal is negative, for simplicity use -signal
filename = "/home/gf/src/Python/Python-in-the-lab/Bk/F64ac_0.03_time_200.dat"
data = -np.loadtxt(filename)
with open(filename) as f:
    header = f.readline()
max_time = float(header[1:])
time = np.linspace(0, max_time, len(data))
fig = plt.figure(figsize=(18,8))
plt.plot(time, data, 'k')
data_mean = np.mean(data)
plt.plot(time, data_mean * np.ones_like(time), '--r', lw=2)

# ahah, what if we want to consider just the values above the average, and calculate the new mean?
# mask = data < data_mean
# data_masked = ma.masked_array(data, mask)
data_masked = ma.masked_less_equal(data, data_mean)
new_mean = data_masked.mean()
plt.plot(time, new_mean * np.ones_like(time), '-r', lw=2);
```

```python
data_masked[160:230]
```

    masked_array(data=[--, --, --, --, --, --, --, --, --, --, --, --, --, --,
                       --, --, --, --, --, --, --, --, --, --, --, --, --,
                       1.770092773437532e-07, --, --, --, --, --, --, --, --,
                       --, --, --, --, --, --, --, --, --, --,
                       2.770092773437532e-07, 3.170092773437532e-07,
                       1.370092773437531e-07, 1.470092773437531e-07,
                       1.870092773437532e-07, --, --, --, --, --, --, --, --,
                       --, --, 1.470092773437531e-07, 1.670092773437531e-07,
                       --, --, --, --, --, --, --],
                 mask=[ True,  True,  True,  True,  True,  True,  True,  True,
                        True,  True,  True,  True,  True,  True,  True,  True,
                        True,  True,  True,  True,  True,  True,  True,  True,
                        True,  True,  True, False,  True,  True,  True,  True,
                        True,  True,  True,  True,  True,  True,  True,  True,
                        True,  True,  True,  True,  True,  True, False, False,
                       False, False, False,  True,  True,  True,  True,  True,
                        True,  True,  True,  True,  True, False, False,  True,
                        True,  True,  True,  True,  True,  True],
           fill_value=1e+20)

```python
# It would be nice to plot the data above the original mean
# i.e. cut the data below it
fig = plt.figure(figsize=(18,8))
plt.plot(time, data, 'k')
plt.plot(time, data_masked.filled(data_mean), '-r', lw=2);
```

# Scipy

[Scipy](http://scipy.org/) SciPy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering. In particular, these are some of the core packages

* [The tutorial](https://docs.scipy.org/doc/scipy/reference/) Let's explore it, it is full of sub-packages
* [The scipy lectures](http://www.scipy-lectures.org/) Very nice!!!
* [Do you know Sympy?](http://www.sympy.org/en/index.html)

```python
import sympy as sp
sp.init_printing()
from IPython.display import display

x = sp.symbols("x")
f = sp.sin(x)
display(f, sp.diff(f, x))
g = sp.diff(sp.sin(x)*sp.exp(-x), x)
display(g)
g_int = sp.integrate(g, x)
display(g_int)
```

```python
out = sp.integrate(sp.sin(x**2), (x, -sp.oo, sp.oo))
display(out)
# Not bad...
```

## HDF5, h5py, and pytables

The [h5py](http://docs.h5py.org/)  package is a Pythonic interface to the HDF5 binary data format.

HDF5 lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want.

[PyTables](https://www.pytables.org/) is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data.

PyTables is built on top of the HDF5 library, using the Python language and the NumPy package. It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code (generated using Cython), makes it a fast, yet extremely easy to use tool for interactively browse, process and search very large amounts of data

* [What is HDF5](https://support.hdfgroup.org/HDF5/whatishdf5.html)
* [Who uses HDF5](https://support.hdfgroup.org/HDF5/users5.html)

## Pickle

The [Pickle](https://docs.python.org/3/library/pickle.html) module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” [1] or “flattening”, however, to avoid confusion, the terms used here are “pickling” and “unpickling”.

```python
import pickle
# Let's save the masked array we made at the beginning
print(data_masked.shape)
with open("masked_array.pkl", 'wb') as f: # note the wb
    pickle.dump(data_masked, f)
```

    (32768,)

```python
# Let's check it if we really wrote the file
import glob
glob.glob1(".", "masked*")
```

    ['masked_array.pkl']

```python
# Try to unpickle
with open("masked_array.pkl", 'rb') as pickle_file:
    m_array = pickle.load(pickle_file)
```

```python
m_array[450:480]
```

    masked_array(data=[--, --, --, --, --, --, --, --, 2.070092773437532e-07,
                       2.970092773437532e-07, 2.170092773437532e-07,
                       1.970092773437532e-07, 1.870092773437532e-07, --, --,
                       --, 1.770092773437532e-07, --, --, --, --, --, --, --,
                       --, --, --, --, --, 1.370092773437531e-07],
                 mask=[ True,  True,  True,  True,  True,  True,  True,  True,
                       False, False, False, False, False,  True,  True,  True,
                       False,  True,  True,  True,  True,  True,  True,  True,
                        True,  True,  True,  True,  True, False],
           fill_value=1e+20)

```python
data_masked[450:480]
```

    masked_array(data=[--, --, --, --, --, --, --, --, 2.070092773437532e-07,
                       2.970092773437532e-07, 2.170092773437532e-07,
                       1.970092773437532e-07, 1.870092773437532e-07, --, --,
                       --, 1.770092773437532e-07, --, --, --, --, --, --, --,
                       --, --, --, --, --, 1.370092773437531e-07],
                 mask=[ True,  True,  True,  True,  True,  True,  True,  True,
                       False, False, False, False, False,  True,  True,  True,
                       False,  True,  True,  True,  True,  True,  True,  True,
                        True,  True,  True,  True,  True, False],
           fill_value=1e+20)

## Scikit-image

The [Scikit-Image](https://scikit-image.org/) is a collection of algorithms for image processing. It is available free of charge and free of restriction. We pride ourselves on high-quality, peer-reviewed code, written by an active community of volunteers.

```python
from skimage import data, filters

image = data.coins()
# ... or any other NumPy array!
edges = filters.sobel(image)
fig, axs = plt.subplots(1,2, figsize=(16,8))
axs[0].imshow(image, cmap='gray')
axs[1].imshow(edges, cmap='gray')
plt.show()
```

## Mahotas

[Mahotas](https://mahotas.readthedocs.io/en/latest/#) is a computer vision and image processing library for Python.

It includes many algorithms implemented in C++ for speed while operating in numpy arrays and with a very clean Python interface.

Mahotas currently has over 100 functions for image processing and computer vision and it keeps growing. Some examples of mahotas functionality:

* watershed
*        convex points calculations.
*        hit & miss. thinning
*        Zernike & Haralick, local binary patterns, and TAS features.
*        morphological processing
*        Speeded-Up Robust Features (SURF), a form of local features
*        thresholding
*        convolution.
*        Sobel edge detection.

## Open-cv

[OpenCV-python](https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_tutorials.html) is based on the [Open Source Computer Vision Library](http://opencv.org), an open-source BSD-licensed library that includes several hundreds of computer vision algorithms. The document describes the so-called OpenCV 2.x API, which is essentially a C++ API, as opposite to the C-based OpenCV 1.x API. The latter is described in opencv1x.pdf.

OpenCV has a modular structure, which means that the package includes several shared or static libraries. The following modules are available:

* core - a compact module defining basic data structures, including the dense multi-dimensional array Mat and basic functions used by all other modules.
* imgproc - an image processing module that includes linear and non-linear image filtering, geometrical image transformations (resize, affine and perspective warping, generic table-based remapping), color space conversion, histograms, and so on.
* video - a video analysis module that includes motion estimation, background subtraction, and object tracking algorithms.
* calib3d - basic multiple-view geometry algorithms, single and stereo camera calibration, object pose estimation, stereo correspondence algorithms, and elements of 3D reconstruction.
* features2d - salient feature detectors, descriptors, and descriptor matchers.
* objdetect - detection of objects and instances of the predefined classes (for example, faces, eyes, mugs, people, cars, and so on).
* highgui - an easy-to-use interface to simple UI capabilities.
* videoio - an easy-to-use interface to video capturing and video codecs.
* gpu - GPU-accelerated algorithms from different OpenCV modules.

## Scikit-learn

[Scikit-learn](https://scikit-learn.org/stable/index.html) is the Machine Learning in Python

*    Simple and efficient tools for data mining and data analysis
*    Accessible to everybody, and reusable in various contexts
*    Built on NumPy, SciPy, and matplotlib
*    Open source, commercially usable - BSD license

## Scikit-Cuda

[Scikit-cuda](https://scikit-cuda.readthedocs.io/en/latest/) provides Python interfaces to many of the functions in the CUDA device/runtime, CUBLAS, CUFFT, and CUSOLVER libraries distributed as part of NVIDIA’s CUDA Programming Toolkit, as well as interfaces to select functions in the CULA Dense Toolkit. Both low-level wrapper functions similar to their C counterparts and high-level functions comparable to those in NumPy and Scipy are provided.

## PyCuda

[PyCuda](https://mathema.tician.de/software/pycuda/) PyCUDA lets you access Nvidia‘s CUDA parallel computation API from Python

```python
try:
    import pycuda.autoinit
    import pycuda.driver as drv
    from pycuda.compiler import SourceModule

    mod = SourceModule("""
__global__ void multiply_them(float *dest, float *a, float *b)
{
  const int i = threadIdx.x;
  dest[i] = a[i] * b[i];
}
""")

    multiply_them = mod.get_function("multiply_them")

    a = numpy.random.randn(400).astype(numpy.float32)
    b = numpy.random.randn(400).astype(numpy.float32)

    dest = np.zeros_like(a)
    multiply_them(
        drv.Out(dest), drv.In(a), drv.In(b),
        block=(400,1,1), grid=(1,1))

    print(dest-a*b)
except:
    print("pyCuda not installed")
```

    pyCuda not installed

## and so on...

[Check this](https://stxnext.com/blog/2018/09/20/most-popular-python-scientific-libraries/) and [also here](https://medium.com/activewizards-machine-learning-company/top-15-python-libraries-for-data-science-in-in-2017-ab61b4f9b4a7)

And, finally, if you are lost there is always [PyPi](https://pypi.org/)

```python

```


# Continuous System

## Imports, stuff

```python
from sympy import *
init_printing(use_latex=True)
from IPython.display import Latex
%matplotlib inline
```

## Symbols

```python
x, w2 = symbols('x omega^2')
L, m, EJ = symbols('L m EJ', positive = True)
A, B, C, D, ld, LD = symbols('A B C D lambda Lambda')
f, φ = symbols('f phi')
```

## Supported mass and stiffness of support

```python
mass_coeff = 8
stiff_coeff = 24
k = stiff_coeff*EJ/L**3
M = mass_coeff*m*L
```

## General solution and its derivatives

```python
f0 = A*cos(ld*x) + B*sin(ld*x) + C*cosh(ld*x) + D*sinh(ld*x)
f1 = f0.diff(x)
f2 = f1.diff(x)
f3 = f2.diff(x)
display(Eq(φ,f0))
```

### Left boundary conditions
The eigenfunction and its second derivative must be zero when `0` is substituted for `x`, we solve for `A` and `C` and put the solution in the variable `AC`.
We substitute our solution in the eigenfunctions and all of its derivatives.

```python
AC = solve((f0.subs(x,0), f2.subs(x,0)), A, C, dict=True)
f0, f1, f2, f3 = [f.subs(AC[0]) for f in (f0, f1, f2, f3)]
display(Eq(φ, f0))
```

### First, simpler boundary condition at the right end, $x=L$.
The second derivative must be equal to zero,  so we solve and  substitute, also substitute $\lambda L$ with $\Lambda$

```python
D = solve(f2.subs(x, L), D, dict=True)
f0, f1, f2, f3 = [f.subs(D[0]).subs(L,LD/ld) for f in (f0, f1, f2, f3)]
display(Latex('With $\\Lambda = \\lambda\\,L$ it is'))
display(Eq(φ, f0.simplify()))
```

### Last boundary conditions, equation of wave numbers
The last equation is an equation of equilibrium
$$V(t) + k\, v(t) + M\,\ddot v(t) = 0$$
(all the forces are directed upwards).

With $v(t)=\phi(x)\,\sin\omega t$, the shear is $V = -EJ\, v''' = -EJ\, \phi'''(x)\sin\omega t$ and
the inertial force is $M\,\ddot v= -M\,\phi\,\omega^2\sin\omega t$ that can be rewritten taking into account that $\omega^2=\lambda^4EJ/m$: $\text{ }M\,\ddot v= - M/m\,EJ\,\lambda^4\phi\sin\omega t$.

Let's write the expanded equation, collecting all the terms that are no $\Lambda$:

```python
eq = (f0*k - f0*M*ld**4*EJ/m - EJ*f3).subs(x, L).subs(L, LD/ld)
display(Eq(eq.expand().collect(B).collect(ld).collect(EJ), 0))
```

We have a non trivial solution when the term in brackets is equal to zero, to have the bracketed term we must divide both members by $B\,EJ\, \lambda^3$

```python
eq = (eq/EJ/ld**3/B).expand()
display(Eq(eq,0))
```

The behavior near $\Lambda=0$ is led by the last term that goes like $48/\Lambda^2$, so to have a nice plot we multiply everything by $\Lambda^2$

```python
display(Eq(symbols('f'), (eq*LD**2).expand()))
plot(eq*LD**2, (LD, 0, 2));
```

and see that there is a root between 1.25 and 1.5.  If we were interested in upper roots, we can observe that all the terms in the LHS of our determinantal equations are bounded for increasing $\Lambda$ except for the first one, that grows linearly, so to investigate the other roots we may divide the equation by $\Lambda$ to remove that trend...

```python
display(Eq(symbols('f'), (eq/LD).expand()))
plot(eq/LD, (LD, 2, 10));
```

All the RHS terms except the first have $\Lambda$ in the denominator and are bounded, so the asymptotic behaviour is controlled by $\Lambda_{n+1}=n\pi$.

```python
from scipy.optimize import bisect
f = lambdify(LD, eq, modules='math')
l1 = bisect(f, 0.5, 1.5)
Latex(r'$\lambda_1=%.6f\,\frac{1}{L}, \quad\omega_1^2=%.6f\,\frac{EJ}{mL^4}$'%(l1, l1**4))
```

$\lambda_1=1.302466\,\frac{1}{L}, \quad\omega_1^2=2.877834\,\frac{EJ}{mL^4}$

## Rayleigh Quotient

Using $v=\frac xL\sin\omega t$ (that is, a rigid rotation about the left hinge) we have
$$T_\text{max}=\frac12\omega^2\Big(\int_0^Lm\left(\frac xL\right)^2dx + M\,1^2\Big) =
\frac12\omega^2\Big(\frac13+8\Big)mL
$$
and
$$V_\text{max}=\frac12\Big(\int_0^L EJ\left(\frac xL\right)''^2 + k\,1^2\Big) =
\frac12\Big(0+24\Big)\,\frac{EJ}{L^3}.
$$

Equating the maximum energies and solving for $\omega^2$ gives
$$ \omega^2 = \frac{24\,EJ/L^3}{\frac{25}3\,mL} = 3\,\frac{24}{25}\,\frac{EJ}{mL^4}=\ldots
$$

```python
display(Latex(r'$\omega^2_{R00} = %.3f\,\frac{EJ}{mL^4}$'%(3*24/25)))
```

$\omega^2_{R00} = 2.880\,\frac{EJ}{mL^4}$

We can say that the RQ check reinforces our previouos finding...

```python

```


# Computational and Numerical Methods
## Group 16
### Set 6 (10-09-2018):   Lagrange and Newton Interpolation
#### Vidhin Parmar   201601003
#### Parth Shah      201601086

<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import math
from scipy.interpolate import lagrange as l
from sympy import *

def plot(x, y,lab):
    plt.plot(x, y, label=lab)
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()

def lag(x, y, xp, laglab):
    poly = l(x, y)
    plt.plot(xp, poly(xp), label=laglab)
    plt.grid(True)
    plt.legend(loc='best')
    print("The Lagrangian polynomial is:")
    print(poly)

def lagwithfunc(f, x, y, xp, lab):
    poly = l(x, y)
    plt.plot(xp, f(xp), label = lab)
    plt.plot(xp, poly(xp), label="Lagrange Polynomial")
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()
    print("The Lagrangian polynomial is:")
    print(poly)

def divdifcoef(x, y):
    '''x : array of data points
       y : array of f(x)  '''
    x.astype(float)
    y.astype(float)
    n = len(x)
    a = []
    for i in range(n):
        a.append(y[i])
    for j in range(1, n):
        for i in range(n-1, j-1, -1):
            a[i] = float(a[i]-a[i-1])/float(x[i]-x[i-j])

    return np.array(a) # array of coefficients

def divdifEval(a, x, r):
    ''' a : array returned by function coef()
        r : the node to interpolate at  '''

    x.astype(float)
    n = len( a ) - 1
    temp = a[n]
    for i in range( n - 1, -1, -1 ):
        temp = temp * ( r - x[i] ) + a[i]
    return temp # y_value interpolation
```

# Q1: $\sqrt{x}$

```python
f = lambda x: x**0.5
lagwithfunc(f, np.array([1, 4]), np.array([1, 2]), np.arange(0,5,0.01), "y=x^0.5")
```

# Q2: $e^{x}$

```python
xp = np.arange(0, 1.5, 0.01)
lag(np.array([0.82, 0.83]), np.array([2.270500, 2.293319]), xp, "First order Lagrange Polynomial")
plt.plot(xp, np.exp(xp), label="y = e^x")
lag(np.array([0.82, 0.83, 0.84]), np.array([2.270500, 2.293319,2.316367]), xp, "Second order Lagrange Polynomial")
plt.show()
```

# Q3 x = [0, 1, 2] y = [-1, -1, 7]

```python
lag(np.array([0, 1, 2]), np.array([-1, -1, 7]), np.arange(0,5,0.01), "Lagrange Polynomial")
x = np.array([0, 1, 2])

a = divdifcoef(x, np.array([-1, -1, 7]))
xp = np.arange(0,5,0.01)
plt.plot(xp, divdifEval(a, x, xp), label = "Newton's Divided difference")
plt.grid(True)
plt.legend(loc='best')
plt.show()
print("Newton's divided difference coefficients are:")
print(a)
```

We see that the interpolating function turns out to be the same for the Lagrange method and the Newton Divided Difference method.

# Q4 Interpolation of $\frac{1}{x}$:

```python
xp = np.arange(1, 6, 0.01)
x = np.array([3.35, 3.4, 3.5, 3.6])
y = np.array([0.298507, 0.294118, 0.285714, 0.277778])
lag(x[0:2], y[0:2], xp, "Linear Lagrange Polynomial")
plt.plot(xp, 1/xp, label="y = 1/x")
a = divdifcoef(x[0:2], y[0:2])
plt.plot(xp, divdifEval(a, x[0:2], xp), label = "Linear Newton's Divided difference")
print("Linear Newton's divided difference coefficients are:")
print(a)
print("\n")

plt.plot(xp, 1/xp, label="y = 1/x")
lag(x[0:3], y[0:3], xp, "Quadratic Lagrange Polynomial")
a = divdifcoef(x[0:3], y[0:3])
plt.plot(xp, divdifEval(a, x[0:3], xp), label = "Quadratic Newton's Divided difference")
print("Quadratic Newton's divided difference coefficients are:")
print(a)
print("\n")

plt.plot(xp, 1/xp, label="y = 1/x")
lag(x[0:4], y[0:4], xp, "Cubic Lagrange Polynomial")
a = divdifcoef(x[0:4], y[0:4])
plt.plot(xp, divdifEval(a, x[0:3], xp), label = "Cubic Newton's Divided difference")
print("Cubic Newton's divided difference coefficients are:")
print(a)
plt.grid(True)
plt.legend(loc='best')
plt.show()
```

Lagrange interpolation is mostly just useful for theory. Computing with it requires huge numbers and catastrophic cancellations. In floating point arithmetic this is very bad. It does have some small advantages: for instance, the Lagrange approach amounts to diagonalizing the problem of finding the coefficients, so it takes only linear time to find the coefficients. This is good if you need to use the same set of points repeatedly. But all of these advantages do not make up for the problems associated with trying to actually evaluate a Lagrange interpolating polynomial.

With Newton interpolation, you get the coefficients reasonably fast (quadratic time), the evaluation is much more stable (roughly because there is usually a single dominant term for a given x), the evaluation can be done quickly and straightforwardly using Horner's method, and adding an additional node just amounts to adding a single additional term. It is also fairly easy to see how to interpolate derivatives using the Newton framework.

# Q5 x = [0, 1, 2, 2.5, 3, 3.5, 4] y = [2.5, 0.5, 0.5, 1.5, 1.5, 1.125, 0]

```python
def lag(x, y, xp):
    poly = l(x, y)
    plt.plot(xp, poly(xp))
    plt.grid(True)
    plt.legend(loc='best')
    print("The Lagrangian polynomial is:")
    print(poly)
x = np.array([0, 1, 2, 2.5, 3, 3.5, 4])
y = np.array([2.5, 0.5, 0.5, 1.5, 1.5, 1.125, 0])
xp = np.arange(0, 4, 0.01)
yp = np.interp(xp, x, y)
plt.plot(xp, yp, label = "PLI")
lag(x[0:3], y[0:3], xp)
a = divdifcoef(x[0:3], y[0:3])
plt.plot(xp, divdifEval(a, x[0:3], xp), label= "x[0, 2] LP")
print()
print("x[0:3] Newton's divided difference coefficients are:")
print(a)
print("\n")

lag(x[3:5], y[3:5], xp)
a = divdifcoef(x[3:5], y[3:5])
plt.plot(xp, divdifEval(a, x[3:5], xp), label = "x[2, 3] LP")
print()
print("x[3:5] Newton's divided difference coefficients are:")
print(a)
print("\n")

lag(x[5:7], y[5:7], xp)
a = divdifcoef(x[5:7], y[5:7])
plt.plot(xp, divdifEval(a, x[5:7], xp), label= "x[3, 4] LP")
print()
print("x[5:7] Newton's divided difference coefficients are:")
print(a)
plt.grid(True)
plt.legend(loc='best')
plt.show()
```

## Now comparing with Piecewise Linear Interpolation

```python
a = divdifcoef(x[0:3], y[0:3])
plt.plot(xp, divdifEval(a, x[0:3], xp) - yp, label = "x[0:3] - PLI")
a = divdifcoef(x[3:5], y[3:5])
plt.plot(xp, divdifEval(a, x[3:5], xp) - yp, label = "x[3:5] - PLI")
a = divdifcoef(x[5:7], y[5:7])
plt.plot(xp, divdifEval(a, x[5:7], xp) - yp, label = "x[5:7] - PLI")
plt.grid(True)
plt.legend(loc='best')
plt.show()
```

The Lagrange Polynomial/Newton Divided Difference is closest to the Piecewise Linear function in the interval in which the Polynomial is made i.e. The Lagrangian Polynomial made by taking two points is closest to the Piecewise Linear function between those points.


```python
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import scipy.fftpack
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
# import ipyvolume as ipv
%matplotlib inline
import discretisedfield.tools as dft
import discretisedfield as df
import oommfc as oc
import micromagneticmodel as mm
```

A common problem in magnetics is the study of the Heisenberg Hamiltonian in 3-D space, commonly on a regular lattice:

\begin{equation}
\mathcal{H} = -\frac{1}{2}\sum_{ij}S_{i} J_{ij} S_{j} - h \sum_{i} S_i - \sum_{i} S_i \sum_{j} \frac{3(S_j \cdot \hat{r}_{ij})\hat{r}_{ij}}{r_{ij}^5} - \frac{S_j}{r_{ij}^3}
\end{equation}

Commonly the dipolar field calculation is often neglected for large scale calculatiins because the computational complexity of calculating the interaction has $\mathcal{O}(N^2)$ complexity. In some cases, particularly where modelling 2-D systems, it is approximated by including an energy term in the Hamiltonian which has the same form as the Uniaxial Anisotropy, however it has been shown that in 3-D cases this is *not* sufficient, and can give incorrect results when modelling magnetisation dynamics with the Landau-Lifshitz-Gilbert equation. The dipolar interaction is crucial to the stabilisation of many interesting features in the magnetic field, such as domain walls, vortices and skyrmions.

In this notebook, we aim to show how the field can be calculated much more quickly, using a Fast Fourier Transform (FFT) convolution approach.

To start, lets consider a three-dimensional cubic lattice of spins, with $n_x \times n_y \times n_z$ in total.

If we index each axis as (i, j, k) We can give each spin a unique index, given by its position in the lattice, as
\begin{equation}
\text{I} = n_x\times n_y\times k + n_x\times j + i
\end{equation}

The position of a given spin $I$ is then just given by:
\begin{equation}
\mathbf{r_I} = (i\,a_x, j\,a_y, k\,a_z)
\end{equation}
where $a_x, a_y, a_z$ are the distances between adjacent cubic unit cells along each coordinate axis.

Below, we just plot the magnetisation for a uniformly magnetised sample:

```python
import numpy as np

nx = ny = nz = 5
lenx = 2*nx - 1
leny = 2*ny - 1
lenz = 2*nz - 1
shape = (lenx, leny, lenz)
ax = 1
ay = 1
az = 1

np.random.seed(0)

mx = np.random.uniform(-1, 1, (nz, ny, nz))
my = np.random.uniform(-1, 1, (nz, ny, nz))
mz = np.random.uniform(-1, 1, (nz, ny, nz))
mod = np.sqrt(mx ** 2 + my ** 2 + mz ** 2)
mx /= mod
my /= mod
mz /= mod
#mx = np.ones((nx, ny, nz))
#my = np.ones((nx, ny, nz)) * 2
#mz = np.ones((nx, ny, nz)) * 3

Hx = np.zeros_like(mx)
Hy = np.zeros_like(my)
Hz = np.zeros_like(mz)

Hx_fft = np.zeros_like(mx)
Hy_fft = np.zeros_like(my)
Hz_fft = np.zeros_like(mz)
```

```python
def H_direct(mx, my, mz, Hx, Hy, Hz):
    Hx[:] = 0
    Hy[:] = 0
    Hz[:] = 0
    for k in range(nz):
        for j in range(ny):
            for i in range(nx):
                for kp in range(nz):
                    for jp in range(ny):
                        for ip in range(nx):
                            x = (i - ip)*ax
                            y = (j - jp)*ay
                            z = (k - kp)*az
                            dr = np.sqrt(x**2 + y**2 + z**2)
                            # Spins don't interact with themselves, so skip if on same position
                            if (dr == 0):
                                continue
                            r = np.array([x, y, z])
                            S = np.array([mx[kp, jp, ip], my[kp, jp, ip], mz[kp, jp, ip]])
                            H = 1e-7 * (3*(r.dot(S))*r / dr**5 - S/dr**3)
                            Hx[k, j, i] += H[0]
                            Hy[k, j, i] += H[1]
                            Hz[k, j, i] += H[2]

def Nxx_calc(x, y, z):
    x2 = x*x
    y2 = y*y
    z2 = z*z

    # x, y, z - coords in mesh
    # mesh.r(point, origin=pmin)
    r2 = x2 + y2 + z2

    return (2 * x2 - y2 - z2) / r2**2.5 if r2 != 0 else 0

def Nxy_calc(x, y, z):
    r2 = x*x + y*y + z*z

    return 3*x*y / r2**2.5 if r2 != 0 else 0

def ComputeTensor(nx, ny, nz, ax, ay, az):
    lenx = 2*nx - 1
    leny = 2*ny - 1
    lenz = 2*nz - 1
    total = lenx*leny*lenz
    shape = (lenx, leny, lenz)
    rshape = (lenx, leny, lenz//2 + 1)
    Nxx = np.zeros(shape)
    Nxy = np.zeros_like(Nxx)
    Nxz = np.zeros_like(Nxx)
    Nyy = np.zeros_like(Nxx)
    Nyz = np.zeros_like(Nxx)
    Nzz = np.zeros_like(Nxx)
    for k in range(lenz):
        for i in range(lenx):
            for j in range(leny):
                # Notice that this spans the whole range of -nx*ax to nx*ax
                x = (i - nx + 1) * ax
                y = (j - ny + 1) * ay
                z = (k - nz + 1) * az
                Nxx[k, j, i] = Nxx_calc(x, y, z)
                Nyy[k, j, i] = Nxx_calc(y, x, z)
                Nzz[k, j, i] = Nxx_calc(z, y, x)
                Nxy[k, j, i] = Nxy_calc(x, y, z)
                Nxz[k, j, i] = Nxy_calc(x, z, y)
                Nyz[k, j, i] = Nxy_calc(y, z, x)
    print(Nxx.shape)
    Nxxt = np.fft.rfftn(Nxx)
    print(Nxxt.shape)
    Nxyt = np.fft.rfftn(Nxy)
    Nxzt = np.fft.rfftn(Nxz)
    Nyyt = np.fft.rfftn(Nyy)
    Nyzt = np.fft.rfftn(Nyz)
    Nzzt = np.fft.rfftn(Nzz)
    return (Nxx, Nxy, Nxz, Nyy, Nyz, Nzz), (Nxxt, Nxyt, Nxzt, Nyyt, Nyzt, Nzzt)

untransformed, tensor = ComputeTensor(nx, ny, nz, ax, ay, az)

def H_FFT(mx, my, mz, tensor, Hx, Hy, Hz):
    Hx[:] = 0
    Hy[:] = 0
    Hz[:] = 0
    Nxxt, Nxyt, Nxzt, Nyyt, Nyzt, Nzzt = tensor
    nz, ny, nx = mx.shape
    shape = (lenx, leny, lenz)
    mx_padded = np.zeros(shape)
    my_padded = np.zeros(shape)
    mz_padded = np.zeros(shape)
    mx_padded[:nx, :ny, :nz] = mx*1e-7

    my_padded[:nx, :ny, :nz] = my*1e-7
    mz_padded[:nx, :ny, :nz] = mz*1e-7
    Mxt = np.fft.rfftn(mx_padded)
    Myt = np.fft.rfftn(my_padded)
    Mzt = np.fft.rfftn(mz_padded)

    Hxt = (Nxxt*Mxt + Nxyt*Myt + Nxzt*Mzt)
    Hyt = (Nxyt*Mxt + Nyyt*Myt + Nyzt*Mzt)
    Hzt = (Nxzt*Mxt + Nyzt*Myt + Nzzt*Mzt)
    hxc = np.fft.irfftn(Hxt, s=mx_padded.shape)
    hyc = np.fft.irfftn(Hyt, s=mx_padded.shape)
    hzc = np.fft.irfftn(Hzt, s=mx_padded.shape)

    for k in range(nz):
        for j in range(ny):
            for i in range(nx):
                Hx[k, j, i] = hxc[k + nz - 1, j + ny - 1, i + nx - 1]
                Hy[k, j, i] = hyc[k + nz - 1, j + ny - 1, i + nx - 1]
                Hz[k, j, i] = hzc[k + nz - 1, j + ny - 1, i + nx - 1]
    return hxc, hyc, hzc
```

    (9, 9, 9)
    (9, 9, 5)

```python
mesh = df.Mesh(p1=(0,0,0), p2=(5,5,5), cell=(1,1,1))
```

```python
def f(x, y, z):
    eps = 1e-18  # to avoid zero division; similar to Albert et al. 2015
    return (abs(y) / 2 * (z**2 - x**2) * np.arcsinh(abs(y) / (np.sqrt(x**2 + z**2) + eps))
            + abs(z) / 2 * (y**2 - x**2) * np.arcsinh(abs(z) / (np.sqrt(x**2 + y**2) + eps))
            - abs(x * y * z) * np.arctan(abs(y * z) / (abs(x) * np.sqrt(x**2 + y**2 + z**2) + eps))
            + 1/6 * (2 * x**2 - y**2 - z**2) * np.sqrt(x**2 + y**2 + z**2))

def g(x, y, z):
    eps = 1e-18
    return (x * y * z * np.arcsinh(z / (np.sqrt(x**2 + y**2) + eps))
            + y / 6 * (3 * z**2 - y**2) * np.arcsinh(x / (np.sqrt(y**2 + z**2) + eps))
            + x / 6 * (3 * z**2 - x**2) * np.arcsinh(y / (np.sqrt(x**2 + z**2) + eps))
            - z**3 / 6 * np.arctan(x * y / (z * np.sqrt(x**2 + y**2 + z**2) + eps))
            - z * y**2 / 2 * np.arctan(x * z / (y * np.sqrt(x**2 + y**2 + z**2) + eps))
            - z * x**2 / 2 * np.arctan(y * z / (x * np.sqrt(x**2 + y**2 + z**2) + eps))
            - x * y * np.sqrt(x**2 + y**2 + z**2) / 3
    )

def N_element(x, y, z, mesh, function):
    dx, dy, dz = mesh.cell
    value = 0.
    for i in np.rollaxis(np.indices((2,) * 6), 0, 7).reshape(64, 6):
        #print(x + (i[0] - i[3]) * dx, y + (i[1] - i[4]) * dy , z + (i[2] - i[5]) * dz, (-1)**np.sum(i), function(x + (i[0] - i[3]) * dx, y + (i[1] - i[4]) * dy , z + (i[2] - i[5]) * dz))
        value += (-1)**np.sum(i) * function(x + (i[0] - i[3]) * dx, y + (i[1] - i[4]) * dy , z + (i[2] - i[5]) * dz)
    return - value / (4 * np.pi * np.prod(mesh.cell))

def N_calc_new(mesh):
    def _inner(p):
        x, y, z = p
        return (N_element(x, y, z, mesh, f),  # Nxx
                N_element(y, z, x, mesh, f),  # Nyy
                N_element(z, x, y, mesh, f),  # Nzz
                N_element(x, y, z, mesh, g),  # Nxy
                N_element(x, z, y, mesh, g),  # Nxz
                N_element(y, z, x, mesh, g))  # Nyz
    return _inner
```

```python
def Nxx_calc(x, y, z):
    x2 = x*x
    y2 = y*y
    z2 = z*z

    # x, y, z - coords in mesh
    # mesh.r(point, origin=pmin)
    r2 = x2 + y2 + z2

    return (2 * x2 - y2 - z2) / r2**2.5 if r2 != 0 else 0

def Nxy_calc(x, y, z):
    r2 = x*x + y*y + z*z

    return 3*x*y / r2**2.5 if r2 != 0 else 0

def N_calc(p):
    x, y, z = p
    return (Nxx_calc(x, y, z), Nxx_calc(y, z, x), Nxx_calc(z, x, y),
            Nxy_calc(x, y, z), Nxy_calc(x, z, y), Nxy_calc(y, z, x))

def ComputeTensor_new(mesh):
    p1 = [(-i + 1) * j - j/2 for i, j in zip(mesh.n, mesh.cell)]
    p2 = [(i - 1) * j + j/2 for i, j in zip(mesh.n, mesh.cell)]
    n = [2*i - 1 for i in mesh.n]
    mesh_new = df.Mesh(p1=p1, p2=p2, n=n)

    return df.Field(mesh_new, dim=6, value=N_calc_new(mesh_new)).fftn
    # Nxxt, Nyyt, Nzzt, Nxyt, Nxzt, Nyzt
    #   0     1     2     3     4     5

def H_FFT(mx, my, mz, tensor, Hx, Hy, Hz):
    Hx[:] = 0
    Hy[:] = 0
    Hz[:] = 0
    Nxxt, Nxyt, Nxzt, Nyyt, Nyzt, Nzzt = tensor
    nz, ny, nx = mx.shape
    shape = (lenx, leny, lenz)
    mx_padded = np.zeros(shape)
    my_padded = np.zeros(shape)
    mz_padded = np.zeros(shape)
    mx_padded[:nx, :ny, :nz] = mx*1e-7

    my_padded[:nx, :ny, :nz] = my*1e-7
    mz_padded[:nx, :ny, :nz] = mz*1e-7
    Mxt = np.fft.rfftn(mx_padded)
    Myt = np.fft.rfftn(my_padded)
    Mzt = np.fft.rfftn(mz_padded)

    Hxt = (Nxxt*Mxt + Nxyt*Myt + Nxzt*Mzt)
    Hyt = (Nxyt*Mxt + Nyyt*Myt + Nyzt*Mzt)
    Hzt = (Nxzt*Mxt + Nyzt*Myt + Nzzt*Mzt)
    hxc = np.fft.irfftn(Hxt, s=mx_padded.shape)
    hyc = np.fft.irfftn(Hyt, s=mx_padded.shape)
    hzc = np.fft.irfftn(Hzt, s=mx_padded.shape)

    for k in range(nz):
        for j in range(ny):
            for i in range(nx):
                Hx[k, j, i] = hxc[k + nz - 1, j + ny - 1, i + nx - 1]
                Hy[k, j, i] = hyc[k + nz - 1, j + ny - 1, i + nx - 1]
                Hz[k, j, i] = hzc[k + nz - 1, j + ny - 1, i + nx - 1]
    return hxc, hyc, hzc
```

```python
m_field = df.Field(mesh, dim=3, value=np.stack((mx, my, mz), axis=3), norm=1)
```

```python
system = mm.System(name='test')
system.m = m_field
system.energy = mm.Demag()
```

```python
oc_field = oc.compute(system.energy.demag.effective_field, system)
```

    Running OOMMF (ExeOOMMFRunner) [2021/08/03 17:50]... (1.1 s)

```python
tensor_new = ComputeTensor_new(mesh)
```

```python
tensor_dft = dft.demag_tensor(mesh)
```

```python
def H_FFT_new(m, tensor):
    m_pad = m.pad({d: (0, m.mesh.n[i] - 1) for d, i in zip(['x', 'y', 'z'], range(3))}, mode='constant')
    m_pad_fft = m_pad.fftn

    hx = tensor.array[..., 0] * m_pad_fft.x.array.squeeze() + tensor.array[..., 3] * m_pad_fft.y.array.squeeze() + tensor.array[..., 4] * m_pad_fft.z.array.squeeze()
    hy = tensor.array[..., 3] * m_pad_fft.x.array.squeeze() + tensor.array[..., 1] * m_pad_fft.y.array.squeeze() + tensor.array[..., 5] * m_pad_fft.z.array.squeeze()
    hz = tensor.array[..., 4] * m_pad_fft.x.array.squeeze() + tensor.array[..., 5] * m_pad_fft.y.array.squeeze() + tensor.array[..., 2] * m_pad_fft.z.array.squeeze()
    H = df.Field(m_pad_fft.mesh, dim=3, value=np.stack((hx, hy, hz), axis=3)).ifftn
    return df.Field(m.mesh, dim=3, value=H.array[m.mesh.n[0] - 1:, m.mesh.n[1] - 1:, m.mesh.n[2] - 1:, :]).real
```
H_direct(mx, my, mz, Hx, Hy, Hz)hxc, hyc, hzc = H_FFT(mx, my, mz, tensor, Hx_fft, Hy_fft, Hz_fft)

```python
H_new = H_FFT_new(m_field, tensor_new)
```

```python
H_dft = dft.demag_field(m_field, tensor_dft)
```

```python
H_new.plane(z=1.5).mpl()
```

```python
oc_field.plane(z=1.5).mpl()
```

```python
H_dft.plane(z=1.5).mpl()
```

```python

```


# SIR Differential Equations

In this notebook we will model the spread of disease using the Kermack-McKendrick Model model which is a compartment model.

```python
## Library
import numpy as np
import math
import matplotlib.pyplot as plt
```

The model is a system of three non-linear first order differential equations:
\begin{equation}
\begin{array}{cl}
\frac{dS}{dt}=-\frac{\beta IS}{N},\\
\frac{dI}{dt}=\frac{\beta IS}{N}-\gamma I,\\
\frac{dR}{dt}=\gamma I
\end{array}
\end{equation}
where
S is the people who are susceptible in the population,
I is the number of people infected,
R is the number of removed population (either by death or recovery), and
N is the sum of these three $$ N=S(t_0)+I(t_0)+R(t_0).$$
$\beta$ is the infection rate, and $\gamma$ is the recovery rate.
$$ R_{0}={\frac{\beta }{\gamma }},$$

the so-called basic reproduction number (also called basic reproduction ratio).

```python
beta=1
gamma=1

```

## Discrete Domain
To numiercally approximate the solution we discretise the continuous domain using a step size, $h=0.001$,
which gives
\begin{equation}
t_i=0+i h,
\end{equation}
for $i=0,...n$.
The figure below illustrate the discrete domain.

```python
h=0.001
t=np.arange(0,10+h/2,h)
fig = plt.figure(1,figsize=(5,4))
ax = fig.add_subplot(1,1,1)
ax.plot(t,0*t,'o:',color='k')
ax.set_xlabel('Time (yrs)')
plt.title('Discrete Time Domain')
plt.show()
```

```python
n=len(t) # Number of Steps
S=np.zeros(n) # Numerical Solution Hares
S[0]=99 # Intial Hare population
I=np.zeros(n) # Numerical Solution Lynxes
I[0]=1# Initial Lynx population
R=np.zeros(n) # Numerical Solution Lynxes
R[0]=0# Initial Lynx population

N=100

```

We numerically approximate the differential equation using Eulers method to give the difference equation
\begin{equation}
S[i+1] = S[i]+ h*\big(-beta*S[i]*I[i]/N \big)
\end{equation}

\begin{equation}
I[i+1] = I[i]+ h*\big(\beta*S[i]*I[i] /N- gamma I[i] \big)
\end{equation}

\begin{equation}
R[i+1] = R[i]+ h*\big(gamma I[i] \big)
\end{equation}

where $h$ is the stepsize and  $H[i]$ and $L[i]$ is the numerical approximation of the susceptible $S(t_i)$, infected $I(t_i)$  and recovered $R(t_i)$ at time $t_i$, for $i=0,...,n$

```python
for i in range (0,n-1):
    S[i+1]=S[i]+h*(-beta*S[i]*I[i]/N)
    I[i+1]=I[i]+h*(beta*S[i]*I[i]/N-gamma*I[i])# INSERT EQUATION HERE
    R[i+1]=R[i]+h*gamma*I[i]

```

## Results

```python
## Plotting Figure
fig = plt.figure(1,figsize=(10,4))

ax = fig.add_subplot(1,1,1)

ax.plot(t,S,':',color='green',label='Sus')
ax.plot(t,I,':',color='red',label='Inf')
ax.plot(t,R,':',color='blue',label='Rec')

ax.legend()
ax.set_xlabel('Time (yrs)')
ax.set_ylabel('Population (tens of thousands)')
#ax.legend(loc='best')
plt.tight_layout()
plt.subplots_adjust(top=0.85)
plt.show()

```

## Phase Plane Plot

```python
## Plotting Figure
fig = plt.figure(1,figsize=(5,5))

ax = fig.add_subplot(1,1,1)

ax.plot(I,S,'-',color='black',label='I vs S')
ax.plot(I,R,'-',color='red',label='I vs R')

ax.set_xlabel('Infected Population (tens of thousands)')
ax.set_ylabel('Population (tens of thousands)')
ax.legend(loc='best')
plt.tight_layout()
plt.ylim(0,110)
plt.xlim(0,110)

plt.show()
```

# References

[1] Weisstein, Eric W. "Kermack-McKendrick Model." From MathWorld--A Wolfram Web Resource. https://mathworld.wolfram.com/Kermack-McKendrickModel.html

[2]  Wikipedia contributors. (2021, February 28). Compartmental models in epidemiology. In Wikipedia, The Free Encyclopedia. Retrieved 18:12, March 15, 2021, from https://en.wikipedia.org/w/index.php?title=Compartmental_models_in_epidemiology&oldid=1009414539

```python

```


# Spline transformations of points

```python
import numpy as np
%matplotlib notebook
import matplotlib as mpl
import matplotlib.pyplot as plt
import summerschool as ss
import os
```

## Arbitrary vector fields have too many degrees of freedom

If we have an arbitrary vector field $v(x)$, and we want to match a set of points $X$ to a set of points $Y$, we can just define the vector field as
\begin{align*}
v(x) &=
\left\{
\begin{array}{ll}
Y_i-X_i & x=X_i\\
\text{anything} & \text{otherwise}
\end{array}
\right.
\end{align*}

Usually we want to use vector fields to find out how images are deforming at points other than the labeled landmarks.  These vector fields have too many degres of freedom.  We need to enforce smoothness.

## Energy methods for enforcing smoothness
Let's define the energy of a vector field by
\begin{align*}
E &= \frac{1}{2}\int v^T(x) A v(x) dx
\end{align*}
where $A$ is a differential operator.

For example, if $v$ represents the displacement of points in an elastic body, then elastic energy is given by
\begin{align*}
\frac{1}{2}\int k\left|\frac{d}{dx}v(x)\right|^2 &= \int k\frac{d}{dx}v^T(x)\frac{d}{dx}v(x) dx\\
&= \frac{1}{2}\int  v^T(x) \left( -k \frac{d^2}{dx^2}\right)v(x)\\
&= \frac{1}{2}\int  v^T(x)Av(x)\\
\end{align*}
for $A = -k \frac{d^2}{dx^2}$.  Here we used the integration by parts, assuming 0 boundary conditions.

A common choice for $A$ is identity minus Laplacian times a constant raised to some integer power: $A = (id - a^2\Delta)^p$ for some choice of $a$ (with units of space) and $p$ (an integer, typically 2 or 4).

For a well defined positive energy, $A$ should be symmetric (i.e. a Hermitian operator).

We can now pose new landmark matching problem: minimize $\frac{1}{2} \int v^T(x) A v(x)dx$ subject to $v(X_i) = Y_i-X_i$.  How to solve this?

## Green's kernels for solutions to energy problems
Write down the minimization problem with constraints using Lagrange multipliers.  We minimize the cost
\begin{align*}
C = \frac{1}{2}\int v^T(x) A v(x) dx +  \sum_i \lambda_i[(Y_i - X_i) - v(X_i)]
\end{align*}
where $\lambda_i$ are the Lagrange multipliers.

We look for a stationary solution by taking perturbations $v \mapsto v + \epsilon \delta v$, and insiting the change be zero for any perturbation.
\begin{align*}
&\frac{d}{d\epsilon}\frac{1}{2} \int (v+\epsilon \delta v)^T(x) A (v+\epsilon \delta v)(x) dx +  \sum_i \lambda_i[(Y_i - X_i) - v(X_i) - \epsilon \delta v(X_i)] \bigg|_{\epsilon = 0}\\
&= \int Av^T(x) \delta v(x) dx + \sum_i \lambda_i \delta v(X_i)
\end{align*}

Let's use a Dirac $\delta$ to put everything under the same integral.
\begin{align*}
&= \int Av^T(x) \delta v(x)  - \sum_i \lambda_i \delta v(x) \delta (x-X_i)dx\\
&=\int \left(Av(x) - \sum_i \lambda_i \delta(x-X_i)\right)^T \delta v(x) dx
\end{align*}
If this variation is to be 0 for any perturbation, we require
\begin{align}
Av(x) = \sum_i \lambda_i \delta(x-X_i)
\end{align}

To solve this, we need to invert the operator $A$.  We'll assume that there is a function $K(x,y)$ such that $A K(x,y) = \delta(x,y)$.  This is called the Green's function of the operator.  The Green's functions for many operators can be looked up https://en.wikipedia.org/wiki/Green%27s_function.  Sometimes a convenient way to find Green's functions is by using Fourier transforms.  Let's plug in the Green's function
\begin{align*}
Av(x) &= \sum_i \lambda_i A K(x,X_i)\\
&= A\left(\sum_i \lambda_i K(x,X_i)\right)
\end{align*}
This will be true if
\begin{align*}
v(x) &= \sum_i \lambda_i K(x,X_i)
\end{align*}

This gives us the form of the solution.  Now we have to find the values of the Lagrange multipliers that will satisfy the constraints.  We require
\begin{align*}
v(X_j) &= \sum_i \lambda_i K(X_j,X_i)\\
&= Y_j-X_j
\end{align*}
This is just a linear equation, it can be written in matrix form.
\begin{align*}
\lambda \bar K = Y-X \implies \lambda = (Y-X)\bar K^{-1}
\end{align*}
where $\bar K$ is the matrix whose $i$th row and $j$th column is $K(X_i,X_j)$.

Note that the solution to the problem can be written in terms of $K$ only, not $A$.  In many cases we'll simply choose a Green's kernel, rather than an energy operator.

```python
f,ax = plt.subplots(1,3)
# simple example, match X to Y
X = np.array([[0,0],[1,0],[2,0]]).transpose()
Y = np.array([[0,1.1],[0.9,0.5],[3,-0.1]]).transpose()
n = X.shape[1]

# use a Gaussian kernel
k = lambda x, y: np.exp(-np.sum((x-y)**2)/(2.0*1.0**2))

dx = 0.05
stride = 8
x0 = np.arange(-1,4,dx)
x1 = np.arange(-1,4,dx)
X0,X1 = np.meshgrid(x0,x1,indexing='ij')
ss.plot_grid(X0,X1,ax=ax[0],rstride=stride,cstride=stride)
hx = ax[0].scatter(X[0,:],X[1,:],color='c',edgecolor='k')
hy = ax[0].scatter(Y[0,:],Y[1,:],color='r',edgecolor='k')
ax[0].set_aspect('equal')
ax[0].legend([hx,hy],['X','Y'])
ax[0].set_title('Matching problem')

# define a kernel matrix
K = np.empty((n,n))
for i in range(n):
    for j in range(n):
        K[i,j] = k(X[:,i],X[:,j])
Lambda = np.linalg.solve(K,(Y-X).transpose()).transpose()
# this defines a function v(x) = sum_i lambda_i k(x_i)
# let's use it to deform the grid for visualization
phiX0 = np.array(X0)
phiX1 = np.array(X1)
for i in range(len(x0)):
    for j in range(len(x1)):
        # this point
        p = np.array([X0[i,j],X1[i,j]])
        for k_ in range(n):
            phiX0[i,j] += Lambda[0,k_]*k(p,X[:,k_])
            phiX1[i,j] += Lambda[1,k_]*k(p,X[:,k_])
ss.plot_grid(phiX0,phiX1,ax=ax[1],rstride=stride,cstride=stride)
hy = ax[1].scatter(Y[0,:],Y[1,:],color='r',edgecolor='k')
ax[1].set_aspect('equal')
ax[1].set_title('The deformation')

ax[2].quiver(X0[::stride,::stride],X1[::stride,::stride],phiX0[::stride,::stride]-X0[::stride,::stride],
             phiX1[::stride,::stride]-X1[::stride,::stride],
             scale=stride)
ax[2].set_aspect('equal')
ax[2].set_title('The vector field')
```

    <IPython.core.display.Javascript object>

    Text(0.5,1,'The vector field')

```python
# now our 3D data
fname = os.path.join('hipAmygLandmarks', 'hippocampus_01_landmarks.txt')
X = ss.load_landmarks(fname)
fname = os.path.join('hipAmygLandmarks', 'hippocampus_01_surface.byu')
V0,F0 = ss.load_surface(fname)

# get a landmarks
fname = os.path.join('hipAmygLandmarks', 'hippocampus_02_landmarks.txt')
Y = ss.load_landmarks(fname)
fname = os.path.join('hipAmygLandmarks', 'hippocampus_02_surface.byu')
V1,F1 = ss.load_surface(fname)

n = X.shape[1]

# define a kernel matrix
k = lambda x, y: np.exp(-np.sum((x-y)**2)/(2.0*5.0**2))
K = np.empty((n,n))
for i in range(n):
    for j in range(n):
        K[i,j] = k(X[:,i],X[:,j])
Lambda = np.linalg.solve(K,(Y-X).transpose()).transpose()

# calculate the deformed vertices
phiV0 = np.array(V0)
for i in range(V0.shape[1]):
    # this point
    for k_ in range(n):
        phiV0[:,i] += Lambda[:,k_]*k(V0[:,i],X[:,k_])
# plot our results
f = plt.figure()
ax = f.add_subplot(1,2,1,projection='3d')
ax.plot_trisurf(V0[0,:],V0[1,:],V0[2,:],triangles=F0,color='c',alpha=0.5)
hx = ax.scatter(X[0,:],X[1,:],X[2,:],color='c',edgecolor='k')
ax.plot_trisurf(V1[0,:],V1[1,:],V1[2,:],triangles=F1,color='r',alpha=0.5)
hy = ax.scatter(Y[0,:],Y[1,:],Y[2,:],color='r',edgecolor='k')
ax.legend([hx,hy],['X','Y'])
ss.axis_equal(ax)

ax = f.add_subplot(1,2,2,projection='3d')
ax.plot_trisurf(V0[0,:],V0[1,:],V0[2,:],triangles=F0,color='c',alpha=0.1)
ax.plot_trisurf(phiV0[0,:],phiV0[1,:],phiV0[2,:],triangles=F0,color='b',alpha=0.5)
ax.plot_trisurf(V1[0,:],V1[1,:],V1[2,:],triangles=F1,color='r',alpha=0.5)
ss.axis_equal(ax)
```

    <IPython.core.display.Javascript object>


Solution to: [Day 7: Pearson Correlation Coefficient I](https://www.hackerrank.com/challenges/s10-pearson-correlation-coefficient/problem)

Table of Contents

- Table of Contents
- Notes
    - Covariance
    - Pearson Correlation Coefficient
- Solution
    - Imports
    - Input
    - Covariance
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Notes

## Covariance
This is a measure of how two random variables change together, or the strength of their correlation.

Consider two random variables, X and Y, each with n values (i.e., x1, x2, x3, ... xn  and y1, y2, y3, ... yn ).
The covariance of X and Y can be found using the following formula:

\begin{equation}
\large
cov(X, Y) = \frac
{1}{n}
\sum_{1}^{n} (x - \bar{x}) (y - \bar{y})
\end{equation}

## Pearson Correlation Coefficient
The Pearson correlation coefficient, ρxy, is given by:

\begin{equation}
\large
\text{p}(xy) = \frac
{cov(X, Y)}
{\sigma x * \sigma y}
\end{equation}

You may also see ρ(xy) written as r(xy).

# Solution

## Imports

```python
from typing import Tuple
```

## Input

```python
def get_input() -> Tuple[int, list, list]:
	"""Returns input for Pearson Correlation Coefficient I

	Returns:
		Tuple[int, list, list]: num items, X, Y
	"""
	num_items = int(input())
	x = [float(val) for val in input().split()]
	y = [float(val) for val in input().split()]
	return num_items, x, y
```

## Covariance

```python
def calc_mean(x: list) -> float:
	"""Returns mean of list

	Args:
		x (list): List to calculate mean

	Returns:
		float: Mean of x
	"""
	return sum(x) / len(x)
```

```python
def calc_sd(x: list) -> float:
	"""Returns standard deviation of list.

	Args:
		x (list): List to calculate sd

	Returns:
		float: Sd of list
	"""
	x_mean = calc_mean(x)

	sd_num = 0
	for item in x:
		sd_num += (item - x_mean) ** 2

	return (sd_num / len(x)) ** (1/2)
```

```python
def calc_cov(num_items: int, x: list, y: list) -> float:
	"""Returns covariance for x and y

	Args:
		num_items (int): length of x and y
		x (list): series 1
		y (list): series 2

	Returns:
		float: covariance between x and y
	"""
	assert len(x) == len(y)
	mean_x, mean_y = calc_mean(x), calc_mean(y)

	cov_total = 0
	for i in range(num_items):
		cov_total += (x[i] - mean_x) * (y[i] - mean_y)
	return cov_total / num_items
```

```python
def calc_pearson_coef(num_items: int, x: list, y: list) -> float:
	"""Returns pearson's coefficient between two lists."""
	cov = calc_cov(num_items, x, y)
	sd_x, sd_y = calc_sd(x), calc_sd(y)

	return cov / (sd_x * sd_y)
```

```python
def print_to_scale(num: int) -> None:
	"""Prints number to 3 decimal places.

	Args:
		num (int): Number to print
	"""
	print(f"{num :.3f}")
```

## Main

```python
def main():
	num_items, x, y = get_input()
	pearson_coef = calc_pearson_coef(num_items, x, y)
	print_to_scale(pearson_coef)
```

```python
if __name__ == "__main__":
	main()
```

    10
    10 9.8 8 7.8 7.7 7 6 5 4 2
    200 44 32 24 22 17 15 12 8 4
    0.612


# The Solution of Nonlinear Equations

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

One of the most frequently occurring problems in scientific work is to find the roots of equations of the form
\begin{equation}
    f(x) = 0,
\end{equation}
i.e., zeros of the function $f(x)$.

## The Existence of Root
The first step to solving an equation is to verify that a root exists.

### Theorem
   Let $f$ be a continuous function on $[a, b]$, satisfying $f(a)f(b) < 0$. Then $f$ has a root between $a$ and $b$, that is, there exists a number $x^{*}$ satisfying $a < x^{*} < b$ and $f(x^{*}) = 0$.

__Notes:__
* This theorem is just the simplest one for the existence of roots.

## The Bisection Method
Let us see one of the simplest root finding methods: the bisecion method.

### Bisection Method
**Algorithm:**
```python
    Given initial interval [a, b] such that f (a)f (b) < 0
    while (b − a)/2 > TOL:
        c = (a + b)/2
    if f (c) = 0:
        stop
    if f(a)f(c) < 0:
        b=c
    else:
        a=c
    The final interval [a,b] contains a root.
    The approximate root is (a + b)/2.
```

**Python implementation:**

```python
# environment setting, before any codes
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
plt.rcParams['axes.unicode_minus']=False
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from IPython.display import clear_output, display
```

### The Python Code
Here, we give the code of bisection method as follows:
```python
    def bisect(f, a, b, tol):
```
* $f$ is the function $f(x)$
* $a$ and $b$ are for the interval $[a, b]$
* $tol$ is the tolerence error for the numerical solution.

```python
def bisect(f, a, b, tol):
    if (np.sign(f(a)*f(b)) >= 0):      # wrong input
        print ('$f(a)f(b) <0$ not satisfied')
        return
    fa = f(a)
    fb = f(b)
    while ((b - a)/2) > tol:
        c = (a + b)/2;
        fc = f(c)
        if fc == 0:                 # c is a solution, done
            return c
        if (np.sign(fc*fa) < 0):       # new interval [a, c]
            b = c;
            fb = fc
        else:                       # new interval [c, b]
            a = c
            fa = fc
    return (a + b)/2                #new midpoint is best estimate
```

### Demostration
**Demonstration version:**

```python
def bisect_iters(f, a, b, iters):
    fa = f(a)
    fb = f(b)

    x = np.zeros(iters+1)
    for i in range(iters):
        c = (a + b)/2
        x[i] = c
        fc = f(c)
        if fc == 0:                 # c is a solution, done
            return c
        if (np.sign(fc*fa) < 0):       # new interval [a, c]
            b = c;
            fb = fc
        else:                       # new interval [c, b]
            a = c
            fa = fc
    x[iters] = (a + b)/2
    return x              #new midpoint is best estimate
```

**Exmaple**
    Use Bisection Method to find the root of $f(x) = e^{-x} - \sin\left(\frac{\pi x}{2}\right)$ on $[0,1]$.

```python
def fun(x):
    return np.exp(-x) - np.sin(np.pi/2*x)
```

```python
def Iteration_demo(f, a, b, iters):
    x = np.linspace(a, b, 1000)
    plt.plot(x, f(x), 'k', linewidth=3)
    plt.plot(x, 0*x, 'k-.', linewidth=3)
    x = bisect_iters(f, a, b, iters)
    x_ref = bisect(f, a, b, 1.e-14)

    #print (x)
    plt.xlim(a,b)
    plt.ylim(min(f(a),f(b)),max(f(a),f(b)))
    #plt.ylim(1,2)
    for i in range(iters+1):
        plt.plot(x, f(x), 'r*', markersize=12, linewidth=3)

    print ('Reference Solution: ', x_ref)
    print ('no.    solution    error bound    error')
    for i in range(iters+1):
        print ("%3d  %12.10f   %7.2e    %7.2e" % \
                   (i, x[i], (b-a)/2**(i+1), np.abs(x_ref - x[i])))

a = 0
b = 1
Iteration_demo(fun, a, b, 10)
```

```python
w = interactive(Iteration_demo, f=fixed(fun), a=fixed(0), b=fixed(1), iters=widgets.IntSlider(min=0,max=50,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=50), Output()), _dom_classes=('widget-intera…

**Exmaple**
    Use Bisection Method to find the root of $f(x) = x^3 + x - 1$ on $[0,1]$.

```python
def fun2(x):
    return x**3 + x - 1

w = interactive(Iteration_demo, f=fixed(fun2), a=fixed(0), b=fixed(1), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Exmaple**
    Use Bisection Method to find the root of __linear equation__ $f(x) = x - 1/3$ on $[0,1]$.  What do you think of the performance?

```python
def fun3(x):
    return x - 0.50000000001

w = interactive(Iteration_demo, f=fixed(fun3), a=fixed(0), b=fixed(1), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

## Fixed-Point Iteration (FPI)

### FPI Algorithm
First, one can rewrite the equation $f(x) = 0$ as a fixed point problem $g(x) = x$, Fixed-Point Iteration proceeds by starting with an initial guess $x_0$ and iterating the function $g$.
\begin{align}
    x_0 & = \text{initial guess} \\
    x_{n+1} & = g(x_n) \,\,\text{for}\,\,n = 0, 1,\ldots,
\end{align}

### The Python Code
The code of FPI is very easy, in each iteration, just do a function evaluation.

```python
def fixed_iters(g, x0, iters):
    x = np.zeros(iters+1)
    x[0] = x0
    for i in range(iters):
        x[i+1] = g(x[i])
    return x

def g(x):
    return np.cos(x)

fixed_iters(g, 1., 20)
```

    array([1.        , 0.54030231, 0.85755322, 0.65428979, 0.79348036,
           0.70136877, 0.76395968, 0.72210243, 0.75041776, 0.73140404,
           0.74423735, 0.73560474, 0.74142509, 0.73750689, 0.74014734,
           0.7383692 , 0.7395672 , 0.73876032, 0.73930389, 0.73893776,
           0.7391844 ])

### Demostration
**Demonstration version:**

```python
def Iteration_demo2(f, a, b, g, x0, iters):
    x = np.linspace(a, b, 1000)
    plt.plot(x, f(x), 'k', linewidth=3)
    plt.plot(x, 0*x, 'k-.', linewidth=3)
    x = fixed_iters(g, x0, iters)
    x_ref = bisect(f, a+1.e-8, b, 1.e-14)

    #print (x)
    plt.xlim(a,b)
    plt.ylim(min(f(a),f(b)),max(f(a),f(b)))
    for i in range(iters+1):
        plt.plot(x, f(x), 'r*', markersize=12, linewidth=3)

    print ('Reference Solution: ', x_ref)
    print ('no.    solution    |x_{n+1} - x_{n}|    error')
    for i in range(iters+1):
        if (i < iters):
            print ("%3d  %14.12f      %7.2e       %7.2e" % \
                   (i, x[i], np.abs(x[i+1]-x[i]), np.abs(x_ref - x[i])))
        else:
            print ("%3d  %14.12f      %7.2e       %7.2e" % \
                   (i, x[i], np.abs(g(x[i])-x[i]), np.abs(x_ref - x[i])))
```

**Example** Use Fixed Point Iteration to find a root of $f(x) = x^3 + 4x^2 - 10$ on $[1, 2]$ with $g_1(x) = \sqrt{\frac{10}{x} - 4x}$ and $g_2(x) = \sqrt{\frac{10}{x+4}}$, the initial guess $x_0 = 1.5$.

```python
def f(x):
    return x**3 + 4*x**2 - 10

a = 1; b = 2;

def g1(x):
    return np.sqrt(10/x - 4*x)

def g2(x):
    return np.sqrt(10/(x+4))

x0 = 1.5

w = interactive(Iteration_demo2, f=fixed(f), a=fixed(a), b=fixed(b), g=fixed(g2), x0 = fixed(x0), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

__Better than bisection?__
* Solving the above example by using bisection method.

```python
w = interactive(Iteration_demo, f=fixed(f), a=fixed(1), b=fixed(2), iters=widgets.IntSlider(min=0,max=50,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=50), Output()), _dom_classes=('widget-intera…

**Example** Find the fixed points of $g(x) = 2.8x - x^2$

```python
def f(x):
    return 1.8*x - x**2

a = 0; b = 2;

def g(x):
    return 2.8*x - x**2

x0 = -1

w = interactive(Iteration_demo2, f=fixed(f), a=fixed(a), b=fixed(b), g=fixed(g), x0 = fixed(x0), iters=widgets.IntSlider(min=0,max=40,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=40), Output()), _dom_classes=('widget-intera…

**Example** Use Fixed Point Iteration to find the positive root of $f(x) = x^2 - 3$ with $g_1(x) = x - \frac{1}{4}(x^2-3)$ and $g_2(x) = \frac{1}{2}\left(x + \frac{3}{x}\right)$.

```python
def f(x):
    return x**2 - 3

a = 0; b = 2

def g1(x):
    return x - 0.25*(x**2-3)

def g2(x):
    return 0.5*(x + 3/x)

x0 = 2

w = interactive(Iteration_demo2, f=fixed(f), a=fixed(a), b=fixed(b), g=fixed(g1), x0 = fixed(x0), iters=widgets.IntSlider(min=0,max=40,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=40), Output()), _dom_classes=('widget-intera…

## Newton's Method
### Newton's Method
Newton's method is a particular kind o af fixed point iteration, with $g(x) = x - \frac{f(x)}{f'(x)}$.
\begin{align}
    x_0 & = \text{initial guess} \\
    x_{n+1} & = x_n - \frac{f(x_n)}{f'(x_{n})} \,\,\text{for}\,\,n = 0, 1,\ldots,
\end{align}

### The Python Code
Similar as the code of FPI, in each iteration, the Newton's method just need do a function evaluation and compute its derivative.

```python
def Newton_iters(f, f_der, x0, iters):
    x = np.zeros(iters+1)
    x[0] = x0
    for i in range(iters):
        x[i+1] = x[i] - f(x[i])/f_der(x[i])
    return x

def f(x):
    return x**2 - 3

def f_der(x):
    return 2*x

Newton_iters(f, f_der, 1., 20)
```

    array([1.        , 2.        , 1.75      , 1.73214286, 1.73205081,
           1.73205081, 1.73205081, 1.73205081, 1.73205081, 1.73205081,
           1.73205081, 1.73205081, 1.73205081, 1.73205081, 1.73205081,
           1.73205081, 1.73205081, 1.73205081, 1.73205081, 1.73205081,
           1.73205081])

**Example** Find the positive root of $f(x) = x^2 - a$ by Newton's Methods

The iterations is
\begin{equation}
    x_{n+1} = \left(x_n + \frac{a}{x_n}\right)/2
\end{equation}

```python
def Iteration_demo3(f, f_der, a, b, x0, x_star, iters):
    x = np.linspace(a, b, 1000)
    plt.plot(x, f(x), 'k', linewidth=3)
    plt.plot(x, 0*x, 'k-.', linewidth=3)
    x = Newton_iters(f, f_der, x0, iters)
    #x_ref = bisect(f, a+1.e-8, b, 1.e-14)

    #print (x)
    plt.xlim(a,b)
    #plt.ylim(min(f(a),f(b)),max(f(a),f(b)))
    for i in range(iters+1):
        plt.plot(x, f(x), 'r*', markersize=12, linewidth=3)

    print ('Reference Solution: ', x_star)
    print ('no.    solution    |x_{n} - x_{*}|    e_{n}/e_{n-1}^2   e_{n}/e_{n-1}')
    for i in range(iters+1):
        if (i < 1):
            print ("%3d  %14.12f      %7.2e" % \
                   (i, x[i], np.abs(x_star - x[i])))
        else:
            print ("%3d  %14.12f      %7.2e           %6.4f          %6.4f" % \
                   (i, x[i], np.abs(x_star - x[i]), np.abs(x_star - x[i])/(x_star - x[i-1])**2, \
                   np.abs(x_star - x[i])/np.abs(x_star - x[i-1])))
```

```python
def f(x):
    return x**2 - 2

def f_der(x):
    return 2*x

x0 = -3
x_star = np.sqrt(2)
w = interactive(Iteration_demo3, f=fixed(f), f_der=fixed(f_der), a=fixed(a), b=fixed(b), x0 = fixed(x0), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Example** Find the root of $f(x) = x^3 + 10x - 20$ around $x_0 = 1.5$ by Newton's Methods

The iterations is
\begin{equation}
    x_{n+1} = x_n - \frac{x^3 + 10x - 20}{3x^2 + 10}
\end{equation}

```python
def f(x):
    return x**3 + 10*x - 20

def f_der(x):
    return 3*x**2 + 10

x0 = 100
a = 1
b = 2
x_star = bisect(f, a, b, 1.e-14)
w = interactive(Iteration_demo3, f=fixed(f), f_der=fixed(f_der), a=fixed(a), b=fixed(b), x0 = fixed(x0), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=10,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=10), Output()), _dom_classes=('widget-intera…

**Example** Find the root of $f(x) = x^3 - 3x + 2$ around $x_0 = 1.1$ by Newton's Methods

The iterations is
\begin{equation}
    x_{n+1} = x_n - \frac{x^3 - 3x + 2}{3x^2 - 3}
\end{equation}

```python
def f(x):
    return x**3 - 3*x +2

def f_der(x):
    return (3*x**2 - 3)/1.

x0 = -1.5
a = -3.
b = 3.
x_star = -2.
w = interactive(Iteration_demo3, f=fixed(f), f_der=fixed(f_der), a=fixed(a), b=fixed(b), x0 = fixed(x0), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=30,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=30), Output()), _dom_classes=('widget-intera…

**Modified Newton Method**
Find the root of $f(x) = x^3 - 3x + 2$ around $x_0 = 1.1$ by Modified Newton Methods

The iterations is
\begin{equation}
    x_{n+1} = x_n - \frac{x^3 - 3x + 2}{x^2 - 1}
\end{equation}

```python
def f(x):
    return x**3 - 3*x +2

def f_der(x):
    return (3*x**2 - 3)/2.

x0 = 1.1
a = -3.
b = 3.
x_star = 1.
w = interactive(Iteration_demo3, f=fixed(f), f_der=fixed(f_der), a=fixed(a), b=fixed(b), x0 = fixed(x0), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=30,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=30), Output()), _dom_classes=('widget-intera…

### Secant Method
One drawback of Newton's method is that it needs the function $f(x)$ and its derivative function $f'(x)$.  What if you do not have the derivative function $f'(x)$?  One way to avoid this issue is to use the so-called secant method.

__Secant Method:__
\begin{align}
    x_0, x_1 & = \text{initial guesses} \\
    x_{n+1} & = x_n - \frac{f(x_n)(x_n - x_{n-1}}{f(x_{n})-f(x_{n-1})} \,\,\text{for}\,\,n = 1, 2,\ldots,
\end{align}

__The Python Code__

```python
def secant_iters(f, x0, x1, iters):
    x = np.zeros(iters+1)
    x[0] = x0
    if (iters >= 1):
        x[1] = x1
        for i in range(1,iters):
            x[i+1] = x[i] - f(x[i])*(x[i]-x[i-1])/(f(x[i]) - f(x[i-1]))
    return x

def f(x):
    return x**2 - 3

secant_iters(f, 1., 2., 10)
```

    array([1.        , 2.        , 1.66666667, 1.72727273, 1.73214286,
           1.73205068, 1.73205081, 1.73205081, 1.73205081, 1.73205081,
           1.73205081])

```python
def Iteration_demo4(f, a, b, x0, x1, x_star, iters):
    x = np.linspace(a, b, 1000)
    plt.plot(x, f(x), 'k', linewidth=3)
    plt.plot(x, 0*x, 'k-.', linewidth=3)
    x = secant_iters(f, x0, x1, iters)
    #x_ref = bisect(f, a+1.e-8, b, 1.e-14)

    #print (x)
    plt.xlim(a,b)
    #plt.ylim(min(f(a),f(b)),max(f(a),f(b)))
    for i in range(iters+1):
        plt.plot(x, f(x), 'r*', markersize=12, linewidth=3)

    print ('Reference Solution: ', x_star)
    print ('no.    solution    |x_{n} - x^{*}|    e_{n}/e_{n-1}^1.618   e_{n}/e_{n-1}')
    for i in range(iters+1):
        if (i < 1):
            print ("%3d  %14.12f      %7.2e" % \
                   (i, x[i], np.abs(x_star - x[i])))
        else:
            print ("%3d  %14.12f      %7.2e           %6.4f          %6.4f" % \
                   (i, x[i], np.abs(x_star - x[i]), np.abs(x_star - x[i])/np.abs(x_star - x[i-1])**1.618, \
                   np.abs(x_star - x[i])/np.abs(x_star - x[i-1])))
```

```python
def f(x):
    return x**3 - 3*x +2

x0 = -1.5
x1 = -2.5
a = -3.
b = 3.
x_star = -2

w = interactive(Iteration_demo4, f=fixed(f), a=fixed(a), b=fixed(b), x0 = fixed(x0), x1 = fixed(x1), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

## Assignment
### Assignment 1:
For equation $x^2−3x+2−e^x =0$
1) Fixed Point Iteration:
    \begin{equation*}
        x_{n+1} = g(x_n)
    \end{equation*}
     with $g(x) = (x^2 + 2^x + 2 - e^x)/5$

2) Newton's Methods:
\begin{equation*}
x_{n+1} = x_n - \frac{x^2−3x+2−e^x}{2x -3 - e^x}
\end{equation*}

```python
def f(x):
    return x**2 - 3*x +2 - np.exp(x)

def g(x):
    return x + 0.2*(x**2 - 3*x +2 - np.exp(x))

a = 0; b = 1;
x0 = 0.5

w = interactive(Iteration_demo2, f=fixed(f), a=fixed(a), b=fixed(b), g=fixed(g), x0 = fixed(x0), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
def f(x):
    return x**2 - 3*x +2 - np.exp(x)

def f_der(x):
    return 2*x - 3 - np.exp(x)

x0 = 0.5
a = 0.
b = 1.
x_star = bisect(f, a, b, 1.e-14)
w = interactive(Iteration_demo3, f=fixed(f), f_der=fixed(f_der), a=fixed(a), b=fixed(b), x0 = fixed(x0), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=30,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=30), Output()), _dom_classes=('widget-intera…

### Assignment 2:
Design a fixed point iteration method to solve equation $x^3 +2x^2 +10x −20 = 0$

```python
def f(x):
    return x**3 + 2*x**2 + 10*x - 20

def g(x):
    return x - 0.02*(x**3 + 2*x**2 + 10*x -20)

a = 1; b = 2;
x0 = 1.5

w = interactive(Iteration_demo2, f=fixed(f), a=fixed(a), b=fixed(b), g=fixed(g), x0 = fixed(x0), iters=widgets.IntSlider(min=0,max=30,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=30), Output()), _dom_classes=('widget-intera…

```python
def f(x):
    return x**3 + 2*x**2 + 10*x - 20

def f_der(x):
    return 3*x**2 + 4*x + 10

x0 = 1.5
a = 1.
b = 2.
x_star = bisect(f, a, b, 1.e-14)
w = interactive(Iteration_demo3, f=fixed(f), f_der=fixed(f_der), a=fixed(a), b=fixed(b), x0 = fixed(x0), \
                x_star = fixed(x_star), iters=widgets.IntSlider(min=0,max=30,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=30), Output()), _dom_classes=('widget-intera…

### Assignment 3
Consider Fixed-Point Iteration applied to $g(x) = 1 − 5x + \frac{15}{2}x^2 − \frac{5}{2}x^3$ . (a) Show that
$1 − \sqrt{3/5}$, $1$, and $1 + \sqrt{3/5}$ are fixed points. (b) Show that none of the three fixed points is
locally convergent.

Futher, find initial guesses for which FPI (c) cycles endlessly through numbers in the interval $(0,1)$, (d) the same as (c), but the interval is $(1, 2)$ (e) diverges to infinity. Cases (c) and (d) are examples of chaotic dynamics. In all three cases, FPI is unsuccessful.

```python
def f(x):
    return 1 - 6*x + 7.5*x**2 - 2.5*x**3

a = 0; b = 2

def g(x):
    return 1 - 5*x + 7.5*x**2 - 2.5*x**3

x0 = 0.5
x_star = 1 - np.sqrt(3./5)
w = interactive(Iteration_demo2, f=fixed(f), a=fixed(a), b=fixed(b), g=fixed(g), x0 = fixed(x0), iters=widgets.IntSlider(min=0,max=80,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=80), Output()), _dom_classes=('widget-intera…

```python
x = fixed_iters(g, x0, 120)
plt.plot(np.arange(np.size(x)),x)
```

```python

```


# Reinforcment Learning: Defining Concepts Mathematically

This blog post is intended to give the reader a baseline understanding of the concepts underlying modern reinforcement learning algorithms. In particular, this blog post will reintroduce the concepts touched upon in our **first blog post**, although this time we will use math to build up a repertoire of definitions helping us to understand more advanced concepts and algorithms. Thus our first post is not mandatory to understand reinforcement learning, yet it is very helpful for newcomers to build up a solid intuition.

# The base Concepts

To start off, we will once again begin by examining the agent-environment duality. A reinforcement learning problem is defined by both the **environment** and the **agent**, whose actions effect said environment. We can now define the interaction of both: given a **state** $s \in \mathcal{S}$ it is the agents task to choose an **action** $a \in \mathcal{A}$. The environment will then give a **reward** $r \in  \mathcal{R}$ to the agent and determine the next state $s' \in \mathcal{S}$ based on the state $s$ and the action $a$ chosen by the agent. While doing so $\mathcal{S}$ constitutes the set of all possible states, $\mathcal{A}$ the set of all possible actions and $\mathcal{R}$ the set of all possible rewards. This relationship can also be easily illustrated.

If we now define $s_1$ as the starting state and $r_t, s_t, a_t$ as the state, action or reward at timestep $t = 1, 2, \ldots, T$ with the condition that $s_T$ is a terminal state, we can show the entire interaction sequence between agent and environment, also called one **episode** (sometimes called "trial" or "trajectory"), as follows:

\begin{equation}
    \tau := (s_1, a_1, r_2, s_2, a_2, r_3, \ldots s_{T-1}, a_{T-1}, r_{T}, s_{T})
\end{equation}

Note that $s_t, a_t, r_t$ can also be denoted by capital letters $S_t, A_t, R_t$.

## The Environment

The environment can be modeled using a transition function $P(s', r | s, a)$ which gives us the probability to go to $s'$ as the next state with reward $r$ under the condition that the current state is $s$ and the agent chose action $a$. From this function we can also derive the state-transition function $P(s' | s, a)$ as well as the reward function $R(r | s, a)$.

\begin{align}
    P(s' | s, a) &:= P_{s, s'}^a = \sum_{r \in \mathcal{R}} P(s', r | s, a) \\
    R(r | s, a) &:=  \mathbb{E} [ r | s, a ] = \sum_{r \in \mathcal{R}} r \sum_{s' \in \mathcal{S}} P(s', r | s, a) \\
\end{align}

$R$ is the expected value for the reward $r$ given $s$ and $a$. A single **transition** in the environment can thus be specified by a tuple $(s, a, r, s')$.

### Markov Decision Processes

Given an episode $\tau$ in combination with a transition function $P(s' | s, a)$ we can observe:

\begin{equation}
        s_{t+1} \sim P(s_{t+1} | s_t, a_t) = P(s_{t+1} | s_t, a_t, s_{t-1}, a_{t-1}) = P(s_{t+1} | s_t, a_t, s_{t-1}, a_{t-1},...,s_1, a_1)
\end{equation}

Therefore $s_{t+1}$ is only dependent on timestep $t$ and its associated state $s_t$ and action $a_t$. The future state is independent from the past and only dependent on the present. This property is also called the **Markov Property** and enables us to view man reinforcement learning problems as **Markov Decision Processes**. While not important for simple algorithms, more advanced approaches utilise this view and its mathematical implications to further refine the learning process.

### Model-based vs Model-free

Depending on how much the agent knows about the **transition function** $P$ and the **reward function** $R$ we can already classify reinforcement learning algorithms as:

- Model-based: The agent has knowledge of the environment. This can be further subdivided:
    - Model is given: The agent knows $P$ and $R$ perfectly. AlphaGo can be seen as such an example as the program is perfectly aware of the Go rules.
    - Model is learned: While $P$ and $R$ are not known at the start, the agent learns these functions during the learning process.
- Model-free: The agent does not have and does not attempt to learn $P$ and $R$. Agent57 (the successor of DQN) developed by Google DeepMind falls into this category as the agent plays 57 different Atari games, showing above human performance and learning only using high-dimensional sensory input in the form of the screen pixels.

## The Agent

As stated above, the agents goal is to choose an action $a$ given a state $s$. We can formalize this using a function $\pi$ also known as **policy**. The policy can take one of two forms:

- deterministic: $a = \pi(s)$ (also written as $\mu (s)$)
- stochastic: $\pi(a | s)$ is the chance that $a$ is chosen under the condition $s$. In this case the derived action is denoted by $a \sim \pi(\cdot | s)$.

The goal of our reinforcement learning algorithms will always be to develop a policy $\pi$ which is as close as possible to the optimal policy $\pi_*$. One of the main differences in the various algorithms is the method used to determine this policy $\pi$.

To gain a better understanding of these methods it can be helpful to imagine how a human would go about formulating a strategy in such circumstances. This is especially intuitive when looking at games as reinforcement learning problems where the programm effectively replaces a human player.

### Value Function

One approach a human might use is to assign a value to each state based on how "good" that state is for generating future rewards. The policy could then simply consist of taking the action which leads to the next state with the highest value weighted by probabilities. Although informal, this describes the concept of the **state-value function** $V(s)$ (also known as value function).

Yet before we can formally describe $V(s)$ we need to define the future reward or **return** $G_t$, which measures the discounted rewards after timestep $t$ in an episode:

\begin{align}
    G_t := r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \ldots = \sum_{k = 0}^\infty \gamma^k r_{t+1+k}
\end{align}

The form of $G_t$ above is called the **infinite horizon discounted-return**. If we instead summed up only to a constant $T$ instead of $\infty$ we would get the **finite horizon discounted-return**.

The factor $\gamma \in [0, 1]$ is a hyperparameter discounting rewards baseed on how far in the future they occur. Doing this has multiple practical reasons as we may prefer immediate benefits over drawn out ones (such as winning a game quickly). Further, this also provides mathematical advantages when encountering loops in the transition sequence and further allows:

\begin{align}
    G_t &= r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \ldots \\
    &= r_{t+1} + \gamma (r_{t+2} + \gamma r_{t+3} + \ldots) \\
    &= r_{t+1} + \gamma G_{t+1}
\end{align}

Using the return, we can now define an equation for $V(s)$:

\begin{align}
    V(s) &:= \mathbb{E}[G_t | s_t = s] \\
    V_\pi(s) &= \sum_{a} \pi(a | s) \sum_{s', r} P(s', r | s, a) (r + \gamma V_\pi(s'))
\end{align}

Note the dependency of $V(s)$ on an episode or a policy. The second equation here is also called a **Bellman equation**. Therefore a policy is required to determine the state-value function and we can define $V^*(s)$ as the function generated by the opotimal policy $\pi_*$. Once a function $V(s)$ is given we can derive a greedy policy $\pi$ accordingly:

\begin{align}
    \pi(s) = \text{arg} \max_{a} \sum_{s', r} P(s', r | s, a) (r + \gamma V(s'))
\end{align}

If we can find an algorithm in which $V_\pi(s)$, generated by the current policy $\pi$, converges on $V_*(s)$ we can thus derive the optimal policy $\pi_*$. This approach is realized in the **policy-iteration** algorithm which alternates between generating a state-value function from a policy and generating a policy from the state-value function.

### Action-Value Function

Instead of only evaluating states, it is also possible to evaluate state-actions pairs, resulting in the **action-value function** $Q(s, a)$. We can define $Q(s, a)$ similarly to $V(s)$ and logically connect both functions for a given policy:

\begin{align}
    Q(s, a) &:= \mathbb{E}[G_t | s_t = s, a_t = a] \\
    V_\pi(s) &= \sum_{a} \pi(a | s) Q_\pi(s, a) \\
    Q_\pi(s, a) &= \sum_{s', r} P(s', r | s, a) (r + \gamma V_\pi(s'))
\end{align}

We can also define the **Bellman equation**:

\begin{align}
    Q_\pi(s, a) &= \sum_{s', r} P(s', r | s, a) (r + \gamma \sum_{a'} \pi(a' | s) Q_\pi(s', a')) \\
\end{align}

The action-value function can also be used to greedily generate a policy:

\begin{align}
    \pi(s) = \text{arg} \max_{a} Q(s, a)
\end{align}

## Approaches to Reinforcement Learning

In theory, every problem we describe using this framework could be seen as a simple planning problem, only requiring the maximum solution to the Bellman equation to derive an optimal policy. In practice, however, the number of possible states and actions as well as the limited knowledge of the environment makes this unfeasible, forcing us to find other ways of converging on the optimal policy. Therefore algorithms tend to optimize the state-value function, the action-value function or the policy directly. If the policy is directly optimized, it tends to consist of another function with a given set of parameters that can be optimized.

Since we have defined the most important terms, we can now delve deeper into specific algorithms and approaches. This will further our understanding of reinforcement learning and detail the implementation of

## References

- [Lilian Wang: A (Long Peek into Reinforcment Learning)](https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html)

- [OpenAI: Spinning Up, Part 1: Key Concepts in RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#optional-formalism)


## Logisitic Population Differential Equations

In this notebook we will a simple population equation [1]

```python
## Library
import numpy as np
import math
import matplotlib.pyplot as plt
```

## Logistic Population Equation
The logistic growth differential equation is a non-linear first order differential equation
\begin{equation}
\frac{d p}{dt}=\alpha_{Birth} p-\alpha_{Death} p^2
\end{equation}
for $a\leq t \leq b$ years, where $\alpha_{Birth}$ is the birth rate and $\alpha_{Death}$ is the death rate, with the initial condition
$$ p(a)=A. $$
As the equation is non-linear we must numverically approximate the solution.
In this simulation we set the parameters of the different equations:
*  as $\alpha_{Birth}=0.3$ and $\alpha_{Death}=0.01$, with the inital condition $p(0)=3$, over the time period $0\leq t \leq 10$.

```python
alpha_Birth=4
alpha_Death=0.2
```

## Discrete Domain
To numerically approximate the solution we discretise the continuous domain using a step size, $h=1$,
which gives
\begin{equation}
t_i=0+i h,
\end{equation}
for $i=0,...10$.
The figure below illustrate the discrete domain.

```python
h=0.1
t_end=30
t=np.arange(0,t_end+h/2,h)
fig = plt.figure(1,figsize=(5,4))
ax = fig.add_subplot(1,1,1)
ax.plot(t,0*t,'o:',color='k')
ax.set_xlabel('Time (yrs)')
plt.title('Discrete Time Domain')
plt.show()
```

```python
N=int(t_end/h) # Number of Steps
P=np.zeros(N+1) # Numerical Solution P
P[0]=3

```

We numerically approximate the differential equation using Eulers method to give the difference equation
\begin{equation}
\frac{P_{i+1}-P_i}{h}=\alpha_{Birth}P_i-\alpha_{Death}P_i^2
\end{equation}
re-arranging
\begin{equation}
\color{red}{P_{i+1}}=P_i+h\big(\alpha_{Birth}P_i-\alpha_{Death}P_i^2\big)
\end{equation}

where $h$ is the stepsize and where $P_i$ is the numerical approximation of $p(t_i)$ at time $t_i$, for $i=0,...,10$

```python
for i in range (0,N):
    P[i+1]=P[i]+h*(alpha_Birth*P[i]-alpha_Death*P[i]*P[i])

```

## Results
The plot bellow shows the Numerical solution $P_i$ of the Logistic Population as a function of time $t$.

```python
## Plotting Figure
fig = plt.figure(1,figsize=(4,4))

ax = fig.add_subplot(1,1,1)

ax.plot(t,P,'o:',color='red',label='Population')
ax.legend()

ax.set_xlabel('Time (yrs)')
ax.set_ylabel('Population (billions)')
#ax.legend(loc='best')
ax.set_title('Numerical Solution, P')

plt.tight_layout()
plt.subplots_adjust(top=0.85)
plt.show()

```

# References

[1] Stover, Christopher and Weisstein, Eric W. "Population Growth." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/PopulationGrowth.html

```python

```

.ipynb_checkpoints/05_Logistic Growth with Seasonal-checkpoint.ipynb