<p align="center">

</p>

## Data Analytics

### Parametric Distributions  in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Parametric Distributions

Here's a demonstration of making and general use of parametric distributions in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics at the Cockrell School of Engineering at the University of Texas at Austin.

#### Parametric Distributions

We will cover the following distributions:

* Uniform
* Triangular
* Gaussian
* Log Normal

We will demonstrate:

* distribution parameters
* forward and inverse operators
* summary statistics

I have a lecture on these parametric distributions available on [YouTube](https://www.youtube.com/watch?v=U7fGsqCLPHU&t=1687s).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - unconv_MV_v4.csv at https://git.io/fhHLT.

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # for randon numbers
import seaborn as sns                     # for matrix scatter plots
from scipy import linalg                  # for linear regression
from sklearn import preprocessing
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

### Uniform Distribution

Let's start with the most simple distribution.

* by default a random number is uniform distributed

* this ensures that enough random samples (Monte Carlo simulations) will reproduce the distribution

\begin{equation}
x_{\alpha}^{s} = F^{-1}_x(p_{\alpha}), \quad X^{s} \sim F_X
\end{equation}

#### Random Samples

Let's demonstrate the use of the command:

```python
uniform.rvs(size=n, loc = low, scale = interval, random_state = seed)
```

Where:

* size is the number of samples

* loc is the minimum value

* scale is the range, maximum value minus the minimum value

* random_state is the random number seed

We will observe the convergence of the samples to a uniform distribution as the number of samples becomes large.

We will make a compact set of code by looping over all the cases of number of samples

* we store the number of samples cases in the list called ns

* we store the samples as a list of lists, called X_uniform

```python
from scipy.stats import uniform

low = 0.05; interval = 0.20; ns = [1e1,1e2,1e3,1e4,1e5,1e6]; X_uniform = []

index = 0
for n in ns:
    X_uniform.append(uniform.rvs(size=int(ns[index]), loc = low, scale = interval).tolist())
    plt.subplot(2,3,index+1)
    GSLIB.hist_st(X_uniform[index],loc,loc+interval,log=False,cumul = False,bins=20,weights = None,xlabel='Values',title='Distribution, N = ' + str(int(ns[index])))
    index = index + 1

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.3, top=1.6, wspace=0.2, hspace=0.3)
```

We can observe that by drawing more Monte Carlo simulations, we more closely approximate the original uniform parametric distribution.

#### Forward Distribution

Let's demonstrate the forward operator. We can take any value and calculate the associated:

* density (probability density function)
* cumulative probability

The transform for the probability density function is:

\begin{equation}
p = f_x(x)
\end{equation}

where $f_x$ is the PDF and $p$ is the density for value, $x$.

and for the cumulative distribution function is:

\begin{equation}
P = F_x(x)
\end{equation}

where $F_x$ is the CDF and $P$ is the cumulative probability for value, $x$.

```python
x_values = np.linspace(0.0,0.3,100)
p_values = uniform.pdf(x_values, loc = low, scale = interval)
P_values = uniform.cdf(x_values, loc = low, scale = interval)

plt.subplot(1,2,1)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3, label='uniform PDF'); plt.title('Uniform PDF'); plt.xlabel('Values'); plt.ylabel('Density')

plt.subplot(1,2,2)
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.3, label='uniform CDF'); plt.title('Uniform CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.8, top=0.8, wspace=0.2, hspace=0.3)
```

#### Inverse Distribution

Let's know demonstrate the reverse operator for the uniform distribution:

\begin{equation}
X = F^{-1}_X(P)
\end{equation}

```python
p_values = np.linspace(0.01,0.99,100)
x_values = uniform.ppf(p_values, loc = low, scale = interval)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3, label='uniform pdf')
```

#### Summary Statistics

We also have a couple of convience member functions to return the statistics from the parametric distribution:

* mean
* median
* mode
* variance
* standard deviation

Let's demonstrate a few of these methods.

```python
uniform.stats(loc = low, scale = interval, moments = 'mvsk')
```

returns a tuple with the mean, variance, skew and kurtosis (centered 1st, 2nd, 3rd and 4th moments)

```python
print('Stats: mean, variance, skew and kurtosis = ' + str(uniform.stats(loc = low, scale = interval, moments = 'mvsk')))
```

    Stats: mean, variance, skew and kurtosis = (array(0.15), array(0.00333333), array(0.), array(-1.2))

We can confirm this by calculating the centered variance (regular variance) with this member function:

```python
uniform.var(loc = low, scale = interval)
```

```python
print('The variance is ' + str(round(uniform.var(loc = low, scale = interval),4)) + '.')
```

    The variance is 0.0033.

We can also directly calculate the:

* standard deviation - std
* mean - mean
* median - median

We can also calculate order of a non-centered moment. The moment method allows us to calculate an non-centered moment of any order. Try this out.

```python
m_order = 4
print('The ' + str(m_order) + 'th order non-centered moment is ' + str(uniform.moment(n = m_order, loc = low, scale = interval)))
```

    The 4th order non-centered moment is 0.0009762500000000001

#### Symmetric Interval

We can also get the symmetric interval (e.g. prediction or confidence intervals) for any alpha level.

* Note the program mislabels the value as alpha, it is actually the significance level (1 - alpha)

```python
level = 0.95
print('The interval at alpha level ' + str(round(1-level,3)) + ' is ' + str(uniform.interval(alpha = alpha,loc = low,scale = interval)))
```

    The interval at alpha level 0.05 is (0.05500000000000001, 0.245)

#### Triangular Distribution

The great thing about parametric distributions is that the above member functions are the same!

* we can plug and play other parametric distributions and repeat the above.

This time we will make it much more compact!

* we will import the triangular distribution as my_dist and call the same functions as before
* we need a new parameter, the distribution mode (c parameter)

```python
from scipy.stats import triang as my_dist                                    # import traingular dist as my_dist
dist_type = 'Triangular'                                                     # give the name of the distribution for labels
low = 0.05; mode = 0.20; c = 0.10                                            # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, loc = low, c = mode, scale = interval)      # calculate density for each x value
P_values = my_dist.cdf(x_values, loc = low, c = mode, scale = interval)      # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' PDF'); plt.xlabel('Values'); plt.ylabel('Density')

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

p_values = np.linspace(0.00001,0.99999,100)                                 # get an array of p-values
x_values = my_dist.ppf(p_values, loc = low, c = mode, scale = interval)      # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3, label='uniform pdf')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.8, top=0.8, wspace=0.2, hspace=0.3); plt.title('Sampling Inverse ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

print('The mean is ' + str(round(uniform.mean(loc = low, scale = interval),4)) + '.') # calculate stats and symmetric interval
print('The variance is ' + str(round(uniform.var(loc = low, scale = interval),4)) + '.')
print('The interval at alpha level ' + str(round(1-level,3)) + ' is ' + str(uniform.interval(alpha = alpha,loc = low,scale = interval)))
```

#### Gaussian Distribution

Let's now use the Gaussian parametric distribution.

* we will need the parameters mean and the variance

We will apply the forward and reverse operations and calculate the summary statistics.

```python
from scipy.stats import norm as my_dist                                      # import traingular dist as my_dist
dist_type = 'Gaussian'                                                       # give the name of the distribution for labels
mean = 0.15; stdev = 0.05                                                    # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, loc = mean, scale = stdev)                  # calculate density for each x value
P_values = my_dist.cdf(x_values, loc = mean, scale = stdev)                  # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' PDF'); plt.xlabel('Values'); plt.ylabel('Density')

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

p_values = np.linspace(0.00001,0.99999,100)                                  # get an array of p-values
x_values = my_dist.ppf(p_values, loc = mean, scale = stdev)                  # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3, label='uniform pdf')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.8, top=0.8, wspace=0.2, hspace=0.3); plt.title('Sampling Inverse ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

print('The mean is ' + str(round(my_dist.mean(loc = mean, scale = stdev),4)) + '.') # calculate stats and symmetric interval
print('The variance is ' + str(round(my_dist.var(loc = mean, scale = stdev),4)) + '.')
print('The interval at alpha level ' + str(round(1-level,3)) + ' is ' + str(my_dist.interval(alpha = alpha,loc = mean,scale = stdev)))
```

#### Log Normal Distribution

Now let's check out the log normal distribution.

* We need the parameters $\mu$ and $\sigma$

```python
from scipy.stats import lognorm as my_dist                                   # import traingular dist as my_dist
dist_type = 'Log Normal'                                                     # give the name of the distribution for labels
mu = np.log(0.10); sigma = 0.2                                               # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, s = sigma, scale = np.exp(mu))              # calculate density for each x value
P_values = my_dist.cdf(x_values, s = sigma, scale = np.exp(mu))              # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' PDF'); plt.xlabel('Values'); plt.ylabel('Density')

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.3); plt.title('Sampling ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

p_values = np.linspace(0.00001,0.99999,100)                                        # get an array of p-values
x_values = my_dist.ppf(p_values,  s = sigma, scale = np.exp(mu))             # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.3, label='uniform pdf')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.8, top=0.8, wspace=0.2, hspace=0.3); plt.title('Sampling Inverse ' + str(dist_type) + ' CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

#print('The mean is ' + str(round(my_dist.mean(loc = mean, scale = stdev),4)) + '.') # calculate stats and symmetric interval
#print('The variance is ' + str(round(my_dist.var(loc = mean, scale = stdev),4)) + '.')
#print('The interval at alpha level ' + str(round(1-level,3)) + ' is ' + str(my_dist.interval(alpha = alpha,loc = mean,scale = stdev)))
```

There are many other parametric distributions that we could have included.  Also we could have demonstrated the distribution fitting.

#### Comments

This was a basic demonstration of working with parametric distributions.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Energy Minimization Assignment, PharmSci 175/275
##### By David Mobley (UCI), Jan. 2018

##### Adapted with permission from an assignment by M. Scott Shell (UCSB)

## Overview
In this assignment, you will begin with a provided template and several functions, as well as a Fortran library, and add additional code to perform a conjugate-gradient minimization. That is, you will write a conjugate-gradient minimizer. You will then apply this minimize to generate Lennard-Jones clusters with varying numbers of particles, and look at how the energy varies as a function of cluster size.

The Jupyter notebook for this assignment is laid out with action items YOU need to take labeled as Step 1-7. These are interspersed with background information on the problem, some examples, and a sandbox section to tinker with some of the functions. So read on, and watch for the sections which require your action.

## What are Lennard-Jones clusters?

Clusters are small, stable packings of (often spherical) particles. These particles could be colloidal particles, nanoparticles, etc. There has been considerable work spent studying these clusters over the years, from atomic sizes up to colloidal particles in the nanometer to micrometers scale. Cluster analysis is important to understanding a range of phenomena, including structures of solids, aggregation and precipitation of particles, the structure of nanomaterials, self-assembly behavior of synthetic and biomolecular systems, and diffusion in dense liquids.

A cluster can be characterized by the number and type of particles and the energetic interactions between them. Here, we will examine Lennard-Jones (LJ) clusters, which are clusters of simple attractive spherical particles with interactions modeled by the Lennard-Jones interaction.

For LJ clusters, there are cluster sizes of unusual stability. These are called magic number clusters and correspond to cluster sizes where the packing of atoms is particularly efficient, leading to very favorable energies and hence exceptional stability. The most stable such clusters are built from an icosahedral arrangement of particles, and the first few such **magic numbers** for cluster sizes of icosahedral geometries are 13, 19, 38, 55, and 75.

These clusters are still interesting from a basic physical chemistry point of view, but our interest here is mainly in (a) energy minimization algorithms, and (b) learning how to do non-trivial numerics in Python.

## Here, we will energy minimize Lennard-Jones clusters of different sizes

Here, we will examine different numbers of particles and attempt to find the minimum energy cluster for each number of particles. Our energy model will use the LJ potential in its dimensionless form (meaning that we have changed units so that all of the constants are hidden in the units). We denote this by putting a star on the potential:

\begin{equation}
U^* = \sum \limits_{i<j} 4(r_{ij}^{-12} - r_{ij}^{-6})
\end{equation}

We will start with a random initial configuration of particles, and try to use an energy minimization algorithm to find the most stable configuration of particles. But when there are more than just a few particles, there will be more than one local minimum, so there is no guarantee the energy minimizer will find the global minimum. In such cases, we will need to minimize from random initial configurations many times in order to ensure we locate the global minimum.
There is also the possibility of forming multiple separate clusters separated by a significant difference. This is not unlikely, since the LJ interaction is only very weakly attractive at large distances. So, to ensure we form single clusters, we will use a weak biasing potential to pull all of the particles towards the origin, biasing the system towards forming a single cluster. Otherwise, the LJ potential will tend to be too weak to pull together very distant particles in these tests. We will use a harmonic biasing potential, so that the total potential energy (“force field”) is:

\begin{equation}
U^* = \sum\limits_i \alpha |\mathbf{r}_i|^2 + \sum \limits_{i<j} 4(r_{ij}^{-12} - r_{ij}^{-6})
\end{equation}

Here we will use $\alpha = 0.0001 N^{-2/3}$ where $N$ is the number of particles; this will be a very small number. This particular form is selected so that the energy due to this term for a cluster of $N$ particles is, on average, constant regardless of $N$.

## Additional details
For this assignment, your job is to perform a conjugate-gradient minimization of Lennard-Jones particles which are initially randomly distributed in space. I am providing several items for you:
* A Fortran library (emlib) which you can use within Python to calculate energies and forces
* A Python library (pos_to_pdb.py) which you can use to output structures to your disk to visualize motion of your particles (for example with PyMol) if you so desire
* A template for your assignment (below) in iPython notebook format; this also will be posted on the class website in plain Python format in case my experiment with iPython notebooks fails here.
 * This template contains some code which will get you started, including code for a line search minimization.
 * It also contains places for you to write the functions (outlined below) you need to write to complete the assignment.
* Here we will be using position and force arrays to represent positions/forces on all of the atoms in the system. Make sure you are [comfortable with that notation](https://github.com/MobleyLab/drug-computing/blob/master/uci-pharmsci/lectures/energy_minimization/energy_minimization.ipynb) before beginning.

## A quick (but important) note on Notebook usage:
iPython notebooks such as this one often contain a variety of cells containing code. These are normally intended to be run, which you can when you have an individual cell selected, by hitting the button at the top with a 'play' symbol, or by typing shift-enter. If you do NOT do so on each of the cells defining variables/functions which will be used here, then you will encounter an error about undefined variables when you run the later cells.

## Your step 1 for the assignment: Start by doing some file bookkeeping:
 * Find `emlib.f90` and optional utility `pos_to_pdb.py` in this directory.
 * In the command prompt navigate to that folder and type 'f2py -c -m emlib emlib.f90' which should compile the fortran library for use within python (For more on F2Py, refer to the [f2py documentation](https://docs.scipy.org/doc/numpy-dev/f2py/)). In OS X, this may require you to install the (free) XCode developer tools (available from the Mac App store) and the command-line developer tools first (the latter via `xcode-select --install`). In Linux it should just work. Windows would be a hurdle.
 * In your command prompt, start theis Jupyter notebook (in OSX this would be something like 'Jupyter notebook 272EnergyMinimization'), which should open it in your web browser; you're running it already unless you are looking at the HTML version of this file.

Template Python code for the assignment is provided below. I suggest making a new notebook which is a copy of this one (perhaps containing your name in the filename) and working from there.

 ## Next, we prep Python for the work:

 First we import the numpy numerical library we're going to need, as well as the compiled Fortran library emlib

```python
import numpy as np
import emlib
from pos_to_pdb import *
#This would allow you to export coordinates if you want, later
```

## Important technical note: Unit masses, etc.

Note that all of the following code will assume unit atomic masses, such that forces and accelerations are equal -- that is, instead of $F=ma$ we write $F=a$ assuming that $m=1$. We also drop most constants. This is a relatively common trick in physics when you are interested only in seeing how the basic equations work, and not in quantitative comparison with experimental numbers. It can be called using "dimensionless units".

## Then we define the LineSearch function:

Here is the `LineSearch` function which is provided for you. Read the prototype (definition) and documentation to understand what it needs and what it will do (note that you do NOT need to read all the code):

```python
def LineSearch(Pos, Dir, dx, EFracTol, Accel = 1.5, MaxInc = 10.,
               MaxIter = 10000):
    """Performs a line search along direction Dir.
    Input:
       Pos: starting positions, (N,3) array
       Dir: (N,3) array of gradient direction
       dx: initial step amount, a float
       EFracTol: fractional energy tolerance
       Accel: acceleration factor
       MaxInc: the maximum increase in energy for bracketing
       MaxIter: maximum number of iteration steps
Output:
       PEnergy: value of potential energy at minimum along Dir
       PosMin: minimum energy (N,3) position array along Dir
"""
    #start the iteration counter
    Iter = 0

    #find the normalized direction
    NormDir = Dir / np.sqrt(np.sum(Dir * Dir))

    #take the first two steps and compute energies
    Dists = [0., dx]
    PEs = [emlib.calcenergy(Pos + NormDir * x) for x in Dists]

    #if the second point is not downhill in energy, back
    #off and take a shorter step until we find one
    while PEs[1] > PEs[0]:
        Iter += 1
        dx = dx * 0.5
        Dists[1] = dx
        PEs[1] = emlib.calcenergy(Pos + NormDir * dx)

    #find a third point
    Dists.append( 2. * dx )
    PEs.append( emlib.calcenergy(Pos + NormDir * 2. * dx) )

    #keep stepping forward until the third point is higher
    #in energy; then we have bracketed a minimum
    while PEs[2] < PEs[1]:
        Iter += 1

        #find a fourth point and evaluate energy
        Dists.append( Dists[-1] + dx )
        PEs.append( emlib.calcenergy(Pos + NormDir * Dists[-1]) )

        #check if we increased too much in energy; if so, back off
        if (PEs[3] - PEs[0]) > MaxInc * (PEs[0] - PEs[2]):
            PEs = PEs[:3]
            Dists = Dists[:3]
            dx = dx * 0.5
        else:
            #shift all of the points over
            PEs = PEs[-3:]
            Dists = Dists[-3:]
            dx = dx * Accel

    #we've bracketed a minimum; now we want to find it to high
    #accuracy
    OldPE3 = 1.e300
    while True:
        Iter += 1
        if Iter > MaxIter:
            print("Warning: maximum number of iterations reached in line search.")
            break

        #store distances for ease of code-reading
        d0, d1, d2 = Dists
        PE0, PE1, PE2 = PEs

        #use a parobolic approximation to estimate the location
        #of the minimum
        d10 = d0 - d1
        d12 = d2 - d1
        Num = d12*d12*(PE0-PE1) - d10*d10*(PE2-PE1)
        Dem = d12*(PE0-PE1) - d10*(PE2-PE1)
        if Dem == 0:
            #parabolic extrapolation won't work; set new dist = 0
            d3 = 0
        else:
            #location of parabolic minimum
            d3 = d1 + 0.5 * Num / Dem

        #compute the new potential energy
        PE3 = emlib.calcenergy(Pos + NormDir * d3)

        #sometimes the parabolic approximation can fail;
        #check if d3 is out of range < d0 or > d2 or the new energy is higher
        if d3 < d0 or d3 > d2 or PE3 > PE0 or PE3 > PE1 or PE3 > PE2:
            #instead, just compute the new distance by bisecting two
            #of the existing points along the line search
            if abs(d2 - d1) > abs(d0 - d1):
                d3 = 0.5 * (d2 + d1)
            else:
                d3 = 0.5 * (d0 + d1)
            PE3 = emlib.calcenergy(Pos + NormDir * d3)

        #decide which three points to keep; we want to keep
        #the three that are closest to the minimum
        if d3 < d1:
            if PE3 < PE1:
                #get rid of point 2
                Dists, PEs = [d0, d3, d1], [PE0, PE3, PE1]
            else:
                #get rid of point 0
                Dists, PEs = [d3, d1, d2], [PE3, PE1, PE2]
        else:
            if PE3 < PE1:
                #get rid of point 0
                Dists, PEs = [d1, d3, d2], [PE1, PE3, PE2]
            else:
                #get rid of point 2
                Dists, PEs = [d0, d1, d3], [PE0, PE1, PE3]

        #check how much we've changed
        if abs(OldPE3 - PE3) < EFracTol * abs(PE3):
            #the fractional change is less than the tolerance,
            #so we are done and can exit the loop
            break
        OldPE3 = PE3

    #return the position array at the minimum (point 1)
    PosMin = Pos + NormDir * Dists[1]
    PEMin = PEs[1]

    return PEMin, PosMin
```

## Step 2: Write a function to assign random initial positions to your atoms

We need a function that can randomly place N atoms in a box with sides of length L. Write a function based on a tool from the numpy 'random' module to do this. Some hints are in order:

* NumPy contains a ‘random’ module which is good for obtaining random numbers and/or arrays. For example, if you have imported numpy as np, then np.random.random(shape) returns a random array with the specified shape (i.e. ‘np.random.random(3)’ would be a 3x1 array) with elements randomly selected between 0 to 1. Try this out:

```python
a = np.random.random(3)
print("a=\n",a)
b = np.random.random((2,3))
print("b=\n",b)
```

    a=
     [ 0.27969529  0.37836589  0.96785443]
    b=
     [[ 0.37068791  0.64081204  0.21422213]
     [ 0.471194    0.28575791  0.54468387]]

* Note that in your function, you want the numbers to run from 0 to L. You might try out what happens if you multiply 'a' and 'b' in the code above by some number.

Now, write your function. I've written the doc string and some comments for you, but you have to fill in its inner workings:

```python
def InitPositions(N, L):
    """Returns an array of initial positions of each atom,
placed randomly within a box of dimensions L.
Input:
    N: number of atoms
    L: box width
Output:
    Pos: (N,3) array of positions
"""
    #### WRITE YOUR CODE HERE ####
    ## In my code, I can accomplish this function in 1 line
    ## using a numpy function.
    ## Yours can be longer if you want. It's more important that it be right than that it be short.

    return Pos
```

## Step 3: Write the Conjugate Gradient function described below

Fill in code for the ConjugateGradient function below based on the discussion in class and below, supplemented by your reading of Leach's book (and other online sources if needed). Some additional guidance and hints are warranted first.

### Hints for ConjugateGradient:
* As discussed/demonstrated above, a LineSearch function is already provided for you here
* You will need to remember that our force and position arrays give the force/position of all of the atoms in the system (x, y, and z components) as [introduced previously](https://github.com/MobleyLab/drug-computing/blob/master/uci-pharmsci/lectures/energy_minimization/energy_minimization.ipynb).
* Keep careful track of which quantities you expect to be vectors/arrays versus scalars. You may wish to check that quantities are what you expect they are.
* You are going to want to keep doing line searches until the energy stops changing. Write a loop to do this, and store your evaluated energies as you go.
* A fortran library, `emlib`, is provided for you to calculate energies and forces. You should be able to ask for 'help(emlib)' for info on usage. You can also look directly at the Fortran code if you would like, though this may be less helpful.
* You can get the potential energy and forces using the provided library using functions from emlib. For example, if `Pos` is an array of positions:

   `PEnergy, Forces = emlib.calcenergyforces(Pos)`

   `Forces = emlib.calcforces( Pos )`
* Conjugate gradient does not specify an initial direction. Your initial search should be in the direction of the force.
* After the initial line search, subsequent line search directions $i$ should be found using this expression for $v_i$, the direction matrix:

   \begin{equation}
   \mathbf{v}_i^N = \mathbf{f}_i^N + \gamma_i \mathbf{v}_{i-1}^N
   \end{equation}

   where

   \begin{equation}
   \gamma_i = \frac{ (\mathbf{f}_i^N-\mathbf{f}_{i-1}^N) \mathbf{f}_i^N}{\mathbf{f}_{i-1}^N \mathbf{f}_{i-1}^N}
   \end{equation}

   Note that here, $\mathbf{f}_i^N$ denotes the force on the particles at step $i$ (and hence it has 3N dimensions - $x$, $y$, and $z$ for each particle) and $\mathbf{f}_{i-1}^N$ is the force at the last ($i-1$) step, [as discussed previously](https://github.com/MobleyLab/drug-computing/blob/master/uci-pharmsci/lectures/energy_minimization/energy_minimization.ipynb). Note that the forces are a collection of vectors, one vector for the force on each particle. $\gamma_i$ should be just a number (scalar). Hint: Note that if you have a force array consisting of a set of vectors, the product you want inside the equation for $\gamma_i$ should be an element-by-element multiplication, not a dot or inner product.  **Be sure to see the helpful tips about how to calculate this which were given in the energy minimization lecture**!

* You want to end up at the point, in your code, where you can obtain the new direction by calculating something like `Dir = newForces + gamma * Dir`
* Continue successive line searches in your CG minimization until the fractional change in energy on subsequent searches is less than the tolerance. That is, you'll set it up to use an `EFracTolCG` variable and continue until this criteria is met (where $U_i$ is the potential energy at the present step):
\begin{equation}
\left|U_i-U_{i-1}\right| < EFracTolCG \times \left| U_i\right|
\end{equation}

* To debug your code, you will probably want to initially use `print` statements in the minimization routine to monitor the energy as a function of step to make sure it's doing the right thing!

### Now actually write ConjugateGradient:

```python
def ConjugateGradient(Pos, dx, EFracTolLS, EFracTolCG):
    """Performs a conjugate gradient search.
Input:
    Pos: starting positions, (N,3) array
    dx: initial step amount
    EFracTolLS: fractional energy tolerance for line search
    EFracTolCG: fractional energy tolerance for conjugate gradient
Output:
    PEnergy: value of potential energy at minimum
    Pos: minimum energy (N,3) position array
"""
    #### WRITE YOUR CODE HERE ####
    ## In my code, I can accomplish this function in 10 lines ###

    #A return statement you may/will use to finish things off
    return PEnergy, Pos
```

## Step 4: Energy minimize a variety of clusters, storing energies

Write code to use the functions you wrote above, plus the emlib module, to energy minimize clusters of various sizes. Loop over clusters from size N=2 to (and including) N=25. For each particle number, do the following:
* Perform K (to be specified below in the section on graphing) minimizations, each starting from a different random configuration of particles
* Store the K energies to a list
* Display the minimum, average, and standard deviation of the minimized energies for the trials. Note standard deviations can be calculated with the numpy.std function (`np.std()`)
* After doing this, you'll be tasked with making some plots.

Use the following settings:
* `dx = 0.001`
* `EFracTolCG = 1.0e-10`
* `EFracTolLS = 1.0e-8`
* And place the particles with L chosen such that the average number density of particles ($N/V$, where $V=L^3$) is $0.001$. That is, for every $N$, solve for $L$ such that $N/L^3 = 0.001$.

These are relatively typical settings for this kind of a system.

**I suggest you do this first for just one N and K to make sure it works**. Then set up a loop over N and perhaps (if you like) a loop over K as well.  Reserve the large K values until you are absolutely certain it’s working. Note that if the computational time becomes prohibitive (i.e. if it runs more than overnight, or your computer is having difficulties handling the lode) we can migrate your simulations to GreenPlanet.

You can easily add visualization of a trajectory by adding, within your ConjugateGradient function’s central loop, a call to the PosToPDB function of the pos_to_pdb module. Assuming you’ve done ‘from pos_to_pdb import *’ you’d add something like:

`PosToPDB( Pos, L, ‘mytrajectory.pdb’)`

within the loop inside your ConjugateGradient minimizer. This will write out each step of the minimization as a separate frame in a pdb file, which you can download with scp and view in PyMol to see exactly what’s going on.
Note that visualization (really, the file I/O and coordinate conversions) will slow things considerably, so I suggest you only do this in one specific case to check out what’s going on, or to troubleshoot if things don't appear to be working. It should also be possible to add interactive visualization via `nglview` here, though I've not done that for you.

* Hint: **You MAY want to use Python's pickle module to save out your data at the end of your calculations, since the next step involves plotting your data and you may want to easily be able to read it back in**. At the very least - whether you save it to disk or not - you'll want to store it (specifically, the minimum and average energies at each N) to variables for later reuse. If you had the variable `energies` containing all of the energies obtained at K = 10000 you might dump this using:

```python
import pickle
file = open('energies.pickle', "w")
pickle.dump( energies, file)
file.close()
#To load again, use:
#file = open("energies.pickle", "r")
#energies = pickle.load(file)
#file.close()
```

### Write your code here:

```python
#Your energy minimization code here
#This will be the longest code you write in this assignment
```

## Step 5: Graph your findings

Plot the minimum and average energies as a function of N for each of K=100, 1000, and 10000. The last case may be fairly time consuming (i.e. several hours) and should be done without output of pdb files for visualization (since this can slow it down).

Use matplotlib/PyLab to make these plots.

**Hint: If your minimizations are proceeding extremely slowly, it may mean you have an error in calculation of gamma**, such that even K=100 or K=10 could take a very long time. Ensure you have implemented the equation for gamma correctly. Even with a correct gamma value, this will take considerable time for the larger N values.

```python
#Your code for this here
```

## Step 6: Compare with what's expected

Compare your results (your minimum energy at each N value) with the known global minimum energies, via a plot and by commenting on the results. These are from ( Leary, J. Global Optimization 11:35 (1997)). Add this curve to your graph. Why might your results be higher?

```python
#Write code here to add these to your graph
```

## Step 7: Fit a curve to the data

(**For undergraduates, this section is optional.**)

Based on macroscopic arguments, the energy of a cluster could scale with both surface area (via a surface tension) and volume (via an energy density for bulk) of the cluster. So we could model the minimum energy as depending on the cluster size in this way:
\begin{equation}
U_{min} \propto a + b N^{2/3} +cN
\end{equation}

Fit this equation to your data in the K=10000 case. You can do this using a least-squares fit, for example using fitting functions within SciPy (`optimize.leastsq`, for example, or similar functions in `scipy.stats`. A fairly dated tutorial is [here](http://www.tau.ac.il/~kineret/amit/scipy_tutorial) (sec 5.4), or see [stack overflow](https://stackoverflow.com/questions/19791581/how-to-use-leastsq-function-from-scipy-optimize-in-python-to-fit-both-a-straight).

**Once you perform the fit, plot the difference between the true minimum energy and the expected energy from this equation as a function of N. Can you identify the magic numbers from this curve?**

```python
#Your code here
```

## Step 8: Submit your results

Submit (via Canvas) your graphs, with axes and curves labeled. Also attach a copy of your Python code (this notebook or, if you prefer, a separate script). Specifically, you should be including:
* Your Python code
* A graph (or graphs) of minimum and average energies as a function of N for K=100, 1000, and 10000. (Note you can use savefig to save graphs). These can be included in this notebook, or saved separately.
* A graph comparing your results for the K=10000 case with the expected results from Leary, as a function of N, overlaid with a fitted curve using the equation above (Step 7); undergrads can omit the fit if desired as this is optional. Provide the parameters determined in your fit.
* Give brief discussion of why your results might be higher than those of Leary. Comment on the magic numbers (step 7).

### Please give clear answers to questions given above, ideally in a box below or a separate report

Specifically, explain whether you can identify the magic numbers, how you can tell what they are, etc. If you are not certain whether your results are right, explain any reservations you have and why, discuss where you think you might have gone wrong, etc. Basically, be sure to answer all my questions AND give a very brief discussion of your results. The main ways to lose points on this assignment are to:
- not finish
- submit late
- not discuss your results
I tend to be generous with partial credit for people whose work is mostly correct/mostly complete.


Solution to: [Day 2: Basic Probability](https://www.hackerrank.com/challenges/s10-mcq-1/tutorial)

Table of Contents

- Table of Contents
- Probability Notes
    - Event, Sample Space, and Probability
    - Fundamental probability rules
    - Sample question
    - Sample question 2
- Math Solution
    - Information
    - Formula:
    - Implementation
- Solution
    - Imports
    - Process
    - Probability of dice rolls at target
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Probability Notes

## Event, Sample Space, and Probability
In probability theory, an **experiment** is any procedure that can be infinitely repeated and has a well-defined set of possible outcomes, known as the sample space, **S**. We define an event to be a set of outcomes of an experiment (also known as a subset of **S**) to which a probability (numerical value) is assigned.

The probability of the occurrence of an event, A, is:

\begin{equation}
\large
\mathrm{P}(A) =
\frac
{\text{Number of favorable outcomes}}
{\text{Total number of outcomes}}
$$

### Fundamental probability rules
Here are the first two fundamental rules of probability:
1. Any probability, **P(A)** , is a number between 0 and 1 *(0 <= P(A) <= 1)*.
2. The probability of the sample space, **S**, is 1 *(P(S) = 1)*.

So how do we bridge the gap between the value of **P(A)** and the sample space **P(S)**?
We know that **P(A)** is the probability that event will occur, then we define $P(A^{c})$ - the complement of **P(A)** to be the probability that event **P(A)** will not occur.

If our sample space is composed of the probabilities of P(A)'s occurrence and non-occurrence, we can then say:

\begin{equation}
\large
P(A) + P(A^{c}) = 1
\end{equation}

or, that the sum of all possible outcomes of A in the sample space is equal to 1.

This is the third fundamental rule of probability:

\begin{equation}
\large
\text{P}(A^{c}) = 1 - \text{P(A)}
\end{equation}

### Sample question
Find the probability of getting an odd number when rolling a 6-sided fair die.

Given the above question, we can extract the following:
- Experiment: rolling a 6-sided die.
- Sample space (1): .
- Event (3): that the number rolled is odd (i.e., 1, 3, 5).

P(A) = (number of favorable outcomes) / (total number of outcomes) = P(A) / P(S) = 3/6 = 1/2

\begin{equation}
\normalsize
P(A) = \frac
{\text{number of favorable outcomes}}
{\text{total number of outcomes}}
\end{equation}

\begin{equation}
\normalsize
= \frac
{P(A)}{P(S)}
\end{equation}

\begin{equation}
\normalsize
= \frac{3}{6}
= \frac{1}{2}
\end{equation}

### Sample question 2

Let A and B be two events such that P(A) = 2/5 and P(B) = 4/5. If the probability of the occurrence of either A or B is 3/5, find the probability of the occurrence of both A and B together (i.e., P(A n B)).

We can use our fundamental rules of probability to solve this problem:

\begin{equation}
\large
|A \cup B| = |A| + |B| - |A \cap B|
\end{equation}

\begin{equation}
\large
P(A \cup B) = P(A) + P(B) - P(A \cap B)
\end{equation}

\begin{equation}
\large
P(A \cap B) = P(A) + P(B) - P(A \cup B)
\end{equation}

\begin{equation}
\large
P(A \cap B) = \frac{2}{5} + \frac{4}{5} - \frac{3}{5}
\end{equation}

\begin{equation}
\large
= \frac{3}{5}
\end{equation}

# Math Solution

## Information
In a single roll of 2 fair (evenly-weighted) six-sided dice, find the probability that their sum will be at most 9.
- Experiment: rolling 2 dice
- Sample space(S) = possible outcomes = 6 * 6 = 36
- Event P(A + B) <= 9

## Formula:

\begin{equation}
\large
P(A) = \frac{\text{Number of favorable events}}{\text{Probability Space}}
\end{equation}

\begin{equation}
\large
P(A) = \frac{P(A)}{P(S)}
\end{equation}

\begin{equation}
\large
P(A) = P(S) - P(A^{c})
\end{equation}

We can also find $P(A^{C})$, which is easier to calculate given the experiment context.

## Implementation
*Determine the probability of rolling 10+*

\begin{equation}
\large
P(A) = P(S) - P(A^{C})
\end{equation}

\begin{equation}
\large
P(A^{C}) = P( (4, 6)) + P( (5, 5), (5, 6)) + P( (6, 4), (6, 5), (6, 6))
\end{equation}

\begin{equation}
\large
= \frac{1 + 2 + 3}{36}
\end{equation}

\begin{equation}
\large
= 1 - \frac{6}{36}
\end{equation}

\begin{equation}
\large
= \frac{30}{36}
= \frac{5}{6}
\end{equation}

\begin{equation}
\large
= 0.833
\end{equation}

# Python Solution

## Imports

```python
from typing import List
```

## Process
- Iterate through all possible rolling combinations and create list of values <= 9
- Divide by number of total events

## Probability of dice rolls at target

```python
def get_possible_combinations(num_dice: int, dice_face: int, target: int) -> List[List[int]]:
	"""Returns Cartesian product of all possible dice rolls.

	Uses two helper functions:
		- get_possible_rolls, generator func to yield all die rolls.
		- recurse_roll_combos, helper recursion to create combinations for `num_dice`
	"""
	def get_possible_rolls(dice_face: int) -> int:
		"""Generator funct to yield possible rolls."""
		for i in range(1, dice_face + 1): yield i

	def recurse_roll_combos(processed_dice: int, all_roll_combos: list) -> List[List[int]]:
		"""Helper func to recurse through all dice."""
		if processed_dice == num_dice:		## Base case
			return all_roll_combos

		new_roll_combos = []
		for roll in all_roll_combos:
			roll_combo = [roll + [val] for val in get_possible_rolls(dice_face)]
			new_roll_combos += (roll_combo)

		processed_dice += 1
		return recurse_roll_combos(processed_dice= processed_dice, all_roll_combos = new_roll_combos)

	first_rolls = [[val] for val in get_possible_rolls(dice_face)]
	roll_combinations = recurse_roll_combos( 1, first_rolls)

	assert (len(roll_combinations) == dice_face ** num_dice)
	return roll_combinations
```

## Main

```python
def main():
	num_dice = 2
	dice_face = 6
	target = 9

	## Get Combinations
	roll_combos = get_possible_combinations(num_dice, dice_face, target)

	## Count combos at target
	num_under_target = 0
	for combo in roll_combos:
		if sum(combo) <= target:
			num_under_target += 1

	## Return P(A) / P(S)
	prob = num_under_target / len(roll_combos)
	print( round(prob, 3))
```

```python
if __name__ == "__main__":
	main()
```

    0.833


## Multidimensional search with gradient-search methods
## The objective is to find a minimum of a multivariate function

Luca Magri (lm547@cam.ac.uk)

(With many thanks to Professor Gábor Csányi.)

Multivariate function = multi-variable function = function that depends on two variables at least

## Direct search for multi-variable functions

- We assume we can evaluate the function $f(x)$ that is to be minimised
- If we do not have access to the gradient, we can perform a __direct search__
- A generalisation of the interval search for multiple variables is the Nelder-Mead method, which is robust but slow

- For example, we rarely have access to the gradient when the objective function is the result of a complex simulation, e.g.,"lift on an aircraft wing", "melting point of a material", "efficiency of a circuit design", etc.

 - Aside: Sometimes we create a __surrogate model__, which approximates the objective function, but provides analytical gradients. __Machine Learning__ methods provide surrogate models.

## Search directions for multi-variable functions
- Access to the gradient enables __gradient search__, which is more efficient than direct (gradient-free) search

- Access to the Hessian (the gradient of the gradient) can improve convergence significantly
 - The computation of the Hessian is straightforward to do for problems with only a few variables
 - The computation of the Hessian may be computationally expensive for problems with many variables, say, thousands or more

 - Even a rough approximation of the Hessian can be beneficial (see "preconditioning")

## Lectures 3 and 4: List of contents
In these lectures, we will analyse some common search-direction methods for multidimensional search

1. Steepest descent method

1. Newton-Raphson method

1. Barzilai-Borwein (BB) method

1. Conjugate-gradient (CG) method

1. Newton-Gauss method

## Nomenclature

- $f(x): \mathbb{R}^N\rightarrow\mathbb{R}$ is a nonlinear function, which we want to minimize

- $x^*\in\mathbb{R}^N$ is a minimum of $f$

- $\nabla f = \begin{pmatrix}
\frac{\partial f}{\partial x_1},
\frac{\partial f}{\partial x_2},
\ldots,
\frac{\partial f}{\partial x_N}
 \end{pmatrix}^T = \frac{\partial f}{\partial x_i}$, $i=1,2,\ldots, N$ is the gradient

- $H=\nabla(\nabla f(x))=\nabla^2 f(x)$ is the Hessian $$H_{i,j}=\frac{\partial^2 f}{\partial x_i\partial x_j}\;\;\;\;\;i,j=1,2,\ldots,N$$

- $k=0,1,2,\ldots$ is the iteration step

- $d_k$ is the $k$-th search direction

- $\alpha_k$ is the step size to take along the $k$-th search direction

- h.o.t. stands for "higher order terms"

## Revision on Taylor expansion for multi-variable functions
- A function that is differentiable infinite times can be expressed as a Taylor series around a point
- We will consider terms up to second order
- The Taylor expansion of the multi-variable function $f(x)$ around the point $x_k$ is
\begin{align}
f(x) = f(x_k) + \nabla f(x_k)^T (x - x_k) + \frac12 (x-x_k)^T H(x_k) (x-x_k) + h.o.t.
\end{align}

- The term $\nabla f(x_k)^T (x - x_k)$ is called first-order term
- The term $\frac12 (x-x_k)^T H(x_k) (x-x_k)$ is called second-order term
- h.o.t. stands for higher-order terms

## Steepest descent method
We exploit the fact that the gradient is the direction of maximum first-order change of a multi-variable function $f(x)$
1. Start with an initial guess for the minimum $x_0$
1. Set the search direction as the negative gradient $$d_k \equiv -\nabla f(x_k)$$
1. Determine by _line search_ the step size $\alpha_k$ that minimizes $$f(x) \;\;\; \textrm{along} \;\;\;d_k$$
1. Update the solution $x_{k+1} = x_k + \alpha_k d_k$
1. If not converged, back to 2.

- Notice __nested structure__:
 - The main loop in step 2 changes the _direction_
 - The inner loop in step 3 is optimisation in 1D

 - How accurate should this inner optimisation (line search) be?
 - Ideally,
 $$\frac{d f(x_k+\alpha_k d_k)}{d\alpha_k} =
 0$$
 which implies that the gradient at $x_k+\alpha_k d_k$ is orthogonal to $d_k$, i.e., successive search directions are orthogonal to each other
 $$ \boxed{d^T_k d_{k+1} =0}$$

 - Proof (optional):
 \begin{align}
 \frac{d f(x_k+\alpha_k d_k)}{d\alpha_k} &=0\\
 \frac{df(x_{k+1})}{d\alpha}& =0 \\
 \frac{\partial f(x_{k+1})}{\partial x_{k+1}}\cdot\frac{\partial x_{k+1}}{\partial\alpha} & =0 \\
 \nabla f(x_{k+1}) \cdot \frac{\partial (x_{k}+\alpha_k d_k)}{\partial\alpha} & =0\\
  -d_{k+1} \cdot d_k & =0\\
 \end{align}
 Therefore
 $d^T_k d_{k+1} =0$

 - Geometric interpretation: If the gradient $-d_{k+1}$ had a component along $x_k+\alpha_k d_k$, then $x_k+\alpha_k d_k$ could not be a minimum of $f(x_k+\alpha_k d_k)$, which contradicts the hypothesis. Hence, the gradient must have a zero component along $x_k+\alpha_k d_k$, which means that it must be orthogonal to $d_k$.

- Practically, an approximate line search is performed
  $$\frac{d f(x_k+\alpha_k d_k)}{d\alpha_k} \approx
 0$$
 hence
 $$ \boxed{d^T_k d_{k+1} \approx0 }$$
- If we approximate the function with a second-order Taylor expansion, the step size is
$$ \alpha_k= - \frac{\nabla f(x_k)^Td_k}{d_k^TH(x_k)d_k}$$
   However, if you have the Hessian $H$, you may want to use better search methods (e.g., Newton-Raphson)
 - If the approximation is poor, the algorithm might not converge
 - If the approximation is too accurate, it could be a waste of resources because we are only making progress along the current search direction
 - Tradeoff: $f(x_k+\alpha_k d_k)$ is often approximated by a quadratic polynomial, or a maximum number of iterations is set

Example with the Rosenbrock function
$$ f(x_1, x_2) = (1-x_1)^2 + a(x_2-x_1^2)^2$$
 where $x_1$ and $x_2$ are the two variables on which $f$ depend (in this example, they have nothing to do with the iteration), and $a=10$
- The gradient is
\begin{align}\nabla f(x_1,x_2)
& = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\\frac{\partial f}{\partial x_2} \end{bmatrix} \\
& = \begin{bmatrix} \frac{\partial \left((1-x_1)^2 + a(x_2-x_1^2)^2\right)}{\partial x_1} \\\frac{\partial \left( (1-x_1)^2 + a(x_2-x_1^2)^2\right)}{\partial x_2} \end{bmatrix} \\
& = \begin{bmatrix}  -2(1-x_1)-4ax_1(x_2-x_1^2) \\ 2a(x_2-x_1^2) \end{bmatrix}
\end{align}

```python
from pylab import *
import numpy as np
# function we will optimize
def Rosen(x,a=10):
    return (1-x[0])**2 + a*(x[1]-x[0]**2)**2
def dRosen(x,a=10):
    return np.array([-2.0*(1.0-x[0]) - 2.0*a*(x[1]-x[0]**2)*2.0*x[0],2.0*a*(x[1]-x[0]**2)])
```

    /home/nbuser/anaconda2_501/lib/python2.7/site-packages/matplotlib/font_manager.py:281: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      'Matplotlib is building the font cache using fc-list. '

```python
def linesearch_golden_section(f, x, d, alpha_init=None, xtol=1e-8):

    def fa(a):
        return f(x+a*d)

    # initial point
    alpha0 = 0
    f0 = fa(alpha0)

    # now find a bracket for the minimum
    if alpha_init == None:
        alpha1 = 1.0
    else:
        alpha1 = alpha_init

    # check to see if the point at alpha1 goes down, if not find one that does
    while fa(alpha1) > f0:
        alpha1 /= 2.0
        print (alpha0, f0, alpha1, fa(alpha1))

    f1 = fa(alpha1)

    # set up golden section ratio
    r = (np.sqrt(5)-1)/2.0
    # now find the outer bracket alpha2 where the function goes up again
    alpha2 = alpha1/r
    while fa(alpha2) < f1:
        alpha1 = alpha2
        alpha2 = alpha1/r
    f2 = fa(alpha2)

    # now we have three points in Golden Section ratio, (alpha0, alpha1, alpha2) such that the function goes down then up,
    # so it must have a minimum in between

    # now loop until convergence
    while abs(alpha0-alpha2) > xtol:
        # get 4th point
        alpha3 = alpha0*r+alpha2*(1-r); f3=fa(alpha3) # 0,3,1,2

        # depending on where the function value falls, update the brackets
        if f3 < f1:
            alpha2=alpha1; f2=f1
            alpha1=alpha3; f1=f3
        else:
            alpha0=alpha2; f0=f2
            alpha2=alpha3; f2=f3
    return 0.5*(alpha0+alpha2)

def steepest_descent(f, df, x0, ftol, xtol):
    traj = []

    # get initial direction
    x = x0[:]
    d = -1.0*df(x)
    alpha_init = 1e-6
    i = 0
    #print i, f(x), np.linalg.norm(d), x[:], d[:]
    traj.append((x[0],x[1]))

    # loop until convergence
    while np.linalg.norm(d) > ftol:
        i += 1
        # do line search in current direction
        alpha = linesearch_golden_section(f, x, d, alpha_init, xtol)
        # update estimate
        x += alpha*d
        # get new direction
        d = -1.0*df(x)

        #print i, f(x), np.linalg.norm(d), x[:], d[:]
        traj.append((x[0],x[1]))

    return x[:],traj
```

```python
x,y = np.meshgrid(np.linspace(-1.7,1.7,100), np.linspace(-0.8,3,100))
R = [[Rosen(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]

fig = figure(figsize=(12,8))
contourf(x, y, R, np.logspace(-1,2, 8))
plt.axes().set_aspect('equal', 'datalim')
xmin, traj = steepest_descent(Rosen,dRosen, np.array([-1.8,1.8]), 1e-3, 1e-8)

plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

Example with quadratic function
\begin{align}
f(x_1,x_2) & =x_1^2+x_2^2-1.9x_1x_2\\
 & = \frac{1}{2}\begin{bmatrix} x_1 \;\;\;x_2\end{bmatrix} \begin{bmatrix} 2 & -1.9\\ -1.9& 2\end{bmatrix}  \begin{bmatrix} x_1 \\x_2\end{bmatrix}\\
\end{align}

```python
def Q(x):
    return x[0]**2+ x[1]**2 - 1.9*x[0]*x[1]
def dQ(x):
    return np.array([2*x[0]-1.9*x[1], 2*x[1]-1.9*x[0]])

x, y = np.meshgrid(np.linspace(-1.1,1.1,50), np.linspace(-1.1,1.1,50))
Qxy = [[Q(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
figure(figsize=(12,8))
contour(x, y, Qxy, linspace(0, 0.1, 6))
title("Quadratic - Steepest descent")
axis('equal')
#axis([-1.5, 1.5, -1.1, 1.1])

xmin, traj = steepest_descent(Q, dQ, np.array([-0.8, -0.95]), 1e-2, 1e-2)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

## Convergence of steepest descent

- This algorithm converges __linearly__ ($p = 1$)

$$ \left\| x_{k+1}-x^*\right\| = \beta \left\| x_k - x^*\right\| $$

- $\beta$ is close to 1, hence the steepest descent is slow. As a consequence of [Kantorovich Inequality](http://mathworld.wolfram.com/KantorovichInequality.html), it can be shown that
$$ \beta \le \left[\frac{k-1}{k+1}\right]^2 $$
where $k$ is the condition number, which is the ratio between the largest and smallest eigenvalues of $H(x^*)$.

- In typical engineering problems, this ratio approaches 1 as the number of variables increases.
- Quadratic problems with ill-conditioned matrices lead to poor convergence of steepest descent.
  - Loosely speaking, ill-conditioned quadratic functions have isocontours that resemble elongated valleys.
  - Isocontour = contour, which is defined as $f(x)=c$ where $c$ is a constant

## Newton-Raphson method

- We assume we can evaluate the gradient, $\nabla f(x)$, and the Hessian, $H(x)=\nabla^2f(x)$
- $f(x)$ is approximated by a quadratic function using the function value, gradient and Hessian at the current location
- Mathematically, the function is Taylor expanded around $x_k$

$$ f(x) = f(x_k) + \nabla f(x_k)^T (x - x_k) + \frac12 (x-x_k)^T H(x_k) (x-x_k) + h.o.t. $$

- The higher order terms (h.o.t.) are neglected and $f$ is approximated by $q$

$$ f(x)\approx q(x) = f(x_k) + \nabla f(x_k)^T (x - x_k) + \frac12 (x-x_k)^T H(x_k) (x-x_k) $$

 - The minimum of $q(x)$ occurs when $\nabla q(x) = 0$ $$
\begin{array}
\\
\nabla q(x) &= \nabla f(x_k) + H(x_k) (x-x_k) = 0\\
x &= x_k - [H(x_k)]^{-1} \nabla f(x_k)\\
\end{array}
$$

- We use the minimum of $q(x)$ as our next guess for the minimum of $f(x)$ $$
x=x_{k+1} = x_k + \alpha_k d_k
$$

 - Therefore, the search direction is $
d_k = - [H(x_k)]^{-1} \nabla f(x_k)
$ and the step size is $\alpha_k = 1$.

```python
def ddQ(x):
    return np.matrix([[2, -1.9], [-1.9, 2]])

def newton(f, df, ddf, x0, ftol, xtol=1e-3):
    traj = []

    # get initial direction
    x = np.array(x0[:])
    d = -1.0*asarray(np.matmul(np.linalg.inv(ddf(x)),df(x)))[0]

    i = 0
    #print i, f(x), np.linalg.norm(d), x[:], d[:]
    traj.append((x[0],x[1]))

    # loop until convergence
    while np.linalg.norm(d) > ftol:
        i += 1
        # update estimate
        x += d
        # get new direction
        d = -1.0*asarray(np.matmul(np.linalg.inv(ddf(x)),df(x)))[0]

        #print i, f(x), np.linalg.norm(d), x[:], d[:]
        traj.append((x[0],x[1]))

    return x[:],traj
```

```python
x, y = np.meshgrid(np.linspace(-1.1,1.1,50), np.linspace(-1.1,1.1,50))
Qxy = [[Q(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
figure(figsize=(12,8))
contour(x, y, Qxy, linspace(0, 0.1, 6), color="k")
title("Quadratic")
axis('equal')
xmin, traj = newton(Q, dQ, ddQ, np.array([-1.75, -0.95]), 1e-2, 1e-2)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

## Convergence of Newton-Raphson method

- It is a __quadratically convergent__ method ($p=2$)

$$ \left\| x_{k+1} - x^*\right\| \leq \beta \left\| x_k - x^*\right\|^2 $$

- Therefore, it converges to the minimum in one iteration in a quadratic function
- It needs more iterations in a non-quadratic nonlinear function

Newton-Raphson method is not very robust:
- It may never converge if the initial guess is not appropriate
<!--- Once you are quite close to the answer (in the _asymptotic limit_)--->
- If $H(x)$ is not positive definite, the method can converge to a maximum, or a saddle point, etc.
- It can take the iterations away from a minimum

Example with Rosenbrock function
- The gradient is
$$ \nabla f(x_1,x_2)
= \begin{bmatrix}  -2(1-x_1)-4ax_1(x_2-x_1^2) \\ 2a(x_2-x_1^2) \end{bmatrix}
$$
- The Hessian is
\begin{align}\nabla^2 f(x_1,x_2)
&= \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1\partial x_2}\\ \frac{\partial^2 f}{\partial x_2\partial x_1}& \frac{\partial^2 f}{\partial x_2^2} \end{bmatrix}\\
&= \begin{bmatrix} \frac{\partial \left(-2(1-x_1)-4ax_1(x_2-x_1^2)\right)}{\partial x_1} \;\;& \frac{\partial \left(-2(1-x_1)-4ax_1(x_2-x_1^2)\right)}{\partial x_2}\\ \frac{\partial \left(2a(x_2-x_1^2)\right)}{\partial x_1}& \frac{\partial \left(2a(x_2-x_1^2)\right)}{\partial x_2} \end{bmatrix}\\
&=  \begin{bmatrix} 2-4a(x_2-x_1^2)+8ax_1^2 \;\;& -4ax_1\\ -4ax_1 &  2a \end{bmatrix}
\end{align}
 If the function has continuous second derivatives (such as the function of this example), the mixed partial derivatives are equal to each other.
 - Hence, the Hessian is symmetric (aside: [Schwarz's theorem](https://en.wikipedia.org/wiki/Symmetry_of_second_derivatives#Schwarz's_theorem))

```python
def ddRosen(x,a=10):
    return np.matrix([[2.0 - 4.0*a*(x[1]-x[0]**2) - 4.0*a*(-2.0*x[0])*x[0], -4.0*a*x[0]],
                      [-4.0*a*x[0]                                        ,   2.0*a ]])
```

```python
x,y = np.meshgrid(np.linspace(-1.7,1.7,100), np.linspace(-0.8,3,100))
R = [[Rosen(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]

fig = figure(figsize=(12,8))
contour(x, y, R, np.logspace(-1,2, 8), color="k")
title("Rosenbrock - Newton's method")

xmin, traj = newton(Rosen, dRosen, ddRosen, np.array([-1.3, 1.3]), 1e-1, 1e-8)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

## Newton-Raphson method with line search

- This is the natural extension to the previous method
  - The step size $\alpha_k$ is found by a line search at each iteration

<!---$$
x_{k+1} = x_k + \alpha_k d_k
$$
With the direction given by
$$
d_k = - [H(x_k)]^{-1} \nabla f(x_k)
$$
Optimise $\alpha$ in each iteration. --->

- This method
 - often speeds up the convergence
  - will find a minimum, not a saddle point or a maximum (when it converges)
  - can fail (try the initial guess $[-1.5,1.5]$ in the code)<!---, direction points uphill!--->
  - "Trust region" methods do not allow too big a step, which reduces the failure of the method

```python
def newton_raphson(f, df, ddf, x0, ftol, xtol=1e-3):
    traj = []

    # get initial direction
    x = np.array(x0[:])
    d = -1.0*asarray(np.matmul(np.linalg.inv(ddf(x)),df(x)))[0]

    i = 0
    #print i, f(x), np.linalg.norm(d), x[:], d[:]
    traj.append((x[0],x[1]))

    alpha_init = 1e-6

    # loop until convergence
    while np.linalg.norm(d) > ftol:
        i += 1
        # do line search in current direction
        alpha = linesearch_golden_section(f, x, d, alpha_init, xtol)
        #print alpha
        # update estimate
        x += alpha*d
        # get new direction
        d = -1.0*asarray(np.matmul(np.linalg.inv(ddf(x)),df(x)))[0]

        #print i, f(x), np.linalg.norm(d), x[:], d[:]
        traj.append((x[0],x[1]))
        print x
        w, v = np.linalg.eig(ddf(x))
        print("Eigenvalues:", w)

    return x[:],traj

```

```python
x,y = np.meshgrid(np.linspace(-1.7,1.7,100), np.linspace(-0.8,3,100))
R = [[Rosen(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]

fig = figure(figsize=(12,8))
contour(x, y, R, np.logspace(-1,2, 30), color="k")
title("Rosenbrock - Newton-Raphson method")

xmin, traj = newton_raphson(Rosen, dRosen, ddRosen, np.array([-1.3, 1.3]), 1e-1, 1e-8)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

## Caveat on Newton-type methods

- Computing and inverting the Hessian requires
    - Computation $\sim O(N^2)$
    - Inversion   $\sim O(N^3)$
- In engineering applications, $N$ can be thousands, millions, billions
- If $H$ is sparse, this helps speed up computation and inversion

## Barzilai-Borwein method

- Reminder: The __secant method__  approximates the Hessian by finite difference to yield

$$
x = x_1 - f'(x_1) \frac{x_1-x_0}{f'(x_1)-f'(x_0)}
$$

- There is a multidimensional generalisation (Barzilai & Borwein, IMA J. Num. Analys., __8(1)__, 141-148, 1998)

$$
x = x_1 - \nabla f(x_1) \frac{(x_1-x_0)^T\left(\nabla f(x_1)-\nabla f(x_0)\right)}{\left|\nabla f(x_1)-\nabla f(x_0))\right|^2}
$$

- Initialised with a single steepest descent step
- Only requires a single evaluation of the gradient in each iteration
- Usually more stable than Newton's method and other Quasi-Newton methods
- Not much is known about its convergence

```python
def bb(f, df, x0, dftol, xtol):
    traj = []

    # get initial direction
    x = x0[:]
    alpha_init = 1e-1
    i = 0
    traj.append((x[0],x[1]))

    # one step of steepest descent
    x1 = x[:]
    df1 = df(x1)
    #print i, f(x1), np.linalg.norm(df1), x1[:], df1[:]
    alpha = linesearch_golden_section(f, x1, -1.0*df1, alpha_init, xtol)
    x2 = x1-alpha*df1
    df2 = df(x2)
    i += 1
    #print i, f(x2), np.linalg.norm(df2), x2[:], df2[:]

    traj.append((x2[0],x2[1]))
    # loop until convergence
    while np.linalg.norm(df2) > dftol:
        i += 1
        # update estimate
        x = x2 - df2 * np.dot(x2-x1,df2-df1)/np.linalg.norm(df2-df1)**2
        # get new direction
        x1 = x2[:] ; df1 = df2[:]
        x2 = x[:]  ; df2 = df(x)

        #print i, f(x), np.linalg.norm(d), x[:], d[:]
        traj.append((x[0],x[1]))

    return x[:],traj

x,y = np.meshgrid(np.linspace(-1.7,1.7,100), np.linspace(-0.8,3,100))
R = [[Rosen(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
```

```python
fig = figure(figsize=(12,8))

#fig, ax = plt.subplots(figsize=(16,12))
#CS = ax.contour(x, y, R)
#ax.clabel(CS, inline=1, fontsize=10)
contour(x, y, R, np.logspace(-1,2, 10), color="k")
title("Rosenbrock - Barzilai-Borwein method")

xmin, traj = bb(Rosen, dRosen, np.array([-1.3, 1.3]), 1e-1, 1e-8)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

## Comparison of the methods for a quadratic function

- Consider minimizing the quadratic function $f(x) = \frac12 x^T A x - b^T x$
- Example with quadratic function
\begin{align}
f(x_1,x_2) & =x_1^2+x_2^2-1.9x_1x_2\\
 & = \frac{1}{2}\begin{bmatrix} x_1 \;\;\;x_2\end{bmatrix} \begin{bmatrix} 2 & -1.9\\ -1.9& 2\end{bmatrix}  \begin{bmatrix} x_1 \\x_2\end{bmatrix}\\
\end{align}

```python
x, y = np.meshgrid(np.linspace(-1.1,1.1,50), np.linspace(-1.1,1.1,50))
Qxy = [[Q(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
figure(figsize=(12,8))
contour(x, y, Qxy, linspace(0, 0.1, 6), color="k")
title("Quadratic")
axis('equal')
#axis([-1.5, 1.5, -1.1, 1.1])

xmin, traj = steepest_descent(Q, dQ, np.array([-0.75, -0.95]), 1e-2, 1e-2)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o", label="Steepest Descent")

xmin, traj = newton(Q, dQ, ddQ, np.array([-0.75, -0.95]), 1e-2, 1e-2)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "b-o", label="Newton")

xmin, traj = bb(Q, dQ, np.array([-0.75, -0.95]), 1e-2, 1e-2)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "g-o", label="Barzilai-Borwein")

legend()

show()
```

- The steepest descent takes many steps (''zig-zagging'')
 - This behaviour deteriorates further as the condition number increases (longer-narrower valleys)
- Newton's method converges in one iteration because the order of convergence is $p=2$
 - This is because the Taylor series of a quadratic function is _exact_ at second order
- The Barzilai-Borwein method is a good compromise
 - Takeaway: An approximated Hessian can greatly improve convergence

# Conjugate Gradients

- Example with quadratic function
\begin{align}
f(x_1,x_2) & =x_1^2+x_2^2-1.9x_1x_2\\
 & = \frac{1}{2}\begin{bmatrix} x_1 \;\;\;x_2\end{bmatrix} \begin{bmatrix} 2 & -1.9\\ -1.9& 2\end{bmatrix}  \begin{bmatrix} x_1 \\x_2\end{bmatrix}\\
\end{align}

```python
x, y = np.meshgrid(np.linspace(-1,1,50), np.linspace(-1,1,50))
Qxy = [[Q(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
figure(figsize=(9,6))
contour(x, y, Qxy, linspace(0, 0.1, 6), color="k")
title("Quadratic")
axis('equal')

plot([-0.75, -0.85, 0], [-0.95, -0.85, 0], "r-o")

show()
```

- Example: $$ f(x, y) = x^2 + 50 y^2$$
 - The Hessian is $H=\nabla (\nabla f(x,y,))=\nabla \begin{pmatrix}2x\\ 100y\end{pmatrix}=\begin{pmatrix}2& 0\\0& 100\end{pmatrix}$
 - The eigenvalues are $\lambda_1=2$, $\lambda_2=100$
 - The condition number is $k=\lambda_{max}/\lambda_{min}=50$.
 - The eigenvectors for $\lambda_1$ and $\lambda_2$ are, respectively $$ \begin{pmatrix} 1\\0\end{pmatrix}\;\;\; \textrm{and}\;\;\;\begin{pmatrix} 0\\1\end{pmatrix}$$
 - The direction of maximum variation is $y$, hence, the ellipsoid is elongated in the $x$ direction

```python
x, y = np.meshgrid(np.linspace(-1,1,50), np.linspace(-0.2,0.2,50))
def aQ(x):
    return x[0]**2+50*x[1]**2
Qxy = [[aQ(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
figure(figsize=(9,1.5))
contour(x, y, Qxy, linspace(0, 1, 6), color="k")
title("Quadratic")
axis('equal')

plot([-0.75, -0.0, 0], [-0.095, -0.095, 0], "r-o")

show()
```

## Conjugate Gradients as a search method

- If we could step along the eigendirections, the convergence would be faster than steepest descent's
- Eigendirections are good search directions even in ill-conditioned quadratic functions because they represent the principal axes of ellipsoids
 - This is mathematically a good idea, but computationally the calculation of eigenvectors is $\sim O(N^3)$, which is the same as computing the Hessian.
 - Therefore, we do not want to use eigenvectors

<!--- - Things would _also_ be OK, if the quadratic was along the axes (i.e. $A$ is a diagonal matrix), and we stepped along the axis directions: --->

- Can we find a new coordinate system in which the search directions are orthogonal to each other?

Consider a set of directions $\{d_i\}$ for which

$$
d_i^T A d_j = 0 \quad \mathrm{for}\quad i\neq j
$$
- This is called __conjugacy__. (If $\{d_i\}$ are eigenvectors, then by definition $Ad_i=\lambda_id_i$, where $\lambda_i$ is the eigenvalue. Therefore $d_i^TAd_j=\lambda_i\lvert\lvert d_i^Td_i\lvert\lvert$. This shows that eigenvectors fullfil the conjugacy condition. Therefore, if we start from an eigendirection, the conjugacy condition will find the next eigenvector, and so on.)

- Collect the directions into a matrix,
$$ S =
\begin{bmatrix}
\uparrow & \uparrow & \uparrow & \uparrow \\
d_1 & d_2 &\ldots& d_N \\
\downarrow & \downarrow & \downarrow & \downarrow
\end{bmatrix}
$$

- Transform the coordinates into $y = S^{-1} x$, and set up the problem in the new coordinate system $y$:

$$ f(x) = f(S y) = \frac12 (Sy)^T A(Sy) - b^T Sy = \frac12 y^T (S^T A S) y - (S^T b)^T y$$

\begin{align}
S^T A S & =
\begin{bmatrix}
\uparrow & \uparrow & \uparrow & \uparrow \\
d_1 & d_2 &\ldots& d_N \\
\downarrow & \downarrow & \downarrow & \downarrow
\end{bmatrix}^T
A
\begin{bmatrix}
\uparrow & \uparrow & \uparrow & \uparrow \\
d_1 & d_2 &\ldots& d_N \\
\downarrow & \downarrow & \downarrow & \downarrow
\end{bmatrix}
& \\
&\\
&=\begin{bmatrix} d_1^TAd_1 & d_1^TAd_2 &\ldots & d_1^TAd_N \\
d_2^TAd_1 & d_2^TAd_2 & \ldots & d_2^TAd_N \\
\vdots & \vdots & \ddots &\vdots\\
d_N^TAd_1 & d_N^TAd_2&\ldots&d_N^TAd_N
\end{bmatrix}
& \\
&\\
&=\begin{bmatrix} d_1^TAd_1 & 0 &\ldots & 0 \\
0 & d_2^TAd_2 & \ldots & 0 \\
\vdots & \vdots & \ddots &\vdots\\
0 & 0&\ldots&d_N^TAd_N
\end{bmatrix}
\end{align}

 - The matrix $S^T A S$ is diagonal because of the the conjugacy condition we imposed
 - This is the critical property, which is central to the conjugate gradient

- The problem in the $y$ space is a quadratic form along the axes
 - The axes of the $y$ space are the search directions $\{d_i\}$.

- How can we construct such a set of directions (ruling out the eigendirections, which would require the diagonalization of $A$)?

## Example on the difference between eigenvectors and conjugate vectors

- Consider the quadratic form expressed in the $x$ coordinate system
$$ f(x) = \frac{1}{2} x^T \underbrace{\begin{bmatrix} 3 & -1\\ -1& 3\end{bmatrix}}_{=A} x$$

- First, we discuss the eigendecomposition
\begin{align}
\Lambda & = Q^{-1}AQ \\
& \\
\begin{bmatrix}2 & 0\\ 0 & 4 \end{bmatrix}
&=
\begin{bmatrix}\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}} &\frac{1}{\sqrt{2}}\end{bmatrix}
A
\begin{bmatrix}\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}}\\\frac{1}{\sqrt{2}} &\frac{1}{\sqrt{2}}\end{bmatrix}
\end{align}
 - The quadratic form in the space spanned by eigenvectors has coordinates $z$ and is diagonal
\begin{align}
f(Qz) & = \frac{1}{2} z^TQ^T A Qz\\
& = \frac{1}{2} z^T \Lambda z
\end{align}
- Second, we discuss the conjugate directions. A matrix $S$ of conjugate directions that are not eigenvectors is
\begin{align}
S & = \begin{bmatrix}1 & -5\\ 2 & 1 \end{bmatrix}
\end{align}
The first column is $d_1$, the second column is $d_2$, which are conjugate to each other. Thus,
\begin{align}
D = S^T A S & = \begin{bmatrix}1 & 2\\ -5 & 1 \end{bmatrix} A \begin{bmatrix}1 & -5\\ 2 & 1 \end{bmatrix}\\
& = \begin{bmatrix}11 & 0\\ 0 & 88 \end{bmatrix}
\end{align}
 - The quadratic form in the space spanned by conjugate directions has coordinates $y$ and is diagonal
\begin{align}
f(y) & = \frac{1}{2} y^TS^T A Sy\\
& = \frac{1}{2} y^T D y
\end{align}
 Although diagonal, matrix $D$ does not contain the eigenvalues of $A$, thus, the conjugate directions $d_1$ and $d_2$ are not the  eigenvectors of $A$.

## Finding the conjugate directions with the conjugate gradient method iteratively

- At iteration $k$, we have the gradient $\nabla f(x_k)$ and the previous search directions $\{d_1 \ldots d_{k-1}\}$.

- We construct the new search direction as

$$
d_k = -\nabla f(x_k) + \beta_k d_{k-1}
$$

 Note that $d_k$ has "memory" of the previous direction $d_{k-1}$
- We impose the conjugacy condition
\begin{align}
0 &= d_{k-1}^T A d_k\\
0 &= d_{k-1}^T A (-\nabla f(x_k) + \beta_k d_{k-1})
\end{align}
  which yields

$$\beta_k = \frac{d_{k-1}^T A \nabla f(x_k)}{d_{k-1}^T A d_{k-1}}$$
- It can be shown that with this choice of $\beta_k$, the search direction $d_k$ is conjugate to _all_ the previous directions

Algorithm:

1. Initialise with steepest descent $d_0 = -\nabla f(x_0)$
1. Update location and determine $\alpha_k$ with line search $$x_{k+1} = x_k + \alpha_k d_k$$
1. Find new search direction: with this choice of $d_0$, it can be shown that  $$d_{k+1} = -\nabla f(x_{k+1}) + \left[\frac{\vert\lvert\nabla f(x_{k+1})\vert\lvert}{\vert\lvert\nabla f(x_k)\vert\lvert}\right]^2 d_k$$
 (Optional: A derivation of the identity $\beta_k = \left[\frac{\vert\lvert\nabla f(x_{k+1})\vert\lvert}{\vert\lvert\nabla f(x_k)\vert\lvert}\right]^2$ can be found on pages 270-271 in Luenberger, D. & Ye, Y. Linear and Nonlinear Programming, third edition, Springer. See the course booklist.)
1. Back to 2 until convergence.

```python
def conjugate_gradients(f, df, x0, ftol, xtol=1e-3, n_restart=None):
    traj = []

    # get initial direction
    x = x0[:]
    df_new = df(x)
    d = -1.0*df_new
    alpha_init = 1e-6
    i = 0
    #print i, f(x), np.linalg.norm(d), x[:], d[:]
    traj.append((x[0],x[1]))

    # loop until convergence
    while np.linalg.norm(df_new) > ftol:
        i += 1
        # do line search in current direction
        alpha = linesearch_golden_section(f, x, d, alpha_init, xtol)
        # update estimate
        x += alpha*d
        # get beta
        df_old = df_new[:]
        df_new = df(x)
        beta = (np.linalg.norm(df_new)/np.linalg.norm(df_old))**2
        # get new direction
        if n_restart == None or i%n_restart != 0:
            d = beta*d
        d += -1.0*df_new

        #print i, f(x), np.linalg.norm(d), x[:], d[:]
        traj.append((x[0],x[1]))

    return x[:],traj

```

```python
x, y = np.meshgrid(np.linspace(-1,1,50), np.linspace(-1,1,50))
Qxy = [[Q(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]
figure(figsize=(12,12))
contour(x, y, Qxy, linspace(0, 0.1, 6), color="k")
title("Quadratic - Conjugate Gradients")
axis('equal')

xmin, traj = conjugate_gradients(Q, dQ, np.array([-1.01, -1.03]), 1e-2, 1e-8)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")

show()
```

```python
x,y = np.meshgrid(np.linspace(-1.7,1.7,100), np.linspace(-0.8,3,100))
R = [[Rosen(np.array([x[j,i],y[j,i]])) for i in range(size(x,0)) ] for j in range(size(x,1))]

fig = figure(figsize=(12,8))
contour(x, y, R, np.logspace(-1,2, 8), color="k")
title("Rosenbrock - Conjugate Gradients")
xmin, traj = conjugate_gradients(Rosen,dRosen, np.array([-1.3,1.3]), 1e-1, 1e-8,2)
plot([traj[i][0] for i in range(len(traj))], [traj[i][1] for i in range(len(traj))], "r-o")
show()
```

## Conjugate gradients final remarks

- It converges after $N$ iterations for an $N$ dimensional quadratic form with exact line search
- For a non-quadratic nonlinear function, search directions are approximately conjugate
- Numerical errors lead to _loss of conjugacy_
 - The algorithm can be periocally _reset_ by setting $d_k=-\nabla f(x_k)$
- The convergence is linear, $$ \|x_{k+1}-x^*\| = \beta \|x_k - x^*\|$$
 - But $\beta$ can be very small, $\beta \approx 0$. This is called "superlinear convergence"
- For large ill-conditioned problems, it can be shown that $$\beta \approx 1-\frac{2}{\sqrt{\kappa}}$$ where $\kappa$ is the condition number.
 - $\beta$ is smaller than the steepest descent's
- The CG method does _not_ require the Hessian and its inversion
- The CG method was originally invented to solve linear systems $Ax = b$.
  - Note the relationship with the quadratic problem

\begin{align}
f(x) &= \frac12 x^T A x - b^T x\\
\nabla f(x) &= Ax - b
\end{align}

## The conjugate gradient method is a Krylov subspace method

Consider the first two iterations

1. Start at $x_0$
2. $d_0 = b-A x_0$
3. $x_1 = x_0 + \alpha_1 d_0$
4. \begin{align}
d_1 &= b-A x_1  + \beta_1 d_0 \\
& = b-A ( x_0 + \alpha_1 d_0 ) + \beta_1 d_0  \\
& = b-A ( x_0 + \alpha_1 (b-A x_0 ) ) + \beta_1 (b-A x_0 ) \\
\end{align}

- The direction after $k$ steps is a linear combination of the vectors

$$
A x_0, A^2 x_0, A^3 x_0, \ldots, A^k x_0
$$

- The corresponding subspace is called __Krylov subspace__
- We often solve $Ax = b$ in the Krylov subspace for large matrices
- There is a large variety of methods called "Krylov subspace methods".
  - The conjugate gradient is one of them.

## Summary of multi-dimensional search-direction methods

| Method | Advantages | Disadvantages
|:-|:-|:-
| Steepest descent | Only needs gradient | Linear convergence<br>slow for ill-conditioned problem |
| Newton-Raphson  | Quadratic Convergence | Needs Hessian<br>can diverge |
| Newton-Raphson with line search | Quadratic Convergence<br> more reliable | Needs Hessian |
| Conjugate gradient | Faster than Steepest Descent | Linear convergence |
| Barzilai-Borwein | Often very stable,<br>faster than Steepest Descent | Convergence analysis<br>difficult in general |

## Least squares fitting of a model to data

One common optimization task involves a cost function that enables a simplification.
Let

- $y_j$ be the measured data, $j = 1,2,\ldots m$
- $\phi(x)$ be the (generally nonlinear) model we are fitting, with $\phi_j(x)$ being the  model's prediction for $y_j$
- $x = [x_1,\ldots,x_n]^T$ be the vector of independent parameters, which can be tuned for the model to fit the data

The elements of the __residual__ vector $r(x)$ are the errors in the model's predictions for each piece of data

$$ r_j(x) = \phi_j(x) - y_j $$

- For all the pieces of data, the total error is

$$ f(x) = \sum_{j=1}^m r_j^2(x)  = r(x)^T r(x) $$

- The optimization task is to minimize $f(x)$.

The gradient of $f(x)$ is
\begin{align}
\nabla f(x) &= \frac{\partial f}{\partial x_i}\;\;\;i=1,2,\ldots,m\\
& = \frac{\partial}{\partial x_i}\left(\sum_{j=1}^m r_j(x)^2\right)\;\;\;i=1,2,\ldots,m \\
& = 2 \sum_{j=1}^m \frac{\partial r_j(x)}{\partial x_i} r_j(x) \;\;\;i=1,2,\ldots,m \\
&= 2 J(x)^T r(x)
\end{align}

where $J(x)$ is the __Jacobian matrix__ of $r(x)$

$$
J(x) =
\begin{bmatrix}
\frac{\partial r_1}{\partial x_1} & \cdots & \frac{\partial r_1}{\partial x_n}\\
\vdots & \ddots & \vdots\\
\frac{\partial r_m}{\partial x_1} & \cdots & \frac{\partial r_m}{\partial x_n}\\
\end{bmatrix}
$$

The Hessian matrix $H(x)$ of $f(x)$ is expressed as

$$ H(x) = \nabla (\nabla f(x)) = 2 J(x)^T J(x) + 2 \sum_{j=1}^m r_j(x) R_j(x) $$

where $R_j(x)$ is the Hessian of the residual $r_j(x)$, i.e., $R_j(x) = \nabla(\nabla r_j(x))$

## The Gauss-Newton method

- Let us assume that the residuals at the optimum $r_j(x^*)$ are small, i.e., the model gives a good fit
- Therefore $R_j$ can be neglected. The Hessian $H(x)$ can be approximated as
$$H(x) \approx \tilde H(x) = 2J(x)^T J(x)$$
for $x$ near the minimum $x^*$, with $\tilde H(x)$ being positive definite.

- In the Gauss-Newton method, the search direction is then
$$
\begin{array}
~d_k &= -\left[\tilde H(x_k)\right]^{-1} \nabla f(x_k)\\
 &= - \left[ 2J(x_k)^T J(x_k)\right]^{-1} \nabla f(x_k) \\
  &= - \frac{1}{2}\left[J(x_k)^T J(x_k)\right]^{-1} 2J(x_k)^T r(x_k) \\
 &= -J(x_k)^{+} r(x_k)\\
\end{array}
$$
where $J^+=\left[J(x_k)^T J(x_k)\right]^{-1} J(x_k)^T$ is the pseudoinverse matrix.

-  The sequence of estimates is $$x_{k+1} = x_k + d_k$$

In order to find the search directions $d_k$ at each iteration, the following _linear problem_ needs to be solved,
\begin{align}
d_k& = -J(x_k)^{-1}[J(x_k)^{T}]^{-1}J(x_k)^Tr(x_k)\\
&=-J(x_k)^{-1}r(x_k)
\end{align}
Therefore
$$J(x_k) d_k = -r(x_k)$$
which is usually easy.

- The rank of $J$ is the smallest between the number of fitting parameters, $n$, and the pieces of data, $m$.
- We do not need to invert any large ill-conditioned matrix
- The Gauss-Newton methods converges nearly quadratically, if the residuals are small

## You have now the tools to do all examples paper 3

## Answers to common questions asked by students after lectures 3 and 4

1. The search directions of the steepest descent in the Rosenbrock function example do not _look_ orthogonal to each other in the final plot. Why?
 - With an exact line search, the steepest descent directions are orthogonal to each other.
 - The directions do not look orthogonal to each other in the plot, but this is only an "optical illusion"
 - Make the aspect ratio of the figure equal, and you will see the orthogonality. E.g.,  use fig = figure(figsize=(12,12))

1. Consider the minimisation of an $N$-dimensional function where $N>2$. Is the line search an $N-1$-dimensional search?
 - No, it is not. The line search is _always_ a one-dimensional  minimisation problem.

1. Can the algorithms of these lectures find global minimum?
 - These algorirthm are designed to solve $\nabla f(x)=0$, therefore, they are designed to find stationary points.
 - If the function is convex, the local minimum is the global minimum.

1. Are the conjugate directions eigenvectors of the Hessian?
 -    Only in one case: when we choose the eigenvectors as conjugate directions. The eigenvectors diagonalise the Hessian _and_ the terms of the diagonal are the eigenvalues of the Hessian.
  (Remember that an eigenvalue $\lambda$ is defined by the equation $\det(A-\lambda I)=0$ where $I$ is the identity matrix.)
 - If the conjugate directions are not eigenvectors, which is the typical case, the answer is no. The conjugate directions diagonalise the Hessian _but_ the terms of the diagonal are not the eigenvalues of the Hessian.

5. How can we derive $\beta_k$ for the conjugate gradient method? (This is optional.)

   For brevity, we define $g_{k}:=\nabla f(x_k)=Ax_k-b$. From the conjugate gradient method, $g_{k+1}^T g_{k}=0$ and $g_k^Td_i=0$ if $i<k$, which are the two properties we use for this proof. So:
   \begin{align}
   g_{k+1} &= Ax_{k+1} - b \\
   & = A(x_{k+1}-x_k) + Ax_k - b \\
   & = A(\alpha_kd_k) + g_k
   \end{align}
   Take the inner product with $g_{k+1}$
   \begin{align}
   g_{k+1}^T g_{k+1} &= \alpha_k g_{k+1}^T Ad_k + \underbrace{g_{k+1}^Tg_k}_{=0} \\
   &= \left(-\frac{g_k^Td_k}{d_k^T A d_k}\right) g_{k+1}^T Ad_k
   \end{align}
   Hence
      \begin{align}
   -\frac{g_{k+1}^T g_{k+1}}{g_k^T d_k}
   &= \frac{g_{k+1}^T Ad_k}{d_k^T A d_k}
   \end{align}
   Note that $$ g_k^T d_k = g_k^T (-g_k + \beta_{k}d_{k-1})=-g_k^Tg_k\;\;\;\;\textrm{because we start with $d_0=-g_0$}$$ therefore
    \begin{align}
   \frac{g_{k+1}^T g_{k+1}}{g_k^T g_k}
   &= \frac{g_{k+1}^T Ad_k}{d_k^T A d_k}
   \end{align}
   We recognize the right-hand side as $\beta_{k+1}$, which proves the identity.


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section10.2, chauffage d'un conducteur électrique

>  Le développement fait dans Transport Phenomena sera répété ici en développant les solutions avec le calculateur formel sympy et en traçant la solution avec sympy.plot. Les développements sont très similaires à la section 2.3, Les coordonnées cylindriques sont utilisées, évidemment. Remarquez bien comment la courbure de la géométrie a un effet sur le bilan.

```python
#
# Pierre Proulx
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
%matplotlib inline
```

```python
# Paramètres, variables et fonctions
#
r,delta_r,L,k,S_c,R=sp.symbols('r,delta_r,L,k,S_c,R')
C1=sp.symbols('C1')
qr=sp.symbols('q_r')
T=sp.Function('T')(r)
```

```python
#
# Équation du bilan d'énergie
#
eq=2*sp.pi*r*L*qr(r)-2*sp.pi*(r+delta_r)*L*qr(r+delta_r)    # conduction
eq=eq+2*sp.pi*r*delta_r*L*S_c                                #source
#
# pose de la limite quand dr tends vers 0
#
eq=eq/(2*sp.pi*L*delta_r)
display(eq)
```

```python
eq0=sp.Limit(eq,delta_r,0)
display(eq0)
eq0=sp.limit(eq,delta_r,0)    # encore une fois, remarquez ici la différence entre les fonctions Limit et limit
display(eq0)
eq1=eq0.doit()
display(eq1)
```

```python
eq2=sp.dsolve(eq1)
display(eq2)
```

```python
# Loi de Fourier.
Fourier=-k*sp.Derivative(T,r)
eq2=eq2.subs(qr(r),Fourier)
display(eq2)
```

```python
eq2=sp.dsolve(eq2)
display('Le profil de température général, avant de poser les conditions aux limites',eq2)
eq2=eq2.rhs #
```

```python
T_0=sp.symbols('T_0')
condition_1=sp.Eq(eq2.diff(r).subs(r,0),0)     # flux nul en r=0
condition_2=sp.Eq(eq2.subs(r,R),T_0)           # température connue en r=R
display('Les conditions aux limites, en r=0 flux nul et en r=R T=T_0',condition_1,condition_2)
constante=sp.solve([condition_1,condition_2],sp.symbols('C1,C2'),dict=True)

display('Les constantes',constante[0])  # les constantes apparaissent dans une liste de 1 élément, dans un dictionnaire
constante=constante[0] #
```

```python
T=eq2.subs(constante)
display(T)
```

```python
I,k_e=sp.symbols('I k_e')
source=I**2/k_e
T=T.subs(S_c,source)
display(T.simplify())
```

> #### Exemple 10.2-2
>> La différence est maintenant que la condition à la paroi est la loi de refroidissement de Newton au lieu de température fixée à T_0 , c'est l'équation 10.2-22:

>><center>
en r=R, $-k\frac {dT}{dr}=h(T-T_{air})$
</center>

>> On peut reprendre la solution au moment ou on a posé cette condition

```python
# constante=sp.solve(sp.Eq(eq2.rhs.subs(r,R),T_0),'C1',dict=True)   # T=T_0 à la paroi
#
# remplacé par
#
eqx=eq2
display(eq2)
T_air,h=sp.symbols('T_air h')
condition_1=sp.Eq(-k*eqx.diff(r).subs(r,R)-h*(eqx.subs(r,R)-T_air),0)
condition_2=sp.Eq(eq2.diff(r).subs(r,0),0)
display('Les conditions aux limites, en r=R flux de Fourier=flux de Newton et flux = 0 en r=0',
        condition_1,condition_2)
constanten=sp.solve([condition_1,condition_2],sp.symbols('C1,C2'),dict=True)
constanten=constanten[0]            # sort de la liste
display('les constantes avec une condition de coefficient de transfert en r=R',constanten)
T=eq2.subs(constanten)
display(T)
```