```python
#%matplotlib nbagg
from __future__ import division
from IPython.display import HTML
from IPython.display import display
from scipy.optimize import *
import scipy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

## Question 1: Poiseuille's method for determining viscosity

The  volume flow rate, ${\displaystyle\frac{{\rm d}V}{{\rm d}t}}$, of fluid flowing smoothly through a horizontal tube of length $L$ and radius $r$ is given by Poiseuille's equation:
\begin{equation}
\frac{{\rm d}V}{{\rm d}t}=\frac{\pi\rho g h r^4}{8\eta L},
\end{equation}
where $\eta$ and $\rho$ are the viscosity and density, respectively, of the fluid,  $h$ is the head of pressure across the tube, and $g$ the acceleration due to gravity.
<bf>
In an experiment the graph of the flow rate versus height has a slope measured to 7%, the length is known to 0.5%, and the radius to 8%.
<bf>
Required:
<bf>

(i) What is the fractional precision to which the viscosity is known?
<bf>
<bf>
(ii) If more experimental time is available, should this be devoted to
>(a) collecting more flow-rate data,
<bf>
>(b) measuring the length,
<bf>
>(c) the radius of the tube?

### (i) What is the fractional precision to which the viscosity is known? Express your answer as a DECIMAL.

```python
def one_i():
    'Returns fractional precision of viscosity'

    fractional_precision = np.sqrt((16*0.08**2)+(0.07**2)+(0.005**2))
    percentage_error = fractional_precision*100

    return fractional_precision
one_i()
```

### (ii) If more experimental time is available, should this be devoted to
>(a) collecting more flow-rate data,
<bf>
>(b) measuring the length,
<bf>
>(c) the radius of the tube?

```python
def one_ii():
    '''Your function should return a string of A,B or C'''

    comment = 'C'

    return comment
one_ii()
```

## Question 2: Functional error approach for Van der Waals calculation

The Van der Waals equation of state is a correction to the ideal gas law, given by the equation,

\begin{equation}
(P+\frac{a}{V_m^2})(V_m-b) = RT,
\end{equation}

where $P$ is the pressure, $V_m$ is the molar volume, $T$ is the absolute temperature, $R$ is the universal gas constant with $a$ and $b$ being species-specific Van der Waals coefficents.

A sample of Nitrogen was measured in an experiment as,
<bf>
>Molar Volume $V_m$ = $(2.000 \pm 0.003)$x$10^{-4}m^3mol^{-1}$
<bf>
>Absolute Temperature $T$ = $298.0\pm0.2K$

and the constants are,

<bf>
$a$ = $(1.408$x$10^{-1}) m^6mol^{-2}Pa$
<bf>
$b$ = $(3.913$x$10^{-5}) m^3mol^{-1}$
<bf>
$R$ = $(8.3145) JK^{-1}mol^{-1}$
<bf>

Required:
<bf>
>(i) From the given data, calculate the pressure giving your answer in MPa.
<bf>
>(ii) Calculate the uncertainty in the pressure by using the functional approach for error propagation.
<bf>
>(iii) Repeat the calculations above for
>>$V_m = (2.000\pm0.003)\times10^{-3}\,{\rm m}^{3}\,{\rm mol}^{-1}$ and  $T=400.0 \pm 0.2K$.

### (i) From the given data, calculate the pressure giving your answer in MPa.

```python
def two_i():
    '''Your function should return the pressure in MPa'''

    R = 8.3145
    T,alpha_t = 298,0.2
    v_m,alpha_vm = 2*10**-4,0.003*10**-4
    b = 3.913*10**-5
    a = 1.408*10**-1
    # Note: the answer is divided by 10**6 in order to have the correct units
    pressure = (((R*T)/(v_m-b))-(a/(v_m**2)))/10**6
    return pressure
two_i()
```

### (ii) Calculate the uncertainty in the pressure by using the functional approach for error propagation.

```python
def two_ii():
    '''Your function should return the uncertainty'''

    R = 8.3145
    T,alpha_t = 298,0.2
    v_m,alpha_vm = 2*10**-4,0.003*10**-4
    b = 3.913*10**-5
    a = 1.408*10**-1
    pressure = two_i()*10**6 # pressure is not in Pascals

    alpha_v= ((R*T)/((v_m+alpha_vm)-b))-(a/((v_m+alpha_vm)**2))-pressure
    alpha_t = ((R*(T+alpha_t))/(v_m-b))-(a/(v_m**2))-pressure
    uncertainty = (np.sqrt((alpha_v**2)+(alpha_t**2)))/10**6

    return uncertainty
two_ii()
```

### (iii) Repeat the calculations above for
>$V_m = (2.000\pm0.003)\times10^{-3}\,{\rm m}^{3}\,{\rm mol}^{-1}$ and  $T=400.0 \pm 0.2K$.

```python
def two_iii():
    '''Your function should return both the pressure and the uncertainty'''
    pressure_2 = 0
    uncertainty_2 = 0

    R = 8.3145
    T,alpha_t = 400,0.2
    v_m,alpha_vm = 2*10**-3,0.003*10**-3
    b = 3.913*10**-5
    a = 1.408*10**-1

    pressure_2 = (((R*T)/(v_m-b))-(a/(v_m**2)))
    alpha_v= ((R*T)/((v_m+alpha_vm)-b))-(a/((v_m+alpha_vm)**2))-pressure_2
    alpha_t = ((R*(T+alpha_t))/(v_m-b))-(a/(v_m**2))-pressure_2
    uncertainty_2 = (np.sqrt((alpha_v**2)+(alpha_t**2)))/1000000
    pressure_2 = pressure_2/10**6

    return pressure_2,uncertainty_2
two_iii()
```

## Question 3: Reverse Engineering the Incredible Goal
The separation of the posts is 7.32m, and the ball is struck from a point 22m from the near post and 29m from the far post.

Required:

> (i) Plot a graph of $\alpha_\theta$ on the y-axis vs $\alpha_L$ on the x-axis for the range of values presented in the `alpha_ls` array.
<bf>
>(ii) To what (common) precision must these three lengths be known to justify quoting the angle to 11 significant figures?

*[Hint: use the functional approach using the values for the errors in the length measurements as the provided `alpha_ls` array]*

### (i) Plot a graph of $\alpha_\theta$ on the y-axis vs $\alpha_L$ on the x-axis for the range of values presented in the `alpha_ls` array.

```python
def three_i():
    '''Your function should plot the graph and return the alpha_thetas array.'''

    a = 22
    b = 29
    c = 7.32
    alpha_ls = [0.1, 0.01, 0.001, 0.0001, 1e-05, 1e-06, 1e-07, 1e-08, 1e-09, 1e-10, 1e-11, 1e-12]
    alpha_thetas = []

    # Cosine rule
    theta_rad = np.arccos(((b**2)+(c**2)-(a**2))/(2*b*c))

    for i in alpha_ls:
        error = i
        # Perform the functional approach
        alpha_az = np.arccos(((b**2)+(c**2)-((a+error)**2))/(2*b*c))-theta_rad
        alpha_bz = np.arccos((((b+error)**2)+(c**2)-(a**2))/(2*(b+error)*c))-theta_rad
        alpha_cz = np.arccos(((b**2)+((c+error)**2)-(a**2))/(2*b*(c+error)))-theta_rad
        alpha_theta = np.sqrt(((alpha_az)**2)+((alpha_bz)**2)+((alpha_cz)**2))
        alpha_thetas.append(alpha_theta)
    plt.plot(alpha_ls,alpha_thetas)

    #Plot on a log scale
    plt.yscale('log')
    plt.xscale('log')
    plt.xlabel(r'$\alpha_L$')
    plt.ylabel(r'$\alpha_\theta$')
    plt.show()

    return alpha_thetas
three_i()
```

### (ii) To what (common) precision must these three lengths be known to justify quoting the angle to 11 significant figures?

For the angle to be quoted to 11 significant figures, the error must be in the 10th decimal place (since the angle is in radians and we are assuming the angle to be >1 radian). This is the case first when the error in the angle is 7x$10^{-10}$, which corresponds to a common precision in the lenghts of $10^{-9}$. If the angle were <1 radian then the precision would be $10^{-10}$.

## Question 4: Linear Regression/Weighted Fit

The data  plotted in Fig 6.1(d) relating to the degradation of the signal to noise ratio from a frequency to voltage converter near harmonics of the mains frequency are listed below.
\begin{equation}
\begin{array}{lcccccc}
\hline
{\rm frequency~(Hz)} &10&20&30&40&50&60\\
{\rm voltage~(mV)} &16&45&64&75&70&115\\
{\rm error~(mV)}   &5&5&5&5&30&5\\
\hline
{\rm frequency~(Hz)} &70&80&90&100&110&\\
{\rm voltage~(mV)} &142&167&183&160&221&\\
{\rm error~(mV)}   &5&5&5&30&5&\\
\hline
\end{array}
\end{equation}

This data is also contained in the file 'linear_regression.csv'.

Required:
<bf>
> (i) Calculate the best-fit gradient and intercept and associated errors using a weighted fit. You may use the `curve_fit` function.

```python
data = pd.read_csv('linear_regression.csv')
frequencies = data.iloc[:,0]
voltages = data.iloc[:,1]
voltage_errors = data.iloc[:,2]

def f(frequencies, a,b):
    return a*frequencies+b

def best_fit_params():
    '''Your function should return the gradient,gradient_error,intercept,intercept_error'''
    gradient = 0
    gradient_error = 0
    intercept = 0
    intercept_error = 0

    # Using the curve_fit function
    popt, pcov = curve_fit(f, frequencies, voltages, sigma = voltage_errors)
    perr = np.sqrt(np.diag(pcov))
    gradient = popt[0]
    gradient_error = perr[0]
    intercept = popt[1]
    intercept_error = perr[1]

    return(gradient,gradient_error,intercept,intercept_error)

best_fit_params()
```

## Question 5: Error bars from a $\chi^2$ minimisation
|--|Unweighted | Weighted |
| --- | --- | --- |
| Gradient | (1.9$\pm$0.2)mV/Hz | (2.03$\pm$0.05)mV/Hz |
| Intercept | (0$\pm$1)x10mV | (-1$\pm$3)mV |

Required:
<bf>
>(i) For the data set of the previous question, write your own code to perform a $\chi^2$ minimisation. You may use the `mininmize` function, but not the `curve_fit` function.
<bf>
>(ii) Verify that $\chi^{2}_{\rm{min}}$ is obtained for the same values of the parameters as are listed in the table above.
<bf>
>(iii) By following the procedure of $\chi^2\rightarrow\chi^2_{\rm min}+1$ outlined in Section 6.5 in Hughes and Hase and the figure above, check that the error bars for $m$ and $c$ are in agreement with the table above.  Include explicitly the first 5 steps of the procedure shown in the figure above for the calculation of the slope.

### (i) For the data set of the previous question, write code to perform a $\chi^2$ minimisation.

```python
data = pd.read_csv('linear_regression.csv')
frequencies = data.iloc[:,0]
voltages = data.iloc[:,1]
voltage_errors = data.iloc[:,2]
# Note: it is useful to normalise the data in order to help avoid a large
# dependence on the initial starting position
voltages = voltages
voltage_errors = voltage_errors

def chisqfunc(x):
    'Linear chi squared fitting function'
    a,b = x
    model = a + b*frequencies
    chisq = np.sum(((voltages - model)/voltage_errors)**2)
    return chisq

# Set starting position from which to minimize
x0 = np.array([0,0])

# Use scipy.optimize.minimize
result =  minimize(chisqfunc, x0)
a,b=result.x

# Good practice to plot graph to show the fit to check its quality
plt.errorbar(frequencies,voltages, label = 'data', yerr=voltage_errors,fmt='o',capsize=2.5)
plt.plot(frequencies,a+b*frequencies, label = 'chi squared fit')
plt.xlabel('Frequency (Hz)')
plt.ylabel('Voltage (mV)')
plt.legend()
plt.show()

print ('The gradient is {} mV/Hz and the intecept is {} mV'.format(b,a))
```

Note: the fit is good and the only points further away from the line are the two with very large error bars.

### (ii) Verify that $\chi^{2}_{\rm{min}}$ is obtained for the same values of the parameters as are listed in the table above.

```python
data = pd.read_csv('linear_regression.csv')

def five_ii():
    '''Your function must return the gradient and intercept'''

    gradient = b
    intercept = a

    return gradient,intercept
five_ii()
```

### (iii) By following the procedure of $\chi^2\rightarrow\chi^2_{\rm min}+1$ outlined in Section 6.5 in Hughes and Hase and the figure above, check that the error bars for $m$ and $c$ are in agreement with the table above.  Include explicitly the first 5 steps of the procedure shown in the figure above for the calculation of the slope.

```python
def search(precision, z_min, f_min):

    z_new = z_min
    f_new = f_min
    m_stop = 0
    c_stop = 0
    while(m_stop == 0 or c_stop == 0):
        m_stop = 1
        c_stop = 1

        # There are more optimal ways than doing a while(1) function but this suffices for this solution
        while (1):
            # changing gradient
            z_new[1] = z_new[1] + (1 / precision)

            f_new = chisqfunc(z_new)

            if (f_new - f_min) >= 1 :
                z_new[1] = z_new[1] - (1 / precision)
                break;
            else :
                m_stop = 0

        while (1):
            # changing intercept
            z_new[0] = z_new[0] - (1 / precision)

            f_new = chisqfunc(z_new)

            if (f_new - f_min) >= 1 :
                z_new[0] = z_new[0] + (1 / precision)
                break;
            else:
                c_stop = 0

    return z_new

def five_iii():
    '''
    Adapted from one of the submitted homework solutions - thank you to whoever did this
    The function has been adapted to make it slightly more efficient but I liked the custom search function
    '''
    error_m = 0
    error_c = 0

    x0 = [0,0]
    result_weight = minimize(chisqfunc, x0)

    precision = 1000
    result_shift_weight = search(1000, result_weight.x, result_weight.fun)

    result_weight = minimize(chisqfunc, x0)

    error_m_weight = round(abs(result_shift_weight[1] - result_weight.x[1]), 2)
    error_c_weight = round(abs(result_shift_weight[0] - result_weight.x[0]), 0)

    error_m = error_m_weight
    error_c = error_c_weight

    return error_m, error_c

five_iii()
```

## Question 6- Strategies for error bars

\begin{equation}
\begin{array}{lccccc}
\hline
x &1&2&3&4&5\\
y &51&103&150&199&251\\
\alpha_{y}   &1&1&2&2&3\\
\hline
x &6&7&8&9&10\\
y &303&347&398&452&512\\
\alpha_{y}   &3&4&5&6&7\\
\hline
\end{array}
\end{equation}

Required:
<bf>
>(i) Calculate the weighted best-fit values of the slope, intercept, and their uncertainties.
<bf>
>(ii) If the data set had been homoscedastic, with all the errors equal, $\alpha_{y}=4$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.
<bf>
>(iii) If the experimenter took greater time to collect the first and last data points, for which $\alpha_{y}=1$, at the expense of all of the other data points, for which $\alpha_{y}=8$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.
<bf>
>(iv) Comment on your results.
<bf>
>(v) Plot the original data from the table including error bars. On the same plot, show the fitted function calculated in (i).

### (i) Calculate the weighted best-fit values of the slope, intercept, and their uncertainties.

```python
xs = [1,2,3,4,5,6,7,8,9,10]
ys = [51,103,150,199,251,303,347,398,452,512]
ay = [1,1,2,2,3,3,4,5,6,7]

def f(x,a,b):
    return(a*x+b)

def six_i():
    'Returns slope, intercept, slope uncertainty and intercept uncertainty'
    errors = [1,1,2,2,3,3,4,5,6,7]
    slope = 0
    intercept = 0
    slope_uncertainty = 0
    intercept_uncertainty = 0

    popt, pcov = curve_fit(f, xs, ys, sigma = errors,p0 = [49,1.7])
    perr = np.sqrt(np.diag(pcov))
    slope = popt[0]
    intercept = popt[1]
    slope_uncertainty = perr[0]
    intercept_uncertainty = perr[1]

    return slope,intercept,slope_uncertainty,intercept_uncertainty
six_i()
```

### (ii) If the data set had been homoscedastic, with all the errors equal, $\alpha_{y}=4$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.

```python
def six_ii():
    errors = [4,4,4,4,4,4,4,4,4,4]
    slope = 0
    intercept = 0
    slope_uncertainty = 0
    intercept_uncertainty = 0

    popt, pcov = curve_fit(f, xs, ys, sigma = errors)
    perr = np.sqrt(np.diag(pcov))
    slope = popt[0]
    intercept = popt[1]
    slope_uncertainty = perr[0]
    intercept_uncertainty = perr[1]

    return slope,intercept,slope_uncertainty,intercept_uncertainty

six_ii()
```

### (iii) If the experimenter took greater time to collect the first and last data points, for which $\alpha_{y}=1$, at the expense of all of the other data points, for which $\alpha_{y}=8$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.

```python
def six_iii():
    errors = [1,8,8,8,8,8,8,8,8,1]
    slope = 0
    intercept = 0
    slope_uncertainty = 0
    intercept_uncertainty = 0

    popt, pcov = curve_fit(f, xs, ys, sigma = errors)
    perr = np.sqrt(np.diag(pcov))
    slope = popt[0]
    intercept = popt[1]
    slope_uncertainty = perr[0]
    intercept_uncertainty = perr[1]
    comment = 'In this case, the error on the fit parameters is lower due to reducing the error of these datapoints.'

    return slope,intercept,slope_uncertainty,intercept_uncertainty
six_iii()
```

### (iv) Comment on your results.

Comparing the uncertainties of the fits found in parts (i), (ii) and (iii), we find that the uncertainty on the slope is smallest for (iii). I.e. having small uncertainties for the extremal points is important for a good determination of the slope. Precise measurements of the data points close to x=0 reduce the uncertainty on the intercept. This is why we find relatively small (and similar) uncertainties on the intercept in scenarios (i) and (iii), where the uncertainty on the y-position of the x=1 data point is 1 in both cases.

### (v) Plot the original data from the table including error bars. On the same plot, show the fitted function calculated in (i).

```python
m,c,_,_ = six_i()
plt.errorbar(xs,ys,yerr=ay,fmt='o',capsize=2.5, label = 'data')
plt.plot(np.arange(11),m*np.arange(11)+c, label = 'weighted fit')
plt.legend()
plt.show
```

Core 1/data analysis/ipython notebook/Assignment 2 Model Solutions.ipynb


## Heat Equation
#### Parameter estimation for the heat equation (no source) using Gaussian processes (Backward Euler scheme)

#### Problem Setup

$u_t - \alpha u_{xx} = 0$

$u(x,t) = e^{-t}sin(\frac{x}{\sqrt{\alpha}})$

$u_0(x) := u(x,0) = sin(\frac{x}{\sqrt{\alpha}})$

$x \in [0, 2\pi \sqrt{\alpha}], t \in [0,1]$

Using the backward Euler scheme, the equation can be re-written as:

$\frac{u_n - u_{n-1}}{\tau} - \alpha \frac{d^2}{dx^2}u_n = 0$

and so:

$u_n - \tau \alpha \frac{d^2}{dx^2}u_n = u_{n-1}$

Consider $u_n$ to be a Gaussian processes.

$u_n \sim \mathcal{GP}(0, k_{uu}(x_i, x_j, \theta))$

And the linear operator:

$\mathcal{L}_x^\alpha = \cdot - \tau \alpha \frac{d^2}{dx^2}\cdot$

so that

$\mathcal{L}_x^\alpha u_n = u_{n-1}$

Problem at hand: estimate $\alpha$.

For the sake of simplicity, take $u := u_n$ and $f := u_{n-1}$.

#### step 1: Simulate data

Take data points at $t = 0$ for $(u_{n-1})$ and $t = \tau$ for $(u_n)$, where $\tau$ is the time step.

$\alpha = 1$ and $x \in (0,2\pi)$.

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
def get_simulated_data(tau_param, alpha_param):
    x = 2*np.pi*np.sqrt(alpha_param)*np.random.rand(20)
    y_u = np.exp(-tau_param)*np.sin(x/np.sqrt(alpha_param))
    y_f = np.sin(x/np.sqrt(alpha_param))
    return (x, y_u, y_f)

(x, yu, yf) = get_simulated_data(0.2, 1)
```

```python
plt.plot(x, yu, 'ro')
plt.show()
```

```python
plt.plot(x, yf, 'bo')
plt.show()
```

#### Step 2:Evaluate kernels

$k_{uu}(x_i, x_j; \theta) = \theta exp(-\frac{1}{2}(x_i-x_j)^2)$

```python
x_i, x_j, theta, alpha, tau = sp.symbols('x_i x_j theta alpha tau')
kuu_sym = theta*sp.exp(-1/(2)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta), kuu_sym, "numpy")
def kuu(x, theta):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta)
    return k
```

$k_{ff}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\phi \mathcal{L}_{x_j}^\phi k_{uu}(x_i, x_j; \theta) \\
= \mathcal{L}_{x_i}^\phi \left( k_{uu} - \tau \alpha \frac{\partial^2}{\partial x_j^2}k_{uu} \right) \\
 = k_{uu} - \tau \alpha \left( \frac{\partial^2}{\partial x_j^2} + \frac{\partial^2}{\partial x_i^2} \right)k_{uu} + \tau^2 \alpha^2 \frac{\partial^2}{\partial x_i^2}\frac{\partial^2}{\partial x_j^2}k_{uu}$

```python
kff_sym = kuu_sym \
        - tau*alpha*(sp.diff(kuu_sym, x_j, x_j) \
        + sp.diff(kuu_sym, x_i, x_i)) \
        + tau**2*alpha**2*sp.diff(kuu_sym, x_j, x_j, x_i, x_i)
kff_fn = sp.lambdify((x_i, x_j, theta, alpha, tau), kff_sym, "numpy")
def kff(x, theta, alpha, tau):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, alpha, tau)
    return k
```

$k_{fu}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\alpha k_{uu}(x_i, x_j; \theta) \\
= k_{uu} - \tau \alpha \frac{\partial^2}{\partial x_i^2}k_{uu} $

```python
kfu_sym = kuu_sym - tau*alpha*sp.diff(kuu_sym, x_i, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, alpha, tau), kfu_sym, "numpy")
def kfu(x, theta, alpha, tau):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kfu_fn(x[i], x[j], theta, alpha, tau)
    return k
```

```python
def kuf(x, theta, alpha, tau):
    return kfu(x,theta,alpha,tau).T
```

#### Step 3: Compute NLML

```python
def nlml(params, x, y1, y2, tau, s):
    params = np.exp(params)
    K = np.block([
        [kuu(x, params[0]) + s*np.identity(x.size), kuf(x, params[0], params[1], tau)],
        [kfu(x, params[0], params[1], tau), kff(x, params[0], params[1], tau) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
nlml((-0.5, 0), x, yu, yf, 0.2, 1e-6)
```

    210.9806883320852

#### Step 4: Optimise hyperparameters

##### Comparison of different optimisers

```python
%%timeit
minimize(nlml, np.random.rand(2), args=(x, yu, yf, 0.2, 1e-6), method="Nelder-Mead")
```

    3.02 s ± 329 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
nlml_wp = lambda params: nlml(params, x, yu, yf, 0.2, 1e-6)
param_bounds = np.array([[-3,0], [-5,1]])
%timeit minimize(nlml_wp, np.random.rand(2), method="TNC", bounds = param_bounds)
```

    The slowest run took 5.00 times longer than the fastest. This could mean that an intermediate result is being cached.
    8.28 s ± 3.19 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

##### Take minimum over multiple optimisations

```python
def minimize_restarts(x, y_u, y_f, tau, n = 10):
    nlml_wp = lambda params: nlml(params, x, y_u, y_f, tau, 1e-6)
    all_results = []
    for it in range(0,n):
        all_results.append(minimize(nlml_wp, np.random.rand(2), method="Nelder-Mead"))
    filtered_results = [m for m in all_results if 0 == m.status]
    return min(filtered_results, key = lambda x: x.fun)
```

```python
a = minimize_restarts(x, yu, yf, 0.2)
```

```python
a
```

     final_simplex: (array([[-1.89175125,  0.10118041],
           [-1.89165494,  0.10118074],
           [-1.89165615,  0.10118167]]), array([-191.19706526, -191.19706523, -191.19706518]))
               fun: -191.19706525924465
           message: 'Optimization terminated successfully.'
              nfev: 102
               nit: 52
            status: 0
           success: True
                 x: array([-1.89175125,  0.10118041])

##### Estimated value of $\alpha$

```python
np.exp(a.x[1])
```

    1.1064762467831075

### Some analysis [to-do]

##### Predicted values of $\alpha$ against time steps.

```python
import time
np.random.seed(1234)
tau_val = np.array([0.2, 0.1, 0.02, 0.01, 0.002, 0.0015, 0.001])
alpha_est = np.zeros(tau_val.size)
alpha_sim = 1
```

```python
for i in np.arange(tau_val.size):
    (loop_x, loop_yu, loop_yf) = get_simulated_data(tau_val[i], alpha_sim)
    a = minimize_restarts(loop_x, loop_yu, loop_yf, tau_val[i])
    alpha_est[i] = np.exp(a.x[1])
```

```python
f = plt.figure()
plt.plot(tau_val, np.abs(alpha_est-1))
plt.xlabel("time step")
plt.ylabel("error in estimate")
plt.show()
```

```python
alpha_est
```

    array([1.10705342, 1.05064343, 1.00880534, 1.00392676, 1.00145057,
           0.98632044, 0.99522722])

##### Convergence plot

```python
fn_values_convergence = []
def callback_fn(params):
    global fn_values_convergence
    fn_values_convergence.append(nlml_wp(params))

minimize(nlml_wp, np.random.rand(2), method="Nelder-Mead", callback=callback_fn)
```

     final_simplex: (array([[-1.89172459,  0.10118035],
           [-1.89180805,  0.10118075],
           [-1.89177548,  0.10118143]]), array([-191.19706526, -191.19706524, -191.19706523]))
               fun: -191.19706525651753
           message: 'Optimization terminated successfully.'
              nfev: 131
               nit: 71
            status: 0
           success: True
                 x: array([-1.89172459,  0.10118035])

```python
plt.plot(fn_values_convergence)
plt.show()
```

```python

```


# Lecture 2: Integral equations/quadratures

## Previous lecture
- Intro
- IE for external problems

## Todays lecture
- IE discretization
- Nystrom, collocation, Galerkin method

## Simplest integral equation

The simplest integral equation we talked about reads

$$\int_{\partial \Omega} \frac{q(y)}{\Vert x - y \Vert} dy = f(x), \quad x \in \partial \Omega,$$
where $\Omega$ is a certain domain in 3D.

The general form of a first-kind integral equation is

$$
\int_{\partial \Omega} q(y) G(x, y) dy = f(x), \quad x \in \partial \Omega,
$$
where $G(x, y)$ is a **Green function** of the PDE.

## Typical Green functions
The Green function of a PDE operator $A$ satisfies
$$A G(x, y) = \delta(x-y), $$
i.e. it plays the role of the "inverse" of the operator without boundary conditions.

It is also called **free-space** Green function. The Green function can also incorporate the boundary conditions.

## Green functions with boundary conditions and reflection method

Suppose we want to find a function $G(x, y)$ such that

$$\Delta_x G(x, y) = \delta (x - y), \; G(x, y) \mid_{x \in \partial \Omega} = 0. $$

If we found such a function, solution of

$$\Delta u = f,$$

is given by the **explicit integral**

$$u(x) = \int_{\Omega} G(x, y) f(y)  dy.$$

## Reflection method

In a few cases, you can write down such a Green function, typically based on the **method of reflections**.

For a Dirictlet bouundary conditions, we can use not only charge located at $y$, but also many other charges that lie outside $\Omega$ and give the required b.c.

- A plane - two charges are ok.
- A cube - many reflections, infinite series

## Discretization
Now to discretization of the equation
$$\int_{\partial \Omega} \frac{q(y)}{\Vert x - y \Vert} dy = f(x), \quad x \in \partial \Omega.$$

Steps of discretization:
1. Select a finite-dimensional subspace, spanned by basis function $\phi_i, i = 1, \dots, N$. Typically, $\phi_i$ are **local basis functions** defined on a certain mesh $\mathcal{T}$ over the surface.
2. Look for the solution in the form
   $$q(y) \approx \sum_{i=1}^N q_i \phi_i(y)$$

Note, that the basis functions can be even **piecewise-constant**, compared to PDEs, where **generally** they have to be at least piecewise-linear for Finite Element Method (they can be piecewise-constant for **Discontinious Galerkin**, but that is more an exception)

## Discretization (2)
$$\int_{\partial \Omega} \frac{q(y)}{\Vert x - y \Vert} dy = f(x), \quad x \in \partial \Omega,$$

Suppose that $\mathrm{supp}(\phi_i) = T_i$ (i.e., local support). Then,

$$ \sum_{i=1}^N q_i \int_{T_i} \frac{\phi_i(y)}{\Vert x - y \Vert} dy = f(x).$$

This has to hold for all $x$, i.e. it is **infinite amount** of equations. There are two ways of dealing with this situation:

1. Galerkin method
2. Collocation method.

Elements $T_i$ are elements of mesh $\mathcal{T}$. Usually they are triangles, but other configuration cam be used.

## Galerkin method
The Galerkin method takes  the equation
$$ \sum_{i=1}^N q_i \int_{T_i} \frac{\phi_i(y)}{\Vert x - y \Vert} dy = f(x),$$
and multiplies it by $\phi_j(x)$ and integrates over $T_j$. Then we get

$$
\sum_{i=1}^N q_i \int_{T_j} \int_{T_i} \frac{\phi_j(x) \phi_i(y)}{\Vert x - y \Vert} dx dy = \int_{T_j} f(x) \phi_j(x) dx.
$$

This is a **square system** of linear equations of the form
$$
   A q = f,
$$
where
$$
A = [a_{ji}], \; a_{ji} = \int_{T_j} \int_{T_i} \frac{\phi_j(x) \phi_i(y)}{\Vert x - y \Vert} dx dy, \quad
f_j = \int_{T_j} f(x) \phi_j(x) dx
$$
Everything is fine, the matrix is symmetric (it can be also shown that it is positive definite, let us do that), but the evaluation of the element requires
**4D** integral to be evaluated. That is main computational problem, thus engineers often use a much more simple collocation method.

## Collocation method
Collocation method consists of taking the equation
$$ \sum_{i=1}^N q_i \int_{T_i} \frac{\phi_i(y)}{\Vert x - y \Vert} dy = f(x),$$
at some **collocation points** $x_j$.

Consider **piecewise-constant case**, i.e. $\phi_i(y) = 1$ on $T_i$ and $0$ otherwise. Then the most natural choice for collocation points are **centers of the triangles**,
and we again get a **square** linear system

$$A q = f,$$

where

$$
A = [a_{ji}], \; a_{ji} = \int_{T_i} \frac{dy}{\Vert x_j - y \Vert}, \quad f_j = f(x_j)
$$
i.e. it has the meaning of **uniformly charged** triangle.

Note that this matrix is **non-symmetric**, so you can not use symmetric-oriented methods to solve linear system, but integrals are in 2D domain.

Also if number of collocation points is more than number of triangles $T_i$, then we have to solve least squares problem, because linear system becomes rectangular and underdetermined.

## Collocation method as Petrov-Galerkin method

- In Galerkin method, function $f(x)$ is projected onto local basis functions $\phi_j(x)$:
$$
f_j = \int_{T_j} f(x) \phi_j(x) dx
$$
- In collocation method, function $f(x)$ is projected onto the delta functions $\delta(x - x_j)$, where $x_j$ are collocation points:
$$
f_j = \int_{T_j} f(x) \delta (x - x_j) dx = f(x_j)
$$

## Nystrom method

The Nystrom method is just another simplication of the collocation method (as collocation method can be considered as a quadrature applied to the Galerkin method).

We just approximate

$$
    A_{ij}  = \int_{T_i} \frac{dy}{\Vert x_j - y \Vert} dy \approx S_i \frac{1}{\|x_j - y_i\|}.
$$

If the **source triangle** $T_i$ is sufficiently far from the **receiver** $x_j$, indeed, this can be a very good approximation.

If $y_i$ lies inside the triangle, then the approximation can be  really bad.

## Locally corrected Nystrom

Typically, a **locally corrected Nystrom** is used: much more advanced quadratures are used in the **close zone**, where the singularity of the integrals appears. The concept is as follows:

- To evaluate the matrix-by-vector product $Aq$ it is split into the **far part** and **close part**.
- Close part is evaluated separately, far is evaluated using a simple quadrature

**Far-close** splitting is the key to **fast methods**!

## Computing the integrals
The main art about Galerkin/collocation methods is **computation of singular integrals**, which can not be handled by standard Gauss quadratures in the close zone.

There are several ways

- Analytic evaluation
- Subtraction of singularity, singularity is integrated analytically, remainder by Gauss
- Gauss-Duffy quadrature
- Schwab-Sauter quadrature
- Rokhlin generalized quadratures

## Model integral

We will study the integrals of the form

$$
\int^b_a \int^d_c \frac{\phi(x) \phi(y)}{\Vert x - y \Vert} dy,
$$

where $\phi(x), \phi(y)$ are standard local basis function (constant or linear).

## Computing the simplest integral
We can actually compute the anti-derivative of the integral in question, by taking successive integrals

$$
   \frac{1}{\sqrt{x^2 + y^2}}
$$
\begin{equation}
\begin{split}
F(x, y) = \int \int \frac{1}{\sqrt{x^2 + y^2}} dx dy = \\
-y +
y \log(x + \sqrt{x^2 + y^2}) + x \log(y + \sqrt{x^2 + y^2}),
\end{split}
\end{equation}

and the final integral is just
$$
   I(x_0, y_0, x_1, y_1) = F(x_0, y_0) + F(x_1, y_1) - F(x_0, y_1) - F(x_1, y_0),
$$

(generalization of the Newton-Leibniz formula)

## Smarter than Nystrom
You can be **a little bit** smarter: approximate $1/r$ by something that can be well-integrated.

By a sum of separable functions!

$$\frac{1}{\Vert x - y\Vert } \approx \sum_{\alpha=1}^r u_{\alpha}(x) u_{\alpha}(y)$$

We have used the symmetry.

Such representation is obviously useful, since in Galerkin formula we can replace 4D integration by 2D integration,

$$
\int \int \phi_i(x) \phi_j(y) u_{\alpha}(x) u_{\alpha}(y) dx dy = \int \phi_i(x) u_{\alpha}(x) dx \int \phi_j(y) u_{\alpha}(y) dy.
$$

But how to compute these functions $u_{\alpha}$?

## Exponential sums

A very useful approach is based on so-called **exponential sums**. A simple identity has the form

$$\frac{1}{\sqrt{r}} = \frac{1}{\sqrt{\pi}} \int^{\infty}_0 \frac{e^{-pr}}{\sqrt{p}} dp.$$

It can be very useful in separating variables.

## A quadrature

Suppose we have a quadrature for the integral:

$$
\frac{1}{\sqrt{r}} = \frac{1}{\sqrt{\pi}} \int^{\infty}_0 \frac{e^{-pr}}{\sqrt{p}} dp. \approx \sum_{k} w_k e^{-p_k r}.$$

Then put

$$
r = x^2 + y^2
$$

and we get the sum of  Gaussians  on the right.

## Sum of Gaussians

$$\frac{1}{\sqrt{r}} \approx \sum_{k} w_k e^{-p_k (x^2 + y^2)},$$

and the integrals reduce to the 1D integrals

$$
   \int_a^b e^{-p_k x^2} dx,
$$

and this can be expressed via the  Error functions  (and there is a standard numpy ```erfc``` function)

## But how to find $w_k$ and $p_k$?

We reduced the problem to the problem of creating a good quadrature rule for the integral

$$\int^{\infty}_0 \frac{e^{-px}}{\sqrt{p}} dp$$

You have to do the $p = e^t$ (why do we do so?) and we have the new integral

$$
\int^{\infty}_{-\infty} e^{-e^{t} x + t/2} dt
$$

and this integral is then approximated by **trapezoidal rule** $t_k = a_t + k h_t, \quad h_t = (b_t - a_t)/K$

The integral is doubly exponentially decaying, and also its Fourier transform is decaying exponentially fast.

Thus, trapezoidal rule has **exponential convergence** in this setting

A short demo...

```python
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline
a = -30
b = 30
n_list = [10, 100, 150]
plt.figure(figsize=(10,8))
for n in n_list:
    t = np.linspace(a, b, n)
    h = t[1] - t[0]
    w = np.ones(n) * h
    w[0] = h/2
    w[n-1] = h/2
    w = w * np.exp(0.5 * t)/np.sqrt(math.pi)
    ps = np.exp(t)
    x = np.linspace(0.01, 1)
    fun = 1.0/np.sqrt(x)
    appr = 0 * x
    for i in xrange(n):
        appr = appr + w[i] * np.exp(-(x) * ps[i])
    plt.plot(x, np.abs(appr - fun), label="$n$ = {}".format(n))
# ax.plot(x, appr, label='appr')
# ax.plot(x, fun, label='fun')
plt.yscale('log')
#ax.set_title('Approximation error by %d Gaussians' % n)
plt.ylabel('Error', fontsize=24)
plt.xlabel("$x$", fontsize=24)
plt.legend(fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
plt.tight_layout()
```

## A sidenote about fitting by sum of exponentials
- Approximation of a given function by a sum of exponentials is a ill-posed problem
- Original method by Prony (1795)
- Recent work by [G. Beylkin, et. al, Approximation by exponential sums revisited](https://www.google.ru/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0CCYQFjAB&url=http%3A%2F%2Famath.colorado.edu%2Fpub%2Fwavelets%2Fpapers%2FBEY-MON-2010.pdf&ei=Cok1VZHkJKLnygO7hYCgBw&usg=AFQjCNGRcxece5lCuy9N7oheTBOzOpcSYw&sig2=Rf3DPGBAMmYO4parmQXzFQ&bvm=bv.91071109,d.bGQ)

## Integration over triangles

Exponential sums are well-applicable to the integration over **rectangles**. But the most typical choice is the integration over **triangles**.

Consider the standard triangle
$$
T = \{ (x, y) \; | \; 0 \leq x \leq 1, \; 0 \leq y \leq x\}.
$$

Consider the integral $$\int_{T} h dS,$$

which can be written as

$$
    \int^{1}_{0} dx \int^x_0 h(x, y)dy.
$$

## Duffy transformation

Duffy transformation is a **mapping** from rectangle to triangle, given by formulas

$$x = x, \quad y = x u, \quad 0 \leq u \leq 1,$$

i.e. it is a mapping from $(x, u)$ to $(x, y)$.

## Duffy transformation(2)
Then, we have

$$\int^{1}_0 \int^{1}_0 h(x, xu) x dx du.$$

Recall the singularity:

$$h(x, y) = \frac{g(x, y)}{\sqrt{x^2 + y^2}},$$

therefore the $x$ cancels out and we have

$$\int_0^1 \int_0^1 \frac{g(x, xu)}{\sqrt{1 + u^2}} dx du, $$

which is **non-singular**.

## Efficient quadratures (papers)
- C. Schwab,  S. Sauter [Efficient automatic quadrature in 3-d Galerkin BEM](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45.4312&rep=rep1&type=pdf)
- A. Polimeridis, Traianos V Yioultsis [On the direct evaluation of weakly singular integrals in Galerkin mixed potential integral equation formulations](http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4618709)
- [Vico-Greengard-Ferrando quadrature](https://arxiv.org/pdf/1604.03155.pdf)

## Vico-Greengard-Ferrando quadrature
Suppose we have to compute
$$f(x) = \int G(x-y) \rho (y) dy, $$
where $G(x-y)$ is a spherically symmetric Green function of some differential operator, that is
$$G(r) = G(|r|)$$

Idea: use **Fourier transform**
$$\widehat{(f \ast g)} = {\hat f} {\hat g}$$

This suggests that we can compute $f(x)$ as
$$f(x) = \Big(\frac{1}{2\pi}\Big)^3 \int e^{i s \cdot x} \widehat{\rho}(s) \widehat{G}(s) ds$$

However, there is a problem: $\widehat{G}(s)$ is typically very singular, e.g. for Helmholtz differential operator $\Delta + k^2$
$$G(s) = \frac{1}{k^2 - s^2},$$
there is a **sphere** of singular points.
So how we can avoid it?

## Vico-Greengard-Ferrando quadratures (2)
We can use the following simple observation. Suppose that $\rho(x)$ is $0$ outside of cube $[0, 1]^3$. Then, in this cube we have the following identity:

$$
\int G(x-y) \rho (y) dy \equiv \int G_L(x-y) \rho (y)  dy,
$$
with
$$
G_L(r) = G(r) \chi_{[0, L]}(|r|), \quad L = \sqrt{3}
$$

and $\chi_D(x)$ is a charactersitic function of a set $D$,
and identity holds because in the cube $[0, 1]^3$ distance between any two points doesn't exceed $\sqrt{3}$.

So we can use $G_L$ instead of $G$. $G_L$ has **compact support** and it's Fourier transform is **smooth**!

E.g. for Helmholtz differential operator:

```python
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline

def G_L(x, k, L):
    return (-1 + np.exp(1.0j * L * k) * (np.cos(L * x) - 1.0j * k/x * np.sin(L * x)))/(k**2 - x**2)

x = np.linspace(-20., 20., 1000)
plt.plot(x, np.real(G_L(x, 4.0, np.sqrt(3))), label='Re $G_L$')
plt.plot(x, np.imag(G_L(x, 4.0, np.sqrt(3))), label='Im $G_L$')
plt.legend()
```

## Vico-Greengard-Ferrando quadratures (3)

So we need to compute
$$f(x) = \Big(\frac{1}{2\pi}\Big)^3 \int e^{i s \cdot x} \widehat{\rho}(s) \widehat{G_L}(s) ds, $$
with $G_L(s)$ being **smooth**. This can be performed using trapezoidal rule.

## Rokhlin generalized quadratures

Finally, I will discuss a general quadrature idea. For a particular **kernel** of the integral operator, the quadrature **points and weights** in

$$\int K f \approx \sum_i w_i (K f)(x_i),$$

have to be defined only once.

## Idea in 1D

I will illustrate the idea in 1D to compute

$$\int^1_0 f(x) dx \approx \sum_{j=1}^M f(x_j) w_j$$

and this quadrature has to integrate certain subset of functions $f_1, \ldots, f_n$,

i.e.

$$\sum_{j=1}^M f_i(x_j) w_j = \int^1_0 f_i(x) dx.$$

## Assumptions

The quadrature formula with nodes $x_1, \ldots, x_m$ and weights $w_1, \ldots, w_m$:

- Integrates $f_i, i = 1, \ldots, n$ and their products
- $f_i$ form **orthonormal basis set** in $L_2[0, 1]$
- $m \gg n$

## Linear system for the weights

Then we have a linear $n \times m$ system for the weights, which is **underdetermined**.

Specifically,

$$\sum_{j=1}^m f_i(x_j) \sqrt{w_j} v_j  = b_n.$$

Since the quadrature formula integrates the products, the matrix has singular values $0$ or $1$ (prove!).

We can also set most of the solution coefficients to zero, and leave only $m$ non-zero entries.

If we denote the positions by $i_1, \ldots, i_n$, the final quadrature will read

$$\int^1_0 f(x) dx \approx \sum_{k=1}^n f(x_{i_k}) z_{k} \sqrt{w_{i_k}}.$$

## Summary of the generalized quadrature idea
- Start with an orthonormal basis set
- Select very large quadrature with many points
- Select a "pruned" quadrature.

The approach can be generalized to singular integrals.

## Summary of the lecture

- Basic discretization schemes (Galerkin, collocation, Nystrom)
- Some approaches to compute singular integrals

## Next lecture
- Fast methods: computation of the convolution via FFT, idea of precorrected FFT.
- More integral equation kernels

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("./styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


 TP-Projet d'optimisation numérique

 Année 2020-2021 - 2e année département Sciences du Numérique

 Mouddene Hamza

 Tyoubi Anass

# Algorithme de Newton
## Implémentation

1. Coder l’algorithme de Newton local tel que décrit dans la section *Algorithme de Newton* (fichier `Algorithme_De_Newton.jl`)

2. Tester l’algorithme sur les fonctions $f_{1}$ , $f_{2}$ avec les points initiaux $x_{011}$ , $x_{012}$ (pour $f_{1}$ ) et $x_{021}$ , $x_{022}$ , $x_{023}$ (pour $f_{2}$ ) donnés en Annexe A.

```julia
using LinearAlgebra
using Markdown                             # Pour que les docstrings en début des fonctions ne posent
                                           # pas de soucis. Ces docstrings sont utiles pour générer
                                           # la documentation sous GitHub
include("Algorithme_De_Newton.jl")

# Affichage les sorties de l'algorithme des Régions de confiance
function my_afficher_resultats(algo, nom_fct, point_init, xmin, fxmin, flag, sol_exacte, nbiters)
    println("-------------------------------------------------------------------------")
    printstyled("Résultats de : ", algo, " appliqué à ", nom_fct, " au point initial ", point_init, ":\n", bold=true, color=:blue)
    println("  * xsol = ", xmin)
    println("  * f(xsol) = ", fxmin)
    println("  * nb_iters = ", nbiters)
    println("  * flag = ", flag)
    println("  * sol_exacte : ", sol_exacte)
end

println("\n\n\nFonction f0")
f0(x) =  sin(x)
# la gradient de la fonction f0
grad_f0(x) = cos(x)
# la hessienne de la fonction f0
hess_f0(x) = -sin(x)
sol_exacte = -pi/2
options = []

x0 = sol_exacte
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f0, grad_f0, hess_f0, x0, options)
my_afficher_resultats("Newton", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = -pi / 2 + 0.5
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f0, grad_f0, hess_f0, x0, options)
my_afficher_resultats("Newton", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = pi / 2
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f0, grad_f0, hess_f0, x0, options)
my_afficher_resultats("Newton", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

println("\n\n\nFonction f1")
# -----------
f1(x) = 2 * (x[1] + x[2] + x[3] - 3) ^ 2 + (x[1] - x[2]) ^ 2 + (x[2] - x[3]) ^ 2
# la gradient de la fonction f1
grad_f1(x) = [4 * (x[1] + x[2] + x[3] - 3) + 2 * (x[1] - x[2]); 4 * (x[1] + x[2] + x[3] - 3) - 2 * (x[1] - x[2]) + 2*(x[2]-x[3]); 4*(x[1]+x[2]+x[3]-3)-2*(x[2]-x[3])]
# la hessienne de la fonction f1
hess_f1(x) = [6 2 4; 2 8 2; 4 2 6]
sol_exacte = [1 ,1 ,1]
options = []

x0 = [1; 0; 0]
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f1, grad_f1, hess_f1, x0, options)
my_afficher_resultats("Newton", "f1", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = [10; 3; -2.2]
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f1, grad_f1, hess_f1, x0, options)
my_afficher_resultats("Newton","f1", x0, xmin, f_min, flag, sol_exacte, nb_iters)

println("\n\n\nFonction f2")
# -----------
f2(x) = 100 * (x[2] - x[1] ^ 2) ^ 2 + (1 - x[1]) ^ 2
grad_f2(x) = [-400 * x[1] * (x[2] - x[1] ^ 2) - 2 * (1 - x[1]) ; 200 * (x[2] -x[1]^2)]
hess_f2(x) = [-400 * (x[2] - 3 * x[1] ^ 2) + 2  -400 * x[1]; -400 * x[1]  200]
sol_exacte = [1, 1, 1]
options = []

x0 = [-1.2; 1]
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("Newton", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = [10; 0]
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("Newton", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = [0; ((1 / 200) + (1 / (10 ^ 12)))]
xmin, f_min, flag, nb_iters = Algorithme_De_Newton(f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("Newton", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)
```

## Interprétation

Justifier

1. les résultats obtenus pour l'exemple $f_0$ ci-dessus;

2. que l’algorithme implémenté converge en une itération pour $f_{1}$;

3. que l’algorithme puisse ne pas converger pour $f_{2}$ avec certains points initiaux.

## Réponses

    les résultats obtenus pour l'exemple  $f_0$  ci-dessus sont plutôt cohérentes, tel que :

        \begin{equation}
            \begin{split}
                &f_0 : \mathbb{R} \rightarrow \mathbb{R}\\
                &x \rightarrow \sin(x)
            \end{split}
        \end{equation}

        Pour le premier cas, le point initiale est égale à la solution exacte $x_{001} = -\dfrac{\pi}{2}$, puis on trouve que le minimum de la fonction $\sin$ est $-1$ sans aucune itération.

        Pour le deuxième résultat on s'écarte de $\dfrac{1}{2}$ du point critique qui est un minimum selon le résultat du premier calcul, après $3$ itérations on retrouve la solution exacte.

        Finalement, on prend un $x_{002} = \dfrac{\pi}{2}$ qui représente un maximum pour la fonction $\sin$, comme $x_{003}$ est dèja un point critique, l'algorithme de Newton s'arrête. Dans cette situation, il faut aller chercher plutôt la condition à l'ordre $2$ pour trouver le minimum de la fonction $\sin$.

        L’algorithme implémenté converge en une itération pour  $f_1$, tel que :

        \begin{equation}
            \begin{split}
                &f_1 : \mathbb{R^{3}} \rightarrow \mathbb{R}\\
                &(x_1, x_2, x_3) \rightarrow 2(x_1 + x_2 + x_3 - 3) ^ 2 + (x_1 - x_2) ^ 2 + (x_2 - x_3) ^ 2
            \end{split}
        \end{equation}

        Pour le points initiaux $x_{011} = \begin{bmatrix}1\\0\\0\end{bmatrix}$ et $x_{012} = \begin{bmatrix}10\\3\\-2.2\end{bmatrix}$, $f_1$ est une forme quadratique donc elle est égale à son développement de Taylor a l'ordre $2$.

        On remarque que l’algorithme appliqué à $f_2$ tel que :

        \begin{equation}
            \begin{split}
                &f_2 : \mathbb{R^{2}} \rightarrow \mathbb{R}\\
                &(x_1, x_2) \rightarrow 100(x_2 - x_1 ^ 2) ^ 2 + (1 - x_1) ^ 2
            \end{split}
        \end{equation}

        converge pour les deux premiers points initiaux $x_{021} = \begin{bmatrix}-1.2\\1\end{bmatrix}$ et $x_{022} = \begin{bmatrix}10\\0\end{bmatrix}$ mais pas pour le dernier point $x_{023} = \begin{bmatrix}0\\\dfrac{1}{200} + \dfrac{1}{10^{12}}\end{bmatrix}$ l'interpreteur Julia signale une erreur : SingularException, cela vient du fait que le point initial est un point critique du coup dans le calcul $d_k$ solution du système : $\nabla^{2} f (x_{k}) d_{k} = - \nabla f (x_{k})$, on se trouve avec un système qui n'est pas inversible.

# Régions de confiance avec pas de cauchy

## Implémentation

1. Coder l'algorithme du pas de Cauchy d’un sous-problème de
régions de confiance (fichier `Pas_De_Cauchy.jl`). Tester sur les quadratiques proposées en Annexe B.

2. Coder l'algorithme de régions de confiance (fichier `Regions_De_Confiance.jl`). Tester sur les problèmes de l’Annexe A.

```julia
include("Pas_De_Cauchy.jl")

println("Pas de cauchy\n\n\nB. Cas tests pour le calcul du pas de Cauchy")

println("\n\nQuadratique 1")
g1 = [0; 0]
H1 = [7 0; 0 2]
println("Cauchy g1 H1\n",Pas_De_Cauchy(g1, H1, 1),"\n")

println("\n\nQuadratique 2")
g2 = [6; 2]
H2 = [7 0; 0 2]
println("Cauchy g2 H2\n",Pas_De_Cauchy(g2, H2, 1),"\n")

println("\n\nQuadratique 3")
g3 = [-2; 1]
H3 = [-2 0; 0 10]
println("Cauchy g3 H3\n",Pas_De_Cauchy(g3, H3, 1),"\n\n")
```

    Pas de cauchy

    B. Cas tests pour le calcul du pas de Cauchy

    Quadratique 1
    Cauchy g1 H1
    ([0.0, 0.0], 0)

    Quadratique 2
    Cauchy g2 H2
    ([-0.9230769230769234, -0.30769230769230776], 1)

    Quadratique 3
    Cauchy g3 H3
    ([0.8944271909999159, -0.4472135954999579], -1)

```julia
# Vos tests
include("Regions_De_Confiance.jl")

println("RC-Pas de cauchy\n\n\nFonction f0")
# -----------
sol_exacte = -pi / 2
options = []

x0 = sol_exacte
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f0, grad_f0, hess_f0, x0, options)
my_afficher_resultats("cauchy", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = -pi / 2 + 0.5
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f0, grad_f0, hess_f0, x0, options)
my_afficher_resultats("cauchy", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = pi / 2
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f0, grad_f0, hess_f0, x0, options)
my_afficher_resultats("cauchy", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

println("\n\n\nFonction f1")
# -----------
sol_exacte = [1, 1, 1]
options = []

x0 = [1; 0; 0]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f1, grad_f1, hess_f1, x0, options)
my_afficher_resultats("cauchy", "f1", x0, xmin, f_min, flag, sol_exacte, nb_iters)
x0 = [10; 3; -2.2]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f1, grad_f1, hess_f1, x0, options)
my_afficher_resultats("cauchy", "f1", x0, xmin, f_min, flag, sol_exacte, nb_iters)

println("\n\n\nFonction f2")
# -----------
sol_exacte = [1, 1, 1]
options = []

x0 = [-1.2; 1]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("cauchy", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)
x0 = [10; 0]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("cauchy", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)
x0 = [0; ((1 / 200) + (1 / (10 ^ 12)))]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("cauchy", f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("cauchy", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)
```

    RC-Pas de cauchy

    Fonction f0
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f0 au point initial -1.5707963267948966:
      * xsol = -1.5707963267948966
      * f(xsol) = -1.0
      * nb_iters = 1
      * flag = 0
      * sol_exacte : -1.5707963267948966
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f0 au point initial -1.0707963267948966:
      * xsol = -1.5707963267949088
      * f(xsol) = -1.0
      * nb_iters = 3
      * flag = 0
      * sol_exacte : -1.5707963267948966
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f0 au point initial 1.5707963267948966:
      * xsol = 1.5707963267948966
      * f(xsol) = 1.0
      * nb_iters = 1
      * flag = 0
      * sol_exacte : -1.5707963267948966

    Fonction f1
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f1 au point initial [1, 0, 0]:
      * xsol = [1.0000558873349883, 0.999992420017735, 0.9999289527004819]
      * f(xsol) = 9.090411079109608e-9
      * nb_iters = 26
      * flag = 2
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f1 au point initial [10.0, 3.0, -2.2]:
      * xsol = [1.000049795462743, 0.9999961002424803, 0.9999424049876057]
      * f(xsol) = 6.0401046516733e-9
      * nb_iters = 28
      * flag = 2
      * sol_exacte : [1, 1, 1]

    Fonction f2
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f2 au point initial [-1.2, 1.0]:
      * xsol = [0.9975452532046185, 0.9950891951120907]
      * f(xsol) = 6.03116510222528e-6
      * nb_iters = 8546
      * flag = 2
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f2 au point initial [10, 0]:
      * xsol = [0.9973371221017552, 0.9946738223584641]
      * f(xsol) = 7.096562862872902e-6
      * nb_iters = 1212
      * flag = 2
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : cauchy appliqué à f2 au point initial [0.0, 0.0050000000010000005]:
      * xsol = [0.9980249833119488, 0.9960352320621435]
      * f(xsol) = 3.935418182296104e-6
      * nb_iters = 3198
      * flag = 2
      * sol_exacte : [1, 1, 1]

## Interprétation

1. Quelle relation lie la fonction test $f_1$ et son modèle de Taylor à l’ordre 2 ? Comparer alors les performances de Newton et RC-Pas de Cauchy sur cette fonction.

2. Le rayon initial de la région de confiance est un paramètre important dans l’analyse
de la performance de l’algorithme. Sur quel(s) autre(s) paramètre(s) peut-on jouer
pour essayer d’améliorer cette performance ? Étudier l’influence d’au moins deux de
ces paramètres.

## Réponses

        \begin{equation}
            \begin{split}
                &f_1 : \mathbb{R^{3}} \rightarrow \mathbb{R}\\
                &(x_1, x_2, x_3) \rightarrow 2(x_1 + x_2 + x_3 - 3) ^ 2 + (x_1 - x_2) ^ 2 + (x_2 - x_3) ^ 2
            \end{split}
        \end{equation}

        La fonction f1 est égale à son développement de Taylor à l’ordre 2. On remarque que l'algorithme de RC-Pas de Cauchy converge pour le point initial $x_{011} = \begin{bmatrix}1\\0\\0\end{bmatrix}$ en $26$ itérations, ainsi que $29$ itérations pour le point initial $x_{012} = \begin{bmatrix}10\\3\\-2.2\end{bmatrix}$., Alors que pour l'algorithme de Newton ça converge en une seule itération. Donc, pour cet exemple, l'algorithme de Newton est plus performant que l'algorithme de RC-Pas de cauchy.

        Le rayon initial de la région de confiance est un paramètre important dans l’analyse
de la performance de l’algorithme. mais on peut jouer sur d'autres paramètres pour essayer d’améliorer cette performance tels que le rayon de confiance maximal, le facteur d’agrandissementde la region de confiance,le facteur de reduction de la region de confiance, le critère d’agrandissement de la region de confiance et le critère de reduction de la region de confiance.

# Régions de confiance avec gradient conjugué tronqué

## Implémentation

1. Implémenter l’algorithme du Gradient Conjugué Tronqué, en se basant sur le cours (fichier `Gradient_Conjugue_Tronque.jl`).
On validera les résultats sur les fonctions de l’Annexe C.

2. Intégrer finalement l’algorithme du Gradient Conjugué Tronqué dans le code de
régions de confiance, et appliquer ce code pour résoudre les exemples proposés en
Annexe A.

```julia
include("Gradient_Conjugue_Tronque.jl")

println("Gradient conjugue tronqué\n\n\nC. Cas tests pour la résolution du sous-problème par l’algorithme du Gradient Conjugué Tronqué")

println("\n\nQuadratique 4")
g4 = [0; 0]
H4 = [-2 0; 0 10]
println("Cauchy g4 H4\n", Gradient_Conjugue_Tronque(g4, H4, []), "\n")

println("\n\nQuadratique 5")
g5 = [2; 3]
H5 = [4 6; 6 5]
println("\nCauchy g5 H5\n", Gradient_Conjugue_Tronque(g5, H5, []), "\n")

println("\n\nQuadratique 6")
g6 = [2; 0]
H6 = [4 0; 0 -15]
println("\nCauchy g6 H6\n", Gradient_Conjugue_Tronque(g6, H6, []), "\n\n")
```

    Gradient conjugue tronqué

    C. Cas tests pour la résolution du sous-problème par l’algorithme du Gradient Conjugué Tronqué

    Quadratique 4
    Cauchy g4 H4
    [0.0, 0.0]

    Quadratique 5

    Cauchy g5 H5
    [1.1782448197996298, -1.6160876042514951]

    Quadratique 6

    Cauchy g6 H6
    [-0.5, 0.0]

```julia
# Vos tests
include("Regions_De_Confiance.jl")

println("RC-GCT\n\n\nFonction f1")
# -----------
sol_exacte = [1,1,1]
options = []

x0 = [1; 0; 0]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("gct", f1, grad_f1, hess_f1, x0, options)
my_afficher_resultats("gct", "f1", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = [10; 3; -2.2]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("gct", f1, grad_f1, hess_f1, x0, options)
my_afficher_resultats("gct", "f1", x0, xmin, f_min, flag, sol_exacte, nb_iters)

println("\n\n\nFonction f2")
# -----------
sol_exacte = [1, 1, 1]
options = []

x0 = [-1.2; 1]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("gct", f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("gct", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = [10; 0]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("gct", f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("gct", "f0", x0, xmin, f_min, flag, sol_exacte, nb_iters)

x0 = [0; ((1 / 200) + (1 / (10 ^ 12)))]
xmin, f_min, flag, nb_iters = Regions_De_Confiance("gct", f2, grad_f2, hess_f2, x0, options)
my_afficher_resultats("gct", "f2", x0, xmin, f_min, flag, sol_exacte, nb_iters)
```

    RC-GCT

    Fonction f1
    -------------------------------------------------------------------------
    Résultats de : gct appliqué à f1 au point initial [1, 0, 0]:
      * xsol = [1.0000000000000007, 1.0, 1.0]
      * f(xsol) = 2.0214560696288428e-30
      * nb_iters = 1
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : gct appliqué à f1 au point initial [10.0, 3.0, -2.2]:
      * xsol = [0.9999999999999996, 1.0000000000000004, 1.0000000000000009]
      * f(xsol) = 2.5637979419682884e-30
      * nb_iters = 3
      * flag = 0
      * sol_exacte : [1, 1, 1]

    Fonction f2
    -------------------------------------------------------------------------
    Résultats de : gct appliqué à f0 au point initial [-1.2, 1.0]:
      * xsol = [0.9999999999999402, 0.9999999999997743]
      * f(xsol) = 1.1277385526166413e-24
      * nb_iters = 32
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : gct appliqué à f0 au point initial [10, 0]:
      * xsol = [1.0000000002404108, 1.0000000004684448]
      * f(xsol) = 7.311578825077101e-20
      * nb_iters = 45
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : gct appliqué à f2 au point initial [0.0, 0.0050000000010000005]:
      * xsol = [0.9999999999998994, 0.9999999999996207]
      * f(xsol) = 3.1813581453548166e-24
      * nb_iters = 19
      * flag = 0
      * sol_exacte : [1, 1, 1]

## Interprétation

1. Comparer la décroissance obtenue avec celle du pas de Cauchy, en retournant, dans
un premier temps le dernier itéré admissible à courbure positive (c’est à dire, que si
l’une ou l’autre des deux conditions (b) ou (d) sont rencontrées dans l’algorithme 3,
alors on ne calcule pas ``σ_{j}`` et on retourne le dernier itéré ``s_{j}`` directement).

2. Comparer la décroissance obtenue avec celle du pas de Cauchy, en imposant la sortie
dans l’algorithme 3 au bout d’une itération seulement. Que remarquez vous ?

3. Comparer la décroissance obtenue avec celle du pas de Cauchy dans le cas général.

4. Quels sont les avantages et inconvénients des deux approches ?

## Réponses

1. Rappelant que la fonction $f_2$ s'écrit sous la forme suivante :

\begin{equation}
    \begin{split}
        &f_2 : \mathbb{R^{2}} \rightarrow \mathbb{R}\\
        &(x_1, x_2) \rightarrow 100(x_2 - x_1 ^ 2) ^ 2 + (1 - x_1) ^ 2
    \end{split}
\end{equation}

On applique cette fonction au point initial $x_{022} = \begin{bmatrix}10\\0\end{bmatrix}$, en retournant le dernier itéré admissible à courbure positive, le résultat de la décroissance obtenue est le suivant :

| RC-Pas de cauchy | RC-Gradient tronqué conjugué |
|:----------------:|:----------------------------:|
|     591198.87    |           591198.87          |
|     328991.13    |           337809.30          |
|     64598.674    |           65927.96           |
| 12522.069        | 5134.153                     |
| 2336.1646        | 1.160071                     |

On remarque que la décroissance de l'algorithme "Pas de cauchy" et celle de l'algorithme "gct" sont à peu près Égaux.

2. En imposant la sortie dans l’algorithme "gct" au bout d’une itération seulement, le résultat de la décroissance obtenue est le suivant :

| RC-Pas de cauchy | RC-Gradient tronqué conjugué |
|:----------------:|:----------------------------:|
|     591198.87    |           591198.87          |
|     328991.13    |           337809.30          |
|     64598.674    |           65927.96           |
| 12522.069        | 5134.153                     |
| 2336.1646        | 1.160071                     |

3. Dans le cas général, le résultat de la décroissance obtenue est le suivant :

| RC-Pas de cauchy | RC-Gradient tronqué conjugué |
|:----------------:|:----------------------------:|
|     591198.87    |           591198.87          |
|     328991.13    |           337809.30          |
|     64598.674    |           65927.96           |
| 12522.069        | -175.9230                    |
| 2336.1646        | -7.765696                    |
| 387.54032        | 1.5709228                    |

4. Chaque approche a des points faibles et forts, la région de confiance avec le pas de Cauchy est une approche qui converge rapidement en unre seule itération, mais malheureusement sa décroissance est faible. En revanche, la régions de confiance avec le gradient tronqué conjugué a une convergence plutôt lente mais une décroissance plus importane que le pas de Cauchy.

# Lagrangien augmenté

## Implémentation
1.Choisir des critères d’arrêt pour la convergence de l'algorithme.

2.Implémenter l'algorithme du lagrangien augmenté, en utilisant les différentes méthodes
qui ont été vues en première partie pour la résolution de la suite de problémes sans
contraintes (fichier `Lagrangien_Augmente.jl`)

3.Tester les différentes variantes sur les problèmes en Annexe D.

```julia
# Vos tests
include("Lagrangien_Augmente.jl")
include("Regions_De_Confiance.jl")
include("Gradient_Conjugue_Tronque.jl")
include("Pas_De_Cauchy.jl")

"Retour sur f1"
contrainte(x) = x[1] + x[3] - 1
grad_contrainte(x) = [1; 0; 1]
hess_contrainte(x) = [0 0 0; 0 0 0; 0 0 0]

"réalisable"
xc = [0; 1; 1]
sol_exacte = [1, 1, 1]
xmin, f_min, flag, nb_iters = Lagrangien_Augmente("gct", f1, contrainte, grad_f1, hess_f1, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_gct", "f1", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("cauchy", f1, contrainte, grad_f1, hess_f1, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_cauchy", "f1", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("newton", f1, contrainte, grad_f1, hess_f1, grad_contrainte, hess_contrainte , xc, [])
my_afficher_resultats("Lagrangien_Augmente_newton", "f1", xc, xmin, f_min, flag, sol_exacte, nb_iters)

"non réalisable"
xc = [0.5; 1.25; 1]
sol_exacte = [1, 1, 1]

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("gct", f1, contrainte, grad_f1, hess_f1, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_gct", "f1", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("cauchy", f1, contrainte, grad_f1, hess_f1, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_cauchy", "f1", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("newton", f1, contrainte, grad_f1, hess_f1, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_newton", "f1", xc, xmin, f_min, flag, sol_exacte, nb_iters)

"Retour sur f2"
contrainte(x) =  (x[1] ^ 2) + (x[2] ^ 2) - 1.5
grad_contrainte(x) = [2 * x[1]; 2 * x[2]]
hess_contrainte(x) = [2 0; 0 2]

"non réalisable"
xc = [1; 0]
sol_exacte = [1, 1, 1]

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("gct", f2, contrainte, grad_f2, hess_f2, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_gct", "f2", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("cauchy", f2, contrainte, grad_f2, hess_f2, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_cauchy", "f2", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("newton", f2, contrainte, grad_f2, hess_f2, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_newton", "f2", xc, xmin, f_min, flag, sol_exacte, nb_iters)

"réalisable"
xc = [sqrt(3) / 2; sqrt(3) / 2]
sol_exacte = [1, 1]

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("gct", f2, contrainte, grad_f2, hess_f2, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_gct", "f2", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("cauchy", f2, contrainte, grad_f2, hess_f2, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_cauchy", "f2", xc, xmin, f_min, flag, sol_exacte, nb_iters)

xmin, f_min, flag, nb_iters = Lagrangien_Augmente("newton", f2, contrainte, grad_f2, hess_f2, grad_contrainte, hess_contrainte, xc, [])
my_afficher_resultats("Lagrangien_Augmente_newton", "f2", xc, xmin, f_min, flag, sol_exacte, nb_iters)
```

    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_gct appliqué à f1 au point initial [0, 1, 1]:
      * xsol = [0, 1, 1]
      * f(xsol) = 3
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_cauchy appliqué à f1 au point initial [0, 1, 1]:
      * xsol = [0, 1, 1]
      * f(xsol) = 3
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_newton appliqué à f1 au point initial [0, 1, 1]:
      * xsol = [0, 1, 1]
      * f(xsol) = 3
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_gct appliqué à f1 au point initial [0.5, 1.25, 1.0]:
      * xsol = [0.5, 1.25, 1.0]
      * f(xsol) = 0.75
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_cauchy appliqué à f1 au point initial [0.5, 1.25, 1.0]:
      * xsol = [0.5, 1.25, 1.0]
      * f(xsol) = 0.75
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_newton appliqué à f1 au point initial [0.5, 1.25, 1.0]:
      * xsol = [0.5, 1.25, 1.0]
      * f(xsol) = 0.75
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_gct appliqué à f2 au point initial [1, 0]:
      * xsol = [1, 0]
      * f(xsol) = 100
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_cauchy appliqué à f2 au point initial [1, 0]:
      * xsol = [1, 0]
      * f(xsol) = 100
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_newton appliqué à f2 au point initial [1, 0]:
      * xsol = [1, 0]
      * f(xsol) = 100
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_gct appliqué à f2 au point initial [0.8660254037844386, 0.8660254037844386]:
      * xsol = [0.8660254037844386, 0.8660254037844386]
      * f(xsol) = 1.3641386247653273
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_cauchy appliqué à f2 au point initial [0.8660254037844386, 0.8660254037844386]:
      * xsol = [0.8660254037844386, 0.8660254037844386]
      * f(xsol) = 1.3641386247653273
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1]
    -------------------------------------------------------------------------
    Résultats de : Lagrangien_Augmente_newton appliqué à f2 au point initial [0.8660254037844386, 0.8660254037844386]:
      * xsol = [0.8660254037844386, 0.8660254037844386]
      * f(xsol) = 1.3641386247653273
      * nb_iters = 0
      * flag = 0
      * sol_exacte : [1, 1]

## Interprétation
 1.Commenter les résultats obtenus, en étudiant notamment les valeurs de $\lambda_k$ et $\mu_k$.

 2.Étudier l'influence du paramètre $\tau$ dans la performance de l'algorithme.

 3.**Supplémentaire** :
      Que proposez-vous comme méthode pour la résolution des problèmes avec
      des contraintes à la fois d'égalité et d'inégalité ? Implémenter (si le temps le permet)
      ce nouvel algorithme

## Vos réponses?

        Les résultas obtenus sont cohérentes, en revanche, la vitesse de convergence est très faible par rapport aux versions précedentes.

    On constate que le $\lambda$ en sortie est très grand, donc la solution du probleme sans contrainte ne satisfait pas aux contraintes appliquees.

    $\mu$ réduit significativement le nombre d’iterations realisées et permet egalement de rendre soluble des résolutions qui prenaient trop de temps. On note également que le $\mu$ final a augmenté à chaque fois, donc les contraintes n’étaient pas respectées et qu’il a fallu pénaliser plusieurs fois afin de les respecter.

L'augmentation de $\tau$ améliore la convergence de l’algorithme. le critère d’arrêt sur la précision de la fonction nous permet de s’attacher plus à la justesse du résultat. La valeur de $\tau$ la plus importante est celle qui fournie donc les meilleures résultats.


# Purpose

The purpose of this notebook is to demonstrate how to simulate data appropriate for use with [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) (LDA) to learn topics. There are a lot of moving parts involved with LDA, and it makes very strong assumptions about how word, topics and documents are distributed. In a nutshell, the distributions are all based on the [Dirichlet-Multinomial distribution](https://en.wikipedia.org/wiki/Dirichlet-multinomial_distribution), and so if you understand that compound distribution, you will have an easier time understanding how to sample the topics (from the document) and the words (from the topic). At any rate, the Wikipedia site does a good enough job to enumerate the moving parts; here they are for completeness.

* $K$ is the number of topics
* $N$ is the number of words in a document; sometimes also denoted as $V$; when the number of words vary from document to document, then $N_d$ is the number of words for the $d$ document; here we assume $N$, $V$ and $N_d$ are all the same
* $M$ is the number of documents
* $\alpha$ is a vector of length $K$ on the priors of the $K$ topics; these alpha are `sparse` (less than 1)
* $\beta$ is a vector of length $N$ on the priors of the $N$ words; typically these are `symmetric` (all set to the same value e.g. 0.001)
* $\theta$ is the $M$ by $K$ matrix of document-topic (documents to topics) where each element is $P(K=k|D=d)$
* $\varphi$ is the $K$ by $V$ matrix of topic-word (topics to words) where each element is $P(W=w|K=k)$

The Wikipedia article states the sampling as follows.

$
\begin{align}
\boldsymbol\varphi_{k=1 \dots K} &\sim \operatorname{Dirichlet}_V(\boldsymbol\beta) \\
\boldsymbol\theta_{d=1 \dots M} &\sim \operatorname{Dirichlet}_K(\boldsymbol\alpha) \\
z_{d=1 \dots M,w=1 \dots N_d} &\sim \operatorname{Categorical}_K(\boldsymbol\theta_d) \\
w_{d=1 \dots M,w=1 \dots N_d} &\sim \operatorname{Categorical}_V(\boldsymbol\varphi_{z_{dw}})
\end{align}
$

Note the following.

* $z_{dw} \in [1 \ldots K]$ ($z_{dw}$ is an integer between 1 and $K$) and serves as a pointer back to $\varphi_k$ (the k-th row in $\varphi$ that you will use as priors to sample the words)
* $w_{dw} \in [1 \ldots N]$ ($w_{dw}$ is an integer between 1 and $N$) which is the n-th word
* $z_{dw}$ is actually sampled from $\operatorname{Multinomial}(\boldsymbol\theta_d)$ taking the arg max, e.g. $z_{dw} \sim \underset{\theta_d}{\operatorname{arg\,max}}\ \operatorname{Multinomial}(\boldsymbol\theta_d)$
* $w_{dw}$ is actually sampled from $\operatorname{Multinomial}(\boldsymbol\varphi_{z_{dw}})$ taking the arg max, e.g. $z_{dw} \sim \underset{\boldsymbol\varphi_{w_{dw}}}{\operatorname{arg\,max}}\ \operatorname{Multinomial}(\boldsymbol\varphi_{z_{dw}})$

The code below should make it clear as there are a lot of sub-scripts and moving parts.

# Simulate the data

Let's get ready to sample. Note the following.

* $K = 10$ (ten topics)
* $N = 100$ (one hundred words)
* $M = 1000$ (one thousand documents)
* $\alpha = [0.1, 0.2, 0.3, 0.4, 0.025, 0.015, 0.37, 0.88, 0.03, 0.08]$ (10 sparse priors on topics)
* $\beta = [0.001 \ldots 0.001]$ (100 symetric priors on words)

Below, we store the sampled documents and associated words in

* `texts` as string literal (e.g. w1 w1 w83 ....)
* `docs` as a dictionary of counts (e.g. { 1: 2, 83: 1, ...})

The matrices

* `C` stores the counts
* `X` stores the [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) values

```python
%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import dirichlet, multinomial
from scipy.sparse import lil_matrix
import pandas as pd
from sklearn.feature_extraction.text import TfidfTransformer

np.random.seed(37)

# number of topics
K = 10
# number of words
N = 100
# number of documents
M = 1000

# priors on K topics
a = np.array([0.1, 0.2, 0.3, 0.4, 0.025, 0.015, 0.37, 0.88, 0.03, 0.08])
# priors on N words
b = np.full((1, N), 0.001, dtype=float)[0]

# distribution of words in topic k
phi = np.array([dirichlet.rvs(b)[0] for _ in range(K)])

# distribution of topics in document d
theta = np.array([dirichlet.rvs(a)[0] for _ in range(M)])

# simulate the documents
texts = []
docs = []

# for each document
for i in range(M):
    d = {}
    t = []

    # for each word
    for j in range(N):
        # sample the possible topics
        z_ij = multinomial.rvs(1, theta[i])
        # get the identity of the topic; the one with the highest probability
        topic = np.argmax(z_ij)

        # sample the possible words from the topic
        w_ij = multinomial.rvs(1, phi[topic])
        # get the identity of the word; the one with the highest probability
        word = np.argmax(w_ij)

        if word not in d:
            d[word] = 0
        d[word] = d[word] + 1
        t.append('w{}'.format(word))
    docs.append(d)
    texts.append(' '.join(t))

# make a nice matrix
# C is a matrix of word counts (rows are documents, columns are words, elements are count values)
C = lil_matrix((M, N), dtype=np.int16)
for i, d in enumerate(docs):
    counts = sorted(list(d.items()), key=lambda tup: tup[0])
    for tup in counts:
        C[i, tup[0]] = tup[1]

# X is a matrix of tf-idf (rows are documents, columns are words, elements are tf-idf values)
X = TfidfTransformer().fit_transform(C)
```

# Gaussian mixture models (GMMs)

Let's see if GMMs can help us recover the number of topics using the [AIC](https://en.wikipedia.org/wiki/Akaike_information_criterion) score to guide us.

```python
from scipy.sparse.linalg import svds
from sklearn.mixture import GaussianMixture

def get_gmm_labels(X, k):
    gmm = GaussianMixture(n_components=k, max_iter=200, random_state=37)
    gmm.fit(X)
    aic = gmm.aic(X)
    print('{}: aic={}'.format(k, aic))
    return k, aic

U, S, V = svds(X, k=20)
gmm_scores = [get_gmm_labels(U, k) for k in range(2, 26)]
```

    2: aic=-91377.4925931899
    3: aic=-115401.48064693023
    4: aic=-140093.33933540556
    5: aic=-140323.78987370015
    6: aic=-141875.7608870883
    7: aic=-148775.55233751616
    8: aic=-144864.34044251204
    9: aic=-145063.4922621106
    10: aic=-150715.19037699007
    11: aic=-152996.5234889565
    12: aic=-155759.24880410862
    13: aic=-154738.52657589084
    14: aic=-155298.3570419242
    15: aic=-155273.86266190943
    16: aic=-158229.54424744606
    17: aic=-158801.92826365907
    18: aic=-158146.93107164893
    19: aic=-157399.88209837917
    20: aic=-158964.20247723104
    21: aic=-156443.29839085325
    22: aic=-156545.28924475564
    23: aic=-156265.51016605442
    24: aic=-155860.4914350854
    25: aic=-157396.56289736537

# k-means clustering (KMC)

Let's see if KMC can help us to recover the number of topics using the [Silhouette score](https://en.wikipedia.org/wiki/Silhouette_%28clustering%29) to guide us.

```python
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

def get_kmc(X, k):
    model = KMeans(k, random_state=37)
    model.fit(X)
    labels = model.predict(X)
    score = silhouette_score(X, labels)
    print('{}: score={}'.format(k, score))
    return k, score

kmc_scores = [get_kmc(X, k) for k in range(2, 26)]
```

    2: score=0.22136552497539078
    3: score=0.2606191325546754
    4: score=0.2985364557161296
    5: score=0.32764563696557253
    6: score=0.34711980577628615
    7: score=0.36212754809252495
    8: score=0.3693035922796191
    9: score=0.3118628444238988
    10: score=0.32070416934016466
    11: score=0.3056882384904699
    12: score=0.28297903762485543
    13: score=0.28462816984240946
    14: score=0.2747613933318139
    15: score=0.2787478862359055
    16: score=0.27452088253304896
    17: score=0.2548015324435892
    18: score=0.25961952207924777
    19: score=0.25650479556223627
    20: score=0.251690199350559
    21: score=0.2566617758778615
    22: score=0.25866268014756943
    23: score=0.24607465357359543
    24: score=0.24936289940720038
    25: score=0.2579644562276278

# LDA modeling

Here, we will use LDA topic modeling technique and the [coherence score](https://radimrehurek.com/gensim/models/coherencemodel.html) to guide us recovering the number of topics.

```python
from gensim import corpora
from gensim.models import LdaModel
from gensim.models.coherencemodel import CoherenceModel

def learn_lda_model(corpus, dictionary, k):
    lda = LdaModel(corpus,
               id2word=dictionary,
               num_topics=k,
               random_state=37,
               iterations=100,
               passes=5,
               per_word_topics=False)
    cm = CoherenceModel(model=lda, corpus=corpus, coherence='u_mass')
    coherence = cm.get_coherence()
    print('{}: {}'.format(k, coherence))
    return k, coherence

T = [t.split(' ') for t in texts]

dictionary = corpora.Dictionary(T)
corpus = [dictionary.doc2bow(text) for text in T]

lda_scores = [learn_lda_model(corpus, dictionary, k) for k in range(2, 26)]
```

    2: -7.112621491925517
    3: -6.770771537876562
    4: -6.654850158110881
    5: -6.495525290205532
    6: -6.592127872424598
    7: -6.4394384370150854
    8: -6.431505215171467
    9: -6.376827700591723
    10: -6.207008469326988
    11: -6.235774265382583
    12: -6.260016126133871
    13: -6.254881861190534
    14: -6.550148968159432
    15: -6.6008249817300415
    16: -6.560176401338963
    17: -6.549319733769306
    18: -6.669241989725236
    19: -6.767980107114681
    20: -6.760030166955779
    21: -6.8683429580215165
    22: -6.6622351856878375
    23: -6.773847370134338
    24: -6.735329093161339
    25: -6.676802294304821

# Visualize the techniques and scores versus the number of topics

Here, we visualize the scores (GMM AIC, KMC Silhouette and LDA Coherence) versus the number of topics (k). For AIC, the lower the score, the better; for silhouette, the higher the better; for coherence, the higher the better. It seems that KCM's silhouette does not really agree with AIC or coherence; and AIC and coherence (although negative correlated) seem to hint at the same number of topics.

When relying on LDA and coherence, k=10 is the highest, as we'd expect since we simulated the data from 10 latent/hidden topics.

```python
def plot_scores(scores, ax, ylabel):
    _x = [s[0] for s in scores]
    _y = [s[1] for s in scores]

    ax.plot(_x, _y, color='tab:blue')
    ax.set_xlabel('k')
    ax.set_ylabel(ylabel)
    ax.set_title('{} vs k'.format(ylabel))

fig, ax = plt.subplots(1, 3, figsize=(15, 5))
plot_scores(gmm_scores, ax[0], 'GMM AIC')
plot_scores(kmc_scores, ax[1], 'KMC Sillhouette')
plot_scores(lda_scores, ax[2], 'LDA Coherence')

plt.tight_layout()
```

# Visualize the topics

This visualization tool allows us to `interrogate` the topics. As we hover over each topic, the words most strongly associated with them are show.

```python
import pyLDAvis.gensim
import warnings

warnings.filterwarnings('ignore')

lda = LdaModel(corpus,
               id2word=dictionary,
               num_topics=10,
               random_state=37,
               iterations=100,
               passes=5,
               per_word_topics=False)

lda_display = pyLDAvis.gensim.prepare(lda, corpus, dictionary, sort_topics=False)
pyLDAvis.display(lda_display)
```

<link rel="stylesheet" type="text/css" href="https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css">

# Close to real-world example

Here's a list of 10 book titles when searching on `programming` and `economics` from Amazon (5 each). Again, when the number of topics is k=2, that model has the highest coherence score.

```python
import nltk
from nltk.corpus import wordnet as wn
from nltk.stem import PorterStemmer

def clean(text):
    t = text.lower().strip()
    t = t.split()
    t = remove_stop_words(t)
    t = [get_lemma(w) for w in t]
    t = [get_stem(w) for w in t]
    return t

def get_stem(w):
    return PorterStemmer().stem(w)

def get_lemma(w):
    lemma = wn.morphy(w)
    return w if lemma is None else lemma

def remove_stop_words(tokens):
    stop_words = nltk.corpus.stopwords.words('english')
    return [token for token in tokens if token not in stop_words]

texts = [
    'The Art of Computer Programming',
    'Computer Programming Learn Any Programming Language In 2 Hours',
    'The Self-Taught Programmer The Definitive Guide to Programming Professionally',
    'The Complete Software Developers Career Guide How to Learn Your Next Programming Language',
    'Cracking the Coding Interview 189 Programming Questions and Solutions',
    'The Economics Book Big Ideas Simply Explained',
    'Economics in One Lesson The Shortest and Surest Way to Understand Basic Economics',
    'Basic Economics',
    'Aftermath Seven Secrets of Wealth Preservation in the Coming Chaos',
    'Economics 101 From Consumer Behavior to Competitive Markets Everything You Need to Know About Economics'
]

texts = [clean(t) for t in texts]

dictionary = corpora.Dictionary(texts)
dictionary.filter_extremes(no_below=3)
corpus = [dictionary.doc2bow(text) for text in texts]

lda_scores = [learn_lda_model(corpus, dictionary, k) for k in range(2, 10)]
```

    2: -26.8263021597115
    3: -26.863492751597203
    4: -26.88208804754005
    5: -26.848616514842924
    6: -26.9006833434829
    7: -26.874118634993117
    8: -26.88208804754005
    9: -26.863492751597203

Learn the model with 2 topics.

```python
lda = LdaModel(corpus,
               id2word=dictionary,
               num_topics=2,
               random_state=37,
               iterations=100,
               passes=20,
               per_word_topics=False)
```

Print what the model predicts for each book title. Note the 9-th book title is a tie (50/50)? Otherwise, all the predictions (based on highest probabilities) are correct.

```python
corpus_lda = lda[corpus]

for d in corpus_lda:
    print(d)
```

    [(0, 0.25178075), (1, 0.74821925)]
    [(0, 0.16788821), (1, 0.8321118)]
    [(0, 0.25178382), (1, 0.74821615)]
    [(0, 0.2517796), (1, 0.7482204)]
    [(0, 0.25178117), (1, 0.74821883)]
    [(0, 0.748248), (1, 0.25175205)]
    [(0, 0.83213073), (1, 0.16786923)]
    [(0, 0.7482476), (1, 0.25175238)]
    [(0, 0.5), (1, 0.5)]
    [(0, 0.8321299), (1, 0.16787016)]

The first topic is about `econom` (economics) and the second about `programming`, as we'd expect. Observe how each topic has a little of the other's words? This observation is the result of the assumption from LDA that documents are a mixture of topics and topics have distributions over words.

```python
lda.print_topics()
```

    [(0, '0.926*"econom" + 0.074*"program"'),
     (1, '0.926*"program" + 0.074*"econom"')]

This book title is a `holdout` title from the economics search result. It is correctly placed in the 0-th topic (economics).

```python
lda[dictionary.doc2bow(clean('Naked Economics Undressing the Dismal Science'))]
```

    [(0, 0.7482481), (1, 0.2517519)]

This book title is a `holdout` title from the programming search result. It is correctly placed in the 1-st topic (programming).

```python
lda[dictionary.doc2bow(clean('Elements of Programming Interviews in Python The Insiders Guide'))]
```

    [(0, 0.2517816), (1, 0.7482184)]

Since this example is trivial, the visualization is not very interesting, but displayed below anyways.

```python
lda_display = pyLDAvis.gensim.prepare(lda, corpus, dictionary, sort_topics=False)
pyLDAvis.display(lda_display)
```

<link rel="stylesheet" type="text/css" href="https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css">

```python

```


# Physics 256
## Simpson's Rule

## Last Time

- Solving for the magnetic field
- Approximating integrals
 - Rectangular rule
 - Trapezoidal rule

## Today

- Simpson's rule
- Improper and divergent integrals

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('notebook');
%config InlineBackend.figure_format = 'retina'
colors = ["#2078B5", "#FF7F0F", "#2CA12C", "#D72827", "#9467BE", "#8C574B",
            "#E478C2", "#808080", "#BCBE20", "#17BED0", "#AEC8E9", "#FFBC79",
            "#98E08B", "#FF9896", "#C6B1D6", "#C59D94", "#F8B7D3", "#C8C8C8",
           "#DCDC8E", "#9EDAE6"]
```

## Better Quadrature

We want to come up with a better scaling approximation to the definite integral:

\begin{equation}
I = \int_a^b f(x) dx
\end{equation}

where we break up the region of integration into $N$ equally sized regions of size:

\begin{equation}
\Delta x = \frac{b-a}{N}.
\end{equation}

Our previous methods approximated the integral of a single panel $I_i = \int_{x_i}^{x_{i+1}} f(x) dx$ using a $0^{th}$ or $1^{st}$ order polynomial.  Now, let's use a $2^{nd}$ order polynomial:

\begin{equation}
P(x) = \alpha + \beta x + \gamma x^2
\end{equation}

where we need to fix the coefficients $\alpha,\beta,\gamma$ by matching:

\begin{equation}
\int_{x_i}^{x+2} P(x) dx \approx \int_{x_i}^{x_{i+2}} f(x) dx .
\end{equation}

The final answer is:

\begin{equation}
I_i + I_{i+1} \approx \frac{\Delta x}{3} \left[f(x_i) + 4 f(x_{i+1}) + f(x_{i+2}) \right]
\end{equation}

which needs to be summed over all panels:

\begin{equation}
I_{\rm simps} \approx \frac{\Delta x}{3} \left[f(a) + f(b) + 4\sum_{i=1}^{N/2} f(x_{2i-1}) + 2\sum_{i=1}^{N/2-1} f(x_{2i}) \right]
\end{equation}

Note: we need an even number of panels for this to work.

Let's code it up!

```python
def simpsons_rule(f,x,*params):
    '''The trapezoidal rule for numerical integration of f(x) over x.'''
    a,b = x[0],x[-1]
    Δx = x[1] - x[0]
    N = x.size

    #I = (f(a,*params) + f(b,*params))/3.0
    #I += (4.0/3.0)*np.sum([f(a + j*Δx,*params) for j in range(1,N,2)])
    #I += (2.0/3.0)*np.sum([f(a + j*Δx,*params) for j in range(2,N,2)])

    I = (f(a,*params) + f(b,*params))/3.0
    I += (4.0/3.0)*sum([f(a+i*Δx,*params) for i in range(1,N,2)])
    I += (2.0/3.0)*sum([f(a+i*Δx,*params) for i in range(2,N,2)])

    return Δx*I
```

Programming challenge

Use Simpson's rule to evaluate the error function in the range $x\in [0,1]$. Make sure to keep the panel width $\Delta x$ fixed for each value of $x$. Compare with the exact result from scipy.special.

\begin{equation}
\mathrm{erf}(x) = \frac{2}{\sqrt{\pi}} \int_0^x \mathrm{e}^{-t^2} dt
\end{equation}

```python
from scipy.constants import pi as  π
from scipy.special import erf

def erf_kernel(t):
    '''The error function kernel.'''
    return  (2.0/np.sqrt(π))*np.exp(-t*t)

Δx = 0.001
x = np.linspace(0,1,20)
erf_approx = np.zeros_like(x)

for j,cx in enumerate(x[1:]):
    N = int(cx/Δx)
    if N % 2: N += 1
    x_int = np.linspace(0,cx,N)
    erf_approx[j+1] = simpsons_rule(erf_kernel,x_int)

# plot the results and compare with the 'exact' value
plt.plot(x,erf_approx,'o', mec=colors[0], mfc=colors[0], mew=1, ms=8, label="Simpson's Rule")
plt.plot(x,erf(x), color=colors[1],zorder=0, label='scipy.special.erf')
plt.legend(loc='lower right')
plt.xlabel('x')
plt.ylabel('erf(x)')
```

## Improper Integrals

### Infinite Intervals

Suppose we want to evaluate a definite integral on a semi-infinte interval: $[a,\infty]$ or $[-\infty,b]$.  We can proceed by finding a function $\phi$ which maps the semi-infinite region to a finite one.

\begin{equation}
I = \int_a^b f(x) dx
\end{equation}

with $a=-\infty$ or $b = \infty$ but not both.

#### (i) $a\ne 0 \text{ and } b \ne 0$

Consider $y = \phi(x) = \frac{1}{x}$, then:

\begin{align}
I &= \int_a^b f(x) dx \newline
&= \int_{1/a}^{1/b} dy \left(-\frac{1}{y^2}\right) f\left(\frac{1}{y}\right) \newline
&= \int_{1/b}^{1/a} \frac{dy}{y^2} f\left(\frac{1}{y}\right) .
\end{align}

#### (ii) $a = 0 \text{ and } b = \infty$

Consider $y = \phi(x) = \frac{x}{1+x}$, then:

\begin{align}
y (1+x) &= x \newline
x(1-y) &= y \newline
x &= \phi^{-1}(y) = \frac{y}{1-y}
\end{align}

so:

\begin{equation}
dx = dy \left[\frac{1}{1-y} + \frac{y}{(1-y)^2} \right] \Rightarrow dx = \frac{dy}{(1-y)^2}.
\end{equation}

Finally

\begin{align}
I &= \int_0^\infty f(x) dx \newline
&= \int_{0}^{1} dy\frac{dy}{(1-y)^2} f\left(\frac{y}{1-y}\right) .
\end{align}

### Example
Evaluate the integral:

\begin{equation}
I = \int_0^\infty \frac{dx}{\sqrt{x^4+1}}.
\end{equation}

Initially it looks like we have a problem as we can't directly evaluate $y/(1-y)$ when $y\to 1$ numerically. However, defind a new function:

\begin{align}
g(y) &= \frac{1}{(1-y)^2} f\left(\frac{y}{1-y}\right) \newline
 &= \frac{1}{\sqrt{y^4 + (1-y)^4}}
\end{align}

which has no numerical singularities.

```python
def f(x):
    return 1.0/np.sqrt(x**4 + 1)

def g(y):
    return 1.0/np.sqrt(y**4 + (1-y)**4)

y = np.linspace(0,1,100000)
print(simpsons_rule(g,y))
```

    1.85408467737

## Scipy.Integrate

Much of what we have learned has already been coded up for you in [`scipy.integrate`](https://docs.scipy.org/doc/scipy-0.18.1/reference/integrate.html) and you should learn how to use these routines.  The workhorse is the [`quad`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.integrate.quad.html#scipy.integrate.quad) method which is fast, highly accurate and very flexible.  It returns a tuple giving the result of the integral and an estimate of the error.

```python
from scipy import integrate
print(integrate.quad(f, 0, np.inf))
```

    (1.854074677301372, 2.425014482827383e-10)

```python

```


# 04:  Metropolis Monte Carlo for Classical Systems

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
try: plt.style.use('./mc_notebook.mplstyle')
except: pass
```

## Evaluating Phase Space Integrals

We have seen that Monte Carlo methods are an ideal way to sample high-dimensional integrals.  We now ask if this can be applied to the phase space integrals that appear when computing equilbrium observables in classical statistical mechanics for $N$ interacting particles.

Starting with the general $N$-body Hamiltonian:

\begin{equation}
H = -\frac{\hbar^2}{2m} \sum_{i=1}^{N} \nabla_i^2 + \frac{1}{2}\sum_{i,j} V_{ij}
\end{equation}

we are interested in the expectation value of some observable $\mathcal{O}$:

\begin{equation}
\langle \mathcal{O} \rangle = \frac{1}{\mathcal{Z}} \int d^{d} x_1 \cdots \int d^{d} x_N\, \mathcal{O}(\mathbf{x}) \mathrm{e}^{-\beta V(\mathbf{x})}
\end{equation}

where the partition function is given by $(\beta = 1/k_{\rm B} T)$:

\begin{equation}
\mathcal{Z} = \int d^{d} x_1 \cdots \int d^{d} x_N\,  \mathrm{e}^{-\beta V(\mathbf{x})}.
\end{equation}

We can immediatly identify this with our previous importance sampling scheme:

\begin{equation}
\langle \mathcal{O} \rangle = \int d^{d} x_1 \cdots \int d^{d} x_N\, \mathcal{O}(\mathbf{x}) \pi( \mathbf{x}) \simeq \frac{1}{M} \sum_m \mathcal{O}(\mathbf{x}_m)
\end{equation}

where

\begin{equation}
\pi(\mathbf{x}) = \frac{\mathrm{e}^{-\beta V(\mathbf{x})}}{\mathcal{Z}}
\end{equation}

is the Boltzmann weight of configuraiton $\mathbf{x}$.

### Key Problem:

How do we effeciently sample the correct Boltzmann probability distribution?

Solved by:

[N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, *Equation of State Calculations by Fast Computing Machines*, J. Chem. Phys. **21**, 1087 (1953).](https://aip.scitation.org/doi/abs/10.1063/1.1699114)

## Markov Chain Monte Carlo

Since $\pi(\mathbf{x})$ is an extremely complicated $dN$-dimensional probability distribution there is no hope of direct sampling.  However, we can recall our strategy to the *pebble throwing game* when we imagined that the entire area couldn't be reached with a single toss: **Markov chain sampling**.

```python
from IPython.display import display

try:
    import graphviz
    markov_chain = graphviz.Digraph()
    markov_chain.attr(rankdir='LR')
    for i in range(2**3):
        markov_chain.node(name="x%d"%i, label="<<i><b>x</b></i><sub>%d</sub>>"%i, shape='circle')

    for i in range(2**3-1):
        markov_chain.edge("x%d"%i, "x%d"%(i+1))

    display(markov_chain)

except:
    pass
```

The goal is to build a Markov Chain consisting of a sequence of valid configuraitons $\mathbf{x}_n$ such that $\pi(\mathbf{x}_n)$ reflects the stationary probability of the system being in configuraiotn $\mathbf{x}_n$.

We achieve this via a set of *transition probabilities* $p_{n,m} = p(\mathbf{x}_n \to \mathbf{x}_{m})$ which encode the liklihood of moving from configuration $\mathbf{x}_n \to \mathbf{x}_{m}$.  We require:

### 1. Normalization

\begin{equation}
\sum_{m} p_{n,m} = 1
\end{equation}

### 2. Detailed Balance

The probability of configuration $\mathbf{x}_n$, $\pi_n \equiv \pi(\mathbf{x}_n)$, is generated via all possible ways of obtaining it (i.e. the target distribution is stationary):

\begin{align*}
\pi_n &= \sum_{m} \pi_{m} p_{m,n} \newline
\pi_n(1-p_{n,n}) &= \sum_{m\ne n} \pi_{m} p_{m,n} .
\end{align*}

Now utilizing the normalization condition:
\begin{equation}
\sum_{m} p_{n,m} = 1 \Rightarrow 1-p_{nn} = \sum_{m\ne n} p_{n,m}
\end{equation}

above we can write:

\begin{equation}
\pi_n \sum_{m \ne n} p_{n,m} = \sum_{m\ne n} \pi_{m} p_{m,n}.
\end{equation}

One obvious solution to this equation which is sufficient but not necessary is to equate each term individually (this is *detailed balance*):

\begin{equation}
\boxed{\pi_n  p_{n,m} = \pi_{m} p_{m,n}}.
\end{equation}

### 3. Ergodicity

Any configuraiton is reachable from any other, i.e. $\forall m,n,\ \exists a \, : (p^a)_{m,n} \ne 0$. The graph corresponding to the Markov chain is connected.

## Metropolis Algorithm

Teller's solution to generating a Markov Chain satisfying these conditions was to use the rejection sampling scheme we have already employed in combination with choosing the probability to move from configuration $\mathbf{x}_n \to \mathbf{x}_m$ to be composite:

\begin{equation}
\mathcal{P}(\mathbf{x}_n \to \mathbf{x}_m) = \mathcal{A}(\mathbf{x}_n \to \mathbf{x}_m) p(\mathbf{x}_n \to \mathbf{x}_m)
\end{equation}

where $\mathcal{A}(\mathbf{x}_n \to \mathbf{x}_m)$ is an *a priori* proposal probability and $p(\mathbf{x}_n \to \mathbf{x}_m)$ is the acceptance probability.

Our *detailed balance* condition requires:

\begin{equation}
\pi(\mathbf{x}_n) \mathcal{P}(\mathbf{x}_n \to \mathbf{x}_m) =\pi(\mathbf{x}_m) \mathcal{P}(\mathbf{x}_m \to \mathbf{x}_n)
\end{equation}

\begin{equation}
\frac{p(\mathbf{x}_n \to \mathbf{x}_m)}{p(\mathbf{x}_m \to \mathbf{x}_n)} = \frac{\pi(\mathbf{x}_m)\mathcal{A}(\mathbf{x}_m \to \mathbf{x}_n)}{\pi(\mathbf{x}_n)\mathcal{A}(\mathbf{x}_n \to \mathbf{x}_m)}
\end{equation}

which is satified by the choice:

\begin{equation}
p(\mathbf{x}_n \to \mathbf{x}_m) = \text{min}\left[1,\frac{\pi(\mathbf{x}_m)\mathcal{A}(\mathbf{x}_m \to \mathbf{x}_n)}{\pi(\mathbf{x}_n)\mathcal{A}(\mathbf{x}_n \to \mathbf{x}_m)}\right].
\end{equation}

The idea is to :

1. Starting from configuration $\mathbf{x}_n$.
2. Propose some change to the configuration with *a priori* proposal probability $\mathcal{A}(\mathbf{x}_n \to \mathbf{x}_m)$.
3. Accept the change with probability $p(\mathbf{x}_n \to \mathbf{x}_m)$ given by the expression above where $\pi(\mathbf{x})$ is known (i.e. the Boltzmann weight) and $\mathcal{A}(\mathbf{x}_n \to \mathbf{x}_m)$ can be chosen.

```python
try:
    markov_chain_snippet = graphviz.Digraph()
    markov_chain_snippet.attr(rankdir='LR')

    markov_chain_snippet.node(name="x%d"%0, label="⋯", shape='none')
    markov_chain_snippet.node(name="x%d"%1, label="<<i><b>x</b></i><sub><i>n</i></sub>>", shape='circle')
    markov_chain_snippet.node(name="x%d"%2, label="<<i><b>x</b></i><sub><i>m</i></sub>>", shape='circle')
    markov_chain_snippet.node(name="x%d"%3, label="⋯", shape='none')

    for i in range(3):
        markov_chain_snippet.edge("x%d"%i, "x%d"%(i+1))

    display(markov_chain_snippet)
except:
    pass
```

### How do we sample the $p$ distribution?

For a given configuration $\mathbf{x}_n$, some update scheme that changes to configuration $\mathbf{x}_m$, and known functions $\pi$ and $\mathcal{A}$, we can sample $p$ via rejection:

1. Compute $p$ as a real number using the formula above.
2. Generate a uniform random number $r \in \mathcal{U}_{[0,1)}$
3. If $r < p$ accept the configuration change to $\mathbf{x}_m$; otherwise reject and return to configuration $\mathrm{x}_n$.

#### Let's test this approach for the 2D classical Ising Model!