# An Introduction to Plotting and Data Analysis in Python 5

Clyde Fare and João Pedro Malhado, Imperial College London (contact: [python@imperial.ac.uk](python@imperial.ac.uk))

Exercise 1 uses material adapted from work by Paul Wilde. Exercise 2 is adapted and uses the dataset from the article F. J. Anscombe, The American Statistician, 27(1):17-21 (1973). This notebook is licensed under a [Creative Commons Attribution 4.0 (CC-by) license](http://creativecommons.org/licenses/by/4.0/)

Let's load the pylab and non-linear fitting machinery and get that out of the way.

```python
%pylab inline
from scipy.optimize import curve_fit
```

## Exercise 1

In this exercise we will be studying the kinetics of the decomposition reaction of N2O5

$$2 N_2 O_{5 (g)} \longrightarrow 4NO_{2 (g)} + O_{2 (g)}$$

The rate of a chemical reaction is how fast the concentration of one of the reactants or products changes during the chemical reaction. In the case of the reaction under study the rate of the reaction is proportional to the concentration of N2O5, it is said to obey a first order rate law:

$$rate=- \frac{d[N_2 O_5]}{d t} =k [N_2 O_5] ,$$

where *k* is the *rate constant*. This is a differential equation and its solution gives us the concentration of N2O5 as a function of time.

$$[N_2 O_5]=[N_2 O_5]_0 e^{-k t}￼$$

where [N2O5]0 is the initial concentration and *t* is the time. (If you want to know where the solution comes from click [here](https://wiki.ch.ic.ac.uk/wiki/index.php?title=Solving_the_Rate_Equation).)

We are interested in determining the value of the rate constant *k*. In order to do this, the concentration of N2O5 was recorded as a function of time at 65°C, and stored in file [N2O5vst_65C.dat](files/N2O5vst_65C.dat), where the first column is time in seconds and the second column is concentration in mol.dm-3.

* Load the data and plot the concentration of N2O5 against time, label the axes and include the units in the label.

```python

```

* Create an appropriate python function for the variation of [N2O5] with time that through non linear fitting will allow us to extract the rate constant

* Fit the function to the data and report the value of the rate constant and the uncertainty associated with it.

* Superimpose the fit with the experimental data to check your fit is meaningful.

```python

```

```python

```

```python

```

Some other colleagues followed the same procedure as above but at different temperatures. The following table shows values of the rate constant and its uncertainty for different temperatures.

| T/°C | k/s-1 | σk/s-1 |
|------------------------------------------------------------------|
| 20       | 0.0000188        | 0.0000089                          |
| 30       | 0.0000694        | 0.0000073                          |
| 40       | 0.000260         | 0.000023                           |
| 50       | 0.000888         | 0.000067                           |
| 60       | 0.00293          | 0.00021                            |

The variation of the rate constant with temperature can give important information about the chemical reaction. The rate constant of an elementary reaction varies in general with temperature according to the Arrhenius equation that you will cover in more detail in your kinetics lectures

$$k(T)=A e^{-\frac{E_a}{R T}} ,$$

where *Ea* is activation energy, *A* is the so called pre-exponential factor (which is related to the frequency of molecular collisions) and *R* is the ideal gas constant.

* Create an appropriate function that through non linear fitting will allow us to extract the pre-exponential factor and the activation energy.

* From the rate constant values on the table above, and the rate constant value you have determined, fit the data to the determine the pre-exponential factor and the activation energy for this reaction.

* Plot values predicted from your fitted function alongside the experimental values to visualise your result.

*Note 1*: The uncertainties associated with each rate constant are different, you should take this into account.

*Note 2*: Units of temperature must be consistent with units of *R*.

```python

```

```python

```

```python

```

If you are having difficulties finding a suitable initial guess for the non-linear Arrhenius fit, or if you managed to extract your activation energy quickly try a different approach:

The Arrhenius equation is a non-linear function of *T*, but can be converted into a linear form by taking the natural logarithm of both sides of the equation:

$$\ln(k)=\ln(A)-\frac{E_a}{R}\frac{1}{T} .$$

We obtain a linear relationship between $\ln(k)$ and $\frac{1}{T}$, where the intercept is $\ln(A)$ and the slope $-\frac{E_a}{R}$. It should be noted that it is only possible in a few cases to transform a non-linear equation into a linear one in this way.

* Plot $\ln(k)$ against $\frac{1}{T}$
* Use polyfit to extract the gradient and intercept of the $\ln(k)$ vs. $\frac{1}{T}$ data (ignore any uncertainties associated with the data when performing this fit)
* Use these to determine the activation energy and the pre-exponential factor.
* How do your results compare with the non-linear fit above?

```python

```

```python

```

```python

```

In the previous question we performed a fit without taking the uncertainties associated with the data into account. This is equivalent to considering all the *ln(k)* data points have the same uncertainty. From the *T* vs *k* table above we know this is not the case, and we should take the uncertainties into account. Note however that *σk* is not the same as *σln(k)*, and also *σln(k)≠ln(σk)*! There are [general methods](http://en.wikipedia.org/wiki/Propagation_of_uncertainty#Simplification) to propagate the uncertainty of a value upon a mathematical operation. For the present purposes however, we will focus on the case of the logarithm at hand:

$$\sigma_{\ln(k)}=\frac{\sigma_k}{k} .$$

* Plot the values of *ln(k)* as a function of *1/T* with appropriate error bars. Note the values of the error bars at low *T* values. What do you expect the relative contribution of these data points to the linear fit to be?

* Fit the data taking into account the error bars (recall the differences between functions *polyfit* and *curve_fit* in dealing with data point uncertainty). How do the results obtained via this procedure compare with the ones obtained via direct exponential fit?

```python

```

```python

```

## Exercise 2

The dataset in the CSV file anscombe.csv is formed by 4 sets of x and y values in the form

Set 1	Set 2	Set 3	Set 4

x	y	x	y	x	y	x	y

* Load the data onto the notebook
* Calculate the mean of x and y for all sets

```python

```

```python

```

* Do a linear fit of y against x for each set.

```python

```

* To better organize the data, create an array of the form:

$$\begin{align}\text{array}([&[\bar{x}_1,\bar{y}_1,m_1,b_1],\\&[\bar{x}_2,\bar{y}_2,m_2,b_2],\\&...])\end{align}$$

where m1, b1, etc. are respectively the slope and intercept of the fit to data set 1

```python

```

Compare the values you obtained for each data set. Based on your analysis so far, could you conclude that the four sets of data correspond to the same experiment and agree with each other?

* Plot your data along with your linear fit in each case.

```python

```

Would you now reach the same conclusion to the question above?

*Conclusion: You should always plot your data! Do not blindly rely on statistical measures.*

For which of the datasets is the linear fit meaningful? Plotting the residuals helps you answering this question.

```python

```


# El método de Newton para intervalos

Igual que en la clase anterior, supondremos que $f(x)$ es una función continuamente diferenciable y que existe un valor $x^*$ donde $f(x^*)=0$ (es decir, $x^*$ es una raíz de $f$). Aquí abordaremos de nuevo cómo encontrar las raíces de $f$ dentro de un intervalo inicial $X$. Supondremos además que existe una extensión del intervalo $F'$ para la derivada $f'$, y que (por el momento) éste no contiene al cero: $0\notin F'(X)$.

La clave del método de Newton para intervalos está en aplicar el *teorema del valor medio*, que nos asegura que para cualquier $x\in X$ se cumple que

\begin{equation}
f(x) = f(x^*) + f'(\xi) \cdot (x-x^*),
\end{equation}

para algúna $\xi$ entre $x$ y $x^*$. Aquí, $\xi$ es un valor desconocido, pero podemos utilizar el hecho de que está contenido en el intervalo: $\xi \in X$.

Por lo tanto, obtenemos

\begin{equation}
x^* = x - \frac{f(x)}{f'(\xi)} \in x - \frac{f(x)}{F'(X)} =: N(X,x),
\end{equation}
donde hemos definido un operador $N$ que actúa sobre un intervalo y *cualquier* punto en el intervalo.

Si suponemos que $x^*\in X$, entonces $x^* \in N(X,x)\cap X$ para toda $x\in X$.

Una elección particular es $x = m := \mathrm{mid}(X)$, el punto medio del intervalo $X$. Entonces obtenemos el llamado *operador de Newton para intervalos*:

\begin{equation}
N(X) := N(X,m) = m - \frac{f(m)}{F'(X)}.
\end{equation}

Nota que cuando implementamos esto en la computadora, en general $f(m)$ no se podrá calcular exactamente, por lo cual es necesario convertir $m$ en un intervalo $M := [m,m]$, y usar la extensión natural $F(M)$, así que tenemos finalmente

\begin{equation}
N(X) := M - \frac{F(M)}{F'(X)},
\end{equation}
donde ahora todos son intervalos.

Ahora podemos definir una sucesión de intervalos a partir de un intervalo inicial $X_0$, dada por $X_{k+1} := X_k \cap N(X_k)$. Por construcción, si $x^*\in X_0$ entonces $x^*\in X_k$ para toda $k$. Entonces, si $X_0$ contiene a una raíz, la raíz se mantiene dentro de la secuencia de intervalos, que de hecho forman una secuencia anidada que converge a $x^*$. Entonces, si controlamos que de alguna manera el diámetro de los intervalos $x_k$ disminuya, obtendremos cotas precisas para $x^*$. Esto es el contenido del **Teorema del método de Newton para intervalos**.

Más aún, si $f(x)$ es doblemente continua y diferenciable y $N(X)$ está bien definida en el intervalo $X$, se puede demostrar que

1. Si $N(X)\cap X = \emptyset$, es decir, $N(X)$ no se intersecta con $X$, entonces $X$ no contiene ninguna raíz de $f$.

2. Si $N(X) \subseteq X$, entonces $X$ contiene exactamente una raíz de $f$.

[1] Escribe una función que calcule el operador de Newton para un intervalo $X$, dadas $f(x)$ y $f'(x)$.

[2] Implementa el método de Newton para intervalos para encontrar las raíces de $f(x) = x^3 - 1$ a partir de $X=[-3,3]$. Muestra gráficamente la implementación del método. En cada iteración subsecuente del método, ilustra qué le pasa al diámetro de la refincación del intervalo que vas obteniendo.

[3] Usando tu implementación, *demuestra* que en el mismo intervalo $X=[-3,3]$ que $g(x) = x^2+1$ no tiene ninguna raíz. Ilustra gráficamente esto.

Hint: En el caso en que $0\in F'(x)$ las hipótesis del teorema no se cumplen. En ese caso hay que dividir el intervalo (bisección por ejemplo) y tratar de que las hipótesis del teorema se cumplan, para poder aplicarlo. Extiende tus funciones para que incluyan esta situación. En ese caso, cada subdivisión debe ser probada si puede o no tener una raíz.

[4] Considera la familia de polinomios de Wilkinson definidos por $W_n(x) = \prod_{i=1}^n(x-i)$. Partiendo de un intervalo simétrico alrededor de cero, implementa el método de Newton para encontrar sus raíces, utilizando diferenciación automática.

## División extendida

Hasta ahora, sólo hemos podido tratar el caso en el cual la derivada $F'(X)$ no contiene $0$. Sin embargo, resulta que es posible tratar también este caso, mediante "división extendida".

[5] Supón que $F'(X)$ sea un intervalo, digamos $F'(X) = [-a, b]$, que contiene $0$ (con $a, b > 0$).

(i) Definiendo $1/A$ como el conjunto $\{1/x: x \in A \}$, evalúa $1/F'(X)$.

(ii) Define una función que implementa esta "división extendida" de intervalos (o, más bien, inversa extendida).

[6] Resulta que el método de Newton sigue funcionando si utilizamos esta división extendida cuándo sea apropiado. Impleméntalo para encontrar *todas* las raíces de una función en un intervalo dado.

Nota que hay casos en los cuales no ocurre ninguna de las posibilidades (1) ni (2) en el teorema del método de Newton para intervalos. ¿Qué se puede hacer en este caso?

[7] Implementa pruebas (tests) para tu código.

notebooks/10. Metodo de Newton para intervalos.ipynb


# Logistic Regression

    Notebook version: 1.0 (Oct 16, 2015)

    Author: Jesús Cid Sueiro (jcid@tsc.uc3m.es)

    Changes: v.1.0 - First version

```python
# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
import csv
import random
import matplotlib
import matplotlib.pyplot as plt
import pylab

import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model

```

## 1. Introduction

### 1.1. Binary classification and decision theory

The goal of a classification problem is to assign a *class* or *category* to every *instance* or *observation* of a data collection. Here, we will assume that every instance ${\bf x}$ is an $N$-dimensional vector in $\mathbb{R}^N$, and that the class $y$ of sample ${\bf x}$ is an element of a binary set ${\mathcal Y} = \{0, 1\}$. The goal of a classifier is to predict the true value of $y$ after observing ${\bf x}$. We will denote as $\hat{y}$ the classifier output or *decision*. If $y=\hat{y}$, the decision is an *hit*, otherwise $y\neq \hat{y}$ and the decision is an *error*.

Decision theory provides a solution to the classification problem in situations where the relation between instance ${\bf x}$ and its class $y$ is given by a known probabilistic model: assume that every tuple $({\bf x}, y)$ is an outcome of a random vector $({\bf X}, Y)$ with joint distribution $p_{{\bf X},Y}({\bf x}, y)$. A natural criteria for classification is to select predictor $\hat{Y}=f({\bf x})$ in such a way that the probability or error, $P\{\hat{Y} \neq Y\}$ is minimum. Noting that
$$
P\{\hat{Y} \neq Y\} = \int P\{\hat{Y} \neq Y | {\bf x}\} p_{\bf X}({\bf x}) d{\bf x}
$$
the optimal decision is got if, for every sample ${\bf x}$, we make decision minimizing the condicional error probability:
\begin{align}
\hat{y}^* &= \arg\min_{\hat{y}} P\{\hat{y} \neq Y |{\bf x}\} \\
          &= \arg\max_{\hat{y}} P\{\hat{y} = Y |{\bf x}\} \\
\end{align}
Thus, the optimal decision rule can be expressed as
$$
P_{Y|X}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  P_{Y|X}(0|{\bf x})
$$
or, equivalently
$$
P_{Y|X}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$
The classifier implementing this decision rule is usually named MAP (Maximum A Posterior).

### 1.2. Parametric classification.

Classical decision theory is grounded on the assumption that the probabilistic model relating the observed sample ${\bf X}$ and the true hypothesis $Y$ is known. Unfortunately, this is unrealistic in many applications, where the only available information to construct the classifier is a dataset $\mathcal S = \{({\bf x}^{(k)}, y^{(k)}), \,k=1,\ldots,K\}$ of intances and their respective class labels.

A more realistic formulation of the classification problem is the following: given a dataset $\mathcal S = \{({\bf x}^{(k)}, y^{(k)}) \in {\mathbb{R}}^N \times {\mathcal Y}, \, k=1,\ldots,K\}$ of independent and identically distributed (i.i.d.) samples from an ***unknown*** distribution $p_{{\bf X},Y}({\bf x}, y)$, predict the class $y$ of a new sample ${\bf x}$ with the minimum probability of error.

Since the probabilistic model generating the data is unknown, the MAP decision rule cannot be applied. However, many classification algorithms use the dataset to obtain an estimate of the posterior class probabilities, and apply it to implement an approximation to the MAP decision maker.

Parametric classifiers based on this idea assume, additionally, that the posterior class probabilty satisfies some parametric formula:
$$
P_{Y|X}(1|{\bf x},{\bf w}) = f_{\bf w}({\bf x})
$$
where ${\bf w}$ is a vector of parameters. Therefore, the dataset ${\mathcal S}$ is used to compute an estimate $\hat{\bf w}$ of ${\bf w}$ and then apply the decision rule
$$
f_{\hat{\bf w}}({\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

In this lesson we explore one of the most popular model-based parametric classification methods: logistic regression.

## 2. Logistic regression.

### 2.1. The logistic function

The logistic regression model assumes that the binary class label $Y \in \{0,1\}$ of observation $X\in \mathbb{R}^N$ satisfies the expression.
$$p_{Y|{\bf X},W}(1|{\bf x}, {\bf w}) = f({\bf w}^\intercal{\bf x})$$
$$p_{Y|{\bf,X},W}(0|{\bf x}, {\bf w}) = 1-f({\bf w}^\intercal{\bf x})$$
where ${\bf w}$ is a parameter vector and $f(·)$ is the *logistic* function, which is defined by
$$f(t) = \frac{1}{1+\exp(-t)}$$

It is straightforward to see that the logistic function has the following properties:

- **P1**: Probabilistic output: $\quad 0 \le f(t) \le 1$
- **P2**: Symmetry: $\quad f(-t) = 1-f(t)$
- **P3**: Monotonicity: $\quad f'(t) = f(t)·(1-f(t)) \ge 0$

In the following we define a logistic function in python, and use it to plot a graphical representation.

**Exercise 1**: Verify properties P2 and P3.

```python
# Define the logistic function
def logistic(x):
   p = 1.0 / (1 + np.exp(-x))
   return p

# Plot the logistic function
t = np.arange(-6, 6, 0.1)
z = logistic(t)

#z = np.divide(1.0, 1 + np.exp(-t))
plt.plot(t, z)
plt.xlabel('$t$', fontsize=14)
plt.ylabel('$\phi(t)$', fontsize=14)
plt.title('The logistic function')
plt.grid()
```

### 2.2. Classifiers based on the logistic model.

The MAP classifier under a logistic model will have the form
$$p_{Y|{\bf X},W}(1|{\bf x}, {\bf w}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad \frac{1}{2} $$
that is
$$f({\bf w}^\intercal{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad \frac{1}{2} $$
therefore
$$
2 \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad
1 + \exp(-{\bf w}^\intercal{\bf x}) $$
which is equivalent to
$${\bf w}^\intercal{\bf x}
\quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad
0 $$
Therefore, the classifiers based on the logistic regression model are given by linear decision boundaries passing through the origin, ${\bf x} = {\bf 0}$.

```python
# Weight vector:
w = [1, 4, 8]   # Try different weights

# Create a regtangular grid.
x_min = -1
x_max = 1
dx = x_max - x_min
h = float(dx) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
Z = logistic(w[0] + w[1]*xx0 + w[2]*xx1)

# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')
```

### 3.3. Nonlinear classifiers.

The logistic model can be extended to construct non-linear classifiers by using non-linear data transformations. A general form for a nonlinear logistic regression model is

$$p_{Y|{\bf X},W}(1|{\bf x}, {\bf w}) = f({\bf w}^\intercal{\bf z}({\bf x})) $$

where ${\bf z}({\bf x})$ is an arbitrary nonlinear transformation. The boundary decision in that case is given by equation

$$
{\bf w}^\intercal{\bf z}({\bf x}) = 0
$$

** Exercise 2**: Modify the code above to generate a 3D surface plot of the polynomial logistic regression model given by

$$
P(1|{\bf x}) = f(1 + 10 x_0 + 10 x_1 - 20 x_0^2 + 5 x_0 x_1 + x_1^2)
$$

```python
# SOLUTION TO THE EXERCISE
# Weight vector:
w = [1, 10, 10, -20, 5, 1]   # Try different weights

# Create a regtangular grid.
x_min = -1
x_max = 1
dx = x_max - x_min
h = float(dx) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
Z = logistic(w[0] + w[1]*xx0 + w[2]*xx1 + w[3]*np.multiply(xx0,xx0) +
             w[4]*np.multiply(xx0,xx1) + w[3]*np.multiply(xx1,xx1))

# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')
```

## 3. Inference

Remember that the starting idea of parametric classification is to use data in a training set $\mathcal S = \{({\bf x}_k, y_k) \in {\mathbb{R}}^N \times \{0,1\}, k=1,\ldots,K\}$ to estimate the unknown parameter ${\bf w}$, and use the estimate $\hat{\bf w}$ to compute the label prediction for any new observation as

$$\hat{y} = \arg\max_y P_{Y|{\bf X},{\bf W}}(y|{\bf x},\hat{\bf w}).$$

To do so, we will make the following assumptions:
- **A1**. The samples in ${\mathcal S}$ are i.i.d.
- **A2**. Target $Y^{(k)}$ only depends on ${\bf x}^{(k)}$, but not on ${\bf x}^{(l)}$ for any $l\neq k$.
- **A3**. (Logistic Regression): target $Y^{(k)}$ satisfies a logistic model: $$P_{Y|{\bf X},{\bf W}}(1|{\bf x}, {\bf w}) = f({\bf w}^\intercal{\bf z}({\bf x})).$$

In the following we discuss different approaches to the estimation of ${\bf w}$.
* Maximum Likelihood (ML) estimat: ${\bf w}_{\text{ML}} = \arg\max_{\bf w} p_{{\mathcal S}|{\bf W}}({\mathcal S}|{\bf w})$
* Maximum A Posteriori estimate (MAP): ${\bf w}_{\text{MAP}} = \arg\max_{\bf w} p_{{\bf W}|{\mathcal S}}({\bf w}|{\mathcal S})$
* Bayesian estimate: ${\bf w}_{\text{B}} = \mathbb{E}\{{\bf W}|{\mathcal S}\}$

For the mathematical derivation of the logistic regression algorithm, the following representation of the logistic model will be useful: noting that

$$p_{Y|{\bf,X},W}(0|{\bf x}, {\bf w}) = 1-f({\bf w}^\intercal{\bf z}({\bf x}))
= f(-{\bf w}^\intercal{\bf z}({\bf x}))$$

we can write

$$p_{Y|{\bf X},W}(y|{\bf x}, {\bf w}) = f(\overline{y}{\bf w}^\intercal{\bf z}({\bf x}))$$

where $\overline{y} = 2y-1$ is a *symmetrized label* ($\overline{y}\in\{-1, 1\}$).

### 3.1. ML estimation.

The ML estimate is defined as

$${\bf w}_{\text{ML}}
   = \arg\min_{\bf w} L({\bf w})
$$

where $L({\bf w})$ is the negative log-likelyhood function, given by

$$
L({\bf w}) = - \log\left(p\left(y^{(1)},\ldots,y^{(K)}|
     {\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)\right)
$$

This function can be computed as follows. Using assumption A1,

$$
L({\bf w}) = - \log\left(\prod_{k=1}^K p\left(y^{(k)}|{\bf x}^{(1)},\ldots,{\bf x}^{(K)},{\bf w}\right)\right),
$$

using A2,

\begin{align}
L({\bf w})
    &= - \log\left(\prod_{k=1}^K p_{Y|{\bf X},{\bf W}}\left(y^{(k)}|{\bf x}^{(k)},{\bf w}\right)\right) \\
    &= - \sum_{k=1}^K\log\left(p_{Y|{\bf X},{\bf W}}\left(y^{(k)}|{\bf x}^{(k)},{\bf w}\right)\right)
\end{align}

and, using A3 (the logistic model)

\begin{align}
L({\bf w})
    &= - \sum_{k=1}^K\log\left(f\left(\overline{y}^{(k)}{\bf w}^\intercal {\bf z}^{(k)}\right)\right)   \\
    &= \sum_{k=1}^K\log\left(1+\exp\left(-\overline{y}^{(k)}{\bf w}^\intercal {\bf z}^{(k)}\right)\right)
\end{align}

where ${\bf z}^{(k)}={\bf z}({\bf x}^{(k)})$.

It can be shown that $L({\bf w})$ is a convex and differentiable function of ${\bf w}$. Therefore, its minimum is a point with zero gradient.

\begin{align}
\nabla_{\bf w} L(\hat{\bf w}_{\text{ML}})
    &= - \sum_{k=1}^K
       \frac{\exp\left(-\overline{y}^{(k)}\hat{\bf w}_{\text{ML}}^\intercal {\bf z}^{(k)}\right) \overline{y}^{(k)} {\bf z}^{(k)}}
       {1+\exp\left(-\overline{y}^{(k)}\hat{\bf w}_{\text{ML}}^\intercal {\bf z}^{(k)}
       \right)} = \\
    &= - \sum_{k=1}^K \left(y^{(k)}-f(\hat{\bf w}_{\text{ML}}^T {\bf z}^{(k)})\right) {\bf z}^{(k)} = 0
\end{align}

Unfortunately, $\hat{\bf w}_{\text{ML}}$ cannot be taken out from the above equation, and some iterative optimization algorithm must be used to estimate the minimum.

### 3.2. Gradient descent.

A simple iterative optimization algorithm is  gradient descent.

\begin{align}
{\bf w}_{n+1} = {\bf w}_n - \rho_n \nabla_{\bf w} L({\bf w}_n)
\end{align}

where $\rho_n >0$ is the *learning step*.

Applying the gradient descent rule to logistic regression, we get the following algorithm:

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n \sum_{k=1}^K \left(y^{(k)}-f({\bf w}_n^\intercal {\bf z}^{(k)})\right) {\bf z}^{(k)}
\end{align}

Defining vectors

\begin{align}
{\bf y} &= (y^{(1)},\ldots,y^{(K)})^\intercal \\
\hat{\bf p}_n &= (f({\bf w}_n^\intercal {\bf x}^{(1)}), \ldots, f({\bf w}_n^\intercal {\bf z}^{(K)}))^\intercal
\end{align}
and matrix
\begin{align}
{\bf Z} = \left({\bf z}^{(1)},\ldots,{\bf z}^{(K)}\right)^\intercal
\end{align}

we can write

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

In the following, we will explore the behavior of the gradient descend method using the Iris Dataset.

#### 3.2.1 Example: Iris Dataset.

As an illustration, consider the  Iris dataset , taken from the  UCI Machine Learning repository. This data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant (*setosa*, *versicolor* or *virginica*). Each instance contains 4 measurements of given flowers: sepal length, sepal width, petal length and petal width, all in centimeters.

We will try to fit the logistic regression model to discriminate between two classes using only two attributes.

First, we load the dataset and split them in training and test subsets.

```python
# Adapted from a notebook by Jason Brownlee
def loadDataset(filename, split):
    xTrain = []
    cTrain = []
    xTest = []
    cTest = []

    with open(filename, 'rb') as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
    for i in range(len(dataset)-1):
        for y in range(4):
            dataset[i][y] = float(dataset[i][y])
        item = dataset[i]
        if random.random() < split:
            xTrain.append(item[0:4])
            cTrain.append(item[4])
        else:
            xTest.append(item[0:4])
            cTest.append(item[4])
    return xTrain, cTrain, xTest, cTest

with open('iris.data', 'rb') as csvfile:
    lines = csv.reader(csvfile)

xTrain_all, cTrain_all, xTest_all, cTest_all = loadDataset('iris.data', 0.66)
nTrain_all = len(xTrain_all)
nTest_all = len(xTest_all)
print 'Train: ' + str(nTrain_all)
print 'Test: ' + str(nTest_all)
```

    Train: 93
    Test: 57

Now, we select two classes and two attributes.

```python
# Select attributes
i = 0 # Try 0,1,2,3
j = 1 # Try 0,1,2,3 with j!=i

# Select two classes
c0 = 'Iris-versicolor'
c1 = 'Iris-virginica'

# Select two coordinates
ind = [i, j]

# Take training test
X_tr = np.array([[xTrain_all[n][i] for i in ind] for n in range(nTrain_all)
                  if cTrain_all[n]==c0 or cTrain_all[n]==c1])
C_tr = [cTrain_all[n] for n in range(nTrain_all)
          if cTrain_all[n]==c0 or cTrain_all[n]==c1]
Y_tr = np.array([int(c==c1) for c in C_tr])
n_tr = len(X_tr)

# Take test set
X_tst = np.array([[xTest_all[n][i] for i in ind] for n in range(nTest_all)
                 if cTest_all[n]==c0 or cTest_all[n]==c1])
C_tst = [cTest_all[n] for n in range(nTest_all)
         if cTest_all[n]==c0 or cTest_all[n]==c1]
Y_tst = np.array([int(c==c1) for c in C_tst])
n_tst = len(X_tst)
```

#### 3.2.2. Data normalization

Normalization of data is a common pre-processing step in many machine learning algorithms. Its goal is to get a dataset where all input coordinates have a similar scale. Learning algorithms usually show less instabilities and convergence problems when data are normalized.

We will define a normalization function that returns a training data matrix with zero sample mean and unit sample variance.

```python
def normalize(X, mx=None, sx=None):

    # Compute means and standard deviations
    if mx is None:
        mx = np.mean(X, axis=0)
    if sx is None:
        sx = np.std(X, axis=0)

    # Normalize
    X0 = (X-mx)/sx

    return X0, mx, sx
```

Now, we can normalize training and test data. Observe in the code that the same transformation should be applied to training and test data. This is the reason why normalization with the test data is done using the means and the variances computed with the training set.

```python
# Normalize data
Xn_tr, mx, sx = normalize(X_tr)
Xn_tst, mx, sx = normalize(X_tst, mx, sx)
```

The following figure generates a plot of the normalized training data.

```python
# Separate components of x into different arrays (just for the plots)
x0c0 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==0]
x1c0 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==0]
x0c1 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==1]
x1c1 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==1]

# Scatterplot.
labels = {'Iris-setosa': 'Setosa',
          'Iris-versicolor': 'Versicolor',
          'Iris-virginica': 'Virginica'}
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
```

In order to apply the gradient descent rule, we need to define two methods:
 - A `fit` method, that receives the training data and returns the model weights and the value of the negative log-likelihood during all iterations.
 - A `predict` method, that receives the model weight and a set of inputs, and returns the posterior class probabilities for that input, as well as their corresponding class predictions.

```python
def logregFit(Z_tr, Y_tr, rho, n_it):

    # Data dimension
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):

        # Compute posterior probabilities for weight w
        p1_tr = logistic(np.dot(Z_tr, w))
        p0_tr = logistic(-np.dot(Z_tr, w))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p1_tr)) - np.dot((1-Y_tr).T, np.log(p0_tr))

        # Update weights
        w += rho*np.dot(Z_tr.T, Y_tr - p1_tr)

    return w, nll_tr

def logregPredict(Z, w):

    # Compute posterior probability of class 1 for weights w.
    p = logistic(np.dot(Z, w))

    # Class
    D = [int(round(pn)) for pn in p]

    return p, D
```

We can test the behavior of the gradient descent method by fitting a logistic regression model with ${\bf z}({\bf x}) = (1, {\bf x}^\intercal)^\intercal$.

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 200   # Number of iterations

# Compute Z's
Z_tr = np.c_[np.ones(n_tr), Xn_tr]
Z_tst = np.c_[np.ones(n_tst), Xn_tst]
n_dim = Z_tr.shape[1]

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The optimal weights are:"
print w
print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])
```

#### 3.2.3. Free parameters

Under certaing conditions, the gradient descent method can be shown to converge assymtotically (i.e. as the number of iterations goes to infinity) to the ML estimate of the logistic model. However, in practice, the final estimate of the weights ${\bf w}$ depend on several factors:

- Number of iterations
- Initialization
- Learning step

**Exercise**: Visualize the variability of gradient descent caused by initializations. To do so, fix the number of iterations to 200 and the learning step, and execute the gradient descent 100 times, storing the training error rate of each execution. Plot the histogram of the error rate values.

Note that you can do this exercise with a loop over the 100 executions, including the code in the previous code slide inside the loop, with some proper modifications. To plot a histogram of the values in array `p` with `n`bins, you can use `plt.hist(p, n)`

##### 3.2.3.1. Learning step

The learning step, $\rho$, is a free parameter of the algorithm. Its choice is critical for the convergence of the algorithm. too large values of $\rho$ make the algorithm diverge. For too small values, the convergence is too slown and more iterations are required for a good convergence.

**Exercise 3**: Observe the evolution of the negative log-likelihood with the number of iterations, for different values of $\rho$. It is easy to check that, for $\rho$ large enough, the gradient descent method does not converge. Can you estimate (through manual observation) and approximate value of $\rho$ stating a boundary between convergence and non-convergence?

**Exercise 4**: In this exercise we explore the influence of the learning step more sistematically. Use the code in the previouse exercises to compute, for every value of $\rho$, the average error rate over 100 executions. Plot the average error rate vs. $\rho$.

Note that you should explore the values of $\rho$ in a logarithmic scale. For instance, you can take $\rho = 1, 1/10, 1/100, 1/1000, \ldots$

In practice, the selection of $\rho$ may be a matter of trial an error. Also there is some theoretical evidence that the learning step should decrease along time up to cero, and the sequence $\rho_n$ should satisfy two conditions:
- C1: $\sum_{n=0}^{\infty} \rho_n^2 < \infty$ (decrease slowly)
- C2: $\sum_{n=0}^{\infty} \rho_n = \infty$ (do not decrease too much slowly)

For instance, we can take $\rho_n= 1/n$. Another common choice is $\rho_n = \alpha/(1+\beta n)$ where $\alpha$ and $\beta$ are also free parameters that can be selected by trial and error with some heuristic method.

#### 3.2.4. Visualizing the posterior map.

We can also visualize the posterior probability map estimated by the logistic regression model for the estimated weights.

```python
# Create a regtangular grid.
x_min, x_max = Xn_tr[:, 0].min(), Xn_tr[:, 0].max()
y_min, y_max = Xn_tr[:, 1].min(), Xn_tr[:, 1].max()
dx = x_max - x_min
dy = y_max - y_min
h = dy /400
xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                     np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
X_grid = np.array([xx.ravel(), yy.ravel()]).T

# Compute Z's
Z_grid = np.c_[np.ones(X_grid.shape[0]), X_grid]

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)

# Put the result into a color plot
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
pp = pp.reshape(xx.shape)
plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
```

#### 3.2.5. Polynomial Logistic Regression

The error rates of the logitic regression model can be potentially reduce by using polynomial transformations.

To compute the polinomial transformation up to a given degree, we can use the `PolynomialFeatures` method in `sklearn.preprocessing`.

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
g = 5 # Degree of polynomial

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The optimal weights are:"
print w
print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])

```

Visualizing the posterior map we can se that the polynomial transformations produces nonlinear decision boundaries.

```python
# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 8, 4  # Set figure size
for i in [1, 2]:
    ax = plt.subplot(1,2,i)
    ax.plot(x0c0, x1c0,'r.', label=labels[c0])
    ax.plot(x0c1, x1c1,'g+', label=labels[c1])
    ax.set_xlabel('$x_' + str(ind[0]) + '$')
    ax.set_ylabel('$x_' + str(ind[1]) + '$')
    ax.axis('equal')
    if i==1:
        ax.contourf(xx, yy, pp, cmap=plt.cm.copper)
    else:
        ax.legend(loc='best')
        ax.contourf(xx, yy, np.round(pp), cmap=plt.cm.copper)
```

## 4. Regularization and MAP estimation.

An alternative to the ML estimation of the weights in logistic regression is Maximum A Posteriori estimation. Modelling the logistic regression weights as a random variable with prior distribution $p_{\bf W}({\bf w})$, the MAP estimate is defined as

$$
\hat{\bf w}_{\text{MAP}} = \arg\max p({\bf w}|{\mathcal S})
$$

The posterior density $p({\bf w}|{\mathcal S})$ is related to the likelihood function and the prior density of the weights, $p_{\bf W}({\bf w})$ through the Bayes rule

$$
p({\bf w}|{\mathcal S}) =
    \frac{p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
          p_{\bf W}({\bf w})}
         {p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)}\right)}
$$

In general, the denominator in this expression cannot be computed analytically. However, it is not required for MAP estimation because it does not depend on ${\bf w}$. Thus, we can compute the MAP estimate as
\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\max_{\bf w}
         p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
         p_{\bf W}({\bf w}) \\
  &= \arg\max_{\bf w} \left\{
         \log\left(p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right) \right)
         + \log\left( p_{\bf W}({\bf w})\right)
         \right\}  \\
  &= \arg\min_{\bf w} \left\{L({\bf w}) - \log\left( p_{\bf W}({\bf w})\right)
         \right\}
\end{align}
where $L(·)$ is the negative log-likelihood function.
Two common choices for the prior are

- **Gaussian**: ${\bf W}$ is a zero-mean Gaussian random variable with variance matrix $v{\bf I}$. Thus:
$$
p_{\bf W}({\bf w}) = \frac{1}{(2\pi v)^{N/2}} \exp\left(-\frac{1}{2v}\|{\bf w}\|^2\right)
$$
and the MAP estimate is
\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2
         \right\}
\end{align}
where $C = 2v$. Noting that
$$\nabla_{\bf w}\left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2\right\}
= - {\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right) + \frac{2}{C}{\bf w},
$$
the gradient descent rule for MAP estimation becomes
\begin{align}
{\bf w}_{n+1} &= \left(1-\frac{2\rho_n}{C}\right){\bf w}_n
    + \rho_n {\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

- **Laplacian**: ${\bf W}$ is a multivariate zero-mean Laplacian distribution given by
$$
p_{\bf W}({\bf w}) = \frac{1}{(2 C)^{N}} \exp\left(-\frac{1}{C}\|{\bf w}\|_1\right)
$$
(where $\|{\bf w}\|=|w_1|+\ldots+|w_N|$ is the $L_1$ norm of ${\bf w}$) and the MAP estimate is

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|_1
         \right\}
\end{align}

The additional term introduced by the prior in the optimization algorithm is usually named the *regularization term*. It is usually very effective to avoid overfitting when the dimension of the weight vectors is high. Parameter $C$ is named the *inverse regularization strength*.

**Exercise 5**: Derive the gradient descent rules for MAP estimation of the logistic regression weights with Laplacian prior.

## 5. Other optimization algorithms

### 5.1. Stochastic Gradient descent.

Stochastic gradient descent (SGD) is based on the idea of using a single sample at each iteration of the learning algorithm. The SGD rule for ML logistic regression is

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf z}^{(n)} \left(y^{(n)}-\hat{p}^{(n)}_n\right)
\end{align}

Once all samples in the training set have been applied, the algorith can continue by applying the training set several times.

The computational cost of each iteration of SGD is much smaller than that of gradient descent, though it usually needs more iterations to converge.

**Exercise 5**: Modify logregFit to implement an algorithm that applies the SGD rule one to every sample in the training set.

### 5.2. The Newton's method

Assume that the function to be minimized, $C({\bf w})$, can be approximated by its second order Taylor series expansion around ${\bf w}_0$

$$
C({\bf w}) \approx C({\bf w}_0)
+ \nabla_{\bf w}^\intercal C({\bf w}_0)({\bf w}-{\bf w}_0)
+ \frac{1}{2}({\bf w}-{\bf w}_0)^\intercal{\bf H}({\bf w}_0)({\bf w}-{\bf w}_0)
$$

where ${\bf H}({\bf w}_k)$ is the  *Hessian* matrix of $C$ at ${\bf w}_k$. Then, the minimum of C around ${\bf w}_0$ can be approximated as
$$
{\bf w}^* = {\bf w}_0 - {\bf H}({\bf w}_0)^{-1} \nabla_{\bf w}^\intercal C({\bf w}_0)
$$

Since the second order polynomial is only an approximation to $C$, ${\bf w}^*$ is only an approximation to the optimal weight vector, but we can expect ${\bf w}^*$ to be closer to the minimizer of $C$ than ${\bf w}_0$. Thus, we can repeat the process, computing a second order approximation around ${\bf w}^*$ and a new approximation to the minimizer.

The  Newton's method is based on this idea. At each optization step, the function to be minimized is approximated by a second order approximation using a Taylor series expansion around the current estimate. As a result, the learning rules becomes

$$\hat{\bf w}_{n+1} = \hat{\bf w}_{n} - \rho_n {\bf H}({\bf w}_k)^{-1} \nabla_{{\bf w}}C({\bf w}_k)
$$

For instance, for the MAP estimate with Gaussian prior, the *Hessian* matrix becomes

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + \sum_{k=1}^K f({\bf w}^T {\bf z}^{(k)}) \left(1-f({\bf w}^T {\bf z}^{(k)})\right){\bf z}^{(k)} ({\bf z}^{(k)})^\intercal
$$

Defining diagonal matrix

$$
{\mathbf S}({\bf w}) = \text{diag}\left(f({\bf w}^T {\bf z}^{(k)}) \left(1-f({\bf w}^T {\bf z}^{(k)})\right)\right)
$$

the Hessian matrix can be written in more compact form as

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + {\bf Z}^\intercal {\bf S}({\bf w}) {\bf Z}
$$

Therefore, the Newton's algorithm for logistic regression becomes

\begin{align}
\hat{\bf w}_{n+1} = \hat{\bf w}_{n} +
\rho_n
\left(\frac{2}{C}{\bf I} + {\bf Z}^\intercal {\bf S}(\hat{\bf w}_{n})
{\bf Z}
\right)^{-1}
{\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

Some variants of the Newton method are implemented in the  Scikit-learn  package.

```python
def logregFit2(Z_tr, Y_tr, rho, n_it, C=1e4):

    # Compute Z's
    r = 2.0/C
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):
        p_tr = logistic(np.dot(Z_tr, w))

        sk = np.multiply(p_tr, 1-p_tr)
        S = np.diag(np.ravel(sk.T))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p_tr)) - np.dot((1-Y_tr).T, np.log(1-p_tr))

        # Update weights
        invH = np.linalg.inv(r*np.identity(n_dim) + np.dot(Z_tr.T, np.dot(S, Z_tr)))

        w += rho*np.dot(invH, np.dot(Z_tr.T, Y_tr - p_tr))

    return w, nll_tr
```

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
C = 1000
g = 4

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(X_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(X_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit2(Z_tr, Y_tr2, rho, n_it, C)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])
```

## 6. Logistic regression in Scikit Learn.

The  scikit-learn  package includes an efficient implementation of  logistic regression. To use it, we must first create a classifier object, specifying the parameters of the logistic regression algorithm.

```python
# Create a logistic regression object.
LogReg = linear_model.LogisticRegression(C=1.0)

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Fit model to data.
LogReg.fit(Z_tr, Y_tr)

# Classify training and test data
D_tr = LogReg.predict(Z_tr)
D_tst = LogReg.predict(Z_tst)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)

# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
dd = LogReg.predict(Z_grid)
pp = LogReg.predict_proba(Z_grid)[:,1]
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 8, 4  # Set figure size
for i in [1, 2]:
    ax = plt.subplot(1,2,i)
    ax.plot(x0c0, x1c0,'r.', label=labels[c0])
    ax.plot(x0c1, x1c1,'g+', label=labels[c1])
    ax.set_xlabel('$x_' + str(ind[0]) + '$')
    ax.set_ylabel('$x_' + str(ind[1]) + '$')
    ax.axis('equal')
    if i==1:
        ax.contourf(xx, yy, pp, cmap=plt.cm.copper)
    else:
        ax.legend(loc='best')
        ax.contourf(xx, yy, np.round(pp), cmap=plt.cm.copper)

```


# Import

```python
#region
import matplotlib.pyplot as plt
import math

from sympy import *
import matplotlib.pyplot as plt
from numpy import linspace
import numpy as np
#endregion
t = symbols('t')
f = symbols('f', cls=Function)
```

# Input

```python
#read input
#region
def ReadArray(f):
    line = f.readline()
    result = list(map(lambda x: float(N(x)), line.split()))
    return result

def ReadMatrix(f):
    listCoef = []
    line = f.readline()
    while(line.strip() != ''):
        coef = list(map(lambda x: float(N(x)), line.split()))
        listCoef.append(coef)
        line = f.readline()
    #print('listCoef: ')
    #print(listCoef)
    return listCoef

def RandN(listCoef):
        # R & N
    R = listCoef[0][0]
    N = math.inf
    for coef in listCoef:
        if(R > coef[0]): R = coef[0]
        coef.pop(0)
        if(N > len(coef)): N = len(coef)
    if R <= 0:
        raise ValueError("invalid input: bán kính <= 0")

    return (R,N)

#endregion
```

# Hàm chính

```python
def calculate(initial, listCoef, N):
    result = initial              # mảng kết quả c_i
    k=len(listCoef)-1             # mảng mảng hệ số a_i và f
    for n in range(0,N-k):
        c=0
        offset = 1;
        for i in range(n+1,n+k+1): offset *= i
        #start calculating c_{n+k}
        for m in range(0,n+1):
            mult = 1
            for i in range(0,k):
                c += listCoef[i][n-m] * result[m+i] * mult
                mult *= m+i+1
        c= (listCoef[k][n]-c)/offset           # -1*n! / (n+k)!
        result.append(c)
    return result

#Program
def Polynomial(inputPath):
    f = open(inputPath,"r")
    initial = ReadArray(f)
    listCoef = ReadMatrix(f)
    f.close()
    R,N = RandN(listCoef)
    result = calculate(initial, listCoef, N)
    return (R, result)
def Restore(array):
    3
```

# Plot and save

```python
#region
def Save(result, outputPath, mode):
    f = open(outputPath, mode)
    f.write("Radius of convergence = " + str(result[0]) + ", Result: \n");
    f.write(str(result[1]))
    f.write("gsdfdgsdfg")
    f.close()

def Plotf(f, interval):
    t_vals = linspace(interval[0], interval[1], 1000)
    lam_x = lambdify(t, f, modules = ['numpy'])
    x_vals = lam_x(t_vals)
    plt.plot(t_vals, x_vals)

def Plot(result, start, end, g = None):
    f = 0
    power = 0
    for i in result:
        f += i * (t ** power)
        power += 1
    Plotf(f, (start, end))
    if g is not None:
        Plotf(g, (start, end))
    return f
#endregion

#Frobenius
```

# Maclaurin Series

```python
t = symbols('t')
test = "input.txt"
def Series(f):
     return series(f,x,n=15)
f = sin(3*x)
file = open(test, "w")
string = str(Series(f))
file.write(string)
file.close()
```

```python
file = open(test, "r")
arr = file.read().split("x**")
print(arr)
```

    ['3*x - 9*', '3/2 + 81*', '5/40 - 243*', '7/560 + 243*', '9/4480 - 2187*', '11/492800 + 6561*', '13/25625600 + O(', '15)']

# Test

```python
test1 = 'example1.txt'
test2 = 'example2.txt'

output = 'outputPath_1.txt'
```

```python
R, array = Polynomial(test2)
print("Radius of convergence = " ,str(R), ", Result:")
np.set_printoptions(precision=1)
print(np.array(array))

f = Plot(array, -2 , 2, g = sin(3*t))
print(f.evalf(2))
Save((R,array),output,"w")

```

```python
R, array = Polynomial(test1)
print("Radius of convergence = " ,str(R), ", Result:")
np.set_printoptions(precision=1)
print(np.array(array))

f = Plot(array, -2 , 2, g = sin(3*t))
print(f.evalf(2))
Save((R,array),output,"w")

```

```python
Plot([1,2], -2 , 2, g = sin(3*t))
```

```python
from sympy import sympify
coeffs = list(polyExpand.as_coefficients_dict().values())
print(coeffs[2]) # order n = 2
```

```python

```

Topic 5 - Solving Differential Equations/26.2.PowerSeries/PowerSeries/PowerSeries.ipynb


# Principle of least action

> Marcos Duarte
> Laboratory of Biomechanics and Motor Control ([http://demotu.org/](http://demotu.org/))
> Federal University of ABC, Brazil

The [principle of least action](https://en.wikipedia.org/wiki/Principle_of_least_action) applied to the movement of a mechanical system states that "the average kinetic energy less the average potential energy is as little as possible for the path of an object going from one point to another" (Prof. Richard Feynman in [The Feynman Lectures on Physics](http://www.feynmanlectures.caltech.edu/II_19.html)).

This principle is so fundamental that it can be used to derive the equations of motion of a system, for example, independent of the Newton's laws of motion. Let's now see the principle of least action in mathematical terms.

The difference between the kinetic and potential energy in a system is known as the [Lagrange or Lagrangian function](https://en.wikipedia.org/wiki/Lagrangian_mechanics):

\begin{equation}
\mathcal{L} = T - V
\label{eq_lagrange}
\end{equation}

The [principle of least action](https://en.wikipedia.org/wiki/Principle_of_least_action) states that the actual path which a system follows between two points in the time interval $t_1$ and $t_2$ is such that the integral

\begin{equation}
\mathcal{S}\; =\; \int _{t_1}^{t_2} \mathcal{L} \; dt
\label{eq_action}
\end{equation}

is stationary, meaning that $\delta \mathcal{S}=0$ (i.e., the value of $\mathcal{S}$ is an extremum), and it can be shown in fact that the value of this integral is a minimum for the actual path of the system. The integral above is known as the action integral and $\mathcal{S}$ is known as the action.

For a didactic demonstration that the integral above is stationary, see [The Feynman Lectures on Physics](http://www.feynmanlectures.caltech.edu/II_19.html).

## References

- Feynman R, Leighton R, Sands M (2013) [The Feynman Lectures on Physics - HTML edition](http://www.feynmanlectures.caltech.edu/).
- Taylor JR (2005) [Classical Mechanics](https://books.google.com.br/books?id=P1kCtNr-pJsC). University Science Books.


# Optimizing Python Code

```python
import cProfile
import random
import math
import numpy as np
import time
import multiprocessing as mp
import cython
import matplotlib.pyplot as plt
from scipy.special import k1e, kve
from scipy.integrate import quad
from PIL import Image

from ipywidgets import interact

%matplotlib inline
%load_ext Cython
```

    The Cython extension is already loaded. To reload it, use:
      %reload_ext Cython

## Introduction

Often times (especially in `Python`) one finds that their code just isn't fast enough and they would like to speed up their code. Say, they're running some sort of scan which is taking many hours to run. How might they reduce this time to something more managable, like several minutes? In these tutorials, we will investigate techniques to speed up code in python.

But before we dig in, we will need to set some ground rules:
1. Make sure your code is working **BEFORE** trying to optimize.
2. Profile your code to find where your code is the slowest.
3. Try using `NumPy` first.
4. Try other options

<div class="alert alert-warning">
==WARNING==

In regards to point #2:

In Donald Knuth's paper "Structured Programming With Go To Statements", he wrote:

"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
</div>

The take away from the above statements is: you should find where you code is taking the longest first before trying to optimize. You may spend a lot of time optimizing what you *think* is the slow part of your code only to realize when you're finished that your optimizations we're did very little. To discover where you should look to optimize, one should **profile** their code.

## Profiling

Profiling you code is very easy in python. One can simply import the `cProfile` module from the python standard library and run `cProfile.run(...)` where the `...` is the code you would like to profile.

To demonstrate how one might use `cProfile` to find slow parts of their code, let's revisit the Ising model. Recall that the Ising model consists of $N$ spins which interact with their nearest neighbors with an interaction strength $J$. If these spins have a magnetic moment, $\mu$, they also interact with external magentic fields $B$. The Hamiltonian for this system was given by:

\begin{align}
    H = -J\sum_{i=0}^{N-1}S_{i}S_{i+1} - \mu B\sum_{i=0}^{N-1}S_{i}
\end{align}

where $S_{i}=1,-1$ and $S_{N + i} \equiv S_{i}$ for periodic boundary conditions. When we investigating the thermodynamic properties Ising model, we used the Metropolis algorithm to relax the spin-chain to an equilibrium state. This algorithm worked applying the following steps:
1. Pick a random spin. Call it $S_{k}$
2. Compute the change in energy, $\Delta E$, needed to flip $S_{k}$
3. If $\Delta E$ < 0, flip the spin. If $\Delta E \geq 0$, we draw a random uniform number $r$. If $e^{-\beta\Delta E}\geq r$, then we flip the spin.
4. Repeat 1-3 many times.
5. Average over the last several microstates to determine the macro thermodynamic property in question.

Let's implement steps 1-4. First, let's construct a function which computes the change in energy of the spin-chain if we flip spin $i$. For our first pass, we will compute the energy for the original state, then recompute the energy for the new state. We will first write a function to compute the energy:

```python
def compute_energy(spin_chain, J, B, mu):
    """
    Compute the energy of the spin configuration.
    """
    n = len(spin_chain)
    energy = 0.0
    for i in range(n):
        # Compute the energy from spin-spin interactions
        spinspin = -J * spin_chain[i - 1] * spin_chain[i]
        # Compute the energy from spin-magnetic field interactions
        mag = -mu * B * spin_chain[i]
        energy += spinspin + mag
    return energy
```

Now, let's write a function to compute the change in energy if we perform a flip:

```python
def compute_energy_change(i, spin_chain, J, B, mu):
    """
    Compute the energy of the spin configuration.
    """
    # Compute the energy of the state
    Ei = compute_energy(spin_chain, J, B, mu)
    # Compute the energy of the flipped state
    spin_chain[i] *= -1
    Ef = compute_energy(spin_chain, J, B, mu)
    # Compute the difference and restore state:
    dE = Ef - Ei
    spin_chain[i] *= -1
    return dE
```

Next, let's implement a function to update the state of our system using the metropolis algorithm:

```python
def update_spin(spin_chain, J, B, mu, T):
    """
    Update a spin in the spin chain.
    """
    n = len(spin_chain)
    # Use `random` to choose an integer from 0, n-1
    i = random.randint(0, n-1)
    # Compute change in energy
    dE = compute_energy_change(i, spin_chain, J, B, mu)
    # Apply Metropolis
    if math.exp(-dE / T) <= random.random():
        spin_chain[i] *= -1
```

Now let's profile the `update_spin` function and see how things are going:

```python
nspins = 1000
J = 1.0
B = 0.0
mu = 0.0
T = 1.0
spin_chain = [random.choice([-1, 1]) for _ in range(nspins)]

cProfile.run('[update_spin(spin_chain, J, B, mu, T) for _ in range(10000)]')
```

             140263 function calls in 4.518 seconds

       Ordered by: standard name

       ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        20000    4.461    0.000    4.463    0.000 <ipython-input-481-4e573e4722c4>:1(compute_energy)
        10000    0.009    0.000    4.472    0.000 <ipython-input-482-0a6d95fc7e8e>:1(compute_energy_change)
        10000    0.014    0.000    4.513    0.000 <ipython-input-487-99df470a5c94>:1(update_spin)
            1    0.004    0.004    4.518    4.518 <string>:1(<listcomp>)
            1    0.000    0.000    4.518    4.518 <string>:1(<module>)
        10000    0.009    0.000    0.019    0.000 random.py:172(randrange)
        10000    0.004    0.000    0.023    0.000 random.py:216(randint)
        10000    0.007    0.000    0.010    0.000 random.py:222(_randbelow)
            1    0.000    0.000    4.518    4.518 {built-in method builtins.exec}
        30000    0.003    0.000    0.003    0.000 {built-in method builtins.len}
        10000    0.002    0.000    0.002    0.000 {built-in method math.exp}
        10000    0.001    0.000    0.001    0.000 {method 'bit_length' of 'int' objects}
            1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        10259    0.002    0.000    0.002    0.000 {method 'getrandbits' of '_random.Random' objects}
        10000    0.001    0.000    0.001    0.000 {method 'random' of '_random.Random' objects}

Here is what everything means:
- **ncalls**: for the number of calls.
- **tottime**: for the total time spent in the given function (and excluding time made in calls to sub-functions)
- **percall**: is the quotient of tottime divided by ncalls
- **cumtime**: is the cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions.
- **percall**: is the quotient of cumtime divided by primitive calls
- **filename:lineno(function)**: provides the respective data of each function

So we can see that 10000 calls to `update_spin` takes about 4.4 seconds. However, we see something interesting. Nearly ALL of that time is spent in `compute` energy (see `tottime`.) So, if we would like to optimize this function, its pretty obvious where to start! We should optimize `compute_energy`. But, if we think a bit harder, we only want compute energy to compute the change in energy of the system. Since only one spin changes at a time, we can bypass `compute_energy` all together and come up with a simpler way to compute the change in energy.

Looking more closely at the Hamiltonian, one finds that, if we flip spin $k$, the change in energy of the system is:

\begin{align}
    \Delta E = 2 S_{k} \left[J\left(S_{k+1} + S_{k-1}\right) + \mu B\right]
\end{align}

Thus, to determine the change in energy of the system, simply need to compute the above. Let's make a new version of our `compute_energy_change` function:

```python
def compute_energy_change2(k, spin_chain, J, B, mu):
    if k == len(spin_chain) - 1:
        return 2 * spin_chain[k] * (J * (spin_chain[0] + spin_chain[k-1]) + mu * B)
    else:
        return 2 * spin_chain[k] * (J * (spin_chain[k+1] + spin_chain[k-1]) + mu * B)
```

As we should always do, let's make sure this new implementation agrees with the previous implemenation:

```python
print(compute_energy_change(1, spin_chain, J, B, mu))
print(compute_energy_change2(1, spin_chain, J, B, mu))
```

    -4.0
    -4.0

Good! Now, let's make a new version of our update function which uses the new version of the `compute_energy_change` function:

```python
def update_spin2(spin_chain, J, B, mu, T):
    """
    Update a spin in the spin chain.
    """
    n = len(spin_chain)
    # Use `random` to choose an integer from 0, n-1
    i = random.randint(0, n-1)
    # Compute change in energy
    dE = compute_energy_change2(i, spin_chain, J, B, mu)
    # Apply Metropolis
    if math.exp(-dE / T) <= random.random():
        spin_chain[i] *= -1
```

And let's give it a whirl:

```python
nspins = 1000
J = 1.0
B = 0.0
mu = 0.0
T = 1.0
spin_chain = [random.choice([-1, 1]) for _ in range(nspins)]

cProfile.run('[update_spin2(spin_chain, J, B, mu, T) for _ in range(10000)]')
```

             110254 function calls in 0.047 seconds

       Ordered by: standard name

       ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        10000    0.006    0.000    0.007    0.000 <ipython-input-489-6d62445ad29f>:1(compute_energy_change2)
        10000    0.012    0.000    0.043    0.000 <ipython-input-493-e94d820e9b94>:1(update_spin2)
            1    0.004    0.004    0.047    0.047 <string>:1(<listcomp>)
            1    0.000    0.000    0.047    0.047 <string>:1(<module>)
        10000    0.008    0.000    0.017    0.000 random.py:172(randrange)
        10000    0.004    0.000    0.021    0.000 random.py:216(randint)
        10000    0.006    0.000    0.009    0.000 random.py:222(_randbelow)
            1    0.000    0.000    0.047    0.047 {built-in method builtins.exec}
        20000    0.002    0.000    0.002    0.000 {built-in method builtins.len}
        10000    0.001    0.000    0.001    0.000 {built-in method math.exp}
        10000    0.001    0.000    0.001    0.000 {method 'bit_length' of 'int' objects}
            1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        10250    0.002    0.000    0.002    0.000 {method 'getrandbits' of '_random.Random' objects}
        10000    0.001    0.000    0.001    0.000 {method 'random' of '_random.Random' objects}

😮... This is about 100 times faster! This illustrates how we can pinpoint the slowest parts of our code and optimize to achieve faster code.

## `NumPy`

Often times when you have slow `Python` code in scientific computing its because you're not using enough `NumPy`! A lot of `NumPy` is written is pure `c`. That means that when you do operations on arrays, these operations are often being done in pure `c` behind the scenes. If we take adavantage of this, we can often achieve large speed ups.

To illustrate this, let's return to the Monte-carlo integration example from one of the PSETs. Remember that Monte-Carlo integration works by approximating an integral $I$ using:

\begin{align}
    I = \int_{\Omega}d^{d}\vec{x}f(\vec{x})\approx \frac{V}{N}\sum_{i=1}^{N}f(\vec{x}_{i})
\end{align}

where $\vec{x}_{i}$ is a random vector inside our region of integration $\Omega$ with volume $V$. For simplicity, we will take $\Omega$ to be a unit hyper-cube (so $V = 1$.) Let's write a function which will use MC integration to integrate some function over a unit hyper-cube.

```python
def integrate_mc(f, ndim, npts):
    """
    Integrate a function over a hyper-cube using Monte-Carlo.

    Parameters
    ----------
    f: callable
        Function to integrate
    ndim: int
        Number of dimensions of the system
    npts: int
        Number of points to use in integration.
    """
    fsum = 0.0
    f2sum = 0.0
    for k in range(npts):
        # Generate a random vector
        pts = [random.random() for i in range(ndim)]
        # Evaluate the function
        fval = f(pts)
        fsum += fval
        f2sum += fval**2
    # Compute the integral estimate
    integral = fsum / npts
    avg2 = f2sum / npts
    std = math.sqrt(avg2 - integral**2)
    error = std / math.sqrt(npts)

    return integral, error
```

Let's use the same function from the PSET, i.e.:

\begin{align}
f(x) = \left(\sum_{i=1}^{10}x_{i}\right)^2
\end{align}

```python
def f(x):
    fval = 0.0
    for i in range(len(x)):
        fval += x[i]
    return fval**2
```

Now let's time how long this function takes:

```python
t0 = time.time()
npts = 5000000
integral, error = integrate_mc(f, 10, npts)
t1 = time.time()
print(f"integral = {integral}, error = {error}, time = {t1-t0}")
```

    integral = 25.825734006092446, error = 0.004112657973875036, time = 10.836636781692505

Let's think about what we are doing in the above example. We are first taking `d=ndim` random number:
\begin{align}
    r_{1}, r_{2},\dots, r_{d}
\end{align}
Then, we are summing them:
\begin{align}
    r_{1}, r_{2},\dots, r_{d} \to r_{1} + r_{2}+ \cdots + r_{d}
\end{align}
Then we square the sum:
\begin{align}
    r_{1} + r_{2}+ \cdots + r_{d} \to (r_{1} + r_{2}+ \cdots + r_{d})^2
\end{align}

One might notice that all of these operations can be done in pure `NumPy`! Let's see what this looks like in `NumPy`:
```python
# Generate `d=ndim` random number: (r1, r2,..., rd)
>>> nums = np.random.rand(ndim)
array([0.26975253, 0.73287346, 0.56619396, 0.36911862, 0.59555464,...])
# Sum them
>>> tsum = np.sum(nums)
2.533493203408602
# Square
>>> tsum**2
6.41858781171758
```
This can be combined into one line as:
```python
>>> np.sum(np.random.rand(ndim))**2
6.41858781171758
```

That was easy! Now, we would like to do this many times. `NumPy` can do this very efficiently. First, let's make an array of `N`x`d` random number:
```python
N = 10
d = 5
>>> rands = np.random.rand(10, 5)
array([[0.26710951, 0.42828954, 0.10725534, 0.90864514],
       [0.89081527, 0.85347667, 0.81367381, 0.53985759],
       [0.11190761, 0.6044705 , 0.10671305, 0.57558664],
       [0.72345535, 0.27989919, 0.42372606, 0.18112357],
       [0.09991332, 0.90279081, 0.65644979, 0.84177427],
       [0.29823545, 0.3388897 , 0.56091701, 0.98475297],
       [0.55350661, 0.47200534, 0.39264102, 0.33956616],
       [0.56339476, 0.92145093, 0.13305375, 0.68434001],
       [0.8425773 , 0.69433766, 0.29323947, 0.32693491],
       [0.43869687, 0.55223718, 0.29408006, 0.41584071]])
```
Now, for each of these rows, we would like to sum up all the entries. This can be done again using `np.sum`. But we need to specify that we would like to sum the columns. This is done by specifying `np.sum(..., axis=1)`. Here, `axis=1` means columns (`axis = 0` means rows.) Let's see what happens:
```python
>>> np.sum(rands, axis=1)
array([1.71129954, 3.09782335, 1.39867779, 1.60820416, 2.50092819,
       2.18279512, 1.75771913, 2.30223945, 2.15708934, 1.70085482])
```
Now, we would like to square each value. We again just use `**`:
```python
>>> np.sum(rands, axis=1)**2
array([2.92854612, 9.59650949, 1.95629956, 2.58632063, 6.25464183,
       4.76459453, 3.08957656, 5.30030649, 4.65303442, 2.8929071 ])
```
Lastly, to get the integral estimate, we can use `np.average`. All together, this looks like:
```python
>>> integral = np.average(np.sum(np.random.rand(10, 5), axis=1)**2)
4.402273672551671
```
We can similarly get the error estimate using `np.std`. Let's see how pure `NumPy` compares to our `integrate_mc` function:

```python
t0 = time.time()
npts = 5000000
vals = np.sum(np.random.rand(npts, 10), axis=1)**2
integal = np.average(vals)
error = np.std(vals) / np.sqrt(npts)
t1 = time.time()

print(f"integral = {integal}, error = {error}, time = {t1 - t0}")
```

    integral = 25.835523342296007, error = 0.0041132507547658095, time = 0.4805889129638672

That was about 50 times faster! Let's try and example:

**Excercise**

Now let's get our hands dirty and see if we can integrate another function. Using the above as a template, use `NumPy` to compute the following integral:
\begin{align}
    I = \prod_{i=1}^{5}\int_{0}^{1}dx_{i}x_{i}^2 = \int_{0}^{1}dx_{1}\cdots\int_{0}^{1}dx_{5}\ x_{1}^{2}\cdots x_{5}^2
\end{align}

Compare you answer with the exact answer, which is $\frac{1}{3^5}$. (**Hint**: NumPy has a product function: `np.prod`.)

```python
# Your answer here...
```

## `Multiproccesing`

Basic setup:

Import the module (call it `mp` for short):
```python
import multiprocessing as mp
```
Construct a `Pool`:
```python
num_pools = ...
pool = mp.Pool(num_pools)
```
The above will generate `num_pools` number of proccess which can run independently. In essence, this will be the number of simultaneous `Python` programs that can run at the same time. Typically, I set this the number number of logical cores on the machine I'm working on. To see how many logical cores you have, you can use: `mp.cpu_count()`.

Start running some functions! Say we have some function called `compute` which takes in some arguments. We can start running this function by using:
```python
result = pool.apply_async(compute, args=(...))
```
Here we are running the function `compute` in a seperate python program and storing the result from `compute` in `result`. Next, if we don't want to add anything more to the pool, then we can close the pool:
```python
pool.close()
```
This tells the pool that we won't be adding anything else. Lastly, we will stop the current python program until all the pools have finished their computations using the `join` method:
```python
pool.join()
```
Now, we can get the result once finished!
```python
res = result.get()
```

Let's give this a shot using our Monte-Carlo integrator. We will construct a function which will compute the sum of function values and the sum of function values squared (needed for error estimate). Then, take the total number of evaluations that we need to perform and divide up the work among the different workers in the pool. Once all the workers have finished computing there portion of the sum, we will compute the integral and error.

```python
def generate_fs(f, ndim, npts):
    """
    Integrate a function over a hyper-cube using Monte-Carlo.

    Parameters
    ----------
    f: callable
        Function to integrate
    ndim: int
        Number of dimensions of the system
    npts: int
        Number of points to use in integration.
    """
    fsum = 0.0
    f2sum = 0.0
    for k in range(npts):
        # Generate a random vector
        pts = [random.random() for i in range(ndim)]
        # Evaluate the function
        fval = f(pts)
        fsum += fval
        f2sum += fval**2
    return np.array([fsum, f2sum])

def multiproccessing_integrate_mc(f, ndim, npts):
    """
    Integrate a function over a hyper-cube using Monte-Carlo.

    Parameters
    ----------
    f: callable
        Function to integrate
    ndim: int
        Number of dimensions of the system
    npts: int
        Number of points to use in integration.
    """
    ncpus = mp.cpu_count()
    pool = mp.Pool(ncpus)
    pts_per_cpu = int(npts / ncpus)
    results = []
    for i in range(ncpus):
        results.append(pool.apply_async(generate_fs, args=(f, ndim, pts_per_cpu)))
    pool.close()
    pool.join()

    tup = sum([res.get() for res in results]) / npts
    integral = tup[0]
    avg2 = tup[1]
    std = math.sqrt(avg2 - integral**2)
    error = std / math.sqrt(npts)

    return integral, error
```

```python
t0 = time.time()
npts = 5000000
res = multiproccessing_integrate_mc(f, 10, npts)
t1 = time.time()
print(f"result={res}, time={t1-t0}")
```

    result=(25.83404179977455, 0.0041171372755293085), time=1.7203221321105957

Here we get an answer that is about 5 times faster!

**Excercise**

Below, we have a function called `complicated_function` that takes a number, then runs for 2 seconds and then returns a number for you. Suppose we need to call this function 10 times. That would take about 20 seconds. Use `multiproccessing` to start up a pool of workers to work on these 10 computations simultaneously and print out the results.

```python
def complicated_function(i):
    time.sleep(2)
    return i
```

```python
t0 = time.time()
results = [complicated_function(i) for i in range(10)]
t1 = time.time()
print(f'results = {results}, time = {t1 - t0}')
```

    results = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], time = 20.028277158737183

```python
t0 = time.time()
ncpus = mp.cpu_count()
pool = mp.Pool(ncpus)
results = []
for i in range(10):
    results.append(pool.apply_async(complicated_function, args=(i,)))
pool.close()
pool.join()

results = [res.get() for res in results]

t1 = time.time()
print(f'results = {results}, time = {t1 - t0}')
```

    results = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], time = 2.13602614402771

## `Cython`

Here we will describe how to make `Python` lightning fast by using `Cython` https://cython.org/. `Cython` is a hybrid language which looks very similar to `Python` but allows you to add types and call `c/c++` functions. `Cython` code gets compiled into optimized `c` code which is then called from python.

In jupyter, all of this is very easy. We just need to have `Cython` installed (easier said than done on Windows...) and annotate a cell with a magic function `%%cython`. We can then write `Cython` code in that cell and call the `Cython` code from the rest of our jupyter notebook! Let's take a look at a very simple example.

### Warm-up

We will start with a silly example. Suppose you want to sum up the first `N` integers in `Python`. That is, you want to know the result of:

\begin{align}
S_N = \sum_{i=1}^{N}i
\end{align}

(of course we know the analytic answer, but bare with me.) In `Python` we can write a function which takes `N` as a parameter and runs a quick for loop to sum over the integers between 1 and `N`. This looks like:

```python
def pysum(N):
    int_sum = 0
    for i in range(N + 1):
        int_sum += i
    return int_sum
```

Let's see how long this function takes to run with `N` = 50 millon:

```python
%timeit pysum(int(5e7))
```

    2.34 s ± 32.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```cython
%%cython

def cysum(unsigned long long n):
    # Add types to int_sum and i (note unsigned long long is a 64-bit unsiged integer)
    # with a maximum value of about 10^19
    cdef unsigned long long int_sum = 0
    cdef unsigned long long i
    for i in range(n + 1):
        int_sum += i
    return int_sum

```

```python
%timeit cysum(int(5e7))
```

    157 ns ± 0.95 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Okay...well...that was about 10 millon times faster... Although this example is a bit contrived and this is obviously not the typical speed up that you will find when using `Cython`, it shows that you can achieve MUCH faster code.

Typicaly, you can see anything from 10-1000 times speed up with `Cython`.

### Ising model revisited

Let's revist the Ising model. But let's step our game up. Let's do the 2D Ising mode. Everything is the same as in the 1D Ising model, but with an extra dimension. We will need code to do the following:

- **`initialize_lattice`**: A function to create a random initial state of spins. All spins will be either 1 or -1.
- **`update_lattice_point`**: A function to update the state of a spin using the Metropolis–Hastings algorithm.
- **`update_lattice`**: A function to update all of the spins on the lattice (this is a bit different than what we did in Lecture and in the PSET, but instead of picking a random spin to update, we will update them all for each 'step')

And thats it! For simplicity, we will assume no external magnetic field. Let's write:

```python
def initialize_lattice(N, M):
    """
    Returns an NxM lattice of spins with spin -1 or 1.
    """
    return np.random.choice([-1, 1], size=(N, M))

def update_lattice_point(lattice, n, m, beta):
    """
    Update the spin located at (n, m) using the Metropolis–Hastings algorithm.

    Parameters
    ----------
    lattice: np.ndarry of shape (N, M)
        Lattice of spins
    n: int
        Row where spin is located.
    m: int
        Column where spin is located.
    beta: float
        Inverse temperature of the system in units of energy.
    """
    N, M = lattice.shape
    # Compute energy needed to flip spin.
    dE = 2 * lattice[n, m] * (
        lattice[(n + 1) % N, m]
        + lattice[(n - 1) % N, m]
        + lattice[n, (m + 1) % M]
        + lattice[n, (m - 1) % M]
    )
    # Decide if we should flip using Metropolis–Hastings
    if np.exp(-dE * beta) > np.random.rand():
        lattice[n, m] *= -1

def update_lattice(lattice, beta):
    """
    Update every point on the lattice once in-place and
    return the lattice.

    Paramters
    ----------
    lattice: np.ndarry of shape (N, M)
        Lattice of spins
    beta: float
        Inverse temperature of the system in units of energy.
    """
    N, M = lattice.shape
    for n_offset in range(2):
        for m_offset in range(2):
            for n in range(n_offset, N, 2):
                for m in range(m_offset, M, 2):
                    update_lattice_point(lattice, n, m, beta)
    return lattice
```

Now, let's write a function to display the lattice. We will use `Pillow` for this example (you can also just use `matplotlib.pyplot.imshow`, but for this case, `Pillow` will be smoother)

```python
def disply_lattice(lattice):
    """
    Generate an image of the lattice in gray scale
    """
    # Return an image of the lattice. Pillow would like values
    # from 0 to 255
    return Image.fromarray(np.uint8((lattice + 1) * 0.5 * 255))

def disply_lattice_sequence(images):
    """
    Display a set of images of the lattice
    """
    def _show(frame=(0, len(images)-1)):
        return disply_lattice(images[frame])
    return interact(_show)
```

Now, let's initialize the lattice and update every point 50 times.

```python
t0 = time.time()
images = [initialize_lattice(200, 200)]
for i in range(50):
    images.append(update_lattice(images[-1].copy(), 0.7))
t1 = time.time()
print(t1 - t0)
```

    11.547499179840088

Takes about 10 seconds...not so good for making changes to the code...But let's take a look at the sequence of images:

```python
disply_lattice_sequence(images)
```

    interactive(children=(IntSlider(value=25, description='frame', max=50), Output()), _dom_classes=('widget-inter…

    <function __main__.disply_lattice_sequence.<locals>._show(frame=(0, 50))>

Very cool!

Now, let's make this ligthning fast! Away we go...

```cython
%%cython

# Import Cython's NumPy
import numpy as np
cimport numpy as np
# Import the c version of exp
from libc.math cimport exp
# Import the c random number generator
from libc.stdlib cimport rand
# Import cython for decorators
cimport cython

# The c random number generator generates an integer
# from 0 to RAND_MAX. We need to know this number.
# The bellow code pulls in this value for the 'limits.h'
# c standard library.
cdef extern from "limits.h":
    int RAND_MAX

@cython.boundscheck(False)
@cython.wraparound(False)
cdef cy_update_lattice_point(np.int64_t[:,:] lattice, int n, int m, double beta):
    # Give types to all the things...
    cdef unsigned int N = lattice.shape[0]
    cdef unsigned int M = lattice.shape[1]

    # Instead of doing (n + 1) % N, we will determine
    # n + 1 and n - 1 ourselves
    cdef int nmax = n + 1
    cdef int nmin = n - 1
    if n == N - 1:
        nmax = 0
    elif n == 0:
        nmin = N - 1

    cdef int mmax = m + 1
    cdef int mmin = m - 1
    if m == M - 1:
        mmax = 0
    elif m == 0:
        mmin = M - 1

    cdef int dE = 2 * lattice[n, m] * (
          lattice[nmax, m]
        + lattice[nmin, m]
        + lattice[n, mmax]
        + lattice[n, mmin]
    )
    # Equivalent to exp(-dE * beta) > np.random.rand()
    if exp(-dE * beta) * RAND_MAX > rand():
        lattice[n, m] = -lattice[n, m]

def cy_update_lattice(np.int64_t[:,:] lattice, double beta):
    cdef int N = lattice.shape[0]
    cdef int M = lattice.shape[1]
    cdef int n, m
    cdef int n_offset, m_offset

    for n_offset in range(2):
        for m_offset in range(2):
            for n in range(n_offset, N, 2):
                for m in range(m_offset, M, 2):
                    cy_update_lattice_point(lattice, n, m, beta)
    return np.array(lattice)
```

Let's see how this new verison compares to the 'pure' `Python` version:

```python
lattice = initialize_lattice(200, 200)
%timeit update_lattice(lattice, 0.4) # = 🐢
%timeit cy_update_lattice(lattice, 0.4) # = 🔥
```

    235 ms ± 6.44 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    657 µs ± 9.74 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

🏃♀ About 500 times faster!!

```python
images = [initialize_lattice(200, 200)]
for i in range(100):
    images.append(cy_update_lattice(images[-1].copy(), 0.8))
```

🔥 Going so fast we're burnin' up 🔥

```python
disply_lattice_sequence(images)
```

    interactive(children=(IntSlider(value=50, description='frame'), Output()), _dom_classes=('widget-interact',))

    <function __main__.disply_lattice_sequence.<locals>._show(frame=(0, 100))>