## Example 1: `Hello world`

```python
# Print the character string `Hello world!`

```

## Example 2: Make a plot of the US unemployment rate

Download unemployment rate data from FRED (https://fred.stlouisfed.org/series/UNRATE/) and make a well-labeled plot.

```python
# Import the pandas library as pd

# Import the plotting library matplotlib.pyplot as plt

# Download the unemployment rate data (PROVIDED)
unemployment = pd.read_csv('https://fred.stlouisfed.org/data/UNRATE.txt',skiprows=24,sep='\s+',index_col=0,parse_dates = True)

# Print first 5 rows of unemployment variable

```

```python
# Create a well-labeled plot of the US unemployment rate

# Save the figure to the current working directory at 120 dots per inch resolution

```

## Example 2: Date and time

Use the `datetime` module to print the current date and time

```python
# Import the datetime module

# Create a variable called `today` that stores the current date and time

# Print the value of `today`

```

```python
# Print today's date formatted as: month day, year

```

```python
# Print the time component of `today` as: HH:MM:SS

```

## Example 3: Approximate a geometric series

Consider the infinite geometric series:

\begin{align}
\sum_{k=0}^{\infty} r^k & = r + r + r^2 + r^3 + \cdots
\end{align}

If $|r|<1$, then the series converges to:

\begin{align}
\frac{1}{1+r}
\end{align}

We can verify this fact numerically by computing the *truncated* series, i.e., compute the series to a finite number of terms. We will do this for $r=0.2$. The idea is to create a variable $s$ that is initially equal to zero. Then we let $k$ increment from 0, 1, up to $N$ and each time add $r^k$ to $s$.

```python
# Set the value of r

# Create a variable `N` that stores the desired number of terms to compute

# Initialize a variable that will store the value of the summation. Set equal to tzero

# Iterate over values of k fron 0 to N. Add the new term to the sum and print the current value of the sum

```

## Example 4: Draw random values from a normal probability distribution

The `numpy` library has a bunch of powerful tools for working with numbers. Here we'll use `numpy` to draw a random sample from the normal probability distribution. Specifically, 100 values from $\mathcal{N}(2,0.2^2)$.

```python
# Import the numpy numerical tools library as np

# Set the seed or state of the numpy random number generator

# Create a variable called 'x' that contains 100 draws from the normal(2,0.2^2) didistribution

```

```python
# Print the mean of the values in the variable 'x'

```

```python
# Print the standard deviation of the values in the variable 'x'

```


# Lecture 8, Optimality conditions

We are still studying the full problem
$$
\begin{align} \
\min \quad &f(x)\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_k(x) = 0\text{ for all }k=1,\ldots,K\\
&x\in \mathbb R^n.
\end{align}
$$

In order to identify which points are optimal, we want to define similar conditions as there are for unconstrained problems through the gradient:

>If $x$ is a  local optimum to function $f$, then $\nabla f(x)=0$.

## KKT conditions

**Theorem (First order Karush-Kuhn-Tucker (KKT) Necessary Conditions)** Let $x^**$ be a local minimum for problem
$$
$$
\begin{align} \
\min \quad &f(x)\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_k(x) = 0\text{ for all }k=1,\ldots,K\\
&x\in \mathbb R^n.
\end{align}
$$
$$

and assume that $x^*$ satisfies some regularity conditions (see e.g., https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions#Regularity_conditions_.28or_constraint_qualifications.29). Then there exists unique Lagrance multiplier vectors $\lambda^* = (\lambda^*_1,\ldots,\lambda_J^*)$ and $\mu^*=(\mu_1^*,\ldots,\mu_K^*)$ such that
$$
\begin{align}
&\nabla_xL(x,\lambda,\mu) = 0\\
&\mu_j^*\geq0,\text{ for all }j=1,\ldots,J\\
&\mu_j^*g_j(x)=0,\text{for all }j=1,\ldots,J,
\end{align}
$$
where $$L(x,\lambda,\mu) = f(x)- \sum_{k=1}^K\mu_kg_k(x) -\sum_{j=1}^J\lambda_jh_j(x)$$ and $A(x^*)$ is the set of active constraints at $x^*$.

The coefficients $\lambda$ and $\mu$ are called the KKT multipliers.

The first equality $\nabla_xL(x,\lambda,\mu) = 0$ is called stationary rule and the requirement $\mu_j^*g_j(x)=0,\text{for all }j=1,\ldots,J$ is called the complementary rule.

## Example

Consider the optimization problem
$$
\begin{align}
\min &\qquad (x_1^2+x^2_2+x^2_3)\\
\text{s.t}&\qquad -3+x_1+x_2+x_3\geq 0.
\end{align}
$$
Let us verify the KKT necessary conditions for the local optimum $x^*=(1,1,1)$.

We can see that
$$
L(x,\lambda,\mu) = (x_1^2+x_2^2+x_3^3)+\mu_1(3-x_1-x_2-x_3)
$$
and thus
$$
\nabla_x L(x,\lambda,\mu) = (2x_1-\mu_1,2x_2-\mu_1,2x_3-\mu_1)
$$
and if $\nabla_x L((1,1,1),\lambda,\mu)=0$, then
$$
2-\mu_1=0 $$
which holds when $$
\mu_1=2.
$$
In addition to this, we can see that $-3+x^*_1+x^*_2+x^*_3= 0$. Thus, the completementary rule holds even though $\mu_1\neq 0$.

## Example 2

Let us check the KKT for a solution that is not a local optimum. Let us have $x^*=(0,1,1)$.

We can easily see that in this case, the conditions are

$$\left\{
\begin{array}{c}
-\mu_1 = 0\\
2-\mu_1=0
\end{array}
\right.
$$
Clearly, there does not exist a $\mu_1\in \mathbb R$ such that these equalities would hold.

## Example 3

Let us check the KKT for another solution that is not a local optimum. Let us have $x^*=(2,2,2)$.

We can easily see that in this case, the conditions is
$$
4-\mu_1 = 0
$$
Now, $\mu_1=4$ satisfies this equation. However, now
$$
\mu_1(-3+x^*_1+x^*_2+x^*_3)=4(-3+6) = 12 \neq 0.
$$
Thus, the completementary rule fails and the KKT conditions are not true.

## Geometric interpretation of the KKT conditions

## Stationary rule

The stationary rule can be written as, there exist $\lambda',\mu$ so that
$$
-\nabla f(x) = -\sum_{j=1}^K\mu_j\nabla g_j(x) + \sum_{k=1}^K\lambda'_k\nabla h_k(x).
$$
Notice that we have slightly different $\lambda'$.

Now, remember that the $-\nabla v(x)$ gives us the direction of reduction of the function $v$ for all functions.

Thus, the above equation means that the direction of reduction of the function $-\nabla f(x)$ is countered by the direction of the reduction of the inequality constraints $-\nabla g_j(x)$ and the directions of either growth (or reduction, since $\lambda'$ can be negative) of the equality constraints $\nabla h_k(x)$.

**This means that the function cannot get reduced without reducing the inequality constraints (making the solution infeasible, if already at the bound), or increasing or decreasing the equality constraints (making, thus, the solution again infeasible).**

#### With just one inequality constraint this means that the negative gradients of $f$ and $g$ must point to the same direction.

#### With equality constraints this means that the negative of the function and the gradiemt of the equality constraint must either to the same or opposite directions

## Complementary conditions
Another way of saying complementary condition
$$
\mu_jg_j(x) = 0 \text{ for all } j=1,\ldots,J
$$
is to say that both $\mu_j$ and $g_j(x)$ cannot be positive at the same time. Especially, if $\mu_j>0$, then $g_j(x)=0$.

**This means that if we want to use the gradient of a constraint for countering the reduction of the function, then the constraint must be at the border.**


# Data Fitting Exercises

# 2. Linear Least-Squares

### How to define a line of “best fit”

In the previous exercise, you used the `linregress` function to calculate a line of best fit to your experimental data. But what is this function actually doing, and how do we define the quality of fit for a particular straight line?

In this exercise, you will work through the maths behind linear regression, and then learn how to implement this using functions in Python as a more general optimisation problem.

### The maths

Linear regression addresses the problem of how to find a line of “best fit” for some data that is appriximately described by a straight line. For example. we might have the data shown in the figure below:

and we want to find the straight line that best describes the relationship between $x$ and $y$:

How do we find the “best” line for this particular data set?

The equation of a straight line is

\begin{equation}
y=mx+c.
\end{equation}

This is a mathematical function that defines a set of $y$ values from a set of $x$ values, for any pair or parameters, $m$ and $c$.

More generally, we can say that $y$ is a function of some model parameters, $P$, and the input $x$:

\begin{equation}
y = f(P,x).
\end{equation}

In our example of a straight line, the parameter set $P$ contains the slope and intercept of the line:

\begin{equation}
P=\left\{m,c\right\},
\end{equation}

and the function $f$ is

\begin{equation}
f(P,x) = P_0 x + P_1.
\end{equation}

Now $P_0$ is the slope, and $P_1$ is the intercept.

In principle, we could choose any combination of $P_0$ and $P_1$ for our model parameters. Every combination gives a different straight line, which might be a better, or worse, fit to our experimental data.

To decide which set of model parameters best describes our real data, we need a way to score the agreement between the model $f(P,x)$ and the data. A common way to quantify the error between the model and the real data is to calculate the **sum of squared errors**. For every $x$ value in our input data set, we can calculate the $y_\mathrm{predicted}$ value predicted by the model. The difference between this predicted value and the real $y$ value for this data point is the **error** for that data point.

To quantify the overall error of this particular model, we can add up the *squares* of all the error terms:

\begin{equation}
\mathrm{error} = \sum_i\left[y_i - f(P,x_i)\right]^2
\end{equation}

Adding *squares* means that positive and negative errors contribute equally to the total error.

We can now determine mathematically whether one model is a better “fit” to the data than another. A better quality of fit will give us a smaller error. The **best** fit to the data (for this particular model function) is given by the set of parameters that **minimises** the error. The procedure of finding a set of model parameters that minimises the sum of squared errors is often called **least-squares** fitting.

### The code

In Python the model straight-line function can be expressed using the following function:
>```python
def model_function( P, x ):
    m = P[0]
    c = P[1]
    y = m * x + c
    return y
```

If you compare this to the mathematical function definition above, you should see the same structure in both the mathematical description and the Python representation.

The function `model_function()` also looks similar to the `line()` function you used in Exercise 1.

```python
def line( m, c, x ):
    y = m * x + c
    return y
```

The only difference is that in `model_function()` we pass in the model **parameters** as a **list**. This is then unpacked inside the function, instead of passing in $m$ and $c$ separately.

In the code cell below, define the functions model_function() and line().
Using x = 2, check that both these functions return the same $y$ values for pairs of parameters, $m$ and $c$. For the model_function() function, remember that the parameters need to be passed in as a **list**.

```python

```

We can also write a Python function that calculates the error between the predictions of our model function and the real $y$ values:

>```python
def error_function( P, x, y ):
    y_predicted = model_function( P, x )
    error_terms = y - y_predicted
    total_error = np.sum( error_terms**2 )
    return total_error
```

* The first line calls the `model_function()` function with the list of model parameters, $P$, and the input $x$ values. This returns a set of *predicted* $y$ values for this particular model.

* The second line calculates the error terms, i.e. the differences between the predicted and actual $y$ values.

* The third line calculates the sum of squared errors.

* The fourth line returns the error.

By copying the appropriate code from Exercise 1, or writing it from scratch, read in the experimental data from 'data/equilibrium_constant.dat' and plot it as points.

Using the model_function() function, generate a series of lines with different parameters, and plot these against the experimental data.

Define the function error_function(). For each of your guessed model parameter sets, calculate the error. You should see that models that appear closer to the experimental data give you lower errors.

```python

```

You should have seen how changing the parameters for your model lets you generate different lines, that each do better or worse jobs of describing the experimental data. In each case, the quality of fit between your model and the data is given by the error, calculated with your error function.

The final step in the problem is to find the **least-squares** solution. This corresponds to the set of model parameters that **minimise** the result from your error function.

Finding the minimum or maximum of a function is a common mathematical problem, and there are a large number of algorithms for doing this numerically, using computers.

The `scipy` module contains functions for doing exactly this, in `scipy.optimize`. For this exercise, you will import the `minimize()` function, and use this to find the parameters that minimise the output of your error function. You can import the `minimize()` function as follows:

>```python
from scipy.optimize import minimize
```

The `minimize()` function can be given a [large number of options](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html), but the simplest usage requires three arguments:
>```python
minimize( function_to_minimise, initial_guess, other_arguments )
```

- The first argument, `function_to_minimise`, is the name of the function to minimise. For this exercise, this is `error_function`.
- The second argument, `initial_guess`, is a starting guess for the parameters that you want to fit. e.g. you might guess that `P=[1.0,1.0]`.
- The third argument is a list of any other arguments that need to be passed into the function you are minimising. In this exercise, your function `error_function` takes three arguments: `error_function( P, x, y )`: The first is the set of model parameters, which you want to change during the fitting. The second and third are the **observed** $x$ and $y$ values (here these are $1/T$ and $\ln{K}$) for each data point. These do not change during the fit, and get passed into `minimize()` as a list in the `other_arguments` position.

e.g. if you have stored the $1/T$ ($x$ values) and $\ln{K}$ ($y$ values) for the experimental data in numpy arrays called, respectively, `inverse_temperature` and `ln_K` then you could write

>```python
P_initial = [ 1.0, 1.0 ]
other_args = inverse_temperature, ln_K
minimize( error_function, P_initial, other_args )
```

The `other_arg = inverse_temperature, ln_K` command stores the $x$ and $y$ data values in a data type called a **tuple**. From the perspective of this exercise, a tuple is like a list, in that it can store multiple values (here you store two `numpy` arrays).

In the following code cell, import the minimize function, and use it to minimise your error function, with the experimental data you loaded previously.

```python

```

Your output should look like:
```
      fun: 0.026018593683863903
 hess_inv: array([[  1.53892987e+06,  -4.95784961e+03],
       [ -4.95784961e+03,   1.60346929e+01]])
      jac: array([  1.53901055e-07,  -1.45286322e-07])
  message: 'Optimization terminated successfully.'
     nfev: 60
      nit: 4
     njev: 15
   status: 0
  success: True
        x: array([ 6917.390252  ,   -21.05540855])
```
You will see that a lot of information about the fit is produced. For this exercise, the important parts of the output are the `message`, which tells you whether the optimisation was successful, and the result labelled `x`, which gives the input parameters that minimise your error function. The value of the error function with these optimised model parameters is given by the `fun` result.

You can access specific parts of the output by appending the variable names to the end of your `minimize()` call, e.g. to get the optimal parameters you can write

>```python
minimize( error_function, P_initial, other_args ).x
```

Repeat your optimisation, and store the optimised model parameters in a variable P_opt. Check these values against the fitted slope and intercept you found in Exercise 1.