# Mining

## Objective and Prerequisites

This model is an example of a production planning problem. In such problems, decisions must be made regarding which products to produce, and which resources to use to produce those products. These problems are common across a broad range of manufacturing and mining industries.

### What You Will Learn
In this example, we will model and solve a multi-period production planning problem. In this case, the aim of the application is to optimize mine production across a number of mines over a five-year period. The company has different mines, which can be opened, operational, or closed. Each mine varies in the quality of its ore and the amount of ore that can be extracted. The quality of ore required to meet production goals varies each year. The aim is to create an optimal production plan for the next five years to maximize the total profit, while considering production capacity, mine restrictions, and mine costs.

More information on this type of model can be found in the fifth edition of Model Building in Mathematical Programming, by H. Paul Williams.

This modeling example is at the intermediate level, where we assume that you know Python and are familiar with the Gurobi Python API. In addition, you should have some knowledge about building mathematical optimization models.

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

---
## Problem Description

A mining company needs to create a five-year operating plan for a certain area with four mines in it.

They can only operate a maximum of three mines in this area in any given year. However, even though a mine may not operate in a given year, the company still must pay royalties on that mine if there is an expectation that it will operate again in the future. Otherwise, it can be permanently closed and no more royalties need to be paid.

The yearly royalties due for each open mine (operating or not) are as follows:

|  | Royalties |
| --- | --- |
| Mine 1 | $\$5 Million$ |
| Mine 2 | $\$4 Million$ |
| Mine 3 | $\$4 Million$ |
| Mine 4 | $\$5 Million$ |

There is a maximum amount of ore that can be extracted from each mine in a given year. These limits are as follows:

|  | Max Production |
| --- | --- |
| Mine 1 | $2.0\times10^6$ Tons |
| Mine 2 | $2.5\times10^6$ Tons |
| Mine 3 | $1.3\times10^6$ Tons |
| Mine 4 | $3.0\times10^6$ Tons |

Each mine produces a different grade of ore. This grade is measured on a scale such that blending ores together results in a linear combination of the quality requirements. For example, if equal quantities of ore from two different mines were combined, the resulting ore would have a grade that is the average of the grade for each of the two ores. The grade for each mine’s ore is as follows:

|  | Ore Quality |
| --- | --- |
| Mine 1 | 1.0 |
| Mine 2 | 0.7 |
| Mine 3 | 1.5 |
| Mine 4 | 0.5 |

Each year, the ore produced from each operating mine must be combined to produce ore of a certain grade. The yearly objectives for the combined ore are as follows:

|  | Quality Target |
| --- | --- |
| Year 1 | 0.9 |
| Year 2 | 0.8 |
| Year 3 | 1.2 |
| Year 4 | 0.6 |
| Year 5 | 1.0 |

The final blended ore sells for $\$10$/ton. Revenues and costs for future years are discounted at the rate of $10\%$ per annum.

The key question for the mining company is: Which mines should be operated each year and how much ore should be extracted from each mine?

This problem is based on a larger one faced by the firm of English China Clays, which had to in decide which mines to work. In that problem (in the 1970s), the goal was to work up to four mines out of 20 in each year.

---
## Model Formulation

### Sets and Indices

$t \in \text{Years}=\{1,2,\dots,5\}$: Set of years.

$m \in \text{Mines}=\{1,2,\dots,4\}$: Set of mines.

### Parameters

$\text{price} \in \mathbb{R}^+$: Selling price (in USD) of one ton of blended ore.

$\text{max_mines} \in \mathbb{N}$: Maximum number of mines that can operate in any given year.

$\text{royalties}_m \in \mathbb{R}^+$: Yearly royalties (in USD) for having mine $m$ open.

$\text{capacity}_m \in \mathbb{R}^+$: Maximum tons of ore that can be extracted from mine $m$ in any given year.

$\text{quality}_m \in \mathbb{R}^+$: Quality of the ore extracted from mine $m$.

$\text{target} \in \mathbb{R}^+$: Quality target of the blended ore in year $t$.

$\text{time_discount}_t \in [0,1] \subset \mathbb{R}^+$: Time discount for revenue and cost in year $t$.

### Decision Variables

$\text{blend}_t \in \mathbb{R}^+$: Tons of blended ore in year $t$.

$\text{extract}_{t,m} \in \mathbb{R}^+$: Tons of ore extracted from mine $m$ in year $t$.

$\text{working}_{t,m} \in \{0,1\}$: 1 if mine $m$ is working in year $t$, 0 otherwise.

$\text{available}_{t,m} \in \{0,1\}$: 1 if mine $m$ is open in year $t$, 0 otherwise.

### Objective Function

- **Profit**: Maximize the total profit (in USD) of the planning horizon.

\begin{equation}
\text{Maximize} \quad Z = \sum_{t \in \text{Years}}\sum_{m \in \text{Mines}}{\text{time_discount}_t*(\text{price}*\text{blend}_t-\text{royalties}_m*\text{extract}_{t,m})}
\tag{0}
\end{equation}

### Constraints

- **Operating Mines**: The total number of operating mines in year $t$ cannot exceed the limit.

\begin{equation}
\sum_{m \in \text{Mines}}{\text{working}_{t,m}} \leq \text{max_mines} \quad \forall t \in \text{Years}
\tag{1}
\end{equation}

- **Quality**: The final quality of the ore blended in year $t$ must meet the target.

\begin{equation}
\sum_{m \in \text{Mines}}{\text{quality}_m*\text{extract}_{t,m}} = \text{target}_t*\text{blended}_t \quad \forall t \in \text{Years}
\tag{2}
\end{equation}

- **Mass Conservation**: Total tons of ore extracted in year $t$ should be equal to the Tons of the ore blended in that year.

\begin{equation}
\sum_{m \in \text{Mines}}{\text{extract}_{t,m}} = \text{blend}_t \quad \forall t \in \text{Years}
\tag{3}
\end{equation}

- **Mine Capacity**: Total tons of ore extracted from mine $m$ in year $t$ cannot exceed the yearly capacity of that mine.

\begin{equation}
\sum_{m \in \text{Mines}}{\text{extract}_{t,m}} \leq \text{capacity}_m*\text{working}_{t,m} \quad \forall t \in \text{Years}
\tag{4}
\end{equation}

- **Open to Operate**: Mine $m$ can be operated in year $t$ only if it is open in that year.

\begin{equation}
\text{working}_{t,m} \leq \text{available}_{t,m} \quad \forall (t,m) \in \text{Years} \times \text{Mines}
\tag{5}
\end{equation}

- **Shut Down**: If mine $m$ is closed in year $t$, it cannot be opened again in the future.

\begin{equation}
\text{available}_{t+1,m} \leq \text{available}_{t,m} \quad \forall (t < 5,m) \in \text{Years} \times \text{Mines}
\tag{6}
\end{equation}

---
## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
import numpy as np
import pandas as pd

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input Data
We define all the input data of the model.

```python
# Parameters

years = [1, 2, 3, 4, 5]
mines = [1, 2, 3, 4]

royalties = {1: 5e6, 2: 4e6, 3: 4e6, 4: 5e6}
capacity = {1: 2e6, 2: 2.5e6, 3: 1.3e6, 4: 3e6}
quality  = {1: 1.0, 2: 0.7, 3: 1.5, 4: 0.5}
target = {1: 0.9, 2: 0.8, 3: 1.2, 4: 0.6, 5: 1.0}
time_discount = {year: (1/(1+1/10.0)) ** (year-1) for year in years}

max_mines = 3
price = 10
```

**Note:** The present value of a future amount at period $n$ is given by the formula: $\text{present_value} = \frac{1}{(1+\text{interest_rate})^n}*\text{future_value} $

## Model Deployment
We create a model and the variables. For each year and each mine, we have (i) a variable that captures production, in millions of tons, (ii) a decision variable which tells us if the mine is open, and (iii) a decision variable which tells us if the mine is operational.

```python
mining = gp.Model('Mining')

blend = mining.addVars(years, name="Blend")
extract = mining.addVars(years, mines, name="Extract")
working = mining.addVars(years, mines, vtype=GRB.BINARY, name="Working")
available = mining.addVars(years, mines, vtype=GRB.BINARY, name="Available")
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-

Next, we insert the constraints.

In each year only three mines can be operational.

```python
#1. Operating Mines

OperatingMines = mining.addConstrs((working.sum(year, '*') <= max_mines for year in years), "Operating_mines")
```

The quality of the ore from the mines multiplied by the amount that is mined must equal the needed blend quality multiplied by the quantity of blended ore.
This ensures that the quality standards are satisfied.

```python
#2. Quality

Quality = mining.addConstrs((gp.quicksum(quality[mine]*extract[year, mine] for mine in mines)
                   == target[year]*blend[year] for year in years), "Quality")
```

The following constraint ensures that the tonnage of blended ore in each year equals the combined tonnage of the constituents.

```python
#3. Mass Conservation

MassConservation = mining.addConstrs((extract.sum(year, '*') == blend[year] for year in years), "Mass Conservation")
```

The following constraint ensures that the mine can extract no more than the extract limit and also that there is only an output if the mine is operational.

```python
#4. Mine Capacity

MineCapacity = mining.addConstrs((extract[year, mine] <= capacity[mine]*working[year, mine] for year, mine in extract), "Capacity")
```

The following constraint ensures that when the mine is operational, it also needs to be open.

```python
# Open to operate
OpenToOperate = mining.addConstrs((working[year, mine] <= available[year, mine] for year, mine in available), "Open to Operate")
```

The following constraint forces a mine to be closed in all years subsequent to that in which it is first closed. If the mine is closed, it cannot be re-opened later:

```python
# Shutdown Mine
ShutdownMine = mining.addConstrs((available[year+1, mine] <= available[year, mine]
                   for year, mine in available if year < years[-1]), "Shut down")
```

The total profit consists of the income from selling the blended ore minus the royalties payable. This is to be maximized. It can be written:

```python
#0. Objective function
obj = gp.quicksum(price*time_discount[year]*blend[year] for year in years) \
- gp.quicksum(royalties[mine] * time_discount[year] * available[year, mine] for year, mine in available)
mining.setObjective(obj, GRB.MAXIMIZE)
```

Next, the optimization process starts and Gurobi finds the optimal solution.

```python
mining.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 71 rows, 65 columns and 182 nonzeros
    Model fingerprint: 0xb3698537
    Variable types: 25 continuous, 40 integer (40 binary)
    Coefficient statistics:
      Matrix range     [5e-01, 3e+06]
      Objective range  [7e+00, 5e+06]
      Bounds range     [1e+00, 1e+00]
      RHS range        [3e+00, 3e+00]
    Found heuristic solution: objective -0.0000000
    Presolve removed 13 rows and 13 columns
    Presolve time: 0.00s
    Presolved: 58 rows, 52 columns, 135 nonzeros
    Variable types: 16 continuous, 36 integer (36 binary)

    Root relaxation: objective 1.577309e+08, 40 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 1.5773e+08    0    4   -0.00000 1.5773e+08      -     -    0s
    H    0     0                    1.189074e+08 1.5773e+08  32.7%     -    0s
    H    0     0                    1.302711e+08 1.5773e+08  21.1%     -    0s
    H    0     0                    1.393620e+08 1.5524e+08  11.4%     -    0s
         0     0 1.5462e+08    0    4 1.3936e+08 1.5462e+08  10.9%     -    0s
    H    0     0                    1.468620e+08 1.5462e+08  5.28%     -    0s

    Cutting planes:
      Implied bound: 4
      MIR: 3
      Flow cover: 1
      RLT: 3
      Relax-and-lift: 3

    Explored 1 nodes (48 simplex iterations) in 0.06 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 5: 1.46862e+08 1.39362e+08 1.30271e+08 ... -0

    Optimal solution found (tolerance 1.00e-04)
    Best objective 1.468619743642e+08, best bound 1.468619743642e+08, gap 0.0000%

---
## Analysis

The optimal solution results in a profit of $\$146.8620$ million with the following production plan for each mine in each year (quantities expressed in millions of tons):

### Extraction Plan
This plan determines the millions of tons of ore extracted from each mine (columns)  in each year (rows) of the planning horizon. For example, 1.3 millions of tons of ore will be extracted from mine 3 during the year 2.

```python
rows = years.copy()
columns = mines.copy()
extraction = pd.DataFrame(columns=columns, index=rows, data=0.0)

for year, mine in extract.keys():
    if (abs(extract[year, mine].x) > 1e-6):
        extraction.loc[year, mine] = np.round(extract[year, mine].x / 1e6, 2)
extraction
```

      1
      2
      3
      4

      1
      2.00
      0.00
      1.3
      2.45

      2
      0.00
      2.50
      1.3
      2.20

      3
      1.95
      0.00
      1.3
      0.00

      4
      0.12
      2.50
      0.0
      3.00

      5
      2.00
      2.17
      1.3
      0.00

### Sales Plan
This plan defines the millions of tons of blended ore to sell during each year of the planning horizon. For example, we plan to sell 5.62 million tons of blended ore during year 4.

```python
rows = years.copy()
sales = pd.DataFrame(columns=['Sales'], index=rows, data=0.0)

for year in blend.keys():
    if (abs(blend[year].x) > 1e-6):
        sales.loc[year, 'Sales'] = np.round(blend[year].x / 1e6, 2)
sales
```

      Sales

      1
      5.75

      2
      6.00

      3
      3.25

      4
      5.62

      5
      5.47

**Note:** If you want to write your solution to a file, rather than print it to the terminal, you can use the model.write() command. An example implementation is:

`mining.write("mining-output.sol")`

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition (Page 261-262, 357)

Copyright © 2020 Gurobi Optimization, LLC

```python

```


# Session 5: Gradient Descent

## House sale-value prediction using the Boston housing dataset

------------------------------------------------------
*ATDST, 2017-2018*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

## Importing Packages

```python
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from sklearn import linear_model

%matplotlib inline
```

Today, we will continue with the example we used for session 2: Predicting house values using the average number of rooms in the [Boston housing dataset](https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.names).

Attribute Information can be found [here](https://www.kaggle.com/c/boston-housing).

### Loading Data

We will manage the database using [Panda's library and Dataframes](http://pandas.pydata.org/pandas-docs/stable/tutorials.html)

```python
housing_data=pd.read_csv('./Boston_train.csv')
```

We divide the whole data set in **80% training** and **20% test**:

```python
N = len(housing_data)

N_train = np.round(0.8 * N).astype(np.int32)

np.random.seed(seed=10) #To fix the random seed

mask = np.random.permutation(len(housing_data))

regression_data_frame = housing_data[['rm','medv']].iloc[list(mask[0:N_train])]

X_0 = np.array(regression_data_frame['rm'])
Y = np.array(regression_data_frame['medv'])

regression_data_frame_test = housing_data[['rm','medv']].iloc[list(mask[N_train:-1])]

X_0_test = np.array(regression_data_frame_test['rm'])
Y_test = np.array(regression_data_frame_test['medv'])

```

# Numerical Optimization with Gradient Descend

As we know, the Ridge regression optimization problem
$$\boldsymbol{\theta}_\lambda = \arg \min_{\theta} \frac{1}{N} \left[\sum_{i=1}^{N} (y^{(i)}-\boldsymbol{\theta}^T\mathbf{x}^{(i)})^2 + \lambda \sum_{j=1}^{D+1} \theta_j^2\right],$$
can be solved using the **normal equation**
$$\boldsymbol{\theta}_\lambda = (\mathbf{X}^T\mathbf{X} + \mathbf{D}_\lambda)^{-1}\mathbf{X}^T\mathbf{y},$$

However, as we advanced in Session 2, computing the normal equation has many important drawbacks:
- You need to keep the full target matrix $\mathbf{X}_{N\times (D+1)}$ in memory, which can be huge in large datasets!
- You need to invert the matrix $(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}$ $\Rightarrow$ $\mathcal{O}(D^3)$ complexity.
- For small data sets ($N\leq D$), $(\mathbf{X}^T\mathbf{X})^{-1}$ can be non-invertible!
- Once you get new data, how do you update $\boldsymbol{\theta}^*$?

Today, we will learn how to apply a very simple algorithm to perform numerical optimization: **gradient descent** (GD). GD is one of the cornerstones of **convex optimization**, but it is also widely used for **non-convex optimization** for instance in [deep learning with neural networks](https://www.youtube.com/watch?v=IHZwWFHWa-w). Be aware that GD is probably the simplest numerical optimization algorithm  and that there is a whole [field](https://web.stanford.edu/group/sisl/k12/optimization/#!index.md) devoted to optimize numerically a function.

In a nutshell, GD tries to iteratively converge to the minimum of a function $f(\boldsymbol{\theta})$ by iteratively applying the following update rule:

$$\boldsymbol{\theta}_{\ell+1} = \boldsymbol{\theta}_\ell - \alpha \nabla f(\boldsymbol{\theta})|_{\boldsymbol{\theta}_\ell},$$

where $\nabla(\cdot)$ is the gradient operator and $\alpha$ is the learning rate. **Setting the learning rate** is a problem in general. Despite we won't cover the topic in detail and just restrict to simple checks, be aware that there are many modifications to GD that attempt to authomatically tune $\alpha$, like the [line search](https://people.maths.ox.ac.uk/hauser/hauser_lecture2.pdf) method.

#### Check out this [beautiful post](http://www.benfrederickson.com/numerical-optimization/)!

___

# Solving the Ridge regression problem using GD

If we define

$$ J(\boldsymbol{\theta}) = \frac{1}{N} \left[\sum_{i=1}^{N} (y^{(i)}-\boldsymbol{\theta}^T\mathbf{x}^{(i)})^2 + \lambda \sum_{j=1}^{D+1}\theta_j^2\right],$$

then it is easy to check that, for $m>0$:

\begin{align}
\frac{\partial J(\boldsymbol{\theta})}{\partial \theta_m}= \frac{2}{N} \left[\lambda\theta_m-\sum_{i=1}^{N} x_m^{(i)}\left(y^{(i)}-\boldsymbol{\theta}^T\mathbf{x}^{(i)}\right)\right]= \frac{2}{N} \left[\lambda\theta_m - \mathbf{e}^T \mathbf{X}_{:,m}\right],
\end{align}
where $\mathbf{e}=\mathbf{y}-\boldsymbol{\theta}^T \mathbf{X}$ is the error vector and $\mathbf{X}_{:,m}$ is the $m$-th column of the normalized training feature matrix. For $m=0$ the first term is not present, since the intercept is not regularized:
\begin{align}
\frac{\partial J(\boldsymbol{\theta})}{\partial \theta_m}= \frac{-2}{N} \left[\sum_{i=1}^{N} x_m^{(i)}\left(y^{(i)}-\boldsymbol{\theta}^T\mathbf{x}^{(i)}\right)\right]= \frac{-2}{N} \left[\mathbf{e}^T \mathbf{X}_{:,m}\right],
\end{align}

Note that in both cases ** the gradient vanishes when the error is zero**.

Lets now program the GD method in a modular way. First, we incorporate some useful functions from the ** previous session**. They will be included as a Python library file.

```python
import Ridge_functions as ridge   # You can find details in provided file S4_Ridge_functions.py
```

## Compute Gradient

** First ** you have to program a function that computes the gradient and takes as input arguments
- The feature matrix $\mathbf{X}$
- The error vector $\mathbf{e}$
- The value of $\lambda$
- The current value of $\boldsymbol{\theta}$
and returns as output the $(D+1)$-dimensional gradient of $J(\boldsymbol{\theta})$.

```python
def compute_gradient(feature_matrix,error,l,T):
    ## YOUR CODE HERE

    gradient = l*T - error @ feature_matrix
    gradient[0] -= l*T[0]
    gradient *= 2.0/feature_matrix.shape[0]

    return(gradient)
```

## Numerical evaluation of the gradient

Before moving forward, let's run a numerical check to verify that the gradient you compute in the function above is correct.

1) Fix a given arbitrary value of $\boldsymbol{\theta}^o$.

2) Introduce a small distortion in one of the components of $\boldsymbol{\theta}$. For instance, $\theta_m^+=\theta^o_m$ for $m\neq 1$ and $\theta^+_1 = \theta^o_1+\epsilon$, where $\epsilon=10^{-3}$. Evaluate the cost function $J(\boldsymbol{\theta}^+)$.

3) Define a new vector $\boldsymbol{\theta}^-$ such that $\theta_m^-=\theta^o_m$ for $m\neq 1$ and $\theta^-_1 = \theta^o_1-\epsilon$, where $\epsilon=10^{-3}$. Evaluate the cost function $J(\boldsymbol{\theta}^-)$.

4) Verify that

$$\frac{\partial J(\boldsymbol{\theta})}{\partial \theta_1} \approx \frac{J(\boldsymbol{\theta}^+)-J(\boldsymbol{\theta}^-)}{2\epsilon}$$,
where the derivative in the left hand side is given by the second component of the gradient computed using your function above. **Repeat the experiment using the first component of the gradient, i.e., the derivative w.r.t. $\theta_0$**.

```python
# Lets fix a degree
deg = 2
l= 1.0
np.random.seed(seed=10) #To fix the random seed

T = np.random.rand(3)

# This is how we would compute the cost function at T and obtain the normalized train feature matrix
# J_train,_,F_train,_ = ridge.eval_J_Ridge_given_T(X_0,X_0_test,deg,Y,Y_test,l,T)

# Your code here
mod = np.zeros(T.shape)
epsilon = 1e-2
index = 0
mod[index] += epsilon

T_p = T + mod
T_m = T - mod

J_p,_,F_train,_ = ridge.eval_J_Ridge_given_T(X_0,X_0_test,deg,Y,Y_test,l,T_p)
J_m,_,_,_ = ridge.eval_J_Ridge_given_T(X_0,X_0_test,deg,Y,Y_test,l,T_m)

error = (Y - ridge.LS_evaluate(F_train,T))

gradient = compute_gradient(F_train,error,l,T)

print("The gradient at position %d is %f" %(index,gradient[index]))

print("The approximate gradient at position %d is %f" %(index,(J_p-J_m)/(2*epsilon))) # Your code here

```

    The gradient at position 1 is -12.406842
    The approximate gradient at position 1 is -12.406842

## Implementing GD

Now, complete the following function to estimate the Ridge solution using gradient descend. The inputs are:

- The normalized train feature matrix $\mathbf{X}$
- The target vector $\mathbf{Y}$
- The initial value of theta $\boldsymbol{\theta}$
- The step size $\alpha$
- The stopping tolerance: a threshold that halts the algorithm when the norm of the gradient is below this limit
- The maximum number of iterations

```python
def regression_gradient_descent(F_train, Y, T_0, l, step_size, tolerance, iter_max,verbose=True,period_verbose=1000):
    converged = False
    T = np.array(T_0) # make sure it's a numpy array
    it=0

    while not converged:

        # First, compute the error vector
        error= (Y - ridge.LS_evaluate(F_train,T)) #YOUR CODE HERE

        # Second compute the gradient vector
        gradient= compute_gradient(F_train,error,l,T) #YOUR CODE HERE

        # Finally, update the theta vector
        T=T- step_size * gradient #YOUR CODE HERE

        grad_norm=np.linalg.norm(gradient)

        if(verbose==True and it % period_verbose ==0):
            J_Ridge = ridge.J_error_L2(Y,ridge.LS_evaluate(F_train,T),T,l)
            print ("Iterations = %d" %(it))
            print ("Gradient norm %f" %(grad_norm))
            print ("Ridge cost function %f" %(J_Ridge))

        if grad_norm < tolerance:
            converged = True
        elif it > iter_max:
            converged = True
        else:
            it=it+1

    if(converged==True):
        J_Ridge = ridge.J_error_L2(Y,ridge.LS_evaluate(F_train,T),T,l)
        print ("Iterations = %d" %(it))
        print ("Gradient norm %f" %(grad_norm))
        print ("Ridge cost function %f" %(J_Ridge))

    return(T)
```

## Evaluate solution and compare with normal equation

Run an example to verify that the GD solution is close to the one predicted by the normal equation. Investigate the effect of the step size.

```python
# Lets fix a degree
deg = 5
l= 1.0
T = np.random.rand(6)

#Cost function at initial T and feature matrix
J_0,_,F_train,_ = ridge.eval_J_Ridge_given_T(X_0,X_0_test,deg,Y,Y_test,l,T)

step_size = 1e-02
iter_max = 5e4
tolerance = 1e-03
period= 1e04

T_opt = regression_gradient_descent(F_train, Y, T, l, step_size, tolerance, iter_max,verbose=True,period_verbose=period)

T_normal = ridge.Ridge_solution(F_train,Y,l)

plt.stem(T_normal,'r',label='Normal Equation')
plt.plot(T_opt,'b',label='Gradient Descent')
plt.legend()
```

```python

```

Notebooks/Session 5 Gradient Descent/.ipynb_checkpoints/S5-Gradiend Descend (Complete)-checkpoint.ipynb


# Notwendige Importe

Die nächste Zelle muss ausgeführt werden. Sinnvoll ist es, vorher einmal das Menü

`cell | All Output | Clear`

aufzurufen. Dadurch werden alle Ausgaben gelöscht. Sonst sieht man nicht mehr, ob eine bestimmte Zelle schon ausgeführt wurde.

```python
from sympy import *
init_printing()

from vierfeldertafel_v01 import Vierfelder_Tafel
```

# Voraussetzungen

**Ereignisse** werden immer mit $A$ und $B$ bezeichnet. Zusätzlich gibt es die **Gegenereignisse** $\bar{A}$ und $\bar{B}$.

Die Wahrscheinlichkeiten werden nach einem bestimmten Schema benannt:

**Grundwahrscheinlichkeiten**

- $P(A)$: `a1`, $P(\bar{A})$: `a2`
- $P(B)$: `b1`, $P(\bar{B})$: `b2`

**Schnittwahrscheinlichkeiten**

- $P(A\cap B)$: `a1b1`
- $P(\bar{A}\cap B)$: `a2b1`
- $P(A\cap \bar{B})$: `a1b2`
- $P(\bar{A}\cap \bar{B})$: `a2b2`

**Bedingte Wahrscheinlichkeiten:**

- $P_A(B)$: `b1_a1`
- $P_{\bar{A}}(B)$: `b1_a2`
- $P_B(A)$: `a1_b1`
- $P_{\bar{B}}(\bar{A})$: `b2_a2`

**Achtung:** Hier ändert sich die Reihenfolge in der Schreibweise!

# Erstes Beispiel:

- $P(A) = 0.3$,
- $P_A(B) = 0.4$
- $P_{\bar A}(\bar{B}) = 0.7$

```python
v = Vierfelder_Tafel(
    a1=0.3,
    b1_a1=0.4,
    b2_a2=0.7
)

v.anzahl_loesungen
```

Nach Eingabe der Werte sollte man immer die Anzahl der Lösungen überprüfen. In den meisten Fällen gibt es nur eine Lösung. Es gibt aber auch Fälle, in denen mehr als eine Lösung existieren kann.

Die Lösungen werden -beginnend mit 0- durchnummeriert. Der vollständige Aufruf zur Ausgabe einer Vierfeldertafel lautet:

`v.tafel(loesung_idx=0,digits=None)`

Die Methode `v.tafel()` hat zwei optionale Argumente, nämlich der Index der Lösung (in der Regel 0) und die Anzahl der Ziffern (`digits`). Wird hier nichts angegeben, so wird mit der Standardgenauigkeit gerechnet.

```python
v.tafel(digits=5)
```

Manchmal möchte man die Zeilen und Spalten vertauschen. Das geht über die Mehode `T`:

```python
v.tafel(digits=5).T
```

Für eine Vierfeldertafel gibt es immer zwei Wahrscheinlichkeitsbäume. Der eine geht von den Ereignissen $A$ und $\bar{A}$ aus, der andere von den Ereignissen $B$ und $\bar{B}$.

```python
v.tree_a(digits=5)
```

```python
v.tree_b(digits=5)
```

# Zweites Beispiel

- $P(A) = \frac{1}{3}$
- $P_A(B) = \frac{1}{4}$
- $P_B(A) = \frac{1}{5}$

```python
v = Vierfelder_Tafel(
    a1 = Rational(1,3),
    b1_a1 = Rational(1,4),
    a1_b1 = Rational(1,5)
)
v.anzahl_loesungen
```

```python
v.tafel()
```

```python
v.tree_a()
```

```python
v.tree_b()
```


# Density Operator and Matrix

## Imports

```python
from IPython.display import display
```

```python
# TODO: there is a bug in density.py that is preventing this from working, uncomment to reproduce
# from sympy import init_printing
# init_printing(use_latex=True)
```

```python
from sympy import *
from sympy.core.trace import Tr
from sympy.physics.quantum import *
from sympy.physics.quantum.density import *
from sympy.physics.quantum.spin import (
    Jx, Jy, Jz, Jplus, Jminus, J2,
    JxBra, JyBra, JzBra,
    JxKet, JyKet, JzKet,
)
```

## Basic density operator

Create a density matrix using symbolic states:

```python
psi = Ket('psi')
phi = Ket('phi')
```

```python
d = Density((psi,0.5),(phi,0.5));
d
```

    Density((|psi>, 0.5),(|phi>, 0.5))

```python
d.states()
```

    (|psi>, |phi>)

```python
d.probs()
```

    (0.5, 0.5)

```python
d.doit()
```

    0.5*|phi><phi| + 0.5*|psi><psi|

```python
Dagger(d)
```

    Density((|psi>, 0.5),(|phi>, 0.5))

```python
A = Operator('A')
```

```python
d.apply_op(A)
```

    Density((A*|psi>, 0.5),(A*|phi>, 0.5))

## Density operator for spin states

Now create a density operator using spin states:

```python
up = JzKet(S(1)/2,S(1)/2)
down = JzKet(S(1)/2,-S(1)/2)
```

```python
d2 = Density((up,0.5),(down,0.5)); d2
```

    Density((|1/2,1/2>, 0.5),(|1/2,-1/2>, 0.5))

```python
represent(d2)
```

    Matrix([
    [0.5,   0],
    [  0, 0.5]])

```python
d2.apply_op(Jz)
```

    Density((Jz*|1/2,1/2>, 0.5),(Jz*|1/2,-1/2>, 0.5))

```python
qapply(_)
```

    Density((hbar*|1/2,1/2>/2, 0.5),(-hbar*|1/2,-1/2>/2, 0.5))

```python
qapply((Jy*d2).doit())
```

    0.5*Jy*|1/2,-1/2><1/2,-1/2| + 0.5*Jy*|1/2,1/2><1/2,1/2|

## Evaluate entropy of the density matrices

```python
entropy(d2)
```

    log(2)/2

```python
entropy(represent(d2))
```

    log(2)/2

```python
entropy(represent(d2,format="numpy"))
```

    (0.69314718055994529-0j)

```python
entropy(represent(d2,format="scipy.sparse"))
```

    (0.69314718055994529-0j)

## Density operators with tensor products

```python
A, B, C, D = symbols('A B C D',commutative=False)

t1 = TensorProduct(A,B,C)

d = Density([t1, 1.0])
d.doit()

t2 = TensorProduct(A,B)
t3 = TensorProduct(C,D)

d = Density([t2, 0.5], [t3, 0.5])
d.doit()
```

    0.5*(A*Dagger(A))x(B*Dagger(B)) + 0.5*(C*Dagger(C))x(D*Dagger(D))

```python
d = Density([t2+t3, 1.0])
d.doit()
```

    1.0*(A*Dagger(A))x(B*Dagger(B)) + 1.0*(A*Dagger(C))x(B*Dagger(D)) + 1.0*(C*Dagger(A))x(D*Dagger(B)) + 1.0*(C*Dagger(C))x(D*Dagger(D))

## Trace operators on density operators with spin states

```python
d = Density([JzKet(1,1),0.5],[JzKet(1,-1),0.5]);
t = Tr(d);
t
```

    Tr(Density((|1,1>, 0.5),(|1,-1>, 0.5)))

```python
t.doit()
```

    1.00000000000000

## Partial Trace on density operators with mixed state

```python
A, B, C, D = symbols('A B C D',commutative=False)

t1 = TensorProduct(A,B,C)

d = Density([t1, 1.0])
d.doit()

t2 = TensorProduct(A,B)
t3 = TensorProduct(C,D)

d = Density([t2, 0.5], [t3, 0.5])
d
```

    Density((AxB, 0.5),(CxD, 0.5))

```python
tr = Tr(d,[1])
tr.doit()
```

    0.5*A*Dagger(A)*Tr(B*Dagger(B)) + 0.5*C*Dagger(C)*Tr(D*Dagger(D))

## Partial trace on density operators with spin states

```python
tp1 = TensorProduct(JzKet(1,1), JzKet(1,-1))
```

Trace out the `0` index:

```python
d = Density([tp1,1]);
t = Tr(d,[0])
t
```

    Tr((|1,1>x|1,-1>, 1))

```python
t.doit()
```

    |1,-1><1,-1|

Trace out the `1` index:

```python
t = Tr(d,[1])
t
```

    Tr((|1,1>x|1,-1>, 1))

```python
t.doit()
```

    |1,1><1,1|

## Examples of `qapply()` on density matrices with spin states

```python
psi = Ket('psi')
phi = Ket('phi')

u = UnitaryOperator()
d = Density((psi,0.5),(phi,0.5)); d

qapply(u*d)
```

    O*Density((|psi>, 0.5),(|phi>, 0.5))

```python
up = JzKet(S(1)/2, S(1)/2)
down = JzKet(S(1)/2, -S(1)/2)
d = Density((up,0.5),(down,0.5))

uMat = Matrix([[0,1],[1,0]])
qapply(uMat*d)
```

    Matrix([
    [                                          0, Density((|1/2,1/2>, 0.5),(|1/2,-1/2>, 0.5))],
    [Density((|1/2,1/2>, 0.5),(|1/2,-1/2>, 0.5)),                                           0]])

## Example of `qapply()` on density matrices with qubits

```python
from sympy.physics.quantum.gate import UGate
from sympy.physics.quantum.qubit import Qubit

uMat = UGate((0,), Matrix([[0,1],[1,0]]))
d = Density([Qubit('0'),0.5],[Qubit('1'), 0.5])
d
```

    Density((|0>, 0.5),(|1>, 0.5))

```python
#after applying Not gate
qapply(uMat*d)
```

    U((0,),Matrix([
    [0, 1],
    [1, 0]]))*Density((|0>, 0.5),(|1>, 0.5))


(nm_interpolation_curve_fitting)=
# Interpolation and curve fitting
```{index} Interpolation and curve fitting
```

Real world environment always generate data, and an important part of a scientist's job is to find patterns in these observations. Python provides several libraries which help to make predictions based on real world data. While Python can make predictions based on existing data point, the human eye is still probably currently the best judge on whether those predictions are sensible or not.

Let's consider a discrete set of data, and let's start with a dataset with 2 dimensions. For example, we could consider the global temperature change over the years. On the x-axis, we could plot the year and on the y-axis, we could plot the temperature. Every year will have a corresponding temperature.

The 1st data point will be data point 0, the 2nd data point will be data point 1, and the 3rd data point will be data point 2, etc. and the last datapoint will be called datapoint \\(N\\). In total, we will have \\(N+1\\) data points. For every data point, there will be a year (\\(x\\) value) and a temperature (\\(y\\) value) assigned.

Since there is an \\(x\\) value and a \\(y\\) value for every data point, we could call the \\(x\\) value of the 1st data point, or data point 0, to be \\(x_0\\), and the \\(y\\) value of the 1st data point, or data point 0, to be \\(y_0\\). We could call the \\(x\\) value of the 2nd data point, or data point 1, to be \\(x_1\\), and the \\(y\\) value of the 2nd data point, or data point 1, to be \\(y_1\\), so on and so forth. We will call the \\(x\\) value of the last data point, or data point \\(N\\), to be \\(x_N\\), and the \\(y\\) value of the last data point, or data point \\(N\\), to be \\(y_N\\).

Therefore, our two dimensional dataset can be described as

\\[ \text{Dataset} = \{ (x_0, y_0),(x_1, y_1),(x_2, y_2),(x_3, y_3), \ldots, (x_N,y_N) \}.\\]

We could define an index \\(i\\), which goes from \\(0\\) to \\(N\\), and describe our dataset as a set of datapoints with their respective \\(x\\) and \\(y\\), which have their index going from \\(0\\) to \\(N\\). Expressed mathematically:

\\[ \text{Dataset} = (x_i, y_i),\quad \text{where} \quad i=0,\ldots,N,\\]

or even simpler

\\[ (x_i, y_i),\quad i=0,\ldots,N,.\\]

## Interpolation vs. curve fitting

The terms curve fitting and interpolation are often used interchangeably, but they are actually very different. In both interpolation and curve fitting, we assume there is some kind of function or relationship between the \\(x\\) value and the \\(y\\) value (and maybe even \\(z\\) if the dataset is 3 dimensional, and even more if the dataset has even more dimensions in machine learning and artificial intelligence). From the limited datapoints that are available, we seek to find the best approximate of this function or relationship.

In our temperature example where we plotted temperature against year, we seek to answer the question if there is a relationship between the temperature and the year. Has the temperature increased through the years, or has it decreased. Maybe the temperature is unrelated to the year? Maybe our data server went up in flames (this apparently happens way more often than you think, that's why invest in a proper [RAID configuration](https://en.wikipedia.org/wiki/Standard_RAID_levels)) and we lost the temperature data for some of the years, could we maybe use the relationship we deduced from the rest of the data points to give an estimate of what the temperature for those years without data to be?

```{admonition} Curve fitting or interpolation?

In both curve fitting and interpolation, you begin with some discrete data points.

With **interpolation**, you seek a function that typically *goes through all of the datapoints*, regardless of what the function will end up looking like. Interpolation is typically used when there is great confidence that there are *no measurement errors* in the data points, and there is *no ambiguity* in mapping \\(x\\) to \\(y\\), meaning that there are no multiple values of \\(y\\) for a single \\(x\\), similar to how mathematical functions are defined.

With **curve fitting**, you simply seek some function that will be a best fit for all the datapoints, and *does not* neccesarily have to *go through all the datapoints*. Curve fitting is typically used when there are *significant error bars* in the data points, and there is *ambiguity* in mapping \\(x\\) to \\(y\\).
```

Although it may seem that interpolation is better than curve fitting since it goes through all the data points, it should be noted that real world data almost always contains some sort of measurement error. Because interpolation tries to fit all the data points, the function produced through interpolation can vary widly even with the slightest change in the data point. Because curve fitting does not seek to fit all the data points, curve fitting functions are less susceptible to drastic changes when there are changes in the data points.

Interpolation has the "inter" word prefix to it, because using the function you got from interpolation, you can make prediction about the \\(y\\) value within the range of the given data points. For example, if I had the temperature data every 5 years since 1990 (1990, 1995, 2000, 2005, 2010, 2015 and 2020), then interpolation would allow me to make prediction about the temperature data in other years within the range of the data points, so from 1990 to 2020, meaning the years 1991, 1992, 1993, 1994, 1996, 1997, 1998, 1999, 2001, 2002, 2003, 2004, 2006, 2007, 2008, 2009, 2011, 2012, 2013, 2014, 2016, 2017, 2018, 2019.

Both interpolation and curve fitting can be done in various different ways. This notebook will introduce some of these ways using NumPy and SciPy libraries:

```python
import numpy as np
import scipy
import matplotlib.pyplot as plt
```

(nm_interpolation)=
## Interpolation
```{index} Interpolation
```
### Mathematical expression

As stated previously, interpolation seeks to find a function that maps \\(x\\) value in the datapoints to the \\(y\\) value in the corresponding datapoints. Therefore, interpolation can be expressed as:

\\[y_i = f(x_i),\quad \forall i, \quad\text{[\\(\forall \equiv\\) "for all"]},\\]

where the function \\(f\\) is called the [interpolant](http://mathworld.wolfram.com/Interpolant.html).

Because it is called interpolation, we can only make predictions for values of \\(x\\) within the range of \\(x_i\\):

\\[ \min(x_i)<x<\max(x_i).\\]

The requirement for distinct \\(x\\) locations means that we have a constraint on the \\(x_i\\)'s which can be written as

\\[x_0 < x_1 < \ldots < x_N.\\]

### Simple example

The simplest form of interpolation is linear interpolation, which is basically joining all the data points together in a line, and then creating a piecewise function.

Suppose we have a set of arbitrary data:

\\[\text{Dataset} = \{ (0.5, 0.5), (2.0, 0.4), (4.0, 0.3), (5.0, 0.1), (7.0, 0.9), (9.0, 0.8) \}.\\]

The \\(x\\) values and \\(y\\) values of the dataset are
\\[\text{Dataset x-values} = \{ 0.5, 2.0, 4.0, 5.0, 7.0, 9.0 \},\\\\\\
\text{Dataset y-values} = \{ 0.5, 0.4, 0.3, 0.1, 0.9, 0.8 \}.\\]

For linear piecewise interpolation, we begin with the first 2 points in the dataset, and then the next 2 points, and the next 2 points and so on and so forth. For the first 2 points, we find a linear equation that is determined by these two points. Linear equations are expressed as:

\\[y = mx + c,\\]

where \\(m\\) is the gradient, \\(c\\) is the \\(y\\)-intercept, and \\(m\\) and \\(c\\) can be found by

\\[m = \frac{y_2 - y_1}{x_2 - x_1},\\\\\\
c = y_1 - mx_1 \text{ or } c = y_2 - mx_2.\\]

For the first 2 points \\((0.5, 0.5)\\) and \\((2.0, 0.4)\\)  we obtain \\(m = -\frac{1}{15}\\) and \\(c =\frac{8}{15}\\).

For the range \\(0.5<x<2.0\\), the function can be written as

\\[f(x) = -\frac{1}{15}x + \frac{8}{15},\quad 0.5<x<2.0.\\]

For the next 2 points \\((2.0, 0.4)\\) and \\((4.0, 0.3)\\) we do the same, and we obtain \\(m = -\frac{1}{20}\\) and \\(c =\frac{1}{2}\\).

For the range \\(2.0<x<4.0\\), the function can be written as

\\[f(x) = -\frac{1}{20}x + \frac{1}{2},\quad 2.0<x<4.0.\\]

We can combine the two partial functions above, and obtain

$$
  f(x) =
  \left\{\begin{array}
    -\frac{1}{15}x + \frac{8}{15}, \quad 0.5<x<2.0 \\
    -\frac{1}{20}x + \frac{1}{2}, \quad 2.0<x<4.0
  \end{array}\right\}.
$$

We can continue the above process for the next 2 points, and the next 2 points, and eventually we will get a piecewise linear function:

$$
  f(x) =
  \left\{\begin{array}
    -\frac{1}{15}x + \frac{8}{15},\quad 0.5<x<2.0 \\
    -\frac{1}{20}x + \frac{1}{2},\quad 2.0<x<4.0 \\
    \quad\quad\cdots
  \end{array}\right\}.
$$

We can write our own code to obtain coefficients \\(m\\) and \\(c\\) for the points given in the dataset above:

```python
x = np.array([0.5, 2.0, 4.0, 5.0, 7.0, 9.0])
y = np.array([0.5, 0.4, 0.3, 0.1, 0.9, 0.8])

m_list = np.zeros(len(x))
c_list = np.zeros(len(x))

for i in range(len(x)-1):

    m = (y[i+1]-y[i])/(x[i+1]-x[i])
    c1 = (y[i]-m*x[i])
    c2 = (y[i+1]-m*x[i+1])

    print("For (%.1f, %.1f) and (%.1f, %.1f), m = %.2f and c1 = %.2f, c2 = %.2f."
             % (x[i], y[i], x[i+1], y[i+1], m, c1, c2))

    m_list[i] = m
    c_list[i] = c1
```

    For (0.5, 0.5) and (2.0, 0.4), m = -0.07 and c1 = 0.53, c2 = 0.53.
    For (2.0, 0.4) and (4.0, 0.3), m = -0.05 and c1 = 0.50, c2 = 0.50.
    For (4.0, 0.3) and (5.0, 0.1), m = -0.20 and c1 = 1.10, c2 = 1.10.
    For (5.0, 0.1) and (7.0, 0.9), m = 0.40 and c1 = -1.90, c2 = -1.90.
    For (7.0, 0.9) and (9.0, 0.8), m = -0.05 and c1 = 1.25, c2 = 1.25.

Piecewise linear interpolation can be easily done in Python. First, let's begin with plotting the points on their own. Python will automatically join the points together with lines unless otherwise specified. The `"o"` was used in the `plt.plot()` to ensure that bullets were shown instead of lines.

```python
# We will want to overlay a plot of the raw data a few times below so
# let's do this via a function that we can call repeatedly
def plot_raw_data(x, y, ax):
    """plot x vs y on axes ax,
    add axes labels and turn on grid
    """
    ax.plot(x, y, 'ko', label='raw data')
    ax.set_xlabel('x', fontsize=12)
    ax.set_ylabel('y', fontsize=12)
    ax.grid(True)
```

```python
# Set up a figure
fig = plt.figure(figsize=(7, 5))
ax1 = fig.add_subplot(111)
ax1.margins(0.1)

# Overlay raw data
plot_raw_data(x, y, ax1)

# Add a legend
ax1.legend(loc='best', fontsize=12)

# Add a figure title
ax1.set_title('Raw data', fontsize=12)

plt.show()
```

Now, let's plot the piecewise linear function. We will use the function from above, so be sure to run the cell above so that the function is loaded.

```python
fig = plt.figure(figsize=(7, 5))
ax1 = fig.add_subplot(111)
ax1.margins(0.1)

# Plot a piecewise-linear approximation.
# We get this simply by connecting the points with straight lines
# and this is the default behaviour of the plotting routine
ax1.plot(x, y, 'b', label='p/w linear interpolant')

# The previous raw data is overlain using the previous function
plot_raw_data(x, y, ax1)

# Add a legend
ax1.legend(loc='best', fontsize=12)

# Add a figure title
ax1.set_title('Raw data and its p/w linear interpolant', fontsize=12)

plt.show()
```

## Polynomial interpolation

Suppose we are given a set of \\(N+1\\) data points \\((x_i, y_i)\\) (with distinct \\(x_i\\)'s).

Now suppose we construct a polynomial of degree \\(N\\):
\\[ P_N(x) = a_0 + a_1 x + a_2 x^2 + a_3 x^3 + \ldots a_N x^N,\\]
where \\(a_0, \, a_1, \, \ldots, \, a_N\\) are the coefficients of our polynomial.

Note that there are \\(N+1\\) of these coefficients available to us with a degree \\(N\\) polynomial. This number of free parameters (\\(N+1\\)) agreeing exactly with the number of data points (\\(N+1\\)) is important as it's exactly the right number to allow us to determine the coefficients uniquely - essentially when we substitute in our data we can write this information as a linear matrix system we can invert (as its square system) for the coefficients.

We recognise this \\(P_N\\\) as the simple expression for a polynomial we have probably been introduced to before.

### How this problem leads to a square matrix system?

Assume we have two pieces of information and want to fit a linear line - let's write these out:

\\[ y_0 = a_0 + a_1x_0,\\\\\\
y_1 = a_0 + a_1x_1. \\]

We are assuming we know the \\(x\\)'s and the \\(y\\)'s, and we want to find the \\(a\\)'s.

There are multiple ways we could solve this, one way is to think about how you solve simultaneous equations - e.g. rearrange the second equation to give an expression for \\(a_1\\) in terms of \\(a_0\\), and so on.

However this isn't easy to do with large amounts of data. Instead we can note that this is equivalent to forming and solving the linear system

\begin{equation}
\begin{pmatrix}
1 & x_0  \\\\\\
1 & x_1
\end{pmatrix}
\begin{pmatrix}
a_0\\\\\\
a_1
\end{pmatrix}=
\begin{pmatrix}
y_0\\\\\\
y_1
\end{pmatrix}.
\end{equation}

Note that we can also interpret this as a linear combination of a *basis* made up of single-term polynomials:

\\[1,\ x,\ x^2,\ \ldots,\ x^N.\\]
These single term polynomials are also referred to as monomials. Our linear example from earlier is an example of this with two free parameters.

While this is perhaps the simplest (it's consistent with the simplest definition of a polynomial we are used to seeing) and the most obvious way to proceed, we should ask ourselves can we do better? Can we try to use a different (less obvious) basis that might mean we don't have a (potentially large, and difficult) matrix system to solve for the coefficients?

## Lagrange polynomial
The answer is yes, and [Lagrange polynomials](https://mathworld.wolfram.com/LagrangeInterpolatingPolynomial.html) are a particularly popular choice for constructing an interpolant for a given data set.

```{margin} Note

Note that this is not a typo and we are not re-using notation here - we will see that the weights in this approach actually are the same as the \\(y_i\\)'s making up the data we are interpolating!

```
Given a set of \\((N+1)\\) points as above, the Lagrange polynomial is defined as the linear combination

\\[L(x) = \sum_{i=0}^{N} y_i \ell_i(x),\\]
where the \\(\ell_i(x)\\) are a new choice for our basis functions (different to the monomials, but the same idea in that they form a basis), and the \\(y_i\\) are the \\(N+1\\) weights/coefficients corresponding to this basis.

The whole point of this approach is that we no longer have to compute the weights by inverting a matrix system as we had to above with monomials.

Now, by construction in this approach, we know the weights directly from the given data.

The functions \\(\ell_i(x)\\) are known as the Lagrange basis polynomials and are defined by the product

\\[\ell_i(x) := \prod_{\begin{smallmatrix}0\le m\le N\\ m\neq i\end{smallmatrix}} \frac{x-x_m}{x_i-x_m} = \frac{(x-x_0)}{(x_i-x_0)} \cdots \frac{(x-x_{i-1})}{(x_i-x_{i-1})} \frac{(x-x_{i+1})}{(x_i-x_{i+1})} \cdots \frac{(x-x_N)}{(x_i-x_N)},\\]
where \\(0\le i\le N\\).

If \\(N=0\\) (i.e. if there is only one point in the data set), there is only one Lagrange basis polynomial defined by convention as \\(\ell_0=1\\).

Notice from the definition of these basis functions the clear requirement that no two \\(x_i\\) are the same, \\(x_i - x_m \neq 0\\), so this expression is always well-defined (i.e. we never get a divide by zero!).

The reason pairs \\(x_i = x_j\\) with \\(y_i\neq y_j\\) are not allowed is that no interpolation function \\(L\\) such that \\(y_i = L(x_i)\\) would exist; a function can only return a single unique value for each argument \\(x_i\\) (this uniqueness is part of the definition of a function).

On the other hand, if also \\(y_i = y_j\\), then those two points would actually be one single point - we would thus have redundant data and really we could throw one away and would be looking for a degree \\(N-1\\) degree to interpolate \\(N\\) distinct data points.

For all \\(j\neq i\\), \\(\ell_j(x)\\) includes the term \\((x-x_i)\\) in the numerator (the thing on the top), so the whole product will be zero when evaluated at \\(x=x_i\\):

\\[\ell_{j\ne i}(x_i) =
\prod_{\begin{smallmatrix}0\le m\le N\\ m\neq j\end{smallmatrix}}
 \frac{x_i-x_m}{x_j-x_m} = \frac{(x_i-x_0)}{(x_j-x_0)} \cdots \frac{(x_i-x_i)}{(x_j-x_i)} \cdots \frac{(x_i-x_N)}{(x_j-x_N)} = 0.\\]

On the other hand a basis function \\(i\\) evaluated at location \\(x_i\\) returns 1:

\\[\ell_i(x_i) =
\prod_{\begin{smallmatrix}0\le m\le N\\ m\neq i\end{smallmatrix}}
\frac{x_i-x_m}{x_i-x_m} = 1.\\]

In other words, all of the basis functions/polynomials are zero at the exact locations of the data (\\(x=x_i\\)), except for \\(\ell_i(x)\\), for which it holds that \\(\ell_i(x_i)=1\\), because it lacks the \\((x-x_i)\\) term in the product.

It follows that

\\[y_i \ell_i(x_i)=y_i,\\]

and therefore at each point \\(x_i\\)

\\[L(x_i)=0+0+\dots + y_i + 0 +\ldots +0=y_i,\\]

showing that \\(L\\) does indeed interpolate (i.e. pass through) the data points exactly.

To help illustrate our discussion lets go back to our simple example from earlier. Note that we can use SciPy to evaluate the individual Lagrange basis function and polynomial.

For example, we can use [`scipy.interpolate.lagrange`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.lagrange.html) to generate the Lagrange polynomial for a dataset as shown in the next cell.

```python
import scipy.interpolate as si

### Our raw data from earlier - you can also test on our three data point example
xi = np.array([0.5, 2.0, 4.0, 5.0, 7.0, 9.0])
yi = np.array([0.5, 0.4, 0.3, 0.1, 0.9, 0.8])

# Create the Lagrange polynomial for the given points.
lp = si.lagrange(xi, yi)
# above we executed 'import scipy.interpolate as si'
# and so this line is calling the 'lagrange' function from the
# 'interpolate' sub-package within scipy.

# Evaluate this function at a high resolution (100 points here) so that
# we get a smooth well-resolved line when we plot our polynomial
x = np.linspace(0.4, 9.1, 100)

# set up the figure
fig = plt.figure(figsize=(7, 5))
ax1 = fig.add_subplot(111)
ax1.margins(0.1)

# Actually plot (x,y)=(x,lp(x)) on the axes with the label ax1
ax1.plot(x, lp(x), 'b', label='Lagrange interpolating polynomial')

# Overlay raw data on the same axes
plot_raw_data(xi, yi, ax1)
ax1.set_title('Lagrange interpolating polynomial (SciPy)', fontsize=16)

# Add a legend
ax1.legend(loc='best', fontsize=14)

plt.show()
```

### Error in Lagrange interpolation

Note that it can be proven that in the case where we are interpolating a known function (e.g. a complicated non-polynomial function such as \\(\exp\\) or \\(\sin\\)) by a simpler polynomial, the error at any point we evaluate the interpolant at is proportional to:

- the distance of that point from any of the data points (which makes sense as the error is obviously zero at the data points),
- and to the \\((N+1)\\)-th derivative of that function evaluated at some location within the bounds of the data, i.e. the more complicated (sharply varying) the function is, the higher the error could be.

This result is sometimes called the [Lagrange remainder theorem](https://en.wikipedia.org/wiki/Polynomial_interpolation#Interpolation_error).

### Exercise

We will sample the function \\(y(x)=x^3\\) at the points \\(x=(1,2,3)\\) and construct Lagrange polynomials - \\(L_0\\) (the constant interpolant going through the \\(x=2\\) data point only), \\(L_1\\) (the linear interpolant going through the \\(x=1\\) and \\(x=3\\) points) and \\(L_2\\) (the quadratic interpolant going through all three points).

```python
# this is the function we are going to approximate with low degree polynomials
def func_x3(x):
    return x**3

# as we will plot our approximation several times let's write a small function to do this
def plot_approximation(f, xi, ax):
    # Relatively fine x points for plotting our functions
    x = np.linspace(0.5, 3.5, 100)
    # Plot the original function
    ax.plot(x, f(x), 'k', label = 'Original\nfunction')

    # construct and plot the Lagrange polynomial
    lp = si.lagrange(xi, f(xi))
    # evaluate and plot the Lagrange polynomial at the x points
    ax.plot(x, lp(x), 'b', label = 'Lagrange poly.\ninterpolant')

    # shade the region between the two to emphasise the difference
    ax.fill_between(x, f(x), lp(x))

    # add some axis labels
    ax.set_xlabel('$x$', fontsize=14)
    ax.set_ylabel('$f(x), \; P_N(x)$', fontsize=14)

    # and add on top the interpolation points
    ax.plot(xi, f(xi), 'ko')

    # and a legend
    ax.legend(loc='best', fontsize=10)

# set up our figs for plotting - we want three subplots arranged in a 1x3 grid
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(10, 4))
# add some padding otherwise axes the labels can overlap with the next subplot
fig.tight_layout(w_pad=4)

# L0
plot_approximation(func_x3, np.array([2., ]), ax1)
ax1.set_title('Approximating a cubic with a constant', fontsize=12)

# L1
plot_approximation(func_x3, np.array([1., 3.]), ax2)
ax2.set_title('Approximating a cubic with a linear', fontsize=12)

# L2
plot_approximation(func_x3, np.array([1., 2., 3.]), ax3)
ax3.set_title('Approximating a cubic with a quadratic', fontsize=12)

plt.show()
```

## Newton polynomial
Calculating the Newton polynomial (also called [Newton's divided difference interpolation polynomial](https://mathworld.wolfram.com/NewtonsDividedDifferenceInterpolationFormula.html)) yields the same polynomial as the Lagrange polynomial method (remember that the polynomial of minimum degree to pass through each data point is unique), but is arguably easier to implement.

To derive this approach we write our degree \\(N\\) polynomial in the following form

\\[ P_N(x) = a_0 +(x-x_0)a_1 + (x-x_0)(x-x_1)a_2 + \cdots + (x-x_0)(x-x_1)\ldots(x-x_N)a_N,\\]
where \\\(a_0, a_1, \ldots, a_N\\_ are our \\(N+1\\) free parameters we need to find using the \\(N+1\\) pieces of information we have in the given data.

Efficient derivation of an algorithm to compute the Newton polynomial follows from noticing that we can write this polynomial in a [recursive form](https://en.wikipedia.org/wiki/Recursion_(computer_science)).

Consider for example a case with $N=3$:

\\[P_3(x) = a_0 +(x-x_0)a_1 + (x-x_0)(x-x_1)a_2 + (x-x_0)(x-x_1)(x-x_2)a_3,\\\\\\
P_3(x)= a_0 +(x-x_0)[a_1 + (x-x_1)[a_2 + (x-x_2)a_3]].\\]

Notice that substituting in the \\(x_i\\) values leads to a set of simultaneous equations where we can easily evaluate the unknowns \\(a_0, a_1, \ldots\\) using 'back (or forward) substitution'.

1. Substitute \\(x=x_0\\): We have \\(a_0 = P_3(x_0)\\), and we know that our interpolant \\(P_3(x)\\) evaluated at \\(x_0\\) must return \\(y_0\\). Hence,
    \\[a_0 = y_0.\\]
2. Now substitute \\(x=x_1\\). We have \\(P_3(x_1) = a_0 +(x_1-x_0)a_1 = y_0 +(x_1-x_0)a_1\\), the LHS of this is \\(y_1\\), and we know everything on the RHS as we have already calculated \\(a_0 = y_0\\). We can thus trivially rearrange to yield
    \\[ a_1 = \frac{(y_1 - y_0)}{(x_1-x_0)}.\\]
2. Substituting \\(x=x_2\\) yields
    \\[y_2 = P_3(x_2) = a_0 +(x_2-x_0)[a_1 + (x_2-x_1)a_2] = y_0 + (x_2-x_0)\left[ \frac{(y_1 - y_0)}{(x_1-x_0)} + (x_2-x_1)a_2\right],\\\\\\
\implies a_2 = \frac{ \frac{(y_2 - y_0)}{(x_2-x_0)} - \frac{(y_1 - y_0)}{(x_1-x_0)}}{x_2-x_1}.\\]

3. And so on.

To define an algorithm for this method in general let's first introducing the following [divided difference notation](https://en.wikipedia.org/wiki/Divided_differences):

\\[ \Delta y_i = \frac{y_i-y_0}{x_i-x_0},\quad i=1,2,\ldots, N,\\\\\\
\Delta^2 y_i = \frac{\Delta y_i-\Delta y_1}{x_i-x_1},\quad i=2, 3,\ldots, N,\\\\\\
\vdots\\\\\\
\Delta^N y_N = \frac{\Delta^{N-1} y_N-\Delta^{N-1} y_{N-1}}{x_N-x_{N-1}}.\\]

With a bit of thought we can hopefully see from the above example that the coefficients of the interpolating polynomial in the general case are given by

\\[a_0=y_0,\quad a_1 = \Delta y_1, \quad a_2 = \Delta^2 y_2, \quad \ldots \quad a_N = \Delta^N y_N.\\]

Our own code below shows how to calculate Newton polynomial coefficients.

```python
# Consider the above example data again
xi = np.array([0.5, 2.0, 4.0, 5.0, 7.0, 9.0])
yi = np.array([0.5, 0.4, 0.3, 0.1, 0.9, 0.8])

def calculate_newton_poly_coeffs(xi, yi):
    """ Evaluate the coefficients a_i recursively using Newton's method
    """
    # Initialise the array 'a' with yi, but take a copy to ensure we don't
    # overwrite our yi data!
    a = yi.copy()

    # We have N+1 data points, and so
    N = len(a) - 1

    # for each k, we compute Δ^k y_i from the a_i = Δ^(k-1) y_i of the previous iteration:
    for k in range(1, N+1):
        # but only for i>=k
        for i in range(k, N+1):
            a[i] = (a[i] - a[k-1])/(xi[i]-xi[k-1])

    return a

# Given the coefficients a, and the data locations xi,
# define a function to evaluate the Newton polynomial
# at locations given in the array x.
# NB. this is just an evaluation of the P_n(x) = ... formula
# given at the start of this section.

def eval_newton_poly(a, xi, x):
    """ Function to evaluate the Newton polynomial
    at x, given the data point xi and the polynomial coeffs a
    """
    N = len(xi) - 1  # polynomial degree
    # recursively build up polynomial evaluated at x
    P = a[N]
    for k in range(1, N+1):
        P = a[N-k] + (x - xi[N-k])*P
    return P
```

And now we can plot the result:

```python
# set up figure
fig = plt.figure(figsize=(7, 5))
ax1 = fig.add_subplot(111)

# add a small margin
ax1.margins(0.1)

# Evaluate the coefficients of the Newton polynomial
a = calculate_newton_poly_coeffs(xi, yi)
# Evaluate the polynomial at high resolution and plot
x = np.linspace(0.4, 9.1, 100)
ax1.plot(x, eval_newton_poly(a, xi, x), 'b', label='Newton polynomial')

# Overlay raw data
plot_raw_data(xi, yi, ax1)
ax1.set_title('Newton interpolating polynomial (our implementation)', fontsize=16)

# Add a legend
ax1.legend(loc='best', fontsize=14)

plt.show()
```

(nm_curve_fitting)=
## Curve fitting
```{index} Curve fitting
```

```{margin} Linear regression
Note that curve fitting is related to the topic of {ref}`regression analysis <linear_regress>`. [Least squares approximation](https://en.wikipedia.org/wiki/Least_squares) used in linear regression is a method of minimising the sum of the squares of the differences between the prediction and real data. Fitting a polynomial to data in a least squares sense is an example of what can be termed polynomial regression.
```

In [curve fitting](https://en.wikipedia.org/wiki/Curve_fitting), we want to find a function that approximates the data in some sense, but does not go through all the point. In this case we no longer have the requirement that

\\[x_0 < x_1 < \ldots < x_N,\\]

and can consider the data simply as a cloud of points.

This is the most typical case for real world data which contains variability and noise and could additionally give rise to multiple different measurements (i.e. \\(y\\) values) at the same \\(x\\) location. With real data this fitting of data to a function often has the effect of smoothing complex or noisy data.

### Polynomial curve fitting

Imagine having several points on a graph, and you want a curve that could go through all these points. The curve could be described through a mathematical function that could create a curve. Remember that straight lines could be considered a special case of a curve, much like how in topology, the square is a special case of a rectangle, and a rectangle is a special case of a parallelogram, and parallelogram a special case of a quadrilateral. One of the mathematical function that could create curves is a polynominal function.

A first degree polynomial function is basically a straight line, and is written in the form of:
\\[ y = mx + c.\\]

A second degree polynominal function is basically a curve with a single bend, and is written in the form of:
\\[ y = ax^2 + bx + c.\\]

A 3 degree polynominal function is a curve with two bends, and is written in the form of:
\\[ y = ax^3 + bx^2 + cx + d.\\]

Thus, a polynominal function of \\(n\\) degree is a curve with \\(n-1\\) bends.

```python
# A snippet to showing polynominal degree and number of bends

def poly_plot(xmin, xmax, number_of_points, a, b, c, d, ax):
    xi = np.linspace(xmin, xmax)
    yi = a*xi**3 + b*xi**2 + c*xi + d
    ax.plot(xi, yi, color="blue")
    ax.set_xlabel("x")

fig, axes = plt.subplots(1, 3, figsize=(10, 3))
ax1 = axes[0]
ax2 = axes[1]
ax3 = axes[2]

poly_plot(-10, 10, 100, 0, 0, 3, 4, ax1)
ax1.set_title("Straight line polynominal\nhas 0 bends")

poly_plot(-10, 10, 100, 0, 1, 1, -6, ax2)
ax2.set_title("Quadratic polynominal\nhas 1 bend")

poly_plot(-10, 10, 100, 1, 2, -5, -6, ax3)
ax3.set_title("Cubic polynominal\nhas 2 bends")

ax1.set_ylabel("y")

plt.show()
```

### Simple example

While we could just use the default plotting method to demonstrate linear interpolation, to demonstrate linear curve-fitting we need to do a bit more work.

The example below demonstrates how we can use `numpy.polyfit` to do this for us. The documentation can be found [here](https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html).

```python
numpy.polyfit(x, y, deg, rcond=None, full=False, w=None, cov=False)
```
```{admonition} Least squares polynomial fit

Fit a polynomial p(x) = p[0] * x**deg + ... + p[deg] of degree deg to points (x, y). Returns a vector of coefficients p that minimises the squared error in the order deg, deg-1, … 0.
```

`numpy.polyfit` returns the coefficients of the polynomial fit to the data. We can then use `numpy.poly1d` to turn this into a function we can easily evaluate, as seen in the next example.

```python
# Fit a polynomial of degree 1, i.e. a straight line, to our (x, y) data from above
degree = 1
poly_coeffs = np.polyfit(x, y, degree)
print('poly_coeffs: ',poly_coeffs)

# Use poly1d to turn the coeffs into a function, p1, we can evaluate
p1 = np.poly1d(poly_coeffs)
```

    poly_coeffs:  [0.0508044  0.26714649]

We can plot the raw data and the linear fit:

```python
# Set up figure
fig = plt.figure(figsize=(7, 5))
ax1 = fig.add_subplot(111)
ax1.margins(0.1)

# Plot the linear fit - define 100 evenly spaced points (x) covering our
# x extent and plot our linear polynomial evaluated at these points (p1(x))
# of course 100 is overkill for this linear example
xx = np.linspace(0., 9.5, 100)
ax1.plot(xx, p1(xx), 'b', label=r'$y = {0:.4f}x+{1:.4f}$'.format(poly_coeffs[0],
                                                                 poly_coeffs[1]))

# Overlay raw data
plot_raw_data(x, y, ax1)

# Add a legend
ax1.legend(loc='best', fontsize=12)

# Add a figure title
ax1.set_title('Raw data and the corresponding linear best fit line', fontsize=12)

plt.show()
```

A curve that goes through multiple points will have to bend and twist itself so that it goes through these points. Therefore, a curve with a higher polynominal degree, and thus a larger number of bends, will be able to bend and twist itself to better fit the points.

```python
# Consider the above example data again
xi = np.array([0.5,2.0,4.0,5.0,7.0,9.0])
yi = np.array([0.5,0.4,0.3,0.1,0.9,0.8])

# Calculate coefficients of polynomial degree 0 - ie a constant value.
poly_coeffs=np.polyfit(xi, yi, 0)

# Construct a polynomial function which we can use to evaluate for arbitrary x values.
p0 = np.poly1d(poly_coeffs)

# Fit a polynomial degree 1 - ie a straight line.
poly_coeffs=np.polyfit(xi, yi, 1)
p1 = np.poly1d(poly_coeffs)

# Quadratic
poly_coeffs=np.polyfit(xi, yi, 2)
p2 = np.poly1d(poly_coeffs)

# Cubic
poly_coeffs=np.polyfit(xi, yi, 3)
p3 = np.poly1d(poly_coeffs)

# Set up figure
fig = plt.figure(figsize=(7, 5))
ax1 = fig.add_subplot(111)
ax1.margins(0.1)

x = np.linspace(0.4, 9.1, 100)

ax1.plot(x, p0(x), 'k', label='Constant')
ax1.plot(x, p1(x), 'b', label='Linear')
ax1.plot(x, p2(x), 'r', label='Quadratic')
ax1.plot(x, p3(x), 'g', label='Cubic')

# Overlay raw data
plot_raw_data(xi, yi, ax1)

ax1.legend(loc='best', fontsize = 12)
ax1.set_title('Polynomial approximations of differing degree', fontsize=16)

plt.show()
```

### Squared error calculation

As described in the docs ([`numpy.polyfit`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.polyfit.html)), least squares fitting minimises the sum of the squares of the differences between the data provided and the polynomial approximation, i.e. it minimises the expression

\\[E = \sum_{i=0}^{N} (P(x_i) - y_i)^2,\\]

where \\(P(x_i)\\) is the value of the polynomial function that has been fit to the data evaluated at point \\(x_i\\), and \\(y_i\\) is the \\(i^{th}\\) data value.

<p style="text-align:center;"></p>

Source: [Wikipedia](https://en.wikipedia.org/wiki/Linear_least_squares)

Let's write a function that evualuates the squared error and use it to evaluate error for each of the polynomials calculated above.

```python
# We use the square of the difference to ensure each contribution
# to the total error is positive, otherwise errors of different signs
# could/would cancel out giving a false estimate of how good our approximation is

def sqr_error(p, xi, yi):
    """"function to evaluate the sum of square of errors"""
    # first compute the square of the differences
    diff2 = (p(xi)-yi)**2
    # and return their sum
    return diff2.sum()

print("Constant fit error = ", sqr_error(p0, xi, yi))
print("Linear fit error = ", sqr_error(p1, xi, yi))
print("Quadratic fit error = ", sqr_error(p2, xi, yi))
print("Cubic fit error = ", sqr_error(p3, xi, yi))
```

    Constant fit error =  0.4600000000000001
    Linear fit error =  0.3329889923793396
    Quadratic fit error =  0.1994782421425494
    Cubic fit error =  0.1573034366232376

(nm_extrapolation)=
## Extrapolation

Interpolation by definition is used to estimate \\(y\\) for values of \\(x\\) within the bounds of the available data (here \\([0.5,9]\\)) with some confidence. **Extrapolation** on the other hand is the process of estimating (e.g. using the interpolating function) \\(y\\) outside the bounds of the available data. However, extrapolation requires a great deal of care as it will become increasingly inaccurate as you go further out of bounds.

We can recreate the plots for different degrees of polynomial, setting x-range from -2 to 11.0 to see how extrapolation changes the result.

```python
# Xonsider the above example data again
xi = np.array([0.5, 2.0, 4.0, 5.0, 7.0, 9.0])
yi = np.array([0.5, 0.4, 0.3, 0.1, 0.9, 0.8])

# Let's set up some space to store all the polynomial coefficients
# there are some redundancies here, and we have assumed we will only
# consider polynomials up to degree N
N = 6
poly_coeffs = np.zeros((N, N))

for i in range(N):
    poly_coeffs[i, :(i+1)] = np.polyfit(xi, yi, i)

# Plot over a couple of different x ranges
x1 = np.linspace(-2., 11., 100)
x2 = np.linspace(0., 9.5, 100)

# Set up figure
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))

for i in range(N):
    p = np.poly1d(poly_coeffs[i, :(i+1)])
    ax1.plot(x1, p(x1), label='Degree %i' % i)

# Overlay raw data
plot_raw_data(xi, yi, ax1)
# Add a legend
ax1.legend(loc='best')
ax1.set_title('Extrapolation example - wider interval', fontsize=14)

# NB. if you set the limits of the x data to numpy.linspace(0., 9.5, 100)
# and replot this will result in you zooming in on the data to better see
# how each polynomial fits the data, but less clearly shows the major
# problem with extrapolation.

for i in range(N):
    p = np.poly1d(poly_coeffs[i, :(i+1)])
    ax2.plot(x2, p(x2), label='Degree %i' % i)

# Overlay raw data
plot_raw_data(xi, yi, ax2)
# Add a legend
ax2.legend(loc='best')
ax2.set_title('Extrapolation example - narrower interval', fontsize=14)

plt.show()
```


# Shallow water equations with a tracer

We can augment the shallow water equations discussed in [Shallow_water.ipynb](Shallow_water.ipynb) with a quantity $\phi(x,t)$ that measures the concentration of a tracer that is advected with the fluid motion (but has no influence on the fluid dynamics).  If $\phi$ is measured in units of mass per unit volume (which is really per unit area of the vertical cross section in this one-dimensional example) then the mass per unit length along the $x$ axis is given by $h(x,t)\phi(x,t)$.  The quantity $\phi$ satisfies the variable coefficient advection equation in advective (non-conservative) form:
$$
\phi_t(x,t) + u(x,t)\phi_x(x,t) = 0.
$$
This is also called the "color equation", since we can think of $\phi$ as measuring the concentration of a dye that changes the color of the water.  We will use this interpretation in the plots below.  By setting the intitial conditions $\phi(x,0)$ to be piecewise constant with different values corresponding to different colors, we can visualize the motion of the fluid better.   We will use two shades of blue for the water that is initially to the left of a dam at $x=0$ and two lighter shades of blue for the water that is initially to the right.

The quantity $h\phi$ satisfies the conservative form of the advection equation,
$$
(h\phi)_t + (uh\phi)_x = 0.
$$
This can be derived by combining the color equation with the conservation of mass equation $h_t +(hu)_x  = 0$.  Since $h\phi$ measures the concentration of dye per unit length in $x$, and we assume molecules of dye are not created or destroyed, it makes sense that this is the conserved quantity.

The full system of equations in conservation form is thus:
\begin{align} \label{SW-tracer}
\begin{split}
h_t + (hu)_x &=0\\
(hu)_t + \left(hu^2 + \frac 1 2 gh^2\right)_x &=0\\
(h\phi)_t + (uh\phi)_x &= 0
\end{split}
\end{align}

The Riemann solution for this system has 3 waves.  Two of these waves, with speeds $u\pm c$ (where $c=\sqrt{gh}$) are just the same waves we studied in the shallow water system.  We refer to these as the 1-wave and 2-wave, to maintain consistency with the last chapter.  The additional wave appearing in this system (the *tracer* wave) moves at the fluid velocity $u$ and carries only a jump in the tracer concentration $\phi$.  It is our first example of what is known as a contact discontinuity or linearly degenerate wave.  Because the fluid velocity is constant across this wave, tracer characteristics travel parallel to the wave on either side.  Thus this family does not admit shock or rarefaction waves.  The 1- and 2-waves can of course each be either a shock wave or rarefaction wave depending on the initial data, as explained in [Shallow_water.ipynb](Shallow_water.ipynb).  Below we consider a dam break problem in which case there is one of each.

```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = 'svg'
from ipywidgets import widgets, fixed
from ipywidgets import interact

from exact_solvers import shallow_water
demo_plot = shallow_water.make_demo_plot_function
```

## Dam break problem

Here we plot the solution to the dam break Riemann problem for the shallow water equations with the addition of a tracer.  The tracer is now also take to be piecewise constant, with light blue representing the value $\phi_l$ taken in the water the left of the dam, while dark blue represents the value $\phi_r$ in the water on the right side of the dam.

The structure of the depth and velocity are exactly the same as seen and discussed in detail in [Shallow_water.ipynb](Shallow_water.ipynb), and the value of $\phi$ is constant across the 1-rarefaction and 3-shock waves.    The discontinuity in $\phi$ shows the propagation of the 2-wave in the Riemann solution, the contact discontinuity across which $\phi$ is discontinuous while  $h$ and $u$ are continuous.

```python
shallow_water.plot_riemann_SW(h_l=3, h_r=1, u_l=0., u_r=0., tracer=True)
```

## Hyperbolic structure
We can write (\ref{SW-tracer}) in the canonical form $q_t + f(q)_x = 0$ if we define
\begin{align}
q & = \begin{pmatrix} h \\ hu \\ h \phi \end{pmatrix} & f & = \begin{pmatrix} hu \\ hu^2 + \frac{1}{2}gh^2 \\ uh\phi\end{pmatrix}.
\end{align}
In terms of the conserved quantities, the flux is
\begin{align}
f(q) & = \begin{pmatrix} q_2 \\ q_2^2/q_1 + \frac{1}{2}g q_1^2 \\ q_3 q_2/q_1 \end{pmatrix}
\end{align}
Thus the flux jacobian is
\begin{align}
f'(q) & = \begin{pmatrix} 0 & 1 & 0  \\ -(q_2/q_1)^2 + g q_1 & 2 q_2/q_1 & 0 \\ -q_3 q_2/q_1^2 & q_3/q_1 & q_2/q_1 \end{pmatrix}
        = \begin{pmatrix} 0 & 1 & 0 \\ -u^2 + g h & 2 u & 0 \\ -u\phi & \phi & u \end{pmatrix}.
\end{align}
Its eigenvalues are
\begin{align} \label{SW:char-speeds}
    \lambda_1 & = u - \sqrt{gh} & \lambda_\text{tracer} & = u & \lambda_2 & = u + \sqrt{gh},
\end{align}
with corresponding eigenvectors
\begin{align} \label{SW:fjac-evecs}
    r_1 & = \begin{bmatrix} 1 \\ u-\sqrt{gh} \\ \phi \end{bmatrix} &
    r_\text{tracer} & = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} &
    r_2 & = \begin{bmatrix} 1 \\ u+\sqrt{gh} \\ \phi \end{bmatrix}
\end{align}
We see immediately that the tracer wave carries only a jump in the tracer itself.  It may seem puzzling that the third entry of $r_1$ and $r_3$ is nonzero.  Notice that each of these waves carries a jump in $h$; since $\phi$ is constant across each of these waves, there must be a corresponding jump in $q_3=\phi h$.

## Linear degeneracy
The speed of the tracer wave depends only on $u$, and $u$ is constant across the tracer wave.  This means that tracer characteristics can neither converge nor diverge due to a jump in $\phi$, but rather run parallel to it.  Characteristic fields with this property are referred to as being **linearly degenerate**.  We will see another example of a linearly degenerate family in the next chapter.


# Computational Astrophysics
## Numerical Integration 01

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the numerical integration techniques.

---

One of the most common tasks in astrophysics is to evaluate integrals such as
\begin{equation}
I = \int_a^b f(x) dx ,
\end{equation}
and, in many cases, it can not be done analytically. The integrand may be given as an analytical function $f(x)$ or a discrete set of values $f(x_i)$ and we will describe the methods to evaluate numerically the integration in both cases.

---
## Piecewise Interpolation and Quadratures

Any integration method which uses a weighted sum to evaluate numerically the integral is called a **quadrature rule**. Suppose that we know (or can evaluate) the integrand $f(x)$ at a finite set of **nodes**, $\{x_j\}$ with $j=0,\cdots,n$ in the interval $[a,b]$ and such that $x_0 = a$ and $x_n = b$. This gives a set of $n+1$ nodes or equivalently $n$ intervals. A discrete approximation of the integral of this function is given by the **rectangle rule**,
\begin{equation}
I = \int_a^b f(x) dx \approx  \Delta x \sum_{i=0}^{n-1} f(x_i),
\end{equation}
where the width of the intervals is $\Delta x = \frac{b-a}{n}$. It is clear, from the definition of an integral, that this approximation converges to the real integral when $n\rightarrow \infty$, i.e. when $\Delta x \rightarrow 0$. Although the rectangle rule may give a good approximation to the integral, it can be improved by using an interpolated function in each interval. These methods involving interpolation of polynomials are called, in general, **Newton-Cotes quadratures**

---
### Midpoint Rule

The simplest modification to the rectangle rule described above is to use the central value of the function $f(x)$ in each interval instead of a value at one of the nodes. Hence, if it is possible to evaluate the integrand at the midpoint of each interval, the approximation of the integral is given by
\begin{equation}
I = \int_{a}^{b} f(x) dx = \sum _{i=0}^{n-1} (x_{i+1} - x_i)  f(\bar{x}_i ),
\end{equation}
where $\bar{x}_i = \frac{x_i + x_{i+1}}{2}$ is the midpoint of the interval $[x_i, x_{i+1}]$.

In order to estimate the error associated with this method, we use a Taylor expansion of the integrand in an interval $[x_i, x_{i+1}]$ about its midpoint $\bar{x}_i$,
\begin{equation}
f(x) = f(\bar{x}_i) + f'(\bar{x}_i)(x-\bar{x}_i) + \frac{f''(\bar{x}_i)}{2}(x-\bar{x}_i)^2 + \frac{f'''(\bar{x}_i)}{6}(x-\bar{x}_i)^3 + ...
\end{equation}

Integrating this expression from $x_i$ to $x_{i+1}$, and noting that the odd-order terms vanish, we get
\begin{equation}
\int_{x_i}^{x_{i+1}} f(x)dx = f(\bar{x}_i)(x_{i+1}-x_i) + \frac{f''(\bar{x}_i)}{24}(x_{i+1}-x_i)^3 + ...
\end{equation}

This expansion shows that the error induced by the approximation in each interval is of the order of $\varepsilon_i =  (x_{i+1}-x_i)^3$. Since the total integral is obtained as a sum of $n$ integrals like this one, the total error is of the order of $\varepsilon = n \varepsilon_i $.

When the nodes are equally spaced, we can write the size of the intervals as the constant  $h = \frac{b - a}{n}$ and therefore, the error associated with each interval is $\varepsilon_i =\frac{(b - a)^3}{n^3} = h^3$, while the total error of the quadrature is $\varepsilon = n \varepsilon_i = \frac{(b - a)^3}{n^2} = nh^3$.

#### Example. Integration with the midpoint rule.

We will read a data set from a .txt file and numerically integrate using the midpoint rule. Since the function is given as data points (and not as an analytical function), we cannot evaluate its value at the midppoint. Therefore, we will use the value of the function at the first point in each interval to calcualte the aprtial sums.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Reading the data
data = np.loadtxt('data_points1.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    h = x[i+1] - x[i]
    Integral = Integral + h*f[i]

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Midpoint Rule Integration')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'The result of the numerical integration of the discretely-sampled function')
print(f'between x = {x[0]:.1f} and x = {x[len(x)-1]:.1f} is I = {Integral:.5e}')
```

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Reading the data
data = np.loadtxt('data_points2.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    h = x[i+1] - x[i]
    Integral = Integral + h*f[i]

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Midpoint Rule Integration')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'The result of the numerical integration of the discretely-sampled function')
print(f'between x = {x[0]:.1f} and x = {x[len(x)-1]:.1f} is I = {Integral:.5e}')
```

---
### Trapezoidal Rule

The next generalization of the rectangle rule is to approximate $f(x)$ with a linear polynomial on each interval.  This is known as the **trapezoidal rule** and the corresponding quadrature is given by

\begin{equation}
I = \int_{a}^{b} f(x) dx = \sum _{i=0}^{n-1} \frac{1}{2} (x_{i+1} - x_i)  \left[ f(x_{i+1}) + f(x_i) \right] .
\end{equation}

Contrary to the midpoint rule, this method doesn't need to evaluate the integrand in the midpoint, but only needs its values at the nodes.

#### Example. Integration with the trapezoidal rule.

We will read a data set from a .txt file and numerically integrate using the trapezoidal rule.

```python
import numpy as np
import matplotlib.pyplot as plt

# Reading the data
data = np.loadtxt('data_points1.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    h = x[i+1] - x[i]
    f_mean = (f[i] + f[i+1])/2
    Integral = Integral + h*f_mean

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i+1]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i+1]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i+1]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Trapezoidal Rule Integration')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'The result of the numerical integration of the discretely-sampled function')
print(f'between x = {x[0]:.1f} and x = {x[len(x)-1]:.1f} is I = {Integral:.5e}')
```

```python
import numpy as np
import matplotlib.pyplot as plt

# Reading the data
data = np.loadtxt('data_points2.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    h = x[i+1] - x[i]
    f_mean = (f[i] + f[i+1])/2
    Integral = Integral + h*f_mean

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i+1]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i+1]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i+1]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Trapezoidal Rule Integration')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'The result of the numerical integration of the discretely-sampled function')
print(f'between x = {x[0]:.1f} and x = {x[len(x)-1]:.1f} is I = {Integral:.5e}')
```

---
## Simpson's Rule

Simpson's Rule is a method that estimates the integral of a function $f(x)$ by approximating it as a second order polynomial in each interval.

If we have three known values of the function $f_1 =f(x_1)$, $f_2 =f(x_2)$ and $f_3 =f(x_3)$ at the points $x_1 < x_2 < x_3$, we fit a second order polynomial with the form
\begin{equation}
p_2 (x) = A (x-x_1)^2 + B (x-x_1) + C .
\end{equation}

Integrating this polynomial in the interval $[x_1 , x_3]$, we obtain the value
\begin{equation}
\int_{x_1}^{x^3} p_2 (x) dx = \frac{x_3 - x_1}{6} \left( f_1 + 4f_2 + f_3 \right) + \mathcal{O} \left( (x_3 - x_1)^5 \right)
\end{equation}

---
### Simpson's Rule with evenly spaced points

If there are $N$ evenly spaced nodes in the integration interval, or equivalently $n=N-1$ intervals with constant width $h$, the total integral by Simpson's rule is writen

\begin{equation}
I = \int_a^b f(x) dx \approx \frac{h}{3} \sum_{i=0}^{\frac{n-2}{2}} \left[ f(x_{2i}) + 4f(x_{2i+1}) + f(x_{2i+2}) \right] .
\end{equation}

The numerical error in each interval is of the order of $h^5$ and therefore, the total integral will have an error of the order of $n h^5 = \frac{(a-b)^5}{n^4}$.

#### Example. Integration with the Simpson's rule.

We will read a data set from a .txt file and numerically integrate using Simpson's rule.

```python
import numpy as np
import matplotlib.pyplot as plt

def quadraticInterpolation(x1, x2, x3, f1, f2, f3, x):
    p2 = (((x-x2)*(x-x3))/((x1-x2)*(x1-x3)))*f1 + (((x-x1)*(x-x3))/((x2-x1)*(x2-x3)))*f2 +\
         (((x-x1)*(x-x2))/((x3-x1)*(x3-x2)))*f3
    return p2

# Reading the data
data = np.loadtxt('data_points1.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)
n = N-1

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(int((n-2)/2 +1)):
    h = x[2*i] - x[2*i+1]
    Integral = Integral + h*(f[2*i] + 4*f[2*i+1] + f[2*i+2])/3

    x_interval = np.linspace(x[2*i],x[2*i+2],6)
    y_interval = quadraticInterpolation(x[2*i], x[2*i+1], x[2*i+2], f[2*i], f[2*i+1], f[2*i+2], x_interval)
    plt.plot(x_interval, y_interval,'r')
    plt.fill_between(x_interval, y_interval, color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Simpson\'s Rule Integration')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'The result of the numerical integration of the discretely-sampled function')
print(f'between x = {x[0]:.1f} and x = {x[len(x)-1]:.1f} is I = {Integral:.5e}')
```

```python
import numpy as np
import matplotlib.pyplot as plt

def quadraticInterpolation(x1, x2, x3, f1, f2, f3, x):
    p2 = (((x-x2)*(x-x3))/((x1-x2)*(x1-x3)))*f1 + (((x-x1)*(x-x3))/((x2-x1)*(x2-x3)))*f2 +\
         (((x-x1)*(x-x2))/((x3-x1)*(x3-x2)))*f3
    return p2

# Reading the data
data = np.loadtxt('data_points2.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)
n = N-1

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(int((n-2)/2 +1)):
    h = x[2*i] - x[2*i+1]
    Integral = Integral + h*(f[2*i] + 4*f[2*i+1] + f[2*i+2])/3

    x_interval = np.linspace(x[2*i],x[2*i+2],6)
    y_interval = quadraticInterpolation(x[2*i], x[2*i+1], x[2*i+2], f[2*i], f[2*i+1], f[2*i+2], x_interval)
    plt.plot(x_interval, y_interval,'r')
    plt.fill_between(x_interval, y_interval, color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Simpson\'s Rule Integration')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'The result of the numerical integration of the discretely-sampled function')
print(f'between x = {x[0]:.1f} and x = {x[len(x)-1]:.1f} is I = {Integral:.5e}')
```

---
### Simpson's Rule for non-equally spaced data

When the known nodes of $f(x)$ are non-equally spaced, the Simpson's rule must be modified to be

\begin{equation}
I = \int_a^b f(x) dx \approx \sum_{i=0}^{\frac{n-2}{2}} \left[ \alpha f(x_{2i}) + \beta f(x_{2i+1}) +\gamma f(x_{2i+2}) \right]
\end{equation}
where
\begin{align}
\alpha = &\frac{-h_{2i+1}^2 + h_{2i+1} h_{2i} + 2 h_{2i}^2}{6 h_{2i}}   \\
\beta = &\frac{ (h_{2i+1} + h_{2i})^3 }{6 h_{2i+1} h_{2i}} \\
\gamma =&  \frac{2 h_{2i+1}^2 + h_{2i+1} h_{2i} - h_{2i}^2}{6 h_{2i+1}}
\end{align}
and $h_j = x_{j+1} - x_j$.


```python
#Exercise 2:

! pip install sympy
! pip install numpy
! pip install scipy
! pip install matplotlib
! pip install random
import matplotlib.pyplot as plt
from scipy.optimize import minimize
import numpy as np
import time
import random as r
import sympy as sp
from sympy import symbols
```

    Requirement already satisfied: sympy in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (1.6.2)
    Requirement already satisfied: mpmath>=0.19 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from sympy) (1.1.0)
    Requirement already satisfied: numpy in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (1.19.2)
    Requirement already satisfied: scipy in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (1.5.2)
    Requirement already satisfied: numpy>=1.14.5 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from scipy) (1.19.2)
    Requirement already satisfied: matplotlib in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (3.3.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (2.4.7)
    Requirement already satisfied: numpy>=1.15 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (1.19.2)
    Requirement already satisfied: certifi>=2020.06.20 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (2020.6.20)
    Requirement already satisfied: kiwisolver>=1.0.1 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (1.2.0)
    Requirement already satisfied: pillow>=6.2.0 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (8.0.0)
    Requirement already satisfied: cycler>=0.10 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from matplotlib) (2.8.1)
    Requirement already satisfied: six in /opt/miniconda3/envs/conda_env/lib/python3.7/site-packages (from cycler>=0.10->matplotlib) (1.15.0)
    ERROR: Could not find a version that satisfies the requirement random (from versions: none)
    ERROR: No matching distribution found for random

```python

#Case of The Banana (Rosenbrock) Function
#The gradient descent approach will be used to  minimise the function

from sympy.plotting import plot3d
x1 , x2 = symbols ( 'x1 , x2' )
f1 = 100*(x2 - x1**2)**2 + (1 - x1)**2
plot3d ( f1 , (x1 , -5 ,5) ,(x2 , -5,5) )
```

```python

def rosenbrock(X):  #cost function
    """Rosenbrock Equation"""
    x1=X[0]
    x2=X[1]
    return 100.0*(x2-x1**2.0)**2 + (1-x1)**2

def rosenbrock_gradient(X):#gradient function
    """Partial  derivatives of the Rosenbrock Equation."""
    x1=X[0]
    x2=X[1]
    dfx1=(-400*x1*x2) + (400*x1**3) + (2*x1 -2) #partial  derivative with respect to X1
    dfx2=200*x2 -200*x1**2 #partialderivative with respect to  X2
    return np.array([dfx1,dfx2])
def result(p):
    """Tracing the states of optimization."""
    global ps
    ps.append(rosenbrock(p))
    return ps

# Optimising and Plotting
fig=plt.figure(figsize=(13,5))
for i in range(10):
        Random_start=np.array([np.random.uniform(-4,4),np.random.uniform(-4,4)])
        ps=[]
        start_time = time.time()
        random.seed(250)
        optimized_rosenbrock= minimize(rosenbrock,Random_start,method="BFGS", jac=rosenbrock_gradient, callback=result)
        print(f"--- time used for function call in Run {i+1} is %s seconds ---" % (time.time() - start_time))
        print("the run's number:",i+1)
        print("the starting point is:",Random_start)
        print("the number of iterations is:",optimized_rosenbrock.nit)
        print("the optimal points are:",optimized_rosenbrock.x)
        print("the optimal  value is %.6f"%optimized_rosenbrock.fun)
        print("\n")

        rosebfgs=fig.add_subplot(121)
        rosebfgs.set_title("Rosenbrock Optimization with BFGS ",fontweight="bold")
        rosebfgs.set_ylabel("values")
        rosebfgs.set_xlabel("Iterations")
        rosebfgs.grid(color='lightgray', alpha=0.7)
        rosebfgs.plot(ps,marker='', label=f"Run {i+1}")
        rosebfgs.plot(0,ps[0],marker='', color=plt.gca().lines[-1].get_color())
        rosebfgs.plot(len(ps)-1,ps[-1],marker='.',color=plt.gca().lines[-1].get_color())
        rosebfgs.grid(color='lightgray', alpha=0.5)

        #setting parameters for log plot
        log_rosebfgs=fig.add_subplot(122)
        log_rosebfgs.grid(color='lightgray', alpha=0.5)
        log_rosebfgs.set_title("Rosenbrook Optimization with BFGS(log)",fontweight="bold")
        log_rosebfgs.set_ylabel("values")
        log_rosebfgs.set_xlabel("iterations")
        log_rosebfgs.plot(ps,marker='', label=f"Run {i+1}")
        log_rosebfgs.plot(0,ps[0],marker='.', color=plt.gca().lines[-1].get_color())
        log_rosebfgs.plot(len(ps)-1,ps[-1],marker='.',color=plt.gca().lines[-1].get_color())
        log_rosebfgs.set_yscale('log')
        log_rosebfgs.legend(loc="lower right",bbox_to_anchor=(1.40,-0.0),borderaxespad=0., ncol=1, title="Runs")

```

```python

```

```python

```

```python
#Case of eggcrate function
#The gradient descent approach will be used to  minimise the function
x , y = symbols ( 'x , y' )
f2 = x**2 + y**2 + 25*(sp.sin(x)**2 + sp.sin(y)**2)
plot3d ( f2 , (x , -5 ,5) ,(y, -5,5) )
```

```python

def eggcrate(X):  #cost function
    """ Eggcrate function"""
    x=X[0]
    y=X[1]
    return x**2 + y**2 + 25*(np.sin(x)**2 + np.sin(y)**2)

def eggcrate_gradient(X):#gradient function
    """ Partial  derivatives of the Eggcrate Equation."""
    x=X[0]
    y=X[1]
    fdx= 2*x + 25*(2*np.sin(x)*np.cos(x))#partial  derivative with respect to x
    fdy=2*y +25*(2*np.sin(y)*np.cos(y)) #partial derivative with respect to  y
    return np.array([fdx,fdy])

es=[]
def tracker(e):
    """Tracing the states of optimization."""
    global es
    es.append(eggcrate(e))
    return es

# Optimising and Plotting
fig=plt.figure(figsize=(17,5))

for i in range(10):
        r.seed(250)
        Random_start1=np.array([np.random.uniform(-2*np.pi, 2*np.pi,),np.random.uniform(-2*np.pi, 2*np.pi,)])
        print("the run's number is:",i+1)
        print("the starting point is:",Random_start1)
        es=[]
        start_time1 = time.time()
        optimized_eggcrate= minimize(eggcrate,Random_start1,method="BFGS", jac=eggcrate_gradient, callback=tracker)
        print(f"--- time used for function call in run {i+1} is %s seconds ---" % (time.time() - start_time1))
        print("it took",optimized_eggcrate.nit,"iterations")
        print("the optimal points are:",optimized_eggcrate.x)
        print("the optimal  value is:%.6f"%optimized_eggcrate.fun)
        print(optimized_eggcrate.message)
        print("\n")
        eggbfgs=fig.add_subplot(121)
        eggbfgs.set_title("Eggcrate Optimization using BFGS ",fontweight="bold")
        eggbfgs.set_ylabel("values")
        eggbfgs.set_xlabel("Iterations")
        eggbfgs.grid(color='lightgray', alpha=0.7)
        eggbfgs.plot(es,marker='', label=f"Run {i+1}")
        eggbfgs.plot(0,es[0],marker='.', color=plt.gca().lines[-1].get_color())
        eggbfgs.plot(len(es)-1,es[-1],marker='.',color=plt.gca().lines[-1].get_color())
        eggbfgs.grid(color='lightgray', alpha=0.5)

        #setting parameters for log plot
        log_eggbfgs=fig.add_subplot(122)
        log_eggbfgs.grid(color='lightgray', alpha=0.5)
        log_eggbfgs.set_title("Eggcrate Optimization with BFGS(log)",fontweight="bold")
        log_eggbfgs.set_ylabel("values")
        log_eggbfgs.set_xlabel("iterations")
        log_eggbfgs.plot(es,marker='', label=f"Run {i+1}")
        log_eggbfgs.plot(0,es[0],marker='.', color=plt.gca().lines[-1].get_color())
        log_eggbfgs.plot(len(es)-1,es[-1],marker='.',color=plt.gca().lines[-1].get_color())
        log_eggbfgs.set_yscale('log')
        log_eggbfgs.legend(loc="lower right",bbox_to_anchor=(1.25,-0.0),borderaxespad=0., ncol=1, title="Runs")

```

```python

```

```python

```

```python

```

```python
#Case of Golinski.
#The SLSQP solver was used

def Golinski(X):
    """ Cost Function"""
    x1=X[0]
    x2=X[1]
    x3=X[2]
    x4=X[3]
    x5=X[4]
    x6=X[5]
    x7=X[6]
    Gol=0.7854*(x1*x2**2)*(3.3333*x3**2+14.9334*x3-43.0934)      -1.5079*x1*(x6**2+x7**2)+7.477*(x6**3+x7**3)      +0.7854*(x4*x6**2+x5*x7**2)
    return Gol

#the following functions cons1 to cons11 are the constraints
def cons1(X):
    return 1-(27/(X[0]*X[1]**2*X[2]))
def cons2(X):
    return 1-(397.5/(X[0]*X[1]**2*X[2]**2))
def cons3(X):
    return 1-(1.93*X[3]**3/(X[1]*X[2]*X[5]**4))
def cons4(X):
    return 1-(1.93*X[4]**3/(X[1]*X[2]*X[6]**4))
def cons5(X):
    return 1-(np.sqrt((745*X[3]/(X[1]*X[2]))**2 + 16.9*10**6)/(110.0*X[5]**3))
def cons6(X):
    return 1-(np.sqrt((745*X[4]/(X[1]*X[2]))**2+157.5*10**6)/(85.0*X[6]**3))
def cons7(X):
    return 1-(X[1]*X[2]/40)
def cons8(X):
    return 1-(5*X[1]/X[0])
def cons9(X):
    return 1-(X[0]/(12*X[1]))
def cons10(X):
    return 1-((1.5*X[5]+1.9)/X[3])
def cons11(X):
    return 1-((1.1*X[6]+1.9)/X[4])

bound=np.array([[2.6,3.6],[0.7,0.8],[17.0,28.0],[7.3,8.3],[7.3,8.3],[2.9,3.9],[5.0,5.9]]) #bounds for each decision variable x1 to x7

cons=({'type':'ineq',  'fun':  cons1},
      {'type':'ineq',  'fun':  cons2},
      {'type':'ineq',  'fun':  cons3},
      {'type':'ineq',  'fun':  cons4},
      {'type':'ineq',  'fun':  cons5},
      {'type':'ineq',  'fun':  cons6},
      {'type':'ineq',  'fun':  cons7},
      {'type':'ineq',  'fun':  cons8},
      {'type':'ineq',  'fun':  cons9},
      {'type':'ineq',  'fun':  cons10},
      {'type':'ineq',  'fun':  cons11})

#tracing the results of the SLSQP
def golinski_tracker(X):
    global Nfeval
    Nfeval=1
    global funcvals
    funcvals.append(Golinski(X))
    Nfeval += 1
    return funcvals

# Optimising and Plotting
fig=plt.figure(figsize=(17,5))
for i in range(10):
        funcvals = []      # funcvals array for each run

        Xo=np.array([np.random.uniform(2.6,3.61),np.random.uniform(0.7,0.81),np.random.uniform(17.0,28.0),
                np.random.uniform(7.3,8.31),np.random.uniform(7.3,8.31),np.random.uniform(2.9,3.9),
                np.random.uniform(5.0,5.91)])
        print(f"the run's number{i+1}:")
        print("the starting point is:[{0:3.4f}, {1: 3.4f},{2: 3.4f},{3: 3.4f},{4: 3.4f},{5: 3.4f},{6: 3.4f}]"
              .format(Xo[0],Xo[1],Xo[2],Xo[3],Xo[4],Xo[5],Xo[6]))

    # defining the global variables so as to change them in local scope
        #funcvals.append(Golinski(Xo))
        start_time=time.time()
        GoSolution = minimize(Golinski, Xo, bounds=bound, constraints=cons,
                       callback=golinski_tracker, method='SLSQP',options={'disp':False,'full_output':True})
        funcvals.append(Golinski(GoSolution.x))
        end_time=time.time()

        print("the number of iterations is:",GoSolution.nit)
        print("the optimal points are:[{0:3.4f}, {1: 3.4f},{2: 3.4f},{3: 3.4f},{4: 3.4f},{5: 3.4f},{6: 3.4f}]"
              .format(GoSolution.x[0],GoSolution.x[1],GoSolution.x[2],GoSolution.x[3],GoSolution.x[4],GoSolution.x[5],GoSolution.x[6]))
        print("the optimal  value is:%.4f"%GoSolution.fun)
        print(GoSolution.message)
        print("--- time used for function call in run is %s seconds ---" % (end_time- start_time))
        print("\n")

    # Plot the graph for each run on single canvas (figure)
        ax1=fig.add_subplot(121)
        ax1.set_title("Golinski Optimization using SLSQP ",fontweight="bold")
        ax1.set_ylabel("values")
        ax1.set_xlabel("Iterations")
        ax1.grid(color='lightgray', alpha=0.7)
        ax1.plot(funcvals,marker='', label=f"Run {i+1}")
        ax1.plot(0,funcvals[0],marker='.', color=plt.gca().lines[-1].get_color())
        ax1.plot(len(funcvals)-1,funcvals[-1],marker='.',color=plt.gca().lines[-1].get_color())
        ax1.grid(color='lightgray', alpha=0.5)

    #setting parameters for log plot
        ax2=fig.add_subplot(122)
        ax2.grid(color='lightgray', alpha=0.5)
        ax2.set_title("Golinski Optimization with SLSQP(log)",fontweight="bold")
        ax2.set_ylabel("values")
        ax2.set_xlabel("iterations")
        ax2.plot(funcvals,marker='', label=f"Run {i+1}")
        ax2.plot(0,funcvals[0],marker='', color=plt.gca().lines[-1].get_color())
        ax2.plot(len(funcvals)-1,funcvals[-1],marker='.',color=plt.gca().lines[-1].get_color())
        ax2.set_yscale('log')
        ax2.legend(loc="lower right",bbox_to_anchor=(1.25,-0.0),borderaxespad=0., ncol=1, title="Runs")

```

```python

```