# Convex Optimization Project Proposal
This notebook describes a convex optimization problem to be studied in a one-week project as part of the course Convex Optimization.

Group: Mads Holst Aagaard Madsen, Emilie Knudsen Brun, and Oliver Lylloff

The optimization problem could be characterized as *covariance matrix fitting* where measured data, given by a  covariance matrix, is used to estimate a source covariance matrix through a propagation model.

Consider a measured (hermitian) covariance matrix $P\in\mathbb{C}^{M\times M}$ and a sound propagation model defined by transfer functions

$$g_{m,n} = \frac{r_{m,n}}{r_{0,n}}e^{i\omega r_{m,n}/c}\quad m=1,...,M,\;n=1,...,N, $$

where $\omega = 2\pi f$ is the angular frequency, $c = 343 m/s$ is the speed of sound, $r_{0,n}$ is the distance from the center of the microphone array to the $n$'th source point, and $r_{m,n}$ is the distance between the $m$'th microphone and $n$'th source point. The transfer functions take into account the propagation of sound from the sources to the microphones. These are arranged in a transfer matrix $G\in\mathbb{C}^{N\times M}$. The model can be decribed by

$$P = GXG^H$$

where $X\in \mathbb{C}^{N\times N}$ is a covariance matrix at the source plane which is to be estimated.

The optimization problem can be stated as a SDP:

$$\begin{align}
\begin{aligned}
\underset{X}{\mathrm{minimize}}\quad & \|P-GXG^H\|_F^2\\
\mathrm{subject \ to} \quad & X\succeq 0\\
\end{aligned}
\end{align}
$$

Several options for the solution can be persued:
* $X$ is diagonal and sparse. In this case the number of variables can be reduced greatly. Additionally a penalty function with regularization parameter $\lambda$ can be added to the objective function, e.g.,

$$\begin{align}
\begin{aligned}
\underset{X}{\mathrm{minimize}}\quad & \|P-GXG^H\|_F^2+\lambda\;\text{tr(diag}(X))\\
\mathrm{subject \ to} \quad & X\succeq 0\\
\end{aligned}
\end{align}
$$

* Reducing the number of variables by introducing a matrix $M$ of same dimensions as $P$ and writing $X = G^HMG+D$, i.e.,

$$\begin{align}
\begin{aligned}
\underset{X}{\mathrm{minimize}}\quad & \|P-G(G^HMG+D)G^H\|_F^2+\lambda\sum_{i=1}^n(D_{ii}+e_i^TG^HMGe_i)\\
\mathrm{subject \ to} \quad & M\succeq 0\\
\quad & D_{ii}\geq0
\end{aligned}
\end{align}
$$

and using $e_i^TG^HMGe_i = \sum_i tr(MGe_ie_iG^H)$, where $e_i$ is a unit-vector. Use resulting $M$ to obtain $X$.

* First-order methods and warmstart. The optimization problem should be solved multiple times for different $\omega$'s. Exploit warmstart capability of first-order methods and the assumption that covariance matrices does not change much between subsequent $\omega$'s.

### Data analysis and modelling
The visualization of noise using a microphone array is produced by a beamforming algorithm. One such example is given below:

```julia
function beamformer(Ns,X,Y,z0,f,rn,CSM)
    const M = size(rn,1)    # Number of microphones
    const omega = 2pi*f     # Angular frequency
    const c = 343           # Speed of sound

    r0 = sqrt(X.^2 + Y.^2 .+ z0^2)

    # Allocation of arrays
    rmn = Array(Float64,Ns,Ns,M);
    gj = Array(Complex128,Ns,Ns,M);
    b = Array(Complex128,Ns,Ns);

    # Compute transfer functions
    for i in 1:Ns
        for j in 1:Ns
            for m in 1:M
                rmn[i,j,m] = sqrt((X[i,j]-rn[m,1])^2+(Y[i,j]-rn[m,2])^2 + z0^2);
                gj[i,j,m] = (rmn[i,j,m]/r0[i,j])*exp(im*omega*rmn[i,j,m]/c);
            end
        end
    end

    CSM[eye(Bool,M)] = 0;    # Diagonal removal

    for i = 1:Ns
        for j = 1:Ns
            b[i,j] = (dot(vec(gj[i,j,:]),CSM*vec(gj[i,j,:])))/(M^2-M);
        end
    end
    return b,gj
end
```

    beamformer (generic function with 1 method)

The experimental data used for this project is obtained from the Python package [Acoular](www.acoular.org). Time sampling of 56 microphones arranged in a microphone array can be found on the [Acoular Github page (a 6 MB download stored in HDF5 format)](https://github.com/acoular/acoular/blob/master/examples/example_data.h5?raw=true). The data is averaged and tranformed into the frequecy domain to obtain a covariance matrix.

Below is an example of the covariance matrix and a beamforming map. Solving the optimization problem results in a matrix $X$ with a diagonal that can be used to visualize the noise distribution in the source plane with a much higher resolution than that of the beamforming map.

```julia
using MAT, GR
inline()

file = matopen("CSM_TE.mat")
CSM_TE = read(file, "CSM")
close(file)

file = matopen("micgeom.mat")
rn = read(file, "micgeom")
close(file)

Ns = 50    # i.e. N = Ns^2

id = 80
df = 47
f = id*df
rx = linspace(-0.6,0,Ns)
ry = linspace(-0.3,0.3,Ns)
X,Y = (Float64[i for i in rx, j in ry],Float64[j for i in rx, j in ry])

b,gj = beamformer(Ns,X,Y,0.68,f,rn,CSM_TE[:,:,id])

# Plot example

GR.figure(size=(750,300))
GR.subplot(1,2,1)
GR.heatmap(abs(flipdim(CSM_TE[:,:,id],1)),title="CSM matrix, rank = $(rank(CSM_TE[:,:,id]))")
GR.subplot(1,2,2)
GR.contourf(rx,ry,real(b)./maximum(real(b)),xlabel="x",ylabel="y",title="Beamforming at $f Hz")
```

## Background

The generation of electricity from renewable energy sources is more important than ever following the global agreement signed by 195 countries at the COP21 climate conference in Paris, December 2015. The long term goal, to limit the global average temperature increase to 2$^{\circ}$C, requires governments around the world to limit their dependence on fossil fuels and adapt renewable energy sources.

This goal requires development and installation of more wind turbines (onshore and offshore) in the following years. Noise regulations on wind turbines near urban areas can restrict the installation of new wind turbines and limit the potential energy production. The main noise from wind turbines is aerodynamic and arises from turbulence caused by the air flow around the wind turbine blades in motion. The aerodynamic noise increases with the size of the wind turbine and with a demand for more productive, and thus larger wind turbines, there is an increased interest in quantifying the aerodynamic noise from wind turbine blades. In the design stage, wind turbine blades undergo a wealth of tests and simulations to ensure that the specific design is efficient. A wind tunnel is used to test the aerodynamic properties by applying a controlled airflow to a section of the blade. The wind tunnel can also be used to localize and quantify the noise sources with a microphone array, however, this requires specially designed wind tunnels.

In typical experimental setups, a wind turbine wing section, i.e., airfoil, is placed in such a wind tunnel and a controlled air flow is applied. The aerodynamic noise generated on the airfoil can be measured with a microphone array.

## References
[Hoeltgen, L., Breuß, M., Herold, G., & Sarradj, E. (2016). Sparse l1 Regularisation of Matrix Valued Models for Acoustic Source Characterisation. arXiv preprint arXiv:1607.00171.](https://arxiv.org/abs/1607.00171)

[Herold, G., Sarradj, E., & Geyer, T. (2013). Covariance Matrix Fitting for Aeroacoustic Application.](http://www-docs.tu-cottbus.de/aeroakustik/public/veroeffentlichungen/herold_cmf_daga2013.pdf)

[Yardibi, T., Li, J., Stoica, P., & Cattafesta, L. N. (2008). Sparsity constrained deconvolution approaches for acoustic source mapping. Journal of the Acoustical Society of America, 123(5), 2631–2642.](http://dx.doi.org.proxy.findit.dtu.dk/10.1121/1.2896754)

```julia

```


# Guide to Linear Regression in Python

This notebook is designed to guide you in learning how to do a basic linear regression using Python. The notebook will be divided into two parts. In the first, we will create data with some known function. Then, we will add some random fluctuations to the data and see if we can recover the function. In the second part, we will use a public dataset to test this schema and perform the linear regression.

This notebook will use a variety of packages.

* [Numpy](https://numpy.org/doc/stable/reference/index.html) for array handling and basic numerical functions.
* [SciKit-Learn](https://scikit-learn.org/stable/modules/classes.html) to perform the [linear regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression) and provide [some data](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets)
* [Matplotlib](https://matplotlib.org/stable/contents.html) for plotting and its simplified plotting module [Pyplot](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.html?highlight=pyplot#module-matplotlib.pyplot)

## Imports

We will begin by *importing* the necessary packages below.

```python
# Import Numpy
import numpy as np   # For arrays and basic numerical computation

# Import Packages from SciKit-Learn (sklearn)
# sklearn is such a large project that objects must be imported directly
from sklearn import datasets
from sklearn.linear_model import LinearRegression

# Import Matplotlib
from matplotlib import pyplot as plt
```

## Part 1 - Known Functions

### Example - Velocity as a Function of Position

Suppose we have a particle traveling along the $x$ axis and get the following data.

| x (m) | v (m/s) |
| ----- | ------- |
|  0.00 |    3.04 |
|  1.00 |    4.96 |
|  2.00 |    7.19 |
|  3.00 |    9.03 |
|  4.00 |   10.84 |
|  5.00 |   13.11 |
|  6.00 |   15.39 |
|  7.00 |   17.28 |
|  8.00 |   18.79 |
|  9.00 |   20.62 |
| 10.00 |   22.81 |

As it turns out, this data was simply generated by the equation

\begin{equation}
    v(x) = 2 x + 3
\end{equation}

with some additional random, gaussian noise. Therefore, when we do linear regression on this data, we *should* recover a slope of 2 and an offset of 3.

Normally, your data might be entered on a spreadsheet which could be ported with either [pandas](https://pandas.pydata.org/) or Google's gsheet plugin, but we will enter the data manually below.

SciKit-Learn requires that data be entered as 2D arrays where the "observations" appear on each row and the "features" on each column. Therefore, we will cast our data into just such an array.

```python
# Store the Positions
# We will store this a 2D row vector first the transpose it to a column vector
# A 1D array is created with one set of brackets whereas a 2D row vector is
#   stored with two brackets
# Note the `T` operator at the end of the parentheses. This is what is performing
#   the transposition
# Although this is not incredibly relevant now, python/numpy assumes numerical data
#   without decimals is integer data. Therefore, my dtype statement is telling
#   numpy that the data should be treated as a float (number with decimals)
X = np.array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]], dtype='float64').T

# Store the Velocities
# This procedure matches the statement above
# Note that statements in parentheses can be broken over multiple lines
# This is called implicit line breaking
Y = np.array([[
    3.04, 4.96, 7.19, 9.03, 10.84, 13.11, 15.39, 17.28, 18.79, 20.62, 22.81
]]).T
```

Now that we have stored our data in the variables `X` and `Y`, we can plot the scatter data before the regression to get a sense for what the data look like. Since this is a simple plot, we will use the pyplot [scatter](scatter) function. I'll also plot the function from which the data were generated with a simple [line plot](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html#matplotlib.pyplot.plot).

```python
# Plot the Sample Data as a scatter plot
# The scatter function takes the independent variable first and the dependent second
# The label keyword is for displaying the legend later
_ = plt.scatter(X, Y, label='Sample Data')

# Plot the Function
# I'll generate another set of variables to make this plot
xLine = np.arange(-2., 12)  # Generates data from -2 to 11 with stepsize 1
yLine = 2*xLine + 3         # Create y from the x data and known function
_ = plt.plot(xLine, yLine, ':k', label='v(x)')

# Set the Limits for the axis
plt.xlim(-2, 11)
plt.ylim( 0, 25)

# Label Everything!
plt.xlabel('Position (m)')
plt.ylabel('Velocity (m/s)')
plt.title("Measuring a Particle's Velocity")

# Add the grid if wanted
# Uncomment the next line if you want the plot to be gridded
# plt.grid()

# Create the Legend
_ = plt.legend()
```

Now that we are comfortable with plotting, let's move on the the linear regression.

```python
# Create/Fit the Model
# sklearn is built around a framework of fitting data to models. Here, we are trying
#   to fit our data to a linear model with linear regression.
# Below, we are creating a linear regression model, fitting the model to our data
#   then we are storing that information in the variable named model
model = LinearRegression().fit(X, Y)

# Print out the fit
# Note that the slope parameter is stored as the model.coef_ and the
#   intercept is stored as model.intercept_
# We store the equation in a string for future use
eqStr = 'v(x) = {:.2f} x + {:.2f}'.format(model.coef_.item(), model.intercept_.item())
print('Model Fit')
print('v(x) =   m  x +   b')
print(eqStr)
```

    Model Fit
    v(x) =   m  x +   b
    v(x) = 1.98 x + 3.13

As can be seen above, we were able to approximate the generating function based on the given data. Although the "random" fluctuations created a slight error in our coeficient and intercept, we got close to the predicted value.

Now, let's recreate the plot above with our regression too.

```python
# Plot the Sample Data as a scatter plot
_ = plt.scatter(X, Y, label='Sample Data')

# Plot the Regression Model
# For this, we need an xLine that is a 2D column vector like above
# We will calculate the y values by predicting with our model
xLine = np.arange(-2., 12)[:, np.newaxis]  # Generates data from -2 to 11 with stepsize 1
yLine = model.predict(xLine)               # Create y from the x data and known function
_ = plt.plot(xLine, yLine, 'C1', label='Trendline')

# Add the equation string we printed above on the graph
_ = plt.text(-1, 15, eqStr)

# Plot the Function
yLine = 2*xLine + 3         # Create y from the x data and known function
_ = plt.plot(xLine, yLine, ':k', label='v(x)')

# Set the Limits for the axis
plt.xlim(-2, 11)
plt.ylim( 0, 25)

# Label Everything!
plt.xlabel('Position (m)')
plt.ylabel('Velocity (m/s)')
plt.title("Measuring a Particle's Velocity")

# Add the grid if wanted
# Uncomment the next line if you want the plot to be gridded
# plt.grid()

# Create the Legend
_ = plt.legend()
```

As you can see, the model fits the data well and almost perfectly recovers the generating function.

### Your Turn

Now it's your turn, input the following data into the variables below and let my code from above take care of the rest.

| x (m) | v (m/s) |
| ----- | ------- |
|  0.00 |    2.09 |
|  2.00 |    7.91 |
|  4.00 |   14.45 |
|  6.00 |   20.07 |
|  8.00 |   25.63 |
| 10.00 |   32.25 |
| 12.00 |   38.91 |
| 14.00 |   44.66 |
| 16.00 |   49.51 |
| 18.00 |   55.11 |
| 20.00 |   61.56 |

```python
# Store the Positions
X = np.array([[]], dtype='float64').T

# Store the Velocities
Y = np.array([[

]]).T

# Create/Fit the Model
model = LinearRegression().fit(X, Y)

# Print out the fit
eqStr = 'v(x) = {:.2f} x + {:.2f}'.format(model.coef_.item(), model.intercept_.item())
print('Model Fit')
print('v(x) =   m  x +   b')
print(eqStr)

# Plot the Sample Data as a scatter plot
_ = plt.scatter(X, Y, label='Sample Data')

# Plot the Regression Model
xLine = np.arange(-2., 22)[:, np.newaxis]  # Generates data from -2 to 11 with stepsize 1
yLine = model.predict(xLine)               # Create y from the x data and known function
_ = plt.plot(xLine, yLine, 'C1', label='Trendline')

# Add the equation string we printed above on the graph
_ = plt.text(-1, 40, eqStr)

# Plot the Function
yLine = 3*xLine + 2         # Create y from the x data and known function
_ = plt.plot(xLine, yLine, ':k', label='v(x)')

# Set the Limits for the axis
plt.xlim(-2, 21)
plt.ylim( 0, 65)

# Label Everything!
plt.xlabel('Position (m)')
plt.ylabel('Velocity (m/s)')
plt.title("Measuring a Particle's Velocity")

# Add the grid if wanted
# Uncomment the next line if you want the plot to be gridded
# plt.grid()

# Create the Legend
_ = plt.legend()
```


## Homework

### Ice Cream Sales in Inferenceville

You have been hired to investigate a disturbing connection between ice cream sales and crime in Inferenceville. You are given a report that describes the joint distribution over random variable $S$, representing ice cream sales, and random variable $C$, representing crime. Each variable takes on a value of “low" or “high", which we'll represent with $0$ and $1$ respectively. The joint distribution (estimated from data) is as follows:

(a) Are random variables $S$ and $C$ independent?

[$\times    $] Yes

[$\checkmark$] No

```python
def is_independent(p_X_Y):
    """
    Returns true if the given join prbability distribution is independet.

    >>> import numpy as np
    >>> p_X_Y = np.array([[0.4, 0.1], [0.25, 0.25]])
    >>> is_independent(p_X_Y)
    False

    >>> p_X_Y = np.array([[0.72, 0.08], [0.18, 0.02]])
    >>> is_independent(p_X_Y)
    True
    """

    import numpy as np
    p_X = p_X_Y.sum(axis=1)
    p_Y = p_X_Y.sum(axis=0)
    Δ = np.outer(p_X, p_Y) - p_X_Y
    return (np.linalg.norm(Δ, np.inf) - 0.00001) < 0

if __name__ == "__main__":
    import doctest
    doctest.testmod()
```

```python
from sympy import init_printing
init_printing()
```

```python
import numpy as np
p_S_C = np.array([[0.4, 0.1], [0.25, 0.25]])
is_independent(p_S_C)
```

    False

(b) After further investigation, you discover information about the temperature, represented by $T$. This random variable also takes on values $0$ or $1$ corresponding again to “low" and “high". You are able to obtain the conditional distribution $p_{S,C \mid T}(s,c \mid t)$, shown below.

Are random variables $S$ and $C$ conditionally independent given $T$?

[$\checkmark$] Yes

[$\times    $] No

```python
import numpy as np
p_T_1 = np.array([[0.72, 0.08], [0.18, 0.02]])
p_T_2 = np.array([[0.08, 0.12], [0.32, 0.48]])
is_independent(p_T_1) and is_independent(p_T_2)
```

    True

(c) Determine the distribution $p_T$ from the tables above. Express your answer as a Python dictionary. The keys should be the Python integers $0$ and $1$.

Assuming $p_T(0) = p$ then $p_T(1) = (1-p)$. Now by the law of total probability,

$$\begin{align}
p_{S,C}(0,0) &= p_{S,C\mid T}(0,0 | 0)p_T(0) + p_{S,C\mid T}(0,0 | 1)p_T(1) \\
         0.4 &= 0.72p + 0.08(1-p)\\
\end{align}$$

```python
from sympy.solvers import solve
from sympy import Symbol
p = Symbol('p')
p = solve(0.08*(1-p) + 0.72*p - 0.4, p)[0]; p
```

```python
p_T = {0: p, 1: 1-p}; p_T
```

```python

```


# Ridge Regression with Gradient Descent

### Author: Juan Solorio

-----

# Overview
In this exercise, I will implement a first version of my own gradient descent algorithm to
solve the ridge regression problem. I will keep improving and extending this gradient descent optimization algorithm. In this notebook, I will implement a basic version of the algorithm.

## Objectives
- Mathematically define _Objective Function_ for Ridge Regression ($F\beta$)
    - Compute gradiant $\nabla F$
- Create functions for algorithm:
    - Objective function
    - Gradient function
    - Gradient Descent funtion
- Observe (plot) the convergence in terms of the function values and the gradients
    and tune for the optimal hyperparameters for step-size ($\eta$) and normalization ($\lambda$)
- Compare to _sklearn_

## Environment Setup - *Importing Libraries*

```python
# needed libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn.linear_model import Ridge
import time

%matplotlib inline
plt.style.use('ggplot')
```

# Background and Theory

We start by defining the ___Linear Regression___ as a supervised learning algorithm and we write the mathematical algorithm as:
$$
y = b_0*x + b_1
$$
> y - target value
$b_0$ - slope
x - predictor variable
$b_1$ - intercept

where $m$ and $b$ are the variables the algorithm will try to predict from the data.

We can generalize this equation for multiple variables and write our _cost function_ to optimize our algorithm as:

$$
F(\beta) = \frac{1}n\sum_{i=1}^n(y_i - h(x_i))^2 \\
\\
\\
h(x_i) = x_1\beta_1 + x_2\beta_2 + ... + x_{i,j}\beta_j
$$

Where $h(x_i)$ is the predicted value for the function, $\beta$ are the weights for the individual variables, and $y_i$ is the target value of the data.

We can finalize this by writing the equation in the expanded form:
$$
F(\beta) = \frac{1}n\sum_{i=1}^n(y_i - \sum_{j=1}^dx_{ij}\beta_j)^2
$$

## Lasso Regression

Linear Regression treats all the features equally and finds unbiased weights to minimizes the cost function.
In _Ridge Regression_ , there is an addition of l2 penalty ( square of the magnitude of weights ) in the cost function of Linear Regression:

$$
\lambda\|\beta\|^2_2

$$

Where $\lambda$ is a hyperparameter to be tuned and this is done so that the model does not overfit the data.

We can write the final objective equation as:

$$
F(\beta) = \frac{1}n\sum_{i=1}^n(y_i - \sum_{j=1}^dx_{ij}\beta_j)^2 + \lambda\sum_{j=1}^d\beta_j^2
$$

## Gradient Descent
Gradient descent is an optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent as defined by the negative of the gradient. In machine learning, we use gradient descent to update the parameters or weights ($\beta$) of our model.

In order to  minimize with respect to $\beta$, we want to take the derivative of $F(\beta)$ and set it equal to zero.

The derivative for $n=1$ and $d=1$ yields:
\begin{equation}
\frac{dF}{d\beta} = \frac{2}n(y - x\beta) + 2\lambda\beta
\end{equation}

and for $n>1$ and $d>1$

$$
\frac{\partial F}{\partial \beta_j} = \frac{\partial}{\partial \beta_j} (\frac{1}n\sum_{i=1}^n(y_i - x^T_{i}\beta_j)^2 + \lambda\|\beta\|^2_2)
$$
$$
= \frac{1}n(\sum_{i=1}^n \frac{\partial}{\partial \beta_j} (y_i - x^T_{i}\beta_j)^2 + \frac{\partial}{\partial \beta_j} \lambda\|\beta\|^2_2)
$$
$$
= -\frac{2}n(\sum_{i=1}^n x_{ij} (y_i - x^T_{i}\beta_j)) + 2\lambda\sum_{j=1}^d\beta_j
$$

In Matrix terms, $F(\beta)$ can be interpreted as:

$$
F(\beta) = \frac{1}{n}<Y - X\beta,Y-X\beta> + \lambda\|\beta\|^2_2
$$

Taking the derivative of the matrix form of $F(\beta)$ leads us to the following:

$$
\nabla F = \frac{\partial}{\partial\beta} F= - \frac{2}{n}X^T(y + X\beta) + 2\lambda\beta
$$

Which is also known as the _gradient_ of the _objective function_ $F$.

# Algorithm Functions Definitions

* Objective function for Ridge Regression $F(\beta)$:

>$$
F(\beta) = \frac{1}n\sum_{i=1}^n(y_i - X^T_{ij} \cdot \beta_j)^2 + \lambda\|\beta_j\|^2_2
$$

```python
def obj_fx(beta,lamda,X,y):
    """
    Linear regression with Ridge penalty L1:
    F(b) = 1/n sum((y - x dotproduct b)^2) + lamda*norm(b)^2

    Parameters
    ----------
    beta : arr
        array of values for weights
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data

    Returns
    -------
    int
        computation of the objective function

    """

    # dot product can be accomplish by 'numpy_arrayA @ numpy_arrayB' or 'numpy_arrayA.dot(numpy_arrayB)'
    return 1/len(y) * sum((y - X @ beta)**2) + lamda*np.linalg.norm(beta)**2
```

* Gradient of objective funtion - $\nabla F$:
>$$
\nabla F = \frac{\partial}{\partial\beta} F= - \frac{2}{n}X^T(y + X\beta) + 2\lambda\beta
$$

```python
def gradient_fx(beta,lamda,X,y):
    """
    Computes gradient of the Linear regression with Ridge penalty L1 function:
    grad F(b) = -2/n (x.T dotproduct (y - x dotproduct b)) + 2*lamda*b

    Parameters
    ----------
    beta : arr
        array of values for weights
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data

    Returns
    -------
    int
        computation of the gradient of Ridge regression objective function

    """
    return (-2/len(y)) * X.T @ (y - X @ beta) + 2*lamda*beta
```

* Gradient Descent Algorith:
>`Gradient Descent algorithm with fixed constant step-size
__input__  step-size $\eta$
__initialization__ $\beta_0 = 0$
__repeat for__ t = 0, 1, 2, . . .
    $\beta_{t+1} = \beta_t − \eta \nabla F(\beta_t)$
__until__ the stopping criterion $\|\nabla F\| \leq \epsilon$ is satisfied.

```python
def gradient_descent(beta_init, eta ,lamda,X,y,epsilon=0.005):
    """
    Computes gradient descent with a fixed step size eta and stopping condition norm gradient F < epislon

    Parameters
    ----------
    beta_init : arr
        array of values for weights as starting point
    eta : int
        interger value for step size parameter
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data
    epsilon : int
        interger value for stopping parameter condition, defaul = 0.005

    Returns
    -------
     beta_vals: Matrix
         Estimated betas at each iteration, with the most recent values in the last row
    """
    # setting initial value of betas
    beta = beta_init
    # gradient calculation for starting values
    gradient = gradient_fx(beta,lamda,X,y)
    # list to save beta values
    beta_vals = [beta_init]
    # loop for stopping criterion epsilon
    while np.linalg.norm(gradient) > epsilon:
        # updating values for beta and gradient
        beta = beta - eta*gradient
        gradient = gradient_fx(beta,lamda,X,y)

        # appending values
        beta_vals.append(beta)
    return np.array(beta_vals)
```

# Implementation
For our data, we'll use the ***Hitters*** dataset from `'https://raw.githubusercontent.com/selva86/datasets/master/Hitters.csv'`, and we'll drop any rows with `NA` values.

```python
# Load the data
hitters = pd.read_csv('https://raw.githubusercontent.com/selva86/datasets/master/Hitters.csv',
                      sep=',', header=0)
hitters = hitters.dropna()
hitters.head()
```

      AtBat
      Hits
      HmRun
      Runs
      RBI
      Walks
      Years
      CAtBat
      CHits
      CHmRun
      CRuns
      CRBI
      CWalks
      League
      Division
      PutOuts
      Assists
      Errors
      Salary
      NewLeague

      1
      315
      81
      7
      24
      38
      39
      14
      3449
      835
      69
      321
      414
      375
      N
      W
      632
      43
      10
      475.0
      N

      2
      479
      130
      18
      66
      72
      76
      3
      1624
      457
      63
      224
      266
      263
      A
      W
      880
      82
      14
      480.0
      A

      3
      496
      141
      20
      65
      78
      37
      11
      5628
      1575
      225
      828
      838
      354
      N
      E
      200
      11
      3
      500.0
      N

      4
      321
      87
      10
      39
      42
      30
      2
      396
      101
      12
      48
      46
      33
      N
      E
      805
      40
      4
      91.5
      N

      5
      594
      169
      4
      74
      51
      35
      11
      4408
      1133
      19
      501
      336
      194
      A
      W
      282
      421
      25
      750.0
      A

We are going to attempt to use our Ridge Regression algorithm to try and predict the _Salary_ of a player based on all the other features in the dataset. We want to create our _Features_ (X) and _Target_ (y) datasets.

```python
# Creating matrix X with the predictors and
# and vector y with the response
X = hitters.drop('Salary', axis=1)
# need to transform our labels for 'League','Division','NewLeague' to bool or {1,0} values
X = pd.get_dummies(X, drop_first=True)
y = hitters.Salary

X.head(3)
```

      AtBat
      Hits
      HmRun
      Runs
      RBI
      Walks
      Years
      CAtBat
      CHits
      CHmRun
      CRuns
      CRBI
      CWalks
      PutOuts
      Assists
      Errors
      League_N
      Division_W
      NewLeague_N

      1
      315
      81
      7
      24
      38
      39
      14
      3449
      835
      69
      321
      414
      375
      632
      43
      10
      1
      1
      1

      2
      479
      130
      18
      66
      72
      76
      3
      1624
      457
      63
      224
      266
      263
      880
      82
      14
      0
      1
      0

      3
      496
      141
      20
      65
      78
      37
      11
      5628
      1575
      225
      828
      838
      354
      200
      11
      3
      1
      0
      1

We now need to split our data into _train_ and _test_ sets for both the features and target variables. Given the values in our _Features_ vary in size or magnitude, we need to standardize our features by using the sklearn function `preprocessing.StandardScaler()`.

```python
# Dividing the data into train and test sets.
# By default,  it is a 75-25 split between train-test
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# Standarizing the data
scaler = preprocessing.StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
scaler = preprocessing.StandardScaler().fit(y_train.values.reshape(-1, 1))
y_train = scaler.transform(y_train.values.reshape(-1, 1)).reshape((-1))
y_test = scaler.transform(y_test.values.reshape(-1, 1)).reshape((-1))
```

```python
def convergence_plots(x_vals, lambduh, X,y,title):
    """
    Plot the convergence in terms of the function values and the gradients
    Input:
    - x_vals: Values the gradient descent algorithm stepped to
    """
    n, d = x_vals.shape
    fs = np.zeros(n)
    grads = np.zeros((n, d))
    for i in range(n):
        fs[i] = obj_fx(x_vals[i], lambduh,X,y)
        grads[i, :] = gradient_fx(x_vals[i], lambduh,X,y)
    grad_norms = np.linalg.norm(grads, axis=1)

    fig_dims = (15, 3)
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=fig_dims)

    ax1.plot(fs)
    ax1.set_xlabel('Iteration')
    ax1.set_ylabel('Objective value')

    ax2.plot(grad_norms)
    ax2.set_xlabel('Iteration')
    ax2.set_ylabel('Norm of gradient')
    plt.suptitle(title,fontsize=16)
    plt.subplots_adjust(left=0.2, wspace=0.8, top=0.8)
    plt.show()

```

```python
eta = 0.05
lambdas = [-0.1, 0.005,0.001,0.01,0.05,1]
d = X_train.shape[1]
beta_init = np.zeros(d)
for lambduh in lambdas:
    # store starting time
    begin = time.time()
    betas = gradient_descent(beta_init=beta_init, eta=eta, lamda=lambduh,X=X_train,y=y_train)
    title=r'Function Value and Norm of Gradient Convergence $\lambda =$ {:.3f}'.format(lambduh)
    convergence_plots(betas, lambduh,X_train, y_train, title)

    time.sleep(1)
    # store end time
    end = time.time()

    # total time taken
    print(f"Total runtime of the program is {end - begin}")
```

We see that the _Objective value_ of the function diverges when the normalization parameter $\lambda$ is negative and that it converges to about _0.5_ when tha values of $\lambda=[0.001,0.05]$. The best run time occured for $\lambda=0.01$, so we'll use this as our _Optimal_ normalization parameter value.

# Comparing to _Scikit-Learn_

We can compute the ridge regression model from sklearn by using the `Ridge` function from **sklearn.linear_model** and compare how our model performed.

_Note that the scikit-learn [objective function](http://scikit-learn.org/stable/modules/linear_model.html#ridge-regression) is_ :

$$
F^*(\beta) = \min_{\beta \in \mathbb{R}^d} \sum_{i=1}^n(y_i - \sum_{j=1}^dx_{ij}\beta_j)^2 + \alpha\sum_{j=1}^d\beta_j^2
$$

when comparing sklearn's objective function $F^*(\beta)$ to our objective function $F(\beta)$, we get that:
> $\lambda = \frac{\alpha}{n} \Longrightarrow \alpha = n * \lambda$

```python
# number of samples n
n = len(y_train)
lambduh = 0.01
betas = gradient_descent(beta_init=beta_init, eta=eta, lamda=lambduh,X=X_train,y=y_train)
# setting alpha for sklearn ridge
alpha = n*lambduh
ridge = Ridge(alpha=alpha, fit_intercept=False) # initializing ridge
ridge.fit(X_train, y_train) # training ridge model
# printing beta values
print("Sklearn Final weights (betas): ", ridge.coef_)
print("Our Final weights (betas): ", betas[-1])

```

    Sklearn Final weights (betas):  [-0.49683318  0.53848371  0.14026802 -0.09125083  0.01367849  0.28982753
     -0.10561524 -0.22109841  0.49775465 -0.05474063  0.53958547  0.06559602
     -0.37105594  0.14704651  0.09250802 -0.09098076  0.04637135 -0.12793725
     -0.01498699]
    Our Final weights (betas):  [-0.48507163  0.51545023  0.1336241  -0.07385865  0.01975817  0.28526168
     -0.11787016 -0.12322043  0.4547416  -0.05871632  0.49069089  0.06869243
     -0.36495034  0.14618957  0.08881195 -0.09213196  0.04644374 -0.12970797
     -0.0157561 ]

```python
print("Objective value from Sklearn F*(beta): ", obj_fx(ridge.coef_, lambduh,X_train,y_train))
print("Objective value from our F(beta): ", obj_fx(betas[-1], lambduh,X_train,y_train))

```

    Objective value from Sklearn F*(beta):  0.5261102696078659
    Objective value from our F(beta):  0.5263969771608789

The values for our final _feature weights_ ($\beta$ 's) and _objective function_ from our algorithms match pretty well with those produced by _Sklearn_ functions.

We can now predict values for our _tests_ datasets and see how well our model's predictions compare to the expected values. Sklearn has the built in function `.predict` for all it's supervised learning models, we can recreate the function by computing the _dot product_ of the features and the weights.

```python
def predict_values(betas, X):
    """
    Function calculates the predicted target values for dot product for given betas and features (X)

    Parameters
    ----------
    beta_init : arr
        array of values for weights
    X : arr
        array of features from data

    Returns
    -------
     Calculated predicted values for target variable, from dot product of features and weights
    """
    return X.dot(betas)
```

```python
# getting a dataframe of the predicted values for both my own algorithm and sklearn
# for the test dataset
predicted_data = pd.DataFrame(np.array([ridge.predict(X_test), predict_values(betas[-1],X_test), y_test]).T,
                              columns=['Sklearn Ridge','Own Ridge','Expected Values']).melt(id_vars=['Expected Values']).rename(
    columns={'variable':'ML algorithm','value':'Predicted Values'})
predicted_data.head()
```

      Expected Values
      ML algorithm
      Predicted Values

      0
      0.289100
      Sklearn Ridge
      0.030257

      1
      -0.079887
      Sklearn Ridge
      1.323443

      2
      -0.811811
      Sklearn Ridge
      -1.026751

      3
      1.402108
      Sklearn Ridge
      0.250032

      4
      -0.769468
      Sklearn Ridge
      0.114917

```python
# plot comparing both how well do my algorithm compares to sklean's predictions
# and comparing how well the predictions are compared to the actual expected values (diagonal scatter)
sns.color_palette("colorblind")
g = sns.FacetGrid(predicted_data,hue='ML algorithm', height = 7,aspect=1)
g = g.map(sns.scatterplot, "Expected Values", "Predicted Values", edgecolor="w",s=100,alpha=.8)
plt.plot(predicted_data['Expected Values'],predicted_data['Expected Values'],'k.',linewidth=1.5,
         alpha=.4,label='If Perfect Prediction')
plt.legend()
plt.show()
```

We can again see that the values from our model compared very well to those from the sklearn model, and both models aline with the expected values' overall trend, but do appear to have greater error or diverge from the expected values the higher the "Salary".

Let's finally just check the distribution of the percent difference for the predicted from expected values, we'll first rescale the predicted values back to the original sample values.

```python
# scaling back to the actual salary values
salary_sklearn = np.round(scaler.inverse_transform(ridge.predict(X_test)))
salary_own = np.round(scaler.inverse_transform(predict_values(betas[-1],X_test)))
salary = scaler.inverse_transform(y_test)
```

```python
# checking the error distribution
print("Percent of values with less than 50%% error: %2d %%"%(100*np.mean(abs(salary-salary_own)/salary <= 0.5)))
sns.histplot(((salary-salary_own)/salary))
```

We can see that about 57% of our values have less than or equal to "50%" difference from the expected values, and from the scatter plot again we can see how at smaller salaries we have some higher predicted salaries and we tend to under predict values as the salaries increase.

1-Python/1 - Ridge Regression w Gradient Descent.ipynb


# Horner's Method

which also call as QinJiuSao's Method in China, because QinJiuSao pubulished the method in 1247 while Horner pubilshed in 1819.

Horner's Method is an algorithm for calculating polynomials in a computational efficient way.

$$p(x) = \sum_{i=0}^{n}a_i * x^{n-i}$$

Intuitively, it seams we need $\frac{n(n+1)}{2}$ times Multiplies and $n$ times Additons. But things chanegd if it is transformed to the follow expression.
$$p(x) = (...(a_0x+a_1)x + a_2)...+a_{n-1})x+a_n$$

Obviously, it is just a recursion defination of the previous polynomial which can be present as below.

$$
\begin{equation}
\left\{
             \begin{array}{lr}
             b_0=a_0, &  \\
             b_i = b_{i-1} * x^* + a_i, & i = 1,2,...,n
             \end{array}
\right.
\end{equation}
$$

Evidently, it just takes $n$ times Multiplies and $n$ times Additions.

# 秦九韶算法

用于计算多项式，将原来的需要$n^2$的运算复杂度降低到了$n$。算法的迭代表达式如上公式。


# Optimizing Stiffness Constants

Suppose we have $n$ blocks, each of width $w$, at positions $x\in\mathbb{R}^n$.
The blocks are positioned between two walls at $0$ and $l$.
The leftmost block is connected to the left wall via a spring with stiffness coefficient $k_1$,
the rightmost block is connected to the right wall via a spring with stiffness coefficient $k_{n+1}$,
and the $i$th block is connected to the $(i+1)$th block via a spring with stiffness coefficient $k_{i+1}$.
The equilibrium position of all the blocks can be found by solving the optimization problem
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \frac{1}{2}k_1x_1^2 + \frac{1}{2}k_2(x_2-x_1)^2 + \ldots + \frac{1}{2}k_{n+1}(l-x_n)^2\\[.2cm]
\mbox{subject to} & x_1 \geq w/2, \quad x_n \leq l - w/2, \\
& x_i - x_{i-1} \geq w, \quad i=2,\ldots,n-1,
\end{array}
\label{eq:prob}
\end{equation}
with variable $x$ and solution denoted $x^\star$.
The objective is the potential energy of the system, and the constraints express
the fact that the blocks have a width, and cannot penetrate each other or the walls.

```python
# NOTE: this notebook requires ffmpeg, which can be installed on ubuntu with "sudo apt install ffmpeg"
# and on mac with "brew install ffmpeg"
import cvxpy as cp
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from cvxpylayers.tensorflow import CvxpyLayer
import matplotlib.patches as patches
from matplotlib import animation, rc
rc('animation', html='html5')
```

We can easily set up this problem as a CVXPY problem, with parameter $k$ and variable $x$:

```python
n = 5
l = 1
w = .05

k = cp.Parameter(n + 1, nonneg=True)
x = cp.Variable(n + 2)
objective = cp.sum(cp.multiply(k, .5 * cp.square(cp.diff(x))))
constraints = [x[0] == 0, x[-1] == l] + [x[i] - x[i-1] >= w for i in range(1, n+2)]
prob = cp.Problem(cp.Minimize(objective), constraints)
```

We can easily convert this problem to a `tensorflow` `CvxpyLayer` in one line.
The layer maps stiffness coefficients to block positions, or $x^\star(k)$.

```python
prob_tf = CvxpyLayer(prob, [k], [x])
```

Our goal is to tune the stiffness coefficients such that the block positions are close to target positions,
or solve the problem
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \|x^\star(k) - x^\mathrm{targ}\|_2^2,
\end{array}
\end{equation}
with variable $k\in\mathbb{R}^{n+1}$.

```python
tf.random.set_seed(1)
k = tf.constant(tf.ones([n + 1], dtype=tf.float64))
x_targ = tf.sort(tf.random.uniform([n], dtype=tf.float64))
x_targ
```

    <tf.Tensor: id=21, shape=(5,), dtype=float64, numpy=array([0.04779355, 0.28081525, 0.32104743, 0.33551271, 0.80628428])>

We can (approximately) solve this problem via gradient descent:

```python
x, = prob_tf(k)
x_np = x.numpy()

fig, ax = plt.subplots()
ax.set_xlim((0, 1))
ax.set_ylim((0, w*4))
rects = []
for i in range(n):
    plt.axvline(x_targ[i].numpy(), c='k')
    rect = patches.Rectangle((x_np[i + 1] - w/2, 0), w, w, linewidth=1, edgecolor='k', facecolor='none')
    ax.add_patch(rect)
    rects.append(rect)

def animate(i):
    global k
    with tf.GradientTape() as tape:
        tape.watch(k)
        x, = prob_tf(k)
        loss = tf.reduce_sum((x[1:-1] - x_targ)**2)
    grad, = tape.gradient(loss, [k])
    k = k - .1 * grad
    k = tf.maximum(k, 0.0)
    for i in range(n):
        rects[i].xy = (x.numpy()[i + 1] - w/2, 0)
    return rects

anim = animation.FuncAnimation(fig, animate,
                               frames=300, interval=50, blit=True)
anim
```


# Lecture 5

This lecture covered linear difference equations (recurrence equations). Below one of the examples from the lecture is solved using SymPy.

## Student grant example

The difference equation for the student grant example from the lecture notes has the form

$$
y_{n} - 2.95 y_{n-1} + 2 y_{n-2} = −(63.685)(1.07^{n})
$$

We investigate and solve this equation using SymPy. As before, we first need to import SymPy. We will also import a `Fraction` object to represent fractions without Python evaluating fraction numerically.

```
from sympy import *
init_printing()
from IPython.display import display

from fractions import Fraction
```

We now define $n$ as an integer symbol and $y$ as a function:

```
n = Symbol("n", integer=True)
y = Function("y")
```

Now, we define the RHS of the difference equation:

```
f = y(n) - Fraction(295, 100)*y(n - 1) + 2*y(n - 2)
display(f)
```

We'll now solve the homogeneous version of the equation using `rsolve`. To compare to the solution in the lecture notes, we'll also evaluate the the solution in floating point:

```
eqn = Eq(f, 0)
soln = rsolve(eqn, y(n))
display(soln)
soln.evalf()
```

The solution is the same as in the lecture notes.

We now consider the non-homogeneous case:

```
# Create non-homogeneous equation
eqn = Eq(f, -63.685*(1.07**n))
display(eqn)
```

We now solve the non-homogeneous equation, on this occasion providing the initial conditions $y(0) = 2000$ and $y(1) = 2200$:

```
soln = rsolve(eqn, y(n), init={y(0) : 2000, y(1) : 2200})
display(soln)
soln.evalf()
```

```
# Plot
%matplotlib inline
plot(soln, (n, 0, 12), xlabel="year (n)", ylabel="fee")

# Evaluate fee at 10 years
print("Fee after 10 years (n=10): {}".format(soln.subs(n, 10).evalf()))
```


# Conditioning of evaluating tan()

```python
import numpy as np
import matplotlib.pyplot as pt
```

Let us estimate the sensitivity of evaluating the $\tan$ function:

```python
x = np.linspace(-5, 5, 1000)
pt.ylim([-10, 10])
pt.plot(x, np.tan(x))
```

```python
x = np.pi/2 - 0.0001
#x = 0.1
x
```

```python
np.tan(x)
```

```python
dx = 0.00005
np.tan(x+dx)
```

## Condition number estimates

### From evaluation data

```python

```

### Using the derivative estimate

```python
import sympy as sp

xsym = sp.Symbol("x")

f = sp.tan(xsym)
df = f.diff(xsym)
df
```

Evaluate the derivative estimate. Use `.subs(xsym, x)` to substitute in the value of `x`.

```python

```

```python

```

cleared-demos/error_and_fp/Conditioning of Evaluating tan.ipynb


The model used to solve a generic instance of a 9 by 9 Sudoku is the following:
$\begin{align}
    & \max z = \sum_{i \in I}\sum_{j \in J}\sum_{n \in N} x_{ijn}
\end{align}$
$\text{subject to:}\\$
$\begin{align}
    & \quad \sum_{j=1}^{9} x_{ijn} = 1, \forall i, \forall n   \\
    & \quad \sum_{i=1}^{9} x_{ijn} = 1, \forall j, \forall n   \\
    & \quad \sum_{i=3r - 2}^{3r}\sum_{j=3c - 2}^{3c} x_{ijn} = 1, \forall n, \forall r, \forall c   \\
    & \quad \sum_{n=1}^{9} x_{ijn} = 1, \forall i, \forall j   \\
\end{align}
$
where $i, j$ and $n$ vary in $\{1, \dots ,9\}$, whereas $r$ and $c$ vary in $\{1, \dots ,3\}$.
The binary variables $x_{ijn}$ are equal to $1$ if cell $(i, j)$ takes value $n$, and $0$ otherwise.
Constraints $(2)$ impose that each value $n$ appears only once in a row; constraints $(3)$ impose that each value $n$ appears only once in a column; constraints $(4)$ impose that each value $n$ appears only once in a $3$ by $3$ sub-square; finally, constraints $(5)$ impose that no values assignments overlap.
Notice that the objective function in $(1)$ could also have been set constant. This formulation has been chosen in order not to cause Pyomo to emit warnings.


## Recap Matrix Multiplication
Produces a matrix from two matrices

The number of columns in the first matrix (m) must be equal to the number of rows in the second matrix. The result matrix has the number of rows of the first (l) and the number of columns of the second matrix (n).

If $\mathbf{A}$ is an $m \times n$ matrix and $\mathbf{B}$ is an $n \times p$ matrix
$$
\mathbf{A}=\begin{pmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\
 a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
 a_{m1} & a_{m2} & \cdots & a_{mn} \\
\end{pmatrix},\quad\mathbf{B}=\begin{pmatrix}
 b_{11} & b_{12} & \cdots & b_{1p} \\
 b_{21} & b_{22} & \cdots & b_{2p} \\
\vdots & \vdots & \ddots & \vdots \\
 b_{n1} & b_{n2} & \cdots & b_{np} \\
\end{pmatrix}
$$

the matrix product $\mathbf{C} = \mathbf{AB}$ is defined to be the $m \times p$

$\mathbf{C}=\begin{pmatrix}
 c_{11} & c_{12} & \cdots & c_{1p} \\
 c_{21} & c_{22} & \cdots & c_{2p} \\
\vdots & \vdots & \ddots & \vdots \\
 c_{m1} & c_{m2} & \cdots & c_{mp} \\
\end{pmatrix}$

such that
$$c_{ij} = a_{i1}b_{1j} + a_{i2}b_{2j} +\cdots + a_{in}b_{nj}= \sum_{k=1}^n a_{ik}b_{kj} \;\;\; \text{for}\ i = 1, ..., m\ \text{and}\ j = 1, ..., p.$$

The corresponding operation on vectors is referred to as dot product or scalar product. If two vectors $\mathbf{\color{red}a} = \left[\color{red}a_\color{red}1, \color{red}a_\color{red}2, ..., \color{red}a_\color{red}n\right]$ and $\mathbf{\color{blue}b} = \left[\color{blue}b_\color{blue}1, \color{blue}b_\color{blue}2, ..., \color{blue}b_\color{blue}n\right]$

are identified as row matrices:

$$\begin{align}
\mathbf{\color{red}a} \cdot \mathbf{\color{blue}b} &= \mathbf{\color{red}a}\mathbf{\color{blue}b}^\top\\
&=\sum_{i=1}^n {\color{red}a}_i{\color{blue}b}_i\\
&={\color{red}a}_1{\color{blue}b}_1+{\color{red}a}_2{\color{blue}b}_2+\cdots+{\color{red}a}_n{\color{blue}b}_n \\
\end{align}$$

Source: https://en.wikipedia.org/wiki/Matrix_multiplication and https://en.wikipedia.org/wiki/Dot_product

```python

```


# Lab 3
## Introduction
In this lab we will analyse population dynamics under the logisitic model with managed harvesting.

First import the modules we need.

```python
from plotly.figure_factory import create_quiver
from plotly import graph_objs as go
from numpy import meshgrid, arange, sqrt, linspace
from scipy.integrate import odeint
```

## Harvesting of fish
A population of fish in a lake, left to its own devices, is modelled by the logistic differential equation
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y),
\end{align}
where the population $y$ is in units of thousands of fish and time $t$ is measured in years.

First define a function for $\mathrm{d}y/\mathrm{d}x$ in terms of $y$ and $x$.

```python
def diff_eq(y, x):
    return 4 * y * (1 - y)
```

Next define a function that creates a Plotly Figure object that contains a slope field and, optionally, a few solutions to initial value problems.

It automates a few things we did in the last lab.

- `diff_eq` is the differential equation to be plotted
- `x` and `y` should be outputs from `meshgrid`.
- `args` is any additional arguments to `diff_eq` (we will use that below).
- `initial_values` is a list (or array) of starting $y$ values from which approximate solutions will start. The corresponding $x$ value is the minimum element of `x`.

Note that the numerical solutions will plotted for the whole range of $x$ values in `x`, so if they blow up you will probably get a warning and less-than-useful plot.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
```

The slope field below should hopefully give you some idea for the fish population dynamics.

Note that we use `arange` rather than `linspace` this week so that we can carefully control the increments between our grid points. `arange(0, 1.1, 0.25)` returns an array that starts with 0 and increments by 0.25 until it exceeds 1.1.

The plot also contains the solution curves for
$y(0) = 1$ and $y(0) = 0.4$. Edit the cell to also include the solution curve for $y(0)=1.4$.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, initial_values=(0.4, 1, 1.4))
fig.show('png')
```

### Equilibrium solutions
Looking back to our differential equation, $\mathrm{d}y/\mathrm{d}t = 0$ when $y(t) = 0$ or $y(t) = 1$. Looking at the slope field, we see that the equilibrium solution $y(t) = 1$ is stable (this is the carrying capacity here, corresponding to 1000 fish), whereas the equilibrium solution $y(t) = 0$ is unstable. Any non-zero initial population will eventually stabilise at 1000 fish.

### What will happen if harvesting is now commenced at a steady rate?
For the simplest harvesting model, assume that $H$ units (thousands) of fish are taken
continuously (smoothly) over the year, rather than at one instant each year.
Note that the units of $H$ are the same as those of $\mathrm{d}y/\mathrm{d}t$, thousands of fish per year, so we simply subtract $H$ from the RHS of our existing equation to give the DE with harvesting as
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y) - H.
\end{align}
Again, the (constant) equilibrium solutions are found by setting $\mathrm{d}y/\mathrm{d}t = 0$, giving from the quadratic formula (check this),
\begin{align}
y(t) = \frac{4\pm\sqrt{16-16H}}{8} = \frac{1\pm\sqrt{1-H}}{2}.
\end{align}
What happens after harvesting starts will depend on the equilibrium solutions, their
stability and the initial number of fish $y(0)$.

Start by redefining `diff_eq` to include the `H` parameter. Note that defining `diff_eq` again overides our original definition.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
```

Now set $H = 0.6$ and plot the slope field. This is done by setting `args=(0.6,)` when we call `create_slope_field`. This is exactly how you would pass additional arguments like this one to `odeint` if you were calling it directly.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,))
fig.show('png')
```

From the solutions to the quadratic equation above, the equilibrium solutions of the DE are found to be $y(t) \approx 0.184$ and $y(t) \approx 0.816$. The previous equilibrium solution with no harvesting at $y(t) = 0$ has moved up to $y(t) \approx 0.184$, while the previous equilibrium solution with no harvesting at $y(t) = 1$ has moved down to $y(t) \approx 0.816$.

From the slope field, we see that the equilibrium solution $y(t) \approx 0.184$ is unstable, whereas the equilibrium solution $y(t) \approx 0.816$ is stable. If the population ever falls below about 0.184, or 184 fish, it will then drop to 0. This is a new feature, introduced by harvesting.

In the cell below, use `create_slope_field` to experiment by plotting the solutions to the initial value problems $y(0)=0.183$ and $y(0)=0.25$. Extend the $x$ range of your slope field until the top line is close to equlibrium. Note that if you extend it too far you will break `odeint` (why?). You may also like to increase the increments in `arange` to make the plot clearer.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.25))
fig.show('png')
```

## Exercises

In this lab you will experiment with the population dynamics given by the logistic equation with harvesting that we started analysing in the lab.

This week the questions will be a combination of plots and written answers.

1. Assume that the harvest is 600 fish per year. **On the same figure,**
  a. plot the slope field,
  b. plot the equilibrium solutions that we found in above, and
  c. plot the solution curves for $y(0)=1$, $y(0)=0.3$, and $y(0)=0.15$.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.816, 1, 0.3, 0.15))
fig.show('png')
```

1.  d. In the cell below, describe the behaviour of the fish population for each of these five initial numbers of fish.

At the equilibrium points the fish population will stagnate and remain constant. At the starting point of 1 it will decline towards 0.816 and at 0.3 it will increase towards it. For 0.15 it will continue to decline until there are no fish left.

2. a. i. Assume that $H=0.8$. Plot the slope field and five solutions, one for each equilibrium solution and one for each region between, above, or below them. You can use the equation from the lab to calculate the equilibrium solutions.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.8,), initial_values=(0.276, 0.724, 1, 0.4, 0.2))
fig.show('png')
```

2. a. ii. In the cell below, describe the limiting behaviour of each line.

At the equilibrium points the fish population will stagnate and remain constant. At the starting point of 1 it will decline towards 0.724 and at 0.1 it will increase towards it. For 0.2 it will continue to decline until there are no fish left.

2. b. i. Assume that $H=1$. Plot the slope field and three solutions for the equilibrium solution and the regions above and below it.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(1,), initial_values=(0.5, 0.8, 0.3))
fig.show('png')
```

2. b. ii. Describe the limiting behaviour of each line.

all values trend towards 0.5.

2. c. i. Assume that  𝐻=1.2. Plot the slope field and two or three solutions.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(1.2,), initial_values=(0.5,))
fig.show('png')
```

2. c. iii. Describe the limiting behaviour of the lines.

regardless of input fish population tends towards extinction

3. Summarize what happens to the equilibrium solutions and their stability as $H$
is increased from 0 to beyond 1. Refer to your plots to support your answers.

the stable soultion, or the stable population number decreases and the unstable number increases until 1 where there is a single semi-stable soultion.

4. What is a reasonable strategy for sustainable fishing in this case?
Don’t forget to allow qualitatively for minor catastrophes, such as disease or temporary overfishing.

dont allow for fishing above 1000 fish to keep the population at a stable rate, assuming it stated above 500 fish.

```python

```


# Problem 3.6 Conditional Distribution

Show that
$$P_{C\mid E=2}\sim\mathcal{N}\left(\frac{8}{17},\frac{1}{17}\right)$$
for the SCM
\begin{align}
C:=&\mathcal{N}(0,1)\\
E:=&4C+\mathcal{N}(0,1)
\end{align}

## Proof

The SCM can be seen as a [multivariate normal distribution](https://en.wikipedia.org/wiki/Multivariate_normal_distribution)
$$
P\left(\begin{matrix}C\\E\end{matrix}\right)=
\mathcal{N}\left(\begin{bmatrix}\mu_C\\\mu_E\end{bmatrix},\Sigma\right)
$$
where $\mu_C=\mu_E=0$ because both distributions have an expecation of zero and $\Sigma$ is the covariance matrix which is
\begin{align}
\Sigma:=&\begin{bmatrix}\sigma^2_C&\rho\sigma_C\sigma_E\\\rho\sigma_C\sigma_E&\sigma_C^2\end{bmatrix}\\
=&\begin{bmatrix}1&14\\14&4^2+1\end{bmatrix}
\end{align}

For the bivariate case the conditional distribution is defined as
$${\displaystyle X_{1}\mid X_{2}=a\ \sim \ {\mathcal {N}}\left(\mu _{1}+{\frac {\sigma _{1}}{\sigma _{2}}}\rho (a-\mu _{2}),\,(1-\rho ^{2})\sigma _{1}^{2}\right).}$$

Applied to $C\mid E=2$ that yields $$\mathcal{N}\left(\frac{8}{17},\frac{1}{17}\right)$$


```
%pylab inline
import numpy as np
```

# Using finite differences to approximate derivatives

The material in this notebook is a supplement to chapter 1 of the text.

Suppose that you're given a table of values of some "secret" function $f(x)$, but you are not given the function itself.  For example:

```
x_values = (0,1,2,3,4)
y_values = (0,1,8,27,64)
```

In order to understand the function that generated these values, we could plot them:

```
plt.plot(x_values,y_values,'-o')
```

In the plot, the points have been connected by a line, and you might use the slope of this line if you wanted to estimate the derivative of the function.  For instance, to estimate the slope at $x=2$, you could use the slope of the line between $x=2$ and $x=3$:

```
plt.plot(x_values,y_values,'-o')
plt.hold(True)
plt.plot(x_values[2:4],y_values[2:4],'-r',linewidth=5)
```

$$\left. f'(x) \right|_{x=2} \approx \frac{f(3)-f(2)}{3-2} = 19.$$

Alternatively, you might use the slope of the line between $x=1$ and $x=2$:

```
plt.plot(x_values,y_values,'-o')
plt.hold(True)
plt.plot(x_values[1:3],y_values[1:3],'-r',linewidth=5)
```

$$\left. f'(x) \right|_{x=2} \approx \frac{f(2)-f(1)}{2-1} = 7.$$

Remember that the derivative is defined as the limit of a formula rather like our first approximation above (referred to as a *forward difference*):

$$ f'(x) = \lim_{h\to 0} \frac{f(x+h)-f(x)}{(x+h)-x} = \lim_{h\to 0} \frac{f(x+h)-f(x)}{h}.$$

Notice that the fraction in the definition is just the slope of the line connecting the values of $f$ at $x$ and $x+h$.  We could equally well define the derivative in terms of a *backward difference*, similar to our second formula above:

$$ f'(x) = \lim_{h\to 0} \frac{f(x)-f(x-h)}{x-(x-h)} = \lim_{h\to 0} \frac{f(x)-f(x-h)}{h}.$$

Yet another way of defining the derivative would be to use the slope of the line connecting points $x+h$ and $x-h$, leading to

$$ f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x-h)}{2h}.$$

Here the factor $2h$ in the denominator is just the length of the interval $(x-h,x+h)$.  Using this formula with our values gives

```
plt.plot(x_values,y_values,'-ok')
plt.hold(True)
plt.plot((x_values[1],x_values[3]),(y_values[1],y_values[3]),'-r',linewidth=5)
```

$$\left. f'(x) \right|_{x=2} \approx \frac{f(3)-f(1)}{3-1} = 13.$$

When we only know the values of a function at a finite set of points, we can't compute these limits, so it makes sense to use a finite value of $h$ in order to approximate the derivative.  This approach is known as the *finite difference method*.  Soon we will use it to solve differential equations, but first let's examine its effectiveness in approximating derivatives.

You may have guessed that the "secret" function that generated these values is $f(x)=x^3$.  Let's compare our three estimates of the derivative with the true derivative: $f'(2) = 3\cdot 2^2 = 12$.  Clearly the third formula (known as a *centered difference*) is the best approximation.  We can also see this by plotting the approximations and the tangent line (whose slope is that of the true derivative):

```
def f(x):
    return x**3

x = np.linspace(0,4)
tangent = 8 + 12*(x-2)
plot(x,f(x),'k',linewidth=2)
plt.plot(x_values[2:4],y_values[2:4],'-g',linewidth=3)
plt.plot(x_values[1:3],y_values[1:3],'-b',linewidth=3)
plt.plot((x_values[1],x_values[3]),(y_values[1],y_values[3]),'-r',linewidth=3)
plt.plot(x,tangent,'--k',linewidth=3)
plt.legend(['f(x)','Forward','Backward','Centered','Tangent'],loc='best')
plt.axis((0.5,3.5,0,40))
```

It's clear that the centered difference approximation (slope of the red line) is closest to the derivative (slope of the dotted line).  Would this be true if we picked another function $f(x)$, or if we used function values at different points?  Let's answer the second question with an experiment.

```
x = 2.
df = 12.

for h in (0.1, 0.05, 0.025):
    forward = (f(x+h)-f(x))/h
    forward_error = forward - df
    print forward_error
```

It seems clear from the results that reducing $h$ by a factor of two also reduces the error in the forward difference approximation by a factor of two.

**Now add backward and centered difference approximations to the code above and try to figure out what happens to their errors as $h$ decreases.  Make a table like Table 1.1 on page 5 of the textbook.**  Is the centered difference always the most accurate?  Do you think it will still be the most accurate if we continue reducing $h$?

**Now redefine the function $f(x)$ to be $x^2$ and generate the same table.  Do you see anything surprising?  Can you explain it?**  If not, make a guess.

## Estimating truncation errors

The error made by a finite difference approximation is called *truncation error*.  Why?  Well, we can estimate the error by expanding each function value in a *Taylor series*:

$$f(x+h) = f(x) + h f'(x) + \frac{1}{2}h^2 f''(x) + \frac{1}{6} h^3 f'''(x) + {\mathcal O}(h^4)$$

Here ${\mathcal O}(h^4)$ indicates that the rest of the terms in the series vanish at least as quickly as $h^4$ when $h\to 0$ (see Appendix A of the text).
Substituting this series in our forward difference formula gives
\begin{align}
\frac{f(x+h) - f(x)}{h} & = \frac{f(x) + h f'(x) + \frac{1}{2}h^2 f''(x) + \frac{1}{6} h^3 f'''(x) + {\mathcal O}(h^4) - f(x)}{h} \\
& = f'(x) + \frac{1}{2}h f''(x) + \frac{1}{6} h^2 f'''(x) + {\mathcal O}(h^3) \\
& = f'(x) + \frac{1}{2}h f''(x) + {\mathcal O}(h^2).
\end{align}

This analysis confirms our intuition that the forward difference approximates $f'(x)$, but it tells us much more.  Most importantly, it shows that the largest term in the error in this approximation is proportional to $h$.  That's why we saw that decreasing $h$ by a factor of two caused the error to decrease by the same amount.

Notice that if we truncated the Taylor series after the first term, we would get the forward difference formula exactly.  That is why the error is referred to as *truncation error* -- it's the error we get from truncating an infinite series.  The term $\frac{1}{2} h f''(x)$ is referred to as the *leading truncation error* because when $h$ is very small we expect that term to be much bigger than all the ones that come after it.

Since in our simple example we know the function $f(x)$, we could evaluate error terms in the series above to get a better approximation of the error.  But typically we won't know what the function is (if we did, why would we need finite differences?), so we'll be most interested in knowing what power of $h$ multiplies the leading error term.

**Write down the Taylor series for $f(x-h)$ about $x$.  Use that (and the series above) to work out the leading truncation error terms for the backward and centered difference formulas.**  Does your result for the centered formula allow you to explain the results you found before when applying it to $f(x)=x^2$?

## Polynomial interpolation and finite difference formulas

In sections 1.2-1.5 of the text, a method for finding finite difference formulas is given, based on Taylor series.  Here we will approach the same topic from a different route, using polynomial interpolation.  The idea is this: to approximate the derivative of a function whose point values are known, we first find a polynomial that interpolates those values, then evaluate the derivative of that polynomial.

To begin, suppose we are given three function values: $f(x_0-h), f(x_0), f(x_0+h)$, which we'll denote by $(f_1,f_2,f_3)$.  We wish to find a polynomial that passes through these three points.  As you may know, a set of $n$ values uniquely defines a polynomial of degree $n-1$, so we will look for a quadratic polynomial.  To make the computation simpler, we'll write it this way: $p(x) = a + b (x-x_0) + c (x-x_0)^2$.  We know that $p$ and $f$ must agree at the three given points, which means
\begin{align}
a + b(-h) + c (-h)^2  & = f_1 \\
a + b(0) + c(0)^2 & = f_2 \\
a + b (h) + c (h)^2 & = f_3 \\
\end{align}

or simply

\begin{align}
a - hb + h^2c  & = f_1 \\
a & = f_2 \\
a + hb + h^2c & = f_3 \\
\end{align}

We can rewrite this system of equations in matrix form:

\begin{align}
\begin{pmatrix}
1 & -h & h^2 \\
1 & 0   & 0 \\
1 & h & h^2 \\
\end{pmatrix}
\begin{pmatrix} a \\ b \\ c \end{pmatrix}
& = \begin{pmatrix} f_1 \\ f_2 \\ f_3 \end{pmatrix}
\end{align}

This is a linear system that we can solve for the coefficients $a,b,c$ in terms of $h$, and the values of $f$.  The result is

\begin{align}
a & = f_2, & b & = \frac{f_3-f_1}{2h}, & c & = \frac{f_1 -2 f_2 + f_3}{2h^2}.
\end{align}

Thus

\begin{align}
p(x) = f_2 + \frac{x-x_0}{2h}(f_3-f_1) + \frac{(x-x_0)^2}{2h^2}(f_1-2f_2+f_3).
\end{align}

Now suppose we wish to approximate the derivative of $f$.  We have an easy approach: use the derivative of $p$:

\begin{align}
p'(x) = \frac{f_3-f_1}{2h} + \frac{x-x_0}{h^2}(f_1 -2 f_2 + f_3).
\end{align}

In particular, if want to approximate the derivative at $x=x_0$, we get simply

$$f'(x) \approx p'(x_0) = \frac{f_3-f_1}{2h} = \frac{f(x_0+h) - f(x_0-h)}{2h}.$$

**Do you recognize this approximation?  Now use $p(x)$ to get an approximation for the second derivative of $f(x)$.  Write down the resulting formula.**

<font color='red'>
Derive a formula for $f''(x)$ based on the values $f(x), f(x-h), f(x-3h)$ by determining the interpolating polynomial and differentiating it.  How accurate is your formula?
</font>

**Can you think of a similar approach that would allow you to derive rules for numerical quadrature (integration)?**

```

```


## Higher Order Integration Schemes

The composite trapezoidal rule is functional, but not all that accurate.  At this point, the only method we have to improve accuracy is to divide the interval into smaller segments.  We can, however, use results for different step sizes $h$ to improve both results by eliminating the lowest order error.  To illustrate, consider the trapezoidal rule for a step $h_1=(b-a)$ and then $h_2=(b-a)/2$,

$$
\begin{align}
I_1 &=\int_a^b f(x) dx = \frac{h_1}{2}\left[f(a)+f(b)\right] - (b-a)\frac{h_1^2}{12} f''(\eta_1),\\
I_2 &=\int_a^b f(x) dx = \frac{h_2}{2}\left[f(a)+2f(a+h)+f(b)\right] - (b-a)\frac{h_2^2}{12} f''(\eta_2).
\end{align}
$$

The problem with attempting to use these two equations to eliminate the error term lies in the fact that $\eta_1$ and $\eta_2$ are not likely to be the same.  However, it is possible to show that, for a suitably smooth function, the error term can be replaced by a power series om even powers of $h$, namely

$$
E(h) = - (b-a)\frac{h^2}{12} f''(\eta_1) = \sum_{j=1}^n K_j h^{2j} + \mathcal{O}(h^{2n+2}),
$$

where the $K_j$ are constants, independent of the division $h$. Using this, and $h_2=h_1/2$ we can rewrite the two trapezoidal methods above as

$$
\begin{align}
I_1 &= \frac{h_1}{2}\left[f(a)+f(b)\right] + K_1 h_1^2 + K_2 h_1^4+ \mathcal{O}(h_1^6),\\
I_2 &= \frac{h_2}{2}\left[f(a)+2f(a+h_2)+f(b)\right] + K_1 h_2^2 + K_2 h_2^4 + \mathcal{O}(h_2^6),\\
&= \frac{h_1}{4}\left[f(a)+2f(a+h_2)+f(b)\right] + K_1 \frac{h_1^2}{4} + K_2 h_2^4+\mathcal{O}(h_1^6).
\end{align}
$$

Now note that

$$
\int_a^b f(x) dx=\frac{4 I_2 - I_1}{3} = \frac{h_2}{3}\left[f(a) + 4 f(a+h_2) + f(b) \right] - \frac{12}{3} K_2 h_2^4 + \mathcal{O}(h_2^6).
$$

The resulting numerical integration scheme is known as *Simpson's Rule* and the error is now $\mathcal{O}(h^4)$ rather than the $\mathcal{O}(h^2)$ for the (composite) trapezoidal rule.

This technique of eliminating error terms using results from two step sizes is generally referred to as Richardson's extrapolation.  We can also use this method to get rid of higher order error terms as well which brings us to *Romberg's Method*.

## Romberg's Method

We start by using the composite trapezoidal rule for $m_1=1,\, m_2=2,\, m_3=4,\,\cdots m_n=2^{n-1}$ subintervals with regularly spaced points $h_k=\frac{b-a}{m_k}$ apart. In particular, when we do this we don't want to recompute $f(x)$ any more than necessary so we need to write the trapezoidal rule for the different intervals in such a way that we use our previous work:

$$
\begin{align}
R_{1,1} &= \frac{h_1}{2}\left[f(a)+f(b)\right],\\
R_{2,1} &= \frac{h_2}{2}\left[f(a)+f(b) + 2 f(a+h_2)\right],\\
&= \frac{1}{2}\left[ \frac{h_1}{2}\left\{ f(a)+f(b)\right\} + 2h_2 f(a+h_2)\right],\\
&= \frac{1}{2}\left[ R_{1,1} + h_1 f(a+h_2)\right],\\
R_{3,1} &= \frac{h_3}{2}\left[f(a)+f(b) + 2\left\{f(a+h_3)+f(a+2 h_3) + f(a+3 h_r)\right\}\right],\\
&= \frac{1}{2}\left[ R_{2,1} + h_2 \left\{ f(a+h_3)+ f(a+3 h_r)\right\}\right],\\
&\vdots \\
R_{k,1} &= \frac{1}{2}\left[ R_{k-1,1}+h_{k-1} \sum_{i=1}^{2^{k-2}} f(a+(2i-1)h_k) \right].
\end{align}
$$

We now apply the extrapolation technique to refine our results using

$$
R_{k,2}=\frac{4 R_{k,1}-R_{k-1,1}}{3},
$$

on each successive pair of our previous results.

We then apply the extrapolation technique to successivly higher orders using

$$
R_{i,j} = \frac{4^{j-1}R_{i,j-1}-R_{i-1,j-1}}{4^{j-1}-1},\qquad j=2,3,\cdots,n.
$$

**Example**  Suppose we want to compute $\int_0^\pi \sin x\, dx$  (which in this case we know is $2$).  We construct the following tableau:

$$
\begin{align}
{\begin{array}{ccccccc}
    R_{1,1}    &         &         &         &         & &\\
    \downarrow & \rangle & R_{2,2} &         &         & &\\
    R_{2,1}    &         &         & \rangle & R_{3,3} & &\\
    \downarrow & \rangle & R_{3,2} &         &         & \rangle & R_{4,4}\\
    R_{3,1}    &         &         & \rangle & R_{4,3} & &\\
    \downarrow & \rangle & R_{4,2} &         &         & &\\
    R_{4,1}    &         &         &         &         &\\
  \end{array} }
\end{align}
$$

With actual numbers, this translates to

$$
\begin{align}
{\begin{array}{ccccccc}
    0    &         &         &         &         & &\\
    \downarrow & \rangle & 2.094... &         &         & &\\
    1.57079...    &         &         & \rangle & 1.9985... & &\\
    \downarrow & \rangle & 2.0045... &         &         & \rangle & 2.0000055...\\
    1.896...    &         &         & \rangle & 1.999983... & &\\
    \downarrow & \rangle & 2.00026... &         &         & &\\
    1.974...    &         &         &         &         &\\
  \end{array} }
\end{align}
$$

We see that even though the underlying composite trapezoidal rule results in the first column are not all that accurate, extrapolation of those results to the fourth column gives  six digits of accuracy.  It should also be clear that we can get an estimate for our (forward) error by comparing the difference between our best two results, in this case $R_{4,4}$ and $R_{4,3}$ where $|R_{4,4}-R_{4,3}|\sim 2\times 10^{-5}$.  Here, this is the error in $R_{4,3}$ and so an overestimate of the error for $R_{4,4}$.

While these are fairly easy to code up yourself, there are also scipy versions of all of these integration routines as illustrated below where we use the various schemes to integrate $\frac{1}{\sqrt{\pi}}e^{-x^2}$ from $0$ to $2$.  We start with the results of the cumulative trapezoidal and cumulative Simpson's rule with our interval divided into eight segments:

```python
import numpy as np

def f(x):
   return 1/np.sqrt(np.pi) * np.exp(-x**2)

x = np.linspace(0, 2, num=9, endpoint=True)
print("sample points: ",x)

y = f(x)
from scipy import integrate
I1 = integrate.trapezoid(y, x)
I2 = integrate.simpson(y, x)
print("trapezoidal = ",I1, ", Simpsons = ",I2)
```

    sample points:  [0.   0.25 0.5  0.75 1.   1.25 1.5  1.75 2.  ]
    trapezoidal =  0.49744809484415425 , Simpsons =  0.4976521729751664

The accuracy of these results is difficult to judge, other than by comparing them which gives us an estimate for the trapezoidal rule.  In contrast, the Romberg integration keeps adding a row to the tableau until the results converge:

```python
from scipy import integrate

normaldist = lambda x: 1/np.sqrt(np.pi) * np.exp(-x**2)
result = integrate.romberg(normaldist, 0, 2, show=True)
```

    Romberg integration of <function vectorize1.<locals>.vfunc at 0x0000020286FC9AF0> from [0, 2]

     Steps  StepSize   Results
         1  2.000000  0.574523
         2  1.000000  0.494815  0.468246
         4  0.500000  0.496836  0.497509  0.499460
         8  0.250000  0.497448  0.497652  0.497662  0.497633
        16  0.125000  0.497607  0.497661  0.497661  0.497661  0.497661
        32  0.062500  0.497648  0.497661  0.497661  0.497661  0.497661  0.497661
        64  0.031250  0.497658  0.497661  0.497661  0.497661  0.497661  0.497661  0.497661

    The final result is 0.4976611325094085 after 65 function evaluations.

Our previous results from the trapezoidal and Simpson's rule correspond to the first two results in the fourth row.

```python

```


dS/dt=-bSI+gI, dI/dt=bSI-gI (uso b para beta y g para gamma)

```python
from sympy import *
from sympy.abc import S,I,t,b,g
```

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
import pylab as pl
```

```python
#puntos criticos
P=-b*S*I+g*I
Q=b*S*I-g*I
#establecer P(S,I)=0 y Q(S,I)=0
Peqn=Eq(P,0)
Qeqn=Eq(Q,0)
print(solve((Peqn,Qeqn),S,I))
#matriz Jacobiana
J11=diff(P,S)
J12=diff(P,I)
J21=diff(Q,S)
J22=diff(Q,I)
J=Matrix([[J11,J12],[J21,J22]])
pprint(J)
```

    [(S, 0), (g/b, I)]
    ⎡-I⋅b  -S⋅b + g⎤
    ⎢              ⎥
    ⎣I⋅b   S⋅b - g ⎦

```python
#J en el punto critico
Jc1=J.subs([(S,S),(I,0)])
pprint(Jc1)
pprint(Jc1.eigenvals())
pprint(Jc1.eigenvects())
Jc2=J.subs([(S,g/b),(I,I)])
pprint(Jc2)
pprint(Jc2.eigenvals())
pprint(Jc2.eigenvects())
```

    ⎡0  -S⋅b + g⎤
    ⎢           ⎥
    ⎣0  S⋅b - g ⎦
    {0: 1, S⋅b - g: 1}
    ⎡⎛      ⎡⎡1⎤⎤⎞  ⎛            ⎡⎡-1⎤⎤⎞⎤
    ⎢⎜0, 1, ⎢⎢ ⎥⎥⎟, ⎜S⋅b - g, 1, ⎢⎢  ⎥⎥⎟⎥
    ⎣⎝      ⎣⎣0⎦⎦⎠  ⎝            ⎣⎣1 ⎦⎦⎠⎦
    ⎡-I⋅b  0⎤
    ⎢       ⎥
    ⎣I⋅b   0⎦
    {0: 1, -I⋅b: 1}
    ⎡⎛      ⎡⎡0⎤⎤⎞  ⎛         ⎡⎡-1⎤⎤⎞⎤
    ⎢⎜0, 1, ⎢⎢ ⎥⎥⎟, ⎜-I⋅b, 1, ⎢⎢  ⎥⎥⎟⎥
    ⎣⎝      ⎣⎣1⎦⎦⎠  ⎝         ⎣⎣1 ⎦⎦⎠⎦

Los puntos criticos son no hiperbolicos, por lo que no cumplen el teorema de Hartmann.

```python
b=1
g=1
def dx_dt(x,t):
    return [ -b*x[0]*x[1]+g*x[1] , b*x[0]*x[1]-g*x[1] ]
#trayectorias en tiempo hacia adelante
ts=np.linspace(0,10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#trayectorias en tiempo hacia atras
ts=np.linspace(0,-10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#etiquetas de ejes y estilo de letra
plt.xlabel('S',fontsize=20)
plt.ylabel('I',fontsize=20)
plt.tick_params(labelsize=12)
plt.ticklabel_format(style="sci", scilimits=(0,0))
plt.xlim(0,100000)
plt.ylim(0,100000)
#campo vectorial
X,Y=np.mgrid[0:100000:15j,0:100000:15j]
u=-b*X*Y+g*Y
v=b*X*Y-g*Y
pl.quiver(X,Y,u,v,color='dimgray')
plt.savefig("SISinf.pdf",bbox_inches='tight')
plt.show()
```

Analsis de Bifurcaciones

El sistema tiene dos puntos criticos, (S,0) y (gamma/beta,I), el primer punto no depende de gamma ni beta, por lo que no cambia.


I assume Spong angles are $\theta_1$ and $\theta_2$.

```python
from __future__ import print_function, division
from sympy import *
init_printing(use_unicode=True)
```

```python
m1, l1, lc1, I1, theta1, theta1dot = symbols('m1 l1 l_{c1} I1 theta1 thetadot1')
m2, lc2, I2, theta2, theta2dot = symbols('m2 l_{c2} I2 theta2 thetadot2')
```

```python
M = Matrix([[m2*l1*(l1+2*lc2*cos(theta2))+I1+I2,
             m2*l1*lc2*cos(theta2)+I2],
            [m2*l1*lc2*cos(theta2)+I2,
             I2]])
```

```python
M
```

```python
C = Matrix([[-2*m2*l1*lc2*sin(theta2)*theta2dot,
             -m2*l1*lc2*sin(theta2)*theta2dot],
            [m2*l1*lc2*sin(theta2)*theta1dot,
             0]])
```

```python
C
```

```python
g = symbols('g')
G = g*Matrix([[-m1*lc1*sin(theta1)-m2*(l1*sin(theta1)+lc2*sin(theta1+theta2))],
              [-m2*lc2*sin(theta1+theta2)]])
```

```python
G
```

Now, let's change the notations to the one that Russ used [here](http://underactuated.csail.mit.edu/underactuated.html?chapter=acrobot) by having ${}^{\texttt{Russ}}I={}^{\texttt{Spong}}I+ml_c^2$ and ${}^{\texttt{Russ}}\theta_1={}^{\texttt{Spong}}\theta_1+\pi/2$

```python
q1, q1dot = symbols('q1 q1dot')
q2, q2dot = symbols('q2 q2dot')
```

```python
M.subs([(I1,I1+m1*lc1**2), (I2,I2+m2*lc2**2),
        (theta1,q1+pi/2), (theta2,q2)])
```

```python
C.subs([(I1,I1+m1*lc1**2), (I2,I2+m2*lc2**2),
        (theta1,q1+pi/2), (theta2,q2)])
```

```python
G.subs([(I1,I1+m1*lc1**2), (I2,I2+m2*lc2**2),
        (theta1,q1+pi/2), (theta2,q2)])
```


# 3.2 Linear Regression Models and Least Squares

We have an input vector $X^T=(X_1,...,X_p)$ and want to predict a real-valued output $Y$.The linear regression model has the form:

$$f(X) = B_0 + \sum_{j=1}^p {X_j\beta_j}$$

 Typically we have a set of training data $(x_1, y_1)...(x_N, y_n)$ from which to estimate the parameters $\beta$. The most popular estimation method is *least squares*, in which we pick $\beta$ to minimize the residual sum of squares, (3.2):
$$
\begin{align}
RSS(\beta)&=\sum_{i=1}^N(y_i-f(x_i))\\
&=\sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^p{x_{ij}\beta_j})^2
\end{align}
$$

How do we minimize (3.2)? We can write the (3.2) using matrix, (3.3):
$$RSS(\beta)=(\mathbf{y}-\mathbf{X}\beta)^T(\mathbf{y}-\mathbf{X}\beta)$$

Differentiating with respect to $\beta$ we obtain:
$$
\begin{align}
\frac{\partial{RSS}}{\partial\beta} = -2\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)
\end{align}
$$

Assuming that **X** has full column rank, and hence the second derivative is positive definite:
$$\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)=0$$

and the unique solution is:
$$\hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

The predicted value at an input vector $x_0$ are given by $\hat{f}(x_0)=(1:x_0)^T\hat{\beta}$:

$$\hat{y}=\mathbf{X}\hat{\beta}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

The matrix $\mathbf{H}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ is sometimes called the "hat" matrix.

**Geometrical representation of the least squares:** We denote the column vectors of **X** by $x_0, x_1, ..., x_p$. These vectors span a subspace of $\mathcal{R}^N$, also referred as the column space of **X**. We minimize $RSS(\beta)=||\mathbf{y}-\mathbf{X}\beta||^2$ by choosing $\hat{\beta}$ so that the residual vector $\mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to this subspace and the orthogonality is expressed by $\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)=0$. The hat matrix **H** is the projection matrix.

**Sampling properties of $\hat{\beta}$**: In order to pin down the sampling properties of $\hat{\beta}$, we assume that the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and that the $x_i$ are fixed. The variance-covariance matrix is given by (3.8):

$$
\begin{align}
Var(\hat{\beta}) &= E\left[(\hat{\beta}-E(\hat{\beta}))(\hat{\beta}-E(\hat{\beta})^T)\right]\\
&= E\left[(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{\varepsilon}\mathbf{\varepsilon}^T\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\right]\\
&= \sigma^2(\mathbf{X}^T\mathbf{X})^{-1}
\end{align}
$$

One estimates the variance $\sigma^2$ by:
$$
\hat{\sigma}^2 = \frac{1}{N-p-1} \sum_{i=1}^N(y_i-\hat{y_i})^2
$$

The N-p-1 rather than N in the denominator makes $\hat{\sigma}^2$ an unbiased estimate of $\sigma^2$: $E(\hat{\sigma}^2)=\sigma^2$.

*Proof*:
$$
\begin{align}
\hat{\varepsilon} &= \mathbf{y} - \mathbf{\hat{y}}\\
&= \mathbf{X}\beta + \varepsilon - \mathbf{X}\hat{\beta}\\
&= \mathbf{X}\beta + \varepsilon - \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T(\mathbf{X}\beta + \varepsilon)\\
&= \varepsilon - \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon\\
&= (\mathbf{I}_n - \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T)\varepsilon\\
&= (\mathbf{I}_n - \mathbf{H})\varepsilon
\end{align}
$$

and we would like to find $Var(\hat{\varepsilon})=E(\hat{\varepsilon}^T\hat{\varepsilon})$:

$$
\begin{align}
E[\hat{\varepsilon}^T\hat{\varepsilon}]
&= E\left[\varepsilon^T(\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H})\varepsilon\right]\\
&= E\left[tr(\varepsilon^T(\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H})\varepsilon)\right]\\
&= E\left[tr(\varepsilon\varepsilon^T(\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H}))\right]\\
&= \sigma^2E\left[tr((\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H}))\right]\\
&= \sigma^2E\left[tr(\mathbf{I}_n - \mathbf{H})\right]\\
&= \sigma^2E\left[tr(\mathbf{I}_n) - tr(\mathbf{I}_{p+1})\right]\\
&= \sigma^2(n-p-1)
\end{align}
$$

Note that, both $\mathbf{H}$ and $\mathbf{I_n}-\mathbf{H}$ are:

- Symmetry matrix, i.e $\mathbf{H}^T=\mathbf{H}$

- Idempotent matrix, i.e $\mathbf{H}^2=\mathbf{H}$

**Inferences about the parameters and the model:** We now assume that deviations of Y around its expectations and Gaussian. Hence (3.9):

$$
\begin{align}
Y &=E(Y|X_1,...,X_p)+\varepsilon\\
&= \beta_0 + \sum_{j=1}^P{X_j\beta_j} + \varepsilon
\end{align}
$$

where $\varepsilon \sim N(0, \sigma^2) $

Under (3.9), it is easy to show that (3.10):
$$
\hat{\beta} \sim N(\beta, (\mathbf{X}^T\mathbf{X})^{-1}\sigma^2)
$$

Also (3.11):
$$(N-p-1)\hat{\sigma}^2 \sim \sigma^2\chi_{N-p-1}^2$$

a chi-squared distribution with N-p-1 degrees of freedom and $\hat{\beta}$ and $\hat{\sigma}$ are statistically independent.

**Hypothesis test:** To test $H_0: \beta_j = 0$ we form the standardized coefficient or *Z-score*:
$$
z_j=\frac{\hat{B}_j}{\hat{\sigma}\sqrt{v_j}}
$$

where $v_j$ is the jth diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$. Under the null hypothesis $z_j$ is distributed as $t_{N-p-1}$, and hence a large value of $z_j$ will lead to rejection. If $\hat{\sigma}$ is replaced by $\sigma$ then $z_j$ is a standard normal distribution. The difference between tail quantiles of a t-distribution and a standard normal become negligible as the sample size increases, see the Figure (3.3) below:

```python
# Figure 3.3
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm, t

fig = plt.figure(figsize = (12, 8))
axes = fig.add_subplot(1, 1, 1)

z = np.linspace(1.9, 3, 500)

normal_probabilities = 1 - norm.cdf(z) + norm.cdf(-z)
t_30_probabilities = 1 - t.cdf(z, 30) + t.cdf(-z, 30)
t_100_probabilities = 1 - t.cdf(z, 100) + t.cdf(-z, 100)

axes.plot(z, normal_probabilities, color='C0', label = 'normal')
axes.plot(z, t_30_probabilities, color='C1', label = '$t_{30}$')
axes.plot(z, t_100_probabilities, color='C2', label = '$t_{100}$')

xlim = axes.get_xlim()

for y in [0.01, 0.05]:
    axes.plot(xlim, [y, y], '--', color = 'gray',
              scalex = False, scaley = False)

    for index, probs in enumerate([normal_probabilities, t_30_probabilities,
                                   t_100_probabilities]):
        x = z[np.abs(probs - y).argmin()]
        axes.plot([x, x], [0, y], '--', color = f"C{index}",
                  scalex = False, scaley = False)

axes.legend()
axes.set_xlabel('Z')
axes.set_ylabel('Tail Probabilities')
plt.show()
```

**Test for the significance of groups of coefficients simultaneously**: We use the F-statistics (3.13):

$$F=\frac{(RSS_0-RSS_1) / (p_1 - p_0)}{RSS_1/(N-p_1-1)}$$

Where $RSS_1$ is for the bigger model with $p_1+1$ parameters and $RSS_0$ for the nested smaller model with $p_0+1$ parameters. Under the null hypothesis that the smaller model is correct, the F statistic will have a $F_{p_1-p_0,N-p_1-1}$ distribution. The $z_j$ in (3.13) is equivalent to the F statistic for dropping the single coefficient $\beta_j$ from the model.

Similarly, we can isolate $\beta_j$ in (3.10) to obtain $1-2\alpha$ confidence interval (3.14)
$$(\hat{\beta_j} - z^{(1-\alpha)}v_j^{\frac{1}{2}}\hat{\sigma}, \hat{\beta_j} + z^{(1-\alpha)}v_j^{\frac{1}{2}}\hat{\sigma})$$

In a similar fashion we can obtain an approximate confidence set for the entire parameter vector $\beta$ (3.15):
$$ C_{\beta} = \{{\beta|(\hat{\beta}-\beta)^T\mathbf{X}^T\mathbf{X}(\hat{\beta}-\beta)} \le \hat{\sigma}^2{\chi_{p+1}^2}^{(1-\alpha)} \}$$

where ${\chi_{l}^2}^{(1-\alpha)}$ is the $1-\alpha$ percentile of the chi-squared distribution of $l$ degrees of freedom.


## Regresión lineal

**Temas Selectos de Modelación Numérica**

Facultad de Ciencias, UNAM

Semestre 2021-2

En este notebook aprenderemos como hacer una regresión lineal por el método de mínimos cuadrados y por el método matricial.

No olvides resolver los ejercicios de tarea al final del notebook. Entrega tu solución en un notebook en la carpeta de Classroom con el nombre `apellido_nombre_tarea05.ipynb`.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## 1. Mínimos cuadrados

Ajuste de rectas de la forma $y=mx+b$ por mínimos cuadrados. La idea detrás de este método es que queremos encontrar la pendiente $m$ y la ordenada al origen $b$ que nos dan la recta que minimiza la suma de los cuadrados de las distancias entre los puntos (digamos, datos) y la recta ajustada:

Es decir, que cuando sumemos el cuadrado de todas las distancias de los puntos a la recta (líneas azules) el valor que obtengamos sea el más pequeño posible (este tipo de problemas se llaman problemas de optimización).

Se pueden ajustar todo tipo de funciones por este método, no sólo rectas, pero para el caso particular de la recta, se obtiene que la pendiente y la ordenada al origen de la recta que minimiza el cuadrado de las distancias se calcula como:

\begin{align}
\tag{1}
m =\frac{N \sum(x_iy_i) − \sum x_i\sum y_i}{N \sum(x_i^2) − (\sum x_i)^2} \label{eq1}\\
\end{align}

\begin{align}
b = \frac{\sum y_i − m \sum x_i}{N} \tag{2}
\end{align}

en donde $N$ es el número de mediciones o puntos, $x_i$, $y_i$ son las mediciones y las sumas ($\sum$) son sobre todas las mediciones.

**OJO**: Debido a que no es necesario graficar los datos para realizar un ajuste por mínimos cuadrados, se puede caer en errores graves como tratar de ajustar una recta a un conjunto de mediciones cuya relación no es lineal. Por eso **es muy importante graficar** los datos y asegurarse de que la relación entre las variables es lineal antes de aplicar el método de mínimos cuadrados.

Siguiendo las ecuaciones anteriores definamos una función `reg_lineal` que calcule a pendiente y ordenada al origen de la recta que mejor se ajusta a los "datos" usando el método de mínimos cuadrados:

```python
def reg_lineal(X,Y):
    '''Esta función calcula la pendiente y la ordenada al origen de la recta y=mx+b por mínimos
    cuadrados a partir de los vectores de mediciones X y Y.
    Input:
    X - arreglo de numpy 1D
    Y - arreglo de numpy 1D del mismo tamaño que X.
    Output:
    m, b : Escalares, la pendiente y ordenada al origen.
    '''
    N = len(X) # numero de valores del vector X
    sum_xy = np.sum(X*Y) # suma de todos los Xi*Yi
    sum_x = np.sum(X) # suma de todas las X
    sum_y = np.sum(Y) # suma de todas las Y
    sum_x2 = np.sum(X**2) # suma de todas las Xˆ2

    m = ((N*sum_xy) - (sum_x*sum_y)) / ((N*sum_x2) - (sum_x**2))
    b = (sum_y - (m*sum_x)) / N
    return(m, b)
```

Probemos nuestra función que calcula la regresión lineal. Para ello generamos un vector X y un vector f(X)=Y de la siguiente manera:

```python
X = np.linspace(1,10,10)
Y = 1 + 2*X + 1*np.random.randn(1) # f(x)=y=1+2x+d

plt.plot(X,Y,'o')
plt.xlabel('x')
plt.ylabel('y=f(x)')
plt.show()
```

Ahora podemos probar la función `reg_lineal` usando X y Y:

```python
m, b = reg_lineal(X,Y)
print('La pendiente m es %f y la ordenada b es %f' %(m,b))
Y2 = m*X+b

plt.plot(X,Y,'o', label='Y')
plt.plot(X,Y2,'-',label='regresión' )
plt.xlabel('x')
plt.ylabel('y=f(x)')
plt.legend()
plt.show()
```

## 2. Método matricial
(Nota: El material de esta sección fue tomado del blog [cmdlinetips](https://cmdlinetips.com/2020/03/linear-regression-using-matrix-multiplication-in-python-using-numpy/))

También podemos hacer regresiones lineales usando el método matricial. Recordemos que en una regresión lineal queremos ajustar nuestros datos, observaciones, etc. usando el modelo lineal $$y=\beta_0+\beta_1X+\epsilon$$ y estimar los parámetros del modelo $\beta_0$ y $\beta_1$ que son la ordenada al origen y la pendiente, respectivamente.

Podemos combinar las "variables predictivas", en este caso X, en una matriz que tiene un vector columna lleno de unos (lo que multiplica a $\beta_0$) y X (lo que multiplica a $\beta_1$):

```python
X_mat = np.vstack((np.ones(len(X)), X)).T # en este caso usamos la función vstack
                                          # y el método T (transponer) para obtener las dimensiones
                                          # adecuadas
```

Con un poco de álgebra lineal y el objetivo de minimizar el error cuadrático medio del sistema de ecuaciones lineales llegamos a que podemos calcular el valor de los parámetros $\hat{\beta}=(\beta_0, \beta_1)$ de la forma:

$$\hat{\beta}=(X^T. X)^{-1}. X^T. Y$$

Podemos implementar esta ecuación usando las funciones para la inversa de una matriz y multiplicación matricial del módulo de álgebra lineal de numpy `linalg`:

```python
beta = np.linalg.inv(X_mat.T.dot(X_mat)).dot(X_mat.T).dot(Y)

print('La pendiente beta_1 es %f y la ordenada beta_0 es %f' %(beta[1], beta[0]))
```

    La pendiente beta_1 es 2.000000 y la ordenada beta_0 es 2.756253

que son los mismos valores para la pendiente y ordenada al origen que encontramos usando la función `reg_lineal`. Ahora usemos estos parámetros para estimar los valores de Y:

```python
Y_mat = X_mat.dot(beta)

plt.plot(X,Y,'o', label='Y')
plt.plot(X,Y_mat,'-',label='regresión método matricial' )
plt.xlabel('x')
plt.ylabel('y=f(x)')
plt.legend()
plt.show()
```

## 3. Ejemplo usando ambos métodos

```python
X = np.linspace(1,10)
Y = 1 + 2*X + 3*X**2 + 20*np.random.randn(1)

# Método matricial - ahora tenemos otro término, Xˆ2
X_mat = np.vstack((np.ones(len(X)), X, X**2)).T
beta = np.linalg.inv(X_mat.T.dot(X_mat)).dot(X_mat.T).dot(Y)
Y_mat = X_mat.dot(beta)

# Función reg_lineal
m, b = reg_lineal(X,Y)
Y2 = m*X+b

plt.plot(X,Y,'o',label='datos')
plt.plot(X,Y_mat, label='Método matricial')
plt.plot(X,Y2, label='Mínimos cuadrados - recta')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend()
plt.show()
```

Ups, claramente ajustar los datos a una recta no era una buena idea. Por eso es importante hacer una inspección gráfica de lo que queremos ajustar.

### Ejercicios de tarea:

1. Desarrollar las expresiones para la ordenada al origen y la pendiente en una recta usando el método de mínimos cuadrados. (Entregar el desarrollo en un archivo aparte).

2. Programar una función que calcule la regresión de un polinomio de grado 3. En un notebook define la función y puébala usando el polinomio de grado 3 $y = 2x^3+5x^2-11x+7$.

```python

```