```python
from gio import GIO
import numpy as np

##Data from Chan et al. (2018)'s Example 1##
#Notice that we have numpy arrays here.
A = np.array([[2,5],[2,-3],[2,1],[-2,-1]])
b = np.array([[10],[-6],[4],[-10]])
x0 = np.array([[2.5],[3]])

##Creating the GIO object and Generating all of the x^0-ep* for all GIO Models##
#.GIO_all_measures() calls all of the GIO methods
gio_testing = GIO(A,b,x0)
gio_testing.GIO_all_measures()
print("This is x^0-ep* for p=1: ",gio_testing.x0_epsilon_p[0])
print("This is x^0-ep* for p=2: ",gio_testing.x0_epsilon_p[1])
print("This is x^0-ep* for p='inf': ",gio_testing.x0_epsilon_p[2])
print("This is x^0-ep* for absolute duality: ",gio_testing.x0_epsilon_a)
print("This is x^0-ep* for relative duality: ",gio_testing.x0_epsilon_r)
```

    This is x^0-ep* for p=1:  [[ 2.5       ]
     [ 3.66666667]]
    This is x^0-ep* for p=2:  [[ 2.19230769]
     [ 3.46153846]]
    This is x^0-ep* for p='inf':  [[ 2.1]
     [ 3.4]]
    This is x^0-ep* for absolute duality:  [array([[ 2.1],
           [ 3.4]])]
    This is x^0-ep* for relative duality:  [array([[ 3.16666667],
           [ 3.66666667]])]

```python
##Because we are storing the calculated attributes in lists, we see that,
##for the last two print statements in the previous cell, a list was returned
##with an array inside of it.  We can solve this problem by add an index element [0]
print("This is x^0-ep* for absolute duality: ",gio_testing.x0_epsilon_a[0])
print("This is x^0-ep* for relative duality: ",gio_testing.x0_epsilon_r[0])
```

    This is x^0-ep* for absolute duality:  [[ 2.1]
     [ 3.4]]
    This is x^0-ep* for relative duality:  [[ 3.16666667]
     [ 3.66666667]]

```python
##The GIO class also handles cases when you might minimally project onto multiple
##hyperplanes. The class will put the multiple istar indices into the .istar_multi
##attribute but will choose the first istar index as the one to continue calculations.

##The class will also output a message to the user to let the user know that
##this has occurred.

A_1 = np.array([[-1,-1],[1,-1],[1,1],[-1,1]])
b_1 = np.array([[-1],[-1],[-1],[-1]])
x0_1 = np.array([[0],[0.7]])

```

The corresponding feasible region set up for the above $A$ and $b$ is as follows:

\begin{equation}
-x_1 - x_2 \geq -1
\end{equation}
\begin{equation}
x_1 - x_2 \geq -1
\end{equation}
\begin{equation}
x_1 + x_2 \geq -1
\end{equation}
\begin{equation}
-x_1 + x_2 \geq -1
\end{equation}

We can then demonstrate the feasible region using the image below (with approximate placement of $x^0$ below). The numbering corresponds to the order of the constraints below (and then the 0 indexing of Python).

```python
##We will create another GIO object to find the optimal epsilon* under the p=1 norm.
##Due to the position of x0_1 and the properties of the p=1 norm, we know that this
##will result into a projection onto multiple hyperplanes

#We will need to bring in everything again because we had a break in the code
#cells (with the markup cell above)

from gio import GIO
import numpy as np

A_1 = np.array([[-1,-1],[1,-1],[1,1],[-1,1]])
b_1 = np.array([[-1],[-1],[-1],[-1]])
x0_1 = np.array([[0],[0.7]])

gio_multi_project_testing = GIO(A_1,b_1,x0_1)
gio_multi_project_testing.GIO_p(1,'F')
```

    Under the inf dual norm, x^0 has been projected onto multiple hyperplanes.  For now, we will choose the first i index and will put the rest of the indices in the istar_multi attribute.

```python
##As we can see, a message is printed to the user when x^0 is projected onto multiple
##hyperplanes.
##We can see the multiple istar if we print the .istar_multi attribute
print(gio_multi_project_testing.istar_multi)
```

    [array([0, 1], dtype=int64)]

```python
##To obtain the raw indices, index into the list
print("Raw Indices: ", gio_multi_project_testing.istar_multi[0])
##To obtain only one of the indices, do a double index
print("Single index: ",gio_multi_project_testing.istar_multi[0][0])
```

    Raw Indices:  [0 1]
    Single index:  0

```python
##We see though that only one epsilon* is calculated and thus only one x^0-epsilon*
##is also calculated
print("This is the epsilon* for p=1: ",gio_multi_project_testing.epsilon_p)
print("This is the (x^0 - epsilon*): ",gio_multi_project_testing.x0_epsilon_p)
```

    This is the epsilon* for p=1:  [array([[-0.3],
           [-0. ]])]
    This is the (x^0 - epsilon*):  [array([[ 0.3],
           [ 0.7]])]

```python
###Calculating rho values###
from gio import GIO
import numpy as np

##Chan et al. (2018)'s Example 1##
##We have changed x0 to demonstrate the rho_p_approx <= rho_p relationship
A = np.array([[2,5],[2,-3],[2,1],[-2,-1]])
b = np.array([[10],[-6],[4],[-10]])
x0 = np.array([[4],[1]])

GIO_rho_test = GIO(A,b,x0)
GIO_rho_test.calculate_rho_p(2) #notice that we did not
                #have to specify if_append='F' because
                #we set a default value in the code
GIO_rho_test.calculate_rho_p_approx(2)
print("This is rho_p exact:",GIO_rho_test.rho_p)
print("This is rho_p - rho_p_approx:",(GIO_rho_test.rho_p[0] - GIO_rho_test.rho_p_approx[0]))

GIO_rho_test.calculate_rho_a()
print("This is rho_a:",GIO_rho_test.rho_a)
GIO_rho_test.calculate_rho_r()
print("This is rho_r:",GIO_rho_test.rho_r)

```

    This is rho_p exact: [0.73886235873548278]
    This is rho_p - rho_p_approx: 0.0232038316283
    This is rho_a: [0.7119341563786008]
    This is rho_r: [0.8851674641148326]

```python
###Testing some values for Validating rho_r#####
from gio import GIO
import numpy as np

##Chan et al. (2018)'s Example 1##
A = np.array([[2,5],[2,-3],[2,1],[-2,-1]])
b = np.array([[10],[-6],[4],[-10]])
x0_1 = np.array([[3],[2]])
x0_2 = np.array([[4],[1]])
x0_3 = np.array([[2],[2]])

validate_rho_r = GIO(A,b,x0_1)
validate_rho_r.calculate_rho_r()
print("This is rho_r for x0_1:",validate_rho_r.rho_r)

validate_rho_r = GIO(A,b,x0_2)
validate_rho_r.calculate_rho_r()
print("This is rho_r for x0_2:",validate_rho_r.rho_r)

validate_rho_r = GIO(A,b,x0_3)
validate_rho_r.calculate_rho_r()
print("This is rho_r for x0_3:",validate_rho_r.rho_r)

```

    This is rho_r for x0_1: [0.71428571428571441]
    This is rho_r for x0_2: [0.8851674641148326]
    Under the b dual norm, x^0 has been projected onto multiple hyperplanes.  For now, we will choose the first i index and will put the rest of the indices in the istar_multi attribute.
    This is rho_r for x0_3: [0.18644067796610142]

```python
###Structural Considerations for GIO Models###
from gio import GIO
import numpy as np
import pyomo.environ as pyo

A = np.array([[2,5],[2,-3],[2,1],[-2,-1]])
b = np.array([[10],[-6],[4],[-10]])
x0 = np.array([[2.5],[3]])

structural_ep_gio_test = GIO(A,b,x0) #instigating the GIO object

###Step 1: Create the Base Model###
#The attribute where the model is stored is called GIO_struc_ep, which is
#the Pyomo model.
structural_ep_gio_test.GIO_structural_epsilon_setup()

### Step 2: If desired, Add constraints on Epsilon ###
### to GIO_struc_ep ###
### The variable for epsilon is called ep ###
### The index set for variables is varindex ###
def ep_constraint(model):  #should provide the details of the index sets and the numvar parameters
    return model.ep[1] <= model.ep[2] #specifically did not ID the epsilon as nonnegative in gio.py
structural_ep_gio_test.GIO_struc_ep.constraint_ep = pyo.Constraint(rule=ep_constraint)

def neg_ep(model,i):
    return model.ep[i] <= 0
structural_ep_gio_test.GIO_struc_ep.neg_ep = pyo.Constraint(\
            structural_ep_gio_test.GIO_struc_ep.varindex,rule=neg_ep)

### Step 3: Solve the Model ###
#You can specify p as p=1,2,infty
#The solver will generate some output letting you know about infeasibility
#but we also produce output too, letting you know which constraint number
#in terms of Python indexing that, when forced to be an equality,
#results in infeasibility
structural_ep_gio_test.GIO_structural_epsilon_solve(2)

print("*************************************************")
print("The c vector found by this workflow is:",structural_ep_gio_test.c_p[0])

###########################################################################################
#We can also generate the same results as the close form solution
structural_ep_no_added_constraints_test = GIO(A,b,x0)
structural_ep_no_added_constraints_test.GIO_structural_epsilon_setup()
structural_ep_no_added_constraints_test.GIO_structural_epsilon_solve(2)

print("The c vector found by this workflow (not adding epsilon constraints) is:",\
     structural_ep_no_added_constraints_test.c_p[0])

```

    WARNING: Loading a SolverResults object with a warning status into
        model=unknown;
            message from solver=Ipopt 3.11.1\x3a Converged to a locally infeasible
            point. Problem may be infeasible.
    We have infeasibility for constraint= 0 .  Putting FLAG in the  container_for_obj_vals vector
    WARNING: Loading a SolverResults object with a warning status into
        model=unknown;
            message from solver=Ipopt 3.11.1\x3a Converged to a locally infeasible
            point. Problem may be infeasible.
    We have infeasibility for constraint= 1 .  Putting FLAG in the  container_for_obj_vals vector
    WARNING: Loading a SolverResults object with a warning status into
        model=unknown;
            message from solver=Ipopt 3.11.1\x3a Converged to a locally infeasible
            point. Problem may be infeasible.
    We have infeasibility for constraint= 2 .  Putting FLAG in the  container_for_obj_vals vector
    *************************************************
    The c vector found by this workflow is: [[-0.66666667]
     [-0.33333333]]
    The c vector found by this workflow (not adding epsilon constraints) is: [[ 0.4]
     [-0.6]]

```python

```


```python
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression
```

# Linear Regression

### Contents
* [Bivariate Linear Regresion](#Bivariate-Linear-Regresion)
* [Bivariate Linear Regresion with Scikit Learn](#Bivariate-Linear-Regresion-with-Scikit-Learn)
* [Bivariate Linear Regression with Linear Algebra](#Bivariate-Linear-Regression-with-Linear-Algebra)
* [Multiple Regression](#Multiple-Regression)
* [Basic Model Validation](#Basic-Model-Validation)
* [Interpreting Coefficients](#Interpreting-Coefficients)

### Why is it so Important?

* **Popularity** - Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.
* **Interpretability** - Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.
* **Simplicity** - A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients.

### What is it?
Linear Regression is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.

### Synonyms for "y variable"

- Dependent Variable
- Predicted Variable
- Target Variable
- Response Variable
- Outcome Variable
- Measured Variable
- Explained Variable
- Label

### Synonyms for "x variable(s)"

- Indepent Variable
- Explanatory Variable
- Regressor
- Covariate

[Wikipedia Independent and Dependent Variables](https://en.wikipedia.org/wiki/Dependent_and_independent_variables)

# Bivariate Linear Regresion

You must first identify the component you are trying to predict based on the other information you have.

Then, Linear Regression seeks to **estimate** the slope, $m$, and intercept, $b$, values that describe a line of best fit:
\begin{align}
y = mx + b
\end{align}

### Notation

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.

- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.

We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that.

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

# Bivariate Linear Regresion with Scikit-Learn

```python
# Linear Regression Example
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/'
                 'datasets/master/Ice_Cream_Sales.csv')

print(df.shape)
df.head(11)
```

    (11, 2)

      Farenheit
      Dollars

      0
      32
      200

      1
      35
      400

      2
      47
      500

      3
      55
      600

      4
      73
      1100

      5
      78
      1200

      6
      81
      1500

      7
      86
      1700

      8
      89
      1800

      9
      90
      1600

      10
      110
      2000

```python
plt.scatter(df.Farenheit, df.Dollars)
plt.ylabel("Farenheit")
plt.xlabel("Dollars")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.

Before we can pass our x and y values into Scikit-Learn we need our data to be the right shape.

[Linear Regression on Pandas DataFrame using Sci-kit Learn - stackoverflow](https://stackoverflow.com/questions/29934083/linear-regression-on-pandas-dataframe-using-sci-kit-learn)

```python
# Capital X, lowercase y because we typically have more than one x variable
# and our X dataframe will be a matrix rather than a vector.
X = df[['Farenheit']].values
y = df[['Dollars']].values

print('Our data needs to have this shape to be accepted by sklearn:',
      f'X: {X.shape}', f'y: {y.shape}', sep='\n')

model = LinearRegression()
model.fit(X, y)

print("Slope Coefficient:", model.coef_[0])
print("Intercept Value:", model.intercept_)
```

    Our data needs to have this shape to be accepted by sklearn:
    X: (11, 1)
    y: (11, 1)
    Slope Coefficient: [24.68849397]
    Intercept Value: [-596.20648399]

Now we can use these estimated values to plot our line of best fit.

"Predicted values" in statistics are denoted with a caret or "hat" symbol over them:

$\hat{m}, \hat{b}$

```python
m_hat = model.coef_[0]
b_hat = model.intercept_

# Plot the points
plt.scatter(df.Farenheit, df.Dollars)

# Plot the line
y_hat = [m_hat*x + b_hat for x in X]
plt.plot(X, y_hat)

# Graph axes and title
plt.ylabel("Dollars")
plt.xlabel("Farenheit")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

## Making Predictions

Now that we have established a line of best fit that summarizes the relationship that we see in the data, we can use the line of best fit in order to make predictions. How do we make predictions? We plug x values into our equation and see what the predicted y value is.

Suppose that the weather forecast said that the temperature outside tomorrow was going to be 70 degrees Farenheit. What would we predict that our ice cream sales would be?

```python
# Make a single prediction for 70 degrees farenheit
x_value = 70

# Plug our temperature into our equation (model)
prediction = m_hat*x_value + b_hat
print("Prediction", prediction)

# Plot the points
plt.scatter(df.Farenheit, df.Dollars)

# Plot the line
y_hat = [m_hat*x + b_hat for x in X]
plt.plot(X, y_hat)
plt.plot(x_value, prediction, 'ro')

# Graph axes and title
plt.ylabel("Dollars")
plt.xlabel("Farenheit")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

## Predicted Values

You'll notice that our predicted point (red) lies on our line of best fit. The line of best fit represents all of the possible predictions that we could make given some x value. Points that represent our predictions that lie on the line of best fit are called "predicted values" or "fitted values."

## Residual Error

The residual error is the distance between points in our dataset and our regression line. How would we calculate this? Well what if we made a prediction for an x value that we already had in our dataset? Lets make a prediction for 55 degrees Farenheit.

```python
# Make a single prediction for 55 degrees farenheit
point = df.loc[df['Farenheit']==55].values[0]
x_actual = point[0]
y_actual = point[1]

prediction = m_hat * x_actual + b_hat
print("Prediction", prediction)

# Plot the points
plt.scatter(df.Farenheit, df.Dollars)

# Plot the line
y_hat = [m_hat*x + b_hat for x in X]
plt.plot(X, y_hat)
plt.plot(x_value, prediction, 'ro')

# Graph axes and title
plt.ylabel("Farenheit")
plt.xlabel("Dollars")
plt.title("Ice Cream Sales and Temperature")

# Plot the error
plt.arrow(x_actual, y_actual, 0, prediction-600, color='r')
plt.show()
```

The difference between the predicted and measured points is what is called the **residual error** - sometimes just called the **residual**. Some amount of error is unavoidable when making predictions, but we want our line of best fit to minimize this error (make it as small as possible). That's what makes it the line of best fit.

# Bivariate Linear Regression with Linear Algebra

### Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit.

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

This can be cumputed using linear algebra in process known as the "Least Squares Solution:"

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### $y$ Vector

The $y$ vector is simple in that it's just a vector of all of our measured y values from our dataset.

### $X$ Matrix

The $X$ matrix is a little bit more complicated since it needs to contain values that can be used to find both our slope and our intercept values. All that this means is that we have to add an extra column of 1s in addition to our regular columns of X values. Without this we wouldn't be able to computer our intercept value.

\begin{align}
y=mx+b1
\end{align}

### $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

# Bivariate Regression with numpy

```python
df['Ones'] = np.ones(11)
df.head()
```

      Farenheit
      Dollars
      Ones

      0
      32
      200
      1.0

      1
      35
      400
      1.0

      2
      47
      500
      1.0

      3
      55
      600
      1.0

      4
      73
      1100
      1.0

```python
## Get our X matrix and y vector in the correct shape
X = df[['Ones', 'Farenheit']].values
Y = df['Dollars'].values.reshape(-1, 1)
print('X:', X, sep='\n')

X_transpose = X.T
print("\nX Transpose:", X_transpose, sep='\n')

X_transpose_X = np.matmul(X_transpose, X)
print("\nX Transpose X", X_transpose_X, sep='\n')

X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
print("\nX Transpose X Inverse", X_transpose_X_inverse, sep='\n')

X_transpose_Y = np.matmul(X_transpose, Y)
print("\nX Transpose Y", X_transpose_Y, sep='\n')

beta_hat = np.matmul(X_transpose_X_inverse, X_transpose_Y)
print("\nBeta Hat", beta_hat, sep='\n')
```

    X:
    [[  1.  32.]
     [  1.  35.]
     [  1.  47.]
     [  1.  55.]
     [  1.  73.]
     [  1.  78.]
     [  1.  81.]
     [  1.  86.]
     [  1.  89.]
     [  1.  90.]
     [  1. 110.]]

    X Transpose:
    [[  1.   1.   1.   1.   1.   1.   1.   1.   1.   1.   1.]
     [ 32.  35.  47.  55.  73.  78.  81.  86.  89.  90. 110.]]

    X Transpose X
    [[1.1000e+01 7.7600e+02]
     [7.7600e+02 6.0974e+04]]

    X Transpose X Inverse
    [[ 8.89637865e-01 -1.13221862e-02]
     [-1.13221862e-02  1.60494908e-04]]

    X Transpose Y
    [[  12600.]
     [1042700.]]

    Beta Hat
    [[-596.20648399]
     [  24.68849397]]

# Multiple Regression
Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression - NOT MULTIVARIATE!

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of  𝑥  values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.

# Multiple Regression with SciKit-Learn

Predicting **systolic blood** pressure from **age** and **weight**:

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon $

```python
import pandas as pd

df = pd.read_excel('https://college.cengage.com/mathematics/brase/'
                   'understandable_statistics/7e/students/datasets/'
                   'mlr/excel/mlr02.xls', names=['y', 'age', 'weight'])
print(df.shape)
df.head()
```

    *** No CODEPAGE record, no encoding_override: will use 'ascii'
    (11, 3)

      y
      age
      weight

      0
      132
      52
      173

      1
      143
      59
      184

      2
      153
      67
      194

      3
      162
      73
      211

      4
      154
      64
      196

```python
from sklearn.linear_model import LinearRegression

# The Scikit-Learn code is the same every time.
y = df[['y']].values
X = df[['age','weight']].values

print(y.shape)
print(X.shape)

model = LinearRegression()
model.fit(X, y)

beta_0 = model.intercept_
beta_i = model.coef_[0]

print("Slope Coefficients: ", beta_i)
print("Intercept Value: ", beta_0)
```

    (11, 1)
    (11, 2)
    Slope Coefficients:  [0.86141469 0.3348592 ]
    Intercept Value:  [30.99410295]

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

# Plot Points on 3D plot
ax.scatter(df['age'], df['weight'], df['y'], c='red')

# Set axes to size of scatter data
x1 = np.array(ax.get_xlim())
y1 = np.array(ax.get_ylim())
xx, yy = np.meshgrid(x1,y1)
zz = beta_i[0]*xx + beta_i[1]*yy + beta_0[0]

# Plot Plane
plt3d = plt.gca(projection='3d')

# Add Opacity to Plane
plt3d.plot_surface(xx, yy, zz,alpha=0.5)
plt.title("Regression Plane")

plt.show()
```

# Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

$\hat{\beta} = \frac{Cov(x,y)}{Var(y)}$

Going back to the two equations for the two models that we have estimated so far, lets replace their beta values with their actual values to see if we can make sense of how to interpret these beta coefficients.

### Bivariate Model

$y_i = \beta_0 + \beta_1temperature + \epsilon$

$sales_i = -596.2 + 24.69temperature + \epsilon$

What might $\beta_0$ in this model represent? It represents the level of sales that we would have if temperature were 0. Since this is negative one way of interpreting it is that it's so cold outside that you would have to pay people to eat ice cream. A more appropriate interpretation is probably that the ice cream store owner should close his store down long before the temperature reaches 0 degrees farenheit (-17.7 celsius). The owner can compare his predicted sales with his costs of doing business to know how warm the weather has to get before he should open his store.

What might the $beta_1$ in this model represent? it represents the increase in sales for each degree of temperature increase. For every degree that the temperature goes up outside he has $25 more in sales.

### Multiple Regression Model

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon$

$BloodPressure_i = 30.99+ .86age_i + .33weight_i + \epsilon$

The interpretation of coefficients in this example are similar. The intercept value repesents the blood pressure a person would have if they were 0 years old and weighed 0 pounds. This not a super useful interpretation. If we look at our data it is unlikely that we have any measurements like these in the dataset. This means that our interpretation of our intercept likely comes from extrapolating the regression line (plane). Coefficients having straightforward interpretations is a strength of linear regression if we're careful about extrapolation and only interpreting our data within the context that it was gathered.

The interpretation of our other coefficients can be a useful indicator for how much a person similar to those in our dataset's blood pressure will go up on average with each additional year of age and pound of weight.

# R Squared:  $R^2$

$R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely.

# Basic Model Validation

One of the downsides of relying on $R^2$ too much is that although it tells you when you're fitting the data well, it doesn't tell you when you're *overfitting* the data. The best way to tell if you're overfitting the data is to get some data that your model hasn't seen yet, and evaluate how your predictions do. This is essentially what "model validation" is.

### Steps of Basic Model Validation

1. Split your labeled data into two halves.
2. Fit your linear regression model to one of the two halves. (We'll call this the training dataset)
3. Run the second half of your data through your model and make predictions. (On the test dataset)
4. Compare your predictions with the true values via some measure of "accuracy." (Root Mean Squared Error)

# Model Validation with SciKit-Learn

```python
# http://lib.stat.cmu.edu/datasets/boston

import pandas as pd
from sklearn.datasets import load_boston
```

```
Variables in order:
 CRIM     per capita crime rate by town
 ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
 INDUS    proportion of non-retail business acres per town
 CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
 NOX      nitric oxides concentration (parts per 10 million)
 RM       average number of rooms per dwelling
 AGE      proportion of owner-occupied units built prior to 1940
 DIS      weighted distances to five Boston employment centres
 RAD      index of accessibility to radial highways
 TAX      full-value property-tax rate per $10,000
 PTRATIO  pupil-teacher ratio by town
 B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
 LSTAT    % lower status of the population
 MEDV     Median value of owner-occupied homes in $1000's
```

```python
# Our target variable is MEDV
# MEDV: Median value of owner-occupied homes in $1000s

boston_dataset = load_boston()

df = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
df['MEDV'] = boston_dataset.target
print(df.shape)
df.head()
```

    (506, 14)

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      MEDV

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

```python
import seaborn as sns

sns.set(style="ticks", color_codes=True)

# Generate a list of column headers not including price
x_columns = df.columns.drop('MEDV')

# Only plot the scatterplot of x variables with our y variable
fig = sns.pairplot(data=df, y_vars=['MEDV'], x_vars=x_columns)
```

```python
from sklearn.model_selection import train_test_split

y = df['MEDV']
X = df.drop('MEDV', axis=1)

X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.3)

print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
```

    (354, 13)
    (152, 13)
    (354,)
    (152,)

```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

model = LinearRegression()
model.fit(X_train, Y_train)

y_test_predict = model.predict(X_test)
RMSE = np.sqrt(mean_squared_error(Y_test, y_test_predict))
R2 = r2_score(Y_test, y_test_predict)

print("RMSE:", RMSE)
print("R^2:", R2)
```

    RMSE: 5.516101900280473
    R^2: 0.6526542967095016

```python
df.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      MEDV

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.613524
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.601545
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.677083
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000


# Notebook 4: Linear Regression (Ising)

## Learning Goal
Let us now apply linear regression to an example that is familiar from Statistical Mechanics: the Ising model. The goal of this notebook is to revisit the concepts of in-sample and out-of-sample errors, as well as $L2$- and $L1$-regularization, in an example that is more intuitive to physicists.

## Overview
Consider the 1D Ising model with nearest-neighbor interactions

$$H[\boldsymbol{S}]=-J\sum_{j=1}^L S_{j}S_{j+1}$$

on a chain of length $L$ with periodic boundary conditions and $S_j=\pm 1$ Ising spin variables. In one dimension, this paradigmatic model has no phase transition at finite temperature.

### Exercises (optional): ###
We invite the reader who is unfamiliar with the property of the Ising model to solve the following problems.

 Compute the partition function of the Ising model in one dimension at inverse temperature $\beta$ when $L\rightarrow\infty$ (thermodynamic limit):
    $$Z=\sum_S \exp(-\beta H[S]).$$
Here the sum is carried over all $2^L$ spin configurations.

 Compute the model's magnetization $M=\langle\sum_i S_i\rangle$ in the same limit ($L\rightarrow\infty$). The expectation is taken with respect to the Boltzmann distribution:
    $$p(S)=\frac{\exp(-\beta H[S])}{Z}$$

 How does $M$ behave as a function of the temperature $T=\beta^{-1}$?

For a more detailed introduction we refer the reader to consult one of the many textbooks on the subject (see for instance Goldenfeld, Lubensky, Baxter , etc.).

### Learning the Ising model ###

Suppose your boss set $J=1$, drew a large number of spin configurations, and computed their Ising energies. Then, without telling you about the above Hamiltonian, he or she handed you a data set of $i=1\ldots n$ points of the form $\{(H[\boldsymbol{S}^i],\boldsymbol{S}^i)\}$. Your task is to learn the Hamiltonian using Linear regression techniques.

```python
import numpy as np
import scipy.sparse as sp
np.random.seed(12)

import warnings
# Comment this to turn on warnings
warnings.filterwarnings('ignore')

### define Ising model aprams
# system size
L=40

# create 10000 random Ising states
states=np.random.choice([-1, 1], size=(10000,L))

def ising_energies(states):
    """
    This function calculates the energies of the states in the nn Ising Hamiltonian
    """
    L = states.shape[1]
    J = np.zeros((L, L),)
    for i in range(L):
        J[i,(i+1)%L]=-1.0 # interaction between nearest-neighbors

    # compute energies
    E = np.einsum('...i,ij,...j->...',states,J,states)

    return E
# calculate Ising energies
energies=ising_energies(states)
```

## Recasting the problem as a Linear Regression
First of all, we have to decide on a model class (possible Hamiltonians) we use to fit the data. In the absence of any prior knowledge, one sensible choice is the all-to-all Ising model

$$
H_\mathrm{model}[\boldsymbol{S}^i] = - \sum_{j=1}^L \sum_{k=1}^L J_{j,k}S_{j}^iS_{k}^i.
$$
Notice that this model is uniquely defined by the non-local coupling strengths $J_{jk}$ which we want to learn. Importantly, this model is linear in ${\mathbf J}$ which makes it possible to use linear regression.

To apply linear regression, we would like to recast this model in the form
$$
H_\mathrm{model}^i \equiv \mathbf{X}^i \cdot \mathbf{J},
$$

where the vectors $\mathbf{X}^i$ represent all two-body interactions $\{S_{j}^iS_{k}^i \}_{j,k=1}^L$, and the index $i$ runs over the samples in the data set. To make the analogy complete, we can also represent the dot product by a single index $p = \{j,k\}$, i.e. $\mathbf{X}^i \cdot \mathbf{J}=X^i_pJ_p$. Note that the regression model does not include the minus sign, so we expect to learn negative $J$'s.

```python
# reshape Ising states into RL samples: S_iS_j --> X_p
states=np.einsum('...i,...j->...ij', states, states)
shape=states.shape
states=states.reshape((shape[0],shape[1]*shape[2]))
# build final data set
Data=[states,energies]
```

## Numerical Experiments

As we already mentioned a few times in the review, learning is not fitting: the subtle difference is that once we fit the data to obtain a candidate model, we expect it to generalize to unseen data not used for the fitting procedure. For this reason, we begin by specifying a training and test data sets

```python
# define number of samples
n_samples=400
# define train and test data sets
X_train=Data[0][:n_samples]
Y_train=Data[1][:n_samples] #+ np.random.normal(0,4.0,size=X_train.shape[0])
X_test=Data[0][n_samples:3*n_samples//2]
Y_test=Data[1][n_samples:3*n_samples//2] #+ np.random.normal(0,4.0,size=X_test.shape[0])
```

# Evaluating the performance:  coefficient of determination $R^2$

In what follows the model performance (in-sample and out-of-sample) is evaluated using the so-called coefficient of determination, which is given by:
\begin{align}
R^2 &= \left(1-\frac{u}{v}\right),\\
u&=(y_{pred}-y_{true})^2\\
v&=(y_{true}-\langle y_{true}\rangle)^2
\end{align}

The best possible score is 1.0 but it can also be negative. A constant model that always predicts the expected value of $y$, $\langle y_{true}\rangle$, disregarding the input features, would get a $R^2$ score of 0.

## Applying OLS, Ridge regression and LASSO:

```python
from sklearn import linear_model
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
import seaborn
%matplotlib inline

# set up Lasso and Ridge Regression models
leastsq=linear_model.LinearRegression()
ridge=linear_model.Ridge()
lasso = linear_model.Lasso()

# define error lists
train_errors_leastsq = []
test_errors_leastsq = []

train_errors_ridge = []
test_errors_ridge = []

train_errors_lasso = []
test_errors_lasso = []

# set regularisation strength values
lmbdas = np.logspace(-4, 5, 10)

#Initialize coeffficients for ridge regression and Lasso
coefs_leastsq = []
coefs_ridge = []
coefs_lasso=[]

for lmbda in lmbdas:

    ### ordinary least squares
    leastsq.fit(X_train, Y_train) # fit model
    coefs_leastsq.append(leastsq.coef_) # store weights
    # use the coefficient of determination R^2 as the performance of prediction.
    train_errors_leastsq.append(leastsq.score(X_train, Y_train))
    test_errors_leastsq.append(leastsq.score(X_test,Y_test))

    ### apply RIDGE regression
    ridge.set_params(alpha=lmbda) # set regularisation parameter
    ridge.fit(X_train, Y_train) # fit model
    coefs_ridge.append(ridge.coef_) # store weights
    # use the coefficient of determination R^2 as the performance of prediction.
    train_errors_ridge.append(ridge.score(X_train, Y_train))
    test_errors_ridge.append(ridge.score(X_test,Y_test))

    ### apply LASSO regression
    lasso.set_params(alpha=lmbda) # set regularisation parameter
    lasso.fit(X_train, Y_train) # fit model
    coefs_lasso.append(lasso.coef_) # store weights
    # use the coefficient of determination R^2 as the performance of prediction.
    train_errors_lasso.append(lasso.score(X_train, Y_train))
    test_errors_lasso.append(lasso.score(X_test,Y_test))

    ### plot Ising interaction J
    J_leastsq=np.array(leastsq.coef_).reshape((L,L))
    J_ridge=np.array(ridge.coef_).reshape((L,L))
    J_lasso=np.array(lasso.coef_).reshape((L,L))

    cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')

    fig, axarr = plt.subplots(nrows=1, ncols=3)

    axarr[0].imshow(J_leastsq,**cmap_args)
    axarr[0].set_title('OLS \n Train$=%.3f$, Test$=%.3f$'%(train_errors_leastsq[-1], test_errors_leastsq[-1]),fontsize=16)
    axarr[0].tick_params(labelsize=16)

    axarr[1].imshow(J_ridge,**cmap_args)
    axarr[1].set_title('Ridge $\lambda=%.4f$\n Train$=%.3f$, Test$=%.3f$' %(lmbda,train_errors_ridge[-1],test_errors_ridge[-1]),fontsize=16)
    axarr[1].tick_params(labelsize=16)

    im=axarr[2].imshow(J_lasso,**cmap_args)
    axarr[2].set_title('LASSO $\lambda=%.4f$\n Train$=%.3f$, Test$=%.3f$' %(lmbda,train_errors_lasso[-1],test_errors_lasso[-1]),fontsize=16)
    axarr[2].tick_params(labelsize=16)

    divider = make_axes_locatable(axarr[2])
    cax = divider.append_axes("right", size="5%", pad=0.05, add_to_figure=True)
    cbar=fig.colorbar(im, cax=cax)

    cbar.ax.set_yticklabels(np.arange(-1.0, 1.0+0.25, 0.25),fontsize=14)
    cbar.set_label('$J_{i,j}$',labelpad=15, y=0.5,fontsize=20,rotation=0)

    fig.subplots_adjust(right=2.0)

    plt.show()
```

To quantify learning, we also plot the in-sample and out-of-sample errors

```python
# Plot our performance on both the training and test data
plt.semilogx(lmbdas, train_errors_leastsq, 'b',label='Train (OLS)')
plt.semilogx(lmbdas, test_errors_leastsq,'--b',label='Test (OLS)')
plt.semilogx(lmbdas, train_errors_ridge,'r',label='Train (Ridge)',linewidth=1)
plt.semilogx(lmbdas, test_errors_ridge,'--r',label='Test (Ridge)',linewidth=1)
plt.semilogx(lmbdas, train_errors_lasso, 'g',label='Train (LASSO)')
plt.semilogx(lmbdas, test_errors_lasso, '--g',label='Test (LASSO)')

fig = plt.gcf()
fig.set_size_inches(10.0, 6.0)

#plt.vlines(alpha_optim, plt.ylim()[0], np.max(test_errors), color='k',
#           linewidth=3, label='Optimum on test')
plt.legend(loc='lower left',fontsize=16)
plt.ylim([-0.1, 1.1])
plt.xlim([min(lmbdas), max(lmbdas)])
plt.xlabel(r'$\lambda$',fontsize=16)
plt.ylabel('Performance',fontsize=16)
plt.tick_params(labelsize=16)
plt.show()
```

## Understanding the results

Let us make a few remarks:

(i) the inverse (see [Scikit documentation](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model)) regularization parameter $\lambda$ affects the Ridge and LASSO regressions at scales, separated by a few orders of magnitude. Notice that this is different for the data considered in Notebook 3 __Section VI: Linear Regression (Diabetes)__. Therefore, it is considered good practice to always check the performance for the given model and data with $\lambda$ over multiple decades.

(ii) at $\lambda\to 0$ and $\lambda\to\infty$, all three models overfit the data, as can be seen from the deviation of the test errors from unity (dashed lines), while the training curves stay at unity.

(iii) While the OLS and Ridge regression test curves are monotonic, the LASSO test curve is not -- suggesting the optimal LASSO regularization parameter is $\lambda\approx 10^{-2}$. At this sweet spot, the Ising interaction weights ${\bf J}$ contain only nearest-neighbor terms (as did the model the data was generated from).

__Gauge degrees of freedom__: recall that the uniform nearest-neighbor interactions strength $J_{j,k}=J$ which we used to generate the data was set to unity, $J=1$. Moreover, $J_{j,k}$ was NOT defined to be symmetric (we only used the $J_{j,j+1}$ but never the $J_{j,j-1}$ elements). The colorbar on the matrix elements plot above suggest that the OLS and Ridge regression learn uniform symmetric weights $J=-0.5$. There is no mystery since this amounts to taking into account both the $J_{j,j+1}$ and the $J_{j,j-1}$ terms, and the weights are distributed symmetrically between them. LASSO, on the other hand, can break this symmetry (see matrix elements plots for $\lambda=0.001$ and $\lambda=0.01$). Thus, we see how different regularization schemes can lead to learning equivalent models but in different gauges. Any information we have about the symmetry of the unknown model that generated the data has to be reflected in the definition of the model and the regularization chosen.

### Exercises: ###

 Plot a histogram of the distribution of the components of ${\bf J}$ at different values of the number of training sample (one can go up to $2\times 10^4$). What happens with the sampling noise as the number of samples is increased/decreased for the three types of regression considered? How do the matrix elements plots above change?

 Try to learn the underlying model of the data, assuming it lies within the class of one-body Hamiltonians, i.e. make the ansatz
$$H_\mathrm{model}[\boldsymbol{S}^i] = \sum_{j=1}^L h_jS_{j}$$
for some unknown field $h_j$. How well can you explain the data? How well does the model generalize? Study these problems by playing with the size of the data set. Try out all three regression models and determine which one does the best. What is the relationship to Mean-Field Theory of this model?

```python

```


# Packages

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import yahoofinancials
import seaborn as sn
from scipy.stats import anderson,kstest,shapiro, probplot

from IPython.core.display import HTML
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

*Discrete Return*
Misal $S(t)$ adalah di saham pada saat $t$ maka *return* diperoleh sebagai berikut
\begin{equation}
\frac{S(t+\Delta t)-S(t)}{S(t)}
\end{equation}

```python
stock = 'TLKM.JK'
df = yf.Ticker(stock).history(period = '1y')
df['return'] = df['Close'].pct_change()
df['return']
```

    Date
    2021-02-11         NaN
    2021-02-15    0.009404
    2021-02-16   -0.003106
    2021-02-17   -0.018692
    2021-02-18    0.009524
                    ...
    2022-02-07    0.014184
    2022-02-08   -0.013986
    2022-02-09    0.011820
    2022-02-10    0.042056
    2022-02-11   -0.002242
    Name: return, Length: 248, dtype: float64

```python
df['return'].plot()
plt.ylabel('% Change')
plt.title('Discrete Return'+ stock)
plt.show()
```

Log Return didefinisikan sebagai berikut :
\begin{equation}
\ln\left(\frac{S(t+\Delta t)}{S(t)}\right)
\end{equation}

```python
df['log_return'] = np.log(df['Close']) - np.log(df['Close'].shift(1))
df['log_return'].plot()
plt.ylabel('Log Return')
plt.title('Log Return ' + stock)
plt.show()
```

Di kalkulus dipelajari bahwa
\begin{align*}
\ln(1+x) = x - \frac{x^2}{2} + \frac{x^3}{3} - \dots
\end{align*}
Sehingga dapat ditinjau pula persamaan
\begin{align*}
\ln\left(\frac{S(t+\Delta t)}{S(t)}\right)= \ln\left(\frac{S(t+\Delta t)-S(t)+S(t)}{S(t)}\right) = \ln\left(1+\frac{S(t+\Delta t)-S(t)}{S(t)}\right) \approx \frac{S(t+\Delta t)-S(t)}{S(t)}
\end{align*}
untuk suatu nilai $\frac{S(t+\Delta t) - S(t)}{S(t)}$ yang kecil

# Distribusi dari *Return* dan *Log Return*

```python
ax1 = plt.subplot(121)
df['return'].hist(density = True)
sn.kdeplot(df['return'])
ax2 = plt.subplot(122)
probplot(df['return'].dropna(), plot= plt)

plt.show()
```

```python
print('Return mean :', np.round(df['return'].mean(),6))
print('Return variance :', np.round(df.var()['return'],6))
```

    Return mean : 0.001182
    Return variance : 0.000323

```python
ax1 = plt.subplot(121)
df['log_return'].hist(density = True)
sn.kdeplot(df['log_return'])
ax2 = plt.subplot(122)
probplot(df['log_return'].dropna(), plot= plt)

plt.show()
```

```python
print('Return mean :', np.round(df['log_return'].mean(),6))
print('Return variance :', np.round(df.var()['log_return'],6))
```

    Return mean : 0.001023
    Return variance : 0.000316


```python
import numpy as np
import scipy.integrate as integrate
import scipy.special as special
import sympy as sym
from scipy.misc import derivative
```

```python
arquivo_1 = np.loadtxt("./amostra1.dat")
y = np.linspace(-2, 4, arquivo_1.size)
```

```python
'''

Letra A)

Podemos resolver integrais de duas formas, numericamente ou simbolicamente. No caso numérico solucionamos uma
integral definida, isto é, que tem definido o intervalo de integração, desta forma o resultado é um número.
Existem muitos métodos numéricos baseados em como subdividir o intervalo em polígonos a serem somados e
alguns utilizando MMC. Já no caso simbólico nós podemos resolver integrais indefinidas, logo encontrar a
função que define o resultado da integral. Utilize o numpy e o scipy para calcular a integral numérica do
conjunto de pontos interpolados entre os valores x=-2 e x=4, com um total de 106 pontos no intervalo, gerados
pelas funções Spline do exercício 1. Para o caso simbólico, utilize o sympy para calcular a integral
indefinida ∫arctan (x2)dx

'''
print("Usando numpy : ", np.trapz(y, x = [-2, 4]))

x = sym.Symbol('x')
print("Calculando a integral : ", sym.integrate(sym.atan(x**2), x))

```

    Usando numpy :  60006.0
    Calculando a integral :  x*atan(x**2) - sqrt(2)*log(4*x**2 - 4*sqrt(2)*x + 4)/4 + sqrt(2)*log(4*x**2 + 4*sqrt(2)*x + 4)/4 - sqrt(2)*atan(x**2)/2 - sqrt(2)*atan(sqrt(2)*x + 1)

```python
'''

Letra B)
Encontre a derivada da função arctan (x2) utilizando o sympy e o scipy.

'''

def f(x):
    return np.arctan(x**2)

print("Calculando a derivada usando Sympy \n : ", sym.diff(sym.atan(x**2), x))

print("Calculando a derivada usando Scipy \n : ", derivative(f, 1.0, dx=1e-6))
```

    Calculando a derivada usando Sympy
     :  2*x/(x**4 + 1)
    Calculando a derivada usando Scipy
     :  0.9999999999732445

```python

```

Métodos Computacionais em Astronomia/lista6/.ipynb_checkpoints/questao2-checkpoint.ipynb


### Scalar factor of the Universe

```python
import numpy as np
from pylab import *
from scipy.integrate import odeint
```

In this notebook, we solve for the scale factor of the Universe based on the Standard Model of Cosmology, often called $\Lambda$CDM model. We take numerical values from the following:

[1] Lyth, D. H., & Liddle, A. R. (2009). The primordial density perturbation: Cosmology, inflation and the origin of structure. Cambridge University Press.

The Friedmann equation is given by
\begin{equation}
H(a)^2 = H_0^2 \left( \Omega_{r0} a^{-4} + \Omega_{m0} a^{-3} + \Omega_{\Lambda 0} \right)
\end{equation}
where $H$ is the Hubble parameter, and $\Omega_{r0}$, $\Omega_{m0}$, and $\Omega_{\Lambda 0}$ are the radiation, matter, and the vacuum (cosmological constant) energy densities, respectively, today. We refer to the following values tabulated in appendix B of Ref. [1]:
\begin{eqnarray}
\Omega_{r0} &=& 8.47 \times 10^{-5} \\
\Omega_{m0} &=& 0.276 \\
\Omega_{\Lambda 0} &=& 1 - \Omega_{r0} - \Omega_{m0} \\
H_0 &=& 70 \ \text{km} / \text{s} / \text{Mpc}  .
\end{eqnarray}

Noting that the Hubble parameter $H$ is related to the scale factor $a$ as
\begin{equation}
H = \frac{\dot{a}}{a}
\end{equation}
where an overdot denotes derivative with respect to the comoving time $t$, then the Friedmann equation can be written down as
\begin{equation}
\dot{a} = a H_0 \sqrt{\left( \Omega_{r0} a^{-4} + \Omega_{m0} a^{-3} + \Omega_{\Lambda 0} \right)} .
\end{equation}
This is the expression that we input into $odeint$. In the following code, we input this differential equation.

```python
# here we setup the constants and the ode

omega_r = 8.47e-5
omega_m = 0.276
omega_vac = 1. - omega_r - omega_m
H_0 = 1. # rescaled to unity for efficient numerics

a_0 = 1. # initial condition on the scale factor today

def f(y, t):
    return y*H_0*np.sqrt( omega_r*(y**(-4.)) + omega_m*(y**(-3.)) + omega_vac )

time_points = np.linspace(1., 0.01, 100)
```

Note that by setting $H_0$ to unity, we work in units where time is measured in $H_0^{-1} \sim 14$ billion years. Also, we are integrating backwards in time, starting from the present.

With this said, we obtain the scale factor $a(t)$ of the Universe as follows.

```python
rc('xtick', labelsize = 20) # for the tick marks
rc('ytick', labelsize = 20)

a_lcdm = odeint(f, a_0, time_points) # odeint does its job

plot(time_points, a_lcdm, 'r-', linewidth = 3.0)
ylim(0.01, 1) # aesthetics
xlim(0.01, 1)
xlabel(r'time (14 byr)', fontsize = 20)
ylabel('scale factor', fontsize = 20)
show()
```

So, yeah. This is the scale factor $a(t)$ of the Universe. From this, one could think of the size of the universe as $V(t) \sim a(t)^3$.

The expansion history can be divided into three eras (1) radiation (2) matter and (3) dark energy era, depending on the Universe's energy content. The first era, which comes just right after the Big Bang and primordial inflation, is radiation domination, where $a(t) \sim t^{1/2}$. Then comes matter era, as radiation cools down much faster than matter, during which $a(t) \sim t^{2/3}$. Finally, and today, after both radiation and matter domination, comes dark energy era, where the Universe is dominated by an invisible, negative pressure fluid that sources the observed cosmic acceleration.


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function
from __future__ import absolute_import

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Interpolation

There are times when you have estimates for the values of a function for specific inputs. The values of the function may be obtained in a variety of ways either through experiment or through the use of other approximation techniques.  Our goal in this chapter is to explore techniques that allow us to determine a new function whose values match the known observations at a set of predetermined input values. We first formally define the term we will use to describe the process.

**Definition:**  Given a discrete set of values $y_i$ at locations $x_i$, an *interpolant* is a (piece-wise) continuous function $f(x)$ that passes exactly through the data (*i.e.* $f(x_i) = y_i$).

**Example 0** The linear polynomial

$$
    P_1(x) = 2(x-1)+3
$$

interpolates the coordinates $(1,3)$ and $(3,7)$.

In general a polynomial of degree $N$ can be used to interpolate $N+1$ data points. There are many different kinds of functions to use to interpolate values, but here we focus on polynomials.

## Applications

 - Data filling
 - Function approximation
 - Fundamental component of other algorithms
   - Root finding (secant method)
   - Optimization, minima/maxima (successive parabolic interpolation)
   - Numerical integration and differentiation
   - The Finite Element Method

## Polynomial Interpolation

**Theorem:**  There is a *unique* polynomial of degree $N$, $P_N(x)$, that passes exactly through $N + 1$ values $y_1, y_2, \ldots, y_N, y_{N+1}$ at *distinct* points $x_1, x_2, \ldots, x_N, x_{N+1}$.

Consequence of the number of unknowns in $P_N(x)$.

#### Example 1: 2 Points

Given points are $(x_0, y_0)$ and $(x_1, y_1)$ which will lead to a line:

Define $P_1(x) = p_1 x + p_0$ and use the two points to find $p_0$ and $p_1$:

We first note that we have two equations and two unknowns. The two equations can be found by assuming the function $P_1(x)$ interpolates the two data points
$$
    \begin{align}
       y_0 &= p_1 x_0 + p_0, \\
       y_1 &= p_1 x_1 + p_0.
    \end{align}
$$
In this example we will solve the first equation for $p_0$, substitute the result into the second equation, and then solve for $p_1$.

$$y_0 = p_1 x_0 + p_0 \quad \Rightarrow \quad p_0 = y_0 - p_1 x_0$$

$$\begin{aligned}
    y_1 &= p_1 x_1 + p_0 & \Rightarrow \\
    y_1 &= p_1 x_1 + y_0 - p_1 x_0 & \Rightarrow \\
    p_1 &= \frac{y_1 - y_0}{x_1 - x_0} & \Rightarrow \\
    p_0 &= y_0 - \frac{y_1 - y_0}{x_1 - x_0} x_0 &
\end{aligned}$$

$$P_1(x) = \frac{y_1 - y_0}{x_1 - x_0} x + y_0 - \frac{y_1 - y_0}{x_1 - x_0} x_0 = \frac{y_1 - y_0}{x_1 - x_0} (x - x_0) + y_0$$

#### Example 2:  3 Points

Given points are $(x_0, y_0)$, $(x_1, y_1)$, and $(x_2, y_2)$ which will lead to quadratic polynomial:

Define $P_2(x) = p_0 x^2 + p_1 x + p_2$ leading to the equations

$$y_0 = p_2 x_0^2 + p_1 x_0 + p_0$$

$$y_1 = p_2 x_1^2 + p_1 x_1 + p_0$$

$$y_2 = p_2 x_2^2 + p_1 x_2 + p_0$$

This gets complicated quickly! Note, we have three equations and three unknowns, and the previous system is a linear system of three equations.
and in general, the problem will reduce to a linear system
$$
    A(\mathbf{x})\mathbf{p} = \mathbf{y}
$$
A more general approach to solving the system will be explored later, but first it is important to determine whether or not the system even has a solution.

### Proof - Uniqueness of Polynomial Interpolants

Let

$$\mathcal{P}_N(x) = \sum^N_{n=0} p_n x^n $$
or
$$\mathcal{P}_N(x) = p_0 + p_1 x + \cdots + p_{N - 1} x^{N - 1} + p_{N} x^N$$

and require $\mathcal{P}_N(x_i) = y_i$ for $i=0,1,\ldots,N$ and $x_i \neq x_j ~~~ \forall i,j$.

### Preliminaries: Monomial Basis

We can think of  $\mathcal{P}_N(x) = \sum^N_{n=0} p_n x^n$ as a polynomial,  or more fundamentally as a *linear combination* of a set of simpler functions, the monomials

$$1, x, x^2, x^3, \ldots, x^{N-1}, x^N$$

with weights

$$p_0, p_1, p_2, p_3, \ldots, p_{N-1}, \text{and } p_N$$

respectively.

### Linear independence of the Monomials

The monomials, form a *linearly independent* set of functions such that no monomial $x^n$ can be written as a linear combination of any other monomial.  We can see this graphically, for the first few monomials

```python
x = numpy.linspace(-1,1,100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
for n in range(4):
    axes.plot(x,x**n,label='$x^{}$'.format(n))
axes.set_xlabel('x')
axes.grid()
axes.legend(loc='best')
axes.set_title('The First 4 Monomials')
plt.show()
```

But more fundamentally. A set of functions is linearly independent if the only linear combination that add to form the zero function, e.g.

$$
    P_N(x) = p_0 1 + p_1 x + p_2 x^2 + \ldots + p_n x^n = 0
$$

is if all the coefficients $p_i = 0$, $\forall i=0,\ldots N$

**Theorem**:  The monomials $x^0,\ldots, x^n$ are linear independent.

**Proof**: consider $P_N(x) = 0$ for all $x$.  Since the polynomials (and monomials) are differentiable at least $n$ times,  differentiate $n$ times to yield
$$
    P^{(n)}_N(x) = n!p_n = 0
$$
which implies $p_n=0$.

Using  this result and differentiating $n-1$ times shows $p_{n-1}=0$, which by induction gives all $p_i = 0$.

Put another way,  the only $n$th degree polynomial that is zero everywhere is if all coefficients are zero.

#### The Fundamental theorem of algebra

Every $n$th degree polynomial has exactly  $n$ complex roots, i.e.

$$
    P_N(x) = (x - a_1)(x - a_2)\ldots(x - a_n)
$$
for $a_i\in \mathbb{C}$.  Therefore, a non-trivial $n$th order polynomial can only be zero at $n$ points.

### Proof - Uniqueness of Polynomial Interpolants

Let

$$\mathcal{P}_N(x) = \sum^N_{n=0} p_n x^n $$

**interpolate** the $N+1$ points $y_i$ at $x_i$.

i.e.
$$\mathcal{P}_N(x_i) = y_i,\quad \mathrm{for}\quad i=0,1,\ldots,N$
$$
and $x_i \neq x_j ~~~ \forall i,j$.

Assume there exists another polynomial

$$Q_N(x) = \sum^N_{n=0} q_n x^n$$

that passes through the same set of points such that $Q_N(x_i) = y_i$.  Now compute $T_N(x) = \mathcal{P}_N(x) - Q_N(x)$:

Now, by construction,  $T_N(x_i) = 0$ which implies that it is equal to zero at $n+1$ points.  However,

$$T_N(x) = \mathcal{P}_N(x) - Q_N(x) = \sum^N_{n=0} p_n x^n - q_n x^n = \sum^N_{n=0} (p_n - q_n) x^n$$

is a $n$th order polynomial which has at most $n$ real roots.  The only way to reconcile this is if T_n(x) = 0, for all $x$, and therefore $p_n - q_n = 0$ individually and therefore $\mathcal{P}_N(x) = Q_N(x)$.

#### Example 3:  Monomial Basis

Consider $\mathcal{P}_3(x) = p_0 + p_1 x + p_2 x^2 + p_3 x^3$ with the four data points $(x_i, y_i), ~~ i = 0,1,2,3$.  We have four equations and four unknowns as expected:

$$\mathcal{P}_3(x_0) = p_0 + p_1 x_0 + p_2 x_0^2 + p_3 x_0^3 = y_0$$

$$\mathcal{P}_3(x_1) = p_0 + p_1 x_1 + p_2 x_1^2 + p_3 x_1^3 = y_1$$

$$\mathcal{P}_3(x_2) = p_0 + p_1 x_2 + p_2 x_2^2 + p_3 x_2^3 = y_2$$

$$\mathcal{P}_3(x_3) = p_0 + p_1 x_3 + p_2 x_3^2 + p_3 x_3^3 = y_3$$

Lets rewrite these as a matrix equation:

$$\mathbf{x} = \begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ x_3 \end{bmatrix} \quad \mathbf{y} = \begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix} \quad \mathbf{p} = \begin{bmatrix} p_0 \\ p_1 \\ p_2 \\ p_3 \end{bmatrix}$$

When we write the system in matrix/vector form the matrix that arises is called *Vandermonde* matrix:

$$
V = \begin{bmatrix}
     1 & x_0 & x_0^2 & x_0^3 \\
     1 & x_1 & x_1^2 & x_1^3 \\
     1 & x_2 & x_2^2 & x_2^3 \\
     1 & x_3 & x_3^2 & x_3^3
\end{bmatrix}.
$$

We can now write the system of linear equations as $V \mathbf{p} = \mathbf{y}$:

$$\begin{bmatrix}
     1 & x_0 & x_0^2 & x_0^3 \\
     1 & x_1 & x_1^2 & x_1^3 \\
     1 & x_2 & x_2^2 & x_2^3 \\
     1 & x_3 & x_3^2 & x_3^3
\end{bmatrix} \begin{bmatrix} p_0 \\ p_1 \\ p_2 \\ p_3 \end{bmatrix} = \begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix}.$$

**Note**: the columns  of $V$ are simply the monomial functions sampled at the discrete points $x_i$.  Because the monomials are linearly independent, so are the columns of $V$

$$\begin{bmatrix}
     1 & x_0 & x_0^2 & x_0^3 \\
     1 & x_1 & x_1^2 & x_1^3 \\
     1 & x_2 & x_2^2 & x_2^3 \\
     1 & x_3 & x_3^2 & x_3^3
\end{bmatrix} \begin{bmatrix} p_0 \\ p_1 \\ p_2 \\ p_3 \end{bmatrix} = \begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix}$$

- What happens if we have redundant data?  Either $(x_i, y_i)$ is repeated or for one $i$ we have two values of $y$.
- What if we have more points then the order of polynomial we want?
- How does this relate to solving the above linear system of equations?

Vandermonde matrices in general are defined as

$$V = \begin{bmatrix}
    1      & x_0    & x_0^2  & \cdots & x_0^N  \\
    1      & x_1    & x_1^2  & \cdots & x_1^N  \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    1      & x_m    & x_m^2  & \cdots & x_m^N  \\
\end{bmatrix}
$$

where $V$ is a $m \times n$ matrix with points $(x_i, y_i)$ for $i = 0, 1, 2, 3, \ldots m$ and for an order $N$ polynomial $\mathcal{P}_N(x)$.

### Finding $p_i$

Finding the coefficients of $\mathcal{P}_N(x)$ can be done by solving the system outlined above.  There are functions in `numpy` that can do this for us such as:
 - `numpy.polyfit(x, y, x.shape[0] - 1)`
 - `numpy.vander(x, N=None)` to construct the matrix and use a linear solver routine.

We can also use a different **basis** that might be easier to use.

### Basis

**Def:**  A basis for a $N$ dimensional vector space is a set of linearly independent vectors that span the space.

The monomials,  $1,x,\ldots, x^n$, form the usual basis for the vector space of $n$th degree polynomials $P_N(x)$.

**Example** $P_2(x)$ is the space of all quadratic functions. i.e. $P_2(x) = \mathrm{span}< 1,x,x^2>$

$$
    P_2(x) = p_0 + p_1 x + p_2 x^2
$$

i.e for every vector $\mathbf{p}\in\mathbb{R}^3$, there is a unique quadratic function in $P_2(x)$.  (we say $P_2$ is *isomorphic* to $\mathbb{R}^3$ and is a three dimensional function space).

**However**,  the monomials are not the only basis for $P_N$

### Lagrange Basis

Given $N+1$ points $(x_0,y_0), (x_1,y_1), \ldots, (x_{N},y_{N})$ again assuming the $x_i$ are all unique, the interpolating polynomial $\mathcal{P}_N(x)$ can be written as

$$\mathcal{P}_N(x) = \sum^{N}_{i=0} y_i \ell_i(x)$$

where

$$\ell_i(x) = \prod^{N}_{j=0, j \neq i} \frac{x - x_j}{x_i - x_j} = \frac{x - x_0}{x_i - x_0} \frac{x - x_1}{x_i - x_1} \cdots \frac{x - x_{i-1}}{x_i - x_{i-1}}\frac{x - x_{i+1}}{x_i - x_{i+1}} \cdots \frac{x - x_{N}}{x_i - x_{N}}$$

are the **Lagrange Polynomials**

### Lagrange Polynomials

$$\ell_i(x) = \prod^{N}_{j=0, j \neq i} \frac{x - x_j}{x_i - x_j} $$

A Key property of the Lagrange polynomials is that

$$
    \ell_i(x_j) = \delta_{ij} = \left\{\begin{matrix}
            0 & i\neq j \\
            1 & i=j\\
            \end{matrix}\right.
$$
which is why the weights in $P_N(x)$ are simply the $y$ values of the interpolant

### Solving for the coefficients of $P_N(x)$

In general,   if

$$
    P_N(x) = \sum_{n=0}^N w_j\phi_j(x)
$$
where $\phi_j(x)$ is any basis function for $P_N$ (i.e. monomial, Lagrange, and there are many more). Then finding the unique set of weights for the interpolating polynomial through $N+1$ distinct data points $(x_i, y_i)$, just reduces to solving $N+1$ linear equations $y_i = P_N(x_i)$.

For the monomial basis this reduces to the linear system

$$
    V(\mathbf{x})\mathbf{w} = \mathbf{y}
$$

What is the matrix for the Lagrange Basis?

### Visualizing the Lagrange Polynomials

```python
# ====================================================
# Compute the Lagrange basis (\ell_i(x))
def lagrange_basis(x, data):
    """Compute Lagrange basis at x given data"""
    basis = numpy.ones((data.shape[0], x.shape[0]))
    for i in range(data.shape[0]):
        for j in range(data.shape[0]):
            if i != j:
                basis[i, :] *= (x - data[j, 0]) / (data[i, 0] - data[j, 0])
    return basis

# ====================================================
# Calculate full polynomial
def poly_interpolant(x, data):
    """Compute polynomial interpolant of (x,y) using Lagrange basis"""
    P = numpy.zeros(x.shape[0])
    basis = lagrange_basis(x, data)
    for n in range(data.shape[0]):
        P += basis[n, :] * data[n, 1]
    return P

# ====================================================
```

```python
x_data = numpy.array([0., 1., 2., 3.])
y_data = numpy.ones(x_data.shape)
data = numpy.array([x_data, y_data]).T

x = numpy.linspace(x_data.min(),x_data.max(),100)
```

```python
# ====================================================
# Plot individual basis functions
fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
basis = lagrange_basis(x, data)
for i in range(len(x_data)):
    axes.plot(x, basis[i, :], label="$\ell_{%s}(x)$" % i)

axes.set_title("Lagrange Basis $\ell_i(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$\ell_i(x)$")
axes.grid()
axes.legend(loc='best')
plt.show()
```

### Linear Independence of the Lagrange Polynomials

Because the weights of each basis function in the Lagrange basis is just the $y$ value at the interpolation points,  it is straightforward to show that the Lagrange polynomials are linearly independent.  I.e. the statement

$$
    \sum_{n=0}^N w_j\phi_j(x) =0
$$
is equivalent to interpolating the zero function,  where all the $w_j =0$

**Example 0 Revisited** In example 0 above the linear polynomial that interpolates the coordinates $(1,3)$ and $(3,7)$ was simply stated as
$$
    P_1(x) = 2(x-1)+3.
$$
Another way to look at this example is to first note that when we add two linear polynomials
the result is another linear polynomial. The first polynomial to define interpolates $(1,1)$
and $(3,0)$,
$$
    \ell_0(x) = \frac{x-3}{1-3}.
$$
The second polynomial to define interpolates $(1,0)$ and $(3,1)$,
$$
    \ell_1(x) = \frac{x-1}{3-1}.
$$

A linear combination of these two functions can be defined that will interpolate the points $(1,3)$ and $(3,7)$,
$$
    P_1(x) = 3\cdot\ell_0(x) + 7\cdot\ell_1(x).
$$
The graphs of these functions are shown below.

```python
# =============================================================
# Plot the two example basis functions in the current example
x = numpy.linspace(1.0, 3.0, 2)

fig = plt.figure(figsize=(8, 6))

axes = fig.add_subplot(1, 1, 1)
axes.set_ylim([0,9])
axes.plot(x, (x-3)/(-2), color='r', label="$\ell_{%s}(x)$" % 0)
axes.plot(x, (x-1)/(2),  color='b', label="$\ell_{%s}(x)$" % 1)
axes.plot(x, 3*(x-3)/(-2) + 7*(x-1)/(2),color='g',label='interpolant')
axes.set_title("Interpolant for (1,3) and (3,7)")
axes.set_xlabel("x")
axes.grid()
plt.show()
```

#### Example 4:  $N = 1$ Lagrange Polynomial

Given 2 points $(x_0, y_0)$ and $(x_1, y_1)$ the Lagrange form of $\mathcal{P}_N(x)$ is given by

$$\ell_0(x) = \frac{x - x_1}{x_0 - x_1}$$

and

$$\ell_1(x) = \frac{x - x_0}{x_1 - x_0}$$

so that

$$\mathcal{P}_1(x) = \ell_0(x) \cdot y_0 + \ell_1(x) \cdot y_1 = \frac{x - x_1}{x_0 - x_1} \cdot y_0 + \frac{x - x_0}{x_1 - x_0} \cdot y_1$$

One important aspect of Lagrange polynomials to note is that the $\ell_i(x)$ functions are exactly 1 when $x  = x_i$ and that every other $\ell_j(x)$ where $j \neq i$ is 0.

```python
data = numpy.array([[-1.5, -0.5], [0.0, 0.5]])
```

```python
# data = numpy.array([[-1.5, -0.5], [0.0, 0.5], [-0.5, 1.0]])
N = data.shape[0] - 1
M = data.shape[0]
x = numpy.linspace(-2.0, 2.0, 100)

# Plot individual basis functions
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
basis = lagrange_basis(x, data)
for i in range(N + 1):
    axes.plot(x, basis[i, :], label="$\ell_{%s}(x)$" % i)
axes.grid()
axes.set_title("Lagrange Basis $\ell_i(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$\ell_i(x)$")
axes.legend(loc=8)

# Plot full polynomial P_N(x)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, poly_interpolant(x, data), label="$P_{%s}(x)$" % N)
for point in data:
    axes.plot(point[0], point[1], 'ko')
axes.set_title("$P_N(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$P_N(x)$")
axes.grid()

plt.show()
```

#### Example 5:  Interpolate four points from $sin(\pi x)$

Use four points to approximate $\sin$ on the interval $x \in [-1, 1]$.  What is the behavior as $N \rightarrow \infty$?  Also plot the error between $f(x)$ and the interpolant $P_N(x)$.

```python
num_points = 21
```

```python
# num_points = 5
# num_points = 6
# num_points = 20
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(-1, 1, num_points)
data[:, 1] = numpy.sin(2.0 * numpy.pi * data[:, 0])
N = data.shape[0] - 1 # Degree of polynomial
M = data.shape[0]
x = numpy.linspace(-1.0, 1.0, 100)

# ====================================================
# Plot individual basis functions
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
basis = lagrange_basis(x, data)
for i in range(N + 1):
    axes.plot(x, basis[i, :], label="$\ell_{%s}(x)$" % i)

axes.set_title("Lagrange Basis $\ell_i(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$\ell_i(x)$")
axes.legend(loc=1)
axes.grid()

# Plot full polynomial P_N(x)
axes = fig.add_subplot(1, 2, 2)
axes.plot(x, poly_interpolant(x, data), label="$P_{%s}(x)$" % N)
axes.plot(x, numpy.sin(2.0 * numpy.pi * x), 'r--', label="True $f(x)$")
for point in data:
    axes.plot(point[0], point[1], 'ko')
axes.set_title("$P_N(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$P_N(x)$")
axes.legend(loc=1)
axes.grid()

plt.show()
```

#### Example 6: Runge's Function

Interpolate $f(x) = \frac{1}{1 + 25 x^2}$ using 6 points of your choosing on $x \in [-1, 1]$.

Try it with 11 points.

Keep increasing the number of points and see what happens.

```python
def f(x):
    return 1.0 / (1.0 + 25.0 * x**2)

x = numpy.linspace(-1, 1, 100)
# x = numpy.linspace(-2, 2, 100)

num_points = 15
```

```python
# num_points = 10
# num_points = 20
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(-1, 1, num_points)
data[:, 1] = f(data[:, 0])
N = data.shape[0] - 1

# Plot the results
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, poly_interpolant(x, data), 'b', label="$P_6(x)$")
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(data[:, 0], data[:, 1], 'ro', label="data")

axes.set_title("Interpolation of Runge's function")
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=1)
axes.grid()

plt.show()
```

#### Example 7: Weierstrass "Monster" Function

Defined as
$$
    f(x) = \sum^\infty_{n=0} a^n \cos(b^n \pi x)
$$
such that
$$
    0 < a < 1 \quad \text{and} \quad a b > 1 + \frac{3\pi}{2}.
$$
This function is continuous everywhere but not differentiable anywhere.

```python
def f(x, a=0.9, N=100):
    summation = 0.0
    b = (1.0 + 3.0 / 2.0 * numpy.pi) / a + 0.01
    print(b)
    for n in range(N + 1):
        summation += a**n * numpy.cos(b**n * numpy.pi * x)
    return summation

x = numpy.linspace(-1, 1, 1000)
# x = numpy.linspace(-2, 2, 100)

num_points = 10
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(-1, 1, num_points)
data[:, 1] = f(data[:, 0])
N = data.shape[0] - 1

# Plot the results
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, poly_interpolant(x, data), 'b', label="$P_6(x)$")
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(data[:, 0], data[:, 1], 'ro', label="data")

axes.set_title("Interpolation of Runge's function")
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=1)

plt.show()
```

### Rules of Thumb

 - Avoid high-order interpolants when possible!  Keep increasing the number of points and see what happens.
 - Avoid extrapolation - Increase the range of $x$ in the above example and check how good the approximation is beyond our sampling interval

### Error Analysis

**Theorem:** Lagrange Remainder Theorem - Let $f(x) \in C^{N+1}[-1, 1]$, then
$$
    f(x) = \mathcal{P}_N(x) + R_N(x)
$$
where $\mathcal{P}_N(x)$ is the interpolating polynomial and
$$
    R_N(x) = Q(x) \frac{f^{(N+1)}(c)}{(N+1)!} \quad \text{with} \quad c \in [-1,1]
$$
with
$$
    Q(x) = \prod^N_{i=0} (x - x_i) = (x-x_0)(x-x_1)\cdots(x-x_N) .
$$

A few things to note:
 - For Taylor's theorem note that $Q(x) = (x - x_0)^{N+1}$ and the error only vanishes at $x_0$.
 - For Lagrange's theorem the error vanishes at all $x_i$.
 - To minimize $R_N(x)$ requires minimizing $|Q(x)|$ for $x \in [-1, 1]$.

#### Minimizing $R_N(x)$

Minimizing the error $R_N(x)$ in Lagrange's theorem is equivalent to minimizing $|Q(x)|$ for $x \in [-1, 1]$.

Minimizing error $\Leftrightarrow$ picking roots of $Q(x)$ or picking the points where the interpolant data is located.  How do we this?

### Chebyshev Polynomials

*Chebyshev polynomials* $T_N(x)$ are another basis that can be used for interpolation.

First 5 polynomials
$$T_0(x) = 1$$

$$T_1(x) = x$$

$$T_2(x) = 2 x^2 - 1$$

$$T_3(x) = 4 x^3 - 3 x$$

$$T_4(x) = 8x^4 - 8x^2 + 1$$

```python
def cheb_poly(x, N):
    """Compute the *N*th Chebyshev polynomial and evaluate it at *x*"""
    T = numpy.empty((3, x.shape[0]))
    T[0, :] = numpy.ones(x.shape)
    T[1, :] = x
    if N == 0:
        return T[0, :]
    elif N == 1:
        return T[1, :]
    else:
        for k in range(2, N + 1):
            T[2, :] = 2.0 * x * T[1, :] - T[0, :]
            T[0, :] = T[1, :]
            T[1, :] = T[2, :]

        return T[2, :]
```

```python
x = numpy.linspace(-1, 1, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

for n in range(5):
    axes.plot(x, cheb_poly(x, n), label="$T_%s$" % n)

axes.set_ylim((-1.1, 1.1))
axes.set_title("Chebyshev Polynomials")
axes.set_xlabel("x")
axes.set_ylabel("$T_N(x)$")
axes.legend(loc='best')
axes.grid()

plt.show()
```

1. Chebyshev nodes of the 1st kind (roots)
$$
    x_k = \cos \left (\frac{(2 k - 1) \pi}{2 N} \right ) \quad k = 1, \ldots, N
$$
1. Chebyshev nodes of the 2nd kind (extrema)
$$
    x_k = \cos \left( \frac{k \pi}{N} \right) \quad k = 0, \ldots, N
$$

```python
N = 4
x_extrema = numpy.cos(numpy.arange(N + 1) * numpy.pi / N)
x_nodes = numpy.cos((2.0 * numpy.arange(1, N + 1) - 1.0) / (2.0 * N) * numpy.pi)
```

```python
fig = plt.figure()
# fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 1, 1)

# Plot points
axes.plot(x_extrema, numpy.zeros(N+1), 'ro')
axes.plot(x_nodes, numpy.zeros(N), 'bo')

# Plot some helpful lines
axes.plot((-1.0, -1.0), (-1.1, 1.1), 'k--')
axes.plot((1.0, 1.0), (-1.1, 1.1), 'k--')
axes.plot((-1.0, 1.0), (0.0, 0.0), 'k--')
for i in range(x_extrema.shape[0]):
    axes.plot((x_extrema[i], x_extrema[i]), (-1.1, 1.1), 'r--')
    axes.plot(x_extrema[i], cheb_poly(x_extrema, N)[i], 'ro')

print('Nodes   = {}'.format(numpy.sort(x_nodes)))
print('Extrema = {}'.format(numpy.sort(x_extrema)))
#print(numpy.cos(x_extrema))

# Plot Chebyshev polynomial
x_hat = numpy.linspace(-1, 1, 1000)
axes.plot(x_hat, cheb_poly(x_hat, N), 'k')
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))

# Labels
axes.set_title("Chebyshev Nodes and Extrema, N={}".format(N), fontsize="20")
axes.set_xlabel("x", fontsize="15")
axes.set_ylabel("$T_{N+1}(x)$", fontsize="15")

plt.show()
```

```python
# First-kind Nesting (3 x)
fig = plt.figure()
# fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 1, 1)

N = 5
factor = 3
x_1 = numpy.cos((2.0 * numpy.arange(1, N + 1) - 1.0) / (2.0 * N) * numpy.pi)
x_2 = numpy.cos((2.0 * numpy.arange(1, factor * N + 1) - 1.0) / (2.0 * factor * N) * numpy.pi)

axes.plot(x_1, numpy.zeros(N), "o", color="r", markerfacecolor="lightgray", markersize="15")
axes.plot(x_2, numpy.zeros(N * factor), 'kx', markersize="10")

x_hat = numpy.linspace(-1, 1, 1000)
axes.plot(x_hat, cheb_poly(x_hat, N), 'k')
axes.plot(x_hat, cheb_poly(x_hat, factor * N), 'k')
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.set_title("Nesting of 1st and 2nd Kind Chebyshev Polynomials")
axes.set_xlabel("$x$")
axes.set_ylabel("$T_N(x)$")
plt.show()
```

#### Properties of Chebyshev Polynomials
1. Defined by a recurrence relation

   $$T_k(x) = 2 x T_{k-1}(x) - T_{k-2}(x)$$

2. Leading coefficient of $x^N$ in $T_N(x)$ is $2^{N-1}$ for $N \geq 1$

3. Extreme values:

   $$|T_N(x)| \leq 1 \quad \text{for} \quad  -1 \leq x \leq 1$$

#### Properties of Chebyshev Polynomials
4. Minimax principle:  The polynomial

   $$T(x) = \frac{T_{N+1}(x)}{2^N}$$

   is a *monic polynomial*, a univariate function with the leading coefficient equal to 1, with the property that

   $$
       \max |T(x)| \leq \max |Q(X)| \quad \text{for} \quad  x \in [-1, 1], \quad \text{and}
   $$
   $$
       \max |T(x)| = \frac{1}{2^N}
   $$

 Recall that the remainder term in the Lagrange Remainder Theorem was
   $$
        R_N(x) = Q(x) \frac{f^{(N+1)}(c)}{(N+1)!} \quad \text{with} \quad c \in [-1,1]
   $$
   with
   $$
        Q(x) = \prod^N_{i=0} (x - x_i) = (x-x_0)(x-x_1)\cdots(x-x_N) .
   $$

#### Error Analysis Redux

Given that the Chebyshev polynomials are a minimum on the interval $[-1, 1]$ we would like $T(x) = Q(x)$.

Since we only know the roots of $Q(x)$ (the points where the interpolant data is located) we require these points to be the roots of the Chebyshev polynomial $T_{N+1}(x)$ therefore enforcing $T(x) = Q(x)$.

The zeros of $T_N(x)$ in the interval $[-1, 1]$ can be shown to satisfy
$$
    x_k = \cos\left( \frac{(2k - 1) \pi}{2 N} \right ) \quad \text{for} \quad k=1, \ldots, N
$$
These nodal points (sampling the function at these points) can be shown to minimize interpolation error.

```python
x = numpy.linspace(0, numpy.pi, 100)
N = 15

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1, aspect="equal")

axes.plot(numpy.cos(x), numpy.sin(x), 'r--')
axes.plot(numpy.linspace(-1.1, 1.1, 100), numpy.zeros(x.shape), 'r')
for k in range(1, N + 1):
    location = [numpy.cos((2.0 * k - 1.0) * numpy.pi / (2.0 * N)),
                numpy.sin((2.0 * k - 1.0) * numpy.pi / (2.0 * N))]
    axes.plot(location[0], location[1], 'ko')
    axes.plot(location[0], 0.0, 'ko')
    axes.plot([location[0], location[0]], [0.0, location[1]], 'k--')

axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-0.1, 1.1))
plt.show()
```

#### Summary

1. Minimizing the error in Lagrange's theorem is equivalent to minimizing
$$
    |Q(x)| \quad \text{for} \quad x \in [-1, 1].
$$
1. We know Chebyshev polynomials are a minimum on the interval $[-1, 1]$ so we would like to have $T(x) = Q(x)$.
1. Since we only know the roots of $Q(x)$ (the points where the interpolant data is located) we require these points to be the roots of the Chebyshev polynomial $T_{N+1}(x)$ therefore enforcing $T(x) = Q(x)$.
1. The zeros of $T_N(x)$ in the interval $[-1, 1]$ can be shown to satisfy
$$
    x_k = \cos\left( \frac{(2k - 1) \pi}{2 N} \right ) \quad \text{for} \quad k=1, \ldots, N
$$
These nodal points (sampling the function at these points) can be shown to minimize interpolation error.

#### Notes
 - The Chebyshev nodes minimize interpolation error for any polynomial basis (due to uniqueness of the interpolating polynomial, any polynomial that interpolates these points are identical regardless of the basis).
 - Chebyshev nodes uniquely define the Chebyshev polynomials.
 - The boundedness properties of Chebyshev polynomials are what lead us to the roots as a minimization but there are other used for these orthogonal polynomials.
 - There are two kinds of Chebyshev nodes and therefore two definitions.

```python
# Runge's function again
def f(x):
    return 1.0 / (1.0 + 25.0 * x**2)

# Parameters
x = numpy.linspace(-1, 1, 100)
num_points = 5
```

```python
# ============================================================
#  Equidistant nodes
equidistant_data = numpy.empty((num_points, 2))
equidistant_data[:, 0] = numpy.linspace(-1, 1, num_points)
equidistant_data[:, 1] = f(equidistant_data[:, 0])
N = equidistant_data.shape[0] - 1

P_lagrange = poly_interpolant(x, equidistant_data)

# ============================================================
#  Chebyshev nodes
chebyshev_data = numpy.empty((num_points, 2))
chebyshev_data[:, 0] = numpy.cos((2.0 * numpy.arange(1, num_points + 1) - 1.0) * numpy.pi / (2.0 * num_points))
chebyshev_data[:, 1] = f(chebyshev_data[:, 0])

P_cheby1 = poly_interpolant(x, chebyshev_data)

# Fit directly with Chebyshev polynomials
coeff = numpy.polynomial.chebyshev.chebfit(chebyshev_data[:, 0], chebyshev_data[:, 1], N)
P_cheby2 = numpy.polynomial.chebyshev.chebval(x, coeff)

# Check on unique polynomials
#print(numpy.allclose(P_cheby1, P_cheby2))

# calculate errornorms for different interpolants
equidistant_err = numpy.linalg.norm(P_lagrange - f(x))
cheb_err = numpy.linalg.norm(P_cheby1 - f(x))

# ============================================================
# Plot the results
fig = plt.figure(figsize=(16,6))
fig.subplots_adjust(hspace=.5)

axes = fig.add_subplot(1, 2, 1)
axes.plot(x, P_lagrange, 'b', label="$P_%s(x)$" % N)
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(equidistant_data[:, 0], equidistant_data[:, 1], 'ro', label="data")
axes.set_title("Interpolation at Equispaced Points: err = {}".format(equidistant_err))
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=8)
#print('Equispaced error = {}'.format(numpy.linalg.norm(P_lagrange - f(x))))

axes = fig.add_subplot(1, 2, 2)
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(x, P_cheby1, 'b', label="$P_%s(x)$" % N)
axes.plot(chebyshev_data[:, 0], chebyshev_data[:, 1], 'ro', label="data")

axes.set_title("Interpolation at Chebyshev Points: err = {}".format(cheb_err))
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=1)
#print('Chebyshev error = {}'.format(numpy.linalg.norm(P_cheby1 - f(x))))

plt.show()
```

## Piece-Wise Polynomial Interpolation

Given $N$ points, use lower order polynomial interpolation to fit the function in pieces.  We can choose the order of the polynomials and the continuity.

 - $C^0$: Interpolant is continuous
   - Linear interpolation
   - Quadratic interpolation
 - $C^1$: Interpolation and 1st derivative are continuous
   - Cubic Hermite polynomials (PCHiP)
 - $C^2$: Interpolation, 1st and 2nd derivatives are continuous
   - Cubic splines

### Piece-Wise Linear

Given a segment between point $(x_k, y_k)$a nd $(x_{k+1}, y_{k+1})$ define the segment as

$$\mathcal{P}_k(x) = \frac{y_{k+1} - y_k}{x_{k+1} - x_k} (x - x_k) + y_k$$

The final interpolant $\mathcal{P}(x)$ is then defined on $[x_k, x_{k+1}]$ using this function.

```python
data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)
N = data.shape[0] - 1

# Lagrange Basis
P_lagrange = poly_interpolant(x, data)

# C^0 Piece-wise linear
# P_pw_linear = numpy.interp(x, data[:, 0], data[:, 1])
P_linear = numpy.zeros(x.shape)
for n in range(1, N + 1):
    P_linear += ((data[n, 1] - data[n - 1, 1]) / (data[n, 0] - data[n - 1, 0]) * (x - data[n - 1, 0])
                         + data[n - 1, 1]) * (x > data[n - 1, 0]) * (x <= data[n, 0])

```

```python
# Add end points for continuity
P_linear += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_linear += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])

# Plot
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ko')
axes.plot(x, P_lagrange, 'b--')
axes.plot(x, P_linear, 'r')

axes.set_title("Interpolated Data - $C^0$ Linear")
axes.set_xlabel("x")
axes.set_ylabel("$P_1(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])

plt.show()
```

### Piece-Wise Overlapping Polynomials

In sets of three points $(x_{k+1}, y_{k+1})$, $(x_{k}, y_{k})$, and $(x_{k-1}, y_{k-1})$, find quadratic interpolant and define final interpolant $P(x)$ using the quadratic interpolant $\mathcal{P}_k(x)$ on $[x_{k-1}, x_{k+1}]$.

```python
data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)
N = data.shape[0] - 1

# This isn't overlapping, it's more like C_0 P_2
# C^0 Piece-wise quadratic
P_quadratic = numpy.zeros(x.shape)
for k in range(1, N + 1, 2):
    p = numpy.polyfit(data[k - 1:k + 2, 0], data[k - 1:k + 2, 1], 2)
    P_quadratic += numpy.polyval(p, x) * (x > data[k - 1, 0]) * (x <= data[k + 1, 0])
# Add end points for continuity
P_quadratic += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_quadratic += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])

# Plot
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ko')
axes.plot(x, P_lagrange, 'b--')
axes.plot(x, P_quadratic, 'r')

axes.set_title("Interpolated Data - $C^0$ Quadratic")
axes.set_xlabel("x")
axes.set_ylabel("$P_3(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])

plt.show()
```

### Piece-Wise $C^1$ Cubic Interpolation

For the previous two cases we had discontinous 1st derivatives!  We can make this better by constraining the polynomials to be continuous at the boundaries of the piece-wise intervals.

Given a segment between points $(x_k, y_k)$ and $(x_{k+1}, y_{k+1})$ we want to fit a cubic function between the two points.

$$\mathcal{P}_k(x) = p_0 + p_1 x + p_2 x^2 + p_3 x^3$$

$$\mathcal{P}_k(x_k) = y_k, \quad \mathcal{P}_k(x_{k+1}) = y_{k+1}$$

Now we have 4 unknowns but only two data points!  Constraining the derivative at each interval end will lead to two new equations and therefore we can solve for the interpolant.

$$\frac{\text{d}}{\text{dx}} \mathcal{P}_k(x_k) = d_k, \quad \frac{\text{d}}{\text{dx}} \mathcal{P}_k(x_{k+1}) = d_{k+1}$$

where we need to prescribe the $d_k$s.  Since we know the polynomial we can write these 4 equations as

$$\begin{aligned}
p_0 + p_1 x_k + p_2 x_k^2 + p_3 x_k^3 &= y_k \\
p_0 + p_1 x_{k+1} + p_2 x_{k+1}^2 + p_3 x_{k+1}^3 &= y_{k+1} \\
p_1 +  2p_2 x_k + 3 p_3 x_k^2 &= d_k \\
p_1 + 2 p_2 x_{k+1} + 3 p_3 x_{k+1}^2 &= d_{k+1}
\end{aligned}$$

Rewriting this as a system we get
$$\begin{bmatrix}
    1 & x_k & x_k^2 & x_k^3 \\
    1 & x_{k+1} & x_{k+1}^2 & x_{k+1}^3 \\
    0 & 1 & 2 x_k & 3 x_k^2 \\
    0 & 1 & 2 x_{k+1} & 3 x_{k+1}^2
\end{bmatrix} \begin{bmatrix}
p_0 \\ p_1 \\ p_2 \\ p_3
\end{bmatrix} = \begin{bmatrix}
y_k \\ y_{k+1} \\ d_k \\ d_{k+1}
\end{bmatrix}$$

A common simplification to the problem description re-parameterizes the locations of the points such that $s \in [0, 1]$ and recast the problem with $(0, y_k)$ and $(1, y_{k+1})$.  This simplifies the above system to
$$\begin{bmatrix}
    1 & 0 & 0 & 0 \\
    1 & 1 & 1 & 1 \\
    0 & 1 & 0 & 0 \\
    0 & 1 & 2 & 3
\end{bmatrix} \begin{bmatrix}
p_0 \\ p_1 \\ p_2 \\ p_3
\end{bmatrix} = \begin{bmatrix}
y_k \\ y_{k+1} \\ d_k \\ d_{k+1}
\end{bmatrix}$$

which can be solved to find

$$\begin{aligned}
\mathcal{P}(s) &= (1-s)^2 (1 + 2s) y_k + s^2 (3 - 2 s) y_{k+1} + s (1 - s)^2 d_k - s^2 (1 - s)d_{k+1}\\
\mathcal{P}'(s) &= 6s(s-1) y_k + 6s(1-s) y_{k+1} + (s-1)(3s-1) d_k - s(3s-2) d_{k+1}\\
\mathcal{P}''(s) &= 6 (1-2s)(y_{k+1} - y_k) + (6s - 4) d_k + (6s-2) d_{k+1}
\end{aligned}$$

Now, how to choose $d_k$?

#### PCHIP

Piecewise Cubic Hermite Interpolation Polynomial

 - Picks the slope that preserves monotonicity
 - Also tried to preserve the shape of the data
 - Note that in general this interpolant is $\mathcal{P}_k(x) \in C^1$

```python
import scipy.interpolate as interpolate

data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)

# C^1 Piece-wise PCHIP
P_pchip = interpolate.pchip_interpolate(data[:, 0], data[:, 1], x)
```

```python
# Plot
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ro')
axes.plot(x, P_pchip, 'r')

axes.set_title("Interpolated Data - $C^1$ Cubic PCHIP")
axes.set_xlabel("x")
axes.set_ylabel("$P_3(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])
axes.grid()
plt.show()
```

#### Cubic Splines

Enfores continuity on second derivatives as well:

$$\mathcal{P}''_{k}(x_{k}) = \mathcal{P}''_{k-1}(x_k)$$

From our generalization before we know

$$\mathcal{P}''(s) = 6 (1-2s)(y_{k+1} - y_k) + (6s - 4) d_k + (6s-2) d_{k+1}$$

and our constraint now becomes

$$\mathcal{P}''_{k}(0) = \mathcal{P}''_{k-1}(1)$$

$$\mathcal{P}''_{k-1}(1) = 6 (1-2 \cdot 1)(y_{k} - y_{k-1}) + (6\cdot 1 - 4) d_{k-1} + (6\cdot 1-2) d_{k}$$

$$\mathcal{P}''_{k}(0) = 6 (1-2 \cdot 0)(y_{k+1} - y_k) + (6\cdot 0 - 4) d_k + (6\cdot 0-2) d_{k+1}$$

$$-6(y_{k} - y_{k-1}) + 2 d_{k-1} + 4 d_{k} = 6 (y_{k+1} - y_k) - 4 d_k -2  d_{k+1}$$

We now have constraints on choosing the $d_k$ values.  Note that we still need to prescribe them at the boundaries of the full interval.

This forms a linear set of equations for the $d_k$s based on the $y_k$ values and can be reformulated into a tri-diagonal linear system

$$\begin{bmatrix}
         & \ddots  & \ddots  & \ddots  & &  &\\
         & 0 & 2 & 8 & 2 & 0 & & \\
         & & 0 & 2 & 8 & 2 & 0 & & & \\
         & &   & 0 & 2 & 8 & 2 & 0 & & \\
         & &    & & & \ddots & \ddots & \ddots &
\end{bmatrix}\begin{bmatrix}
    \vdots \\ d_{k-1} \\ d_{k} \\ d_{k+1} \\ \vdots
\end{bmatrix} = \begin{bmatrix}
    \vdots \\ 6 (y_{k} - y_{k-2}) \\ 6 (y_{k+1} - y_{k-1}) \\ 6 (y_{k+2} - y_{k}) \\\vdots
\end{bmatrix}$$

The boundaries are still left unconstrained and we must pick some rule to specify the derivatives there.

```python
import scipy.interpolate as interpolate

data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)

# C^2 Piece-wise Splines
# Note that to get an interpolant we need to set the smoothing
# parameters *s* to 0
P_spline = interpolate.UnivariateSpline(data[:, 0], data[:, 1], s=0)
```

```python
# Plot
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ro')
axes.plot(x, P_spline(x), 'r', label = '$C^2$')
axes.plot(x, P_pchip, 'b--', label = 'Pchip')
axes.set_title("Interpolated Data - $C^2$ Cubic Splines")
axes.set_xlabel("x")
axes.set_ylabel("$P_3(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])
axes.grid()
axes.legend(loc='best')

plt.show()
```

### Let's compare all of these methods

```python
import scipy.interpolate as interpolate

data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)

# Lagrange Basis
N = data.shape[0] - 1

lagrange_basis = numpy.ones((N + 1, x.shape[0]))
for i in range(N + 1):
    for j in range(N + 1):
        if i != j:
            lagrange_basis[i, :] *= (x - data[j, 0]) / (data[i, 0] - data[j, 0])

# Calculate full polynomial
P_lagrange = numpy.zeros(x.shape[0])
for n in range(N + 1):
    P_lagrange += lagrange_basis[n, :] * data[n, 1]

# C^0 Piece-wise linear
# P_pw_linear = numpy.interp(x, data[:, 0], data[:, 1])
P_linear = numpy.zeros(x.shape)
for n in range(1, N + 1):
    P_linear += ((data[n, 1] - data[n - 1, 1]) / (data[n, 0] - data[n - 1, 0]) * (x - data[n - 1, 0])
                         + data[n - 1, 1]) * (x > data[n - 1, 0]) * (x <= data[n, 0])

# Add end points for continuity
P_linear += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_linear += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])
# C^0 Piece-wise quadratic
P_quadratic = numpy.zeros(x.shape)
for k in range(1, N + 1, 2):
    p = numpy.polyfit(data[k - 1:k + 2, 0], data[k - 1:k + 2, 1], 2)
    P_quadratic += numpy.polyval(p, x) * (x > data[k - 1, 0]) * (x <= data[k + 1, 0])
# Add end points for continuity
P_quadratic += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_quadratic += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])

# C^1 Piece-wise PCHIP
P_pchip = interpolate.pchip_interpolate(data[:, 0], data[:, 1], x)

# C^2 Piece-wise Splines
P_spline = interpolate.UnivariateSpline(data[:, 0], data[:, 1], s=0)

# Plot
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ko', label="Data")
axes.plot(x, P_lagrange, 'y', label="Lagrange")
axes.plot(x, P_linear, 'g', label="PW Linear")
axes.plot(x, P_quadratic, 'r', label="PW Quadratic")
axes.plot(x, P_pchip, 'c', label="PW Cubic - PCHIP")
axes.plot(x, P_spline(x), 'b', label="PW Cubic - Spline")
axes.grid()

axes.set_title("Interpolated Data - Method Comparisons")
axes.set_xlabel("x")
axes.set_ylabel("$P(x)$")
axes.legend(loc='best')
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])

plt.show()
```

## Relationship to Regression

What if we have more data and want a lower degree polynomial but do not want to use a piece-wise defined interpolant?
Regression techniques are often used to minimize a form of error between the data points $y_i$ at $x_i$ with an approximating function $f(x_i)$.  Note that this is NOT interpolation anymore!

### Least-Squares

One way of doing this is to require that we minimize the least-squares error
$$
    E = \left( \sum^m_{i=1} |y_i - f(x_i)|^2 \right )^{1/2}.
$$
where as before we have data $y_i$ at locations $x_i$ and an approximating function $f(x_i)$.

From the beginning of our discussion we know we can write the interpolant as a system of linear equations which we can then solve for the coefficients of a monomial basis.  If we wanted to fit a line
$$
    \mathcal{P}_1(x) = p_0 + p_1 x
$$
to $N$ data points we would have
$$
   \begin{bmatrix}
        1 & x_1 \\
        1 & x_2 \\
        \vdots & \vdots \\
        1 & x_N
   \end{bmatrix} \begin{bmatrix}
       p_0 \\ p_1
   \end{bmatrix} = \begin{bmatrix}
       y_1 \\ y_2 \\ \vdots \\ y_N
   \end{bmatrix}
$$
or
$$
    A p = y
$$
What's wrong with this system?

This leads to the likelihood that there is no solution to the system as
$$
    A \in \mathbb{R}^{N \times 2}, p \in \mathbb{R}^{2 \times 1}, \text{  and  } y \in \mathbb{R}^{N \times 1}.
$$

Instead we can solve the related least-squares system
$$
    A^T A p = A^T y
$$
whose solution minimizes the least-square error defined before as $E$.

Note:  In general, this is not the most stable way to solve least squares problems,  in general, using an orthogonalization technique like $QR$ factorization is better numerically.

```python
# Linear Least Squares Problem
N = 50
x = numpy.linspace(-1.0, 1.0, N)
y = x + numpy.random.random((N))
```

```python
A = numpy.ones((x.shape[0], 2))
A[:, 1] = x
p = numpy.linalg.solve(numpy.dot(A.transpose(), A), numpy.dot(A.transpose(), y))
#p = numpy.linalg.lstsq(A, y, rcond=None)[0]
f = lambda x: p[0] + p[1] * x
E = numpy.linalg.norm(y - f(x), ord=2)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, y, 'ko')
axes.plot(x, f(x), 'r')
axes.set_title("Least Squares Fit to Data, err={}".format(E))
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Themes and variations

You can play all sorts of games, whether they are justified by the data or not,  for example we can fit the same random data with a function like

$$
    f(x) = p_0 + p_1\tanh(x)
$$

which is still a linear problem for the coefficients $p_0$ and $p_1$, however the vandermonde matrix now has columns of $\mathbf{1}$ and $\tanh\mathbf{x}$.

```python
# Linear Least Squares Problem
A = numpy.ones((x.shape[0], 2))
A[:, 1] = numpy.tanh(x)
p = numpy.linalg.solve(numpy.dot(A.transpose(), A), numpy.dot(A.transpose(), y))
# p = numpy.linalg.lstsq(A, y)[0]
f = lambda x: p[0] + p[1] * numpy.tanh(x)
E = numpy.linalg.norm(y - f(x), ord=2)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, y, 'ko')
axes.plot(x, f(x), 'r')
axes.set_title("Least Squares Fit to Data, err = {}".format(E))
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Let ye be warned...

(Original image can be found at [Curve Fitting](https://xkcd.com/2048/).)