_Lambda School Data Science_

# Ordinary Least Squares Regression - notes

## What is Linear Regression?

Linear Regression is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.

### Synonyms for "y variable"
- Dependent Variable
- Response Variable
- Outcome Variable
- Predicted Variable
- Measured Variable
- Explained Variable
- Label
- Target

### Synonyms for "x variable"
- Independent Variable
- Explanatory Variable
- Regressor
- Covariate
- Feature

# Simple Linear Regresion (bivariate)

## Making Predictions

Say that we were trying to create a model that captured the relationship between temperature outside and ice cream sales. In Machine Learning our goal is often different that of other flavors of Linear Regression Analysis, because we're trying to fit a model to this data with the intention of making **predictions** on new data (in the future) that we don't have yet.

## What are we trying to predict?

So if we had measured ice cream sales and the temprature outside on 11 different days, at the end of our modeling **what would be the thing that we would want to predict? - Ice Cream Sales or Temperature?**

We would probably want to be measuring temperature with the intention of using that to **forecast** ice cream sales. If we were able to successfully forecast ice cream sales from temperature, this might help us know beforehand how much ice cream to make or how many cones to buy or on which days to open our store, etc. Being able to make predictions accurately has a lot of business implications. This is why making accurate predictions is so valuable (And in large part is why data scientists are paid so well).

### Y Variable Intuition

We want the thing that we're trying to predict to serve as our **y** variable. This is why it's sometimes called the "predicted variable." We call it the "dependent" variable because our prediction for how much ice cream we're going to sell "depends" on the temperature outside.

### X Variable Intuition

All other variables that we use to predict our y variable (we're going to start off just using one) we call our **x** variables. These are called our "independent" variables because they don't *depend* on y, they "explain" y. Hence they are also referred to as our "explanatory" variables.

```python
%matplotlib inline
from ipywidgets import interact
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import statsmodels.api as sm
```

```python
columns = ['Year','Incumbent Party Candidate','Other Candidate','Incumbent Party Vote Share']

data = [[1952,"Stevenson","Eisenhower",44.6],
        [1956,"Eisenhower","Stevenson",57.76],
        [1960,"Nixon","Kennedy",49.91],
        [1964,"Johnson","Goldwater",61.34],
        [1968,"Humphrey","Nixon",49.60],
        [1972,"Nixon","McGovern",61.79],
        [1976,"Ford","Carter",48.95],
        [1980,"Carter","Reagan",44.70],
        [1984,"Reagan","Mondale",59.17],
        [1988,"Bush, Sr.","Dukakis",53.94],
        [1992,"Bush, Sr.","Clinton",46.55],
        [1996,"Clinton","Dole",54.74],
        [2000,"Gore","Bush, Jr.",50.27],
        [2004,"Bush, Jr.","Kerry",51.24],
        [2008,"McCain","Obama",46.32],
        [2012,"Obama","Romney",52.00]]

df = pd.DataFrame(data=data, columns=columns)
```

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share

      0
      1952
      Stevenson
      Eisenhower
      44.60

      1
      1956
      Eisenhower
      Stevenson
      57.76

      2
      1960
      Nixon
      Kennedy
      49.91

      3
      1964
      Johnson
      Goldwater
      61.34

      4
      1968
      Humphrey
      Nixon
      49.60

      5
      1972
      Nixon
      McGovern
      61.79

      6
      1976
      Ford
      Carter
      48.95

      7
      1980
      Carter
      Reagan
      44.70

      8
      1984
      Reagan
      Mondale
      59.17

      9
      1988
      Bush, Sr.
      Dukakis
      53.94

      10
      1992
      Bush, Sr.
      Clinton
      46.55

      11
      1996
      Clinton
      Dole
      54.74

      12
      2000
      Gore
      Bush, Jr.
      50.27

      13
      2004
      Bush, Jr.
      Kerry
      51.24

      14
      2008
      McCain
      Obama
      46.32

      15
      2012
      Obama
      Romney
      52.00

```python
df.plot(x='Year', y='Incumbent Party Vote Share');
```

```python
df['Incumbent Party Vote Share'].describe()
```

    count    16.000000
    mean     52.055000
    std       5.608951
    min      44.600000
    25%      48.350000
    50%      50.755000
    75%      55.495000
    max      61.790000
    Name: Incumbent Party Vote Share, dtype: float64

```python
target = 'Incumbent Party Vote Share'
df['Prediction'] = df[target].mean()
df['Error'] = df['Prediction'] - df[target]
```

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error

      0
      1952
      Stevenson
      Eisenhower
      44.60
      52.055
      7.455
      7.455

      1
      1956
      Eisenhower
      Stevenson
      57.76
      52.055
      -5.705
      5.705

      2
      1960
      Nixon
      Kennedy
      49.91
      52.055
      2.145
      2.145

      3
      1964
      Johnson
      Goldwater
      61.34
      52.055
      -9.285
      9.285

      4
      1968
      Humphrey
      Nixon
      49.60
      52.055
      2.455
      2.455

      5
      1972
      Nixon
      McGovern
      61.79
      52.055
      -9.735
      9.735

      6
      1976
      Ford
      Carter
      48.95
      52.055
      3.105
      3.105

      7
      1980
      Carter
      Reagan
      44.70
      52.055
      7.355
      7.355

      8
      1984
      Reagan
      Mondale
      59.17
      52.055
      -7.115
      7.115

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      52.055
      -1.885
      1.885

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      52.055
      5.505
      5.505

      11
      1996
      Clinton
      Dole
      54.74
      52.055
      -2.685
      2.685

      12
      2000
      Gore
      Bush, Jr.
      50.27
      52.055
      1.785
      1.785

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      52.055
      0.815
      0.815

      14
      2008
      McCain
      Obama
      46.32
      52.055
      5.735
      5.735

      15
      2012
      Obama
      Romney
      52.00
      52.055
      0.055
      0.055

```python
df['Error'].sum()
```

    -1.4210854715202004e-14

```python
df['Absolute Error'] = df['Error'].abs()
df['Absolute Error'].sum()
```

    72.82

```python
df['Absolute Error'].mean()
```

    4.55125

```python
mean_absolute_error(y_true=df[target], y_pred=df['Prediction'])
```

    4.55125

```python
r2_score(y_true=df[target], y_pred=df['Prediction'])
```

    0.0

```python
columns = ['Year','Average Recent Growth in Personal Incomes']

data = [[1952,2.40],
        [1956,2.89],
        [1960, .85],
        [1964,4.21],
        [1968,3.02],
        [1972,3.62],
        [1976,1.08],
        [1980,-.39],
        [1984,3.86],
        [1988,2.27],
        [1992, .38],
        [1996,1.04],
        [2000,2.36],
        [2004,1.72],
        [2008, .10],
        [2012, .95]]

growth = pd.DataFrame(data=data, columns=columns)
```

```python
df = df.merge(growth)
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes

      0
      1952
      Stevenson
      Eisenhower
      44.60
      52.055
      7.455
      7.455
      2.40

      1
      1956
      Eisenhower
      Stevenson
      57.76
      52.055
      -5.705
      5.705
      2.89

      2
      1960
      Nixon
      Kennedy
      49.91
      52.055
      2.145
      2.145
      0.85

      3
      1964
      Johnson
      Goldwater
      61.34
      52.055
      -9.285
      9.285
      4.21

      4
      1968
      Humphrey
      Nixon
      49.60
      52.055
      2.455
      2.455
      3.02

      5
      1972
      Nixon
      McGovern
      61.79
      52.055
      -9.735
      9.735
      3.62

      6
      1976
      Ford
      Carter
      48.95
      52.055
      3.105
      3.105
      1.08

      7
      1980
      Carter
      Reagan
      44.70
      52.055
      7.355
      7.355
      -0.39

      8
      1984
      Reagan
      Mondale
      59.17
      52.055
      -7.115
      7.115
      3.86

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      52.055
      -1.885
      1.885
      2.27

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      52.055
      5.505
      5.505
      0.38

      11
      1996
      Clinton
      Dole
      54.74
      52.055
      -2.685
      2.685
      1.04

      12
      2000
      Gore
      Bush, Jr.
      50.27
      52.055
      1.785
      1.785
      2.36

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      52.055
      0.815
      0.815
      1.72

      14
      2008
      McCain
      Obama
      46.32
      52.055
      5.735
      5.735
      0.10

      15
      2012
      Obama
      Romney
      52.00
      52.055
      0.055
      0.055
      0.95

```python
feature = 'Average Recent Growth in Personal Incomes'
```

```python
df.plot(x=feature, y=target, kind='scatter');
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.

## The Equation for a Line

As we know a common equation for a line is:

\begin{align}
y = mx + b
\end{align}

Where $m$ is the slope of our line and $b$ is the y-intercept.

If we want to plot a line through our cloud of points we figure out what these two values should be. Linear Regression seeks to **estimate** the slope and intercept values that describe a line that best fits the data points.

```python
m = 4
b = 44
df['Prediction'] = m * df[feature] + b
df['Error'] = df['Prediction'] - df[target]
df['Absolute Error'] = df['Error'].abs()
df['Absolute Error'].sum()
```

    45.27999999999999

```python
df['Absolute Error'].mean()
```

    2.829999999999999

```python
r2_score(y_true=df[target], y_pred=df['Prediction'])
```

    0.5178779627255485

```python
ax = df.plot(x=feature, y=target, kind='scatter')
df.plot(x=feature, y='Prediction', kind='line', ax=ax);
```

```python
def regression(m, b):
    df['Prediction'] = m * df[feature] + b
    df['Error'] = df['Prediction'] - df[target]
    df['Absolute Error'] = df['Error'].abs()
    sum_absolute_error = df['Absolute Error'].sum()

    title = f'Sum of absolute errors: {sum_absolute_error}'
    ax = df.plot(x=feature, y=target, kind='scatter', title=title, figsize=(7, 7))
    df.plot(x=feature, y='Prediction', kind='line', ax=ax)

regression(m=3, b=46)
```

## Residual Error

The residual error is the distance between points in our dataset and our regression line.

```python
def regression(m, b):
    df['Prediction'] = m * df[feature] + b
    df['Error'] = df['Prediction'] - df[target]
    df['Absolute Error'] = df['Error'].abs()
    sum_absolute_error = df['Absolute Error'].sum()

    title = f'Sum of absolute errors: {sum_absolute_error}'
    ax = df.plot(x=feature, y=target, kind='scatter', title=title, figsize=(7, 7))
    df.plot(x=feature, y='Prediction', kind='line', ax=ax)

    for x, y1, y2 in zip(df[feature], df[target], df['Prediction']):
        ax.plot((x, x), (y1, y2), color='grey')

regression(m=3.5, b=46)
```

```python
interact(regression, m=(-10,10,0.5), b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes

      0
      1952
      Stevenson
      Eisenhower
      44.60
      50.0
      5.40
      5.40
      2.40

      1
      1956
      Eisenhower
      Stevenson
      57.76
      50.0
      -7.76
      7.76
      2.89

      2
      1960
      Nixon
      Kennedy
      49.91
      50.0
      0.09
      0.09
      0.85

      3
      1964
      Johnson
      Goldwater
      61.34
      50.0
      -11.34
      11.34
      4.21

      4
      1968
      Humphrey
      Nixon
      49.60
      50.0
      0.40
      0.40
      3.02

      5
      1972
      Nixon
      McGovern
      61.79
      50.0
      -11.79
      11.79
      3.62

      6
      1976
      Ford
      Carter
      48.95
      50.0
      1.05
      1.05
      1.08

      7
      1980
      Carter
      Reagan
      44.70
      50.0
      5.30
      5.30
      -0.39

      8
      1984
      Reagan
      Mondale
      59.17
      50.0
      -9.17
      9.17
      3.86

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      50.0
      -3.94
      3.94
      2.27

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      50.0
      3.45
      3.45
      0.38

      11
      1996
      Clinton
      Dole
      54.74
      50.0
      -4.74
      4.74
      1.04

      12
      2000
      Gore
      Bush, Jr.
      50.27
      50.0
      -0.27
      0.27
      2.36

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      50.0
      -1.24
      1.24
      1.72

      14
      2008
      McCain
      Obama
      46.32
      50.0
      3.68
      3.68
      0.10

      15
      2012
      Obama
      Romney
      52.00
      50.0
      -2.00
      2.00
      0.95

```python
df['Square Error'] = df['Error'] **2
```

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes
      Square Error

      0
      1952
      Stevenson
      Eisenhower
      44.60
      54.400
      9.800
      9.800
      2.40
      96.040000

      1
      1956
      Eisenhower
      Stevenson
      57.76
      56.115
      -1.645
      1.645
      2.89
      2.706025

      2
      1960
      Nixon
      Kennedy
      49.91
      48.975
      -0.935
      0.935
      0.85
      0.874225

      3
      1964
      Johnson
      Goldwater
      61.34
      60.735
      -0.605
      0.605
      4.21
      0.366025

      4
      1968
      Humphrey
      Nixon
      49.60
      56.570
      6.970
      6.970
      3.02
      48.580900

      5
      1972
      Nixon
      McGovern
      61.79
      58.670
      -3.120
      3.120
      3.62
      9.734400

      6
      1976
      Ford
      Carter
      48.95
      49.780
      0.830
      0.830
      1.08
      0.688900

      7
      1980
      Carter
      Reagan
      44.70
      44.635
      -0.065
      0.065
      -0.39
      0.004225

      8
      1984
      Reagan
      Mondale
      59.17
      59.510
      0.340
      0.340
      3.86
      0.115600

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      53.945
      0.005
      0.005
      2.27
      0.000025

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      47.330
      0.780
      0.780
      0.38
      0.608400

      11
      1996
      Clinton
      Dole
      54.74
      49.640
      -5.100
      5.100
      1.04
      26.010000

      12
      2000
      Gore
      Bush, Jr.
      50.27
      54.260
      3.990
      3.990
      2.36
      15.920100

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      52.020
      0.780
      0.780
      1.72
      0.608400

      14
      2008
      McCain
      Obama
      46.32
      46.350
      0.030
      0.030
      0.10
      0.000900

      15
      2012
      Obama
      Romney
      52.00
      49.325
      -2.675
      2.675
      0.95
      7.155625

```python
def regression(m, b):
    df['Prediction'] = m * df[feature] + b
    df['Error'] = df['Prediction'] - df[target]
    df['Absolute Error'] = df['Error'].abs()
    df['Square Error'] = df['Error'] **2
    sum_square_error = df['Square Error'].sum()

    title = f'Sum of square errors: {sum_square_error}'
    ax = df.plot(x=feature, y=target, kind='scatter', title=title, figsize=(7, 7))
    df.plot(x=feature, y='Prediction', kind='line', ax=ax)

    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)

    for x, y1, y2 in zip(df[feature], df[target], df['Prediction']):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))
```

```python
interact(regression, m=(-10,10,0.5), b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

```python
regression(3, 46.5)
```

```python
b = 46
ms = np.arange(-10,10,0.5)
sses = []

for m in ms:
    predictions = m * df[feature] + b
    errors = predictions - df[target]
    square_errors = errors ** 2
    sse = square_errors.sum()
    sses.append(sse)

hypotheses = pd.DataFrame({'Slope': ms})
hypotheses['Intercept'] = b
hypotheses['Sum of Square Errors'] = sses

hypotheses
```

      Slope
      Intercept
      Sum of Square Errors

      0
      -10.0
      46
      15205.34740

      1
      -9.5
      46
      14085.60595

      2
      -9.0
      46
      13009.27500

      3
      -8.5
      46
      11976.35455

      4
      -8.0
      46
      10986.84460

      5
      -7.5
      46
      10040.74515

      6
      -7.0
      46
      9138.05620

      7
      -6.5
      46
      8278.77775

      8
      -6.0
      46
      7462.90980

      9
      -5.5
      46
      6690.45235

      10
      -5.0
      46
      5961.40540

      11
      -4.5
      46
      5275.76895

      12
      -4.0
      46
      4633.54300

      13
      -3.5
      46
      4034.72755

      14
      -3.0
      46
      3479.32260

      15
      -2.5
      46
      2967.32815

      16
      -2.0
      46
      2498.74420

      17
      -1.5
      46
      2073.57075

      18
      -1.0
      46
      1691.80780

      19
      -0.5
      46
      1353.45535

      20
      0.0
      46
      1058.51340

      21
      0.5
      46
      806.98195

      22
      1.0
      46
      598.86100

      23
      1.5
      46
      434.15055

      24
      2.0
      46
      312.85060

      25
      2.5
      46
      234.96115

      26
      3.0
      46
      200.48220

      27
      3.5
      46
      209.41375

      28
      4.0
      46
      261.75580

      29
      4.5
      46
      357.50835

      30
      5.0
      46
      496.67140

      31
      5.5
      46
      679.24495

      32
      6.0
      46
      905.22900

      33
      6.5
      46
      1174.62355

      34
      7.0
      46
      1487.42860

      35
      7.5
      46
      1843.64415

      36
      8.0
      46
      2243.27020

      37
      8.5
      46
      2686.30675

      38
      9.0
      46
      3172.75380

      39
      9.5
      46
      3702.61135

```python
hypotheses.plot(x='Slope', y='Sum of Square Errors',
                title=f'Intercept={b}');
```

```python
X = df[[feature]]
y = df[target]

X.shape, y.shape
```

    ((16, 1), (16,))

```python
model = LinearRegression()
model.fit(X, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
model.coef_, model.intercept_
```

    (array([3.06052805]), 46.247648016800795)

```python
model.predict([[1]])
```

    array([49.30817607])

```python
model.predict([[2]])
```

    array([52.36870413])

```python
model.predict([[3]])
```

    array([55.42923218])

```python
model.predict(X)
```

    array([53.59291535, 55.09257409, 48.84909686, 59.13247113, 55.49044274,
           57.32675957, 49.55301832, 45.05404208, 58.06128631, 53.1950467 ,
           47.41064868, 49.43059719, 53.47049423, 51.51175627, 46.55370082,
           49.15514967])

```python
df['Prediction'] = model.predict(X)
```

```python
ax = df.plot(x=feature, y=target, kind='scatter', title='sklearn LinearRegression')
df.plot(x=feature, y='Prediction', kind='line', ax=ax);
```

```python
df['Error'] = df['Prediction'] - y
```

```python
df['Absolute Error'] = df['Error'].abs()
df['Square Error'] = df['Error'] ** 2
```

```python
df['Square Error'].mean()
```

    12.392042143389562

```python
mean_squared_error(y_true=y, y_pred=model.predict(X))
```

    12.39204214338956

```python
np.sqrt(mean_squared_error(y, model.predict(X)))
```

    3.5202332512760512

# R Squared:  $R^2$

One final attribute of linear regressions that we're going to talk about today is a measure of goodness of fit known as $R^2$ or R-squared. $R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely.

```python
model.score(X, y)
```

    0.5798462099485426

```python
r2_score(y, model.predict(X))
```

    0.5798462099485426

### Statsmodels

https://www.statsmodels.org/dev/examples/notebooks/generated/ols.html

```python
model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

                                    OLS Regression Results
    ======================================================================================
    Dep. Variable:     Incumbent Party Vote Share   R-squared:                       0.580
    Model:                                    OLS   Adj. R-squared:                  0.550
    Method:                         Least Squares   F-statistic:                     19.32
    Date:                        Mon, 29 Apr 2019   Prob (F-statistic):           0.000610
    Time:                                17:28:59   Log-Likelihood:                -42.839
    No. Observations:                          16   AIC:                             89.68
    Df Residuals:                              14   BIC:                             91.22
    Df Model:                                   1
    Covariance Type:                    nonrobust
    =============================================================================================================
                                                    coef    std err          t      P>|t|      [0.025      0.975]
    -------------------------------------------------------------------------------------------------------------
    const                                        46.2476      1.622     28.514      0.000      42.769      49.726
    Average Recent Growth in Personal Incomes     3.0605      0.696      4.396      0.001       1.567       4.554
    ==============================================================================
    Omnibus:                        5.392   Durbin-Watson:                   2.379
    Prob(Omnibus):                  0.067   Jarque-Bera (JB):                2.828
    Skew:                          -0.961   Prob(JB):                        0.243
    Kurtosis:                       3.738   Cond. No.                         4.54
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

    /usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:2389: FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
      return ptp(axis=axis, out=out, **kwargs)
    /usr/local/lib/python3.6/dist-packages/scipy/stats/stats.py:1416: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
      "anyway, n=%i" % int(n))

# The Anatomy of Linear Regression

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.

- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.

# More Formal Notation

We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that.

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

# How Does it do it?

## Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit.

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

## Linear Algebra!

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python
X = sm.add_constant(df[feature]).values
print('X')
print(X)

y = df[target].values[:, np.newaxis]
print('y')
print(y)

X_transpose = X.T
print('X Transpose')
print(X_transpose)

X_transpose_X = X_transpose @ X
print('X Transpose X')
print(X_transpose_X)

X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
print('X Transpose X Inverse')
print(X_transpose_X_inverse)

X_transpose_y = X_transpose @ y
print('X Transpose y')
print(X_transpose_y)

beta_hat = X_transpose_X_inverse @ X_transpose_y
print('Beta Hat')
print(beta_hat)
```

    X
    [[ 1.    2.4 ]
     [ 1.    2.89]
     [ 1.    0.85]
     [ 1.    4.21]
     [ 1.    3.02]
     [ 1.    3.62]
     [ 1.    1.08]
     [ 1.   -0.39]
     [ 1.    3.86]
     [ 1.    2.27]
     [ 1.    0.38]
     [ 1.    1.04]
     [ 1.    2.36]
     [ 1.    1.72]
     [ 1.    0.1 ]
     [ 1.    0.95]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]
     [46.32]
     [52.  ]]
    X Transpose
    [[ 1.    1.    1.    1.    1.    1.    1.    1.    1.    1.    1.    1.
       1.    1.    1.    1.  ]
     [ 2.4   2.89  0.85  4.21  3.02  3.62  1.08 -0.39  3.86  2.27  0.38  1.04
       2.36  1.72  0.1   0.95]]
    X Transpose X
    [[16.    30.36 ]
     [30.36  86.821]]
    X Transpose X Inverse
    [[ 0.18575056 -0.06495418]
     [-0.06495418  0.03423145]]
    X Transpose y
    [[ 832.88  ]
     [1669.7967]]
    Beta Hat
    [[46.24764802]
     [ 3.06052805]]

    /usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:2389: FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
      return ptp(axis=axis, out=out, **kwargs)

# Multiple Regression

Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression - NOT MULTIVARIATE!

```python
df.sort_values(by='Square Error', ascending=False)
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes
      Square Error
      US Military Fatalities per Million

      0
      1952
      Stevenson
      Eisenhower
      44.60
      53.592915
      8.992915
      8.992915
      2.40
      80.872526
      190

      4
      1968
      Humphrey
      Nixon
      49.60
      55.490443
      5.890443
      5.890443
      3.02
      34.697316
      146

      11
      1996
      Clinton
      Dole
      54.74
      49.430597
      -5.309403
      5.309403
      1.04
      28.189758
      0

      5
      1972
      Nixon
      McGovern
      61.79
      57.326760
      -4.463240
      4.463240
      3.62
      19.920515
      0

      12
      2000
      Gore
      Bush, Jr.
      50.27
      53.470494
      3.200494
      3.200494
      2.36
      10.243163
      0

      15
      2012
      Obama
      Romney
      52.00
      49.155150
      -2.844850
      2.844850
      0.95
      8.093173
      5

      1
      1956
      Eisenhower
      Stevenson
      57.76
      55.092574
      -2.667426
      2.667426
      2.89
      7.115161
      0

      3
      1964
      Johnson
      Goldwater
      61.34
      59.132471
      -2.207529
      2.207529
      4.21
      4.873184
      1

      8
      1984
      Reagan
      Mondale
      59.17
      58.061286
      -1.108714
      1.108714
      3.86
      1.229246
      0

      2
      1960
      Nixon
      Kennedy
      49.91
      48.849097
      -1.060903
      1.060903
      0.85
      1.125515
      0

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      47.410649
      0.860649
      0.860649
      0.38
      0.740716
      0

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      53.195047
      -0.744953
      0.744953
      2.27
      0.554955
      0

      6
      1976
      Ford
      Carter
      48.95
      49.553018
      0.603018
      0.603018
      1.08
      0.363631
      2

      7
      1980
      Carter
      Reagan
      44.70
      45.054042
      0.354042
      0.354042
      -0.39
      0.125346
      0

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      51.511756
      0.271756
      0.271756
      1.72
      0.073851
      4

      14
      2008
      McCain
      Obama
      46.32
      46.553701
      0.233701
      0.233701
      0.10
      0.054616
      14

```python
"""
Fatalities denotes the cumulative number of American military
fatalities per millions of US population the in Korea, Vietnam,
Iraq and Afghanistan wars during the presidential terms
preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and
2012 elections.

http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf
"""

columns = ['Year','US Military Fatalities per Million']

data = [[1952,190],
        [1956,  0],
        [1960,  0],
        [1964,  1],
        [1968,146],
        [1972,  0],
        [1976,  2],
        [1980,  0],
        [1984,  0],
        [1988,  0],
        [1992,  0],
        [1996,  0],
        [2000,  0],
        [2004,  4],
        [2008, 14],
        [2012,  5]]

deaths = pd.DataFrame(data=data, columns=columns)
```

```python
df = df.merge(deaths)
```

```python
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']

target = 'Incumbent Party Vote Share'

X = df[features]
y = df[target]

model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

                                    OLS Regression Results
    ======================================================================================
    Dep. Variable:     Incumbent Party Vote Share   R-squared:                       0.871
    Model:                                    OLS   Adj. R-squared:                  0.851
    Method:                         Least Squares   F-statistic:                     43.77
    Date:                        Mon, 29 Apr 2019   Prob (F-statistic):           1.68e-06
    Time:                                17:57:20   Log-Likelihood:                -33.412
    No. Observations:                          16   AIC:                             72.82
    Df Residuals:                              13   BIC:                             75.14
    Df Model:                                   2
    Covariance Type:                    nonrobust
    =============================================================================================================
                                                    coef    std err          t      P>|t|      [0.025      0.975]
    -------------------------------------------------------------------------------------------------------------
    const                                        46.6621      0.937     49.806      0.000      44.638      48.686
    Average Recent Growth in Personal Incomes     3.4820      0.408      8.527      0.000       2.600       4.364
    US Military Fatalities per Million           -0.0537      0.010     -5.408      0.000      -0.075      -0.032
    ==============================================================================
    Omnibus:                        2.475   Durbin-Watson:                   2.607
    Prob(Omnibus):                  0.290   Jarque-Bera (JB):                0.666
    Skew:                           0.096   Prob(JB):                        0.717
    Kurtosis:                       3.981   Cond. No.                         110.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

    /usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:2389: FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
      return ptp(axis=axis, out=out, **kwargs)
    /usr/local/lib/python3.6/dist-packages/scipy/stats/stats.py:1416: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
      "anyway, n=%i" % int(n))

```python
model = LinearRegression()
model.fit(X, y)
print('Intercept:', model.intercept_)
pd.Series(model.coef_, features)
```

    Intercept: 46.662064098015115

    Average Recent Growth in Personal Incomes    3.481956
    US Military Fatalities per Million          -0.053661
    dtype: float64

```python
np.sqrt(mean_squared_error(y, model.predict(X)))
```

    1.9528760602450586

# Train / Test Split

```python
train = df.query('Year < 2008')
test  = df.query('Year >= 2008')
X_train = train[features]
y_train = train[target]
X_test  = test[features]
y_test  = test[target]
X_train.shape, y_train.shape, X_test.shape, y_test.shape
```

    ((14, 2), (14,), (2, 2), (2,))

```python
model.fit(X_train, y_train)
model.predict(X_test)
```

    array([45.86970509, 49.39965918])

```python
test
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes
      Square Error
      US Military Fatalities per Million

      14
      2008
      McCain
      Obama
      46.32
      46.553701
      0.233701
      0.233701
      0.10
      0.054616
      14

      15
      2012
      Obama
      Romney
      52.00
      49.155150
      -2.844850
      2.844850
      0.95
      8.093173
      5

### More about the "Bread & Peace" model
- https://fivethirtyeight.com/features/what-do-economic-models-really-tell-us-about-elections/
- https://statmodeling.stat.columbia.edu/2007/12/15/bread_and_peace/
- https://avehtari.github.io/RAOS-Examples/ElectionsEconomy/hibbs.html
- https://douglas-hibbs.com/
- http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf

# Dimensionality in Linear Regression!

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of $x$ values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.

# Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

\begin{align}
\hat{\beta} = \frac{Cov(x,y)}{Var(y)}
\end{align}

Going back to the two equations for the two models that we have estimated so far, lets replace their beta values with their actual values to see if we can make sense of how to interpret these beta coefficients.

## Bivariate Model

$y_i = \beta_0 + \beta_1temperature + \epsilon$

$sales_i = -596.2 + 24.69temperature + \epsilon$

What might $\beta_0$ in this model represent? It represents the level of sales that we would have if temperature were 0. Since this is negative one way of interpreting it is that it's so cold outside that you would have to pay people to eat ice cream. A more appropriate interpretation is probably that the ice cream store owner should close his store down long before the temperature reaches 0 degrees farenheit (-17.7 celsius). The owner can compare his predicted sales with his costs of doing business to know how warm the weather has to get before he should open his store.

What might the $beta_1$ in this model reprsent? it represents the increase in sales for each degree of temperature increase. For every degree that the temperature goes up outside he has $25 more in sales.

## Multiple Regression Model

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon$

$BloodPressure_i = 30.99+ .86age_i + .33weight_i + \epsilon$

The interpretation of coefficients in this example are similar. The intercept value repesents the blood pressure a person would have if they were 0 years old and weighed 0 pounds. This not a super useful interpretation. If we look at our data it is unlikely that we have any measurements like these in the dataset. This means that our interpretation of our intercept likely comes from extrapolating the regression line (plane). Coefficients having straightforward interpretations is a strength of linear regression if we're careful about extrapolation and only interpreting our data within the context that it was gathered.

The interpretation of our other coefficients can be a useful indicator for how much a person similar to those in our dataset's blood pressure will go up on average with each additional year of age and pound of weight.

# Basic Model Validation

One of the downsides of relying on $R^2$ too much is that although it tells you when you're fitting the data well, it doesn't tell you when you're *overfitting* the data. The best way to tell if you're overfitting the data is to get some data that your model hasn't seen yet, and evaluate how your predictions do. This is essentially what "model validation" is.

# Why is Linear Regression so Important?

## Popularity

Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.

## Interpretability

Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.

## Simplicity

A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients.


P1 discontinuous SWIPDG, stationary linear elliptic ESV2007 problem
==================================

This example is about approximating the solution $u$ of the elliptic problem

$$\begin{align}
  -\nabla\cdot( \kappa \nabla u ) &= f   &&\text{in } \Omega\\
                                u &= g_D &&\text{on }\partial\Omega
\end{align}$$

with datafunction as defined in `dune/gdt/test/linearelliptic/problems/ESV2007.hh` (see below) using a piecewise linear block SWIPDG scheme, as in `dune/gdt/test/linearelliptic/discretizers/block-ipdg.hh`.

```python
import numpy as np
```

```python
from dune.xt.common import init_logger, init_mpi
init_mpi()
init_logger()
```

$$\begin{align}
  \Omega &= [-1, 1]^2\\
  \Gamma_D &= \partial\Omega\\
  \Gamma_N &= \emptyset
\end{align}$$

```python
from dune.xt.grid import (make_cube_dd_subdomains_grid__2d_simplex_aluconform as make_grid,
                          make_boundary_info_on_dd_subdomain_layer as make_boundary_info)

# the inner_boundary_segment_index must be larger than the number of boundary intersections of the fine grid
# a good guess is std::numeric_limits<size_t>::max() - 42 = 18446744073709551615 - 42 = 18446744073709551573
inner_boundary_id = 18446744073709551573
grid = make_grid(lower_left=[-1, -1], upper_right=[1, 1], num_elements=[4, 4], num_refinements=2,
                 num_partitions=[2, 2], num_oversampling_layers=0, inner_boundary_segment_index=inner_boundary_id)
grid.visualize('../block_swipdg_esv2007_grid', with_coupling=True)

boundary_info = make_boundary_info(grid, {'type': 'xt.grid.boundaryinfo.boundarysegmentindexbased',
                                          'default': 'dirichlet',
                                          'neumann': '[{} {}]'.format(inner_boundary_id, inner_boundary_id+1)})
```

$$\begin{align}\lambda(x) &:= 1\\
\kappa(x) &:= [1, 0; 0, 1] \in \mathbb{R}^{2\times 2}\\
f(x) &:= \tfrac{1}{2} \pi^2 \cos(\tfrac{1}{2} \pi x_0) \cos(\tfrac{1}{2} \pi x_1)\\
g_D(x) &:= 0\end{align}$$

Note that the `grid` is only provided to select the correct _type_ of function; these functions do not rely on the actual grid object.

```python
from dune.xt.functions import make_constant_function_1x1, make_constant_function_2x2, make_expression_function_1x1

lmbda = make_constant_function_1x1(grid, 1.0, name='diffusion')
kappa = make_constant_function_2x2(grid, [[1., 0.], [0., 1.]], name='diffusion')
f = make_expression_function_1x1(grid, 'x', '0.5*pi*pi*cos(0.5*pi*x[0])*cos(0.5*pi*x[1])', order=3, name='force')
g_D = make_constant_function_1x1(grid, 0.0, name='dirichlet')
```

```python
from dune.xt.la import (IstlRowMajorSparseMatrixDouble as Matrix,
                        IstlDenseVectorDouble as Vector)
from dune.gdt import make_block_dg_dd_subdomain_part_to_1x1_fem_p1_space as make_block_space

block_space = make_block_space(grid)

print('preparing local and coupling containers ...')
local_patterns = [block_space.local_space(ii).compute_pattern('face_and_volume')
                  for ii in np.arange(block_space.num_blocks)]
coupling_patterns = {}
for ii in np.arange(grid.num_subdomains):
    for jj in grid.neighboring_subdomains(ii):
            coupling_patterns[(ii, jj)] = block_space.compute_coupling_pattern(ii, jj, 'face')
# if we did CG locally, we would additionally need those:
#boundary_patterns = {}
#for ii in grid.boundary_subdomains():
#    boundary_patterns[ii] = block_space.compute_boundary_pattern(ii, 'face')

local_matrices = [None]*grid.num_subdomains
local_vectors = [None]*grid.num_subdomains
coupling_matrices = {}
for ii in np.arange(grid.num_subdomains):
    local_matrices[ii] = Matrix(block_space.local_space(ii).size(),
                                block_space.local_space(ii).size(),
                                local_patterns[ii])
    local_vectors[ii] = Vector(block_space.local_space(ii).size())
for ii in np.arange(grid.num_subdomains):
    for jj in grid.neighboring_subdomains(ii):
        coupling_matrices[(ii, jj)] = Matrix(block_space.local_space(ii).size(),
                                             block_space.local_space(jj).size(),
                                             coupling_patterns[(ii, jj)])
# if we did CG locally, we would additionally need boundary_matrices
# and boundary_vectors (for non-trivial boundary data)
```

```python
from dune.xt.grid import (make_apply_on_inner_intersections_2d_simplex_aluconformgrid_dd_subdomain_coupling_part
                          as make_apply_on_inner_intersections)

from dune.gdt import (make_elliptic_swipdg_affine_factor_matrix_operator as make_elliptic_swipdg_matrix_operator,
                      make_l2_volume_vector_functional,
                      make_local_elliptic_swipdg_affine_factor_inner_integral_operator as make_local_elliptic_swipdg_coupling_operator,
                      make_local_coupling_two_form_assembler,
                      make_system_assembler)

print('assembling local containers ...')

def assemble_local_contributions(subdomain):
    ipdg_operator = make_elliptic_swipdg_matrix_operator(lmbda, kappa, boundary_info, local_matrices[subdomain],
                                                         block_space.local_space(subdomain))
    l2_functional = make_l2_volume_vector_functional(f, local_vectors[subdomain], block_space.local_space(subdomain))
    local_assembler = make_system_assembler(block_space.local_space(subdomain))
    local_assembler.append(ipdg_operator)
    local_assembler.append(l2_functional)
    local_assembler.assemble()

for ii in np.arange(grid.num_subdomains):
    assemble_local_contributions(ii)

print('assembling coupling matrices ...')

local_ipdg_coupling_operator = make_local_elliptic_swipdg_coupling_operator(lmbda, kappa)
local_coupling_assembler = make_local_coupling_two_form_assembler(local_ipdg_coupling_operator)
apply_on_inner_intersections = make_apply_on_inner_intersections()

def assemble_coupling_contributions(subdomain, neighboring_subdomain):
    coupling_assembler = block_space.coupling_assembler(subdomain, neighboring_subdomain)
    coupling_assembler.append(local_coupling_assembler,
                              local_matrices[subdomain],
                              local_matrices[neighboring_subdomain],
                              coupling_matrices[(subdomain, neighboring_subdomain)],
                              coupling_matrices[(neighboring_subdomain, subdomain)],
                              apply_on_inner_intersections) # The coupling assembler visits only coupling intersections
    # between the subdomains, those only once ond those are marked as inner. Thus we need to apply on inner intersections
    # to assemble primally.

for ii in np.arange(grid.num_subdomains):
    for jj in grid.neighboring_subdomains(ii):
        if ii < jj: # Assemble primally (visit each intersection only once).
            assemble_coupling_contributions(ii, jj)
```

```python
print('creating global container ...')

from dune.xt.la import IstlDenseVectorDouble, IstlRowMajorSparseMatrixDouble, SparsityPatternDefault

global_pattern = SparsityPatternDefault(block_space.mapper.size)
for ii in np.arange(grid.num_subdomains):
    block_space.mapper.copy_local_to_global(local_patterns[ii], ii, global_pattern)
    for jj in grid.neighboring_subdomains(ii):
        block_space.mapper.copy_local_to_global(coupling_patterns[(ii, jj)], ii, jj, global_pattern)
        block_space.mapper.copy_local_to_global(coupling_patterns[(jj, ii)], jj, ii, global_pattern)

system_matrix = IstlRowMajorSparseMatrixDouble(block_space.mapper.size, block_space.mapper.size, global_pattern)
rhs_vector = IstlDenseVectorDouble(block_space.mapper.size, 0.)
for ii in np.arange(grid.num_subdomains):
    block_space.mapper.copy_local_to_global(local_matrices[ii], local_patterns[ii], ii, system_matrix)
    block_space.mapper.copy_local_to_global(local_vectors[ii], ii, rhs_vector)
    for jj in grid.neighboring_subdomains(ii):
        block_space.mapper.copy_local_to_global(coupling_matrices[(ii, jj)],
                                                coupling_patterns[(ii, jj)],
                                                ii, jj, system_matrix)
        block_space.mapper.copy_local_to_global(coupling_matrices[(jj, ii)],
                                                coupling_patterns[(jj, ii)],
                                                jj, ii, system_matrix)
```

```python
from dune.xt.la import make_solver
from dune.gdt import make_discrete_function

print('solving linear system (of size {}x{}) ...'.format(block_space.mapper.size, block_space.mapper.size))
solution = IstlDenseVectorDouble(block_space.mapper.size, 0.0)
make_solver(system_matrix).apply(rhs_vector, solution)
make_discrete_function(block_space, solution, 'solution').visualize('../block_swipdg_esv2007_solution')
```


```python
import matplotlib.pyplot as plt
# from networkx import minimum_cut, DiGraph
from mip import *

from src.read_instance import MDOVRP
```

```python
filepath = "Instancias/ES-n78-m2-Q10138.txt"
# filepath = "Instancias/Vrp-Set-A/A-n80-m2-Q60.vrp"
```

```python
N, D, V, Q, q, c, coord_x, coord_y = MDOVRP(filepath)
```

Usando a formulação $MDOVRP_{2i− flv}$ de Lalla-Ruiz e Mes (2019)

    Lalla-Ruiz, Eduardo, and Martijn Mes. "Mathematical formulations and improvements for the multi-depot open vehicle routing problem." Optimization Letters 15 (2021): 271-286.

```python
#cria o modelo
model = Model('PRVMD', solver_name = GUROBI)

# Variaveis de decisao
x = [[model.add_var(var_type=BINARY) if i!=j else model.add_var(lb=0, ub=0) for i in V] for j in V]
u = [[model.add_var(var_type=CONTINUOUS) if i!=j else model.add_var(lb=0, ub=0) for i in V] for j in V]
y = [model.add_var(var_type=BINARY) for i in N]
w = [model.add_var(var_type=BINARY) for i in N]
```

    Set parameter WLSAccessID
    Set parameter WLSSecret
    Set parameter LicenseID
    Academic license - for non-commercial use only - registered to thiago.giachetto@aluno.ufop.edu.br

\begin{equation}
\label{eq:9}
    x_{ij} \in \{0, 1\}, \quad \forall i, j \in V
\end{equation}

\begin{equation}
\label{eq:10}
    u_{ij} \geq 0, \quad \forall i, j \in V
\end{equation}

\begin{equation}
\label{eq:10.1}
    w_{j} \in \{0, 1\}, \quad \forall j \in N
\end{equation}

```python
# Funcao objetivo
A = 1
B = 10

model.objective = minimize(A*xsum(x[i][j] * c[i,j] for i in V for j in V if i != j) - B*xsum(q[j]*w[j] for j in N))
```

\begin{equation}
\label{eq:1}
 Minimizar \quad A\sum_{i \in V }\sum_{j \in V} c_{ij}x_{ij} - B\sum_{j \in N} q_{j}w_{j}
\end{equation}

```python
# Restricao 2
for j in N:
    model += xsum(x[i][j] for i in V if i != j) == w[j]
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} = w_{j}, \quad \forall j \in N
    \tag{2}
\end{equation}

```python
# Restricao 18

for j in N:
    model += (xsum(x[i][j] for i in V if i != j) - xsum(x[j][i] for i in N if i != j)) >= 0
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} - \sum_{i \in N, i \neq j}x_{ji} \geq 0, \quad \forall j \in N
    \tag {18}
\end{equation}

```python
# Restricao 19
for i in V:
    for j in V:
        if i != j:
            model += x[i][j] + x[j][i] <= 1
```

\begin{equation}
    x_{ij} + x_{ji} \leq 1, \quad \forall i, j \in V, i \neq j
    \tag {19}
\end{equation}

```python
# Restricao 20
# Ninguem volta para o depósito
model += xsum(x[j][k] for k in D for j in V) == 0
```

\begin{equation}
    \sum_{k \in D}\sum_{j \in V}x_{jk} = 0
    \tag {20}
\end{equation}

```python
# Restricao 21 modificada -> se cidade não é atendida, restrição é esquecida
for j in N:
    model += (xsum(u[i][j] for i in V if i != j) - xsum(u[j][i] for i in V if i != j) - q[j]) >= -Q*(1 - w[j])
```

\begin{equation}
    (\sum_{i \in V, i \neq j}u_{ij} - \sum_{i \in V, i \neq j}u_{ji}) - q_{j} \geq -Q(1 - w_{j}), \quad \forall j \in N
    \tag {21}
\end{equation}

    Wolsey, Laurence A. Integer programming (pp. 11). John Wiley & Sons, 2020.

```python
# restrição que liga x e w
for j in N:
    xsum(x[i][j] for i in V if i != j) == w[j]
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} = w_{j}, \quad \forall j \in N
    \tag {21}
\end{equation}

```python
#Restricao 22
for i in N:
    for j in N:
        model += (Q - q[i])*x[i][j] >= u[i][j]
```

\begin{equation}
    (Q - q_i) \cdot x_{ij} \geq u_{ij}, \quad \forall i, j \in N
    \tag {22}
\end{equation}

```python
# Restricao 23
for k in D:
    for j in N:
        model += Q * x[k][j] >= u[k][j]
```

\begin{equation}
    Q \cdot x_{kj} \geq u_{kj}, \quad \forall k \in D, j \in N
    \tag {23}
\end{equation}

```python
# # Restricao 8
# menor distancia de algum deposito ate a cidade
d = [min([c[j, i] for j in D]) for i in N]
# menor distancia entre cidade i e quaisquer outra cidade
r = [min([c[j, i] for j in N if i != j]) for i in N]
# máxima menor distancia entre duas cidades
M = max(r)

# d[i] >= r[i] <=> é mais longe ir de algum depósito do q de outra cidade => y in {0, 1}
# d[i] < r[i] <=> é mais perto sair de algum depósito do q de qualquer outra cidade => y = 1
for i in N:
    model += d[i] + M * y[i] >= r[i]*w[i]
```

\begin{equation}
    d_{i} + M y_{i} \geq r_{i}w_{i}, \quad \forall k \in N
    \tag {8}
\end{equation}

```python
# Restricao 9

for i in N:
    k_l = D[np.argmin([c[j, i] for j in D], axis=0)]
    model += x[k_l][i] >= y[i]
```

\begin{equation}
    x_{k'i} \geq y_{i}, \quad \forall i \in N, k' = argmin(c_{ki})_{k \in D}
    \tag {9}
\end{equation}

```python
# limite de carros total
NUM_VEICULOS = 10

model += xsum(x[k][i] for k in D for i in N) <= NUM_VEICULOS
```

```python
# Restricao na demanda relacionado ao limite de caminhões
model += xsum(q[j]*w[j] for j in N) <= NUM_VEICULOS*Q
```

```python
# Restricao de distancia maxima entre duas cidades

for i in V:
    for j in V:
        if i != j:
            model += c[i,j]*x[i][j] <= 180
```

```python
# # Restricao para atender todas as cidades

# for j in N:
#     model += w[j] == 1
```

```python
# Resolve o modelo
# model.optimize(max_seconds=7200)
model.optimize(max_seconds=120)
print("status: {} objective value : {} best possible: {}".format(model.status, model.objective_value, model.objective_bound))
```

    Set parameter TimeLimit to value 120
    Set parameter NodeLimit to value 1073741824
    Set parameter SolutionLimit to value 1073741824
    Set parameter IntFeasTol to value 1e-06
    Set parameter Method to value 3
    Gurobi Optimizer version 9.5.1 build v9.5.1rc2 (linux64)
    Thread count: 2 physical cores, 4 logical processors, using up to 4 threads
    Academic license - for non-commercial use only - registered to thiago.giachetto@aluno.ufop.edu.br
    Optimize a model with 18323 rows, 12320 columns and 59808 nonzeros
    Model fingerprint: 0x691a671d
    Variable types: 6162 continuous, 6158 integer (6158 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+04]
      Objective range  [3e+00, 5e+04]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 1e+05]
    Found heuristic solution: objective 0.0000000
    Presolve removed 13871 rows and 6537 columns
    Presolve time: 0.13s
    Presolved: 4452 rows, 5783 columns, 21573 nonzeros
    Variable types: 2832 continuous, 2951 integer (2951 binary)
    Found heuristic solution: objective -78976.95200

    Concurrent LP optimizer: dual simplex and barrier
    Showing barrier log only...

    Root barrier log...

    Ordering time: 0.02s

    Barrier statistics:
     AA' NZ     : 4.358e+04
     Factor NZ  : 1.867e+05 (roughly 6 MB of memory)
     Factor Ops : 1.833e+07 (less than 1 second per iteration)
     Threads    : 1

                      Objective                Residual
    Iter       Primal          Dual         Primal    Dual     Compl     Time
       0  -1.42994101e+07 -2.02944792e+10  1.48e+04 1.13e+04  1.08e+07     0s
       1  -4.93473482e+06 -1.10263979e+10  4.15e+03 3.64e-11  2.73e+06     0s
       2  -9.89396005e+05 -2.20008295e+09  1.53e+02 1.44e-09  1.95e+05     0s
       3  -8.03924464e+05 -4.99293892e+08  2.51e+00 2.27e-10  3.18e+04     0s
       4  -8.01870282e+05 -2.35705411e+07  9.08e-01 2.91e-11  1.44e+03     0s
       5  -8.09731973e+05 -3.30949502e+06  2.73e-03 1.02e-10  1.57e+02     0s
       6  -8.24420250e+05 -1.65305135e+06  1.21e-04 2.91e-11  5.20e+01     0s
       7  -8.26375963e+05 -9.86494312e+05  8.12e-05 7.28e-12  1.00e+01     0s
       8  -8.29699534e+05 -8.73138153e+05  2.16e-07 1.14e-13  2.72e+00     0s
       9  -8.30362757e+05 -8.54795813e+05  3.70e-08 5.68e-14  1.53e+00     0s
      10  -8.31378240e+05 -8.43818932e+05  1.36e-08 5.68e-14  7.80e-01     0s
      11  -8.32058821e+05 -8.40449153e+05  9.74e-09 2.84e-14  5.26e-01     0s
      12  -8.33208052e+05 -8.38930087e+05  4.01e-09 1.42e-14  3.59e-01     0s
      13  -8.34024668e+05 -8.37174469e+05  2.11e-09 2.84e-14  1.98e-01     0s
      14  -8.34467789e+05 -8.36250981e+05  1.42e-09 2.84e-14  1.12e-01     0s
      15  -8.34864631e+05 -8.35802816e+05  8.19e-10 2.84e-14  5.88e-02     0s
      16  -8.35057930e+05 -8.35660651e+05  5.45e-10 7.28e-12  3.78e-02     0s
      17  -8.35324375e+05 -8.35580992e+05  1.83e-10 2.84e-14  1.61e-02     0s
      18  -8.35434026e+05 -8.35516544e+05  4.54e-11 5.68e-14  5.18e-03     0s
      19  -8.35456125e+05 -8.35496978e+05  2.34e-11 2.84e-14  2.56e-03     0s
      20  -8.35464898e+05 -8.35478095e+05  1.14e-11 5.68e-14  8.28e-04     0s
      21  -8.35470855e+05 -8.35474635e+05  8.37e-12 7.28e-12  2.37e-04     0s
      22  -8.35473584e+05 -8.35473911e+05  5.50e-11 5.68e-14  2.05e-05     0s
      23  -8.35473784e+05 -8.35473795e+05  2.12e-11 1.14e-13  7.25e-07     0s
      24  -8.35473793e+05 -8.35473793e+05  1.67e-11 7.28e-12  1.32e-09     0s

    Barrier solved model in 24 iterations and 0.35 seconds (0.24 work units)
    Optimal objective -8.35473793e+05

    Solved with barrier

    Root relaxation: objective -8.354738e+05, 1305 iterations, 0.23 seconds (0.15 work units)

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 -835473.79    0  143 -78976.952 -835473.79   958%     -    0s
    H    0     0                    -834379.1550 -835473.79  0.13%     -    1s
         0     0 -835397.15    0  163 -834379.16 -835397.15  0.12%     -    1s
    H    0     0                    -834847.2710 -835397.15  0.07%     -    1s
         0     0 -835396.15    0  166 -834847.27 -835396.15  0.07%     -    1s
         0     0 -835396.15    0  166 -834847.27 -835396.15  0.07%     -    1s
         0     0 -835364.41    0  149 -834847.27 -835364.41  0.06%     -    2s
         0     0 -835363.99    0  149 -834847.27 -835363.99  0.06%     -    2s
         0     0 -835363.99    0  149 -834847.27 -835363.99  0.06%     -    2s
         0     0 -835358.49    0  183 -834847.27 -835358.49  0.06%     -    2s
         0     0 -835358.06    0  176 -834847.27 -835358.06  0.06%     -    2s
         0     0 -835358.03    0  185 -834847.27 -835358.03  0.06%     -    2s
         0     0 -835335.07    0  185 -834847.27 -835335.07  0.06%     -    2s
    H    0     0                    -834868.9210 -835335.07  0.06%     -    2s
         0     0 -835330.90    0  184 -834868.92 -835330.90  0.06%     -    2s
         0     0 -835330.56    0  185 -834868.92 -835330.56  0.06%     -    2s
         0     0 -835330.56    0  186 -834868.92 -835330.56  0.06%     -    2s
         0     0 -835325.95    0  187 -834868.92 -835325.95  0.05%     -    2s
         0     0 -835325.91    0  187 -834868.92 -835325.91  0.05%     -    2s
         0     0 -835323.06    0  180 -834868.92 -835323.06  0.05%     -    2s
         0     0 -835322.49    0  180 -834868.92 -835322.49  0.05%     -    3s
         0     0 -835322.43    0  181 -834868.92 -835322.43  0.05%     -    3s
         0     0 -835315.83    0  176 -834868.92 -835315.83  0.05%     -    3s
         0     0 -835315.43    0  180 -834868.92 -835315.43  0.05%     -    3s
         0     0 -835315.42    0  180 -834868.92 -835315.42  0.05%     -    3s
         0     0 -835311.61    0  177 -834868.92 -835311.61  0.05%     -    3s
         0     0 -835311.40    0  178 -834868.92 -835311.40  0.05%     -    3s
         0     0 -835310.37    0  186 -834868.92 -835310.37  0.05%     -    3s
         0     0 -835310.37    0  186 -834868.92 -835310.37  0.05%     -    3s
         0     2 -835304.25    0  186 -834868.92 -835304.25  0.05%     -    4s
        28    55 -835250.87   12  130 -834868.92 -835286.27  0.05%   138    5s
       909   895 -835053.78  201  186 -834868.92 -835276.58  0.05%  45.1   10s
    H  910   851                    -834870.9770 -835252.67  0.05%  45.0   11s
    H  911   808                    -834957.5470 -835196.81  0.03%  45.0   12s
    H  914   769                    -835011.5020 -835182.99  0.02%  44.8   12s
       923   775 -835141.14   98  160 -835011.50 -835170.89  0.02%  44.4   15s
    H  950   760                    -835048.6390 -835170.89  0.01%  51.9   18s
       993   897 -835167.57   32   94 -835048.64 -835170.89  0.01%  56.1   20s
    H 1401   908                    -835052.2090 -835170.74  0.01%  62.1   23s
      1578  1155 -835164.18   30  153 -835052.21 -835168.55  0.01%  64.1   25s
      2624  1649 -835119.09   34  130 -835052.21 -835166.34  0.01%  59.0   30s
      3426  2224 -835116.82   49   88 -835052.21 -835163.55  0.01%  60.5   35s
      4123  2685 -835098.40   82   66 -835052.21 -835161.62  0.01%  59.5   40s
      4818  3122 -835098.75   43  127 -835052.21 -835160.75  0.01%  59.5   47s
      5253  3393 -835138.92   46  127 -835052.21 -835159.78  0.01%  59.9   51s
      5633  3772 -835093.11   39  152 -835052.21 -835159.35  0.01%  60.9   55s
      6242  4287 -835085.26   54  119 -835052.21 -835158.83  0.01%  61.9   60s
      7043  4822 -835121.41   45   91 -835052.21 -835158.42  0.01%  62.5   65s
      7230  4830 -835091.75   55  189 -835052.21 -835158.42  0.01%  64.2   70s
      7238  4839 -835158.42   27  215 -835052.21 -835158.42  0.01%  64.8   76s
      7244  4845 -835158.42   29  219 -835052.21 -835158.42  0.01%  65.1   82s
      7266  4859 -835158.38   32  207 -835052.21 -835158.42  0.01%  66.0   85s
      7426  5140 -835152.31   45  177 -835052.21 -835158.42  0.01%  68.2   93s
      7638  5414 -835137.50   76  123 -835052.21 -835158.42  0.01%  71.4   99s
      8116  5514 -835144.71   33  166 -835052.21 -835158.42  0.01%  73.6  102s
      8521  5556 -835153.65   32  187 -835052.21 -835157.65  0.01%  74.1  105s
      8763  5721 -835142.14   33  187 -835052.21 -835156.79  0.01%  75.6  110s
      9471  6144 -835132.99   44  200 -835052.21 -835154.19  0.01%  75.6  117s
      9816  6205 -835149.43   36  187 -835052.21 -835153.58  0.01%  76.1  120s

    Cutting planes:
      Gomory: 15
      Lift-and-project: 83
      Implied bound: 15
      MIR: 169
      Flow cover: 106
      Zero half: 4
      RLT: 7
      Relax-and-lift: 8

    Explored 10032 nodes (769883 simplex iterations) in 120.02 seconds (127.11 work units)
    Thread count was 4 (of 4 available processors)

    Solution count 10: -835052 -835049 -835012 ... 0

    Time limit reached
    Best objective -8.350522090000e+05, best bound -8.351535772783e+05, gap 0.0121%
    status: OptimizationStatus.FEASIBLE objective value : -835052.2089999999 best possible: -835153.5772782869

---

## Testando resultados

```python
# dist percorrida
sum([(1 if x[i][j].x>0.5 else 0) * c[i,j] for i in V for j in V if i != j])
```

    2457.7910000000006

```python
# demanda total atendida
print(f"demanda atendida: {sum(q[j]*w[j].x for j in N)} | limite demanda: {NUM_VEICULOS*Q} | total demanda: {sum(q)}")
```

    demanda atendida: 83751.0 | limite demanda: 101380 | total demanda: 83751

```python
# # --- PLOTA O GRAFICO ---
# plt.figure(figsize=(10, 6))
# plt.scatter(coord_x[0:], coord_y[0:])
# for i in N:
#     plt.annotate(f"{i}", (coord_x[i], coord_y[i]))
# for i in D:
#     plt.plot(coord_x[i], coord_y[i], c = 'r', marker = 's')

# for i in V:
#     for j in V:
#         if x[i][j].x > 0:
#             plt.plot([coord_x[i], coord_x[j]], [coord_y[i], coord_y[j]], c='g', zorder=0)

# #escala dos eixos
# plt.yticks([i for i in range(1,80,20)]);
# plt.xticks([i for i in range(1,80,20)]);
```

```python
# para onde cada veículo vai
for k in D:
    print(f"{k} - ", end='')
    print([i for i, el in enumerate(u[k]) if el.x > 0])
```

    76 - [6, 28]
    77 - [8, 16, 17, 22, 32, 61, 62, 69]

```python
# caminho dos carros olhando arcos
for k in D:
    print(f"Depot: {k}")
    for i, el in enumerate(x[k]):
        if el.x > 0:
            visitados = [i]
            next_list = [j for j, el in enumerate(x[i]) if el.x > 0]
            while len(next_list) > 0:
                if len(next_list) > 1:
                    print(f"ERROR: {next_list}")
                next_el = next_list[0]
                visitados.append(next_el)
                next_list = [j for j, el in enumerate(x[next_el]) if el.x > 0]
            print(visitados)
```

    Depot: 76
    [6, 56, 44, 35, 58]
    [28, 4, 71]
    Depot: 77
    [8, 59, 75, 38, 12, 55, 54, 19]
    [16]
    [17, 50, 39, 3, 66, 13, 7]
    [22, 45, 20, 14, 29, 37, 33, 31, 21, 23]
    [32, 60, 15, 9, 51, 47]
    [61, 68, 46, 26, 64, 1, 74, 2, 24, 57, 49, 48]
    [62, 36, 34, 10, 53, 5, 43]
    [69, 25, 30]

```python
# caminho dos carros olhando demanda
for k in D:
    print(f"Depot: {k}")
    for i, el in enumerate(u[k]):
        if int(el.x) > 0:
            visitados = [(i, int(el.x))]
            next_list = [(j, int(el.x)) for j, el in enumerate(u[i]) if int(el.x) > 0]
            while len(next_list) > 0:
                if len(next_list) > 1:
                    print(f"ERROR: {next_list}")
                next_el, value = next_list[0]
                visitados.append((next_el, value))
                next_list = [(j, int(el.x)) for j, el in enumerate(u[next_el]) if int(el.x) > 0]
            print([(*el, q[el[0]]) for el in visitados])
```

    Depot: 76
    [(6, 10138, 1790), (56, 8348, 1446), (44, 6902, 2610), (35, 2907, 2189), (58, 718, 719)]
    [(28, 10138, 3319), (4, 6819, 1107), (71, 1256, 1257)]
    Depot: 77
    [(8, 10138, 446), (59, 9399, 1202), (75, 8197, 899), (38, 7298, 1518), (12, 5780, 990), (55, 4790, 1574), (54, 3216, 1431), (19, 1785, 1786)]
    [(16, 10138, 4506)]
    [(17, 10138, 2744), (50, 7394, 1231), (39, 6163, 871), (3, 5292, 2371), (66, 2172, 835), (13, 1337, 765), (7, 572, 573)]
    [(22, 10138, 2245), (45, 7893, 1030), (20, 6858, 846), (14, 6012, 641), (29, 5371, 1510), (37, 3861, 1820), (33, 2041, 749), (31, 1292, 546), (21, 746, 307), (23, 439, 440)]
    [(32, 10138, 992), (60, 9146, 821), (15, 8325, 4446), (9, 3829, 687), (51, 3142, 1118), (47, 2024, 2025)]
    [(61, 10138, 855), (68, 9283, 873), (46, 8410, 924), (26, 7247, 801), (64, 6446, 582), (1, 5864, 797), (74, 5067, 576), (2, 4491, 647), (24, 3844, 1649), (57, 2195, 520), (49, 1675, 416), (48, 1259, 1260)]
    [(62, 10138, 2271), (36, 7867, 854), (34, 7013, 1180), (10, 5256, 2227), (53, 3029, 1479), (5, 1550, 543), (43, 1007, 1008)]
    [(69, 10138, 3760), (25, 6378, 1292), (30, 5086, 835)]

```python
print("Cidades não atendidas")

# cidades não atendidas (olhando arestas)
print(f"aresta:  {([j for j in N if sum([x[i][j].x for i in V]) < 0.5])}")

# nao atendidos olhando demanda
print(f"demanda: {[j for j in N if (q[j] - (sum([u[i][j].x for i in V if i != j]) - sum([u[j][i].x for i in N if i != j]))) > 0.05]}")

# nao atendidos olhando w
print(f"w:       {[j for j in N if w[j].x < 0.5]}")
```

    Cidades não atendidas
    aresta:  [0, 11, 18, 27, 40, 41, 42, 52, 63, 65, 67, 70, 72, 73]
    demanda: []
    w:       [0, 11, 18, 27, 40, 41, 42, 52, 63, 65, 67, 70, 72, 73]

```python
# nao atendidos olhando w
for j in N:
    if w[j].x < 0.5:
        print(f"{j}-> demanda: {q[j]}")
```

    0-> demanda: 0
    11-> demanda: 0
    18-> demanda: 0
    27-> demanda: 0
    40-> demanda: 0
    41-> demanda: 0
    42-> demanda: 0
    52-> demanda: 0
    63-> demanda: 0
    65-> demanda: 0
    67-> demanda: 0
    70-> demanda: 0
    72-> demanda: 0
    73-> demanda: 0

```python
# nao atendidos olhando demanda
for j in N:
    enter = (sum([u[i][j].x for i in V if i != j]) - sum([u[j][i].x for i in N if i != j]))
    if (q[j] - enter) > 0.05:
        print(f"{j}-> enter: {enter}  need: {q[j]}")
```

```python
# nro de veiculos
sum(x[k][i].x for k in D for i in N)
```

    10.0

```python
# cidades que tem mais de dois arcos ligadas a ela (depositos podem ter mais)
print([j for j in V if len([i for i in V if x[j][i].x > 0.5 or x[i][j].x > 0.5]) > 2])
```

    [77]

```python
print(f"distancia media : {sum([c[el] for el in c])/len(c)}")
print(f"distancia maxima: {max([c[el] for el in c])}")
print(f"distancia minima: {min([c[el] for el in c])}")
```

    distancia media : 194.7773214990138
    distancia maxima: 538.506
    distancia minima: 0.0


# Burgers' equation

```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = 'svg'
from ipywidgets import interact
from ipywidgets import widgets
from ipywidgets import FloatSlider, fixed
from exact_solvers import burgers
from exact_solvers import burgers_demos
```

In this chapter, we study a simple scalar nonlinear conservation law: Burgers' equation.  Burgers' equation models momentum transport in a fluid of uniform density and pressure, and it is the simplest equation that captures some key features of gas dynamics or shallow water flow.

To examine the Python code for this chapter, see:

 - [exact_solvers/burgers.py](exact_solvers/burgers.py)
 - [exact_solvers/burgers_demos.py](exact_solvers/burgers_demos.py)

Burgers' equation has been used extensively for developing both theory and numerical methods, and it will allow us to further explore the Riemann problem for nonlinear conservation laws. The Burgers' equation is given by
\begin{align}
q_t + \left(\frac{1}{2}q^2\right)_x = 0.
\label{burgers0}
\end{align}
It is a scalar conservation law with the nonlinear flux term $f(q)=q^2/2$. The quasilinear form is obtained by applying the chain rule to differentiate the flux term:
\begin{align*}
q_t + qq_x = 0.
\end{align*}
This equation looks very similar to the advection equation, with the difference that the advection speed at each point is given by the solution $q$.  The nonlinearity in the Burgers flux term is essentially the same as in the momentum conservation equation in fluid dynamics, so studying its dynamics provides a guideline to understand the more complex nonlinear systems arising in the study of shallow water or Euler equations, as we pursue further in [Shallow_water](Shallow_water.ipynb) and [Euler](Euler.ipynb).

In [Traffic_flow](Traffic_flow.ipynb) we consider another scalar nonlinear conservation law that has a similar structure to Burgers' equation and a nice physical interpretation.

## Shock formation

As the speed of propagation in Burgers' equation is given by $q$ itself, the peak of a wave travels faster than the rest. This behavior is illustrated in the next figure.  The dashed line shows the initial condition while the solid lines show the solution at later times.

```python
interact(burgers_demos.multivalued_solution,
         t=FloatSlider(min=0.,max=9.,value=7.75), fig=fixed(0));
```

Notice that at first $q$ remains single-valued for every $x$. However, after some time the crest of the wave overtakes the leading edge. After this time, we obtain a triple-valued solution for certain values of $x$, which is unphysical since momentum is single-valued. The first time this overtaking happens is referred to as the *breaking time* -- a reference to waves breaking on the beach. It is also the point where the conservation law, in its differential form, breaks down and where the characteristics cross each other for the first time.   When characteristics cross, a shock wave forms. Mathematically, imposing a shock will avoid the problem of the solution being multivalued.  Where should the shock be located?

If we replace part of the multivalued solution interval with a shock, some mass will be removed (area $A_1$ in the figure below) and some mass will be added (area $A_2$ in the figure below). In the live notebook, the figure below shows possible locations for the shock at a given time; in order to maintain conservation of the integral of $q$, the shock must be placed so that these two areas are equal.

```python
interact(burgers_demos.shock_location,
         xshock = FloatSlider(min=6,max=10,step=0.25,value=7.75),
         fig=fixed(0));
```

This geometric reasoning provides a nice intuition for the shock location, but is cumbersome in practice.  To determine the path of the shock we can use the *Rankine-Hugoniot jump condition,* which we will derive in [Traffic_flow](Traffic_flow.ipynb), and requires that the jump in flux across a propagating shock must be related to the jump in $q$ by
$$
s(q_r-q_l)  = f(q_r) - f(q_l),
$$
at each instant in time, where $s$ is the shock speed at this time.

Since the flux for Burgers' equation is $f(q) = q^2/2$, this gives
\begin{align*}
s(q_r-q_l) & = \frac{1}{2} (q_r^2 - q_l^2) \\
\Rightarrow \ \ \ \ s &= \frac{1}{2}(q_l + q_r).
\end{align*}
In general (as in the image above) the states $q_l$ and $q_r$ just to the left and right of the shock will not be constant and the speed of a shock will change in time.

### Shock solution
For Burgers' equation, (or any scalar hyperbolic conservation law with a convex flux function, such as the traffic flow problem considered in [Traffic_flow](Traffic_flow.ipynb)), the solution of the Riemann problem will always consist of a single wave, which may be either a shock or a rarefaction.  The analysis above already yields the solution to the Riemann problem in the case that the resulting wave is a shock.  The entropy condition, as we will explain below, indicates that a shock will form when $f'(q_l) > f'(q_r)$; i.e. when $q_l>q_r$.  In this case, the solution is
\begin{align*}
q(x,t) =
\begin{cases}
q_l \quad \text{if} \ \ x/t<s \\
q_r \quad \text{if} \ \ x/t>s.
\end{cases}
\end{align*}
Below, we plot the solution of the Burgers' equation for an initial condition that leads to a shock (i.e. with $q_l>q_r$).

```python
interact(burgers_demos.shock(),
         t=widgets.FloatSlider(min=0,max=1.0,value=0.5),
         which_char=widgets.Checkbox(value=True,
                                     description='Show characteristics'));
```

## Rarefaction wave
In the previous figure with the hump as the initial condition, we observed that a shock formed on the right side of the hump. However, on the left side, the characteristics spread out and will never cross. This part of the solution is therefore a rarefaction wave. This is the kind of behavior we will observe in the solution of the Riemann problem when $q_l<q_r$.

In the next figure, we consider such a Riemann problem. As time evolves, a rarefaction forms following the advection speed $q$ given by the quasi-linear equation. Therefore, for time $t$, the solution must propagate a distance $x=qt$, so the solution along the rarefaction is then $q = x/t$. As $q_l<q_r$, the smallest and largest displacements are given by $q_l t$ and $q_r t$, respectively.

```python
interact(burgers_demos.rarefaction_figure,
         t = FloatSlider(min=0,max=9,value=5));
```

The full rarefaction solution for the Burgers' Riemann problem is then simply given by
\begin{align*}
q(x,t) =
\begin{cases}
q_l, \quad \text{for} \ \ x<q_l t \\
x/t, \quad \text{for} \ \ q_l t \le x \le q_r t \\
q_r, \quad \text{for} \ \ x>q_r t.
\end{cases}
\end{align*}

As we will see in [Traffic_flow](Traffic_flow.ipynb), the rarefaction solution is always a self-similar solution. This means that it can be expressed as a function of the ratio between position and time $q(x,t) = \tilde{q}(x/t)$, so it remains the same when rescaling both $x$ and $t$ by the same factor. This is a consequence of $q$ being the same along the characteristic in the rarefaction fan. In the Burgers' equation, the form of the rarefaction is particularly simple since the advection speed is simply $q$.

### Rarefaction solution
In the figure below, we plot a solution of the Riemann problem with $q_l<q_r$. In this case, we observe a rarefaction.

```python
interact(burgers_demos.rarefaction(),
         t=widgets.FloatSlider(min=0,max=1.0,value=0.5),
         which_char=widgets.Checkbox(value=True,
                                     description='Show characteristics'));
```

## Weak solutions

As we mentioned before, the differential form of the equation breaks down in the presence of shocks/discontinuities. However, the integral form of the conservation law remains valid. We will derive a specific form of the integral conservation law that is particularly useful to prove mathematical results.
We first integrate the general conservation law $q_t+f(q)_x=0$ from $x=x_1$ to $x=x_2$ and $t=t_1$ to $t=t_2$
\begin{align}
\int_{t_1}^{t_2}\int_{x_1}^{x_2} [q_t+f(q)_x] dx dt = 0,
\label{eq:Burgersintclaw}
\end{align}
This integral can be rewritten in terms of an indicator function $\phi(x,t)$ with compact support in $[x_1,x_2]\times[t_1,t_2]$ (defined to be 1 in this region, zero elsewhere):
\begin{align*}
\int_{0}^{\infty}\int_{-\infty}^{\infty} [q_t+f(q)_x]\phi(x,t) dx dt = 0.
\end{align*}

We can generalize this result by replacing $\phi(x,t)$ by a smooth function with compact support on some region of the $x-t$ plane. In this case, integrating by parts yield
\begin{align*}
\int_{0}^{\infty}\int_{-\infty}^{\infty} [q\phi_t+f(q)\phi_x] dx dt = -\int_{-\infty}^{\infty}q(x,0)\phi(x,0)dx,
\end{align*}
where now the derivatives are on $\phi(x,t)$ and not on $q$ or $f(q)$, so the equation still makes sense for discontinuous $q$. The function $q(x,t)$ is called a *weak solution* of the conservation law if this equation holds for all functions $\phi(x,t)$ that are continuously differentiable and with compact support (bump functions). Note this rules out the original interval chosen in Eq. \ref{eq:Burgersintclaw} since its $\phi$ would not be smooth. However, we can approximate this function arbitrarily well by a smooth function. Note any weak solution is a solution of any form of the integral conservation law and vice versa.

Weak solutions are thus allowed to include discontinuities.  The shock solution presented above, for instance, is a weak solution of the Burgers' equation.  After characteristics cross, we cannot have strong solutions of the conservation law, and we must resort to weak solutions. However, a potential problem with the notion of weak solutions is that in some cases the weak solution is not unique and it is possible to determine more than one function that satisifes the condition above. This generally happens when discontinuous initial data should lead to a rarefaction wave solution but there is also a discontinous weak solution that is allowed mathematically but is not physically correct.

### An unphysical weak solution

We previously mentioned that for Burgers' equation, when $q_l<q_r$, we expect a rarefaction to be the correct solution. This is because if we smeared out the initial data just slightly then there would be smoothly varying characteristics emerging from each point $x$ and these characteristics would spread out.  However, for exactly discontinuous initial data, we could also assume the solution consists of a propagating discontinuity and then use the equation in its conservative form to derive the Rankine-Hugoniot conditions, as done in previous chapters and in (LeVeque, 2002). This is mathematically correct and one obtains the same relation for the shock speed as before, $s = (q_l + q_r)/2$ , so the solution is simply
\begin{align*}
q(x,t) =
\begin{cases}
q_l \quad \text{if} \ \ x/t<s \\
q_r \quad \text{if} \ \ x/t>s.
\end{cases}
\end{align*}
This unphysical solution is also referred to as an expansion shock, and it is also a weak solution to Burgers' equation. In the next figure, we plot this weak solution.

```python
interact(burgers_demos.unphysical(),
         t=widgets.FloatSlider(min=0,max=1.0,value=0.5),
         which_char=widgets.Checkbox(value=True,
                                     description='Show characteristics'));
```

Note the speed of the characteristics with respect to the shock. The spreading characteristics clearly indicate that the correct solution should be a rarefaction. In order to be able to specify which of the weak solutions is physically correct, we need to derive a mathematical condition from our physical intuition gained from observing the behavior of the characteristics. This condition is referred as the entropy condition.

### The entropy condition

Any condition that allows us to determine whether a weak solution to a conservation law is actually physically correct is called an *admissibility condition* or more often an *entropy condition*.  This name comes from gas dynamics, where the physical entropy must increase in the gas passing through a shock, according to the second law of thermodynamics.  A discontinuity that violates this is non-physical. A mathematical version of an "entropy function" can be defined for other conservation laws.  More discussion of this and several other formulations with more detailed explanations are available in the literature, e.g., (LeVeque, 2002).

In the context of the Riemann problem, the entropy condition allows us to determine if the physical solution should involve a shock or a rarefaction. In the case of scalar conservation laws, the entropy condition is quite simple and can be formulated in terms of the flux function of the conservation law as follows: the solution of a scalar Riemann problem will consist of a shock only if
$$
f'(q_l) > f'(q_r).
$$
In other words, the solution is a shock only if the characteristics are going into the shock as time progresses. If the characteristics are spreading out/going out of the shock as in the last example, one would naturally expect a rarefaction to be the correct solution. In the case of Burgers' equation, the flux function is $f(q)=q^2/2$, so the correct solution is a shock only if
$$
q_l > q_r,
$$
which can be clearly observed in the interactive solution below.   In later chapters, we will see how this condition generalizes to systems of conservation laws.

### Convexity

The flux $f(q) = q^2/2$ for Burgers' equation is a convex function, since $f''(q) = 1 > 0$ for all values of $q$. This means that as $q$ varies between $q_l$ and $q_r$ the the characteristic speed $f'(q) = q$ is either monotonically increasing (if $q_l < q_r$) or monotomically decreasing (if $q_l > q_r$).  Because of this the solution is always either a single rarefaction wave or a single shock wave.

The flux $f(\rho) = \rho(1-\rho)$ simple traffic flow model considered in [Traffic_flow](Traffic_flow.ipynb) also has the property that $f'(\rho)$ varies monotonically with $\rho$ (in the context of hyperbolic PDEs, a flux function is referred to as *convex* if either $f''(q)\ge0$ for all $q$ or $f''(q)\le 0$ for all $q$).  Since $f''$ is always negativefor the traffic flow model, the correct Riemann solution is a shock if $\rho_l < \rho_r$, or a rarefaction wave if $\rho_l > \rho_r$.

The solution to the Riemann problem can be much more complicated if $f'(q)$ is not monotonically varying between $q_l$ and $q_r$, i.e. if $f''(q)$ changes sign.  In this case the Riemann solution can consist of multiple shock and rarefaction waves.  The nonconvex case is explored further in [Nonconvex_scalar](Nonconvex_scalar.ipynb).

## Interactive solution and examples
The live notebook next shows a full interactive solution for the Riemann problem for Burgers' equation. The values of the initial conditions and the time can be modified to observe their effect on the characteristic structure and the solution.

```python
burgers.riemann_solution_interact()
```

## Animations of more complex solutions
We can now explore a few more examples that are representative of phenomena we will observe in more complicated systems, with more interesting initial data that the single jump discontinuity of a Riemann problem.  For the animations shown below, the solution is computed numerically using [PyClaw](http://www.clawpack.org/pyclaw/).

### Example 1: Bump initial condition
The animation below, or on [this webpage](http://www.clawpack.org/riemann_book/html/burgers_animation0.html),  shows the solution of Burgers' equation for an initial Gaussian hump.

```python
burgers_demos.bump_animation(numframes = 50)
```

### Example 3: Three state solutions

When three piecewise constant values are given as initial condition, one can solve a Riemann problem across each discontinuity. For instance, consider the Burgers' equation with its initial condition given by

\begin{align*}
q_0(x) =
\begin{cases}
q_l \quad \text{if} \ \ x < -1 \\
q_m \quad \text{if} \ \ -1\le x \le 1 \\
q_r \quad \text{if} \ \ x > 1.
\end{cases}
\end{align*}

We can decompose this into two Riemann problems, one at $x=-1$ and another one at $x=1$. Each of these two Riemann problems will yield either a shock or a rarefaction. The interesting part is what happens when the generated shocks or rarefactions interact. Following the animation below, let us assume that both Riemann problems produce a right-propagating shock. However, the shock generated at $x=-1$ propagates faster, so it will eventually reach the shock originally generated at $x=1$. At the point in time when the shocks collide, one can simply restate the problem again as a Riemann problem and solve the whole problem analytically. We again use PyClaw to solve the problem and show an animated solution. The first plot shows the solution $q(x)$ evolving through time, while the second one shows the characteristic structure in the $x/t$ plane, where the shocks are shown as wide lines and the characteristics as thin lines. In this figure, one can clearly see the two shocks collide to become a new one. In the second plot, the time is marked by an horizontal dashed line. Note one can also solve this problem analytically.  This animation can be viewed in the live notebook on or [this webpage](http://www.clawpack.org/riemann_book/html/burgers_animation1.html).

```python
burgers_demos.triplestate_animation(ql = 4.0, qm = 2.0,
                                    qr = 0.0, numframes = 50)
```

As one would expect, this can become more complicated when one or both of the waves generated are given by rarefactions. The animation below, or on [this webpage](http://www.clawpack.org/riemann_book/html/burgers_animation2.html), shows how a right-propagating shock can overtake a rarefaction. However, there are consequences for the shock after interacting with the rarefaction; its speed is changed. This is clear in the $x/t$ plane, where the slope of the shock is changed after interacting with the rarefaction. Note the shock is again marked as a wider line.

```python
burgers_demos.triplestate_animation(ql = 4.0, qm = -1.5,
                                    qr = 0.5, numframes = 50)
```

Analogously, one can also have a rarefaction colliding into a shock. As we can see in the animation below, or [this webpage](http://www.clawpack.org/riemann_book/html/burgers_animation3.html), this can even change the direction in which the shock is propagating. This happens because the velocity of the shock depends on the slopes of the impinging characteristics. When a shock interacts with a rarefaction, the varying slopes of the characteristic curves produce a constant change of the shock propagation velocity. Try modifying the values of $q_l$, $q_m$ and $q_r$ in order to see what other behavior can be observed. Can one observe two interacting rarefactions?

```python
burgers_demos.triplestate_animation(ql = -1, qm = 3.0,
                                    qr = -2, numframes = 50)
```