Lambda School Data Science

*Unit 2, Sprint 1, Module 2*

---

# Regression 2
- Do train/test split
- Use scikit-learn to fit a multiple regression
- Understand how ordinary least squares regression minimizes the sum of squared errors
- Define overfitting/underfitting and the bias/variance tradeoff

### Setup

Run the code cell below. You can work locally (follow the [local setup instructions](https://lambdaschool.github.io/ds/unit2/local/)) or on Colab.

Libraries:
- matplotlib
- numpy
- pandas
- plotly
- scikit-learn

```python
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'

# If you're working locally:
else:
    DATA_PATH = '../data/'

# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
```

# Do train/test split

## Overview

### Predict Elections! 🇺🇸🗳

How could we try to predict the 2020 US Presidential election?

According to Douglas Hibbs, a political science and economics professor, you can [explain elections with just two features, "Bread and Peace":](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:
>
> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

Let's look at the data that Hibbs collected and analyzed:

```python
import pandas as pd
df = pd.read_csv(DATA_PATH+'elections/bread_peace_voting.csv')
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share

      0
      1952
      Stevenson
      Eisenhower
      2.40
      190
      44.60

      1
      1956
      Eisenhower
      Stevenson
      2.89
      0
      57.76

      2
      1960
      Nixon
      Kennedy
      0.85
      0
      49.91

      3
      1964
      Johnson
      Goldwater
      4.21
      1
      61.34

      4
      1968
      Humphrey
      Nixon
      3.02
      146
      49.60

      5
      1972
      Nixon
      McGovern
      3.62
      0
      61.79

      6
      1976
      Ford
      Carter
      1.08
      2
      48.95

      7
      1980
      Carter
      Reagan
      -0.39
      0
      44.70

      8
      1984
      Reagan
      Mondale
      3.86
      0
      59.17

      9
      1988
      Bush, Sr.
      Dukakis
      2.27
      0
      53.94

      10
      1992
      Bush, Sr.
      Clinton
      0.38
      0
      46.55

      11
      1996
      Clinton
      Dole
      1.04
      0
      54.74

      12
      2000
      Gore
      Bush, Jr.
      2.36
      0
      50.27

      13
      2004
      Bush, Jr.
      Kerry
      1.72
      4
      51.24

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20

Data Sources & Definitions

- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33

Here we have data from the 1952-2016 elections. We could make a model to predict 1952-2016 election outcomes — but do we really care about that?

No, not really. We already know what happened, we don't need to predict it.

This is explained in [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/), Chapter 2.2, Assessing Model Accuracy:

> In general, we do not really care how well the method works training on the training data. Rather, _we are interested in the accuracy of the predictions that we obtain when we apply our method to previously unseen test data._ Why is this what we care about?
>
> Suppose that we are interested in developing an algorithm to predict a stock’s price based on previous stock returns. We can train the method using stock returns from the past 6 months. But we don’t really care how well our method predicts last week’s stock price. We instead care about how well it will predict tomorrow’s price or next month’s price.
>
> On a similar note, suppose that we have clinical measurements (e.g. weight, blood pressure, height, age, family history of disease) for a number of patients, as well as information about whether each patient has diabetes. We can use these patients to train a statistical learning method to predict risk of diabetes based on clinical measurements. In practice, we want this method to accurately predict diabetes risk for _future patients_ based on their clinical measurements. We are not very interested in whether or not the method accurately predicts diabetes risk for patients used to train the model, since we already know which of those patients have diabetes.

So, we're really interested in the 2020 election — but we probably don't want to wait until then to evaluate our model.

There is a way we can estimate now how well our model will generalize in the future. We can't fast-forward time, but we can rewind it...

We can split our data in **two sets.** For example:
1. **Train** a model on elections before 2008.
2. **Test** the model on 2008, 2012, 2016.

This "backtesting" helps us estimate how well the model will predict the next elections going forward, starting in 2020.

This is explained in [_Forecasting,_ Chapter 3.4,](https://otexts.com/fpp2/accuracy.html) Evaluating forecast accuracy:

> The accuracy of forecasts can only be determined by considering how well a model performs on new data that were not used when fitting the model.
>
>When choosing models, it is common practice to separate the available data into two portions, training and test data, where the training data is used to estimate any parameters of a forecasting method and the test data is used to evaluate its accuracy. Because the test data is not used in determining the forecasts, it should provide a reliable indication of how well the model is likely to forecast on new data.
>
>
>
>The size of the test set is typically about 20% of the total sample, although this value depends on how long the sample is and how far ahead you want to forecast. The following points should be noted.
>
>- A model which fits the training data well will not necessarily forecast well.
>- A perfect fit can always be obtained by using a model with enough parameters.
>- Over-fitting a model to data is just as bad as failing to identify a systematic pattern in the data.
>
>Some references describe the test set as the “hold-out set” because these data are “held out” of the data used for fitting. Other references call the training set the “in-sample data” and the test set the “out-of-sample data”. We prefer to use “training data” and “test data” in this book.

## Follow Along

Split the data in two sets:
1. Train on elections before 2008.
2. Test on 2008 and after.

```python
train = df[df['Year'] < 2008]
# train = df.query('Year' < 2008)
test = df[df['Year'] >= 2008]
```

How many observations (rows) are in the train set? In the test set?

```python
len(train), len(test)
```

    (14, 3)

Note that this volume of data is at least two orders of magnitude smaller than we usually want to work with for predictive modeling.

There are other validation techniques that could be used here, such as [time series cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split), or [leave-one-out cross validation](https://scikit-learn.org/stable/modules/cross_validation.html#leave-one-out-loo) for small datasets. However, for this module, let's start simpler, with train/test split.

Using a tiny dataset is intentional here. It's good for learning because we can see all the data at once.

## Challenge

In your assignment, you will do train/test split, based on date.

# Use scikit-learn to fit a multiple regression

## Overview

We've done train/test split, and we're ready to fit a model.

We'll proceed in 3 steps. The first 2 are review from the previous module. The 3rd is new.

- Begin with baselines (0 features)
- Simple regression (1 feature)
- Multiple regression (2 features)

## Follow Along

### Begin with baselines (0 features)

What was the average Incumbent Party Vote Share, in the 1952-2004 elections?

```python
train['Incumbent Party Vote Share'].mean()
```

    52.46857142857142

What if we guessed this number for every election? How far off would this be on average?

```python
from sklearn.metrics import mean_absolute_error

# Arrange y target vectors
target = 'Incumbent Party Vote Share'
y_train = train[target]
y_test = test[target]

# Get mean baseline
print('Mean Baseline (using 0 features)')
guess = y_train.mean()

# Train Error
y_pred = [guess] * len(y_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error (1952-2004 elections): {mae:.2f} percentage points')

# Test Error
y_pred = [guess] * len(y_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error (2008-16 elections): {mae:.2f} percentage points')
```

    Mean Baseline (using 0 features)
    Train Error (1952-2004 elections): 4.85 percentage points
    Test Error (2008-16 elections): 3.63 percentage points

```python
(guess - y_train).abs().mean()
```

    4.846938775510204

### Simple regression (1 feature)

Make a scatterplot of the relationship between 1 feature and the target.

We'll use an economic feature: Average Recent Growth in Personal Incomes. ("Bread")

```python
import pandas as pd
import plotly.express as px

px.scatter(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004',
    trendline='ols',  # Ordinary Least Squares
)
```

1952 & 1968 are outliers: The incumbent party got fewer votes than predicted by the regression. What do you think could explain those years? We'll come back to this soon, but first...

Use scikit-learn to fit the simple regression with one feature.

Follow the [5 step process](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html#Basics-of-the-API), and refer to [Scikit-Learn LinearRegression documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

```python
# 1. Import the appropriate estimator class from Scikit-Learn
from sklearn.linear_model import LinearRegression

# 2. Instantiate this class
model = LinearRegression()

# 3. Arrange X features matrices (already did y target vectors)
features = ['Average Recent Growth in Personal Incomes']
X_train = train[features]
X_test = test[features]
print(f'Linear Regression, dependent on: {features}')

# 4. Fit the model
model.fit(X_train, y_train)
y_pred = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error: {mae:.2f} percentage points')

# 5. Apply the model to new data
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error: {mae:.2f} percentage points')
```

    Linear Regression, dependent on: ['Average Recent Growth in Personal Incomes']
    Train Error: 2.65 percentage points
    Test Error: 1.80 percentage points

How does the error compare to the baseline?

### Multiple regression (2 features)

Make a scatterplot of the relationship between 2 features and the target.

We'll add another feature: US Military Fatalities per Million. ("Peace" or the lack thereof.)

Rotate the scatterplot to explore the data. What's different about 1952 & 1968?

```python
px.scatter_3d(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004'
)
```

Use scikit-learn to fit a multiple regression with two features.

```python
# TODO: Complete this cell

# Re-arrange X features matrices
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']
print(f'Linear Regression, dependent on: {features}')

X_train = train[features]
X_test = test[features]

# Fit the model
model.fit(X_train, y_train)
y_pred_train = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred_train)
print(f'Train Error: {mae:.2f} percentage points')

# Apply the model to new data
y_pred_test = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred_test)
print(f'Test Error: {mae:.2f} percentage points')

```

    Linear Regression, dependent on: ['Average Recent Growth in Personal Incomes', 'US Military Fatalities per Million']
    Train Error: 1.33 percentage points
    Test Error: 1.63 percentage points

How does the error compare to the prior model?

### Plot the plane of best fit

For a regression with 1 feature, we plotted the line of best fit in 2D.

(There are many ways to do this. Plotly Express's `scatter` function makes it convenient with its `trendline='ols'` parameter.)

For a regression with 2 features, we can plot the plane of best fit in 3D!

(Plotly Express has a `scatter_3d` function but it won't plot the plane of best fit for us. But, we can write our own function, with the same "function signature" as the Plotly Express API.)

```python
import itertools
import numpy as np
import plotly.express as px
import plotly.graph_objs as go
from sklearn.linear_model import LinearRegression

def regression_3d(df, x, y, z, num=100, **kwargs):
    """
    Visualize linear regression in 3D: 2 features + 1 target

    df : Pandas DataFrame
    x : string, feature 1 column in df
    y : string, feature 2 column in df
    z : string, target column in df
    num : integer, number of quantiles for each feature
    """

    # Plot data
    fig = px.scatter_3d(df, x, y, z, **kwargs)

    # Fit Linear Regression
    features = [x, y]
    target = z
    model = LinearRegression()
    model.fit(df[features], df[target])

    # Define grid of coordinates in the feature space
    xmin, xmax = df[x].min(), df[x].max()
    ymin, ymax = df[y].min(), df[y].max()
    xcoords = np.linspace(xmin, xmax, num)
    ycoords = np.linspace(ymin, ymax, num)
    coords = list(itertools.product(xcoords, ycoords))

    # Make predictions for the grid
    predictions = model.predict(coords)
    Z = predictions.reshape(num, num).T

    # Plot predictions as a 3D surface (plane)
    fig.add_trace(go.Surface(x=xcoords, y=ycoords, z=Z))

    return fig
```

```python
regression_3d(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004'
)
```

Where are 1952 & 1968 in relation to the plane? Which elections are the biggest outliers now?

Roll over points on the plane to see predicted incumbent party vote share (z axis), dependent on personal income growth (x axis) and military fatatlies per capita (y axis).

### Get and interpret coefficients

During the previous module, we got the simple regression's coefficient and intercept. We plugged these numbers into an equation for the line of best fit, in slope-intercept form: $y = mx + b$

Let's review this objective, but now for multiple regression.

What's the equation for the plane of best fit?

$y = \beta_0 + \beta_1x_1 + \beta_2x_2$

Can you relate the intercept and coefficients to what you see in the plot above?

```python
model.intercept_, model.coef_
```

    (46.25489966153873, array([ 3.59004735, -0.05315709]))

```python
beta0 = model.intercept_
beta1, beta2 = model.coef_
print(f'y = {beta0} + {beta1}x1 + {beta2}x2')
```

    y = 46.25489966153873 + 3.5900473494560536x1 + -0.05315709351049324x2

```python
# This is easier to read
print('Intercept:', '\t'*4 + ' '*4, f'{model.intercept_:.6f}')
coefficients = pd.Series(model.coef_, features)
print(coefficients.to_string())
```

    Intercept: 				     46.254900
    Average Recent Growth in Personal Incomes    3.590047
    US Military Fatalities per Million          -0.053157

One of the coefficients is positive, and the other is negative. What does this mean?

What does the model predict if income growth=0%, fatalities=0

```python
model.predict([[0, 0]])
```

    array([46.25489966])

Income growth = 1% (fatalities = 0)

```python
model.predict([[1, 0]])
```

    array([49.84494701])

The difference between these predictions = ?

```python
model.predict([[1, 0]]) - model.predict([[0, 0]])
```

    array([3.59004735])

What if... income growth = 2% (fatalities = 0)

```python
model.predict([[2, 0]])
```

    array([53.43499436])

The difference between these predictions = ?

```python
model.predict([[2, 0]]) - model.predict([[1, 0]])
```

    array([3.59004735])

What if... (income growth=2%) fatalities = 100

```python
model.predict([[2, 100]])
```

    array([48.11928501])

The difference between these predictions = ?

```python
model.predict([[2, 100]]) - model.predict([[2, 0]])
```

    array([-5.31570935])

What if income growth = 3% (fatalities = 100)

```python
model.predict([[3, 100]])
```

    array([51.70933236])

The difference between these predictions = ?

```python
model.predict([[3, 100]]) - model.predict([[2, 100]])
```

    array([3.59004735])

What if (income growth = 3%) fatalities = 200

```python
model.predict([[3, 200]])
```

    array([46.39362301])

The difference between these predictions = ?

```python
model.predict([[3, 200]]) - model.predict([[3, 100]])
```

    array([-5.31570935])

## Challenge

In your assignment, you'll fit a Linear Regression with at least 2 features.

# Understand how ordinary least squares regression minimizes the sum of squared errors

## Overview

So far, we've evaluated our models by their absolute error. It's an intuitive metric for regression problems.

However, ordinary least squares doesn't directly minimize absolute error. Instead, it minimizes squared error.

In this section, we'll introduce two new regression metrics:

- Squared error
- $R^2$

We'll demostrate two possible methods to minimize squared error:

- Guess & check
- Linear Algebra

## Follow Along

### Guess & Check

This function visualizes squared errors. We'll go back to simple regression with 1 feature, because it's much easier to visualize.

Use the function's m & b parameters to "fit the model" manually. Guess & check what values of m & b minimize squared error.

```python
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

def squared_errors(df, feature, target, m, b):
    """
    Visualize linear regression, with squared errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    df.plot.scatter(feature, target, ax=ax)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    ax.plot(x, y_pred)

    # Plot squared errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    # Print regression metrics
    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)
```

Here's what the mean baseline looks like:

```python
feature = 'Average Recent Growth in Personal Incomes'
squared_errors(train, feature, target, m=0, b=y_train.mean())
```

Notice that $R^2$ is exactly zero.

[$R^2$ represents the proportion of the variance for a dependent variable that is explained by the independent variable(s).](https://en.wikipedia.org/wiki/Coefficient_of_determination)

The mean baseline uses zero independent variables and explains none of the variance in the dependent variable, so its $R^2$ score is zero.

The highest possible $R^2$ score is 1. The lowest possible *Train* $R^2$ score with ordinary least squares regression is 0.

In this demo, it's possible to get a negative Train $R^2$, if you manually set values of m & b that are worse than the mean baseline. But that wouldn't happen in the real world.

However, in the real world, it _is_ possible to get a negative *Test/Validation* $R^2$. It means that your *Test/Validation* predictions are worse than if you'd constantly predicted the mean of the *Test/Validation* set.

---

Now that we've visualized the squared errors for the mean baseline, let's guess & check some better values for the m & b parameters:

```python
squared_errors(train, feature, target, m=3, b=46)
```

You can run the function repeatedly, with different values for m & b.

How do you interpret each metric you see?

- Mean Squared Error
- Root Mean Squared Error
- Mean Absolute Error
- $R^2$

Does guess & check really get used in machine learning? Sometimes! Some complex functions are hard to minimize, so we use a sophisticated form of guess & check called "gradient descent", which you'll learn about in Unit 4.

GridSearchCV, RandomizedSearchCV

Fortunately, we don't need to use guess & check for ordinary least squares regression. We have a solution, using linear algebra!

### Linear Algebra

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

#### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

#### Lets calculate our $\beta$ parameters with numpy!

```python
# This is NOT something you'll be tested on. It's just a demo.

# X is a matrix. Add column of constants for fitting the intercept.
def add_constant(X):
    constant = np.ones(shape=(len(X),1))
    return np.hstack((constant, X))
X = add_constant(train[features].values)
print('X')
print(X)

# y is a column vector
y = train[target].values[:, np.newaxis]
print('y')
print(y)

# Least squares solution in code
X_transpose = X.T
X_transpose_X = X_transpose @ X
X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
X_transpose_y = X_transpose @ y
beta_hat = X_transpose_X_inverse @ X_transpose_y

print('Beta Hat')
print(beta_hat)
```

    X
    [[  1.     2.4  190.  ]
     [  1.     2.89   0.  ]
     [  1.     0.85   0.  ]
     [  1.     4.21   1.  ]
     [  1.     3.02 146.  ]
     [  1.     3.62   0.  ]
     [  1.     1.08   2.  ]
     [  1.    -0.39   0.  ]
     [  1.     3.86   0.  ]
     [  1.     2.27   0.  ]
     [  1.     0.38   0.  ]
     [  1.     1.04   0.  ]
     [  1.     2.36   0.  ]
     [  1.     1.72   4.  ]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]]
    Beta Hat
    [[46.25489966]
     [ 3.59004735]
     [-0.05315709]]

```python
# Scikit-learn gave the exact same results!
model.intercept_, model.coef_
```

    (46.25489966153873, array([ 3.59004735, -0.05315709]))

# Define overfitting/underfitting and the bias/variance tradeoff

## Overview

Read [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off). Jake VanderPlas explains overfitting & underfitting:

> Fundamentally, the question of "the best model" is about finding a sweet spot in the tradeoff between bias and variance. Consider the following figure, which presents two regression fits to the same dataset:
>
>
>
> The model on the left attempts to find a straight-line fit through the data. Because the data are intrinsically more complicated than a straight line, the straight-line model will never be able to describe this dataset well. Such a model is said to _underfit_ the data: that is, it does not have enough model flexibility to suitably account for all the features in the data; another way of saying this is that the model has high _bias_.
>
> The model on the right attempts to fit a high-order polynomial through the data. Here the model fit has enough flexibility to nearly perfectly account for the fine features in the data, but even though it very accurately describes the training data, its precise form seems to be more reflective of the particular noise properties of the data rather than the intrinsic properties of whatever process generated that data. Such a model is said to _overfit_ the data: that is, it has so much model flexibility that the model ends up accounting for random errors as well as the underlying data distribution; another way of saying this is that the model has high _variance_.

VanderPlas goes on to connect these concepts to the "bias/variance tradeoff":

> From the scores associated with these two models, we can make an observation that holds more generally:
>
>- For high-bias models, the performance of the model on the validation set is similar to the performance on the training set.
>
>- For high-variance models, the performance of the model on the validation set is far worse than the performance on the training set.
>
> If we imagine that we have some ability to tune the model complexity, we would expect the training score and validation score to behave as illustrated in the following figure:
>
>
>
> The diagram shown here is often called a validation curve, and we see the following essential features:
>
>- The training score is everywhere higher than the validation score. This is generally the case: the model will be a better fit to data it has seen than to data it has not seen.
>- For very low model complexity (a high-bias model), the training data is under-fit, which means that the model is a poor predictor both for the training data and for any previously unseen data.
>- For very high model complexity (a high-variance model), the training data is over-fit, which means that the model predicts the training data very well, but fails for any previously unseen data.
>- For some intermediate value, the validation curve has a maximum. This level of complexity indicates a suitable trade-off between bias and variance.
>
>The means of tuning the model complexity varies from model to model.

So far, our only "means of tuning the model complexity" has been selecting one feature or two features for our linear regression models. But we'll quickly start to select more features, and more complex models, with more "hyperparameters."

This is just a first introduction to underfitting & overfitting. We'll continue to learn about this topic all throughout this unit.

## Follow Along

Let's make our own Validation Curve, by tuning a new type of model complexity: polynomial degrees in a linear regression.

Go back to the the NYC Tribeca condo sales data

```python
# Read NYC Tribeca condo sales data, from first 4 months of 2019.
# Dataset has 90 rows, 9 columns.
df = pd.read_csv(DATA_PATH+'condos/tribeca.csv')
assert df.shape == (90, 9)

# Arrange X features matrix & y target vector
features = ['GROSS_SQUARE_FEET']
target = 'SALE_PRICE'
X = df[features]
y = df[target]
```

Do random [train/test split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=11)
```

Repeatedly fit increasingly complex models, and keep track of the scores

```python
from IPython.display import display, HTML
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

# Credit for PolynomialRegression: Jake VanderPlas, Python Data Science Handbook, Chapter 5.3
# https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#Validation-curves-in-Scikit-Learn
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))

polynomial_degrees = range(1, 10, 2)
train_r2s = []
test_r2s = []

for degree in polynomial_degrees:
    model = PolynomialRegression(degree)
    display(HTML(f'Polynomial degree={degree}'))

    model.fit(X_train, y_train)
    train_r2 = model.score(X_train, y_train)
    test_r2 = model.score(X_test, y_test)
    display(HTML(f'<b style="color: blue">Train R2 {train_r2:.2f}</b>'))
    display(HTML(f'<b style="color: red">Test R2 {test_r2:.2f}</b>'))

    plt.scatter(X_train, y_train, color='blue', alpha=0.5)
    plt.scatter(X_test, y_test, color='red', alpha=0.5)
    plt.xlabel(features)
    plt.ylabel(target)

    x_domain = np.linspace(X.min(), X.max())
    curve = model.predict(x_domain)
    plt.plot(x_domain, curve, color='blue')
    plt.show()
    display(HTML('<hr/>'))

    train_r2s.append(train_r2)
    test_r2s.append(test_r2)

display(HTML('Validation Curve'))
plt.plot(polynomial_degrees, train_r2s, color='blue', label='Train')
plt.plot(polynomial_degrees, test_r2s, color='red', label='Test')
plt.xlabel('Model Complexity (Polynomial Degree)')
plt.ylabel('R^2 Score')
plt.legend()
plt.show()
```

As model complexity increases, what happens to Train $R^2$ and Test $R^2$?

# Review

In your assignment, you'll continue to **predict how much it costs to rent an apartment in NYC,** using the dataset from renthop.com.

- Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.
- Engineer at least two new features.
- Fit a linear regression model with at least two features.
- Get the model's coefficients and intercept.
- Get regression metrics RMSE, MAE, and $R^2$, for both the train and test sets.

You've been provided with a separate notebook for your assignment, which has all the instructions and stretch goals. What's the best test MAE you can get? Share your score and features used with your cohort on Slack!

# Sources

#### Train/Test Split
- James, Witten, Hastie, Tibshirani, [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/), Chapter 2.2, Assessing Model Accuracy
- Hyndman, Athanasopoulos, [_Forecasting,_ Chapter 3.4,](https://otexts.com/fpp2/accuracy.html) Evaluating forecast accuracy

#### Bias-Variance Tradeoff
- Jake VanderPlas, [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off), Hyperparameters and Model Validation

#### "Bread and Peace" Background
- Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

#### "Bread and Peace" Data Sources & Definitions
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33


### Постановка задачи
Рассмотрим несколько моделей линейной регрессии, чтобы выяснить более оптимальную для первых 20 зданий.

Данные:
* http://video.ittensive.com/machine-learning/ashrae/building_metadata.csv.gz
* http://video.ittensive.com/machine-learning/ashrae/weather_train.csv.gz
* http://video.ittensive.com/machine-learning/ashrae/train.0.csv.gz
Соревнование: https://www.kaggle.com/c/ashrae-energy-prediction/

© ITtensive, 2020

```python
import pandas as pd
from pandas.tseries.holiday import USFederalHolidayCalendar as calendar
import numpy as np
from scipy.interpolate import interp1d
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Lasso, Ridge, ElasticNet, BayesianRidge
```

```python
def reduce_mem_usage (df):
    start_mem = df.memory_usage().sum() / 1024**2
    for col in df.columns:
        col_type = df[col].dtypes
        if str(col_type)[:5] == "float":
            c_min = df[col].min()
            c_max = df[col].max()
            if c_min > np.finfo("f2").min and c_max < np.finfo("f2").max:
                df[col] = df[col].astype(np.float16)
            elif c_min > np.finfo("f4").min and c_max < np.finfo("f4").max:
                df[col] = df[col].astype(np.float32)
            else:
                df[col] = df[col].astype(np.float64)
        elif str(col_type)[:3] == "int":
            c_min = df[col].min()
            c_max = df[col].max()
            if c_min > np.iinfo("i1").min and c_max < np.iinfo("i1").max:
                df[col] = df[col].astype(np.int8)
            elif c_min > np.iinfo("i2").min and c_max < np.iinfo("i2").max:
                df[col] = df[col].astype(np.int16)
            elif c_min > np.iinfo("i4").min and c_max < np.iinfo("i4").max:
                df[col] = df[col].astype(np.int32)
            elif c_min > np.iinfo("i8").min and c_max < np.iinfo("i8").max:
                df[col] = df[col].astype(np.int64)
        elif col == "timestamp":
            df[col] = pd.to_datetime(df[col])
        elif str(col_type)[:8] != "datetime":
            df[col] = df[col].astype("category")
    end_mem = df.memory_usage().sum() / 1024**2
    print('Потребление памяти меньше на', round(start_mem - end_mem, 2), 'Мб (минус', round(100 * (start_mem - end_mem) / start_mem, 1), '%)')
    return df
```

```python
buildings = pd.read_csv("http://video.ittensive.com/machine-learning/ashrae/building_metadata.csv.gz")
weather = pd.read_csv("http://video.ittensive.com/machine-learning/ashrae/weather_train.csv.gz")
energy = pd.read_csv("http://video.ittensive.com/machine-learning/ashrae/train.0.csv.gz")
energy = energy[(energy["building_id"]<20)]
energy = pd.merge(left=energy, right=buildings, how="left",
                   left_on="building_id", right_on="building_id")
energy = energy.set_index(["timestamp", "site_id"])
weather = weather.set_index(["timestamp", "site_id"])
energy = pd.merge(left=energy, right=weather, how="left",
                  left_index=True, right_index=True)
energy.reset_index(inplace=True)
energy = energy.drop(columns=["meter", "site_id", "year_built",
                              "square_feet", "floor_count"], axis=1)
del buildings
del weather
energy = reduce_mem_usage(energy)
print (energy.info())
```

    Потребление памяти меньше на 10.39 Мб (минус 70.5 %)
    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 175680 entries, 0 to 175679
    Data columns (total 11 columns):
     #   Column              Non-Null Count   Dtype
    ---  ------              --------------   -----
     0   timestamp           175680 non-null  datetime64[ns]
     1   building_id         175680 non-null  int8
     2   meter_reading       175680 non-null  float16
     3   primary_use         175680 non-null  category
     4   air_temperature     175620 non-null  float16
     5   cloud_coverage      99080 non-null   float16
     6   dew_temperature     175620 non-null  float16
     7   precip_depth_1_hr   175660 non-null  float16
     8   sea_level_pressure  173980 non-null  float16
     9   wind_direction      170680 non-null  float16
     10  wind_speed          175680 non-null  float16
    dtypes: category(1), datetime64[ns](1), float16(8), int8(1)
    memory usage: 4.4 MB
    None

```python
energy["hour"] = energy["timestamp"].dt.hour.astype("int8")
energy["weekday"] = energy["timestamp"].dt.weekday.astype("int8")
for weekday in range(0,7):
    energy['is_wday' + str(weekday)] = energy['weekday'].isin([weekday]).astype("int8")
energy["date"] = pd.to_datetime(energy["timestamp"].dt.date)
dates_range = pd.date_range(start='2015-12-31', end='2017-01-01')
us_holidays = calendar().holidays(start=dates_range.min(),
                                  end=dates_range.max())
energy['is_holiday'] = energy['date'].isin(us_holidays).astype("int8")
energy["meter_reading_log"] = np.log(energy["meter_reading"] + 1)
```

```python
energy_train,energy_test = train_test_split(energy[energy['meter_reading'] > 0],test_size=0.2)
```

```python
from sklearn.metrics import *
```

```python
hours = range(0,24)
buildings = range(0,energy_train['building_id'].max() + 1)
lr_columns = ['meter_reading_log','hour','building_id','is_holiday']
for wday in range(0,7):
    lr_columns.append('is_wday' + str(wday))

```

Линейная регрессия
\begin{equation}
z = Ax + By + C, |z-z_0|^2 \rightarrow min
\end{equation}
Лассо + LARS Лассо
\begin{equation}
\frac{1}{2n}|z-z_0|^2 + a(|A|+|B|) \rightarrow min
\end{equation}
Гребневая регрессия
\begin{equation}
|z-z_0|^2 + a(A^2 + B^2) \rightarrow min
\end{equation}
ElasticNet: Лассо + Гребневая регрессия
\begin{equation}
\frac{1}{2n}|z-z_0|^2 + \alpha p|A^2+B^2| + (\alpha - p)(|A|+|B|)/2 \rightarrow min
\end{equation}

```python
lr_models = {
    "LinearRegression":LinearRegression,
    "Lasso-0.01":Lasso,
    "Lasso-0.1":Lasso,
    "Lasso-1.0":Lasso,
    "Ridge-0.01":Ridge,
    "Ridge-0.1":Ridge,
    "Ridge-1.0":Ridge,
    "ELasticNet-1-1":ElasticNet,
    "ELasticNet-0.1-1":ElasticNet,
    "ELasticNet-1-0.1":ElasticNet,
    "ELasticNet-0.1-0.1":ElasticNet,
    "BayesianRidge":BayesianRidge
}
energy_train_lr = pd.DataFrame(energy_train,columns=lr_columns)
```

```python
lr_models_scores = {}
for _ in lr_models:
    lr_model = lr_models[_]
    energy_lr_scores = [[]] * len(buildings)
    for building in buildings:
        energy_lr_scores[building] = [0] * len(hours)
        energy_train_b = energy_train_lr[energy_train_lr['building_id'] == building]
        for hour in hours:
            energy_train_bh = energy_train_b[energy_train_b['hour'] == hour]
            y = energy_train_bh['meter_reading_log']
            x = energy_train_bh.drop(['meter_reading_log','hour','building_id'],axis=1)
            if _ in ['Ridge-0.1','Lasso-0.1']:
                model = lr_model(alpha=0.1,fit_intercept=False).fit(x,y)
            elif _ in ['Ridge-0.01','Lasso-0.01']:
                model = lr_model(alpha=0.01,fit_intercept=False).fit(x,y)
            elif _ == 'ElasticNet-1-1':
                model = lr_model(alpha=1,l1_ratio=1,fit_intercept=False).fit(x,y)
            elif _ == 'ElasticNet-1-0.1':
                model = lr_model(alpha=1,l1_ratio=0.1,fit_intercept=False).fit(x,y)
            elif _ == 'ElasticNet-0.1-1':
                model = lr_model(alpha=0.1,l1_ratio=1,fit_intercept=False).fit(x,y)
            elif _ == 'ElasticNet-0.1-0.1':
                model = lr_model(alpha=0.1,l1_ratio=0.1,fit_intercept=False).fit(x,y)
            else:
                model = lr_model(fit_intercept=False).fit(x,y)
            energy_lr_scores[building][hour] = r2_score(y,model.predict(x))
    lr_models_scores[_] = np.mean(energy_lr_scores)
print(lr_models_scores)
```

    {'LinearRegression': 0.13262979264188432, 'Lasso-0.01': -0.19060262906360775, 'Lasso-0.1': -31.177850789729533, 'Lasso-1.0': -2429.7711638954247, 'Ridge-0.01': 0.13221503036076057, 'Ridge-0.1': 0.09151042473014116, 'Ridge-1.0': -3.6576414865818228, 'ELasticNet-1-1': -2105.6896312858844, 'ELasticNet-0.1-1': -2105.6896312858844, 'ELasticNet-1-0.1': -2105.6896312858844, 'ELasticNet-0.1-0.1': -2105.6896312858844, 'BayesianRidge': 0.13261999641258232}

```python
energy_lr = []
energy_ridge = []
energy_br = []
for building in buildings:
    energy_lr.append([])
    energy_ridge.append([])
    energy_br.append([])
    energy_train_b = energy_train_lr[energy_train_lr['building_id'] == building]
    for hour in hours:
        energy_lr[building].append([0] * (len(lr_columns)-3))
        energy_ridge[building].append([0] * (len(lr_columns)-3))
        energy_br[building].append([0] * (len(lr_columns)-3))
        energy_train_bh = energy_train_b[energy_train_b['hour'] == hour]
        y = energy_train_bh['meter_reading_log']
        if len(y) > 0:
            x = energy_train_bh.drop(['meter_reading_log','hour','building_id'],axis=1)
            model = LinearRegression(fit_intercept=False).fit(x,y)
            energy_lr[building][hour] = model.coef_
            model = Ridge(alpha=0.01,fit_intercept=False).fit(x,y)
            energy_ridge[building][hour] = model.coef_
            model = BayesianRidge(fit_intercept=False).fit(x,y)
            energy_br[building][hour] = model.coef_
print(energy_lr[0][0])
print(energy_ridge[0][0])
print(energy_br[0][0])
```

    [-0.05204313  5.44504565  5.41921165  5.47881611  5.41753305  5.43838778
      5.45137392  5.44059806]
    [-0.04938976  5.44244413  5.41674949  5.47670968  5.41516617  5.43591691
      5.44949479  5.43872264]
    [-0.05138182  5.44439819  5.41859905  5.47829205  5.41694412  5.43777302
      5.45090643  5.44013149]


# Imports and preamble

```python
# !pip install pulp
# !apt-get install glpk-utils
# !apt-get install coinor-cbc
```

    Collecting pulp
      Downloading https://files.pythonhosted.org/packages/fb/34/ff5915ff6bae91cfb7c4cc22c3c369a6aea0b2127045dd5f308a91c260ac/PuLP-2.0-py3-none-any.whl (39.2MB)
        100% |████████████████████████████████| 39.2MB 1.1MB/s
    Requirement already satisfied: pyparsing>=2.0.1 in /home/jefehern/anaconda3/lib/python3.7/site-packages (from pulp) (2.3.1)
    Installing collected packages: pulp
    Successfully installed pulp-2.0
    E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
    E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
    E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
    E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

```python
import numpy as np
import pulp
```

## Kestrel solver

```python
import time
import xmlrpc.client
import sys
import os
import lxml.etree
import lxml.builder
from pulp.solvers import LpSolver_CMD
from pulp.solvers import PulpSolverError
from pulp.constants import *
```

```python
class Kestrel(LpSolver_CMD):
    """
    API Wrapper for Neos Solver XML-RPC API. Compatible with pulp linear programming api.
    Only Xpress solver works; in the future more solvers will be added
    :param ptype: problem type for example 'milp': mixed integer linear programming
    olny tested with 'milp', 'lp'
    :param maximize: 'yes' or 'no' to maximize or minimize respectibly
    :param email: add your email
    :param priority: default 'long', if 'short' is scpecified job will be killed after 5 min
    :param  keepFiles = 0, mip = 1, msg = 0, options = []: neccesary to make LpSolver_CMD  work
    """
    def __init__(self, ptype, maximize=None, email=None, priority=None, keepFiles=0, mip=1, msg=0, options=[]):
        self.ptype = ptype
        if maximize:
            self.maximize = 'yes'
        else:
            self.maximize = "no"
        if email:
            self.email=email
        else:
            raise ValueError('Missing email')
        if priority:
            self.priority = priority
        else:
            self.priority = 'long'
        LpSolver_CMD.__init__(self, './', keepFiles, mip, msg, options)

    def copy(self):
        """Make a copy of self"""
        aCopy = LpSolver_CMD.copy(self)
        aCopy.cuts = self.cuts
        aCopy.presolve = self.presolve
        aCopy.dual = self.dual
        aCopy.strong = self.strong
        return aCopy

    def actualSolve(self, lp, **kwargs):
        """Solve a well formulated lp problem"""
        return self.solve_Kestrel(lp, **kwargs)

    def available(self):
        """True if the solver is available"""
        neos = xmlrpc.client.ServerProxy("https://neos-server.org:3333")
        alive = neos.ping()
        if alive!="NeosServer is alive\n":
            print("Could not make connection to NEOS Server")
            available = False
        else:
            available = True
        return available

    def solve_Kestrel(self, lp):
        vs = lp.writeMPS("problem.mps", rename = 0)
        file = open('problem.mps', "r")
        E = lxml.builder.ElementMaker()
        root = E.document
        field1 = E.category
        field2 = E.solver
        field3 = E.inputMethod
        field4 = E.MPS
        field5 = E.maximize
        field6 = E.nosol
        field7 = E.email
        field8 = E.priority
        xmldoc = root(
                     field1(self.ptype),
                     field2("FICO-Xpress"),
                     field3("MPS"),
                     field4(lxml.etree.CDATA(file.read())),
                     field5(self.maximize),
                     field6("no"),
                     field7(self.email),
                     field8(self.priority))
        xml = lxml.etree.tostring(xmldoc).decode()
        file.close()
        try:
            os.remove('problem.mps')
        except:
            pass
        neos = xmlrpc.client.ServerProxy("https://neos-server.org:3333")
        alive = neos.ping()
        if alive != "NeosServer is alive\n":
            raise RuntimeError("Could not make connection to NEOS Server")
        else:
            (jobNumber, password) = neos.submitJob(xml)
            print("Job number = {} Job password = {}".format(jobNumber, password))
            if jobNumber == 0:
                raise RuntimeError("NEOS Server error: {}".format(password))
            else:
                status=""
                while status!="Done":
                    time.sleep(5)
                    status= neos.getJobStatus(jobNumber, password)
                    print('Solving... Problem status: ', status)
                msg=neos.getFinalResults(jobNumber, password)
                #print(msg.data.decode())
                tmpSol=open('tmpSol.txt', 'w')
                tmpSol.write(msg.data.decode())
                tmpSol.close()
        values= self.readsol_MPS(self.path+'tmpSol.txt')
        try:
            os.remove('tmpSol.txt')
        except:
            pass
        kestrelStatus = {"Done": LpStatusOptimal}
        if status not in kestrelStatus:
            raise PulpSolverError("Unknown status returned by Kestrel: " + statusString)
        lp.status = kestrelStatus[status]
        lp.assignVarsVals(values)
        return lp.status

    def readsol_MPS(self, filename):
        with open(filename,'r') as f:
            values = {}
            while 1:
                l = f.readline()
                if l == "": break
                line = l.split()
                if len(line) and line[0] == 'C':
                    name = line[2]
                    value = float(line[4])
                    values[name] = value
        return values
```

# Job shop Scheduling
Scheduling es la asignación de recursos a lo largo del tiempo. En los Job shop mal programados, no es nada raro que los trabajos esperen durante el 95 por ciento de su ciclo de producción total. Esto resulta en un largo ciclo de flujo de trabajo. Si a esto le sumamos el tiempo de inventario y el tiempo de cobro de cuentas por cobrar, se obtendrá un largo ciclo de flujo de caja. Por lo tanto, el flujo de trabajo es igual al flujo de caja, y el flujo de trabajo es impulsado por el schedule. Un schedule es un calendario para realizar actividades, utilizar recursos o asignar instalaciones.

En un problema de Scheduling tenemos que realizar un conjunto de trabajos en varios procesadores que utilizan otros recursos bajo ciertas restricciones, como restricciones en los tiempos de finalización del trabajo, prioridades entre trabajos (un trabajo no puede comenzar hasta que otro está terminado), etc. El objetivo es optimizar algún criterio, por ejemplo, minimizar el tiempo total de procesamiento, que es el tiempo de finalización del último trabajo (suponiendo que el primer trabajo comienza en el tiempo $0$); o maximizar el número de trabajos procesados.

A continuación formulamos un problema de scheduling muy general que subsume como casos especiales una gran cantidad de problemas de scheduling estudiados en la literatura.

## Fomulacion del problema de Job shop scheduling
Se nos dan $n$ trabajos para ser procesados en $m$ procesadores (máquinas). Sea $M_j \subseteq \mathcal{P} = \{ 1, \cdots, m\}$ el subconjunto de procesadores que pueden cumplir con el trabajo $j \in \mathcal{J} = \{ 1, \cdots, n\}$. Cada trabajo $j$ se caracteriza por los siguientes parámetros:

*   $w_j$: peso del trabajo
*   $r_j, d_j$: fechas de liberación y vencimiento (el trabajo debe procesarse durante el intervalo de tiempo $[r_j, d_j]$
*   $p_{jk}$: tiempo de procesamiento en la maquina $k$

Las relaciones de precedencia entre puestos de trabajo están dadas por un dígrafo acíclico $G = (\mathcal{J}, E)$ definido sobre el conjunto $\mathcal{J}$: paa cada arco $(j_1, j_2) \in \mathcal{J}$, el trabajo $j_2$ no puede empezar hasta que el trabajo $j_1$ termine.

El Job shop Scheduling implica decidir cuándo iniciar cada paso de cada trabajo en su procesador. Por lo tanto, las variables de decisión de la hora de inicio están ahora indexadas tanto por el trabajo como por el procesador:
\begin{equation}
x_{jk} = \text{hora de inicio del trabajo j en el procesador k}
\end{equation}
Los pasos de los distintos trabajos que se programan en job shop deben tener lugar en la secuencia indicada. Es decir, las horas de inicio están sujetas a restricciones de precedencia.
la restriction de precedencia que el trabajo $j$ se debe completar en el procesador $k$ antes de que comience la actividad en $k'$ puede expresarse de la siguiente manera
\begin{equation}
x_{jk} + p_{jk} \leq x_{jk'}
\end{equation}
donde $x_{jk}$ denota el tiempo de inicio del trabajo $j$ en el procesador $k$, $p_{jk}$  es el tiempo de proceso de $j$ en $k$, y $x_{jk'}$  es el tiempo de inicio del trabajo $j$ en el procesador $k'$.
Se puede modelar los conflictos introduciendo variables de decisión discretas
\begin{equation}
        y_{jj'k}=
        \begin{cases}
            1 & \text{si j está programado antes del trabajo j' en el procesador k} \\
            0 &\text{si no}
        \end{cases}
\end{equation}
Asi como las restricciones
\begin{equation}
x_{jk} + p_{jk} \leq x_{j'k} + M(1- y_{jj'k}) \\
x_{j'k} + p_{j'k} \leq x_{jk} + My_{jj'k}
\end{equation}
Una de las características intrigantes de los modelos Job shop Scheduling es la amplia variedad de funciones objetivo que pueden ser apropiadas.

Funcion objetivo |  Ecuacion
--- | -----
Maximum completion time | $\max_j \{ x_{j\hat{k}} + p_{j\hat{k}}\}$
Mean completion time | $\frac{1}{n} \sum_j x_{j\hat{k}} + p_{j\hat{k}}$
Maximum flow time | $\max_j \{ x_{j\hat{k}} + p_{j\hat{k}} - r_j\}$
Mean flow time | $\frac{1}{n} \sum_j x_{j\hat{k}} + p_{j\hat{k}} - r_j$
Maximum lateness | $\max_j \{ x_{j\hat{k}} + p_{j\hat{k}} - d_j\}$
Mean lateness | $\frac{1}{n} \sum_j x_{j\hat{k}} + p_{j\hat{k}} - d_j$
Maximum tardiness| $\max_j \{\max\{0, x_{j\hat{k}} + p_{j\hat{k}} - d_j\}\}$
Mean tardiness| $\frac{1}{n} \sum_j \{\max\{0, x_{j\hat{k}} + p_{j\hat{k}} - d_j\}$

donde $\hat{k}$ es la ultima maquina donde debe ocurrir el trabajo $j$

## Formulacion en Pulp

```python
# rows = jobs
# columns = machines
procesing_times = np.array(
    [
        [29,78,9,36,49,11,62,56,44,21],
        [43,90,75,11,69,28,46,46,72,30],
        [91,85,39,74,90,10,12,89,45,33],
        [81,95,71,99,9,52,85,98,22,43],
        [14,6,22,61,26,69,21,49,72,53],
        [84,2,52,95,48,72,47,65,6,25],
        [46,37,61,13,32,21,32,89,30,55],
        [31,86,46,74,32,88,19,48,36,79],
        [76,69,76,51,85,11,40,89,26,74],
        [85,13,61,7,64,76,47,52,90,45]
    ]
)
```

```python
# rows = jobs
# columns = machine index
order = np.array(
    [
        [0,1,2,3,4,5,6,7,8,9],
        [0,2,4,9,3,1,6,5,7,8],
        [1,0,3,2,8,5,7,6,9,4],
        [1,2,0,4,6,8,7,3,9,5],
        [2,0,1,5,3,4,8,7,9,6],
        [2,1,5,3,8,9,0,6,4,7],
        [1,0,3,2,6,5,9,8,7,4],
        [2,0,1,5,4,6,8,9,7,3],
        [0,1,3,5,2,9,6,7,4,8],
        [1,0,2,6,8,9,5,3,4,7]
    ]
)
```

```python
def job_shop(procesing_times, order):
    n_jobs, n_maq  = procesing_times.shape

    x = {}
    for job in range(n_jobs):
        for maq in range(n_maq):
            x[(job, maq)] = pulp.LpVariable('x_{}_{}'.format(job, maq), 0, None, pulp.LpContinuous)
    y = {}
    for maq in range(n_maq):
        for job in range(n_jobs):
            for job_ in range(n_jobs):
                if job < job_:
                    y[(job, job_, maq)] = pulp.LpVariable('y_{}_{}_{}'.format(job, job_, maq), 0, 1, pulp.LpInteger)
    z = pulp.LpVariable('z', 0, None, pulp.LpContinuous)

    prob = pulp.LpProblem("Job_shop", pulp.LpMinimize)

    # Set objective
    prob += z

    # create precedance constrains
    for job in range(n_jobs):
        for maq in range(1, n_maq):
            prob += x[(job, order[job, maq])] >= x[(job, order[job, maq-1])] + procesing_times[job ,order[job, maq-1]]

    # create conflict constrains
    for maq in range(n_maq):
        for job in range(n_jobs):
            for job_ in range(n_jobs):
                if job < job_:
                    prob += x[(job_, maq)] >= x[(job, maq)] + procesing_times[job, maq] - 9999*y[(job, job_, maq)]
                    prob += x[(job, maq)] >= x[(job_, maq)] + procesing_times[job_, maq] - 9999*(1 - y[(job, job_, maq)])

    # Objective function
    for job in range(n_jobs):
        prob += z >= x[(job, order[job, -1])] + procesing_times[job, order[job, -1]]
    return prob

# Solving
# prob = job_shop(procesing_times, order)
# prob.writeLP("job_shop.lp")
# prob.solve(Kestrel(ptype='milp', email='jefehern@espol.edu.ec'))
```

```python
x_sol = np.zeros((n_jobs, n_maq))
for v in prob.variables():
    if v.name != 'z':
        line = v.name.split(sep='_')
        if line[0] == 'x':
            x_sol[int(line[1]), int(line[2])] = v.varValue
    else:
        z_sol = v.varValue
```

```python
x_sol = np.array(
   [[ 83, 138, 228, 260, 321, 422, 433, 526, 902, 951],
    [188, 480, 283, 469, 370, 644, 583, 672, 727, 439],
    [428, 309, 593, 519, 882, 677, 807, 718, 632, 849],
    [231,  43, 157, 593, 312, 920, 321, 428, 406, 806],
    [ 22,  37,   0, 112, 173,  43, 412, 271, 199, 320],
    [568, 225,  22, 374, 834, 275, 691, 907, 469, 543],
    [ 37,   0,  96,  83, 439, 189, 157, 339, 271, 210],
    [312, 394, 237, 898, 640, 552, 672, 807, 691, 727],
    [112, 227, 358, 296, 672, 347, 543, 583, 946, 469],
    [343, 296, 434, 763, 770, 687, 495, 855, 542, 642]]
)
z_sol = 972
```

```python
import random
import matplotlib.patches as patches
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
```

```python
plt.rcParams["font.size"] = "20"
cmap = plt.get_cmap('Paired')
colors = cmap(np.linspace(0, 1, n_maq))
fig, ax = plt.subplots(1, 1, figsize=(30,10))
ax.set_title('Job shop')
height = 0.9
for job in range(n_jobs):
    for maq in range(n_maq):
        ax.add_patch(patches.Rectangle((x_sol[job, maq], job-0.5),
                                       procesing_times[job, maq],
                                       height,
                                       color=colors[maq],
                                       alpha=1))
ax.set_ylim(-0.5, n_jobs)
ax.set_xlim(0, (x_sol + procesing_times).max())
ax.set_yticks([job for job in range(n_jobs)])
ax.set_yticklabels(['Job {}'.format(job) for job in range(n_jobs, -1, -1)])
plt.legend(handles=[mpatches.Patch(color=colors[maq], label='Maq {}'.format(maq)) for maq in range(n_maq)],
               bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.show()
```

## Shifting bottleneck heuristic
El Shifting Bottleneck Heuristic es un procedimiento destinado a minimizar el tiempo que se tarda en hacer el trabajo, o específicamente, el makepan en un taller de trabajo. El makespan se define como la cantidad de tiempo, de principio a fin, para completar un conjunto de trabajos con varias máquinas en los que el pedido de la máquina está preestablecido para cada trabajo. Suponiendo que los puestos de trabajo compitan realmente por los mismos recursos (máquinas), siempre habrá uno o más recursos que actúen como un "cuello de botella" en el procesamiento. Este procedimiento heurístico o "regla empírica" minimiza el efecto del cuello de botella.

```python
import networkx as nx
```

```python
class Job(object):
    def __init__(self, Id, r, p):
        self.Id = Id
        self.r = r  # route
        self.p = p  # processing times
```

```python
class Jobshop(nx.DiGraph):
    def __init__(self):  # , jobs):
        super().__init__()
        self.machines = {}
        self.add_node("U", p=0)
        self.add_node("V", p=0)

    def handleJobRoutings(self, jobs):
        for j in jobs.values():
            self.add_edge("U", (j.r[0], j.Id))
            for m, n in zip(j.r[:-1], j.r[1:]):
                self.add_edge((m, j.Id), (n, j.Id))
            self.add_edge((j.r[-1], j.Id), "V")

    def handleJobProcessingTimes(self, jobs):
        for j in jobs.values():
            for m, p in zip(j.r, j.p):
                self.add_node((m, j.Id), p=p)

    def makeMachineSubgraphs(self):
        machineIds = set(ij[0] for ij in self if ij[0] not in ("U", "V"))
        for m in machineIds:
            self.machines[m] = self.subgraph(ij for ij in self if ij[0] == m)
            self.machines[m].remove_nodes_from(["U", "V"])

    def addJobs(self, jobs):
        self.handleJobRoutings(jobs)
        self.handleJobProcessingTimes(jobs)
        self.makeMachineSubgraphs()

    def output(self):
        for m in sorted(self.machines):
            for j in sorted(self.machines[m]):
                print("{}: {}".format(j, self.node[j]['C']))
```


# Market Sharing

## Objective and Prerequisites

In this example, we’ll show you how to solve a goal programming problem that involves allocating the retailers to two divisions of a company in order to optimize the trade-offs of several market sharing goals. You’ll learn how to create a mixed integer linear programming model of the problem using the Gurobi Python API and how to find an optimal solution to the problem using the Gurobi Optimizer.

This model is example 13 from the fifth edition of Model Building in Mathematical Programming by H. Paul Williams on pages 267-268 and 322-324.

This modeling example is at the beginner level, where we assume that you know Python and that you have some knowledge about building mathematical optimization models. The reader should also consult the  [documentation](https://www.gurobi.com/resources/?category-filter=documentation)
of the Gurobi Python API.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-MUI-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_MARKET_SHARING_COM_EVAL_GitHub&utm_term=Market_Sharing&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EDU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_MARKET_SHARING_ACADEMIC_EVAL_GitHub&utm_term=Market_Sharing&utm_content=C_JPM) as an *academic user*.

## Problem Description

A large company has two divisions: D1 and D2. The company supplies retailers with oil and spirit. The goal is to allocate each retailer to either division D1 or division D2. The allocated division will be the retailer’s supplier. As far as possible, the allocation must be made so that D1 controls 40% of the market and D2 the remaining 60%. The retailers in the table below are listed as M1 to M23. Each retailer has an estimated market for oil and spirit. Retailers M1 to M8 are in region 1, retailers M9 to M18 are in region 2, and retailers M19 to M23 are in region 3. Certain retailers are considered to have good growth prospects and categorized as group A and the others are in group B. Each retailer has a certain number of delivery points.

We want to make the 40%/60% split between D1 and D2 in each of the following categories:
1. Total number of delivery points
2. Control of spirit market
3. Control of oil market in region 1
4. Control of oil market in region 2
5. Control of oil market in region 3
6. Number of retailers in group A
7. Number of retailers in group B.

There is flexibility in that any market share may vary by $\pm$ 5%. That is, the share can vary between the limits 35%/65% and 45%/55%. The objective is to minimize the sum of the percentage deviations from the 40%/60% split.

## Model Formulation

### Sets and Indices

$r \in \text{Retailers}=\{\ 1,2,...,23\}$

### Parameters

$\text{deliveryPoints}_{r} \in \mathbb{N}^+$: Delivery points of retailer $r$.

$\text{spiritMarket}_{r} \in \mathbb{R}^+$: Spirit market -in millions of gallons, of retailer $r$.

$\text{oilMarket1}_{r} \in \mathbb{R}^+$: Oil market -in millions of gallons of retailer $r$ in Region 1.

$\text{oilMarket2}_{r} \in \mathbb{R}^+$: Oil market -in millions of gallons of retailer $r$ in Region 2.

$\text{oilMarket3}_{r} \in \mathbb{R}^+$: Oil market -in millions of gallons of retailer $r$ in Region 3.

$\text{retailerA}_{r} \in \{0,1\}$: Parameter has a value of 1 if retailer $r$  belongs to group A.

$\text{retailerB}_{r} \in \{0,1\}$: Parameter has a value of 1 if retailer $r$  belongs to group B.

$\text{deliveryPoints40} \in \mathbb{R}^+$: Forty percent of the delivery points.

$\text{deliveryPoints5} \in \mathbb{R}^+$: Five percent of the delivery points.

$\text{spiritMarket40} \in \mathbb{R}^+$: Forty percent of  the spirit market.

$\text{spiritMarket5} \in \mathbb{R}^+$: Five percent of the spirit market.

$\text{oilMarket1_40} \in \mathbb{R}^+$: Forty percent of  the oil market in region 1.

$\text{oilMarket1_5} \in \mathbb{R}^+$: Five percent of  the oil market in region 1.

$\text{oilMarket2_40} \in \mathbb{R}^+$: Forty percent of  the oil market in region 2.

$\text{oilMarket2_5} \in \mathbb{R}^+$: Five percent of  the oil market in region 2.

$\text{oilMarket3_40} \in \mathbb{R}^+$: Forty percent of  the oil market in region 3.

$\text{oilMarket3_5} \in \mathbb{R}^+$: Five percent of  the oil market in region 3.

$\text{retailerA40} \in \mathbb{R}^+$: Forty percent of the number of retailers in group A.

$\text{retailerA5} \in \mathbb{R}^+$: Five percent of the number of retailers in group A.

$\text{retailerB40} \in \mathbb{R}^+$: Forty percent of the number of retailers in group B.

$\text{retailerB5} \in \mathbb{R}^+$: Five percent of the number of retailers in group B.

### Decision Variables

$\text{allocate}_{r} \in \{0,1\}$: This binary variable is equal 1, if retailer r is allocated to Division 1, and 0 if allocated to Division 2.

$\text{deliveryPointsPos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent of the delivery points.

$\text{deliveryPointsNeg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent of the delivery points.

$\text{spiritMarketPos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent  of the spirit market.

$\text{spiritMarketNeg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent  of the spirit market.

$\text{oilMarket1Pos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent  of  the oil market in region 1.

$\text{oilMarket1Neg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent  of  the oil market in region 1.

$\text{oilMarket2Pos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent  of  the oil market in region 2.

$\text{oilMarket2Neg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent  of the oil market in region 2.

$\text{oilMarket3Pos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent  of  the oil market in region 3.

$\text{oilMarket3Neg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent  of the oil market in region 3.

$\text{retailerAPos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent of the number of retailers in group A.

$\text{retailerANeg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent of the number of retailers in group A.

$\text{retailerBPos} \in \mathbb{R}^+$: This decision variable measures the positive deviation of the retailers’ allocation for the goal of satisfying forty percent of the number of retailers in group B.

$\text{retailerBNeg} \in \mathbb{R}^+$: This decision variable measures the negative deviation of the retailers’ allocation for the goal of satisfying forty percent of the number of retailers in group B.

### Constraints

**Delivery points**: The allocation of retailers at Division 1 satisfies as much as possible forty percent of the delivery points.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{deliveryPoints}_{r}*{\text{allocate}_{r}} + \text{deliveryPointsPos} - \text{deliveryPointsNeg}  = \text{deliveryPoints40}
\end{equation}

**Spirit Market**: The allocation of retailers at Division 1 satisfies as much as possible forty percent of the spirit market.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{spiritMarket}_{r}*{\text{allocate}_{r}} + \text{spiritMarketPos} -
\text{spiritMarketNeg}  = \text{spiritMarket40}
\end{equation}

**Oil market region 1**: The allocation of retailers in region 1 at Division 1 satisfies as much as possible forty percent of the oil market in that region.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{oilMarket1}_{r}*{\text{allocate}_{r}} + \text{oilMarket1Pos} -
\text{oilMarket1Neg}  = \text{oilMarket1_40}
\end{equation}

**Oil market region 2**: The allocation of retailers in region 2 at Division 1 satisfies as much as possible forty percent of the oil market in that region.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{oilMarket2}_{r}*{\text{allocate}_{r}} + \text{oilMarket2Pos} -
\text{oilMarket2Neg}  = \text{oilMarket2_40}
\end{equation}

**Oil market region 3**: The allocation of retailers in region 3 at Division 1 satisfies as much as possible forty percent of the oil market in that region.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{oilMarket3}_{r}*{\text{allocate}_{r}} + \text{oilMarket3Pos} -
\text{oilMarket3Neg}  = \text{oilMarket3_40}
\end{equation}

**Group A**: The allocation of retailers at Division 1 satisfies as much as possible forty percent of the retailers in group A.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{retailerA40}_{r}*{\text{allocate}_{r}} + \text{retailerAPos} -
\text{retailerANeg}  = \text{retailerA40}
\end{equation}

**Group B**: The allocation of retailers at Division 1 satisfies as much as possible forty percent of the retailers in group B.

\begin{equation}
\sum_{r \in \text{Retailers}} \text{retailerB40}_{r}*{\text{allocate}_{r}} + \text{retailerBPos} -
\text{retailerBNeg}  = \text{retailerB40}
\end{equation}

**Flexibility**: There is flexibility in that any market share may vary by $\pm$ 5%.

$$
\text{deliveryPointsPos} \leq \text{deliveryPoints5}
$$

$$
\text{deliveryPointsNeg}  \leq \text{deliveryPoints5}
$$

$$
\text{spiritMarketPos} \leq \text{spiritMarket5}
$$

$$
\text{spiritMarketNeg}  \leq \text{spiritMarket5}
$$

$$
\text{oilMarket1Pos} \leq \text{oilMarket1_5}
$$

$$
\text{oilMarket1Neg} \leq \text{oilMarket1_5}
$$

$$
\text{oilMarket2Pos} \leq \text{oilMarket2_5}
$$

$$
\text{oilMarket2Neg} \leq \text{oilMarket2_5}
$$

$$
\text{oilMarket3Pos} \leq \text{oilMarket3_5}
$$

$$
\text{oilMarket3Neg} \leq \text{oilMarket3_5}
$$

$$
\text{retailerAPos} \leq \text{retailerA5}
$$

$$
\text{retailerANeg} \leq \text{retailerA5}
$$

$$
\text{retailerBPos} \leq \text{retailerB5}
$$

$$
\text{retailerBNeg} \leq \text{retailerB5}
$$

### Objective Function

**Minimize deviations**: Minimize the sum of positive and negative deviations.

\begin{equation}
\text{Minimize} \quad  \text{deliveryPointsPos} + \text{deliveryPointsNeg} + \text{spiritMarketPos} + \text{spiritMarketNeg} +
\text{oilMarket1Pos} + \text{oilMarket1Neg}
\end{equation}

$$
+ \text{oilMarket2Pos} + \text{oilMarket2Neg} + \text{oilMarket3Pos} + \text{oilMarket3Neg}
$$

$$
+ \text{retailerAPos} + \text{retailerANeg} + \text{retailerBPos} + \text{retailerBNeg}
$$

## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
import numpy as np
import pandas as pd
from itertools import product

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input data

We define all the input data for the model.

```python
# Create a dictionary to capture the delivery points and spirit market -in millions of gallons.

retailers, deliveryPoints, spiritMarket = gp.multidict({
    (1): [11,34],
    (2): [47,411],
    (3): [44,82],
    (4): [25,157],
    (5): [10,5],
    (6): [26,183],
    (7): [26,14],
    (8): [54,215],
    (9): [18,102],
    (10): [51,21],
    (11): [20,54],
    (12): [105,0],
    (13): [7,6],
    (14): [16,96],
    (15): [34,118],
    (16): [100,112],
    (17): [50,535],
    (18): [21,8],
    (19): [11,53],
    (20): [19,28],
    (21): [14,69],
    (22): [10,65],
    (23): [11,27]
})

# Create a dictionary to capture the oil market -in millions of gallons for region 1.

retailers1,  oilMarket1 = gp.multidict({
    (1): 9,
    (2): 13,
    (3): 14,
    (4): 17,
    (5): 18,
    (6): 19,
    (7): 23,
    (8): 21
})

# Create a dictionary to capture the oil market -in millions of gallons for region 2.

retailers2,  oilMarket2 = gp.multidict({
    (9): 9,
    (10): 11,
    (11): 17,
    (12): 18,
    (13): 18,
    (14): 17,
    (15): 22,
    (16): 24,
    (17): 36,
    (18): 43
})

# Create a dictionary to capture the oil market -in millions of gallons for region 3.

retailers3,  oilMarket3 = gp.multidict({
    (19): 6,
    (20): 15,
    (21): 15,
    (22): 25,
    (23): 39
})

# Create a dictionary to capture retailers in group A.

groupA,  retailerA = gp.multidict({
    (1): 1,
    (2): 1,
    (3): 1,
    (5): 1,
    (6): 1,
    (10): 1,
    (15): 1,
    (20): 1
})

# Create a dictionary to capture retailers in group B.

groupB,  retailerB = gp.multidict({
    (4): 1,
    (7): 1,
    (8): 1,
    (9): 1,
    (11): 1,
    (12): 1,
    (13): 1,
    (14): 1,
    (16): 1,
    (17): 1,
    (18): 1,
    (19): 1,
    (21): 1,
    (22): 1,
    (23): 1
})

# Forty and five percentages of each goal

deliveryPoints40 = 292
deliveryPoints5 = 36.5
spiritMarket40 = 958
spiritMarket5 = 119.75
oilMarket1_40 = 53.6
oilMarket1_5 = 6.7
oilMarket2_40 = 86
oilMarket2_5 = 10.75
oilMarket3_40 = 40
oilMarket3_5 = 5
retailerA40 = 3.2
retailerA5 = 0.4
retailerB40 = 6
retailerB5 = 0.75
```

## Model Deployment

We create a model and the variables. The main decision variable is a binary variable that is equal to 1  when a retailer is allocated to Division 1, and 0 when allocated it to Division 2. The rest of the decision variables measure positive and negative deviations from the seven goals of the 40%/60% split.

```python
model = gp.Model('MarketSharing')

# Allocation of retailers to Division 1.
allocate = model.addVars(retailers, vtype=GRB.BINARY, name="allocate")

# Positive and negative deviation of delivery points goal.

deliveryPointsPos = model.addVar(ub= deliveryPoints5, name='deliveryPointsPos')
deliveryPointsNeg = model.addVar(ub= deliveryPoints5, name='deliveryPointsNeg')

# Positive and negative deviation of spirit market goal.

spiritMarketPos = model.addVar(ub=spiritMarket5, name='spiritMarketPos')
spiritMarketNeg = model.addVar(ub=spiritMarket5, name='spiritMarketNeg')

# Positive and negative deviation of oil market in region 1 goal.

oilMarket1Pos = model.addVar(ub=oilMarket1_5, name='oilMarket1Pos')
oilMarket1Neg = model.addVar(ub=oilMarket1_5, name='oilMarket1Neg')

# Positive and negative deviation of oil market in region 2 goal.

oilMarket2Pos = model.addVar(ub=oilMarket2_5, name='oilMarket2Pos')
oilMarket2Neg = model.addVar(ub=oilMarket2_5, name='oilMarket2Neg')

# Positive and negative deviation of oil market in region 3 goal.

oilMarket3Pos = model.addVar(ub=oilMarket3_5, name='oilMarket3Pos')
oilMarket3Neg = model.addVar(ub=oilMarket3_5, name='oilMarket3Neg')

# Positive and negative deviation of retailers in group A goal.

retailerAPos  = model.addVar(ub=retailerA5, name='retailerAPos')
retailerANeg  = model.addVar(ub=retailerA5, name='retailerANeg')

# Positive and negative deviation of retailers in group B goal.

retailerBPos  = model.addVar(ub=retailerB5, name='retailerBPos')
retailerBNeg  = model.addVar(ub=retailerB5, name='retailerBNeg')
```

    Using license file c:\gurobi\gurobi.lic

The allocation of retailers at Division 1 satisfies as much as possible forty percent of the delivery points.

```python
# Delivery points constraint.

DPConstr = model.addConstr((gp.quicksum(deliveryPoints[r]*allocate[r] for r in retailers)
                            + deliveryPointsPos - deliveryPointsNeg == deliveryPoints40), name='DPConstrs')
```

The allocation of retailers at Division 1 satisfies as much as possible forty percent of the spirit market.

```python
# Spirit market constraint.

SMConstr = model.addConstr((gp.quicksum(spiritMarket[r]*allocate[r] for r in retailers)
                            + spiritMarketPos - spiritMarketNeg == spiritMarket40), name='SMConstr')
```

The allocation of retailers in region 1 at Division 1 satisfies as much as possible forty percent of the oil market in that region.

```python
# Oil market in region 1 constraint.

OM1Constr = model.addConstr((gp.quicksum(oilMarket1[r]*allocate[r] for r in retailers1)
                            + oilMarket1Pos - oilMarket1Neg == oilMarket1_40), name='OM1Constr')
```

The allocation of retailers in region 2 at Division 1 satisfies as much as possible forty percent of the oil market in that region.

```python
# Oil market in region 2 constraint.

OM2Constr = model.addConstr((gp.quicksum(oilMarket2[r]*allocate[r] for r in retailers2)
                            + oilMarket2Pos - oilMarket2Neg == oilMarket2_40), name='OM2Constr')
```

The allocation of retailers in region 3 at Division 1 satisfies as much as possible forty percent of the oil market in that region.

```python
# Oil market in region 3 constraint.

OM3Constr = model.addConstr((gp.quicksum(oilMarket3[r]*allocate[r] for r in retailers3)
                            + oilMarket3Pos - oilMarket3Neg == oilMarket3_40), name='OM3Constr')
```

The allocation of retailers at Division 1 satisfies as much as possible forty percent of the retailers in group A.

```python
# Group A constraint.

AConstr = model.addConstr((gp.quicksum(retailerA[r]*allocate[r] for r in groupA)
                            + retailerAPos - retailerANeg == retailerA40), name='AConstr')
```

The allocation of retailers at Division 1 satisfies as much as possible forty percent of the retailers in group B.

```python
# Group B constraint.

BConstr = model.addConstr((gp.quicksum(retailerB[r]*allocate[r] for r in groupB)
                            + retailerBPos - retailerBNeg == retailerB40), name='BConstr')
```

Minimize the sum of positive and negative deviations.

```python
# Objective function

obj = deliveryPointsPos + deliveryPointsNeg+ spiritMarketPos + spiritMarketNeg + oilMarket1Pos + oilMarket1Neg + oilMarket2Pos + oilMarket2Neg + oilMarket3Pos + oilMarket3Neg + retailerAPos + retailerANeg + retailerBPos + retailerBNeg

model.setObjective(obj)
```

```python
# Verify model formulation

model.write('marketSharing.lp')

# Run optimization engine

model.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 7 rows, 37 columns and 105 nonzeros
    Model fingerprint: 0xa5aab3a9
    Variable types: 14 continuous, 23 integer (23 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 5e+02]
      Objective range  [1e+00, 1e+00]
      Bounds range     [4e-01, 1e+02]
      RHS range        [3e+00, 1e+03]
    Presolve time: 0.00s
    Presolved: 7 rows, 37 columns, 105 nonzeros
    Variable types: 14 continuous, 23 integer (23 binary)

    Root relaxation: objective 0.000000e+00, 13 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0    0.00000    0    7          -    0.00000      -     -    0s
    H    0     0                     131.6000000    0.00000   100%     -    0s
    H    0     0                     111.6000000    0.00000   100%     -    0s
    H    0     0                      70.6000000    0.00000   100%     -    0s
    H    0     0                      66.8000000    0.00000   100%     -    0s
         0     0    0.20000    0    6   66.80000    0.20000   100%     -    0s
    H    0     0                      63.8000000    0.20000   100%     -    0s
    H    0     0                      54.8000000    0.20000   100%     -    0s
    H    0     0                      42.4235294    0.20000   100%     -    0s
    H    0     0                      39.3737179    0.20000  99.5%     -    0s
    H    0     0                      38.8000000    0.20000  99.5%     -    0s
         0     0    0.20000    0    7   38.80000    0.20000  99.5%     -    0s
    H    0     0                      30.8000000    0.20000  99.4%     -    0s
    H    0     0                      24.8000000    0.20000  99.2%     -    0s
         0     0    0.60000    0   11   24.80000    0.60000  97.6%     -    0s
    H    0     0                       7.6000000    0.60000  92.1%     -    0s
         0     0    0.60000    0    7    7.60000    0.60000  92.1%     -    0s
         0     0    0.60000    0   10    7.60000    0.60000  92.1%     -    0s
         0     0    0.60000    0   10    7.60000    0.60000  92.1%     -    0s
         0     0    0.60000    0   10    7.60000    0.60000  92.1%     -    0s
         0     0    0.60000    0    8    7.60000    0.60000  92.1%     -    0s
         0     0    0.67457    0    6    7.60000    0.67457  91.1%     -    0s
         0     0    0.92289    0   11    7.60000    0.92289  87.9%     -    0s
         0     0    6.72359    0    8    7.60000    6.72359  11.5%     -    0s
         0     0    7.38975    0    8    7.60000    7.38975  2.77%     -    0s
         0     2    7.38975    0    8    7.60000    7.38975  2.77%     -    0s

    Cutting planes:
      Gomory: 1
      Cover: 39
      MIR: 24
      StrongCG: 3
      Inf proof: 5
      RLT: 4
      Relax-and-lift: 1

    Explored 439 nodes (2410 simplex iterations) in 0.11 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 10: 7.6 24.8 30.8 ... 131.6

    Optimal solution found (tolerance 1.00e-04)
    Best objective 7.599999999999e+00, best bound 7.599999999999e+00, gap 0.0000%

## Analysis

The allocation of retailers to Division 1 that minimizes the sum of positive and negative deviations from the goal follows. In addition, we show how each goal is within the 35%/45% range values.

```python
# Output reports

print("\n\n_________________________________________________________________________________")
print(f"The optimal allocation of retailers to Division 1 is:")
print("_________________________________________________________________________________")
for r in retailers:
    if(allocate[r].x > 0.5):
        print(f"Retailer{r}")

#print(f"\nThe optimal objective function value is {model.objVal}")
```

    _________________________________________________________________________________
    The optimal allocation of retailers to Division 1 is:
    _________________________________________________________________________________
    Retailer2
    Retailer6
    Retailer7
    Retailer9
    Retailer12
    Retailer13
    Retailer14
    Retailer15
    Retailer23

The following report validates that the goals have been satisfied within acceptable 35% and 45% ranges.

```python
# Test that the solution is within acceptable ranges.

goal_ranges = pd.DataFrame(columns=["Goal", "Min_35", "Actual", "Max_45"])

count = 0

DeliveryPointsGoal = 0
for r in retailers:
    if (allocate[r].x > 0.5):
        DeliveryPointsGoal += deliveryPoints[r]*allocate[r].x

goal_ranges = goal_ranges.append({"Goal": 'Delivery points', "Min_35": round(deliveryPoints40*(0.35/0.40),2), "Actual": round(DeliveryPointsGoal,2), "Max_45": round(deliveryPoints40*(0.45/0.40),2) }, ignore_index=True)
count += 1

spiritMarketGoal = 0
for r in retailers:
    if (allocate[r].x > 0.5):
        spiritMarketGoal += spiritMarket[r]*allocate[r].x

goal_ranges = goal_ranges.append({"Goal": 'Spirit market', "Min_35": round(spiritMarket40*(0.35/0.40),2),
                                  "Actual": round(spiritMarketGoal,2), "Max_45": round(spiritMarket40*(0.45/0.40),2) },
                                 ignore_index=True)
count += 1

oilMarket1Goal = 0
for r in retailers1:
    if (allocate[r].x > 0.5):
        oilMarket1Goal += oilMarket1[r]*allocate[r].x

goal_ranges = goal_ranges.append({"Goal": 'Oil market1', "Min_35": round(oilMarket1_40*(0.35/0.40),2),
                                  "Actual": round(oilMarket1Goal,2), "Max_45": round(oilMarket1_40*(0.45/0.40),2) },
                                 ignore_index=True)
count += 1

oilMarket2Goal = 0
for r in retailers2:
    if (allocate[r].x > 0.5):
        oilMarket2Goal += oilMarket2[r]*allocate[r].x
#
goal_ranges = goal_ranges.append({"Goal": 'Oil market2', "Min_35": round(oilMarket2_40*(0.35/0.40),2),
                                  "Actual": round(oilMarket2Goal,2), "Max_45": round(oilMarket2_40*(0.45/0.40),2) },
                                 ignore_index=True)
count += 1

oilMarket3Goal = 0
for r in retailers3:
    if (allocate[r].x > 0.5):
        oilMarket3Goal += oilMarket3[r]*allocate[r].x
#
goal_ranges = goal_ranges.append({"Goal": 'Oil market3', "Min_35": round(oilMarket3_40*(0.35/0.40),2),
                                  "Actual": round(oilMarket3Goal,2), "Max_45": round(oilMarket3_40*(0.45/0.40),2) },
                                 ignore_index=True)
count += 1

retailerAGoal = 0
for r in groupA:
    if (allocate[r].x > 0.5):
        retailerAGoal += retailerA[r]*allocate[r].x
#
goal_ranges = goal_ranges.append({"Goal": 'Group A', "Min_35": round(retailerA40*(0.35/0.40),2),
                                  "Actual": round(retailerAGoal,2), "Max_45": round(retailerA40*(0.45/0.40),2) },
                                 ignore_index=True)
count += 1

retailerBGoal = 0
for r in groupB:
    if (allocate[r].x > 0.5):
        retailerBGoal += retailerB[r]*allocate[r].x
#
goal_ranges = goal_ranges.append({"Goal": 'Group B', "Min_35": round(retailerB40*(0.35/0.40),2),
                                  "Actual": round(retailerBGoal,2), "Max_45": round(retailerB40*(0.45/0.40),2) },
                                 ignore_index=True)
count += 1

goal_ranges.index=[''] * count
goal_ranges
```

      Goal
      Min_35
      Actual
      Max_45

      Delivery points
      255.50
      290.0
      328.50

      Spirit market
      838.25
      957.0
      1077.75

      Oil market1
      46.90
      55.0
      60.30

      Oil market2
      75.25
      84.0
      96.75

      Oil market3
      35.00
      39.0
      45.00

      Group A
      2.80
      3.0
      3.60

      Group B
      5.25
      6.0
      6.75

## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```