## Logistic Regression

- Models the probability an object belongs to a class
- Values ranges from 0 to 1
- Can use threshold to classify into which classes a class belongs
- An S-shaped curve

$
\begin{align}
\sigma(t) = \frac{1}{1 + e^{-t}}
\end{align}
$

```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import math
```

### How does the Logistic Regression function looks like

```python
from matplotlib import pyplot as plt
import math

x = [i for i in range(-10, 10)]
y = [1.0/(1+math.exp(-1*i)) for i in x]

ax = plt.figure(figsize=(8, 6))
plt.plot(x, y,  color='black')
plt.axhline(0.5)
plt.show()
```

### Read the data and plot

```python
import pandas as pd

# Read the data
df_data = pd.read_csv('../data/2d_classification.csv')

# Plot the data
plt.rcParams['figure.figsize'] = [10, 7] # Size of the plots

colors = {0:'b', 1:'g', 2:'r', 3:'c', 4:'m', 5:'y', 6:'k'}
fig, ax = plt.subplots()
grouped = df_data.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
plt.show()
```

### Another way to plot the data

```python
import pandas as pd
import numpy as np

# Read the data
df_data = pd.read_csv('../data/2d_classification.csv')

colors = {0:'b', 1:'g', 2:'r', 3:'c', 4:'m', 5:'y', 6:'k'}

plt.figure()
unq_labels = np.unique(df_data['label'])
for i in unq_labels:
    df = df_data.loc[df_data['label'] == i][['x','y']]
    x = df['x']
    y = df['y']
    plt.scatter(x, y, c=colors[i], alpha=1)
```

### Just using some friendly variable names

```python
data = df_data[['x','y']].values
label = df_data['label'].values
```

### Our first Logistic Regression model

```python
from sklearn.linear_model import LogisticRegression
from matplotlib import pyplot as plt

log_regr = LogisticRegression()
log_regr.fit(data, label)
predictions = log_regr.predict(data)

df_data['pred_label'] = predictions
```

```python
colors = {0:'b', 1:'g', 2:'r', 3:'c', 4:'m', 5:'y', 6:'k'}

plt.figure(figsize=(8,6))
unq_labels = np.unique(predictions)
for i in unq_labels:
    df = df_data.loc[df_data['pred_label'] == i][['x','y']]
    x = df['x']
    y = df['y']
    plt.scatter(x, y, c=colors[i], alpha=1)
```

#### Problem with the previous approach
- When we develop model, using the complete dataset will not allow us to know the actual performance metrics
- Instead always have a test data for evaluation

<font color='red'> <h1> Go to Notebook '03a - Data Split' to know about data split and return from this point </h1> </font>

## Train-Test Splits

```python
from sklearn.model_selection import train_test_split
data_train, data_test, label_train, label_test = train_test_split(data, label, test_size=0.20,
                                                                  random_state=0, stratify=None)
```

```python
df_before_pred = pd.DataFrame(data_train, columns=['x','y'])
df_before_pred['label'] = label_train

# df_before_pred['label'] = df_before_pred['labels'].apply(lambda x:x+2)
df_before_pred_test = pd.DataFrame(data_test, columns=['x','y'])
df_before_pred_test['label'] = 4

df_before_pred = pd.concat([df_before_pred, df_before_pred_test], ignore_index=True)
```

__Train-test ratio of the split data__

```python
print('Complete data:', data.shape)
print('Train Data:', data_train.shape, 'Test Data:', data_test.shape)
```

    Complete data: (100, 2)
    Train Data: (80, 2) Test Data: (20, 2)

__How the data is split__

```python
from collections import Counter
print('Training Data split', Counter(label_train))
print('Testing Data split', Counter(label_test))
```

    Training Data split Counter({0: 44, 1: 36})
    Testing Data split Counter({1: 12, 0: 8})

#### Stratified split

```python
from sklearn.model_selection import train_test_split
data_train, data_test, label_train, label_test = train_test_split(data, label, test_size=0.20,
                                                                  random_state=0, stratify=label)

print('Complete data:', data.shape)
print('Labels distribution', Counter(label))
print('Train Data:', data_train.shape, 'Test Data:', data_test.shape)

from collections import Counter
print('Training Data split', Counter(label_train))
print('Testing Data split', Counter(label_test))
```

    Complete data: (100, 2)
    Labels distribution Counter({0: 52, 1: 48})
    Train Data: (80, 2) Test Data: (20, 2)
    Training Data split Counter({0: 42, 1: 38})
    Testing Data split Counter({1: 10, 0: 10})

### Logistic regression on the train-test stratified split

```python
from sklearn.linear_model import LogisticRegression
from matplotlib import pyplot as plt

# Run Logistic Regression
log_regr = LogisticRegression()
log_regr.fit(data_train, label_train)
predictions = log_regr.predict(data_test)

data = np.concatenate((data_train, data_test))
label = np.concatenate((label_train, predictions))

df_train_test_pred = pd.DataFrame(data, columns=['x','y'])
df_train_test_pred['label'] = label
```

```python
colors = {0:'b', 1:'g', 2:'r', 3:'c', 4:'m', 5:'y', 6:'k'}

plt.figure(figsize= (16, 6))

plt.subplot(1,2,1)
unq_labels = np.unique(df_before_pred['label'])
for i in unq_labels:
    df = df_before_pred.loc[df_before_pred['label'] == i][['x','y']]
    x = df['x']
    y = df['y']
    plt.scatter(x, y, c=colors[i], alpha=1)

plt.subplot(1,2,2)
unq_labels = np.unique(df_train_test_pred['label'])
for i in unq_labels:
    df = df_train_test_pred.loc[df_train_test_pred['label'] == i][['x','y']]
    x = df['x']
    y = df['y']
    plt.scatter(x, y, c=colors[i], alpha=1)
```

## Find the accuracy

```python
log_regr.score(data_test, label_test)
```

    0.95

## Find the coefficients

```python
log_regr.coef_
```

    array([[-2.87341187, -1.37330706]])

## Find the intercepts

```python
log_regr.intercept_
```

    array([-0.41734966])

```python

```

classification/notebooks/.ipynb_checkpoints/03 - Logistic Regression-checkpoint.ipynb


# Critical Points and Optimization
We've explored various techniques that we can use to calculate the derivative of a function at a specific *x* value; in other words, we can determine the *slope* of the line created by the function at any point on the line.

This ability to calculate the slope means that we can use derivatives to determine some interesting properties of the function.

## Function Direction at a Point
Consider the following function, which represents the trajectory of a ball that has been kicked on a football field:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Run the Python code below to graph this function and see the trajectory of the ball over a period of 10 seconds.

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

By looking at the graph of this function, you can see that it describes a parabola in which the ball rose in height before falling back to the ground. On the graph, it's fairly easy to see when the ball was rising and when it was falling.

Of course, we can also use  derivative to determine the slope of the function at any point. We can apply some of the rules we've discussed previously to determine the derivative function:

- We can add together the derivatives of the individual terms (***-10x2***, ***100x***, and ***3***) to find the derivative of the entire function.
- The *power* rule tells us that the derivative of ***-10x2*** is ***-10 • 2x***, which is ***-20x***.
- The *power* rule also tells us that the derivative of ***100x*** is ***100***.
- The derivative of any constant, such as ***3*** is ***0***.

So:

\begin{equation}k'(x) = -20x + 100 + 0 \end{equation}

Which of course simplifies to:

\begin{equation}k'(x) = -20x + 100 \end{equation}

Now we can use this derivative function to find the slope for any value of ***x***.

Run the cell below to see a graph of the function and its derivative function:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Look closely at the purple line representing the derivative function, and note that it is a constant  decreasing value - in other words, the slope of the function is reducing linearly as x increases. Even though the function value itself is increasing for the first half of the parabola (while the ball is rising), the slope is becoming less steep (the ball is not rising at such a high rate), until finally the ball reaches its apogee and the slope becomes negative (the ball begins falling).

Note also that the point where the derivative line crosses 0 on the y-axis is also the point where the function value stops increasing and starts decreasing. When the slope has a positive value, the function is increasing; and when the slope has a negative value, the function is decreasing.

The fact that the derivative line crosses 0 at the highest point of the function makes sense if you think about it logically. If you were to draw the tangent line representing the slope at each point, it would be rotating clockwise throughout the graph, initially pointing up and to the right as the ball rises, and turning until it is pointing down and right as the ball falls. At the highest point, the tangent line would be perfectly horizontal, representing a slope of 0.

Run the following code to visualize this:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot tangent slopes for x = 2, 5, and 8
x1 = 2
x2 = 5
x3 = 8
plt.plot([x1-1,x1+1],[k(x1)-(kd(x1)),k(x1)+(kd(x1))], color='red')
plt.plot([x2-1,x2+1],[k(x2)-(kd(x2)),k(x2)+(kd(x2))], color='red')
plt.plot([x3-1,x3+1],[k(x3)-(kd(x3)),k(x3)+(kd(x3))], color='red')

plt.show()
```

Now consider the following function, which represents the number of flowers growing in a flower bed before and after the spraying of a fertilizer:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the derivative values
yd = [wd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that the green line represents the function, showing the number of flowers for 10 days before and after the fertilizer treatment. Before treatment, the number of flowers was in decline, and after treatment the flower bed started to recover.

The derivative function is shown in purple, and once again shows a linear change in slope. This time, the slope is increasing at a constant rate; and once again, the derivative function line crosses 0 at the lowest point in the function line (in other words, the slope changed from negative to positive when the flowers started to recover).

## Critical Points
From what we've seen so far, it seems that there is a relationship between a function reaching an extreme value (a maximum or a minimum), and a derivative value of 0. This makes intuitive sense; the derivative represents the slope of the line, so when a function changes from a negative slope to a positive slope, or vice-versa, the derivative must pass through 0.

However, you need to be careful not to assume that just because the derivative is 0 at a given point, that this point represents the minimum or maximum of the function. For example, consider the following function:

\begin{equation}v(x) = x^{3} - 2x + 100 \end{equation}

Run the following Python code to visualize this function and its corresponding derivative function:

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (2*x) + 100

def vd(x):
    return 3*(x**2) - 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-1000, 2000, 100))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that in this case, the purple derivative function line passes through 0 as the green function line transitions from a *concave downwards* slope (a slope that is decreasing) to a *concave upwards* slope (a slope that is increasing). The slope flattens out to 0, forming a "saddle" before the it starts increasing.

What we can learn from this is that interesting things seem to happen to the function when the derivative is 0. We call points where the derivative crosses 0 *critical points*, because they indicate that the function is changing direction. When a function changes direction from positive to negative, it forms a peak (or a *local maximum*), when the function changes direction from negative to positive it forms a trough (or *local minimum*), and when it maintains the same overall direction but changes the concavity of the slope it creates an *inflexion point*.

## Finding Minima and Maxima
A common use of calculus is to find minimum and maximum points in a function. For example, we might want to find out how many seconds it took for the kicked football to reach its maximum height, or how long it took for our fertilizer to be effective in reversing the decline of flower growth.

We've seen that when a function changes direction to create a maximum peak or a minimum trough, the derivative of the function is 0, so a step towards finding these extreme points might be to simply find all of the points in the function where the derivative is 0. For example, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

From this, we've calculated the function for the derivative as:

\begin{equation}k'(x) = -20x + 100 \end{equation}

We can then solve the derivative equation for an f'(x) value of 0:

\begin{equation}-20x + 100 = 0 \end{equation}

We can remove the constant by subtracting 100 to both sides:

\begin{equation}-20x = -100 \end{equation}

Multiplying both sides by -1 gets rid of the negative values (this isn't strictly necessary, but makes the equation a little less confusing)

\begin{equation}20x = 100 \end{equation}

So:

\begin{equation}x = 5 \end{equation}

So we know that the derivative will be 0 when *x* is 5, but is this a minimum, a maximum, or neither? It could just be an inflexion point, or the entire function could be a constant value with a slope of 0) Without looking at the graph, it's difficult to tell.

## Second Order Derivatives
The solution to our problem is to find the derivative of the derivative! Until now, we've found the derivative of a function, and indicated it as ***f'(x)***. Technically, this is known as the *prime* derivative; and it describes the slope of the function. Since the derivative function is itself a function, we can find its derivative, which we call the *second order* (or sometimes just *second*) derivative. This is indicated like this: ***f''(x)***.

So, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Here's the function for the prime derivative:

\begin{equation}k'(x) = -20x + 100 \end{equation}

And using a combination of the power rule and the constant rule, here's the function for the second derivative:

\begin{equation}k''(x) = -20 \end{equation}

Now, without even drawing the graph, we can see that the second derivative has a constant value; so we know that the slope of the prime derivative is linear; and because it's a negative value, we know that it is decreasing. So when the prime derivative crosses 0, it we know that the slope of the function is decreasing linearly; so the point at *x=0* must be a maximum point.

Run the following code to plot the function, the prime derivative, and the second derivative for the kicked ball:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

def k2d(x):
    return -20

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the k'(x) values
yd = [kd(i) for i in x]

# Use the 2-derivative function to get the k''(x)
y2d = [k2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot  k'(x)
plt.plot(x,yd, color='purple')

# Plot k''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

Let's take the same approach for the flower bed problem. Here's the function:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

Using the power rule and constant rule, gives us the prime derivative function:

\begin{equation}w'(x) = 2x + 2 \end{equation}

Applying the power rule and constant rule to the prime derivative function gives us the second derivative function:

\begin{equation}w''(x) = 2 \end{equation}

Note that this time, the second derivative is a positive constant, so the prime derivative (which is the slope of the function) is increasing linearly. The point where the prime derivative crosses 0 must therefore be a minimum. Let's run the code below to check:

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

def w2d(x):
    return 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the w'(x) values
yd = [wd(i) for i in x]

# Use the 2-derivative function to get the w''(x) values
y2d = [w2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot w'(x)
plt.plot(x,yd, color='purple')

# Plot w''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

## Critical Points that are *Not* Maxima or Minima
Of course, it's possible for a function to form a "saddle" where the prime derivative is zero at a point that is not a minimum or maximum. Here's an example of a function like this:

\begin{equation}v(x) = x^{3} - 6x^{2} + 12x + 2 \end{equation}

And here's its prime derivative:

\begin{equation}v'(x) = 3x^{2} - 12x + 12 \end{equation}

Let's find a critical point where v'(x) = 0

\begin{equation}3x^{2} - 12x + 12 = 0 \end{equation}

Factor the x-terms

\begin{equation}3x(x - 4) = 12 \end{equation}

Divide both sides by 3:

\begin{equation}x(x - 4) = 4 \end{equation}

Factor the x terms back again

\begin{equation}x^{2} - 4x = 4 \end{equation}

Complete the square, step 1

\begin{equation}x^{2} - 4x + 4 = 0 \end{equation}

Complete the square, step 2

\begin{equation}(x - 2)^{2} = 0 \end{equation}

Find the square root:

\begin{equation}x - 2 = \pm\sqrt{0}\end{equation}

\begin{equation}x - 2 = +\sqrt{0} = 0, -\sqrt{0} = 0\end{equation}

v'(2) = 0 (only touches 0 once)

Is it a maximum or minimum? Let's find the second derivative:

\begin{equation}v''(x) = 6x - 12\end{equation}

So

\begin{equation}v''(2) = 0\end{equation}

So it's neither negative or positive, so it's not a maximum or minimum.

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (6*(x**2)) + (12*x) + 2

def vd(x):
    return (3*(x**2)) - (12*x) + 12

def v2d(x):
    return (3*(2*x)) - 12

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-5, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Use the derivative function to get the derivative values
y2d = [v2d(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-2000, 2000, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot the derivative
plt.plot(x,y2d, color='magenta')

plt.show()

print ("v(2) = " + str(v(2)))

print ("v'(2) = " + str(vd(2)))

print ("v''(2) = " + str(v2d(2)))

```

## Optimization
The ability to use derivatives to find minima and maxima of a function makes it a useful tool for scenarios where you need to optimize a function for a specific variable.

### Defining Functions to be Optimized
For example, suppose you have decided to build an online video service that is based on a subscription model. You plan to charge a monthly subscription fee, and you want to make the most revenue possible. The problem is that customers are price-sensitive, so if you set the monthly fee too high, you'll deter some customers from signing up. Conversely, if you set the fee too low, you may get more customers, but at the cost of reduced revenue.

What you need is some kind of function that will tell you how many subscriptions you might expect to get based on a given fee. So you've done some research, and found a formula to indicate that the expected subscription volume (in thousands) can be calculated as 5-times the monthly fee subtracted from 100; or expressed as a function:

\begin{equation}s(x) = -5x + 100\end{equation}

What you actually want to optimize is monthly revenue, which is simply the number of subscribers multiplied by the fee:

\begin{equation}r(x) = s(x) \cdot x\end{equation}

We can combine ***s(x)*** into ***r(x)*** like this:

\begin{equation}r(x) = -5x^{2} + 100x\end{equation}

### Finding the Prime Derivative
The function ***r(x)*** will return the expected monthly revenue (in thousands) for any proposed fee (*x*). What we need to do now is to find the fee that yields the maximum revenue. Fortunately, we can use a derivative to do that.

First, we need to determine the prime derivative of ***r(x)***, and we can do that easily using the power rule:

\begin{equation}r'(x) = 2 \cdot -5x + 100\end{equation}

Which is:

\begin{equation}r'(x) = -10x + 100\end{equation}

### Find Critical Points
Now we need to find any critical points where the derivative is 0, as this could indicate a maximum:

\begin{equation}-10x + 100 = 0\end{equation}

Let's isolate the *x* term:

\begin{equation}-10x = -100\end{equation}

Both sides are negative, so we can mulitply both by -1 to make them positive without affecting the equation:

\begin{equation}10x = 100\end{equation}

Now we can divide both sides by 10 to isolate *x*:

\begin{equation}x = \frac{100}{10}\end{equation}

So:

\begin{equation}x = 10\end{equation}

#### Check for a Maximum
We now know that with an *x* value of of **10**, the derivative is 0; or put another way, when the fee is 10, the slope indicating the change in subscription volume is flat. This could potentially be a point where the change in subscription volume has peaked (in other words, a maximum); but it could also be a minimum or just an inflexion point where the rate of change transitions from negative to positive.

To be sure, we can check the second order derivative. We can calculate this by applying the power rule to the prime derivative:

\begin{equation}r''(x) = -10\end{equation}

Note that the second derivative is a constant with a negative value. It will be the same for any point, including our critical point at *x=10*:

\begin{equation}r''(10) = -10\end{equation}

A negative value for the second derivative tells us that the derivative slope is moving in a negative direction at the point where it is 0, so the function value must be at a maximum.

In other words, the optimal monthly fee for our online video service is 10 - this will generate the maximum monthly revenue.

Run the code below to show the function ***r(x)*** as a graph, and verify that the maximum point is at x = 10.

```python
%matplotlib inline

# Create function s
def s(x):
    return (-5*x) + 100

# Create function r
def r(x):
    return s(x) * x

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 21))

# Use the function to get the y values
y = [r(i) for i in x]

# Set up the graph
plt.xlabel('x (monthly fee)')
plt.ylabel('r(x) (revenue in $,000)')
plt.xticks(range(0,22, 1))
plt.yticks(range(0, 600, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

Basics of Calculus by Hiren/02-04-Critical Points and Optimization.ipynb


# Introduction to Mathematical Optimization Modeling

## Objective and prerequisites
The goal of this modeling example is to introduce the key components in the formulation of mixed integer programming (MIP) problems. For each component of a MIP problem formulation, we provide a description,  the associated Python code,  and the mathematical notation describing the component.

To fully understand the content of this notebook, the reader should:

* Be familiar with Python.
* Have a background in any branch of engineering, computer science, economics, statistics, any branch of the “hard” sciences, or any discipline that uses quantitative models and methods.

The reader should also consult the  [documentation](https://www.gurobi.com/resources/?category-filter=documentation)
of the Gurobi Python API.
This notebook is explained in detail in our series of tutorial videos on mixed integer linear programming.
You can watch these videos by clicking
[here](https://www.gurobi.com/resource/tutorial-mixed-integer-linear-programming/)

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

## Problem description

Consider a consulting company that has three open positions: Tester, Java Developer, and Architect. The three top candidates (resources) for the positions are: Carlos, Joe, and Monika. The consulting company administered competency tests to each candidate in order to assess their ability to perform each of the jobs. The results of these tests are called *matching scores*. Assume that only one candidate can be assigned to a job, and at most one job can be assigned to a candidate.

The problem is to determine an assignment of resources and jobs such that each job is fulfilled, each resource is assigned to at most one job, and the total matching scores of the assignments is maximized.

## Mathematical optimization

Mathematical optimization (which is also known as mathematical programming) is a declarative approach where the modeler formulates an  optimization problem that captures the key features of a complex decision problem. The Gurobi Optimizer solves the mathematical optimization problem using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components:

* Sets
* Parameters
* Decision variables
* Constraints
* Objective function(s)

The following Python code imports the Gurobi callable library and imports the ``GRB`` class into the main namespace.

```python
import gurobipy as gp
from gurobipy import GRB
```

## Resource Assignment Problem
### Data
The list $R$ contains the names of the three resources: Carlos, Joe, and Monika.

The list $J$ contains the names of the job positions: Tester, Java Developer, and Architect.

$r \in R$: index and set of resources. The resource $r$ belongs to the set of resources $R$.

$j \in J$: index and set of jobs. The job $j$ belongs to the set of jobs $J$.

```python
# Resource and job sets
R = ['Carlos', 'Joe', 'Monika']
J = ['Tester', 'JavaDeveloper', 'Architect']
```

The ability of each resource to perform each of the jobs is listed in the following matching scores table:

For each resource $r$ and job $j$, there is a corresponding matching score $s$. The matching score $s$ can only take values between 0 and 100. That is, $s_{r,j} \in [0, 100]$ for all resources $r \in R$ and jobs $j \in J$.

We use the Gurobi Python ``multidict`` function to initialize one or more dictionaries with a single statement. The function takes a dictionary as its argument. The keys represent the possible combinations of resources and jobs.

```python
# Matching score data
combinations, scores = gp.multidict({
    ('Carlos', 'Tester'): 53,
    ('Carlos', 'JavaDeveloper'): 27,
    ('Carlos', 'Architect'): 13,
    ('Joe', 'Tester'): 80,
    ('Joe', 'JavaDeveloper'): 47,
    ('Joe', 'Architect'): 67,
    ('Monika', 'Tester'): 53,
    ('Monika', 'JavaDeveloper'): 73,
    ('Monika', 'Architect'): 47
})
```

The following constructor creates an empty ``Model`` object “m”. We specify the model name by passing the string "RAP" as an argument. The ``Model`` object “m” holds a single optimization problem. It consists of a set of variables, a set of constraints, and the objective function.

```python
# Declare and initialize model
m = gp.Model('RAP')
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-

## Decision variables

To solve this assignment problem, we need to identify which resource is assigned to which job. We introduce a decision variable for each possible assignment of resources to jobs. Therefore, we have 9 decision variables.

To simplify the mathematical notation of the model formulation, we define the following indices for resources and jobs:

For example, $x_{2,1}$ is the decision variable associated with assigning the resource Joe to the job Tester. Therefore, decision variable $x_{r,j}$ equals 1 if resource $r \in R$  is assigned to job $j \in J$, and 0 otherwise.

The ``Model.addVars()`` method creates the decision variables for a ``Model`` object.
This method returns a Gurobi ``tupledict`` object that contains the newly created variables. We supply the ``combinations`` object as the first argument to specify the variable indices. The ``name`` keyword is used to specify a name for the newly created decision variables. By default, variables are assumed to be non-negative.

```python
# Create decision variables for the RAP model
x = m.addVars(combinations, name="assign")
```

## Job constraints

We now discuss the constraints associated with the jobs. These constraints need to ensure that each job is filled by exactly one resource.

The job constraint for the Tester position requires that resource 1 (Carlos), resource 2 (Joe), or resource 3 (Monika) is assigned to this job. This corresponds to the following constraint.

Constraint (Tester=1)

$$
x_{1,1} + x_{2,1} + x_{3,1} = 1
$$

Similarly, the constraints for the Java Developer and Architect positions can be defined as follows.

Constraint (Java Developer = 2)

$$
x_{1,2} + x_{2,2} + x_{3,2} = 1
$$

Constraint (Architect = 3)

$$
x_{1,3} + x_{2,3} + x_{3,3} = 1
$$

The job constraints are defined by the columns of the following table.

In general, the constraint for the job Tester can defined as follows.

$$
x_{1,1} + x_{2,1} + x_{3,1} = \sum_{r=1}^{3 } x_{r,1} =  \sum_{r \in R} x_{r,1} = 1
$$

All of the job constraints can be defined in a similarly succinct manner. For each job $j \in J$, take the summation of the decision variables over all the resources. We can write the corresponding job constraint as follows.

$$
\sum_{r \in R} x_{r,j} = 1
$$

The ``Model.addConstrs()`` method of the Gurobi/Python API defines the job constraints of the ``Model`` object “m”. This method returns a Gurobi ``tupledict`` object that contains the job constraints.
The first argument of this method, "x.sum(‘*’, j)", is the sum method and defines the LHS of the jobs constraints as follows:
For each job $j$ in the set of jobs $J$, take the summation of the decision variables over all the resources. The $==$  defines an equality constraint, and the number "1" is the RHS of the constraints.
These constraints are saying that exactly one resource should be assigned to each job.
The second argument is the name of this type of constraints.

```python
# Create job constraints
jobs = m.addConstrs((x.sum('*',j) == 1 for j in J), name='job')
```

## Resource constraints

The constraints for the resources need to ensure that at most one job is assigned to each resource. That is, it is possible that not all the resources are assigned.

For example, we want a constraint that requires Carlos to be assigned to at most one of the jobs: either job 1 (Tester), job 2 (Java Developer ), or job 3 (Architect). We can write this constraint as follows.

Constraint (Carlos=1)

$$
x_{1, 1} + x_{1, 2} + x_{1, 3}  \leq 1.
$$

This constraint is less or equal than 1 to allow the possibility that Carlos is not assigned to any job. Similarly, the constraints for the resources Joe and Monika can be defined as follows:

Constraint (Joe=2)

$$
x_{2, 1} + x_{2, 2} + x_{2, 3}  \leq 1.
$$

Constraint (Monika=3)

$$
x_{3, 1} + x_{3, 2} + x_{3, 3}  \leq 1.
$$

Observe that the resource constraints are defined by the rows of the following table.

The constraint for the resource Carlos can be defined as follows.

$$
x_{1, 1} + x_{1, 2} + x_{1, 3} = \sum_{j=1}^{3 } x_{1,j} = \sum_{j \in J} x_{1,j} \leq 1.
$$

Again, each of these constraints can be written in a succinct manner. For each resource $r \in R$, take the summation of the decision variables over all the jobs. We can write the corresponding resource constraint as follows.

$$
\sum_{j \in J} x_{r,j} \leq  1.
$$

The ``Model.addConstrs()`` method of the Gurobi/Python API defines the resource constraints of the ``Model`` object “m”.
The first argument of this method, "x.sum(r, ‘*’)", is the sum method and defines the LHS of the resource constraints as follows: For each resource $r$ in the set of resources $R$, take the summation of the decision variables over all the jobs.
The $<=$  defines a less or equal constraints, and the number “1” is the RHS of the constraints.
These constraints are saying that each resource can be assigned to at most 1 job.
The second argument is the name of this type of constraints.

```python
# Create resource constraints
resources = m.addConstrs((x.sum(r,'*') <= 1 for r in R), name='resource')
```

## Objective function

The objective function is to maximize the total matching score of the assignments that satisfy the job and resource constraints.

For the Tester job, the matching score is $53x_{1,1}$, if resource Carlos is assigned, or $80x_{2,1}$, if resource Joe is assigned, or $53x_{3,1}$, if resource Monika is assigned.
Consequently, the matching score for the Tester job is as follows, where only one term in this summation will be nonzero.

$$
53x_{1,1} + 80x_{2,1} + 53x_{3,1}.
$$

Similarly, the matching scores for the Java Developer and Architect jobs are defined as follows. The matching score for the Java Developer job is:

$$
27x_{1, 2} + 47x_{2, 2} + 73x_{3, 2}.
$$

The matching score for the Architect job is:

$$
13x_{1, 3} + 67x_{2, 3} + 47x_{3, 3}.
$$

The total matching score is the summation of each cell in the following table.

The goal is to  maximize the total matching score of the assignments. Therefore, the objective function is defined as follows.

\begin{equation}
\text{Maximize} \quad (53x_{1,1} + 80x_{2,1} + 53x_{3,1}) \; +
\end{equation}

\begin{equation}
\quad (27x_{1, 2} + 47x_{2, 2} + 73x_{3, 2}) \; +
\end{equation}

\begin{equation}
\quad (13x_{1, 3} + 67x_{2, 3} + 47x_{3, 3}).
\end{equation}

Each term in parenthesis in the objective function can be expressed as follows.

\begin{equation}
(53x_{1,1} + 80x_{2,1} + 53x_{3,1}) = \sum_{r \in R} s_{r,1}x_{r,1}.
\end{equation}

\begin{equation}
(27x_{1, 2} + 47x_{2, 2} + 73x_{3, 2}) = \sum_{r \in R} s_{r,2}x_{r,2}.
\end{equation}

\begin{equation}
(13x_{1, 3} + 67x_{2, 3} + 47x_{3, 3}) = \sum_{r \in R} s_{r,3}x_{r,3}.
\end{equation}

Hence, the objective function can be concisely written as:

\begin{equation}
\text{Maximize} \quad \sum_{j \in J} \sum_{r \in R} s_{r,j}x_{r,j}.
\end{equation}

The ``Model.setObjective()`` method of the Gurobi/Python API defines the objective function of the ``Model`` object “m”. The objective expression is specified in the first argument of this method.
Notice that both the matching score parameters “score” and the assignment decision variables “x” are defined over the “combinations” keys. Therefore, we use the method “x.prod(score)” to obtain the summation of the elementwise multiplication of the "score" matrix and the "x" variable matrix.
The second argument, ``GRB.MAXIMIZE``, is the optimization "sense." In this case, we want to *maximize* the total matching scores of all assignments.

```python
# Objective: maximize total matching score of all assignments
m.setObjective(x.prod(scores), GRB.MAXIMIZE)
```

We use the “write()” method of the Gurobi/Python API to write the model formulation to a file named "RAP.lp".

```python
# Save model for inspection
m.write('RAP.lp')
```

We use the “optimize( )” method of the Gurobi/Python API to solve the problem we have defined for the model object “m”.

```python
# Run optimization engine
m.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 6 rows, 9 columns and 18 nonzeros
    Model fingerprint: 0xb6602fb2
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [1e+01, 8e+01]
      Bounds range     [0e+00, 0e+00]
      RHS range        [1e+00, 1e+00]
    Presolve time: 0.00s
    Presolved: 6 rows, 9 columns, 18 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    4.6000000e+32   1.800000e+31   4.600000e+02      0s
           5    1.9300000e+02   0.000000e+00   0.000000e+00      0s

    Solved in 5 iterations and 0.01 seconds
    Optimal objective  1.930000000e+02

The ``Model.getVars()`` method of the Gurobi/Python API
retrieves a list of all variables in the Model object “m”. The ``.x`` variable attribute is used to query solution values and the ``.varName`` attribute is used to query the name of the decision variables.

```python
# Display optimal values of decision variables
for v in m.getVars():
    if v.x > 1e-6:
        print(v.varName, v.x)

# Display optimal total matching score
print('Total matching score: ', m.objVal)
```

    assign[Carlos,Tester] 1.0
    assign[Joe,Architect] 1.0
    assign[Monika,JavaDeveloper] 1.0
    Total matching score:  193.0

The optimal assignment is to assign:

* Carlos to the Tester job, with a matching score of 53
* Joe to the Architect job, with a matching score of 67
* Monika to the Java Developer job, with a matching score of 73.

The maximum total matching score is 193.

## Resource Assignment Problem with a budget constraint

Now, assume there is a fixed cost $C_{r,j}$ associated with assigning a resource $r \in R$ to job $j \in J$. Assume also that there is a limited budget $B$ that can be used for job assignments.

The cost of assigning Carlos, Joe, or Monika to any of the jobs is $\$1,000$ , $\$2,000$ , and $\$3,000$  respectively. The available budget is $\$5,000$.

### Data

The list $R$ contains the names of the three resources: Carlos, Joe, and Monika.
The list $J$ contains the names of the job positions: Tester, Java Developer, and Architect.

The Gurobi Python ``multidict`` function initialize two dictionaries:
* "scores" defines the matching scores for each resource and job combination.
* "costs" defines the fixed cost associated of assigning a resource to a job.

```python
# Resource and job sets
R = ['Carlos', 'Joe', 'Monika']
J = ['Tester', 'JavaDeveloper', 'Architect']

# Matching score data
# Cost is given in thousands of dollars
combinations, scores, costs = gp.multidict({
    ('Carlos', 'Tester'): [53, 1],
    ('Carlos', 'JavaDeveloper'): [27, 1],
    ('Carlos', 'Architect'): [13,1],
    ('Joe', 'Tester'): [80, 2],
    ('Joe', 'JavaDeveloper'): [47, 2],
    ('Joe', 'Architect'): [67, 2],
    ('Monika', 'Tester'): [53, 3] ,
    ('Monika', 'JavaDeveloper'): [73, 3],
    ('Monika', 'Architect'): [47, 3]
})

# Available budget (thousands of dollars)
budget = 5
```

The following constructor creates an empty ``Model`` object “m”. The ``Model`` object “m” holds a single optimization problem. It consists of a set of variables, a set of constraints, and the objective function.

```python
# Declare and initialize model
m = gp.Model('RAP2')
```

### Decision variables

The decision variable $x_{r,j}$ is 1 if $r \in R$ is assigned to job $j \in J$, and 0 otherwise.

The ``Model.addVars()`` method defines the decision variables for the model object “m”.

Because there is a budget constraint, it is possible that not all of the jobs will be filled. To account for this, we define a new decision variable that indicates whether or not a job is filled.

Let $g_{j}$ be equal 1 if job $j \in J$ is not filled, and 0 otherwise. This variable is a gap variable that indicates that a job cannot be filled.

***Remark:*** For the previous formulation of the RAP, we defined the assignment variables as non-negative and continuous which is the default value of the ``vtype`` argument of the ``Model.addVars()`` method.
However, in this extension of the RAP, because of the budget constraint we added to the model, we need to explicitly define these variables as binary. The ``vtype=GRB.BINARY`` argument of the ``Model.addVars()`` method defines the assignment variables as binary.

```python
# Create decision variables for the RAP model
x = m.addVars(combinations, vtype=GRB.BINARY, name="assign")

# Create gap variables for the RAP model
g = m.addVars(J, name="gap")
```

### Job constraints

Since we have a limited budget to assign resources to jobs, it is possible that not all the jobs can be filled. For the job constraints, there are two possibilities either a resource is assigned to fill the job, or this job cannot be filled and we need to declare a gap. This latter possibility is captured by the decision variable $g_j$. Therefore, the job constraints are written as follows.

For each job $j \in J$, exactly one resource must be assigned to the job, or the corresponding $g_j$ variable must be set to 1:

$$
\sum_{r \: \in \: R} x_{r,\; j} + g_{j} = 1.
$$

```python
# Create job constraints
jobs = m.addConstrs((x.sum('*',j) + g[j]  == 1 for j in J), name='job')
```

### Resource constraints

The constraints for the resources need to ensure that at most one job is assigned to each resource. That is, it is possible that not all the resources are assigned. Therefore, the resource constraints are written as follows.

For each resource $r \in R$, at most one job can be assigned to the resource:

$$
\sum_{j \: \in \: J} x_{r,\; j} \leq 1.
$$

```python
# Create resource constraints
resources = m.addConstrs((x.sum(r,'*') <= 1 for r in R), name='resource')
```

### Budget constraint

This constraint ensures that the cost of assigning resources to fill job requirements do not exceed the budget available. The costs of assignment and budget are in thousands of dollars.

The cost of filling the Tester job is $1x_{1,1}$, if resource Carlos is assigned, or $2x_{2,1}$, if resource Joe is assigned, or $3x_{3,1}$, if resource Monika is assigned.
Consequently, the cost of filling the Tester job is as follows, where at most one term in this summation will be nonzero.

$$
1x_{1,1} + 2x_{2,1} + 3x_{3,1}.
$$

Similarly, the cost of filling the Java Developer and Architect jobs are defined as follows. The cost of filling the Java Developer job is:

$$
1x_{1, 2} + 2x_{2, 2} + 3x_{3, 2}.
$$

The cost of filling the Architect job is:

$$
1x_{1, 3} + 2x_{2, 3} + 3x_{3, 3}.
$$

Hence, the total cost of filling the jobs should be less or equal than the budget available.

\begin{equation}
(1x_{1,1} + 2x_{2,1} + 3x_{3,1}) \; +
\end{equation}

\begin{equation}
(1x_{1, 2} + 2x_{2, 2} + 3x_{3, 2}) \; +
\end{equation}

\begin{equation}
(1x_{1, 3} + 2x_{2, 3} + 3x_{3, 3}) \leq 5
\end{equation}

Each term in parenthesis in the budget constraint can be expressed as follows.

\begin{equation}
(1x_{1,1} + 2x_{2,1} + 3x_{3,1}) = \sum_{r \in R} C_{r,1}x_{r,1}.
\end{equation}

\begin{equation}
(1x_{1, 2} + 2x_{2, 2} + 3x_{3, 2}) = \sum_{r \in R} C_{r,2}x_{r,2}.
\end{equation}

\begin{equation}
(1x_{1, 3} + 2x_{2, 3} + 3x_{3, 3}) = \sum_{r \in R} C_{r,3}x_{r,3}.
\end{equation}

Therefore, the budget constraint can be concisely written as:

\begin{equation}
\sum_{j \in J} \sum_{r \in R} C_{r,j}x_{r,j} \leq B.
\end{equation}

The ``Model.addConstr()`` method of the Gurobi/Python API defines the budget constraint of the ``Model`` object “m”.
The first argument of this method, "x.prod(costs)", is the prod method and defines the LHS of the budget constraint. The $<=$ defines a less or equal constraint, and the budget amount available is the RHS of the constraint.
This constraint is saying that the total cost of assigning resources to fill jobs requirements cannot exceed the budget available.
The second argument is the name of this constraint.

```python
budget = m.addConstr((x.prod(costs) <= budget), name='budget')
```

## Objective function

The objective function is similar to the RAP. The first term in the objective is the total matching score of the assignments. In this extension of the RAP, it is possible that not all jobs are filled; however, we want to heavily penalize this possibility. For this purpose, we have a second term in the objective function that takes the summation of the gap variables over all the jobs and multiply it by a big penalty $M$.

Observe that the maximum value of a matching score is 100, and the value that we give to $M$ is 101. The rationale behind the value of $M$ is that having gaps heavily deteriorates the total matching scores value.

Consequently, the objective function is to maximize the total matching score of the assignments minus the penalty associated of having gap variables with a value equal to 1.

$$
\max \; \sum_{j \; \in \; J} \sum_{r \; \in \; R} s_{r,j}x_{r,j} -M \sum_{j \in J} g_{j}
$$

```python
# Penalty for not filling a job position
M = 101
```

```python
# Objective: maximize total matching score of assignments
# Unfilled jobs are heavily penalized
m.setObjective(x.prod(scores) - M*g.sum(), GRB.MAXIMIZE)
```

```python
# Run optimization engine
m.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 7 rows, 12 columns and 30 nonzeros
    Model fingerprint: 0xf3c6f8c8
    Variable types: 3 continuous, 9 integer (9 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 3e+00]
      Objective range  [1e+01, 1e+02]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 5e+00]
    Presolve time: 0.00s
    Presolved: 7 rows, 12 columns, 30 nonzeros
    Variable types: 0 continuous, 12 integer (12 binary)
    Found heuristic solution: objective 52.0000000

    Root relaxation: objective 1.350000e+02, 4 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0  135.00000    0    2   52.00000  135.00000   160%     -    0s
         0     0  121.66667    0    7   52.00000  121.66667   134%     -    0s
         0     0     cutoff    0        52.00000   52.00000  0.00%     -    0s

    Cutting planes:
      Gomory: 1
      GUB cover: 1
      RLT: 1

    Explored 1 nodes (7 simplex iterations) in 0.03 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 1: 52

    Optimal solution found (tolerance 1.00e-04)
    Best objective 5.200000000000e+01, best bound 5.200000000000e+01, gap 0.0000%

The definition of the objective function includes the penalty of no filling jobs. However, we are interested in the optimal total matching score value when not all the jobs are filled. For this purpose, we need to compute the total matching score value using the matching score values $s_{r,j}$ and the assignment decision variables $x_{r,j}$.

```python
# Compute total matching score from assignment variables
total_matching_score = 0
for r, j in combinations:
    if x[r, j].x > 1e-6:
        print(x[r, j].varName, x[r, j].x)
        total_matching_score += scores[r, j]*x[r, j].x

print('Total matching score: ', total_matching_score)
```

    assign[Joe,Tester] 1.0
    assign[Monika,JavaDeveloper] 1.0
    Total matching score:  153.0

### Analysis

Recall that the budget is $\$5,000$, and the total  cost associated of allocating the three resources is $\$6,000$. This means that there is not enough budget to allocate the three resources we have. Consequently, the Gurobi Optimizer must choose two resources to fill the jobs demand, leave one job unfilled, and maximize the total matching scores. Notice that the two top matching scores are 80% (Joe for the Tester job) and 73% (Monika for the Java Developer job). Also, notice that the lowest score is 13% (Carlos for the Architect job). Assigning Joe to the Tester job, Monika to the Java Developer job, and nobody to the Architect job costs $\$5,000$  and yields a total matching score of 153. This is the optimal solution found by the Gurobi Optimizer.

Copyright © 2020 Gurobi Optimization, LLC