Lambda School Data Science

*Unit 2, Sprint 1, Module 2*

---

# Regression 2
- Do train/test split
- Use scikit-learn to fit a multiple regression
- Understand how ordinary least squares regression minimizes the sum of squared errors
- Define overfitting/underfitting and the bias/variance tradeoff

### Setup

Run the code cell below. You can work locally (follow the [local setup instructions](https://lambdaschool.github.io/ds/unit2/local/)) or on Colab.

Libraries:
- matplotlib
- numpy
- pandas
- plotly
- scikit-learn

```python
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'

# If you're working locally:
else:
    DATA_PATH = '../data/'

# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
```

# Do train/test split

## Overview

### Predict Elections! 🇺🇸🗳

How could we try to predict the 2020 US Presidential election?

According to Douglas Hibbs, a political science and economics professor, you can [explain elections with just two features, "Bread and Peace":](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:
>
> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

Let's look at the data that Hibbs collected and analyzed:

```python
import pandas as pd
df = pd.read_csv(DATA_PATH+'elections/bread_peace_voting.csv')
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share

      0
      1952
      Stevenson
      Eisenhower
      2.40
      190
      44.60

      1
      1956
      Eisenhower
      Stevenson
      2.89
      0
      57.76

      2
      1960
      Nixon
      Kennedy
      0.85
      0
      49.91

      3
      1964
      Johnson
      Goldwater
      4.21
      1
      61.34

      4
      1968
      Humphrey
      Nixon
      3.02
      146
      49.60

      5
      1972
      Nixon
      McGovern
      3.62
      0
      61.79

      6
      1976
      Ford
      Carter
      1.08
      2
      48.95

      7
      1980
      Carter
      Reagan
      -0.39
      0
      44.70

      8
      1984
      Reagan
      Mondale
      3.86
      0
      59.17

      9
      1988
      Bush, Sr.
      Dukakis
      2.27
      0
      53.94

      10
      1992
      Bush, Sr.
      Clinton
      0.38
      0
      46.55

      11
      1996
      Clinton
      Dole
      1.04
      0
      54.74

      12
      2000
      Gore
      Bush, Jr.
      2.36
      0
      50.27

      13
      2004
      Bush, Jr.
      Kerry
      1.72
      4
      51.24

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20

Data Sources & Definitions

- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33

Here we have data from the 1952-2016 elections. We could make a model to predict 1952-2016 election outcomes — but do we really care about that?

No, not really. We already know what happened, we don't need to predict it.

This is explained in [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/), Chapter 2.2, Assessing Model Accuracy:

> In general, we do not really care how well the method works training on the training data. Rather, _we are interested in the accuracy of the predictions that we obtain when we apply our method to previously unseen test data._ Why is this what we care about?
>
> Suppose that we are interested in developing an algorithm to predict a stock’s price based on previous stock returns. We can train the method using stock returns from the past 6 months. But we don’t really care how well our method predicts last week’s stock price. We instead care about how well it will predict tomorrow’s price or next month’s price.
>
> On a similar note, suppose that we have clinical measurements (e.g. weight, blood pressure, height, age, family history of disease) for a number of patients, as well as information about whether each patient has diabetes. We can use these patients to train a statistical learning method to predict risk of diabetes based on clinical measurements. In practice, we want this method to accurately predict diabetes risk for _future patients_ based on their clinical measurements. We are not very interested in whether or not the method accurately predicts diabetes risk for patients used to train the model, since we already know which of those patients have diabetes.

So, we're really interested in the 2020 election — but we probably don't want to wait until then to evaluate our model.

There is a way we can estimate now how well our model will generalize in the future. We can't fast-forward time, but we can rewind it...

We can split our data in **two sets.** For example:
1. **Train** a model on elections before 2008.
2. **Test** the model on 2008, 2012, 2016.

This "backtesting" helps us estimate how well the model will predict the next elections going forward, starting in 2020.

This is explained in [_Forecasting,_ Chapter 3.4,](https://otexts.com/fpp2/accuracy.html) Evaluating forecast accuracy:

> The accuracy of forecasts can only be determined by considering how well a model performs on new data that were not used when fitting the model.
>
>When choosing models, it is common practice to separate the available data into two portions, training and test data, where the training data is used to estimate any parameters of a forecasting method and the test data is used to evaluate its accuracy. Because the test data is not used in determining the forecasts, it should provide a reliable indication of how well the model is likely to forecast on new data.
>
>
>
>The size of the test set is typically about 20% of the total sample, although this value depends on how long the sample is and how far ahead you want to forecast. The following points should be noted.
>
>- A model which fits the training data well will not necessarily forecast well.
>- A perfect fit can always be obtained by using a model with enough parameters.
>- Over-fitting a model to data is just as bad as failing to identify a systematic pattern in the data.
>
>Some references describe the test set as the “hold-out set” because these data are “held out” of the data used for fitting. Other references call the training set the “in-sample data” and the test set the “out-of-sample data”. We prefer to use “training data” and “test data” in this book.

**How should we split: Randomly? Before/after a given date?**

I recommend you all read a great blog post, [How (and why) to create a good validation set](https://www.fast.ai/2017/11/13/validation-sets/), by fast.ai cofounder Rachel Thomas.

She gives great examples to answer the question “When is a random subset not good enough?” I’m not as opposed to random splits as Rachel Thomas seems to be. But it’s worth thinking about the trade-offs!

Time-based and random splits can both be useful, and you’ll get repeated hands-on practice with both during this unit! (She also talks about the distinction between validation & test sets, which we’ll introduce in the last lesson of this Sprint.)

## Follow Along

Split the data in two sets:
1. Train on elections before 2008.
2. Test on 2008 and after.

```python
# Splitting with slicing syntax
train = df[:14]
test = df[14:]
```

```python
# Splitting with dataframe filtering
train = df[df['Year'] < 2008]
test = df[df['Year'] >= 2008]
```

How many observations (rows) are in the train set? In the test set?

```python

```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20

Note that this volume of data is at least two orders of magnitude smaller than we usually want to work with for predictive modeling.

There are other validation techniques that could be used here, such as [time series cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split), or [leave-one-out cross validation](https://scikit-learn.org/stable/modules/cross_validation.html#leave-one-out-loo) for small datasets. However, for this module, let's start simpler, with train/test split.

Using a tiny dataset is intentional here. It's good for learning because we can see all the data at once.

## Challenge

In your assignment, you will do train/test split, based on date.

# Use scikit-learn to fit a multiple regression

## Overview

We've done train/test split, and we're ready to fit a model.

We'll proceed in 3 steps. The first 2 are review from the previous module. The 3rd is new.

- Begin with baselines (0 features)
- Simple regression (1 feature)
- Multiple regression (2 features)

## Follow Along

### Begin with baselines (0 features)

What was the average Incumbent Party Vote Share, in the 1952-2004 elections?

```python
train['Incumbent Party Vote Share'].mean()
```

    52.46857142857142

What if we guessed this number for every election? How far off would this be on average?

```python
# Arrange y target vectors
target = 'Incumbent Party Vote Share'
y_train = train[target]
y_test = test[target]
```

```python
# Get mean baseline
print('Mean Baseline (using 0 features)')
guess = y_train.mean()
print(guess)
```

    Mean Baseline (using 0 features)
    52.46857142857142

```python
# Train Error
from sklearn.metrics import mean_absolute_error
y_pred = [guess] * len(y_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error (1952-2004 elections): {mae:.2f} percentage points')
```

    Train Error (1952-2004 elections): 4.85 percentage points

```python
# Test Error
y_pred = [guess] * len(y_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error (2008-16 elections): {mae:.2f} percentage points')
```

    Test Error (2008-16 elections): 3.63 percentage points

### Simple regression (1 feature)

Make a scatterplot of the relationship between 1 feature and the target.

We'll use an economic feature: Average Recent Growth in Personal Incomes. ("Bread")

```python
import pandas as pd
import plotly.express as px

px.scatter(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004',
    trendline='ols',  # Ordinary Least Squares
)
```

<meta charset="utf-8" />

1952 & 1968 are outliers: The incumbent party got fewer votes than predicted by the regression. What do you think could explain those years? We'll come back to this soon, but first...

Use scikit-learn to fit the simple regression with one feature.

Follow the [5 step process](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html#Basics-of-the-API), and refer to [Scikit-Learn LinearRegression documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

```python
# 1. Import the appropriate estimator class from Scikit-Learn
from sklearn.linear_model import LinearRegression
```

```python
# 2. Instantiate this class
model = LinearRegression()
```

```python
# 3. Arrange X features matrices (already did y target vectors)
features = ['Average Recent Growth in Personal Incomes']
X_train = train[features]
X_test = test[features]
print(f'Linear Regression, dependent on: {features}')
```

    Linear Regression, dependent on: ['Average Recent Growth in Personal Incomes']

```python
# 4. Fit the model
model.fit(X_train, y_train)
y_pred = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error: {mae:.2f} percentage points')
```

    Train Error: 2.65 percentage points

```python
# 5. Apply the model to new data
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error: {mae:.2f} percentage points')
```

    Test Error: 1.80 percentage points

How does the error compare to the baseline?

### Multiple regression (2 features)

Make a scatterplot of the relationship between 2 features and the target.

We'll add another feature: US Military Fatalities per Million. ("Peace" or the lack thereof.)

Rotate the scatterplot to explore the data. What's different about 1952 & 1968?

```python
px.scatter_3d(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004',

)
```

<meta charset="utf-8" />

Use scikit-learn to fit a multiple regression with two features.

```python
# Re-arrange X features matrices
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']
print(f'Linear Regression, dependent on: {features}')

X_train = train[features]
X_test = test[features]
```

    Linear Regression, dependent on: ['Average Recent Growth in Personal Incomes', 'US Military Fatalities per Million']

```python
# Fit the model
model.fit(X_train, y_train)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

```python
# Check train error
y_pred = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train Error: {mae:.2f} percentage points')
```

    Train Error: 1.33 percentage points

```python
# Apply the model to new data
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test Error: {mae:.2f} percentage points')
```

    Test Error: 1.63 percentage points

How does the error compare to the prior model?

### Plot the plane of best fit

For a regression with 1 feature, we plotted the line of best fit in 2D.

(There are many ways to do this. Plotly Express's `scatter` function makes it convenient with its `trendline='ols'` parameter.)

For a regression with 2 features, we can plot the plane of best fit in 3D!

(Plotly Express has a `scatter_3d` function but it won't plot the plane of best fit for us. But, we can write our own function, with the same "function signature" as the Plotly Express API.)

```python
import itertools
import numpy as np
import plotly.express as px
import plotly.graph_objs as go
from sklearn.linear_model import LinearRegression

def regression_3d(df, x, y, z, num=100, **kwargs):
    """
    Visualize linear regression in 3D: 2 features + 1 target

    df : Pandas DataFrame
    x : string, feature 1 column in df
    y : string, feature 2 column in df
    z : string, target column in df
    num : integer, number of quantiles for each feature
    """

    # Plot data
    fig = px.scatter_3d(df, x, y, z, **kwargs)

    # Fit Linear Regression
    features = [x, y]
    target = z
    model = LinearRegression()
    model.fit(df[features], df[target])

    # Define grid of coordinates in the feature space
    xmin, xmax = df[x].min(), df[x].max()
    ymin, ymax = df[y].min(), df[y].max()
    xcoords = np.linspace(xmin, xmax, num)
    ycoords = np.linspace(ymin, ymax, num)
    coords = list(itertools.product(xcoords, ycoords))

    # Make predictions for the grid
    predictions = model.predict(coords)
    Z = predictions.reshape(num, num).T

    # Plot predictions as a 3D surface (plane)
    fig.add_trace(go.Surface(x=xcoords, y=ycoords, z=Z))

    return fig
```

```python
regression_3d(
    train,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2004'
)
```

<meta charset="utf-8" />

Where are 1952 & 1968 in relation to the plane? Which elections are the biggest outliers now?

Roll over points on the plane to see predicted incumbent party vote share (z axis), dependent on personal income growth (x axis) and military fatatlies per capita (y axis).

### Get and interpret coefficients

During the previous module, we got the simple regression's coefficient and intercept. We plugged these numbers into an equation for the line of best fit, in slope-intercept form: $y = mx + b$

Let's review this objective, but now for multiple regression.

What's the equation for the plane of best fit?

$y = \beta_0 + \beta_1x_1 + \beta_2x_2$

Can you relate the intercept and coefficients to what you see in the plot above?

```python
model.intercept_, model.coef_
```

    (46.25489966153873, array([ 3.59004735, -0.05315709]))

```python
beta0 = model.intercept_
beta1, beta2 = model.coef_
print(f'y = {beta0} + {beta1}x1 + {beta2}x2')
```

    y = 46.25489966153873 + 3.5900473494560536x1 + -0.05315709351049324x2

```python
# This is easier to read
print('Intercept', model.intercept_)
coefficients = pd.Series(model.coef_, features)
print(coefficients.to_string())
```

    Intercept 46.25489966153873
    Average Recent Growth in Personal Incomes    3.590047
    US Military Fatalities per Million          -0.053157

```python
# This does not exactly match correlation
df.corr()['Incumbent Party Vote Share']
```

    Year                                        -0.219182
    Average Recent Growth in Personal Incomes    0.766281
    US Military Fatalities per Million          -0.364377
    Incumbent Party Vote Share                   1.000000
    Name: Incumbent Party Vote Share, dtype: float64

One of the coefficients is positive, and the other is negative. What does this mean?

Let's look at some scenarios. We'll see that one unit's change in an independent variable results in a coefficient worth of change in the dependent variable.

What does the model predict if income growth=0%, fatalities=0

```python
model.predict([[0, 0]])
```

    array([46.25489966])

Income growth = 1% (fatalities = 0)

```python
model.predict([[1, 0]])
```

    array([49.84494701])

The difference between these predictions = ?

```python
model.predict([[1, 0]]) - model.predict([[0, 0]])
```

    array([3.59004735])

What if... income growth = 2% (fatalities = 0)

```python
model.predict([[2, 0]])
```

    array([53.43499436])

The difference between these predictions = ?

```python
model.predict([[2, 0]]) - model.predict([[1, 0]])
```

    array([3.59004735])

What if... (income growth=2%) fatalities = 100

```python
model.predict([[2, 100]])
```

    array([48.11928501])

The difference between these predictions = ?

```python
model.predict([[2, 100]]) - model.predict([[2, 0]])
```

    array([-5.31570935])

What if income growth = 3% (fatalities = 100)

```python
model.predict([[3, 100]])
```

    array([51.70933236])

The difference between these predictions = ?

```python
model.predict([[3, 100]]) - model.predict([[2, 100]])
```

    array([3.59004735])

What if (income growth = 3%) fatalities = 200

```python
model.predict([[3, 200]])
```

    array([46.39362301])

The difference between these predictions = ?

```python
model.predict([[3, 200]]) - model.predict([[3, 100]])
```

    array([-5.31570935])

## Challenge

In your assignment, you'll fit a Linear Regression with at least 2 features.

# Understand how ordinary least squares regression minimizes the sum of squared errors

## Overview

So far, we've evaluated our models by their absolute error. It's an intuitive metric for regression problems.

However, ordinary least squares doesn't directly minimize absolute error. Instead, it minimizes squared error.

In this section, we'll introduce two new regression metrics:

- Squared error
- $R^2$

We'll demostrate two possible methods to minimize squared error:

- Guess & check
- Linear Algebra

## Follow Along

### Guess & Check

This function visualizes squared errors. We'll go back to simple regression with 1 feature, because it's much easier to visualize.

Use the function's m & b parameters to "fit the model" manually. Guess & check what values of m & b minimize squared error.

```python
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

def squared_errors(df, feature, target, m, b):
    """
    Visualize linear regression, with squared errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    df.plot.scatter(feature, target, ax=ax)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    ax.plot(x, y_pred)

    # Plot squared errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    # Print regression metrics
    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)
```

Here's what the mean baseline looks like:

```python
feature = 'Average Recent Growth in Personal Incomes'
squared_errors(train, feature, target, m=0, b=y_train.mean())
```

Notice that $R^2$ is exactly zero.

[$R^2$ represents the proportion of the variance for a dependent variable that is explained by the independent variable(s).](https://en.wikipedia.org/wiki/Coefficient_of_determination)

The mean baseline uses zero independent variables and explains none of the variance in the dependent variable, so its $R^2$ score is zero.

The highest possible $R^2$ score is 1. The lowest possible *Train* $R^2$ score with ordinary least squares regression is 0.

In this demo, it's possible to get a negative Train $R^2$, if you manually set values of m & b that are worse than the mean baseline. But that wouldn't happen in the real world.

However, in the real world, it _is_ possible to get a negative *Test/Validation* $R^2$. It means that your *Test/Validation* predictions are worse than if you'd constantly predicted the mean of the *Test/Validation* set.

---

Now that we've visualized the squared errors for the mean baseline, let's guess & check some better values for the m & b parameters:

```python
squared_errors(train, feature, target, m=3, b=46)
```

You can run the function repeatedly, with different values for m & b.

How do you interpret each metric you see?

- Mean Squared Error
- Root Mean Squared Error
- Mean Absolute Error
- $R^2$

Does guess & check really get used in machine learning? Sometimes! Some complex functions are hard to minimize, so we use a sophisticated form of guess & check called "gradient descent", which you'll learn about in Unit 4.

Fortunately, we don't need to use guess & check for ordinary least squares regression. We have a solution, using linear algebra!

### Linear Algebra

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

#### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

#### Lets calculate our $\beta$ parameters with numpy!

```python
# This is NOT something you'll be tested on. It's just a demo.

# X is a matrix. Add column of constants for fitting the intercept.
def add_constant(X):
    constant = np.ones(shape=(len(X),1))
    return np.hstack((constant, X))
X = add_constant(train[features].values)
print('X')
print(X)

# y is a column vector
y = train[target].values[:, np.newaxis]
print('y')
print(y)

# Least squares solution in code
X_transpose = X.T
X_transpose_X = X_transpose @ X
X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
X_transpose_y = X_transpose @ y
beta_hat = X_transpose_X_inverse @ X_transpose_y

print('Beta Hat')
print(beta_hat)
```

    X
    [[  1.     2.4  190.  ]
     [  1.     2.89   0.  ]
     [  1.     0.85   0.  ]
     [  1.     4.21   1.  ]
     [  1.     3.02 146.  ]
     [  1.     3.62   0.  ]
     [  1.     1.08   2.  ]
     [  1.    -0.39   0.  ]
     [  1.     3.86   0.  ]
     [  1.     2.27   0.  ]
     [  1.     0.38   0.  ]
     [  1.     1.04   0.  ]
     [  1.     2.36   0.  ]
     [  1.     1.72   4.  ]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]]
    Beta Hat
    [[46.25489966]
     [ 3.59004735]
     [-0.05315709]]

```python
# Scikit-learn gave the exact same results!
model.intercept_, model.coef_
```

    (46.25489966153873, array([ 3.59004735, -0.05315709]))

# Define overfitting/underfitting and the bias/variance tradeoff

## Overview

Read [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off). Jake VanderPlas explains overfitting & underfitting:

> Fundamentally, the question of "the best model" is about finding a sweet spot in the tradeoff between bias and variance. Consider the following figure, which presents two regression fits to the same dataset:
>
>
>
> The model on the left attempts to find a straight-line fit through the data. Because the data are intrinsically more complicated than a straight line, the straight-line model will never be able to describe this dataset well. Such a model is said to _underfit_ the data: that is, it does not have enough model flexibility to suitably account for all the features in the data; another way of saying this is that the model has high _bias_.
>
> The model on the right attempts to fit a high-order polynomial through the data. Here the model fit has enough flexibility to nearly perfectly account for the fine features in the data, but even though it very accurately describes the training data, its precise form seems to be more reflective of the particular noise properties of the data rather than the intrinsic properties of whatever process generated that data. Such a model is said to _overfit_ the data: that is, it has so much model flexibility that the model ends up accounting for random errors as well as the underlying data distribution; another way of saying this is that the model has high _variance_.

VanderPlas goes on to connect these concepts to the "bias/variance tradeoff":

> From the scores associated with these two models, we can make an observation that holds more generally:
>
>- For high-bias models, the performance of the model on the validation set is similar to the performance on the training set.
>
>- For high-variance models, the performance of the model on the validation set is far worse than the performance on the training set.
>
> If we imagine that we have some ability to tune the model complexity, we would expect the training score and validation score to behave as illustrated in the following figure:
>
>
>
> The diagram shown here is often called a validation curve, and we see the following essential features:
>
>- The training score is everywhere higher than the validation score. This is generally the case: the model will be a better fit to data it has seen than to data it has not seen.
>- For very low model complexity (a high-bias model), the training data is under-fit, which means that the model is a poor predictor both for the training data and for any previously unseen data.
>- For very high model complexity (a high-variance model), the training data is over-fit, which means that the model predicts the training data very well, but fails for any previously unseen data.
>- For some intermediate value, the validation curve has a maximum. This level of complexity indicates a suitable trade-off between bias and variance.
>
>The means of tuning the model complexity varies from model to model.

So far, our only "means of tuning the model complexity" has been selecting one feature or two features for our linear regression models. But we'll quickly start to select more features, and more complex models, with more "hyperparameters."

This is just a first introduction to underfitting & overfitting. We'll continue to learn about this topic all throughout this unit.

## Follow Along

Let's make our own Validation Curve, by tuning a new type of model complexity: polynomial degrees in a linear regression.

Go back to the the NYC Tribeca condo sales data

```python
# Read NYC Tribeca condo sales data, from first 4 months of 2019.
# Dataset has 90 rows, 9 columns.
df = pd.read_csv(DATA_PATH+'condos/tribeca.csv')
assert df.shape == (90, 9)

# Arrange X features matrix & y target vector
features = ['GROSS_SQUARE_FEET']
target = 'SALE_PRICE'
X = df[features]
y = df[target]
```

Do random [train/test split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=11)
```

Repeatedly fit increasingly complex models, and keep track of the scores

```python
from IPython.display import display, HTML
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

# Credit for PolynomialRegression: Jake VanderPlas, Python Data Science Handbook, Chapter 5.3
# https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#Validation-curves-in-Scikit-Learn
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))

polynomial_degrees = range(1, 10, 2)
train_r2s = []
test_r2s = []

for degree in polynomial_degrees:
    model = PolynomialRegression(degree)
    display(HTML(f'Polynomial degree={degree}'))

    model.fit(X_train, y_train)
    train_r2 = model.score(X_train, y_train)
    test_r2 = model.score(X_test, y_test)
    display(HTML(f'<b style="color: blue">Train R2 {train_r2:.2f}</b>'))
    display(HTML(f'<b style="color: red">Test R2 {test_r2:.2f}</b>'))

    plt.scatter(X_train, y_train, color='blue', alpha=0.5)
    plt.scatter(X_test, y_test, color='red', alpha=0.5)
    plt.xlabel(features)
    plt.ylabel(target)

    x_domain = np.linspace(X.min(), X.max())
    curve = model.predict(x_domain)
    plt.plot(x_domain, curve, color='blue')
    plt.show()
    display(HTML('<hr/>'))

    train_r2s.append(train_r2)
    test_r2s.append(test_r2)

display(HTML('Validation Curve'))
plt.plot(polynomial_degrees, train_r2s, color='blue', label='Train')
plt.plot(polynomial_degrees, test_r2s, color='red', label='Test')
plt.xlabel('Model Complexity (Polynomial Degree)')
plt.ylabel('R^2 Score')
plt.legend()
plt.show()
```

As model complexity increases, what happens to Train $R^2$ and Test $R^2$?

# Review

In your assignment, you'll continue to **predict how much it costs to rent an apartment in NYC,** using the dataset from renthop.com.

- Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.
- Engineer at least two new features.
- Fit a linear regression model with at least two features.
- Get the model's coefficients and intercept.
- Get regression metrics RMSE, MAE, and $R^2$, for both the train and test sets.

You've been provided with a separate notebook for your assignment, which has all the instructions and stretch goals. What's the best test MAE you can get? Share your score and features used with your cohort on Slack!

# Sources

#### Train/Test Split
- James, Witten, Hastie, Tibshirani, [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/), Chapter 2.2, Assessing Model Accuracy
- Hyndman, Athanasopoulos, [_Forecasting,_ Chapter 3.4,](https://otexts.com/fpp2/accuracy.html) Evaluating forecast accuracy
- Rachel Thomas, [How (and why) to create a good validation set](https://www.fast.ai/2017/11/13/validation-sets/)

#### Bias-Variance Tradeoff
- Jake VanderPlas, [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off), Hyperparameters and Model Validation
- StatQuest, [Machine Learning Fundamentals: Bias and Variance](https://youtu.be/EuBBz3bI-aA) (6.5 minutes)

#### "Bread and Peace" Background
- Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)
- Nate Silver, [What Do Economic Models Really Tell Us About Elections?](https://fivethirtyeight.com/features/what-do-economic-models-really-tell-us-about-elections/)

#### "Bread and Peace" Data Sources & Definitions
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33


```python
# -*- coding: utf-8 -*-

%matplotlib inline
!wget https://project-ccap.github.io/data/tiny_celeba.tgz -O tiny_celeba.tgz
!tar -zxf tiny_celeba.tgz
!pip install japanize_matplotlib
```

# DCGAN Tutorial

**Author**: `Nathan Inkawhich` <https://github.com/inkawhich>

# はじめに

本物の有名人の写真を集めたデータセット celeba を用いた，有名人を生成する生成敵対ネットワーク (GAN)。

ここでのコードのほとんどは `pytorch/examples` <https://github.com/pytorch/examples> にある dcgan の実装からのものである。

# 1. 生成敵対ネットワーク Generative Adversarial Networks

## 1.1 GAN とは？

GAN は， 学習データの分布を把握することで， その同じ分布から新しいデータを生成できるように，   深層学習モデルに教えるための枠組みである。
GAN は 2014 年 に Ian Goodfellow によって発明され， 論文 [Generative Adversarial Nets](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf) で初めて説明された。
GAN は *生成器 generator* と *識別器 discriminator* という 2 つの異なるモデルで構成されている。
生成器の仕事は， 学習画像に似た 「偽」の画像を生成することである。
識別器の役割は， ある画像を見て， それが本物の訓練画像なのか， 生成器からの偽物の画像なのかを出力することである。
識別器の仕事は，ある画像を見て、それが本物の訓練画像か， 生成器の偽物の画像かを出力することである。
訓練中， 生成器は常に識別器を出し抜こうと， より良い偽物を生成し， 識別器はより良い探偵になって本物と偽物の画像を正しく分類するように努力する。
このゲームの均衡は， 生成器があたかも訓練データから直接来たかのような完璧な偽物を生成しているときで，  識別器は生成器の出力が本物か偽物かを常に 50 ％の信頼度で推測することになる。

ここでの表記を， 識別器から定義する。
$x$ を画像のデータとする。
$D(x)$ は識別器ネットワークで $x$ が生成器ではなく学習データから来たものである確率を (スカラー) で出力する。
ここでは， 画像を扱っているので $D(x)$ の入力は CHW サイズ 3x64x64 の画像である。
直感的には $D(x)$ は $x$ が学習データから来た場合は HIGH $x$ が生成器から来た場合は LOW になるはずである。
$D(x)$ は， 従来の 2 値分類器と考えることもできる。

生成器の表記法として $z$ を標準正規分布からサンプリングされた潜在空間ベクトルとします。
$G(z)$ は， 潜在ベクトル $z$ をデータ空間にマッピングする生成器関数を表す。
$G$ の目的は， 学習データの分布（$p_{data}$）を推定し， その推定分布（$p_g$）から偽のサンプルを生成することである。

つまり $D(G(z))$ は， 生成器 $G$ の出力が実像である確率(スカラー) である。
[Goodfellow の 論文](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf)  に記載されているように， $D$ と $G$ はミニマックスゲームを行い， $D$ は本物と偽物を正しく分類する確率 ($logD(x)$) を最大化しようとし， $G$ は $D$ の出力が偽物であると予測する確率 ($log(1-D(G(x)))$) を最小化しようとしする。
論文より GAN の損失関数は

\begin{align}
\underset{G}{\text{min}} \underset{D}{\text{max}}V(D,G) = \mathbb{E}_{x\sim p_{data}(x)}\big[\log D(x)\big] + \mathbb{E}_{z\sim p_{z}(z)}\big[\log(1-D(G(z)))\big]\tag{1}
\end{align}

理論的には， このミニマックスゲームの解は $p_g = p_{data}$ となり， 識別器は入力が本物か偽物かをランダムに推測することになる。
しかし GAN の収束理論はまだ盛んに研究されており， 現実にはモデルがこの点まで学習するとは限らない。

# 2. DCGANとは？

DCGAN は上述の GAN を直接拡張したもだが， 識別器と生成器にそれぞれ畳み込み層と畳み込み変換層を明示的に使用している。
DCGAN は Radfordらが論文 [Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks](https://arxiv.org/pdf/1511.06434.pdf)  で初めて発表した。
識別器は strided [`convolution`](https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d)  層， [`batch norm`](https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm2d) 層，
[`LeakyReLU`](https://pytorch.org/docs/stable/nn.html#torch.nn.LeakyReLU)  活性化関数で構成されている。
入力は 3x64x64 の入力画像であり，出力は入力が実データの分布からのものであるというスカラー確率である。
生成器は，[`convolutional-transpose`](https://pytorch.org/docs/stable/nn.html#torch.nn.ConvTranspose2d) 層，バッチノルム層，[`ReLU`](https://pytorch.org/docs/stable/nn.html#relu) 活性化関数で構成されている。
入力は標準正規分布から引き出された潜在的なベクトル $z$ で，出力は 3x64x64 の RGB 画像である。
strided conv-transpose 層により，潜在ベクトルは画像と同じ形状のボリュームに変換される。
この論文では， 最適化手法の設定方法， 損失関数の計算方法， モデルの重みの初期化方法などのヒントも紹介されているが， これらについては後の節で説明する。

```python
from __future__ import print_function
#%matplotlib inline
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation

ngpu = 1
device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

import japanize_matplotlib
from IPython.display import HTML

# Set random seed for reproducibility
manualSeed = 999
#manualSeed = random.randint(1, 10000) # use if you want new results
print("Random Seed: ", manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)

# from https://discuss.pytorch.org/t/how-to-cast-a-tensor-to-another-type/2713/28
torch.set_default_dtype(torch.float32)
```

    Random Seed:  999

# 3. 入力

実行するための入力を定義する。

- **dataroot** - データセットのフォルダのルートへのパス。データセットについては， 次節で詳しく説明する。
- **workers** - DataLoader でデータを読み込むためのワーカースレッド数。
- **batch_size** - 学習に使用するバッチサイズ。DCGAN 論文ではバッチサイズ  128 を使用してる。
- **image_size** - 学習に使用する画像の空間サイズ。
   この実装では，デフォルトで 64x64 に設定されている。
別のサイズにしたい場合は，D と G の構造を変更する必要がある。
   詳細については， <https://github.com/pytorch/examples/issues/70> を参照。
- **nc** - 入力画像の色チャンネル数， カラー画像の場合は 3
- **nz** - 潜在的なベクトルの長さ。
- **ngf** - 生成器で処理される特徴地図の深さに関連
- **ndf** - 識別器を通して伝搬される特徴地図の深さを設定
- **num_epochs** - 実行する学習エポック数。長く学習すれば良い結果が得られるだろうが，それだけ時間もかかる。
- **lr** - 学習時の学習率。DCGAN 論文に記載されているように，この数値は 0.0002 とする。
-  **β1** - Adam 最適化 の β1 ハイパーパラメータ。論文にあるように、この数値は 0.5 であるべき。
- **ngpu** - 利用可能な GPU 数。この値が 0 の場合，コードは CPU モードで実行される。この数字が 0 より大きい場合，その数の GPU で実行される。

```python
#!ls tiny_celeba
```

```python
# Root directory for dataset
dataroot = "data/celeba"
dataroot = 'tiny_celeba' #img_align_celeba'

# Number of workers for dataloader
workers = 2
#workers = 4

# Batch size during training
batch_size = 128

# Spatial size of training images. All images will be resized to this
#   size using a transformer.
image_size = 64

# Number of channels in the training images. For color images this is 3
nc = 3

# Size of z latent vector (i.e. size of generator input)
nz = 100

# Size of feature maps in generator
ngf = 64

# Size of feature maps in discriminator
ndf = 64

# Number of training epochs
num_epochs = 20

# Learning rate for optimizers
lr = 0.0002

# Beta1 hyperparam for Adam optimizers
beta1 = 0.5

# Number of GPUs available. Use 0 for CPU mode.
ngpu = 1
#ngpu = 0
```

# 4. データ

このチュートリアルでは [Celeb-A Faces dataset](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) を使う。
リンク先のサイト，または [Google Drive](https://drive.google.com/drive/folders/0B7EVK8r0v71pTUZsaXdaSnZBZzg) でダウンロードできる。

データセットは *img_align_celeba.zip*というファイル名でダウンロードされる。
ダウンロードが完了したら *celeba* という名前のディレクトリを作成し， そのディレクトリに zip ファイルを解凍する。
そして、このノートブックの *dataroot* 入力を， 先ほど作成した *celeba* ディレクトリに設定してください。
結果として， ディレクトリ構造は次のようになる。

```
/path/to/celeba
       -> img_align_celeba
           -> 188242.jpg
           -> 173822.jpg
           -> 284702.jpg
           -> 537394.jpg
              ...
```

これは重要なステップである。
なぜなら ImageFolder データセットクラスを使用するため， データセットのルートフォルダにサブディレクトリがあることが必要だからである。
これで， データセットの作成， データローダの作成， 実行するデバイスの設定， そして最後に訓練データの一部を可視化することができる。

```python
!mkdir tiny_celeba/img_align_celeba
!mv tiny_celeba/*.jpg tiny_celeba/img_align_celeba/
```

```python
# We can use an image folder dataset the way we have it setup.
# Create the dataset
dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
# Create the dataloader
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)

# Decide which device we want to run on
device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

# Plot some training images
real_batch = next(iter(dataloader))
plt.figure(figsize=(14,14))
plt.axis("off")
plt.title("Training Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(),(1,2,0)))
```

```python
real_batch = next(iter(dataloader))
plt.figure(figsize=(24,12))
plt.axis("off")
plt.title("Training Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:24], padding=2, normalize=True).cpu(),(1,2,0)))
```

```python
plt.imshow(real_batch[0][0].numpy().transpose(1,2,0).clip(0,1))
```

# 5. 実装

入力パラメータの設定とデータセットの準備ができたら，いよいよ実装に入る。
まず， 初期化の方法から始めて， 生成器， 識別器， 損失関数， 学習ループについて詳しく説明する。

# 6. 重みの初期化

DCGAN の論文から， 著者はすべてのモデルの重みを `mean=0`, `stdev=0.02` の正規分布からランダムに初期化することを指定している。
`weights_init` 関数は， 初期化されたモデルを入力とし， この基準を満たすように， すべての畳み込み層， 畳み込み変換層， バッチ正規化層を再初期化する。
この関数は， 初期化後すぐにモデルに適用される。

```python
# custom weights initialization called on netG and netD
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)
```

# 7. 生成器

生成器である $G$ は， 潜在空間ベクトル ($z$) をデータ空間にマッピングするように設計されている。
データは画像なので $z$ をデータ空間に変換するということは， 最終的に学習画像と同じサイズ（3x64x64）の RGB 画像を作成することになる。
$z$ をデータ空間に変換するということは， 最終的に学習画像と同じサイズ（3x64x64）の RGB 画像を作成することを意味する。
実際には 2 次元の畳み込み変換層と 2 次元のバッチ・ノルム層，および ReLU  活性化を組み合わせて，この変換を行っている。
生成器の出力は，tanh 関数を介して入力データの範囲 $[-1,1]$ に戻される。
これは DCGAN 論文の重要な貢献であるため，conv-transpose 層の後のバッチノルム関数の存在は注目に値する。
これは DCGAN 論文の重要な貢献である。
これらの層は， 学習中の勾配の流れを助ける。
DCGAN 論文に掲載されている生成器のイメージを以下に示す。

Radford らの論文より

入力 節で設定した入力 (*nz*, *ngf*, *nc*) が， コード上の生成器アーキテクチャにどのように影響するかに注目。
*nz*は  $z$  入力ベクトルの長さ，*ngf*は，生成器を介して伝搬される特徴地図のサイズ，*nc* は，出力画像のチャンネル数 (RGB 画像の場合は 3 に設定) である。
以下は，この生成器のコードである。

```python
# Generator Code

class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)
```

これで， 生成器を実体化して  `weights_init` 関数を適用することができる。
印刷されたモデルを見て， 生成器オブジェクトがどのように構成されているかを確認してみよう。

```python
# Create the generator
netG = Generator(ngpu).to(device)

# Handle multi-gpu if desired
if (device.type == 'cuda') and (ngpu > 1):
    netG = nn.DataParallel(netG, list(range(ngpu)))

# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.2.
netG.apply(weights_init)

# Print the model
print(netG)
```

# 8. 識別器

前述のように，識別器である $D$ は，画像を入力とし， その入力画像が (偽物ではなく) 本物であるかどうかの確率をスカラーで出力する 2 値分類ネットワークである。
ここでは $D$ は 3x64x64 の入力画像を受け取り，Conv2d，BatchNorm2d，LeakyReLU の各層で処理し，シグモイド活性化関数によって最終的な確率を出力している。
このアーキテクチャは， 必要に応じて層を増やすことができる，strided convolution, BatchNorm, LeakyReLUを使用することに意味がある。
DCGAN の論文では， ダウンサンプルにプーリングではなく strided convolution を使用することは， ネットワークに独自のプーリング関数を学習させることができるため， 良い方法であると述べられている。
また，バッチ正則化とリーキー ReLU は $G$ と $D$ の両方の学習過程にとって重要な， 健全な勾配の流れを促進する。

識別機のコード

```python
class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 4 x 4
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.main(input)
```

これで， 生成器と同様に，判別器を作成し `weights_init`  関数を適用して，モデルの構造を表示することができる。

```python
# Create the Discriminator
netD = Discriminator(ngpu).to(device)

# Handle multi-gpu if desired
if (device.type == 'cuda') and (ngpu > 1):
    netD = nn.DataParallel(netD, list(range(ngpu)))

# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.2.
netD.apply(weights_init)

# Print the model
print(netD)
```

# 8. 損失関数と最適化関数

$D$ と $G$ の設定ができたら， 損失関数と最適化関数を使って学習方法を指定する。
ここでは，2 値化交差エントロピー損失  ([`BCELoss`](https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss)  関数を用いる。
PyTorch では以下のように定義されている:

\begin{align}
\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad\\
l_n = - \left[y_n \cdot \log x_n + (1 - y_n) \cdot \log (1 - x_n) \right]
\end{align}

この関数は， 目的関数の両対数成分 (すなわち $\log(D(x))$ と $\log(1-D(G(z)))$) の計算を提供することに注目。
BCE 方程式のどの部分を $y$ の入力に使用するかを指定することができる。
これは下の訓練ループで実現されるが， どのようにして計算したい成分を選択できるかを理解することは重要である。
ここで重要なのは $y$ を変更するだけで，どの成分を計算するかを選択できることである。

次に， 本物のラベルを 1， 偽物のラベルを 0 と定義する。
これらのラベルは $D$ と $G$ の損失を計算する際に使用され， これは元の GAN 論文でも使用されている慣習である。
最後に $D$ 用と $G$ 用の 2 つの別々の最適化関数を設定する。
DCGAN 論文で指定されているように， どちらも学習率 0.0002, Beta1 =0.5 の Adam 最適化である。
生成器の学習の進行状況を把握するために， ガウス分布 (つまり 固定ノイズ) から引き出された潜在的なベクトルの固定バッチを生成する。
学習ループでは，この 固定ノイズ を定期的に $G$ に入力し，反復していくうちに
ノイズの中から画像が形成されていく様子を見ることができる。

```python
# Initialize BCELoss function
criterion = nn.BCELoss()

# Create batch of latent vectors that we will use to visualize
#  the progression of the generator
fixed_noise = torch.randn(64, nz, 1, 1, device=device)

# Establish convention for real and fake labels during training
real_label = 1
fake_label = 0

# Setup Adam optimizers for both G and D
optimizerD = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))
```

# 9. 訓練

最後に GAN の枠組みのすべての部分が定義されたので，それを訓練することができる。
ハイパーパラメータの設定を間違えると，何が悪かったのかほとんど説明できないままモード崩壊してしまうので GAN の訓練はやや芸術的な形式であることに留意して欲しい。
ここでは Goodfellow の論文のアルゴリズム 1 に忠実に従いつつ [ganhacks](https://github.com/soumith/ganhacks) で示されているいくつかのベストプラクティスを遵守する。
具体的には 「本物と偽物の画像に対して異なるミニバッチを作成する」ことと，G の目的関数が $\log D(G(z))$ を最大化するように調整することである。
学習は大きく 2 つのパートに分かれている。
パート1 では 識別器の更新， パート2 では生成器の更新を行う。

**パート1 - 識別器の訓練**

識別器を学習する目的は， 与えられた入力を正しく本物か偽物かに分類する確率を最大化することである。
Goodfellow の言葉を借りれば「確率的勾配を上昇させて識別器を更新する」ということになる。
実際には $\log(D(x)) + \log(1-D(G(x))$ を最大化したい。
ganhacks からの別のミニバッチの提案により， これを 2 つのステップで計算する。
まず，訓練セットから実サンプルのバッチを構築し，$D$ を前向き経路で処理し，損失 ($\log(D(x))$) を計算し，逆向経路で勾配を計算する。
次に，現在の生成器で偽のサンプルのバッチを作成し，このバッチを $D$ に前向き経路に渡し，損失 ($\log(1-D(G(z)))$) を計算する。
そして，逆向経路処理で勾配を蓄積する。
ここで， すべての本物のバッチとすべての偽物のバッチの両方から蓄積された勾配を使って， 識別機の最適化関数のステップを呼び出す。

**パート2 - 生成器の訓練**

オリジナル論文で述べられているように， 我々は，より良い偽物を生成するために $\log(1-D(G(z)))$ を最小化することで生成器を訓練したいと考えている。
前述したように Goodfellow はこの方法では， 特に学習の初期段階では十分な勾配が得られないことを示した。
そこで，代わりに $\log(D(G(z)))$ を最大化することにした。
このコードでは，パート1 の生成器の出力を識別器で分類し，G の損失を計算し， 実ラベルを GT* として使用し， 逆向経路処理で G の勾配を計算し， 最後に最適化関数のステップで G のパラメータを更新することで，これを実現している。
実ラベルを損失関数の GT ラベルとして使用するのは直感的ではないかもしれないが， これにより BCELoss の$\log(x)$ 部分（$\log(1-x)$ 部分ではなく) を使用することができ，これはまさに我々が望むものである。

最後に，統計情報の報告を行う。
各エポックの終わりに，固定ノイズのバッチをジェネレーターに通して G の訓練の進捗状況を視覚的に確認する。
訓練の統計情報を報告は:

- **Loss_D** - すべての本物のバッチとすべての偽物のバッチの損失の合計として計算された識別器の損失 ($\log(D(x)) + \log(D(G(z)))$).
- **Loss_G** - 生成器の損失は $\log(D(G(z)))$ として計算される。
- **D(x)** - すべての現実のバッチに対する識別器の (バッチ全体の) 平均出力。これは 1 に近い値から始まり G が良くなると理論的には 0.5 に収束するはずで ある。この理由を考えてみよう。
- **D(G(z))** - すべての偽のバッチに対する識別器の平均出力。最初の数字は D が更新される前で 2 番目の数字は D が更新された後。これらの数値は 0 に近いところから始まり G が良くなるにつれて 0.5 に収束するはずである。この理由を考えてみよう。

**注** このステップは， 何回エポックを実行したか， データセットからいくつかのデータを削除したかによって，時間がかかることがある。

```python
# 訓練ループ

# Lists to keep track of progress
img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
# For each epoch
for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        #label = torch.full((b_size,), real_label, device=device)
        label = torch.full((b_size,), real_label, device=device).float()

        # Forward pass real batch through D
        #output = netD(real_cpu).view(-1)
        output = netD(real_cpu).view(-1).float()

        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()

        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()

        # Output training stats
        if i % 50 == 0:
            print(f'[{epoch}/{num_epochs}][{i}/{len(dataloader)}]\t',
                  f'識別器損失: {errD.item():.3f}\t生成器損失: {errG.item():.3f}\t',
                  f'D(x):{D_x:.3f}\tD(G(z)): {D_G_z1:.3f} / {D_G_z2:.3f}')

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1
```

# 10. 結果

最後に，我々の結果を確認しよう。
ここでは 3 つの異なる結果を見てみよう。
まず D と G の損失が訓練中にどのように変化したかを見る。
次に 固定ノイズバッチでの G の出力をエポックごとに可視化する。
そして 3 つ目は，実データのバッチと G の偽データのバッチを比較する。

## 10.1 損失とトレーニング回数の関係

以下は D と G の損失と訓練の反復回数をプロットしたものである。

```python
plt.figure(figsize=(14,8))
plt.title("生成器と識別機の損失値の変化")
plt.plot(G_losses,label="生成器", color='green')
plt.plot(D_losses,label="識別器", color='blue')
plt.xlabel("反復回数")
plt.ylabel("損失")
plt.legend()
plt.show()
```

## 10.2 識別機の学習の視覚化

学習の各エポックの後， 固定ノイズバッチに生成器の出力を保存したことを思い出せ。
ここでは G の学習経過をアニメーションで可視化してみよう。
再生ボタンを押すとアニメーションが始まる。

```python
#%%capture
fig = plt.figure(figsize=(8,8))
plt.axis("off")
ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated=True)] for i in img_list]
ani = animation.ArtistAnimation(fig, ims, interval=1000, repeat_delay=1000, blit=True)

HTML(ani.to_jshtml())
```

## 10.3 本物の画像と偽物の画像の比較

最後に，本物の画像と偽物の画像を並べて見てみまよう。

```python
# Grab a batch of real images from the dataloader
real_batch = next(iter(dataloader))

# Plot the real images
plt.figure(figsize=(15,15))
plt.subplot(1,2,1)
plt.axis("off")
plt.title("Real Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))

# Plot the fake images from the last epoch
plt.subplot(1,2,2)
plt.axis("off")
plt.title("Fake Images")
plt.imshow(np.transpose(img_list[-1],(1,2,0)))
plt.show()
```

```python
#help(netD)
```

```python
print(netD.state_dict(), netG.state_dict())
#print(optimizerD.state_dict(), optimizerG.state_dict())
#!pwd
```

```python
PATH = '2020-0217_011dcgan_faces2.pth'
torch.save({
            'netD_state_dict': netD.state_dict(),
            'netG_state_dict': netG.state_dict(),
            'optimizerD_state_dict': optimizerD.state_dict(),
            'optimizerG_state_dict': optimizerG.state_dict(),
            }, PATH)
```

```python
#!ls -t | head
!file 2020-0217_011dcgan_faces.pth
```

```python
print(list(netD.parameters())[5].size())
```

```python
np.load()
```


# **Quantum K-Means**

```python
pip install qiskit-ibmq-provider
```

    Requirement already satisfied: qiskit-ibmq-provider in /opt/conda/lib/python3.8/site-packages (0.18.3)
    Requirement already satisfied: requests>=2.19 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (2.27.1)
    Requirement already satisfied: websocket-client>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (1.3.1)
    Requirement already satisfied: numpy>=1.13 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (1.22.3)
    Requirement already satisfied: python-dateutil>=2.8.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (2.8.2)
    Requirement already satisfied: qiskit-terra>=0.18.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (0.19.2)
    Requirement already satisfied: urllib3>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (1.26.8)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-ibmq-provider) (1.1.0)
    Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.8/site-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider) (1.16.0)
    Requirement already satisfied: scipy>=1.5 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (1.8.0)
    Requirement already satisfied: tweedledum<2.0,>=1.1 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (1.1.1)
    Requirement already satisfied: ply>=3.10 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (3.11)
    Requirement already satisfied: dill>=0.3 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (0.3.4)
    Requirement already satisfied: stevedore>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (3.5.0)
    Requirement already satisfied: symengine>=0.8 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (0.9.2)
    Requirement already satisfied: python-constraint>=1.4 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (1.4.0)
    Requirement already satisfied: psutil>=5 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (5.9.0)
    Requirement already satisfied: sympy>=1.3 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (1.10)
    Requirement already satisfied: retworkx>=0.10.1 in /opt/conda/lib/python3.8/site-packages (from qiskit-terra>=0.18.0->qiskit-ibmq-provider) (0.11.0)
    Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider) (2021.10.8)
    Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider) (2.0.12)
    Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests>=2.19->qiskit-ibmq-provider) (3.3)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /opt/conda/lib/python3.8/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider) (1.5.0)
    Requirement already satisfied: cryptography>=1.3 in /opt/conda/lib/python3.8/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider) (36.0.1)
    Requirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.8/site-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider) (1.15.0)
    Requirement already satisfied: pycparser in /opt/conda/lib/python3.8/site-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider) (2.21)
    Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from stevedore>=3.0.0->qiskit-terra>=0.18.0->qiskit-ibmq-provider) (5.8.1)
    Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.8/site-packages (from sympy>=1.3->qiskit-terra>=0.18.0->qiskit-ibmq-provider) (1.2.1)
    Note: you may need to restart the kernel to use updated packages.

```python
pip install matlab
```

    Collecting matlab
      Downloading matlab-0.1.tar.gz (538 bytes)
    Building wheels for collected packages: matlab
      Building wheel for matlab (setup.py) ... done
      Created wheel for matlab: filename=matlab-0.1-py3-none-any.whl size=1180 sha256=6f3feb961dd6f5a1288d05aa85864287831ad165f88923f7cd72e85e39990ea9
      Stored in directory: /home/jovyan/.cache/pip/wheels/9d/e3/ca/f9444a09793775674f6d6c3389b6192c72eff2503085389009
    Successfully built matlab
    Installing collected packages: matlab
    Successfully installed matlab-0.1
    Note: you may need to restart the kernel to use updated packages.

```python
import matplotlib.pyplot as plot
import pandas as pand
from qiskit import QuantumRegister, ClassicalRegister
from qiskit import QuantumCircuit
from qiskit import Aer, execute
from numpy import pi
```

    <frozen importlib._bootstrap>:219: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject

```python
figure, axis = plot.subplots()
axis.set(xlabel='Feature 1', ylabel='Feature 2')
```

```python
data_input = pand.read_csv('kmeans_input.csv',
    usecols=['Feature 1', 'Feature 2', 'Class'])
```

```python
isRed = data_input['Class'] == 'Red'
isGreen = data_input['Class'] == 'Green'
isBlack = data_input['Class'] == 'Black'
```

```python
# Filter data
redData = data_input[isRed].drop(['Class'], axis=1)
greenData = data_input[isGreen].drop(['Class'], axis=1)
blackData = data_input[isBlack].drop(['Class'], axis=1)
```

```python
y_p = 0.141
x_p = -0.161
```

```python
xgc = sum(redData['Feature 1']) / len(redData['Feature 1'])
xbc = sum(greenData['Feature 1']) / len(greenData['Feature 1'])
xkc = sum(blackData['Feature 1']) / len(blackData['Feature 1'])
```

```python
# Finding the y-coords of the centroids
ygc = sum(redData['Feature 2']) / len(redData['Feature 2'])
ybc = sum(greenData['Feature 2']) / len(greenData['Feature 2'])
ykc = sum(blackData['Feature 2']) / len(blackData['Feature 2'])
```

```python
# Plotting the centroids
plot.plot(xgc, ygc, 'rx')
plot.plot(xbc, ybc, 'gx')
plot.plot(xkc, ykc, 'kx')
```

```python
plot.plot(x_p, y_p, 'bo')
```

```python
# Setting the axis ranges
plot.axis([-1, 1, -1, 1])
```

```python
plot.show()
```

```python
# Calculating theta and phi values
phi_list = [((x + 1) * pi / 2) for x in [x_p, xgc, xbc, xkc]]
theta_list = [((x + 1) * pi / 2) for x in [y_p, ygc, ybc, ykc]]
```

```python
quantumregister = QuantumRegister(3, 'quantumregister')
```

```python
classicregister = ClassicalRegister(1, 'classicregister')
quantum_circuit = QuantumCircuit(quantumregister, classicregister, name='qc')
backend = Aer.get_backend('qasm_simulator')
quantum_results_list = []
```

```python
for i in range(1, 4):
    quantum_circuit.h(quantumregister[2])

    quantum_circuit.u3(theta_list[0], phi_list[0], 0, quantumregister[0])
    quantum_circuit.u3(theta_list[i], phi_list[i], 0, quantumregister[1])

    quantum_circuit.cswap(quantumregister[2], quantumregister[0], quantumregister[1])
    quantum_circuit.h(quantumregister[2])

    quantum_circuit.measure(quantumregister[2], classicregister[0])

    quantum_circuit.reset(quantumregister)

    job = execute(quantum_circuit, backend=backend, shots=1024)
    result = job.result().get_counts(quantum_circuit)
    quantum_results_list.append(result['1'])

print(quantum_results_list)
```

    /tmp/ipykernel_110/1224122024.py:5: DeprecationWarning: The QuantumCircuit.u3 method is deprecated as of 0.16.0. It will be removed no earlier than 3 months after the release date. You should use QuantumCircuit.u instead, which acts identically. Alternatively, you can decompose u3 in terms of QuantumCircuit.p and QuantumCircuit.sx: u3(ϴ,φ,λ) = p(φ+π) sx p(ϴ+π) sx p(λ) (2 pulses on hardware).
      quantum_circuit.u3(theta_list[0], phi_list[0], 0, quantumregister[0])
    /tmp/ipykernel_110/1224122024.py:6: DeprecationWarning: The QuantumCircuit.u3 method is deprecated as of 0.16.0. It will be removed no earlier than 3 months after the release date. You should use QuantumCircuit.u instead, which acts identically. Alternatively, you can decompose u3 in terms of QuantumCircuit.p and QuantumCircuit.sx: u3(ϴ,φ,λ) = p(φ+π) sx p(ϴ+π) sx p(λ) (2 pulses on hardware).
      quantum_circuit.u3(theta_list[i], phi_list[i], 0, quantumregister[1])

    [79, 64, 115]

    /tmp/ipykernel_110/1224122024.py:5: DeprecationWarning: The QuantumCircuit.u3 method is deprecated as of 0.16.0. It will be removed no earlier than 3 months after the release date. You should use QuantumCircuit.u instead, which acts identically. Alternatively, you can decompose u3 in terms of QuantumCircuit.p and QuantumCircuit.sx: u3(ϴ,φ,λ) = p(φ+π) sx p(ϴ+π) sx p(λ) (2 pulses on hardware).
      quantum_circuit.u3(theta_list[0], phi_list[0], 0, quantumregister[0])
    /tmp/ipykernel_110/1224122024.py:6: DeprecationWarning: The QuantumCircuit.u3 method is deprecated as of 0.16.0. It will be removed no earlier than 3 months after the release date. You should use QuantumCircuit.u instead, which acts identically. Alternatively, you can decompose u3 in terms of QuantumCircuit.p and QuantumCircuit.sx: u3(ϴ,φ,λ) = p(φ+π) sx p(ϴ+π) sx p(λ) (2 pulses on hardware).
      quantum_circuit.u3(theta_list[i], phi_list[i], 0, quantumregister[1])

```python
class_list = ['Red', 'Green', 'Black']
quantum_p_class = class_list[quantum_results_list.index(min(quantum_results_list))]
distances_list = [((x_p - i[0])**2 + (y_p - i[1])**2)**0.5 for i in [(xgc, ygc), (xbc, ybc), (xkc, ykc)]]
classical_p_class = class_list[distances_list.index(min(distances_list))]
```

```python
print("""using quantumdistance algorithm,
 the new data point is related to the""", quantum_p_class,
 'class.\n')
print('Euclidean distances are listed: ', distances_list, '\n')
print("""based on euclidean distance calculations,
 the new data point is related to the""", classical_p_class,
 'class.')
```

    using quantumdistance algorithm,
     the new data point is related to the Green class.

    Euclidean distances are listed:  [0.520285324797846, 0.4905204028376394, 0.7014755294377704]

    based on euclidean distance calculations,
     the new data point is related to the Green class.

```python
figure, axis = plot.subplots()
axis.set(xlabel='Feature 1', ylabel='Feature 2')

data_input = pand.read_csv('kmeans_input.csv',
    usecols=['Feature 1', 'Feature 2', 'Class'])

isRed = data_input['Class'] == 'Red'
isGreen = data_input['Class'] == 'Green'
isBlack = data_input['Class'] == 'Black'

# Filter data
redData = data_input[isRed].drop(['Class'], axis=1)
greenData = data_input[isGreen].drop(['Class'], axis=1)
blackData = data_input[isBlack].drop(['Class'], axis=1)

y_p = 0.141
x_p = -0.161

xgc = sum(redData['Feature 1']) / len(redData['Feature 1'])
xbc = sum(greenData['Feature 1']) / len(greenData['Feature 1'])
xkc = sum(blackData['Feature 1']) / len(blackData['Feature 1'])

# Finding the y-coords of the centroids
ygc = sum(redData['Feature 2']) / len(redData['Feature 2'])
ybc = sum(greenData['Feature 2']) / len(greenData['Feature 2'])
ykc = sum(blackData['Feature 2']) / len(blackData['Feature 2'])

# Plotting the centroids
plot.plot(xgc, ygc, 'rx')
plot.plot(xbc, ybc, 'gx')
plot.plot(xkc, ykc, 'kx')

plot.plot(x_p, y_p, 'bo')

# Setting the axis ranges
plot.axis([-1, 1, -1, 1])

plot.show()

# Calculating theta and phi values
phi_list = [((x + 1) * pi / 2) for x in [x_p, xgc, xbc, xkc]]
theta_list = [((x + 1) * pi / 2) for x in [y_p, ygc, ybc, ykc]]

quantumregister = QuantumRegister(3, 'quantumregister')

classicregister = ClassicalRegister(1, 'classicregister')

quantum_circuit = QuantumCircuit(quantumregister, classicregister, name='qc')

backend = Aer.get_backend('qasm_simulator')

quantum_results_list = []

for i in range(1, 4):
    quantum_circuit.h(quantumregister[2])

    quantum_circuit.u3(theta_list[0], phi_list[0], 0, quantumregister[0])
    quantum_circuit.u3(theta_list[i], phi_list[i], 0, quantumregister[1])

    quantum_circuit.cswap(quantumregister[2], quantumregister[0], quantumregister[1])
    quantum_circuit.h(quantumregister[2])

    quantum_circuit.measure(quantumregister[2], classicregister[0])

    quantum_circuit.reset(quantumregister)

    job = execute(quantum_circuit, backend=backend, shots=1024)
    result = job.result().get_counts(quantum_circuit)
    quantum_results_list.append(result['1'])

print(quantum_results_list)

class_list = ['Red', 'Green', 'Black']

quantum_p_class = class_list[quantum_results_list.index(min(quantum_results_list))]

distances_list = [((x_p - i[0])**2 + (y_p - i[1])**2)**0.5 for i in [(xgc, ygc), (xbc, ybc), (xkc, ykc)]]
classical_p_class = class_list[distances_list.index(min(distances_list))]

print("""using quantumdistance algorithm,
 the new data point is related to the""", quantum_p_class,
 'class.\n')
print('Euclidean distances are listed: ', distances_list, '\n')
print("""based on euclidean distance calculations,
 the new data point is related to the""", classical_p_class,
 'class.')
```


# Block 5: One-dimensional optimal transport
### Alfred Galichon (NYU)
## `math+econ+code' masterclass on matching models, optimal transport and applications
© 2018-2019 by Alfred Galichon. Support from NSF grant DMS-1716489 is acknowledged. Julie Lenoir and James Nesbit contributed.

## Optimal transport I: One-dimensional matching

### Learning Objectives

* Copulas and comonotonicity

* Positive Assortative Matching

* The Wage Equation

### References

* [OTME], Ch. 4 and App. C

* Complements:

     * Becker (1973). A Theory of Marriage: Part I. *JPE*.

     * Sattinger (1993). Assignment Models of the Distribution of Earnings. *JEL*.

     * Gabaix and Landier (2008). Why has CEO Pay Increased So Much? *QJE*.

     * Tervi0 (2008). The Difference That CEOs Make: An Assignment Model Approach. *AER*.

## Motivation

Today we shall study one-dimensional models of matching. The application we shall envision is to the literature on CEO compensation: talented CEOs arguably generate more output if they run large firms than if they run a small business. Hence in these models, the CEO's talent is leveraged by the firm size.

The output of a match is CEO's talent multiplied by a term capturing the firm's size. This surplus function belongs to a class which leads to positive assortative matching: the optimal matching will consists of assigning the most talented CEO with the biggest firm, etc.

The model we shall follows a literature in family economics, see Becker (1973), and labor economics: Sattinger (1993), Tervio (2008), Gabaix and Landier (2008). Our application will be based on Gabaix and Landier's CEO compensation data. The difference with the empirical literature on marriage is that one observes typically less relevant characteristics on CEOs, but we observe the transfers, which are typically not observed whithin the couple.

## Data

For each firm, we observe the size of the firm (market cap) and the compensation.

Distribution of CEO talent (to be defined later) is calibrated in their paper and we will take it as given; so will be the production function.

Our exercise will consist of predicting the CEO compensation. (Actually GL do the opposite: they use the distribution of wages to calibrate the production function).

```R
thePath = getwd()
data = as.matrix(read.csv(paste0(thePath, "/../data_mec_optim/wages_gabaix-landier/data_Gabaix_Landier.csv"), header = TRUE))
head(data)
```

A matrix: 6 × 5 of type dbl

yeara	mktcap	nFIRM	comp	nCEO

2000	17522.96	333	22024.619	 74

1991	19737.49	129	       NA	 NA

1992	23442.23	107	       NA	 NA

1993	23647.43	118	       NA	 NA

1994	22099.51	126	 5147.516	175

1995	22952.51	154	 6280.350	208

## Positive assortative matching

Assume $\mathcal{X}=\mathcal{Y}=\mathbb{R}$. This implies that both workers and firms are characterized by scalar characteristics $x$ and $y$. Restrictive, but most of the applied literature in Economics to this day has
focused on this case, which already generates interesting economic insights.

Assume that there is the same number of workers and firms, and this number is normalized to one. Let $P$ be the probability distribution of the workers types $x$, and $Q$ the probability distribution of the firms types $y$.

E.g. the literature on CEO compensation studies the matching problem of firms and managers: each manager has\ a measure of talent $x\in\mathbb{R}_{+}$ (extra return generated), and each firm has market capitalization $y\in\mathbb{R}_{+}$. Then, the economic value generated by a manager of talent $x$ running a firm of size $y$ is $\Phi\left(  x,y\right)  =xy$. Assume manager of talent $x$ is assigned to firm of size $y=T\left(x\right)  $. The constraint on the assignment map $T$ is that $T\#P=Q$, which means that each firm is run by a manager and each manager runs a firm. Then the total value created is

\begin{align*}
\mathbb{E}_{P}\left[  \Phi\left(  X,T\left(  X\right)  \right)  \right]
=\mathbb{E}\left[  XT\left(  X\right)  \right]  .
\end{align*}

Under a natural assuption on $\Phi$, called \emph{supermodularity}, we will see that the optimal coupling is such that $Y=T\left(  X\right)  $, where $T$ is nondecreasing. $T$ will be given an explicit characterization.

## Copulas

Recall that given a probability distribution $P$ on the real line, one can define the \emph{quantile} of that distribution, denoted $F_{P}^{-1}$, which is a map from $\left[  0,1\right]  $ to $R$, nondecreasing and continuous from the right, and such that if $U\sim\mathcal{U}:=\mathcal{U} \left(\left[0,1\right]\right)$, then

\begin{align*}
F_{P}^{-1}\left(  U\right)  \sim P.
\end{align*}

$F_{P}^{-1}$ is the generalized inverse of the c.d.f. of $P$, $F_{P}$, the proper inverse when $P$ has a positive density, in which case $U=F_{P}\left(X\right)  $. Note also that if $X\sim P$, then $F_{P}\left(  X\right)  $ has a uniform distribution if and only if $P$ has no mass point.

[Quantile representation](#quantileRep) extends to the case of bivariate distributions: for $\pi\in\mathcal{M}\left(  P,Q\right)  $, there exists a pair $\left(  U,V\right)  $ of uniform random variables such that

\begin{align*}
\left(  F_{P}^{-1}\left(  U\right)  ,F_{Q}^{-1}\left(  V\right)  \right)
\sim\pi,
\end{align*}

and the c.d.f. associated with the distribution of $\left(  U,V\right)  $ is called the *copula* associated with distribution $\pi$.

## Comonotonicity

A pair of random variables $\left(  X,Y\right)$ is *comonotone* if there is $U\sim\mathcal{U}$ such that $X=F_{P}^{-1}\left(  U\right)$ and $Y=F_{Q}^{-1}\left(  U\right)  $. Equivalently, $X$ and $Y$ are said to exhibit *Positive Assortative Matching (PAM)*.

The copula associated with a pair of comonotone random variables is the c.d.f. associated with $\left(  U,U\right)  $, which is $F\left(  u,v\right) =\min\left(  u,v\right)  $. This copula is called the *upper Fréchet-Hoeffding copula*.

Note that when the cdf of $X$ is continuous, there is a much simpler equivalent statement of comonotonicity:

---
**Lemma**
If the distribution of $X$ has no mass points, then $X$ and $Y$ are comonotone if and only if there exists a nondecreasing map $T$ such that $Y=T\left(  X\right)  $. Moreover, one can choose $T\left(x\right)  =F_{Q}^{-1}\left(  F_{P}\left(  x\right)  \right).$

---

---
**Proof**

Consider $U\sim\mathcal{U}$ such that $X=F_{P}^{-1}\left(  U\right)  $ and $Y=F_{Q}^{-1}\left(  U\right)  $. If the distribution of $X$ has no mass point, then $U=F_{P}\left(  X\right)  $. Hence, $Y=F_{Q}^{-1}\left(F_{P}\left(  X\right)  \right)  $.

---

### Supermodular surplus

Assume $\Phi$ is \emph{supermodular}, that is, for every scalars $x$, $x^{\prime}$, $y$ and $y^{\prime}$,

\begin{align*}
\Phi\left(  x\vee x^{\prime},y\vee y^{\prime}\right)  +\Phi\left(  x\wedge
x^{\prime},y\wedge y^{\prime}\right)  \geq\Phi\left(  x,y\right)  +\Phi\left(
x^{\prime},y^{\prime}\right)  , \label{supermodularPhi}%
\end{align*}

where $x\vee x^{\prime}$ and $x\wedge x^{\prime}$ denote respectively the maximum and the minimum between scalars $x$ and $x^{\prime}$. When $\Phi$ is twice continuously differentiable (which we will assume from now on), this is equivalent to

\begin{align*}
\frac{\partial^{2}\Phi\left(  x,y\right)  }{\partial x\partial y}\geq0.
\label{SupermodDiff}%
\end{align*}

Assume that there are two types of workers $\mathcal{X=}\left\{\underline{x},\overline{x}\right\}  $ and firms $\mathcal{Y=}\left\{  \underline{y},\overline{y}\right\}  $. An equivalent restatement of the [supermodular condition](#supermodularPhi) is then

\begin{align*}
\overline{x}\geq \underline{x} \text{ and }\overline{y}\geq \underline{y} \text{ implies }\Phi\left(
\overline{x},\overline{y}\right)  +\Phi\left(  \underline{x},\underline{y}\right)
\geq\Phi\left(  \overline{x},\underline{y}\right)  +\Phi\left(  \underline{x},\overline
{y}\right)  \label{PAMequiv}%
\end{align*}

which asserts that the total output created is higher if the high types match together and the low types match together (assortative matching) rather than if mixed high/low pairs are formed.

The following examples of surplus functions are supermodular:

1. Cobb-Douglas function: $\Phi\left(  x,y\right)  =x^{a}y^{b}$ ($x,y\geq 0$), with $a,b\geq 0$,

2. General multiplicative form: $\Phi\left(  x,y\right)  =\zeta\left(x\right)  \xi\left(  y\right)  $ with $\zeta$ and $\xi$ nondecreasing,

3. Leontieff: $\Phi\left(  x,y\right)  =\min\left(  x,y\right)$,

4. C.E.S. function: $\Phi\left(  x,y\right)  =\left(  x^{-\rho}+y^{-\rho}\right)  ^{-1/\rho}$, $\rho\geq 0$,

5. $\Phi\left(  x,y\right)  =\phi\left(  x-y\right)  $ where $\phi$ is concave; in particular, $\Phi\left(  x,y\right)  =-\left\vert x-y\right\vert ^{p}$, $p\geq1$ or $\Phi\left(  x,y\right)  =-\left(  x-y-k\right)  ^{+}$,

6. $\Phi\left(  x,y\right)  =\phi\left(  x+y\right)  $, where $\phi$ convex.

## Rearrangement theorem

---
**Theorem**

1. Assume that $\Phi$ is supermodular. Then the primal of the Monge-Kantorovich problem

    \begin{align*}
    \sup_{\pi\in\mathcal{M}\left(  P,Q\right)  }\mathbb{E}_{\pi}\left[
    \Phi\left(  X,Y\right)  \right]
    \end{align*}

    has a comonotone solution.

2. Conversely, if [MK primal](#mkpb1d) has a comonotone solution for any choice of probability distributions $P$ and $Q$ on the real line, then $\Phi$ is supermodular.

3. If, in addition, $P$ has no mass points, then there is an optimal assignment which is is pure and satisfies $Y=T\left(  X\right)$ where

    \begin{align*}
    T\left(  x\right)  =F_{Q}^{-1}\circ F_{P}\left(  x\right).
    \end{align*}

---
The proof of part 1. is based on the following lemma.

---

**Lemma**

Let $Z_{1}$ and $Z_{2}$ be two Bernoulli random variables of respective success probability $p_{1}$ and $p_{2}$. Then $\mathbb{E}\left[  Z_{1}Z_{2}\right]  \leq\min\left(  p_{1},p_{2}\right)$.

---
---
**Proof**

As $Z_{2}\leq1$, $\mathbb{E}\left[  Z_{1}Z_{2}\right]  \leq\mathbb{E}\left[Z_{1}\right]  =p_{1}$. Similarly $\mathbb{E}\left[  Z_{1}Z_{2}\right]\leq\mathbb{E}\left[  Z_{2}\right]  =p_{2}$. Thus, $\mathbb{E}\left[Z_{1}Z_{2}\right]  \leq\min\left(  p_{1},p_{2}\right)$.

---

We are now ready to sketch the proof of the [Rearrangement Theorem](#thm:MKsupermod).

---
**Proof**

1. Take $U\sim\mathcal{U}$,and $X=F_{P}^{-1}\left(  U\right)  $ and $Y=F_{Q}^{-1}\left(  U\right)$. By the [quantile representation](#quantileRep), $X\sim P$ and $Y\sim Q$ and $\left(  X,Y\right)$ is comonotone by definition. The proof is in three steps.

    **Step 1.** For $a,b\in\mathbb{R}$, consider surplus function $\phi_{ab}\left(x,y\right)  :=1\left\{  x\geq a\right\}  1\left\{  y\geq b\right\}$, and let $Z_{1}=1\left\{  X\geq a\right\}  $ and $Z_{2}=1\left\{  Y\geq b\right\}$. $Z_{1}$ and $Z_{2}$ are two Bernoulli random variables of respective success probability $p_{1}=1-F_{P}\left(  a\right)  $ and $p_{2}=1-F_{Q}\left(b\right)  $, thus $\mathbb{E}\left[  Z_{1}Z_{2}\right]  \leq\min\left(p_{1},p_{2}\right)  $, but a straightforward calculation shows that the inequality actually holds as an equality. Hence $\left(  X,Y\right)$, which is comonotone, is optimal for each surplus function $\phi_{ab}$.

    **Step 2.** Assume $\mathcal{X}=\left[\underline{x},\overline{x}\right]$ and $\mathcal{Y} = \left[ \underline{y}, \overline{y}\right]$ are compact intervals. Then

    \begin{align*}
F\left(  x,y\right)  =\frac{\Phi\left(  x,y\right)  -\Phi\left(  \underline{x},y\right)  -\Phi\left(  x,\underline{y}\right)  +\Phi\left(  \underline{x},\underline{y}\right)  }{\Phi\left(  \overline{x},\overline{y}\right)  -\Phi\left(\underline{x},\overline{y}\right)  -\Phi\left(  \overline{x},\underline{y}\right)+\Phi\left(  \underline{x},\underline{y}\right)  }
    \end{align*}

    is a c.d.f. associated to a probability measure $\zeta$, and hence

    \begin{align*}
F\left(x,y\right) = \int\phi_{ab}\left(  x,y\right)  d\zeta\left(  a,b\right)$.
    \end{align*}

    As a result, if $\pi\in\mathcal{M}\left(  p,q\right)$ is the distribution of $\left(  X,Y\right)  $ where $X$ and $Y$ are comonotone, then

    \begin{align*}
\int F\left(  x,y\right)  d\pi\left(  x,y\right)  \geq\int F\left(
x,y\right)  d\tilde{\pi}\left(  x,y\right)
    \end{align*}

    for every $\tilde{\pi}\in\mathcal{M}\left(  p,q\right)  $. But as $F$ is of the form $F\left(  x,y\right)  =K\Phi\left(  x,y\right)  +f\left(  x\right) +g\left(  y\right)  +c$ with $K>0$, and because $\int\left\{  f\left(x\right)  +g\left(  y\right)  +c\right\}  d\pi\left(  x,y\right)=\int\left\{  f\left(  x\right)  +g\left(  y\right)  +c\right\}  d\tilde{\pi}\left(  x,y\right)  $ for every $\tilde{\pi}\in\mathcal{M}\left(  p,q\right) $, it results that

    \begin{align*}
\int\Phi\left(  x,y\right)  d\pi\left(  x,y\right)  \geq\int\Phi\left(x,y\right)  d\tilde{\pi}\left(  x,y\right)  ~\forall\tilde{\pi}\in \mathcal{M}\left(  p,q\right)
    \end{align*}
    which completes step 2.

    **Step 3.** When $\mathcal{X}$ and $\mathcal{Y}$ are the real line, the result still holds by approximation.

2. The converse follows by taking for $P$ the discrete probability with two mass points \b{x} and $\overline{x}$ with probability 1/2 each, and $Q$ the discrete probability with two mass points \b{y} and $\overline{y}$ also each with probability 1/2. Then if [MK primal](#mkpb1d) has a solution such that $F_{P}^{-1}\left(  U\right)  $ and $Y=F_{Q}^{-1}\left(  U\right)  $, for $U\sim\mathcal{U}\left(  \left[  0,1\right]  \right)  $, it follows that [condition](#PAMequiv) holds.

3. follows from (i) and [Lemma](#lem:comonotone).

---

Note that the assumptions made in the [Rearrangement Theorem](#thm:MKsupermod) do not guarantee that all the optimal assignments are comonotone. Indeed, the trivial example where $\Phi\left(  x,y\right)  =0$ for every $x$ and $y$ provides an example of supermodular surplus function, for which any assignment is optimal. For this reason, we provide a strengthening of the previous result, which ensures uniqueness. We will assume $\Phi$ is strictly supermodular, that is if both $\overline{x}>\underline{x}$ and $\overline{y}>\underline{y}$ hold, then $\Phi\left(\overline{x},\overline{y}\right)  +\Phi\left(\underline{x},\underline{y}\right)  >\Phi\left(\overline{x},\underline{y}\right)  +\Phi\left( \underline{x},\overline{y}\right)$.

---
**Theorem**

Assume that $\Phi$ is strictly supermodular, and $P$ has no mass point. Then the primal [Monge-Kantorovich problem](#mkpb1d) has a unique optimal assignment, and this assignment is characterized by $Y=T\left(  X\right)  $ where $T$ is given by [this](#mongeMap1d).

---

### The wage equation

Assume $\left(u,v\right)$ is a solution to the dual of the Monge-Kantorovich problem

\begin{align}
\inf \,&  \mathbb{E}_{P}\left[  u\left(  X\right)  \right]  +\mathbb{E}_{Q}\left[  v\left(  Y\right)  \right] \\
s.t.~  &  u\left(  x\right)  +v\left(  y\right)  \geq\Phi\left(  x,y\right)
\end{align}

Then $v\left(y\right)$ is interpreted as the value of the problem of a firm of type $y$, choosing the optimal worker $x$. Then the firm's program is

\begin{align*}
v\left(  y\right)  =\max_{x}\left\{  \Phi\left(  x,y\right)  -u\left(
x\right)  \right\}
\end{align*}

thus by first order conditions, one is led to the *wage equation*

\begin{align*}
u^{\prime}\left(  x\right)  =\frac{\partial\Phi}{\partial x}\left(  x,T\left(
x\right)  \right)  , \label{WageEq}%
\end{align*}
where $T$ is given by [this](#mongeMap1d).

---
**Theorem**

1. Assume $\Phi$ is supermodular and continuously differentiable with respect to its first variable. Assume $P$ has no mass point. Then the [dual Monge-Kantorovich problem](#mkdual1d) has a solution $\left(u,v\right)$. Further, $u$ solves the [wage equation](#WageEq). Hence, $u$ is determined up to a constant $c$ by

    \begin{align*}
    u\left(  x\right)  =c+\int_{x_{0}}^{x}\frac{\partial\Phi}{\partial x}\left(t,T\left(  t\right)  \right)  dt.
    \end{align*}

2. Assume further that $Q$ has no mass point, and that $\Phi$ is also continuously differentiable with respect to its second variable. Then $v$ is given by

    \begin{align*}
    v\left(  y\right)  =c^{\prime}+\int_{T\left(  x_{0}\right)  }^{y}\frac{\partial\Phi}{\partial y}\left(  T^{-1}\left(  z\right)  ,z\right)  dz,
    \end{align*}

    where $c$ and $c^{\prime}$ are related by $c+c^{\prime}=\Phi\left(x_{0},T\left(  x_{0}\right)  \right).$

---

### Interpretation: imperfect competition and rents

It is important to distinguish between $\partial\Phi\left(  x,T\left(x\right)  \right)  /\partial x$, which is the partial derivative of $\Phi\left(  x,y\right)  $ applied at $\left(  x,y\right)  =\left(  x,T\left(x\right)  \right)  $, and $d\Phi\left(  x,T\left(  x\right)  \right)  /dx$,which is the total derivative of $\Phi\left(  x,T\left(  x\right)  \right)$ with respect to $x$. One has

\begin{align*}
\frac{d\Phi\left(  x,T\left(  x\right)  \right)  }{dx}=\frac{\partial \Phi\left(  x,T\left(  x\right)  \right)  }{\partial x}+\frac{\partial \Phi\left(  x,T\left(  x\right)  \right)  }{\partial y}T^{\prime}\left(x\right)
\end{align*}

This decomposition has an interesting interpretation in terms of differential rent. The total derivative $d\Phi\left(  x,T\left(  x\right) \right)  /dx$ is the marginal increase in value between a firm run by a manager of talent $x$ and a firm run by a manager of talent $x+dx$. This differential value is split between the manager's differential rent ($\partial\Phi\left(  x,T\left(  x\right)  \right)  /\partial x=u^{\prime}\left(  x\right)  $) and the firm's differential rent ( $\left(  \partial \Phi\left(  x,T\left(  x\right)  \right)  /\partial y\right)  T^{\prime }\left(  x\right)  =dv\left(  T\left(  x\right)  \right)  /dx$).

This discussion highlights the nature of the assignment model, which is a model of imperfect competition. In this model, managers compete against each other, and likewise on the other side of the market. Managers are imperfect substitutes for each other. This imperfect competition is the source of the rent, as made apparent by [wage equation formula](#WageEq).

## Application

Gabaix and Landier (2008) assume that

\begin{align*}
\Phi\left(  x,y\right)  =Cxy^{\gamma}
\end{align*}

where $C$ and $\gamma$ are parameters.

If $X$ is the talent of the CEO, then the distribution of $X$ can be parameterized through its quantile by

\begin{align*}
Q_{X}\left(  t\right)  =X^{\max}-\frac{B}{\beta N}\left(  N\left(  1-t\right) \right)  ^{\beta}
\end{align*}

(such a distribution is consistent with extreme value theory). $N$ is the number of firms and $\beta$ is a parameter.

If $Y$ is the size of the firm, then its distribution is assumed to be Pareto with distribution $1/\alpha$, i.e.

\begin{align*}
Q_{Y}\left(  t\right)  =A\left(  N\left(  1-t\right)  \right)  ^{-\alpha}%
\end{align*}

(they find that $\alpha=1$, i.e. Zipf's law, fits the data quite well).

The wage of CEO indexed by $t\in\left[  0,1\right]  $ is $w\left(t\right)  $; firm $y=Q_{Y}\left(  t\right)  $'s problem is

\begin{align*}
\max_{t\in\left[  0,1\right]  }\left\{  CQ_{X}\left(  t\right)  y^{\gamma}-w\left(  t\right)  \right\}  =\max_{t\in\left[  0,1\right]  }\left\{-C\frac{B}{\beta N}\left(  N\left(  1-t\right)  \right)  ^{\beta}y^{\gamma}-w\left(  t\right)  \right\}
\end{align*}

and by FOC:

\begin{align*}
w^{\prime}\left(  t\right)   &  =BCy^{\gamma}\left(  N\left(  1-t\right)
\right)  ^{\beta-1}\\
&  =A^{\gamma}BC\left(  N\left(  1-t\right)  \right)  ^{-\alpha\gamma+\beta-1}%
\end{align*}

where it is assumed $\alpha\gamma>\beta$. When $N$ is large, this yields

\begin{align*}
w\left(  t\right)  \approx\frac{A^{\gamma}BC}{\alpha\gamma-\beta}\left(N\left(  1-t\right)  \right)  ^{-\alpha\gamma+\beta}
\end{align*}

"Superstar" effect (Rosen, 1981): if $\beta>0$, then the wages are unbounded, even though talent is bounded -- talent is very valuable at the top end of the distribution.

We shall use Gabaix-Landier's calibrated values
\begin{align*}
\beta\simeq2/3,\gamma\simeq1,BC=2.8\ast10^{-6}
\end{align*}

```R
beta = 2/3
B = 1
C = 2.8e-06
alpha = gamma = 1
X_max = 0
```

We will focus on data from year 2003.

```R
year = 2003
data2003 = data[data[, 1] == year, , drop = FALSE]
```

We will normalize compensation to insure compensation and mktcap are in the same order of magnitude. We need to get rid of the `NA`'s in the data (separately for compensation and market cap) and sort in decreasing order.

```R
comp = data2003[, 4]/1000  # insures same order of magnitude
comp = comp[which(!is.na(comp))]
comp = comp[order(comp, decreasing = T)]

mktcap = data2003[, 2]
mktcap = mktcap[which(!is.na(mktcap))]
mktcap = mktcap[order(mktcap, decreasing = T)]
```

```R
N = length(mktcap)
n = 1:N
```

So in order compute the wage function, it remains to find the value of $A$. This will simply be a regression of $\log(S)$ on $n$. Regress firm rank against mktcap:

```R
ols = lm(log(mktcap) ~ log(n))
summary(ols)
```

    Call:
    lm(formula = log(mktcap) ~ log(n))

    Residuals:
         Min       1Q   Median       3Q      Max
    -1.41075 -0.05161  0.03054  0.06535  0.14352

    Coefficients:
                 Estimate Std. Error t value Pr(>|t|)
    (Intercept) 15.514000   0.027602   562.1   <2e-16 ***
    log(n)      -0.982167   0.005195  -189.0   <2e-16 ***
    ---
    Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

    Residual standard error: 0.1133 on 498 degrees of freedom
    Multiple R-squared:  0.9863,	Adjusted R-squared:  0.9862
    F-statistic: 3.574e+04 on 1 and 498 DF,  p-value: < 2.2e-16

```R
A = unname(exp(ols$coefficients[1]))
```

```R
wage = function(x) {
    return((A^gamma * B * C)/(alpha * gamma - beta) * x^(-alpha * gamma + beta))
}
```

We can compute the theoretical wage function and plot it against the realized wage function.

```R
ols$coefficients[1]
```

(Intercept): 15.5140000823903

```R
# Wage estimation

W = mapply(wage, 1:N)

plot(W, type = "l", xlab = "Rank")
points(1:N, comp, pch = 3, col = "blue")
```

```R

```