```python
 # Check that have our correct Kernel running
import sys
print(sys.executable)
print(sys.version)
print(sys.version_info)
```

    /opt/conda/envs/python/bin/python
    3.8.3 (default, Jul  2 2020, 16:21:59)
    [GCC 7.3.0]
    sys.version_info(major=3, minor=8, micro=3, releaselevel='final', serial=0)

# Predicting Solids in Rivers
## Background
Equations used to compute bed load transport in rivers and streams are based upon regression analysis of data collected for variables related to solids load calculations.
However, these methods have been found to be difficult to use, too complex for practical use, and/or inadequate in terms of estimation precision.
\ref developed and deployed a database-search engine used to find approximate solutions of solids load transport based on model-specific input data.
The tool is available via a web interface in an attempt to simplify and streamline sediment transport estimation.
The search processing was all server-side, operating on a single, central database.

Here we will explore the underlying database and replicate some parts of the original work.
###

## Concept of Distance in N-Dimensional Space
The concept of distance is vital to the search engine. In the screening tool the search input values, S, Q, U, and D50 are compared to their commensurate values in the database, and a distance is computed from the search values to values in the database. The nearest values in N-dimensional distance are selected (actually by a sort) and then used for the estimation of unit solids discharge for the search values. The search engine has several different kinds of distances that the engineer may select.

### Minkowski Distance
The distance between the search values and a database record is computed using  (Tan et al., 2008).

\begin{equation}
L_p= (|x_{1,data} - x_{1,search}|^p + |x_{2,data} - x_{2,search}|^p + \dots + |x_{N,data} - x_{N,search}|^p)^{\frac{1}{p}}
\end{equation}

`p` represents a parameter that modifies the Minkowski distance based upon its magnitude.
When p > 0 and integer produces a quantity known as the Minkowski distance.
The Euclidean distance between the two vectors xdata and xsearch, which is the hypotenuse-type distance that engineers are readily familiar with, is the special case of Equation 9 when p = 2, and the distance itself is called the L2-norm.
In many situations, the Euclidean distance is insufficient for capturing the actual distances in a given high-dimensional space if traverse of that space along a hypotenuse is infeasible.
For example, taxi drivers in Manhattan should measure distance not in terms of the length of the straight line to their destination, but in terms of the Manhattan (taxi distance) distance, which takes into account that streets are either orthogonal or parallel to each other.
The taxi distance is also called the L1 norm and is the special case of Equation 9 when p=1.
This distance measures the shortest path along Cartesian axes (like city streets).

When some elements are unknown (as may be the case in our searches) or the noise in the elements is substantial, the Euclidean distance is not the most appropriate measure of distance, hence the value of p is left as a variable (Erickson,2010).
## Data Value Standardization
The variables in the database are not expressed in the same magnitude, range, and scale.
For example, discharge values are several orders of magnitude larger in the database than median grain diameter, hence the two are not directly comparable when computing a distance for the search algorithm.
In such a case, one way to facilitate direct interpretation for comparing composite indices of the original data having different magnitudes and unit systems is to use normalization.
Normalization serves the purpose of bringing the indicators into the same unit scale or unit base and makes distance computations appropriate.
Normalizing data is done using various standardization techniques to assign a value to each variable so that they may be directly compared without unintentional bias due to differences in unit scale.

### Z-score Standardization
Z-score standardization is a commonly used normalization method that converts all indicators to a common scale with an average of zero and standard deviation of one. This transformation is the same as computing a standard-normal score for each data value.
The average of zero avoids the introduction of aggregation distortions stemming from differences in indicators’ means.
The scaling factor is the standard deviation of the indicator across, for instance, the velocities, slopes or unit solids discharges being ranked.
Thus, an indicator with extreme values will have intrinsically a greater effect on the composite indicator.
The raw score on each data entry is converted to a Z-score, then distances are calculated using the Z-scores for each variable rather than the raw value.
Upon completion of the distance calculations and selection of the nearest neighbors, the results are transformed back into the raw values for subsequent presentation.
Equation 10 shows the basic z-score formula used to normalize data sets (TIBCO).
$z = \frac{(x − μ)}{σ}$
 33
Equation 10

### Unit-Interval [0,1] Standardization
An alternate approach considered for the screening algorithm is an option to use a mapping of each variable in the database to a [0,1] scale and linearly weight within the scale.
This standardization has the same goal as Z-score, which is to prevent one variable from overwhelming the distance computations because of its relative magnitude.
The unit interval [0,1] standardization technique differs from the Z-score in that the variability is governed by the minimum and maximum value for each variable, and hence extrapolation is not feasible.
Because extrapolation is likely necessary until new records are added to the database, this standardization method is not appropriate.

### Unstandardized
The unstandardized approach is not apprpriate because discharge and/or velocity completely dominate any search algorithm, almost to the exclusion of the other variables.
The option was useful for method testing and database error detection but is not useful for production application.

```python

```

```python

```

```python

```

### Download Current Database
Download using http:get method to access the public database from the source URL.

```python
import requests # Module to process http/https requests
remote_url="http://54.243.252.9/engr-1330-webroot/9-MyJupyterNotebooks/43-SolidsInRivers/solids_in_rivers.csv"  # set the url
rget = requests.get(remote_url, allow_redirects=True, verify=False)  # get the remote resource, follow imbedded links, ignore the certificate
open('solids_in_rivers.csv','wb').write(rget.content) # extract from the remote the contents, assign to a local file same name
import pandas as pd # Module to process dataframes (not absolutely needed but somewhat easier than using primatives, and gives graphing tools)
```

```python

```

```python
riverdb = pd.read_csv("solids_in_rivers.csv")
#riverdb.head()
#print(riverdb["D50_m"])
riverdb.describe()
```

      ID
      Q_m3_s
      q_m2_s
      U_m_s
      W_m
      H_m
      R_m
      S_m_m
      D16_m
      D50_m
      ...
      Record_Number
      Froude
      GammaS_N_m3
      GammaF_N_m3
      Tau0_kg_m_s2
      TauStar
      Ustar_m_s
      ManningN
      WP_m
      A_m2

      count
      12081.000000
      12401.000000
      12401.000000
      11270.000000
      12401.000000
      11704.000000
      9675.000000
      12289.000000
      7359.000000
      12401.000000
      ...
      12401.000000
      11269.000000
      12401.000000
      12401.000000
      9563.000000
      9563.000000
      9563.000000
      9270.000000
      9382.000000
      11270.000000

      mean
      1940.571724
      195.722908
      0.967272
      0.891899
      30.219469
      0.738820
      0.606682
      0.011017
      0.019633
      0.036385
      ...
      6380.274897
      0.536185
      25629.979244
      9805.728190
      21.922912
      0.210218
      0.114883
      0.039846
      29.447613
      158.835167

      std
      2949.308625
      1294.763761
      2.747765
      0.545093
      94.382748
      1.807773
      1.543387
      0.018917
      0.021628
      0.048645
      ...
      3696.599235
      0.369873
      2214.983569
      128.285394
      31.256190
      0.375793
      0.093251
      0.042504
      103.538520
      992.809279

      min
      1.000000
      0.000500
      0.001094
      0.047000
      0.076200
      0.000000
      0.007562
      0.000002
      0.000066
      0.000011
      ...
      1.000000
      0.026609
      10100.800000
      9745.510000
      0.027512
      0.005681
      0.005246
      0.001850
      0.121153
      0.001200

      25%
      37.000000
      0.041300
      0.071941
      0.490000
      0.710000
      0.124400
      0.070938
      0.001240
      0.002000
      0.000640
      ...
      3101.000000
      0.319346
      25987.600000
      9794.920000
      1.310125
      0.036348
      0.036224
      0.014014
      0.800300
      0.065400

      50%
      114.000000
      0.699000
      0.197778
      0.760000
      4.880000
      0.270000
      0.153412
      0.004000
      0.014000
      0.011700
      ...
      6487.000000
      0.446229
      25987.600000
      9803.980000
      6.218710
      0.067029
      0.078964
      0.021891
      1.676300
      0.763333

      75%
      3554.000000
      8.080000
      0.643761
      1.152320
      13.140000
      0.590000
      0.450000
      0.013000
      0.033000
      0.063000
      ...
      9587.000000
      0.637954
      25987.600000
      9803.980000
      33.615450
      0.222178
      0.183112
      0.048671
      13.457700
      7.174412

      max
      9825.000000
      28822.300000
      101.618000
      4.100000
      1109.420000
      77.000000
      16.246300
      0.200000
      0.098000
      0.220000
      ...
      12687.000000
      5.687350
      41384.100000
      13366.500000
      207.570000
      7.761280
      0.455660
      0.372269
      1142.270000
      18225.400000

8 rows × 27 columns

```python
riverdb.plot.scatter(x='D16_m',y='D50_m')
```

```python
riverdb.plot.scatter(x='R_m',y='q_m2_s')
```

```python

```

```python

```

```python
## Now delete the local copy
! rm solids_in_rivers.csv
```

```python
import math
def phi_value(diameter):
    # diameter in meters
    diameter = diameter*1000.0 # convert to millimeters
#    print(diameter)
    phi_value = 1.0*math.log2(diameter)
    return phi_value
def d_value(phi_val):
    # phi_val is log2(diam), return diam
    d_value = 2.0**phi_val
    return d_value
```

```python
mu = phi_value(0.00032004)
sigma = math.sqrt(1.22)/math.sqrt(2.)
myguess = mu
print(myguess)
print(d_value(myguess))

from scipy.optimize import newton

def f(x):
    global mu,sigma
    quantile = 0.5
    argument = (x - mu)/(math.sqrt(2.0)*sigma)
    normdist = (1.0 + math.erf(argument))/2.0
    return normdist - quantile

phi50 = newton(f, myguess)

def f(x):
    global mu,sigma
    quantile = 0.05
    argument = (x - mu)/(math.sqrt(2.0)*sigma)
    normdist = (1.0 + math.erf(argument))/2.0
    return normdist - quantile

phi05 = newton(f, myguess)

def f(x):
    global mu,sigma
    quantile = 0.16
    argument = (x - mu)/(math.sqrt(2.0)*sigma)
    normdist = (1.0 + math.erf(argument))/2.0
    return normdist - quantile

phi16 = newton(f, myguess)

def f(x):
    global mu,sigma
    quantile = 0.84
    argument = (x - mu)/(math.sqrt(2.0)*sigma)
    normdist = (1.0 + math.erf(argument))/2.0
    return normdist - quantile

phi84 = newton(f, myguess)

def f(x):
    global mu,sigma
    quantile = 0.90
    argument = (x - mu)/(math.sqrt(2.0)*sigma)
    normdist = (1.0 + math.erf(argument))/2.0
    return normdist - quantile

phi90 = newton(f, myguess)

print('d05 = ',d_value(phi05)/1000,'  millimeters')
print('d16 = ',d_value(phi16)/1000,'  millimeters')
print('d50 = ',d_value(phi50)/1000,'  millimeters')
print('d84 = ',d_value(phi84)/1000,'  millimeters')
print('d90 = ',d_value(phi90)/1000,'  millimeters')
```

    -1.6436758641647295
    0.32004
    d05 =  0.00013136495766569197   millimeters
    d16 =  0.00018680794242288665   millimeters
    d50 =  0.00032004   millimeters
    d84 =  0.0005482936125281758   millimeters
    d90 =  0.000640489979928618   millimeters

```python
	D16_m	D50_m	D84_m	D90_m
7185	0.043	0.126	0.28	0.339
```

```python
import math

def normdensity(x,mu,sigma):
    weight = 1.0 /(sigma * math.sqrt(2.0*math.pi))
    argument = ((x - mu)**2)/(2.0*sigma**2)
    normdensity = weight*math.exp(-1.0*argument)
    return normdensity

def normdist(x,mu,sigma):
    argument = (x - mu)/(math.sqrt(2.0)*sigma)
    normdist = (1.0 + math.erf(argument))/2.0
    return normdist
```

```python
math.sqrt(2)
```

    1.4142135623730951

```python

```


```
from sympy import *
from dolfin import Expression
x = symbols('x[0]')
y = symbols('x[1]')
z = symbols('x[2]')
uu = sin(x)*exp(x+y+z)
uv = sin(y)*exp(x+y+z)
uw = sin(z)*exp(x+y+z)
# uu = (x**4+y**5+z**6)
# uv = (x**4+y**5+z**6)
# uw =(x**4+y**5+z**6)
# u = diff(uw,y)-diff(uv,z)
# v = diff(uu,z)-diff(uw,x)
# w = diff(uv,x)-diff(uu,y)
# p = sin(y)+exp(x+y+z)
u = y**1*z**2
v = x**1*z**2
w = 0
p = x*y
print "u:    ",u
print "\n"
print "v:    ",v
print "\n"
print "w:    ",w
print "\n"
print "p:    ",p
print "\n"
L1 = diff(u,x,x)+diff(u,y,y) + diff(u,z,z)
L2 = diff(v,x,x)+diff(v,y,y) + diff(v,z,z)
L3 = diff(w,x,x)+diff(w,y,y) + diff(w,z,z)
print "Laplacian 1:    ",L1
print "\n"
print "Laplacian 2:    ",L2
print "\n"
print "Laplacian 3:    ",L3
print "\n"
A1 = u*diff(u,x)+v*diff(u,y)+w*diff(u,z)
A2 = u*diff(v,x)+v*diff(v,y)+w*diff(v,z)
A3 = u*diff(w,x)+v*diff(w,y)+w*diff(w,z)

print "Advection 1:    ",A1
print "\n"
print "Advection 2:    ",A2
print "\n"
print "Advection 3:    ",A3
print "\n"
P1 = diff(p,x)
P2 = diff(p,y)
P3 = diff(p,z)
print "Pressure 1:    ",P1
print "\n"
print "Pressure 2:    ",P2
print "\n"
print "Pressure 3:    ",P3

 # (exp(x + y)*sin(y) + exp(x + y)*cos(y))**2 - 2*exp(2*x + 2*y)*sin(y)*cos(y)

```

    u:     x[1]*x[2]**2

    v:     x[0]*x[2]**2

    w:     0

    p:     x[0]*x[1]

    Laplacian 1:     2*x[1]

    Laplacian 2:     2*x[0]

    Laplacian 3:     0

    Advection 1:     x[0]*x[2]**4

    Advection 2:     x[1]*x[2]**4

    Advection 3:     0

    Pressure 1:     x[1]

    Pressure 2:     x[0]

    Pressure 3:     0

```
x = symbols('x')
y = symbols('y')
z = symbols('z')
```

```
u.subs(z,y)
```

    x[1]*x[2]**2

```
u
```

    x[1]*x[2]**2

```
u.nsimplify()

```

    x[1]*x[2]**2

```
ccode(u)
```

    'x[1]*pow(x[2], 2)'

```
from dolfin import Expression
```

```
print u
```

    x[1]*x[2]**2

```
x = symbols('x')
y = symbols('y')
z = symbols('z')
```

```
u.replace(x[0],x)
```

```
u.free_symbols
```

    {x[1], x[2]}

```

```

```
A = Matrix([u,v])
```

```
ccode(A)
```

```
def NS2D(case):
    x = symbols('x[0]')
    y = symbols('x[1]')

    if case == 1:
        u = sin(y)*exp(x)
        v = cos(y)*exp(x)
        p = sin(x)*cos(y)
    if case == 2:
        u = pow(y,3)
        v = pow(x,3)
        p = pow(x,2)
    if case == 3:
        uu = cos(x)*exp(x+y)
        u = diff(uu,y)
        v = -diff(uu,x)
        p = pow(x,3)*sin(y)+exp(x+y)

    L1 = diff(u,x,x)+diff(u,y,y)
    L2 = diff(v,x,x)+diff(v,y,y)

    A1 = u*diff(u,x)+v*diff(u,y)
    A2 = u*diff(v,x)+v*diff(v,y)

    P1 = diff(p,x)
    P2 = diff(p,y)

    u0 = Expression((ccode(u),ccode(v)))
    p0 = Expression(ccode(p))

    Laplacian = Expression((ccode(L1),ccode(L2)))
    Advection = Expression((ccode(A1),ccode(A2)))
    gradPres = Expression((ccode(P1),ccode(P2)))

    print "  Exact Solution:\n"
    print "  u = (",str(u),",",str(v),")\n"
    print "  p = (",str(p),")\n"
    return u0, p0, Laplacian, Advection, gradPres
```

```

```

```

```

```
ThreeD(3)
```

      Exact Solution:

      u = ( exp(x[0])*sin(x[1]) , exp(x[0])*cos(x[1]) )

      p = ( sin(x[0])*cos(x[1]) )

    (Coefficient(VectorElement('Lagrange', None, None, 2, None), 15),
     Coefficient(FiniteElement('Lagrange', None, None, None), 16),
     Coefficient(VectorElement('Lagrange', None, None, 2, None), 17),
     Coefficient(VectorElement('Lagrange', None, None, 2, None), 18),
     Coefficient(VectorElement('Lagrange', None, None, 2, None), 19))

## def M2D(case):     x = symbols('x[0]')     y = symbols('x[1]')      if case == 1:         uu = cos(x)*exp(x+y)         u = diff(uu,y)         v = -diff(uu,x)         p = x*sin(2*pi*y)*sin(2*pi*x)     if case == 2:         u = y*y*(y-1)         v = x*x*(x-1)         p = y*(y-1)*x*(x-1)      L1 = diff(v,x,y) - diff(u,y,y)     L2 = diff(u,x,y) - diff(v,x,x)      P1 = diff(p,x)     P2 = diff(p,y)     print ccode(p)      u0 = Expression((ccode(u),ccode(v)))     p0 = Expression(ccode(p))      CurlCurl = Expression((ccode(L1),ccode(L2)))     gradPres = Expression((ccode(P1),ccode(P2)))      print "  Exact Solution:\n"     print "  u = (",str(u),",",str(v),")\n"     print "  p = (",str(p),")\n"     return u0, p0, CurlCurl, gradPres

```
def M2D(case):
    x = symbols('x[0]')
    y = symbols('x[1]')

    if case == 1:
        uu = cos(x)*exp(x+y)
        u = diff(uu,y)
        v = -diff(uu,x)
        p = x*sin(2*pi*y)*sin(2*pi*x)
    if case == 2:
        u = y*y*(y-1)
        v = x*x*(x-1)
        p = y*(y-1)*x*(x-1)

    L1 = diff(v,x,y) - diff(u,y,y)
    L2 = diff(u,x,y) - diff(v,x,x)

    P1 = diff(p,x)
    P2 = diff(p,y)

    print ccode(p)
    u0 = Expression((ccode(u),ccode(v)))
    p0 = Expression(ccode(p))

    CurlCurl = Expression((ccode(L1),ccode(L2)))
    gradPres = Expression((ccode(P1),ccode(P2)))

    print "  Exact Solution:\n"
    print "  u = (",str(u),",",str(v),")\n"
    print "  p = (",str(p),")\n"
    return u0, p0, CurlCurl, gradPres

```

```
p = x*sin(2*pi*y)*sin(2*pi*x)
```

```
s = ccode(p)
s
```

    'x[0]*sin(2*M_PI*x[0])*sin(2*M_PI*x[1])'

```
s.replace('M_PI','pi')
```

    'x[0]*sin(2*pi*x[0])*sin(2*pi*x[1])'


# Lesson 1 - Introduction to Artificial Neural Networks

This lesson will focus on the history of the challenges that researchers in the field on Artificial Neural Networks (ANNs) have met and solved.
The course attendants will understand the foundations on which ANNs have been (and currently are) developed and will thus better appreciate the majority of state-of-the-art solutions.

### Summary

* [Lexicon](#lexicon)
* [Parametric learning](#parametric_learning)
* [1943: Linear Threshold Unit](#ltu)
* [1949: Hebbian learning](#hebb)
* [1957: Perceptron algorithm](#perceptron)
* [1960: Delta rule](#delta_rule)
* [1989: Universal Approximation Theorem](#uat)
* [1986: Backpropagation algorithm](#backprop)

### Lexicon

In order to allow an easy translation from a *classical* statistical framework to the modern *statistical learning* framework, this notebook provides a brief vocabulary which will be used throughout the course.

The usual statistical inference problem is framed as a search for a mapping

$$f: X \to Y$$

where $X$ is a set representing what we can observe in the reality, while $Y$ represents what we would like to know about the reality, and that we suppose related to what we observe.

To put it simply:
* $X$ represents **what we know**
* $Y$ represents **what we want to know**

$X$ is usually a real vector space which coordinates quantify the relevant and quantifiable properties of the problem's domain.

$Y$ is usually a low-dimensional real vector space (for continuous regression) or a finite set (for logistic regression, AKA classification).

The points in $X$ are usually modeled as random vectors $x = (x_1, x_2, \dots x_n) \in \mathbb{R}^n$ whose components are called *independent variables*, *observables* or *predictors* in classical statistics and **inputs** in statistical learning.

The points in $Y$ are usually modeled as random vectors $y = (y_1, y_2, \dots y_m) \in \mathbb{R}^m$ or by a discrete random variable $y \in {Y_1, Y_2, \dots Y_C}$ which components are called *dependent variables*, *targets* or *responses* in classical statistics and **outputs** in statistical learning.

Many classical statistics techniques do not infer $y$ as a direct response to $x$

$$y = f(x)$$

but rather on a preliminary transformation of $x$:

$$y = f(\phi(x))$$

The transformed points $\phi(x)$ are called *latent variables* or *transformed variables* in classical statistics and **features** in statistical learning, while the map $\phi$ is called a **feature map**.

### Parametric learning

Suppose we observe a set $E = \{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \dots (x^{(N)}, y^{(N)})\} \subset X \times Y$; we call this observations **experience**.

We would like to have a function $f: X \to Y$ that satisfies the observed pairs, but that could also be applied confidently to unseen instances $x \in X$; this last capability to extend $f$ to unseen examples is called **generalization**.

When $Y$ is continuous, both univariate and multivariate, this statistical inference task is referred to as **regression**.
If the seeked releationship $f \subset X \times Y$ is linear (i.e. $y_j = \sum_{i=1}^{n} w_{ji} x_i$) we talk about **linear regression**, otherwise the task is called **non-linear regression**.

When $Y$ is a finite set, the problem is called **classification**.
With a slight abuse, we will refer to classification as **logistic regression**, and in particular we will talk about **binomial logistic regression** when $|Y| = 2$ and of **multinomial logistic regression** when $|Y| = k, k > 2$.

A brief note about the etimology of the terms.
The term *regression* was coined in the 19th century by the English statistician Francis Galton to refer to the phenomenon of *regression towards mediocrity in hereditary stature*, an 1886 study in which he extensively used line fitting techniques, but where the term *regression* indicated more an outcome of the analysis than the techniques themselves!
The term *logistic* was introduced in 1844 by the Belgian statistician Pierre Verhulst in his work *Reserches mathematiques sur la loi d'accroisement de la population*, polemicizing against the exponential models for a population's growth that relied on the unreasonable assumption of unlimited resources; he defined more logical hypothesis and produced growth plot with a sigmoid shape, a function that he named *logistic* because it was the result of a model which used *logical* assumptions.
How was this term transferred to the classification problem? As we will see, this *logistic function* will play a critical role for Artificial Neural Networks in general, and for classification tasks in particular.

The general problem of **machine learning** is a particular case of the more general statistical learning framework, since all the experience available to an agent is in the form of digital data.

Classical statistical inference defines the concept of a *parametric model*, a space of functions

$$\mathcal{F}_{\Theta} = \{f_{\theta}: X \to Y\}_{\theta \in \Theta}$$

parametrized by elements of $\Theta$, into which the closest (in a metric spaces sense) approximation $f_{\theta^*}$ to the real map $f: X \to Y$ is searched for by whatever statistical procedure the investigator prefers.
Since machine learning is a subfield of Artificial Intelligence, in this context models are also called **agents**.

Statistical learning defines the concept of a **risk functional** (also called **loss functional** or **cost functional**)

$$L: \mathcal{F}_{\Theta} \to \mathbb{R}$$

that can evaluate models of the chosen family. This concept is critical, since it allows to *translate* the analytical procedure of functional minimization into a *navigation* of the parameter space $\{\theta_0, \theta_1, \dots \theta^*\} \subset \Theta$ intended to find an optimal solution $\theta^*$.
This process of iteratively *modifying the agent's parameters to minimize the risk functional* is called **learning**.

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.preprocessing import scale

# create dataset
X, Y_hat = make_blobs(n_samples=150, n_features=3, centers=2)
X = scale(X)
Y_hat = 2*Y_hat - 1.0

# plot data
fig1 = plt.figure()
ax = fig1.gca(projection='3d')
ax.scatter(X[:, 0], X[:, 1], X[:, 2], c=Y_hat, cmap='bwr')
plt.show()

```

### 1943: Linear Threshold Unit

Idea: the original [paper](https://pdfs.semanticscholar.org/5272/8a99829792c3272043842455f3a110e841b1.pdf) by McCulloch and Pitts essentially stated that **a threshold neuron with binary output implementing a weighted sum of inputs can emulate first-order logic sentences (NOT/AND/OR functions)**.
This ensures that first-order logic statements as *"this point belongs/does not belong to a set"* can be computed by setting suitable thresholds on a linear combination of values.

Let

$$x = (x_1, x_2, \dots x_n) \in \mathbb{R}^n$$

be an input vector.
A **Linear Threshold Unit (LTU)** models the following:

$$LTU(x) = \begin{cases} 1, & \mbox{if } \sum_{i=1}^{n} w_i x_i \geq b \\ -1, & \mbox{if } \sum_{i=1}^{n} w_i x_i < b \end{cases}$$

where $b$ is a specified *firing threshold* (or simply *threshold*) that specifies the amount of *stress* to be accumulated in the neuron under the form of a linear combination of stimuli before the neuron can *fire* a non-zero signal.
Improperly, this function is often referred to as a *Heaviside-activated linear unit*, but the Heaviside function is defined as

$$H(t) = \begin{cases} 0, & \mbox{if } t < 0 \\ \frac{1}{2}, & \mbox{if } t = 0 \\ 1, & \mbox{if } t > 0 \end{cases}$$

Mathematically, a LTU is able to draw an hyperplane in $\mathbb{R}^n$, as shown by the example below.

```python
class LTU():
    def __init__(self, n_dim):
        # init LTU
        self.weights = np.random.randn(n_dim)
        self.bias = np.random.randn()

    def linear(self, x):
        return np.dot(x, self.weights) + self.bias

    def activate(self, linear):
        return np.where(linear >= 0, 1, -1)

    def predict(self, x):
        return self.activate(self.linear(x))

```

```python
ltu = LTU(X.shape[1])

# generate a meshgrid to compute LTU hyperplane
x = np.arange(-2.0, 2.0, 0.1)
y = np.arange(-2.0, 2.0, 0.1)
xx, yy = np.meshgrid(x, y)

# plot LTU hyperplane onto data
fig2 = plt.figure()
ax21 = fig2.gca(projection='3d')
ax21.scatter(X[:, 0], X[:, 1], X[:, 2], c=Y_hat, cmap='bwr')
zz_ltu = -(np.dot(np.transpose(np.vstack([xx[None, :], yy[None, :]]), (1, 2, 0)), ltu.weights[:-1]) + ltu.bias) / ltu.weights[-1]
ax22 = fig2.gca()
ax22.plot_surface(xx, yy, zz_ltu, alpha=0.2)
plt.show()

```

Problem: the threshold of the MPC neuron needs to be set by hand.

### 1949: Hebbian learning

Idea: in his [book](http://s-f-walker.org.uk/pubsebooks/pdfs/The_Organization_of_Behavior-Donald_O._Hebb.pdf) on neuropsychology, on the basis of experimental observations, Hebb stated that **"when the axon of a cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased."**
The main implication of this statement is that connected neurons should exhibit a self-organization property in order to optimize their communication process.

To shape this idea into the mathematical terms of a LTU, Hebb's goal is to find a constant $\xi > 1$ such that

$$\langle x, w \rangle = y = \langle \frac{x}{\xi}, w + \Delta w \rangle$$

That is, to change the strengths $w$ of the connections in such a way that, after the *metabolic change*, the same activation $y$ can be obtained with an *attenuated version* $\frac{x}{\xi}$ of the inputs.
A simple way to do this is aligning or counteraligning the weights with the input pattern $x$

$$\Delta w = \alpha y x$$

where $\alpha$ is a small positive constant called **learning rate**, which guarantees

$$\xi = 1 + \eta \|x\|^2$$

as desired.

Problem: Hebb did not provide a formal procedure to train an LTU neuron automatically.

### 1957: Perceptron algorithm

Idea: Frank Rosenblatt was the first to realize a working Hebbian model based on a LTU neuron.
The basic representation is This *adaptive artificial neuron*, that was capable of learning autonomously, was called [perceptron](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.335.3398&rep=rep1&type=pdf).

The training algorithm proposed by Rosenblatt iteratively presents patterns $x^{(i)}$ to the LTU, which predicts $LTU(x^{(i)}) = y^{(i)} \in \{-1, 1\}$, but hides the known desired target $\hat{y}^{(i)}$ from it; after each presented pattern, the difference $\delta y^{(i)} = \hat{y}^{(i)} - y^{(i)}$ determines wether the LTU response should have been higher or lower; this **error signal** is finally used to determine which connections to inhibit ($|w + \Delta w| < |w|$) and which to excite ($|w + \Delta w| > |w|$):

$$\Delta w = \alpha \delta y^{(i)} x^{(i)}$$

The main difference with the Hebbian learning rule is that the perceptron learning is actually *anti-Hebbian* in that it tries to force the LTU to close the existing gap with a known pattern (**supervised learning**), while Hebbian learning creates a self-stimulating loop (the goal is the reduction of the intensity of the input).

The perceptron algorithm can also handle **multivariate logistic regression**: supposing we want to predict $\hat{y} = (\hat{y}_1, \hat{y}_2)$, we could use two distinct (and not mutually interconnected) perceptrons $LTU_1(x) = y_1$ and $LTU_2(x) = y_2$ to model the two components of the output.
This ensemble of units is a first example of a **layer** of artificial neurons, which is a foundamental concept for modern Artificial Neural Networks research.

```python
class Perceptron(LTU):
    def __init__(self, n_dim, alpha=0.001, n_epochs=10):
        LTU.__init__(self, n_dim)
        self.bias = 0
        self.alpha = alpha
        self.n_epochs = n_epochs
        # record "heatmaps" of updates
        self.heatmaps = list()

    def train_one_step(self, x, y_hat):
        y = self.predict(x)
        delta = y_hat - y
        delta_w = self.alpha * delta * x
        delta_b = self.alpha * delta
        self.weights += delta_w
        self.bias += delta_b
        # store "heatmap" of current update
        self.heatmaps.append(np.array([x, delta_w]))

    def train(self, X, Y_hat):
        for i_epoch in range(self.n_epochs):
            for (x, y_hat) in zip(X, Y_hat):
                self.train_one_step(x, y_hat)
            print('Epoch {:2d} - Accuracy: {:6.2f}%'.format(i_epoch+1, self.accuracy(X, Y_hat)))

    def accuracy(self, X, Y_hat):
        Y = self.predict(X)
        accuracy = 100 * (np.sum(np.where(Y==Y_hat, 1, 0)) / len(Y))
        return accuracy

```

```python
perceptron = Perceptron(X.shape[1])

# plot initial hyperplane onto data
fig3 = plt.figure()
ax31 = fig3.gca(projection='3d')
ax31.scatter(X[:, 0], X[:, 1], X[:, 2], c=Y_hat, cmap='bwr')
zz_perceptron = -(np.dot(np.transpose(np.vstack([xx[None, :], yy[None, :]]), (1, 2, 0)), perceptron.weights[:-1]) + perceptron.bias) / perceptron.weights[-1]
ax32 = fig3.gca()
ax32.plot_surface(xx, yy, zz_perceptron, alpha=0.2)

perceptron.train(X, Y_hat)

# plot "heatmaps"
heatmaps = np.array(perceptron.heatmaps)
fig4, (ax41, ax42) = plt.subplots(1, 2)
ax41.imshow(heatmaps[:10, 0, :], cmap='hot', vmin=-2.0, vmax=2.0)
ax42.imshow(heatmaps[:10, 1, :], cmap='hot', vmin=-2.0*(perceptron.alpha*2.0), vmax=2.0*(perceptron.alpha*2.0))
plt.show()

# plot trained hyperplane onto data
fig5 = plt.figure()
ax51 = fig5.gca(projection='3d')
ax51.scatter(X[:, 0], X[:, 1], X[:, 2], c=Y_hat, cmap='bwr')
zz_perceptron = -(np.dot(np.transpose(np.vstack([xx[None, :], yy[None, :]]), (1, 2, 0)), perceptron.weights[:-1]) + perceptron.bias) / perceptron.weights[-1]
ax52 = fig5.gca()
ax52.plot_surface(xx, yy, zz_perceptron, alpha=0.2)

plt.show()

```

Problem: the perceptron is constrained to use a step activation function, and is thus constrained to solve classifiction problems.

### 1960: Delta rule
Idea: a [paper](http://www.dtic.mil/dtic/tr/fulltext/u2/241531.pdf) by Bernard Widrow and Tedd Hoff described a way to modify the perceptron algorithm showing that **the step activation function was not a necessary condition for Hebbian learning**.

The two key tools on which the Delta rule relies are the chain rule for a composed function $f(w) = f(\phi(w))$

$$\frac{\partial f}{\partial w} = \frac{\partial f}{\partial \phi} \frac{\partial \phi}{\partial w}$$

and the steepest descent numerical optimization method

$$\Delta w = -\alpha \nabla_w f$$

which allow to replace the error signal $\delta y^{(i)}$ in a suitable manner.
This is where the Delta rule algorithms takes its name.

The Widrow-Hoff neuron, called ADAptive LINE neuron or **Adaline**, differs from a LTU in that its output is computed withouth thresholding the internal neuron score.
This implies that the error signal $\delta y^{(i)}$ is not constrained to the set $\{-2, +2\}$, but can be proportional to the distance from the desired output $\delta y^{(i)} \propto (\hat{y}^{(i)} - y^{(i)})$.
This effect is obtained applying a Least Mean Squres (LMS) loss

$$L(x) = \frac{1}{2} (\hat{y} - y)^2$$

to the Adaline's output $Adaline(x) = y$.
This yields the update:

$$\begin{align} \Delta w_{ij} &= -\alpha \frac{\partial L}{\partial y_j} \frac{\partial y_j}{\partial w_{ij}} \\ &= -\alpha (-(\hat{y}_j - y_j)) x_i \end{align}$$

An advantage of this model is that **it can handle also linear regression**: this depends on the fact that the desired response $\hat{y}$ can be whichever real number, and is not constrained to be a binary variable.

```python
# noisy line dataset
def noisy_line(noise=0.01):
    n_samples = 150
    X = np.random.uniform(low=-1.0, high=1.0, size=(n_samples, 1))
    theta = np.random.uniform(low=-0.2, high=0.2, size=(2, 2))
    Y_hat = theta[0, :] + X * theta[1, :]
    Y_hat += noise * np.random.randn(n_samples, 1)
    return theta, X, Y_hat

theta, X, Y_hat = noisy_line()

# plot data
fig6 = plt.figure()
ax = fig6.gca(projection='3d')
ax.scatter(X, Y_hat[:, 0], Y_hat[:, 1])
plt.show()

```

```python
class Adaline():
    def __init__(self, in_dim, out_dim, alpha=0.01, n_epochs=10):
        self.weights = 0.01 * np.random.randn(in_dim, out_dim)
        self.biases = np.zeros((1, out_dim))
        # training hyperparameters
        self.alpha = alpha
        self.n_epochs = n_epochs
        self.losses = list()

    def preprocess(self, x):
        if x.ndim == 1:
            x = x.reshape((x.shape[0], -1))
        return x

    def linear(self, x):
        return self.biases + np.dot(x, self.weights)

    def activate(self, linear):
        return linear

    def predict(self, x):
        x = self.preprocess(x)
        return self.activate(self.linear(x))

    def train_one_step(self, x, y_hat):
        y = self.predict(x)
        delta_y = y_hat - y
        loss = np.sum(np.square(delta_y)) / 2
        grad_y = -delta_y
        grad_w = x.T * grad_y
        grad_b = grad_y
        delta_w = -self.alpha * grad_w
        delta_b = -self.alpha * grad_b
        self.weights += delta_w
        self.biases += delta_b

    def train(self, X, Y_hat):
        for i_epoch in range(self.n_epochs):
            for (x, y_hat) in zip(X, Y_hat):
                self.train_one_step(x, y_hat)
            loss = self.loss(X, Y_hat)
            self.losses.append(loss)
            print('Epoch {:2d} - Loss: {:6.2f}'.format(i_epoch+1, loss))

    def loss(self, X, Y_hat):
        Y = self.predict(X)
        delta = Y_hat - Y
        loss = np.sum(np.square(delta)) / 2
        return loss

```

```python
adaline = Adaline(X.shape[1], Y_hat.shape[1])

# generate a mesh to compute Adaline predictions
x = np.linspace(-1.0, 1.0)

# plot untrained Adaline predictions onto data
fig7 = plt.figure()
ax71 = fig7.gca(projection='3d')
ax71.scatter(X, Y_hat[:, 0], Y_hat[:, 1])
y = adaline.predict(x)
ax72 = fig7.gca()
ax72.plot(x, y[:, 0], y[:, 1], c='r')

adaline.train(X, Y_hat)

# plot trained Adaline predictions onto data
fig8 = plt.figure()
ax81 = fig8.gca(projection='3d')
ax81.scatter(X, Y_hat[:, 0], Y_hat[:, 1])
y = adaline.predict(x)
ax82 = fig8.gca()
ax82.plot(x, y[:, 0], y[:, 1], c='r')
plt.show()

fig9 = plt.figure()
plt.plot(adaline.losses)
plt.show()

print('Bias (real): ', theta[0, :])
print('Bias (learnt): ', adaline.biases)
print('Weights (real): ', theta[1:, :])
print('Weights (learnt) : ', adaline.weights)

```

Problem: both the perceptron and Adaline are intrinsically uncapable of non-linear discrimination.

Consider the case of a circular dense blob of points surrounded by a circular crown of qualitatively different points.
As humans, we could easily discriminate them in an analytic way by a transformation from a Cartesian coordinate system to a polar one:

$$\begin{align} \rho &: \mathbb{R}^2 \to \mathbb{R} \\ &(x_1, x_2) \mapsto \rho = \sqrt{x_1^2 + x_2^2} \end{align}$$

The polar reference allows for a straightforward classification:

$$f((x_1, x_2)) = \begin{cases} +1 , & \mbox{if } \rho((x_1, x_2)) < \bar{\rho} \\ -1 , & \mbox{if } \rho((x_1, x_2)) \geq \bar{\rho} \end{cases}$$

where $\bar{\rho}$ is a suitable threshold.
The perceptron and Adaline have no hope to solve this problem.

```python
from sklearn.datasets import make_circles

# generate non-linearly separable data
X, Y_hat = make_circles(n_samples=150, noise=0.1, factor=0.2)

# plot non-linearly separable data
fig10 = plt.figure()
ax101 = fig10.gca()
ax101.scatter(X[:, 0], X[:, 1], c=Y_hat, cmap='bwr')
plt.show()

```

### 1989: Universal Approximation Theorem

Idea: George Cybenko proved a [critical result](http://web.eecs.umich.edu/~cscott/smlrg/approx_by_superposition.pdf) about multilayer neural networks with at least one hidden layer.
In particular, he found that **given an arbitrary continuous function $\phi$, a two-layers Artificial Neural Network with sigmoid activation functions and a sufficiently large amount of neurons in the hidden layer is capable of approximating $\phi$ arbitrarily close**.
A few years later, Hornik [generalized the result](http://zmjones.com/static/statistical-learning/hornik-nn-1991.pdf) to other classes of activation functions: sigmoid functions are not a necessary condition to learn non-linearities.

Problem: the missing link is an extension of the perceptron algorithm and the Delta rule to multilayer networks.

### 1986: Backpropagation algorithm

Idea: David Rumelhart and Geoffrey Hinton [extended the Delta rule](https://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf) to compute the inifinitesimal changes to be applied to an arbitrary network's parameters in order to reduce the error at its task.

This method can be used to efficiently train multilayer neural networks.
The possibility to train models with more than just one hidden layer enables the construction of **Deep Neural Networks** (DNN for short), Artificial Neural Networks that try to exploit the UAT not by adding more and more neurons to a single hidden layer, but instead **create a hierarchy of two or more transformations** to ease the process of extracting the desired mapping $f: X \to Y$.

To summarize:
* single-layer models are equivalent to statistical inference that does not use any intermediate feature map;
* two-layers models (i.e. models with just a single hidden layer, also called *shallow models*) are equivalent to statistical inference that uses just a single feature map;
* three-or-more-layers models (also called *deep models*) are equivalent to statistical inference that uses many intermediate feature maps.

As a conclusive note for this lesson, we will give analytical formulas for the backpropagation algorithm.
Suppose we have a $L$-feature maps model

$$f(x) = f(\phi_{L}(\phi_{L-1}((\dots \phi_1(x, w_{(1)})), w_{(L-1)}), w_{(L)})$$

To adjust the arbitrary $k$-th feature parameters, the backpropagation algorithm works in two steps.
The first is using the chain rule to compute the partial derivative of the model loss with respect to the parameters to be adjusted:

$$\frac{\partial L}{\partial w_{(k)}} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial \phi_{L}} \left( \prod_{i=L}^{i=k+1}\frac{\partial \phi_{i}}{\partial \phi_{i-1}} \right) \frac{\partial \phi_{k}}{\partial w_{(k)}}$$

The second step consists in applying whichever numerical optimization algorithm the investigator estimates suitable for the problem; for example, using steepest gradient descent yields an update:

$$\Delta w_{(k)} = -\alpha \frac{\partial L}{\partial w_{(k)}}$$

### References

**History of single-layer ANNs**: this nice [blog post](http://sebastianraschka.com/Articles/2015_singlelayer_neurons.html) by Sebastian Raschka provides additional examples and visualizations.

**Backpropagation**: I definitely suggest to read [Stanford's lesson](http://cs231n.github.io/optimization-2/) for an intuitive explanation (or watch its [Youtube version](https://www.youtube.com/watch?v=59Hbtz7XgjM) for the laziest) and Michael Nielsen's [online book](http://neuralnetworksanddeeplearning.com/) for a better mathematical understanding.


# EFFECTIVE GROWTH

1. Here we analyse the effective growth function for the midpoint and the unequal-time power spectrum, to understand why they are equal at some scale.

2. We also plot the power spectra as a function of redshift at a fixed k to understand when the unequal-time correlations stop being important.

```python
import numpy as np
import pandas
from scipy.interpolate import interp1d
from astropy.cosmology import FlatLambdaCDM
from skypy.power_spectrum import growth_function
```

```python
%matplotlib inline
import matplotlib.pyplot as plt

plt.rcParams.update({
    'text.usetex': False,
    'font.family': 'serif',
    'legend.frameon': False,
    'legend.handlelength': 1.5,
})
```

## 1. Power spectra

```python
import sys
sys.path.append("../unequalpy")
from matter import matter_power_spectrum_1loop as Petc
from matter import matter_unequal_time_power_spectrum as Puetc
from approximation import geometric_approx as Pgeom
from approximation import midpoint_approx as Pmid
```

* Cosmology and normilised growth

```python
cosmo = FlatLambdaCDM(H0=67.11, Ob0=0.049, Om0= 0.2685)
```

```python
def Dz(redshift):
    return growth_function(redshift, cosmo)/growth_function(0, cosmo)
```

* Load FAST-PT loop contributions

```python
d = np.loadtxt('../data/Pfastpt.txt',unpack=True)
k = d[:, 0]
pk, p22, p13 = d[:, 1], d[:, 2], d[:, 3]
p11_int = interp1d( k, pk, fill_value="extrapolate")
p22_int = interp1d( k, p22, fill_value="extrapolate")
p13_int = interp1d( k, p13, fill_value="extrapolate")
powerk = (p11_int, p22_int, p13_int)
```

* Mean redshift and redshift separations

```python
zm = 0.5
dz = 0.1 * np.arange(0, 5)
```

* Relative error

```python
def error(p1, p2, reference):
    return np.abs(p1 - p2) / reference
```

## 2. Effective growth function

* Equal-time version $$D_{eff}(k, z)^2 = \frac{P(k,z)}{P(k, 0)}$$

* Unequal-time version $$D_{eff}(k, z_1, z_2)^2 = \frac{P(k,z_1, z_2)}{P(k, 0)}$$

```python
def Det_eff(wavenumber, z):
    return Petc(wavenumber, Dz(z), powerk) / Petc(wavenumber, 1, powerk)
```

```python
def Dgeom_eff(wavenumber, z1, z2):
    return np.sqrt( Petc(wavenumber, Dz(z1), powerk) * Petc(wavenumber, Dz(z2), powerk) ) / Petc(wavenumber, 1, powerk)
```

```python
def Duet_eff(wavenumber, z1, z2):
    return Puetc(wavenumber, Dz(z1), Dz(z2), powerk) / Petc(wavenumber, 1, powerk)
```

```python
zv = 3*np.logspace(-3,0)
```

```python
dD1 = error(Det_eff(0.1, zv), Duet_eff(0.1, zv - 0.1, zv + 0.1),  Duet_eff(0.1, zv - 0.1, zv + 0.1))
dD2 = error(Det_eff(0.2, zv), Duet_eff(0.2, zv - 0.1, zv + 0.1), Duet_eff(0.2, zv - 0.1, zv + 0.1))

dDg1 = error(Dgeom_eff(0.1, zv - 0.1, zv + 0.1), Duet_eff(0.1, zv - 0.1, zv + 0.1),  Duet_eff(0.1, zv - 0.1, zv + 0.1))
dDg2 = error(Dgeom_eff(0.2, zv - 0.1, zv + 0.1), Duet_eff(0.2, zv - 0.1, zv + 0.1), Duet_eff(0.2, zv - 0.1, zv + 0.1))

dD12 = error(Det_eff(0.1, zv), Duet_eff(0.1, zv - 0.2, zv + 0.2), Duet_eff(0.1, zv - 0.2, zv + 0.2))
dD22 = error(Det_eff(0.2, zv), Duet_eff(0.2, zv - 0.2, zv + 0.2), Duet_eff(0.2, zv - 0.2, zv + 0.2))

dDg12 = error(Dgeom_eff(0.1, zv - 0.2, zv + 0.2), Duet_eff(0.1, zv - 0.2, zv + 0.2), Duet_eff(0.1, zv - 0.2, zv + 0.2))
dDg22 = error(Dgeom_eff(0.2, zv - 0.2, zv + 0.2), Duet_eff(0.2, zv - 0.2, zv + 0.2), Duet_eff(0.2, zv - 0.2, zv + 0.2))
```

```python
plt.figure(figsize=(6, 4))
plt.plot(zv[:-5], dD1[:-5], 'k', label='Mid $k=0.1\; h/Mpc$, $\Delta z = 0.1$')
plt.plot(zv[:-5], dDg1[:-5], 'r', label='Geometric')

plt.plot(zv[:-5], dD2[:-5], '--k', label='Mid $k=0.2\; h/Mpc$', lw=0.25)
plt.plot(zv[:-5], dDg2[:-5], '--r', label='Geometric', lw=0.25)

plt.ylabel('$\Delta D_{eff}/ D_{u}^2$')
plt.xlabel('Mean redshift $z$')

plt.yscale('log')

plt.title("Approximations = unequal-time")
plt.legend(frameon=False)
# plt.savefig('plots/approx.pdf')

plt.show()
```

## 3. Differences between Midpoint and Geometric

#### 3.2. Geometric - Midpoint, toy model

* Power spectrum

$$|P_{geom} - P_{mid}| \approx (\Delta z D`(z_m))^2$$

```python
from scipy.misc import derivative
```

```python
def deff(z, k=0.1):
    return Det_eff(k, z)
def deffu(z, deltaz=0.1, k=0.1):
    return Duet_eff(k, z-deltaz, z+deltaz)
```

```python
Dp = derivative(deff, zv)
Dpu = derivative(deffu, zv)
```

    /Users/c49734lf/miniconda3/lib/python3.7/site-packages/skypy/power_spectrum/_growth.py:112: RuntimeWarning: invalid value encountered in power
      growth_factor = np.power(omega_m_z, gamma)
    /Users/c49734lf/miniconda3/lib/python3.7/site-packages/skypy/power_spectrum/_growth.py:167: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
      the requested tolerance from being achieved.  The error may be
      underestimated.
      integral = integrate.quad(integrand, z, z_upper)[0]

```python
def DeltaP(Deltaz, dp):
    return np.square(Deltaz * dp)
```

```python
def deffk(z):
    return deff(k, z)
```

```python
Dp05 = derivative(deffk, 0.5)
Dpu05 = derivative(deffu, 0.5)
```

```python
ddp = error(DeltaP(0.1, Dp), DeltaP(0.1, Dpu), DeltaP(0.1, Dpu))
pg = np.sqrt( Petc(0.1, Dz(zv -0.1), powerk) * Petc(0.1, Dz(zv + 0.1), powerk) )
pm = Petc(0.1, Dz(zv), powerk)
pg_pm = np.absolute(pg - pm)
```

```python
fig, (ax0, ax1) = plt.subplots(ncols=2, constrained_layout=True, figsize=(10,4))

ax0.loglog(k, DeltaP(0.1, Dp05), 'k', label='$\Delta z = 0.1$')
ax0.loglog(k, DeltaP(0.2, Dp05), 'r', label='$\Delta z = 0.2$')

ax0.set_xlabel('Wavenumber, $k \; (h/Mpc)$')
ax0.set_ylabel('$\Delta z ^2 D_{eff}^{\'}(k, z_m)^2$')
ax0.set_title("Fixed $z_m=0.5$")
ax0.legend(frameon=False)

ax1.plot(zv, DeltaP(0.1, Dp), 'k', label='$\Delta z = 0.1$')
ax1.plot(zv, DeltaP(0.2, Dp), 'r', label='$\Delta z = 0.2$')
ax1.set_yscale('log')

ax1.set_xlabel('Mean redshift, $z_m$')
ax1.set_title("Fixed $k = 0.1 h/Mpc$")
ax1.legend(frameon=False)

# plt.savefig('plots/deltap.pdf')

plt.show()
```

```python
fig, (ax0, ax1) = plt.subplots(ncols=2, constrained_layout=True, figsize=(10,4))

ax0.loglog(k, DeltaP(0.1, Dp[37]) * Petc(k, Dz(zv[37]), powerk), 'k', label='$\Delta z = 0.1$')
ax0.loglog(k, DeltaP(0.2, Dp[37]) * Petc(k, Dz(zv[37]), powerk), 'r', label='$\Delta z = 0.2$')

ax0.set_xlabel('Wavenumber, $k$ $(h/Mpc)$')
ax0.set_ylabel('$|P_{geom} - P_{mid}| $')
ax0.set_title("Fixed $z_m=0.5$")
ax0.legend(frameon=False)

ax1.plot(zv, DeltaP(0.1, Dp) * pm, 'k', label='$\Delta z = 0.1$')
ax1.plot(zv, DeltaP(0.2, Dp) * pm, 'r', label='$\Delta z = 0.2$')
ax1.set_yscale('log')

ax1.set_xlabel('Mean redshift, $z$')
ax1.set_title("Fixed k $0.1 h/Mpc$")
ax1.legend(frameon=False)

# plt.savefig('plots/deltap.pdf')

plt.show()
```

 * Delta Cl's

 Using a toy model where
 \begin{equation}
    n(z) = \delta _D (z - z_a) + \delta _D (z - z_b)
\end{equation}

```python
import scipy.stats as ss

def plot_normal(x_range, mu=0, sigma=1, cdf=False, **kwargs):
    '''
    Plots the normal distribution function for a given x range
    If mu and sigma are not provided, standard normal is plotted
    If cdf=True cumulative distribution is plotted
    Passes any keyword arguments to matplotlib plot function
    '''
    x = x_range
    if cdf:
        y = ss.norm.cdf(x, mu, sigma)
    else:
        y = ss.norm.pdf(x, mu, sigma)
    plt.plot(x, y, **kwargs)
```

```python
x = np.linspace(-5, 5, 5000)
plot_normal(x, -2, 0.08, lw=1)
plot_normal(x, 2, 0.08, lw=1)

plt.xlabel('Redshift, $z$')
plt.ylabel('$n(z)$')
plt.title('Toy universe')

plt.xticks((-2, 2), ('$z_a$', '$z_b$'))
plt.yticks([])

# plt.savefig('plots/toy.pdf')

plt.show()
```

```python
from scipy.special import spherical_jn
from scipy.integrate import quad, simps
```

```python
zm = 0.5
dz = 0.1
za = zm - dz
zb = zm + dz
```

```python
ell = np.linspace(0.2, 2000)
```

```python
def jl(multipole, wavenumber, redshift):
    return spherical_jn(multipole, wavenumber * redshift)
```

```python
def DCl_toy(multipole , zmean=0.5, sepz=0.1):
    jla = jl(multipole, k, zmean - sepz)
    jlb = jl(multipole, k, zmean + sepz)
    integrandk = jla * jlb * Petc(k, 1, powerk) / k**2

    def d(z):
        return Det_eff(sepz, z)
    dp = derivative(d, zmean)
    return np.square(sepz * dp) * simps(integrandk, k)

def DCl(multipole , zmean=0.5, sepz=0.1):
    jla = jl(multipole, k, zmean - sepz)
    jlb = jl(multipole, k, zmean + sepz)
    pg = np.sqrt( Petc(k, Dz(zmean -sepz), powerk) * Petc(k, Dz(zmean + sepz), powerk) )
    pm = Petc(k, Dz(zmean), powerk)
    pg_pm = np.absolute(pg - pm)
    integrandk = jla * jlb * pg_pm / k**2

    return simps(integrandk, k)
```

```python
DCLv = []
for i in range(2000):
#     print(DCl(i))
    DCLv.append(DCl_toy(i))
```

```python
DCLv2 = []
for i in range(2000):
#     print(DCl(i))
    DCLv2.append(DCl_toy(i, sepz=0.2))
```

```python
lll=np.arange(2000)
```

```python
plt.plot(lll[5:], DCLv[5:], '--k', label='$\Delta z = 0.1$', lw=0.5)
plt.plot(lll[5:], DCLv2[5:], ':r', label='$\Delta z = 0.2$', lw=0.25)

plt.yscale('log')

plt.xlabel('Multipoles, $\ell$')
plt.ylabel('$|C_{\ell}^{geom} - C_{\ell}^{mid}|$')
plt.xlim(5,1000)

plt.title("Mean redshift, $z_m=0.5$")
plt.legend(frameon=False);
# plt.savefig('plots/deltacl.pdf')

plt.show()
```


# Block 5: One-dimensional optimal transport
### Alfred Galichon (NYU)
## `math+econ+code' masterclass on matching models, optimal transport and applications
© 2018-2020 by Alfred Galichon. Support from NSF grant DMS-1716489 is acknowledged. Julie Lenoir and James Nesbit contributed.

## Optimal transport I: One-dimensional matching

### Learning Objectives

* Copulas and comonotonicity

* Positive Assortative Matching

* The Wage Equation

### References

* [OTME], Ch. 4 and App. C

* Complements:

     * Becker (1973). A Theory of Marriage: Part I. *JPE*.

     * Sattinger (1993). Assignment Models of the Distribution of Earnings. *JEL*.

     * Gabaix and Landier (2008). Why has CEO Pay Increased So Much? *QJE*.

     * Tervi0 (2008). The Difference That CEOs Make: An Assignment Model Approach. *AER*.

## Motivation

Today we shall study one-dimensional models of matching. The application we shall envision is to the literature on CEO compensation: talented CEOs arguably generate more output if they run large firms than if they run a small business. Hence in these models, the CEO's talent is leveraged by the firm size.

The output of a match is CEO's talent multiplied by a term capturing the firm's size. This surplus function belongs to a class which leads to positive assortative matching: the optimal matching will consists of assigning the most talented CEO with the biggest firm, etc.

The model we shall follows a literature in family economics, see Becker (1973), and labor economics: Sattinger (1993), Tervio (2008), Gabaix and Landier (2008). Our application will be based on Gabaix and Landier's CEO compensation data. The difference with the empirical literature on marriage is that one observes typically less relevant characteristics on CEOs, but we observe the transfers, which are typically not observed whithin the couple.

## Data

For each firm, we observe the size of the firm (market cap) and the compensation.

Distribution of CEO talent (to be defined later) is calibrated in their paper and we will take it as given; so will be the production function.

Our exercise will consist of predicting the CEO compensation. (Actually GL do the opposite: they use the distribution of wages to calibrate the production function).

```python
# import gurobipy as grb
import pandas as pd
import numpy as np
# import scipy.sparse as sparse
import os
```

```python
thepath = os.path.join(os.getcwd(),'..')
data = pd.read_csv(os.path.join(thepath, "data_mec_optim/wages_gabaix-landier/data_Gabaix_Landier.csv"))
```

```python
data.head()
```

      yeara
      mktcap
      nFIRM
      comp
      nCEO

      0
      2000
      17522.963
      333.0
      22024.6190
      74.0

      1
      1991
      19737.494
      129.0
      NaN
      NaN

      2
      1992
      23442.229
      107.0
      NaN
      NaN

      3
      1993
      23647.432
      118.0
      NaN
      NaN

      4
      1994
      22099.512
      126.0
      5147.5156
      175.0

## Positive assortative matching

Assume $\mathcal{X}=\mathcal{Y}=\mathbb{R}$. This implies that both workers and firms are characterized by scalar characteristics $x$ and $y$. Restrictive, but most of the applied literature in Economics to this day has
focused on this case, which already generates interesting economic insights.

Assume that there is the same number of workers and firms, and this number is normalized to one. Let $P$ be the probability distribution of the workers types $x$, and $Q$ the probability distribution of the firms types $y$.

E.g. the literature on CEO compensation studies the matching problem of firms and managers: each manager has\ a measure of talent $x\in\mathbb{R}_{+}$ (extra return generated), and each firm has market capitalization $y\in\mathbb{R}_{+}$. Then, the economic value generated by a manager of talent $x$ running a firm of size $y$ is $\Phi\left(  x,y\right)  =xy$. Assume manager of talent $x$ is assigned to firm of size $y=T\left(x\right)  $. The constraint on the assignment map $T$ is that $T\#P=Q$, which means that each firm is run by a manager and each manager runs a firm. Then the total value created is

\begin{align*}
\mathbb{E}_{P}\left[  \Phi\left(  X,T\left(  X\right)  \right)  \right]
=\mathbb{E}\left[  XT\left(  X\right)  \right]  .
\end{align*}

Under a natural assuption on $\Phi$, called \emph{supermodularity}, we will see that the optimal coupling is such that $Y=T\left(  X\right)  $, where $T$ is nondecreasing. $T$ will be given an explicit characterization.

## Copulas

Recall that given a probability distribution $P$ on the real line, one can define the \emph{quantile} of that distribution, denoted $F_{P}^{-1}$, which is a map from $\left[  0,1\right]  $ to $R$, nondecreasing and continuous from the right, and such that if $U\sim\mathcal{U}:=\mathcal{U} \left(\left[0,1\right]\right)$, then

\begin{align*}
F_{P}^{-1}\left(  U\right)  \sim P.
\end{align*}

$F_{P}^{-1}$ is the generalized inverse of the c.d.f. of $P$, $F_{P}$, the proper inverse when $P$ has a positive density, in which case $U=F_{P}\left(X\right)  $. Note also that if $X\sim P$, then $F_{P}\left(  X\right)  $ has a uniform distribution if and only if $P$ has no mass point.

[Quantile representation](#quantileRep) extends to the case of bivariate distributions: for $\pi\in\mathcal{M}\left(  P,Q\right)  $, there exists a pair $\left(  U,V\right)  $ of uniform random variables such that

\begin{align*}
\left(  F_{P}^{-1}\left(  U\right)  ,F_{Q}^{-1}\left(  V\right)  \right)
\sim\pi,
\end{align*}

and the c.d.f. associated with the distribution of $\left(  U,V\right)  $ is called the *copula* associated with distribution $\pi$.

## Comonotonicity

A pair of random variables $\left(  X,Y\right)$ is *comonotone* if there is $U\sim\mathcal{U}$ such that $X=F_{P}^{-1}\left(  U\right)$ and $Y=F_{Q}^{-1}\left(  U\right)  $. Equivalently, $X$ and $Y$ are said to exhibit *Positive Assortative Matching (PAM)*.

The copula associated with a pair of comonotone random variables is the c.d.f. associated with $\left(  U,U\right)  $, which is $F\left(  u,v\right) =\min\left(  u,v\right)  $. This copula is called the *upper Fréchet-Hoeffding copula*.

Note that when the cdf of $X$ is continuous, there is a much simpler equivalent statement of comonotonicity:

---
**Lemma**
If the distribution of $X$ has no mass points, then $X$ and $Y$ are comonotone if and only if there exists a nondecreasing map $T$ such that $Y=T\left(  X\right)  $. Moreover, one can choose $T\left(x\right)  =F_{Q}^{-1}\left(  F_{P}\left(  x\right)  \right).$

---

---
**Proof**

Consider $U\sim\mathcal{U}$ such that $X=F_{P}^{-1}\left(  U\right)  $ and $Y=F_{Q}^{-1}\left(  U\right)  $. If the distribution of $X$ has no mass point, then $U=F_{P}\left(  X\right)  $. Hence, $Y=F_{Q}^{-1}\left(F_{P}\left(  X\right)  \right)  $.

---

### Supermodular surplus

Assume $\Phi$ is \emph{supermodular}, that is, for every scalars $x$, $x^{\prime}$, $y$ and $y^{\prime}$,

\begin{align*}
\Phi\left(  x\vee x^{\prime},y\vee y^{\prime}\right)  +\Phi\left(  x\wedge
x^{\prime},y\wedge y^{\prime}\right)  \geq\Phi\left(  x,y\right)  +\Phi\left(
x^{\prime},y^{\prime}\right)  , \label{supermodularPhi}%
\end{align*}

where $x\vee x^{\prime}$ and $x\wedge x^{\prime}$ denote respectively the maximum and the minimum between scalars $x$ and $x^{\prime}$. When $\Phi$ is twice continuously differentiable (which we will assume from now on), this is equivalent to

\begin{align*}
\frac{\partial^{2}\Phi\left(  x,y\right)  }{\partial x\partial y}\geq0.
\label{SupermodDiff}%
\end{align*}

Assume that there are two types of workers $\mathcal{X=}\left\{\underline{x},\overline{x}\right\}  $ and firms $\mathcal{Y=}\left\{  \underline{y},\overline{y}\right\}  $. An equivalent restatement of the [supermodular condition](#supermodularPhi) is then

\begin{align*}
\overline{x}\geq \underline{x} \text{ and }\overline{y}\geq \underline{y} \text{ implies }\Phi\left(
\overline{x},\overline{y}\right)  +\Phi\left(  \underline{x},\underline{y}\right)
\geq\Phi\left(  \overline{x},\underline{y}\right)  +\Phi\left(  \underline{x},\overline
{y}\right)  \label{PAMequiv}%
\end{align*}

which asserts that the total output created is higher if the high types match together and the low types match together (assortative matching) rather than if mixed high/low pairs are formed.

The following examples of surplus functions are supermodular:

1. Cobb-Douglas function: $\Phi\left(  x,y\right)  =x^{a}y^{b}$ ($x,y\geq 0$), with $a,b\geq 0$,

2. General multiplicative form: $\Phi\left(  x,y\right)  =\zeta\left(x\right)  \xi\left(  y\right)  $ with $\zeta$ and $\xi$ nondecreasing,

3. Leontieff: $\Phi\left(  x,y\right)  =\min\left(  x,y\right)$,

4. C.E.S. function: $\Phi\left(  x,y\right)  =\left(  x^{-\rho}+y^{-\rho}\right)  ^{-1/\rho}$, $\rho\geq 0$,

5. $\Phi\left(  x,y\right)  =\phi\left(  x-y\right)  $ where $\phi$ is concave; in particular, $\Phi\left(  x,y\right)  =-\left\vert x-y\right\vert ^{p}$, $p\geq1$ or $\Phi\left(  x,y\right)  =-\left(  x-y-k\right)  ^{+}$,

6. $\Phi\left(  x,y\right)  =\phi\left(  x+y\right)  $, where $\phi$ convex.

## Rearrangement theorem

---
**Theorem**

1. Assume that $\Phi$ is supermodular. Then the primal of the Monge-Kantorovich problem

    \begin{align*}
    \sup_{\pi\in\mathcal{M}\left(  P,Q\right)  }\mathbb{E}_{\pi}\left[
    \Phi\left(  X,Y\right)  \right]
    \end{align*}

    has a comonotone solution.

2. Conversely, if [MK primal](#mkpb1d) has a comonotone solution for any choice of probability distributions $P$ and $Q$ on the real line, then $\Phi$ is supermodular.

3. If, in addition, $P$ has no mass points, then there is an optimal assignment which is is pure and satisfies $Y=T\left(  X\right)$ where

    \begin{align*}
    T\left(  x\right)  =F_{Q}^{-1}\circ F_{P}\left(  x\right).
    \end{align*}

---
The proof of part 1. is based on the following lemma.

---

**Lemma**

Let $Z_{1}$ and $Z_{2}$ be two Bernoulli random variables of respective success probability $p_{1}$ and $p_{2}$. Then $\mathbb{E}\left[  Z_{1}Z_{2}\right]  \leq\min\left(  p_{1},p_{2}\right)$.

---
---
**Proof**

As $Z_{2}\leq1$, $\mathbb{E}\left[  Z_{1}Z_{2}\right]  \leq\mathbb{E}\left[Z_{1}\right]  =p_{1}$. Similarly $\mathbb{E}\left[  Z_{1}Z_{2}\right]\leq\mathbb{E}\left[  Z_{2}\right]  =p_{2}$. Thus, $\mathbb{E}\left[Z_{1}Z_{2}\right]  \leq\min\left(  p_{1},p_{2}\right)$.

---

We are now ready to sketch the proof of the [Rearrangement Theorem](#thm:MKsupermod).

---
**Proof**

1. Take $U\sim\mathcal{U}$,and $X=F_{P}^{-1}\left(  U\right)  $ and $Y=F_{Q}^{-1}\left(  U\right)$. By the [quantile representation](#quantileRep), $X\sim P$ and $Y\sim Q$ and $\left(  X,Y\right)$ is comonotone by definition. The proof is in three steps.

    **Step 1.** For $a,b\in\mathbb{R}$, consider surplus function $\phi_{ab}\left(x,y\right)  :=1\left\{  x\geq a\right\}  1\left\{  y\geq b\right\}$, and let $Z_{1}=1\left\{  X\geq a\right\}  $ and $Z_{2}=1\left\{  Y\geq b\right\}$. $Z_{1}$ and $Z_{2}$ are two Bernoulli random variables of respective success probability $p_{1}=1-F_{P}\left(  a\right)  $ and $p_{2}=1-F_{Q}\left(b\right)  $, thus $\mathbb{E}\left[  Z_{1}Z_{2}\right]  \leq\min\left(p_{1},p_{2}\right)  $, but a straightforward calculation shows that the inequality actually holds as an equality. Hence $\left(  X,Y\right)$, which is comonotone, is optimal for each surplus function $\phi_{ab}$.

    **Step 2.** Assume $\mathcal{X}=\left[\underline{x},\overline{x}\right]$ and $\mathcal{Y} = \left[ \underline{y}, \overline{y}\right]$ are compact intervals. Then

    \begin{align*}
F\left(  x,y\right)  =\frac{\Phi\left(  x,y\right)  -\Phi\left(  \underline{x},y\right)  -\Phi\left(  x,\underline{y}\right)  +\Phi\left(  \underline{x},\underline{y}\right)  }{\Phi\left(  \overline{x},\overline{y}\right)  -\Phi\left(\underline{x},\overline{y}\right)  -\Phi\left(  \overline{x},\underline{y}\right)+\Phi\left(  \underline{x},\underline{y}\right)  }
    \end{align*}

    is a c.d.f. associated to a probability measure $\zeta$, and hence

    \begin{align*}
F\left(x,y\right) = \int\phi_{ab}\left(  x,y\right)  d\zeta\left(  a,b\right)$.
    \end{align*}

    As a result, if $\pi\in\mathcal{M}\left(  p,q\right)$ is the distribution of $\left(  X,Y\right)  $ where $X$ and $Y$ are comonotone, then

    \begin{align*}
\int F\left(  x,y\right)  d\pi\left(  x,y\right)  \geq\int F\left(
x,y\right)  d\tilde{\pi}\left(  x,y\right)
    \end{align*}

    for every $\tilde{\pi}\in\mathcal{M}\left(  p,q\right)  $. But as $F$ is of the form $F\left(  x,y\right)  =K\Phi\left(  x,y\right)  +f\left(  x\right) +g\left(  y\right)  +c$ with $K>0$, and because $\int\left\{  f\left(x\right)  +g\left(  y\right)  +c\right\}  d\pi\left(  x,y\right)=\int\left\{  f\left(  x\right)  +g\left(  y\right)  +c\right\}  d\tilde{\pi}\left(  x,y\right)  $ for every $\tilde{\pi}\in\mathcal{M}\left(  p,q\right) $, it results that

    \begin{align*}
\int\Phi\left(  x,y\right)  d\pi\left(  x,y\right)  \geq\int\Phi\left(x,y\right)  d\tilde{\pi}\left(  x,y\right)  ~\forall\tilde{\pi}\in \mathcal{M}\left(  p,q\right)
    \end{align*}
    which completes step 2.

    **Step 3.** When $\mathcal{X}$ and $\mathcal{Y}$ are the real line, the result still holds by approximation.

2. The converse follows by taking for $P$ the discrete probability with two mass points \b{x} and $\overline{x}$ with probability 1/2 each, and $Q$ the discrete probability with two mass points \b{y} and $\overline{y}$ also each with probability 1/2. Then if [MK primal](#mkpb1d) has a solution such that $F_{P}^{-1}\left(  U\right)  $ and $Y=F_{Q}^{-1}\left(  U\right)  $, for $U\sim\mathcal{U}\left(  \left[  0,1\right]  \right)  $, it follows that [condition](#PAMequiv) holds.

3. follows from (i) and [Lemma](#lem:comonotone).

---

Note that the assumptions made in the [Rearrangement Theorem](#thm:MKsupermod) do not guarantee that all the optimal assignments are comonotone. Indeed, the trivial example where $\Phi\left(  x,y\right)  =0$ for every $x$ and $y$ provides an example of supermodular surplus function, for which any assignment is optimal. For this reason, we provide a strengthening of the previous result, which ensures uniqueness. We will assume $\Phi$ is strictly supermodular, that is if both $\overline{x}>\underline{x}$ and $\overline{y}>\underline{y}$ hold, then $\Phi\left(\overline{x},\overline{y}\right)  +\Phi\left(\underline{x},\underline{y}\right)  >\Phi\left(\overline{x},\underline{y}\right)  +\Phi\left( \underline{x},\overline{y}\right)$.

---
**Theorem**

Assume that $\Phi$ is strictly supermodular, and $P$ has no mass point. Then the primal [Monge-Kantorovich problem](#mkpb1d) has a unique optimal assignment, and this assignment is characterized by $Y=T\left(  X\right)  $ where $T$ is given by [this](#mongeMap1d).

---

### The wage equation

Assume $\left(u,v\right)$ is a solution to the dual of the Monge-Kantorovich problem

\begin{align}
\inf \,&  \mathbb{E}_{P}\left[  u\left(  X\right)  \right]  +\mathbb{E}_{Q}\left[  v\left(  Y\right)  \right] \\
s.t.~  &  u\left(  x\right)  +v\left(  y\right)  \geq\Phi\left(  x,y\right)
\end{align}

Then $v\left(y\right)$ is interpreted as the value of the problem of a firm of type $y$, choosing the optimal worker $x$. Then the firm's program is

\begin{align*}
v\left(  y\right)  =\max_{x}\left\{  \Phi\left(  x,y\right)  -u\left(
x\right)  \right\}
\end{align*}

thus by first order conditions, one is led to the *wage equation*

\begin{align*}
u^{\prime}\left(  x\right)  =\frac{\partial\Phi}{\partial x}\left(  x,T\left(
x\right)  \right)  , \label{WageEq}%
\end{align*}
where $T$ is given by [this](#mongeMap1d).

---
**Theorem**

1. Assume $\Phi$ is supermodular and continuously differentiable with respect to its first variable. Assume $P$ has no mass point. Then the [dual Monge-Kantorovich problem](#mkdual1d) has a solution $\left(u,v\right)$. Further, $u$ solves the [wage equation](#WageEq). Hence, $u$ is determined up to a constant $c$ by

    \begin{align*}
    u\left(  x\right)  =c+\int_{x_{0}}^{x}\frac{\partial\Phi}{\partial x}\left(t,T\left(  t\right)  \right)  dt.
    \end{align*}

2. Assume further that $Q$ has no mass point, and that $\Phi$ is also continuously differentiable with respect to its second variable. Then $v$ is given by

    \begin{align*}
    v\left(  y\right)  =c^{\prime}+\int_{T\left(  x_{0}\right)  }^{y}\frac{\partial\Phi}{\partial y}\left(  T^{-1}\left(  z\right)  ,z\right)  dz,
    \end{align*}

    where $c$ and $c^{\prime}$ are related by $c+c^{\prime}=\Phi\left(x_{0},T\left(  x_{0}\right)  \right).$

---

### Interpretation: imperfect competition and rents

It is important to distinguish between $\partial\Phi\left(  x,T\left(x\right)  \right)  /\partial x$, which is the partial derivative of $\Phi\left(  x,y\right)  $ applied at $\left(  x,y\right)  =\left(  x,T\left(x\right)  \right)  $, and $d\Phi\left(  x,T\left(  x\right)  \right)  /dx$,which is the total derivative of $\Phi\left(  x,T\left(  x\right)  \right)$ with respect to $x$. One has

\begin{align*}
\frac{d\Phi\left(  x,T\left(  x\right)  \right)  }{dx}=\frac{\partial \Phi\left(  x,T\left(  x\right)  \right)  }{\partial x}+\frac{\partial \Phi\left(  x,T\left(  x\right)  \right)  }{\partial y}T^{\prime}\left(x\right)
\end{align*}

This decomposition has an interesting interpretation in terms of differential rent. The total derivative $d\Phi\left(  x,T\left(  x\right) \right)  /dx$ is the marginal increase in value between a firm run by a manager of talent $x$ and a firm run by a manager of talent $x+dx$. This differential value is split between the manager's differential rent ($\partial\Phi\left(  x,T\left(  x\right)  \right)  /\partial x=u^{\prime}\left(  x\right)  $) and the firm's differential rent ( $\left(  \partial \Phi\left(  x,T\left(  x\right)  \right)  /\partial y\right)  T^{\prime }\left(  x\right)  =dv\left(  T\left(  x\right)  \right)  /dx$).

This discussion highlights the nature of the assignment model, which is a model of imperfect competition. In this model, managers compete against each other, and likewise on the other side of the market. Managers are imperfect substitutes for each other. This imperfect competition is the source of the rent, as made apparent by [wage equation formula](#WageEq).

## Application

Gabaix and Landier (2008) assume that

\begin{align*}
\Phi\left(  x,y\right)  =Cxy^{\gamma}
\end{align*}

where $C$ and $\gamma$ are parameters.

If $X$ is the talent of the CEO, then the distribution of $X$ can be parameterized through its quantile by

\begin{align*}
Q_{X}\left(  t\right)  =X^{\max}-\frac{B}{\beta N}\left(  N\left(  1-t\right) \right)  ^{\beta}
\end{align*}

(such a distribution is consistent with extreme value theory). $N$ is the number of firms and $\beta$ is a parameter.

If $Y$ is the size of the firm, then its distribution is assumed to be Pareto with distribution $1/\alpha$, i.e.

\begin{align*}
Q_{Y}\left(  t\right)  =A\left(  N\left(  1-t\right)  \right)  ^{-\alpha}%
\end{align*}

(they find that $\alpha=1$, i.e. Zipf's law, fits the data quite well).

The wage of CEO indexed by $t\in\left[  0,1\right]  $ is $w\left(t\right)  $; firm $y=Q_{Y}\left(  t\right)  $'s problem is

\begin{align*}
\max_{t\in\left[  0,1\right]  }\left\{  CQ_{X}\left(  t\right)  y^{\gamma}-w\left(  t\right)  \right\}  =\max_{t\in\left[  0,1\right]  }\left\{-C\frac{B}{\beta N}\left(  N\left(  1-t\right)  \right)  ^{\beta}y^{\gamma}-w\left(  t\right)  \right\}
\end{align*}

and by FOC:

\begin{align*}
w^{\prime}\left(  t\right)   &  =BCy^{\gamma}\left(  N\left(  1-t\right)
\right)  ^{\beta-1}\\
&  =A^{\gamma}BC\left(  N\left(  1-t\right)  \right)  ^{-\alpha\gamma+\beta-1}%
\end{align*}

where it is assumed $\alpha\gamma>\beta$. When $N$ is large, this yields

\begin{align*}
w\left(  t\right)  \approx\frac{A^{\gamma}BC}{\alpha\gamma-\beta}\left(N\left(  1-t\right)  \right)  ^{-\alpha\gamma+\beta}
\end{align*}

"Superstar" effect (Rosen, 1981): if $\beta>0$, then the wages are unbounded, even though talent is bounded -- talent is very valuable at the top end of the distribution.

We shall use Gabaix-Landier's calibrated values
\begin{align*}
\beta\simeq2/3,\gamma\simeq1,BC=2.8\ast10^{-6}
\end{align*}

```python
β = 2/3
B = 1
C = 2.8e-06
α = γ = 1
X_max = 0
```

We will focus on data from year 2003.

```python
year = 2003
data2003 = data[data['yeara'] == 2003]
```

We will normalize compensation to insure compensation and mktcap are in the same order of magnitude. We need to get rid of the `NA`'s in the data (separately for compensation and market cap) and sort in decreasing order.

```python
comp = data2003['comp']/1000  # insures same order of magnitude
comp = comp[comp.notnull()].sort_values(ascending=False)

mktcap = data2003['mktcap']
mktcap = mktcap[mktcap.notnull()].sort_values(ascending=False)
```

```python
N=mktcap.size
n= np.arange(1,N + 1)
```

So in order compute the wage function, it remains to find the value of $A$. This will simply be a regression of $\log(S)$ on $n$. Regress firm rank against mktcap:

```python
import statsmodels.api as sm
n = np.log(n)
n = sm.add_constant(n)
model = sm.OLS(np.log(mktcap), n).fit()
model.summary()
```

OLS Regression Results

  Dep. Variable:	         mktcap	        R-squared:         	    0.986

  Model:	                   OLS	         Adj. R-squared:    	    0.986

  Method:	             Least Squares	    F-statistic:       	 3.574e+04

  Date:	             Tue, 21 Jan 2020	   Prob (F-statistic):	    0.00

  Time:	                 09:01:26	       Log-Likelihood:    	   380.55

  No. Observations:	         500	        AIC:               	   -757.1

  Df Residuals:	             498	        BIC:               	   -748.7

  Df Model:	                   1

  Covariance Type:	      nonrobust

    	       coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  const	    15.5140	     0.028	   562.058	  0.000	    15.460	    15.568

  x1	       -0.9822	     0.005	  -189.048	  0.000	    -0.992	    -0.972

  Omnibus:	       582.902	   Durbin-Watson:     	    0.098

  Prob(Omnibus):	  0.000	    Jarque-Bera (JB):  	 67515.074

  Skew:	          -5.276	    Prob(JB):          	     0.00

  Kurtosis:	      58.941	    Cond. No.          	     29.9

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
A = np.exp(model.params[0])
```

We can compute the theoretical wage function and plot it against the realized wage function.

```python
def wage(x):
    return ((A**γ * B * C)/(α * γ - β) * x**(-α * γ + β))
```

```python
W = list(map(wage, range(1,N + 1)))
```

```python
import matplotlib.pyplot as plt
plt.plot(range(1, N+1), W)
plt.plot(range(1, N+1), comp)
```