# More Mixed Logit
The purpose of this notebook is to prodvide more examples of estimating Normal Error Component Mixed Logit Models. The types of models to be estimated in this notebook include models that use mixing distributions to account for heteroskedasticity and/or non-zero correlations between the error terms of one's various utility equations.

As in some of the other example notebooks, the dataset being used is the "Travel Mode Choice" dataset from Greene and Hensher. It is described on the statsmodels website, and their description is reproduced below in full. The basic specification to be estimated will come from the example given in Econometric Analysis by William H. Greene (2011), section 23.11.7.

    The data, collected as part of a 1987 intercity mode choice study, are a sub-sample of 210 non-business
    trips between Sydney, Canberra and Melbourne in which the traveler chooses a mode from four alternatives
    (plane, car, bus and train). The sample, 840 observations, is choice based with over-sampling of the
    less popular modes (plane, train and bus) and under-sampling of the more popular mode, car. The level of
    service data was derived from highway and transport networks in Sydney, Melbourne, non-metropolitan N.S.W.
    and Victoria, including the Australian Capital Territory.

    Number of observations: 840 Observations On 4 Modes for 210 Individuals.
    Number of variables: 8
    Variable name definitions::

        individual = 1 to 210
        mode =
            1 - air
            2 - train
            3 - bus
            4 - car
        choice =
            0 - no
            1 - yes
        ttme = terminal waiting time for plane, train and bus (minutes); 0
               for car.
        invc = in vehicle cost for all stages (dollars).
        invt = travel time (in-vehicle time) for all stages (minutes).
        gc = generalized cost measure:invc+(invt*value of travel time savings)
            (dollars).
        hinc = household income ($1000s).
        psize = traveling group size in mode chosen (number).

    Source

    Greene, W.H. and D. Hensher (1997) Multinomial logit and discrete choice models in Greene, W. H. (1997)
    LIMDEP version 7.0 user’s manual revised, Plainview, New York econometric software, Inc. Download from
    on-line complements to Greene, W.H. (2011) Econometric Analysis, Prentice Hall, 7th Edition (data table
    F18-2) http://people.stern.nyu.edu/wgreene/Text/Edition7/TableF18-2.csv

# Load needed packages

```python
# Note the ordered dictionaries are used to store our
# model specifications
from collections import OrderedDict

# Pandas is used for data input/output
import pandas as pd
# Numpy is used for vectorized math operations
import numpy as np
# Statsmodels is used to retrieve the dataset
import statsmodels.datasets

# Pylogit is used to estimate the model
import pylogit as pl
```

# Load the dataset

```python
# Access the dataset
mode_data = statsmodels.datasets.modechoice.load_pandas()
# Get a pandas dataframe of the mode choice data
mode_df = mode_data["data"]
# Look at a few rows of the data
mode_df.head()
```

      individual
      mode
      choice
      ttme
      invc
      invt
      gc
      hinc
      psize

      0
      1
      1
      0
      69
      59
      100
      70
      35
      1

      1
      1
      2
      0
      34
      31
      372
      71
      35
      1

      2
      1
      3
      0
      35
      25
      417
      70
      35
      1

      3
      1
      4
      1
      0
      10
      180
      30
      35
      1

      4
      2
      1
      0
      64
      58
      68
      68
      30
      2

# Specify and Estimate a Multinomial Logit (MNL) Model
Even though we want to estimate a mixed logit model, we first start by speficying a basic MNL model. This will give us "good" starting values for the mixed logit model.

The utility equations used by Greene are as follows. Note that $i$ refers to a particular individual and $j$ refers to a particular travel mode. Moreover, $d_{i, \textrm{condition}}$ is an indicator that equals 1 for individual $i$ when the condition is true.

$\begin{align}
U_{ij} &= \alpha _{\textrm{air}} d_{i,\ j=\textrm{air}} +
          \alpha _{\textrm{train}} d_{i,\ j=\textrm{train}} +
          \alpha _{\textrm{bus}} d_{i,\ j=\textrm{bus}} + \\
          &\quad \ \beta _{\textrm{generalized_cost}} \textrm{GC}_{ij} +
          \beta _{\textrm{terminal_time}} \textrm{TTME}_{ij} +
          \beta _{\textrm{household_income}} d_{i,\ j=\textrm{air}} \textrm{HINC}_{i} +
          \epsilon _{ij}
\end{align}
$

In other words, there is an alternative-specific-constant (i.e. an intercept) for the air, train, and bus modes, and the car is used as the reference mode. Moreover, the alternative-specific variables (i.e. the generalized cost and terminal time variables) are specified has having generic effects on all modes. That is, the impact of a one-minute increase in terminal time or a 1\$ increase in "generalized cost" on the utility of a given mode is a-priori assumed to be the same for all modes. In contrast to the alternative-specific variables, household income is an individual characteristic and it is a-priori assumed to only affect the utility of traveling by air.

```python
# Initialize the Ordered Dictionaries to store the model
# specification and parameter names
specification_dict = OrderedDict()
name_dict = OrderedDict()

# Add the variables to the specification and name dictionaries.
specification_dict["intercept"] = [1, 2, 3]
name_dict["intercept"] = ["ASC Air", "ASC Train", "ASC Bus"]

specification_dict["gc"] = [[1, 2, 3, 4]]
name_dict["gc"] = ["Generalized Cost (all modes)"]

specification_dict["ttme"] = [[1, 2, 3, 4]]
name_dict["ttme"] = ["Terminal Time (all modes)"]

specification_dict["hinc"] = [1]
name_dict["hinc"] = ["Household Income (air mode)"]

```

```python
# Estimate the MNL model
# Initialize the model object
mnl_model = pl.create_choice_model(data=mode_df,
                                   alt_id_col="mode",
                                   obs_id_col="individual",
                                   choice_col="choice",
                                   specification=specification_dict,
                                   model_type="MNL",
                                   names=name_dict)

# Create the initial values for the parameters
# being estimated
num_params = 6
initial_values = np.zeros(num_params)

# Estimate the mnl_model
mnl_model.fit_mle(initial_values)

# View the estimation results
mnl_model.get_statsmodels_summary()
```

    Log-likelihood at zero: -291.1218
    Initial Log-likelihood: -291.1218
    Estimation Time: 0.01 seconds.
    Final log-likelihood: -199.1284

    /Users/timothyb0912/anaconda/lib/python2.7/site-packages/scipy/optimize/_minimize.py:382: RuntimeWarning: Method BFGS does not use Hessian information (hess).
      RuntimeWarning)

Multinomial Logit Model Regression Results

  Dep. Variable:	         choice	            No. Observations:  	    210

  Model:	         Multinomial Logit Model	   Df Residuals:      	    204

  Method:	                  MLE	             Df Model:          	     6

  Date:	             Sat, 19 Nov 2016	       Pseudo R-squ.:     	   0.316

  Time:	                 19:01:17	           Pseudo R-bar-squ.: 	   0.295

  converged:	              True	             Log-Likelihood:    	 -199.128

   	                          	              LL-Null:           	 -291.122

                	                  coef	     std err	      z	      P>|z|	 [95.0% Conf. Int.]

  ASC Air	                          5.2074	     0.779	     6.684	  0.000	     3.681     6.734

  ASC Train	                        3.8690	     0.443	     8.731	  0.000	     3.001     4.738

  ASC Bus	                          3.1632	     0.450	     7.025	  0.000	     2.281     4.046

  Generalized Cost (all modes)	    -0.0155	     0.004	    -3.517	  0.000	    -0.024    -0.007

  Terminal Time (all modes)	       -0.0961	     0.010	    -9.207	  0.000	    -0.117    -0.076

  Household Income (air mode)	      0.0133	     0.010	     1.295	  0.195	    -0.007     0.033

# Estimate a Mixed Logit Model with heteroskedastic errors
One way to allow the variances of the utilities for the various alternatives to be different is to add a mean zero normal error component to each utility. One's utility equations would then be as follows:
$\begin{align}
U_{ij} &= \alpha _{\textrm{air}} d_{i,\ j=\textrm{air}} +
          \alpha _{\textrm{train}} d_{i,\ j=\textrm{train}} +
          \alpha _{\textrm{bus}} d_{i,\ j=\textrm{bus}} + \\
          &\quad \ \beta _{\textrm{generalized_cost}} \textrm{GC}_{ij} +
          \beta _{\textrm{terminal_time}} \textrm{TTME}_{ij} +
          \beta _{\textrm{household_income}} d_{i,\ j=\textrm{air}} \textrm{HINC}_{i} +\\
          &\quad \ \sigma_{\textrm{air}} d_{i,\ j=\textrm{air}} \eta_{ij} + \sigma_{\textrm{train}} d_{i,\ j=\textrm{train}} \eta_{ij} + \sigma_{\textrm{bus}} d_{i,\ j=\textrm{bus}} \eta_{ij} + \sigma_{\textrm{car}} d_{i,\ j=\textrm{car}} \eta_{ij} + \epsilon_{ij}\\
\textrm{where }& \eta_{ij} \sim N\left(0, 1\right)
\end{align}
$

There are two things to note about the utility equations above. First, as noted by Walker et al. (2007) not all of the $\sigma_j$'s are identifiable. As demonstrated below, this will be resolved by constraining the smallest $\sigma_j$ to zero. Secondly, note that the utility equations above are formally equivalent to treating this as a random-coefficients model, where we view the alternative specific constants as being randomly distributed. This can be seen by re-writing the utility equations as follows:
$\begin{align}
U_{ij} &= \beta _{\textrm{generalized_cost}} \textrm{GC}_{ij} +
          \beta _{\textrm{terminal_time}} \textrm{TTME}_{ij} +
          \beta _{\textrm{household_income}} d_{i,\ j=\textrm{air}} \textrm{HINC}_{i} +\\
          &\quad \ \sigma_{\textrm{air}} d_{i,\ j=\textrm{air}} \tilde{\eta}_{ij} + \sigma_{\textrm{train}} d_{i,\ j=\textrm{train}} \tilde{\eta}_{ij} + \sigma_{\textrm{bus}} d_{i,\ j=\textrm{bus}} \tilde{\eta}_{ij} + \sigma_{\textrm{car}} d_{i,\ j=\textrm{car}} \tilde{\eta}_{ij} + \epsilon_{ij}\\
\textrm{where }& \tilde{\eta}_{ij} \sim N\left(\alpha _{j}, 1\right)\\
          &\quad \ j \in \left\lbrace \textrm{air, train, bus, car}\right\rbrace\\
          &\quad \ \alpha _{\textrm{car}} = 0
\end{align}
$

From this perspective, using a mixed-logit model to account for heteroskedasticity is equivalent to using a mixed-logit model where the alternative-specific constants are viewed as being randomly distributed throughout the population.

Programmatically, PyLogit implements mixed logit models as random-coefficients models, so we will use this second viewpoint to specify the model to be estimated.

In particular, we will instruct PyLogit to treat the alternative specific constants as normally distributed random variables, where we will estimate the mean and standard deviations of the distributions. Note that in order to estimate the $\sigma_{\textrm{car}}$ of the car alternative, we need to have a variable for the alternative specific constant of the car. We will therefore add the car to the intercept specification, but we will then constrain it to zero during estimation (and as noted in the equation above).

As in Walker et al. (2007), we will first attempt to estimate all the standard deviations. Then (for identification purposes) we will re-estimate the model, constraining the smallest standard deviation from the first estimation.

Sources:

Joan L. Walker, Moshe Ben-Akiva, and Denis Bolduc. "Identification of parameters in normal error component logit-mixture (NECLM) models." Journal of Applied Econometrics, 22(6):1095–1125, September 2007. ISSN 1099-1255. doi: 10.1002/jae.971. URL http://onlinelibrary.wiley. com/doi/10.1002/jae.971/abstract.

```python
# Add the intercept for car to the specification dictionary
heteroskedastic_specification = OrderedDict()
heteroskedastic_names = OrderedDict()

heteroskedastic_specification["intercept"] = [1, 2, 3, 4]
heteroskedastic_names["intercept"] = ["ASC Air", "ASC Train", "ASC Bus", "ASC Car"]

for key in specification_dict:
    if key != "intercept":
        heteroskedastic_specification[key] = specification_dict[key]
        heteroskedastic_names[key] = name_dict[key]

# Create a list of the variables whose standard deviations
# are to be estimated
mixing_variables = heteroskedastic_names["intercept"]

heteroskedastic_model_0 = pl.create_choice_model(data=mode_df,
                                                 alt_id_col="mode",
                                                 obs_id_col="individual",
                                                 choice_col="choice",
                                                 specification=heteroskedastic_specification,
                                                 model_type="Mixed Logit",
                                                 names=heteroskedastic_names,
                                                 mixing_id_col="individual",
                                                 mixing_vars=mixing_variables)

# Create the initial values for the parameters
# being estimated. Note that we start at the MNL estimates
# and that we need to have initial variables for the
# standard deviation parameters that we'll estimate as well
# as for intercept for car (which we'll constrain to zero)
initial_values = np.concatenate((mnl_model.params.values[:3],
                                 np.zeros(1),  # for the car intercept
                                 mnl_model.params.values[3:],
                                 np.zeros(len(mixing_variables))))

# Estimate the Mixed logit model.
# We set a seed for reproducibility, and we constrain the intercept
# for the car utility, i.e. position 4 (or in python, array index 3).
heteroskedastic_model_0.fit_mle(initial_values,
                                seed=26,
                                num_draws=500,
                                constrained_pos=[3])

# View the estimation results
heteroskedastic_model_0.get_statsmodels_summary()
```

    Log-likelihood at zero: -291.1218
    Initial Log-likelihood: -199.1284
    Estimation Time: 4.66 seconds.
    Final log-likelihood: -195.7015

Mixed Logit Model Regression Results

  Dep. Variable:	      choice	         No. Observations:  	    210

  Model:	         Mixed Logit Model	   Df Residuals:      	    199

  Method:	               MLE	          Df Model:          	    11

  Date:	          Sat, 19 Nov 2016	    Pseudo R-squ.:     	   0.328

  Time:	              19:01:22	        Pseudo R-bar-squ.: 	   0.290

  converged:	           False	         Log-Likelihood:    	 -195.701

   	                       	           LL-Null:           	 -291.122

                	                  coef	     std err	      z	      P>|z|	 [95.0% Conf. Int.]

  ASC Air	                          4.6890	     1.503	     3.119	  0.002	     1.743     7.635

  ASC Train	                        5.2413	     1.113	     4.708	  0.000	     3.059     7.423

  ASC Bus	                          4.2436	     1.063	     3.991	  0.000	     2.159     6.328

  ASC Car	                               0	       nan	       nan	    nan	       nan       nan

  Generalized Cost (all modes)	    -0.0333	     0.010	    -3.274	  0.001	    -0.053    -0.013

  Terminal Time (all modes)	       -0.1156	     0.019	    -6.126	  0.000	    -0.153    -0.079

  Household Income (air mode)	      0.0381	     0.025	     1.497	  0.135	    -0.012     0.088

  Sigma ASC Air	                   -3.3812	     1.038	    -3.257	  0.001	    -5.416    -1.347

  Sigma ASC Train	                  0.0660	     4.236	     0.016	  0.988	    -8.237     8.369

  Sigma ASC Bus	                   -0.0380	     5.384	    -0.007	  0.994	   -10.591    10.515

  Sigma ASC Car	                    0.3973	     2.326	     0.171	  0.864	    -4.162     4.957

From the estimation results above, we can see that $\sigma_{\textrm{bus}}$ is the smallest standard deviation. We will therefore constrain this $\sigma_j$ to zero and re-estimate the model. Note also that the values of $\sigma_j$ are reported as positive or negative. This is because PyLogit performed an unconstrained maximization, and the normal distribution is symmetric, so it doesn't matter if we show multiply by negative one or not.

```python
# Initialize a new model
heteroskedastic_model_1 = pl.create_choice_model(data=mode_df,
                                                 alt_id_col="mode",
                                                 obs_id_col="individual",
                                                 choice_col="choice",
                                                 specification=heteroskedastic_specification,
                                                 model_type="Mixed Logit",
                                                 names=heteroskedastic_names,
                                                 mixing_id_col="individual",
                                                 mixing_vars=mixing_variables)

# Create the initial values for the parameters
# being estimated. Note that we start at the MNL estimates
# and that we need to have initial variables for the
# standard deviation parameters that we'll estimate as well
# as for intercept for car (which we'll constrain to zero)
initial_values = np.concatenate((mnl_model.params.values[:3],
                                 np.zeros(1),  # for the car intercept
                                 mnl_model.params.values[3:],
                                 np.zeros(len(mixing_variables))))

# Estimate the Mixed logit model.
# We set a seed for reproducibility, and we constrain the parameter
# in position 4, i.e. the intercept for the car utility, and in
# position -2, i.e. the standard deviation for the bus.
heteroskedastic_model_1.fit_mle(initial_values,
                                seed=26,
                                num_draws=500,
                                constrained_pos=[3, -2])

# View the estimation results
heteroskedastic_model_1.get_statsmodels_summary()
```

    Log-likelihood at zero: -291.1218
    Initial Log-likelihood: -199.1284
    Estimation Time: 4.72 seconds.
    Final log-likelihood: -195.6216

Mixed Logit Model Regression Results

  Dep. Variable:	      choice	         No. Observations:  	    210

  Model:	         Mixed Logit Model	   Df Residuals:      	    199

  Method:	               MLE	          Df Model:          	    11

  Date:	          Sat, 19 Nov 2016	    Pseudo R-squ.:     	   0.328

  Time:	              19:01:27	        Pseudo R-bar-squ.: 	   0.290

  converged:	           False	         Log-Likelihood:    	 -195.622

   	                       	           LL-Null:           	 -291.122

                	                  coef	     std err	      z	      P>|z|	 [95.0% Conf. Int.]

  ASC Air	                          4.6678	     1.598	     2.922	  0.003	     1.536     7.799

  ASC Train	                        5.4416	     1.180	     4.612	  0.000	     3.129     7.754

  ASC Bus	                          4.4082	     1.121	     3.931	  0.000	     2.210     6.606

  ASC Car	                               0	       nan	       nan	    nan	       nan       nan

  Generalized Cost (all modes)	    -0.0353	     0.011	    -3.168	  0.002	    -0.057    -0.013

  Terminal Time (all modes)	       -0.1184	     0.021	    -5.748	  0.000	    -0.159    -0.078

  Household Income (air mode)	      0.0415	     0.027	     1.527	  0.127	    -0.012     0.095

  Sigma ASC Air	                   -3.6390	     1.099	    -3.312	  0.001	    -5.793    -1.485

  Sigma ASC Train	                 -0.1010	     3.871	    -0.026	  0.979	    -7.688     7.486

  Sigma ASC Bus	                         0	       nan	       nan	    nan	       nan       nan

  Sigma ASC Car	                   -0.6942	     1.569	    -0.443	  0.658	    -3.769     2.380

Given that the estimated $\sigma_{\textrm{train}}$ and $\sigma_{\textrm{car}}$ parameters are so close to zero and have such high p-values, we eliminate those variables only retain $\sigma_{\textrm{air}}$.

```python
# Initialize a new model. Note we use the original specification and
# original name dictionary because we are not estimating a Sigma ASC Car
# and those dictionaries already contain all the needed information to
# estimate Sigma ASC Air
heteroskedastic_model_2 = pl.create_choice_model(data=mode_df,
                                                 alt_id_col="mode",
                                                 obs_id_col="individual",
                                                 choice_col="choice",
                                                 specification=specification_dict,
                                                 model_type="Mixed Logit",
                                                 names=name_dict,
                                                 mixing_id_col="individual",
                                                 mixing_vars=["ASC Air"])

# Create the initial values for the parameters
# being estimated. Note that we start at the estimates
# from the last model, using the estimated ASC's (minus the car ASC),
# the level-of-service coefficients, and Sigma Air.
initial_values = np.concatenate((heteroskedastic_model_1.params.values[:3],
                                 heteroskedastic_model_1.params.values[4:8]))

# Estimate the Mixed logit model. The seed is for reproducibility
heteroskedastic_model_2.fit_mle(initial_values,
                                seed=26,
                                num_draws=500)

# View the estimation results
heteroskedastic_model_2.get_statsmodels_summary()
```

    Log-likelihood at zero: -291.1218
    Initial Log-likelihood: -195.8153
    Estimation Time: 1.53 seconds.
    Final log-likelihood: -195.6975

Mixed Logit Model Regression Results

  Dep. Variable:	      choice	         No. Observations:  	    210

  Model:	         Mixed Logit Model	   Df Residuals:      	    203

  Method:	               MLE	          Df Model:          	     7

  Date:	          Sat, 19 Nov 2016	    Pseudo R-squ.:     	   0.328

  Time:	              19:01:28	        Pseudo R-bar-squ.: 	   0.304

  converged:	           True	          Log-Likelihood:    	 -195.697

   	                       	           LL-Null:           	 -291.122

                	                  coef	     std err	      z	      P>|z|	 [95.0% Conf. Int.]

  ASC Air	                          4.6406	     1.255	     3.698	  0.000	     2.181     7.100

  ASC Train	                        5.1455	     0.671	     7.670	  0.000	     3.831     6.460

  ASC Bus	                          4.1621	     0.626	     6.646	  0.000	     2.935     5.389

  Generalized Cost (all modes)	    -0.0326	     0.007	    -4.698	  0.000	    -0.046    -0.019

  Terminal Time (all modes)	       -0.1142	     0.012	    -9.664	  0.000	    -0.137    -0.091

  Household Income (air mode)	      0.0372	     0.025	     1.508	  0.132	    -0.011     0.085

  Sigma ASC Air	                   -3.3096	     0.841	    -3.934	  0.000	    -4.959    -1.661

# Estimate a Mixed Logit Model with Nesting (i.e. correlated error terms)
In certain cases, one might wish to allow for correlated utilities. In particular, we use the $\epsilon_{ij}$ terms to represent the unobserved factors that affect the utility gained by an individual $i$ if they choose alternative $j$. It may be quite reasonable to expect the unobserved factors that contribute to the utility of alternative $j$ (say the Train alternative) to be correlated with the unobserved factors that contribute to the utility of alternative $k$ (say the Bus alternative). For instance, we do not observe factors such as the access and egress time to the terminals for the train and bus modes. Given that these are ground transportation modes that require smaller terminals than airplanes, it is likely that the train and bus terminals are close to the center of downtown Sydney and downtown Melbourne, and thus likely to be closer to one's destination as opposed to the outskirts of the cities. Since these unobserved factors are likely to be correlated, the utilities of the two modes are likely to be correlated.

One way to account for such correlation is through the addition of shared error components in one's utility equations. In our example specifically, we can add a error component to the train and bus utilities as follows:
$\begin{align}
U_{ij} &= \beta _{\textrm{generalized_cost}} \textrm{GC}_{ij} +
          \beta _{\textrm{terminal_time}} \textrm{TTME}_{ij} +
          \beta _{\textrm{household_income}} d_{i,\ j=\textrm{air}} \textrm{HINC}_{i} +\\
          &\quad \ \sigma_{\textrm{air}} d_{i,\ j=\textrm{air}} \tilde{\eta}_{ij} + \sigma_{\textrm{nest}} d_{i,\ j \in \textrm{(train, bus)}} \xi + \epsilon_{ij}\\
\textrm{where }& \tilde{\eta}_{ij} \sim N\left(\alpha _{j}, 1\right)\\
          &j \in \left\lbrace \textrm{air, train, bus, car}\right\rbrace\\
          &\alpha _{\textrm{car}} = 0 \\
          & \xi \sim N\left( 0, 1 \right)
\end{align}
$
Note that we have removed the $\sigma_j$ parameters for all $j \neq \textrm{air}$ because of the estimation results from above.

As before, we will need to implement this specification through a random-coefficients perspective, so we will add a dummy variable to the model that is $1$ if the alternative is train or bus, and $0$ otherwise. Then, when treating the coefficient on this dummy variable as a random coefficient, we will constrain the mean to be zero, and estimate the standard deviation. This is analagous to the way that we estimated the standard deviation for the car alternative, even though the "ASC Car" coefficient was constrained to be zero.

```python
# Add a "nesting variable" to the dataset by first
# initializing the variable for all modes, then
# setting the veriable equal to 1, only for the
# train (2) and bus(3) modes
mode_df["train_bus_nesting_dummy"] = 0
mode_df.loc[mode_df["mode"].isin([2, 3]),
            "train_bus_nesting_dummy"] = 1

# Initialize the specification and name dictionaries
# for the model to be estimated.
nesting_specification = OrderedDict()
nesting_names = OrderedDict()

# As with 'heteroskedastic model 2', we build off the
# original specification and name dictionaries because
# we are not estimating a value for Sigma ASC Car.
for key in specification_dict:
    nesting_specification[key] = specification_dict[key]
    nesting_names[key] = name_dict[key]

# Note that we are adding a single subset-specific coefficient to the
# nesting dummy. This coefficient (which will be constrained to zero), will
# be the mean of a randomly distributed nesting variable.
nesting_specification["train_bus_nesting_dummy"] = [[2, 3]]
nesting_names["train_bus_nesting_dummy"] = ["Nesting Parameter"]
```

```python
# Create a list of the variables whose standard deviations
# are to be estimated
mixing_vars_nesting = ["ASC Air", "Nesting Parameter"]

nesting_model = pl.create_choice_model(data=mode_df,
                                       alt_id_col="mode",
                                       obs_id_col="individual",
                                       choice_col="choice",
                                       specification=nesting_specification,
                                       model_type="Mixed Logit",
                                       names=nesting_names,
                                       mixing_id_col="individual",
                                       mixing_vars=mixing_vars_nesting)

# Create the initial values for the parameters
# being estimated.
initial_values = np.concatenate((heteroskedastic_model_2.params.values[:6],
                                 np.zeros(1),  # for the nesting dummy
                                 heteroskedastic_model_2.params.values[-1:],
                                 np.zeros(1)))  # for the Sigma Nesting Dummy

# Estimate the Mixed logit model.
# We set a seed for reproducibility, and we constrain the parameter
# in position 7, i.e. the nesting dummy.
nesting_model.fit_mle(initial_values,
                     seed=28,
                     num_draws=500,
                     constrained_pos=[6])

# View the estimation results
nesting_model.get_statsmodels_summary()
```

    Log-likelihood at zero: -291.1218
    Initial Log-likelihood: -195.5820
    Estimation Time: 1.38 seconds.
    Final log-likelihood: -195.4987

Mixed Logit Model Regression Results

  Dep. Variable:	      choice	         No. Observations:  	    210

  Model:	         Mixed Logit Model	   Df Residuals:      	    201

  Method:	               MLE	          Df Model:          	     9

  Date:	          Sat, 19 Nov 2016	    Pseudo R-squ.:     	   0.328

  Time:	              19:01:30	        Pseudo R-bar-squ.: 	   0.298

  converged:	           True	          Log-Likelihood:    	 -195.499

   	                       	           LL-Null:           	 -291.122

                	                  coef	     std err	      z	      P>|z|	 [95.0% Conf. Int.]

  ASC Air	                          4.7379	     1.422	     3.333	  0.001	     1.952     7.524

  ASC Train	                        5.2137	     0.945	     5.519	  0.000	     3.362     7.065

  ASC Bus	                          4.2274	     0.911	     4.639	  0.000	     2.441     6.013

  Generalized Cost (all modes)	    -0.0329	     0.009	    -3.463	  0.001	    -0.051    -0.014

  Terminal Time (all modes)	       -0.1157	     0.018	    -6.307	  0.000	    -0.152    -0.080

  Household Income (air mode)	      0.0358	     0.025	     1.411	  0.158	    -0.014     0.085

  Nesting Parameter	                     0	       nan	       nan	    nan	       nan       nan

  Sigma ASC Air	                   -3.3560	     1.016	    -3.305	  0.001	    -5.346    -1.365

  Sigma Nesting Parameter	          0.4213	     1.863	     0.226	  0.821	    -3.230     4.072

As the estimation results show, $\sigma_{\textrm{Nesting Parameter}}$ is not significantly different from zero at any common alpha-level. This indicates that the utilities of the train and bus coefficients may not actually be correlated, as we might have guessed a priori. This result could have been anticipated by considering the fact that this model is a restricted version of the "Heteroskedastic Model 1" that was estimated earlier. Here we are restricting $\sigma_{\textrm{train}}$ and $\sigma_{\textrm{bus}}$ terms to be the same value, and we are calling this value $\sigma_{\textrm{Nesting Parameter}}$. Given that neither parameter in the unrestricted model had a statistically significant difference from zero, we could anticipate that the restricted model would also not result in a parameter estimate that had a statistically significant difference from zero.

examples/notebooks/More Mixed Logit--Heteroskedasticity and Nesting.ipynb


.. meta::
   :description: A guide which introduces the most important steps to get started with pymoo, an open-source multi-objective optimization framework in Python.

.. meta::
   :keywords: Multi-objective Optimization, Python, Evolutionary Computation, Optimization Test Problem, Hypervolume

# Part I: A Constrained Bi-objective Optimization Problem

In the following, we investigate exemplarily a bi-objective optimization with two constraints.
We have tried to select a suitable optimization problem with enough complexity for demonstration purposes, but not too difficult to lose track of the overall idea. Its definition is given by:

As already discussed in the Preface, let an optimization problem be defined by:

\begin{align}
\begin{split}
\min \quad& f_{m}(x) \quad \quad \quad \quad m = 1,..,M  \\[4pt]
\text{s.t.}   \quad& g_{j}(x) \leq 0  \quad \; \; \,  \quad j = 1,..,J \\[2pt]
\quad& h_{k}(x) = 0        \quad  \; \; \quad k = 1,..,K \\[4pt]
\quad& x_{i}^{L} \leq x_{i} \leq x_{i}^{U}  \quad i = 1,..,N \\[2pt]
\quad& x \in \Omega
\end{split}
\end{align}

The the example problem to be solved in this getting started guide is given by:

\begin{align}
\begin{split}
\min \;\; & f_1(x) = 100 \,(x_1^2 + x_2^2) \\
\max \;\; & f_2(x) = -(x_1-1)^2 - x_2^2 \\[1mm]
\text{s.t.} \;\; & g_1(x) = 2 \, (x_1 - 0.1) \, (x_1 - 0.9) \leq 0\\
& g_2(x) = 20 \, (x_1 - 0.4) \, (x_1 - 0.6) \geq 0\\[1mm]
& -2 \leq x_1 \leq 2 \\
& -2 \leq x_2 \leq 2\\[1mm]
& x \in \mathbb{R}
\end{split}
\end{align}

The problem consists of two objectives ($M=2$) where $f_1(x)$ is minimized and $f_2(x)$ maximized. The optimization is subject to two inequality constraints ($J=2$) where $g_1(x)$ is formulated as a less than and $g_2(x)$ as a greater than constraint. The problem is defined with respect to two variables ($N=2$), $x_1$ and $x_2$, both in the range $[-2,2]$. The problem does not contain any equality constraints ($K=0$).

Let us analyze where the Pareto-optimal solutions have to lie. The first objective $f_1$ is minimized at $(0,0)$, whereas the second object $f_2$ at $(1, 0)$. Because both functions are of quadratic nature, the optimum is given by a straight line between the two optima. This means all Pareto-optimal solutions (ignoring the constraints for now) have in common that $x_2=0$ and $x_1 \in (0,1)$. The first constraint only relies on $x_1$ and is satisfied if $x_1 \in (0.1,0.9)$. The second constraint $g_2$ is satisfied for $x_1 \in (0.4,0.6)$.
This means analytically, the pareto-optimal set is  given by $PS = \{(x_1, x_2) \,|\, (0.1 \leq x_1 \leq 0.4) \lor (0.6 \leq x_1 \leq 0.9) \, \land \, x_2 = 0\}$.

The figure below shows the problem's functions in the design space and might help to see the relationship between the functions intuitively. The contour lines of the objective function $f_1(x)$ is represented by a solid and $f_2(x)$ by a dashed line. The constraints $g_1(x)$ and $g_2(x)$ are parabolas which intersect the $x_1$-axis at $(0.1, 0.9)$ and $(0.4, 0.6)$. A thick orange line illustrates the Pareto-optimal set. When considering both constraints together, the Pareto-set shown in orange is split into two parts as analytically derived above.

```python
import numpy as np

X1, X2 = np.meshgrid(np.linspace(-2, 2, 500), np.linspace(-2, 2, 500))

F1 = X1**2 + X2**2
F2 = (X1-1)**2 + X2**2
G = X1**2 - X1 + 3/16

G1 = 2 * (X1[0] - 0.1) * (X1[0] - 0.9)
G2 = 20 * (X1[0] - 0.4) * (X1[0] - 0.6)

import matplotlib.pyplot as plt
plt.rc('font', family='serif')

levels = [0.02, 0.1, 0.25, 0.5, 0.8]
plt.figure(figsize=(7, 5))
CS = plt.contour(X1, X2, F1, levels, colors='black', alpha=0.5)
CS.collections[0].set_label("$f_1(x)$")

CS = plt.contour(X1, X2, F2, levels, linestyles="dashed", colors='black', alpha=0.5)
CS.collections[0].set_label("$f_2(x)$")

plt.plot(X1[0], G1, linewidth=2.0, color="green", linestyle='dotted')
plt.plot(X1[0][G1<0], G1[G1<0], label="$g_1(x)$", linewidth=2.0, color="green")

plt.plot(X1[0], G2, linewidth=2.0, color="blue", linestyle='dotted')
plt.plot(X1[0][X1[0]>0.6], G2[X1[0]>0.6], label="$g_2(x)$",linewidth=2.0, color="blue")
plt.plot(X1[0][X1[0]<0.4], G2[X1[0]<0.4], linewidth=2.0, color="blue")

plt.plot(np.linspace(0.1,0.4,100), np.zeros(100),linewidth=3.0, color="orange")
plt.plot(np.linspace(0.6,0.9,100), np.zeros(100),linewidth=3.0, color="orange")

plt.xlim(-0.5, 1.5)
plt.ylim(-0.5, 1)
plt.xlabel("$x_1$")
plt.ylabel("$x_2$")

plt.legend(loc='upper center', bbox_to_anchor=(0.5, 1.12),
          ncol=4, fancybox=True, shadow=False)

plt.tight_layout()
plt.show()
```

Next, we derive the Pareto-front by mapping the Pareto-set to the objective space. The Pareto-front equation is based on $f_2$ depending on the variable of $f_1$. We know that at the optimum $x_2=0$ which means we can simplify the objective functions to $f_1(x) = 100 \; x_1^2$ and $f_2(x) = -(x_1-1)^2$. The first objective $f_1$ can be reformulated to $x_1 = \sqrt{\frac{f_1}{100}}$ and then be put into the second objective which results in

$$f_2 = -\left(\sqrt{\frac{f_1}{100}}-1\right)^2$$

The equation defines the shape, however, next all possible values for $f_1$ need to be defined. As shown before the Pareto-set is defined for $(0.1 \leq x_1 \leq 0.4) \lor (0.6 \leq x_1 \leq 0.9) \, \land \, x_2 = 0$.
If we plug in the values for $x_1$ into $f_1$ we get the points of interest $[1,16]$ and $[36,81]$.
Thus the Pareto-front is given by:

```python
import numpy as np
import matplotlib.pyplot as plt

plt.figure(figsize=(7, 5))

f2 = lambda f1: - ((f1/100) ** 0.5 - 1)**2
F1_a, F1_b = np.linspace(1, 16, 300), np.linspace(36, 81, 300)
F2_a, F2_b = f2(F1_a), f2(F1_b)

plt.rc('font', family='serif')
plt.plot(F1_a,F2_a, linewidth=2.0, color="green", label="Pareto-front")
plt.plot(F1_b,F2_b, linewidth=2.0, color="green")

plt.xlabel("$f_1$")
plt.ylabel("$f_2$")

plt.legend(loc='upper center', bbox_to_anchor=(0.5, 1.10),
          ncol=4, fancybox=True, shadow=False)

plt.tight_layout()
plt.show()
```

As a quick check, we shall verify if this is a non-dominated set. Keeping in mind that the first objective is minimized and the second maximized for this optimization problem, a better solution lies on the top left. This means the derived Pareto-front makes sense.


Text provided under a Creative Commons Attribution license, CC-BY.  All code is made available under the FSF-approved MIT license.  (c) Lorena A. Barba, 2013. Thanks: Gilbert Forsyth for help writing the notebooks. NSF for support via CAREER award #1149784.
[@LorenaABarba](https://twitter.com/LorenaABarba)

12 steps to Navier-Stokes
=====
***

We continue our journey to solve the Navier-Stokes equation with Step 4. But don't continue unless you have completed the previous steps! In fact, this next step will be a combination of the two previous ones. The wonders of *code reuse*!

Step 4: Burgers' Equation
----
***

You can read about Burgers' Equation on its [wikipedia page](http://en.wikipedia.org/wiki/Burgers'_equation).

Burgers' equation in one spatial dimension looks like this:

$$\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = \nu \frac{\partial ^2u}{\partial x^2}$$

As you can see, it is a combination of non-linear convection and diffusion. It is surprising how much you learn from this neat little equation!

We can discretize it using the methods we've already detailed in Steps [1](http://nbviewer.ipython.org/urls/github.com/barbagroup/CFDPython/blob/master/lessons/01_Step_1.ipynb) to [3](http://nbviewer.ipython.org/urls/github.com/barbagroup/CFDPython/blob/master/lessons/04_Step_3.ipynb).  Using forward difference for time, backward difference for space and our 2nd-order method for the second derivatives yields:

$$\frac{u_i^{n+1}-u_i^n}{\Delta t} + u_i^n \frac{u_i^n - u_{i-1}^n}{\Delta x} = \nu \frac{u_{i+1}^n - 2u_i^n + u_{i-1}^n}{\Delta x^2}$$

As before, once we have an initial condition, the only unknown is $u_i^{n+1}$. We will step in time as follows:

$$u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)$$

###Initial and Boundary Conditions

To examine some interesting properties of Burgers' equation, it is helpful to use different initial and boundary conditions than we've been using for previous steps.

Our initial condition for this problem is going to be:

\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi &=& \exp \bigg(\frac{-x^2}{4 \nu} \bigg) + \exp \bigg(\frac{-(x-2 \pi)^2}{4 \nu} \bigg)
\end{eqnarray}

This has an analytical solution, given by:

\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi &=& \exp \bigg(\frac{-(x-4t)^2}{4 \nu (t+1)} \bigg) + \exp \bigg(\frac{-(x-4t -2 \pi)^2}{4 \nu(t+1)} \bigg)
\end{eqnarray}

Our boundary condition will be:

$$u(0) = u(2\pi)$$

This is called a *periodic* boundary condition. Pay attention! This will cause you a bit of headache if you don't tread carefully.

###Saving Time with SymPy

The initial condition we're using for Burgers' Equation can be a bit of a pain to evaluate by hand.  The derivative $\frac{\partial \phi}{\partial x}$ isn't too terribly difficult, but it would be easy to drop a sign or forget a factor of $x$ somewhere, so we're going to use SymPy to help us out.

[SymPy](http://sympy.org/en/) is the symbolic math library for Python.  It has a lot of the same symbolic math functionality as Mathematica with the added benefit that we can easily translate its results back into our Python calculations (it is also free and open source).

Start by loading the SymPy library, together with our favorite library, NumPy.

```
import numpy as np
import sympy
```

We're also going to tell SymPy that we want all of its output to be rendered using $\LaTeX$. This will make our Notebook beautiful!

```
from sympy import init_printing
init_printing(use_latex=True)
```

Start by setting up symbolic variables for the three variables in our initial condition and then type out the full equation for $\phi$.  We should get a nicely rendered version of our $\phi$ equation.

```
x, nu, t = sympy.symbols('x nu t')
phi = sympy.exp(-(x-4*t)**2/(4*nu*(t+1))) + sympy.exp(-(x-4*t-2*np.pi)**2/(4*nu*(t+1)))
phi
```

It's maybe a little small, but that looks right.  Now to evaluate our partial derivative $\frac{\partial \phi}{\partial x}$ is a trivial task.

```
phiprime = phi.diff(x)
phiprime
```

If you want to see the unrendered version, just use the Python print command.

```
print phiprime
```

    -(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 12.5663706143592)*exp(-(-4*t + x - 6.28318530717959)**2/(4*nu*(t + 1)))/(4*nu*(t + 1))

###Now what?

Now that we have the Pythonic version of our derivative, we can finish writing out the full initial condition equation and then translate it into a usable Python expression.  For this, we'll use the *lambdify* function, which takes a SymPy symbolic equation and turns it into a callable function.

```
from sympy.utilities.lambdify import lambdify

u = -2*nu*(phiprime/phi)+4
print u
```

    -2*nu*(-(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 12.5663706143592)*exp(-(-4*t + x - 6.28318530717959)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)))/(exp(-(-4*t + x - 6.28318530717959)**2/(4*nu*(t + 1))) + exp(-(-4*t + x)**2/(4*nu*(t + 1)))) + 4

###Lambdify

To lambdify this expression into a useable function, we tell lambdify which variables to request and the function we want to plug them in to.

```
ufunc = lambdify((t, x, nu), u)
print ufunc(1,4,3)
```

    3.49170664206

###Back to Burgers' Equation

Now that we have the initial conditions set up, we can proceed and finish setting up the problem.  We can generate the plot of the initial condition using our lambdify-ed function.

```
import matplotlib.pyplot as plt
%matplotlib inline

###variable declarations
nx = 101
nt = 100
dx = 2*np.pi/(nx-1)
nu = .07
dt = dx*nu

x = np.linspace(0, 2*np.pi, nx)
#u = np.empty(nx)
un = np.empty(nx)
t = 0

u = np.asarray([ufunc(t, x0, nu) for x0 in x])
u
```

    array([ 4.        ,  4.06283185,  4.12566371,  4.18849556,  4.25132741,
            4.31415927,  4.37699112,  4.43982297,  4.50265482,  4.56548668,
            4.62831853,  4.69115038,  4.75398224,  4.81681409,  4.87964594,
            4.9424778 ,  5.00530965,  5.0681415 ,  5.13097336,  5.19380521,
            5.25663706,  5.31946891,  5.38230077,  5.44513262,  5.50796447,
            5.57079633,  5.63362818,  5.69646003,  5.75929189,  5.82212374,
            5.88495559,  5.94778745,  6.0106193 ,  6.07345115,  6.136283  ,
            6.19911486,  6.26194671,  6.32477856,  6.38761042,  6.45044227,
            6.51327412,  6.57610598,  6.63893783,  6.70176967,  6.76460125,
            6.82742866,  6.89018589,  6.95176632,  6.99367964,  6.72527549,
            4.        ,  1.27472451,  1.00632036,  1.04823368,  1.10981411,
            1.17257134,  1.23539875,  1.29823033,  1.36106217,  1.42389402,
            1.48672588,  1.54955773,  1.61238958,  1.67522144,  1.73805329,
            1.80088514,  1.863717  ,  1.92654885,  1.9893807 ,  2.05221255,
            2.11504441,  2.17787626,  2.24070811,  2.30353997,  2.36637182,
            2.42920367,  2.49203553,  2.55486738,  2.61769923,  2.68053109,
            2.74336294,  2.80619479,  2.86902664,  2.9318585 ,  2.99469035,
            3.0575222 ,  3.12035406,  3.18318591,  3.24601776,  3.30884962,
            3.37168147,  3.43451332,  3.49734518,  3.56017703,  3.62300888,
            3.68584073,  3.74867259,  3.81150444,  3.87433629,  3.93716815,  4.        ])

```
plt.figure(figsize=(11,7), dpi=100)
plt.plot(x,u, marker='o', lw=2)
plt.xlim([0,2*np.pi])
plt.ylim([0,10]);
```

This is definitely not the hat function we've been dealing with until now. We call it a "saw-tooth function".  Let's proceed forward and see what happens.

### Periodic Boundary Conditions

One of the big differences between Step 4 and the previous lessons is the use of *periodic* boundary conditions.  If you experiment with Steps 1 and 2 and make the simulation run longer (by increasing `nt`) you will notice that the wave will keep moving to the right until it no longer even shows up in the plot.

With periodic boundary conditions, when a point gets to the right-hand side of the frame, it *wraps around* back to the front of the frame.

Recall the discretization that we worked out at the beginning of this notebook:

$$u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)$$

What does $u_{i+1}^n$ *mean* when $i$ is already at the end of the frame?

Think about this for a minute before proceeding.

```
for n in range(nt):
    un = u.copy()
    for i in range(nx-1):
        u[i] = un[i] - un[i] * dt/dx *(un[i] - un[i-1]) + nu*dt/dx**2*\
                (un[i+1]-2*un[i]+un[i-1])
    u[-1] = un[-1] - un[-1] * dt/dx * (un[-1] - un[-2]) + nu*dt/dx**2*\
                (un[0]-2*un[-1]+un[-2])

u_analytical = np.asarray([ufunc(nt*dt, xi, nu) for xi in x])
```

```
plt.figure(figsize=(11,7), dpi=100)
plt.plot(x,u, marker='o', lw=2, label='Computational')
plt.plot(x, u_analytical, label='Analytical')
plt.xlim([0,2*np.pi])
plt.ylim([0,10])
plt.legend();
```

***

What next?
----

The subsequent steps, from 5 to 12, will be in two dimensions. But it is easy to extend the 1D finite-difference formulas to the partial derivatives in 2D or 3D. Just apply the definition — a partial derivative with respect to $x$ is the variation in the $x$ direction *while keeping $y$ constant*.

Before moving on to [Step 5](http://nbviewer.ipython.org/urls/github.com/barbagroup/CFDPython/blob/master/lessons/07_Step_5.ipynb), make sure you have completed your own code for steps 1 through 4 and you have experimented with the parameters and thought about what is happening. Also, we recommend that you take a slight break to learn about [array operations with NumPy](http://nbviewer.ipython.org/urls/github.com/barbagroup/CFDPython/blob/master/lessons/07_Step_5.ipynb).

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#CD2305;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #CD2305;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


```
#hide
!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
```

```
#hide
from fastai.gen_doc.nbdoc import *
```
[[chapter_foundations]]
# A Neural Net from the Foundations

This chapter begins a journey where we will dig deep into the internals of the models we used in the previous chapters. We will be covering many of the same things we've seen before, but this time around we'll be looking much more closely at the implementation details, and much less closely at the practical issues of how and why things are as they are.

We will build everything from scratch, only using basic indexing into a tensor. We]ll write a neural net from the ground up, then implement backpropagation manually, so we know exactly what's happening in PyTorch when we call `loss.backward`. We'll also see how to extend PyTorch with custom *autograd* functions that allow us to specify our own forward and backward computations.

## Building a Neural Net Layer from Scratch

Let's start by refreshing our understanding of how matrix multiplication is used in a basic neural network. Since we're building everything up from scratch, we'll use nothing but plain Python initially (except for indexing into PyTorch tensors), and then replace the plain Python with PyTorch functionality once we've seen how to create it.

### Modeling a Neuron

A neuron receives a given number of inputs and has an internal weight for each of them. It sums those weighted inputs to produce an output and adds an inner bias. In math, this can be written as:

$$ out = \sum_{i=1}^{n} x_{i} w_{i} + b$$

if we name our inputs $(x_{1},\dots,x_{n})$, our weights $(w_{1},\dots,w_{n})$, and our bias $b$. In code this translates into:

```python
output = sum([x*w for x,w in zip(inputs,weights)]) + bias
```

This output is then fed into a nonlinear function called an *activation function* before being sent to another neuron. In deep learning the most common of these is the *rectified Linear unit*, or *ReLU*, which, as we've seen, is a fancy way of saying:
```python
def relu(x): return x if x >= 0 else 0
```

A deep learning model is then built by stacking a lot of those neurons in successive layers. We create a first layer with a certain number of neurons (known as *hidden size*) and link all the inputs to each of those neurons. Such a layer is often called a *fully connected layer* or a *dense layer* (for densely connected), or a *linear layer*.

It requires to compute, for each `input` in our batch and each neuron with a give `weight`, the dot product:

```python
sum([x*w for x,w in zip(input,weight)])
```

If you have done a little bit of linear algebra, you may remember that having a lot of those dot products happens when you do a *matrix multiplication*. More precisely, if our inputs are in a matrix `x` with a size of `batch_size` by `n_inputs`, and if we have grouped the weights of our neurons in a matrix `w` of size `n_neurons` by `n_inputs` (each neuron must have the same number of weights as it has inputs) and all the biases in a vector `b` of size `n_neurons`, then the output of this fully connected layer is:

```python
y = x @ w.t() + b
```

where `@` represents the matrix product and `w.t()` is the transpose matrix of `w`. The output `y` is then of size `batch_size` by `n_neurons`, and in position `(i,j)` we have (for the mathy folks out there):

$$y_{i,j} = \sum_{k=1}^{n} x_{i,k} w_{k,j} + b_{j}$$

Or in code:

```python
y[i,j] = sum([a * b for a,b in zip(x[i,:],w[j,:])]) + b[j]
```

The transpose is necessary because in the mathematical definition of the matrix product `m @ n`, the coefficient `(i,j)` is:

```python
sum([a * b for a,b in zip(m[i,:],n[:,j])])
```

So the very basic operation we need is a matrix multiplication, as it's what is hidden in the core of a neural net.

### Matrix Multiplication from Scratch

Let's write a function that computes the matrix product of two tensors, before we allow ourselves to use the PyTorch version of it. We will only use the indexing in PyTorch tensors:

```
import torch
from torch import tensor
```

We'll need three nested `for` loops: one for the row indices, one for the column indices, and one for the inner sum. `ac` and `ar` stand for number of columns of `a` and number of rows of `a`, respectively (the same convention is followed for `b`), and we make sure calculating the matrix product is possible by checking that `a` has as many columns as `b` has rows:

```
def matmul(a,b):
    ar,ac = a.shape # n_rows * n_cols
    br,bc = b.shape
    assert ac==br
    c = torch.zeros(ar, bc)
    for i in range(ar):
        for j in range(bc):
            for k in range(ac): c[i,j] += a[i,k] * b[k,j]
    return c
```

To test this out, we'll pretend (using random matrices) that we're working with a small batch of 5 MNIST images, flattened into 28×28 vectors, with linear model to turn them into 10 activations:

```
m1 = torch.randn(5,28*28)
m2 = torch.randn(784,10)
```

Let's time our function, using the Jupyter "magic" command `%time`:

```
%time t1=matmul(m1, m2)
```

    CPU times: user 1.15 s, sys: 4.09 ms, total: 1.15 s
    Wall time: 1.15 s

And see how that compares to PyTorch's built-in `@`:

```
%timeit -n 20 t2=m1@m2
```

    14 µs ± 8.95 µs per loop (mean ± std. dev. of 7 runs, 20 loops each)

As we can see, in Python three nested loops is a very bad idea! Python is a slow language, and this isn't going to be very efficient. We see here that PyTorch is around 100,000 times faster than Python—and that's before we even start using the GPU!

Where does this difference come from? PyTorch didn't write its matrix multiplication in Python, but rather in C++ to make it fast. In general, whenever we do computations on tensors we will need to *vectorize* them so that we can take advantage of the speed of PyTorch, usually by using two techniques: elementwise arithmetic and broadcasting.

### Elementwise Arithmetic

All the basic operators (`+`, `-`, `*`, `/`, `>`, `<`, `==`) can be applied elementwise. That means if we write `a+b` for two tensors `a` and `b` that have the same shape, we will get a tensor composed of the sums the elements of `a` and `b`:

```
a = tensor([10., 6, -4])
b = tensor([2., 8, 7])
a + b
```

    tensor([12., 14.,  3.])

The Booleans operators will return an array of Booleans:

```
a < b
```

    tensor([False,  True,  True])

If we want to know if every element of `a` is less than the corresponding element in `b`, or if two tensors are equal, we need to combine those elementwise operations with `torch.all`:

```
(a < b).all(), (a==b).all()
```

    (tensor(False), tensor(False))

Reduction operations like `all()`, `sum()` and `mean()` return tensors with only one element, called rank-0 tensors. If you want to convert this to a plain Python Boolean or number, you need to call `.item()`:

```
(a + b).mean().item()
```

    9.666666984558105

The elementwise operations work on tensors of any rank, as long as they have the same shape:

```
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
m*m
```

    tensor([[ 1.,  4.,  9.],
            [16., 25., 36.],
            [49., 64., 81.]])

However you can't perform elementwise operations on tensors that don't have the same shape (unless they are broadcastable, as discussed in the next section):

```
n = tensor([[1., 2, 3], [4,5,6]])
m*n
```

With elementwise arithmetic, we can remove one of our three nested loops: we can multiply the tensors that correspond to the `i`-th row of `a` and the `j`-th column of `b` before summing all the elements, which will speed things up because the inner loop will now be executed by PyTorch at C speed.

To access one column or row, we can simply write `a[i,:]` or `b[:,j]`. The `:` means take everything in that dimension. We could restrict this and take only a slice of that particular dimension by passing a range, like `1:5`, instead of just `:`. In that case, we would take the elements in columns or rows 1 to 4 (the second number is noninclusive).

One simplification is that we can always omit a trailing colon, so `a[i,:]` can be abbreviated to `a[i]`. With all of that in mind, we can write a new version of our matrix multiplication:

```
def matmul(a,b):
    ar,ac = a.shape
    br,bc = b.shape
    assert ac==br
    c = torch.zeros(ar, bc)
    for i in range(ar):
        for j in range(bc): c[i,j] = (a[i] * b[:,j]).sum()
    return c
```

```
%timeit -n 20 t3 = matmul(m1,m2)
```

    1.7 ms ± 88.1 µs per loop (mean ± std. dev. of 7 runs, 20 loops each)

We're already ~700 times faster, just by removing that inner `for` loop! And that's just the beginning—with broadcasting we can remove another loop and get an even more important speed up.

### Broadcasting

As we discussed in <<chapter_mnist_basics>>, broadcasting is a term introduced by the [NumPy library](https://docs.scipy.org/doc/) that describes how tensors of different ranks are treated during arithmetic operations. For instance, it's obvious there is no way to add a 3×3 matrix with a 4×5 matrix, but what if we want to add one scalar (which can be represented as a 1×1 tensor) with a matrix? Or a vector of size 3 with a 3×4 matrix? In both cases, we can find a way to make sense of this operation.

Broadcasting gives specific rules to codify when shapes are compatible when trying to do an elementwise operation, and how the tensor of the smaller shape is expanded to match the tensor of the bigger shape. It's essential to master those rules if you want to be able to write code that executes quickly. In this section, we'll expand our previous treatment of broadcasting to understand these rules.

#### Broadcasting with a scalar

Broadcasting with a scalar is the easiest type of broadcating. When we have a tensor `a` and a scalar, we just imagine a tensor of the same shape as `a` filled with that scalar and perform the operation:

```
a = tensor([10., 6, -4])
a > 0
```

    tensor([ True,  True, False])

How are we able to do this comparison? `0` is being *broadcast* to have the same dimensions as `a`. Note that this is done without creating a tensor full of zeros in memory (that would be very inefficient).

This is very useful if you want to normalize your dataset by subtracting the mean (a scalar) from the entire data set (a matrix) and dividing by the standard deviation (another scalar):

```
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
(m - 5) / 2.73
```

    tensor([[-1.4652, -1.0989, -0.7326],
            [-0.3663,  0.0000,  0.3663],
            [ 0.7326,  1.0989,  1.4652]])

What if have different means for each row of the matrix? in that case you will need to broadcast a vector to a matrix.

#### Broadcasting a vector to a matrix

We can broadcast a vector to a matrix as follows:

```
c = tensor([10.,20,30])
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
m.shape,c.shape
```

    (torch.Size([3, 3]), torch.Size([3]))

```
m + c
```

    tensor([[11., 22., 33.],
            [14., 25., 36.],
            [17., 28., 39.]])

Here the elements of `c` are expanded to make three rows that match, making the operation possible. Again, PyTorch doesn't actually create three copies of `c` in memory. This is done by the `expand_as` method behind the scenes:

```
c.expand_as(m)
```

    tensor([[10., 20., 30.],
            [10., 20., 30.],
            [10., 20., 30.]])

If we look at the corresponding tensor, we can ask for its `storage` property (which shows the actual contents of the memory used for the tensor) to check there is no useless data stored:

```
t = c.expand_as(m)
t.storage()
```

     10.0
     20.0
     30.0
    [torch.FloatStorage of size 3]

Even though the tensor officially has nine elements, only three scalars are stored in memory. This is possible thanks to the clever trick of giving that dimension a *stride* of 0 (which means that when PyTorch looks for the next row by adding the stride, it doesn't move):

```
t.stride(), t.shape
```

    ((0, 1), torch.Size([3, 3]))

Since `m` is of size 3×3, there are two ways to do broadcasting. The fact it was done on the last dimension is a convention that comes from the rules of broadcasting and has nothing to do with the way we ordered our tensors. If instead we do this, we get the same result:

```
c + m
```

    tensor([[11., 22., 33.],
            [14., 25., 36.],
            [17., 28., 39.]])

In fact, it's only possible to broadcast a vector of size `n` with a matrix of size `m` by `n`:

```
c = tensor([10.,20,30])
m = tensor([[1., 2, 3], [4,5,6]])
c+m
```

    tensor([[11., 22., 33.],
            [14., 25., 36.]])

This won't work:

```
c = tensor([10.,20])
m = tensor([[1., 2, 3], [4,5,6]])
c+m
```

If we want to broadcast in the other dimension, we have to change the shape of our vector to make it a 3×1 matrix. This is done with the `unsqueeze` method in PyTorch:

```
c = tensor([10.,20,30])
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
c = c.unsqueeze(1)
m.shape,c.shape
```

    (torch.Size([3, 3]), torch.Size([3, 1]))

This time, `c` is expanded on the column side:

```
c+m
```

    tensor([[11., 12., 13.],
            [24., 25., 26.],
            [37., 38., 39.]])

Like before, only three scalars are stored in memory:

```
t = c.expand_as(m)
t.storage()
```

     10.0
     20.0
     30.0
    [torch.FloatStorage of size 3]

And the expanded tensor has the right shape because the column dimension has a stride of 0:

```
t.stride(), t.shape
```

    ((1, 0), torch.Size([3, 3]))

With broadcasting, by default if we need to add dimensions, they are added at the beginning. When we were broadcasting before, Pytorch was doing `c.unsqueeze(0)` behind the scenes:

```
c = tensor([10.,20,30])
c.shape, c.unsqueeze(0).shape,c.unsqueeze(1).shape
```

    (torch.Size([3]), torch.Size([1, 3]), torch.Size([3, 1]))

The `unsqueeze` command can be replaced by `None` indexing:

```
c.shape, c[None,:].shape,c[:,None].shape
```

    (torch.Size([3]), torch.Size([1, 3]), torch.Size([3, 1]))

You can always omit trailing colons, and `...` means all preceding dimensions:

```
c[None].shape,c[...,None].shape
```

    (torch.Size([1, 3]), torch.Size([3, 1]))

With this, we can remove another `for` loop in our matrix multiplication function. Now, instead of multiplying `a[i]` with `b[:,j]`, we can multiply `a[i]` with the whole matrix `b` using broadcasting, then sum the results:

```
def matmul(a,b):
    ar,ac = a.shape
    br,bc = b.shape
    assert ac==br
    c = torch.zeros(ar, bc)
    for i in range(ar):
#       c[i,j] = (a[i,:]          * b[:,j]).sum() # previous
        c[i]   = (a[i  ].unsqueeze(-1) * b).sum(dim=0)
    return c
```

```
%timeit -n 20 t4 = matmul(m1,m2)
```

    357 µs ± 7.2 µs per loop (mean ± std. dev. of 7 runs, 20 loops each)

We're now 3,700 times faster than our first implementation! Before we move on, let's discuss the rules of broadcasting in a little more detail.

#### Broadcasting rules

When operating on two tensors, PyTorch compares their shapes elementwise. It starts with the *trailing dimensions* and works its way backward, adding 1 when it meets empty dimensions. Two dimensions are *compatible* when one of the following is true:

- They are equal.
- One of them is 1, in which case that dimension is broadcast to make it the same as the other.

Arrays do not need to have the same number of dimensions. For example, if you have a 256×256×3 array of RGB values, and you want to scale each color in the image by a different value, you can multiply the image by a one-dimensional array with three values. Lining up the sizes of the trailing axes of these arrays according to the broadcast rules, shows that they are compatible:

```
Image  (3d tensor): 256 x 256 x 3
Scale  (1d tensor):  (1)   (1)  3
Result (3d tensor): 256 x 256 x 3
```

However, a 2D tensor of size 256×256 isn't compatible with our image:

```
Image  (3d tensor): 256 x 256 x   3
Scale  (1d tensor):  (1)  256 x 256
Error
```

In our earlier examples we had with a 3×3 matrix and a vector of size 3, broadcasting was done on the rows:

```
Matrix (2d tensor):   3 x 3
Vector (1d tensor): (1)   3
Result (2d tensor):   3 x 3
```

As an exercise, try to determine what dimensions to add (and where) when you need to normalize a batch of images of size `64 x 3 x 256 x 256` with vectors of three elements (one for the mean and one for the standard deviation).

Another useful wat of simplifying tensor manipulations is the use of Einstein summations convention.

### Einstein Summation

Before using the PyTorch operation `@` or `torch.matmul`, there is one last way we can implement matrix multiplication: Einstein summation (`einsum`). This is a compact representation for combining products and sums in a general way. We write an equation like this:

```
ik,kj -> ij
```

The lefthand side represents the operands dimensions, separated by commas. Here we have two tensors that each have two dimensions (`i,k` and `k,j`).  The righthand side represents the result dimensions, so here we have a tensor with two dimensions `i,j`.

The rules of Einstein summation notation are as follows:

1. Repeated indices are implicitly summed over.
1. Each index can appear at most twice in any term.
1. Each term must contain identical nonrepeated indices.

So in our example, since `k` is repeated, we sum over that index. In the end the formula represents the matrix obtained when we put in `(i,j)` the sum of all the coefficients `(i,k)` in the first tensor multiplied by the coefficients `(k,j)` in the second tensor... which is the matrix product! Here is how we can code this in PyTorch:

```
def matmul(a,b): return torch.einsum('ik,kj->ij', a, b)
```

Einstein summation is a very practical way of expressing operations involving indexing and sum of products. Note that you can have just one member on the lefthand side. For instance, this:

```python
torch.einsum('ij->ji', a)
```

returns the transpose of the matrix `a`. You can also have three or more members. This:

```python
torch.einsum('bi,ij,bj->b', a, b, c)
```

will return a vector of size `b` where the `k`-th coordinate is the sum of `a[k,i] b[i,j] c[k,j]`. This notation is particularly convenient when you have more dimensions because of batches. For example, if you have two batches of matrices and want to compute the matrix product per batch, you would could this:

```python
torch.einsum('bik,bkj->bij', a, b)
```

Let's go back to our new `matmul` implementation using `einsum` and look at its speed:

```
%timeit -n 20 t5 = matmul(m1,m2)
```

    68.7 µs ± 4.06 µs per loop (mean ± std. dev. of 7 runs, 20 loops each)

As you can see, not only is it practical, but it's *very* fast. `einsum` is often the fastest way to do custom operations in PyTorch, without diving into C++ and CUDA. (But it's generally not as fast as carefully optimized CUDA code, as you see from the results in "Matrix Multiplication from Scratch".)

Now that we know how to implement a matrix multiplication from scratch, we are ready to build our neural net—specifically its forward and backward passes—using just matrix multiplications.

## The Forward and Backward Passes

As we saw in <<chapter_mnist_basics>>, to train a model, we will need to compute all the gradients of a given a loss with respect to its parameters, which is known as the *backward pass*. The *forward pass* is where we compute the output of the model on a given input, based on the matrix products. As we define our first neural net, we will also delve into the problem of properly initializing the weights, which is crucial for making training start properly.

### Defining and Initializing a Layer

We will take the example of a two-layer neural net first. As we've seen, one layer can be expressed as `y = x @ w + b`, with `x` our inputs, `y` our outputs, `w` the weights of the layer (which is of size number of inputs by number of neurons if we don't transpose like before), and `b` is the bias vector:

```
def lin(x, w, b): return x @ w + b
```

We can stack the second layer on top of the first, but since mathematically the composition of two linear operations is another linear operation, this only makes sense if we put something nonlinear in the middle, called an activation function. As mentioned at the beginning of the chapter, in deep learning applications the activation function most commonly used is a ReLU, which returns the maximum of `x` and `0`.

We won't actually train our model in this chapter, so we'll use random tensors for our inputs and targets. Let's say our inputs are 200 vectors of size 100, which we group into one batch, and our targets are 200 random floats:

```
x = torch.randn(200, 100)
y = torch.randn(200)
```

For our two-layer model we will need two weight matrices and two bias vectors. Let's say we have a hidden size of 50 and the output size is 1 (for one of our inputs, the corresponding output is one float in this toy example). We initialize the weights randomly and the bias at zero:

```
w1 = torch.randn(100,50)
b1 = torch.zeros(50)
w2 = torch.randn(50,1)
b2 = torch.zeros(1)
```

Then the result of our first layer is simply:

```
l1 = lin(x, w1, b1)
l1.shape
```

    torch.Size([200, 50])

Note that this formula works with our batch of inputs, and returns a batch of hidden state: `l1` is a matrix of size 200 (our batch size) by 50 (our hidden size).

There is a problem with the way our model was initialized, however. To understand it, we need to look at the mean and standard deviation (std) of `l1`:

```
l1.mean(), l1.std()
```

    (tensor(0.0019), tensor(10.1058))

The mean is close to zero, which is understandable since both our input and weight matrices have means close to zero. But the standard deviation, which represents how far away our activations go from the mean, went from 1 to 10. This is a really big problem because that's with just one layer. Modern neural nets can have hundred of layers, so if each of them multiplies the scale of our activations by 10, by the end of the last layer we won't have numbers representable by a computer.

Indeed, if we make just 50 multiplications between `x` and random matrices of size 100×100, we'll have:

```
x = torch.randn(200, 100)
for i in range(50): x = x @ torch.randn(100,100)
x[0:5,0:5]
```

    tensor([[nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan],
            [nan, nan, nan, nan, nan]])

The result is `nan`s everywhere. So maybe the scale of our matrix was too big, and we need to have smaller weights? But if we use too small weights, we will have the opposite problem—the scale of our activations will go from 1 to 0.1, and after 100 layers we'll be left with zeros everywhere:

```
x = torch.randn(200, 100)
for i in range(50): x = x @ (torch.randn(100,100) * 0.01)
x[0:5,0:5]
```

    tensor([[0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.]])

So we have to scale our weight matrices exactly right so that the standard deviation of our activations stays at 1. We can compute the exact value to use mathematically, as illustrated by Xavier Glorot and Yoshua Bengio in ["Understanding the Difficulty of Training Deep Feedforward Neural Networks"](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf). The right scale for a given layer is $1/\sqrt{n_{in}}$, where $n_{in}$ represents the number of inputs.

In our case, if we have 100 inputs, we should scale our weight matrices by 0.1:

```
x = torch.randn(200, 100)
for i in range(50): x = x @ (torch.randn(100,100) * 0.1)
x[0:5,0:5]
```

    tensor([[ 0.7554,  0.6167, -0.1757, -1.5662,  0.5644],
            [-0.1987,  0.6292,  0.3283, -1.1538,  0.5416],
            [ 0.6106,  0.2556, -0.0618, -0.9463,  0.4445],
            [ 0.4484,  0.7144,  0.1164, -0.8626,  0.4413],
            [ 0.3463,  0.5930,  0.3375, -0.9486,  0.5643]])

Finally some numbers that are neither zeros nor `nan`s! Notice how stable the scale of our activations is, even after those 50 fake layers:

```
x.std()
```

    tensor(0.7042)

If you play a little bit with the value for scale you'll notice that even a slight variation from 0.1 will get you either to very small or very large numbers, so initializing the weights properly is extremely important.

Let's go back to our neural net. Since we messed a bit with our inputs, we need to redefine them:

```
x = torch.randn(200, 100)
y = torch.randn(200)
```

And for our weights, we'll use the right scale, which is known as *Xavier initialization* (or *Glorot initialization*):

```
from math import sqrt
w1 = torch.randn(100,50) / sqrt(100)
b1 = torch.zeros(50)
w2 = torch.randn(50,1) / sqrt(50)
b2 = torch.zeros(1)
```

Now if we compute the result of the first layer, we can check that the mean and standard deviation are under control:

```
l1 = lin(x, w1, b1)
l1.mean(),l1.std()
```

    (tensor(-0.0050), tensor(1.0000))

Very good. Now we need to go through a ReLU, so let's define one. A ReLU removes the negatives and replaces them with zeros, which is another way of saying it clamps our tensor at zero:

```
def relu(x): return x.clamp_min(0.)
```

We pass our activations through this:

```
l2 = relu(l1)
l2.mean(),l2.std()
```

    (tensor(0.3961), tensor(0.5783))

And we're back to square one: the mean of our activations has gone to 0.4 (which is understandable since we removed the negatives) and the std went down to 0.58. So like before, after a few layers we will probably wind up with zeros:

```
x = torch.randn(200, 100)
for i in range(50): x = relu(x @ (torch.randn(100,100) * 0.1))
x[0:5,0:5]
```

    tensor([[0.0000e+00, 1.9689e-08, 4.2820e-08, 0.0000e+00, 0.0000e+00],
            [0.0000e+00, 1.6701e-08, 4.3501e-08, 0.0000e+00, 0.0000e+00],
            [0.0000e+00, 1.0976e-08, 3.0411e-08, 0.0000e+00, 0.0000e+00],
            [0.0000e+00, 1.8457e-08, 4.9469e-08, 0.0000e+00, 0.0000e+00],
            [0.0000e+00, 1.9949e-08, 4.1643e-08, 0.0000e+00, 0.0000e+00]])

This means our initialization wasn't right. Why? At the time Glorot and Bengio wrote their article, the popular activation in a neural net was the hyperbolic tangent (tanh, which is the one they used), and that initialization doesn't account for our ReLU. Fortunately, someone else has done the math for us and computed the right scale for us to use. In ["Delving Deep into Rectifiers: Surpassing Human-Level Performance"](https://arxiv.org/abs/1502.01852) (which we've seen before—it's the article that introduced the ResNet), Kaiming He et al. show that we should use the following scale instead: $\sqrt{2 / n_{in}}$, where $n_{in}$ is the number of inputs of our model. Let's see what this gives us:

```
x = torch.randn(200, 100)
for i in range(50): x = relu(x @ (torch.randn(100,100) * sqrt(2/100)))
x[0:5,0:5]
```

    tensor([[0.2871, 0.0000, 0.0000, 0.0000, 0.0026],
            [0.4546, 0.0000, 0.0000, 0.0000, 0.0015],
            [0.6178, 0.0000, 0.0000, 0.0180, 0.0079],
            [0.3333, 0.0000, 0.0000, 0.0545, 0.0000],
            [0.1940, 0.0000, 0.0000, 0.0000, 0.0096]])

That's better: our numbers aren't all zeroed this time. So let's go back to the definition of our neural net and use this initialization (which is named *Kaiming initialization* or *He initialization*):

```
x = torch.randn(200, 100)
y = torch.randn(200)
```

```
w1 = torch.randn(100,50) * sqrt(2 / 100)
b1 = torch.zeros(50)
w2 = torch.randn(50,1) * sqrt(2 / 50)
b2 = torch.zeros(1)
```

Let's look at the scale of our activations after going through the first linear layer and ReLU:

```
l1 = lin(x, w1, b1)
l2 = relu(l1)
l2.mean(), l2.std()
```

    (tensor(0.5661), tensor(0.8339))

Much better! Now that our weights are properly initialized, we can define our whole model:

```
def model(x):
    l1 = lin(x, w1, b1)
    l2 = relu(l1)
    l3 = lin(l2, w2, b2)
    return l3
```

This is the forward pass. Now all that's left to do is to compare our output to the labels we have (random numbers, in this example) with a loss function. In this case, we will use the mean squared error. (It's a toy problem, and this is the easiest loss function to use for what is next, computing the gradients.)

The only subtlety is that our outputs and targets don't have exactly the same shape—after going though the model, we get an output like this:

```
out = model(x)
out.shape
```

    torch.Size([200, 1])

To get rid of this trailing 1 dimension, we use the `squeeze` function:

```
def mse(output, targ): return (output.squeeze(-1) - targ).pow(2).mean()
```

And now we are ready to compute our loss:

```
loss = mse(out, y)
```

That's all for the forward pass—let's now look at the gradients.

### Gradients and the Backward Pass

We've seen that PyTorch computes all the gradients we need with a magic call to `loss.backward`, but let's explore what's happening behind the scenes.

Now comes the part where we need to compute the gradients of the loss with respect to all the weights of our model, so all the floats in `w1`, `b1`, `w2`, and `b2`. For this, we will need a bit of math—specifically the *chain rule*. This is the rule of calculus that guides how we can compute the derivative of a composed function:

$$(g \circ f)'(x) = g'(f(x)) f'(x)$$

> j: I find this notation very hard to wrap my head around, so instead I like to think of it as: if `y = g(u)` and `u=f(x)`; then `dy/dx = dy/du * du/dx`. The two notations mean the same thing, so use whatever works for you.

Our loss is a big composition of different functions: mean squared error (which is in turn the composition of a mean and a power of two), the second linear layer, a ReLU and the first linear layer. For instance, if we want the gradients of the loss with respect to `b2` and our loss is defined by:

```
loss = mse(out,y) = mse(lin(l2, w2, b2), y)
```

The chain rule tells us that we have:
$$\frac{\text{d} loss}{\text{d} b_{2}} = \frac{\text{d} loss}{\text{d} out} \times \frac{\text{d} out}{\text{d} b_{2}} = \frac{\text{d}}{\text{d} out} mse(out, y) \times \frac{\text{d}}{\text{d} b_{2}} lin(l_{2}, w_{2}, b_{2})$$

To compute the gradients of the loss with respect to $b_{2}$, we first need the gradients of the loss with respect to our output $out$. It's the same if we want the gradients of the loss with respect to $w_{2}$. Then, to get the gradients of the loss with respect to $b_{1}$ or $w_{1}$, we will need the gradients of the loss with respect to $l_{1}$, which in turn requires the gradients of the loss with respect to $l_{2}$, which will need the gradients of the loss with respect to $out$.

So to compute all the gradients we need for the update, we need to begin from the output of the model and work our way *backward*, one layer after the other—which is why this step is known as *backpropagation*. We can automate it by having each function we implemented (`relu`, `mse`, `lin`) provide its backward step: that is, how to derive the gradients of the loss with respect to the input(s) from the gradients of the loss with respect to the output.

Here we populate those gradients in an attribute of each tensor, a bit like PyTorch does with `.grad`.

The first are the gradients of the loss with respect to the output of our model (which is the input of the loss function). We undo the `squeeze` we did in `mse`, then we use the formula that gives us the derivative of $x^{2}$: $2x$. The derivative of the mean is just $1/n$ where $n$ is the number of elements in our input:

```
def mse_grad(inp, targ):
    # grad of loss with respect to output of previous layer
    inp.g = 2. * (inp.squeeze() - targ).unsqueeze(-1) / inp.shape[0]
```

For the gradients of the ReLU and our linear layer, we use the gradients of the loss with respect to the output (in `out.g`) and apply the chain rule to compute the gradients of the loss with respect to the output (in `inp.g`). The chain rule tells us that `inp.g = relu'(inp) * out.g`. The derivative of `relu` is either 0 (when inputs are negative) or 1 (when inputs are positive), so this gives us:

```
def relu_grad(inp, out):
    # grad of relu with respect to input activations
    inp.g = (inp>0).float() * out.g
```

The scheme is the same to compute the gradients of the loss with respect to the inputs, weights, and bias in the linear layer:

```
def lin_grad(inp, out, w, b):
    # grad of matmul with respect to input
    inp.g = out.g @ w.t()
    w.g = inp.t() @ out.g
    b.g = out.g.sum(0)
```

We won't linger on the mathematical formulas that define them since they're not important for our purposes, but do check out Khan Academy's excellent calculus lessons if you're interested in this topic.

### Sidebar: SymPy

SymPy is a library for symbolic computation that is extremely useful library when working with calculus. Per the [documentation](https://docs.sympy.org/latest/tutorial/intro.html):

> : Symbolic computation deals with the computation of mathematical objects symbolically. This means that the mathematical objects are represented exactly, not approximately, and mathematical expressions with unevaluated variables are left in symbolic form.

To do symbolic computation, we first define a *symbol*, and then do a computation, like so:

```
from sympy import symbols,diff
sx,sy = symbols('sx sy')
diff(sx**2, sx)
```

$\displaystyle 2 sx$

Here, SymPy has taken the derivative of `x**2` for us! It can take the derivative of complicated compound expressions, simplify and factor equations, and much more. There's really not much reason for anyone to do calculus manually nowadays—for calculating gradients, PyTorch does it for us, and for showing the equations, SymPy does it for us!

### End sidebar

Once we have have defined those functions, we can use them to write the backward pass. Since each gradient is automatically populated in the right tensor, we don't need to store the results of those `_grad` functions anywhere—we just need to execute them in the reverse order of the forward pass, to make sure that in each function `out.g` exists:

```
def forward_and_backward(inp, targ):
    # forward pass:
    l1 = inp @ w1 + b1
    l2 = relu(l1)
    out = l2 @ w2 + b2
    # we don't actually need the loss in backward!
    loss = mse(out, targ)

    # backward pass:
    mse_grad(out, targ)
    lin_grad(l2, out, w2, b2)
    relu_grad(l1, l2)
    lin_grad(inp, l1, w1, b1)
```

And now we can access the gradients of our model parameters in `w1.g`, `b1.g`, `w2.g`, and `b2.g`.

We have sucessfuly defined our model—now let's make it a bit more like a PyTorch module.

### Refactoring the Model

The three functions we used have two associated functions: a forward pass and a backward pass. Instead of writing them separately, we can create a class to wrap them together. That class can also store the inputs and outputs for the backward pass. This way, we will just have to call `backward`:

```
class Relu():
    def __call__(self, inp):
        self.inp = inp
        self.out = inp.clamp_min(0.)
        return self.out

    def backward(self): self.inp.g = (self.inp>0).float() * self.out.g
```

`__call__` is a magic name in Python that will make our class callable. This is what will be executed when we type `y = Relu()(x)`. We can do the same for our linear layer and the MSE loss:

```
class Lin():
    def __init__(self, w, b): self.w,self.b = w,b

    def __call__(self, inp):
        self.inp = inp
        self.out = inp@self.w + self.b
        return self.out

    def backward(self):
        self.inp.g = self.out.g @ self.w.t()
        self.w.g = self.inp.t() @ self.out.g
        self.b.g = self.out.g.sum(0)
```

```
class Mse():
    def __call__(self, inp, targ):
        self.inp = inp
        self.targ = targ
        self.out = (inp.squeeze() - targ).pow(2).mean()
        return self.out

    def backward(self):
        x = (self.inp.squeeze()-self.targ).unsqueeze(-1)
        self.inp.g = 2.*x/self.targ.shape[0]
```

Then we can put everything in a model that we initiate with our tensors `w1`, `b1`, `w2`, `b2`:

```
class Model():
    def __init__(self, w1, b1, w2, b2):
        self.layers = [Lin(w1,b1), Relu(), Lin(w2,b2)]
        self.loss = Mse()

    def __call__(self, x, targ):
        for l in self.layers: x = l(x)
        return self.loss(x, targ)

    def backward(self):
        self.loss.backward()
        for l in reversed(self.layers): l.backward()
```

What is really nice about this refactoring and registering things as layers of our model is that the forward and backward passes are now really easy to write. If we want to instantiate our model, we just need to write:

```
model = Model(w1, b1, w2, b2)
```

The forward pass can then be executed with:

```
loss = model(x, y)
```

And the backward pass with:

```
model.backward()
```

### Going to PyTorch

The  `Lin`, `Mse` and `Relu` classes we wrote have a lot in common, so we could make them all inherit from the same base class:

```
class LayerFunction():
    def __call__(self, *args):
        self.args = args
        self.out = self.forward(*args)
        return self.out

    def forward(self):  raise Exception('not implemented')
    def bwd(self):      raise Exception('not implemented')
    def backward(self): self.bwd(self.out, *self.args)
```

Then we just need to implement `forward` and `bwd` in each of our subclasses:

```
class Relu(LayerFunction):
    def forward(self, inp): return inp.clamp_min(0.)
    def bwd(self, out, inp): inp.g = (inp>0).float() * out.g
```

```
class Lin(LayerFunction):
    def __init__(self, w, b): self.w,self.b = w,b

    def forward(self, inp): return inp@self.w + self.b

    def bwd(self, out, inp):
        inp.g = out.g @ self.w.t()
        self.w.g = self.inp.t() @ self.out.g
        self.b.g = out.g.sum(0)
```

```
class Mse(LayerFunction):
    def forward (self, inp, targ): return (inp.squeeze() - targ).pow(2).mean()
    def bwd(self, out, inp, targ):
        inp.g = 2*(inp.squeeze()-targ).unsqueeze(-1) / targ.shape[0]
```

The rest of our model can be the same as before. This is getting closer and closer to what PyTorch does. Each basic function we need to differentiate is written as a `torch.autograd.Function` object that has a `forward` and a `backward` method. PyTorch will then keep trace of any computation we do to be able to properly run the backward pass, unless we set the `requires_grad` attribute of our tensors to `False`.

Writing one of these is (almost) as easy as writing our original classes. The difference is that we choose what to save and what to put in a context variable (so that we make sure we don't save anything we don't need), and we return the gradients in the `backward` pass. It's very rare to have to write your own `Function` but if you ever need something exotic or want to mess with the gradients of a regular function, here is how to write one:

```
from torch.autograd import Function

class MyRelu(Function):
    @staticmethod
    def forward(ctx, i):
        result = i.clamp_min(0.)
        ctx.save_for_backward(i)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        i, = ctx.saved_tensors
        return grad_output * (i>0).float()
```

The structure used to build a more complex model that takes advantage of those `Function`s is a `torch.nn.Module`. This is the base structure for all models, and all the neural nets you have seen up until now were from that class. It mostly helps to register all the trainable parameters, which as we've seen can be used in the training loop.

To implement an `nn.Module` you just need to:

- Make sure the superclass `__init__` is called first when you initiliaze it.
- Define any parameters of the model as attributes with `nn.Parameter`.
- Define a `forward` function that returns the output of your model.

As an example, here is the linear layer from scratch:

```
import torch.nn as nn

class LinearLayer(nn.Module):
    def __init__(self, n_in, n_out):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(n_out, n_in) * sqrt(2/n_in))
        self.bias = nn.Parameter(torch.zeros(n_out))

    def forward(self, x): return x @ self.weight.t() + self.bias
```

As you see, this class automatically keeps track of what parameters have been defined:

```
lin = LinearLayer(10,2)
p1,p2 = lin.parameters()
p1.shape,p2.shape
```

    (torch.Size([2, 10]), torch.Size([2]))

It is thanks to this feature of `nn.Module` that we can just say `opt.step()` and have an optimizer loop through the parameters and update each one.

Note that in PyTorch, the weights are stored as an `n_out x n_in` matrix, which is why we have the transpose in the forward pass.

By using the linear layer from PyTorch (which uses the Kaiming initialization as well), the model we have been building up during this chapter can be written like this:

```
class Model(nn.Module):
    def __init__(self, n_in, nh, n_out):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out))
        self.loss = mse

    def forward(self, x, targ): return self.loss(self.layers(x).squeeze(), targ)
```

fastai provides its own variant of `Module` that is identical to `nn.Module`, but doesn't require you to call `super().__init__()` (it does that for you automatically):

```
class Model(Module):
    def __init__(self, n_in, nh, n_out):
        self.layers = nn.Sequential(
            nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out))
        self.loss = mse

    def forward(self, x, targ): return self.loss(self.layers(x).squeeze(), targ)
```

In the last chapter, we will start from such a model and see how to build a training loop from scratch and refactor it to what we've been using in previous chapters.

## Conclusion

In this chapter we explored the foundations of deep learning, beginning with matrix multiplication and moving on to implementing the forward and backward passes of a neural net from scratch. We then refactored our code to show how PyTorch works beneath the hood.

Here are a few things to remember:

- A neural net is basically a bunch of matrix multiplications with nonlinearities in between.
- Python is slow, so to write fast code we have to vectorize it and take advantage of techniques such as elementwise arithmetic and broadcasting.
- Two tensors are broadcastable if the dimensions starting from the end and going backward match (if they are the same, or one of them is 1). To make tensors broadcastable, we may need to add dimensions of size 1 with `unsqueeze` or a `None` index.
- Properly initializing a neural net is crucial to get training started. Kaiming initialization should be used when we have ReLU nonlinearities.
- The backward pass is the chain rule applied multiple times, computing the gradients from the output of our model and going back, one layer at a time.
- When subclassing `nn.Module` (if not using fastai's `Module`) we have to call the superclass `__init__` method in our `__init__` method and we have to define a `forward` function that takes an input and returns the desired result.

## Questionnaire

1. Write the Python code to implement a single neuron.
1. Write the Python code to implement ReLU.
1. Write the Python code for a dense layer in terms of matrix multiplication.
1. Write the Python code for a dense layer in plain Python (that is, with list comprehensions and functionality built into Python).
1. What is the "hidden size" of a layer?
1. What does the `t` method do in PyTorch?
1. Why is matrix multiplication written in plain Python very slow?
1. In `matmul`, why is `ac==br`?
1. In Jupyter Notebook, how do you measure the time taken for a single cell to execute?
1. What is "elementwise arithmetic"?
1. Write the PyTorch code to test whether every element of `a` is greater than the corresponding element of `b`.
1. What is a rank-0 tensor? How do you convert it to a plain Python data type?
1. What does this return, and why? `tensor([1,2]) + tensor([1])`
1. What does this return, and why? `tensor([1,2]) + tensor([1,2,3])`
1. How does elementwise arithmetic help us speed up `matmul`?
1. What are the broadcasting rules?
1. What is `expand_as`? Show an example of how it can be used to match the results of broadcasting.
1. How does `unsqueeze` help us to solve certain broadcasting problems?
1. How can we use indexing to do the same operation as `unsqueeze`?
1. How do we show the actual contents of the memory used for a tensor?
1. When adding a vector of size 3 to a matrix of size 3×3, are the elements of the vector added to each row or each column of the matrix? (Be sure to check your answer by running this code in a notebook.)
1. Do broadcasting and `expand_as` result in increased memory use? Why or why not?
1. Implement `matmul` using Einstein summation.
1. What does a repeated index letter represent on the left-hand side of einsum?
1. What are the three rules of Einstein summation notation? Why?
1. What are the forward pass and backward pass of a neural network?
1. Why do we need to store some of the activations calculated for intermediate layers in the forward pass?
1. What is the downside of having activations with a standard deviation too far away from 1?
1. How can weight initialization help avoid this problem?
1. What is the formula to initialize weights such that we get a standard deviation of 1 for a plain linear layer, and for a linear layer followed by ReLU?
1. Why do we sometimes have to use the `squeeze` method in loss functions?
1. What does the argument to the `squeeze` method do? Why might it be important to include this argument, even though PyTorch does not require it?
1. What is the "chain rule"? Show the equation in either of the two forms presented in this chapter.
1. Show how to calculate the gradients of `mse(lin(l2, w2, b2), y)` using the chain rule.
1. What is the gradient of ReLU? Show it in math or code. (You shouldn't need to commit this to memory—try to figure it using your knowledge of the shape of the function.)
1. In what order do we need to call the `*_grad` functions in the backward pass? Why?
1. What is `__call__`?
1. What methods must we implement when writing a `torch.autograd.Function`?
1. Write `nn.Linear` from scratch, and test it works.
1. What is the difference between `nn.Module` and fastai's `Module`?

### Further Research

1. Implement ReLU as a `torch.autograd.Function` and train a model with it.
1. If you are mathematically inclined, find out what the gradients of a linear layer are in mathematical notation. Map that to the implementation we saw in this chapter.
1. Learn about the `unfold` method in PyTorch, and use it along with matrix multiplication to implement your own 2D convolution function. Then train a CNN that uses it.
1. Implement everything in this chapter using NumPy instead of PyTorch.

```

```