# Ordered Logit Model

```python
import pandas  as pd
import numpy as np
import biogeme.database as db
import biogeme.biogeme as bio
import biogeme.models as models
import biogeme.distributions as dist
from biogeme.expressions import Beta, DefineVariable, log, Elem
import seaborn as sns
import matplotlib.pyplot as plt
```

**Import data**

```python
pandas = pd.read_excel("data/BDA-Despistes_DCM_AE_V5.xlsx")
pandas = pandas[pandas['WEIGHT'].notna()]
pandas['Hora'] = pandas['Hora'].str.replace(':','')
pandas['Hora'] = pandas['Hora'].astype('float')
pandas['WINTER'] = pandas['WINTER ']
pandas['TALUDE_VALETA'] = pandas['TALUDE+VALETA']
pandas = pandas[pandas['Hora'].notna()].reset_index()
database = db.Database("data/roadside", pandas)
```

## Let's see what this dataset has

In Portugal, injury crash severities are registered by the police
and categorised as a function of the victims’ stay in hospital and
outcome:
* A minor injury is registered when a victim requires hospital
treatment but stays there for less than 24 h;
* A severe injury refers to a victim who is registered as a hospital
in-patient and stays there for more than a day; and
* A fatality means a victim who dies as a consequence of crash
injuries within 30 days of occurrence of the crash.

Portuguese crash data structure allows for categorising ROR
accidents by driver injury severity and by the severity of the most
seriously injured occupant. Data was provided by the Guarda
Nacional Republicana (GNR), which is a police force responsible for
maintaining security and public order as well as protecting and
defending the population and their property. The GNR is also
responsible for road crash investigation and data collection.
Whenever an accident occurs GNR officers compile an accident
report that includes detailed information such as the injury
severity of each vehicle occupant, driver characteristics, factors
leading to the accident, weather conditions, and a sketch of the
accident scene.

Roque, C., et al., Detecting unforgiving roadside contributors through the severity analysis of ran-off-road
crashes. Accid. Anal. Prev. (2015), http://dx.doi.org/10.1016/j.aap.2015.02.012

```python
pandas
```

      index
      ID
      CHOICE
      CHOI_MAX
      CHOICE3
      CHOICE3_MAX
      CHOICE_F
      CHOICE_G
      CHOICE_L
      CHOICE_P
      ...
      YOUNG
      OLD
      RAIN
      PEAK1
      PEAK2
      PEAK_HOUR
      AE
      TALUDE+VALETA
      WINTER
      TALUDE_VALETA

      0
      1
      2
      3
      3
      2
      2
      0
      0
      1
      0
      ...
      0
      0
      0
      0
      1
      1
      1
      0
      0
      0

      1
      2
      3
      4
      3
      3
      2
      0
      0
      0
      1
      ...
      0
      0
      0
      0
      0
      0
      1
      0
      1
      0

      2
      3
      4
      3
      3
      2
      2
      0
      0
      1
      0
      ...
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0

      3
      4
      5
      3
      3
      2
      2
      0
      0
      1
      0
      ...
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0

      4
      5
      6
      3
      3
      2
      2
      0
      0
      1
      0
      ...
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      885
      921
      922
      4
      3
      3
      2
      0
      0
      0
      1
      ...
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0

      886
      922
      923
      4
      3
      3
      2
      0
      0
      0
      1
      ...
      0
      1
      0
      0
      0
      0
      1
      0
      0
      0

      887
      923
      924
      2
      2
      1
      1
      0
      1
      0
      0
      ...
      0
      1
      0
      0
      0
      0
      1
      0
      0
      0

      888
      924
      925
      3
      3
      2
      2
      0
      0
      1
      0
      ...
      0
      0
      0
      1
      0
      1
      0
      0
      0
      0

      889
      925
      926
      4
      3
      3
      2
      0
      0
      0
      1
      ...
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0

890 rows × 74 columns

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['CHOICE'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
#chart.set_xticklabels(['Fatal', 'SeriousInjury', 'LightInjury', 'PDO']);
```

```python
plt.figure(figsize=(15,5))
chart = sns.countplot(pandas['AGE'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['YOUNG'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['OLD'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['TIPESTRA'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
#chart.set_xticklabels(['Fatal', 'SeriousInjury', 'LightInjury', 'PDO']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['AE'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['PEAK_HOUR'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['METEO'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
#chart.set_xticklabels(['Fatal', 'SeriousInjury', 'LightInjury', 'PDO']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['CAPOTA'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['NO_ALCOHOL'])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

```python
plt.figure(figsize=(10,5))
chart = sns.countplot(pandas['WINTER '])
chart.set_xticklabels(chart.get_xticklabels(), rotation=30, horizontalalignment='right');
chart.set_xticklabels(['No', 'Yes']);
```

**Use collumn names as variables**

```python
globals().update(database.variables)
```

```python
pandas = database.data
```

```python
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

pandas.describe()
```

      index
      ID
      CHOICE
      CHOI_MAX
      CHOICE3
      CHOICE3_MAX
      CHOICE_F
      CHOICE_G
      CHOICE_L
      CHOICE_P
      CHOICE_MF
      CHOICE_MG
      CHOICE_ML
      CHOICE_MP
      CHOICE3_FG
      CHOICE3_L
      CHOICE3_P
      CHOICE3_MFG
      CHOICE3_ML
      CHOICE3_MP
      GENDER
      AGE
      WEIGHT
      Dia
      mês
      ano
      DATE
      WEEKDAY
      DIAUTIL
      Hora
      Minuto
      segundos
      TIME
      SUNRISE
      SUNSET
      NIGHT
      TIPESTRA
      FAIXA
      METEO
      ALINH
      LVEL
      VEL120
      VEL90
      LUMINO
      TIPAC
      NOBST
      BARMET
      BARRIG
      TALUDE
      VEDACA
      VALETA
      VEIC1
      VEIC2
      VEIC3
      VEIC4
      NOCUP
      CAPOTA
      NDEFV
      TXALC
      TXMUL
      TXPRIS
      NO_ALCOHOL
      WINTER
      SUMMER
      YOUNG
      OLD
      RAIN
      PEAK1
      PEAK2
      PEAK_HOUR
      AE
      TALUDE+VALETA
      WINTER
      TALUDE_VALETA

      count
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.0
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.00000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000
      890.000000

      mean
      463.185393
      464.185393
      3.096629
      2.914607
      2.103371
      1.923596
      0.006742
      0.058427
      0.766292
      0.168539
      0.008989
      0.082022
      0.894382
      0.014607
      0.065169
      0.766292
      0.168539
      0.091011
      0.894382
      0.014607
      0.362921
      38.522472
      2.202382
      15.664045
      6.725843
      2009.458427
      40169.388764
      3.844944
      0.675281
      12.384270
      24.486517
      0.0
      0.532079
      0.296597
      0.799093
      0.325843
      1.347191
      1.914607
      1.423596
      1.462921
      259.773034
      0.826966
      0.008989
      913.839326
      16.008989
      1.440449
      0.359551
      0.231461
      0.115730
      0.031461
      0.028090
      0.748315
      0.187640
      0.014607
      0.002247
      1.541573
      0.352809
      0.931461
      0.051360
      0.010112
      0.020225
      0.965169
      0.306742
      0.320225
      0.196629
      0.094382
      0.383146
      0.15618
      0.125843
      0.282022
      0.835955
      0.143820
      0.306742
      0.143820

      std
      267.464751
      267.464751
      0.494844
      0.354165
      0.472518
      0.316058
      0.081876
      0.234681
      0.423426
      0.374555
      0.094435
      0.274553
      0.307521
      0.120040
      0.246962
      0.423426
      0.374555
      0.287787
      0.307521
      0.120040
      0.481113
      13.998797
      0.614833
      9.007468
      3.604320
      0.498549
      204.867961
      2.100566
      0.468533
      5.804907
      17.649071
      0.0
      0.243611
      0.022819
      0.061079
      0.468952
      0.899767
      0.279623
      0.597430
      0.741867
      322.298997
      0.378489
      0.094435
      278.861683
      119.748264
      0.675244
      0.480139
      0.422003
      0.320081
      0.174657
      0.165323
      0.434225
      0.390644
      0.120040
      0.047378
      0.829003
      0.478113
      1.639919
      0.298132
      0.100107
      0.140847
      0.183456
      0.461400
      0.466825
      0.397673
      0.292524
      0.486427
      0.36323
      0.331858
      0.450237
      0.370525
      0.351105
      0.461400
      0.351105

      min
      1.000000
      2.000000
      1.000000
      1.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      17.000000
      1.020000
      1.000000
      1.000000
      2009.000000
      39814.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.0
      0.000000
      0.259722
      0.718750
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000
      50.000000
      0.000000
      0.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.00000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000

      25%
      233.250000
      234.250000
      3.000000
      3.000000
      2.000000
      2.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      27.000000
      2.000000
      8.000000
      4.000000
      2009.000000
      39996.250000
      2.000000
      0.000000
      8.000000
      10.000000
      0.0
      0.357639
      0.276389
      0.733333
      0.000000
      1.000000
      2.000000
      1.000000
      1.000000
      120.000000
      1.000000
      0.000000
      999.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.00000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000

      50%
      463.500000
      464.500000
      3.000000
      3.000000
      2.000000
      2.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      36.000000
      2.000000
      15.000000
      7.000000
      2009.000000
      40169.000000
      4.000000
      1.000000
      13.000000
      25.000000
      0.0
      0.541667
      0.297917
      0.804861
      0.000000
      1.000000
      2.000000
      1.000000
      1.000000
      120.000000
      1.000000
      0.000000
      999.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.00000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000

      75%
      694.750000
      695.750000
      3.000000
      3.000000
      2.000000
      2.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.000000
      1.000000
      48.000000
      2.000000
      24.000000
      10.000000
      2010.000000
      40338.000000
      6.000000
      1.000000
      17.000000
      40.000000
      0.0
      0.714410
      0.316667
      0.865972
      1.000000
      1.000000
      2.000000
      2.000000
      2.000000
      120.000000
      1.000000
      0.000000
      999.000000
      2.000000
      2.000000
      1.000000
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000
      0.000000
      0.000000
      0.000000
      2.000000
      1.000000
      1.000000
      0.000000
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000
      0.000000
      0.000000
      1.000000
      0.00000
      0.000000
      1.000000
      1.000000
      0.000000
      1.000000
      0.000000

      max
      925.000000
      926.000000
      4.000000
      4.000000
      3.000000
      3.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      83.000000
      3.500000
      31.000000
      12.000000
      2010.000000
      40543.000000
      7.000000
      1.000000
      23.000000
      57.000000
      0.0
      0.993056
      0.329861
      0.878472
      1.000000
      9.000000
      2.000000
      5.000000
      3.000000
      999.000000
      1.000000
      1.000000
      999.000000
      999.000000
      4.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      7.000000
      1.000000
      6.000000
      2.990000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.00000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000
      1.000000

**Create parameters to be estimated**

```python
tau1 = Beta('tau1', -1, None, 0, 0)

delta2 = Beta('delta2', 1, 0, None, 0)
tau2 = tau1 + delta2

delta3 = Beta('delta3', 2, 0, None, 0)
tau3 = tau2 + delta3

B_ROLLOVER = Beta('B_ROLLOVER', 0, None, None, 0)
B_SPEEDLIMIT120 = Beta('B_SPEEDLIMIT120', 0, None, None, 0)
```

**Define the utility function**

\begin{align}
V & = \beta_{rollover}X_{rollover}} + \beta_{120}X_{120} \\
\end{align}

```python
U = B_ROLLOVER * CAPOTA + B_SPEEDLIMIT120 * VEL120
```

**Associate utility functions with choice probabilities**

```python
ChoiceProba = {
    1: 1 - dist.logisticcdf(U - tau1),
    2: dist.logisticcdf(U - tau1) - dist.logisticcdf(U - tau2),
    3: dist.logisticcdf(U - tau2) - dist.logisticcdf(U - tau3),
    4: dist.logisticcdf(U - tau3)}
```

**Define the model**

```python
logprob = log(Elem(ChoiceProba, CHOICE))
```

**Define the Biogeme object**

```python
biogeme  = bio.BIOGEME(database, logprob, removeUnusedVariables=False)

biogeme.modelName = "roadside_ordered"
```

**Estimate the model**

```python
biogeme.generateHtml = True
biogeme.generatePickle = False

results = biogeme.estimate()

print(f"HTML file:    {results.data.htmlFileName}")
print(f"Pickle file:  {results.data.pickleFileName }")
```

    HTML file:    roadside_ordered~06.html
    Pickle file:  None

**Print results**

```python
pandasResults = results.getEstimatedParameters()
print(pandasResults)
```

                        Value   Std err     t-test       p-value  Rob. Std err  Rob. t-test  Rob. p-value
    B_ROLLOVER      -0.811606  0.176468  -4.599154  4.242093e-06      0.176113    -4.608434  4.057138e-06
    B_SPEEDLIMIT120 -0.119254  0.205153  -0.581293  5.610429e-01      0.195180    -0.610995  5.412030e-01
    delta2           2.340009  0.389813   6.002897  1.938280e-09      0.389365     6.009803  1.857487e-09
    delta3           4.377749  0.161946  27.032181  0.000000e+00      0.161973    27.027582  0.000000e+00
    tau1            -5.457001  0.453228 -12.040298  0.000000e+00      0.435426   -12.532565  0.000000e+00

```python
gs = results.getGeneralStatistics()

for k,v in gs.items():
    print("{}= {}".format(k.ljust(45),v[0]))
```

    Number of estimated parameters               = 5
    Sample size                                  = 890
    Excluded observations                        = 0
    Init log likelihood                          = -1061.5663994215017
    Final log likelihood                         = -614.8420363957916
    Likelihood ratio test for the init. model    = 893.4487260514202
    Rho-square for the init. model               = 0.4208162233367141
    Rho-square-bar for the init. model           = 0.41610620236890206
    Akaike Information Criterion                 = 1239.6840727915833
    Bayesian Information Criterion               = 1263.6401801052143
    Final gradient norm                          = 0.0029065761458367204
    Nbr of threads                               = 8

## Let's build a second model, with more variables

```python
B_WINTER = Beta('B_WINTER', 0, None, None, 0)
B_OLD = Beta('B_OLD', 0, None, None, 0)
B_YOUNG = Beta('B_YOUNG', 0, None, None, 0)
B_RAIN = Beta('B_RAIN', 0, None, None, 0)
B_AE = Beta('B_AE', 0, None, None, 0)
B_NO_ALCOHOL = Beta('B_NO_ALCOHOL', 0, None, None, 0)
B_SPEEDLIMIT90 = Beta('B_SPEEDLIMIT90', 0, None, None, 0)
B_TALUDE_VALETA = Beta('B_TALUDE_VALETA', 0, None, None, 0)
```

```python
U = B_ROLLOVER * CAPOTA + \
    B_SPEEDLIMIT120 * VEL120 + \
    B_SPEEDLIMIT90 * VEL90 + \
    B_WINTER * WINTER + \
    B_OLD * OLD + \
    B_YOUNG * YOUNG + \
    B_RAIN * RAIN + \
    B_AE * AE + \
    B_NO_ALCOHOL * NO_ALCOHOL + \
    B_TALUDE_VALETA * TALUDE_VALETA
```

```python
ChoiceProba = {
    1: 1 - dist.logisticcdf(U - tau1),
    2: dist.logisticcdf(U - tau1) - dist.logisticcdf(U - tau2),
    3: dist.logisticcdf(U - tau2) - dist.logisticcdf(U - tau3),
    4: dist.logisticcdf(U - tau3)}
```

```python
logprob = log(Elem(ChoiceProba, CHOICE))
```

```python
biogeme  = bio.BIOGEME(database, logprob, removeUnusedVariables=False)

biogeme.modelName = "roadside_ordered"
```

```python
biogeme.generateHtml = True
biogeme.generatePickle = False

results = biogeme.estimate()

print(f"HTML file:    {results.data.htmlFileName}")
print(f"Pickle file:  {results.data.pickleFileName }")
```

    HTML file:    roadside_ordered~07.html
    Pickle file:  None

```python
pandasResults = results.getEstimatedParameters()
pandasResults
```

      Value
      Std err
      t-test
      p-value
      Rob. Std err
      Rob. t-test
      Rob. p-value

      B_AE
      -2.804544
      1.385569
      -2.024109
      4.295891e-02
      1.205453
      -2.326547
      1.998938e-02

      B_NO_ALCOHOL
      0.089008
      0.445765
      0.199674
      8.417356e-01
      0.489613
      0.181792
      8.557460e-01

      B_OLD
      0.238587
      0.270444
      0.882204
      3.776667e-01
      0.294924
      0.808979
      4.185271e-01

      B_RAIN
      0.114333
      0.173732
      0.658103
      5.104721e-01
      0.169951
      0.672743
      5.011107e-01

      B_ROLLOVER
      -0.814045
      0.178774
      -4.553479
      5.276605e-06
      0.178241
      -4.567099
      4.945208e-06

      B_SPEEDLIMIT120
      2.676493
      1.386086
      1.930972
      5.348646e-02
      1.207666
      2.216253
      2.667420e-02

      B_SPEEDLIMIT90
      1.895840
      1.406187
      1.348213
      1.775898e-01
      1.228578
      1.543117
      1.228024e-01

      B_TALUDE_VALETA
      -0.132874
      0.230832
      -0.575633
      5.648633e-01
      0.226779
      -0.585919
      5.579297e-01

      B_WINTER
      0.303483
      0.180408
      1.682202
      9.252964e-02
      0.176858
      1.715967
      8.616805e-02

      B_YOUNG
      0.010284
      0.200436
      0.051307
      9.590805e-01
      0.178488
      0.057616
      9.540542e-01

      delta2
      2.348522
      0.390519
      6.013847
      1.811714e-09
      0.391632
      5.996758
      2.012962e-09

      delta3
      4.422593
      0.164831
      26.831025
      0.000000e+00
      0.164681
      26.855592
      0.000000e+00

      tau1
      -5.281491
      0.625643
      -8.441694
      0.000000e+00
      0.640451
      -8.246512
      2.220446e-16

```python
gs = results.getGeneralStatistics()

for k,v in gs.items():
    print("{}= {}".format(k.ljust(45),v[0]))
```

    Number of estimated parameters               = 13
    Sample size                                  = 890
    Excluded observations                        = 0
    Init log likelihood                          = -1061.5663994215017
    Final log likelihood                         = -610.1374173509528
    Likelihood ratio test for the init. model    = 902.8579641410979
    Rho-square for the init. model               = 0.42524799420606585
    Rho-square-bar for the init. model           = 0.4130019396897545
    Akaike Information Criterion                 = 1246.2748347019055
    Bayesian Information Criterion               = 1308.560713717346
    Final gradient norm                          = 0.002452270583563102
    Nbr of threads                               = 8


# Different Classifiers with scikit-learn

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import datasets, tree, preprocessing, neighbors
# produce vector inline graphics
from IPython.display import set_matplotlib_formats, display, Code
from sklearn.model_selection import train_test_split, cross_val_predict, cross_val_score, StratifiedKFold
from sklearn.pipeline import make_pipeline
set_matplotlib_formats('svg')
from wittgenstein import RIPPER
from ipywidgets import interact
import ipywidgets as widgets
import seaborn as sns
```

## Normalization of Attributes

For this notebook, we consider the binary classification problem on the [breast cancer dataset](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)):

```python
# load the dataset from sklearn
dataset = datasets.load_breast_cancer()

# create a DataFrame to help further analysis
df = pd.DataFrame(data=dataset.data, columns=dataset.feature_names)
df['target'] = dataset.target
df.target = df.target.astype('category')
df.target.cat.categories = dataset.target_names
df  # display an overview of the data
```

The goal is to train a classifier to diagnose whether a breast mass is malignant or benign. The target class distribution is shown as follows:

```python
display(df.target.value_counts())
df.target.value_counts().plot(kind='bar', title='counts of different classes')
```

The input features are characteristics of cell images obtained by [fine needle analysis (FNA)](https://en.wikipedia.org/wiki/Fine-needle_aspiration):

```python
%%html
<a href="https://www.researchgate.net/figure/FNA-biopsy-samples-of-benign-left-and-malignant-center-and-right-breast-tumor-cells_fig1_261959799"></a>
<p>FNA biopsy samples of benign (left) and malignant (center and right) breast tumor cells.</p>
<p>Eickhoff, Carsten. (2014). Crowd-powered experts: helping surgeons interpret breast cancer images. ACM International Conference Proceeding Series. 53-56. 10.1145/2594776.2594788.</p>
```

FNA biopsy samples of benign (left) and malignant (center and right) breast tumor cells.

Eickhoff, Carsten. (2014). Crowd-powered experts: helping surgeons interpret breast cancer images. ACM International Conference Proceeding Series. 53-56. 10.1145/2594776.2594788.

The following function displays the statistics of the features grouped by the class values:

```python
def show_feature_statistics(df):
    df.groupby('target').boxplot(rot=90,
                                 layout=(1, 3),
                                 figsize=(12, 5),
                                 fontsize=7)

show_feature_statistics(df)
```

From the above plots, it can be observed that the attributes `mean area` and `worst area` have much larger ranges than other features have.

**Exercise** Is it true that a feature with the larger range is a better feature? Why?

YOUR ANSWER HERE

### Min-max Normalization

We can normalize a numeric feature to the unit interval as follows:

\begin{align}
X_i\in [X_i^{\min}, X_i^{\max}] \mapsto \frac{X_i - X_i^{\min}}{X_i^{\max} - X_i^{\min}} \in [0,1].
\end{align}

To perform the normalization on i.i.d. samples $x_{ij}$ indexed by $j$, we use the mapping

\begin{align}
x_{ij} \mapsto \frac{x_{ij} - \min_j x_j}{\max_j x_j - \min_j x_j}.
\end{align}

An implementation is as follows:

```python
def minmax_normalize(df, suffix=' (min-max normalized)'):
    ''' Returns a DataFrame with numerical attributes of the input DataFrame
    min-max normalized.

    Parameters
    ----------
    df (DataFrame): Input to be min-max normalized. May contain both numeric
        and categorical attributes.
    suffix (string): Suffix to append to the names of normalized attributes.

    Returns
    -------
    A new DataFrame which retains the categorical attributes but with the
    numeric attributes replaced by their min-max normalization.
    The normalized features are renamed with the suffix appended to the end of
    their original names.
    '''
    df_minmax_normalized = df.copy()
    min_values = df.min()  # Categorical feature target skipped automatically

    # min-max normalize
    df_minmax_normalized[min_values.index] = ((df[min_values.index] - min_values) /
                            (df.max() - min_values)).copy()

    # rename normalized features
    df_minmax_normalized.rename(
        columns={c: c + suffix
                 for c in min_values.index},
        inplace=True)

    return df_minmax_normalized
```

It is a good idea to rename the normalized features to differentiate them from the original features. The following plots the statistics of the normalized features.

```python
df_minmax_normalized = minmax_normalize(df)
assert df_minmax_normalized.target.to_numpy().base is df.target.to_numpy().base
show_feature_statistics(df_minmax_normalized)
```

After normalization, we can see how instances of different classes differ in different input features other than `mean area` and `worst area`. In particular, both `mean-concavity` and `worst-concavity` are substantially higher for malignant examples than for benign examples. Such details are hard to see in the plots before normalization.

### Standard Normalization

For features with unbounded support and a distribution close to the normal distribution, one may use the $z$-score/standard normalization instead of min-max normalization:

\begin{align}
X_i \mapsto \frac{X_i - \mu_{X_i}}{\sigma_{X_i}},
\end{align}

where $\mu_{X_i}$ and $\sigma_{X_i}$ are the expectation and standard deviation of the feature $X_i$ respectively. The i.i.d.\ samples can be transformed using sample mean and standard deviation instead. It is okay to use a biased estimate for standard deviation and variance.

To inspect the distribution of each feature, we can use the function [`displot`](https://seaborn.pydata.org/generated/seaborn.displot.html) provided by the package [`seaborn`](https://seaborn.pydata.org).

```python
import seaborn as sns

@interact(feature=dataset.feature_names, kernel_density_estimation=True, group_by_class=False)
def plot_distribution(feature, kernel_density_estimation, group_by_class):
    sns.displot(data=df, x=feature, col='target' if group_by_class else None, kde=kernel_density_estimation, height=3.5)
```

Play with the above widgets to check if the distributions of a feature look like the bell-shaped normal distribution.

**Exercise** Complete the function `standard_normalize` as follows:

- Assign to `df_standard_normalized` a new data `DataFrame` same as `df` but with all its numeric attributes standard normalized.
- You may use the methods `mean` and `std`.
- Rename the normalized features by appending `suffix` to their names.

```python
def standard_normalize(df, suffix=' (standard normalized)'):
    ''' Returns a DataFrame with numerical attributes of the input DataFrame
    standard normalized.

    Parameters
    ----------
    df (DataFrame): Input to be standard normalized. May contain both numeric
        and categorical attributes.
    suffix (string): Suffix to append to the names of normalized attributes.

    Returns
    -------
    A new DataFrame which retains the categorical attributes but with the
    numeric attributes replaced by their standard normalization.
    The normalized features are renamed with the suffix appended to the end of
    their original names.
    '''
    # YOUR CODE HERE
    raise NotImplementedError()
    return df_standard_normalized

df_standard_normalized = standard_normalize(df)
show_feature_statistics(df_standard_normalized)
```

```python
# tests
assert np.isclose(df_standard_normalized.mean(), 0).all()
```

## Nearest Neighbor Classification

To create a $k$-nearest-neighbor ($k$-NN) classifier, we can use `sklearn.neighbors.KNeighborsClassifier`. The following fits a $1$-NN classifier to the entire dataset and returns its training accuracy.

```python
from sklearn import neighbors

X, Y = df[dataset.feature_names], df.target
kNN1 = neighbors.KNeighborsClassifier(n_neighbors=1)
kNN1.fit(X, Y)

print('Training accuracy: {:0.3g}'.format(kNN1.score(X, Y)))
```

**Exercise** Why is the training accuracy for $1$-NN $100\%$? Explain according to how 1-NN works.

YOUR ANSWER HERE

To avoid overly-optimistic performance estimates, the following uses 10-fold cross validations to compute the accuracies of 1-NN trained on datasets with and without normalization.

```python
cv = StratifiedKFold(n_splits=10, random_state=0, shuffle=True)

dfs = {
    'None': df,
    'Min-max': df_minmax_normalized,
    'Standard': df_standard_normalized
}

acc = pd.DataFrame(columns=dfs.keys())
for norm in dfs:
    acc[norm] = cross_val_score(
        kNN1,
        dfs[norm].loc[:, lambda df: ~df.columns.isin(['target'])],
        # not [dataset.feature_names] since normalized features are renamed
        dfs[norm]['target'],
        cv=cv)

acc.agg(['mean', 'std']).round(5)
```

### Data Leak

The accuracies computed previously appear to show that normalization improves the performance of 1-NN. However, there are two subtle issues:

1. The standard deviations are quite large compared to the difference in performance. The difference in accuracies may be due to the random sampling in obtaining the data and splitting the data.
2. The normalization factors are calculated from the entire datasets. Hence, in cross-validation, the normalized data for training a classifier indeed depends on the data for testing the classifier. This can lead to an overly-optimistic estimate.

There is not much we can do to resolve the first issue, other than collecting more data. Repeating the cross-validation with different random seeds do not help as that only smooth out the randomness in splitting, not sampling.

For the second issue can be easily resolved, however, by computing the normalization factors from training set instead of the entire dataset:
- Like the filtered classifier in Weka, `sklearn.pipeline` provides the function `make_pipeline` to combine a filter with a classifier.
- `sklearn.preprocessing` provides different filters for normalizing features.

```python
from sklearn.pipeline import make_pipeline
from sklearn import preprocessing

kNN1_standard_normalized = make_pipeline(preprocessing.StandardScaler(), kNN1)
acc['Standard'] = cross_val_score(kNN1_standard_normalized, X, Y, cv=cv)
acc['Standard'].agg(['mean', 'std']).round(5)
```

The variance still appears to too large to conclude the improve in performance. Indeed, the accuracy for the pipelined classifier gets even larger than the earlier overly-optimistic accuracy. Like the experimenter interface in Weka, a proper way to compare the performance of different classifiers is to use the [paired t-test](https://towardsdatascience.com/inferential-statistics-series-t-test-using-numpy-2718f8f9bf2f).

**Exercise** Similar to the above cell, correct the accuracies in `acc['Min-max']` to use `preprocessing.MinMaxScaler` as part of a pipeline for the 1-NN classifier.

```python
# YOUR CODE HERE
raise NotImplementedError()
acc['Min-max'].agg(['mean', 'std']).round(5)
```

```python
# tests
```

### Decision Regions

Since `sklearn` does not provide any function to plot the decision regions of a classifier, we provide the function `plot_decision_regions` in a new module `util` defined in [`util.py`](util.py) of the current directory.

```python
from util import plot_decision_regions
?plot_decision_regions
```

```python
Code(filename='util.py')
```

from joblib import Memory # for caching
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import os

os.makedirs('private', exist_ok=True)
memory = Memory(location="private",verbose=0)

@memory.cache
def plot_decision_regions(X, Y, clf, target_names=None, ax=None, N=200):
    """Plot the decision boundaries of a classifier on a 2D dataset.

    Parameters
    ----------
    X (array-like): 2D input features.
    Y (array-like): 1D target values for X.
    clf (sklean Classifier): A classifier trained to predicts Y from X.
    target_names (array-like): array of possible target names. If None, infer from Y.
    ax (axis): axis to plot the boundaries.
    N: number of points for each dimension to scan for the decision boundaries.

    Return
    ------
    axis: axis for the plot of the decision boundaries
    """
    def color(target_array):
        return ((np.asarray(target_array).reshape(-1, 1) == target_names.reshape(1, -1)) *
                np.arange(len(target_names)).reshape(1, -1)).sum(axis=-1)

    if ax is None:
        ax = plt.gca()

    X_, Y_ = np.asarray(X), np.asarray(Y)

    X_min = X_.min(axis=0)
    X_max = X_.max(axis=0)
    x1, x2 = np.meshgrid(np.linspace(X_min[0], X_max[0], N),
                         np.linspace(X_min[1], X_max[1], N))

    if target_names is None:
        target_names = np.unique(Y_)

    c = color(Y_)
    yhat = color(clf.predict(np.c_[x1.ravel(), x2.ravel()])).reshape(x1.shape)

    ax.contourf(x1, x2, yhat, alpha=0.4)
    scatter = ax.scatter(X_[:, 0], X_[:, 1], c=c, edgecolor='w', s=20)
    ax.set_xlim(X_min[0], X_max[0])
    ax.set_ylim(X_min[1], X_max[1])
    ax.add_artist(
        ax.legend(scatter.legend_elements()[0],
                  target_names,
                  loc="upper left",
                  title="Classes"))
    return ax

The following plots the decision region for a selected pair of input features.

```python
@interact(normalization=['None', 'Min-max', 'Standard'],
          feature1=dataset.feature_names,
          feature2=dataset.feature_names,
          k=widgets.IntSlider(1, 1, 5, continuous_update=False),
          resolution=widgets.IntSlider(1, 1, 4, continuous_update=False))
def decision_regions_kNN(normalization,
                         feature1=dataset.feature_names[0],
                         feature2=dataset.feature_names[1],
                         k=1,
                         resolution=1):
    scaler = {
        'Min-max': preprocessing.MinMaxScaler,
        'Standard': preprocessing.StandardScaler
    }
    kNN = neighbors.KNeighborsClassifier(n_neighbors=k)
    if normalization != 'None':
        kNN = make_pipeline(scaler[normalization](), kNN)
    kNN.fit(df[[feature1, feature2]], df.target)
    plt.figure(figsize=(10, 10))
    ax = plot_decision_regions(df[[feature1, feature2]],
                               df.target,
                               kNN,
                               N=resolution * 100)
    ax.set_title('Decision region for {}-NN'.format(k))
    ax.set_xlabel(feature1)
    ax.set_ylabel(feature2)
```

Interact with the widgets to:

- Learn the effect on the decision regions/boundaries with different normalizations and choices of $k$.
- Learn to draw the decision boundaries for $1$-NN with min-max normalization.

To avoid repeated computations, `plot_decision_regions` is [a decorated function](https://realpython.com/primer-on-python-decorators/) with its return values memorized/cached. To clear the cached plots, run:

```python
plot_decision_regions.clear()
```

**Exercise** Complete the following code to plot the decision regions for decision trees. Afterwards, explain whether the decision regions change for different normalizations.

```python
@interact(normalization=['None', 'Min-max', 'Standard'],
          feature1=dataset.feature_names,
          feature2=dataset.feature_names,
          resolution=widgets.IntSlider(1, 1, 4, continuous_update=False))
def decision_regions_kNN(normalization,
                         feature1=dataset.feature_names[0],
                         feature2=dataset.feature_names[1],
                         resolution=1):
    scaler = {
        'Min-max': preprocessing.MinMaxScaler,
        'Standard': preprocessing.StandardScaler
    }
    # YOUR CODE HERE
    raise NotImplementedError()
    plt.figure(figsize=(10, 10))
    ax = plot_decision_regions(df[[feature1, feature2]],
                               df.target,
                               DT,
                               N=resolution * 100)
    ax.set_title('Decision region for Decision Tree')
    ax.set_xlabel(feature1)
    ax.set_ylabel(feature2)
```

YOUR ANSWER HERE

```python
plot_decision_regions.clear()
```

## Rule-based Classification

Although RIPPER is not available in `sklearn`, there is an implementation in [`wittgenstein`](https://pypi.org/project/wittgenstein/).

```python
from wittgenstein import RIPPER

ripper = RIPPER(random_state=0)
pos_class=dataset.target_names[0]
ripper.fit(df, class_feat = 'target', pos_class=pos_class)
ripper.out_model()
```

The above is the rule for detecting the positive class. To evaluate the performance:

```python
scores = cross_val_score(ripper, df[dataset.feature_names], df.target == pos_class)
print(f'Accuracy: {scores.mean():.3g}')
```

**Challenge** The `predict` method of `ripper` seems to fail, as the following code does not show a correct decision region. As a challenge, try to identify the issue from the [github repository](https://github.com/imoscovitz/wittgenstein).

```python
@interact(normalization=['None', 'Min-max', 'Standard'],
          feature1=dataset.feature_names,
          feature2=dataset.feature_names,
          resolution=widgets.IntSlider(1, 1, 4, continuous_update=False))
def decision_regions_kNN(normalization,
                         feature1=dataset.feature_names[0],
                         feature2=dataset.feature_names[1],
                         resolution=1):
    scaler = {
        'Min-max': preprocessing.MinMaxScaler,
        'Standard': preprocessing.StandardScaler
    }
    ripper = RIPPER(random_state=0)
    if normalization != 'None':
        ripper = make_pipeline(scaler[normalization](), ripper)
    X = df[[feature1, feature2]].to_numpy()
    Y = df.target == pos_class
    ripper.fit(X, Y)
    ripper.out_model()
    plt.figure(figsize=(10, 10))
    ax = plot_decision_regions(X,
                               Y,
                               ripper,
                               N=resolution * 100)
    ax.set_title('Decision region for RIPPER')
    ax.set_xlabel(feature1)
    ax.set_ylabel(feature2)

plot_decision_regions.clear()
```

Tutorial4/Different Classifiers with scikit-learn.ipynb


[](https://kaggle.com/kernels/welcome?src=https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/tutorials/W0D3_LinearAlgebra/W0D3_Tutorial3.ipynb)

# Bonus Tutorial: Discrete Dynamical Systems
**Week 0, Day 3: Linear Algebra**

**By Neuromatch Academy**

__Content creators:__ Ella Batty

__Production editors:__ Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

In this tutorial, we will start to gain an intuition for how eigenvalues and eigenvectors can be helpful for understanding dynamical systems. We will focus on a discrete dynamical system consisting of two neurons.

By the end of the tutorial, you will:

* Predict whether the firing rates of interconnected model neurons will explode or decay based on the eigenvalues of the weight matrix.
* Apply ideas from previous tutorials (linear combination, basis vectors, etc) to understand a new concept

---
# Setup

```python
# Imports

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
```

```python
#@title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Plotting functions

def plot_circuit_responses(u, W, eigenstuff = False, xlim='default', ylim='default'):
    fig, ax = plt.subplots(1, 1, figsize=(10,10))

    # Set up axis limits
    if xlim =='default':
      extreme = np.maximum(np.abs(np.min(u)), np.max(u))
      xlim = [- extreme, extreme]
    if ylim == 'default':
      extreme = np.maximum(np.abs(np.min(u)), np.max(u))
      ylim = [- extreme, extreme]

    # Set up look
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)
    cs = plt.rcParams['axes.prop_cycle'].by_key()['color']*10
    ax.set_xlim(xlim)
    ax.set_ylim(ylim)

    # Set up tracking textz
    tracker_text = ax.text(.5, .9, "", color='w', fontsize=20, verticalalignment='top', horizontalalignment='left', transform=ax.transAxes)

    # Plot eigenvectors
    if eigenstuff:
      eigvals, eigvecs = np.linalg.eig(W)

      if np.abs(eigvals[0]) < np.abs(eigvals[1]):
        lc1 = 'c'
        lc2 = 'g'
      else:
        lc1 = 'g'
        lc2 = 'c'

      ax.plot(np.arange(-10000, 10000)*eigvecs[0, 0], np.arange(-10000, 10000)*eigvecs[1, 0],lc1, alpha=.5, label = r'$\mathbf{v}_1$')
      ax.plot(np.arange(-10000, 10000)*eigvecs[0, 1], np.arange(-10000, 10000)*eigvecs[1, 1], lc2, alpha=.5, label = r'$\mathbf{v}_2$')

      ax.legend()

    # Set up scatter
    cmap = plt.cm.coolwarm
    norm = plt.Normalize(vmin=0, vmax=u.shape[1])
    ax.plot(u[0,:], u[1, :], alpha=.4, zorder = 0)
    scatter = ax.scatter(u[0, :], u[1, :], alpha=1, c = cmap(norm(np.arange(u.shape[1]))), zorder = 1)

    ax.set(xlabel = 'Neuron 1 Firing Rate ($a_1$)', ylabel = 'Neuron 2 Firing Rate ($a_2$)', title = 'Neural firing over time')

    fig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap),
                ax=ax, label = 'Time step')
```

```python
#@title Helper functions

def get_eigval_specified_matrix(target_eig):
  """Generates matrix with specified eigvals

    Args:
      target_eig (list): list of target eigenvalues, can be real or complex,
              should be length 2 unless you desire repeated eigenvalues
              with the same eigenvector, in which case length 1

    Returns:
      ndarray: 2 x 2 matrix with target eigvals

  """

  # Set up two eigenvectors
  V = np.array([[1, 1], [-1, 1]]).astype('float')
  for i in range(2):
    V[:,i] = V[:,i]/np.linalg.norm(V[:,i])

  # Get matrix with target eigenvalues
  if type(target_eig[0]) == int or type(target_eig[0]) == float:

    if len(target_eig) == 2: # distinct eigvecs (not necessarily distinct eigvals)

        D = np.diag(target_eig)
        A = V @ D @ np.linalg.inv(V)

    else: # repeated with same vec
      summed = 2*target_eig[0]

      a = summed-3
      d = 3
      bc = target_eig[0]**2 - a*d
      factors = [n for n in range(1, bc+ 1) if bc % n == 0]
      b = factors[int(np.floor(len(factors)/2))]
      c = bc/-b

      A = np.array([[a, b], [c, d]])

  elif type(target_eig[0]) == complex:

      C = [np.real(V[:,0]), np.real(V[:,1])]
      B = np.array([[np.real(target_eig[0]), np.imag(target_eig[0])], [-np.imag(target_eig[0]), np.real(target_eig[0])]]).squeeze()
      A = C @ B @ np.linalg.inv(C)

  return A
```

---

# Section 1: Defining a neural circuit

**Please note a correction to the video appearing below it**

```python
# @title Video 1: A neural circuit
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1564y1b78X", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="XJu2bQgnn9M", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers our neural circuit we will be working with in this tutorial.

**Please note this correction to the video: I had wrong subscripts for weights in the equations and matrix. These should be:**

$$a_{1, t} = w_{1, 1}a_{1, t-1} + w_{2, 1}a_{2, t-1} $$
$$a_{2, t} = w_{1, 2}a_{1, t-1} + w_{2, 2}a_{2, t-1} $$
$$\mathbf{W} = \begin{bmatrix} w_{1, 1} & w_{2, 1} \\ w_{1, 2} & w_{2, 2} \end{bmatrix}, \mathbf{a}_{t} =   \begin{bmatrix} a_{1, t} \\ a_{2, t} \end{bmatrix}$$

 Click here for text recap of video

In previous tutorials, we have looked at static models of LGN neurons based on the responses of retinal neurons. By static, I mean that we just looked at a single time point.

Let's now introduce the concept of time. We will chop time up into little bins and look at the activity of neurons in each bin. That is, we will work in a **discrete** time framework. For example, if each bin is 1 second long, we will look at the firing rate of each neuron at intervals of a second.

Instead of examining pre- and post- synaptic neurons, we will examine two neurons in one area that are connected. In our model, the activity of neuron 1 at one time bin depends on the activities of both neurons during the previous time bin multiplied by the respective weights from itself and neuron 2. It might seem weird for a neuron to have a weight to itself - this is abstracting away some biological details but basically conveys how much the neural activity depends on its history. (Throughout this course, we'll see lots of neuron models and how some model biological detail more faithfully while others abstract.)

We will refer to the activity of neuron i during time bin j as $a_{i, j}$. The weight from neuron y to neuron x will be $w_{y, x}$. With this helpful notation, we can write an equation for the activity of neuron 1 at time bin t:
$$a_{1, t} = w_{1, 1}a_{1, t-1} + w_{2, 1}a_{2, t-1} $$

And the symmetric model is true of neuron 2:
$$a_{2, t} = w_{1, 2}a_{1, t-1} + w_{2, 2}a_{2, t-1} $$

This is already a mess of subscript numbers - luckily we can use matrices and vectors once again and our model becomes:

$$\mathbf{a}_{t} = \mathbf{W}\mathbf{a}_{t-1} $$
where:
$$\mathbf{W} = \begin{bmatrix} w_{1, 1} & w_{2, 1} \\ w_{1, 2} & w_{2, 2} \end{bmatrix}, \mathbf{a}_{t} =   \begin{bmatrix} a_{1, t} \\ a_{2, t} \end{bmatrix}$$

It turns out that this is a **discrete dynamical system**. Dynamical systems are concerned with how quantities evolve other time, in this case our neural firing rates. When we model the evolution of quantities over time using a discrete time framework, it is, unsurprisingly, a discrete dynamical system. We will see continuous dynamical systems (where we embrace the full continuity of time) tomorrow and later in the comp neuro course during W2D2: Linear Dynamics.

## Coding Exercise 1: Implementing the circuit

In this exercise, you will implement the function `circuit_implementation`. Given a weight matrix, initial activities at time 0, and a number of time bins to model, this function calculates the neural firing rates at each time bin.

We will use initial firing rates of 1 for both neurons:
$$\mathbf{a}_0 = \begin{bmatrix}
1 \\
1  \\
\end{bmatrix}$$
and the weight matrix:

$$\mathbf{W} = \begin{bmatrix} 1 & 0.2 \\
0.1 & 1 \\ \end{bmatrix}$$

We will look at activity over 30 time steps. As before, we will allow our firing rates to be negative, despite this not being possible biologically.

```python
def circuit_implementation(W, a0, T):
  """ Simulate the responses of N neurons over time given their connections

  Args:
    W (ndarray): weight matrix of synaptic connections, should be N x N
    a0 (ndarray): initial condition or input vector, should be N,
    T (scalar): number of time steps to run simulation for

  Returns:
    a (ndarray): the neural responses over time, should be N x T

  """

  # Compute the number of neurons
  N = W.shape[0]

  # Initialize empty response array and initial condition
  a = np.zeros((N, T))
  a[:, 0]  = a0

  #################################################
  ## TODO for students ##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: Complete circuit_implementation")
  #################################################

  # Loop over time steps and compute u(t+1)
  for i_t in range(1, T):
      a[:, i_t] = ...

  return a

# Define W, u0, T
W = np.array([[1, .2], [.1, 1]])
a0 = np.array([1, 1])
T = 30

# Get neural activities
a = circuit_implementation(W, a0, T)

# Visualize neural activities
plot_circuit_responses(a, W)
```

```python
# to_remove solution
def circuit_implementation(W, a0, T):
  """ Simulate the responses of N neurons over time given their connections

  Args:
    W (ndarray): weight matrix of synaptic connections, should be N x N
    a0 (ndarray): initial condition or input vector, should be N,
    T (scalar): number of time steps to run simulation for

  Returns:
    a (ndarray): the neural responses over time, should be N x T

  """

  # Compute the number of neurons
  N = W.shape[0]

  # Initialize empty response array and initial condition
  a = np.zeros((N, T))
  a[:, 0]  = a0

  # Loop over time steps and compute u(t+1)
  for i_t in range(1, T):
      a[:, i_t] = W @ a[:, i_t-1]

  return a

# Define W, u0, T
W = np.array([[1, .2], [.1, 1]])
a0 = np.array([1, 1])
T = 30

# Get neural activities
a = circuit_implementation(W, a0, T)

# Visualize neural activities
with plt.xkcd():
  plot_circuit_responses(a, W)
```

The firing rates of both neurons are exploding to infinity over time. Let's now see what happens with a different weight matrix:

$$\mathbf{W} = \begin{bmatrix} 0.2 & 0.1 \\
1 & 0.2 \\ \end{bmatrix}$$

```python
# @markdown Execute this cell to visualize activity over time

# Define W, a0, T
W = np.array([[.2, .1], [1, .2]])
a0 = np.array([1, 1])
T = 30

# Get neural activities
a = circuit_implementation(W, a0, T)

# Visualize neural activities
plot_circuit_responses(a, W)
```

We can see that with this weight matrix, the firing rates are decaying towards zero. It turns out that we could have predicted this by looking at the eigenvalues of the weight matrices, as we'll see in the next section.

---
# Section 2: Understanding dynamics using eigenstuff

As we'll see in this section, eigenvectors and eigenvalues are incredibly useful for understanding the evolution of the neural firing rates, and discrete dynamical systems in general.

## Section 2.1: Initial firing rates along an eigenvector

**Please note this correction to the video: the equation at the bottom of slide 115 (at 2:31) should be: $a_i = W^ia_0$**

```python
# @title Video 2: Looking at activity along an eigenvector
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Tf4y1b7de", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="BnSvcWbYf8g", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers what happens if our initial firing rates fall along one of the eigenvectors of the weight matrix.

 Click here for text recap of video

**Rewriting our circuit equation**

In our neural circuit, we are modeling the activities at a time step as:
$$\mathbf{a}_{t} = \mathbf{W}\mathbf{a}_{t-1} $$

Let's start at time  step 1:
$$\mathbf{a}_{1} = \mathbf{W}\mathbf{a}_{0} $$

And move on to time step 2:
$$\mathbf{a}_{2} = \mathbf{W}\mathbf{a}_{1} $$

In the above equation, we can subsitute in $\mathbf{a}_{1} = \mathbf{W}\mathbf{a}_{0}$:
$$\mathbf{a}_{2} = \mathbf{W}\mathbf{W}\mathbf{a}_{0} = \mathbf{W}^2 \mathbf{a}_{0}$$

We can keep doing this with subsequent time steps:
$$\mathbf{a}_{3} = \mathbf{W}\mathbf{a}_{2} = \mathbf{W}\mathbf{W}^2 \mathbf{a}_{0} = \mathbf{W}^3\mathbf{a}_{0}  $$
$$\mathbf{a}_{4} = \mathbf{W}\mathbf{a}_{3} = \mathbf{W}\mathbf{W}^3 \mathbf{a}_{0} = \mathbf{W}^4\mathbf{a}_{0}  $$

This means that we can write the activity at any point as:
$$\mathbf{a}_{i} = \mathbf{W}^i\mathbf{a}_{0}  $$

**Initial firing rates along an eigenvector**

Remember from the last tutorial, that an eigenvector of matrix $\mathbf{W}$ is a vector that becomes a scalar multiple (eigenvalue) of itself when multiplied by that matrix:

$$\mathbf{W}\mathbf{v} = \lambda\mathbf{v}$$

Let's look at what happens if the initial firing rates in our neural circuit lie along that eigenvector, using the same substitution method as in the previous section:
$$\mathbf{a}_{0} = \mathbf{v} $$
$$\mathbf{a}_{1} = \mathbf{W}\mathbf{a}_0 = \mathbf{W}\mathbf{v} = \lambda\mathbf{v} $$
$$\mathbf{a}_{2} = \mathbf{W}\mathbf{a}_1 = \mathbf{W}\lambda\mathbf{v} = \lambda\mathbf{W}\mathbf{v} = \lambda^2\mathbf{v}$$
$$\mathbf{a}_{3} = \mathbf{W}\mathbf{a}_2 = \mathbf{W}\lambda^2\mathbf{v} = \lambda^2\mathbf{W}\mathbf{v} = \lambda^3\mathbf{v}$$
$$...$$
$$\mathbf{a}_i = \lambda^i\mathbf{v}$$

The activities at any time step equal a scalar times the initial activities. In other words, if the initial activities lie along an eigenvector, the activities will only evolve along that eigenvector.

### Interactive demo 2.1: Changing the eigenvalue

Let's visualize what happens if the initial activities of the neurons lie along an eigenvector and think about how this depends on the eigenvalue.

The interactive demo below is the same visualization you saw in Section 1, but now we also plot the eigenvectors $\mathbf{v}_1$ and $\mathbf{v}_2$.

Questions:
1.  What happens if the eigenvalue is large (2)?
2.  What happens if you move the eigenvalue from 2 to towards 0?
3.  What happens with negative eigenvalues?

```python
# @markdown Execute this cell to enable the widget. Please be patient for a few seconds after you change the slider

@widgets.interact(eigenvalue = widgets.FloatSlider(value=0.5, min=-2, max=2, step=0.2))
def plot_system(eigenvalue):

  # Get weight matrix with specified eigenvalues
  W = get_eigval_specified_matrix([eigenvalue, eigenvalue])

  # Get initial condition
  u0 = np.array([1, 1])

  # Get neural activities
  u = circuit_implementation(W, u0, 10)

  # Visualize neural activities
  plot_circuit_responses(u, W, eigenstuff = True, xlim = [-15, 15], ylim = [-15, 15])
```

```python
# to_remove explanation

# 1) With the eigenvalue = 2, the activities of the neurons explode towards infinity, along
#.   the eigenvector.

# 2) At eigenvalue = 1, there is a shift in what happens. With the eigenvalue above 1,
#.  the activites always explode. Once the eigenvalue is below 1, the activities decay to 0.
#.  If the eigenvalue equals 1, the activities never differ from the initial condition.
#.  This makes sense with the equation above. Lambda is raised to a power when computing activities:
#.  if it's a fraction, this term will get smaller so the activities will. If above 1, this term
#.   will explore so the activities will.

# 3) If the eigenvalue is between -1 and 0, the neural activities jump across the
#.   origin repeatedly along the eigenvector but eventually decay to 0. If the eigenvalue is below -1, the
#.   activities jump across the origin repeatedly along the eigenvector but explode to
#.   positive or negative infinity. Once again, this makes sense if you think through the equation above.
```

## Section 2.2: Understanding general dynamics using eigenstuff

```python
# @title Video 3: Understanding general dynamics using eigenstuff
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1fX4y1c7y7", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="66DExLQPzPI", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers how the eigenvalues and eigenvectors of the weight matrix affect the neural activities more generally.

 Click here for text recap of video

We now know that if our initial activities (or initial condition) fall on an eigenvector of $\mathbf{W}$, the activities will evolve along that line, either exploding to infinity if the absolute value of the eigenvalue is above 1 or decaying to the origin it it is below 1. What if our initial condition doesn't fall along the eigenvector though?

To understand what will happen, we will use the ideas of basis vectors and linear combinations from Tutorial 1.

Let's assume for now that our weight matrix has two distinct eigenvectors ($\mathbf{v}_1$ and $\mathbf{v}_2$) with corresponding eigenvalues $\lambda_1$ and $\lambda_2$, and that these eigenvectors form a basis for 2D space. That means we can write any vector in 2D space as a linear combination of our eigenvectors, including our initial activity vector:

$$\mathbf{a}_0 = c_1\mathbf{v}_1 + c_2\mathbf{v}_2 $$

Let's compute the next time step, using our previous strategy of substitution:
$$\begin{align}
\mathbf{a}_1 &= \mathbf{W}\mathbf{a}_0
\\ &= \mathbf{W}(c_1\mathbf{v}_1 + c_2\mathbf{v}_2) \\ &= c_1\mathbf{W}\mathbf{v}_1 + c_2\mathbf{W}\mathbf{v}_2 \\ &= c_1\lambda_1\mathbf{v}_1 + c_2\lambda_2\mathbf{v}_2 \end{align} $$

All activities can be written as:
$$\mathbf{a}_i = c_1\lambda_1^i\mathbf{v}_1 + c_2\lambda_2^i\mathbf{v}_2 $$

 We'll see what this means for our system in the next demo.

Before we get to the neural circuit, refresh your memory on linear combinations briefly by looking at our widget from tutorial 1 below. What happens when the absolute values of both scalar multiples of both vectors is big? What about when one is big and one is small?

```python
 #@markdown Make sure you execute this cell to enable the widget! Move the sliders for “a” and “b”. After releasing the slider, be patient for a couple of seconds to see the desired change.

def plot_arrows(x, y, a_times_x, b_times_y):
  fig, ax = plt.subplots(figsize=(10, 7))

  ax.spines['top'].set_color('none')
  ax.spines['bottom'].set_position('zero')
  ax.spines['left'].set_position('zero')
  ax.spines['right'].set_color('none')

  ax.set_aspect('equal', adjustable='box')
  ax.set(xlim = [-10, 10], ylim = [-10, 10], xticks = np.arange(-10, 10), yticks = np.arange(-10, 10), xticklabels = [], yticklabels = [])

  ax.grid(alpha=.4)

  z = a_times_x + b_times_y
  z_arr = ax.arrow(0, 0, z[0], z[1], width=.08, color='k', length_includes_head = True);

  x_orig, = ax.plot([0, x[0]], [0, x[1]], '--', color='#648FFF')
  y_orig, = ax.plot([0, y[0]], [0, y[1]], '--',  color='#DC267F')

  ax_arr = ax.arrow(0, 0, a_times_x[0], a_times_x[1], width=.08, color='#648FFF', length_includes_head = True);
  by_arr = ax.arrow(0, 0, b_times_y[0], b_times_y[1], width=.08, color='#DC267F', length_includes_head = True);

  ax.plot([a_times_x[0], z[0]], [a_times_x[1], z[1]], '--k')
  ax.plot([b_times_y[0], z[0]], [b_times_y[1], z[1]], '--k')

  leg = ax.legend([x_orig, y_orig, ax_arr, by_arr, z_arr], [r"$\mathbf{x}$", r"$\mathbf{y}$", r"$a\mathbf{x}$", r"$b\mathbf{y}$", r"$\mathbf{z} = a\mathbf{x} + b\mathbf{y}$"], handlelength = 2, fontsize = 25, loc = 'center left', bbox_to_anchor=(1.05, .5))
  for handle, label in zip(leg.legendHandles, leg.texts):
      try:
          label.set_color(handle.get_facecolor())
      except:
          label.set_color(handle.get_color())
      #handle.set_visible(False)

@widgets.interact(a = widgets.FloatSlider(value=1.0, min=-4, max=4, step=0.1), b = widgets.FloatSlider(value=1.0, min=-4, max=4, step=0.1))
def plot_linear_combination(a, b):
  x = np.array([3, 1])
  y = np.array([-1, 2])

  plot_arrows(x, y, a*x, b*y)
```

### Interactive demo 2.2: Changing both eigenvalues

In the demo below, you can now change both eigenvalues and the initial condition (with `a0_1` setting neuron 1 initial activity and `a0_2` setting neuron 2 initial activity). We will only look at positive eigenvalues to keep things a little more simple. We also make sure the second eigenvalue is always the smaller one (just for better visualization purposes).

Think each of the following questions through based on the equation we just arrived at and then play with the demo to see if you are correct.
$$\mathbf{a}_i = c_1\lambda_1^i\mathbf{v}_1 + c_2\lambda_2^i\mathbf{v}_2 $$

1.  What will happen when both eigenvalues are greater than 1? Does this depend on initial condition?  Set eigenvalue1 to 2 and eigenvalue2 to 1.2 and try out different initial conditions. What do you see?
2.  What will happen when both eigenvalues are less than 1?
3. What happens if one eigenvalue is below 1 and the other is above 1?

```python
# @markdown Execute this cell to enable the widget (there is a small lag so be patient after changing sliders)
eigenvalue1 = widgets.FloatSlider(value=0.5, min=0.2, max=2, step=0.2)
eigenvalue2 = widgets.FloatSlider(value=0.2, min=0.2, max=0.5, step=0.2)
a0_1 = widgets.FloatSlider(value=1, min=-5, max=5, step=0.2)
a0_2 = widgets.FloatSlider(value=2, min=-5, max=5, step=0.2)

def update_range(*args):
    eigenvalue2.max = eigenvalue1.value - 0.2
eigenvalue1.observe(update_range, 'value')

def plot_system(eigenvalue1, eigenvalue2, a0_1, a0_2):

  # Get initial condition
  a0 = np.array([a0_1, a0_2])

  # Get weight matrix with specified eigenvalues
  W = get_eigval_specified_matrix([eigenvalue1, eigenvalue2])

  # Get neural activities
  u = circuit_implementation(W, a0, 10)

  # Visualize neural activities
  plot_circuit_responses(u, W, eigenstuff = True, xlim = [-15, 15], ylim = [-15, 15])

widgets.interact(plot_system, eigenvalue1 = eigenvalue1, eigenvalue2 = eigenvalue2, a0_1 = a0_1, a0_2 = a0_2)
```

```python
# to_remove explanation

# 1) If both eigenvalues are above 1, the neural activities will eventually explode
#.   to infinity or negative infinity, depending on initial conditions. Tthe exact trajectory
#.   is drawn towards the eigenvector with the larger eigenvalue. This is because the larger eigenvalue
#.   will increasingly dominate the other one as it is raised to increasingly larger powers.

# 2) If both eigenvalues are below 1, the neural activity will eventually decay to 0.

#. 3) The activities will eventually explode to positive or negative infinity, unless
#.   the initial condition lies exactly on the eigenvector with the small eigenvalue. If the
#.   initial condition is near to that eigenvector, the trajectory will first go towards
#.   the origin before exploding.
```

## Section 2.3: Complex eigenvalues

We've been hiding some complexity from you up until now, namely that eigenvalues can be complex. Complex eigenvalues result in a very specific type of dynamics: rotations.

We will not delve into the proof or intuition behind this here as you'll encounter complex eigenvalues in dynamical systems in W2D2: Linear Dynamics.

Instead, we will simply demonstrate how the nature of the rotations depends on the complex eigenvalues in the animation below. We plot a 3-neuron circuit to better show the rotations. We illustrate each of the following:

*   Complex eigenvalues with an absolute value equal to 1 result in a sustained rotation in 3D space.

*   Complex eigenvalues with an absolute value below 1 result in a rotation towards the origin.

*   Complex eigenvalues with an absolute value above 1 result in a rotation towards the positive/negative infinity.

---
# Summary

You have seen how we can predict what happens in a discrete dynamical system with an update rule of:
$$ \mathbf{a}_t = \mathbf{W}\mathbf{a}_{t-1}$$

The most important takeaway is that inspecting eigenvalues and eigenvectors enables you to predict how discrete dynamical systems evolve. Specifically:

*  If all eigenvalues are real and have absolute values above 1, the neural activities explode to infinity or negative infinity.

*   If all eigenvalues are real and have absolute values below 1, the neural activities decay to 0.

*   If all eigenvalues are real and at least one has an absolute value above 1, the neural activities explode to infinity or negative infinity, except for special cases where the initial condition lies along an eigenvector with an eigenvalue whose absolute value is below 1.

* If eigenvalues are complex, the neural activities rotate in space and decay or explode depending on the amplitude of the complex eigenvalues.

* Even finer details of the trajectories can be predicted by examining the exact relationship of eigenvalues and eigenvectors.

Importantly, these ideas extend far beyond our toy neural circuit. Discrete dynamical systems with the same structure of update rule are common. While the exact dependencies on eigenvalues will change, we will see that we can still use eigenvalues/vectors to understand continuous dynamical systems in W2D2: Linear Dynamics.