# Vector-space models: dimensionality reduction

```python
__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Spring 2021"
```

## Contents

1. [Overview](#Overview)
1. [Set-up](#Set-up)
1. [Latent Semantic Analysis](#Latent-Semantic-Analysis)
  1. [Overview of the LSA method](#Overview-of-the-LSA-method)
  1. [Motivating example for LSA](#Motivating-example-for-LSA)
  1. [Applying LSA to real VSMs](#Applying-LSA-to-real-VSMs)
  1. [Other resources for matrix factorization](#Other-resources-for-matrix-factorization)
1. [GloVe](#GloVe)
  1. [Overview of the GloVe method](#Overview-of-the-GloVe-method)
  1. [GloVe implementation notes](#GloVe-implementation-notes)
  1. [Applying GloVe to our motivating example](#Applying-GloVe-to-our-motivating-example)
  1. [Testing the GloVe implementation](#Testing-the-GloVe-implementation)
  1. [Applying GloVe to real VSMs](#Applying-GloVe-to-real-VSMs)
1. [Autoencoders](#Autoencoders)
  1. [Overview of the autoencoder method](#Overview-of-the-autoencoder-method)
  1. [Testing the autoencoder implementation](#Testing-the-autoencoder-implementation)
  1. [Applying autoencoders to real VSMs](#Applying-autoencoders-to-real-VSMs)
1. [Other methods](#Other-methods)

## Overview

The matrix weighting schemes reviewed in the first notebook for this unit deliver solid results. However, they are not capable of capturing higher-order associations in the data.

With dimensionality reduction, the goal is to eliminate correlations in the input VSM and capture such higher-order notions of co-occurrence, thereby improving the overall space.

As a motivating example, consider the adjectives _gnarly_ and _wicked_ used as slang positive adjectives.  Since both are positive, we expect them to be similar in a good VSM. However, at least stereotypically, _gnarly_ is Californian and _wicked_ is Bostonian. Thus, they are unlikely to occur often in the same texts, and so the methods we've reviewed so far will not be able to model their similarity.

Dimensionality reduction techniques are often capable of capturing such semantic similarities (and have the added advantage of shrinking the size of our data structures).

## Set-up

* Make sure your environment meets all the requirements for [the cs224u repository](https://github.com/cgpotts/cs224u/). For help getting set-up, see [setup.ipynb](setup.ipynb).

* Make sure you've downloaded [the data distribution for this course](http://web.stanford.edu/class/cs224u/data/data.tgz), unpacked it, and placed it in the current directory (or wherever you point `DATA_HOME` to below).

```python
from torch_glove import TorchGloVe
import numpy as np
from np_glove import GloVe
import os
import pandas as pd
import scipy.stats
from torch_autoencoder import TorchAutoencoder
import utils
import vsm
```

```python
# Set all the random seeds for reproducibility:

utils.fix_random_seeds()
```

```python
DATA_HOME = os.path.join('data', 'vsmdata')
```

```python
yelp5 = pd.read_csv(
    os.path.join(DATA_HOME, 'yelp_window5-scaled.csv.gz'), index_col=0)
```

```python
yelp20 = pd.read_csv(
    os.path.join(DATA_HOME, 'yelp_window20-flat.csv.gz'), index_col=0)
```

```python
giga5 = pd.read_csv(
    os.path.join(DATA_HOME, 'giga_window5-scaled.csv.gz'), index_col=0)
```

```python
giga20 = pd.read_csv(
    os.path.join(DATA_HOME, 'giga_window20-flat.csv.gz'), index_col=0)
```

## Latent Semantic Analysis

Latent Semantic Analysis (LSA) is a prominent dimensionality reduction technique. It is an application of __truncated singular value decomposition__ (SVD) and so uses only techniques from linear algebra (no machine learning needed).

### Overview of the LSA method

The central mathematical result is that, for any matrix of real numbers $X$ of dimension $m \times n$, there is a factorization of $X$ into matrices $T$, $S$, and $D$ such that

$$X_{m \times n} = T_{m \times m}S_{m\times m}D_{n \times m}^{\top}$$

The matrices $T$ and $D$ are  __orthonormal__ – their columns are length-normalized and orthogonal to one another (that is, they each have cosine distance of $1$ from each other). The singular-value matrix $S$ is a diagonal matrix arranged by size, so that the first dimension corresponds to the greatest source of variability in the data, followed by the second, and so on.

Of course, we don't want to factorize and rebuild the original matrix, as that wouldn't get us anywhere. The __truncation__ part means that we include only the top $k$ dimensions of $S$. Given our row-oriented perspective on these matrices, this means using

$$T[1{:}m, 1{:}k]S[1{:}k, 1{:}k]$$

which gives us a version of $T$ that includes only the top $k$ dimensions of variation.

To build up intuitions, imagine that everyone on the Stanford campus is associated with a 3d point representing their position: $x$ is east–west, $y$ is north–south, and $z$ is zenith–nadir. Since the campus is spread out and has relatively few deep basements and tall buildings, the top two dimensions of variation will be $x$ and $y$, and the 2d truncated SVD of this space will leave $z$ out. This will, for example, capture the sense in which someone at the top of Hoover Tower is close to someone at its base.

### Motivating example for LSA

We can also return to our original motivating example of _wicked_ and _gnarly_. Here is a matrix reflecting those assumptions:

```python
gnarly_df = pd.DataFrame(
    np.array([
        [1,0,1,0,0,0],
        [0,1,0,1,0,0],
        [1,1,1,1,0,0],
        [0,0,0,0,1,1],
        [0,0,0,0,0,1]], dtype='float64'),
    index=['gnarly', 'wicked', 'awesome', 'lame', 'terrible'])

gnarly_df
```

      0
      1
      2
      3
      4
      5

      gnarly
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0

      wicked
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0

      awesome
      1.0
      1.0
      1.0
      1.0
      0.0
      0.0

      lame
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0

      terrible
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0

No column context includes both _gnarly_ and _wicked_ together so our count matrix places them far apart:

```python
vsm.neighbors('gnarly', gnarly_df)
```

    gnarly      0.000000
    awesome     0.292893
    wicked      1.000000
    lame        1.000000
    terrible    1.000000
    dtype: float64

Reweighting doesn't help. For example, here is the attempt with Positive PMI:

```python
vsm.neighbors('gnarly', vsm.pmi(gnarly_df))
```

    gnarly      0.000000
    awesome     0.292893
    wicked      1.000000
    lame        1.000000
    terrible    1.000000
    dtype: float64

However, both words tend to occur with _awesome_ and not with _lame_ or _terrible_, so there is an important sense in which they are similar. LSA to the rescue:

```python
gnarly_lsa_df = vsm.lsa(gnarly_df, k=2)
```

```python
vsm.neighbors('gnarly', gnarly_lsa_df)
```

    gnarly      0.0
    wicked      0.0
    awesome     0.0
    lame        1.0
    terrible    1.0
    dtype: float64

### Applying LSA to real VSMs

Here's an example that begins to convey the effect that this can have empirically.

First, the original count matrix:

```python
vsm.neighbors('superb', yelp5).head()
```

    superb         0.000000
    phenomenal     0.012159
    outstanding    0.012170
    fantastic      0.015080
    terrific       0.016488
    dtype: float64

And then LSA with $k=100$:

```python
yelp5_svd = vsm.lsa(yelp5, k=100)
```

```python
vsm.neighbors('superb', yelp5_svd).head()
```

    superb         0.000000
    outstanding    0.009402
    phenomenal     0.009575
    fantastic      0.012573
    terrific       0.014234
    dtype: float64

A common pattern in the literature is to apply PMI first. The PMI values tend to give the count matrix a normal (Gaussian) distribution that better satisfies the assumptions underlying SVD:

```python
yelp5_pmi = vsm.pmi(yelp5, positive=False)
```

```python
yelp5_pmi_svd = vsm.lsa(yelp5_pmi, k=100)
```

```python
vsm.neighbors('superb', yelp5_pmi_svd).head()
```

    superb         0.000000
    phenomenal     0.040337
    outstanding    0.066857
    terrific       0.068770
    exceptional    0.077025
    dtype: float64

```python
yelp5_pmi_svd
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      90
      91
      92
      93
      94
      95
      96
      97
      98
      99

      ):
      1.437366
      4.557234
      34.581427
      9.904594
      -8.836850
      4.783611
      -5.923757
      2.365317
      7.153208
      -3.319591
      ...
      0.161369
      3.228617
      -0.961274
      0.749047
      1.480295
      1.744890
      -0.898080
      -1.584570
      -0.278718
      -0.342861

      );
      -7.331655
      13.317027
      26.784239
      5.835746
      0.027927
      2.612011
      -2.192629
      -0.625491
      6.705506
      -3.204643
      ...
      -0.236299
      -0.076758
      -0.232908
      2.437669
      -0.948463
      0.215719
      -0.099084
      -0.859263
      -0.878322
      1.954915

      ..
      3.575590
      -11.150100
      0.966626
      2.349962
      0.525188
      3.748029
      1.536606
      7.153553
      1.394925
      5.795305
      ...
      -1.613267
      0.504108
      -1.515366
      2.197602
      -2.012526
      -0.583324
      1.020110
      1.433105
      -0.981065
      -0.795687

      ...
      1.664805
      -4.708962
      5.065324
      0.086736
      0.153097
      2.753349
      -0.292617
      9.542095
      2.394887
      2.240579
      ...
      -1.766309
      0.502418
      -1.171912
      2.057251
      -1.546699
      0.204402
      0.735876
      1.326382
      -0.626003
      -1.043260

      :(
      14.331547
      -4.051552
      5.137324
      6.297147
      5.410325
      3.981367
      -8.355681
      13.946015
      6.456173
      -7.361078
      ...
      -1.545323
      3.536775
      1.428976
      4.285336
      0.113137
      -1.455572
      0.270583
      -1.169132
      -1.769784
      1.236976

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      zero
      42.922118
      2.265832
      -14.099342
      13.495838
      -0.442400
      -17.403860
      -5.249248
      2.670833
      11.660561
      0.130391
      ...
      5.798536
      5.045399
      -2.892084
      2.124796
      -3.291423
      -3.334803
      1.771991
      -0.510973
      1.014089
      4.155700

      zinc
      -9.117357
      6.573923
      4.816907
      -4.864993
      -2.093645
      2.011948
      -1.687539
      -0.235476
      -4.776726
      2.372491
      ...
      -2.486255
      -0.123244
      -0.886462
      0.020586
      -0.876515
      0.173563
      2.597190
      1.248349
      -0.986036
      -2.080470

      zombie
      0.020414
      28.218548
      10.308875
      -5.465735
      3.253382
      4.134910
      7.776402
      8.265688
      -2.723968
      3.467228
      ...
      2.290680
      -0.096824
      -2.651254
      2.515903
      0.775413
      -8.932563
      0.587445
      -0.743377
      0.531828
      2.809857

      zone
      22.817412
      25.879715
      2.803584
      -11.637706
      7.990066
      3.861264
      -1.410138
      6.432013
      -1.879044
      -4.352102
      ...
      1.634279
      2.009811
      -0.138421
      0.331866
      -2.127628
      -0.542791
      2.962847
      1.934855
      -0.921823
      2.389236

      zoo
      21.876643
      22.017796
      9.407152
      -16.740864
      1.089437
      3.624366
      -4.857669
      3.518619
      -8.020991
      -3.000732
      ...
      5.110539
      -5.617114
      -2.495130
      -2.850441
      2.527100
      1.498996
      3.236248
      1.660042
      -1.213899
      -0.642195

6000 rows × 100 columns

### Other resources for matrix factorization

The [sklearn.decomposition](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) module contains an implementation of LSA ([TruncatedSVD](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html#sklearn.decomposition.TruncatedSVD)) that you might want to switch to for real experiments:

* The `sklearn` version is more flexible than the above in that it can operate on both dense matrices (Numpy arrays) and sparse matrices (from Scipy).

* The `sklearn` version will make it easy to try out other dimensionality reduction methods in your own code; [Principal Component Analysis (PCA)](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA) and [Non-Negative Matrix Factorization (NMF)](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html#sklearn.decomposition.NMF) are closely related methods that are worth a look.

## GloVe

### Overview of the GloVe method

[Pennington et al. (2014)](http://www.aclweb.org/anthology/D/D14/D14-1162.pdf) introduce an objective function for semantic word representations. Roughly speaking, the objective is to learn vectors for words $w_{i}$ and $w_{j}$ such that their dot product is proportional to their probability of co-occurrence:

$$w_{i}^{\top}\widetilde{w}_{k} + b_{i} + \widetilde{b}_{k} = \log(X_{ik})$$

The paper is exceptionally good at motivating this objective from first principles. In their equation (6), they define

$$w_{i}^{\top}\widetilde{w}_{k} = \log(P_{ik}) = \log(X_{ik}) - \log(X_{i})$$

If we allow that the rows and columns can be different, then we would do

$$w_{i}^{\top}\widetilde{w}_{k} = \log(P_{ik}) = \log(X_{ik}) - \log(X_{i} \cdot X_{*k})$$

where, as in the paper, $X_{i}$ is the sum of the values in row $i$, and $X_{*k}$ is the sum of the values in column $k$.

The rightmost expression is PMI by the equivalence $\log(\frac{x}{y}) = \log(x) - \log(y)$, and hence we can see GloVe as aiming to make the dot product of two learned vectors equal to the PMI!

The full model is a weighting of this objective:

$$\sum_{i, j=1}^{|V|} f\left(X_{ij}\right)
  \left(w_i^\top \widetilde{w}_j + b_i + \widetilde{b}_j - \log X_{ij}\right)^2$$

where $V$ is the vocabulary and $f$ is a scaling factor designed to diminish the impact of very large co-occurrence counts:

$$f(x)
\begin{cases}
(x/x_{\max})^{\alpha} & \textrm{if } x < x_{\max} \\
1 & \textrm{otherwise}
\end{cases}$$

Typically, $\alpha$ is set to $0.75$ and $x_{\max}$ to $100$ (though it is worth assessing how many of your non-zero counts are above this; in dense word $\times$ word matrices, you could be flattening more than you want to).

### GloVe implementation notes

* The implementation in `np_glove.GloVe` is the most stripped-down, bare-bones version of the GloVe method I could think of. As such, it is quite slow.

* `torch_glove.py` includes a vectorized implementation that is much, much faster, so we'll mainly use that. It is based on the method and implementation introduced in [Dingwall and Potts 2018](https://www.aclweb.org/anthology/N18-2034/).

* For really large jobs, [the official C implementation released by the GloVe team](http://nlp.stanford.edu/projects/glove/) is probably the best choice.

### Applying GloVe to our motivating example

GloVe should do well on our _gnarly/wicked_ evaluation, though you will see a lot variation due to the small size of this VSM:

```python
gnarly_glove_mod = GloVe(n=2, max_iter=1000)
```

```python
gnarly_glove = gnarly_glove_mod.fit(gnarly_df)
```

    Finished epoch 999 of 1000; error is 7.405087355137681e-059

```python
vsm.neighbors('gnarly', gnarly_glove)
```

    gnarly      0.000000
    awesome     0.176390
    wicked      0.309273
    lame        0.418618
    terrible    0.557261
    dtype: float64

### Testing the GloVe implementation

It is not easy analyze GloVe values derived from real data, but the following little simulation suggests that `np_glove.glove` is working as advertised: it does seem to reliably deliver vectors whose dot products are proportional to the log co-occurrence probability:

```python
glove_test_count_df = pd.DataFrame(
    np.array([
        [10.0,  2.0,  3.0,  4.0],
        [ 2.0, 10.0,  4.0,  1.0],
        [ 3.0,  4.0, 10.0,  2.0],
        [ 4.0,  1.0,  2.0, 10.0]]),
    index=['A', 'B', 'C', 'D'],
    columns=['A', 'B', 'C', 'D'])
```

```python
glove_test_count_df
```

      A
      B
      C
      D

      A
      10.0
      2.0
      3.0
      4.0

      B
      2.0
      10.0
      4.0
      1.0

      C
      3.0
      4.0
      10.0
      2.0

      D
      4.0
      1.0
      2.0
      10.0

```python
glove_test_mod = GloVe(n=4, max_iter=1000)
```

```python
glove_test_df = glove_test_mod.fit(glove_test_count_df)
```

    Converged on iteration 847 with error 9.943994380114379e-06

The `score` method for `GloVe` performs the correlation test that seems appropriate for GloVe. We can roughly summarize the test as follows: calculate the Pearson correlation of `G.dot(G.T)` and `log P(X)`, where `G` is the learned GloVe embedding and `log P(X)` gives the log probabilities. The test exempts `0` count cells when calculating the correlation.

```python
glove_test_mod.score(glove_test_count_df)
```

    0.9573164440840088

### Applying GloVe to real VSMs

The `vsm.glove` implementation is too slow to use on real matrices. The version in the `torch_glove` module is significantly faster, making its use possible even without a GPU (and it will be very fast indeed on a GPU machine):

```python
glove_model = TorchGloVe(max_iter=100)

yelp5_glv = glove_model.fit(yelp5)
```

    Finished epoch 100 of 100; error is 1713142.87555

We can again use the `score` method to see how we did:

```python
glove_model.score(yelp5)
```

    0.3325292290071531

```python
vsm.neighbors('superb', yelp5_glv).head()
```

    superb       0.000000
    famous       0.004735
    knowledge    0.005022
    express      0.005296
    sea          0.005308
    dtype: float64

## Autoencoders

An autoencoder is a machine learning model that seeks to learn parameters that predict its own input. This is meaningful when there are intermediate representations that have lower dimensionality than the inputs. These provide a reduced-dimensional view of the data akin to those learned by LSA, but now we have a lot more design choices and a lot more potential to learn higher-order associations in the underyling data.

### Overview of the autoencoder method

The module `torch_autoencoder` uses PyToch to implement a simple one-layer autoencoder:

$$
\begin{align}
h &= \mathbf{f}(xW + b_{h}) \\
\widehat{x} &= hW^{\top} + b_{x}
\end{align}$$

Here, we assume that the hidden representation $h$ has a low dimensionality like 100, and that $\mathbf{f}$ is a non-linear activation function (the default for `TorchAutoencoder` is `tanh`). These are the major design choices internal to the network. It might also be meaningful to assume that there are two matrices of weights $W_{xh}$ and $W_{hx}$, rather than using $W^{\top}$ for the output step.

The objective function for autoencoders will implement some kind of assessment of the distance between the inputs and their predicted outputs. For example, one could use the one-half mean squared error:

$$\frac{1}{m}\sum_{i=1}^{m} \frac{1}{2}(\widehat{X[i]} - X[i])^{2}$$

where $X$ is the input matrix of examples (dimension $m \times n$) and $X[i]$ corresponds to the $i$th example.

When you call the `fit` method of `TorchAutoencoder`, it returns the matrix of hidden representations $h$, which is the new embedding space: same row count as the input, but with the column count set by the `hidden_dim` parameter.

For much more on autoencoders, see the 'Autoencoders' chapter of [Goodfellow et al. 2016](http://www.deeplearningbook.org).

### Testing the autoencoder implementation

Here's an evaluation that is meant to test the autoencoder implementation – we expect it to be able to full encode the input matrix because we know its rank is equal to the dimensionality of the hidden representation.

```python
def randmatrix(m, n, sigma=0.1, mu=0):
    return sigma * np.random.randn(m, n) + mu

def autoencoder_evaluation(nrow=1000, ncol=100, rank=20, max_iter=20000):
    """This an evaluation in which `TorchAutoencoder` should be able
    to perfectly reconstruct the input data, because the
    hidden representations have the same dimensionality as
    the rank of the input matrix.
    """
    X = randmatrix(nrow, rank).dot(randmatrix(rank, ncol))
    ae = TorchAutoencoder(hidden_dim=rank, max_iter=max_iter)
    ae.fit(X)
    X_pred = ae.predict(X)
    mse = (0.5 * (X_pred - X)**2).mean()
    return(X, X_pred, mse)
```

```python
ae_max_iter = 100

_, _, ae = autoencoder_evaluation(max_iter=ae_max_iter)

print("Autoencoder evaluation MSE after {0} evaluations: {1:0.04f}".format(
    ae_max_iter, ae))
```

    Finished epoch 100 of 100; error is 0.0015143734635785222

    Autoencoder evaluation MSE after 100 evaluations: 0.0008

### Applying autoencoders to real VSMs

You can apply the autoencoder directly to the count matrix, but this could interact very badly with the internal activation function: if the counts are all very high or very low, then everything might get pushed irrevocably towards the extreme values of the activation.

Thus, it's a good idea to first normalize the values somehow. Here, I use `vsm.length_norm`:

```python
yelp5_l2 = yelp5.apply(vsm.length_norm, axis=1)
```

```python
yelp5_l2_ae = TorchAutoencoder(
    max_iter=100, hidden_dim=50, eta=0.001).fit(yelp5_l2)
```

    Stopping after epoch 28. Training loss did not improve more than tol=1e-05. Final error is 0.0003644404678198043.

```python
vsm.neighbors('superb', yelp5_l2_ae).head()
```

    superb         0.000000
    fantastic      0.000021
    outstanding    0.000023
    terrific       0.000025
    phenomenal     0.000036
    dtype: float64

This is very slow and often seems not to work all that well. To speed things up, one can first apply LSA or similar:

```python
yelp5_l2_svd100 = vsm.lsa(yelp5_l2, k=100)
```

```python
yelp_l2_svd100_ae = TorchAutoencoder(
    max_iter=1000, hidden_dim=50, eta=0.01).fit(yelp5_l2_svd100)
```

    Stopping after epoch 56. Training loss did not improve more than tol=1e-05. Final error is 0.0014757063472643495.

```python
vsm.neighbors('superb', yelp_l2_svd100_ae).head()
```

    superb         0.000000
    phenomenal     0.014526
    outstanding    0.016242
    fantastic      0.017960
    terrific       0.022512
    dtype: float64

## Other methods

Learning word representations is one of the most active areas in NLP right now, so I can't hope to offer a comprehensive summary. I'll settle instead for identifying some overall trends and methods:

* The label __word2vec__ picks out a family of models in which the embedding for a word $w$ is trained to predict the words that co-occur with $w$. This intuition can be cashed out in numerous ways. The skip-gram model, due to [Mikolov et al. 2013](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality), is arguably the closest to the models discussed above. For instance, see [Levy and Golberg 2014](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization) for a proof that the skip-gram model's objective reduces to PMI shifted by a constant value.

* The LexVec model of [Salle et al. 2016](https://www.aclweb.org/anthology/P16-2068/) combines the core insight of GloVe (learn vectors that approximate PMI) with the insight from word2vec that we should additionally try to push words that don't appear together farther apart in the VSM. (GloVe simply ignores 0 count cells and so can't do this.)

* Many apparently diverse models can be expressed as matrix factorization methods like SVD/LSA. See especially
[Singh and Gordon 2008](http://www.cs.cmu.edu/~ggordon/singh-gordon-unified-factorization-ecml.pdf),
[Levy and Golberg 2014](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization), [Cotterell et al. 2017](https://www.aclweb.org/anthology/E17-2028/).

* Subword modeling ([reviewed briefly in the previous notebook](vsm_01_distributional.ipynb#Subword-information)) is increasingly yielding dividends. (It would already be central if most of NLP focused on languages with complex morphology!) Check out the papers at the Subword and Character-Level Models for NLP Workshops: [SCLeM 2017](https://sites.google.com/view/sclem2017/home), [SCLeM 2018](https://sites.google.com/view/sclem2018/home).

* Contextualized word representations have proven valuable in many contexts. These methods do not provide representations for individual words, but rather represent them in their linguistic context. This creates space for modeling how word senses vary depending on their context of use. See [vsm_04_contextualreps.ipynb](vsm_04_contextualreps.ipynb) for techniques for using such models to create VSMs like those explored above.


Observationally, studies of doppler shift maps have shown that many dense cores have measured non-negligible,systematic velocities. This is important because the centrifugal force introduces an additional support for the sphere to counter the gravitational infall.

While it is easy to think about a rotating sphere, putting rotation on a Cartesian grid is a bit more tricky, in particular, we need to enforce:

   1. Solid body rotation $\leftrightarrow d\Omega/dr = 0$.
   2. Everything should be rotating in a plane $v_z=0$ (angular momentum in the $z$-direction).

We can get the $\Omega$ from either the energy-balance or the force-balance approach, where $\beta$ is our tuning parameter and determines whether the sphere is rotationally-supported or collapsing.
\begin{align}
\beta = \frac{E_{k,rot}}{E_{p,grav}}
\\ \boxed{\Omega \approx\sqrt{\frac{\beta GM}{R^3}}}
\end{align}
Given x, y, z,$\Omega$ , we can get:
\begin{align}
\theta = cos^{-1}(\frac{z}{r})
\\ \boxed{\phi = tan^{-1}(\frac{y}{x})}
\\ r = \sqrt{x^2+y^2+z^2}
\end{align}
From the conservation of angular momentum (things should rotate in a plane), $V_z = 0$.  Enforcing solid body rotation  $$\Omega = \frac{|V|}{r}= \frac{\sqrt{V_x^2+V_y^2}}{r}$$.

We can solve this by two approaches to enforce circular rotation:

###  $\vec{\rho}\perp\vec{v}$ Approach
Let $\vec{\rho}$ be the cylindrical radial vector
\begin{align}
\vec{\rho} = \vec{r}sin\theta
\\ \vec{\rho}\cdot \vec{v} = 0
\\ V_x  = -V_y tan\phi
\end{align}
when $\theta \neq 0,\pi$. (Note that Method 2 doesn't have this constraint on $\theta$. )

### Geometric Approach
Looking at a top-down view, we can determine geometrically:
\begin{align}
V_y = - v sin\phi
\\ V_x  = v cos\phi
\end{align}
Dividing the two we get: $V_x  = -V_y tan\phi$.
\begin{align}
\boxed{V_y = - \Omega r sin\phi}
\\ \boxed{V_x  =  \Omega r cos\phi}
\end{align}

```python
%pylab inline
from  scripts.plotSim import *
```

    Populating the interactive namespace from numpy and matplotlib

Setup :

    ./setup RotatingSinkSphere -3d +usm --maxblock=500 -auto ; cd object/;make -j8;cd ..;

### Initial Conditions

~~~fortran
!Computed from 3*G*M/(rmax/1.057e-17)**3
omega = sqrt(2.2196e-25*beta_param)
...
rr = sqrt(xx**2 + yy**2 + zz**2)
phi =  atan(yy/xx)
....
velxZone = abs(omega*rr*sin(phi))
velyZone = abs(omega*rr*cos(phi))

if (xx<0) then
   velyZone=-velyZone
endif
if (yy>0) then
   velxZone=-velxZone
endif
if (rc >=18.719) then
   velxZone = velxZone*exp(-(rc-18.719)/9.35)
   velyZone = velyZone*exp(-(rc-18.719)/9.35)
endif
velzZone = 0.0
~~~

Note that we need to explicitly specify the direction of the velocities because the arctangent that is computed in fortran only return ranges from $\epsilon[0,\pi/2]$. We also damp the velocity outside of the sphere, so that the velocity at the edge of the box is not too high. (The only important physics in the problem is the sphere itself rotating)

### Running the example

Code example up to now is in ``RotatingSinkSphere/``.

```python
cd ../proj/dlee/FLASH4.3/object/
```

    /global/project/projectdirs/astro250/dlee/FLASH4.3/object

```python
plot_dens(612)
```

<br>

```python
from IPython.display import YouTubeVideo
YouTubeVideo("XnIdxSPN0_A",width=520,height=420)
```


# Tutorial 1: GLMs for Encoding
**Week 1, Day 4: Generalized Linear Models**

**By Neuromatch Academy**

__Content creators:__ Pierre-Etienne H. Fiquet, Ari Benjamin, Jakob Macke

__Content reviewers:__ Davide Valeriani, Alish Dipani, Michael Waskom, Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

# Tutorial Objectives

*Estimated timing of tutorial: 1 hour, 15 minutes*

This is part 1 of a 2-part series about Generalized Linear Models (GLMs), which are a fundamental framework for supervised learning.

In this tutorial, the objective is to model a retinal ganglion cell spike train by fitting a temporal receptive field. First with a Linear-Gaussian GLM (also known as ordinary least-squares regression model) and then with a Poisson GLM (aka "Linear-Nonlinear-Poisson" model). In the next tutorial, we’ll extend to a special case of GLMs, logistic regression, and learn how to ensure good model performance.

This tutorial is designed to run with retinal ganglion cell spike train data from [Uzzell & Chichilnisky 2004](https://journals.physiology.org/doi/full/10.1152/jn.01171.2003?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%20%200pubmed).

*Acknowledgements:*

- We thank EJ Chichilnisky for providing the dataset. Please note that it is provided for tutorial purposes only, and should not be distributed or used for publication without express permission from the author (ej@stanford.edu).
- We thank Jonathan Pillow, much of this tutorial is inspired by exercises asigned in his 'Statistical Modeling and Analysis of Neural Data' class.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/upyjz/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

# Setup

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
from scipy.io import loadmat
```

```python
#@title Figure settings
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Plotting Functions

def plot_stim_and_spikes(stim, spikes, dt, nt=120):
  """Show time series of stim intensity and spike counts.

  Args:
    stim (1D array): vector of stimulus intensities
    spikes (1D array): vector of spike counts
    dt (number): duration of each time step
    nt (number): number of time steps to plot

  """
  timepoints = np.arange(120)
  time = timepoints * dt

  f, (ax_stim, ax_spikes) = plt.subplots(
    nrows=2, sharex=True, figsize=(8, 5),
  )
  ax_stim.plot(time, stim[timepoints])
  ax_stim.set_ylabel('Stimulus intensity')

  ax_spikes.plot(time, spikes[timepoints])
  ax_spikes.set_xlabel('Time (s)')
  ax_spikes.set_ylabel('Number of spikes')

  f.tight_layout()

def plot_glm_matrices(X, y, nt=50):
  """Show X and Y as heatmaps.

  Args:
    X (2D array): Design matrix.
    y (1D or 2D array): Target vector.

  """
  from matplotlib.colors import BoundaryNorm
  from mpl_toolkits.axes_grid1 import make_axes_locatable
  Y = np.c_[y]  # Ensure Y is 2D and skinny

  f, (ax_x, ax_y) = plt.subplots(
    ncols=2,
    figsize=(6, 8),
    sharey=True,
    gridspec_kw=dict(width_ratios=(5, 1)),
  )
  norm = BoundaryNorm([-1, -.2, .2, 1], 256)
  imx = ax_x.pcolormesh(X[:nt], cmap="coolwarm", norm=norm)

  ax_x.set(
    title="X\n(lagged stimulus)",
    xlabel="Time lag (time bins)",
    xticks=[4, 14, 24],
    xticklabels=['-20', '-10', '0'],
    ylabel="Time point (time bins)",
  )
  plt.setp(ax_x.spines.values(), visible=True)

  divx = make_axes_locatable(ax_x)
  caxx = divx.append_axes("right", size="5%", pad=0.1)
  cbarx = f.colorbar(imx, cax=caxx)
  cbarx.set_ticks([-.6, 0, .6])
  cbarx.set_ticklabels(np.sort(np.unique(X)))

  norm = BoundaryNorm(np.arange(y.max() + 1), 256)
  imy = ax_y.pcolormesh(Y[:nt], cmap="magma", norm=norm)
  ax_y.set(
    title="Y\n(spike count)",
    xticks=[]
  )
  ax_y.invert_yaxis()
  plt.setp(ax_y.spines.values(), visible=True)

  divy = make_axes_locatable(ax_y)
  caxy = divy.append_axes("right", size="30%", pad=0.1)
  cbary = f.colorbar(imy, cax=caxy)
  cbary.set_ticks(np.arange(y.max()) + .5)
  cbary.set_ticklabels(np.arange(y.max()))

def plot_spike_filter(theta, dt, **kws):
  """Plot estimated weights based on time lag model.

  Args:
    theta (1D array): Filter weights, not including DC term.
    dt (number): Duration of each time bin.
    kws: Pass additional keyword arguments to plot()

  """
  d = len(theta)
  t = np.arange(-d + 1, 1) * dt

  ax = plt.gca()
  ax.plot(t, theta, marker="o", **kws)
  ax.axhline(0, color=".2", linestyle="--", zorder=1)
  ax.set(
    xlabel="Time before spike (s)",
    ylabel="Filter weight",
  )

def plot_spikes_with_prediction(
    spikes, predicted_spikes, dt, nt=50, t0=120, **kws):
  """Plot actual and predicted spike counts.

  Args:
    spikes (1D array): Vector of actual spike counts
    predicted_spikes (1D array): Vector of predicted spike counts
    dt (number): Duration of each time bin.
    nt (number): Number of time bins to plot
    t0 (number): Index of first time bin to plot.
    kws: Pass additional keyword arguments to plot()

  """
  t = np.arange(t0, t0 + nt) * dt

  f, ax = plt.subplots()
  lines = ax.stem(t, spikes[:nt], use_line_collection=True)
  plt.setp(lines, color=".5")
  lines[-1].set_zorder(1)
  kws.setdefault("linewidth", 3)
  yhat, = ax.plot(t, predicted_spikes[:nt], **kws)
  ax.set(
      xlabel="Time (s)",
      ylabel="Spikes",
  )
  ax.yaxis.set_major_locator(plt.MaxNLocator(integer=True))
  ax.legend([lines[0], yhat], ["Spikes", "Predicted"])

  plt.show()
```

```python
#@title Data retrieval and loading
import os
import hashlib
import requests

fname = "RGCdata.mat"
url = "https://osf.io/mzujs/download"
expected_md5 = "1b2977453020bce5319f2608c94d38d0"

if not os.path.isfile(fname):
  try:
    r = requests.get(url)
  except requests.ConnectionError:
    print("!!! Failed to download data !!!")
  else:
    if r.status_code != requests.codes.ok:
      print("!!! Failed to download data !!!")
    elif hashlib.md5(r.content).hexdigest() != expected_md5:
      print("!!! Data download appears corrupted !!!")
    else:
      with open(fname, "wb") as fid:
        fid.write(r.content)
```

---
# Section 1: Linear-Gaussian GLM

```python
# @title Video 1: Linear Gaussian model
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV17T4y1E75x", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="Yv89UHeSa9I", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Section 1.1: Load retinal ganglion cell activity data

*Estimated timing to here from start of tutorial: 10 min*

In this exercise we use data from an experiment that presented a screen which randomly alternated between two luminance values and recorded responses from retinal ganglion cell (RGC), a type of neuron in the retina in the back of the eye. This kind of visual stimulus is called a "full-field flicker", and it was presented at ~120Hz (ie. the stimulus presented on the screen was refreshed about every 8ms). These same time bins were used to count the number of spikes emitted by each neuron.

The file `RGCdata.mat` contains three variablies:

- `Stim`, the stimulus intensity at each time point. It is an array with shape $T \times 1$, where $T=144051$.

-  `SpCounts`, the binned spike counts for 2 ON cells, and 2 OFF cells. It is a $144051 \times 4$ array, and each column has counts for a different cell.

- `dtStim`, the size of a single time bin (in seconds), which is needed for computing model output in units of spikes / s. The stimulus frame rate is given by `1 / dtStim`.

Because these data were saved in MATLAB, where everything is a matrix, we will also process the variables to more Pythonic representations (1D arrays or scalars, where appropriate) as we load the data.

```python
data = loadmat('RGCdata.mat')  # loadmat is a function in scipy.io
dt_stim = data['dtStim'].item()  # .item extracts a scalar value

# Extract the stimulus intensity
stim = data['Stim'].squeeze()  # .squeeze removes dimensions with 1 element

# Extract the spike counts for one cell
cellnum = 2
spikes = data['SpCounts'][:, cellnum]

# Don't use all of the timepoints in the dataset, for speed
keep_timepoints = 20000
stim = stim[:keep_timepoints]
spikes = spikes[:keep_timepoints]
```

Use the `plot_stim_and_spikes` helper function to visualize the changes in stimulus intensities and spike counts over time.

```python
plot_stim_and_spikes(stim, spikes, dt_stim)
```

### Coding Exercise 1.1: Create design matrix

Our goal is to predict the cell's activity from the stimulus intensities preceding it. That will help us understand how RGCs process information over time. To do so, we first need to create the *design matrix* for this model, which organizes the stimulus intensities in matrix form such that the $i$th row has the stimulus frames preceding timepoint $i$.

In this exercise, we will create the design matrix $\mathbf{X}$ using $d=25$ time lags. That is, $\mathbf{X}$ should be a $T \times d$ matrix. $d = 25$ (about 200 ms) is a choice we're making based on our prior knowledge of the temporal window that influences RGC responses. In practice, you might not know the right duration to use.

The last entry in row `t` should correspond to the stimulus that was shown at time `t`, the entry to the left of it should contain the value that was show one time bin earlier, etc. Specifically, $X_{ij}$ will be the stimulus intensity at time $i + d - 1 - j$.

Note that for the first few time bins, we have access to the recorded spike counts but not to the stimulus shown in the recent past. For simplicity we are going to assume that values of `stim` are 0 for the time lags prior to the first timepoint in the dataset. This is known as "zero-padding", so that the design matrix has the same number of rows as the response vectors in `spikes`.

Your task is is to complete the function below to:

  - make a zero-padded version of the stimulus
  - initialize an empty design matrix with the correct shape
  - **fill in each row of the design matrix, using the zero-padded version of the stimulus**

To visualize your design matrix (and the corresponding vector of spike counts), we will plot a "heatmap", which encodes the numerical value in each position of the matrix as a color. The helper functions include some code to do this.

```python
def make_design_matrix(stim, d=25):
  """Create time-lag design matrix from stimulus intensity vector.

  Args:
    stim (1D array): Stimulus intensity at each time point.
    d (number): Number of time lags to use.

  Returns
    X (2D array): GLM design matrix with shape T, d

  """

  # Create version of stimulus vector with zeros before onset
  padded_stim = np.concatenate([np.zeros(d - 1), stim])

  #####################################################################
  # Fill in missing code (...),
  # then remove or comment the line below to test your function
  # raise NotImplementedError("Complete the make_design_matrix function")
  #####################################################################

  # Construct a matrix where each row has the d frames of
  # the stimulus proceeding and including timepoint t
  T = len(stim)  # Total number of timepoints (hint: number of stimulus frames)
  X = np.zeros((T, d))
  for t in range(T):
      X[t] = X[t] = padded_stim[t:t + d]

  return X

# Make design matrix
X = make_design_matrix(stim)

# Visualize
plot_glm_matrices(X, spikes, nt=50)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D4_GeneralizedLinearModels/solutions/W1D4_Tutorial1_Solution_ce5673f1.py)

*Example output:*

## Section 1.2: Fit Linear-Gaussian regression model

*Estimated timing to here from start of tutorial: 25 min*

First, we will use the design matrix to compute the maximum likelihood estimate for a linear-Gaussian GLM (aka "general linear model"). The maximum likelihood estimate of $\theta$ in this model can be solved analytically using the equation you learned about on Day 3:

\begin{align}
\boldsymbol{\hat \theta} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}
\end{align}

Before we can apply this equation, we need to augment the design matrix to account for the mean of $y$, because the spike counts are all $\geq 0$. We do this by adding a constant column of 1's to the design matrix, which will allow the model to learn an additive offset weight. We will refer to this additional weight as $b$ (for bias), although it is alternatively known as a "DC term" or "intercept".

```python
# Build the full design matrix
y = spikes
constant = np.ones_like(y)
X = np.column_stack([constant, make_design_matrix(stim)])

# Get the MLE weights for the LG model
theta = np.linalg.inv(X.T @ X) @ X.T @ y
theta_lg = theta[1:]
```

Plot the resulting maximum likelihood filter estimate (just the 25-element weight vector $\theta$ on the stimulus elements, not the DC term $b$).

```python
plot_spike_filter(theta_lg, dt_stim)
```

### Coding Exercise 1.2: Predict spike counts with Linear-Gaussian model

Now we are going to put these pieces together and write a function that outputs a predicted spike count for each timepoint using the stimulus information.

Your steps should be:

- Create the complete design matrix
- Obtain the MLE weights ($\boldsymbol{\hat \theta}$)
- Compute $\mathbf{\hat y} = \mathbf{X}\boldsymbol{\hat \theta}$

```python
def predict_spike_counts_lg(stim, spikes, d=25):
  """Compute a vector of predicted spike counts given the stimulus.

  Args:
    stim (1D array): Stimulus values at each timepoint
    spikes (1D array): Spike counts measured at each timepoint
    d (number): Number of time lags to use.

  Returns:
    yhat (1D array): Predicted spikes at each timepoint.

  """
  ##########################################################################
  # Fill in missing code (...) and then comment or remove the error to test
  # raise NotImplementedError("Complete the predict_spike_counts_lg function")
  ##########################################################################

  # Create the design matrix
  y = spikes
  constant = np.ones_like(y)
  X = np.column_stack([constant, make_design_matrix(stim)])

  # Get the MLE weights for the LG model
  theta =  np.linalg.inv(X.T @ X) @ X.T @ y

  #print( np.linalg.inv(X.T @ X))
  rho =  np.corrcoef(np.hstack(X.T), np.hstack(X))
  print(rho)

  # Compute predicted spike counts
  yhat = X @ theta

  return yhat

# Predict spike counts
predicted_counts = predict_spike_counts_lg(stim, spikes)

# Visualize
plot_spikes_with_prediction(spikes, predicted_counts, dt_stim)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D4_GeneralizedLinearModels/solutions/W1D4_Tutorial1_Solution_823fa455.py)

*Example output:*

Is this a good model? The prediction line more-or-less follows the bumps in the spikes, but it never predicts as many spikes as are actually observed. And, more troublingly, it's predicting *negative* spikes for some time points.

The Poisson GLM will help to address these failures.

### Bonus challenge

The "spike-triggered average" falls out as a subcase of the linear Gaussian GLM: $\mathrm{STA} = \mathbf{X}^T \mathbf{y} \,/\, \textrm{sum}(\mathbf{y})$, where $\mathbf{y}$ is the vector of spike counts of the neuron. In the LG GLM, the term $(\mathbf{X}^T\mathbf{X})^{-1}$ corrects for potential correlation between the regressors. Because the experiment that produced these data used a white noise stimulus, there are no such correlations. Therefore the two methods are equivalent. (How would you check the statement about no correlations?)

---
# Section 2: Linear-Nonlinear-Poisson GLM

*Estimated timing to here from start of tutorial: 36 min*

```python
# @title Video 2: Generalized linear model
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1mz4y1X7JZ", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="wRbvwdze4uE", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Section 2.1: Nonlinear optimization with `scipy.optimize`

*Estimated timing to here from start of tutorial: 45 min*

Before diving into the Poisson GLM case, let us review the use and importance of convexity in optimization:
- We have seen previously that in the Linear-Gaussian case, maximum likelihood  parameter estimate can be computed analytically. That is great because it only takes us a single line of code!
- Unfortunately in general there is no analytical solution to our statistical estimation problems of interest. Instead, we need to apply a nonlinear optimization algorithm to find the parameter values that minimize some *objective function*. This can be extremely tedious because there is no general way to check whether we have found *the optimal solution* or if we are just stuck in some local minimum.
- Somewhere in between these two extremes, the special case of convex objective function is of great practical importance. Indeed, such optimization problems can be solved very reliably (and usually quite rapidly too!) using some standard software.

Notes:
- a function is convex if and only if its curve lies below any chord joining two of its points
- to learn more about optimization, you can consult the book of Stephen Boyd and Lieven Vandenberghe [Convex Optimization](https://web.stanford.edu/~boyd/cvxbook/).

Here we will use the `scipy.optimize` module, it contains a function called [`minimize`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) that provides a generic interface to a large number of optimization algorithms. This function expects as argument an objective function and an "initial guess" for the parameter values. It then returns a dictionary that includes the minimum function value, the parameters that give this minimum, and other information.

Let's see how this works with a simple example. We want to minimize the function $f(x) = x^2$:

```python
f = np.square

res = minimize(f, x0=2)
print(
  f"Minimum value: {res['fun']:.4g}",
  f"at x = {res['x']}",
)
```

    Minimum value: 3.566e-16 at x = [-1.88846401e-08]

When minimizing a $f(x) = x^2$, we get a minimum value of $f(x) \approx 0$ when $x \approx 0$. The algorithm doesn't return exactly $0$, because it stops when it gets "close enough" to a minimum. You can change the `tol` parameter to control how it defines "close enough".

A point about the code bears emphasis. The first argument to `minimize` is not a number or a string but a *function*. Here, we used `np.square`. Take a moment to make sure you understand what's going on, because it's a bit unusual, and it will be important for the exercise you're going to do in a moment.

In this example, we started at $x_0 = 2$. Let's try different values for the starting point:

```python
start_points = -1, 1.5

xx = np.linspace(-2, 2, 100)
plt.plot(xx, f(xx), color=".2")
plt.xlabel("$x$")
plt.ylabel("$f(x)$")

for i, x0 in enumerate(start_points):
  res = minimize(f, x0)
  plt.plot(x0, f(x0), "o", color=f"C{i}", ms=10, label=f"Start {i}")
  plt.plot(res["x"].item(), res["fun"], "x", c=f"C{i}", ms=10, mew=2, label=f"End {i}")
  plt.legend()
```

The runs started at different points (the dots), but they each ended up at roughly the same place (the cross): $f(x_\textrm{final}) \approx 0$. Let's see what happens if we use a different function:

```python
g = lambda x: x / 5 + np.cos(x)
start_points = -.5, 1.5

xx = np.linspace(-4, 4, 100)
plt.plot(xx, g(xx), color=".2")
plt.xlabel("$x$")
plt.ylabel("$f(x)$")

for i, x0 in enumerate(start_points):
  res = minimize(g, x0)
  plt.plot(x0, g(x0), "o", color=f"C{i}", ms=10, label=f"Start {i}")
  plt.plot(res["x"].item(), res["fun"], "x", color=f"C{i}", ms=10, mew=2, label=f"End {i}")
  plt.legend()
```

Unlike $f(x) = x^2$, $g(x) = \frac{x}{5} + \cos(x)$ is not *convex*. We see that the final position of the minimization algorithm depends on the starting point, which adds a layer of comlpexity to such problems.

### Coding Exercise 2.1: Fitting the Poisson GLM and prediction spikes

In this exercise, we will use [`scipy.optimize.minimize`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) to compute maximum likelihood estimates for the filter weights in the Poissson GLM model with an exponential nonlinearity (LNP: Linear-Nonlinear-Poisson).

In practice, this will involve filling out two functions.

- The first should be an *objective function* that takes a design matrix, a spike count vector, and a vector of parameters. It should return a negative log likelihood.
- The second function should take `stim` and `spikes`, build the design matrix and then use `minimize` internally, and return the MLE parameters.

What should the objective function look like? We want it to return the negative log likelihood: $-\log P(y \mid \mathbf{X}, \theta).$

In the Poisson GLM,

\begin{align}
\log P(\mathbf{y} \mid \mathbf{X}, \theta) = \sum_t \log P(y_t \mid \mathbf{x_t},\theta),
\end{align}

where

\begin{align}
P(y_t \mid \mathbf{x_t}, \theta) = \frac{\lambda_t^{y_t}\exp(-\lambda_t)}{y_t!} \text{, with rate } \lambda_t = \exp(\mathbf{x_t}^{\top} \theta).
\end{align}

Now, taking the log likelihood for all the data we obtain:
$\log P(\mathbf{y} \mid X, \theta) = \sum_t( y_t \log\left(\lambda_t) - \lambda_t - \log(y_t !)\right).$

Because we are going to minimize the negative log likelihood with respct to the parameters $\theta$, we can ignore the last term that does not depend on $\theta$. For faster implementation, let us rewrite this in matrix notation:

\begin{align}
\mathbf{y}^T \log(\mathbf{\lambda}) - \mathbf{1}^T \mathbf{\lambda} \text{, with  rate } \mathbf{\lambda} = \exp(\mathbf{X}^{\top} \theta)
\end{align}

Finally, don't forget to add the minus sign for your function to return the negative log likelihood.

```python
def neg_log_lik_lnp(theta, X, y):
  """Return -loglike for the Poisson GLM model.

  Args:
    theta (1D array): Parameter vector.
    X (2D array): Full design matrix.
    y (1D array): Data values.

  Returns:
    number: Negative log likelihood.

  """
  #####################################################################
  # Fill in missing code (...), then remove the error
  #raise NotImplementedError("Complete the neg_log_lik_lnp function")
  #####################################################################

  # Compute the Poisson log likeliood
  rate = np.exp(X @ theta)
  log_lik = y.T @ np.log(rate) - np.sum(rate)

  return -log_lik

def fit_lnp(stim, spikes, d=25):
  """Obtain MLE parameters for the Poisson GLM.

  Args:
    stim (1D array): Stimulus values at each timepoint
    spikes (1D array): Spike counts measured at each timepoint
    d (number): Number of time lags to use.

  Returns:
    1D array: MLE parameters

  """
  #####################################################################
  # Fill in missing code (...), then remove the error
  #raise NotImplementedError("Complete the fit_lnp function")
  #####################################################################

  # Build the design matrix
  y = spikes
  constant = np.ones_like(y)
  X = np.column_stack([constant, make_design_matrix(stim)])

  # Use a random vector of weights to start (mean 0, sd .2)
  x0 = np.random.normal(0, .2, d + 1)

  # Find parameters that minmize the negative log likelihood function
  res = minimize(neg_log_lik_lnp, x0, args=(X, y))

  return res['x']

# Fit LNP model
theta_lnp = fit_lnp(stim, spikes)

# Visualize
plot_spike_filter(theta_lg[1:], dt_stim, color=".5", label="LG")
plot_spike_filter(theta_lnp[1:], dt_stim, label="LNP")
plt.legend(loc="upper left");
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D4_GeneralizedLinearModels/solutions/W1D4_Tutorial1_Solution_f09f1cc5.py)

*Example output:*

Plotting the LG and LNP weights together, we see that they are broadly similar, but the LNP weights are generally larger. What does that mean for the model's ability to *predict* spikes? To see that, let's finish the exercise by filling out the `predict_spike_counts_lnp` function:

```python
def predict_spike_counts_lnp(stim, spikes, theta=None, d=25):
  """Compute a vector of predicted spike counts given the stimulus.

  Args:
    stim (1D array): Stimulus values at each timepoint
    spikes (1D array): Spike counts measured at each timepoint
    theta (1D array): Filter weights; estimated if not provided.
    d (number): Number of time lags to use.

  Returns:
    yhat (1D array): Predicted spikes at each timepoint.

  """
  ###########################################################################
  # Fill in missing code (...) and then remove the error to test
  #raise NotImplementedError("Complete the predict_spike_counts_lnp function")
  ###########################################################################

  y = spikes
  constant = np.ones_like(spikes)
  X = np.column_stack([constant, make_design_matrix(stim)])
  if theta is None:  # Allow pre-cached weights, as fitting is slow
    theta = fit_lnp(X, y, d)

  yhat = np.exp(X @ theta)

  return yhat

# Predict spike counts
yhat = predict_spike_counts_lnp(stim, spikes, theta_lnp)

# Visualize
plot_spikes_with_prediction(spikes, yhat, dt_stim)

#Bonus
print(np.var(yhat))
yhat_lg = predict_spike_counts_lg(stim, spikes, d=25)
print(np.var(yhat_lg))

```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D4_GeneralizedLinearModels/solutions/W1D4_Tutorial1_Solution_ae48f475.py)

*Example output:*

We see that the LNP model does a better job of fitting the actual spiking data. Importantly, it never predicts negative spikes!

*Bonus:* Our statement that the LNP model "does a better job" is qualitative and based mostly on the visual appearance of the plot. But how would you make this a quantitative statement?

---
# Summary

*Estimated timing of tutorial: 1 hour, 15 minutes*

In this first tutorial, we used two different models to learn something about how retinal ganglion cells respond to a flickering white noise stimulus. We learned how to construct a design matrix that we could pass to different GLMs, and we found that the Linear-Nonlinear-Poisson (LNP) model allowed us to predict spike rates better than a simple Linear-Gaussian (LG) model.

In the next tutorial, we'll extend these ideas further. We'll meet yet another GLM — logistic regression — and we'll learn how to ensure good model performance even when the number of parameters `d` is large compared to the number of data points `N`.

---
# Notation

\begin{align}
y &\quad \text{measurement or response, here: spike count}\\
T &\quad \text{number of time points}\\
d &\quad \text{input dimensionality}\\
\mathbf{X} &\quad \text{design matrix, dimensions: } T \times d\\
\theta &\quad \text{parameter}\\
\hat \theta &\quad \text{estimated parameter}\\
\hat y &\quad \text{estimated response}\\
P(\mathbf{y} \mid \mathbf{X}, \theta) &\quad \text{probability of observing response } y \text{ given design matrix } \mathbf{X} \text{ and parameters } \theta \\
\mathrm{STA} &\quad \text{spike-triggered average}\\
b &\quad \text{bias weight, intercept}\\
\end{align}


# Numerical optimization

You will learn to solve non-convex multi-dimensional optimization problems using numerical optimization with multistart and nesting (**scipy.optimize**). You will learn simple function approximation using linear interpolation (**scipy.interp**).

**Links:**

1. **scipy.optimize:** [overview](https://docs.scipy.org/doc/scipy/reference/optimize.html) + [tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html)
2. **scipy.interp:** [overview](https://docs.scipy.org/doc/scipy/reference/interpolate.html) + [tutorial](https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html)

**Useful note:** [Numerical Optimization in MATLAB](http://web.econ.ku.dk/munk-nielsen/notes/noteOptimization.pdf) (by Anders Munk-Nielsen)

```python
import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# Introduction

All **optimization problems** are characterized by:

1. Control vector (choices), $\boldsymbol{x} \in \mathbb{R}^k$
2. Objective function (payoff) to minimize, $f:\mathbb{R}^k \rightarrow \mathbb{R}$ (differentiable or not)
3. Constraints, i.e. $\boldsymbol{x}  \in C \subseteq \mathbb{R}^k$ (linear or non-linear interdependence)

**Maximization** is just **minimization** of $-f$.

> Note that $f$ might also take other inputs (parameters or a dataset), but these are fixed, and therefore not variables we optimize over.

All **optimizers** (minimizers) follow the structure:

1. Make initial guess
2. Evaluate the function (and perhaps gradients)
3. Check for convergence
4. Update guess and return to step 2

**Convergence:** "Small" change in function value since last iteration (or zero gradient).

**Characteristics** of optimizers:

1. Use gradients or not.
2. Allow for specifying bounds.
3. Allow for specifying general constraints.

**Gradients** provide useful information, but can be costly to compute (using analytical formula or numerically).

**Penalty terms** can (sometimes) instead be used to enforce bounds and constraints.

**Optimizers** you should know:

1. **Nelder-Mead:**
 * **Pro:** Robust (to e.g. noise in objective function) and does not require derivatives.
 * **Con:** Slow convergence. No bounds or constraints.
2. **Newton-CG:**
 * **Pro:** Require few iterations. Very precise with analytical hessian for smooth functions.
 * **Con:** Costly computation of hessian. No bounds or constraints.
3. **BFGS:** (like newton, but with smart computation of hessian)
  * **Pro:** Require few function evaluations.
  * **Con:** No bounds or constraints.
4. **L-BFGS-B:** Like BFGS, but allows for bounds.
5. **SLSQP:**
  * **Pro:** Bounds and constraints in multiple dimensions.
  * **Con:** Not as efficient as BFGS.

## Gradient based optimizers

Let us look at the idea behind gradient based optimizers.

**One dimensional intuition:** Consider the second-order Taylor approximation around $x_n$:

$$
f_T(x) = f_T(x_n + \Delta x) \approx f(x_n)+ f^{\prime}(x_n) \Delta x + \frac{1}{2} f^{\prime\prime}(x_n) (\Delta x)^2
$$

Find the minimum wrt. to $\Delta x$ by solving the FOC:

$$
0 = \frac{d}{d\Delta x} f_T(x) = f^{\prime}(x_n) + f^{\prime\prime}(x_n) \Delta x \Leftrightarrow \Delta x = -\frac{f^{\prime}(x_n)}{f^{\prime\prime}(x_n)}
$$

**Algorithm:** `minimize_newton()`

1. Choose tolerance $\epsilon>0$, guess on $\boldsymbol{x}_0$, compute $f(\boldsymbol{x}_0)$, and set $n=1$.
2. Compute $\nabla f(\boldsymbol{x}_{n-1})$ (gradient/jacobian) and $\boldsymbol{H}f(\boldsymbol{x}_{n-1})$ (hessian).
3. Compute new guess

  $$
  \boldsymbol{x}_{n} = \boldsymbol{x}_{n-1} - [\boldsymbol{H}f(\boldsymbol{x}_{n-1})]^{-1} \nabla f(\boldsymbol{x}_{n-1})
  $$

3. If $|f(\boldsymbol{x}_n)-f(\boldsymbol{x}_{n-1})| < \epsilon$ then stop.
5. Set $n = n + 1$ and return to step 2.

```python
def minimize_newton(f,x0,jac,hess,max_iter=500,tol=1e-8):
    """ minimize function with Newtons' algorithm

    Args:

        f (callable): function
        x0 (np.ndarray): initial values
        jac (callable): jacobian
        hess (callable): hessian
        max_iter (int): maximum number of iterations
        tol (float): tolerance

    Returns:

        x (np.ndarray): minimum
        n (int): number of iterations used

    """

    # step 1: initialize
    x = x0
    fx = f(x0)
    n = 1

    # step 2-5: iteration
    while n < max_iter:

        x_prev = x
        fx_prev = fx

        # step 2: evaluate gradient and hessian
        jacx = jac(x_prev)
        hessx = hess(x_prev)

        # step 3: update x
        inv_hessx = linalg.inv(hessx)
        x = x_prev - inv_hessx@jacx

        # step 4: check convergence
        fx = f(x)
        if abs(fx-fx_prev) < tol:
            break

        # step 5: increment n
        n += 1

    return x,n
```

**Algorithm:** `minimize_gradient_descent()`

1. Choose tolerance $\epsilon>0$, potential step sizes, $ \boldsymbol{\alpha} = [\alpha_0,\alpha_1,\dots,\alpha_\#]$, guess on $\boldsymbol{x}_0$, compute $f(\boldsymbol{x}_0)$, and set $n=1$.
2. Compute $\nabla f(\boldsymbol{x}_{n-1})$.
3. Find good step size:

  $$
  \alpha^{\ast} = \arg \min_{\alpha \in \boldsymbol{\alpha}}  f(\boldsymbol{x}_{n-1} - \alpha \nabla f(\boldsymbol{x}_{n-1}))
  $$

4. Compute new guess:

  $$
  \boldsymbol{x}_{n} = \boldsymbol{x}_{n-1} - \alpha^{\ast} \nabla f(\boldsymbol{x}_{n-1})
  $$

5. If $|f(\boldsymbol{x}_n)-f(\boldsymbol{x}_{n-1})| < \epsilon$ then stop.
6. Set $n = n + 1$ and return to step 2.

```python
def minimize_gradient_descent(f,x0,jac,alphas=[0.01,0.05,0.1,0.25,0.5,1],max_iter=500,tol=1e-8):
    """ minimize function with gradient descent

    Args:

        f (callable): function
        x0 (np.ndarray): initial values
        jac (callable): jacobian
        alpha (list): potential step sizes
        max_iter (int): maximum number of iterations
        tol (float): tolerance

    Returns:

        x (np.ndarray): minimum
        n (int): number of iterations used

    """

    # step 1: initialize
    x = x0
    fx = f(x0)
    n = 1

    # step 2-6: iteration
    while n < max_iter:

        x_prev = x
        fx_prev = fx

        # step 2: evaluate gradient
        jacx = jac(x)

        # step 3: find good step size (line search)
        fx_ast = np.inf
        alpha_ast = np.nan
        for alpha in alphas:
            x = x_prev - alpha*jacx
            fx = f(x)
            if fx < fx_ast:
                fx_ast = fx
                alpha_ast = alpha

        # step 4: update guess
        x = x_prev - alpha_ast*jacx

        # step 5: check convergence
        fx = f(x)
        if abs(fx-fx_prev) < tol:
            break

        # d. update i
        n += 1

    return x,n
```

**Many generalizations:**

1. Use both Hessian and line search
2. Stop line search when improvement found
3. Limit attention to a "trust-region"

etc. etc. etc. etc.

# Example: The rosenbrock function

Consider the **rosenbrock function**:

$$
f(\boldsymbol{x}) = f(x_1,x_2) =0.5(1-x_{1})^{2}+(x_{2}-x_{1}^{2})^{2}
$$

with **jacobian** (gradient)

$$
\nabla f(\boldsymbol{x})=\begin{bmatrix}\frac{\partial f}{\partial x_{1}}\\
\frac{\partial f}{\partial x_{2}}
\end{bmatrix}=\begin{bmatrix}-(1-x_{1})-4x_{1}(x_{2}-x_{1}^{2})\\
2(x_{2}-x_{1}^{2})
\end{bmatrix}
$$

and **hessian**:

$$
\boldsymbol{H}f(\boldsymbol{x})=\begin{bmatrix}\frac{\partial f}{\partial x_{1}x_{1}} & \frac{\partial f}{\partial x_{1}x_{2}}\\
\frac{\partial f}{\partial x_{1}x_{2}} & \frac{\partial f}{\partial x_{2}x_{2}}
\end{bmatrix}=\begin{bmatrix}1-4x_{2}+12x_{1}^{2} & -4x_{1}\\
-4x_{1} & 2
\end{bmatrix}
$$

**Note:** Minimum is at $(1,1)$ where $f(1,1)=0$.

**Check jacobian and hessian:**

```python
x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
f = 0.5*(1.0-x1)**2 + (x2-x1**2)**2
```

```python
Df = sm.Matrix([sm.diff(f,i) for i in [x1,x2]])
Df
```

$\displaystyle \left[\begin{matrix}- 4 x_{1} \left(- x_{1}^{2} + x_{2}\right) + 1.0 x_{1} - 1.0\\- 2 x_{1}^{2} + 2 x_{2}\end{matrix}\right]$

```python
Hf = sm.Matrix([[sm.diff(f,i,j) for j in [x1,x2]] for i in [x1,x2]])
Hf
```

$\displaystyle \left[\begin{matrix}12 x_{1}^{2} - 4 x_{2} + 1.0 & - 4 x_{1}\\- 4 x_{1} & 2\end{matrix}\right]$

**Implementation:**

```python
def _rosen(x1,x2):
    return 0.5*(1.0-x1)**2+(x2-x1**2)**2
def rosen(x):
    return _rosen(x[0],x[1])
def rosen_jac(x):
    return np.array([-(1.0-x[0])-4*x[0]*(x[1]-x[0]**2),2*(x[1]-x[0]**2)])
def rosen_hess(x):
    return np.array([[1-4*x[1]+12*x[0]**2,-4*x[0]],[-4*x[0],2]])
```

**3D Plot:**

```python
# a. grids
x1_vec = np.linspace(-2,2,500)
x2_vec = np.linspace(-2,2,500)
x1_grid,x2_grid = np.meshgrid(x1_vec,x2_vec,indexing='ij')
rosen_grid = _rosen(x1_grid,x2_grid)

# b. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid,x2_grid,rosen_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')

# d. invert xaxis
ax.invert_xaxis()

# e. add colorbar
fig.colorbar(cs);
```

**Contour plot:**

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
levels = [1e-6,5*1e-6,1e-5,5*1e-5,1e-4,5*1e-4,1e-3,5*1e-3,1e-2,5*1e-2,1,2,4,6,8,12,16,20]
cs = ax.contour(x1_grid,x2_grid,rosen_grid,levels=levels,cmap=cm.jet)
fig.colorbar(cs);
```

**Newton:**

```python
x0 = np.array([5,4])
x,n = minimize_newton(rosen,x0,rosen_jac,rosen_hess)
print(n,x,rosen(x))
```

    6 [1. 1.] 0.0

**Gradient descent:**

```python
x0 = np.array([5,4])
x,n = minimize_gradient_descent(rosen,x0,rosen_jac,alphas=[0.01,0.05,0.1,0.25,0.5,1])
print(n,x,rosen(x))
```

    173 [1.00020519 1.00053964] 3.7750814497569406e-08

## Scipy minimizers

**Preperation I:** Function for collecting infomation while running optimizing:

```python
# complicated -> not necessary to understand it
def collect(x):

    # globals used to keep track across iterations
    global evals # set evals = 0 before calling optimizer
    global x0
    global x1s
    global x2s
    global fs

    # a. initialize list
    if evals == 0:
        x1s = [x0[0]]
        x2s = [x0[1]]
        fs = [rosen(x0)]

    # b. append trial values
    x1s.append(x[0])
    x2s.append(x[1])
    fs.append(rosen(x))

    # c. increment number of evaluations
    evals += 1
```

**Preperation II:** Function plotting the collected information:

```python
# complicated -> not necessary to understand it
def contour():

    global evals
    global x1s
    global x2s
    global fs

    # a. contour plot
    fig = plt.figure(figsize=(10,4))
    ax = fig.add_subplot(1,2,1)
    levels = [1e-6,5*1e-6,1e-5,5*1e-5,1e-4,5*1e-4,1e-3,5*1e-3,1e-2,5*1e-2,1,2,4,6,8,12,16,20]
    cs = ax.contour(x1_grid,x2_grid,rosen_grid,levels=levels,cmap=cm.jet)
    fig.colorbar(cs)
    ax.plot(x1s,x2s,'-o',ms=4,color='black')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')

    # b. function value
    ax = fig.add_subplot(1,2,2)
    ax.plot(np.arange(evals+1),fs,'-o',ms=4,color='black')
    ax.set_xlabel('iteration')
    ax.set_ylabel('function value')
```

**Nelder-Mead**

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='Nelder-Mead',
                           callback=collect, # call collect() before each iteration
                           options={'disp':True}) # display the results
contour()
```

> **Note:** Does not require a gradient. Slow convergence close to target.

**Newton** (with analytical hessian)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,hess=rosen_hess,
                           method='Newton-CG',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Smoother and faster.

**Newton** (with numerical hessian computed by scipy)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='Newton-CG',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Same as above, but gradient evaluations instead of hessian evaluations.

**BFGS** (with analytical gradient)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Non-smooth, but fast. Very low number of function evaluations.

**BFGS** (with numerical gradient computed by scipy)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True})
contour()
```

> **Note:** Same as above, but more function evaluations.

**L-BFGS-B** (with analytical gradient)

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='L-BFGS-B',
                           bounds=((-3,3),(-3,3)),
                           callback=collect,
                           options={'disp':True})
contour()
```

**SLSQP**

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,jac=rosen_jac,
                           method='SLSQP',
                           bounds=((-2,2),(-2,2)),
                           callback=collect,
                           options={'disp':True})
contour()
```

## Controling the optimizers

> **Note:** See the settings for each optimizer in the [documention](https://docs.scipy.org/doc/scipy/reference/optimize.html).

We can lower the **tolerance**:

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True,'gtol':1e-8}) # note this
contour()
```

We can change the **maximum number of iterations**:

```python
evals = 0
x0 = [-1.5,-1]
result = optimize.minimize(rosen,x0,
                           method='BFGS',
                           callback=collect,
                           options={'disp':True,'maxiter':50}) # note this and warning
contour()
```

## Sombrero function: Local minima and multistart

Consider the **sombrero** function

$$
f(x_1,x_2) = g\Big(\sqrt{x_1^2 + x_2^2}\Big)
$$

where

$$
g(r) = -\frac{\sin(r)}{r+10^{-4}} + 10^{-4}r^2
$$

The **global minimum** of this function is (0,0). But the function also have (infinitely many) **local minima**. How to avoid these?

```python
def _sombrero(x1,x2):
    r = np.sqrt(x1**2 + x2**2)
    return -np.sin(r)/(r+1e-4) + 1e-4*r**2

sombrero = lambda x: _sombrero(x[0],x[1])
```

## 3D plot

```python
# a. grids
x1_vec = np.linspace(-15,15,500)
x2_vec = np.linspace(-15,15,500)
x1_grid_sombrero,x2_grid_sombrero = np.meshgrid(x1_vec,x2_vec,indexing='ij')
sombrero_grid = _sombrero(x1_grid_sombrero,x2_grid_sombrero)

# b. main
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(x1_grid_sombrero,x2_grid_sombrero,sombrero_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')

# d. invert xaxis
ax.invert_xaxis()

# e. colorbar
fig.colorbar(cs);
```

## Multistart - BFGS

**Multi-start:** Draw many random starting values:

```python
np.random.seed(1986)
x0s = -15 + 30*np.random.uniform(size=(5000,2)) # in [-15,15]
xs = np.empty((5000,2))
fs = np.empty(5000)
```

Try to solve with **BFGS** starting from each of these:

```python
fopt = np.inf
xopt = np.nan
for i,x0 in enumerate(x0s):

    # a. optimize
    result = optimize.minimize(sombrero,x0,method='BFGS')
    xs[i,:] = result.x
    f = result.fun

    # b. print first 10 or if better than seen yet
    if i < 10 or f < fopt: # plot 10 first or if improving
        if f < fopt:
            fopt = f
            xopt = xs[i,:]

        print(f'{i:4d}: x0 = ({x0[0]:6.2f},{x0[1]:6.2f})',end='')
        print(f' -> converged at ({xs[i][0]:6.2f},{xs[i][1]:6.2f}) with f = {f:.14f}')

# best solution
print(f'\nbest solution:\n x = ({xopt[0]:6.2f},{xopt[1]:6.2f}) -> f = {fopt:.14f}')
```

       0: x0 = (  2.07,-12.68) -> converged at (  2.26,-13.84) with f = -0.05118272212318
       1: x0 = (  3.25, 11.92) -> converged at (  3.69, 13.53) with f = -0.05118272211067
       2: x0 = (  1.35,  6.08) -> converged at (  1.67,  7.53) with f = -0.12241421149687
       3: x0 = ( -3.42,  4.55) -> converged at ( -4.63,  6.17) with f = -0.12241421149362
       4: x0 = (  5.70,  6.56) -> converged at (  5.06,  5.82) with f = -0.12241421149696
       5: x0 = (  5.71,  9.20) -> converged at (  4.06,  6.56) with f = -0.12241421146807
       6: x0 = ( -6.49,  7.73) -> converged at ( -4.96,  5.91) with f = -0.12241421149702
       7: x0 = ( -5.77, -1.86) -> converged at ( -7.34, -2.37) with f = -0.12241421135584
       8: x0 = (  1.58, -2.01) -> converged at (  0.04, -0.05) with f = -0.99776236017000
       9: x0 = (  2.33, 11.77) -> converged at (  2.72, 13.76) with f = -0.05118272212317
      25: x0 = (  1.41, -3.37) -> converged at (  0.03, -0.06) with f = -0.99776236017097
     150: x0 = (-14.53,  8.53) -> converged at ( -0.06,  0.03) with f = -0.99776236017097
     223: x0 = (  2.87, -0.21) -> converged at ( -0.07,  0.00) with f = -0.99776236017098
     279: x0 = ( -6.90, -8.37) -> converged at ( -0.04, -0.05) with f = -0.99776236017098
     354: x0 = ( -1.24, -2.85) -> converged at ( -0.03, -0.06) with f = -0.99776236017098
    1588: x0 = (-12.93,-11.42) -> converged at ( -0.05, -0.04) with f = -0.99776236017098
    4700: x0 = (  1.06,  0.65) -> converged at (  0.06,  0.04) with f = -0.99776236017098

    best solution:
     x = (  0.06,  0.04) -> f = -0.99776236017098

The solver, wrongly, **converges to many of the local minima**:

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(xs[:,0],xs[:,1]);
```

## Multistart - Nelder-Mead

Try to solve with **Nelder-Mead** starting from each of these:

```python
fopt = np.inf
xopt = np.nan
for i,x0 in enumerate(x0s):

    # a. optimize
    result = optimize.minimize(sombrero,x0,method='Nelder-Mead')
    xs[i,:] = result.x
    f = result.fun

    # b. print first 10 or if better than seen yet
    if i < 10 or f < fopt: # plot 10 first or if improving
        if f < fopt:
            fopt = f
            xopt = xs[i,:]
        print(f'{i:4d}: x0 = ({x0[0]:6.2f},{x0[1]:6.2f})',end='')
        print(f' -> converged at ({xs[i][0]:6.2f},{xs[i][1]:6.2f}) with f = {f:.12f}')

# best solution
print(f'\nbest solution:\n x = ({xopt[0]:6.2f},{xopt[1]:6.2f}) -> f = {fopt:.12f}')
```

       0: x0 = (  2.07,-12.68) -> converged at (  2.20,-13.85) with f = -0.051182722123
       1: x0 = (  3.25, 11.92) -> converged at (  3.43, 13.60) with f = -0.051182722123
       2: x0 = (  1.35,  6.08) -> converged at (  1.39,  7.59) with f = -0.122414211497
       3: x0 = ( -3.42,  4.55) -> converged at ( -3.71,  6.76) with f = -0.122414211497
       4: x0 = (  5.70,  6.56) -> converged at (  6.69,  3.84) with f = -0.122414211497
       5: x0 = (  5.71,  9.20) -> converged at (  4.98, 13.11) with f = -0.051182722123
       6: x0 = ( -6.49,  7.73) -> converged at ( -7.34,  2.37) with f = -0.122414211497
       7: x0 = ( -5.77, -1.86) -> converged at ( -7.45, -2.01) with f = -0.122414211497
       8: x0 = (  1.58, -2.01) -> converged at ( -0.02,  0.06) with f = -0.997762360171
       9: x0 = (  2.33, 11.77) -> converged at (  2.36, 13.83) with f = -0.051182722123

    best solution:
     x = ( -0.02,  0.06) -> f = -0.997762360171

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(xs[:,0],xs[:,1]);
```

# Constraints

Consider the **constrained problem**:

$$
\min_{x_1,x_2,x_3,x_4} x_1x_4(x_1+x_2+x_3) + x_3
$$

subject to

$$
\begin{aligned}
x_1x_2x_3x_4 &\geq 25 \\
x_1^2+x_2^2+x_3^2+x_4^2 &= 40 \\
1 \leq x_1,x_2,x_3,x_4 &\leq 5
\end{aligned}
$$

Define **objective** and **constraints**:

```python
def _objective(x1,x2,x3,x4):
    return x1*x4*(x1+x2+x3)+x3

def objective(x):
    return _objective(x[0],x[1],x[2],x[3])

def ineq_constraint(x):
    return x[0]*x[1]*x[2]*x[3]-25.0 # violated if negative

def eq_constraint(x):
    sum_eq = 40.0
    for i in range(4):
        sum_eq = sum_eq - x[i]**2
    return sum_eq
```

```python
# a. setup
bound = (1.0,5.0)
bounds = (bound, bound, bound, bound)
ineq_con = {'type': 'ineq', 'fun': ineq_constraint}
eq_con = {'type': 'eq', 'fun': eq_constraint}

# b. call optimizer
x0 = (40**(1/8),40**(1/8),40**(1/8),40**(1/8)) # fit the equality constraint
result = optimize.minimize(objective,x0,
                             method='SLSQP',
                             bounds=bounds,
                             constraints=[ineq_con,eq_con],
                             options={'disp':True})

print('\nx = ',result.x)
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 17.01401728904412
                Iterations: 9
                Function evaluations: 55
                Gradient evaluations: 9

    x =  [1.         4.74299967 3.82114994 1.3794083 ]

**Alternative:** Extend the **objective function with a penalty term**, where guesses outside the allowed bounds and constraints are projected into the allowed region, but a (large) penalty term is added to discourage this. Solve this problem with an unconstrained solver. Not always easy to do in practice if there are multiple constraint (see lecture 3 for a simple example).

# Interpolation

**Intermezzo:** To consider dynamic optimization problems, we need to think about interpolation.

**Inputs:**

1. Sorted vector of known points (grid vector), $G = \{G_i\}_{i=0}^{n-1}$
2. Vector of known values (at these points), $F = \{F_i = f(G_i)\}_{i=0}^{n-1}$
3. A new point, `x`

**Algorithm:** `linear_interpolate()`
1. Determine `i`  such that

$$
G_i \leq x < G_{i+1}
$$

2. Compute interpolated value by

$$
y =  F_{i} + \frac{F_{i+1}-F_{i}}{G_{i+1}-G_{i}}(x-G_{i})
$$

**Extrapolation:**

1. Below where $x < G_1$:

$$
y =  F_{0} + \frac{F_{1}-F_{0}}{G_{1}-G_{0}}(x-G_{0})
$$

2. Above where $x > G_{n-2}$:

$$
y =  F_{n-2} + \frac{F_{n-1}-F_{n-2}}{G_{n-1}-G_{n-2}}(x-G_{n-2})
$$

```python
def linear_interpolate(G,F,x):
    """ linear interpolation (and extrapolation)

    Args:

        G (np.ndarray): known points
        F (np.ndarray): known values
        x (float): point to be interpolated

    Returns:

        y (float): intepolated value

    """

    assert len(G) == len(F)
    n = len(G)

    # a. find index in known points
    if x < G[1]: # exprapolation below
        i = 0
    elif x > G[-2]: # extrapolation above
        i = n-2
    else: # true interpolation

        # search
        i = 0
        while x >= G[i+1] and i < n-1:
            i += 1

        assert x >= G[i]
        assert x < G[i+1]

    # b. interpolate
    diff_G = G[i+1]-G[i]
    diff_F = F[i+1]-F[i]
    slope = diff_F/diff_G
    y = F[i] + slope*(x-G[i])

    return y
```

## Example

Consider the following function and known points:

```python
f = lambda x: (x-3)**3 - 3*x**2 + 5*x

G = np.linspace(-5,10,6)
F = f(G)
```

**Simple test:**

```python
for x in [-2.3,4.1,7.5,9.1]:
    true = f(x)
    y = linear_interpolate(G,F,x)
    print(f'x = {x:4.1f} -> true = {true:6.1f}, interpolated = {y:6.1f}')
```

    x = -2.3 -> true = -176.2, interpolated = -193.5
    x =  4.1 -> true =  -28.6, interpolated =  -27.7
    x =  7.5 -> true =  -40.1, interpolated =  -24.5
    x =  9.1 -> true =   24.1, interpolated =   50.7

**Scipy.interpolate:** Use the *RegularGridInterpolator*

```python
# a. construct interpolation function
interp_func = interpolate.RegularGridInterpolator([G],F,
                                                  bounds_error=False,
                                                  fill_value=None)

# bounds_error=False and fill_value=None allow for extrapolation

# b. interpolate
grid = np.linspace(-7,12,500)
interp_values = interp_func(grid)

# c. evaluate true values
true_values = f(grid)

# d. plot true and interpolated values
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(G,F,'o',label='known points')
ax.plot(grid,true_values,'-',lw=1,label='true function')
ax.plot(grid,interp_values,'-',lw=1,label='interpolated values')
ax.legend(loc='lower right',facecolor='white',frameon=True);
```

**Note:**

1. Linear interpolation works best when the function does not curve too much.
2. Extrapolation is much worse than interpolation.

**Multiple dimensions:** Same principle, ``interpolate.RegularGridInterpolator([G1,G2,G3],F)``.

# Dynamic optimization problems

The following subject is hard. But also extremely useful. *If you master this, you can solve (almost) all economic models you meet on your way in life*.

## Problem formulation

Consider a **household** living in two periods.

In the **second period** it gets utility from **consuming** and **leaving a bequest** (warm glow),

$$
\begin{aligned}
v_{2}(m_{2})&= \max_{c_{2}}\frac{c_{2}^{1-\rho}}{1-\rho}+\nu\frac{(m_{2}-c_{2}+\kappa)^{1-\rho}}{1-\rho}\\
\text{s.t.} \\
c_{2} &\in [0,m_{2}]
\end{aligned}
$$

where

* $m_2$ is cash-on-hand
* $c_2$ is consumption
* $\rho > 1$ is the risk aversion coefficient
* $\nu > 0 $ is the strength of the bequest motive
* $\kappa > 0$ is the degree of luxuriousness in the bequest motive
* the constraint ensures the household *cannot* die in debt

The **value function** $v(m_2)$ measures the household's value of having $m_2$ at the beginning of period 2.

```python
def utility(c,rho):
    return c**(1-rho)/(1-rho)

def bequest(m,c,nu,kappa,rho):
    return nu*(m-c+kappa)**(1-rho)/(1-rho)

def v2(c2,m2,rho,nu,kappa):
    return utility(c2,rho) + bequest(m2,c2,nu,kappa,rho)
```

In the **first period**, the household gets utility from consuming and takes into account that it will also live in the next-period, where it receives a stochastic income,

$$
\begin{aligned}
v_{1}(m_{1})&=\max_{c_{1}}\frac{c_{1}^{1-\rho}}{1-\rho}+\beta\mathbb{E}_{1}\left[v_2(m_2)\right]\\&\text{s.t.}&\\
m_2&= (1+r)(m_{1}-c_{1})+y_{2} \\
y_{2}&= \begin{cases}
1-\Delta & \text{with prob. }0.5\\
1+\Delta & \text{with prob. }0.5
\end{cases}\\
c_{1}&\in [0,m_{1}]\\
\end{aligned}
$$

where

* $m_1$ is cash-on-hand in period 1
* $c_1$ is consumption in period 1
* $\beta > 0$ is the discount factor
* $\mathbb{E}_1$ is the expectation operator conditional on information in period 1
* $y_2$ is income in period 2
* $\Delta \in (0,1)$ is the level of income risk (mean-preserving)
* $r$ is the return on savings
* the last constraint ensures the household *cannot* borrow

```python
def v1(c1,m1,rho,beta,r,Delta,v2_interp):

    # a. v2 value, if low income
    m2_low = (1+r)*(m1-c1) + 1-Delta
    v2_low = v2_interp([m2_low])[0]

    # b. v2 value, if high income
    m2_high = (1+r)*(m1-c1) + 1+Delta
    v2_high = v2_interp([m2_high])[0]

    # c. expected v2 value
    v2 = 0.5*v2_low + 0.5*v2_high

    # d. total value
    return utility(c1,rho) + beta*v2
```

## Solve household problem

Choose **parameters**:

```python
rho = 8
kappa = 0.5
nu = 0.1
r = 0.04
beta = 0.94
Delta = 0.5
```

**Solve second period:**

```python
def solve_period_2(rho,nu,kappa,Delta):

    # a. grids
    m2_vec = np.linspace(1e-8,5,500)
    v2_vec = np.empty(500)
    c2_vec = np.empty(500)

    # b. solve for each m2 in grid
    for i,m2 in enumerate(m2_vec):

        # i. objective
        obj = lambda x: -v2(x[0],m2,rho,nu,kappa)

        # ii. initial value (consume half)
        x0 = m2/2

        # iii. optimizer
        result = optimize.minimize(obj,[x0],method='L-BFGS-B',bounds=((1e-8,m2),))

        # iv. save
        v2_vec[i] = -result.fun
        c2_vec[i] = result.x

    return m2_vec,v2_vec,c2_vec

# solve
m2_vec,v2_vec,c2_vec = solve_period_2(rho,nu,kappa,Delta)

# illustration
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(m2_vec,c2_vec)
ax.set_xlabel('$m_2$')
ax.set_ylabel('$c_2$')
ax.set_title('consumption function in period 2')

ax = fig.add_subplot(1,2,2)
ax.plot(m2_vec,v2_vec)
ax.set_xlabel('$m_2$')
ax.set_ylabel('$v_2$')
ax.set_title('value function in period 2')
ax.set_ylim([-40,1]);
```

```python
def solve_period_2(rho,nu,kappa,Delta):

    # a. grids
    m2_vec = np.linspace(1e-8,5,500)
    v2_vec = np.empty(500)
    c2_vec = np.empty(500)

    # b. solve for each m2 in grid
    for i,m2 in enumerate(m2_vec):

        # i. objective
        obj = lambda x: -v2(x[0],m2,rho,nu,kappa)

        # ii. initial value (consume half)
        x0 = m2/2

        # iii. optimizer
        result = optimize.minimize(obj,[x0],method='L-BFGS-B',bounds=((1e-12,m2),))

        # iv. save
        v2_vec[i] = -result.fun
        c2_vec[i] = result.x[0]

    return m2_vec,v2_vec,c2_vec

# solve
m2_vec,v2_vec,c2_vec = solve_period_2(rho,nu,kappa,Delta)

# illustration
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(m2_vec,c2_vec)
ax.set_xlabel('$m_2$')
ax.set_ylabel('$c_2$')
ax.set_title('consumption function in period 2')

ax = fig.add_subplot(1,2,2)
ax.plot(m2_vec,v2_vec)
ax.set_xlabel('$m_2$')
ax.set_ylabel('$v_2$')
ax.set_title('value function in period 2')
ax.set_ylim([-40,1]);
```

**Question:** Why is there a kink in the consumption function?

**Construct interpolator:**

```python
v2_interp = interpolate.RegularGridInterpolator([m2_vec], v2_vec,
                                                bounds_error=False,fill_value=None)
```

**Solve first period:**

```python
def solve_period_1(rho,beta,r,Delta,v2_interp):

    # a. grids
    m1_vec = np.linspace(1e-8,4,100)
    v1_vec = np.empty(100)
    c1_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m1 in enumerate(m1_vec):

        # i. objective
        obj = lambda x: -v1(x[0],m1,rho,beta,r,Delta,v2_interp)

        # ii. initial guess (consume half)
        x0 = m1/2

        # iii. optimize
        result = optimize.minimize(obj,[x0],method='L-BFGS-B',bounds=((1e-12,m1),))

        # iv. save
        v1_vec[i] = -result.fun
        c1_vec[i] = result.x[0]

    return m1_vec,v1_vec,c1_vec

# solve
m1_vec,v1_vec,c1_vec = solve_period_1(rho,beta,r,Delta,v2_interp)

# illustrate
fig = plt.figure(figsize=(10,4))
ax = fig.add_subplot(1,2,1)
ax.plot(m1_vec,c1_vec)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('consumption function in period 1')

ax = fig.add_subplot(1,2,2)
ax.plot(m1_vec,v1_vec)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('value function in period 1')
ax.set_ylim([-40,1]);
```

**Summary:** We can summarize what we have done in a single function doing:

1. Solve period 2 (i.e. find $v_2(m_2)$ og $c_2(m_2)$)
2. Construct interpolator of $v_2(m_2)$
3. Solve period 1 (i.e. find $v_1(m_1)$ og $c_1(m_1)$)

```python
def solve(rho,beta,r,Delta,nu,kappa):

    # a. solve period 2
    m2_vec,v2_vec,c2_vec = solve_period_2(rho,nu,kappa,Delta)

    # b. construct interpolator
    v2_interp = interpolate.RegularGridInterpolator([m2_vec], v2_vec,
        bounds_error=False,fill_value=None)

    # b. solve period 1
    m1_vec,v1_vec,c1_vec = solve_period_1(rho,beta,r,Delta,v2_interp)

    return m1_vec,c1_vec,m2_vec,c2_vec
```

**Plot consumption function for various level of income risk**, i.e varios $\Delta$

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

for Delta in [0.05,0.15,0.25]:
    m1_vec,c1_vec,m2_vec,c2_vec = solve(rho,beta,r,Delta,nu,kappa)
    ax.plot(m1_vec,c1_vec,label=f'$\Delta = {Delta}$')

ax.legend(loc='lower right',facecolor='white',frameon=True)
ax.set_xlabel('$m_1$')
ax.set_ylabel('$c_1$')
ax.set_title('value function in period 1')
ax.set_xlim([0,2])
ax.set_ylim([0,1.5]);
```

**Main takeaway:** The household lower its consumption when risk increases (such as in a recession). This is called **precautionary saving**.

## Simulation

**Step 1:** Solve and construct interpolators:

```python
Delta = 0.5
m1_vec,c1_vec,m2_vec,c2_vec = solve(rho,beta,r,Delta,nu,kappa)

c1_interp = interpolate.RegularGridInterpolator([m1_vec], c1_vec,
                                                bounds_error=False,fill_value=None)

c2_interp = interpolate.RegularGridInterpolator([m2_vec], c2_vec,
                                                bounds_error=False,fill_value=None)
```

**Step 2:** Draw initail distribution of $m_1$ and simulate forward

```python
# a. draw initial
simN = 10000
sim_m1 = np.fmax(np.random.normal(1,0.1,size=simN),0) # "randomly" chosen distribution

# b. period 1
sim_c1 = c1_interp(sim_m1)
sim_a1 = sim_m1-sim_c1

# c. transition to period 2 with random draw
sim_m2 = (1+r)*sim_a1+np.random.choice([0.5,1.5],p=[0.5,0.5])

# d. period 2
sim_c2 = c2_interp(sim_m2)
```

**Step 3:** Plot distributions

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.hist(sim_c1,bins=100,label='period 1')
ax.hist(sim_c2,bins=100,label='period 2')

ax.legend(loc='lower right',facecolor='white',frameon=True)
ax.set_xlabel('$c_t$')
ax.set_ylabel('freq.')
ax.set_title('consumption');
```

**Conclusion:** You can now solve models with complex heterogeneity and uncertainty, and simulate the implied dynamics. By introducing various policies you can quantify their effect not just for the average, but for the full distribution.

# Summary

**This lecture:**

1. Solving multidimensional optimization problems with and without gradients (and hessians)
2. Using multistart to alleviate problems with local minima (due to non-convexities)
3. Linear interpolation between known points
4. Solving dynamic optimization problems backwards period-by-period

**Dynamic optimization:** Extremely useful technique. Can handle multiple periods, multiple states and choices, more shocks etc. You can solve general equilibrium models where the households solve such problems.

**Need more dynamic optimization?** [Mini-Course on Dynamic Programming](https://github.com/NumEconCopenhagen/ConsumptionSavingNotebooks/tree/master/DynamicProgramming#mini-course-in-dynamic-programming)

**Next lecture:** The Need for Speed.


# Fitting global HDX-MS folding kinetics from Lorentzian curve fitting

Using the population fractions of the unfolded, intermediate and folded state from Lorentzian curve fitting, the refolding kinetics were fitted using the ODE model (k1, k-1 and k2).

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
from sympy import Eq
from symfit import Variable, Parameter, D, Fit, ODEModel, exp
from symfit.core.minimizers import *
from symfit.core.objectives import LeastSquares
from smitting.fitting import ODEFit
import os
```

```python
directory = '.'
```

```python
time = np.array([60*60, 30*60, 20*60, 15*60, 10*60, 5*60, 1*60, 40, 20, 0])[::-1]
states = ['FD', 'native', 'i1']
```

```python
files = [f for f  in os.listdir(directory) if 'curve' in f]
files
```

    ['global_PpiB_4C_Fitting_curve', 'Lorentzian_curve_fitting_PpiB WT.ipynb']

```python
# We choose the first file to start with:
data_arr = np.genfromtxt(os.path.join(directory, files[0]))[:, ::-1]
data_arr.shape
```

    (3, 10)

```python
# Plotting the imported data:
plt.figure()
for d, s in zip(data_arr, states):
    plt.plot(time, d, label=s)
_ = plt.legend()
```

    <IPython.core.display.Javascript object>

## ODE model fitting U <-> I -> F

```python
# First we create a dictionary with the names of the states and their data
data = {
    'FD': data_arr[0],
    'Native': data_arr[1],
    'int1': data_arr[2],
}

# Then another dictionary which defines the inital conditions.
# At t=0, the concentration of FD is set to one, other to zero
initial = {'t': 0, 'FD': 1, 'int1':0, 'Native': 0}

# We define the model with a string like this
# This model has an equillibrium between FD and int1 with an irreversible reaction from int1 to the Native
s = 'FD <-> int1 -> Native'

# The fit object is initialized by giving it the timepoints and the data dictionary
fit = ODEFit(time, **data)

# Then we can make the model
fit.make_model(s, initial)
print(fit.model)
```

    Derivative(FD, t; k_FDint1, k_int1FD, k_int1Native) = -10**k_FDint1*FD + 10**k_int1FD*int1
    Derivative(Native, t; k_FDint1, k_int1FD, k_int1Native) = 10**k_int1Native*int1
    Derivative(int1, t; k_FDint1, k_int1FD, k_int1Native) = 10**k_FDint1*FD - 10**k_int1FD*int1 - 10**k_int1Native*int1

```python
# To print the model:
for item in fit.model.model_dict.items():
    display(Eq(item[0], item[1]))
```

$\displaystyle \frac{d}{d t} FD = - 10^{k_{FDint1}} FD + 10^{k_{int1FD}} int_{1}$

$\displaystyle \frac{d}{d t} Native = 10^{k_{int1Native}} int_{1}$

$\displaystyle \frac{d}{d t} int_{1} = 10^{k_{FDint1}} FD - 10^{k_{int1FD}} int_{1} - 10^{k_{int1Native}} int_{1}$

Note that in this model the k-rates are input only as the exponent with base 10. This is because the k-values usually span multiple orders of magnitude and fitting algoritms typically take linear steps instead of logarithmic. This means that if the parameter returned from the fititng is $-2$, the actual value is $10^{\text{-}2}$

```python
# We set all parameters bounds and initial guesses:
for k in fit.parameters:
    fit.set_par(k, value=-2, min=-3.5, max=1)

# To set an individual parameter (ie k value of reaction of intermediate 1 to FD:
fit.set_par('k_int1FD', value=-3, min=-2, max=0)
```

```python
# We can use fit.interactive for checking and adjusting initial guesses
fit.interactive()
```

    <IPython.core.display.Javascript object>

    interactive(children=(FloatSlider(value=-2.0, description='k_int1FD', max=0.0, min=-2.0, step=0.02), FloatSlid…

```python
{k: v.value for k, v in fit.parameters.items()}
```

    {'k_int1FD': -2.0, 'k_FDint1': -0.8, 'k_int1Native': -2.6}

```python
# We make the fit object, using the minimizer DifferentialEvolution. This minimzer searches all parameter
# space for a global minimum. We then use SLSQP in a second step to refine the result. I've not tested all
# possibilities exhaustively so there might be better solutions
fit.make_fit(minimizer=[DifferentialEvolution, SLSQP])
```

```python
fit.execute()
```

```python
print(fit.res)
```

    Scalar Parameters:
    Parameter | Value       | Standard Deviation
    k_FDint1    8.272749e-01  2.499993e-01
    k_int1FD    -1.290588e-01  6.667881e-02
    k_int1Native-2.484892e+00  6.448847e-02
    Status message         Optimization terminated successfully.
    Number of iterations   26
    Objective              <symfit.core.objectives.LeastSquares object at 0x00000249DE3BBCC0>
    Minimizer              ChainedMinimizer(minimizers=[<symfit.core.minimizers.DifferentialEvolution object at 0x00000249DE3C85C0>, <symfit.core.minimizers.SLSQP object at 0x00000249DE3C8A90>])

    Goodness of fit qualifiers:
    chi_squared            0.08454399225070694
    objective_value        0.04227199612535347
    r_squared              0.9718186821282423

```python
for k, v in fit.res.params.items():
    print(k, v, 10**v)
```

    k_FDint1 0.8272749104204046 6.718540050499896
    k_int1FD -0.1290587922137221 0.7429185591763043
    k_int1Native -2.4848921424526376 0.0032742200042265627

The found k value for conversion from `FD` to `int1` is $10^{0.83}$ which is equal to 6.72 $s^{-1}$. This is the largest value in the kinetic scheme which is as expected because the intermediate is formed very fast.

```python
#Plotting the ODE model on the data points
fig, ax = fit.plot_result()
ax.set_xlabel('refolding time (sec)')
ax.set_ylabel('Fraction population')
ax.set_xticks((0,600,1200,1800,2400,3000,3600))

```

    <IPython.core.display.Javascript object>

    [<matplotlib.axis.XTick at 0x249e15a5588>,
     <matplotlib.axis.XTick at 0x249e15a5160>,
     <matplotlib.axis.XTick at 0x249e1591d68>,
     <matplotlib.axis.XTick at 0x249e15ea7b8>,
     <matplotlib.axis.XTick at 0x249e15eac50>,
     <matplotlib.axis.XTick at 0x249e15f3128>,
     <matplotlib.axis.XTick at 0x249e15f35c0>]

```python

```