+ This notebook is part of lecture 26 *Complex matrices and the fast Fourier transform* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, Matrix, symbols, I, sqrt, Rational
from IPython.display import Image
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Complex vectors, matrices
# Fast Fourier transform

## Complex vectors

* Consider the following vector with complex entries (from this point on I will not use the underscore to indicate a vector, so as not to create confusion with the bar, noting complex conjugate, instead, inferring from context)
$$ {z} = \begin{bmatrix} {z}_{1} \\ {z}_{2} \\ \vdots \\ {z}_{n} \end{bmatrix} $$

* The length (actually length squared) of this vector is *no good*, since it should be positive
$$ {z}^{T}{z} $$
* Instead we consider the following
$$ z\bar { z } ={ \left| { z } \right|  }^{ 2 }\\ \therefore \quad \bar { z } ^{ T }z\\ \left[ { \bar { z }  }_{ 1 },{ \bar { z }  }_{ 2 },\dots ,{ \bar { z }  }_{ n } \right] \begin{bmatrix} { z }_{ 1 } \\ { z }_{ 2 } \\ \vdots  \\ { z }_{ n } \end{bmatrix} $$

```python
z = Matrix([1, I]) # I is the sympy symbol for the imaginary number i
z
```

$$\left[\begin{matrix}1\\i\end{matrix}\right]$$

* Let's calculate this manually

```python
z.norm() # The length of a vector
```

$$\sqrt{2}$$

```python
z_cc = Matrix([1, -I])
z_cc
```

$$\left[\begin{matrix}1\\- i\end{matrix}\right]$$

```python
sqrt(z_cc.transpose() * z)
```

$$\left(\left[\begin{matrix}2\end{matrix}\right]\right)^{\frac{1}{2}}$$

* Taking the transpose of the complex conjugate is called the Hermitian
$$ {z}^{H}{z} $$

* We can use the Hermitian for non-complex (or mixed complex) vectors **u** and **v** too
$$ \bar{y}^{T}{x} \\ {y}^{H}{x} $$

```python
from sympy.physics.quantum.dagger import Dagger # A fun way to quickly get the Hermitian
```

```python
Dagger(z)
```

$$\left[\begin{matrix}1 & - i\end{matrix}\right]$$

```python
sqrt(Dagger(z) * z)
```

$$\left(\left[\begin{matrix}2\end{matrix}\right]\right)^{\frac{1}{2}}$$

## Complex symmetric matrices

### The transpose

* If the symmetric matrix has complex entries then AT=A is *no good*

```python
A = Matrix([[2, 3 + I], [3 - I, 5]])
A # A Hermitian matrix
```

$$\left[\begin{matrix}2 & 3 + i\\3 - i & 5\end{matrix}\right]$$

```python
A.transpose() == A
```

    False

```python
Dagger(A)
```

$$\left[\begin{matrix}2 & 3 + i\\3 - i & 5\end{matrix}\right]$$

```python
Dagger(A) == A
```

    True

* This will work for real-values symmetric matrices as well

```python
A = Matrix([[3, 4], [4, 2]])
A
```

$$\left[\begin{matrix}3 & 4\\4 & 2\end{matrix}\right]$$

```python
A.transpose() == A
```

    True

```python
Dagger(A) == A
```

    True

### The eigenvalues and eigenvectors

* Back to the complex matrix A

```python
A = Matrix([[2, 3 + I], [3 - I, 5]])
A
```

$$\left[\begin{matrix}2 & 3 + i\\3 - i & 5\end{matrix}\right]$$

```python
A.eigenvals()
```

$$\begin{Bmatrix}0 : 1, & 7 : 1\end{Bmatrix}$$

$$ A=\begin{bmatrix} 2 & 3+i \\ 3-i & 5 \end{bmatrix}\\ A-\lambda I=\underline { 0 } \\ \left| \begin{bmatrix} 2 & 3+i \\ 3-i & 5 \end{bmatrix}-\begin{bmatrix} \lambda  & 0 \\ 0 & \lambda  \end{bmatrix} \right| =0\\ \begin{vmatrix} 2-\lambda  & 3+i \\ 3-i & 5-\lambda  \end{vmatrix}=0\\ \left( 2-\lambda  \right) \left( 5-\lambda  \right) -\left( 3+i \right) \left( 3-i \right) =0\\ 10-7\lambda +{ \lambda  }^{ 2 }-\left( 9+1 \right) =0\\ { \lambda  }^{ 2 }-7\lambda =0\\ { \lambda  }_{ 1 }=0\\ { \lambda  }_{ 2 }=7 $$

```python
A.eigenvects()
```

$$\begin{bmatrix}\begin{pmatrix}0, & 1, & \begin{bmatrix}\left[\begin{matrix}- \frac{3}{2} - \frac{i}{2}\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}7, & 1, & \begin{bmatrix}\left[\begin{matrix}\frac{3}{5} + \frac{i}{5}\\1\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

```python
S, D = A.diagonalize()
```

```python
S
```

$$\left[\begin{matrix}-3 - i & 3 + i\\2 & 5\end{matrix}\right]$$

```python
D
```

$$\left[\begin{matrix}0 & 0\\0 & 7\end{matrix}\right]$$

* What about S now?
* We have to use its transpose, but it is complex, so we have to take the Hermitian

```python
Dagger(S)
```

$$\left[\begin{matrix}-3 + i & 2\\3 - i & 5\end{matrix}\right]$$

```python
S == Dagger(S) # Don't get confused here, S is not symmetric
```

    False

* Remember that for a symmetric matrix the column vectors in S (usually called Q, the matrix of eigenvectors) are orthogonal, with QTQ=I
* With complex entries we have to consider the Hermitian here, not just the simple transpose
* Here we call Q *unitary*

## The fast Fourier transform

* Look at this special matrix (where we start counting rows and columns at zero)

$$ { F }_{ n }=\begin{bmatrix} W^{ \left( 0 \right) \left( 0 \right)  } & { W }^{ \left( 0 \right) \left( 1 \right)  } & { W }^{ \left( 0 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( 0 \right) \left( n-1 \right)  } \\ W^{ \left( 1 \right) \left( 0 \right)  } & { W }^{ \left( 1 \right) \left( 1 \right)  } & { W }^{ \left( 1 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( 1 \right) \left( n-1 \right)  } \\ { W }^{ \left( 2 \right) \left( 0 \right)  } & { W }^{ \left( 2 \right) \left( 1 \right)  } & { W }^{ \left( 2 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( 2 \right) \left( n-1 \right)  } \\ \vdots  & \vdots  & \vdots  & \dots  & \vdots  \\ { W }^{ \left( n-1 \right) \left( 0 \right)  } & { W }^{ \left( n-1 \right) \left( 1 \right)  } & { W }^{ \left( n-1 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( n-1 \right) \left( n-1 \right)  } \end{bmatrix} \\ \left({F}_{n}\right)_{ij}={W}^{ij}; i,j=0,1,2,\dots,n-1 $$

* W is a special number whose *n*th power equals 1
$$ {W}^{n}=1 \\ W={ e }^{ \frac { i2\pi  }{ n }  }=\cos { \frac { 2\pi  }{ n } +i\sin { \frac { 2\pi  }{ n }  }  }  $$
* It is in the complex plane of course (as written in *sin* and *cos* above)

* Remember than *n* here refers to the size the matrix
* Here it also refers to the *n*th *n* roots (if that makes any sense, else look at the image below)

```python
Image(filename = 'W.png')
```

* So for *n*=4 we will have the following
$$ { F }_{ 4 }=\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 1 } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 2 } } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 3 } } \\ 1 & \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 2 } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 4 } } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 6 } } \\ 1 & \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 3 } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 6 } } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 9 } } \end{bmatrix} $$

* We note that a quarter of the way around is *i*
$$ {e}^{\frac{2\pi{i}}{4}}={i} $$
* We thus have the following
$$ { F }_{ 4 }=\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & i & { i }^{ 2 } & { i }^{ 3 } \\ 1 & { i }^{ 2 } & { i }^{ 4 } & { i }^{ 6 } \\ 1 & { i }^{ 3 } & { i }^{ 6 } & { i }^{ 9 } \end{bmatrix}\\ { F }_{ 4 }=\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & i & -1 & -i \\ 1 & -1 & 1 & -1 \\ 1 & -i & -1 & i \end{bmatrix} $$

* Note how the columns are orthogonal

```python
F = Matrix([[1, 1, 1, 1], [1, I, -1, -I], [1, -1, 1, -1], [1, -I, -1, I]])
F
```

$$\left[\begin{matrix}1 & 1 & 1 & 1\\1 & i & -1 & - i\\1 & -1 & 1 & -1\\1 & - i & -1 & i\end{matrix}\right]$$

```python
F.col(0) # Calling only the selected column (counting starts at 0)
```

$$\left[\begin{matrix}1\\1\\1\\1\end{matrix}\right]$$

* The columns are supposed to be orthogonal, i.e. inner (dot) product should be zero
* Clearly below it is not

```python
F.col(1).dot(F.col(3))
```

$$4$$

* Remember, though, that this is a complex matrix and we have to use the Hermitian

```python
col1 = F.col(1)
col3 = F.col(3)
col1, col3
```

$$\begin{pmatrix}\left[\begin{matrix}1\\i\\-1\\- i\end{matrix}\right], & \left[\begin{matrix}1\\- i\\-1\\i\end{matrix}\right]\end{pmatrix}$$

```python
Dagger(col3), col1
```

$$\begin{pmatrix}\left[\begin{matrix}1 & i & -1 & - i\end{matrix}\right], & \left[\begin{matrix}1\\i\\-1\\- i\end{matrix}\right]\end{pmatrix}$$

```python
Dagger(col3) * col1 # Another way to do the dot product
```

$$\left[\begin{matrix}0\end{matrix}\right]$$

* So, these columns are all orthogonal, but they are not orthonormal
* Note, though that the are all of length 2, so we can normalize each

```python
Rational(1, 2) * F
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{1}{2} & \frac{1}{2} & \frac{1}{2}\\\frac{1}{2} & \frac{i}{2} & - \frac{1}{2} & - \frac{i}{2}\\\frac{1}{2} & - \frac{1}{2} & \frac{1}{2} & - \frac{1}{2}\\\frac{1}{2} & - \frac{i}{2} & - \frac{1}{2} & \frac{i}{2}\end{matrix}\right]$$

* We also note the following
$$ {F}_{n}^{H}{F}_{n}={I} $$
* Just remember to normalize them

```python
Dagger(Rational(1, 2) * F)
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{1}{2} & \frac{1}{2} & \frac{1}{2}\\\frac{1}{2} & - \frac{i}{2} & - \frac{1}{2} & \frac{i}{2}\\\frac{1}{2} & - \frac{1}{2} & \frac{1}{2} & - \frac{1}{2}\\\frac{1}{2} & \frac{i}{2} & - \frac{1}{2} & - \frac{i}{2}\end{matrix}\right]$$

```python
Dagger(Rational(1, 2) * F) * ((Rational(1, 2) * F))
```

$$\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$$

* Now why do we call it *fast* Fourier transform
* Note the following
$$ { W }_{ n }={ e }^{ \frac { 2\pi i }{ n }  }\\ { \left( { W }_{ n } \right)  }^{ p }={ \left( { e }^{ \frac { 2\pi i }{ n }  } \right)  }^{ p }\\ { \left( { W }_{ 64 } \right)  }^{ 2 }={ \left( { e }^{ \frac { 2\pi i }{ 64 }  } \right)  }^{ 2 };\quad n=64,\quad p=2\\ \therefore \quad { \left( { W }_{ 64 } \right)  }^{ 2 }={ W }_{ 32 } $$

* Now we have the following connection between the two
$$ \left[ { F }_{ 64 } \right] =\begin{bmatrix} I & D \\ I & -D \end{bmatrix}\begin{bmatrix} { F }_{ 32 } & 0 \\ 0 & { F }_{ 32 } \end{bmatrix}\left[ P \right] \\ D=\begin{bmatrix} 1 & 0 & 0 & \dots  & 0 \\ 0 & W & 0 & \dots  & 0 \\ 0 & 0 & { W }^{ 2 } & \dots  & 0 \\ \vdots  & \vdots  & \vdots  & \dots  & \vdots  \\ 0 & 0 & 0 & \dots  & { W }^{ 31 } \end{bmatrix}$$
* P is a permutation matrix

* Going down to 16 will include the following
$$ \begin{bmatrix} I & D & 0 & 0 \\ I & -D & 0 & 0 \\ 0 & 0 & I & D \\ 0 & 0 & I & -D \end{bmatrix}\begin{bmatrix} { F }_{ 16 } & 0 & 0 & 0 \\ 0 & { F }_{ 16 } & 0 & 0 \\ 0 & 0 & { F }_{ 16 } & 0 \\ 0 & 0 & 0 & { F }_{ 16 } \end{bmatrix}\left[ P \right]  $$

* The recursive work above leads to decreasing the work that is required for working with these problems

```python

```


Metabolic Control Analysis

```
pip install tellurium
```

    Collecting tellurium
      Downloading https://files.pythonhosted.org/packages/06/d8/581c899fff31b2d3c84b6df210f6f86d2cc5228acb51fc175292f23773ca/tellurium-2.2.0-py3-none-any.whl (118kB)
         |████████████████████████████████| 122kB 7.7MB/s
    Requirement already satisfied: jinja2>=2.9.6 in /usr/local/lib/python3.7/dist-packages (from tellurium) (2.11.3)
    Collecting libroadrunner>=2.0.3
      Downloading https://files.pythonhosted.org/packages/ed/7a/afe6e667e6aa2533665042870df354576d0e9944b6f79ae948ba21404b39/libroadrunner-2.0.5-cp37-cp37m-manylinux2010_x86_64.whl (16.6MB)
         |████████████████████████████████| 16.6MB 185kB/s
    Collecting phrasedml>=1.0.9
      Downloading https://files.pythonhosted.org/packages/c1/bd/eb0c9d729d59fcfb3b8123dbad7dd7801bdb7a6495bc03ef823bbf045b64/phrasedml-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (5.6MB)
         |████████████████████████████████| 5.6MB 40.7MB/s
    Requirement already satisfied: jupyter-core>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from tellurium) (4.7.1)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from tellurium) (2.23.0)
    Collecting python-libnuml>=1.0.0
      Downloading https://files.pythonhosted.org/packages/68/d8/7ebc91b3c925a52799ffa36eb3e33fe2400e7d4984199cd1f4ac38dc377f/python_libnuml-1.1.3-cp37-cp37m-manylinux1_x86_64.whl (2.0MB)
         |████████████████████████████████| 2.0MB 37.9MB/s
    Collecting rrplugins>=2.0.3
      Downloading https://files.pythonhosted.org/packages/07/44/ac0b3505a4044eca46f1793665abc1a55fd2797103b192b0750f76e5c69a/rrplugins-2.0.4-py3-none-manylinux2010_x86_64.whl (36.7MB)
         |████████████████████████████████| 36.7MB 83kB/s
    Requirement already satisfied: jupyter-client>=5.1.0 in /usr/local/lib/python3.7/dist-packages (from tellurium) (5.3.5)
    Requirement already satisfied: ipykernel>=4.6.1 in /usr/local/lib/python3.7/dist-packages (from tellurium) (4.10.1)
    Collecting numpy==1.19.3
      Downloading https://files.pythonhosted.org/packages/65/b3/07864c89acb2a86df6f2e8c9bf091ec5916da58dd3ce3a633a51a02c115e/numpy-1.19.3-cp37-cp37m-manylinux2010_x86_64.whl (14.9MB)
         |████████████████████████████████| 14.9MB 184kB/s
    Collecting python-libcombine>=0.2.2
      Downloading https://files.pythonhosted.org/packages/38/ba/67cf2742ad188147efcca0239c69328921d807ba605a54fbfc1016867462/python_libcombine-0.2.10-cp37-cp37m-manylinux1_x86_64.whl (2.5MB)
         |████████████████████████████████| 2.5MB 44.1MB/s
    Collecting python-libsedml>=2.0.12
      Downloading https://files.pythonhosted.org/packages/d7/dc/b304c05f83f74e4ef4edbe9af6cbef4a456a8c92011abe76343181e24794/python_libsedml-2.0.18-cp37-cp37m-manylinux1_x86_64.whl (3.1MB)
         |████████████████████████████████| 3.1MB 43.3MB/s
    Requirement already satisfied: ipython in /usr/local/lib/python3.7/dist-packages (from tellurium) (5.5.0)
    Requirement already satisfied: pandas>=0.20.2 in /usr/local/lib/python3.7/dist-packages (from tellurium) (1.1.5)
    Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from tellurium) (3.6.4)
    Requirement already satisfied: matplotlib>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from tellurium) (3.2.2)
    Requirement already satisfied: plotly>=2.0.12 in /usr/local/lib/python3.7/dist-packages (from tellurium) (4.4.1)
    Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.7/dist-packages (from tellurium) (1.4.4)
    Collecting antimony>=2.12.0
      Downloading https://files.pythonhosted.org/packages/16/13/849d12029ef311d1677f804d8c19da290dede5f29b82028ecccabf5ffc08/antimony-2.12.0.2-py3-none-manylinux2010_x86_64.whl (5.8MB)
         |████████████████████████████████| 5.8MB 37.0MB/s
    Collecting scipy>=1.5.1
      Downloading https://files.pythonhosted.org/packages/75/91/ee427c42957f8c4cbe477bf4f8b7f608e003a17941e509d1777e58648cb3/scipy-1.6.2-cp37-cp37m-manylinux1_x86_64.whl (27.4MB)
         |████████████████████████████████| 27.4MB 94kB/s
    Collecting python-libsbml>=5.18.0
      Downloading https://files.pythonhosted.org/packages/93/13/2df22b77ee9125381a00aa4dc63de1776bb84d773655d2770416e81416cd/python_libsbml-5.19.0-cp37-cp37m-manylinux1_x86_64.whl (6.2MB)
         |████████████████████████████████| 6.2MB 25.0MB/s
    Collecting sbml2matlab>=0.9.1
      Downloading https://files.pythonhosted.org/packages/51/f1/fd7844a2c2f6ec7b32d37addf22a175e33b999e3b920f7689085f42e1b65/sbml2matlab-1.2.3-cp37-cp37m-manylinux1_x86_64.whl (3.2MB)
         |████████████████████████████████| 3.2MB 47.4MB/s
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2>=2.9.6->tellurium) (1.1.1)
    Requirement already satisfied: traitlets in /usr/local/lib/python3.7/dist-packages (from jupyter-core>=4.3.0->tellurium) (5.0.5)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->tellurium) (2020.12.5)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->tellurium) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->tellurium) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->tellurium) (3.0.4)
    Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.7/dist-packages (from jupyter-client>=5.1.0->tellurium) (22.0.3)
    Requirement already satisfied: tornado>=4.1 in /usr/local/lib/python3.7/dist-packages (from jupyter-client>=5.1.0->tellurium) (5.1.1)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from jupyter-client>=5.1.0->tellurium) (2.8.1)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (0.7.5)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (4.8.0)
    Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (56.0.0)
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (0.8.1)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (1.0.18)
    Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (2.6.1)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython->tellurium) (4.4.2)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.20.2->tellurium) (2018.9)
    Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium) (1.10.0)
    Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium) (1.4.0)
    Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium) (8.7.0)
    Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium) (20.3.0)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium) (1.15.0)
    Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium) (0.7.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.0.2->tellurium) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.0.2->tellurium) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.0.2->tellurium) (0.10.0)
    Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.7/dist-packages (from plotly>=2.0.12->tellurium) (1.3.3)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/dist-packages (from traitlets->jupyter-core>=4.3.0->tellurium) (0.2.0)
    Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect; sys_platform != "win32"->ipython->tellurium) (0.7.0)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->tellurium) (0.2.5)
    ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
    ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
    Installing collected packages: numpy, libroadrunner, phrasedml, python-libnuml, rrplugins, python-libcombine, python-libsedml, antimony, scipy, python-libsbml, sbml2matlab, tellurium
      Found existing installation: numpy 1.19.5
        Uninstalling numpy-1.19.5:
          Successfully uninstalled numpy-1.19.5
      Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    Successfully installed antimony-2.12.0.2 libroadrunner-2.0.5 numpy-1.19.3 phrasedml-1.1.1 python-libcombine-0.2.10 python-libnuml-1.1.3 python-libsbml-5.19.0 python-libsedml-2.0.18 rrplugins-2.0.4 sbml2matlab-1.2.3 scipy-1.6.2 tellurium-2.2.0

```
import sympy
from sympy import init_printing
init_printing()
import tellurium as te
```

## Latex Macros
Macro definitions
* relative values: \rv{x} - the relative value of x
$\newcommand{\rv}[1]{\frac{\delta #1}{#1}}$: $\rv{x}$

* normalized elasticity: \els{i}{x} - how reaction rate i is impacted by $x$
$\newcommand{\els}[2]{\epsilon_{#2}^{#1}}$: $\els{i}{x}$

* unnormalized elasticity: \uels{i}{x} - how reaction rate i is impacted by $x$
$\newcommand{\uels}[2]{\zeta_{#2}^{#1}}$: $\uels{i}{x}$

* term in local equation: \ltm{v}{x} - how a term in a kinetic equation contributes to a change in reaction rate.
$\newcommand{\ltm}[2]{\els{#1}{#2} \rv{#2}}$: $\ltm{v}{x}$

* control coefficient: \ctl{x}{b} - how $x$ changes with a change in $b$
$\newcommand{\ctl}[2]{C_{#2}^{#1}}$: $\ctl{x}{b}$

* change in species concentrations: \dxdt - how $x$ changes with respect to $t$
$\newcommand{\dxdt}[]{ \frac{d {\bf x}}{d t}}$: $\dxdt$

* change in parameter values: \dpdt - how $p$ changes with respect to $t$
$\newcommand{\dxdt}[]{ \frac{d {\bf p}}{d t}}$: $\dpdt$

* total change in reaction rate: \dv
$\newcommand{\dv}[] {d {\bf v} }$: $\dv$

* total change in state: \dx
$\newcommand{\dx}[] {d {\bf x} }$: $\dx$

* reaction rates: \vt -
$\newcommand{\vt}[]{ \bf{v}}$ $\vt$

* change in reaction rate w.r.t. $\bf{x}$: \dvdx -
$\newcommand{\dvdx}[]{ \frac{\partial {\bf v}}{\partial {\bf x}}}$ $\dvdx$

* change in reaction rate w.r.t. $\bf{p}$: \dvdp -
$\newcommand{\dvdp}[]{ \frac{\partial {\bf v}}
{\partial {\bf p}}}$ $\dvdp$

* LaPlace transform of $\bf{x}$: \ltx -
$\newcommand{\ltx}[]{ {\bf \cal{L}(x)}}$ $\ltx$

* LaPlace transform of $\bf{v}$: \ltv -
$\newcommand{\ltv}[]{ {\bf \cal{L}(v)}}$ $\ltv$

* LaPlace transform of $\bf{p}$: \ltp -
$\newcommand{\ltp}[]{ {\bf \cal{L}(p)}}$ $\ltp$

* Transfer function $\bf{x}$ w.r.t. $\bf{p}$,
$\frac{\ltx}{\ltp}$: \tfx -
$\newcommand{\tfx}[]{ {\bf H_x}(s)}$ $\tfx$

* Transfer function for fluxes,
$\frac{\ltv}{\ltp}$: \tfj -
$\newcommand{\tfj}[]{ {\bf H_J}(s)}$ $\tfj$

Writing a matrix
$
\begin{equation*}
A_{m,n} =
\begin{pmatrix}
a_{1,1} & a_{1,2} & \cdots & a_{1,n} \\
a_{2,1} & a_{2,2} & \cdots & a_{2,n} \\
\vdots  & \vdots  & \ddots & \vdots  \\
a_{m,1} & a_{m,2} & \cdots & a_{m,n}
\end{pmatrix}
\end{equation*}
$

## Notation

*   $S_i$ is the $i$-th metabolite.
*   $s_i$ is the concentration of $S_i$.
* $q_i$ is the equilibrium constant for reaction $i$.
* $\bf{N}$ is the stoichiometry matrix. $n \times r$, where $n$ is the number of species and $r$ is the number of reactions.
* $\bf{x}$ is a vector of species for the $n$ species
* $\bf{v}$ is vector of reaction rates for the $r$ reactions
* $\bf{p}$ is the vector of parameters

1.   Relative vs. absolute values. The absolute change in $x$ is $\delta x$. The relative is $\frac{\delta x}{x}$.

# Transfer Function Analysis

## State

1. $\dxdt = \bf{N} \bf{v}(\bf{x}, \bf{p})$.

1. Linearizing the right hand side, $\dxdt \approx \bf{N}\dvdx \dxdt + \bf{N} \dvdp \dpdt$.

1. Letting $A = \bf{N} \dvdx$ and $B = \bf{N} \dvdp$, we have $\dxdt \approx \bf{A}$ $  d \bf{ x }$ $+ {\bf{B}}  d \bf{p}$.

1. From (2), the LaPlace transform of both sides is
$s \cal{L}(\bf{x}) = \bf{N} \dvdx \cal{L}(x) + \bf{N} \dvdp \cal{L}(\bf{p})$. And so, $\bf{H_x}(s) =
\frac{ \ltx }{ \ltp } =
[s \bf{I} - N \dvdx]^{-1}\bf{N} \dvdp$.

1. The canonical transfer function considers a unit change in each reaction w.r.t. $\bf{p}$.
$\tilde{\bf{H}}_x(s) = \bf{H}(\bf{x}) \left( \dvdp \right) ^{-1}$ $=[s \bf{I} - N \dvdx]^{-1}\bf{N}$.

## Flux

1. Linearizing, $\dvdt \approx \dvdx \dxdt +  \dvdp \dpdt$.

1. Letting $C = \dvdx$ and $D = \dvdp$, we have
$\dxdt \approx \bf{C}$ $  d \bf{ x }$ $+ {\bf{D}}$  $d \bf{p}$.

1. From (2), the LaPlace transform of both sides is
$s \ltv = \dvdx \ltx + \dvdp \ltp$
$= \dvdx \tfx \ltp + \dvdp \ltp$.
And so, $\bf{H_J}(s) = \dvdx \tfx + \dvdp$.

1. The canonical transfer function considers a unit change in each reaction w.r.t. $\bf{p}$.
$\bf{\tilde{H}_J}$$(s) = \dvdx \bf{\tilde{H}_x} + \bf{I}$.

# Elasticities

1. $\els{v_i}{s} = \rv{v_i} /\rv{s_i}$
1. Consider elasticities for various kinetics
   1. Zeroth order kinetics: $v = k$
   1. Mass action: $v = k \prod_i r_i^{n_i}$, where $r_i$ is the $i$-th reactant
   1. Hill equation: $v = \frac{k_1 x^n}{k_2 + x^n}$. This includes the Haldane equation.
   1. State based kinetics. $x_0 = \sum_i x_i$.
   $v = k_1 \frac{\sum_{j \in S} k_i x_j^{n_j}}{k_2 + \sum_i k_i x_i^{n_i}}$.

1. Enzyme elascity
   1. Reversible MM: $v = V_f \frac{S/K_S}{1 + S/K_S + P/K_P}$

```
symbols = "x_1, x_2, x_3, k_1, k_2, k_3, n_1, n_2, n_3, k, I, V_f, v, V_r, s, p, k_s, k_p, V_M"
statement = "%s = sympy.symbols('%s')" % (symbols, symbols)
exec(statement)
```

```
mmEx = V_f * (s /k_s)/ (1 + s/k_s + p/k_p)
elasS = sympy.diff(mmEx, s)
elasS.simplify()
```

```
mmRevEx = V_f * (s /k_s)/ (1 + s/k_s + p/k_p) - V_r * (p / k_p)/(1 + s/k_s + p/k_p)
elasRevS = sympy.diff(mmRevEx, s)
elasRevS = elasRevS.simplify()
```

```
n, d = sympy.fraction(elasRevS)
elasRevS = (n.expand() /d) / mmEx
elasRevS.simplify()

```

```

```

## Mass action kinetics

```
mass_action_kin = k*(x_1**n_1)*(x_2**n_2)*(x_3**n_3)
mass_action_kin
```

```
mass_action_els = x_1*sympy.diff(mass_action_kin, x_1)/mass_action_kin
mass_action_els
```

```
def calcEls(expr, ele):
  return ele*sympy.diff(expr, ele)/expr
```

```
calcEls(mass_action_kin, x_2)
```

```
# Subscripted sequence
n, r = sympy.symbols('n, r')
a_n = sympy.Function('a')(n)
terms = 4
short_expr = sympy.Sum(a_n * r**n, (n, 0, terms - 1))
a_seq = [-1, 3, 23, 8]
coeffed_short_expr = short_expr.doit().subs(
    (a_n.subs(n, i), a_seq[i]) for i in range(terms))
coeffed_short_expr.subs(r, r)
```

```
sympy.Sum((I/3)**n,(n,0,sympy.oo)).doit()
```

$\displaystyle \begin{cases} \frac{1}{1 - \frac{I}{3}} & \text{for}\: \frac{\left|{I}\right|}{3} < 1 \\\sum_{n=0}^{\infty} 3^{- n} I^{n} & \text{otherwise} \end{cases}$

## State based kinetics

```
x1, x2, k1, k2, n1, n2 = sympy.symbols("x1, x2, k1, k2, n1, n2")
expr = (k1*x1**n1)/(k1*x1**n1 + k2*x2**n2)
expr
```

```
x1_els = x1*(sympy.diff(expr, x1)/expr)
x1_els
```

This can be simplified by extracting $v$.

# Control Coefficients

1.   There are two kinds of control coefficients, flux and concentration. $C^J_i$ is the flux control and $C^s_i$ is the concentration control for species $s$. We use $C_i^x$ to denote a control coefficient that may be either of the two.

2.   $C^J_i = \frac{\delta J}{J} / \frac{\delta e_i}{e_i}$, where $e_i$ is the amount of the enzyme that catylzes the $i$-th reaction.

1.   Using a first order Taylor's approximation,
$\frac{\delta J}{J} = \sum C^J_i \frac{\delta e_i}{e_i}$.

1. We can linearize reaction rates as well, especially for mass action kinetics. Let $v_i$ be the rate for the $i$-th reaction, then
$\delta v_i = k_{i0} \delta e_i + \sum_{j} \phi_{ij} k_{ij} \delta s_j$, where $\phi_{ij}$ is 1 if metabolite $s_j$ participates in the kinetics for reaction $i$.

1. Suppose we have:
    1.  a linear pathway of irreversible reactions so that at steady state $v_i = v_j$ for all $i,j$
    1. each enzyme affects only one reaction
    1. we make compensating changes in $E_i$ and $E_{i+1}$ such that $\delta J = 0$.

1. Then
    1. $v_i = k_i e_i s_i = k_{i+1} e_{i+1} s_{i+1} = v_{i+1}$
    1. $0 = C_i^J (\epsilon_{e_i}^i \frac{\delta e_i}{e_i} + \epsilon_{s_i}^i \frac{\delta s_i}{s_i}) + C_{i+1}^J (\epsilon_{e_{i+1}}^{i+1} \frac{\delta e_{i+1}}{e_{i+1}} + + \epsilon_{s_{i+1}}^{i+1} \frac{\delta s_{i+1}}{s_{i+1}} )$

$0 = C_i^J \frac{\delta e_i}{e_i} + C_{i+1}^J \frac{\delta e_{i+1}}{e_{i+1}}$

**Issue**: To make use of local equations, I need to have the same units, relative or absolute. There seems to be an inconsistency here.

# General Relationships

1. $\rv{J} = \sum_{s \in \cal{S}} \ctl{J}{s} \rv{s}$ for changes for all $s$ in $\cal{S}$.

1. $\rv{J} = \sum_{e \in \cal{E}} \ctl{J}{e} \rv{e}$ can be viewed as the control exerted by the flux of each reaction (if the enzymes $e$ are in a linear range). This means that if $\rv{e} =1$, then $\ctl{J}{e} / \rv{J^* }$ is the relative effect of the reaction for $e$ on $J$, where $\rv{J^* }$ is result of a unit percent change in each reaction flux.

1. Consider a collection of enzymes $\cal{E}_S$ for reactions that directly affect the concentration of $S$.
Consider an experiment in all fluxes are constant and only the concentration of $S$ changed in a way that compensates for the change in $e$. Then,
$\rv{J} = \sum_{e \in \cal{E}} \ctl{J}{e} \rv{e}$.
But for each reaction in the set, $\rv{v} = \els{v}{e} \rv{e} + \els{v}{s} \rv{s}$.
Thus, $\rv{e} = - \els{v}{s} / \els{v}{e} \rv{s}$ for each $e$.
And so, $0 = \sum_{e \in \cal{E}} \ctl{J}{e} \els{v}{s} / \els{v}{e} \rv{s}$,
which is equivalent to $0 = \sum_{e \in \cal{E}} \ctl{J}{e} \els{v}{s} / \els{v}{e}$. If we are in a linear regime for the enzymes, then
$0 = \sum_{e \in \cal{E}} \ctl{J}{e} \els{v}{s}$.

1. $\els{x}{y} \rv{y}$ is the change in
   $x$ due to $y$ for a single reaction.

1. $\els{x}{y} / \els{x}{z} = \els{z}{y}$

1. If there is a unique enzyme for each reaction, then we can assess he contribution of each reaction using $\rv{J} = \sum_{e \in \cal{E}} \ctl{J}{e} \rv{e}$, where each $e \in \cal{E}$ refers to a distinct reaciton.

1. Consider a linear network with a single reactant and product, unique reactants, reversible reactions, and unique enzymes $e_i$ for each reaction. That is, $\cdots \rightarrow S_1 \xrightarrow{v_1} S_2 \xrightarrow{v_2} S_3 \cdots$.
   1. It must be that $J = v_i$, and so
$\rv{J} = \rv{v_i}$ at steady state.

   1. $\rv{v_i} = \els{v_i}{e_i} \rv{e_i} + \els{v_i}{s_i} \rv{s_i} + \els{v_i}{s_{i+1}} \rv{s_{i+1}}$.

   1. Suppose we conduct an experiment in which we adjust $e_{i-1}$ and $e_i$ so that $\rv{J} = 0$ such that there is a single $i^* > 1$ such that $\rv{s_{i^*}} \neq 0$.

      1. Only reactions $i^* -1$ and $i^* $ are impacted since $\rv{v_i} = 0$ for $i  i^*$ (for the same reason).

      1. $\rv{v_{i^* }} = \els{v_{i^* }}{e_{i^* }} \rv{e_{i^* }} + \els{v_{i^* }}{s_{i^* }} \rv{s_{i^* }} = 0$.

      1. $\els{s_{i^* }}{e_{i^* }} =  \els{v_{i^* }}{e_i^* } / \els{v_{i^* }}{s_{i^* }}$.

      1. $\rv{v_{i^* -1}} = \els{v_{i^* -1 }}{e_{i^* -1}} \rv{e_{i^* -1}} + \els{v_{i ^* -1}}{s_{i^* }} \rv{s_{i^* }} = 0$.

      1. TODO: calculate $\rv{s_i^* }$

      1. As a result, $\rv{e_{i^* }} / \rv{e_{i^* -1}} =
      \phi(v_{i* -1}, e_{i^* -1}, s_{i^* }) / \phi(v_{i* }, e_{i^* }, s_{i^* })$

1. Now consider two adjacent reactions in which we change their respective enzymes in a compensating so that $\rv{J} = 0$. For convenience, we use the subscripts 1 and 2. $\cdots \rightarrow S_1 \xrightarrow{v_1} S_2 \xrightarrow{v_2} S_3 \cdots$. We assume that the experiment is conducted so that only
the $e_i$ and $s_2$ change.
   1. $\rv{J} = \rv{v_1} = \rv{v_2} = 0$
   1. $\rv{v_1} = \ltm{v_1}{s_1} + \ltm{v_1}{e_1} + \ltm{v_1}{s_2} = \ltm{v_1}{e_1} + \ltm{v_1}{s_2} = 0$
   1. $\rv{v_2} = \ltm{v_2}{e_2} + \ltm{v_2}{s_2} = 0$
   1. So, $\ltm{v_1}{e_1} = - \ltm{v_1}{s_2}$ and
   $\ltm{v_2}{e_2} = - \ltm{v_2}{s_2}$. Hence,
   $\rv{e_1} = -\els{v_1}{s_2} / \els{v_1}{e_1} \rv{s_2}$.
   Similarly, $\rv{e_2} = -\els{v_2}{s_2} / \els{v_2}{e_2} \rv{s_2}$.

1. Suppose that enzyme flux elasticity is 1 and we make compensating changes in
a set of enzymes $\cal{E}$ such that $\delta J = 0$ and only one chemical species changes, $S$, and for all reactions $i$ for which $\delta e_i \neq 0$,
there is an $S$ in the reaction. Then,
    * $0 = \rv{J} = \sum_{e \in \cal{E}} \ctl{J}{e} \rv{e}$
    * $\delta v = \delta e + \els{v}{s} \delta s$
    * Assuming that $\delta v = 0$, then $\delta e = - \els{v}{s} \delta s$.
    * $0 = \sum_i - \ctl{J}{v_i} \els{v_i}{s} \delta s
    \equiv \sum_i \ctl{J}{v_i} \els{v_i}{s} = 0$

# Connectivity Theorems

## Framework

1. Flux can change but designated species concentrations do not. (Non-designated species concentrations likely will change.)
1. Consider $S^{\prime} \xrightarrow{v_1} S \xrightarrow{v_2} S^{\prime \prime} \xrightarrow{v_3 }$. The reactions are reversible.

1. Observed that $\rv{J} = \rv{v_i}$.

1. Assume that $\rv{e_1} \neq 0 \neq \rv{e_2}$ so that $\rv{J} = 0$
and so $\rv{v_i} = 0$.
However, $S$ does change.

1. Since $\rv{v_1} = \rv{v_2}$,
$\els{v_1}{e_1} \rv{e_1} - \rv{s}\els{v_1}{s} = -\els{v_2}{e_2} \rv{e_2} - \els{v_2}{s} \rv{s}$.

1. $\rv{s_2} = \els{s_2}{e_2} \rv{e_2} + \els{s}{e_2} \rv {e_2}$.

1. Substituting, we have
$\els{v_1}{e_1} \rv{e_1} = \els{v_2}{e_2} \rv{e_2} + \els{v_2}{s_2}
( \els{s_2}{e_1} \rv{e_1} + \els{s_2}{e_2} \rv {e_2})$.

1. How do I extract $e_i$? Let $r = \rv{e_1} / \rv{e_2}$. Then,
$\els{v_1}{e_1} r = \els{v_2}{e_2}  + \els{v_2}{s_2}
( \els{s_2}{e_1} r + \els{s_2}{e_2})$ and so
$r = \frac{
  \els{v_2}{e_2}  + \els{v_2}{s_2} \els{s_2}{e_2}}{\els{v_1}{e_1} - \els{v_2}{s_2}}$. Notice the singularity when $\els{v1}{e_1} = \els{v_2}{s_2}$. Also, note that the requirement that $\els{v1}{e_1} > \els{v_2}{s_2}$.

# Linear pathways

Consider a linear pathway in which reaction $i$ converts $S_i$ to $S_{i+1}$ at a rate $k_i e_i s_i$, and
and the reverse reaction has kinetics constant $k_{-i}$.
Note that this payway assumes that uni-uni reactions with a moderator (enzyme).

1.    $v_i = e_i(s_i k_i - s_{i+1} k_{-1})$
1. $\rho_i$ is the disequilibrium constant for reaction $i$, where $\frac{s_{i+1}}{s_i}/q_i$. This is in $[0, \infty)$.
    1. $\rho_i = 1$ if $v_i = 0$.
    1. $\rho_i = 0$ if $k_{-i} = 0$. That is, reaction $i$ is irreversible.
    1. $\rho_i = \infty$ if $s_i = 0$.
1. $v_i = k_i (s_i - s_{i+1}/q_i)$
1. Further, $v_i = k_is_i(1 - \rho_i)$

Find the relationships between pairs of $C_i^J$. Consider an experiment in which all metabolite concentrations are kept constant and enzyme concentrations are changed in a manner so that flux does not change.
1. By construction of the experiment, $\rv{J} = \sum C_i^J \els{v_i}{e} \rv{e}$.
1. $v_i = v_j$ for all $i,j$ since this is a linear pathway.
1. $\els{v_i}{e_i} \rv{e_i} = \rv{v_i}$.
1. So, $\rv{J} = \rv{v_j}$ for all reactions in a
linear pathway.
1. We conclude
$\rv{J} = \sum C_i^J \rv{v_i} = \sum C_i^J \rv{J} \equiv 1 = \sum C_i^J$.
1. An easier derivation of the above is to note that
   1. $\rv{J} = \sum C_i^J \rv{v_i}$.
   1. $v_i = J = v_j$, if the stoichiometries are all 1.
   1. $v_i = J =  v_j$ it must be that $\rv{v_i} = \rv{J} = \rv{v_j}$.
   1. Since the change is non-zero, we can divide, which yields the conclusion.

Now consider concentrations.

Now consider a set of compensating changes for $i \in S$.
Note that
$\sum_{i \in S} C_i^x \epsilon_e^i = 0$, where $x$ indicates that the
change may be in flux or concentrations.

1. This can be generalized slight to consider a set of changes in enzyme concentrations that result in a known change, and so the right hand side need not be 0.

# Generalizing Linear Pathways

Can this be generalized further beyond linear pathways? And, should this apply to species as well as flux? For a general reaction network, we can't equate the reaction rates, and so must include the changes in enzymes.

Let $C_i^x$ denote a control coefficient for flux or concentration, and
$i \in S$ be a set of compensating changes.
Note that
$\sum_{i \in S} C_i^x \epsilon_e^i \frac{\delta e_i}{e_i} = 0$.

# One Step Linear Pathway

## State

1. Consider the reversible reaction $X_0 \xrightarrow{v_1} x \xrightarrow{v_2} X_1$, where $X_i$ is a fixed species.
1. $\dvdx =
\begin{equation*}
\begin{pmatrix}
\zeta_x^1  \\
\zeta_x^2   \\
\end{pmatrix}
\end{equation*}
$
1. $N =
\begin{equation*}
\begin{pmatrix}
1 & -1  \\
\end{pmatrix}
\end{equation*}
$
1. $N \dvdx = \zeta_x^1 - \zeta_x^2$
1. $\dvdx N =
\begin{equation*}
\begin{pmatrix}
\zeta_x^1 & -\zeta_x^1 \\
\zeta_x^2 & -\zeta_x^2 \\
\end{pmatrix}
\end{equation*}
$
1. $sI - N \dvdx = s - \zeta_x^1 + \zeta_x^2$
1. $\left( sI - N \dvdx \right)^{-1} = \frac{1}{s - \zeta_x^1 + \zeta_x^2}$
1. ${\cal C}_x = \left( sI - N \dvdx \right) ^{-1} N =
\begin{equation*}
\begin{pmatrix}
\frac{1}{s - \zeta_x^1 + \zeta_x^2} & \frac{-1}{s - \zeta_x^1 + \zeta_x^2}  \\
\end{pmatrix}
\end{equation*}
$

## Flux

$\dv = \dvdx \bf{x} + \dvdp \bf{p}$

$\begin{align}
\dv & = & \dvdx \bf{x} & + & \dvdp \bf{p} \\
s V(s) & = & \dvdx X(s) & +  & \dvdp P(s) \\
s V(s) & = & \dvdx N V(s) & + & \dvdp P(s) \\
\left( sI - \dvdx N \right) V(s)  & = & \dvdp P(s) \\
H_J(s) & = &\left( sI - \dvdx N \right)^{-1} \dvdp
\end{align}
$

# Two Step Linear Pathway

Formulation: $X_0 \xrightarrow{v_1} x_1 \xrightarrow{v_2} x_2 \xrightarrow{v_3} X_1$.

$
\begin{equation*}
N =
\begin{pmatrix}
1 & -1 & 0 \\
0 & 1 & -1 \\
\end{pmatrix}
\end{equation*}
$

$
\begin{equation*}
\dvdx =
\begin{pmatrix}
\uels{1}{1} & \uels{1}{2} \\
\uels{2}{1} & \uels{2}{2} \\
\uels{3}{1} & \uels{3}{2} \\
\end{pmatrix}
\end{equation*}
$

Assuming mass action kinetics:
$
\begin{equation*}
\dvdx =
\begin{pmatrix}
 0 & 0 \\
k_1 & 0 \\
0 & k_2 \\
\end{pmatrix}
\end{equation*}
$

The contribution of each species on other species is
indicated by:
$
\begin{equation*}
N \dvdx =
\begin{pmatrix}
-k_1 & 0 \\
k_1 & -k_2 \\
\end{pmatrix}
\end{equation*}
$

The homogeneous differential equation for this system is
$
\dx =
A \bf{x}
$,
where $A = N \dvdx$.
And, a solution is $c \bf{x}_s e^{- \lambda}$,
where $\lambda$ is an eigenvalue, $\bf{x}_s$ is
its eigenvector, and $c$ is a constant.
Note that the eigenvalues are $-k_1, -k_2$
with the associated
eigenvectors
$
\begin{equation*}
\begin{pmatrix}
\frac{k_2}{k_1} - 1 \\
1
\end{pmatrix}
\end{equation*}
$,
$
\begin{equation*}
\begin{pmatrix}
0 \\
1
\end{pmatrix}
\end{equation*}
$.

That is, the complementary (homogeneous) solution is
$
\begin{equation*}
{\bf x}_c (t) =
c_1
\begin{pmatrix}
\frac{k_2}{k_1} - 1 \\
1
\end{pmatrix}
e^{-k_1 t}
+
c_2
\begin{pmatrix}
0 \\
1
\end{pmatrix}
e^{-k_2 t}
\end{equation*}
$, for $k_1 \neq k_2$.
Note that $c_i = x_i (0)$.

In matrix form,
let
$
\begin{equation*}
{\bf x}_1 (t) =
\begin{pmatrix}
\frac{k_2}{k_1} - 1 \\
0
\end{pmatrix}
e^{-k_1 t}
\end{equation*}
$,
$
\begin{equation*}
{\bf x}_2 (t) =
\begin{pmatrix}
0 \\
1
\end{pmatrix}
e^{-k_2 t}
\end{equation*}
$,
and
$
\begin{equation*}
{\bf X} =
\begin{pmatrix}
{\bf x}_1 & {\bf x}_2
\end{pmatrix}
\end{equation*}
$.

We also have the effect of parameters. In this case, the only parameter that's present is $X_0$.
$
\begin{equation*}
\dvdp =
\begin{pmatrix}
\frac{\partial v_1}{\partial X_0} \\
0 \\
0 \\
\end{pmatrix}
=
\begin{pmatrix}
k_0 \\
0 \\
0 \\
\end{pmatrix}
\end{equation*}
$
The full solution is
$
\begin{align*}
{\bf x}^{\prime} & = & {\bf A} {\bf x} + \dvdp {\bf p} \\
& = & {\bf A} {\bf x} +
\begin{pmatrix}
k_0 \\
0 \\
0 \\
\end{pmatrix} X_0
\end{align*}
$

As is standard practice, we solve for the particular solution.
$
\begin{align*}
{\bf x}_p =
{\bf X}_c {\bf u}(t)
\end{align*}
$,
where $u_i(t)$ is for state variable $i$
and
$
\begin{equation*}
{\bf u}(t) =
\begin{pmatrix}
u_1(t) \\
u_2(t) \\
\end{pmatrix}
\end{equation*}
$.
So,
$
\begin{align*}
{\bf x}_p^{\prime}
=
{\bf A} {\bf x}_p
+ {\bf r}(t)
\end{align*}
$,
where ${\bf r}(t)$ is dimensioned as the state vector.
Substituting,
$
\begin{align*}
\left(
  {\bf X}_c {\bf u}(t)
\right)^{\prime} & = &
A
  {\bf X}_c {\bf u}(t)
+ {\bf r}(t) \\
{\bf X}^{\prime}_c {\bf u}(t) + {\bf X}_c {\bf u}^{\prime}(t)
 & = & A {\bf X}_c {\bf u}(t) + {\bf r} (t) \\
{\bf X}_c {\bf u}^{\prime} (t) & = &  {\bf r} (t) \\
{\bf u}^{\prime} (t) & = & \left( {\bf X}_c \right)^{-1} {\bf r}(t) \\
{\bf u}(t) & = & \int \left( {\bf X}_c \right)^{-1} {\bf r}(t) \\
\end{align*}
$

$
\begin{equation*}
{\bf X}_c =
\left[
\begin{pmatrix}
\frac{k_2}{k_1} - 1 \\
1
\end{pmatrix}
e^{-k_1 t},
\begin{pmatrix}
0 \\
1
\end{pmatrix}
e^{-k_2 t}
\right]
\end{equation*}
$, and so
$
\begin{equation*}
{\bf X}_c^{-1} =
\left[
\begin{pmatrix}
\frac{k_1}{k_2 - k_1}  \\
1
\end{pmatrix}
e^{k_1 t},
\begin{pmatrix}
0 \\
1
\end{pmatrix}
e^{k_2 t}
\right]
\end{equation*}
$

$
\begin{equation*}
{\bf u}(t) =
\int {\bf X}_c^{-1} {\bf r}(t) =
  X_0 \frac{1}{k_1}
\begin{pmatrix}
\frac{k_1}{k_2 - k_1}  \\
1
\end{pmatrix}
e^{k_1 t}
\end{equation*}
$

$
\begin{align*}
{\bf x}_p & =  & {\bf X}_c {\bf u}(t) \\
& = &
\frac{X_0}{k_1}
\begin{pmatrix}
\frac{k_2 - k_1}{k_1}e^{-k_1 t} & 0 \\
e^{-k_1 t} & e^{-k_2 t} \\
\end{pmatrix}
\begin{pmatrix}
\frac{k_1}{k_2 - k_1}  \\
1
\end{pmatrix}
e^{k_1 t}
\end{align*}
$

```
k1, k2, mat = sympy.symbols(["k1", "k2", "mat"])
mat = sympy.Matrix([ [-k1, 0], [k1, -k2]])
mat.eigenvals()
```

```
mat.eigenvects()
```

$\displaystyle \left[ \left( - k_{1}, \  1, \  \left[ \left[\begin{matrix}-1 + \frac{k_{2}}{k_{1}}\\1\end{matrix}\right]\right]\right), \  \left( - k_{2}, \  1, \  \left[ \left[\begin{matrix}0\\1\end{matrix}\right]\right]\right)\right]$

**Find the inhomogeneous solution.**

```
model = """
J0: $X0 -> x1; k0*X0
J1: x1 -> x2; k1*x1
J2: x2 -> $X1; k2*x2

X0 = 1
k0 = 1
k1 = 1
k2 = 2
X1 = 2
"""

rr = te.loada(model)
rr.plot(rr.simulate())
```

# Notes

1. Tellurium can conduct experiments with species fixed by indicating fixed species in the model. Below is an analysis to verify the relationship between changing enzymes and concentrations.

```
import tellurium as te
model = """
R1: $S1 -> S2; e1*k1_f*S1 - k1_r*S2
R2: S2 -> $S3; e2*k2_f*S2 - k2_r*S3

S1 = 10
S2 =0
S3 = 10
k1_f = 1
k1_r = 0.5
k2_f = 2
k2_r = 1
e1 = 1
e2 = 1
"""

rr = te.loada(model)
rr.plot(rr.simulate())
```

```
e1Factor = -rr.getEE("R1", "S2")/rr.getEE("R1", "e1")
print("e1Factor: %f" % e1Factor)
#
e2Factor = -rr.getEE("R2", "S2")/rr.getEE("R2", "e2")
print("e2Factor: %f" % e2Factor)

baselineS2 = rr.getSteadyStateValues()[0]
```

```
fracChange = 0.1
newE1 = e1Factor*fracChange
newE2 = e2Factor*fracChange
rr["e1"] = newE1
rr["e2"] = newE2
rr.reset()
rr.plot(rr.simulate(0, 20, 200))
```

save/Solving Inhomogeneous LTI Differential Equations.ipynb


# Assignment 1

**Submission deadline:**
* **Problems 1-4: last lab session before or on Wednesday, 18.10.17**
* **Problems 5-6: last lab session before or on Wednesday, 25.10.17**

**Points: 10 + 4 bonus points**

Please note: some of the assignments are tedious or boring if you are already a NumPy ninja. The bonus problems were designed to give you a more satisfying alternative.

## Downloading this notebook

This assignment is an Jupyter notebook. Download it by cloning https://github.com/janchorowski/nn_assignments. Follow the instructions in its README for instructions. Whenever possible, add your solutions to the notebook.

Please email us about any problems with it - we will try to correct them quickly. Also, please do not hesitate to use **GitHub’s pull requests** to send us corrections!

```python
# Standard IPython notebook imports
%matplotlib inline

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os

import numpy as np
import matplotlib.pyplot as plt

from sklearn import datasets
```

## Problem 0 [0p]
1. Get familiar with [lab_guide.pdf](https://github.com/janchorowski/nn_assignments/blob/nn17_fall/lab_guide.pdf). It is not obligatory to use the labs remotely and most of the time you will do just fine with your own machine. However, we recommend using labs for the most computationally-intensive tasks such as the final projects.

    Make sure you know how to:
    * load the Anaconda Python distribution with `/pio/os/anaconda/set-env.sh` (confirm with `python -V`)

    (also: look into this `nn_assignments`' `set-env.sh` and see that it loads `Anaconda` for you),
    * connect to labs 110/137 from *outside* of the faculty,
    * copy a file over an ssh tunnel or connect remotely to a Jupyter Notebook,
    * use `GNU screen`, `tmux` and/or `nohup`,
    * leave a process running, disconnect, connect again and verify it is still running.

  In case of problems, ask for help!

2. Read [Jupyter tutorial from Data Analysis in Biological Sciences course at Caltech](http://bebi103.caltech.edu/2015/tutorials/t0b_intro_to_jupyter_notebooks.html) (which itself can be downloaded as a Jupyter notebook). Feel free to skip the tutorial if you have some prior experience with Jupyter notebook.

## Problem 1 [2p]

First, get familiar with Python at https://docs.python.org/2/tutorial/. Then, get
to know the capabilities of NumPy, the prime numerical library of Python http://www.numpy.org/, for instance with the tutorial at http://wiki.scipy.org/Tentative_NumPy_Tutorial.

You might also need:
  1. another intro to NumPy,
http://people.duke.edu/~ccc14/pcfb/numerics.html
  2. a better interactive shell for Python,
http://ipython.org/
  3. access to IPython through an ordinary web browser,
http://ipython.org/notebook.html
  4. a plotting library for Python.
http://matplotlib.org/

**a) Declare variables:**
1. $a=10$,
2. $b=2.5\times 10^{23}$,
3. $c=2+3i$, where $i$ is an imaginary unit,
4. $d=e^{i2\pi/3}$, where $i$ is an imaginary unit, $e$ is the Euler's number (use `exp`, `pi`).

```python
# TODO: Complete the declarations

import cmath
a = 10
b = 2.5*pow(10.0,23.0)
c = complex(2,3)
d = cmath.exp(cmath.pi*2j/3)
```

**b) Declare vectors:**
1. $aVec=\begin{bmatrix} 3.14 & 15 & 9 & 26 \end{bmatrix}$,
2. $bVec=\begin{bmatrix} 2.71 & 8 & 28 & 182 \end{bmatrix}^\intercal$ (column vector),
3. $cVec=\begin{bmatrix} 5 & 4.8 & \cdots & -4.8 & -5 \end{bmatrix}$ (vector of numbers from $5$ to $-5$ decreasing by $0.2$),
4. $dVec=\begin{bmatrix} 10^0 & 10^{0.01} & \cdots & 10^{0.99} & 10^1 \end{bmatrix}$ (logarithmically spaced numbers from 1 to 10, use `logspace` and make sure, that the result has correct length!),
5. $eVec=Hello$ ($eVec$ is a string of characters, thus a vector).

```python
aVec = np.asmatrix((3.14,15,9,26))
bVec = np.transpose(np.array([[2.71,8,28,182]]))
cVec = np.array([x/5 for x in range(25,-26,-1)])
dVec = np.array([10**(x/100) for x in range(0,101)])
eVec = np.array(list('Hello'))

```

**c) Declare matrices:**
1. $aMat=\begin{bmatrix}
                    2      & \cdots & 2 \\
                    \vdots & \ddots & \vdots \\
                    2      & \cdots & 2
                \end{bmatrix}$,

matrix $9\times 9$ filled with 2s (use `ones` or `zeros`),
2. $bMat=\begin{bmatrix}
                    1      & 0      & \cdots &        & 0      \\
                    0      & \ddots & 0      &        & 0      \\
                    \vdots & 0      & 5      & 0      & \vdots \\
                           &        & 0      & \ddots & 0      \\
                    0      &        & \cdots & 0      & 1
                \end{bmatrix}$,

matrix $9\times 9$ filled with zeros, with $\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \end{bmatrix}$ on its diagonal (use `zeros`, `diag`),
3. $cMat=\begin{bmatrix}
                    1      & 11     & \cdots & 91     \\
                    2      & 12     & \ddots & 92     \\
                    \vdots & \vdots & \ddots & \vdots \\
                    10     & 20     & \cdots & 100
                \end{bmatrix}$,

matrix $10\times 10$, columns of which form the vector $1:100$ (use `reshape`),
4. $dMat=\begin{bmatrix}
                    NaN & NaN & NaN & NaN \\
                    NaN & NaN & NaN & NaN \\
                    NaN & NaN & NaN & NaN
                \end{bmatrix}$,

matrix $3\times 4$ filled with `NaN`s (use... `NaN`),
5. $eMat=\begin{bmatrix}
                    13  & -1  & 5  \\
                    -22 & 10  & -87
                \end{bmatrix}$,

6. $fMat$ filled with random natural numbers from $[-3,3]$ (use `rand` and `floor` or `ceil`).

```python
import random
aMat = np.ones((9,9))*2
bMat = np.diag(range(1,6)+range(4,0,-1))
cMat = np.arange(1,101).reshape(10, 10, order='F')
dMat = np.ones((3,4))*np.NaN
eMat = np.array([[13,-1,5],[-22,10,-87]])
fMat = np.array([random.randint(-3,3) for x in range(9)]).reshape(3,3)
```

** d) Declare a multiplication table ** as a $10\times 10$ matrix `mulMat`. Use matrix/vector multiplication.

```python
mulMat = (np.arange(1,11)).reshape(10,1)*(np.arange(1,11))
mulMat
```

    array([[  1,   2,   3,   4,   5,   6,   7,   8,   9,  10],
           [  2,   4,   6,   8,  10,  12,  14,  16,  18,  20],
           [  3,   6,   9,  12,  15,  18,  21,  24,  27,  30],
           [  4,   8,  12,  16,  20,  24,  28,  32,  36,  40],
           [  5,  10,  15,  20,  25,  30,  35,  40,  45,  50],
           [  6,  12,  18,  24,  30,  36,  42,  48,  54,  60],
           [  7,  14,  21,  28,  35,  42,  49,  56,  63,  70],
           [  8,  16,  24,  32,  40,  48,  56,  64,  72,  80],
           [  9,  18,  27,  36,  45,  54,  63,  72,  81,  90],
           [ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100]])

** e) Compute elemwise using values from b).**
For instance, the first element of $xVec[0]$ should be equal to

\begin{equation}
1/(\sqrt{2\pi2.5^2}) e^{-cVec[0]^2 / (2\cdot\pi 2.5^2)}.
\end{equation}

1. $xVec=1/(\sqrt{2\pi2.5^2}) e^{-cVec^2 / (2\cdot\pi 2.5^2)}$
2. $yVec=\sqrt{(aVec^\intercal)^2 + bVec^2}$
3. $zVec=\log_{10}(1/dVec)$, using `log10`

```python
import math
xVec = np.array([1/(2*np.pi*2.5**2)**0.5*np.exp(-x**2/(2*np.pi*2.5**2)) for x in cVec])
yVec = np.array([(aVec.item(i)**2+bVec.item(i)**2)**0.5 for i in range(4)])
zVec = np.array([math.log(1/x,10) for x in dVec])
```

** f) Compute with matrix/vector operations using values from c).**

**NOTE:** Every multiplication (and power) in this subtask is a [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication).
1. $xMat=(aVec\cdot bVec)aMat^2$,
2. $yMat=bVec\cdot aVec$

(remember, that matrix multiplication is not commutative),
4. $zMat=\lvert cMat\rvert (aMat\cdot bMat)^\intercal$, where $\lvert A\rvert$ denotes determinant of $A$ (use `det`).

```python
xMat = (aVec.dot(bVec).item(0))*(aMat).dot(aMat)
yMat = bVec.dot(aVec)
zMat = np.linalg.det(cMat)*(aMat.dot(bMat[:9][:9])).T
```

** g) Declare `ismagic(A)` function ** which checks if matrix $A$ is a [magic square](https://en.wikipedia.org/wiki/Magic_square) and returns a boolean.

```python
magicSquare=np.array([[2,7,6],[9,5,1],[4,3,8]])
B = np.array([[1,2],[1,4]])
def ismagic(A):
    if(len(A)==0):
        return False
    if(np.unique(A).size!=A.size):
        return False

    d1_sum=np.trace(A)
    d2_sum=np.trace(A[:,::-1])

    if (d2_sum!=d1_sum):
        return False
    c_sum = np.sum(A,axis=0)

    if(np.unique(c_sum).size!=1 or c_sum.item(0)!=d1_sum):
        return False

    r_sum = np.sum(A,axis=1)
    if(np.unique(r_sum).size!=1 or r_sum.item(0)!=d1_sum):
        return False

    return True

print(ismagic(magicSquare))
print(ismagic(B))
```

    True
    False

# k-Nearest Neighbors

The following excerpt of code loads the data describing iris flowers
and shows relations between their length and petal width for three
species (namely: setosa, versicolor, virginica).

```python
# pylab - loads numerical and graphical libraries into the IPython notebook
%pylab inline

# sklearn is a large collection of machine learning algorithms
# here we’ll use it only for the built-in iris dataset
from sklearn import datasets

iris = datasets.load_iris()

print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)

petal_length = iris.data[:, iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]

for target in set(iris.target):
    example_ids = target == iris.target
    scatter(petal_length[example_ids], petal_width[example_ids],
            label=iris.target_names[target], color='bgr'[target],
            marker='x', alpha=0.7)
unknown = np.array([
    [1.5, 0.3],
    [4.5, 1.2],
    [5.5, 2.3],
    [5.1, 1.7]
])
scatter(unknown[:, 0], unknown[:, 1], marker='v',
        color='gray', s=50, label='??')
xlabel('petal length (cm)')
ylabel('petal width (cm)')
grid(True)
legend(loc='upper left');
```

Based on these two features, it is easy to distinguish iris setosa from the two remaining species. Yet iris versicolor and virginica remain mixed together.

Looking closely at the plot, we might estimate the species of the selected unknown irises (gray triangles). For three of them the answer seems obvious – they belong in uniformly-colored areas covered by one species only. Yet unknown iris flower in (5.1, 1.7) is troublesome – it lays on the boundary of versicolor and virginica clusters. We can assume, that its species is the one of the closest one to it, coming from the training set (and so having a label).

K-Nearest Neighbors method (http://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) solves the classification problem, i.e. sets class labels (species in case of irises) of a previously unseen sample by choosing the most common class among the top k neighbors of the sample in question (for instance according to the Euclidean distance). Thus, the k-Nearest Neighbors algorithm works as follows. For each unlabeled sample x:
1. Find k nearest neighbors among the labeled samples.
2. Set the most common label among them as label of x.

## Problem 2 [2p]

1.  **[1p]** Load the iris data (in Python it’s built-in into machine learning libraries, use sklearn.datasets.load_iris), the data is also available on-line at https://archive.ics.uci.edu/ml/datasets/Iris

2.  **[1p]** Irises are described with 4 attributes: petal and sepal widths and lengths. We often plot such data as matrices depicting relationships between pairs of attributes (the diagonal of which holds an ordinary histogram). Write code making a plot like the one below. Please pay attention to the details: make a proper legend and correctly label the axes.

```python
from sklearn.datasets import load_iris
iris = load_iris()

data = [iris.data[:, iris.feature_names.index('sepal length (cm)')],
        iris.data[:, iris.feature_names.index('sepal width (cm)')],
        iris.data[:, iris.feature_names.index('petal length (cm)')],
        iris.data[:, iris.feature_names.index('petal width (cm)')]]

f, axarr = plt.subplots(4,4)
for y in range(0,4):
    for x in range(0,4):
        if(x!=y):
            for target in set(iris.target):
                example_ids = target == iris.target
                axarr[x][y].scatter(data[y][example_ids],
                                    data[x][example_ids],
                                    label=iris.target_names[target],
                                    color='bgr'[target],
                                    marker='o',
                                    alpha=0.7)
                axarr[x,y].yaxis.set_major_locator(plt.MaxNLocator(10))
                axarr[x,y].xaxis.set_major_locator(plt.MaxNLocator(10))

for x in range(0,4):
    axarr[x][x].hist(data[x], 10, normed=None, histtype='bar', facecolor='blue', alpha=0.75, ec='black')

axarr[3][0].set_xlabel('sepal length (cm)')
axarr[3][1].set_xlabel('sepal width (cm)')
axarr[3][2].set_xlabel('petal length (cm)')
axarr[3][3].set_xlabel('petal width (cm)')

axarr[0][0].set_ylabel('sepal length (cm)')
axarr[1][0].set_ylabel('sepal width (cm)')
axarr[2][0].set_ylabel('petal length (cm)')
axarr[3][0].set_ylabel('petal width (cm)')

#ax1.ylabel('petal width (cm)')
#axarr[3][2].grid(True)
#f.legend(loc='upper left');
#plt.legend(loc="upper left", title="Legend", fancybox=True)

# Get current size
fig_size = plt.rcParams["figure.figsize"]

# Set figure width to 12 and height to 9
fig_size[0] = 12
fig_size[1] = 9
plt.rcParams["figure.figsize"] = fig_size

handles, labels = axarr[1,0].get_legend_handles_labels()
f.legend(handles, labels, loc='upper center', ncol=3, fontsize='large', scatterpoints=3)

```

## Problem 3 [2p]

Implement the k-Nearest Neighbors algorithm. Try to
take advantage of matrix calculus rather than using for loops.

**Tip:** What is computed by \begin{equation} \sqrt{(X - Y)^T (X - Y)} \end{equation} when both X and Y are vectors?

**Tip:** Try to use broadcasting (NumPy: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) and built-ins sort, numpy.sort, numpy.argsort (sorting), scipy.stats.mode (choosing the most common element of the set).

```python
from scipy.stats import mode
def kNN (Data, Labels, X, k):
    distans = np.sqrt(-2*Data.dot(X.T) + (Data**2).sum(axis=1).reshape(-1,1) + (X**2).sum(axis=1).reshape(1,-1)  )
    inds = distans.argsort(axis=0)[:k]
    nearestLabels=Labels[inds]
    return mode(nearestLabels, axis=0)[0][0]
```

## Problem 4 [2p]
Consider the following experiment:
1. We scramble the data and split it into two parts - training set (66.6% of all samples) and test set (33.4%).
2. Based on the training set, we use the k-NN algorithm to predict the labels on the test set.
3. We then check the number of errors and write it down.

Do this 500 times for k ∈ {1, 3, 5, ..., 19}. Plot a function of the average number of errors
as the function of k. It should be similar to the one below.

## Problem 5 [2p]

Apply the K-Nearest Neighbors (K-NN) algorithm to the MNIST and CIFAR10 datasets.

The MNIST (http://yann.lecun.com/exdb/mnist/) dataset consists of normalized (centered and stretched) scans of hand-written digits. Specifically, each element of the dataset is a 28 × 28 grayscale image, thus having 764 8-bit pixels.

The CIFAR10 (http://www.cs.toronto.edu/~kriz/cifar.html) dataset consists of small, 32 by 32 pixels, RGB images belonging to 10 categories.

1. **[1p]** Download and load the MNIST and CIFAR10 datasets. For both datasets, display a few objects from each of the classes, paying attention to aesthetics and clarity of your presentation. **Note:** the datasets are available on Lab computers. Please use the code below to get started.

2. **[2p]** Apply a k-NN classifier to the MNIST and CIFAR10 datasets. First, divide the training set into two parts, which we will call training and validation. On MNIST use the first 50000 samples for training and the last 10000 for validation. On CIFAR10, use 40000 to train and 10000 for validation. Then find the optimal number of neighbors by assessing the accuracy on the validation set. You do not need to repeat this experiment multiple times. Finally, compute the accuracy on the test set obtained with the best previously chosen number of neighbors. On MNIST you should get about 3% errors, while on CIFAR10 you should get about 70% errors. Why CIFAR10 is harder than MNIST? Pick a few mislabeled samples from the test dataset and plot them along with the correct ones. **Note:**
  * MNIST and CIFAR10 are much larger than the Iris dataset. A good implementation may need a few minutes on Lab computers. Please optimize your algorithm:
  * Compute the distances only once, then test for different values of k.
  * Use vectorized expressions to compute the distance. It is possible to compute all distances between the training and testing points in one expression. Hint: think about the vectorized expression \begin{equation}(X - Y)^T (X - Y)\end{equation}.
  * You can use single precision numbers in computation.
  * If your code is taking a long time to execute, please save its results before the lab session.

**Note:** in NumPy, matrices have its own data type (dtype), which is retained during
calculations. Please pay attention to it. I particular, do not subtract values of data types not
having the sign bit, do not divide integers, etc. Results of such operations will not be
automatically casted to types having the required precision.

```python
try:
    import torchvision

    data_path = os.environ.get('PYTORCH_DATA_PATH', '../data')

    mnist_full_train_dataset = torchvision.datasets.MNIST(
        data_path, train=True, download=True)
    mnist_full_train_data_uint8 = mnist_full_train_dataset.train_data.numpy()
    mnist_full_train_labels_int64 = mnist_full_train_dataset.train_labels.numpy()

    mnist_test_dataset = torchvision.datasets.MNIST(
        data_path, train=False, download=True)
    mnist_test_data_uint8 = mnist_test_dataset.test_data.numpy()
    mnist_test_labels_int64 = mnist_test_dataset.test_labels.numpy()

except ImportError:
    # Load the same data without torchvision (e.g., on a personal machine)
    with np.load('/pio/data/data/nn17/mnist.npz') as data:
        mnist_full_train_data_uint8 = data['train_data']
        mnist_full_train_labels_int64 = data['train_labels']
        mnist_test_data_uint8 = data['test_data']
        mnist_test_labels_int64 = data['test_labels']

# Split train data into train and validation sets
mnist_train_data_uint8 = mnist_full_train_data_uint8[:50000]
mnist_train_labels_int64 = mnist_full_train_labels_int64[:50000]
mnist_valid_data_uint8 = mnist_full_train_data_uint8[50000:]
mnist_valid_labels_int64 = mnist_full_train_labels_int64[50000:]
```

```python
from common.plotting import plot_mat

plot_mat(mnist_train_data_uint8[:20, None], cmap='gray')
```

```python
try:
    import torchvision
    #
    # TODO: load CIFAR10
    #
except ImportError:
    with np.load('/pio/data/data/nn17/cifar.npz') as data:
        cifar_full_train_data_uint8 = data['train_data']
        cifar_full_train_labels_int64 = data['train_labels']
        cifar_test_data_uint8 = data['test_data']
        cifar_test_labels_int64 = data['test_labels']
```

```python
#
# TODO: plot an example of each class on MNIST and on CIFAR-10
#
```

```python
from scipy.stats import mode

def KNN(train_X, train_Y, test_X, test_Y, ks, batch_size=200):
    """
    Compute error rate for various
    """
    errs = np.zeros((len(ks),))
    for i in xrange(0, test_Y.shape[0], batch_size):
        batch_X = test_X[i:i + batch_size]
        batch_Y = test_Y[i:i + batch_size]
        print("Examples %d:%d Computing distances... " %
              (i, i + batch_size), end='')

        #
        # TODO: fill in an efficient distance matrix computation
        #
        dists = TODO

        print("Sorting... ", end='')
        closest = np.argsort(dists, 0)

        print("Computing errors...")
        targets = train_Y[closest]

        for ki, k in enumerate(ks):
            predictions, unused_counts = mode(targets[:k, :], axis=0)
            predictions = predictions.ravel()
            #
            # TODO: fill in error count computation
            #
            errs[ki] += TODO

    errs /= test_Y.shape
    return np.vstack((ks, errs))
```

```python
# Now find the best k on the validation set

mnist_validation_errs = KNN(
    mnist_train_data_uint8, mnist_train_labels_int64,
    mnist_valid_data_uint8, mnist_valid_labels_int64,
    [1, 3, 5, 7, 9])

plot(mnist_validation_errs[0, :], mnist_validation_errs[1, :])
```

```python
# Now use the best k to compute the test error

best_K = TODO

mnist_full_train_data_uint8 = mnist_train_dataset.train_data.numpy()
mnist_full_train_labels_int64 = mnist_train_dataset.train_labels.numpy()

mnist_test_errs = KNN(mnist_full_train_data_uint8,
                      mnist_full_train_labels_int64,
                      mnist_test_data_uint8,
                      mnist_test_labels_int64, [best_K])
print("When k=%d the test error rate is %.1f%%" %
      (mnist_test_errs[0, 0], mnist_test_errs[1, 0] * 100.0))
```

```python
#
# Now repeat the k-NN training for CIFAR10
#
```

# Locality sensitive hashing

Problem 5 was about speeding up the inference using loops implicitly present in matrix multiplication instead of explicit loops in Python. In this problem, we will explore a strategy to truly reduce the total number of computations required to find nearest neighbors without sacrificing too much accuracy.

To speed up nearest neighbor search we will employ *Locality Sensitive Hashing (LSH)* functions. For a given distance metric, the locality sensitive hash should put items that are similar into the same bucket. Notice that this is essentially a design choice opposite to traditional cryptographic hash functions that should amplify the difference of similar inputs (typically we want that small perturbations of data result in large changes to the hash value).

One of the simplest implementations of LSH approximates the cosine distance. Let $x\in \mathbb{R}^N$ and $y\in \mathbb{R}^N$ be two vectors. Their cosine distance is defined as:

\begin{equation}
    d_\text{cos}(x,y) = \frac{x \cdot y}{\|x\| \|y\|} = \cos\left(\theta(x,y)\right),
\end{equation}
where $\theta(x,y)$ is the unsigned angle between $x$ and $y$.

We will construct a family $H$ of hash functions that are an LSH for angle distances (an approximation to cosine distance). Assume $p\in \mathbb{R}^N$ is a random vector (components are sampled from the normal distribution) of length 1. Then define the hash function $h(x) = \text{sgn}(x\cdot p)$, where $\text{sgn()}$ is the sign function. It can be proven that:

\begin{equation}
    p_{h\in H}[h(x)=h(y)] = 1 - \frac{\theta(x,y)}{\pi}.
\end{equation}

The equation means that the probability of a hash collision grows as the the angle between two vectors gets smaller. Therefore, vectors that are close according to the cosine distance will be put with high probability into the same bin (we use the fact that for small $\theta$ we can approximate $\cos(\theta) = 1 - \theta/\pi$.

We will say that a family of randomly chosen hash functions $H$ is $(d_1, d_2, p_1, p_2)$-sensitive with respect to a distance metric $d$ if for any $x$ and $y$:
1. If $d(x,y) \leq d_1$ then $p_{h\in H}[h(x)=h(y)] \geq p_1$.
2. If $d(x,y) \geq d_2$ then $p_{h\in H}[h(x)=h(y)] \leq p_2$.

For example, our family of randomly chosen hyperplanes is $(d_1, d_2, (1-d_1)/\pi, (1-d_2)/\pi)$-sensitive.

Ideally, vectors should be placed into the same bin with a high probability if their distance is smaller than a threshold, and with a low probability if their distance is larger that the threshold. By combining hashing functions we can get closer to this ideal sensitivity.

Given a family of hash functions $H$ with sensitivity $(d_1, d_3, p_1, p_2)$ we can construct a new family $H'$ by combining $r$ functions from $H$:
1. AND: let $h=[h_1, h_2, \ldots, h_r] \in H'$ and $h(x)=h(y)$ if and only if $\forall_i h_i(x)=h_i(y)$. Then $H'$ is $(d_1, d_2, (p_1)^r, (p_2)^r)$-sensitive.
2. OR: let $h=[h_1, h_2, \ldots, h_r] \in H'$ and $h(x)=h(y)$ if and only if $\exists_i h_i(x)=h_i(y)$. Then $H'$ is $(d_1, d_2, 1-(1-p_1)^r, 1-(1-p_2)^r)$-sensitive.

AND makes all probabilities shrink, but properly choosing $r$ we can make the lower probability approach 0 while the higher does not. Conversely, OR makes all probabilities grow, we can make the upper probability approach 1 while the lower does not.

## Problem 6 [2-4p bonus]

1. **[1bp]** **Note:** you can show sketches of proofs for this assignment.
    1. Show that angle between vectors is a metric (https://en.wikipedia.org/wiki/Metric_(mathematics)).

    2. Show that $p_{h\in H}[h(x)=h(y)] = 1 - \frac{\theta(x,y)}{\pi}$ for $h$ computed using a randomly chosen hyperplane.

    3. Show the properties of either AND or OR boosting of LSH.

3. **[1-3bp]** Reimplement k-Nearest Neighbors for MNIST classification using the cosine distance instead of the Euclidean distance. Choose a sensible value of $k$. Use Locality Sensitive Hashing to achieve an error rate no greater than $150\%$ of the original error rate with at least a $90\%$ speedup (i.e., by considering on average at most 5000 training samples per query image). For a few settings plot the speedup-vs-accuracy relation.

  **Note:** points will be awarded based on ingenuity of your solution. Feel free to explore your own ideas!

```python

```