花一天时间把 LaTex 看了
照着下面敲一遍就差不多了
+ [Markdown中的LaTeX格式](https://www.jianshu.com/p/8c46e915c45e)

特殊符号基本都是相应的英文简写
+ [常用数学标记与符号的英文表达](https://zhuanlan.zhihu.com/p/165310516)

# 一、Markdown排版格式

## 1）行内公式排版

$ c = \sqrt{a^{2}+b_{xy}^{2}+e^{x}} $

## 2）块公式排版

$$ c = \sqrt{a^{2}+b_{xy}^{2}+e^{x}} $$

# 二、LaTex的公式规则

## 常用规则

### 1）转义

>以下几个字符：# $ % & ~ _ ^ \ {} 有特殊意义，需要表示这些字符时，需要转义，即在每个字符前加上\
\boxed 命令给公式加一个方框

$ E = mc^{2} $
$\boxed E = mc^{2} $

### 2）希腊字母

### 3）上下标和根号

>用^来表示上标
用_来表示下标
根号用\sqrt来表示

注意：上下标如果多余一个字符或符号，需要用{}括起来

$ \sum_{i=1}^n a_i $

\sqrt[开方次数，默认为2]{开方公式}，例如：

$$ x_{ij}^{2}\quad \sqrt{x}\quad \sqrt[3]{x} $$

其中 \quad 表示添加空格

### 4）分数

> 分数用\frac表示

> 字号工具环境设置：
\dfrac命令把字号设置为独立公式中的大小
>tfrac命令把字号设置为行间公式中的大小

$ \frac{1}{2}\quad \dfrac{1}{2}\quad \tfrac{1}{2}$

$$ \frac{1}{2}\quad \dfrac{1}{2}\quad \tfrac{1}{2} $$

### 5）运算符

> \+ - * / = 直接输入

>特殊运算则用以下特殊命令 \pm \times \div \cdot \cap \cup \geq \leq \neq \approx \equiv

$$ \pm\quad \times\quad \div\quad \cdot\quad \cap\quad \cup\quad \geq\quad \leq\quad \neq\quad \approx\quad \equiv
$$

>和、积、极限、积分等运算符用 \sum \prod \lim \int 表示。这些公式在行内公式被压缩，以适应行高，可以通过\limits 和 \nolimits 命令显示是否压缩

$ \sum \prod \lim \int $

$
\sum\limits_{i=1}^{n} i\quad
\prod\limits_{i=1}^{n}\quad
\lim\limits_{x\to0} x^{2}\quad
\int\limits_{a}^{b} x^{2}dx\quad
\sum_{i=1}^{n} i\quad
\prod_{i=1}^{n}\quad
\lim_{x\to0}x^{2}\quad
\int_{a}^{b} x^{2}dx\quad
$

### 6）多重积分

> 使用如下形式: \int \iint \iiint \idotsint

$$
\int\quad
\iint\quad
\iiint\quad
\idotsint\quad
$$

$
\leftarrow\quad
\Leftarrow\quad
\rightarrow\quad
\Rightarrow\quad
\to\quad
\leftrightarrow\quad
\Leftrightarrow\quad
\longleftrightarrow\quad
\Longleftrightarrow\quad
\longleftarrow\quad
\Longleftarrow\quad
$

### 7）注音和标注

$ \bar{x} $

$ \acute{x} $

$ \mathring{x}$

$ \vec{x}$

$\grave{x}$

$ \dot{x} $

$ \tilde{x} $

$ \ddot{x} $

$ \check{x} $

$ \breve{x} $

$ \dddot{x} $

### 8）分割符

>括号用 () [] {} \langle \rangle 表示 () [] {} ⟨⟩

$ \langle \rangle $

$ \overline{xxx} $

$ \overleftrightarrow{xxx} $

$ \underline{xxx} $

$ \underleftrightarrow{xxx} $

$ \overleftarrow{xxx} $

$ \overbrace{xxx} $

$ \underleftarrow{xxx} $

$ \underbrace{xxx} $

$ \widehat{xxx} $

$ \underrightarrow{xxx} $

$ \widetilde{xxx} $

$$
\Bigg(\quad \bigg(\quad \Big(\quad \big((x) \big)\quad \Big)\quad \bigg)\quad \Bigg)\quad
\Bigg[\quad \bigg[\quad \Big[\quad \big[[x] \big]\quad \Big]\quad \bigg]\quad \Bigg]\quad
\Bigg\{\quad \bigg\{\quad \Big\{\quad \big\{\{x\} \big\}\quad \Big\}\quad \bigg\}\quad \Bigg\}\quad
$$

### 9）省略号

>省略号用 \dots \cdots \vdots \ddots表示，\dots和\cdots的纵向位置不同，前者一般用于有下标的序列

$$ x_1,x_2,\dots,x_n\quad 1,2,\cdots,n\quad \vdots\quad \ddots$$

### 10）空白间距

$$
a\quad b
a\qquad b
a\: b
a\; b
a\, b
ab
a \! b
$$

## 复杂公式

### 1）矩阵

$$
\begin{array}{ccc}
x_1 & x_2 & \dots \\
x_3 & x_4 & \dots \\
\vdots & \vdots & \ddots
\end{array}
$$

$$
\begin{pmatrix}
a & b \\
c & d \\
\end{pmatrix}
\quad
\begin{bmatrix}
a & b \\
c & d \\
\end{bmatrix}
\begin{Bmatrix}
a & b \\
c & d \\
\end{Bmatrix}
\begin{vmatrix}
a & b \\
c & d \\
\end{vmatrix}
\begin{Vmatrix}
a & b \\
c & d \\
\end{Vmatrix}
$$

$$
(
\begin{smallmatrix}
a & b \\
c & d
\end{smallmatrix}
)
$$

### 2）长公式

>无需对齐可使用multline;
需要对齐使用split;
用\\来分行
用&设置对齐位置

$$
\begin{multline}
x = a+b+c+\\
d+e+f+g
\end{multline}
$$

$$
\begin{split}
x = &a + b + c&+\\
&d+e+f&+g
\end{split}
$$

### 3）公式组

$
\begin{gather}
a=b+c+d\\
x=y+z\\
5=4+1
\end{gather}
$

$
\begin{align}
a&=b+c&+d \\
x&=y&+z \\
5&=4&+1
\end{align}
$

### 4）分支公式

>分段函数通常用cases次环境写成分支公式

$
y=\begin{cases}
-x,\quad x\leq 0 \\
x,\quad x>0
\end{cases}
$


+ This notebook is part of lecture 29 *Singular value decomposition* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols, sqrt, Rational
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

+ This chapter starts with explanations using sympy
+ A proper method using numpy and sympy is described in the example problem at the end

# Singular value decomposition (SVD)

## Derivation

* This is the final form of matrix factorization
* The factors are an orthogonal matrix A, a diagonal matrix Σ, and an orthogonal matrix V
$$ {A}={U}{\Sigma}{V}^{T} $$

+ In case the matrix A is symmetric positive definite, the decomposition is akin to the following
$$  {A}={Q}{\Lambda}{Q}^{T} $$

* Consider a vector *v*1 in ℝn row space, transformed into a vector *u*1 in ℝm column space by the matrix A
$$ {u}_{1}={A}{v}_{1} $$

* What we are looking for is an orthogonal basis in ℝn row space, transformed into an orthogonal basis in ℝm column space
$$ {u}_{1}={A}{v}_{1} \\ { v }_{ 1 }\bot { v }_{ 2 };{ u }_{ 1 }\bot { u }_{ 2 }$$

* It's easy to calculate an orthogonal basis in the row space using Gram-Schmidt
* Now, though, we need something special in A that would ensure that the basis *u*i in in ℝm column space is also orthogonal (and at the same time make it orthonormal, so that *v*i ends up as σi*u*i)
* The two nullspaces are not required
* So, we are looking for the following
$$ A\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ { v }_{ 1 } & { v }_{ 2 } & \cdots  & { v }_{ r } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}=\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ { u }_{ 1 } & u_{ 2 } & \cdots  & { u }_{ r } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}\begin{bmatrix} { \sigma  }_{ 1 } & \quad  & \quad  & \quad  & \quad  \\ \quad  & { \sigma  }_{ 2 } & \quad  & \quad  & \quad  \\ \quad  & \quad  & \quad \ddots  & \quad  & \quad  \\ \quad  & \quad  & \quad  & { \sigma  }_{ r }\quad  & \quad  \\ \quad  & \quad  & \quad  & \quad  & \left( 0 \right)  \end{bmatrix} \\ {A}{V}={U}{\Sigma}$$

+ In the case that we are not changing spaces V and U would be the same matrix Q (and then Q-1)

### Example problem explaining the derivation

+ Look at the next matrix A that is square and invertible (i.e. rank 2)

```python
A = Matrix([[4, 4], [-3, 3]])
A
```

* We are looking for *v*1 and *v*2 in the ℝ2 rowspace and *u*1 and *u*2 in the ℝ2 columnspace, as well as the scaling factors *σ*1>0 and *σ*2>0

+ Just to be complete, we extend V until *v*n with zero columns and U with zero columns until *u*m, as well as zeros for Σ to include the nullspaces

+ Now A is not symmetric so that their eigenvectors are not orthogonal (Q), so we can't go that route

+ From above we have the following and because V is square and orthogonal we have
$$ {A}={U}{\Sigma}{V}^{-1} \\ {A}={U}{\Sigma}{V}^{T} $$

+ Multiplying both sides by AT we will have a left-hand side that is square and definte (semi)definte
$$ A=U\Sigma { V }^{ T }\\ { A }^{ T }A=V{ \Sigma  }^{ T }{ U }^{ T }U\Sigma { V }^{ T }\\ \because \quad { U }^{ T }U=I\\ \because \quad { \Sigma  }^{ T }\Sigma =\dots { \sigma  }_{ i }^{ 2 }\dots \\ { A }^{ T }A=V{ \Sigma  }^{ T }\Sigma { V }^{ T } $$

+ Because ATA is now definite (semi)positive, we have a perfect situation akin to being able to use QΛQT
+ The eigenvalues are the squares of the σi values
+ To get U we use AAT and use its eigenvalues and eigenvectors

* All of this is easy to accomplish with the mpmath submodule of sympy

```python
from sympy.mpmath import svd
```

```python
U, S, V = svd(A)
```

```python
U # The numbers round to zero!!! Please see it as zero
```

```python
S # Not the final Sigma matrix
```

```python
V
```

+ There are square roots, so the values are given instead of symbols

* Now let's do it step-by-step

```python
A.transpose() * A
```

```python
(A.transpose() * A).eigenvals()
```

```python
(A.transpose() * A).eigenvects()
```

* These are not normalized, though
* Also remember to take the square roots of the eigenvalues
* ... and to add zeros to incorporate the correct size for *m* and *n*
* ... and to take the transpose

* Now let's tackle U

```python
A * A.transpose()
```

```python
(A * A.transpose()).eigenvals()
```

* The eigenvalues are always the same

```python
(A * A.transpose()).eigenvects()
```

* Also remember to normalize (see example problem below)

* We now have U, Σ and VT (although Σ must still be constructed; see below)

### Example problem to explain the derivation for dependent rows, columns

+ Let's consider this rank=1, 2×2 singular matrix
+ The rowspace is just a line (the second row is a constant multiple of the first)
+ The nullspace of this row picture is a line perpendicular to this
+ The columnspace is also on a line, with the nullspace of AT being a line perpendicular to this

```python
A = Matrix([[4, 3], [8, 6]])
A
```

+ Let's use *svd*() first

```python
U, S, V = svd(A, full_matrices = True, compute_uv = True)
```

* This is likely to be different to the value you calculate for U
* We are talking unit basis vectors, though, which can be in a different direction depending on your choice

```python
U
```

```python
S
```

+ Note that the size of our Σ matrix is wrong
+ It has to be 2×2 and we have to create it from this info
+ Since A has rank = 1 and all off-diagonal entries must be zero, we will only have a value in the first row, first column position
+ Below I show you how to correct this

```python
V
```

```python
A.transpose() * A # Which will be symmetric positive definite and of rank = 1
```

```python
(A.transpose() * A).eigenvals() # One eigenvalue will be zero and the other must then be the trace
```

```python
(A.transpose() * A).eigenvects()
```

```python
A * A.transpose()
```

```python
(A * A.transpose()).eigenvals()
```

```python
(A * A.transpose()).eigenvects()
```

+ Inserted below is the three resultant matrices from our calculations above (normalized, etc)

```python
Matrix([[1 / sqrt(5), 2 / sqrt(5)], [2 / sqrt(5), 1 / sqrt(5)]]), Matrix([[sqrt(125), 0], [0, 0]]), Matrix([[0.8, 0.6], [0.6, -0.8]])
```

+ Now let me show you how to correct the *svd*() solutions

```python
U
```

```python
S
```

```python
S = Matrix([[11.1803398874989, 0], [0, 0]])
S # Composed by hand (proper method further below)
```

```python
V
```

```python
V = Matrix([[-0.8, -0.6], [-0.6, 0.8]])
V # Remember that this is actually V transpose
```

+ Let's calculate AΣVT

```python
U * S * V
```

+ Compensating for rounding, this is the original matrix A

## Summary

+ The orthonormal basis for the rowspace is
$$ {v}_{1},{v}_{2},\dots,{v}_{r} $$
+ The orthonormal basis for the columnspace is
$$ {u}_{1},{u}_{2},\dots,{u}_{r} $$
+ The orthonormal basis for the nullspace is
$$ {v}_{r+1},{v}_{r+2},\dots,{v}_{n} $$
+ The orthonormal basis for the nullspace of AT
$$ {u}_{r+1},{u}_{r+2},\dots,{u}_{m} $$

## Example problem

### Example problem 1

* Find the singular value decomposition of the matrix
$$ \begin{bmatrix}5&5\\-1&7\end{bmatrix} $$

#### Solution

+ First off, I'll show you how to make proper use of numpy and scipy (as opposed to sympy) to solve singular value decomposition problems

```python
from numpy import matrix, transpose # Importing the matrix object and the
# transpose object from numerical python (numpy)
from numpy.linalg import svd, det # Importing the svd and determinant
# methods from the linalg submodule
from scipy.linalg import diagsvd
```

```python
type(transpose) # Type tells us what 'something' is (sometimes)
```

```python
C = matrix([[5, 5], [-1, 7]]) # Using the numpy matrix object
C
```

* We can see from the determinant that the rows and columns are independent

```python
det(C) # Notice the difference in syntax
```

+ Let's calculate U by looking at ATA

```python
transpose(C) *C # Notice the difference in synmtax
```

+ This is symmetric, positive definite
+ One eigenvalue will be 0 and the other, the trace (since they (the eigenvalues) must sum to the trace)
+ Remember that the eigenvalues are the squares of the *σ*i values

+ Now let's put numpy and sympy to good use

```python
U, S, VT = svd(C) # I use the computer variable VT to remind us that
# this is the transpose of V
```

+ S will only indicate the eigenvalues and must be converted to the correct sized matrix

```python
M, N = C.shape # Shape returns a tuple (two values), indicating
# row and column size
M, N
```

```python
Sig = diagsvd(S, M, N) # Creating a m times n matrix from S
Sig
```

```python
VT
```

+ Let's check if it worked!

```python
U * Sig * VT
```

Now, let's use good old sympy

```python
C = Matrix([[5, 5], [-1, 7]])
C
```

+ We need to work with a positive (semi)definite matrix

```python
CTC = C.transpose() * C # Using the computer variable CTC to remind that
# it is C transpose times C
CTC, CTC.det()
```

+ Let's look at the eigenvalues

```python
CTC.eigenvals()
```

+ Σ will contain along its main diagonal the square root of these eigenvalues

```python
Sig = Matrix([[sqrt(20), 0], [0, sqrt(80)]])
Sig
```

+ For V we require the eigenvectors of CTC
+ We need to remember to normalize each vector (dividing each component by the length (norm) of that vector

```python
CTC.eigenvects()
```

+ Let's normalize each *v*i by calculating the length (norm) of each

```python
v1 = Matrix([-3, 1])
v1.norm()
```

```python
v2 = Matrix([Rational(1, 3), 1])
v2.norm()
```

+ We'll get each element of V by dividing by these norms

```python
-3 / v1.norm(), 1 / v1.norm()
```

```python
Rational(1, 3) / v2.norm(), 1 / v2.norm()
```

```python
V = Matrix([[-3 / sqrt(10), 1 / sqrt(10)], [1 / sqrt(10), 3 / sqrt(10)]])
# Just remember to put the elements of V in the correct place
V
```

+ Remember that it is equal to the transpose of V

```python
V == V.transpose()
```

+ Now for U using CCT

```python
CCT = C * C.transpose() # Using the computer variable CCT
CCT
```

+ The eigenvalues will be the same

```python
CCT.eigenvals()
```

```python
CCT.eigenvects()
```

```python
u1 = Matrix([-1, 1])
u2 = Matrix([1, 1])
```

```python
-1 / u1.norm(), 1 / u1.norm()
```

```python
1 / u2.norm(), 1 / u2.norm()
```

```python
U = Matrix([[-sqrt(2) / 2, sqrt(2) / 2], [sqrt(2) / 2, sqrt(2) / 2]])
# Just remember to put the elements of U in the correct place
U
```

+ Let's see if it worked!

```python
U * Sig * V
```

```python

```


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

## Beta Densities with Integer Parameters ##

In the previous section we learned how to work with joint densities, but many of the joint density functions seemed to appear out of nowhere. For example, we checked that the function

$$
f(x, y) = 120x(y-x)(1-y), ~~~~ 0 < x < y < 1
$$

is a joint density, but there was no clue where it came from. In this section we will find its origin and go on to develop an important family of densities on the unit interval.

### Order Statistics of IID Uniform $(0, 1)$ Variables ###
Let $U_1, U_2, \ldots, U_n$ be i.i.d. uniform on $(0, 1)$. Imagine each $U_i$ as the position of a dart thrown at the unit interval. The graph below shows the positions of five such darts, each shown as a star.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
y = 1 - np.ones(5)
x = stats.uniform.rvs(size=5)
order_stats = np.sort(x)
plt.scatter(x, y, marker='*', color='r', s=100)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('Five IID Uniform (0, 1) Variables');
```

Based on the graph above, can you tell which star corresponds to $U_1$? You can't, because $U_1$ could be any of the five stars. So also you can't identify any of the five variables $U_1, U_2, U_3, U_4, U_5$.

What you *can* see, however, is the list of $U_i$'s *sorted in increasing order*. You can see the value of the minimum, the second on the sorted list, the third, the fourth, and finally the fifth which is the maximum.

These are called the *order statistics* of $U_1, U_2, U_3, U_4, U_5$, and are denoted $U_{(1)}, U_{(2)}, U_{(3)}, U_{(4)}, U_{(5)}$.

Remember that because the $U_i$'s are independent random variables with densities, there can't be ties: the chance that two of them are equal is 0.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
order_stats = np.sort(x)
plt.scatter(x, y, marker='*', color='r', s=100)
u_labels = make_array('$U_{(1)}$', '$U_{(2)}$', '$U_{(3)}$', '$U_{(4)}$', '$U_{(5)}$')
for i in range(5):
    plt.text(order_stats[i], -0.0007, u_labels[i], size=16)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('Order Statistics of the Five IID Uniform (0, 1) Variables');
```

In general for $1 \le k \le n$, the *$k$th order statistic* of $U_1, U_2, \ldots, U_n$ is the $k$th value when the $U_i$'s are sorted in increasing order. This can also be thought of as the $k$th *ranked* value when the minimum has rank 1. It is denoted $U_{(k)}$.

### Joint Density of Two Order Statistics ###
Let $n = 5$ as above and let's try to work out the joint density of $U_{(2)}$ and $U_{(4)}$. That's the joint density of the second and fourth values on the sorted list.

The graph below shows the event $\{U_{(2)} \in dx, U_{(4)} \in dy\}$ for values $x$ and $y$ such that $0 < x < y < 1$.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
y = 1 - np.ones(5)
x = make_array(0.1, 0.3, 0.45, 0.7, 0.9)
plt.scatter(x, y, marker='*', color='r', s=100)
plt.plot([0.28, 0.32], [0, 0], color='gold', lw=2)
plt.text(0.28, -0.0007, r'$dx$', size=16)
plt.plot([0.68, 0.72], [0, 0], color='gold', lw=2)
plt.text(0.68, -0.0007, r'$dy$', size=16)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('$n = 5$; $\{ U_{(2)} \in dx, U_{(4)} \in dy \}$');
```

To find $P(U_{(2)} \in dx, U_{(4)} \in dy)$, notice that:
- One of $U_1, U_2, U_3, U_4, U_5$ must be in $dx$; there are 5 ways to choose this one.
- Of the remaining 4 variables, one of them must be in $dy$; there are 4 ways to choose this one.
- Of the remaining 3, one must be in $(0, x)$.
- Of the remaining 2, one must be in $(x, y)$.
- The last one must be in $(y, 1)$.

Thus
$$
P(U_{(2)} \in dx, U_{(4)} \in dy) ~ \sim ~ 5(1dx) \cdot 4(1dy) \cdot 3x \cdot 2(y-x) \cdot 1(1-y)
~ = ~ 120x(y-x)(1-y)dxdy
$$

and therefore the joint density of $U_{(2)}$ and $U_{(4)}$ is given by

$$
f(x, y) = 120x(y-x)(1-y), ~~~ 0 < x < y < 1
$$

This solves the mystery of how the formula arises.

But it also does much more. The *marginal* densities of the order statistics of i.i.d. uniform $(0, 1)$ variables form a family that is important in data science.

### The Density of $U_{(k)}$ ###
Let $U_{(k)}$ be the $k$th order statistic of $U_1, U_2, \ldots, U_n$. We will find the density of $U_{(k)}$ by following the same general process that we followed to find the joint density above.

The graph below displays the event $\{ U_{(k)} \in dx \}$. For the event to occur,
- One of the variables $U_1, U_2, \ldots, U_n$ has to be in $dx$.
- Of the remaining $n-1$ variables, $k-1$ must have values in $(0, x)$ and the rest in $(x, 1)$.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
plt.scatter(0.4, 0, marker='*', color='r', s=100)
plt.plot([0.38, 0.42], [0, 0], color='gold', lw=2)
plt.text(0.38, -0.0007, r'$dx$', size=16)
plt.text(0.1, 0.001, '$k-1$ stars', size=16)
plt.text(0.1, 0.0005, 'in $(0, x)$', size=16)
plt.text(0.6, 0.001, '$n-k$ stars', size=16)
plt.text(0.6, 0.0005, 'in $(x, 1)$', size=16)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('$\{ U_{(k)} \in dx \}$');
```

There are $n$ ways to choose the variable that lands in $dx$. Once that is chosen, $k-1$ of the remaining $n-1$ variables must have values in $(0, x)$. Thus

$$
P(U_{(k)} \in dx) \sim n \cdot 1 dx \cdot \binom{n-1}{k-1}x^{k-1}(1-x)^{n-k}
$$

by the binomial formula.

Therefore the density of $U_{(k)}$ is given by

$$
f_{U_{(k)}} (x) = \frac{n!}{(k-1)!(n-k)!} x^{k-1}(1-x)^{n-k}, ~~~ 0 < x < 1
$$

Let's rewrite the exponents slightly:
$$
f_{U_{(k)}} (x) = \frac{n!}{(k-1)!((n-k+1)-1)!} x^{k-1}(1-x)^{(n-k+1)-1}, ~~~ 0 < x < 1
$$

Because $1 \le k \le n$, we know that $n-k+1$ is a positive integer. Since $n$ is an arbitrary positive integer, so is $n-k+1$.

### Beta Densities ###
We have shown that if $r$ and $s$ are any two positive integers, then the function

$$
f(x) ~ = ~ \frac{(r+s-1)!}{(r-1)!(s-1)!} x^{r-1}(1-x)^{s-1}, ~~~ 0 < x < 1
$$

is a probability density function. This is called the *beta density with parameters $r$ and $s$*.

The order statistic $U_{(k)}$ has the beta density with parameters $k$ and $n-k+1$.

The shape of the density is determined by the factors that involve $x$ and $1-x$. All the factorials are just parts of the constant that make the density integrate to 1.

Notice that the uniform $(0, 1)$ density is the same as the beta density with parameters $r = 1$ and $s = 1$. The uniform $(0, 1)$ density is a member of the *beta family*.

The graph below shows some beta density curves. As you would expect, the beta $(3, 3)$ density is symmetric about 0.5.

```python
x = np.arange(0, 1.01, 0.01)
for i in np.arange(1, 7, 1):
    plt.plot(x, stats.beta.pdf(x, i, 6-i), lw=2)
plt.title('Beta $(i, 6-i)$ densities for $1 \leq i \leq 5$');
```

By choosing the parameters appropriately, you can create beta densities that put much of their mass near a prescribed value. That is one of the reasons beta densities are used to model *random proportions*. For example, if you think that the probability that an email is spam is most likely in the 60% to 90% range, but might be lower, you might model your belief by choosing the density that peaks at around 0.75 in the graph above.

The calculation below shows you how to get started on the process of picking parameters so that the beta density with those parameters has properties that reflect your beliefs.

### The Beta Integral ###
The beta density integrates to 1, and hence for all positive integers $r$ and $s$ we have

$$
\int_0^1 x^{r-1}(1-x)^{s-1}dx ~ = ~ \frac{(r-1)!(s-1)!}{(r+s-1)!}
$$

Thus probability theory makes short work of an otherwise laborious integral. Also, we can now find the expectation of a random variable with a beta density.

Let $X$ have the beta $(r, s)$ density for two positive integer parameters $r$ and $s$. Then

\begin{align*}
E(X) &= \int_0^1 x \frac{(r+s-1)!}{(r-1)!(s-1)!} x^{r-1}(1-x)^{s-1}dx \\
&= \frac{(r+s-1)!}{(r-1)!(s-1)!} \int_0^1 x^r(1-x)^{s-1}dx \\
&= \frac{(r+s-1)!}{(r-1)!(s-1)!} \cdot \frac{r!(s-1)!}{(r+s)!} ~~~~ \text{(beta integral for parameters } r+1 \text{and } s\text{)}\\
&= \frac{r}{r+s}
\end{align*}

You can follow the same method to find $E(X^2)$ and hence $Var(X)$.

The formula for the expectation allows you to pick parameters corresponding to your belief about the random proportion being modeled by $X$. For example, if you think the proportion is likely to be somewhere around 0.4, you might start by trying out a beta prior with $r = 2$ and $s = 3$.

You will have noticed that the form of the beta density looks rather like the binomial formula. Indeed, we used the binomial formula to derive the beta density. Later in the course you will see another close relation between the beta and the binomial. These properties make the beta family one of the most widely used families of densities in machine learning.

```python

```


# A simple example

To illustrate the application of quantile regression neural networks we consider a simple, 1-dimensional regression probelm with heteroscedastic uncertainty:

\begin{align}
y = \sin(x) + \cos(x) \cdot \mathcal{N}(0, 1)
\end{align}

The code below generates the synthetic training data consisting of $10^7$ randomly sampled $x$ values and corresponding $y$ values.

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import quantnn
import quantnn.plotting
quantnn.plotting.set_style()
```

    Bad key "mathtext.fallback" on line 203 in
    /home/simonpf/src/quantnn/quantnn/data/matplotlib_style.rc.
    You probably need to get an updated matplotlibrc file from
    https://github.com/matplotlib/matplotlib/blob/v3.1.3/matplotlibrc.template
    or from the matplotlib source distribution

```python
from quantnn.examples import simple
x, y = simple.create_training_data(10_000_000)
simple.plot_histogram(x, y)
plt.savefig("simple_training_data.png")
```

## Quantile regression neural networks (QRNNs)

The task of predicting the conditional distribution $P(y|x)$ of $y$ given input $x$ can be solved using quantile regression. The ``QRNN`` class of the quantnn package provides an implementation of a Quantile Regression Neural Network (QRNN), i.e. it uses a neural network to learn to predict the quantiles of the posterior distribution $P(y|x)$.

### Model definition and training

The simplest way to train a quantile regression neural network is to use the ``quantnn.QRNN`` class, which provides a simple interface to define and and train QRNNs based on fully-connected, feed-forward neural networks.

To do this all we need to do is to choose which quantiles to predict as well as the basic parameters of the network architecture. More specifically, the following information must be provided to the constructor of the ``QRNN`` class:

1. The list of quantiles that we want to predict.
2. The number of input features (``input_dimensions``). For this simple example this just ``1``.
3. A tuple describing the model architecture ``(n_layers, n_neurons, activation_function)``

> **Note:** quantnn will automatically choose the backend between Keras and PyTorch depending on which one is available on your system. You can specifically choose one over the other using the ``quantnn.qrnn.set_backend`` function.

```python
import quantnn as q
quantiles = np.linspace(0.01, 0.99, 99)

layers = 4
neurons = 128
activation = "relu"
model = (layers, neurons, activation)
qrnn = q.QRNN(quantiles, n_inputs=1, model=model)
```

Finally, we train the neural network on the training data. Here, we use only a subset of the data, in order to speed up the training process. We restart the training three times while lowering the learning rate, which leads to more accurate predicted quantiles.

```python
training_data = (x[::5].reshape(-1, 1), y[::5].reshape(-1, 1))
results = qrnn.train(training_data=training_data,
                     n_epochs=15)
```

    Output()

           Training history

      Epoch          Training loss
 ────────────────────────────────────
     #      LR           Total

      1   0.0005         0.183
      2   0.0005         0.182
      3   0.0005         0.182
      4   0.0005         0.182
      5   0.0005         0.182
      6   0.0005         0.182
      7   0.0005         0.182
      8   0.0005         0.182
      9   0.0005         0.182
     10   0.0005         0.182
     11   0.0005         0.182
     12   0.0005         0.182
     13   0.0005         0.182
     14   0.0005         0.182
     15   0.0005         0.182

### Evaluation

To evaluate the QRNN we generate 1000 new $x$-value and use the ``predict`` method of the trained ``qrnn`` object to predict the quantiles of $y$.

```python
n = 1_000
x_val = np.linspace(-np.pi, np.pi, n)
y_val = simple.create_validation_data(x_val)
y_pred = qrnn.predict(x_val.reshape(-1, 1))
y_mean = qrnn.posterior_mean(y_pred=y_pred)
```

We can now compare the quantiles predicted using the QRNN to the empirical quantiles from the training data. The QRNN manages fairly well to predict the true conditional cumulative distribution function (CDF). With additional optimization of the training process, the

```python
from quantnn.plotting import plot_quantiles
from matplotlib.gridspec import GridSpec

gs = GridSpec(1, 3, width_ratios=[1.0, 1.0, 0.05])
f = plt.figure(figsize=(12, 5))

# Calculate empirical CDF
bins = np.linspace(-np.pi, np.pi, 201)
counts, _, _ = np.histogram2d(y, x, bins=bins)
norm = np.sum(counts * (bins[1:] - bins[:-1]), axis=0, keepdims=True)
counts /= norm
cdf = np.cumsum(counts * (bins[1:] - bins[:-1]), axis=0)

# Plot QRNN quantiles
ax = plt.subplot(gs[0, 0])
m = plot_quantiles(ax, x_val, y_pred[:, 1::6], qrnn.quantiles[1::6])
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2.5, 2.5])
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_title("(a) Predicted CDF", loc="left")
ax.grid(False)

# Plot true cdf
ax = plt.subplot(gs[0, 1])
ax.pcolormesh(bins, bins, cdf, cmap=m.cmap, norm=m.norm)
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2.5, 2.5])
ax.set_xlabel("x")
ax.set_title("(b) True CDF", loc="left")

ax = plt.subplot(gs[0, 2])
plt.colorbar(m, cax=ax, label="$P(Y \leq y | X=x)$")
plt.tight_layout()
```

## Density regression neural network

An alternative way to learn to predict the distribution $P(y|x)$ is to use a neural network to predict a gridded approximation of the probability density function, a technique we will, for simplicity, refer to as Density Regression Neural Network (DRNN).

To create a DRNN model, we need to choose a suitable binning for the output value. Here we simply use an equidistant grid covering the range $-3$ to $3$. Apart from having to provide the bins of the output PDF instead of the quantiles to predict, the definition, training and evaluation of the DRNN works in exactly the same way as for the QRNN.

```python
from quantnn.drnn import DRNN
layers = 4
neurons = 128
activation = "relu"
model = (layers, neurons, activation)
bins = np.linspace(-3, 3, 101)
drnn = DRNN(bins, n_inputs=1, model=model)
```

```python
training_data = (x[::5].reshape(-1, 1), y[::5].reshape(-1, 1))
results = drnn.train(training_data=training_data,
                     n_epochs=15)
```

    Output()

           Training history

      Epoch          Training loss
 ────────────────────────────────────
     #      LR           Total

      1   0.0005         3.727
      2   0.0005         3.567
      3   0.0005         3.563
      4   0.0005         3.561
      5   0.0005         3.559
      6   0.0005         3.558
      7   0.0005         3.558
      8   0.0005         3.557
      9   0.0005         3.557
     10   0.0005         3.556
     11   0.0005         3.556
     12   0.0005         3.556
     13   0.0005         3.556
     14   0.0005         3.556
     15   0.0005         3.555

```python
y_pred = drnn.predict(x_val.reshape(-1, 1))
```

To evaluate the quality of the predicted PDF, we can now simply plot the predicted PDFs to the empirical PDFs of the training data. Similar as for the QRNN, the prediction match the reference values fairly well.

```python
from quantnn.plotting import plot_quantiles
from matplotlib.colors import Normalize
from matplotlib.gridspec import GridSpec

gs = GridSpec(1, 3, width_ratios=[1.0, 1.0, 0.05])
f = plt.figure(figsize=(12, 5))
norm=Normalize(0.0, 0.5)

# Plot DRNN PDF
bin_centers = 0.5 * (drnn.bins[1:] + drnn.bins[:-1])
ax = plt.subplot(gs[0, 0])
m = ax.pcolormesh(x_val, bin_centers, y_pred.T, norm=norm)
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2.5, 2.5])
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_title("(a) Predicted PDF", loc="left")
ax.grid(False)

# Plot true cdf
bins_x = np.linspace(-np.pi, np.pi, 201)
counts, _, _ = np.histogram2d(y, x, bins=((bins, bins_x)))
counts /= np.sum(counts * (bins[1:] - bins[:-1])[:, np.newaxis], axis=0, keepdims=True)

ax = plt.subplot(gs[0, 1])
ax.pcolormesh(bins_x, bins, counts, norm=norm)
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2.5, 2.5])
ax.set_xlabel("x")
ax.set_title("(b) True PDF", loc="left")

ax = plt.subplot(gs[0, 2])
plt.colorbar(m, cax=ax, label="$p(y | x)$")
plt.tight_layout()
```

## Calculating posterior statistics

Both the QRNN and DRNN allow us to predict the posterior distribution of the quantity $y$ given input $x$. In most applications, however, we are not interested in the full distribution but rather specific statistics. The quantnn package provides functions to calculate a range of such statistics in the ``quantnn.quantiles`` module for QRNN predictions and in the ``quantnn.desntiy`` module for predictions produced by a DRNN.

```python
y_pred_qrnn = qrnn.predict(x_val.reshape(-1, 1))
y_pred_drnn = drnn.predict(x_val.reshape(-1, 1))
```

### Mean and median

```python
import quantnn.quantiles as qq
import quantnn.density as qd

qrnn_mean = qrnn.posterior_mean(y_pred=y_pred_qrnn)
drnn_mean = drnn.posterior_mean(y_pred=y_pred_drnn)
qrnn_median = y_pred_qrnn[:, 49]
drnn_median = drnn.posterior_quantiles(y_pred=y_pred_drnn, quantiles=[0.5])
```

```python
gs = GridSpec(2, 2, height_ratios=[1.0, 0.1])
f, axs = plt.subplots(1, 2, figsize=(12, 5))
y_mean = np.sin(x_val)

ax = plt.subplot(gs[0, 0])
ax.plot(x_val, qrnn_mean)
ax.plot(x_val, drnn_mean)
ax.plot(x_val, y_mean, c="k", ls="--", label="Truth")
ax.set_title("(a) Mean", loc="left")

ax = plt.subplot(gs[0, 1])
handles = []
handles += ax.plot(x_val, qrnn_median)
handles += ax.plot(x_val, drnn_median)
handles += ax.plot(x_val, y_mean, c="k", ls="--", label="Truth")
ax.set_title("(b) Median", loc="left")

ax = plt.subplot(gs[1, :])
ax.set_axis_off()
ax.legend(handles=handles, labels=["QRNN", "DRNN", "truth"], ncol=3, loc="center")
```

### Quantiles

```python
quantiles = np.linspace(0.1, 0.9, 9)
quantiles_qrnn = qrnn.posterior_quantiles(y_pred=y_pred_qrnn, quantiles=quantiles)
quantiles_drnn = drnn.posterior_quantiles(y_pred=y_pred_drnn, quantiles=quantiles)
```

```python
gs = GridSpec(1, 3, width_ratios=[1.0, 1.0, 0.05])
f, axs = plt.subplots(1, 2, figsize=(13, 5))
y_mean = np.sin(x_val)

ax = plt.subplot(gs[0, 0])
m = plot_quantiles(ax, x_val, quantiles_qrnn, quantiles)
ax.set_title("(a) QRNN quantiles", loc="left")
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2, 2])
ax.set_xlabel("x")
ax.set_ylabel("y")

ax = plt.subplot(gs[0, 1])
plot_quantiles(ax, x_val, quantiles_drnn, quantiles)
ax.set_title("(b) DRNN quantiles", loc="left")
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2, 2])
ax.set_xlabel("x")
ax.set_ylabel("y")

ax = plt.subplot(gs[0, 2])
plt.colorbar(m, cax=ax, label="$P(Y \leq y | X=x)$")
```

## Binned PDF

```python
bins = drnn.bins
centers = 0.5 * (bins[1:] + bins[:-1])
binned_pdf_qrnn = qq.pdf_binned(y_pred_qrnn.numpy(),
                                qrnn.quantiles,
                                bins)
```

```python
gs = GridSpec(1, 3, width_ratios=[1.0, 1.0, 0.05])
f, axs = plt.subplots(1, 2, figsize=(13, 5))
norm = Normalize(0, 0.5)

ax = plt.subplot(gs[0, 0])
m = ax.pcolormesh(x_val, centers, binned_pdf_qrnn.T, norm=norm)
ax.set_title("(a) QRNN", loc="left")
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2, 2])
ax.set_xlabel("x")
ax.set_ylabel("y")

ax = plt.subplot(gs[0, 1])
m = ax.pcolormesh(x_val, centers, y_pred_drnn.T, norm=norm)
ax.set_title("(b) DRNN", loc="left")
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([-2, 2])
ax.set_xlabel("x")

ax = plt.subplot(gs[0, 2])
plt.colorbar(m, cax=ax, label="$p(y | x)$")
plt.tight_layout()
```

## Classification

Finally, we consider the case of trying to classfying whether the output $y$ corresponding to a  given $x$ is larger than a certain threshold value. For this we can use the ``probability_larger_than`` functions of the ``quantnn.quantiles`` and ``quantnn.density`` modules. This function returns the probability that the output $y$ is larger than the given threshold.

```python
t = 0.1
p_qrnn = qq.probability_larger_than(y_pred_qrnn.numpy(), qrnn.quantiles, t)
p_drnn = qd.probability_larger_than(y_pred_drnn.numpy(), drnn.bins, t)

# Calculate empirical probability
bins = np.linspace(-np.pi, np.pi, 201)
counts, _, _ = np.histogram2d(y, x, bins=bins)
centers = 0.5 * (bins[1:] + bins[:-1])
counts_larger_t = counts * (centers > t).reshape(-1, 1)
p = np.sum(counts_larger_t, axis=0) / np.sum(counts, axis=0)
```

```python
f, ax = plt.subplots(1, 1)
ax.plot(x_val, p_qrnn)
ax.plot(x_val, p_drnn)
ax.plot(centers, p, c="k", ls="--")
ax.set_xlim([-np.pi, np.pi])
ax.set_ylim([0, 1])
ax.set_xlabel("x")
ax.set_ylabel("$P(Y > 0.1 | X = x)$")
```


# 勾配消失問題
多層パーセプトロンでは、層の長さを長くすればするほど表現力は増します。一方で、学習が難しくなるという問題が知られています。

中間層が10層という深い(deepな)多層パーセプトロンを用いて問題を確認してみます。

```python
# このセルはヘルパー関数です。

# ライブラリのインポート
import numpy as np
import matplotlib.pyplot as plt
from keras import backend as K
import tensorflow as tf
tf.compat.v1.disable_eager_execution()

# i 番目のレイヤーの出力を取得する
def getActivation(model, i, x):
    activations = K.function([model.input, K.learning_phase()], [model.layers[i].output, ])
    return activations(x)[0]

# iLayer 番目のレイヤーの勾配(誤差)を取得する
def getGradientNode(model, i, x, t):
    layer_output = model.layers[i].output
    loss = K.mean(K.binary_crossentropy(model.output, K.variable(t)))
    grads = K.gradients(loss, layer_output)[0]
    gradient_function = K.function([model.input], [grads])
    return gradient_function([x])[0]

# iLayer 番目のレイヤーにつながる重み(wij)の勾配を取得する
def getGradientParameter(model, i, x, t):
    weights = model.layers[i].weights[0]  # get only kernel (not bias)
    gradients = K.gradients(model.total_loss, weights)
    input_tensors = model._feed_inputs + model._feed_targets + model._feed_sample_weights
    gradient_function = K.function(input_tensors, gradients)
    return gradient_function([x, np.ones(len(x)), t, 0])[0]

def getRelGradientParameter(model, i, x, t):
    return getGradientParameter(model, i, x, t) / model.get_weights()[i * 2]

# モデルの勾配や活性化関数の出力等をプロット
# 左上: ウェイト(wij)の初期値
# 中上: ウェイト(wij)の微分(dE/dwij)
# 右上: ウェイト(wij)の微分(dE/dwij)の分散
# 左下: 各ノードの出力(sigma(ai))
# 中下: 各ノードでの微分(delta_ij)
# 右下: 各ノードでの微分(delta_ij)の分散
def showGradient(model, x, t):
    fig, ax = plt.subplots(2, 3, figsize=(18, 8))

    iLayers = [0, 3, 6, 10]
    labels = [
        ' 0th layer',
        ' 3th layer',
        ' 6th layer',
        'Last layer',
    ]

    values = [model.get_weights()[i * 2].flatten() for i in iLayers]
    ax[0][0].hist(values, bins=50, stacked=False, density=True, label=labels, histtype='step')
    ax[0][0].set_xlabel('weight')
    ax[0][0].set_ylabel('Probability density')
    ax[0][0].legend(loc='upper left', fontsize='x-small')

    values = [getActivation(model, i, x).flatten() for i in iLayers]
    ax[1][0].hist(values, bins=50, stacked=False, density=True, label=labels, histtype='step')
    ax[1][0].set_xlabel('activation')
    ax[1][0].set_ylabel('Probability density')
    ax[1][0].legend(loc='upper center', fontsize='x-small')

    grads = [np.abs(getGradientParameter(model, i, x, t).flatten()) for i in iLayers]
    grads = [np.log10(x[x > 0]) for x in grads]
    ax[0][1].hist(grads, bins=50, stacked=False, density=True, label=labels, histtype='step')
    ax[0][1].set_xlabel('log10(|gradient of weights|)')
    ax[0][1].set_ylabel('Probability density')
    ax[0][1].legend(loc='upper left', fontsize='x-small')

    grads = [np.abs(getGradientNode(model, i, x, t).flatten()) for i in iLayers]
    grads = [np.log10(x[x > 0]) for x in grads]
    ax[1][1].hist(grads, bins=50, stacked=False, density=True, label=labels, histtype='step')
    ax[1][1].set_xlabel('log10(|gradient of nodes|)')
    ax[1][1].set_ylabel('Probability density')
    ax[1][1].legend(loc='upper left', fontsize='x-small')

    var = [np.abs(getGradientParameter(model, i, x, t).flatten()).var() for i in range(10)]
    ax[0][2].plot(var)
    ax[0][2].set_yscale('log')
    ax[0][2].set_xlabel('Layer')
    ax[0][2].set_ylabel('Variance of gradient of weights')

    var = [np.abs(getGradientNode(model, i, x, t).flatten()).var() for i in range(10)]
    ax[1][2].plot(var)
    ax[1][2].set_yscale('log')
    ax[1][2].set_xlabel('Layer')
    ax[1][2].set_ylabel('Variance of gradient of nodes')

    plt.show()

```

```python
from keras.models import Sequential
from keras.initializers import RandomNormal, RandomUniform
from keras.layers import Dense

# データセットの生成
nSamples = 1000
nFeatures = 50
x = np.random.randn(nSamples, nFeatures)  # 100個の入力変数を持つイベント1000個生成。それぞれの入力変数は正規分布に従う
t = np.random.randint(2, size=nSamples).reshape([nSamples, 1])  # 正解ラベルは0 or 1でランダムに生成

# モデルの定義
activation = 'sigmoid'  # 中間層の各ノードで使う活性化関数
initializer = RandomNormal(mean=0.0, stddev=1.0)  # weight(wij)の初期値。ここでは正規分布に従って初期化する
# initializer = RandomUniform(minval=-1, maxval=1)  # weight(wij)の初期値。ここでは一様分布に従って初期化する

# 中間層が10層の多層パーセプトロン。各レイヤーのノード数は全て50。
model = Sequential()
model.add(Dense(50, activation=activation, kernel_initializer=initializer, input_dim=nFeatures))
for i in range(9):
    model.add(Dense(50, activation=activation, kernel_initializer=initializer))
model.add(Dense(1, activation='sigmoid', kernel_initializer=initializer))
model.compile(loss='binary_crossentropy', optimizer='adam')

# モデルの勾配や活性化関数の出力等をプロット
# 左上: ウェイト(wij)の初期値
# 中上: ウェイト(wij)の微分(dE/dwij)
# 右上: ウェイト(wij)の微分(dE/dwij)の分散
# 左下: 各ノードの出力(sigma(ai))
# 中下: 各ノードでの微分(delta_ij)
# 右下: 各ノードでの微分(delta_ij)の分散
showGradient(model, x, t)

```

左上のプロットはパラメータ($w_{ij}$)の初期値を表しています。指定したとおり、各層で正規分布に従って初期化されています。
左下のプロットは活性化関数の出力($z_i$)を表しています。パラメータ($w_{ij}$)の初期値として正規分布を指定すると、シグモイド関数の出力はそのほとんどが0か1に非常に近い値となっています。シグモイド関数の微分は$\sigma^{'}(x)=\sigma(x)\cdot(1-\sigma(x))$なので、$\sigma(x)$が0や1に近いときは微分値も非常に小さな値となります。
誤差逆伝播の式は
$$
\begin{align}
\delta_{i}^{(k)} &= \sigma^{'}(a_i^{(k)}) \left( \sum_j w_{ij}^{(k+1)} \cdot \delta_{j}^{(k+1)} \right) \\
\frac{\partial E_n}{\partial w_{ij}^{(k)}}  &= \delta_{j}^{(k)} \cdot z_{i}^{(k)}
\end{align}
$$
でした。$\sigma^{'}(a_i^{(k)})$が小さいと後方の層から前方の層に誤差が伝わる際に、値が小さくなってしまいます。
中上、中下のプロットはそれぞれ各層での$\frac{\partial E_n}{\partial w_{ij}^{(k)}}$、$\delta_{i}^{(k)}$を表しています。
前方の層(0th layer)は後方の層と比較して分布の絶対値が小さくなっています。
右上と右下のプロットは各レイヤーでの$\frac{\partial E_n}{\partial w_{ij}^{(k)}}$と$\delta_{i}^{(k)}$の分布の分散を表しています。

このように誤差が前の層にいくにつれて小さくなるため、前の層が後ろの層と比較して学習が進まなくなります。
この問題は勾配消失の問題として知られています。

勾配消失はパラメータの初期値や、活性化関数を変更することによって解決・緩和することがわかっています。
Kerasの
- [初期化のページ](https://keras.io/ja/initializers/)
- [活性化関数のページ](https://keras.io/ja/activations/)

も参考にしながら、この問題の解決を試みてみましょう。

活性化関数・パラメータの初期化方法の変更はそれぞれコード中の"activation"、"initializer"を変更することによって行えます。

例えばパラメータの初期化を(-0.01, +0.01)の一様分布に変更するときは以下のコードのようにすれば良いです。

```python
from keras.models import Sequential
from keras.initializers import RandomNormal, RandomUniform
from keras.layers import Dense

# データセットの生成
nSamples = 1000
nFeatures = 50
x = np.random.randn(nSamples, nFeatures)  # 100個の入力変数を持つイベント1000個生成。それぞれの入力変数は正規分布に従う
t = np.random.randint(2, size=nSamples).reshape([nSamples, 1])  # 正解ラベルは0 or 1でランダムに生成

# モデルの定義
activation = 'sigmoid'  # 中間層の各ノードで使う活性化関数
# initializer = RandomNormal(mean=0.0, stddev=1.0)  # weight(wij)の初期値。ここでは正規分布に従って初期化する
initializer = RandomUniform(minval=-0.01, maxval=0.01)  # weight(wij)の初期値。ここでは一様分布に従って初期化する

# 中間層が10層の多層パーセプトロン。各レイヤーのノード数は全て50。
model = Sequential()
model.add(Dense(50, activation=activation, kernel_initializer=initializer, input_dim=nFeatures))
for i in range(9):
    model.add(Dense(50, activation=activation, kernel_initializer=initializer))
model.add(Dense(1, activation='sigmoid', kernel_initializer=initializer))
model.compile(loss='binary_crossentropy', optimizer='adam')

# モデルの勾配や活性化関数の出力等をプロット
# 左上: ウェイト(wij)の初期値
# 中上: ウェイト(wij)の微分(dE/dwij)
# 右上: ウェイト(wij)の微分(dE/dwij)の分散
# 左下: 各ノードの出力(sigma(ai))
# 中下: 各ノードでの微分(delta_ij)
# 右下: 各ノードでの微分(delta_ij)の分散
showGradient(model, x, t)

```

この例では活性化関数の出力が0.5付近に集中しています。
どのノードも同じ出力をしているということはノード数を増やした意味があまりなくなっており、多層パーセプトロンの表現力が十分に活かしきれていないことがわかります。
また、勾配消失も先程の例と比較して大きくなっています。


## Day 4: Neurons and Networks

Welcome to Day 4! In this section we simulate a network of neurons interacting via synapses.

### How do we model Synapses?

Each synapse is defined by its own set of state variables and differential equations governing their temporal evolution. There are different kinds of synapses - electrical and chemical synapses. Electrical synapses are essentially physical conduits that allow the flow of ions across connected neurons. Chemical synapses are more common in the brain and are more complex than electrical synapses. When an action potential arrives at the axon terminal, it leads to the opening of voltage-gated calcium channels. The incoming calcium triggers neurotransmitter filled vesicles to fuse with the axon terminal membrane and release their cargo into the synaptic cleft. The neurotransmitters diffuse across the cleft and open (or close) ion channels on the post-synaptic neuron. This can cause a depolarization (increase in potential across the post-synaptic neuron's membrane) that makes it easier for the neuron to spike or it can inhibit the neuron and have the opposite effect. In some cases these effects are fast and direct - a neurotransmitter binds to a receptor in the post-synaptic site that causes an influx or efflux of ions and leads to a change in the membrane potential. The effect of a synapse can also be indirect such that neurotransmitters invoke a second messenger cascade that eventually leads to the opening or closing of ion channels in the post-synaptic neuron. Here we model fast excitatory and inhibitory chemical synapses.The network of interactions between neurons will be described by a connectivity matrix. Different connectivity matrices describe the interactions due to different types of synapses.

#### Modelling Synapses

The synaptic current ($I_{syn}$) depends on the difference between the reversal potential ($E_{syn}$) and the value of the membrane potential ($u$). The synaptic current due to neurotransmitter release into the synaptic cleft following an action potential is given by,
\begin{equation}
I_{syn}(t)=g_{syn}(t)(u(t)−E_{syn})
\end{equation}
When the transmitter binds to postsynaptic receptors it causes a transient change in the conductance $g_{syn}$. To capture the dynamics of $g_{syn}$, one models the system using a simple kinetic model where the receptors can be either in an open or a closed state~\cite{Destexhe1994}. The transition between the states is proportional to the concentration of the neurotransmitter $[T]$ in the cleft.
\begin{equation}
\mathrm{C}\underset{\beta}{\stackrel{\alpha[T]}{\rightleftharpoons}} \mathrm{O}
\end{equation}
This may be rewritten in the form of a differential equation.
\begin{eqnarray}\frac{d[O]}{dt}=\alpha[T](1−[O])−\beta[O]\end{eqnarray}

We can now describe the synaptic conductance $g_{syn}(t)=g_{max}[O]$, in terms of the maximal conductance $g_{max}$ and a gating variable $[O]$, where $[O](t)$ is the fraction of open synaptic channels. $\alpha$ is the binding constant, $\beta$ the unbinding constant and $(1−[O])$ the fraction of closed channels where the neurotransmitter can bind. The functional form of T depends on the nature of the synapse.
For cholinergic excitatory synapses, $[T]$ is given by,

\begin{eqnarray}
[T]_{ach} = A\ \Theta(t_{max}+t_{fire}+t_{delay}-t)\ \Theta(t-t_{fire}-t_{delay})
\end{eqnarray}

where, $\Theta (x)$ is the Heaviside step function, $t_{fire}$ is the time of the last presynaptic spike, $t_{delay}$ is the time delay from the time of the last spike to its effect on the postsynaptic neuron and $t_{max}$ is the duration after the spike during which the transmitter remains in the synaptic cleft.
For Fast GABAergic inhibitory synapses, we used the following equation,

\begin{eqnarray}[T]_{gaba} = \frac{1}{1+e^{-\frac{V(t-t_{fire}-t_{delay})-V_0}{\sigma}}}\end{eqnarray}

Note that in order to solve this equation, we need to determine the time when the presynaptic neuron fired ($t_{fire}$). To account for these synaptic interactions between neurons we need to modify the RK4 integrator developed to simulate multiple independent Hodgkin-Huxley neurons.

#### Iterating over conditionals in TensorFlow

In this section we modify the integrator that we coded on Day 2 to account for interactions between neurons. This will require an additional variable that stores the time elapsed from the last presynaptic spike to calculate the  equations. In the modified code we will use the TensorFlow function $\texttt{tf.where()}$ to efficiently assign the indices of neurons that have spiked and those that have not spiked at each time point. To understand the usage and function of $\texttt{tf.where()}$, consider the following example. Say, you have a array $\texttt{x}$ of 10 random numbers between 0 and 1. You want the output of the code to be another array of the same size as $\texttt{x}$ such that the elements of the array are either -10 or 10 depending on whether the corresponding element in $\texttt{x}$ is less or greater than 0.5. The function  $\texttt{tf.where(cond,a,b)}$ outputs an array with elements from $\texttt{a}$ if the condition $\texttt{cond}$ is $\texttt{True}$ and from $\texttt{b}$ if $\texttt{cond}$ is $\texttt{False}$. See the example code below.

```python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
## OR for Tensorflow 2.0 and above ##
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

# create the Tensor with the random variables
x = tf.constant(np.random.uniform(size = (10,)),dtype=tf.float64)

# a list of 10s to select from if true
if_true = tf.constant(10*np.ones((10,)),dtype=tf.float64)

# a list of -10s to select from if false
if_false = tf.constant(-10*np.ones((10,)),dtype=tf.float64)

# perform the conditional masking
selection = tf.where(tf.greater(x,0.5),if_true,if_false)

with tf.Session() as sess:
    x_out = sess.run(x)
    selection_out = sess.run(selection)

print(x_out)
print(selection_out)
```

    [0.63200218 0.92918439 0.54110399 0.32261521 0.73172468 0.44062604
     0.9469465  0.75659112 0.41025333 0.46642094]
    [ 10.  10.  10. -10.  10. -10.  10.  10. -10. -10.]

#### Redesigning the Generalized TensorFlow Integrator
In order to determine whether a particular neuron fired, we introduce a new variable \texttt{fire\_t} that stores the time of the last spike for each neuron.

We modify the code as follows:
1. The Integrator class that we defined earlier now requires two more properties as input, namely, the number of neurons ($\texttt{n}$) and the firing threshold ($\texttt{F_b}$) of each of these neurons. We provide these inputs as arguments to the Integrator class.
2. The state vector will now have an additional set of $\texttt{n}$ variables representing the firing times. These will not be updated by the step function ($\texttt{_step_func}$).
\3. Inside the Integrator class, we have access to the values of the state variable and the change in the state variable since the last iteration. We use this to check if the voltages have crossed the firing threshold. The convention followed in this code is, the first $\texttt{n}$ elements of the state vector are the membrane voltages while the last $\texttt{n}$ elements are the time from the last spike for each of the neurons.
4. The differential update function, step\_func, takes all except the last $\texttt{n}$ values of the state variable and updates it according to the differential equations specified in $\texttt{func}$. The last $\texttt{n}$ variables are updated separately in $\texttt{scan_func}$. It checks if any neuron has crossed its firing threshold and updates the variable $\texttt{fire_t}$ of the appropriate neurons with the current time.

The modifications to the RK4 code implemented earlier is shown below,

```python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
## OR ##
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

%matplotlib inline

def tf_check_type(t, y0): # Ensure Input is Correct
    if not (y0.dtype.is_floating and t.dtype.is_floating):
        raise TypeError('Error in Datatype')

class _Tf_Integrator():

    def __init__(self,n_,F_b):
        # class constructor to get inputs for number of neurons and firing thresholds
        self.n_ = n_
        self.F_b = F_b

    def integrate(self, func, y0, t):
        time_delta_grid = t[1:] - t[:-1]

        def scan_func(y, t_dt):
            # recall the necessary variables
            n_ = self.n_
            F_b = self.F_b

            t, dt = t_dt

            # Differential updation
            dy = self._step_func(func,t,dt,y) # Make code more modular.
            dy = tf.cast(dy, dtype=y.dtype) # Failsafe

            out = y + dy # the result after differential updation

            # Conditional to use specialized Integrator vs Normal Integrator (n=0)
            if n_>0:

                # Extract the last n variables for fire times
                fire_t = y[-n_:]

                # Value of change in firing times if neuron didnt fire = 0
                l = tf.zeros(tf.shape(fire_t),dtype=fire_t.dtype)

                # Value of change in firing times if neuron fired = Current Time - Last Fire Time
                l_ = t-fire_t

                # Check if Voltage is initially less than Firing Threshold
                z = tf.less(y[:n_],F_b)
                # Check if Voltage is more than Firing Threshold after updation
                z_ = tf.greater_equal(out[:n_],F_b)

                df = tf.where(tf.logical_and(z,z_),l_,l)

                fire_t_ = fire_t+df # Update firing time

                return tf.concat([out[:-n_],fire_t_],0)
            else:
                return out

        y = tf.scan(scan_func, (t[:-1], time_delta_grid),y0)

        return tf.concat([[y0], y], axis=0)

    def _step_func(self, func, t, dt, y):
        k1 = func(y, t)
        half_step = t + dt / 2
        dt_cast = tf.cast(dt, y.dtype) # Failsafe

        k2 = func(y + dt_cast * k1 / 2, half_step)
        k3 = func(y + dt_cast * k2 / 2, half_step)
        k4 = func(y + dt_cast * k3, t + dt)
        return tf.add_n([k1, 2 * k2, 2 * k3, k4]) * (dt_cast / 6)

def odeint(func, y0, t, n_, F_b):
    t = tf.convert_to_tensor(t, preferred_dtype=tf.float64, name='t')
    y0 = tf.convert_to_tensor(y0, name='y0')
    tf_check_type(y0,t)
    return _Tf_Integrator(n_, F_b).integrate(func,y0,t)
```

    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:101: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
    Instructions for updating:
    non-resource variables are not supported in the long term

#### Implementing a network of Hodgkin-Huxley Neurons

Recall, each Hodgkin Huxley Neuron in a network with $n$ neurons has 4 dynamical variables $V$, $ m$, $n$, $h$. Each of these variables were represented as $n$-dimensional vectors. Now we need to add some more state variables representing each synapse. The neuron receives excitatory and inhibitory inputs that are introduced as additional synaptic currents $I_{ach}$ and $I_{GABA}$. The equation now reads,

\begin{eqnarray}C_m\frac{dV}{dt} = I_{injected} - I_{Na} - I_K - I_L - I_{ach} - I_{gaba}\end{eqnarray}

For each synapse, we have:

\begin{eqnarray}
\frac{d[O]_{ach/gaba}}{dt} = \alpha (1-[O]_{ach/gaba})[T]_{ach/gaba}-\beta[O]_{ach/gaba}
\end{eqnarray}

\begin{eqnarray}
I_{ach/gaba}(t)=g_{max}[O]_{ach/gaba}(V−E_{ach/gaba})
\end{eqnarray}

#### Synaptic Memory Management

As discussed earlier, there are atmost $n^2$ synapses of each type but at a time, unless the network is fully connected/very dense, mostly we need a very small subset of these synapses. We could, in principle, calculate the dynamics of all $n^2$ but it would be pointless. So have to devise a matrix based sparse-dense coding system for evaluating the dynamics of these variables and also using their values. This will reduce memory usage and minimize number of calculations at the cost of time for encoding and decoding into dense data from sparse data and vice versa. This is why we use a matrix approach, so that tensorflow can speed up the process.

#### Defining the Connectivity

In a network with n neurons, there are at most $n^2$ synapses of each type. The actual number may be much smaller. To illustrate the details of the implementation, consider the following three neuron network. Let $X_1$ be an excitatory neuron that forms a cholinergic synapse, $X_2$ an inhibitory neuron that extends a GABAergic synapse onto $X_3$. The network has the form: $X_1 \rightarrow X_2 \rightarrow X_3$. In defining the connectivity matrix for each synapse type, we set a convention where the presynaptic neurons are indexed by the column number, and the postsynaptic neurons by the row number. Let $X_1$, $X_2$, $X_3$ be indexed as 0, 1 and 2 respectively. The excitatory connectivity matrix takes the form

$$Ach_{n\times n}=
\begin{bmatrix}
0&0&0\\
1&0&0\\
0&0&0\\
\end{bmatrix}
$$

Similarly, the inhibitory connectivity matrix becomes:

$$GABA_{n\times n}=
\begin{bmatrix}
0&0&0\\
0&0&0\\
0&1&0\\
\end{bmatrix}
$$

In the following code we specify the parameters of the synapse. The number of synapses of each type are determined by adding up all the elements of the connectivity matrix. Other parameters are specified as vectors with values for each of the synapses.

```python
n_n = 3 # number of simultaneous neurons to simulate

# Acetylcholine

ach_mat = np.zeros((n_n,n_n))        # Ach Synapse Connectivity Matrix
ach_mat[1,0]=1

## PARAMETERS FOR ACETYLCHLOLINE SYNAPSES ##

n_ach = int(np.sum(ach_mat))     # Number of Acetylcholine (Ach) Synapses
alp_ach = [10.0]*n_ach           # Alpha for Ach Synapse
bet_ach = [0.2]*n_ach            # Beta for Ach Synapse
t_max = 0.3                          # Maximum Time for Synapse
t_delay = 0                          # Axonal Transmission Delay
A = [0.5]*n_n                        # Synaptic Response Strength
g_ach = [0.35]*n_n                   # Ach Conductance
E_ach = [0.0]*n_n                    # Ach Potential

# GABAa

gaba_mat = np.zeros((n_n,n_n))       # GABAa Synapse Connectivity Matrix
gaba_mat[2,1] = 1

## PARAMETERS FOR GABAa SYNAPSES ##

n_gaba = int(np.sum(gaba_mat)) # Number of GABAa Synapses
alp_gaba = [10.0]*n_gaba       # Alpha for GABAa Synapse
bet_gaba = [0.16]*n_gaba       # Beta for GABAa Synapse
V0 = [-20.0]*n_n                     # Decay Potential
sigma = [1.5]*n_n                    # Decay Time Constant
g_gaba = [0.8]*n_n                  # fGABA Conductance
E_gaba = [-70.0]*n_n                # fGABA Potential

```

#### Defining Firing Thresholds

We shall also define a list that stores the firing threshold for every neuron.

```python
## Storing Firing Thresholds ##

F_b = [0.0]*n_n                      # Fire threshold
```

#### Defining Input Current as function of Time

We can store our input to each neuron as a $n\times timesteps$ matrix, say current_input, and extract the input at each time point during dynamical update step using a function which we shall call I_inj_t(t).

```python
def I_inj_t(t):
    # Turn indices to integer and extract from matrix
    index = tf.cast(t/epsilon,tf.int32)
    return tf.constant(current_input.T,dtype=tf.float64)[index]
```

#### Defining the connectivity matrix and calculating currents

For updating the dynamics of synapses, we need only as many variables as the number of synapses $\times$ number of equations required for each synapse. Here our synapse models require only one dynamical variable, the fraction of open channels $[O]$, that we store as an $k$—dimensional vector, where $k$ is the number of synapses. There are two instances where the $[O]$ vector is used. First, to solve the equation for $[o]$ and second, to calculate the synaptic current given by,

$$I_{syn} = \sum_{presynaptic} g_{syn}[O](V-E_{syn})$$

The best way to represent this calculation is to use the connectivity matrix $\mathbf{C}$ for the synapses and the open fraction vector $\vec{[O]}$ to create an open fraction matrix $\mathbf{O}$ and perform the following computations.

$$\mathbf{C}=
\begin{bmatrix}
0&1&...&0\\
0&0&...&1\\
...&...&...&1\\
1&0&0&0
\end{bmatrix}\ \ \ \ \ \ \ \ \vec{[O]}=[O_1,O_2...O_k]\ \ \ \ \ \ \ \
\mathbf{O}=
\begin{bmatrix}
0&O_1&...&0\\
0&0&...&O_a\\
...&...&...&O_b\\
O_k&0&0&0
\end{bmatrix}
$$

$$\vec{[I_{syn}]}=\sum_{columns}\mathbf{O}\diamond(\vec{g}_{syn}\odot(\vec{V}-\vec{E}_{syn}))$$

where $\diamond$ is columnwise multiplication and $\odot$ is elementwise multiplication. $\vec{[I_{syn}]}$ is now the total synaptic current input to the each of the neurons.

##### Steps to calculate the Synaptic Currents

1. Firstly we need to convert from the sparse $[O]$ vector to the dense $\mathbf{O}$ matrix. TensorFlow does not allow to make changes to a defined tensor directly, thus we create a $n^{2}$ vector TensorFlow variable o\_ which we will later reshape to a $n\times n$ matrix.
2. We then flatten the synaptic connectivity matrix and find the indices where there is a connection. For this we use the boolean mask function to choose the correct k (total number of synapses) indices from the range $1$ to $n^2$ and store in the variable ind.
3. Using the scatter\_update function of TensorFlow, we fill the correct indices of the variable o\_ that we created with the values of open fraction from the $[O]$ vector.
4. We now reshape the vector as a $n\times n$ matrix. Since python stores matrices as array of arrays, with each row as an inner array, for performing columnswise multiplication, the easiest way is to tranpose the matrix, so that each column is the inner array, perform element wise multiplication with each inner array and apply transpose again.
5. Finally using reduce\_sum, we sum over the columns to get our $I_{syn}$ vector.

```python
## Acetylcholine Synaptic Current ##

def I_ach(o,V):
    o_ = tf.Variable([0.0]*n_n**2,dtype=tf.float64)
    ind = tf.boolean_mask(tf.range(n_n**2),ach_mat.reshape(-1) == 1)
    o_ = tf.scatter_update(o_,ind,o)
    o_ = tf.transpose(tf.reshape(o_,(n_n,n_n)))
    return tf.reduce_sum(tf.transpose((o_*(V-E_ach))*g_ach),1)

## GABAa Synaptic Current ##

def I_gaba(o,V):
    o_ = tf.Variable([0.0]*n_n**2,dtype=tf.float64)
    ind = tf.boolean_mask(tf.range(n_n**2),gaba_mat.reshape(-1) == 1)
    o_ = tf.scatter_update(o_,ind,o)
    o_ = tf.transpose(tf.reshape(o_,(n_n,n_n)))
    return tf.reduce_sum(tf.transpose((o_*(V-E_gaba))*g_gaba),1)

# For TensorFlow v2.x, use tf.tensor_scatter_nd_update instead of
# tf.scatter_update and make o_ a tf.constant istead of a tf.variable
# Also ind must be reshaped to be a column matrix.

## Other Currents ##

def I_K(V, n):
    return g_K  * n**4 * (V - E_K)

def I_Na(V, m, h):
    return g_Na * m**3 * h * (V - E_Na)

def I_L(V):
    return g_L * (V - E_L)
```

#### Updating Synaptic Variables

To update the synapses we first calculate the values of the presynaptic activation function $[T]$ for both types of synapses. This function determines whether a neuron fired or not and is calculated for each neuron. The values of $[T]$ are then sent to the correct synapses in the form of a $k \times 1$ vector. Recall:

$$[T]_{ach} = A\ \Theta(t_{max}+t_{fire}+t_{delay}-t)\ \Theta(t-t_{fire}-t_{delay})$$

$$[T]_{gaba} = \frac{1}{1+e^{-\frac{V(t)-V_0}{\sigma}}}$$

Once we calculate the values of $[T]$-vector for both types of synapse, we need to redirect them to the correct synapses in a sparse $k \times 1$ vector form.

##### Steps to calculate [T]

1. To calculate $[T]_{ach}$, use a boolean logical_and function to check is the current timepoint t is greater than the last fire time (fire\_t) + delay (t\_delay) and less than last fire time (fire\_t) + delay (t\_delay) + activation length (t\_max) for each neuron as a vector. Use the result of these boolean operations to choose between zero or an constant A. This serves as the heaviside step function. For $[T]_{gaba}$, simply use the V vector to determine T.
2. To determine the [T] vector, we follow as two step process. First we multiply each row of the connectivity matrices $\mathbf{C}$ with the respective $[T]$ vector to get a activation matrix $\mathbf{T}$, and then we just flatten $\mathbf{T}$ and $\mathbf{C}$ and, using tf.boolean\_mask, remove all the zeros from $\mathbf{T}$ to get a $k\times1$ vector which now stores the presynaptic activation for each of the synapses where $k=n_{gaba}$ or $n_{ach}$
3. Calculate the differential change in the open fractions using the $k\times1$ vector.

```python
def dXdt(X, t):
    V = X[:1*n_n]       # First n_n values are Membrane Voltage
    m = X[1*n_n:2*n_n]  # Next n_n values are Sodium Activation Gating Variables
    h = X[2*n_n:3*n_n]  # Next n_n values are Sodium Inactivation Gating Variables
    n = X[3*n_n:4*n_n]  # Next n_n values are Potassium Gating Variables
    o_ach = X[4*n_n : 4*n_n + n_ach] # Next n_ach values are Acetylcholine Synapse Open Fractions
    o_gaba = X[4*n_n + n_ach : 4*n_n + n_ach + n_gaba] # Next n_gaba values are GABAa Synapse Open Fractions
    fire_t = X[-n_n:]   # Last n_n values are the last fire times as updated by the modified integrator

    dVdt = (I_inj_t(t) - I_Na(V, m, h) - I_K(V, n) - I_L(V) - I_ach(o_ach,V) - I_gaba(o_gaba,V)) / C_m

    ## Updation for gating variables ##

    m0,tm,h0,th = Na_prop(V)
    n0,tn = K_prop(V)

    dmdt = - (1.0/tm)*(m-m0)
    dhdt = - (1.0/th)*(h-h0)
    dndt = - (1.0/tn)*(n-n0)

    ## Updation for o_ach ##

    A_ = tf.constant(A,dtype=tf.float64)
    Z_ = tf.zeros(tf.shape(A_),dtype=tf.float64)

    T_ach = tf.where(tf.logical_and(tf.greater(t,fire_t+t_delay),tf.less(t,fire_t+t_max+t_delay)),A_,Z_)
    T_ach = tf.multiply(tf.constant(ach_mat,dtype=tf.float64),T_ach)
    T_ach = tf.boolean_mask(tf.reshape(T_ach,(-1,)),ach_mat.reshape(-1) == 1)

    do_achdt = alp_ach*(1.0-o_ach)*T_ach - bet_ach*o_ach

    ## Updation for o_gaba ##

    T_gaba = 1.0/(1.0+tf.exp(-(V-V0)/sigma))
    T_gaba = tf.multiply(tf.constant(gaba_mat,dtype=tf.float64),T_gaba)
    T_gaba = tf.boolean_mask(tf.reshape(T_gaba,(-1,)),gaba_mat.reshape(-1) == 1)

    do_gabadt = alp_gaba*(1.0-o_gaba)*T_gaba - bet_gaba*o_gaba

    ## Updation for fire times ##

    dfdt = tf.zeros(tf.shape(fire_t),dtype=fire_t.dtype) # zero change in fire_t

    out = tf.concat([dVdt,dmdt,dhdt,dndt,do_achdt,do_gabadt,dfdt],0)
    return out

```

#### Updaing the Gating Variable and the Initial Conditions

As before, we again define functions that return the values of $\tau_m$, $\tau_h$, $\tau_n$, $m_0$, $h_0$, $n_0$, set parameters and initial conditions.

Note: The last firing times are stored in the $n$ elements of the state vector. If we initialize the last firing time as 0, then the second neuron $X_2$ will get an EPSP immediately after the start of the simulation. To avoid this the last firing time should be initialized to a large negative number greater than the duration of the simulation.

```python
def K_prop(V):
    T = 22
    phi = 3.0**((T-36.0)/10)
    V_ = V-(-50)

    alpha_n = 0.02*(15.0 - V_)/(tf.exp((15.0 - V_)/5.0) - 1.0)
    beta_n = 0.5*tf.exp((10.0 - V_)/40.0)

    t_n = 1.0/((alpha_n+beta_n)*phi)
    n_0 = alpha_n/(alpha_n+beta_n)

    return n_0, t_n

def Na_prop(V):
    T = 22
    phi = 3.0**((T-36)/10)
    V_ = V-(-50)

    alpha_m = 0.32*(13.0 - V_)/(tf.exp((13.0 - V_)/4.0) - 1.0)
    beta_m = 0.28*(V_ - 40.0)/(tf.exp((V_ - 40.0)/5.0) - 1.0)

    alpha_h = 0.128*tf.exp((17.0 - V_)/18.0)
    beta_h = 4.0/(tf.exp((40.0 - V_)/5.0) + 1.0)

    t_m = 1.0/((alpha_m+beta_m)*phi)
    t_h = 1.0/((alpha_h+beta_h)*phi)

    m_0 = alpha_m/(alpha_m+beta_m)
    h_0 = alpha_h/(alpha_h+beta_h)

    return m_0, t_m, h_0, t_h

# Initializing the Parameters

C_m = [1.0]*n_n
g_K = [10.0]*n_n
E_K = [-95.0]*n_n

g_Na = [100]*n_n
E_Na = [50]*n_n

g_L = [0.15]*n_n
E_L = [-55.0]*n_n

# Initializing the State Vector

y0 = tf.constant([-71]*n_n+[0,0,0]*n_n+[0]*n_ach+[0]*n_gaba+[-9999999]*n_n,dtype=tf.float64)
```

#### Creating the Current Input and Run the Simulation

We will run an 700 ms simulation with 100ms current injection at neuron $X_1$ of increasing amplitude with 100ms gaps in between.

```python
epsilon = 0.01
t = np.arange(0,700,epsilon)

current_input= np.zeros((n_n,t.shape[0]))
current_input[0,int(100/epsilon):int(200/epsilon)] = 2.5
current_input[0,int(300/epsilon):int(400/epsilon)] = 5.0
current_input[0,int(500/epsilon):int(600/epsilon)] = 7.5

state = odeint(dXdt,y0,t,n_n,F_b)

with tf.Session() as sess:
    # Since we are using variables we have to initialize them
    tf.global_variables_initializer().run()
    state = sess.run(state)
```

#### Visualizing and Interpreting the Output

We overlay the voltage traces for the three neurons and observe the dynamics.

```python
plt.figure(figsize=(12,3))

plt.plot(t,state[:,0],label="$X_1$")
plt.plot(t,state[:,1],label="$X_2$")
plt.plot(t,state[:,2],label="$X_3$")
plt.title("Simple Network $X_1$ --> $X_2$ --◇ $X_3$")
plt.ylim([-90,60])
plt.xlabel("Time (in ms)")
plt.ylabel("Voltage (in mV)")
plt.legend()

plt.tight_layout()
plt.show()
```

We can see that the current injection triggers the firing of action potentials with increasing frequency with increasing current. Also we see that as soon as the first neuron $X_1$ crosses its fireing threshold, an EPSP is triggered in the next neuron $X_2$ causing a firing with a slight delay from the firing of $X_1$. Finally, as the second neuron depolarizes, we see a corresponding hyperpolarization in the next neuron $X_3$ caused by an IPSP. We can also plot the dynamics of the channels itself by plotting o\_ach and o\_gaba which are the 5th and 4th last elements respectively.

```python
plt.figure(figsize=(12,3))

plt.plot(t,state[:,-5],label="$[O]_{ach}$ in $X_2$")
plt.plot(t,state[:,-4],label="$[O]_{gaba}$ in $X_3$")
plt.title("Channel Dynamics in $X_1$ --> $X_2$ --◇ $X_3$")
#plt.ylim([-90,60])
plt.xlabel("Time (in ms)")
plt.ylabel("Fraction of Channels open")
plt.legend()

plt.tight_layout()
plt.show()
```

Thus we are now capable of making complex networks of neurons with both excitatory and inhibitory connections.

# References

(Destexhe, Mainen et al., 1994) Destexhe Alain, Mainen Zachary F. and Sejnowski Terrence J., ``_Synthesis of models for excitable membranes, synaptic transmission and neuromodulation using a common kinetic formalism_'', Journal of Computational Neuroscience, vol. 1, number 3, pp. 195--230,  1994.

Tutorial/Day 4 Neurons and Networks/Day 4.ipynb


# Quaternion Series Quantum Mechanics Doodles

This notebook is about initially playing around with quaternion series as a .

```python
%%capture
%matplotlib inline
import numpy as np
import sympy as sp
import pandas as pd
import matplotlib.pyplot as plt

# To get equations the look like, well, equations, use the following.
from sympy.interactive import printing
printing.init_printing(use_latex=True)
from IPython.display import display

# Tools for manipulating quaternions.
import Q_tools as qt;
```

Start with 4 simple states:

```python
A = qt.QHStates([qt.QH([4,0,0,0]), qt.QH([0,1,0,0])])
B = qt.QHStates([qt.QH([0,0,1,0]), qt.QH([0,0,0,2]), qt.QH([0,3,0,0])])
Op = qt.QHStates([qt.QH([3,0,0,0]), qt.QH([0,1,0,0]), qt.QH([0,0,2,0]), qt.QH([0,0,0,3]), qt.QH([2,0,0,0]), qt.QH([0,4,0,0])])
Op4i = qt.QHStates([qt.QH([0,4,0,0])])
print("A:\n", A)
print("B:\n", B)
print("Op:\n", Op)
print("Op4i:\n", Op4i)
```

What I did with these 4 states was to calculate the combinations using good old paper and pencil. So I did:

<A|A>
(A|A) - my notation for *not* taking the conjugate of the first A
<B|B>
(B|B)
Op|B>=Op|B)
<A|Op
(A|Op
<A|Op|B>
(A|Op|B)

The doodling took one page of work. The work can then be checked in this notebook using variations of a command run like so:

```python
print(qt.QHStates().product(bra=A, ket=B))
```

Good, spots errors.

```python
print(qt.QHStates().product(bra=A, ket=B, operator=Op))
```

I was able to confirm this result was correct. In general though, this stuff is going to get insanely complicated quickly.

## Bonus Problem

See if you can write out an operator that is Hermitian. I have a very specific guess in mind, but have yet to try it.

```python

```


```python
%matplotlib inline
```

# 592B, Class 3.2 (09/16). Fourier series

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.io.wavfile as wavfile

from ipywidgets import interactive
from IPython.display import Audio, display
```

## Review questions

**RQ1. Why didn't the code below work as expected? Can you make some minimal changes to get it working as expected? Hint 1: You might try incrementing the value of `t_stop` and plotting points rather than a line. Hint 2: What happens when you comment out the `plt.xlim` line? .**

```python
fs, hmong_data = wavfile.read('hmong_m6_24_c.wav')
t_start = 0; t_stop = 1

plt.plot(hmong_data[round(t_start*fs):round(t_stop*fs)])
plt.xlim(t_start, t_stop)
plt.ylabel("Amplitude")
plt.xlabel("Time (s)")
# set the title
plt.title("Hmong wave file")
```

**RQ2. Lisa was right on target when she made her comment about asymmetric waveforms arising from varying phase in adding up sinusoids. See for instance [this post](https://www.soundonsound.com/sound-advice/q-why-do-waveforms-sometimes-look-lop-sided) and [this post](http://www.audiomisc.co.uk/asymmetry/asym.html). (The posts are both correct but also misleading in a way we'll discuss when we go over the review questions.) Based on these posts, what do people mean when they are using the term "asymmetric"? Taking the two sinusoids we added up in our last class, what happens to the shape of the waveform when you vary the phase shift between them? Can you make the resulting wave symmetric and/or asymmetric? How does changing the relative phase affect how the superposition of the sinuosids sounds when you play the resulting audio?**

```python
# Here's the code we had last time, to get you started.

fs = 44100 # define the sampling rate, f_s = 44.1 kHz

t_start = 0 # We start sampling at t = 0s
t_stop =  1 # We stop sampling at t = 1s

ns = (t_stop - t_start) * fs - 1

x = np.linspace(t_start, t_stop, ns)

f1 = 440 # frequency of y_1(t)
f2 = 440 # frequency of y_2(t)

y1 = np.sin(2*np.pi*f1*x)
y2 = np.sin(-2*np.pi*f2*x)
```

    0.0

```python
plt.figure("Adding up sines") # Create a new figure

plt.xlim(0,0.01)
plt.plot(x , y1, "-g", label="y1") # plot (x,y1) as a green line
plt.plot(x , y2, "-b", label="y2") # plot (x,y2) as a blue line
plt.plot(x , y1_plus_y2, "-r", label="y1+y2") # plot (x,y2) as a blue line
plt.plot(x,0*x, "--") # I added this so you can see the y-axis

#plt.stem(x,y1, 'r', )
plt.legend(loc="upper right")
plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')
plt.title("Adding up sines")
```

*End review questions*

Below is some code that creates an interactive "widget" for both plotting and playing two sine waves and their superposition (sum).

You need to import these libraries, as we did at the beginning of the notebook:
```python
from ipywidgets import interactive
from IPython.display import Audio, display
```
The code is inspired by this [beat frequencies demo](https://ipywidgets.readthedocs.io/en/stable/examples/Beat%20Frequencies.html).

```python
def plot_play_summed_sines(f1 = 440, f2 = 880, t_start = 0, t_stop = 2, fs = 44100, xlim_max = 0.01):
    x = np.linspace(t_start, t_stop, fs * (t_stop - t_start))
    y1 = np.sin(2*np.pi*f1*x)
    y2 = np.sin(2*np.pi*f2*x)

    plt.xlim(t_start,xlim_max)
    plt.plot(x , y1, "-g", label="y1")
    plt.plot(x , y2, "-b", label="y2")
    plt.plot(x , y1 + y2, "-r", label="y1+y2")
    plt.legend(loc="upper right")
    plt.xlabel('Time (s)')
    plt.ylabel('Amplitude (dB)')
    plt.title("Adding up sines")

    display(Audio(data=y1, rate=fs))
    display(Audio(data=y2, rate=fs))
    display(Audio(data=y1+y2, rate=fs))
```

```python
v = interactive(plot_play_summed_sines, f1=(100,900), f2=(100,900), t_start = (0,0), t_stop = (0,5))
display(v)
```

    interactive(children=(IntSlider(value=440, description='f1', max=900, min=100), IntSlider(value=880, descripti…

## A first look at Fourier series

Recall from last week that a standard definition of a sinusoidal signal (i.e., a sine wave) is given as function of time $t$:

$$y(t) = A \sin (\omega t + \phi) $$

where:
- $A$ is the amplitude
- $\omega$ is the angular frequency, n.b., $\omega = 2\pi f$ (where $f$ is the frequency in cycles per second (Hertz or Hz))
- $\phi$ is the phase shift

One standard definition of a Fourier series is:

\begin{equation}
      f(t) = a_0 + \displaystyle\sum\limits_{n=1}^N \left(a_n\cos(2\pi nt) + b_n\sin(2\pi nt)\right)
\end{equation}

where

$n \in \mathbb{Z}$, and $a_0 \ldots a_N$ and $b_0 \ldots b_N$ are called **Fourier coefficients**. Note that these are amplitude values for each component sinusoidal function in the sum: they provide weights for the individual components in the sum. We'll see later that these determine the amplitude of peaks in a spectrum of $f(t)$.

***Discussion: Why is there no $b_0$ in the formula?***

Let's see what each of the individual $a_n\cos(2\pi nt) + b_n\sin(2\pi nt)$ terms look like, for $a_n = b_n = 1$ for all $n$, for $n <6$, otherwise 0. And let's include the $a_0$ term by starting at $N=0$.

```python
# Inspired by https://matplotlib.org/examples/pylab_examples/subplots_demo.html Three subplots sharing both x/y axes

t_start = 0; t_stop = 1; fs = 1000
x = np.linspace(t_start, t_stop, fs * (t_stop - t_start))
y = [np.sin(2*np.pi*n*x) + np.cos(2*np.pi*n*x) for n in np.arange(0,6)]

f_fourier_series, (ax1, ax2, ax3, ax4, ax5, ax6) = plt.subplots(6, sharex=True, sharey=True)
ax1.plot(x , y[0], "-", label="n = 0")
ax1.set_title('Fourier series up to N = 5, a_n = b_n = 1')
ax2.plot(x , y[1], "-", label="n = 1")
ax3.plot(x , y[2], "-", label="n = 2")
ax4.plot(x , y[3], "-", label="n = 3")
ax5.plot(x , y[4], "-", label="n = 4")
ax6.plot(x , y[5], "-", label="n = 5")

plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')

# Fine-tune figure; make subplots close to each other and hide x ticks for
# all but bottom plot.
f_fourier_series.subplots_adjust(hspace=0)
plt.setp([a.get_xticklabels() for a in f_fourier_series.axes[:-1]], visible=False)

```

These are the first 6 terms in a **set of basis functions**, in particular, the **Fourier basis functions**. Each term, for some given $N$, is a basis function. In the limit, as $N \rightarrow \infty$ we can approximate any function with this set of basis functions. And each of the basis functions are **orthogonal** to one another. For intuition, we'll review vector spaces and orthogonal vectors.

Below, we "multiply" one basis function by another to illustrate orthogonality: the integral of the product of any two of the basis functions is 0.

```python
t_start = 0; t_stop = 3; fs = 1000
x = np.linspace(t_start, t_stop, fs * (t_stop - t_start))
y = [np.sin(2*np.pi*f*x) + np.cos(2*np.pi*f*x) for f in np.arange(0,6)]

f_ortho, (ax1, ax2, ax3, ax4, ax5, ax6) = plt.subplots(6, sharex=True, sharey=True)
ax1.plot(x , y[1]*y[2], "-", label="n = 0")
ax1.plot((t_start, t_stop), (0, 0), 'r-')
ax1.set_title('Fourier series up to N = 5, a_n = b_n = 1')
ax2.plot(x , y[1]*y[3], "-", label="n = 1")
ax2.plot((t_start, t_stop), (0, 0), 'r-')
ax3.plot(x , y[2]*y[3], "-", label="n = 2")
ax3.plot((t_start, t_stop), (0, 0), 'r-')
ax4.plot(x , y[3]*y[4], "-", label="n = 3")
ax4.plot((t_start, t_stop), (0, 0), 'r-')
ax5.plot(x , y[4]*y[5], "-", label="n = 4")
ax5.plot((t_start, t_stop), (0, 0), 'r-')
ax6.plot(x , y[5]*y[5], "-", label="n = 5")
ax6.plot((t_start, t_stop), (0, 0), 'r-')

plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')

# Fine-tune figure; make subplots close to each other and hide x ticks for
# all but bottom plot.
f_ortho.subplots_adjust(hspace=0.5)
plt.setp([a.get_xticklabels() for a in f_ortho.axes[:-1]], visible=False)

```

In your homework, you'll explore building up a complex wave as a Fourier series.

## Determining Fourier coefficients

We'll go over more about determining Fourier coefficients next week, and you'll learn more about them when you read *Who is Fourier? Ch. 1-3*. But for now, I'll just bequeath you formulae for computing these: they involve integration. If you need a review of integration, *Who is Fourier?* is excellent for providing geometric intuition---read it!

\begin{equation}
\label{eq:a0}
a_0 = \frac{1}{T}\int_0^T f(t)dt
\end{equation}

\begin{equation}
\label{eq:an}
a_n = \frac{2}{T}\int_0^T f(t)\cos 2\pi ntdt
\end{equation}

\begin{equation}
\label{eq:bn}
b_n = \frac{2}{T}\int_0^T f(t)\sin 2\pi ntdt
\end{equation}

where $T$ is the period of $f(t)$.

It turns out that there is a way to do symbolic math in python, using the `sympy` library.

Let's examine a Fourier series up to $N=2$ with the following coefficient values:

\begin{align}
a_0 = 8\\
a_1 = 3\\
b_1 = 2\\
a_2 = 2\\
b_2 = 4\\
\end{align}

Knowing that those are the coefficient values, we'll now try to compute $a_0$ using the formula above, and see if we get 8.

```python
# http://www.scipy-lectures.org/packages/sympy.html#integration
import sympy as sym # note: you may need to run conda install sympy in your terminal

x = sym.Symbol('x')
sym.integrate(8*sym.cos(2*sym.pi*0*x) + 0*sym.sin(2*sym.pi*0*x)
              + 3*sym.cos(2*sym.pi*1*x) + 2*sym.sin(2*sym.pi*1*x)
              + 2*sym.cos(2*sym.pi*2*x) + 4*sym.sin(2*sym.pi*2*x),
              (x, 0, 1))
```

$\displaystyle 8$

OK now let's try calculating *b2*.

```python
x = sym.Symbol('x')
2* sym.integrate(sym.sin(2*sym.pi*2*x) * (8*sym.cos(2*sym.pi*0*x) + 0*sym.sin(2*sym.pi*0*x)
              + 3*sym.cos(2*sym.pi*1*x) + 2*sym.sin(2*sym.pi*1*x)
              + 2*sym.cos(2*sym.pi*2*x) + 4*sym.sin(2*sym.pi*2*x)),
              (x, 0, 1))
```

$\displaystyle 4$


#New Plotting Framework for SymPy

## Structure of the Module

This module implements a new plotting framework for SymPy. The central class of the module is the `Plot` class that connects the data representations (subclasses of `BaseSeries`) with different plotting backends. It's not imported by default for backward compatibility with the old module.

Then there are the `plot_*()` functions for plotting different kinds of plots and is better suited for interactive work.

* ``plot``: Plots line plots in 2D.

* ``plot_parametric``: Plots parametric line plots in 2D.

* ``plot_implicit`` : Plots implicit equations and region plots in 2D

* ``plot3d`` : Plots functions of two variables in 3D

* ``plot3d_parametric_line``: Plots line parametric plots in 3D

* ``plot3d_parametric_surface``   : Plots surface parametric plots of functions with two variables in 3D.

##General examples

```
from sympy.plotting import plot, plot_parametric, plot3d, plot3d_parametric_line, plot3d_parametric_surface
```

```
p = plot(x)
```

```
p # the Plot object
```

```
p[0] # one of the data series objects
```

```
p[0].label # an option of the data series
```

```
p.legend # a global option of the plot
```

```
p.legend = True

p.show()
```

You can plot 2D different functions in the same plot.

```
p1 = plot_parametric(x*sin(x),x*cos(x), show=False)

p1.extend(p) # Plot objects are just like lists.

p1.show()
```

```
p1.legend = True

p1.show()
```

```
p1[0].line_color='r'

p1[1].line_color='b' # a constant color

p1.show()
```

```
p1[0].line_color = lambda a : a # color dependent on the parameter

p1.show()
```

```
p1.title = 'Big title'

p1.xlabel = 'the x axis'

p1[1].label = 'straight line'

p1.show()
```

```
p1.aspect_ratio
```

```
p1.aspect_ratio = (1,1)

p1.xlim = (-15,20)

p1.show()
```

Hm, `xlim` does not work in the notebook. Hopefully it works in IPython.

```
p1._backend.ax.get_xlim()
```

Yeah, the backend got the command, but the `inline` backend does not honour it.

## Adding expressions to a plot

```
p = plot(x)

p
```

```
p.extend(plot(x+1, show=False))

p.show()

p
```

```
p.append(plot(x+3, x**2, show=False)[1])

p.show()

p
```

## Different types of plots

###``plot``

The ``plot`` by default uses an recursive adaptive algorithm to plot line plots. The default depth of recursion is 12, which means the function will be sampled at a maximum of $2^{12}$ points.

```
help(plot)
```

```
plot(sin(x**2)) # plots with adaptive sampling and default range of (-10, 10)
```

You can also specify the depth of the recursion. It is also possible to disable adaptive sampling and use uniform sampling with ``nb_of_points``.

```
plot(sin(x**2), depth=7)  #specifying the depth of recursion.
```

```
plot(sin(x**2), adaptive=False, nb_of_points=500)
```

###``plot_parametric``

``plot_parametric`` uses an recursive adaptive sampling algorithm to plot.

```
help(plot_parametric)
```

```
plot_parametric(cos(x), sin(x))
```

Multiple plots.

```
plot_parametric((cos(x), sin(x)), (x, cos(x)))
```

We can combine parametric and line plots into a single plot.

```
p = plot(sin(x), show=False)

p.extend(plot_parametric(cos(x), sin(x), show=False))

p.show()
```

###``plot3d``

```
help(plot3d)
```

```
plot3d(x*y)
```

```
plot3d(x*y, nb_of_points_x=100, nb_of_points_y=50)
```

###``plot3_parametric_line``

```
help(plot3d_parametric_line)
```

```
plot3d_parametric_line(cos(x), sin(x), x)
```

###``plot3d_parametric_surface``

```
help(plot3d_parametric_surface)
```

```
plot3d_parametric_surface(cos(x + y), sin(x - y), x - y)
```

## Complex values

If complex values are encountered, they are discarded while plotting.

```
plot(sqrt(x), (x, -5, 5))
```

## Textplot

There is also the textplot backend that permits plotting in the terminal.

```
pt = plot(sin(x),show=False)
```

```
pt.backend = plot_backends['text']
```

```
pt.show()
```