# 変分ベイズのめも

### 独立な事象の同時確率
xとyが排反である場合
$$
\begin{align}
P(x,y) &= P(x)P(y)
\end{align}
$$

 したがって、
 排反なグループが$Z_{i} (i=1,...,M)$に分割されるとき、以下のように定義される。

$$
\begin{align}
q(Z) &= \prod_{i=1}^M q_{i}(Z_{i})
\end{align}
$$

### 期待値
確率変数が取る値を、確率によって重み付けした平均値

#### 確立分布が一様な場合
$X=x_{i}$となる確率が$p_{i}$であるような確率変数$X$を考えたときに期待値$E[X]$は以下のように定義される

$$
\begin{align}
E(X) &= \sum_{i=1}^n p_{i}x_{i}
\end{align}
$$

#### 確立分布が一様ではない場合
確立分布$P(x)$に対して、ある関数$f(x)$の期待値は以下のように定義される

$$
\begin{align}
<f(x)>_{p(x)} &= \int p(x)f(x)dx
\end{align}
$$

※確立密度関数は積分することで確率が与えられる

### エントロピー
確立分布$p(x)$に対する次のような期待値を以下のように定義する

$$
\begin{align}
H[p] &= -\int p(x)\ln{p(x)}dx
\end{align}
$$

エントロピーのように関数を引数にとって、値を返すものを汎関数という

### KLダイバージェンス
関数同士の類似度を測るためのものである。0に近くなるほど類似度は高い

$$
\begin{align}
KL[q||p] &= -\int q(Z)\ln{\frac{p(Z|X)}{q(Z)}}dZ
\end{align}
$$

$q(x)$に対する$\frac{p(Z|X)}{q(Z)}$の期待値

### 下界
$$
\begin{align}
KL[q||p] &= -\int q(Z)\ln{\frac{p(Z|X)}{q(Z)}}dZ
\end{align}
$$


# Advanced Expression Manipulation Solutions

```python
from sympy import *
x, y, z = symbols('x y z')
init_printing()
from IPython.display import display
```

For each exercise, fill in the function according to its docstring.

## Creating expressions from classes

Create the following objects without using any mathematical operators like `+`, `-`, `*`, `/`, or `**` by explicitly using the classes `Add`, `Mul`, and `Pow`.  You may use `x` instead of `Symbol('x')` and `4` instead of `Integer(4)`.

$$x^2 + 4xyz$$
$$x^{(x^y)}$$
$$x - \frac{y}{z}$$

```python
def explicit_classes1():
    """
    Returns the expression x**2 + 4*x*y*z, built using SymPy classes explicitly.

    >>> explicit_classes1()
    x**2 + 4*x*y*z
    """
    return Add(Pow(x, 2), Mul(4, x, y, z))
```

```python
explicit_classes1()
```

```python
def explicit_classes2():
    """
    Returns the expression x**(x**y), built using SymPy classes explicitly.

    >>> explicit_classes2()
    x**(x**y)
    """
    return Pow(x, Pow(x, y))
```

```python
explicit_classes2()
```

```python
def explicit_classes3():
    """
    Returns the expression x - y/z, built using SymPy classes explicitly.

    >>> explicit_classes3()
    x - y/z
    """
    return Add(x, Mul(-1, Mul(y, Pow(z, -1))))
```

```python
explicit_classes3()
```

## Nested args

```python
expr = x**2 - y*(2**(x + 3) + z)
```

Use nested `.args` calls to get the 3 in expr.

```python
def nested_args():
    """
    Get the 3 in the above expression.

    >>> nested_args()
    3
    """
    expr = x**2 - y*(2**(x + 3) + z)
    return expr.args[1].args[2].args[1].args[1].args[0]
```

```python
nested_args()
```

## Traversal

Write a post-order traversal function that prints each node.

```python
def post(expr):
    """
    Post-order traversal

    >>> expr = x**2 - y*(2**(x + 3) + z)
    >>> post(expr)
    -1
    y
    2
    3
    x
    x + 3
    2**(x + 3)
    z
    2**(x + 3) + z
    -y*(2**(x + 3) + z)
    x
    2
    x**2
    x**2 - y*(2**(x + 3) + z)
    """
    for arg in expr.args:
        post(arg)
    display(expr)
```

```python
expr = x**2 - y*(2**(x + 3) + z)
```

```python
post(expr)
```


# root_finding_graphical_method
Jalankan kode pada sel-sel di bawah ini sehingga identitas Anda tertampilkan dengan benar dan perkirakan posisi akar dari grafik fungsi yang dibuat.

## graphical method
Ulas kembali secara singkat metode grafik dalam memperkirakan posisi akar suatu fungsi dengan menonton video berikut.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('9xRkoYbXOaU', width=600, height=340)
```

Bila video tidak muncul dapat dilihat di https://www.youtube.com/watch?v=9xRkoYbXOaU

## identity
Koreksi kode pada sel berikut sehingga identitas Anda benar tertampilkan.

```python
from IPython.core.display import HTML
HTML("")

import sys, os
sys.path.insert(1, os.path.join(sys.path[0], '../..'))

from src.student import Students as stu
nim = 10200999
print(nim, end=' ')
print(stu[nim]['name'], end=' ')
print("https://github.com/", stu[nim]['github'], sep='')
```

## function
Sebuah fungsi matematika, misalnya saja

\begin{equation}\tag{1}
f(x) = \left[ x - 2.4 e^{-(x - 2.7)} \right] \sin \tfrac23 \pi (x-1.35)
\end{equation}

dapat digambarkan dengan dalam rentang $x \in [x_{\rm beg}, x_{\rm end}]$.

```python
import math

# define a function of x
def f(x):
    y1 = (x -  2.4 * math.exp(-(x - 2.7)))
    y2 = math.sin(2 * math.pi * (x - 1.35) / 3)
    y = y1 * y2
    return y
```

```python
# create data
xbeg = 0
xend = 10
N = 100
dx = (xend - xbeg) / N

xx = []
yy = []

for i in range(0, N + 1):
    x = xbeg + i * dx
    y = f(x)

    xx.append(x)
    yy.append(y)

```

```python
import matplotlib.pyplot as plt

# plot data
plt.ion()
fig, ax = plt.subplots()
ax.set_xlabel("$x$")
ax.set_ylabel("$y$")
plt.plot(xx, yy)

# draw horizontal line
line = plt.Line2D((xbeg, xend), (0, 0), color='#aaa', lw=1, ls='dashed')
plt.gca().add_line(line)

# show plot result
plt.show()
```

## a root
Dengan memilih rentang antara 5 dan 6, perkirakan posisi akarnya dan tunjukkan melalui penggambaran sebuah lingkaran pada posisi tersebut.

```python
import matplotlib.pyplot as plt
from matplotlib.patches import Ellipse

# plot data
plt.ion()
fig, ax = plt.subplots()
ax.set_xlabel("$x$")
ax.set_ylabel("$y$")
plt.plot(xx, yy)

# draw horizontal line
line = plt.Line2D((xbeg, xend), (0, 0), color='#aaa', lw=1, ls='dashed')
plt.gca().add_line(line)

# zoom
plt.xlim([3, 7])
plt.ylim([-7, 7])

# draw a cricle
xroot = 5.5
point = Ellipse(
    xy=(xroot, 0), width=0.1, height=0.6,
    edgecolor='r', fc='None', lw=1
)
plt.gca().add_patch(point)

# show plot result
plt.show()

print("xroot = ", xroot, sep='')
```

Modifikasi kode di atas sehingga lingkaran merah tepat menunjukkan akar yang terletak antara 5 dan 6.


```python
from sympy import *

F, l = var("F, l")
q = F/l
R = q*l

# Reactions:
MA, Ah, Av, Gh, Gv, Bv = var("MA, Ah, Av, Gh, Gv, Bv")

eq1 = Eq(Ah + Gh)
eq2 = Eq(Av + Gv - 2*R)
eq3 = Eq(MA + 2*l*Gv - l*2*R)

eq4 = Eq(- Gh - F - F)
eq5 = Eq(- Gv - R - F - F + Bv)
eq6 = Eq(- R/2 - F - 2*F + 3*Bv)

eqns = [eq1, eq2, eq3, eq4, eq5, eq6]
unks = [MA, Ah, Av, Gh, Gv, Bv]

sol = solve(eqns, unks)
pprint(sol)

# Sections:
x = var("x")
MA,Ah,Av,Gh,Gv,Bv = \
    sol[MA],sol[Ah],sol[Av],sol[Gh],sol[Gv],sol[Bv]

# Region 1:
N = - Ah
Q = - q*x + Av
M = - MA - q*x*x/2 + Av*x

pprint("\nN, Q, M:")
S = Matrix([N, Q, M])
pprint(S)
pprint("\nAt x = 0:")
pprint(S.subs(x,0))
pprint("\nAt x = 3l:")
pprint(S.subs(x,3*l))

# ⎧             23⋅F      7⋅F                -11⋅F       17⋅F⋅l⎫
# ⎨Ah: 2⋅F, Av: ────, Bv: ───, Gh: -2⋅F, Gv: ──────, MA: ──────⎬
# ⎩              6         6                   6           3   ⎭
#
# N, Q, M:
# ⎡          -2⋅F          ⎤
# ⎢                        ⎥
# ⎢       23⋅F   F⋅x       ⎥
# ⎢       ──── - ───       ⎥
# ⎢        6      l        ⎥
# ⎢                        ⎥
# ⎢                       2⎥
# ⎢  17⋅F⋅l   23⋅F⋅x   F⋅x ⎥
# ⎢- ────── + ────── - ────⎥
# ⎣    3        6      2⋅l ⎦
#
# At x = 0:
# ⎡  -2⋅F  ⎤
# ⎢        ⎥
# ⎢  23⋅F  ⎥
# ⎢  ────  ⎥
# ⎢   6    ⎥
# ⎢        ⎥
# ⎢-17⋅F⋅l ⎥
# ⎢────────⎥
# ⎣   3    ⎦
#
# At x = 3l:
# ⎡-2⋅F ⎤
# ⎢     ⎥
# ⎢ 5⋅F ⎥
# ⎢ ─── ⎥
# ⎢  6  ⎥
# ⎢     ⎥
# ⎢4⋅F⋅l⎥
# ⎢─────⎥
# ⎣  3  ⎦

```


```python
import sympy
t, K ,r, P0, C1 = sympy.symbols('t, K, r, P_0, C_1')
P = sympy.Function('P')
edo = P(t).diff(t) - r * P(t) * (1 - P(t)/K)
edo
```

$\displaystyle - r \left(1 - \frac{P{\left(t \right)}}{K}\right) P{\left(t \right)} + \frac{d}{d t} P{\left(t \right)}$

```python
edo_sol = sympy.dsolve(edo, P(t))
edo_sol
```

$\displaystyle P{\left(t \right)} = \frac{K e^{C_{1} K + r t}}{e^{C_{1} K + r t} - 1}$

```python
ini_cond = {P(0): P0}
ini_cond
```

    {P(0): P_0}

```python
C_eq = edo_sol.subs(t,0).subs(ini_cond)
C_eq
```

$\displaystyle P_{0} = \frac{K e^{C_{1} K}}{e^{C_{1} K} - 1}$

```python
import matplotlib.pyplot as plt
import numpy as np
%config InlineBackend.figure_formats = ['svg']

def logistica(t, P0=100, K=1000, r=0.25):
    A = P0 / (P0 - K)
    return K / (1 - np.exp(-r*t) / A)
```

```python
t = np.linspace(0, 40, 100)  # Intervalo de tiempo en que se obtiene la solución
p1 = plt.plot(t, logistica(t, P0 = 100), label=r'$P_0 = 100$')
p1 = plt.plot(t, logistica(t, P0 = 2000), label=r'$P_0 = 1500$')
plt.legend()
plt.show()
```

```python

```


# Intro to TensorFlow

https://www.youtube.com/watch?v=q5iL3XYFv2M

## Backpropagation on zero hidden layer classification case
Suppose we are required to learn the function that maps $x$ (the inputs) to $y$ (the outputs). In this particular instance we restrict ourselves to the case that $y=\sigma(Wx)$. The maths behind regression is shown below:
\begin{align}
z_i =& Wx_i\\
a_i =& \sigma(z_i)
\end{align}
where $\sigma(z_i) = 1/(1+exp(-z_i))$ is the sigmoid function. $a_i$ is commonly known as the activation. **The loss function** is $$\mathcal{L} = \frac{1}{2N}\sum_i (y_i-a_i)^2$$.

**We need to adjust the $W$ to minimise the loss function**. We use the chain rule:
\begin{align}
\frac{d \mathcal{L}}{dW} =& \sum_i \frac{d\mathcal{L}}{da_i}\frac{da_i}{dz_i}\frac{dz_i}{dW}\\
=& \frac{1}{N}\sum_i -(y_i-a_i)\cdot\frac{\exp(-z)}{(1+\exp(-z))^2}\cdot x_i\\
=& \frac{1}{N}\sum_i -(y_i-a_i)\cdot a_i(1-a_i)\cdot x_i
\end{align}

```python
from IPython.display import HTML

HTML('')

```

    /home/maziyar/anaconda3/envs/tf2/lib/python3.6/site-packages/IPython/core/display.py:694: UserWarning: Consider using IPython.display.IFrame instead
      warnings.warn("Consider using IPython.display.IFrame instead")

```python
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)

%matplotlib inline
```

```python
def sigmoid(z):
    return 1/(1+np.exp(-z))

z = np.arange(-5,5,0.1)

plt.plot(z,sigmoid(z))
plt.plot([0,0],[0,1],'r')
plt.plot([-6,6],[0.5,0.5],'r')
```

```python
N = 1000
D = 5

X = 5*np.random.randn(N,D)
w = np.random.randn(D,1)
y = X.dot(w)

y[y<=0] = 0
y[y>0] = 1
```

```python
train_X = X[1:100]
test_X = X[100:]
```

```python
X.shape
```

    (1000, 5)

```python
train_X.shape
```

    (99, 5)

```python
test_X.shape
```

    (900, 5)

```python
def dL_dw(X,e,a):
    return -X.T.dot(e*a*(1-a))/len(X)

def gradient_descent(gamma=5e-1, n_epochs=1000, batch_size=100, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            a = sigmoid(X[idx].dot(w)) # Activation function
            e = y[idx] - a # Really important that you use y_obs and not y (you do not have access to true y)
            #update parameters
            w = w - gamma*dL_dw(X[idx],e,a)
        loss[i] = 0.5*e.T.dot(e)/len(e)
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],w])
```

    array([[-0.34022325, -0.42375968],
           [-0.38179346, -0.45341411],
           [-1.5158064 , -1.79564317],
           [-0.26698786, -0.33009019],
           [ 0.58082039,  0.73282908]])

```python
idx = np.random.choice(len(X),20,replace=False)
a = sigmoid(X[idx].dot(w)) # Activation function
e = y[idx] - a
0.5*e.T.dot(e)/len(e)
```

    array([[0.00948528]])

```python
# Classification error
X_test = 5*np.random.randn(100,D)
y_test = X_test.dot(w)
y_test[y_test<=0] = 0
y_test[y_test>0] = 1

y_inferred = sigmoid(X_test.dot(params[-1])) # Get a probability measure given X
# y_inferred[y_inferred>0.5] = 1
# y_inferred[y_inferred<=0.5] = 0

# np.sum(y_test==y_inferred)
```

## Cross Entropy Error

We shall focus on doing the same as above with a different loss function, the Cross Entropy Loss function.
$$\mathcal{L} = -\frac{1}{N}\sum_i y_i\log(a_i)+(1-y_i)\log(1-a_i)$$.

The following remains the same:
\begin{align}
z_i =& Wx_i\\
a_i =& \sigma(z_i)
\end{align}
and the derivative,
\begin{align}
\frac{d \mathcal{L}}{dW} =& \sum_i \frac{d\mathcal{L}}{da_i}\frac{da_i}{dz_i}\frac{dz_i}{dW}\\
=& \frac{1}{N}\sum_i -\left(\frac{y_i}{a_i}-\frac{1-y_i}{1-a_i}\right)\cdot\frac{\exp(-z)}{(1+\exp(-z))^2}\cdot x_i\\
=& \frac{1}{N}\sum_i -\left(\frac{y_i-a_i}{a_i(1-a_i)}\right)\cdot a_i(1-a_i)\cdot x_i\\
=& \frac{1}{N}\sum_i -(y_i-a_i)\cdot x_i
\end{align}

```python
# from IPython.core.debugger import Tracer;

def dL_dw(X,e,a):
    return -X.T.dot(e)/len(X)

def gradient_descent(gamma=1e-2, n_epochs=1000, batch_size=100, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    # NOTE: I initialised w such that its variance is 1/D
    w = np.random.randn(D,1)*(1/np.sqrt(D))
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            a = sigmoid(X[idx].dot(w)) # Activation function
            e = y[idx] - a # Really important that you use y_obs and not y (you do not have access to true y)
            w = w - gamma*dL_dw(X[idx],e,a)
#
        loss[i] = -np.mean(y[idx]*np.log(a+1e-10)+(1-y[idx])*np.log(1-a+1e-10))
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],w])
```

    array([[-0.11711174, -0.42375968],
           [-0.11644926, -0.45341411],
           [-0.63267484, -1.79564317],
           [-0.131367  , -0.33009019],
           [ 0.20858427,  0.73282908]])

```python
# Classification error
X_test = 5*np.random.randn(100,D)
y_test = X_test.dot(w)
y_test[y_test<=0] = 0
y_test[y_test>0] = 1

y_inferred = sigmoid(X_test.dot(params[-1])) # Get a probability measure given X
y_inferred[y_inferred>0.5] = 1
y_inferred[y_inferred<=0.5] = 0

np.sum(y_test==y_inferred)
```

    97

## Tensorflow Introduction

```python
import tensorflow as tf
```

```python
X.shape
```

    (1000, 5)

```python
train_X.shape
```

    (99, 5)

```python
def tf_train(X_train, y_train, batch_size=20):
    # Dataset (inputs and labels)
    x = tf.compat.v1.placeholder(tf.float32, [None, D])
    y_ = tf.compat.v1.placeholder(tf.float32, [None, 1])

    # random variable
    W = tf.Variable(tf.random.normal([D, 1],stddev=0.1))
    # map X to inferred output function
    a = tf.sigmoid(tf.matmul(x, W))

    # Define loss and optimizer
    cross_entropy = tf.reduce_mean(input_tensor=-(y_*tf.math.log(a)+(1-y_)*tf.math.log(1-a)))
    train_step = tf.compat.v1.train.GradientDescentOptimizer(1e-2).minimize(cross_entropy)

    sess = tf.compat.v1.InteractiveSession()
    tf.compat.v1.initialize_all_variables().run()
    # Train
    for epoch in range(1000):
        idx = np.random.choice(len(X_train),batch_size,replace=False)
        _,l = sess.run([train_step, cross_entropy], feed_dict={x: X_train[idx], y_: y_train[idx]})
        if epoch%100 == 0:
            print('loss: '+str(l))
```

```python
tf.compat.v1.disable_eager_execution()

tf_train(X,y)
```

    WARNING:tensorflow:From /home/maziyar/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
    Instructions for updating:
    If using Keras pass *_constraint arguments to layers.
    WARNING:tensorflow:From /home/maziyar/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/util/tf_should_use.py:198: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.
    loss: 0.79079616
    loss: 0.17373714
    loss: 0.13240728
    loss: 0.16038704
    loss: 0.104678296
    loss: nan
    loss: nan
    loss: nan
    loss: nan
    loss: nan

```python
def tf_train(X_train, y_train, batch_size=20, n_epoch=1000):
    x = tf.compat.v1.placeholder(tf.float32, [None, D])
    y_ = tf.compat.v1.placeholder(tf.float32, [None, 1])

    W = tf.Variable(tf.random.normal([D, 1],stddev=1/np.sqrt(D)))

    # Define loss and optimizer
    z = tf.matmul(x,W)

    cross_entropy = tf.reduce_mean(input_tensor=tf.nn.sigmoid_cross_entropy_with_logits(logits=z,labels=y_))
    train_step = tf.compat.v1.train.GradientDescentOptimizer(1e-2).minimize(cross_entropy)

    sess = tf.compat.v1.InteractiveSession()
    tf.compat.v1.initialize_all_variables().run()
    # Train
    for epoch in range(n_epoch):
        idx = np.random.choice(len(X_train),batch_size,replace=False)
        _,l = sess.run([train_step, cross_entropy], feed_dict={x: X_train[idx], y_: y_train[idx]})
        if epoch%100 == 0:
            print('loss: '+str(l))

    return sess.run(W)
```

```python
w_est = tf_train(X,y)
```

    WARNING:tensorflow:From /home/maziyar/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    loss: 0.82817715

    /home/maziyar/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/client/session.py:1750: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).
      warnings.warn('An interactive session is already active. This can '

    loss: 0.12993214
    loss: 0.1804026
    loss: 0.185333
    loss: 0.03280472
    loss: 0.1189767
    loss: 0.075235054
    loss: 0.053363986
    loss: 0.099958375
    loss: 0.07475308

```python
np.hstack([w_est,w])
```

    array([[-0.28446072, -0.42375968],
           [-0.34989649, -0.45341411],
           [-1.35862863, -1.79564317],
           [-0.23279138, -0.33009019],
           [ 0.5114302 ,  0.73282908]])

```python
# Classification error
X_test = 5*np.random.randn(100,D)
y_test = X_test.dot(w)
y_test[y_test<=0] = 0
y_test[y_test>0] = 1

y_inferred = sigmoid(X_test.dot(w_est)) # Get a probability measure given X
y_inferred[y_inferred>0.5] = 1
y_inferred[y_inferred<=0.5] = 0

np.sum(y_test==y_inferred)
```

    98

```python

```

jupyter/Keras_TensorFlow_Course/Lesson 03 - TF_intro.ipynb


# "Linear Programming in Economics"
> "An example of how linear programming is used to solve economic constrained maximization problems"

- toc: true
- branch: master
- badges: true
- comments: true
- categories: [Linear Programming, Economic problems]
- image: images/linear_programming.png
- hide: false
- search_exclude: true
- metadata_key1: metadata_value1
- metadata_key2: metadata_value2

```python
import numpy as np
import matplotlib.pyplot as plt
import unittest
import seaborn as sns
sns.set_style('darkgrid')
from sympy import symbols, solve
from scipy.optimize import linprog

```

This is an example of a product portfolio choice problem optimizing the following problem of which a firm is able to produce n different products subject to a set of linear technological constraints and which maximizes a linear profit function:

$$
\max(\pi x) \text{ subject to } Ax \le b,
$$

where $ \pi $ denotes the vector of per product unit profits, and matrix $ A $ together with the
vector $ b $ gives the set of technological constraints.

#### An object orientated approach will be used to describe the space of the problem.

```python

class production_portfolio:

    '''This class describes the model of optimal product portfolio choice
       It takes in constraints and prices with the ability to remove each'''

    prices = []
    #The constraints are added as a class attribute, NOT an instance attribute
    #To delete constraints, must use the remove_constraint method individually or the reset method to remove all
    a_matrix = []
    b_matrix = []
    temp = []

    def __init__(self, n):
        '''Initialize the optimal product portfolio choice model'''
        #n is the number of constraints (not including non-negativity constraints)
        #Took out current specification (label) of the model - not needed for our model
        self.n = n

    def __repr__(self):
        '''String representation for the optimal product portfolio choice model'''

        return ('''These are the constraints: \nA:{}\nb:{}\nThese are the prices: {}\nThere are {} constraints
                '''.format(self.a_matrix, self.b_matrix, self.prices,
                                                      self.nr_constraints()))

    def set_prices(self,p):
        '''Set product prices: p is list of nparray of length n'''
        p = np.array(p)
        self.prices.append(list(abs(p)))
        return self.prices

    def add_constraint(self,coef,b):
        '''
        Adds the coefficients and b values to the model. Does not check if it is feasible.
        Run consistency_check below to check feasibility.

        '''
        self.a_matrix.append(coef)
        self.b_matrix.append(b)
        return self.a_matrix, self.b_matrix

    def inception(matrix):
        '''Converts nested lists to a one-nested list(i.e. a matrix form)
           [[[1,5], [4,6]]] -> [[1,5], [4,6]]. Needs to be in this later form for below function'''
        if isinstance(matrix, list):
            temp.append(matrix)
            matrix = matrix[0]
            inception(matrix)
        if isinstance(matrix, int):
            return temp[-1]
        return temp[-2]

    def remove_constraint(self,j):
        '''Removes the jth row constraint of the model'''
        self.a_matrix = inception(self.a_matrix)
        self.a_matrix = np.delete(self.a_matrix, j, axis=0)
        self.b_matrix = np.delete(self.b_matrix, j, axis=0)
        #If prices not equal to the number of products, this sets the remaining equal to 0. Might not be needed
        if len(self.prices)>0:
            self.prices = np.delete(self.prices,j)
            return self.a_matrix, self.b_matrix, self.prices
        return self.a_matrix, self.b_matrix

    def nr_constraints(self):
        '''Number of constraints in the current model'''
        return (np.array(self.a_matrix).shape[0])

    @classmethod
    def reset(self):
        '''Removes all the constraints in the model from the class attribute'''
        self.a_matrix = []
        self.b_matrix = []
        self.prices = []
```

#### A consistency check for the problem needs to be added. How do you tell if a solution exists within the constraints?

```python
upperbound = 20
grid = np.linspace(0, upperbound, 50)
hypercube = [] #This is the space of x
#Each element in grid is multipled by all the other elements in grid to create our hypercube
#Not a proper hypergrid as the dimensions are (2500,2) but is an unstack of each dimension onto a column
for i in grid:
    for j in grid:
        hypercube.append([j,i])

hypercube = np.array(hypercube)
x = hypercube
```

```python
def consistency_check(a, b,x, verbose=False):

    '''Checks if the system of inequality constraints in the model is consistent'''
    # x is a 2500x2 matrix, a is the constraint matrix
    #Each element in x is checked against the equation a*x <= b
    #The first constraint in which this equation is not true, it exits out of the loop and goes to the next x.
    #Repeats until it finds an x that meets all the constraints, then exits the loop with True else False
    for i in range(len(x)):
        for j in range(len(a)):
            if not a[j]@x[i] <= b[j]:
                break
            if j == (len(a)-1):
                if verbose: return print("This is feasible at: \ni: {}, a: {}, b: {}, j:{}".format(i, a[j], b[j], j))
                else: return True
    if verbose:
        return print("This is not feasible")
    else:
        return False
```

```python

class TestDemo(unittest.TestCase):

    def test1(self):
        '''general feasible constraints'''
        a = [[1,1], [3,2],[-1,0],[0,-1]]
        b = [2,5,0,0]
        test = consistency_check(a,b,x)
        self.assertTrue(test)

    def test2(self):
        '''general feasible constraints'''
        a = [[1,1], [1,0], [-1,0], [0,-1]]
        b = [1,0,0,0]
        test = consistency_check(a,b,x)
        self.assertTrue(test)

    def test3(self):
        '''non-feasible constraints with given x'''
        a = [[1,1], [3,2], [-1, 1], [-1,0], [0,-1]]
        b = [2,5,-5,0,0]
        test = consistency_check(a,b,x)
        self.assertFalse(test) #This is not feasible and should return False

    def test4(self):
        '''non-feasible constraints with given x'''
        a = [[-0.5, -1],[1,1],[-1,0],[0,-1]]
        b = [-12,10,0,0]
        test = consistency_check(a,b,x)
        self.assertFalse(test) #This is not feasible and should return False

if __name__ == '__main__':
    unittest.main(argv=['first-arg-is-ignored'], exit=False)
```

    ....
    ----------------------------------------------------------------------
    Ran 4 tests in 0.061s

    OK

## A Graphical Representation of the linear programming problem

```python

class graph_from_matrix:

    def __init__(self, matrix, x, n, contour=None, nonneg=True, xlim=10,ylim=10, verbose=False, solve=False,
                 geq=None, c = None, overlap = False, optimal_level = False, hide_point=False):
        self.matrix = matrix #Matrix of constraints. Must include non-negativity constraints in here.
        self.x = x #Space of solutions
        self.n = n # Number of constraints to solve
        self.nonneg = nonneg #Automatically fills in nonnegativity constraints if true
        self.xlim = xlim #Sets limits on x
        self.ylim = ylim #Sets limits on y
        self.verbose=verbose
        self.solve = solve #If constraints or objective functions are given, can calculate the optimal point
        self.geq = geq #The shaded region between the constraint and axis is flipped for a particular constraint
        self.contour = contour # True or False
        self.c = c #Vector of prices/objective fuction
        self.optimal_level = optimal_level #Given a specific objective function, shows the specific contour plot
        self.overlap = overlap #Visualization tool to view price contour plot if it overlaps constraint
        self.hide_point = hide_point

    def equation(self, matrix):
        '''Takes in a matrix of A and b values, where the last column is the solution (b vector)
        and finds and equation to solve'''
        # Treats x and y as symbols and not variables
        x_symb = symbols('x'); y_symb=symbols('y')
        expr = []
        #Converts the matrices back into equations to solve
        for eq in range(len(self.matrix)):
            expr.append(self.matrix[eq][0]*x_symb
                        + self.matrix[eq][1]*y_symb
                        - self.matrix[eq][2])

        lines = []
        # Solves the equations for y for each equation found above
        for equation in expr:
            lines.append(solve(equation, y_symb))

        if self.verbose: print("These are the equations to solve: {}".format(lines[:-2]))
        # Takes off the last two values which are the non-negativity constraints. They are 0
        return lines[:-2]

    def yvalue(self):
        '''Solves for y given the above equation and the x's formed from the meshed grid'''
        #Calls above function with the given instanced matrix
        equations = self.equation(self.matrix)
        x = self.x #Needed to evaluate the type str below
        y = []
        #Evaluates y for each equation
        for i in equations:
            y.append(eval(str(i[0])))
        # Dimensions equal to amount of constraints solved. Doesn't solve non-negativity constraints
        return y

    def plot(self):
        '''Plots the equations, the mesh grid, contour plots and optimal point/s'''

        fig, ax1 = plt.subplots()
        plt.rcParams['figure.figsize'] = [12, 8]

        #Only used if the constraint overlaps a specific contour plot given by a price vector
        #Plots the overlapped constraint with a small linewidth and the rest as normal
        if self.overlap:
            for i in range(len(self.yvalue())):
                    if i == 2:
                        plt.plot(self.x, self.yvalue()[i], color='black', linewidth=0.01)
                    else:
                        plt.plot(self.x, self.yvalue()[i], color='black')
        else:
            for i in self.yvalue():
                plt.plot(self.x, i, color='black')

        #Fills in each constraint as necessary, geq flips the fill to the opposite side
        #Colours in the feasible set given the constraints
        for i in range(len(self.yvalue())):
            if i == self.geq:
                ax1.fill_between(self.x,max(self.yvalue()[i]),self.yvalue()[i],alpha=0.33,color='g')
            else:
                ax1.fill_between(self.x,0,self.yvalue()[i],alpha=0.33,color='g')

        #Plots the mesh grid, each dimension of a column is a dimension of a hypergrid. In this case = square
        plt.scatter(x[:,0], x[:,1], s=3)

        # Plots lines on axis if non-negativity constraint is needed
        if self.nonneg:
            plt.plot([0]*self.ylim,range(0,self.ylim),color='black')
            plt.plot(range(0,self.xlim),[0]*self.xlim,color='black')

        #Maximum x and y values are equal to the maximum of the constraint matrices
        plt.xlim(0,(self.matrix).max())
        plt.ylim(0,(self.matrix).max())
        plt.xlabel('x')
        plt.ylabel('y')

        #If the objective function and constraints exists, the equation is solved for the optimal point and graphed
        #If the value of the optimal solution does not exist in our space of values (meshgrid)
        #the nearest point that exists to our optimal solution is found and also graphed
        if self.solve:
            point = solver(n=2,c=c,A=A,b=b)['x']
            if not self.hide_point:
                plt.plot(point[0],point[1],'ro')
                plt.plot(nearest(hypercube, point)[0], nearest(hypercube, point)[1], 'bo')

        if self.contour:
            X,Y = np.meshgrid(grid, grid)
            Z = self.contour[0]*X + self.contour[1]*Y

            #If a specific objective function is given, this graphs only that contour line
            if self.optimal_level:
                self.optimal_level = self.contour[0]*point[0]+self.contour[1]*point[1]
                ax1.contour(X,Y,Z, levels=[self.optimal_level],colors='r', linewidths=3)

            #If no specific objective function, maps out a variety of contours
            else:
                ax1.contour(X,Y,Z, levels=30,colors='r')
        return plt.show()
```

### Example problem with a given a, b

```python
x = hypercube
a = np.array([[1,1], [3,2],[-1,0],[0,-1]]) #Constraints/coefficients
b = np.array([2,5,0,0]).reshape(4,1) #Technological constraints
matrix = np.append(a,b,1)

n = 2
obj = graph_from_matrix(matrix, x[:,0], n, contour=[5,3])
obj.plot()
```

```python
#Find all intersection points of the line segments

def intersection_points(n,*args):

    x_points = []
    y_points = []
    for i in range(len(args)):
        for j in range(len(args)):
            if i != j and j >= i:
                #print("Solving for: {} = {}".format(args[i], args[j]))
                x_points.append(solve(args[i]-args[j],symbols('x')))

    for i in range(len(x_points)):
        x_points[i] = float(x_points[i][0])

    if n == 3:
        temp = x_points.copy()
        x_points[2] = temp[1]
        x_points[1] = temp[2]
    #print(x_points, args, sep='\n')
    for x_values, equation in zip(x_points,args):
        x = x_values
        y_points.append(eval(str(equation)))

    #return x_points
    return list(zip(x_points, y_points))

```

```python
intersection = intersection_points(2, obj.equation(matrix)[0][0], obj.equation(matrix)[1][0])
print("The intersection point of the above graph is x: {}, y: {}".format(intersection[0][0], intersection[0][1]))
```

    The intersection point of the above graph is x: 1.0, y: 1.0

## Given a price vector, what is the optimal value?
### Example:

Let $ n=2 $ and the goods production technology
is restricted by

$$
\begin{cases}
y - x  &\le& 6, \\
2x - y &\le&12,
\end{cases}
$$

In addition, there is a resource constraint given by $ x + 2.5y \le 16 $.

Finally, let profit be given by $ \pi(x,y) = 3y + 5x $.

```python
#Model object created within the function itself - mixing OOP and functional programming
def solver(n, c, A, b, method='simplex'):
    ''' Uses linprog to solve optimal production portfolio
    n is number of goods
    c = the profit function
    A is the matrix of constraints
    B is the technology constraint'''
    f = production_portfolio(n)
    for i in range(len(A)):
        f.add_constraint(A[i],b[i])
    A = f.a_matrix
    b = f.b_matrix
    f.prices = c
    c = np.array(c)
    res = linprog(c, A_ub = A, b_ub = b, method=method)
    return res
```

```python
# Finding the optimal product portfolio given constraints and prices
#Constraints
A = [[-1, 1],[2, -1],[1, 2.5],[-1, 0],[0, -1]]
b = [6,12,16,0,0]
#Prices
c = [-5,-3]
f = production_portfolio(3)
point = solver(n=3,c=c,A=A,b=b)['x']
f.prices = c
f
```

    These are the constraints:
    A:[[-1, 1], [2, -1], [1, 2.5], [-1, 0], [0, -1]]
    b:[6, 12, 16, 0, 0]
    These are the prices: [-5, -3]
    There are 5 constraints

```python
print('This is the optimal solution given by the above constraints\nx:{}\ny:{}'.format(round(point[0],3),
                                                                                round(point[1],3)))
```

    This is the optimal solution given by the above constraints
    x:7.667
    y:3.333

### If it's a discrete product portfolio, the solution might not exist within our space of values (the grid/hypercube). Need to find the feasible point closest to the optimal solution

```python

def nearest(x, optimal_points, verbose=False):
        '''As the product space may not be continious, the closest value to the optimal solution is found in our grid'''
        #Takes the absolute difference between the optimal point and all the values of the mesh grid
        #Finds the index of the smallest difference
        index = ((np.abs(x[:,0] - optimal_points[0])) + (np.abs(x[:,1] - optimal_points[1]))).argmin()

        feasible = True
        #For that value, see if it is feasible. If not feasible, delete that value and redo fnding the nearest point
        while feasible:
            # If this evaluates to True, feasible = false. exits loop
            feasible = not (A@np.array(x[index]) <= b.T).all()
            if not feasible:
                break
            if verbose: print('Deleted these: {}'.format(x[index]))
            x = np.delete(x, index, 0)
            index = ((np.abs(x[:,0] - optimal_points[0])) + (np.abs(x[:,1] - optimal_points[1]))).argmin()

        return x[index]
```

```python
#production_portfolio(2).reset()
x = hypercube
#New constraints
a = np.array([[-1, 1],[2, -1],[1, 2.5],[-1, 0],[0, -1]])
b = np.array([6,12,16,0,0]).reshape(5,1)
#Added together into a matrix
matrix = np.append(a,b,1)
#Price vector is negative as it is a maximization problem and by default linprog solves for minimization
c = [-5,-3]
contour = list(np.array(c)*-1)
n = 3

#Matrix = constriants, x=[:,0] hypercube, n = 3, find the optimal point
obj = graph_from_matrix(matrix, x[:,0], n, solve=True, geq=1,
                        contour=contour, c=c, optimal_level=True)
obj.plot()
```

```python
intersection = intersection_points(3, obj.equation(matrix)[0][0], obj.equation(matrix)[1][0], obj.equation(matrix)[2][0])
print('''The intersection points of the above graph are the following:
         \nx1: {} y1: {}\nx2: {} y2: {}\nx3" {} y3: {}'''.format(intersection[0][0], intersection[0][1],
                                                                 intersection[1][0], intersection[1][1],
                                                                 intersection[2][0], intersection[2][1]))
```

    The intersection points of the above graph are the following:

    x1: 18.0 y1: 24.0
    x2: 7.666666666666667 y2: 3.333333333333334
    x3" 0.2857142857142857 y3: 6.2857142857142865

### Short Analysis on the given price ratio

### Example 1: Price Ratio at 2x:5y

Due to prices always being positive, the optimal bundle is monotonic and always limited by the constraints. If the price vector is parallel to the constraint $ x + 2.5y \le 16 $, the optimal bundle will be any point along this constraint between the two extreme points of (7.667, 3.333) and (0.286, 6.286)

```python
x = hypercube
a = np.array([[-1, 1],[2, -1],[1, 2.5],[-1, 0],[0, -1]])
b = np.array([6,12,16,0,0]).reshape(5,1)
matrix = np.append(a,b,1)
contour = [3,7.5]
c = [-3,-7.5]

n = 3
obj = graph_from_matrix(matrix, x[:,0], n, solve=True, geq=1,
                        contour=contour, c=c, optimal_level=True, overlap=True, hide_point=True)
obj.plot()
```

### Example 2: Ratio lower than 2:5

In this example, we pick an objective function where the ratio of price x to price y is marginally smaller than 2:5. We can see that the flatter objective function means (0.286, 6.286) becomes the optimal point.

```python
x = hypercube
a = np.array([[-1, 1],[2, -1],[1, 2.5],[-1, 0],[0, -1]])
b = np.array([6,12,16,0,0]).reshape(5,1)
matrix = np.append(a,b,1)
contour = [5,13]
c = [-5,-13]

n = 3
obj = graph_from_matrix(matrix, x[:,0], n, solve=True, geq=1,
                        optimal_level=True, contour=contour, c=c, overlap=True)
obj.plot()
```


# Mathematical models in Jupyter

## Introduction

Jupyter is a fully-functional alternative to Mathematica or Maple notebooks for developing and analyzing mathematical models in biology (or any other discipine, for that matter). For this, you will need to use a [Computer Algebra System](https://en.wikipedia.org/wiki/Computer_algebra_system) (CAS). A CAS is software that emulates manual (pen-and-paper) manipulations of mathematical expressions. Yes, it can be done, and very effectively, for a vast array of mathematical problems! A CAS combined with a graphing/plotting package like matplotlib gives you a powerful tool for mathematical modelling using a Jupyter notebook.

We will use Python's [SymPy](http://sympy.org/en/index.html) package, which provides powerful CAS features for most common mathematical modelling problems.

There is also [Sage](http://www.sagemath.org/), a more capable CAS. We will not use it here because is not a regular Python package, but rather, uses Python as its programming language. So unlike SymPy it cannot just be loaded in a Jupyter nb with a Python kernel. Instead, you will need to [install its Jupyter kernel](http://doc.sagemath.org/html/en/reference/repl/sage/repl/ipython_kernel/install.html). You can [install and try sage](http://www.sagemath.org/download.html) outside of Jupyter if you want.

So let's use SymPy in Jupyter.

If you used [Anaconda](http://continuum.io/downloads) to install Jupyter, it should already include SymPy, Matplotlib, IPython, NumPy, and other useful packages for scientific computing. If you don't have SymPy for some other reason, install it in Linux/mac using:
```bash
$ sudo apt install python3-sympy
```
Otherwise, follow the instructions [here](http://docs.sympy.org/latest/install.html).

We also need to rum some commands so that the plots appear correctly:

```python
%matplotlib inline
import matplotlib.pyplot as p
```

And import scipy and sympy:

```python
from sympy import *
import scipy as sc
init_printing() # for pretty-printing equations etc
```

## Some preliminaries

Before we get started with our mathematical modelling session in Jupyter, some SymPy preliminaries.

### Symbolic variables

In CAS' like SymPy, we need to create symbolic variables for the mathematical variables we want to work with. A new symbolic variable can be created using `var`. Try this:

```python
x = var('x')
type(x) # check it's class
```

    sympy.core.symbol.Symbol

You can also define multiple symbolic variables at one go:

```python
a, b, c = var("a, b, c")
```

For more info on symbolic variables, [have a look at this](http://docs.sympy.org/latest/gotchas.html#variables).

It is often important to add assumptions (constraints) to our symbolic vars:

```python
x = var('x', real=True)
```

Now check:

```python
x.is_imaginary
```

    False

```python
x = Symbol('x', positive=True)
```

Again, check:

```python
x > 0
```

```python
x < 0
```

### Symbolic equations

We can define the mathematical equations (functions) that we will be using/manipulating as follows:

```python
MyFun = (pi + x)**2; MyFun
```

```python

```

```python
MyFun = N_0 + (N_max - N_0) * exp(-exp(r_max * exp(1) * (t_lag - t)/((N_max - N_0) * log(10)) + 1))
```

See the nice $\LaTeX$ - formatted output: this is where `init_printing()` comes handy.

<div style="padding:6px;background-color:cornsilk;line-height:1.4;">
SymPy has predefined expressions for a number of mathematical constants, such as: `pi` ($\pi$), `e` (exponential), `oo` (infinity).
</div>

You can also get your equation in latex syntax! Try:

```python
latex(MyFun)
```

    '\\left(x + \\pi\\right)^{2}'

That has extra escape slashes for Python to be able to parse it correctly. To display it in its actual form (that you can directly use in a $\LaTeX$ document), `print` it:

```python
print(latex(MyFun))
```

    \left(x + \pi\right)^{2}

### Numerical evaluation

To evaluate an expression numerically we can use the `evalf` function (or `N`). It takes an argument `n` which specifies the number of significant digits.

```python
pi.evalf(n=100) # pi to a 100 places after decimal!
```

`N()` is shorthand alias for `evalf()`:

```python
N(pi, 50)
```

So let's try evaluating our function:

```python
N(MyFun, 5)
```

When we numerically evaluate algebraic expressions we often want to substitute a symbol with a numerical value. In SymPy we do that using the `subs` function:

```python
MyFun.subs(x, 1.5)
```

Now let's evaluate it:

```python
MyFun.subs(x, 1.5)
```

The `subs` function can also be used to substitute mathematical variables or expressions. Let's substitute $x$ with $a+\pi$:

```python
MyFun.subs(x, a+pi)
```

And assign it as a new symbolic equation for using later:

```python
MyFun_new = MyFun.subs(x, a+pi); MyFun_new
```

We can also numerically evaluate the function over a range of values using NumPy arrays:

```python
x_vec = sc.arange(0, 10, 0.1)
```

```python
MyFun_vec = sc.array([N(MyFun.subs(x, xx)) for xx in x_vec]) #Note: using a list comprehension!
```

We can also evaluate the new function `MyFun_new` we created by substitution above:

```python
MyFun_new_vec = sc.array([N((MyFun_new).subs(a, xx)) for xx in x_vec])
```

Now plot the two functions that you evaluated (try adding axes and a legend to these basic plots).

```python
fig, ax = p.subplots()
ax.plot(x_vec, MyFun_vec)
ax.plot(x_vec, MyFun_new_vec)
```

However, numerical evaluation using `evalf()` can be very slow. There is a much more efficient way to do it by using [`lambdify()`](http://docs.sympy.org/latest/modules/utilities/lambdify.html) to "compile" a Sympy expression into a function that is much more efficient to evaluate numerically:

```python
MyFun_lamb = lambdify([x], MyFun, 'numpy')
```

The first argument is a (python) list of variables that `MyFun_lamb` will be a function of. In this case its only $x$. Now we can directly pass a numpy array and MyFun is evaluated more efficiently:

```python
MyFun_vec = MyFun_lamb(x_vec)
```

The speedup when using "lambdified" functions instead of direct numerical evaluation can be significant, often several orders of magnitude. Even in this simple example we get a significant speed up:

```python
%%timeit #remember this?

MyFun_vec = sc.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

    27.7 ms ± 7.48 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%%timeit

MyFun_vec = MyFun_lamb(x_vec)
```

    2.64 µs ± 684 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Fundamental mathematical operations

Let's look at some fundamental methematical operations in SymPy that you will almost certainly use at some point in biological models. You can find a full list and addtional tutorials and examples at [SymPy documentation](http://docs.sympy.org/latest/index.html).

### Expand and factor

```python
expand(MyFun)
```

The `expand` function takes a number of keywords arguments which we can tell the functions what kind of expansions to want to perform; use `help(expand)` (or the [SymPy documentation](http://docs.sympy.org/latest/index.html)) for more info.

You can also print the result of an manipulation in its raw python form:

```python
print(expand(MyFun))
```

    x**2 + 2*pi*x + pi**2

You can also factor using, well, `factor()`:

```python
factor(x**2 + 2*pi*x + pi**2)
```

### Apart and together

To manipulate symbolic expressions of fractions, you can use the `apart` and `together` functions:

```python
f1 = 1/((a+1)*(a+2)); f1
```

```python
apart(f1)
```

```python
f2 = 1/(a+2) + 1/(a+3); f2
```

```python
together(f2)
```

### Simplification

The `simplify` tries to simplify an expression into a nice looking expression, using various techniques. More specific alternatives to the `simplify` functions also exist: `trigsimp`, `powsimp`, `logcombine`, etc. Applying `simplify` to the above example will give the same result as `together`:

```python
simplify(f2)
```

Note that simplify usually combines fractions but does not factor.

In addition to algebraic manipulations, the other main use of CAS is to do calculus, like derivatives and integrals of algebraic expressions.

### Differentiation

Differentiation is usually simple. Use the `diff` function. The first argument is the expression to take the derivative of, and the second argument is the symbol by which to take the derivative:

```python
diff(MyFun_new, a)
```

For higher order derivatives we can do:

```python
diff(MyFun_new, a, a)
```

```python
diff(MyFun_new**2, a, 2) # same as above
```

You can directly apply another manipulation to the result of a previous operation:

```python
expand(diff(MyFun_new**2, a, 2))
```

Calculate the derivative of a multivariate expression:

```python
x, y, z = var("x,y,z")
```

```python
f = sin(x*y) + cos(y*z)
```

$\frac{d^3f}{dxdy^2}$

```python
diff(f, x, 1, y, 2)
```

### Integration

Integration is done in a similar fashion:

```python
MyFun
```

```python
integrate(MyFun, x)
```

By providing limits for the integration variable we can evaluate definite integrals:

```python
integrate(MyFun, (x, -1, 1))
```

and also improper integrals

```python
integrate(exp(-x**2), (x, -oo, oo))
```

Remember, `oo` is the SymPy notation for inifinity.

### Sums and products

You can evaluate sums and products using `Sum`. Note that this function is named `Sum` and not `sum` to avoid namespace conflict.

```python
n = var("n")
```

```python
Sum(1/n**2, (n, 1, 10))
```

```python
Sum(1/n**2, (n,1, 10)).evalf()
```

```python
Sum(1/n**2, (n, 1, oo)).evalf()
```

Products work much the same way:

```python
Product(n, (n, 1, 10)) # 10!
```

### Limits

Limits can be evaluated using the `limit` function. For example,

```python
limit(sin(x)/x, x, 0)
```

We can use 'limit' to check the result of derivation using the `diff` function:

```python
f
```

```python
diff(f, x)
```

$\displaystyle \frac{\mathrm{d}f(x,y)}{\mathrm{d}x} = \frac{f(x+h,y)-f(x,y)}{h}$

```python
h = var("h")
```

```python
limit((f.subs(x, x+h) - f)/h, h, 0)
```

OK!

We can change the direction from which we approach the limiting point using the `dir` keywork argument:

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

### Series

Series expansion is also one of the most useful features of a CAS. In SymPy we can perform a series expansion of an expression using the `series` function:

```python
series(exp(x), x) # this is a classic!
```

By default it expands the expression around $x=0$, but we can expand around any value of $x$ by explicitly include a value in the function call:

```python
series(exp(x), x, 1)
```

Or try:

```python
series(log(x), x, 0) # will not work why?
```

```python
series(log(x), x,1) # this will work, however
```

And we can explicitly define to which order the series expansion should be carried out:

```python
series(exp(x), x, 0, 3)
```

Another way to do the same:

```python
exp(x).series(x,0,3)
```

The series expansion includes the order of the approximation, which is very useful for keeping track of the order of validity when we do calculations with series expansions of different orders:

```python
s1 = cos(x).series(x, 0, 5); s1
```

```python
s2 = sin(x).series(x, 0, 2); s2
```

```python
expand(s1 * s2)
```

If we want to get rid of the order information we can use the `removeO` method:

```python
expand(s1.removeO() * s2.removeO())
```

### Matrix algebra

Matrices are defined using the `Matrix` class:

```python
m11, m12, m21, m22 = var("m11, m12, m21, m22")
b1, b2 = var("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]]) # Again, note: capital M for to avoid namespace conflict
A
```

```python
b = Matrix([[b1], [b2]]); b
```

With `Matrix` class instances we can do the usual matrix algebra operations:

```python
A**2
```

```python
A * b
```

And calculate determinants and inverses, and the like:

```python
A.det()
```

```python
A.inv()
```

### Solving equations

For solving equations and systems of equations we can use the `solve` function:

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 - x**2 - 1, x)
```

System of equations:

```python
solve([x + y - 1, x - y - 1], [x,y])
```

In terms of other symbolic expressions:

```python
solve([x + y - a, x - y - c], [x,y])
```

You can also solve a single, or a system of ordinary differential equations (ODEs) using
[dsolve](http://docs.sympy.org/latest/modules/solvers/ode.html). WE will use this in a couple of the biological examples below.

## Some biological examples

Here are some examples of development and analysis of some fundamental mathematical models in biology.

### One population: Exponential growth

Let's look at how populations grow when there are no environmental constraints. The differential equation model is:

\begin{equation}\label{eq:exp_growth}
	\frac{\text{d}N}{\text{d}t} = r_m N
\end{equation}

where $r_m$ is the intrinsic, constant rate of population gowth (units of 1/time), and $N$ is population size (or biomass abundance). I use the subscript $m$ in $r_m$ to denote both [Malthusian](https://en.wikipedia.org/wiki/Malthusianism) and maximal population growth rate because, in theory, without any constraints, this growth rate is expected to at its theoretical/biological maximum.

Let's solve equation \ref{eq:exp_growth} so that we can calculate population size $N_t$ at any given time $t$ given a starting population size  $N_0$ at time 0.

First assign the symbolic variables:

```python
r_max, N_0, K, t_lag, t  = var("r_max N_0 K t_lag t",real = True) # the real bit is not really necessary here
```

```python
N_0 + (K - N_0) * exp(-exp(r_max * exp(1) * (t_lag - t)/((K - N_0) * log(10)) + 1))
```

```python
r_m, N, t = var("r_m N t",real = True) # the real bit is not really necessary here
```

Now tell SymPy that $N$ is a function:

```python
N = Function('N')
```

Define $N$ is a derivative of $t$

```python
dN_dt = Derivative(N(t), t) - r_m*N(t); dN_dt
```

Note that we have simply re-written the condition that LHS = RHS in eqn \ref{eq:exp_growth}. Now that we have the differential equation set up, we can solve it using `dsolve`. Since this is a simple ODE, Sympy can do it on its own with no hints or guesses (unlike more complex ODEs; [see the documentation](http://docs.sympy.org/latest/modules/solvers/ode.html)):

```python
MyEq_sol = dsolve(dN_dt); MyEq_sol
```

If you remember your high-school calculus, you might recall that $C_1$ here is an arbitrary constant. We now need to re-express it in terms of the initial conditions. We can do so by setting $t = 0$, and then setting that equal to $N(0)$ at time 0:

$$C_1 = N_0$$

That is,

\begin{equation}\label{eq:exp_growth_sol}
N{\left (t \right )} = N_0 e^{r_m t}
\end{equation}

We could use Sympy for this last step as well (using `subs()` like you learned above), but that would be just plain silly -- like using a sledge-hammer to drive in a nail!

We can now have a go at plotting the model (eqn. \ref{eq:exp_growth}), and also the solution eqn. \ref{eq:exp_growth_sol}. First, let's get an approximate solution by using numerical integration (which you learnt in the Advanced Python chapter):

```python
from scipy  import integrate

# parameters
r_m = 1.

# initial conditions
N_0 = 0.1

# The time vector
t_vec = sc.arange(0, 10., 0.01)

def exp_pop(N, t, r_m):
    """The right-hand side of the exponential growth ODE"""
    return r_m*N

N_vec = integrate.odeint(exp_pop, N_0, t_vec, args=(r_m,)) # the comma is needed!

# plot the numerical solution
p.plot(t_vec, N_vec)
p.xlabel('Time') ; p.ylabel('$N$')

# plot analytical solution
p.plot(t_vec, N_0 * sc.exp(r_m * t_vec),'k--')
p.legend(['numerical approximation', 'analytical solution'], loc='best') # draw legend
```

These look practically identical. But they are not:

```python
N_vec - N_0 * sc.exp(r_m * t_vec)
```

    array([[  0.00000000e+00,  -1.00501671e-03,  -2.02013400e-03, ...,
             -2.13744854e+03,  -2.15893125e+03,  -2.18062988e+03],
           [  1.00502283e-03,   6.11819401e-09,  -1.01511118e-03, ...,
             -2.13744753e+03,  -2.15893025e+03,  -2.18062887e+03],
           [  2.02014197e-03,   1.01512526e-03,   7.96376975e-09, ...,
             -2.13744651e+03,  -2.15892923e+03,  -2.18062786e+03],
           ...,
           [  2.13744963e+03,   2.13744863e+03,   2.13744761e+03, ...,
              1.09563845e-03,  -2.14816243e+01,  -4.31802491e+01],
           [  2.15893236e+03,   2.15893136e+03,   2.15893034e+03, ...,
              2.14838268e+01,   1.10688877e-03,  -2.16975180e+01],
           [  2.18063100e+03,   2.18062999e+03,   2.18062898e+03, ...,
              4.31824631e+01,   2.16997432e+01,   1.11834951e-03]])

(Logistic-Population-Growth)=
### One population: Logistic Population growth

Populations eventually run into contraints, even if they can grow exponentially at the start (eqn \ref{eq:exp_growth}). The classical model for logistic growth in population density ($N$) captures this dynamic:

$$
	\frac{\text{d}N}{\text{d}t} = r_m N \left(1-\frac{N}{K}\right)
$$(eq:logist_growth)

where $K$ is the carrying capacity of the environment, while $r_m$ is the same parameter same as above. Let's solve this one as well.

As in the case of the exponential growth above, let's find the solution to this equation for any arbitrary time point $t$.

Again, first we define the vars and the function:

```python
r_m, K, N, t = var("r_m K N t",real = True) # the real bit is not really necessary here

N = Function('N')
```

```python
dN_dt = Derivative(N(t), t) - r_m * N(t) * (1 - N(t) / K); dN_dt
```

Again, as in the exponential growth example, we have simply re-written the condition that LHS = RHS in eqn \ref{eq:logist_growth}. Now we can solve the ODE:

```python
MyEq_sol = dsolve(dN_dt); MyEq_sol
```

This is a bit more complicated than the solution for exponential growth above. But we can solve it the same way. First substitute $t = 0$, and then solve the resulting equation for the initital condition of $N_0$, which then gives This the time-dependent solution:
\begin{equation}
    N_t = \frac{N_0 K\mathrm{e}^{r_m t}}{K + N_0(\mathrm{e}^{r_m t}-1)}
\end{equation}

You can do the last steps to obtain this solution using Sympy as well (I leave it to you to try it).

No let's again compare the analytical solution against the numerical one:

```python
from scipy  import integrate

# parameters
r_m = 1.
K = 10.
# initial condition
N_0 = 0.1

#The time vector
t_vec = sc.arange(0, 10., 0.01)

def log_pop(N, t, r_m, K):
    """The right-hand side of the logistic ODE"""
    return r_m*N*(1-N/K)

N_vec = integrate.odeint(log_pop, N_0, t_vec, args=(r_m, K));

p.plot(t_vec, N_vec) # plot the solution
p.xlabel('Time') ; p.ylabel('$N$')

# plot analytical solution
p.plot(t_vec, K * N_0 * sc.exp(r_m * t_vec)/(K + N_0 * (sc.exp(r_m * t_vec) - 1.)),'k--')
p.legend(['numerical approximation', 'analytical solution'], loc='best') # draw legend
```

### Two interacting populations: The Lotka-Volterra predator-prey model

Now for the classical Lotka-Volterra model that you encountered in the advanced Python week (without logistic growth for the consumer, $C$).

\begin{align}
\frac{dN}{dt} &= r_m N \left(1-\frac{N}{K}\right) - a N C\\
\frac{dC}{dt} &= e a N C - z C
\end{align}

here $r_m$ and $K$ is the Resource's growth rate and carrying capacity respectively as in the Logistic equation, $a$ is the consumer's search rate for the resource, $e$ is consumer's biomass conversion efficiency, and $z$ is it's mortality rate.

To solve this system of ODEs, we will take a different approach -- We will solve for the equilibrum (steady state for the two species' populations). Again, we start by define the vars:

```python
r_m, a, e, z, K, N, C, t = var("r_m, a, e, z, K, N, C, t",real = True)
```

Now define the sysyem of ODEs for Sympy:

```python
dN_dt = r_m * N *(1-N/K) - a * N * C
dC_dt = e * a * N * C - z * C

dC_dt, dN_dt
```

Now define the equilibrium state:

```python
N_eqlb = Eq(dN_dt, 0)
C_eqlb = Eq(dC_dt, 0)
N_eqlb, C_eqlb
```

Solve it:

```python
N_eqlb_sol = solve(N_eqlb, C)
C_eqlb_sol = solve(C_eqlb, N)

N_eqlb_sol, C_eqlb_sol
```

So there is one equilibrium  solution where both species maintain their non-zero populations. That was easy! Now you don't need to guess what parameter values will give you coexistence (both populations remain > 0). Just substitute parameter combinations that satisfy the conditions that

$$ \frac{r_{m} \left(K - N\right)}{K a} > 0, \textrm{ and } \frac{z}{a e} > 0$$

As an exercise try plotting this exact solution for the steady state along with the one you would obtain (asymptotically) using numerical integration of the system of ODEs. Below is code that does the numerical integration (essentially, same as the `LV.py` script).

```python
from scipy import integrate

t_vec = sc.arange(0, 100., 0.01)

# parameters
r_m = 1.
a = 1
e = 0.5
z = .5
K =10

# initial condition: this is an array now!
N0C0 = sc.array([1., 1.])

# the function still receives only `x`, but it will be an array, not a number
def LV(NC, t, r_m, K, a, e, z):
    # Unlike the esponental and logistic growth model, we now need to convert
    # the output to a numpy array as it has two populations.
    return sc.array([ r_m * NC[0]*(1-NC[0]/K) - a * NC[0] * NC[1],
                   e * a * NC[0] * NC[1] - z * NC[1] ])

NC_vec = integrate.odeint(LV, N0C0, t_vec, (r_m, K, a, e, z))
```

Check NC_vec's dimensions:

```python
print(NC_vec.shape)
```

    (10000, 2)

Now let's plot the solution. But first, just for fun, let's change the plot style:

```python
print(p.style.available)
```

    ['bmh', 'seaborn-darkgrid', 'fast', 'seaborn-dark', 'seaborn-notebook', 'classic', 'seaborn', 'ggplot', 'seaborn-white', 'seaborn-pastel', 'seaborn-deep', 'seaborn-dark-palette', 'dark_background', 'seaborn-paper', 'grayscale', 'seaborn-ticks', '_classic_test', 'seaborn-whitegrid', 'Solarize_Light2', 'seaborn-talk', 'fivethirtyeight', 'seaborn-poster', 'seaborn-muted', 'seaborn-colorblind', 'seaborn-bright']

```python
p.style.use('seaborn-darkgrid')
```

```python
p.plot(t_vec, NC_vec)
p.xlabel('Time'); p.ylabel('Population size') # and of y-axis
p.legend(['Resource ($N$)', 'Consumer ($C$)'], loc='best')
```

An useful thing to do here is take a look at the *phase space*, that is, plot only the dependent variables, without respect to time:

```python
p.plot(NC_vec[0,0], NC_vec[0,1], 'o')
print('Initial condition:', NC_vec[0])

p.plot(NC_vec[:,0], NC_vec[:,1])

#Another solution with a different initial condition:
#NC_vec2 = odeint(LV, [2., 4.], t_vec, (r_m, K, a, e, z))
#p.plot(NC_vec2[:,0], NC_vec2[:,1])
#p.plot(NC_vec2[0,0], NC_vec2[0,1], 'o')
#p.xlabel('Resource Population size'); p.ylabel('Consumer Population size') # and of y-axis
```

## Readings and Resources

* [The SymPy documentation](http://sympy.org/en/index.html)
* [SymPy online](http://live.sympy.org)


# Mathematical Theory and Intuition behind Our Improved Model

This markdown explains the mathematical theory behind our model. Note that the actual implementation might be different. For simplicity, we are also not including all the datasets we used (otherwise the equations might be too long).

Let $i$ denote the $i$th date since the beginning of our dataset and $n$ denote the cutoff date for training-testing . Let the range from $n+1$ to $m$ be the range for testing (forecast).

For a given date $i$, let $\textbf{asp}(i)$ be the actual index for S\&P500; $\textbf{fbsp}(i)$ be the index for S\&P500 bases on a training set of the first $i-1$ many days' market data of S\&P500; $\textbf{div}(i)$ be the dividend yield rates of S\&P500; $\textbf{eps}(i)$ be the dividen yield rates of S\&P500; $\textbf{tby}(i)$ be the 10-year treasury bond yield; $\textbf{ffr}(i)$ be the fed fund rates;  $\textbf{fta}(i)$ be the fed total assets.

## Linear Model

\begin{equation}
    \textbf{fb}_1(i,a,b,c,d,e):=\textbf{fbsp}(i)+a*\textbf{tby}(i)+b*\textbf{ffr}(i)+c*\textbf{fta}(i)+d*\textbf{div}(i)+e*\textbf{eps}(i)
\end{equation}

\begin{equation}
    \textbf{E}_n(a,b,c,d,e):= \frac{1}{n-1000}\sum_{i=1000}^{n}
    (\textbf{fb}_1(i,a,b,c,d,e)-\textbf{asp}(i))^2
\end{equation}

attains its minimum. Namely, finding

\begin{equation}
    (a_n,b_n,c_n,d_n,e_n):=\text{argmin} \textbf{E}_n(a,b,c,d,e)
\end{equation}

Note that it doesn't make sense to start with $i=1$ since the fbprophet itself need to use the first $i-1$ of the data for training

## Nonlinear Model

All notations are the same as above.

Here is a different model (nonlinear revision of fbprophet). In this model, we will the dividend yield rate of S\&P 500.

First, what is the dividend yield rate? Dividend is the money that publicly listed companies pay you (usually four times a year) for holding their stocks (until the so-called ex-dividend dates). It's like the interest paid to your saving accounts from the banks. Some companies pay while some don't, especially for the growth tech stocks. From my experience, the impact of bond rates and fed fund rates on the stock market will change when they rise above or fall below the dividend yield rate}. Stock prices fall when those rates rise above the dividend yield rate of SP500 (investor are selling their stocks to buy bonds or save more money in their bank account!).

Based on this idea, it might be useful to consider the differences of those rates and the dividend yield rate of SP500.

Normally an increase in the federal fund rate will result in an increase in bank loan interest rate, which will in turn result in an decrease in the net income of S\&P500-listed companies since they have to pay a higher interest when borrowing money from banks. Based on this thought, I believe it is reasonable to make a correction to $c*\textbf{eps}(i)$ by replacing the term by $c*\textbf{eps}(i)(1+d*\textbf{ffr}(i))$. If my intuition is correct, the generated constant $d$ from the optimization should be a negative number.

$$\textbf{fb}_2(i,a,b,c,d,e):= \textbf{fbsp}(i)*\big[1+a*(\textbf{div}(i)-\textbf{tby}(i))\big]\big[1+b*(\textbf{div}(i)-\textbf{ffr}(i))\big]\\
+c*\textbf{eps}(i)(1+d*\textbf{ffr}(i)+e*\textbf{fta}(i))$$

and consider

$$E_n(a,b,c,d,e) := \frac{1}{n-1000}\sum_{i=1000}^n(\textbf{fb}_2(i,a,b,c,d,e)-\textbf{asp}(i))^2$$

Now find (by approximation, SGD, etc.) $(a_n,b_n,c_n,d_n,e_n):=\text{argmin} E_n(a,b,c,d,e)$

Using $(a_n,b_n,c_n,d_n,e_n)$ as constants, our output will be $\textbf{fb}_2(i,a_n,b_n,c_n,d_n,e_n)$

### The actual implementation of the nonlinear model

For the actual implementation of the nonlinear model, we threw away the higher order terms (the products of three things) since they are relatively smaller quantities

### The mathematical theory behind our nonlinear model:

#### The Taylor's theorem for multivariate functions

Let $f :\mathbb{R}^n → \mathbb R$ be a $k$-times-differentiable function at the point $a ∈ \mathbb{R}^n$. Then there exists $h_a:\mathbb{R}^n → \mathbb R$ such that:

<math>\begin{align}
& f(\boldsymbol{x}) = \sum_{|\alpha|\leq k} \frac{D^\alpha f(\boldsymbol{a})}{\alpha!} (\boldsymbol{x}-\boldsymbol{a})^\alpha  + \sum_{|\alpha|=k} h_\alpha(\boldsymbol{x})(\boldsymbol{x}-\boldsymbol{a})^\alpha, \\
& \mbox{and}\quad \lim_{\boldsymbol{x}\to \boldsymbol{a}}h_\alpha(\boldsymbol{x})=0.
\end{align}</math>

In our model, we think of $\textbf{asp}$ as a function of $\textbf{fbsp}, \textbf{tby}, \textbf{div}, \textbf{ffr}, \textbf{fta}$, etc. Say

$$\textbf{asp}:=F(\textbf{fbsp}, \textbf{tby}, \textbf{div}, \textbf{ffr}, \textbf{fta},\cdots)$$

With the assumption that $F$ here is regular, we can apply the Taylor's theorem for multivariate functions from above to $F$. Ideally, we have to consider all possible products in our implementation. But in our implementation, we chose to make a balance between our intuitive nonlinear model and Taylor's theorem.

## A faster computation scheme
One drawback of the models we proposed about is that we have to call fbprophet thousands of times when we implement it.

Here is a method that reduces the number of times calling fbprophet:

Say in the training process we are considering from i=1,000 to 11,000. Namely based on the current scheme, we have to call fbprophet for 10,000 times.

Instead of this, we make a break-down 10,000=100*100 as follows:

For i=1,000 to 1,100, we only use the first 999 dates for training;

For i=1,100 to 1,200, we only use the first 1,099 dates for training;

.............................................

For i=10,900 to 11,000, we only use the first 10,899 dates for training;

In this way, it seems that we only need to call fbprophet for 100 times. And this doesn't seem harm the accuracy too much.

```python
import pandas as pd
import numpy as np

## For plotting
import matplotlib.pyplot as plt
from matplotlib import style
import datetime as dt
import seaborn as sns
sns.set_style("whitegrid")
```

```python
path = '../Data/dff1.csv'
```

```python
df= pd.read_csv(path, parse_dates=['ds'])
# df = df.rename(columns = {"Date":"ds","Close":"y"})
df = df[['ds', 'y','fbsp', 'diff','tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi',
       'rfs']]
# df
```

```python
df['fbsp_tby'] = df['fbsp'] * df['tby']
df['fbsp_ffr'] = df['fbsp'] * df['ffr']
df['fbsp_div'] = df['fbsp'] * df['div']
df['eps_tby'] = df['eps'] * df['tby']
df['eps_ffr'] = df['eps'] * df['ffr']
df['eps_div'] = df['eps'] * df['div']
```

```python
# cutoff between test and train data
cutoff = len(df) - 252
df_train = df[:cutoff].copy()
df_test = df[cutoff:].copy()
print(cutoff)
```

    2300

```python
df_train.columns
```

    Index(['ds', 'y', 'fbsp', 'diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une',
           'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
           'eps_ffr', 'eps_div'],
          dtype='object')

```python
#possible_features = ['tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti',
#      'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
#      'eps_ffr', 'eps_div']

from itertools import chain, combinations

def powerset(iterable):
    #"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

#print(list(powerset(possible_features)))
```

```python
len(possible_features)
```

    15

```python
from statsmodels.regression.linear_model import OLS

reg_new = OLS((df_train['diff']).copy(),df_train[possible_features].copy()).fit()
print(reg_new.params)

#from the output, we can see it's consistent with sklearn output
```

    tby         305.980303
    ffr         183.608289
    fta           0.000065
    eps          -1.775823
    div        -142.530009
    une         -80.692625
    wti           3.378638
    ppi          -7.923858
    rfs           0.006087
    fbsp_tby     -0.043153
    fbsp_ffr     -0.146870
    fbsp_div     -0.391168
    eps_tby      -2.122760
    eps_ffr       2.390346
    eps_div       3.950103
    dtype: float64

```python
new_coef = reg_new.params
new_possible_feats = new_coef[abs(new_coef)>0].index

power_feats = list(powerset(new_possible_feats))
power_feats.remove(())

power_feats = [ list(feats) for feats in power_feats]
len(power_feats)

```

    32767

```python
AIC_scores = []
parameters = []

for feats in power_feats:
    tmp_reg = OLS((df_train['diff']).copy(),df_train[feats].copy()).fit()
    AIC_scores.append(tmp_reg.aic)
    parameters.append(tmp_reg.params)

Min_AIC_index = AIC_scores.index(min(AIC_scores))
Min_AIC_feats = power_feats[Min_AIC_index]
Min_AIC_params  = parameters[Min_AIC_index]
print(Min_AIC_feats)
print(Min_AIC_params)
```

```python
len(Min_AIC_feats)
```

```python
###After selecting the best features, we report the testing error, and make the plot
AIC_df_test = df_test[Min_AIC_feats]
AIC_pred_test = AIC_df_test.dot(Min_AIC_params)+df_test.fbsp

AIC_df_train = df_train[Min_AIC_feats]
AIC_pred_train = AIC_df_train.dot(Min_AIC_params)+ df_train.fbsp

```

```python
from sklearn.metrics import mean_squared_error as MSE

mse_train = MSE(df_train.y, AIC_pred_train)
mse_test = MSE(df_test.y, AIC_pred_test)

#compare with fbprophet()

fb_mse_train = MSE(df_train.y, df_train.fbsp)
fb_mse_test = MSE(df_test.y, df_test.fbsp)

print(mse_train,fb_mse_train)

print(mse_test,fb_mse_test)
```

```python
df_train.ds
```

```python
plt.figure(figsize=(18,10))

# plot the training data
plt.plot(df_train.ds,df_train.y,'b',
            label = "Training Data")

plt.plot(df_train.ds, AIC_pred_train,'r-',
            label = "Improved Fitted Values by Best_AIC")

# # plot the fit
plt.plot(df_train.ds, df_train.fbsp,'g-',
            label = "FB Fitted Values")

# # plot the forecast
plt.plot(df_test.ds, df_test.fbsp,'g--',
            label = "FB Forecast")
plt.plot(df_test.ds, AIC_pred_test,'r--',
            label = "Improved Forecast by Best_AIC")
plt.plot(df_test.ds,df_test.y,'b--',
            label = "Test Data")

plt.legend(fontsize=14)

plt.xlabel("Date", fontsize=16)
plt.ylabel("SP&500 Close Price", fontsize=16)

plt.show()
```

```python
plt.figure(figsize=(18,10))
plt.plot(df_test.y,label="Training Data")
plt.plot(df_test.fbsp,label="FB Forecast")
plt.plot(AIC_pred_test,label="Improved Forecast by Best_AIC")
plt.legend(fontsize = 14)
plt.show()
```

```python

```

```python
column = ['tby', 'ffr', 'fta', 'eps', 'div', 'une',
                                                  'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']
```

```python
from sklearn import preprocessing
df1_train = df_train[['diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']]

X = preprocessing.scale(df1_train)
from statsmodels.regression.linear_model import OLS

reg_new = OLS((X[:,0]).copy(),X[:,1:].copy()).fit()
print(reg_new.params)
```

    [ 1.50405129  1.03228322  0.27409454  1.17073571  0.31243092 -0.75747342
      0.46988206 -0.39944639  2.10369448 -0.69112943 -2.1804296  -2.38576385
     -1.14196633  1.41832903 -0.34501927]

```python
# Before Covid
# pd.Series(reg_new.params, index=['tby', 'ffr', 'fta', 'eps', 'div', 'une',
#                                                   'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'] )
```

```python
# before covid
coef1 = [ 1.50405129,  1.03228322,  0.27409454,  1.17073571,  0.31243092,
       -0.75747342,  0.46988206, -0.39944639,  2.10369448, -0.69112943,
       -2.1804296 , -2.38576385, -1.14196633,  1.41832903, -0.34501927]
# include covid
coef2 = [ 0.65150054,  1.70457239, -0.1573802 , -0.18007979, -0.15221931,
       -0.62326075,  0.45065894, -0.38972706,  2.87210843, -1.17604495,
       -4.92858316, -2.15459111,  0.11418468,  2.74829778,  0.55520382]
```

```python
# Include Covid
# pd.Series( np.append( ['coefficients (before covid)'], np.round(coef1,3)),  index= np.append(['features'], column) )

```

```python
index1 = ['10 Year U.S Treasury Bond Yield Rates (tby)', 'Federal Funds Rates (ffr)',
        'Federal Total Assets (fta)', 'Earning-Per-Share of S&P 500 (eps)', 'Dividend Yield of S&P 500 (div)',
        'Unemployment Rates (une) ', 'West Texas Intermediate oil index (wit)', 'Producer Price Index (ppi)',
         'Retail and Food Services Sales (rfs)',
         'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'
        ]
```

```python
len(index1)
```

    15

```python
pd.Series(coef2, index =index1)
```

    10 Year U.S Treasury Bond Yield Rates (tby)    0.651501
    Federal Funds Rates (ffr)                      1.704572
    Federal Total Assets (fta)                    -0.157380
    Earning-Per-Share of S&P 500 (eps)            -0.180080
    Dividend Yield of S&P 500 (div)               -0.152219
    Unemployment Rates (une)                      -0.623261
    West Texas Intermediate oil index (wit)        0.450659
    Producer Price Index (ppi)                    -0.389727
    Retail and Food Services Sales (rfs)           2.872108
    fbsp_tby                                      -1.176045
    fbsp_ffr                                      -4.928583
    fbsp_div                                      -2.154591
    eps_tby                                        0.114185
    eps_ffr                                        2.748298
    eps_div                                        0.555204
    dtype: float64

```python
df3 = pd.DataFrame(coef1, index = index1, columns = ['coefficients (before covid)'])
df3['coefficients (include covid)'] =pd.Series(coef2, index =index1)
df3
```

      coefficients (before covid)
      coefficients (include covid)

      10 Year U.S Treasury Bond Yield Rates (tby)
      1.504051
      0.651501

      Federal Funds Rates (ffr)
      1.032283
      1.704572

      Federal Total Assets (fta)
      0.274095
      -0.157380

      Earning-Per-Share of S&P 500 (eps)
      1.170736
      -0.180080

      Dividend Yield of S&P 500 (div)
      0.312431
      -0.152219

      Unemployment Rates (une)
      -0.757473
      -0.623261

      West Texas Intermediate oil index (wit)
      0.469882
      0.450659

      Producer Price Index (ppi)
      -0.399446
      -0.389727

      Retail and Food Services Sales (rfs)
      2.103694
      2.872108

      fbsp_tby
      -0.691129
      -1.176045

      fbsp_ffr
      -2.180430
      -4.928583

      fbsp_div
      -2.385764
      -2.154591

      eps_tby
      -1.141966
      0.114185

      eps_ffr
      1.418329
      2.748298

      eps_div
      -0.345019
      0.555204

```python

```

Code/.ipynb_checkpoints/Scenario2 including the covid period without wti, eps-checkpoint.ipynb


# Hydrogen orbital isosurfaces

© 2020 Lee-Ping Wang and Department of Chemistry, University of California, Davis. All rights reserved.

This Jupyter notebook was written for CHE 2AH, Fall 2020.  It uses Sympy to define the hydrogen atom wavefunctions and PyVista for plotting isosurfaces.

How to use:

1. Evaluate the cell directly below this one to import the libraries and define functions.  If you're interested in coding, feel free to read through the code to get a better handle on what's going on.

2. In the cell below the one containing the codes, set the *n*, *l*, and *m* quantum numbers (be sure that 0 < *l* < *n*, and -*l* < *m* < +*l*), and then evaluate it.  The three-dimensional plot should appear.  Click the "three bars" in the upper left to set the color map, opacity, change the viewpoint, etc.

Note: In the interactive plot, the colors of the x, y, and z axes are red, yellow and green respectively.

```python
import numpy as np
from sympy.physics.hydrogen import Psi_nlm
from sympy import Symbol, lambdify
import pyvista as pv

pv.plotting.rcParams['cmap'] = 'jet'

def get_wavefunction(n, l, m):
    """
    Get functions to evaluate hydrogen atom wavefunction.

    Parameters
    ----------
    n, l, m : int
        Quantum numbers (must obey 0 < L < n, and -L < m < +L)
        (L is capitalized because otherwise it looks like the number 1).

    Returns
    -------
    f : function
        This function takes as input three array arguments containing
        r, theta, and phi values on a grid, and produce the hydrogen atom
        wavefunction values corrresponding to n, l, m quantum numbers.
    """
    r=Symbol("r", real=True, positive=True)
    phi=Symbol("phi", real=True)
    theta=Symbol("theta", real=True)
    # Obtain the symbolic function.
    expr=Psi_nlm(n, l, m, r, phi, theta, 1)
    # The numerical function should take r, theta, phi in the expected order
    f = lambdify([r, theta, phi], expr)
    return f

def getOrbitalIsosurface(n, l, m, make_real=False, isovalue=None, resolution=101):
    """
    Get hydrogen orbital isosurfaces.

    Parameters
    ----------
    n, l, m : int
        Quantum numbers (must obey 0 < L < n, and -L < m < +L).

    make_real : bool
        When set to True:
        Take the linear combination (e^imϕ + e^-imϕ)/√2 or (e^imϕ + e^-imϕ)/√2i
        for positive and negative "m" to provide the "real" linear combinations
        that are familiar to chemists. Both positive and negative isosurfaces
        will be given.

        When set to False:
        Do not take the linear combination and return the single orbital for
        the provided value of "m". Create a single isosurface corresponding to the norm
        and color it by the argument of the complex number.

    isovalue : float
        If provided, the isovalue for which to plot the functions.
        If not provided, a default value is used which is 0.05/2^n
        (i.e. 0.05, 0.025, 0.0125, 0.00625, 0.003125 for n=1, 2, 3, 4, 5)

    resolution : int
        The number of points on the grid in each dimension.
        Above 200, performance starts to get spotty.

    Returns
    ----------
    pl : pyvista.PlotterITK object
        Run pl.show(True) on the returned object to make the 3D plot
    """

    # If the isovalue is not provided, set it automatically.
    if isovalue is None:
        isovalue = 0.05/2**n

    # Set the plot limits (orbital size increases with n)
    if n == 1:
        limit = 5
    elif n == 2:
        limit = 10
    else:
        limit = 20*(n-2)

    # Create grid of x, y, z values.
    rj=resolution*1j
    x, y, z = np.mgrid[-limit:limit:rj, -limit:limit:rj, -limit:limit:rj]

    # Calculate r, theta, phi. The +1e-10 is to avoid singularities
    r = np.sqrt(x**2+y**2+z**2)
    theta = np.arccos(z/(r+1e-10))
    phi = np.arctan2(y, x+1e-10)

    # Create PlotterITK and StructuredGrid object
    pl = pv.PlotterITK()
    grid = pv.StructuredGrid(x, y, z)

    if make_real:
        # Evaluate +|m| and -|m| wavefunctions, and take linear combinations
        f_p = get_wavefunction(n, l, abs(m))
        f_m = get_wavefunction(n, l, -abs(m))
        if m == 0:
            f_array = f_p(r, theta, phi)
        elif m > 0:
            f_array = (f_p(r, theta, phi) + f_m(r, theta, phi))/np.sqrt(2)
        elif m < 0:
            f_array = (f_p(r, theta, phi) - f_m(r, theta, phi))/np.sqrt(2)
        # Take the real and imaginary parts; only one of them should be nonzero
        f_real = np.real(f_array)
        f_imag = np.imag(f_array)
        if np.linalg.norm(f_imag) > 1e-10 and np.linalg.norm(f_real) > 1e-10:
            print(np.linalg.norm(f_imag), np.linalg.norm(f_real))
            raise RuntimeError("At this point, only one of the real or imaginary parts should be nonzero")
        # Because only one part is nonzero, adding them just keeps the nonzero part
        f_data = f_real + f_imag
        # For some reason, flattening in the "normal" order reverses the axes ordering
        grid["vol"] = f_data.flatten(order='F')
        # Create two isosurfaces for positive and negative isovalues
        # (This is because I don't know how to set an initial colormap in the plotter window
        #  and the default colors don't look very good.)
        isosurface1 = grid.contour([isovalue])
        isosurface2 = grid.contour([-isovalue])
        isosurface3 = grid.contour([0])
        pl.add_mesh(isosurface1,color='orange',opacity=1.0)
        pl.add_mesh(isosurface2,color='white',opacity=1.0)
        pl.add_mesh(isosurface3,color='gray',opacity=0.4)
    else:
        # Evaluate wavefunction for the given m
        f = get_wavefunction(n, l, m)
        f_array = f(r, theta, phi)
        # Norm of the complex function
        f_abs = np.absolute(f_array)
        grid["vol"] = f_abs.flatten(order='F')
        isosurface = grid.contour([isovalue])
        # Evaluate the argument of the complex number
        # on the isosurface points and use that as the color map
        x, y, z = (isosurface.points[:,0], isosurface.points[:,1], isosurface.points[:,2])
        r = np.sqrt(x**2+y**2+z**2)
        theta = np.arccos(z/(r+1e-10))
        phi = np.arctan2(y, x+1e-10)
        f_mesh = f(r, theta, phi)
        f_arg = np.angle(f_mesh)
        pl.add_mesh(isosurface, scalars=f_arg)

    return pl
```

```python
#===============================#
#| You may modify these values |#
#===============================#
n = 2
l = 1
m = 0
pl = getOrbitalIsosurface(n, l, m, make_real=True)
pl.show(True)
```

    Viewer(geometries=[{'vtkClass': 'vtkPolyData', 'points': {'vtkClass': 'vtkPoints', 'name': '_points', 'numberO…

```python

```


# Intro to TensorFlow

https://www.youtube.com/watch?v=q5iL3XYFv2M

## Backpropagation on zero hidden layer classification case
Suppose we are required to learn the function that maps $x$ (the inputs) to $y$ (the outputs). In this particular instance we restrict ourselves to the case that $y=\sigma(Wx)$. The maths behind regression is shown below:
\begin{align}
z_i =& Wx_i\\
a_i =& \sigma(z_i)
\end{align}
where $\sigma(z_i) = 1/(1+exp(-z_i))$ is the sigmoid function. $a_i$ is commonly known as the activation. **The loss function** is $$\mathcal{L} = \frac{1}{2N}\sum_i (y_i-a_i)^2$$.

**We need to adjust the $W$ to minimise the loss function**. We use the chain rule:
\begin{align}
\frac{d \mathcal{L}}{dW} =& \sum_i \frac{d\mathcal{L}}{da_i}\frac{da_i}{dz_i}\frac{dz_i}{dW}\\
=& \frac{1}{N}\sum_i -(y_i-a_i)\cdot\frac{\exp(-z)}{(1+\exp(-z))^2}\cdot x_i\\
=& \frac{1}{N}\sum_i -(y_i-a_i)\cdot a_i(1-a_i)\cdot x_i
\end{align}

```python
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)

%matplotlib inline
```

```python
def sigmoid(z):
    return 1/(1+np.exp(-z))

z = np.arange(-5,5,0.1)

plt.plot(z,sigmoid(z))
plt.plot([0,0],[0,1],'r')
plt.plot([-6,6],[0.5,0.5],'r')
```

```python
N = 1000
D = 5

X = 5*np.random.randn(N,D)
w = np.random.randn(D,1)
y = X.dot(w)

y[y<=0] = 0
y[y>0] = 1
```

```python
train_X = X[1:100]
test_X = X[100:]
```

```python
X.shape
```

    (1000, 5)

```python
train_X.shape
```

    (99, 5)

```python
test_X.shape
```

    (900, 5)

```python
def dL_dw(X,e,a):
    return -X.T.dot(e*a*(1-a))/len(X)

def gradient_descent(gamma=5e-1, n_epochs=1000, batch_size=100, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            a = sigmoid(X[idx].dot(w)) # Activation function
            e = y[idx] - a # Really important that you use y_obs and not y (you do not have access to true y)
            #update parameters
            w = w - gamma*dL_dw(X[idx],e,a)
        loss[i] = 0.5*e.T.dot(e)/len(e)
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],w])
```

    array([[-0.31226165, -0.42375968],
           [-0.34929182, -0.45341411],
           [-1.3870532 , -1.79564317],
           [-0.24588416, -0.33009019],
           [ 0.53334152,  0.73282908]])

```python
idx = np.random.choice(len(X),20,replace=False)
a = sigmoid(X[idx].dot(w)) # Activation function
e = y[idx] - a
0.5*e.T.dot(e)/len(e)
```

    array([[ 0.00271073]])

```python
# Classification error
X_test = 5*np.random.randn(100,D)
y_test = X_test.dot(w)
y_test[y_test<=0] = 0
y_test[y_test>0] = 1

y_inferred = sigmoid(X_test.dot(params[-1])) # Get a probability measure given X
# y_inferred[y_inferred>0.5] = 1
# y_inferred[y_inferred<=0.5] = 0

# np.sum(y_test==y_inferred)
```

## Cross Entropy Error

We shall focus on doing the same as above with a different loss function, the Cross Entropy Loss function.
$$\mathcal{L} = -\frac{1}{N}\sum_i y_i\log(a_i)+(1-y_i)\log(1-a_i)$$.

The following remains the same:
\begin{align}
z_i =& Wx_i\\
a_i =& \sigma(z_i)
\end{align}
and the derivative,
\begin{align}
\frac{d \mathcal{L}}{dW} =& \sum_i \frac{d\mathcal{L}}{da_i}\frac{da_i}{dz_i}\frac{dz_i}{dW}\\
=& \frac{1}{N}\sum_i -\left(\frac{y_i}{a_i}-\frac{1-y_i}{1-a_i}\right)\cdot\frac{\exp(-z)}{(1+\exp(-z))^2}\cdot x_i\\
=& \frac{1}{N}\sum_i -\left(\frac{y_i-a_i}{a_i(1-a_i)}\right)\cdot a_i(1-a_i)\cdot x_i\\
=& \frac{1}{N}\sum_i -(y_i-a_i)\cdot x_i
\end{align}

```python
# from IPython.core.debugger import Tracer;

def dL_dw(X,e,a):
    return -X.T.dot(e)/len(X)

def gradient_descent(gamma=1e-2, n_epochs=1000, batch_size=100, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    # NOTE: I initialised w such that its variance is 1/D
    w = np.random.randn(D,1)*(1/np.sqrt(D))
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            a = sigmoid(X[idx].dot(w)) # Activation function
            e = y[idx] - a # Really important that you use y_obs and not y (you do not have access to true y)
            w = w - gamma*dL_dw(X[idx],e,a)
#
        loss[i] = -np.mean(y[idx]*np.log(a+1e-10)+(1-y[idx])*np.log(1-a+1e-10))
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],w])
```

    array([[-0.14663108, -0.42375968],
           [-0.12596242, -0.45341411],
           [-0.54578196, -1.79564317],
           [-0.11315139, -0.33009019],
           [ 0.19399247,  0.73282908]])

```python
# Classification error
X_test = 5*np.random.randn(100,D)
y_test = X_test.dot(w)
y_test[y_test<=0] = 0
y_test[y_test>0] = 1

y_inferred = sigmoid(X_test.dot(params[-1])) # Get a probability measure given X
y_inferred[y_inferred>0.5] = 1
y_inferred[y_inferred<=0.5] = 0

np.sum(y_test==y_inferred)
```

    99

## Tensorflow Introduction

```python
import tensorflow as tf
```

```python
X.shape
```

    (1000, 5)

```python
X_train.shape
```

    (500, 5)

```python
def tf_train(X_train, y_train, batch_size=20):
    # Dataset (inputs and labels)
    x = tf.placeholder(tf.float32, [None, D])
    y_ = tf.placeholder(tf.float32, [None, 1])

    # random variable
    W = tf.Variable(tf.random_normal([D, 1],stddev=0.1))
    # map X to inferred output function
    a = tf.sigmoid(tf.matmul(x, W))

    # Define loss and optimizer
    cross_entropy = tf.reduce_mean(-(y_*tf.log(a)+(1-y_)*tf.log(1-a)))
    train_step = tf.train.GradientDescentOptimizer(1e-2).minimize(cross_entropy)

    sess = tf.InteractiveSession()
    tf.initialize_all_variables().run()
    # Train
    for epoch in range(1000):
        idx = np.random.choice(len(X_train),batch_size,replace=False)
        _,l = sess.run([train_step, cross_entropy], feed_dict={x: X_train[idx], y_: y_train[idx]})
        if epoch%100 == 0:
            print('loss: '+str(l))

```

```python
tf_train(X,y)
```

    WARNING:tensorflow:From <ipython-input-42-20b4b688827a>:16: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.
    loss: 0.903563
    loss: 0.133423
    loss: 0.0812295
    loss: 0.0744086
    loss: 0.0868373
    loss: nan
    loss: nan
    loss: nan
    loss: nan
    loss: nan

```python
def tf_train(X_train, y_train, batch_size=20, n_epoch=1000):
    x = tf.placeholder(tf.float32, [None, D])
    y_ = tf.placeholder(tf.float32, [None, 1])

    W = tf.Variable(tf.random_normal([D, 1],stddev=1/np.sqrt(D)))

    # Define loss and optimizer
    z = tf.matmul(x,W)

    cross_entropy = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=z,labels=y_))
    train_step = tf.train.GradientDescentOptimizer(1e-2).minimize(cross_entropy)

    sess = tf.InteractiveSession()
    tf.initialize_all_variables().run()
    # Train
    for epoch in range(n_epoch):
        idx = np.random.choice(len(X_train),batch_size,replace=False)
        _,l = sess.run([train_step, cross_entropy], feed_dict={x: X_train[idx], y_: y_train[idx]})
        if epoch%100 == 0:
            print('loss: '+str(l))

    return sess.run(W)
```

```python
w_est = tf_train(X,y)
```

    WARNING:tensorflow:From <ipython-input-44-5d078cf0b8ba>:14: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.
    loss: 1.63057
    loss: 0.139019
    loss: 0.179242
    loss: 0.144856
    loss: 0.0849196
    loss: 0.102604
    loss: 0.0781684
    loss: 0.0877755
    loss: 0.0193359
    loss: 0.0764944

```python
np.hstack([w_est,w])
```

    array([[-0.29627892, -0.42375968],
           [-0.34968904, -0.45341411],
           [-1.36344194, -1.79564317],
           [-0.23447008, -0.33009019],
           [ 0.50962275,  0.73282908]])

```python
# Classification error
X_test = 5*np.random.randn(100,D)
y_test = X_test.dot(w)
y_test[y_test<=0] = 0
y_test[y_test>0] = 1

y_inferred = sigmoid(X_test.dot(w_est)) # Get a probability measure given X
y_inferred[y_inferred>0.5] = 1
y_inferred[y_inferred<=0.5] = 0

np.sum(y_test==y_inferred)
```

    99

```python

```

deepschool.io/Lesson 03 - TF_intro.ipynb