```
from __future__ import division
from sympy import *
from IPython.display import display, Math, Latex
from IPython.core.display import display_html
init_session(quiet=True, use_latex='mathjax')
init_printing()
```

    IPython console for SymPy 0.7.6 (Python 2.7.8-32-bit) (ground types: python)

```
def format_deriv(arg, itau, idel):
    """
    A function for giving a nice latex representation of
    the partial derivative in question
    """
    if itau+idel == 1:
        numexp = ''
    else:
        numexp = '^{{{s:d}}}'.format(s=itau+idel)

    if itau == 0:
        tau = ''
    elif itau == 1:
        tau = '\\partial \\tau'
    else:
        tau = '\\partial \\tau^{{{s:d}}}'.format(s=itau)

    if idel == 0:
        delta = ''
    elif idel == 1:
        delta = '\\partial \\delta'
    else:
        delta = '\\partial \\delta^{{{s:d}}}'.format(s=idel)

    temp = '\\frac{{\\partial{{{numexp:s}}} {arg:s}}}{{{{{tau:s}}}{{{delta:s}}}}} = '
    return temp.format(**locals())
```

# Derivatives of $\Delta$

```
B_i, a_i, tau, delta, XX = symbols('B_i, a_i, tau, delta, XX')
theta = symbols('theta', cls=Function)(tau, delta)
Delta = theta**2+B_i*((delta-1)**2)**a_i
display(Math('\\Delta = ' + latex(Delta)))

#c_i, beta_i = symbols('c_i, beta_i')
#_theta = (1-tau) + c_i*((delta-1)**2)**(1/(2*beta_i))

def deriv(idel, itau):
    dd = diff(diff(Delta, tau, itau), delta, idel)
    dd = dd.subs(((delta-1)**2)**(a_i)/(delta-1)**idel, XX)
    dd = factor(dd, XX)
    dd = dd.subs(XX, ((delta-1)**2)**(a_i)/(delta-1)**idel)

    #dd = use(dd, simplify, 1)
    display(Math(format_deriv('\Delta',itau,idel)
                 + latex(dd)))
    #el = diff(diff(Delta, tau, itau), delta, idel)
    #args = dict(c_i = 0.7, beta_i = 0.3, B_i = 0.3, a_i = 3.5, tau = 1.3, delta = 0.9)
    #for n in range(1,5):
    #    for _itau in range(0,n+1):
    #        s = diff(diff(_theta, tau, _itau), delta, n-_itau).subs(args)
    #        el = el.replace(diff(diff(theta, tau, _itau), delta, n-_itau), s)
    #el = el.subs(theta, _theta)
    #print el.subs(args).evalf()

for n in range(1,5):
    for itau in range(0,n+1):
        deriv(itau, n-itau)
```

$$\Delta = B_{i} \left(\left(\delta - 1\right)^{2}\right)^{a_{i}} + \theta^{2}{\left (\tau,\delta \right )}$$

$$\frac{\partial{} \Delta}{{\partial \tau}{}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )}$$

$$\frac{\partial{} \Delta}{{}{\partial \delta}} = \frac{2 B_{i} a_{i} \left(\left(\delta - 1\right)^{2}\right)^{a_{i}}}{\delta - 1} + 2 \theta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{2}} \Delta}{{\partial \tau^{2}}{}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + 2 \left(\frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )}\right)^{2}$$

$$\frac{\partial{^{2}} \Delta}{{\partial \tau}{\partial \delta}} = 2 \left(\theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \theta{\left (\tau,\delta \right )} + \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{2}} \Delta}{{}{\partial \delta^{2}}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \theta{\left (\tau,\delta \right )} + 2 \left(\frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )}\right)^{2} + \frac{2 \left(\left(\delta - 1\right)^{2}\right)^{a_{i}}}{\left(\delta - 1\right)^{2}} \left(2 B_{i} a_{i}^{2} - B_{i} a_{i}\right)$$

$$\frac{\partial{^{3}} \Delta}{{\partial \tau^{3}}{}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \theta{\left (\tau,\delta \right )} + 6 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \theta{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{3}} \Delta}{{\partial \tau^{2}}{\partial \delta}} = 2 \left(\theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \theta{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{3}} \Delta}{{\partial \tau}{\partial \delta^{2}}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \theta{\left (\tau,\delta \right )} + 4 \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \theta{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \theta{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{3}} \Delta}{{}{\partial \delta^{3}}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \theta{\left (\tau,\delta \right )} + 6 \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \theta{\left (\tau,\delta \right )} + \frac{2 \left(\left(\delta - 1\right)^{2}\right)^{a_{i}}}{\left(\delta - 1\right)^{3}} \left(4 B_{i} a_{i}^{3} - 6 B_{i} a_{i}^{2} + 2 B_{i} a_{i}\right)$$

$$\frac{\partial{^{4}} \Delta}{{\partial \tau^{4}}{}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \tau^{4}}  \theta{\left (\tau,\delta \right )} + 8 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \theta{\left (\tau,\delta \right )} + 6 \left(\frac{\partial^{2}}{\partial \tau^{2}}  \theta{\left (\tau,\delta \right )}\right)^{2}$$

$$\frac{\partial{^{4}} \Delta}{{\partial \tau^{3}}{\partial \delta}} = 2 \left(\theta{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta\partial \tau^{3}}  \theta{\left (\tau,\delta \right )} + \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \theta{\left (\tau,\delta \right )} + 3 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + 3 \frac{\partial^{2}}{\partial \delta\partial \tau}  \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \theta{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{4}} \Delta}{{\partial \tau^{2}}{\partial \delta^{2}}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{2}\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + 4 \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + 4 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \theta{\left (\tau,\delta \right )} + 2 \frac{\partial^{2}}{\partial \delta^{2}}  \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \theta{\left (\tau,\delta \right )} + 4 \left(\frac{\partial^{2}}{\partial \delta\partial \tau}  \theta{\left (\tau,\delta \right )}\right)^{2}$$

$$\frac{\partial{^{4}} \Delta}{{\partial \tau}{\partial \delta^{3}}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{3}\partial \tau}  \theta{\left (\tau,\delta \right )} + 6 \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \theta{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \tau} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \theta{\left (\tau,\delta \right )} + 6 \frac{\partial^{2}}{\partial \delta^{2}}  \theta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \theta{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{4}} \Delta}{{}{\partial \delta^{4}}} = 2 \theta{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{4}}  \theta{\left (\tau,\delta \right )} + 8 \frac{\partial}{\partial \delta} \theta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \theta{\left (\tau,\delta \right )} + 6 \left(\frac{\partial^{2}}{\partial \delta^{2}}  \theta{\left (\tau,\delta \right )}\right)^{2} + \frac{2 \left(\left(\delta - 1\right)^{2}\right)^{a_{i}}}{\left(\delta - 1\right)^{4}} \left(8 B_{i} a_{i}^{4} - 24 B_{i} a_{i}^{3} + 22 B_{i} a_{i}^{2} - 6 B_{i} a_{i}\right)$$

# Derivatives of $\theta$

```
A_i, beta_i, tau, delta = symbols('A_i, beta_i, tau, delta')
theta = (1-tau) + A_i*((delta-1)**2)**(1/(2*beta_i))
display(Math('\\theta = ' + latex(theta)))

def deriv(idel, itau):
    display(Math(format_deriv('\\theta',itau,idel)
                 + latex( diff(diff(theta, tau, itau), delta, idel)) ))
    #print diff(diff(theta, tau, itau), delta, idel).subs(dict(A_i = 0.7, beta_i = 0.3, tau = 1.3, delta = 0.9))

for n in range(1,5):
    for itau in range(0,n+1):
        deriv(itau, n-itau)
```

$$\theta = A_{i} \left(\left(\delta - 1\right)^{2}\right)^{\frac{1}{2 \beta_{i}}} - \tau + 1$$

$$\frac{\partial{} \theta}{{\partial \tau}{}} = -1$$

$$\frac{\partial{} \theta}{{}{\partial \delta}} = \frac{A_{i} \left(2 \delta - 2\right) \left(\left(\delta - 1\right)^{2}\right)^{\frac{1}{2 \beta_{i}}}}{2 \beta_{i} \left(\delta - 1\right)^{2}}$$

$$\frac{\partial{^{2}} \theta}{{\partial \tau^{2}}{}} = 0$$

$$\frac{\partial{^{2}} \theta}{{\partial \tau}{\partial \delta}} = 0$$

$$\frac{\partial{^{2}} \theta}{{}{\partial \delta^{2}}} = \frac{A_{i} \left(-1 + \frac{1}{\beta_{i}}\right) \left(\left(\delta - 1\right)^{2}\right)^{\frac{1}{2 \beta_{i}}}}{\beta_{i} \left(\delta - 1\right)^{2}}$$

$$\frac{\partial{^{3}} \theta}{{\partial \tau^{3}}{}} = 0$$

$$\frac{\partial{^{3}} \theta}{{\partial \tau^{2}}{\partial \delta}} = 0$$

$$\frac{\partial{^{3}} \theta}{{\partial \tau}{\partial \delta^{2}}} = 0$$

$$\frac{\partial{^{3}} \theta}{{}{\partial \delta^{3}}} = \frac{A_{i} \left(\left(\delta - 1\right)^{2}\right)^{\frac{1}{2 \beta_{i}}}}{\beta_{i} \left(\delta - 1\right)^{3}} \left(2 - \frac{3}{\beta_{i}} + \frac{1}{\beta_{i}^{2}}\right)$$

$$\frac{\partial{^{4}} \theta}{{\partial \tau^{4}}{}} = 0$$

$$\frac{\partial{^{4}} \theta}{{\partial \tau^{3}}{\partial \delta}} = 0$$

$$\frac{\partial{^{4}} \theta}{{\partial \tau^{2}}{\partial \delta^{2}}} = 0$$

$$\frac{\partial{^{4}} \theta}{{\partial \tau}{\partial \delta^{3}}} = 0$$

$$\frac{\partial{^{4}} \theta}{{}{\partial \delta^{4}}} = \frac{A_{i} \left(\left(\delta - 1\right)^{2}\right)^{\frac{1}{2 \beta_{i}}}}{\beta_{i} \left(\delta - 1\right)^{4}} \left(-6 + \frac{11}{\beta_{i}} - \frac{6}{\beta_{i}^{2}} + \frac{1}{\beta_{i}^{3}}\right)$$

# Derivatives of $\psi$

```
C_i, D_i, tau, delta = symbols('C_i, D_i, tau, delta')
psi = exp( -C_i*(delta-1)**2 -D_i*(tau-1)**2)
display(Math('\\psi = ' + latex(psi)))

def deriv(idel, itau):
    display(Math(format_deriv('\\psi',itau, idel)
                 + latex( diff(diff(psi, tau, itau), delta, idel)).replace(latex(psi),'\psi') ))
    #print diff(diff(psi, tau, itau), delta, idel).subs(dict(C_i = 10, D_i = 275, tau= 1.3, delta = 0.9))

for n in range(1,5):
    for itau in range(0,n+1):
        deriv(itau, n-itau)
```

$$\psi = e^{- C_{i} \left(\delta - 1\right)^{2} - D_{i} \left(\tau - 1\right)^{2}}$$

$$\frac{\partial{} \psi}{{\partial \tau}{}} = - D_{i} \left(2 \tau - 2\right) \psi$$

$$\frac{\partial{} \psi}{{}{\partial \delta}} = - C_{i} \left(2 \delta - 2\right) \psi$$

$$\frac{\partial{^{2}} \psi}{{\partial \tau^{2}}{}} = 2 D_{i} \left(2 D_{i} \left(\tau - 1\right)^{2} - 1\right) \psi$$

$$\frac{\partial{^{2}} \psi}{{\partial \tau}{\partial \delta}} = C_{i} D_{i} \left(2 \delta - 2\right) \left(2 \tau - 2\right) \psi$$

$$\frac{\partial{^{2}} \psi}{{}{\partial \delta^{2}}} = 2 C_{i} \left(2 C_{i} \left(\delta - 1\right)^{2} - 1\right) \psi$$

$$\frac{\partial{^{3}} \psi}{{\partial \tau^{3}}{}} = 4 D_{i}^{2} \left(\tau - 1\right) \left(- 2 D_{i} \left(\tau - 1\right)^{2} + 3\right) \psi$$

$$\frac{\partial{^{3}} \psi}{{\partial \tau^{2}}{\partial \delta}} = - 2 C_{i} D_{i} \left(2 \delta - 2\right) \left(2 D_{i} \left(\tau - 1\right)^{2} - 1\right) \psi$$

$$\frac{\partial{^{3}} \psi}{{\partial \tau}{\partial \delta^{2}}} = 4 C_{i} D_{i} \left(\tau - 1\right) \left(- 2 C_{i} \left(\delta - 1\right)^{2} + 1\right) \psi$$

$$\frac{\partial{^{3}} \psi}{{}{\partial \delta^{3}}} = 4 C_{i}^{2} \left(\delta - 1\right) \left(- 2 C_{i} \left(\delta - 1\right)^{2} + 3\right) \psi$$

$$\frac{\partial{^{4}} \psi}{{\partial \tau^{4}}{}} = 4 D_{i}^{2} \left(4 D_{i}^{2} \left(\tau - 1\right)^{4} - 12 D_{i} \left(\tau - 1\right)^{2} + 3\right) \psi$$

$$\frac{\partial{^{4}} \psi}{{\partial \tau^{3}}{\partial \delta}} = - 4 C_{i} D_{i}^{2} \left(2 \delta - 2\right) \left(\tau - 1\right) \left(- 2 D_{i} \left(\tau - 1\right)^{2} + 3\right) \psi$$

$$\frac{\partial{^{4}} \psi}{{\partial \tau^{2}}{\partial \delta^{2}}} = 4 C_{i} D_{i} \left(2 C_{i} \left(\delta - 1\right)^{2} - 1\right) \left(2 D_{i} \left(\tau - 1\right)^{2} - 1\right) \psi$$

$$\frac{\partial{^{4}} \psi}{{\partial \tau}{\partial \delta^{3}}} = 8 C_{i}^{2} D_{i} \left(\delta - 1\right) \left(\tau - 1\right) \left(2 C_{i} \left(\delta - 1\right)^{2} - 3\right) \psi$$

$$\frac{\partial{^{4}} \psi}{{}{\partial \delta^{4}}} = 4 C_{i}^{2} \left(4 C_{i}^{2} \left(\delta - 1\right)^{4} - 12 C_{i} \left(\delta - 1\right)^{2} + 3\right) \psi$$

# Derivatives of $\alpha_r$

```
n_i, tau, delta = symbols('n_i, tau, delta')
Delta_bi = symbols('Delta_bi', cls=Function)(tau, delta)
psi = symbols('psi', cls=Function)(tau, delta)
alphar = n_i*delta*Delta_bi*psi
display(Math('\\alpha^{{r}}_{{NA,i}} = '
             + latex(alphar).replace(latex(Delta_bi),'\Delta^{b_i}')))

def collector(a):
    return collect(a, Delta_bi)

def deriv(idel, itau):
    dd = diff(diff(alphar, tau, itau), delta, idel)
    dd = simplify(dd)
    dd = use(dd, expand, 2)
    display(Math(format_deriv('\\alpha^{{r}}_{{NA,i}}',itau,idel)
                 + latex(dd).replace(latex(Delta_bi),'\Delta^{b_i}') ))

for n in range(1,5):
    for itau in range(0,n+1):
        deriv(itau, n-itau)
```

$$\alpha^{{r}}_{{NA,i}} = \delta n_{i} \Delta^{b_i} \psi{\left (\tau,\delta \right )}$$

$$\frac{\partial{} \alpha^{{r}}_{{NA,i}}}{{\partial \tau}{}} = \delta n_{i} \left(\Delta^{b_i} \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta^{b_i}\right)$$

$$\frac{\partial{} \alpha^{{r}}_{{NA,i}}}{{}{\partial \delta}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta^{b_i} + \Delta^{b_i} \psi{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{2}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau^{2}}{}} = \delta n_{i} \left(\Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} + 2 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{2}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau}{\partial \delta}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} + \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} + \delta \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} + \Delta^{b_i} \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta^{b_i}\right)$$

$$\frac{\partial{^{2}} \alpha^{{r}}_{{NA,i}}}{{}{\partial \delta^{2}}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} + 2 \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} + 2 \Delta^{b_i} \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} + 2 \psi{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta^{b_i}\right)$$

$$\frac{\partial{^{3}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau^{3}}{}} = \delta n_{i} \left(\Delta^{b_i} \frac{\partial^{3}}{\partial \tau^{3}}  \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta^{b_i} + 3 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 3 \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i}\right)$$

$$\frac{\partial{^{3}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau^{2}}{\partial \delta}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta^{b_i} + \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 2 \delta \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} + 2 \delta \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} + \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} + 2 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{3}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau}{\partial \delta^{2}}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta^{b_i} + 2 \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 2 \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} + \delta \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} + 2 \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + 2 \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} + 2 \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{3}} \alpha^{{r}}_{{NA,i}}}{{}{\partial \delta^{3}}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{3}}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta^{b_i} + 3 \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 3 \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} + 3 \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 3 \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} + 6 \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{4}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau^{4}}{}} = \delta n_{i} \left(\Delta^{b_i} \frac{\partial^{4}}{\partial \tau^{4}}  \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \tau^{4}}  \Delta^{b_i} + 4 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{3}}{\partial \tau^{3}}  \psi{\left (\tau,\delta \right )} + 4 \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta^{b_i} + 6 \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{4}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau^{3}}{\partial \delta}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{4}}{\partial \delta\partial \tau^{3}}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta\partial \tau^{3}}  \Delta^{b_i} + \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{3}}{\partial \tau^{3}}  \psi{\left (\tau,\delta \right )} + 3 \delta \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta^{b_i} + 3 \delta \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta^{b_i} + 3 \delta \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 3 \delta \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + \Delta^{b_i} \frac{\partial^{3}}{\partial \tau^{3}}  \psi{\left (\tau,\delta \right )} + \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta^{b_i} + 3 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 3 \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i}\right)$$

$$\frac{\partial{^{4}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau^{2}}{\partial \delta^{2}}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{4}}{\partial \delta^{2}\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{2}\partial \tau^{2}}  \Delta^{b_i} + 2 \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 2 \delta \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \psi{\left (\tau,\delta \right )} + 2 \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta^{b_i} + 2 \delta \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta^{b_i} + \delta \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 4 \delta \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 2 \Delta^{b_i} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 2 \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta^{b_i} + 2 \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{2}}{\partial \tau^{2}}  \psi{\left (\tau,\delta \right )} + 4 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta^{b_i} + 4 \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i}\right)$$

$$\frac{\partial{^{4}} \alpha^{{r}}_{{NA,i}}}{{\partial \tau}{\partial \delta^{3}}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{4}}{\partial \delta^{3}\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{3}\partial \tau}  \Delta^{b_i} + 3 \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \psi{\left (\tau,\delta \right )} + \delta \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{3}}  \psi{\left (\tau,\delta \right )} + 3 \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta^{b_i} + \delta \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta^{b_i} + 3 \delta \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + 3 \delta \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 3 \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \psi{\left (\tau,\delta \right )} + 3 \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta^{b_i} + 6 \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta\partial \tau}  \psi{\left (\tau,\delta \right )} + 3 \frac{\partial}{\partial \tau} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 6 \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta^{b_i} + 3 \frac{\partial}{\partial \tau} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i}\right)$$

$$\frac{\partial{^{4}} \alpha^{{r}}_{{NA,i}}}{{}{\partial \delta^{4}}} = n_{i} \left(\delta \Delta^{b_i} \frac{\partial^{4}}{\partial \delta^{4}}  \psi{\left (\tau,\delta \right )} + \delta \psi{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{4}}  \Delta^{b_i} + 4 \delta \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{3}}  \psi{\left (\tau,\delta \right )} + 4 \delta \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta^{b_i} + 6 \delta \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 4 \Delta^{b_i} \frac{\partial^{3}}{\partial \delta^{3}}  \psi{\left (\tau,\delta \right )} + 4 \psi{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta^{b_i} + 12 \frac{\partial}{\partial \delta} \Delta^{b_i} \frac{\partial^{2}}{\partial \delta^{2}}  \psi{\left (\tau,\delta \right )} + 12 \frac{\partial}{\partial \delta} \psi{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta^{b_i}\right)$$

# Derivatives of $\Delta^{b_i}$

```
b_i, tau, delta = symbols('b_i, tau, delta')
Delta = symbols('Delta', cls=Function)(tau, delta)

#c_i, beta_i, a_i, d_i = symbols('c_i, beta_i, a_i, d_i')
#_theta = (1-tau) + c_i*((delta-1)**2)**(1/(2*beta_i))
#_Delta = _theta**2+d_i*((delta-1)**2)**a_i
#print _Delta

def deriv(idel, itau):
    dd = diff(diff(Delta**b_i, tau, itau), delta, idel)
    dd = use(dd, simplify, 2)
    dd = use(dd, factor, 2)

    display(Math(format_deriv('\\Delta^{{b_i}}', itau, idel)
                 + latex(simplify(dd)) ))

    #el = diff(diff(Delta**b_i, tau, itau), delta, idel)
    #args = dict(c_i = 0.7, beta_i = 0.3, d_i = 0.3, a_i = 3.5, b_i = 0.875, tau = 1.3, delta = 0.9)
    #for n in range(1,5):
    #    for _itau in range(0,n+1):
    #        s = diff(diff(_Delta, tau, _itau), delta, n-_itau).subs(args)
    #        el = el.replace(diff(diff(Delta, tau, _itau), delta, n-_itau), s)
    #el = el.subs(Delta, _Delta)
    #print el.subs(args).evalf()

for n in range(1,5):
    for itau in range(0,n+1):
        deriv(itau, n-itau)
```

$$\frac{\partial{} \Delta^{{b_i}}}{{\partial \tau}{}} = b_{i} \Delta^{b_{i} - 1}{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}$$

$$\frac{\partial{} \Delta^{{b_i}}}{{}{\partial \delta}} = b_{i} \Delta^{b_{i} - 1}{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{2}} \Delta^{{b_i}}}{{\partial \tau^{2}}{}} = b_{i} \left(b_{i} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} + \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2}\right) \Delta^{b_{i} - 2}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{2}} \Delta^{{b_i}}}{{\partial \tau}{\partial \delta}} = \frac{b_{i}}{\Delta^{3}{\left (\tau,\delta \right )}} \left(\left(b_{i} - 1\right) \Delta^{b_{i} + 1}{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} + \Delta^{b_{i} + 2}{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )}\right)$$

$$\frac{\partial{^{2}} \Delta^{{b_i}}}{{}{\partial \delta^{2}}} = b_{i} \left(b_{i} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} + \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} - \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2}\right) \Delta^{b_{i} - 2}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{3}} \Delta^{{b_i}}}{{\partial \tau^{3}}{}} = b_{i} \left(3 \left(b_{i} - 1\right) \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + \left(b_{i}^{2} - 3 b_{i} + 2\right) \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{3} + \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{b_{i} - 3}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{3}} \Delta^{{b_i}}}{{\partial \tau^{2}}{\partial \delta}} = b_{i} \left(b_{i}^{2} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} + b_{i} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 2 b_{i} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - 3 b_{i} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} + \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 2 \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2}\right) \Delta^{b_{i} - 3}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{3}} \Delta^{{b_i}}}{{\partial \tau}{\partial \delta^{2}}} = b_{i} \left(\left(b_{i}^{2} - 3 b_{i} + 2\right) \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} + \left(2 b_{i} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + b_{i} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} - 2 \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )}\right) \Delta{\left (\tau,\delta \right )} + \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{b_{i} - 3}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{3}} \Delta^{{b_i}}}{{}{\partial \delta^{3}}} = b_{i} \left(3 \left(b_{i} - 1\right) \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} + \left(b_{i}^{2} - 3 b_{i} + 2\right) \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{3} + \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{b_{i} - 3}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{4}} \Delta^{{b_i}}}{{\partial \tau^{4}}{}} = b_{i} \left(6 \left(b_{i}^{2} - 3 b_{i} + 2\right) \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + \left(b_{i}^{3} - 6 b_{i}^{2} + 11 b_{i} - 6\right) \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{4} + \left(4 b_{i} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta{\left (\tau,\delta \right )} + 3 b_{i} \left(\frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )}\right)^{2} - 4 \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta{\left (\tau,\delta \right )} - 3 \left(\frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )}\right)^{2}\right) \Delta^{2}{\left (\tau,\delta \right )} + \Delta^{3}{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \tau^{4}}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{b_{i} - 4}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{4}} \Delta^{{b_i}}}{{\partial \tau^{3}}{\partial \delta}} = b_{i} \left(b_{i}^{3} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{3} + 3 b_{i}^{2} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 3 b_{i}^{2} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - 6 b_{i}^{2} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{3} + b_{i} \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta{\left (\tau,\delta \right )} + 3 b_{i} \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 3 b_{i} \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 9 b_{i} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 9 b_{i} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + 11 b_{i} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{3} + \Delta^{3}{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta\partial \tau^{3}}  \Delta{\left (\tau,\delta \right )} - \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \tau^{3}}  \Delta{\left (\tau,\delta \right )} - 3 \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 3 \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 6 \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 6 \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - 6 \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{3}\right) \Delta^{b_{i} - 4}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{4}} \Delta^{{b_i}}}{{\partial \tau^{2}}{\partial \delta^{2}}} = b_{i} \left(b_{i}^{3} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} + b_{i}^{2} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 4 b_{i}^{2} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + b_{i}^{2} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} - 6 b_{i}^{2} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} + 2 b_{i} \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 2 b_{i} \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta{\left (\tau,\delta \right )} + b_{i} \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 2 b_{i} \Delta^{2}{\left (\tau,\delta \right )} \left(\frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )}\right)^{2} - 3 b_{i} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 12 b_{i} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - 3 b_{i} \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} + 11 b_{i} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} + \Delta^{3}{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{2}\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 2 \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 2 \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta{\left (\tau,\delta \right )} - \Delta^{2}{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} - 2 \Delta^{2}{\left (\tau,\delta \right )} \left(\frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )}\right)^{2} + 2 \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \tau^{2}}  \Delta{\left (\tau,\delta \right )} + 8 \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + 2 \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} - 6 \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \left(\frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )}\right)^{2}\right) \Delta^{b_{i} - 4}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{4}} \Delta^{{b_i}}}{{\partial \tau}{\partial \delta^{3}}} = b_{i} \left(\left(b_{i}^{3} - 6 b_{i}^{2} + 11 b_{i} - 6\right) \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{3} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} + \left(3 b_{i} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta{\left (\tau,\delta \right )} + b_{i} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta{\left (\tau,\delta \right )} + 3 b_{i} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - 3 \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{2}\partial \tau}  \Delta{\left (\tau,\delta \right )} - \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta{\left (\tau,\delta \right )} - 3 \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{2}{\left (\tau,\delta \right )} + 3 \left(b_{i}^{2} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + b_{i}^{2} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} - 3 b_{i} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} - 3 b_{i} \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta\partial \tau}  \Delta{\left (\tau,\delta \right )} + 2 \frac{\partial}{\partial \tau} \Delta{\left (\tau,\delta \right )} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )}\right) \Delta{\left (\tau,\delta \right )} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} + \Delta^{3}{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{3}\partial \tau}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{b_{i} - 4}{\left (\tau,\delta \right )}$$

$$\frac{\partial{^{4}} \Delta^{{b_i}}}{{}{\partial \delta^{4}}} = b_{i} \left(6 \left(b_{i}^{2} - 3 b_{i} + 2\right) \Delta{\left (\tau,\delta \right )} \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{2} \frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )} + \left(b_{i}^{3} - 6 b_{i}^{2} + 11 b_{i} - 6\right) \left(\frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )}\right)^{4} + \left(4 b_{i} \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta{\left (\tau,\delta \right )} + 3 b_{i} \left(\frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )}\right)^{2} - 4 \frac{\partial}{\partial \delta} \Delta{\left (\tau,\delta \right )} \frac{\partial^{3}}{\partial \delta^{3}}  \Delta{\left (\tau,\delta \right )} - 3 \left(\frac{\partial^{2}}{\partial \delta^{2}}  \Delta{\left (\tau,\delta \right )}\right)^{2}\right) \Delta^{2}{\left (\tau,\delta \right )} + \Delta^{3}{\left (\tau,\delta \right )} \frac{\partial^{4}}{\partial \delta^{4}}  \Delta{\left (\tau,\delta \right )}\right) \Delta^{b_{i} - 4}{\left (\tau,\delta \right )}$$

```

```


# Multiple Regression Analysis: Further Issues
## Effects of Data Scaling on OLS Statistics

By analysing an example, we have when the data for **dependent variable** are scaled to $k$ times as before,

- the OLS coefficient estimates are scaled to $\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Exp}{\mathrm{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\AcA}{\mathscr{A}}
\newcommand{\FcF}{\mathscr{F}}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Avar}[2][\,\!]{\mathrm{Avar}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}k$ times, intercept included
- statistical significance won't change
- the standard error are scaled to $k$ times
- $t$ statistics won't change
- the endpoints for the CI are scaled to $k$ times
- $R$-squared won't change
- sum of squared residuals, $\text{SSR}$ are scaled to $k^2$ times
- standard error of the regression, $\hat\sigma = \sqrt{\ffrac{\text{SSR}} {n-k-1}}$, are scaled to $k$ times

Now we scale on of the **independent variable** to its $k$ times, say $x_1$

- only $\hat\beta_1$ is scaled to its $1/k$ times
- also its standard error is scaled to its $1/k$ times
- all other things keep the same

### Beta Coefficients

Standard deviation and mean are key features of the data, so now we introduce regression using $z$-scores, where the variable is ***standardized*** in the sample by *subtracting off its mean* and *dividing by its standard deviation*.

$$\begin{cases}
y_i = \hat\beta_0 + \hat\beta_1 x_{i1} + \hat\beta_2 x_{i2} + \cdots + \hat\beta_k x_{ik} + \hat u_i \\
\bar y = \hat\beta_0 + \hat\beta_1 \bar x_1 + \hat\beta_2 \bar x_{2} + \cdots + \hat\beta_k \bar x_k + 0
\end{cases} \\[1.5em]
\Rightarrow\begin{align}
\ffrac{y_i-\bar y} {\hat\sigma_y} &= \P{\ffrac{\hat\sigma_1} {\hat\sigma_y}} \hat\beta_1 \P{\ffrac{x_{i1} - \bar x_1} {\hat\sigma_1}} + \P{\ffrac{\hat\sigma_2} {\hat\sigma_y}} \hat\beta_2 \P{\ffrac{x_{i2} - \bar x_2} {\hat\sigma_2}} + \cdots + \P{\ffrac{\hat\sigma_k} {\hat\sigma_y}} \hat\beta_k \P{\ffrac{x_{ik} - \bar x_k} {\hat\sigma_k}} + \ffrac{\hat u_i} {\hat\sigma_y} \\
z_y &\equiv \hat b_1 z_1 + \hat b_2 z_2 + \cdots + \hat b_k z_k + \text{error}
\end{align}
$$

where $z_y$ denotes the $z$-score of $y$, $z_i$ for $i=1,2,\dots,k$ is the $z$-score of $x_i$. And the new coefficients are $\hat b_j = \ffrac{\hat\sigma_j} {\hat\sigma_y} \hat\beta_j$, called the ***standardized coefficients*** or ***beta coefficients***.

## More on Functional Form
### More on Using Logarithmic Functional Forms

$Review$

>For model $\log\P{y} = \beta_0 + \beta_1\log\P{x_1} + \beta_2 x_2 + u$, $\beta_1 = \ffrac{\partial \log\P{y}} {\partial \log\P{x_1}}$, the elasticity, presenting the percentage change of $y$ if $x_1$ increases by $1\%$.

Benefits, blahblahblah; Drawbacks, blahblahblah...
***

About rescaling: for variables appearing in logarithmic form, their slope coefficients are invariant to rescalings.

$$ \log\P{y_i} = \beta_0 + \beta_1 x_i + u_i \Rightarrow \log\P{c_iy_i} = \P{\log\P{c_1} +\beta_0} + \beta_1 x_i + u_i \\
y_i = \beta_0 + \beta_1 \log\P{x_i} + u_i \Rightarrow y_i = \P{\beta_1\log\P{c_1} + \beta_0} + \beta_1 \log\P{x_i} + u_i$$

### Models with Quadratics

A simple example, the estimated equation: $\hat y = \hat\beta_0 +\hat\beta_1 x + \hat\beta_2 x^2$. So that the approximation:

$$\Delta \hat y \approx \P{\hat\beta_1 + 2\hat\beta_2 x}\Delta x\Rightarrow \ffrac{\Delta \hat y} {\Delta x} = \hat\beta_1 + 2\hat\beta_2 x$$

Some times there's marginal effects, when $\hat\beta_1\cdot\hat\beta_2 < 0$. The turning point is at

$$x^* = \abs{\ffrac{\hat\beta_1} {2\hat\beta_2}}$$

$Remark$

>More possible model: $\log\P{y} = \beta_0 + \beta_1\log\P{x} +\beta_2\P{\log\P{x}}^2$. It's elasticity is $\beta_1$ when $\beta_2 = 0$ otherwise
>
>$$\ffrac{\partial\log\P{y}} {\partial\log\P{x}} = \beta_1 + 2\beta_2 \log\P{x}$$

### Models with Interaction Terms

Sometimes we have the model like:

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 \cdot x_2 + u$$

then the partial effect of $x_2$ on $y$ is $\ffrac{\Delta y} {\Delta x_2} = \beta_2 + \beta_3 x_1$ meaning that there's a ***interaction effect*** between $x_1$ and $x_2$. And this can only complicate out explanation to the model. Not recommended.

One thing to do for a better explanation is to ***reparameterize*** the model, from the original one to

$$y = \alpha_0 + \delta_1 x_1 + \delta_2 x_2 + \beta_3 \P{x_1 - \mu_1}\P{x_2 - \mu_2} +u$$

- Easy interpretation of all parameters
- Standard errors for partial effects at the mean values available
- If necessary, interaction may be centered at other interesting values

## More on Goodness-of-Fit and Selection of Regressors

$R^2$ is simply an estimate of how much variation in $y$ is explained by $x_1, x_2, \dots,x_k$ in the population. Two forgettable points:

- A high $R^2$ does not imply that there is a causal interpretation因果关系
- A low $R^2$ does not preclude排除 precise estimation of partial effects

### Adjusted $R$-Squared

$Review$

The ordinary $R$-squared: $R^2 = 1-\ffrac{\text{SSR}} {\text{SST}}$, as am estimate for $1-\ffrac{\sigma_{u}^2} {\sigma_{y}^2}$, the ***population ***$R$***-squared***. Here's the adjusted one:

$$\bar R^2 = 1-\ffrac{\ffrac{\text{SSR}} {n-k-1}} {\ffrac{\text{SST}} {n-1}} = 1-\ffrac{\hat \sigma^2} {\ffrac{ \text{SST}} {n-1}}$$

- it imposes a penalty for adding additional independent variables to a model
- Not like $R^2$ a non-decreasing one, $\bar R^2$ will increase $iff$ the $t$ statistic on the new variable is greater than one in absolute value
- could be Negative

$$\bar R^2 = 1 - \P{1-R^2} \ffrac{n-1} {n-k-1}$$

### using Adjusted $R$-Squared to Choose between Nonnested Models

***nonnested models***: Two models are **nonnested models** because neither is a special case of the other.

If we're comparing two nonnested models to decide which one is better,

- if they have different number of parameters, $R^2$ can't reveal a thing. We should take the degree of freedom into account thus use $\bar R^2$
- also not comparable when they differ in their definition of the dependent variable

### Controlling for Too Many Factors in Regression Analysis

Certain variables should not be held fixed, if so, we're ***over controlling***. And it really partially depend on what the problem we are focusing on is.

### Adding Regressors to Reduce the Error Variance

With more regressors

1. more multicollinearity problems
2. reduces the error variance
3. but it's hard to find those variables that are uncorrelated with other regressors

## Prediction and Residual Analysis
### Confidence Intervals for Predictions

The estimated model: $\hat y = \hat\beta_0 + \hat\beta_1 x_1 + \hat\beta_2 x_2 + \cdots + \hat\beta_k x_k$. And the estimated parameter is:

$$\theta_0 = \Exp\SB{y\mid x_1 = c_1, x_2 = c_2,\dots,x_k = c_k} = \beta_0 + \beta_1 c_1 + \cdots + \beta_k c_k$$

Its estimator is $\hat\theta_0 = \hat\beta_0 + \hat\beta_1 c_1 + \cdots + \hat\beta_k c_k$

And the standard error for the confident interval: $\hat\theta_0 \pm z^*\cdot \text{se}\P{\hat\theta_0}$. How to find that? We can write

$$\begin{align}
y &= \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u \\
&= \P{\theta_0 -\beta_1 c_1 - \cdots - \beta_k c_k} + \beta_1 x_1 + \cdots + \beta_k x_k + u\\
&= \theta_0 + \beta_1\P{x_1 - c_1} + \beta_2 \P{x_2 - c_2} + \cdots + \beta_k \P{x_k - c_k} + u
\end{align}$$

Then run regression for $y$ on $x_i-c_i$. The intercept, that's the $\hat\theta_0$. Then we put the *variance in the unobserved error* into consideration.

$x_1^0,x_2^0,\dots,x_k^0$ are the new values of the independent variables, $u^0$ is the unobserved error. Then we have

$$y^0 = \beta_0 + \beta_1 x_1^0 + \cdots + \beta_k x_k^0 +u^0$$

As before, our best prediction of $y^0$ is the expected value of $y^0$ given the explanatory variables. So there's actually a ***prediction error***:

$$\hat e^0 = y^0 - \hat y^0 = \P{\beta_0 + \beta_1 x_1^0 + \cdots + \beta_k x_k^0 } + u^0 - \hat y^0$$

and by the **zero mean** of the unobserved error and the unbiasedness of the parameters we have

$$\Exp\SB{\hat e^0} = \P{\beta_0 + \beta_1 x_1^0 + \cdots + \beta_k x_k^0 } + 0 - \P{\beta_0 + \beta_1 x_1^0 + \cdots + \beta_k x_k^0 } = 0$$

However, its variance is not $0$. The ***variance of the prediction error*** (conditional on all in-sample values of the independent variables) is

$$\Var{\hat e^0} = \Var{\hat y^0} + \Var{u^0} = \Var{\hat y^0} + \sigma^2$$

$$\text{se}\P{\hat e^0} = \CB{\SB{\text{se}\P{\hat y^0}}^2+\hat\sigma^2}^{0.5}$$

Using the same reasoning for the $t$ statistics of the $\hat\beta_j$, $\ffrac{\hat e^0} {\text{se}\P{\hat e^0}}$ has a $t$ distribution with $df = n-\P{k+1}$. Therefore,

$$P\CB{-t_{0.025} \leq \ffrac{\hat e^0} {\text{se}\P{\hat e^0}} \leq t_{0.025}} = 0.95$$

and the ***prediction interval*** is $\hat y^0 \pm t_{0.025} \cdot \text{se} \P{\hat e^0}$, which is *wider* than the confidence interval for $\hat y^0$ because of there's an extra $\hat\sigma^2$ when calculating $\text{se}\P{\hat e^0}$.

### Residual Analysis
### Predicting $y$ When $\log\P{y}$ Is the Dependent Variable

The model is: $\log\P{y} = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k +u$. Then, the predict $\log\P{y}$ is $\widehat{\log\P{y}} = \hat\beta_0 +\hat\beta_1 x_1 + \hat\beta_2 x_2 + \cdots + \hat\beta_k x_k$. However, we CANNOT predict $y$ as $\hat y =\exp\P{\widehat{\log\P{y}}}$. With the assumption $\text{MLR}.1$ through $\text{MLR}.6$, we have

$$\Exp\SB{y \mid \mathbf{x}} = \exp\P{\ffrac{\sigma^2} {2}} \cdot\exp\P{\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k}$$

>$Remark$
>
>Here the term $\exp\P{\ffrac{\sigma^2} {2}}$ can be obtained by the moment generating function for normal distribution $r.v.$

Then the actual prediction should be $\hat y = \exp\P{\ffrac{\hat\sigma^2} {2}} \cdot\exp\P{\widehat{\log\P{y}}}$. And the truth is that this prediction is not unbiased, but consistent.

And while this method depend on the normality of the error term, $u$. Here's an alternative way to avoid that:

If we just assume that $u$ is independent of the explanatory variables, then we have

$$\Exp\SB{y\mid \mathbf{x}} = \alpha_0 \exp\P{\beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k},\bspace \alpha_0 = \Exp\SB{\exp\P{u}}>1$$

If we know what's $\hat\alpha_0$ then we can directly say that $\hat y = \hat\alpha_0\exp\P{\widehat{\log\P{y}}}$. So how to estimate $\alpha_0$ without the normality assumption?

### Smearing Estimate

The first method is based on $\alpha_0 = \Exp\SB{\exp\P{u}}$. We replace the unobserved errors $u_i$ to the corresponding OLS residuals: $\hat u_i = \log\P{y_i} - \hat\beta_0 - \hat\beta_1x_{i1} - \hat\beta_2 x_{i2} - \cdots - \hat\beta_k x_{ik}$, so that

$$\hat\alpha_0 = \ffrac{1} {n} \sum_{i=1}^{n} \exp\P{\hat u_i}$$

A **moments estimator**! And it's consistent but not unbiased, because we have replaced $u_i$  with $\hat u_i$ inside a nonlinear function. This version of $\hat\alpha_0$ is called the ***smearing estimate***.

### Regression Estimate

A different estimate of $\alpha_0$ is based on a *simple regression through the origin*. We first define:

$$m_i = \exp\P{\beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_k x_{ik}}$$

so that $\Exp\SB{y_i\mid m_i} = \alpha_0 m_i$ and if we could really observe what $m_i$ is, we could obtain an *unbiased* estimator of $\alpha_0$ from the regression $y_i$ on $m_i$ without an intercept.

Well, not possible. So we replace the $\beta_j$ with their OLS estimates and obtain $\hat m_i = \exp\P{\widehat{\log\P{y_i}}}$ where, again, $\log\P{y_i}$ are the fitted values from the regression $\log\P{y_i}$ on $x_{i1}, x_{i2}, \dots, x_{ik}$ (here is a different regression with an intercept, just to obtain $\log\P{y_i}$). Then the OLS slope estimate from the simple regression $y_i$ on $m_i$ (without an intercept):

$$\check{\alpha}_0 = \ffrac{\d{\sum_{i=1}^{n} \hat m_i y_i}} {\d{\sum_{i=1}^{n} \hat m_i^2}}$$

And this is the ***regression estimate*** of $\alpha_0$, still, consistent but not unbiased.

$Remark$

> **smearing estimate** can gurantee that the estimates are greater than $1$ while **regression estimate** cna't. **regression estimate** will be much less than $1$ when the assumption of independence between $u$ and $x_j$ is violated.

$Steps$

1. Obtain the fitted values, $\widehat{\log\P{y_i}}$, and residuals, $\hat u_i$, from the regression $\log\P{y}$ on $x_{1}, x_{2}, \dots, x_{k}$, with intercept.
2. Calculate $\hat\alpha_0$ or $\check\alpha_0$
3. For given values of $x_{1}, x_{2}, \dots, x_{k}$, obtain $\widehat{\log\P{y}}$ from $\hat y = \hat\alpha_0\exp\P{\widehat{\log\P{y}}}$
4. Obtain the prediction $\hat y$ also from the preceding equation
***

Further, what's the goodness-of-fit for our $\log\P{y}$. One that is easy to implement, and have the same value whether we estimate $\alpha_0$ as $\hat\alpha_0 = \Exp\SB{\exp\P{u}} = \exp\P{\ffrac{\hat\sigma^2} {2}} $, or $\hat\alpha_0 = \ffrac{1} {n} \sum_{i=1}^{n} \exp\P{\hat u_i}$, or $\check{\alpha}_0 = \ffrac{\d{\sum\nolimits_{i=1}^{n} \hat m_i y_i}} {\d{\sum\nolimits_{i=1}^{n} \hat m_i^2}}$. The $R$-squared is defined as

$$R^2 = 1 - \ffrac{\text{SSR}} {\text{SST}}$$

or, just the square of the correlation between $y_i$ and $\hat y_i$, $\rho^2\P{y_i,\hat y_i}$. So when the independent variable is $\log\P{y}$ we would define

$$R^2 = \rho^2\P{y_i,\hat y_i} = \rho^2\P{y_i,\hat \alpha_0 m_i} = \rho^2\P{y_i,\hat m_i}$$

***


## Selection of primary sampling units (PSUs)

In the sections below, we draw primary sampling units (PSUs) using probability proportional to size (PPS) sampling techniques implemented in the *Sample* class. The class *Sample* has two main methods that is *inclusion_probs* and *select*. The method *inclusion_probs()* computes the probability of selection and *select()* draws the random samples.

The following will illustrate the use of samplics for sample selection. For the illustration,
- We consider a stratified cluster design.
- We will a priori decide how many PSUs to sample from each stratum
- For the clusters selection, we demonstrate PPS methods

This example is not meant to be exhaustif. There are many use cases that are not covered in this tutorial. For example, some PSUs may be segmented due to their size and segments selected in a subsequent step. Segment selection can be done with Samplics in a similar way as the PSUs selection, with PPS or SRS, after the segements have been created by the user.

First, let us import the python packages necessary to run the tutorial.

```python
import numpy as np
import pandas as pd

import samplics
from samplics.datasets import load_psu_frame
from samplics.sampling import SampleSelection
```

### Sample Dataset

The file *sample_frame.csv* - shown below - contains synthetic data of 100 clusters classified by region (East, North, South and West). Clusters represent a group of households. In the file, each cluster has an associated number of households (number_households) and a status variable indicating whether the cluster is in scope or not.

This synthetic data represents a simplified version of enumeration areas (EAs) frames found in many countries and used by major household survey programs such as the Demographic and Health Surveys (DHS), the Population-based HIV Impact Assessment (PHIA) surveys and the Multiple Cluster Indicator Surveys (MICS).

```python
psu_frame_dict = load_psu_frame()
psu_frame = psu_frame_dict["data"]
psu_frame.head(25)
```

      cluster
      region
      number_households_census
      cluster_status
      comment

      0
      1
      North
      105
      1
      NaN

      1
      2
      North
      85
      1
      NaN

      2
      3
      North
      95
      1
      NaN

      3
      4
      North
      75
      1
      NaN

      4
      5
      North
      120
      1
      NaN

      5
      6
      North
      90
      1
      NaN

      6
      7
      North
      130
      1
      NaN

      7
      8
      North
      55
      1
      NaN

      8
      9
      North
      30
      1
      NaN

      9
      10
      North
      600
      1
      due to a large building

      10
      11
      South
      25
      1
      NaN

      11
      12
      South
      250
      1
      NaN

      12
      13
      South
      105
      1
      NaN

      13
      14
      South
      75
      1
      NaN

      14
      15
      South
      205
      1
      NaN

      15
      16
      South
      190
      1
      NaN

      16
      17
      South
      95
      1
      NaN

      17
      18
      South
      85
      1
      NaN

      18
      19
      South
      50
      1
      NaN

      19
      20
      South
      110
      1
      NaN

      20
      21
      South
      130
      1
      NaN

      21
      22
      South
      180
      1
      NaN

      22
      23
      South
      65
      1
      NaN

      23
      24
      South
      75
      1
      NaN

      24
      25
      South
      95
      1
      NaN

Often, sampling frames are not available for the sampling units of interest. For example, most countries do not have a list of all households or people living in the country. Even if such frames exist, it may not be operationally and financially feasible to directly select sampling units without any form of clustering.

Hence, stage sampling is a common strategy used by large household national surveys for selecting samples of households and people. At the first stage, geographic or administrative clusters of households are selected. At the second stage, a frame of households is created from the selected clusters and a sample of households is selected. At the third stage (if applicable), a sample of people is selected from the households in the sample. This is a high level description of the process; usually implementations are much less straightforward and may require many adjustments to address complexities.

### PSU Probability of Selection

At the first stage, we use the proportional to size (pps) method to select a random sample of clusters. The measure of size is the number of households (number_households) as provided in the psu sampling frame. The sample is stratified by region. The probabilities, for stratified pps, is obtained as follow: \begin{equation} p_{hi} = \frac{n_h M_{hi}}{\sum_{i=1}^{N_h}{M_{hi}}} \end{equation} where $p_{hi}$ is the probability of selection for unit $i$ from stratum $h$, $M_{hi}$ is the measure of size (mos), $n_h$ and $N_h$ are the sample size and the total number of clusters in stratum $h$, respectively.

**Important.** The pps method is used in many surveys not just for multistage household surveys. For example, in business surveys, establishments can greatly vary in size; hence pps methods are often use to select samples. Simarly, facility-based surveys can benefit from pps methods when frames with measures of size are available.

### PSU Sample size

For a stratified sampling design, the sample size is provided using a Python dictionary. Python dictionaries allow us to pair the strata with the sample sizes. Let's say that we want to select 3 clusters from stratum *East*, 2 from *West*, 2 from *North* and 3 from *South*. The snippet of code below demonstrates how to create the Python dictionary. Note that it is important to correctly spell out the keys of the dictionary which corresponds to the values of the variable stratum (in our case it's *region*).

```python
psu_sample_size = {"East":3, "West": 2, "North": 2, "South": 3}

print(f"\nThe sample size per domain is: {psu_sample_size}\n")
```

    The sample size per domain is: {'East': 3, 'West': 2, 'North': 2, 'South': 3}

The function *array_to_dict()* converts an array to a dictionnary by pairing the values of the array to their frequency. We can use this function to calculates the number of clusters per stratum and store the result in a Python dictionnary. Then, we modify the values of the dictionnary to create the sample size dictionnary.

If some of the clusters are certainties then an exception will be raised. Hence, the user will have to manually handle the certaininties. Better handling of certainties is planned for future versions of the library *samplics*.

```python
from samplics import array_to_dict

frame_size = array_to_dict(psu_frame["region"])
print(f"\nThe number of clusters per stratum is: {frame_size}")

psu_sample_size = frame_size.copy()
psu_sample_size["East"] = 3
psu_sample_size["North"] = 2
psu_sample_size["South"] = 3
psu_sample_size["West"] = 2
print(f"\nThe sample size per stratum is: {psu_sample_size}\n")
```

    The number of clusters per stratum is: {'East': 25, 'North': 10, 'South': 20, 'West': 45}

    The sample size per stratum is: {'East': 3, 'North': 2, 'South': 3, 'West': 2}

```python
stage1_design = SampleSelection(method="pps-sys", stratification=True, with_replacement=False)

psu_frame["psu_prob"] = stage1_design.inclusion_probs(
    psu_frame["cluster"],
    psu_sample_size,
    psu_frame["region"],
    psu_frame["number_households_census"],
    )

nb_obs = 15
print(f"\nFirst {nb_obs} observations of the PSU frame \n")
psu_frame.head(nb_obs)
```

    First 15 observations of the PSU frame

      cluster
      region
      number_households_census
      cluster_status
      comment
      psu_prob

      0
      1
      North
      105
      1
      NaN
      0.151625

      1
      2
      North
      85
      1
      NaN
      0.122744

      2
      3
      North
      95
      1
      NaN
      0.137184

      3
      4
      North
      75
      1
      NaN
      0.108303

      4
      5
      North
      120
      1
      NaN
      0.173285

      5
      6
      North
      90
      1
      NaN
      0.129964

      6
      7
      North
      130
      1
      NaN
      0.187726

      7
      8
      North
      55
      1
      NaN
      0.079422

      8
      9
      North
      30
      1
      NaN
      0.043321

      9
      10
      North
      600
      1
      due to a large building
      0.866426

      10
      11
      South
      25
      1
      NaN
      0.027523

      11
      12
      South
      250
      1
      NaN
      0.275229

      12
      13
      South
      105
      1
      NaN
      0.115596

      13
      14
      South
      75
      1
      NaN
      0.082569

      14
      15
      South
      205
      1
      NaN
      0.225688

### PSU Selection

In this section, we select a sample of psus using pps methods. In the section above, we have calculated the probabilities of selection. That step is not necessary when using *samplics*. We can use the method *select()* to calculate the probability of selection and select the sample, in one run. As shown below, *select()* method returns a tuple of  three arrays.
* The first array indicates the selected units (i.e. psu_sample = 1 if selected, and 0 if not selected).
* The second array provides the number of hits, useful when the sample is selected with replacement.
* The third array is the probability of selection.

NB: *np.random.seed()* fixes the random seed to allow us to reproduce the random selection.

```python
np.random.seed(23)

psu_frame["psu_sample"], psu_frame["psu_hits"], psu_frame["psu_probs"] = stage1_design.select(
    psu_frame["cluster"],
    psu_sample_size,
    psu_frame["region"],
    psu_frame["number_households_census"]
    )

nb_obs = 15
print(f"\nFirst {nb_obs} observations of the PSU frame with the sampling information \n")
psu_frame.head(nb_obs)
```

    First 15 observations of the PSU frame with the sampling information

      cluster
      region
      number_households_census
      cluster_status
      comment
      psu_prob
      psu_sample
      psu_hits
      psu_probs

      0
      1
      North
      105
      1
      NaN
      0.151625
      0
      0
      0.151625

      1
      2
      North
      85
      1
      NaN
      0.122744
      0
      0
      0.122744

      2
      3
      North
      95
      1
      NaN
      0.137184
      0
      0
      0.137184

      3
      4
      North
      75
      1
      NaN
      0.108303
      0
      0
      0.108303

      4
      5
      North
      120
      1
      NaN
      0.173285
      0
      0
      0.173285

      5
      6
      North
      90
      1
      NaN
      0.129964
      0
      0
      0.129964

      6
      7
      North
      130
      1
      NaN
      0.187726
      1
      1
      0.187726

      7
      8
      North
      55
      1
      NaN
      0.079422
      0
      0
      0.079422

      8
      9
      North
      30
      1
      NaN
      0.043321
      0
      0
      0.043321

      9
      10
      North
      600
      1
      due to a large building
      0.866426
      1
      1
      0.866426

      10
      11
      South
      25
      1
      NaN
      0.027523
      0
      0
      0.027523

      11
      12
      South
      250
      1
      NaN
      0.275229
      0
      0
      0.275229

      12
      13
      South
      105
      1
      NaN
      0.115596
      0
      0
      0.115596

      13
      14
      South
      75
      1
      NaN
      0.082569
      0
      0
      0.082569

      14
      15
      South
      205
      1
      NaN
      0.225688
      0
      0
      0.225688

The default setting ```sample_only=False``` returns the entire frame. We can easily reduce the output data to the sample by filtering i.e. ```psu_sample == 1```. However, if we are only interested in the sample, we could use ```sample_only=True``` when calling *select()*. This will reduce the output data to the sampled units and ```to_dataframe=true``` will convert the data to a pandas dataframe (pd.DataFrame). Note that the columns in the dataframe will be reduced to the minimum.

```python
np.random.seed(23)

psu_sample = stage1_design.select(
    psu_frame["cluster"],
    psu_sample_size,
    psu_frame["region"],
    psu_frame["number_households_census"],
    to_dataframe = True,
    sample_only = True
    )

print("\nPSU sample without the non-sampled units\n")
psu_sample
```

    PSU sample without the non-sampled units

      _samp_unit
      _stratum
      _mos
      _sample
      _hits
      _probs

      0
      7
      North
      130
      1
      1
      0.187726

      1
      10
      North
      600
      1
      1
      0.866426

      2
      16
      South
      190
      1
      1
      0.209174

      3
      24
      South
      75
      1
      1
      0.082569

      4
      29
      South
      200
      1
      1
      0.220183

      5
      34
      East
      305
      1
      1
      0.210587

      6
      45
      East
      450
      1
      1
      0.310702

      7
      52
      East
      700
      1
      1
      0.483314

      8
      64
      West
      300
      1
      1
      0.091673

      9
      86
      West
      280
      1
      1
      0.085561

The systematic selection method can be implemented with or without replacement. The other *samplics* algorithms for selecting sample with unequal probablities of selection are Brewer, Hanurav-Vijayan (hv), Murphy, and Rao-Sampford (rs) methods. As shown below, all these sampling techniques can be specified when extentiating a *Sample* class; then call *select()* to draw samples.

```python
Sample(method="pps-sys", with_replacement=True)
Sample(method="pps-sys", with_replacement=False)
Sample(method="pps-brewer", with_replacement=False)
Sample(method="pps-hv", with_replacement=False) # Hanurav-Vijayan method
Sample(method="pps-murphy", with_replacement=False)
Sample(method="pps-sampford", with_replacement=False) # Rao-Sampford method
```
For example, if we wanted to select the sample using the Rao-Sampford method, we could use the following snippet of code.

```python
np.random.seed(23)

stage1_sampford = SampleSelection(method="pps-rs", stratification=True, with_replacement=False)

psu_sample_sampford = stage1_sampford.select(
    psu_frame["cluster"],
    psu_sample_size,
    psu_frame["region"],
    psu_frame["number_households_census"],
    to_dataframe=True,
    sample_only=False
    )

psu_sample_sampford
```

      _samp_unit
      _stratum
      _mos
      _sample
      _hits
      _probs

      0
      1
      North
      105
      0
      0
      0.151625

      1
      2
      North
      85
      0
      0
      0.122744

      2
      3
      North
      95
      1
      1
      0.137184

      3
      4
      North
      75
      0
      0
      0.108303

      4
      5
      North
      120
      0
      0
      0.173285

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      95
      96
      West
      95
      1
      1
      0.029030

      96
      97
      West
      40
      0
      0
      0.012223

      97
      98
      West
      105
      0
      0
      0.032086

      98
      99
      West
      320
      0
      0
      0.097785

      99
      100
      West
      200
      0
      0
      0.061115

100 rows × 6 columns


 Vaccine Design

This tutorial illustrates the use of epytope for population optimized epitope-based vaccines using OptiTope  (Toussaint, et al. (2008) PLoS Comput Biol, 4(12))

For this tutorial to work, you have to install a MIP solver that is supported by Pyomo like CPLEX, GLPK, or CBC.

This tutorial will entail:
- Simple epitope prediction from a list of peptide sequences and protein sequences
- Simple epitope selection
- Activation/deactivation of optional constraints
- String-of-Beads design

```python
%matplotlib inline
%reload_ext autoreload
%autoreload 2

from epytope.Core import Allele, Peptide, Protein,generate_peptides_from_proteins
from epytope.IO import read_lines, read_fasta
from epytope.EpitopePrediction import EpitopePredictorFactory
from epytope.EpitopeSelection import OptiTope
from epytope.EpitopeAssembly import EpitopeAssembly, EpitopeAssemblyWithSpacer
```

 Chapter 1: Simple Epitope Prediction and preparation

Epitope selection is the most important step for vaccine design. It is concerned with selecting a small set of candidate epitopes to maximize the probability of inducing a long lasting and strong immune response. Epitope Selection is an interface to OptiTope, a highly flexible mathematical framework for epitope selection. OptiTope determines the provably optimal epitope set that maximizes the overall immunogenicity for a target population or a single person and user specified requirements.

We first start with reading in proteins from a FASTA file and constructing peptides from these. Then we have to construct a list of alleles to represent our target population and assign their probability of occurring in that targeted population.

```python
proteins = read_fasta("./data/proteins.fasta", id_position=3, in_type=Protein)
peptides = generate_peptides_from_proteins(proteins,9)

alleles = []
with open("./data/allele_probabilities_europe.csv", "r") as f:
    for l in f:
        allele,prob = l.split()
        alleles.append(Allele(allele,prob=float(prob.strip())))
```

With that in place, we can start predicting the immunogenicity of each peptide HLA allele pair. Since the prediction of immunogenicity is an unsolved problem as of yet, the IC50 value of an epitope HLA allele pair is usually used as approximation. In this tutorial we use `BIMAS` as prediction engine. In addition, we can specify an allele specific binding threshold that is used internally to filter for specific constraints.

```python
bimas = EpitopePredictorFactory("BIMAS")
supported_alleles = filter(lambda a: a.name in bimas.supportedAlleles, alleles)
epi_df =bimas.predict(peptides, alleles=supported_alleles)
epi_df.head()
threshold = {a.name:1 for a in supported_alleles}

```

 Chapter 2: Epitope Selection

OptiTope defines the overall immunogenicity $I(E)$ of an epitope set $E$ as the sum over the immunogenicity of its components weighted by the HLA allele $a \in A$ frequencies $p_a$ of the target population $A$. This is a commonly made assumption due to a lack of understanding the interplay of different epitopes within a vaccine.

\begin{align}
I(E) = \sum_{e \in E}\sum_{a \in A} p_a \cdot i_{e,a}
\end{align}

Together with the prediction results, we are ready to use OptiTope. The simplest model OptiTope offers, is the optimal selections only constraint by the number of allowed selected epitopes:

\begin{align}
\max &\sum_{e \in E}x_e\sum_{a \in A}p_a\cdot i_{e,a}\\
\text{s.t.}\\
&\sum_{e \in E} x_e \le k
\end{align}

```python
optitope = OptiTope(epi_df, k=3, solver="cplex", threshold=threshold)
selected_epitope = optitope.solve()
selected_epitope
```

We can also activate additional constraints. For example we can restrict the selected epitope set to cover a certain percentage of all HLA alleles:

\begin{align}
\max &\sum_{e \in E}x_e\sum_{a \in A}p_a\cdot i_{e,a}\\
\text{s.t.}\\
&\sum_{e \in E}x_e \le k\\
&\sum_{e \in I_a} x_e \ge y_a~ ~ \forall a \in A \\
&\sum_{a \in A} y_a \ge \tau^{\text{HLA}}
\end{align}

```python
optitope.activate_allele_coverage_const(0.8)
optitope.solve()
```

and/or antigens:

\begin{align}
\max& \sum_{e \in E}x_e\sum_{a \in A}p_a\cdot i_{e,a}\\
\text{s.t.}\\
&\sum_{e \in E}x_e \le k\\
&\sum_{e \in I_a} x_e \ge y_a~ ~ \forall a \in A \\
&\sum_{a \in A} y_a \ge \tau^{\text{HLA}}\\
&\sum_{e \in I_g} x_e \ge z_g~ ~ \forall g \in G \\
&\sum_{g \in G} z_g \ge \tau^{\text{Antigen}}
\end{align}

```python
optitope.activate_allele_coverage_const(0.8)
optitope.activate_antigen_coverage_const(0.8)
optitope.solve()
```

you can also again deactivate these optional constraints with:

```python
optitope.deactivate_allele_coverage_const()
optitope.deactivate_antigen_coverage_const()
optitope.solve()
```

OptiTope  offers other optional constraints like epitope coverage or overlapping constraints. Also, the allele probability could be reused to represent for example the HLA expression of a individual, or can be completely ignored (which results in OptiTope to treat each allele uniformly for each locus).

 Chapter 3: Epitope Assembly

Epitope Assembly is concerned with ordering epitopes into a string-of-beads poly-peptide which maximizes the probability that the epitopes will be fully recovered after proteasomal cleavage. This is an important step for vaccine design and has potentially high impact on the efficacy of the designed vaccine. Epitope Assembly formulates the epitope ordering problem as a traveling salesperson problem where epitopes represent the cities to visit and the distances between the cities represent the recovery probabilities.

epytope offers two alternatives for string-of-beads design. A simple string-of-beads with `EpitopeAssembly.EpitopeAssembly`, and a string-of-beads design with additional spacer sequences connecting the epitopes (`EpitopeAssembly.EpitopeAssembly`).

Since these problems are formulated as Traveling salesperson optimization problem, the time to solve an instance can vary dramatically and solving to optimality can be impractical even for small instances. Therefore, epytope offers the possibility to approximate a string-of-beads configuration with the Lin-Kernighan traveling salesman heuristic besides an optimal solution. For that, the LKH implementation must be downloaded, compiled, and globally executable. The source code can be found at http://www.akira.ruc.dk/~keld/research/LKH/.

 Simple String-of-Beads Design

Lets start with the simple approach. For cleavage prediction we use `PCM` but any other cleavage site prediction method available in epytope can be used. To access the cleavage site prediction methods we use the `epytope.CleavagePrediction.CleavageSitePredictorFactory`, similar to the `EpitopePredictorFactory`. All we have to specify is a set of say through OptiTope selected epitopes and a cleavage site predictor. epytope will return a list of Peptides based on their optimal ordering.

```python
from epytope.CleavagePrediction import CleavageSitePredictorFactory

pcm = CleavageSitePredictorFactory("PCM")
SVB = EpitopeAssembly(selected_epitope,pcm,solver="cplex").solve()
print("SVB: ",  "-".join(map(str,SVB)))
```

String-of-Beads Design with Spacer Sequences

Designing spacer sequences within a string-of-beads is more demanding and the model used is restricted to linear prediction function (currently only models derived from `APSSMEpitopePrediction`).

Spacer Design for Epitope Assembly is concerned with determining the optimal amino acid composition and length of a small sequence connecting two epitopes within a string-of-beads poly-peptide and the epitope order to maximizes the probability that the epitopes will be fully recovered after proteasomal cleavage with the smallest neo-immunogenicity.
Hence, we have to specify a linear epitope predictor besides the cleavage site predictor and a HLA allele set for which the spacer sequences are optimized. Here again, epytope offers the possibility to approximate the solution via LKH.

We will use PCM as cleavage site and again BIMAS as epitope binding prediction methods and solve the design problem optimally.

```python
SVB_spacer = EpitopeAssemblyWithSpacer(selected_epitope, pcm, bimas,
                                       alleles, threshold=threshold, solver="cplex").solve()
print "SVB with spacer: ", "-".join(map(str,SVB_spacer))
```


## Bilbao scheme - Evaluation of the iteration matrix

```python
from sympy import symbols, exp, sqrt, cos, series, latex
sigma = symbols('sigma')
omega = symbols('omega')
mass = symbols('mass')
dt = symbols('dt')
from sympy import init_printing
init_printing(use_latex='mathjax')

```

## Ak, ek

```python
expo = sqrt(sigma**2 - omega**2) * dt
Ak = exp(-sigma*dt) * (exp(expo) + exp(-expo))
Ak2 = 2 * exp(-sigma*dt) * cos(dt*sqrt(omega**2 - sigma**2))
Ak = Ak2
ek = exp(-2*sigma*dt)
```

```python
Ak, ek, Ak2
```

$$\left ( 2 e^{- dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )}, \quad e^{- 2 dt \sigma}, \quad 2 e^{- dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )}\right )$$

## $\theta_k, \sigma^*$

```python
theta_k = 2 / (omega ** 2 ) / dt**2 - Ak / (1 + ek - Ak)
theta_k
```

$$- \frac{2 e^{- dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )}}{1 - 2 e^{- dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} + e^{- 2 dt \sigma}} + \frac{2}{dt^{2} \omega^{2}}$$

```python
sigma_star = (1/dt  + omega**2 * dt /2 - theta_k * omega**2 * dt / 2) * (1 - ek) / (1 + ek)
sigma_star.simplify()
```

$$\frac{dt \omega^{2} \left(e^{2 dt \sigma} - 1\right)}{2 e^{2 dt \sigma} - 4 e^{dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} + 2}$$

## $dt\sigma^*$

```python
dtsigmastar = (dt * sigma_star) .simplify()
dtsigmastar
```

$$\frac{dt^{2} \omega^{2} \left(e^{2 dt \sigma} - 1\right)}{2 e^{2 dt \sigma} - 4 e^{dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} + 2}$$

## $\frac{1}{2}(1 - \theta_k) \omega^2 dt^2$

```python
wkk_part1 = (1 - theta_k)/2 * (omega**2) * (dt**2)
wkk_part1.simplify()
```

$$\frac{- \frac{dt^{2} \omega^{2}}{2} e^{2 dt \sigma} - \frac{dt^{2} \omega^{2}}{2} + e^{2 dt \sigma} - 2 e^{dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} + 1}{- e^{2 dt \sigma} + 2 e^{dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} - 1}$$

## wkk = $ m + \frac{1}{2}(1 - \theta_k) \omega^2 dt^2 + dt\sigma^*$

```python
wkk = 1 + dtsigmastar + wkk_part1
wkk.simplify()
```

$$\frac{dt^{2} \omega^{2} e^{2 dt \sigma}}{e^{2 dt \sigma} - 2 e^{dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} + 1}$$

## $\frac{1}{w_{k}}$

```python
inv_wkk = 1 / wkk
inv_wkk.simplify()
```

$$\frac{e^{- 2 dt \sigma}}{dt^{2} \omega^{2}} \left(e^{2 dt \sigma} - 2 e^{dt \sigma} \cos{\left (dt \sqrt{\omega^{2} - \sigma^{2}} \right )} + 1\right)$$

## Forward developments

$dt \sigma^*$

```python
order = 8
```

```python
fd_dtsig = series(dtsigmastar, dt, 0, order)
fd_dtsig
```

$$dt \sigma + \frac{dt^{3} \sigma}{12} \omega^{2} + dt^{5} \left(\frac{\omega^{4} \sigma}{240} - \frac{\omega^{2} \sigma^{3}}{180}\right) + dt^{7} \left(\frac{\omega^{6} \sigma}{6048} - \frac{\omega^{4} \sigma^{3}}{1512} + \frac{\omega^{2} \sigma^{5}}{1890}\right) + \mathcal{O}\left(dt^{8}\right)$$

```python
print(latex(fd_dtsig))
```

    dt \sigma + \frac{dt^{3} \sigma}{12} \omega^{2} + dt^{5} \left(\frac{\omega^{4} \sigma}{240} - \frac{\omega^{2} \sigma^{3}}{180}\right) + dt^{7} \left(\frac{\omega^{6} \sigma}{6048} - \frac{\omega^{4} \sigma^{3}}{1512} + \frac{\omega^{2} \sigma^{5}}{1890}\right) + \mathcal{O}\left(dt^{8}\right)

 $\frac{1}{2}(1 - \theta_k) \omega^2 dt^2$

```python
fd_wkk_part1 = series(wkk_part1, dt, 0, order)
fd_wkk_part1
```

$ wkk $ and $\frac{1}{wkk}$

```python
fd_invwkk = series(inv_wkk, dt, 0, order)
fd_invwkk
```

$$1 - dt \sigma + dt^{2} \left(- \frac{\omega^{2}}{12} + \frac{2 \sigma^{2}}{3}\right) + dt^{3} \left(\frac{\omega^{2} \sigma}{12} - \frac{\sigma^{3}}{3}\right) + dt^{4} \left(\frac{\omega^{4}}{360} - \frac{\omega^{2} \sigma^{2}}{20} + \frac{2 \sigma^{4}}{15}\right) + dt^{5} \left(- \frac{\omega^{4} \sigma}{360} + \frac{\omega^{2} \sigma^{3}}{45} - \frac{2 \sigma^{5}}{45}\right) + dt^{6} \left(- \frac{\omega^{6}}{20160} + \frac{\omega^{4} \sigma^{2}}{630} - \frac{\omega^{2} \sigma^{4}}{126} + \frac{4 \sigma^{6}}{315}\right) + dt^{7} \left(\frac{\omega^{6} \sigma}{20160} - \frac{\omega^{4} \sigma^{3}}{1512} + \frac{\omega^{2} \sigma^{5}}{420} - \frac{\sigma^{7}}{315}\right) + \mathcal{O}\left(dt^{8}\right)$$

```python
print(latex(fd_invwkk))
```

    1 - dt \sigma + dt^{2} \left(- \frac{\omega^{2}}{12} + \frac{2 \sigma^{2}}{3}\right) + dt^{3} \left(\frac{\omega^{2} \sigma}{12} - \frac{\sigma^{3}}{3}\right) + dt^{4} \left(\frac{\omega^{4}}{360} - \frac{\omega^{2} \sigma^{2}}{20} + \frac{2 \sigma^{4}}{15}\right) + dt^{5} \left(- \frac{\omega^{4} \sigma}{360} + \frac{\omega^{2} \sigma^{3}}{45} - \frac{2 \sigma^{5}}{45}\right) + dt^{6} \left(- \frac{\omega^{6}}{20160} + \frac{\omega^{4} \sigma^{2}}{630} - \frac{\omega^{2} \sigma^{4}}{126} + \frac{4 \sigma^{6}}{315}\right) + dt^{7} \left(\frac{\omega^{6} \sigma}{20160} - \frac{\omega^{4} \sigma^{3}}{1512} + \frac{\omega^{2} \sigma^{5}}{420} - \frac{\sigma^{7}}{315}\right) + \mathcal{O}\left(dt^{8}\right)


# MAT 221 Calculus I

## April 9, 2020

Today's Agenda:
1. Continuous Function
2. Intermediate Value Theorem
3. Exercises

## Derivatives of Logarithmic Functions

Implicit differentation, which we explored in the last section, can also be employed to find the derivatives of logarithmic functions, which are of the form $y = \log_a{x}$. This also includes the natural logarithmic function $y = \ln{x}$.

## Proving $\frac{d}{dx} (\log_a{x}) = \frac{1}{x \ln{a}}$

Taking advantage of the fact that $y = \log_a{x}$ can be rewritten as an exponential equation, $a^y = x$, we can state the derivative of $\log_a{x}$ as:

$$ \frac{d}{dx} (\log_a{x}) = \frac{1}{x \ln{a}} $$

This can be proved by applying implicit differentiation. First we find the deriative of $y = a^x$. Start by taking the $\ln{}$ of both sides of the equation:

$$ \ln{y} = \ln{a^x} $$

Then exponentiate both sides:

$$ e^{\ln{y}} = e^{\ln{a^x}} $$

As $a^{\ln{x}}$ = $x \ln{e}$, and $\ln{e} = 1$, we can simplify the left side of the equation to remove the exponent and natural log.

$$ y = e^{\ln{a^x}} $$

Proceed to find the derivative by using the chain rule:

$$ \frac{dx}{dy} = e^{\ln{a^x}}, \qquad y = e^u, \qquad u = \ln{a^x} = x\ln{a} $$

$$ \frac{dy}{du} = e^u, \qquad \frac{du}{dx} = \ln{a} $$

$$ \ln{a} e^u = \ln{a} e^{\ln{a}} $$

Just as $\ln{e} = 1$, $e^{\ln{x}} = x$, thus:

$$ \frac{dy}{dx} \ln{a} \space a^x $$

Returning to the derivative of $y = \log_a{x}$, rewrite the equation:

$$ a^y = x $$

Now that we know $\frac{dy}{dx} \ln{a} \space a^x$, we can derive the equation implicitly with respect to $x$:

$$ \frac{d}{dx} a^y = \frac{d}{dx} x $$

$$ a^y \ln{a} \frac{dy}{dx} = 1 $$

Then, dividing by $a^y \ln{a}$:

$$ \frac{d}{dx} \log_a{x} = \frac{1}{x \ln{a}} $$

## Proving $\frac{d}{dx} (\ln{x}) = \frac{1}{x}$

Start with $y = \ln{x}$. Exponentate both sides of the equation:

$$ e^y = e^{\ln{x}} $$

Recalling that $e^{\ln{x}} = x$:

$$ e^y = x $$

Applying implicit differentiation:

$$ \frac{dy}{dx} e^y = \frac{d}{dx} x $$

$$ e^y \frac{dy}{dx} = 1 $$

Using the fact that $e^y = x$, we can substitute $x$ for $e^y$:

$$ x \frac{dy}{dx} = 1 $$

Divide by $x$:

$$ \frac{dy}{dx} = \frac{1}{x} $$

## Examples

```python
from sympy import symbols, diff, simplify, sin, cos, log, tan, sqrt, init_printing
from mpmath import ln, e

init_printing()
x = symbols('x')
y = symbols('y')
```

### Example 1: Calculate the derivative of $f(x) = \sin{\ln{x}}$

Apply the Chain Rule:

$$ y = \sin{u}, \qquad u = \ln{x} $$

As we proved above, $\frac{du}{dx} = \frac{1}{x}$, therefore:

$$ \frac{dy}{dx} = \frac{1}{x} \cos{\ln{x}} $$

```python
diff(sin(ln(x)))
```

## Example 2: Find the derivative of the function $f(x) = \log_2{(1 - 3x)}$

Using the fact that $\frac{d}{dx} (\log_a{x}) = \frac{1}{x \ln{a}}$, apply the Chain Rule:

$$ y = \log_2{u}, \qquad u = 1 - 3x $$

$$ \frac{dy}{du} = \frac{1}{u \ln{2}}, \qquad \frac{du}{dx} = 3 $$

$$ \frac{dy}{dx} = \frac{3}{(1 - 3x) \ln{2}} $$

```python
diff(log((1 - 3 * x), 2))
```

### Example 3: Find the derivative to the function $\sqrt[5]{\ln{x}}$

First, rewrite the equation as $\sqrt[5]{\ln{x}} = (\ln{x})^{\frac{1}{5}}$. Then, apply the Chain Rule:

$$ y = u^\frac{1}{5}, \qquad u = \ln{x} $$

$$ \frac{dy}{du} = \frac{1}{5} (\ln{x})^{-\frac{4}{5}}, \qquad \frac{du}{dx} = \frac{1}{x} $$

$$ \frac{dy}{dx} = \frac{1}{x} \space \frac{1}{5} (\ln{x})^{-\frac{4}{5}} = \frac{(\ln{x})^{-\frac{4}{5}}}{5x} = \frac{1}{5x \ln^\frac{4}{5}{x}} $$

```python
diff((log(x)) ** (1/5))
```

### Example 4: Find the derivative of $f(x) = \sin{x} \ln{5x}$

Start by using the Product rule, $u \frac{dv}{dx} + v \frac{du}{dx}$:

$$ u = \sin{x}, \qquad v = \ln{5x} $$

$$ \frac{dy}{dx} = (\sin{x}) \frac{dv}{dx} \ln{5x} + (\ln{5x}) \frac{du}{dx} \sin{x} $$

$$ \frac{dy}{dx} = \frac{\sin{x}}{x} + \ln{5x} \cos{x} $$

```python
diff(sin(x) * log(5 * x))
```

### Example 5: Find the derivative of the function $F(x) = \ln{\frac{(2x + 1)^3}{(3x - 1)^4}}$

Here, rather than applying the Chain Rule and then the Quotient Rule to the inner equation, which would result in a very lengthy and tedious derivation, we can take advantage of one of the logarithmic identities, $\ln{ab} = \ln{a} + \ln{b}$. Thus, we can rewrite the function as:

$$ F(x) = \ln{(2x + 1)^3} + \ln{\frac{1}{(3x - 1)^4}} $$

With the rewritten function, we can leverage another logarithmic identity, $\ln{a^b} = b \ln{a}$ to simplify the derivation. We can therefore rewrite the function again as:

$$ F(x) = 3 \ln{(2x + 1)} - 4 \ln{(3x - 1)} $$

Then, proceed to differentiate the function:

$$ \frac{dy}{dx} = 3 \frac{d}{dx} \ln{(2x + 1)} - 4 \frac{d}{dx} \ln{(3x - 1)} $$

$$ \frac{dy}{dx} = 3 \frac{2}{2x + 1} - 4 \frac{3}{3x - 1} = \frac{6}{2x + 1} - \frac{12}{3x - 1} $$

```python
simplify(diff(log((2 * x + 1) ** 3 / (3 * x - 1) ** 4)))
```

## References

Stewart, J. (2007). Essential calculus: Early transcendentals. Belmont, CA: Thomson Higher Education.