```python
import numpy as np
```

Pontos da quadratura gaussiana

```python
# um ponto
kxi1   = np.array([0.0])
alpha1 = np.array([2.0])
# 2 pontos
kxi2   = np.array([-0.577350269189626e0, 0.577350269189626e0])
alpha2 = np.array([1.0, 1.0])
# 3 pontos
kxi3   = np.array([-0.774596669241483e0, 0.0e0              , 0.774596669241483e0])
alpha3 = np.array([ 0.555555555555556e0, 0.888888888888889e0, 0.555555555555556e0])
# 4 pontos
kxi4   = np.array([-0.861136311594053e0, -0.339981043584856e0, 0.339981043584856e0, 0.861136311594053e0])
alpha4 = np.array([ 0.347854845137454e0,  0.652145154862546e0, 0.652145154862546e0, 0.347854845137454e0])
# 5 pontos
kxi5   = np.array([-0.906179845938664e0, -0.538469310105683e0, 0.0e0                , 0.538469310105683e0, 0.906179845938664e0])
alpha5 = np.array([ 0.236926885056189e0,  0.478628670499366e0, 0.568888888888889e0, 0.478628670499366e0, 0.236926885056189e0])

ksis   = [kxi1, kxi2, kxi3, kxi4, kxi5]
alphas = [alpha1, alpha2, alpha3, alpha4, alpha5]
```

 Função que se quer integrar

```python
def f1(e):
  return e**2 - 64*e + 1

def f2(n, e):
  return (e**2 + 1)*(n+1)
```

Quadratura gaussina para o intervalor [-1, 1]

```python
def quadratura_gauss(func, n_pontos=5):
  '''
  ----------------------------------------------------------------
  Quadratura gaussiana no intervalor [-1, 1]
  ----------------------------------------------------------------
  func     - funcao que se quer calcular a integral
  n_pontos - numero de pontos de gaus
  ----------------------------------------------------------------
  Retorna o valor da integral
  ----------------------------------------------------------------
  '''
  pontos = ksis[n_pontos-1]
  pesos  = alphas[n_pontos-1]

  tmp = 0.0e0

  for i in range(0, n_pontos):
    ksi_i   = pontos[i]
    alpha_i = pesos[i]
    tmp += func(ksi_i) * alpha_i

  return tmp

def quadratura_gauss_dupla(func, n_pontos=5):
  '''
  ----------------------------------------------------------------
  Quadratura gaussiana dupla no intervalor [-1, 1]
  ----------------------------------------------------------------
  func     - funcao que se quer calcular a integral
  n_pontos - numero de pontos de gaus
  ----------------------------------------------------------------
  Retorna o valor da integral
  ----------------------------------------------------------------
  '''
  pontos = ksis[n_pontos-1]
  pesos  = alphas[n_pontos-1]

  tmp = 0.0e0
  for i in range(0, n_pontos):
    ksi_i   = pontos[i]
    alpha_i = pesos[i]
    for j in range(0, n_pontos):
      ksi_j   = pontos[j]
      alpha_j = pesos[j]
      tmp += func(ksi_i, ksi_j)*alpha_i*alpha_j

  return tmp
```

Integral simples:

\begin{equation}
valor = \int_{-1}^{1} f(\xi) d\xi
\end{equation}

onde

\begin{equation}
f(\xi) = \xi^2 - 64\xi + 1
\end{equation}

Calculo da integral para os pontos de 1 a 5

```python
for n in [1,2,3,4,5]:
  valor = quadratura_gauss(f1, n)
  print(f'O valor da integral para {n} de integração é = {valor:.14e}')
```

    O valor da integral para 1 de integração é = 2.00000000000000e+00
    O valor da integral para 2 de integração é = 2.66666666666667e+00
    O valor da integral para 3 de integração é = 2.66666666666667e+00
    O valor da integral para 4 de integração é = 2.66666666666667e+00
    O valor da integral para 5 de integração é = 2.66666666666667e+00

Integral dupla:

\begin{equation}
valor = \int_{-1}^{1} \int_{-1}^{1} f(\eta, \xi) d\eta d\xi
\end{equation}

onde

\begin{equation}
f(\xi, \eta) = (\xi^2 +1)(\eta +1)
\end{equation}

Calculo da integral para os pontos de 1 a 5

```python
for n in [1,2,3,4,5]:
  valor = quadratura_gauss_dupla(f2, n)
  print(f'O valor da integral para {n} de integração é = {valor:.14e}')
```

    O valor da integral para 1 de integração é = 4.00000000000000e+00
    O valor da integral para 2 de integração é = 5.33333333333333e+00
    O valor da integral para 3 de integração é = 5.33333333333334e+00
    O valor da integral para 4 de integração é = 5.33333333333333e+00
    O valor da integral para 5 de integração é = 5.33333333333333e+00


```python
import sympy, numpy
from sympy.abc import x, theta
from sympy import init_printing

init_printing()

s_nu = sympy.Symbol("\\nu")
s_nu_p = sympy.Symbol("{\\nu^\\prime}")
s_omega = sympy.Symbol("{\\omega}")
s_U = sympy.Symbol("U")
s_beta = sympy.Symbol("{\\beta}")

s1 = s_nu + s_omega
s2 = s_nu_p + s_omega
s3 = s_nu * s_nu_p * s1 * s2

# Implementation of Eq.(31)
Fud_1 = -s_U + sympy.Rational(1,8) * (s_U**3) * (s_nu**2 + s1**2 + s2**2 + s_nu_p**2)/s3
Fud_2 = sympy.Rational(3,16)* (s_U**5)/s3
Fud_3 = s_beta * ((s_U**2)/4) * (1/(1+sympy.exp(s_beta*s_U/2))) * (2*sympy.KroneckerDelta(s_nu, -s2) + sympy.KroneckerDelta(s_omega,0)) * (1 + s_U**2/(4*s1**2)) * (1 + s_U**2/(4*s2**2))
Fud_4 = -s_beta * ((s_U**2)/4) * (1/(1+sympy.exp(-s_beta*s_U/2))) * (2*sympy.KroneckerDelta(s_nu, s_nu_p) + sympy.KroneckerDelta(s_omega,0)) * (1 + s_U**2/(4*s_nu**2)) * (1 + s_U**2/(4*s2**2))

Fud = Fud_1 + Fud_2 + Fud_3 + Fud_4
Fud
```

```python
beta = 5.0
U = 2.0
```

```python
# Substitute \nu = 1, \nu^\prime=1, \omega=1 into the analytic expression of Eq.(31), where integers much be interpreted as
# fermionic or bosonic Matsubara frequencies.
n, n_p, m = 1, 1, 1
Fud.subs([(s_nu, (2*n+1)*numpy.pi/beta), (s_nu_p,(2*n_p+1)*numpy.pi/beta), (s_omega,(2*m)*numpy.pi/beta), (s_U,2), (s_beta,5)]).evalf()
```

```python
import numpy
import scipy
import sys

def delta(i, j):
    if i==j:
        return 1
    else:
        return 0

# Numerical implementation of Eqs.(30) [Fuu] and (31) [Fud]
def F_ph(n, np, m):
    nu = (2*n+1)*numpy.pi/beta
    nu_p = (2*np+1)*numpy.pi/beta
    omega = 2*m*numpy.pi/beta

    r1 = nu+omega
    r2 = nu_p+omega
    tmp = 1/(nu * r1 * r2 * nu_p)

    Fuu = -0.25*beta*(U**2)*(delta(n,np)-delta(m,0))*(1+0.25*(U/nu)**2)*(1+0.25*(U/r2)**2)

    t1 = 0.125*(U**3)*(nu**2+r1**2+r2**2+nu_p**2)*tmp
    t2 = (3.0/16.0) * (U**5) * tmp
    t3 = beta * 0.25 * (U**2) * (1/(1+numpy.exp(0.5*beta*U))) * (2*delta(nu,-nu_p-m)+delta(m,0)) * (1+0.25*(U/r1)**2) * (1+0.25*(U/r2)**2)
    t4 = - beta * 0.25 * (U**2) * (1/(1+numpy.exp(-0.5*beta*U))) * (2*delta(nu, nu_p)+delta(m,0)) * (1+0.25*(U/nu)**2) * (1+0.25*(U/r2)**2)

    Fud = -U + t1 + t2 + t3 + t4

    return Fuu, Fud

# Single-particle Green's function: poles at omega = +/- U/2
def G1(n):
    iwn = 1J * (2*n+1)*numpy.pi/beta

    return 0.5/(iwn - U/2) + 0.5/(iwn + U/2)

# The second term of the right hand side of Eq.(9)
def G2_conn_ph(n, np, m):
    Fuu, Fud = F_ph(n, np, m)
    leggs = - G1(n) * G1(np) * G1(n + m) * G1(np + m)
    return leggs*Fuu, leggs*Fud

print(F_ph(1,1,1))
print(G2_conn_ph(1,1,1))
```

    (-7.056427666781439, -15.081762560490278)
    ((0.10103064440229036+0j), (0.21593365115633148+0j))

```python
import matplotlib.pylab as plt
import matplotlib.cm as cm
from mpl_toolkits.mplot3d import Axes3D

%matplotlib inline

N = 30
x = numpy.linspace(-N, N, 2*N+1)
z = numpy.zeros((2*N+1, 2*N+1), dtype=float)
m = 10

for i in range(2*N+1):
    for j in range(2*N+1):
        n = x[i]
        np = x[j]
        G2_conn_uu_ph, G2_conn_ud_ph = G2_conn_ph(n, np, m)
        # Eq. (17): density channel
        z[i,j] = (G2_conn_uu_ph + G2_conn_ud_ph).real

vrange = numpy.amax(numpy.abs(z))
plt.imshow(z.T, origin="lower", extent=(numpy.amin(x), numpy.amax(x), numpy.amin(x), numpy.amax(x)), cmap=cm.bwr, vmin=-vrange, vmax=vrange)
plt.colorbar()

plt.legend(loc='best',shadow=True,frameon=False)
```


# 微分方程式の計算について　　　N0.5　－1　定数変化法

###  学籍番号[＿＿＿＿＿＿＿＿＿]クラス[＿＿＿＿＿]　クラス番号[＿＿＿＿＿] 名前[＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿]

###########   1階常微分方程式 定数変化法 (同次形の方程式　としての解を求め,次に定数Cをu(x)に代えて（1）式に代入してu(x)を求めていく

$$ \frac{dy}{dx}+P(x)y = q(x)------------（１）$$

はじめに同次形の方程式　としての解を求める

$$ \frac{dy}{dx}+f(x)y = 0 $$

$$ y{\left (x \right )} = C_{1} e^{- \int f{\left (x \right )}\, dx}----------(2)$$

C1をｘの関数としてu(x)と置き換えてみる
$$ y = u(x)e^{- \int f(x)dx} $$
このｙを(1)式の非同次の式に代入して、
u(x)を求める。
(2)のC1にu(u)を代入すして解とする

```python
from sympy import *
x, n , a ,C1, C2 ,C3 = symbols('x n a C1 C2 C3')
f,g,y = symbols('f g y' ,cls=Function)
init_printing()
m ='5///1'
i =0
```

```python
diffeq = Eq(y(x).diff(x, 1)- y(x)/x ,x)
i =0
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python

```

```python
dsolve(diffeq, y(x))
```

```python
diffeq = Eq(x*y(x).diff(x, 1)+y(x) , x*exp(x))
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python
dsolve(diffeq, y(x))
```

```python
diffeq = Eq(cos(x)* y(x).diff(x,1) - y(x)*sin(x),1)
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python
dsolve(diffeq, y(x))
```

```python
diffeq = Eq(y(x).diff(x, 1)+ 2*x*y(x),(2*x +1)*exp(x))
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python
simplify(dsolve(diffeq, y(x)))
```

```python
diffeq = Eq(y(x).diff(x, 1)+(2/x)*y(x),x**2)
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python
simplify(dsolve(diffeq, y(x)))
```

```python
diffeq = Eq(y(x).diff(x, 1)+y(x) ,x)
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python
reseq = dsolve(diffeq, y(x))
reseq
```

```python
f0 = Eq(reseq.rhs.subs(x,0),0)
c = solve(f0)
c
```

```python
reseq01 = reseq.subs(C1, c[0])
reseq01
```

```python
diffeq = Eq(cos(x)*y(x).diff(x, 1)-sin(x)*y(x) ,1)
i=i+1
print( 'No.',m,'---',i)
diffeq
```

```python
reseq = dsolve(diffeq, y(x))
reseq
```

```python
f0 = Eq(reseq.rhs.subs(x,0),4)
c = solve(f0)
c
```

```python
reseq01 = reseq.subs(C1, c[0])
reseq01
```

```python

```


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from u_xx_f_sympy import model2, x
import sympy as sym
import numpy as np
from fe1D import finite_element1D, mesh_uniform, u_glob
import matplotlib.pyplot as plt

C = 5
D = 2
L = 4

m_values = [0, 1, 2, 3, 4]
d_values = [1, 2, 3, 4]
for m in m_values:
    u = model2(x**m, L, C, D)
    print(('\nm=%d, u: %s' % (m, u)))
    u_exact = sym.lambdify([x], u)

    for d in d_values:
        vertices, cells, dof_map = mesh_uniform(
            N_e=2, d=d, Omega=[0,L], symbolic=False)
        vertices[1] = 3  # displace vertex
        essbc = {}
        essbc[dof_map[-1][-1]] = D

        c, A, b, timing = finite_element1D(
            vertices, cells, dof_map,
            essbc,
            ilhs=lambda e, phi, r, s, X, x, h:
            phi[1][r](X, h)*phi[1][s](X, h),
            irhs=lambda e, phi, r, X, x, h:
            x**m*phi[0][r](X),
            blhs=lambda e, phi, r, s, X, x, h: 0,
            brhs=lambda e, phi, r, X, x, h:
            -C*phi[0][r](-1) if e == 0 else 0,
            intrule='GaussLegendre')

        # Visualize
        # (Recall that x is a symbol, use xc for coordinates)
        xc, u, nodes = u_glob(c, vertices, cells, dof_map)
        u_e = u_exact(xc)
        print(('Max diff at nodes, d=%d:' % d, \
              np.abs(u_exact(nodes) - c).max()))
        plt.figure()
        plt.plot(xc, u, 'b-', xc, u_e, 'r--')
        plt.legend(['finite elements, d=%d' %d, 'exact'],
                   loc='lower left')
        figname = 'tmp_%d_%d' % (m, d)
        plt.savefig(figname + '.png'); plt.savefig(figname + '.pdf')
    for ext in 'pdf', 'png':
        cmd = 'doconce combine_images -2 '
        cmd += ' '.join(['tmp_%d_%d.' % (m, d) + ext
                         for d in d_values])
        cmd += ' u_xx_xm%d_P1to4.' % m + ext
        print(cmd)
        os.system(cmd)

#plt.show()

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/28_U_XX_XM_P1TO4.ipynb


# Fast Radial Potential

The goal is sum up radial potential function with nice properties such as
  - fast evaluation (e.g. low degree polynominal)
  - finite support (to allow linear scaling N-body simulation)
  - smooth derivatives (1st, 2nd)
  - can be computed from $r^2$ rather than $r$ (so save `sqrt()` in vector lenght evaluation)
  - crucial features such as (1) minimum position (2) minimum energy (3) node position can be easily tuned
  - can be factorized for parir of atoms (therefore it can be stored on grid)

```python
# initialize environment
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import sympy as sy
x = np.linspace(0.0,10.0,1000)
dx = x[1]-x[0]

def numDeriv( x, f ):
    return (x[1:]+x[:-1])*0.5, (f[1:]-f[:-1])/(x[1:]-x[:-1])

```

### How to use $|\vec r|^2$ instead of $|\vec r|$ (get rid of `sqrt()`)

$F(\vec r) = {\vec r} (F(r)/r$)

So we have to express $f(r)=F(r)/r$ in terms of only $(r^2)^N$. For example function $F(r) = (1-r^2)$ is not good since $f(r)=(1-r^2)/r$ requires explicit evaluation of $r^{-1}$. But function $F(r) = (1-r^2)*r$ and $F(r) = (1-r^2)/r$ are good since $f(r)=(1-r^2)$ and $f(r)=(1-r^2)/r^2$ can be efficinetly expressed in terms of only $(r^2)^N$. Notice, that for any polynominal radial potential $V(r)$ composed of only $(r^2)^N$ the resulting force ( i.e. derivative $\partial_r r^N = r^{N-1}$ ) always fullfill this condition.

```python
def func1(r):
    r2 = r**2
    E  = 1-r2
    fr = -2
    return E,fr*r

def func2(r):
    r2 = r**2
    E  = (1-r2)**2
    fr  = -2*(1-r2)
    return E,fr*r

def func3(r):
    r2 = r**2
    E  = 0.1*(    (1-r2)*     (4-r2)  )
    fr = 0.1*( -2*(1-r2) + -2*(4-r2)  )
    return E,fr*r

def func4(r):
    r2 = r**2
    E  = 0.1*(    (1-r2)*            (4-r2)**2  )
    fr = 0.1*( -4*(1-r2)*(4-r2) + -2*(4-r2)**2  )
    return E,fr*r

def func5(r):
    r2 = r**2
    E  = 0.05*(    (1-r2)*               (4-r2)**3  )
    fr = 0.05*( -6*(1-r2)*(4-r2)**2 + -2*(4-r2)**3  )
    return E,fr*r

def func6(r):
    r2 = r**2
    E  = 0.025*(    (1-r2)*               (4-r2)**4  )
    fr = 0.025*( -8*(1-r2)*(4-r2)**3 + -2*(4-r2)**4  )
    return E,fr*r

funcs = [func1,func2,func3,func4,func5,func6]
for func in funcs:
    E,F = func(x)
    plt.subplot(2,1,1); plt.plot(x,E);
    plt.subplot(2,1,2); plt.plot(x,-F);

plt.subplot(2,1,1); plt.ylim(-1.0,1.0); plt.xlim(0.0,4.0); plt.grid(); plt.ylabel("Energy"); plt.axhline(0,c='k',ls='--')
plt.subplot(2,1,2); plt.ylim(-4.0,4.0); plt.xlim(0.0,4.0); plt.grid(); plt.ylabel("Force"); plt.axhline(0,c='k',ls='--')

```

## Factorized Polynominals

```python
a = (1.0-x**2)   # this per atom
b = (x-2)**2     # this will be on grid
plt.plot(x,a,label='a')
plt.plot(x,b,label='b')
plt.plot(x,a*b,lw=2,c='k',label='c')
vmax=1.00; plt.ylim(-vmax,vmax); plt.xlim(0.0,4.0); plt.grid(); plt.legend()
```

## Approx exponential

```python
alpha = -1.6

def getMorse( r, R, eps, alpha=alpha ):
    return eps*( np.exp(2*alpha*(r-R)) - 2*np.exp(alpha*(r-R)) )

def fastExp( x, n=4 ):
    e = 1.0 + x/np.power(2.0,n);
    for i in range(n): e*=e
    return e

def getFastMorse( r, R, eps, alpha=alpha, n=4 ):
    expar = fastExp(alpha*(r-R), n=n )
    return eps*( expar*expar - 2*expar )

plt.plot( x, getMorse       ( x, 4.0, 1.0 ), ':k', lw=2, label=('exact') )
for i in range(5):
    plt.plot( x, getFastMorse( x, 4.0, 1.0, n=i ), ls='-', label=('aprox n=%i' %i ) )

vmax=1.00; plt.ylim(-vmax,vmax); plt.xlim(2.0,10.0); plt.grid(); plt.legend()
```

### Approx Gaussian

```python
def Gaussian( r ):
    return np.exp(-r*r)

def fastGauss( r, n=4 ):
    x = r*r
    e = 1.0 - x/np.power(2.0,n);
    for i in range(n): e*=e
    return e

plt.plot( x , Gaussian( x), '--k', label=('exact') )
for i in range(5):
    plt.plot( x, fastGauss( x, n=i ), ls='-', label=('aprox n=%i' %i ) )

plt.ylim(-0.5,1.0); plt.xlim(0.0,4.0); plt.grid(); plt.legend()
```

### Exp power

```python
def fastExp( x, n=4 ):
    e = 1.0 + x/np.power(2.0,n);
    e[e<0] = 0
    for i in range(n): e*=e
    return e

xs = np.linspace(0.0,10.0,300)

plt.plot( xs, np.exp(-xs), '--k', label=('exact') )
for i in range(5):
    plt.plot( xs, fastExp(-xs, n=i ), ls='-', label=('aprox n=%i' %i ) )

plt.ylim(-0.5,1.0); plt.xlim(0.0,6.0); plt.grid(); plt.legend()
```

# Radial polynomynal potential

```python

'''
V  = (A-r2)*(B-r2)**4
fr = ( 4*(A-r2) - (B-r2) )*-2*(B-r2)**3
'''

r, r2, A, B, C = sy.symbols('r r2 A B C')

V = (A-r2)*(B-r2)**2
F = sy.diff(V, r2)
#F = sy.simplify(sy.expand(F))
F = sy.factor(F)
print F
F = F.expand()
print "coefs : "
#print sy.collect(F,r2)
print " ^0 : ",F.coeff(r2, 0)
print " ^1 : ",F.coeff(r2, 1)
print " ^2 : ",F.coeff(r2, 2)
print " ^3 : ",F.coeff(r2, 3)
print " ^4 : ",F.coeff(r2, 4)

print "solve : ", sy.solve(F,r2)
```

```python
def UniPolyPot( r, Rmax=4.0, Rmin=2.0, n=4):
    '''
    potential in form $ V(r) = (A-r^2) * (B-r^2)^n $
    '''
    r2   = r**2
    C    = Rmin**2
    B    = Rmax**2
    print "C=",C,"n=",n," C*(n+1)=",C*(n+1)," B=",B
    A    = (C*(n+1) - B)/n; print "A =",A," R0 =", np.sqrt(A)
    resc = -1/((A-C)*(B-C)**n);  print "resc = ", resc
    ea   = A-r2
    eb   = B-r2
    ebn  = eb**(n-1) * resc
    E    = ea * eb * ebn
    fr   = ( n*ea + eb) * ebn * 2
    return E, fr*r

def plotUniPolyPot( Rmax=4.0, Rmin=2.0, n=4, clr=None ):
    E,F = UniPolyPot(x, Rmax=Rmax, Rmin=Rmin, n=n )
    plt.subplot(2,1,1); plt.plot(x,E, c=clr, label=("%i" %n));
    plt.subplot(2,1,2); plt.plot(x,F, c=clr, label=("%i" %n));
    plt.plot((x[:-1]+x[1:])*0.5,(E[:-1]-E[1:])/dx, ls=":",c=clr, label=("%i" %n));
    plt.axvline(Rmax,c=clr, ls="--")

plotUniPolyPot( Rmax=4.0, Rmin=3.0, n=2, clr='r' )
plotUniPolyPot( Rmax=4.4, Rmin=3.0, n=3, clr='g' )
plotUniPolyPot( Rmax=4.8, Rmin=3.0, n=4, clr='b' )

plt.subplot(2,1,1); plt.ylim(-1.0,1.0); plt.xlim(0.0,6.0); plt.grid(); plt.ylabel("Energy"); plt.axhline(0,c='k',ls='--')
plt.subplot(2,1,2); plt.ylim(-4.0,4.0); plt.xlim(0.0,6.0); plt.grid(); plt.ylabel("Force"); plt.axhline(0,c='k',ls='--')
```

```python
def R4(r,   REmin=2.0, Emin=-0.2, E0=1.0 ):
	REmin2  = REmin**2
	Scaling = (E0-Emin)/(REmin2**2)
	Rmax    = np.sqrt(REmin2+np.sqrt(-(Emin/Scaling)))
	R0      = np.sqrt(REmin2-np.sqrt(-(Emin/Scaling)))
	mask=(r>Rmax)
	r2=r**2
	f   = Scaling*( (REmin2-r2)**2    ) + Emin
	df  = Scaling*( 4*(REmin2-r2)*r   )
	ddf = Scaling*( 4*REmin2-12*r2    )
	f[mask]=0; df[mask]=0; ddf[mask]=0;
	return f,df,ddf, Rmax,R0,REmin

def R4_(r,   R0=1.0, Rmax=2.0, Emin=-0.2 ):
	Rmax2=Rmax**2; R02=R0**2
	REmin2  = (Rmax2 + R02)*0.5
	Scaling = -4*Emin/(Rmax2-R02)**2
	mask=(r>Rmax)
	r2=r**2
	f   = Scaling*( (REmin2-r2)**2    ) + Emin
	df  = Scaling*( 4*(REmin2-r2)*r   )
	ddf = Scaling*( 4*REmin2-12*r2    )
	f[mask]=0; df[mask]=0; ddf[mask]=0;
	return f,df,ddf, Rmax,R0,np.sqrt(REmin2)

def LR2(r, C=-2.5, K=1.0, A=1.0, s=0.1 ):
	r2=r**2
	f   = K*r2  + A/(s+r2)  +   C
	df  = -( 2*K*r - 2*A*r/(s+r2)**2 )
	ddf = -( 2*K   - 2*A/(s+r2)**2  + 8*A*r2/(s+r2)**3 )
	return f,df,ddf,0,0,0

rs   = np.linspace(0,4.0,100)
#func = R4
func = R4_
#func = LR2
f,df,ddf, Rmax,R0,REmin  = func(rs)
dr   = rs[1]-rs[0]
df_  = -(f[2:]-f[:-2])/(2*dr)
ddf_ = (df_[2:]-df_[:-2])/(2*dr)
plt.figure(figsize=(5,15))
plt.subplot(3,1,1); plt.plot(rs,f)   ; plt.axhline(0,ls='--',color='k'); plt.axvline(R0,ls='--',color='k');    plt.axvline(REmin,ls='--',color='k'); plt.axvline(Rmax,ls='--',color='k'); #  plt.ylim(-1,1);
plt.subplot(3,1,2); plt.plot(rs,df)  ; plt.axhline(0,ls='--',color='k'); plt.axvline(REmin,ls='--',color='k'); plt.axvline(Rmax,ls='--',color='k'); plt.plot(rs[1:-1],df_); #  plt.ylim(-5,5);
plt.subplot(3,1,3); plt.plot(rs,ddf) ; plt.axhline(0,ls='--',color='k'); plt.axvline(Rmax,ls='--',color='k');  plt.plot(rs[2:-2],ddf_); #  plt.ylim(-10,10);
plt.show()
```

### Even reciprocal polynominal

```python
'''
$ eps*( A/r^8 - 1/r^6) = eps*(R0^2-r^2)/(r^2)^4 $
$ A = R0^2 $
'''

r, r2, A, B, C = sy.symbols('r r2 A B C')

V = (A-r2)/r2**4  # A/r^8
F = sy.diff(V, r2)
#F = sy.simplify(sy.expand(F))
F = sy.factor(F)
print F
print "solve : ", sy.solve(F,r2)
```

```python
def PolyInvR2( r, Rmin=3.0):
    '''
    '''
    R0    = Rmin*np.sqrt(3.0/4.0)  #*(3.0/4.0)
    print Rmin, R0
    r2    = r**2
    A     = R0**2
    ir2   = 1.0/r2;
    resc  = -1.0/( (1/Rmin**8)*( A-Rmin**2 ) ); print resc
    E     = resc*(ir2**4)*( A-r2 )
    fr    = resc*(ir2**4)*(4*A*ir2-3)*2
    return E, fr*r

def LenardLones( r, Rmin ):
    r2   = r**2
    ir2  = Rmin**2/r2
    ir6  = ir2**3
    resc = 1.0
    E    = resc*( ir6 - 2 )*ir6
    fr   = resc*( ir6 - 1 )*ir6*ir2*(2.0**(1.0/2.5)) # WHY ?
    return E, fr*r

def plotPolyInvR2( Rmin=3.0, clr=None ):
    E,F = PolyInvR2(x, Rmin=Rmin )
    E_,F_ = LenardLones( x, Rmin=Rmin )
    plt.subplot(2,1,1); plt.plot(x,E, c=clr);  plt.plot(x,E_, c=clr, ls='--');
    plt.subplot(2,1,2); plt.plot(x,F, c=clr);  plt.plot(x,F_, c=clr, ls='--');
    plt.plot((x[:-1]+x[1:])*0.5,(E_[:-1]-E_[1:])/dx, ls=":", lw=2.0, c=clr);
    plt.axvline(Rmax,c=clr, ls="--")

plotPolyInvR2( Rmin=3.0, clr='r' )
#plotPolyInvR2( , clr='g' )
#plotPolyInvR2( , clr='b' )

plt.subplot(2,1,1); plt.ylim(-1.0,1.0); plt.xlim(0.0,6.0); plt.grid(); plt.ylabel("Energy"); plt.axhline(0,c='k',ls='--')
plt.subplot(2,1,2); plt.ylim(-1.0,1.0); plt.xlim(0.0,6.0); plt.grid(); plt.ylabel("Force"); plt.axhline(0,c='k',ls='--')
```

# Buckingham potential
https://en.wikipedia.org/wiki/Buckingham_potential

```python
def getBuckingham( r, R0=3.4, eps=0.030, alpha=1.8 ):
    '''
    V   =  eps *(  (6/(a-6)) * exp( a * (1-(R/R0) )  -  (a/(a-6)) *(R0/R)**6   )
    V   =  (eps/(a-6)) *(  6*exp( a * (1-(R/R0) )  -  a*(R0/R)**6 )
    V   =  (eps/(a-6)) *(  6*exp( -(a/R0)*(R0-R) ) -  a*(R0/R)**6 )
    '''
    a    = alpha*R0
    pref = eps/(a-6)
    A = pref * 6
    #B = pref * a * (R0**6)
    B = pref * a
    print R0, eps, alpha, " | ", a, pref, " | ", A, B
    #V = A*np.exp( -alpha*(r-R0) ) + B/(r**6)
    V = A*np.exp( -alpha*(r-R0) ) - B*(R0/r)**6
    return V

eps = 0.03
R0  = 3.4
alpha=1.8

V = getBuckingham( x, R0=R0, eps=eps, alpha=alpha ); #print V
x_,F = numDeriv( x, V )

plt.subplot(2,1,1); plt.plot(x,  V )
plt.subplot(2,1,2); plt.plot(x_, F )

plt.subplot(2,1,1); plt.ylim(-0.1,0.1); plt.xlim(0.0,6.0); plt.grid(); plt.ylabel("Energy"); plt.axhline(0,c='k',ls='--');
plt.axvline( R0,c='k',ls='-'); plt.axhline( -eps,c='k',ls='-')

plt.subplot(2,1,2); plt.ylim(-0.1,0.1); plt.xlim(0.0,6.0); plt.grid(); plt.ylabel("Force"); plt.axhline(0,c='k',ls='--')
```

```python
alpha = -1.6

def getMorse( r, R, eps, alpha=alpha ):
    return eps*( np.exp(2*alpha*(r-R)) - 2*np.exp(alpha*(r-R)) )

def getElec( r, qq, w2=4 ):
    return 14.3996448915*qq/(w2+r*r)

Emorse = getMorse( x, 3.0, 0.095*0.026 )
Eelec  = getElec ( x, -0.4*0.2 )

plt.plot( x, Emorse,        '-r', lw=2, label=('Morse') )
plt.plot( x, Eelec,         '-b', lw=2, label=('Elec') )
plt.plot( x, Emorse+Eelec , '-k', lw=2, label=('Morse+Elec') )

vmax=0.5; plt.ylim(-vmax,vmax); plt.xlim(0.0,10.0); plt.grid(); plt.legend()
```

```python

```

```python

```


```julia
using DifferentialEquations
using Plots
using LaTeXStrings
pgfplotsx()
```

    Plots.PGFPlotsXBackend()

Consider the simple reaction:

\begin{align}
    A + * &\longleftrightarrow A_*\\
    A_*   &\longleftrightarrow B + *\\
\end{align}

Both are elementary steps that occur in the liquid phase, and we will consider it in a few different solvent environments.

```julia
function batch(du, u, p, t)
    MAR = p["MAR"]
    PAR = p["PAR"]

    k20, K10, K20, Ns, NS = PAR

    NA     = u[:,1]
    NB     = u[:,2]

    NT     = NA + NB .+ NS

    XA   = NA./NT
    XB   = NB./NT
    XS   = NS./NT

    #For A in solvent
    A12A = MAR[1]

    #For As in solvent
    A12As = MAR[2]

    #For B in solvent
    A12B = MAR[3]

    #For Transition State 1 in solvent
    A12TS1 = MAR[4]

    #For Transition State 2 in solvent
    A12TS2 = MAR[5]

    gammaA  = exp.(XS.^2*A12A)
    gammaAs = exp.(XS.^2*A12As)
    gammaB  = exp.(XS.^2*A12B)

    gammaTS1 = exp.(XS.^2*A12TS1)
    gammaTS2 = exp.(XS.^2*A12TS2)

    thetastar = 1 ./(1 .+ K10*gammaA./gammaAs.*XA)
    thetaA   = K10*gammaA./gammaAs.*XA.*thetastar

    z2 = 1/K20*gammaB./gammaAs.*XB.*thetastar./thetaA
    r2 = k20*gammaAs./gammaTS2.*thetaA.*(1 .- z2)

    RA  = -r2[1]
    RB  =       r2[1]

    du[1] = RA*Ns
    du[2] = RB*Ns
    return du, r2, z2, thetastar, thetaA
end
```

    batch (generic function with 1 method)

```julia
K    = 1
Ns   = 0.001
k20  = 0.01
V    = 1
NTOT = 100

NA0  = 0.1
NAs0 = 0.0
NB0  = 0.0

NS   = NTOT - NA0 - NB0

var0     = [NA0 NB0]
span     = (0.0, 10000000);
```

```julia
#Solvate transition state relative to reactants
MARSET1 = zeros(5,3)
MARSET1[:,1] = [0.0, 0.0, 0.0, 0.0, 0.0] #no solvation
MARSET1[:,2] = [0.0, 2.0, 0.0, 0.0, 0.0] #destabilize TS1
MARSET1[:,3] = [0.0, -2.0, 0.0, 0.0, 0.0] #stabilize TS1
K10SET       = [10, 1000, 10000, 100000]#10 .^range(1, stop = 4, length = 4)
K20SET       = K./K10SET

labels = Matrix{String}(undef, 1, length(K10SET))

for m = 1:length(K10SET)
    labels[m] = string("K1 = ", K10SET[m])
end

tfine  = 10 .^(range(-8.0, stop = log10(maximum(span)), length = 1000))
Xout   = zeros(length(tfine), size(K10SET, 1), 3)
r2out  = zeros(length(tfine), size(K10SET, 1), 3)
z2out  = zeros(length(tfine), size(K10SET, 1), 3)
cov    = zeros(length(tfine), size(K10SET, 1), 3)

for i = 1:3
    for j = 1:length(K10SET)
        p0   = Dict("MAR" => MARSET1[:,i], "PAR" => [k20, K10SET[j], K20SET[j], Ns, NS])
        prob = ODEProblem(batch, var0, span, p0)
        sol  = solve(prob, Rodas5(), abstol = 1e-16, reltol = 1e-16)
        solf = sol(tfine)
        NA   = solf[1,:]
        NB   = solf[2,:]
        NT   = NA + NB .+ NS

        dut, rt2, zt2, thetastart, thetaAt = batch([0., 0.], [NA NB], p0, tfine)

        ext  = NB/NA0
        cov[:, j, i]  = thetaAt
        Xout[:, j, i]  = ext
        r2out[:, j, i] = rt2
        z2out[:, j, i] = zt2
    end
end

pertX1   = Xout[:,:,2]./Xout[:,:,1]
pertX2   = Xout[:,:,3]./Xout[:,:,1]
pertC1   = cov[:,:,2]./cov[:,:,1]
pertC2   = cov[:,:,3]./cov[:,:,1];
```

```julia
plt1 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    #yscale = :log10,
    #ylim = (1e-1, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = "Conversion",
    fontfamily = "Times Roman",
    legend  = :topleft,
    legendtitle = L"\ \ \ \ K_1^\circ = 10",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(a)",
    titlelocation = :left,
    titlefontsize = 26
)

plt1 = plot!(plt1, tfine/maximum(tfine), Xout[:,1,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")
plt1 = plot!(plt1, tfine/maximum(tfine), Xout[:,1,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt1 = plot!(plt1, tfine/maximum(tfine), Xout[:,1,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")

display(plt1)
savefig(plt1, "CS31.pdf")
```

```julia
plt2 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    #yscale = :log10,
    #ylim = (1e-1, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = "Conversion",
    fontfamily = "Times Roman",
    legend  = :topleft,
    legendtitle = L"\ \ \ \ K_1^\circ = 10^3",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(b)",
    titlelocation = :left,
    titlefontsize = 26
)

plt2 = plot!(plt2, tfine/maximum(tfine), Xout[:,2,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")
plt2 = plot!(plt2, tfine/maximum(tfine), Xout[:,2,1], ls = :solid,      lc = :black, lw = 0.75,  label = L"\textrm{Ideal Solution}")
plt2 = plot!(plt2, tfine/maximum(tfine), Xout[:,2,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")

display(plt2)
savefig(plt2, "CS32.pdf")
```

```julia
plt3 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    #yscale = :log10,
    #ylim = (1e-1, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = "Conversion",
    fontfamily = "Times Roman",
    legend  = :topleft,
    legendtitle = L"\ \ \ \ K_1^\circ = 10^4",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(c)",
    titlelocation = :left,
    titlefontsize = 26
)

plt3 = plot!(plt3, tfine/maximum(tfine), Xout[:,3,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")
plt3 = plot!(plt3, tfine/maximum(tfine), Xout[:,3,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt3 = plot!(plt3, tfine/maximum(tfine), Xout[:,3,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")

display(plt3)
savefig(plt3, "CS33.pdf")
```

```julia
plt4 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    #yscale = :log10,
    #ylim = (1e-1, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = "Conversion",
    fontfamily = "Times Roman",
    legend  = :topleft,
    legendtitle = L"\ \ \ \ K_1^\circ = 10^5",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(d)",
    titlelocation = :left,
    titlefontsize = 26
)

plt4 = plot!(plt4, tfine/maximum(tfine), Xout[:,4,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")
plt4 = plot!(plt4, tfine/maximum(tfine), Xout[:,4,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt4 = plot!(plt4, tfine/maximum(tfine), Xout[:,4,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")

display(plt4)
savefig(plt4, "CS34.pdf")
```

```julia
plt5 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    yscale = :log10,
    ylim = (1e-4, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = L"\theta_{A}",
    fontfamily = "Times Roman",
    legend  = :topleft,
    legendtitle = L"\ \ \ \ K_1^\circ = 10",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(a)",
    titlelocation = :left,
    titlefontsize = 26
)

plt5 = plot!(plt5, tfine/maximum(tfine), cov[:,1,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")
plt5 = plot!(plt5, tfine/maximum(tfine), cov[:,1,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt5 = plot!(plt5, tfine/maximum(tfine), cov[:,1,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")

display(plt5)
savefig(plt5, "CS35.pdf")
```

```julia
plt6 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    yscale = :log10,
    ylim = (1e-4, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = L"\theta_{A}",
    fontfamily = "Times Roman",
    legend  = :bottomright,
    legendtitle = L"\ \ \ \ K_1^\circ = 10^3",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(b)",
    titlelocation = :left,
    titlefontsize = 26
)

plt6 = plot!(plt6, tfine/maximum(tfine), cov[:,2,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")
plt6 = plot!(plt6, tfine/maximum(tfine), cov[:,2,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt6 = plot!(plt6, tfine/maximum(tfine), cov[:,2,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")

display(plt6)
savefig(plt6, "CS36.pdf")
```

```julia
plt7 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    yscale = :log10,
    ylim = (1e-4, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = L"\theta_{A}",
    fontfamily = "Times Roman",
    legend  = :bottomright,
    legendtitle = L"\ \ \ \ K_1^\circ = 10^4",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(c)",
    titlelocation = :left,
    titlefontsize = 26
)

plt7 = plot!(plt7, tfine/maximum(tfine), cov[:,3,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")
plt7 = plot!(plt7, tfine/maximum(tfine), cov[:,3,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt7 = plot!(plt7, tfine/maximum(tfine), cov[:,3,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")

display(plt7)
savefig(plt7, "CS37.pdf")
```

```julia
plt8 = plot(
    xscale  = :log10,
    xlim    = (1e-8, 1.0),
    xminorticks = 4,
    yscale = :log10,
    ylim = (1e-4, 1e1),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 26,
    yguidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = L"\theta_{A}",
    fontfamily = "Times Roman",
    legend  = :bottomright,
    legendtitle = L"\ \ \ \ K_1^\circ = 10^5",
    legendfontsize = 16,
    legendfonthalign = :left,
    title = "(d)",
    titlelocation = :left,
    titlefontsize = 26
)

plt8 = plot!(plt8, tfine/maximum(tfine), cov[:,4,3], ls = :dash,       lc = :blue,  lw = 1.00, label = L"\textrm{Stabilize A}_*")
plt8 = plot!(plt8, tfine/maximum(tfine), cov[:,4,1], ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{Ideal Solution}")
plt8 = plot!(plt8, tfine/maximum(tfine), cov[:,4,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = L"\textrm{Destabilize A}_*")

display(plt8)
savefig(plt8, "CS38.pdf")
```

```julia
function coverage(p)
    MARs = p["MARs"]
    PAR  = p["PAR"]

    K10, NA, NB, NS = PAR

    NT     = NA + NB + NS

    XA   = NA/NT
    XB   = NB/NT
    XS   = NS/NT

    #For A in solvent
    A12A = 0.0

    #For As in solvent
    A12As = MARs

    #For B in solvent
    A12B = 0.0

    gammaA  = exp(XS^2*A12A)
    gammaAs = exp(XS^2*A12As)
    #print(log(gammaA), ' ', log(gammaAs), '\n')

    thetastar = 1/(1 + K10*gammaA/gammaAs*XA)
    thetaA    = K10*gammaA/gammaAs*XA*thetastar
    activityAs = gammaAs*thetaA

    return thetaA, activityAs
end
```

    coverage (generic function with 1 method)

```julia
#Calculate Coverages
MARs = [-3, -2, -1, 0, 1, 2, 3]
K10SET  = 10 .^range(-2, stop = 8, length = 500)
K20SET  = K./K10SET

cov    = zeros(size(K10SET, 1), length(MARs))
act    = zeros(size(K10SET, 1), length(MARs))

for i = 1:length(MARs)
    for j = 1:length(K10SET)
        p0   = Dict("MARs" => MARs[i], "PAR" => [K10SET[j], NA0, NB0, NS])
        ct, at = coverage(p0)
        cov[j,i] = ct
        act[j,i] = at
    end
end
```

```julia
plt9 = plot(
    xscale  = :log10,
    xlim    = (1e-2, 1e8),
    #xticks  = 10. .^range(-2, stop = 7, step = 1),
    xminorticks = 5,
    yscale = :log10,
    ylim = (1e-8, 1e0),
    yminorticks = 5,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 30,
    yguidefontsize = 30,
    tickfontsize = 24,
    xlabel  = L"K_1^\circ",
    ylabel  = L"\theta_A",
    fontfamily = "Times Roman",
    legend  = :bottomright,
    legendfontsize = 18,
    title = "(a)",
    titlelocation = :left,
    titlefontsize = 30
)

plt9 = plot!(plt9, K10SET, cov[:,1:3], ls = :dash,       lc = :auto,  lw = 1.00, labels = [L"\textrm{ln}(\gamma) \approx -3" L"\textrm{ln}(\gamma) \approx -2" L"\textrm{ln}(\gamma) \approx -1"])
plt9 = plot!(plt9, K10SET, cov[:,4],   ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{ln}(\gamma) = \ \ 0")
plt9 = plot!(plt9, K10SET, cov[:,5:7], ls = :dashdotdot, lc = :auto,  lw = 1.00, labels = [L"\textrm{ln}(\gamma) \approx \ \ 1" L"\textrm{ln}(\gamma) \approx \ \ 2" L"\textrm{ln}(\gamma) \approx \ \ 3" ])
display(plt9)
savefig(plt9, "CS39.pdf")
```

```julia
plt10 = plot(
    xscale  = :log10,
    xlim    = (1e-2, 1e8),
    #xticks  = 10. .^range(-2, stop = 7, step = 1),
    xminorticks = 5,
    yscale = :log10,
    ylim = (1e-8, 1e2),
    yminorticks = 5,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 30,
    yguidefontsize = 30,
    tickfontsize = 24,
    xlabel  = L"K_1^\circ",
    ylabel  = L"\textrm{Activity of} \ A_*",
    fontfamily = "Times Roman",
    legend  = :bottomright,
    legendfontsize = 18,
    title = "(b)",
    titlelocation = :left,
    titlefontsize = 30
)

plt10 = plot!(plt10, K10SET, reverse(act[:,5:7], dims = 2), ls = :dashdotdot, lc = :auto,  lw = 1.00, labels = [L"\textrm{ln}(\gamma) \approx \ \ 3" L"\textrm{ln}(\gamma) \approx \ \ 2" L"\textrm{ln}(\gamma) \approx \ \ 1" ])
plt10 = plot!(plt10, K10SET, act[:,4],                      ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{ln}(\gamma) = \ \ 0")
plt10 = plot!(plt10, K10SET, reverse(act[:,1:3], dims = 2), ls = :dash,       lc = :auto,  lw = 1.00, labels = [L"\textrm{ln}(\gamma) \approx -1" L"\textrm{ln}(\gamma) \approx -2" L"\textrm{ln}(\gamma) \approx -3"])
display(plt10)
savefig(plt10, "CS310.pdf")
```

```julia
function rate(p)
    MARs = p["MARs"]
    PAR  = p["PAR"]

    k20, K10, K20, NA, NB, NS = PAR

    NT     = NA + NB + NS

    XA   = NA/NT
    XB   = NB/NT
    XS   = NS/NT

    #For A in solvent
    A12A = 0.0

    #For As in solvent
    A12As = MARs

    #For B in solvent
    A12B = 0.0

    #For TS1 in solvent
    A12TS1 = 0.0

    #For TS2 in solvent
    A12TS2 = 0.0

    gammaA  = exp(XS^2*A12A)
    gammaAs = exp(XS^2*A12As)
    #print(log(gammaAs))
    gammaB  = exp(XS^2*A12B)
    gammaTS1 = exp(XS^2*A12TS1)
    gammaTS2 = exp(XS^2*A12TS2)

    thetastar = 1/(1 + K10*gammaA/gammaAs*XA)
    thetaA    = K10*gammaA/gammaAs*XA*thetastar
    activityAs = gammaAs*thetaA

    r2 = k20*gammaAs/gammaTS2*thetaA

    return thetaA, activityAs, r2
end
```

    rate (generic function with 1 method)

```julia
#Calculate Rates and Coverages
MARs = [-3, -2, -1, 0, 1, 2, 3]
K10SET  = 10 .^range(-2, stop = 8, length = 500)
K20SET  = K./K10SET

labels = Matrix{String}(undef, 1, length(K10SET))

# for m = 1:length(MARs)
#     labels[m] = string("log10gamma = ", MARs[m])
# end

cov    = zeros(size(K10SET, 1), length(MARs))
act    = zeros(size(K10SET, 1), length(MARs))
rat    = zeros(size(K10SET, 1), length(MARs))

for i = 1:length(MARs)
    for j = 1:length(K10SET)
        p0   = Dict("MARs" => MARs[i], "PAR" => [k20, K10SET[j], K20SET[j], NA0, NB0, NS])
        ct, at, rt = rate(p0)
        cov[j,i] = ct
        act[j,i] = at
        rat[j,i] = rt
    end
end
```

```julia
plt11 = plot(
    xscale  = :log10,
    xlim    = (1e-2, 1e8),
    #xticks  = 10. .^range(-2, stop = 7, step = 1),
    xminorticks = 5,
    yscale = :log10,
    ylim = (1e-9, 1e0),
    yminorticks = 5,
    tick_direction = :out,
    grid = :off,
    xguidefontsize = 30,
    yguidefontsize = 30,
    tickfontsize = 24,
    xlabel  = L"K_1^\circ",
    ylabel  = "rate",
    fontfamily = "Times Roman",
    legend  = :bottomright,
    legendfontsize = 18,
    title = "(c)",
    titlelocation = :left,
    titlefontsize = 30
)

plt11 = plot!(plt11, K10SET, reverse(rat[:,5:7], dims = 2), ls = :dashdotdot, lc = :auto,  lw = 1.00, labels = [L"\textrm{ln}(\gamma) \approx \ \ 3" L"\textrm{ln}(\gamma) \approx \ \ 2" L"\textrm{ln}(\gamma) \approx \ \ 1" ])
plt11 = plot!(plt11, K10SET, rat[:,4],                      ls = :solid,      lc = :black, lw = 0.75, label = L"\textrm{ln}(\gamma) = \ \ 0")
plt11 = plot!(plt11, K10SET, reverse(rat[:,1:3], dims = 2), ls = :dash,       lc = :auto,  lw = 1.00, labels = [L"\textrm{ln}(\gamma) \approx -1" L"\textrm{ln}(\gamma) \approx -2" L"\textrm{ln}(\gamma) \approx -3"])

display(plt11)
savefig(plt11, "CS311.pdf")
```

```julia

```


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Important Distributions

Analytic cumulative distribution functions (CDFs) and probability density functions (PDFs) are frequently used as models for practical random processes. They typically describe the statistical properties of a random process by a few parameters. These parameters may be fitted to an actual random process and used in algorithms for statistical signal processing. Alternatively the parameters may be used to generate sample functions with a given distribution. For the following, a real-valued wide-sense stationary (WSS) random process $x[k]$ is assumed.

First a function is defined which plots the PDF and CDF of a given model.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

def plot_pdf_cdf(x, distr):
    plt.figure(figsize = (10, 5))

    plt.subplot(121)
    plt.plot(x, distr.pdf(x))
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$p_x(\theta)$')
    plt.title('PDF')
    plt.grid()

    plt.subplot(122)
    plt.plot(x, distr.cdf(x))
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$P_x(\theta)$')
    plt.title('CDF')
    plt.grid()
```

### Uniform Distribution

#### Definition

The PDF of the [uniform distribution](https://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29) is given as

\begin{equation}
p_x(\theta) = \begin{cases} \frac{1}{b - a} & \text{for } a \leq \theta \leq b \\ 0 & \text{otherwise} \end{cases}
\end{equation}

where $a$ and $b$ with $a \leq b$ denote the lower and upper bound for the amplitude of the random signal $x[k]$. The uniform distribution implies that all amplitudes between these bounds occur with the same probability $\frac{1}{b - a}$. Its CDF can be derived from the PDF by integration over $\theta$

\begin{equation}
P_x(\theta) = \begin{cases} 0 & \text{for } \theta < a \\
\frac{\theta - a}{b - a} & \text{for } a \leq \theta < b \\
1 & \text{for } \theta \geq b \end{cases}
\end{equation}

The PDF/CDF of a uniformly distributed random signal with $a=0$ and $b=1$ is plotted for illustration

```python
plot_pdf_cdf(np.linspace(-.5, 1.5, num=1000), stats.uniform(loc=0, scale=1))
```

The [linear mean](ensemble_averages.ipynb#Linear-mean) is derived from the PDF as

\begin{equation}
\mu_x = \frac{a+b}{2}
\end{equation}

and the variance as

\begin{equation}
\sigma_x^2 = \frac{(b-a)^2}{12}
\end{equation}

**Exercise**

* Derive the linear mean and the variance from the PDF.
* What values does the linear mean and variance for the above plotted example PDF/CDF take?

Solution: The linear mean and the variance can be derived by introducing the PDF into their definitions in terms of ensemble averages. This yields

$$\mu_x[k] = E\{ x[k] \} = \int\limits_{a}^{b} \theta \, \frac{1}{b -a} \; \mathrm{d}\theta = \frac{a + b}{2}$$

and

$$\sigma_x^2 = E\{ x[k] \} - \mu_x^2 = \int\limits_{a}^{b} \theta^2 \, \frac{1}{b -a} \; \mathrm{d}\theta - \frac{(a+b)^2}{4} = \frac{(b-a)^2}{12}$$

It can be concluded from the example PDF that $a=0$ and $b=1$. Introducing these values in above equations derives $\mu_x = \frac{1}{2}$ and $\sigma_x^2 = \frac{1}{12}$.

#### Example

Most software frameworks for numerical mathematics provide functionality to draw samples from a random process for a given PDF. So does [`Numpy`](http://docs.scipy.org/doc/numpy/reference/routines.random.html) or [`scipy.stats`](http://docs.scipy.org/doc/scipy/reference/stats.html#continuous-distributions).

In the following example a large but finite number of samples is drawn from a random process which generates uniformly distributed samples. First a function is defined that estimates the PDF by computing the histogram over the given random samples. The CDF is computed from the estimated PDF by cumulative summation. Both the estimated PDF and CDF are plotted for illustration.

```python
def estimate_plot_pdf_cdf(x, nbins=100):

    plt.figure(figsize = (10, 6))
    plt.hist(x, nbins, density=True)
    plt.title('Estimated PDF')
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$\hat{p}_x(\theta)$')
    plt.grid()

    plt.figure(figsize = (10, 6))
    plt.hist(x, nbins, cumulative=True, density=True)
    plt.title('Estimated CDF')
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$\hat{P}_x(\theta)$')
    plt.grid()

    print('Estimated linear mean: %f' %np.mean(x))
    print('Estimated variance: %f' %np.var(x))
```

For the standard uniform distribution ($a=0$, $b=1$) we get the following results

```python
np.random.seed(1)
estimate_plot_pdf_cdf(stats.uniform.rvs(size=100000, loc=0, scale=1), nbins=100)
```

    Estimated linear mean: 0.499219
    Estimated variance: 0.083730

**Exercise**

* Why is the estimate of the CDF smoother that the estimate of the PDF?
* Change the parameters `loc` and `scale` in above example. How are these parameters related to the upper $b$ and lower bound $a$?
* What changes if you change the length `size` of the random signal?

Solution: The estimate of the CDF is smoother as compared to the estimate of the PDF since the integration performed to compute the CDF from the PDF can be interpreted as low-pass filter. This attenuates the 'fine-structure' in the PDF. The lower bound $a$ is equal to `loc` the upper bound $b$ to `loc+scale`. Increasing the length of the random signal lowers the statistical uncertainty when estimating the PDF.

### Normal Distribution

#### Definition

The PDF of the [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution)/Gaussian distribution  is given as

\begin{equation}
p_x(\theta) = \frac{1}{\sqrt{2 \pi} \sigma_x} \mathrm{e}^{- \frac{(\theta - \mu_x)^2}{2 \sigma_x^2}}
\end{equation}

where $\mu_x$ and $\sigma_x^2$ denote its linear mean and variance, respectively. The central limit theorem states that averages of random variables independently drawn from independent distributions become normally distributed when the number of random variables is sufficiently large. As a result, random signals that are expected to be a combination of many independent processes often have distributions that are approximately normal. Normal distributions are often used to model random processes whose internal structure is not exactly known but can be assumed to be a combination of various random processes. Note that a normal distributed random signals shows a small but finite probability that samples with very high amplitudes occur. This is in contrast to the facts that technical systems can only generate and process signals of finite amplitude. However, the normal distribution is often assumed to provide a reasonable approximation in such cases.

The maximum value of the PDF is located at $\theta = \mu_x$ and is given as

\begin{equation}
p_x(\mu_x) = \frac{1}{\sqrt{2 \pi} \sigma_x}
\end{equation}

The CDF of the normal distribution is derived by integrating the PDF over $\theta$

\begin{align}
P_x(\theta) &= \frac{1}{\sqrt{2 \pi} \sigma_x} \int\limits_{—\infty}^{\theta} \mathrm{e}^{- \frac{(\zeta - \mu_x)^2}{2 \sigma_x^2}} \mathrm{d}\zeta \\
&= \frac{1}{2} \left( 1 + \text{erf}\left( \frac{\theta-\mu_x}{\sqrt{2} \sigma_x} \right)\right)
\end{align}

where $\text{erf}(\cdot)$ denotes the [error function](https://en.wikipedia.org/wiki/Error_function).

The PDF and CDF are illustrated for the standard zero-mean normal distribution ($\mu_x=0$, $\sigma_x^2=1$)

```python
plot_pdf_cdf(np.linspace(-5, 5, num=100), stats.norm(loc=0, scale=1))
```

#### Example

For the standard zero-mean uniform distribution we get the following numerical results when drawing a large number of random samples

```python
estimate_plot_pdf_cdf(stats.norm.rvs(size=100000, loc=0, scale=1), nbins=100)
```

    Estimated linear mean: 0.004897
    Estimated variance: 0.993752

**Exercise**

* Change the linear mean $\mu_x$ and the variance $\sigma_x^2$ by changing the `loc` and `scale` parameter. How is the mean and variance related to these parameters?
* Assume you want to model measurement noise with zero-mean and a given power $P$. How do you have to chose the parameters of the normal distribution?

Solution: The linear mean $\mu_x$ is equal to `loc` and the standard deviation $\sigma_x$ is equal to `scale`. The quadratic mean $E\{x^2[k]\}$ quantifies the average power of a random process. For a zero-mean process the quadratic mean $E\{x^2[k]\}$ is equal to the variance $\sigma_x^2$.

### Laplace Distribution

#### Definition

The PDF of the [Laplace distribution](https://en.wikipedia.org/wiki/Laplace_distribution) is given as

\begin{equation}
p_x(\theta) = \frac{1}{\sqrt{2} \sigma_x} \mathrm{e}^{- \sqrt{2} \frac{|\theta - \mu_x|}{\sigma_x}}
\end{equation}

where $\mu_x$ and $\sigma_x^2$ denote the linear mean and variance, respectively. Laplace distributions are often used to model the amplitude distribution of a speech or music signal. The maximum value of the PDF is located at $\theta = \mu_x$ and is given as

\begin{equation}
p_x(\mu_x) = \frac{1}{\sqrt{2} \sigma_x}
\end{equation}

The CDF can be derived by integrating the PDF over $\theta$

\begin{equation}
P_x(\theta) = \begin{cases}
\frac{1}{2} \mathrm{e}^{\sqrt{2} \frac{\theta - \mu_x}{\sigma_x}} & \text{for } \theta \leq \mu_x \\
1 - \frac{1}{2} \mathrm{e}^{- \sqrt{2} \frac{\theta - \mu_x}{\sigma_x}} & \text{for } \theta > \mu_x
\end{cases}
\end{equation}

For the zero-mean, unit-variance Laplace distribution the PDF/CDF is illustrated. Note the scale parameter in [`scipy.stats.laplace`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.laplace.html#scipy.stats.laplace) is related to the variance by $\lambda = \frac{\sigma_x}{\sqrt{2}}$

```python
plot_pdf_cdf(np.linspace(-5, 5, num=100), stats.laplace(loc=0, scale=1/np.sqrt(2)))
```

#### Example

For the standard zero-mean Laplace distribution we get the following estimates when drawing a large number of random samples

```python
estimate_plot_pdf_cdf(stats.laplace(scale=1/np.sqrt(2)).rvs(size=10000), nbins=100)
```

    Estimated linear mean: -0.006542
    Estimated variance: 1.001713

### Example - Amplitude Distribution of a Speech Signal

Lets estimate the PDF and CDF of a recorded speech signal in order to see if we can model it by one of the PDFs introduced above.

```python
from scipy.io import wavfile

fs, x = wavfile.read('../data/speech_8k.wav')
x = np.asarray(x, dtype=float)/2**15
estimate_plot_pdf_cdf(x, nbins=100)
```

    Estimated linear mean: -0.000067
    Estimated variance: 0.018548

**Exercise**

* Which analytic PDF/CDF can be used to model a speech signal?
* How would you chose the parameters of the distribution to fit the estimated PDF?

Solution: The Laplace distribution fits very well to the estimated PDF. It is frequently used to model the amplitude distribution of a speech signal. For the given speech signal its mean and variance can be approximated as $\mu_x = 0$ and $\sigma_x^2 = 0.02$.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# LDDS

## Tutorial 4: Lagrangian descriptors for a user-defined 1 DoF dynamical system

In this notebook we will show how to calculate Lagrangian descriptors for a 1 DoF dynamical system defined by the user.

We begin by importing the LDDS module, and from it, all the functions necessary to calculate and plot LDs.

```python
import os, sys
sys.path.insert(1, os.pardir)
import ldds
from ldds.base import compute_lagrangian_descriptor
from ldds.tools import draw_all_lds
import numpy as np
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')
```

#### The Morse oscillator

The Hamiltonian function that describes the Morse oscillator is:

\begin{equation*}
 H(x,p_x) = \dfrac{1}{2m}p_x^2 + D\left(1 - e^{-\alpha x}\right)^2
\end{equation*}

where $m$ is the mass, $D$ represents the well depth and $\alpha$ controls the well width. The vector field that defines the dynamical system is given by:

\begin{align}
   \dot{x} & = \dfrac{\partial H}{\partial p_x} = f_1(x,p_x) = \dfrac{p_x}{m} \\[.2cm]
   \dot{p}_x & = -\dfrac{\partial H}{\partial x} = f_2(x,p_x) = -2 D\,\alpha \left(e^{-\alpha x} - e^{-2\alpha x}\right)
\end{align}

Define the `Morse1D` function containing the two-dimensional dynamical system

```python
def Morse1D(t, u, PARAMETERS = [1, 2, 1]):
    """
    Returns a user-defined 1-DoF vector field at time t, for an array of points in phase-space.
    Number of model parameters: 1 . PARAMETERS = [m, D, alpha]
    Functional form: v = ((1/m)*y, -2*D*alpha*(exp(-alpha*x) - exp(-2*alpha*x))), with u = (x, y)

    Parameters
    ----------
    t : float
        fixed time-point of vector field, for all points in phase-space.

    u : array_like, shape(n,)
        points in phase-space to determine vector field at common time t.

    PARAMETERS : list of floats
        vector field parameters

    Returns
    -------
    v : array_like, shape(n,)
        vector field corresponding to points u, in phase-space at time t
    """
    x, y = u.T
    ##############################################
    #
    # CHANGE STUFF BELOW ONLY
    #
    ##############################################
    # Hamiltonian Model Parameter
    m, D, alpha = PARAMETERS
    v = np.column_stack([ (1/m) * y, -2 * D * alpha * (np.exp(-alpha * x) - np.exp(-2 * alpha * x))])
    ##############################################
    #
    # CHANGE STUFF ABOVE ONLY
    #
    ##############################################
    return v
```

### Define 2D grid to compute LDs

```python
# Mesh parameters to calculate LDs

# 2D Domain
x_min,x_max = [-1, 7]
y_min,y_max = [-2.5, 2.5]

# Grid size
Nx, Ny = [500, 500]

grid_parameters = [(x_min, x_max, Nx), (y_min, y_max, Ny)]
```

### Setup vector field and model parameters

```python
# Morse oscillator parameters
m, D, alpha = [1, 2, 1]

vector_field = lambda t,u: Morse1D(t, u, PARAMETERS = [m, D, alpha])
```

### Compute and draw LDs at time $t = t_0$

```python
# Integration parameters (Calculate LDs at time t = t0 by integrating in the interval [t0-tau,t0+tau])
t0 = 0
tau = 15

# LDs definition (we will use the p-value seminorm)
p_value = 0.5

LD_forward = compute_lagrangian_descriptor(grid_parameters, vector_field, tau, p_value)
LD_backward = compute_lagrangian_descriptor(grid_parameters, vector_field, -tau, p_value)

figs = draw_all_lds(LD_forward, LD_backward, grid_parameters, tau, p_value)
```

We change next the model parameters:

```python
# Morse oscillator parameters
m, D, alpha = [1, 1, 1]

vector_field = lambda t,u: Morse1D(t, u, PARAMETERS = [m, D, alpha])
```

### Compute and draw LDs at time $t = t_0$

```python
# Integration parameters (Calculate LDs at time t = t0 by integrating in the interval [t0-tau,t0+tau])
t0 = 0
tau = 15

# LDs definition (we will use the arclength)
p_value = 2

LD_forward = compute_lagrangian_descriptor(grid_parameters, vector_field, tau, p_value)
LD_backward = compute_lagrangian_descriptor(grid_parameters, vector_field, -tau, p_value)

figs = draw_all_lds(LD_forward, LD_backward, grid_parameters, tau, p_value)
```

```python

```


## autodiff32 Documentation

## Introduction

Differentiation, or the process of finding a derivative, is an extremely important mathematical operation with a wide range of applications. The discovery of extrema or zeros of functions is essential in any optimization problem, and the solving of differential equations is fundamental to modern science and engineering. Differentiation is essential in nearly all quantitative disciplines: physicists may take the derivative of the displacement of a moving object with respect to time in order to find the velocity of that object, and data scientists may use derivatives when optimizing weights in a neural network.

Naturally, we would like to compute the derivative as accurately and efficiently as possible. Two classical methods of calculating the derivative have clear shortcomings. Symbolic differentiation (finding the derivative of a given formula with respect to a specified variable, producing a new formula as its output) will be accurate, but can be quite expensive computationally. The finite difference method ($\frac{\partial f}{\partial x} = \frac{f(x+\epsilon)-f(x)}{\epsilon}$ for some small $\epsilon$) does not have this issue, but will be less precise as different values of epsilon will give different results. This brings us to automatic differentiation, a less costly and more precise approach.

__Extension:__ For this project, we have implemented forward-mode automatic differentiation as well as reverse-mode automatic differentiation. It is important to have both methods accessible to have accuracy as well as optimal efficiency.

Automatic differentiation can be used to compute derivatives to machine precision of functions $f:\mathbb{R}^{m} \to \mathbb{R}^{n}$

The forward mode is more efficient when $n\gg m$.
   - This corresponds to the case where the number of functions to evaluate is much greater than the number of inputs.
   - Actually computes the Jacobian-vector product $Jp$.

The reverse mode is more efficient when $n\ll m$.
   - This corresponds to the case where the number of inputs is much greater than the number of functions.
   - Actually computes the Jacobian-transpose-vector product $J^{T}p$.

## Background

Automatic differentiation breaks down the main function into elementary functions, evaluated upon one another. It then uses the chain rule to update the derivative at each step and ends in the derivative of the entire function.

To better understand this process, let's look at an example. Consider the example function

\begin{equation}
f(x) = x + 4sin(\frac{x}{4})
\end{equation}

We would like to compute the derivative of this function at a particular value of x. Let's say that in this case, we are interested in evaluating the derivative at $x=\pi$. In other words, we want to find $f'(\pi)$ where $f'(x) = \frac{\partial f}{\partial x}$

We know how to solve this _symbolically_ using methods that we learned in calculus, but remember, we want to compute this answer as accurately and efficiently as possible, which is why we want to solve it using automatic differentiation.

### The Chain Rule

To solve this using automatic differentiation, we need to find the decomposition of the differentials provied by the chain rule. Remember, the chain rule is a formula for computing the derivative of the composition of two or more functions. So if we have a function $h\left(u\left(t\right)\right)$ and we want the derivative of $h$ with respect to $t$, then we know by the chain rule that the derivative is $\dfrac{\partial h}{\partial t} = \dfrac{\partial h}{\partial u}\dfrac{\partial u}{\partial t}.$ The chain rule can also be expanded to deal with multiple arguments and vector inputs (in which case we would be calculating the _gradient)_.

Our function $f(x)$ is composed of elemental functions for which we know the derivatives. We will separate out each of these elemental functions, evaluating the derivative at each step using the chain rule.

### Forward-mode differentiation

Using forward-mode differentiation, the evaluation trace for this problem looks like:

| Trace    | Elementary Operation | Derivative | $f'(a)$ |
| :------: | :----------------------: | :------------------------------:  | :------------------------------: |
| $x_{3}$  | $\pi$ | $1$ | $\pi$ | $1$ |
| $x_{0}$  | $\frac{x_{3}}{4}$ | $\frac{\dot{x}_{3}}{4}$ | $\frac{1}{4}$ |
| $x_{1}$  | $\sin\left(x_{0}\right)$ | $\cos\left(x_{0}\right)\dot{x}_{0}$ | $\frac{\sqrt{2}}{8}$|
| $x_{2}$  | $4x_{1}$ | $4\dot{1}_{3}$ | $\frac{\sqrt{2}}{2}$ |
| $x_{4}$  | $x_{2} + x_{3}$| $\dot{x}_{2} + \dot{x}_{3}$ | $1 + \frac{\sqrt{2}}{2}$ |

This evaluation trace provides some intuition for how forward-mode automatic differentiation is used to calculate the derivative of a function evaluated at a certain value ($f'(\pi) = 1 + \frac{\sqrt{2}}{2}$).

</img>
*Figure 1: Forward-mode computational graph for example above*

### Reverse-mode automatic differentiation

Using reverse-mode differentiation, sometimes called backpropagation, the trace on the right is evaluated top to bottom intsead.

| Trace    | Elementary Operation | Derivative                             | $f'(a)$ |
| :------: | :----------------------: | :------------------------------: | :------------------------------: |
| $x_{3}$  | $\pi$ | $\frac{\partial{x_0}}{\partial{x_3}}\bar{x_0} + \bar{x_4} = \cos(\frac{\pi}{4}) + 1$ | $1 + \frac{\sqrt{2}}{2}$ |
| $x_{0}$  | $\frac{x_{3}}{4}$ | $\frac{\partial{x_1}}{\partial{x_0}}\bar{x_1} = 4\cos(\frac{\pi}{4})$ | $2\sqrt{2}$ |
| $x_{1}$  | $\sin\left(x_{0}\right)$ | $\frac{\partial{x_2}}{\partial{x_1}}\bar{x_2}$| $4$|
| $x_{2}$  | $4x_{1}$ | $\frac{\partial{x_4}}{\partial{x_2}}\bar{x_4}$ | $1$ |
| $x_{4}$  | $x_{2} + x_{3}$| $1$ | $1$ | $1$

</img>
*Figure 2: Reverse-mode computational graph for example above*

You may notice that when we computed the derivative above, we "seeded" the derivative with a value of 1. This seed vector doesn't have to be 1, but the utility of using a unit vector becomes apparent when we consider a problem involving directional derivatives.

The definition of the directional derivative (where $p$ is the seed vector)
$$D_{p}x_{3} = \sum_{j=1}^{2}{\dfrac{\partial x_{3}}{\partial x_{j}}p_{j}}$$

can be expanded to

\begin{align}
  D_{p}x_{3} &= \dfrac{\partial x_{3}}{\partial x_{1}}p_{1} + \dfrac{\partial x_{3}}{\partial x_{2}}p_{2} \\
             &= x_{2}p_{1} + x_{1}p_{2}
\end{align}

If we choose $p$ to be a the unit vector, we can see how this is beneficial:

$p = \left(1,0\right)$ gives $\dfrac{\partial f}{\partial x}$

$p = \left(0,1\right)$ gives $\dfrac{\partial f}{\partial y}$

So to summarize, the forward mode of automatic differentiation is really computing the _product of the gradient of our function with the seed vector:_

$$D_{p}x_{3} = \nabla x_{3}\cdot p.$$

If our function is a vector, then the forward mode actually computes $Jp$ where $J = \dfrac{\partial f_{i}}{\partial x_{j}}, \quad i = 1,\ldots, n, \quad j = 1,\ldots,m$ is the Jacobian matrix. Often we will really only want the "action" of the Jacobian on a vector, so we will just want to compute the matrix-vector product $Jp$ for some vector $p$. Using the same logic, the reverse mode actually computes the Jacobian-transpose-vector product $J^{T}p$.

Automatic differentiation can be used to compute derivatives to machine precision of functions $f:\mathbb{R}^{m} \to \mathbb{R}^{n}$

## How to Use autodiff32
### Installation

**1) Create a virtual environment (optional)**

From the terminal, create a virtual environment:

_(The command below will create the virtual environment in your present working directory, so consider moving to a project folder or a known location before creating the environment)_

```virtualenv env```

activate the virtual environment:

```source env/bin/activate```

if you plan to launch a jupyter notebook using this virtual environment, run the following to install and set up jupyter in your virtual environment:

```python -m pip install jupyter```

```python -m ipykernel install --user --name=env```

**3) Install the autodiff32 package**

In the terminal, type:
```pip install autodiff32```
Package dependencies will be taken care of automatically!

_(Alternatively, it is also possible to install the autodiff32 package by downloading this GitHub repository. If you choose that method, use the requirements.txt file to ensure you have installed all necessary dependencies.)_

## Tutorial

It is easy to use the autodiff32 package in a Jupyter notebook, as we will demonstrate here:

_(Alternatively, you can start a Python interpreter by typing ```Python``` into the terminal, or work from your favorite Python IDE.)_

_(Remember, if you are using a virtual environment, follow steps 1 through 3 above and then type ```jupyter notebook``` into your terminal to launch a notebook. Inside the notebook, switch the kernel to that of your virtual environment.)_

```python
pip install autodiff32
```

    Collecting autodiff32
      Downloading https://files.pythonhosted.org/packages/b6/8c/88fccbf0cc72968be09e75c7bd9f6811d6c675bba90d825e2fc5dfdd145e/autodiff32-0.1.2-py3-none-any.whl
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from autodiff32) (1.17.4)
    Installing collected packages: autodiff32
    Successfully installed autodiff32-0.1.2

```python
import autodiff32 as ad
import math # only necessary for this particular example

"""
Initialize an AutoDiff object
with the number you would like to pass into your function
"""
X = ad.AutoDiff(math.pi)

"""
Define your function of interest
"""
func = X + 4*ad.sin(X/4)
"""
Look at the derivative of your function
evaluated at the number you gave above
"""
print("derivative:",func.der)

"""
Look at the value of your function
evaluated at the number you gave:
"""
print("value:", func.val)
```

    derivative: 1.7071067811865475
    value: 5.970019778335983

Notice that this is the same equation used in our example above: $f(x) = x + 4(sin(\frac{x}{4}))$. Just for fun, let's see if the derivative that we calculated in the evolution trace is the same as the result using autodiff32:

```python
print("autodiff32 derivative:", func.der)
print("evolution trace derivative:", 1+math.sqrt(2)/2)
```

    autodiff32 derivative: 1.7071067811865475
    evolution trace derivative: 1.7071067811865475

We can see that the derivative calculated using autodiff32 is the same as the derivative calulated by walking manually through the evolution trace!

Now what if your function if interest has a **vector input** (X has more than one value)? In that case, use the following workflow:

```python
import numpy as np
X = ad.AutoDiff(np.array([1,2,3]))
func = X**2
print("value:", func.val)
print("derivative:",func.der)
```

    value: [1 4 9]
    derivative: [2. 4. 6.]

Notice that there are three values and three derivatives. This is because your function and its derivative have been evaluated at the three values you provided.

Now what if your function if interest is a **multivariate function** (has more than just an X variable)? In that case, use the following workflow:

```python
X,Y = ad.Multi_AutoDiff_Creator(X = 2, Y = 4).Vars
func = X**2 + 3*Y
print("value:", func.val)
print("derivative:",func.der)
```

    value: 16
    derivative: [4. 3.]

Notice that the derivative has two values. This is the derivative of your function with respect to X and Y, evaluated at the values of X and Y you provided.

Now what if you actually have **multiple functions of interest**? In that case, use the following workflow:

```python
X,Y = ad.Multi_AutoDiff_Creator(X = 2, Y = 4).Vars
func = np.array([X+Y, 2*X*Y]) # two functions!

# get value and derivatives of function separately
print("value of first function:", func[0].val)
print("derivative of first function:",func[0].der)

print("\nvalue of second function:", func[1].val)
print("derivative of second function:",func[1].der)

# return the Jacobian matrix
J = ad.Jacobian(func)
print("\nJacobian matrix:\n",J.value())
```

    value of first function: 6
    derivative of first function: [1. 1.]

    value of second function: 16
    derivative of second function: [8. 4.]

    Jacobian matrix:
     [[1. 1.]
     [8. 4.]]

Notice that you have an additional option here! You can return the values and derivatives of the functions as you would normally (except that you indicate the index of the function when asking for the value or derivative), _or_ you can return the **Jacobian matrix** which contains the derivatives with respect to X and Y for each of the functions.

Now what if your function if interest is a **multivariate function AND it has vector inputs**? In that case, use the following workflow:

_Please note that the workflow here is significantly different from the rest of the package!_

_This is due to the complexities of handling the derivatives of vector inputs for multivariate functions. We wanted to give you, the user, as much functionality as possible, even if that meant sacrificing a bit in user-friendliness._

```python
# For a single multivariate function evaluated at vector value inputs

# define your variables and their vector values
X = [1,2,3]
Y = [2,3,3]
Z = [3,5,3]
W = [3,5,3]

# put them together in a list, in the order they will be used in your function!
VarValues = [X, Y, Z, W]

# define your function
# Vars[0] represents X, Vars[1] represents Y, etc.
func = lambda Vars:3*Vars[0] + 4*Vars[1] + 4*Vars[2]**2 + 3*Vars[3]

# find the values and derivatives
Values, Derivatives = ad.MultiVarVector_AutoDiff_Evaluate(VarValues,func)

print("values:\n", Values)
print("\nderivatives:\n", Derivatives)

```

    values:
     [ 56 133  66]

    derivatives:
     [[ 3.  4. 24.  3.]
     [ 3.  4. 40.  3.]
     [ 3.  4. 24.  3.]]

Now what if you have **multiple multivariate functions of interest with vector inputs**? In that case, use the following workflow:

_Please note that the workflow here is significantly different from the rest of the package!_

_This is due to the complexities of handling the derivatives of vector inputs for multivariate functions. We wanted to give you, the user, as much functionality as possible, even if that meant sacrificing a bit in user-friendliness._

```python
# For a single multivariate function evaluated at vector value inputs

# define your variables and their vector values
X = [1,2,3]
Y = [2,3,3]
Z = [3,5,3]
W = [3,5,3]

# put them together in a list, in the order they will be used in your function!
VarValues = [X, Y, Z, W]

# define your functions
# Vars[0] represents X, Vars[1] represents Y, etc.
func = lambda Vars:np.array([3*Vars[0] + 4*Vars[1] + 4*Vars[2]**2 + 3*Vars[3],    # first function
                                     5*Vars[0] + 6*Vars[1] + 7*Vars[2]**2 + 1*Vars[3]])   # second function
# find the values and derivatives
Values, Derivatives = ad.MultiVarVector_AutoDiff_Evaluate(VarValues,func)

print("values:\n", Values)
print("\nderivatives:\n", Derivatives)
```

    values:
     [[ 56  83]
     [133 208]
     [ 66  99]]

    derivatives:
     [[[ 3.  4. 24.  3.]
      [ 5.  6. 42.  1.]]

     [[ 3.  4. 40.  3.]
      [ 5.  6. 70.  1.]]

     [[ 3.  4. 24.  3.]
      [ 5.  6. 42.  1.]]]

**Extension Usage Demo**

```python

# For univariate /Multivariate Scalar Function
Graph = ad.ComputationalGraph()
X = ad.Node(value = 3, Graph = Graph)
Y = ad.Node(value = 4, Graph = Graph)
Z = ad.Node(value = 1, Graph = Graph)

G = 2*X + 3*Y*Z + 2*Z
Graph.ComputeValue()
Graph.ComputeGradient(-1)
print("Derivative for X is: ",X.deri)
print("Derivative for Y is: ",Y.deri)
print("Derivative for Z is: ",Z.deri)
print("Value of Function is: ",G.value)
```

    Derivative for X is:  2.0
    Derivative for Y is:  3.0
    Derivative for Z is:  14.0
    Value of Function is:  20

```python
#For univariate or Multivariate Vector Functions with single value
import numpy as np
Graph = ad.ComputationalGraph()
X = ad.Node(value = 3, Graph = Graph)
Y = ad.Node(value = 4, Graph = Graph)
Z = ad.Node(value = 1, Graph = Graph)

G = np.array([-2*ad.sinr(X),  #please use sinr for sin operation on the node
               2*Y + Z*Y,
               3*X+3*Y*X+2*Z])

Func = ad.ReverseVecFunc(G,X =X,Y= Y,Z= Z)
Value ,Derivative = Func.value(Graph)
print("The value for the vector function is: ")
print(Value)
print("The derivative for the vector function is: ")
print(Derivative)
```

    The value for the vector function is:
    [-0.28224002 12.         47.        ]
    The derivative for the vector function is:
    [[ 1.97998499  0.          0.        ]
     [ 0.          3.          4.        ]
     [15.          9.          2.        ]]

```python
#SERIES OF VALUES For vector functions
D = 3 # number of variables
x = [1,2,3]
y = [6,7,4]
z = [3,8,1]
Values = np.array([x,y,z])
G = np.array([-2*X,  #please use sinr for sin operation on the node
               2*Y + Z*Y,
               3*X+3*Y*X+2*Z])
Func = ad.ReverseVecFunc(G,X =X,Y= Y,Z= Z)
Vals,Deris=Func.Seriesvalue(Values,D,Graph)
print("The value for the vector function is: ")
print(Vals)
print("The derivative for the vector function is: ")
print(np.array(Deris))
```

    The value for the vector function is:
    [[-2 30 27]
     [-4 70 64]
     [-6 12 47]]
    The derivative for the vector function is:
    [[[-2.  0.  0.]
      [ 0.  5.  6.]
      [21.  3.  2.]]

     [[-2.  0.  0.]
      [ 0. 10.  7.]
      [24.  6.  2.]]

     [[-2.  0.  0.]
      [ 0.  3.  4.]
      [15.  9.  2.]]]

```python
import numpy as np
Graph = ad.ComputationalGraph()
X = ad.Node(value = 3, Graph = Graph)
F = 3*X**2
Xvals= np.array([[3,2,4]]) #Please input Xvals as a 2 dimensional array
Vals, deri = Graph.SeriesValues(Xvals,1,Graph)
print(Vals)
print(deri)
```

    [27, 12, 48]
    [[18.0], [12.0], [24.0]]

## Software Organization
### Directory Structure
Our structure is as follows:

    /cs207-FinalProject
        README.md
        LICENSE
        .gitignore
        .travis.yml
        setup.py
        requirements.txt
        docs/
            milestone1.ipynb
            milestone2.ipynb
            documentation.ipynb
        autodif32/
            __init__.py
            AutoDiffObj.py
            Elementary.py
            ElementaryReverse.py
            Graph.py
            JacobianVectorFunc.py
            MultivariateVarCreator.py
            MultivariateVectorAutoDiffEvaluate.py
            MultivariateVectorVarCreator.py
            ReverseJacobVectorFunc.py
            ReverseMode.py
        test/
            __init__.py
            autodiffobj_test.py
            elementary_test.py
            Reverse_test.py
            JacobianVectorFunc_test.py

### Modules

The ```AutoDiffObj``` module creates an AutoDiff object based on the scalar value you would like to evaluate a function and its derivative at. It overloads the basic operations including multiply, add, negative, subtract, powers, division, and equality. It also includes a Jacobian method, which returns the Jacobian of the function. If the function is univariate, then the Jacobian is just the derivative. If the function is multivariate (not yet implemented), then the Jacobian will be an array.

The ```Elementary``` module implements some of the elementary functions for use in the forward mode of autodifferentiation, including exp, log, sqrt, sin, cos, tan, asin, acos, atan, sinh, cosh and tanh.

The ```ElementaryReverse``` module implements some of the elementary functions for use in the reverse mode of autodifferentiation. These functions are made distinct from those in the Elementary module by an r at the end of the function name (expr, logr, sqrtr, sinr, cosr, tanr, asinr, acosr, atanr, sinhr, coshr and tanhr).

The ```Graph``` module implements the ComputationalGraph class which allows each node to be recorded in sequence in a graph for later use in value and derivative computation. This module also implements the ComputeValue and ComputeGradient functions, which compute the value and gradient using reverse mode and leveraging the computational graph.

The ```JacobianVectorFunc``` module implements the Jacobian method for vector inputs.

The ```MultivariateVarCreator``` module takes in the values of each variable in a user defined multivariate function, and returns len(kwargs) number of AutoDiff class variables with derivative (seed) as an np.array. The MultivariateVarCreator class acts as a helper function for user to create multiple AutoDiff Objects conveniently (instead of manually create many AutoDiff objects) to use in the evaluation of the multivariate function. Please note that the implementation of multivariate functions is still in progress.

The ```MultivariateVectorAutoDiffEvaluate``` module implements the MultiVarVector_AutoDiff_Evaluate function, which autodifferentiates and evaluates a multivariate function using forward mode at user defined vector input values.

The ```MultivariateVectorVarCreator``` module implements the Multi_Vector_AutoDiff_Creator class, which instantiates multiple AutoDiff objects (for use in multivariate functions).

The ```ReverseJacobVectorFunc``` implements the ReverseVecFunc class, which takes the vector function and variables as inputs, and computes the value and the Jacobian matrix using the reverse mode of automatic differentiation.

The ```ReverseMode``` module implements the node class, which is a single AutoDiff Object used for reverse mode.

### Testing Suite
Our testing files live in the `test/` directory. The tests are run using pytest.

### Installation procedure

1) For general users, install the autodiff32 package using pip (see 'How to Use autodiff32' above for complete instructions)

```pip install autodiff32```

2) For developers, feel free to clone this repository, and use the requirements.txt file to ensure you have installed all necessary dependencies.

## Implementation details

The current implementation of AutoDiff32 allows for scalar univariate inputs to functions. Core classes include the AutoDiff class, . AutoDiff32 is externally dependent on numpy, and this dependency has been automatically taken care of in the released version of the package (as well as in the requirements.txt file if the user chooses to manually download the package). AutoDiff32 has implemented a number of elementary functions, as listed above in the description of the basic modules.

We plan to continue development of the AutoDiff32 package to allow for vector inputs as well as multivariate inputs. This will require robust Jacobian matrix functionality.

In addition to the currently implemented forward mode, we plan to build out the reverse mode of auto differentiation as an advanced feature.

### Forward mode Details:

The core data structure (and also external dependencies) for our implementation will be numpy arrays, and the core classes we have implemented are described below:

1) An ***AutoDiff class*** which stores the current value and derivative for the current node. The class method will conatin overloading operators such as plus, minus, multiply, etc.

```python
"""
Initialize an Automatic Differentiation object which stores its current value and derivative

Note that the derivative needs to not be a scalar value

For multivariable differentiation problems of a scalar function, the derivative will be a vector
"""

def __init__(self,value,der=1) :
  #Store the value of the Autodiff object
  #Store derivative of autodiff object (default value is 1)

#overloading operators to enable basic operations between classes and numbers (not exhaustive):

"""
These methods will differentiate cases in which other is a scalar, a vector, a class, or any child class

All of these operators will return AutoDiff classes
"""
def __mult__(self,other):
def __rmult__(self,other):
def __radd__(self,other):
def __add__(self,other):

```

Multivariate functions (both scalar and vector) can be also evaluated using the AutoDiff class as below, and the resulting Jacobian will be a vector array:

```python
X = AutoDiff(1,np.array([1,0]))
Y =  AutoDiff(1,np.array([0,1]))

func = X + 2*Y
```

While this way of defining and evaluating multivariate functions is feasible, it is very inconvenient for the users to have to keep track of the dimensionality of the derivatives. For,example, the above func definition will raise an error if Y's derivative is defined as np.array([0,0,1]). This potential problem in dimensionality will also be a cause difficulties in the error handling process in the code.

The way we tackle this problem is to create a helper class called **Multi_AutoDiff_Creator** as described below:

2) A ***Multi_AutoDiff_Creator class*** which helps the user create variable objects from a multivariable function

```python

"""
This class helps users initialize different variables from a multivariable function without explicitly specifying them using separate AutoDiff classes

It will need to import AutoDiff Class
"""
	def __init__(self,*args,**kwargs):
    '''
    INPUT : variables as kwargs such as (X=3,Y =4)
    RETURN : X number of autodiff objects with length kwargs and each with its 'vector' derivatives

    '''
    #Demo and comparison

    '''initiate a multivariable function using Multi_AutoDiff_Creator class'''
    X,Y= Multi_AutoDiff_Creator(X = 1., Y=3.).Vars #X,Y are autodiff object with derivative [1,0] and [0,1]
    func = X + 2*Y*X
```

Notice that this class only serves as a **helper class** to ensure that every variable created has the correct format of dimensionality. The class itself has no influence on the forward mode differentation process.

For better calculation of the derivatives of our elementary functions, we also introduce our elementary function methods.

3) An ***Elementary function*** file which calculate derivatives for elementary functions as described previously
```python
    #Elementary Function Derivative (not exhaustive):
    def exp(self,ADobj):
    def tan(self,ADobj):
    def sin(self,ADobj):
    def cos(self.ADobj):
    '''
   RETURN an AutoDiff object if ADobj is an AutoDiff object with related value and derivative for the particular elementary functions
   Return np.exp/tan/sin(ODobj) if ODobj is a number
    '''

```

4) A ***Jacobian*** class which helps the user compute the Jacobian matrix for vector functions evaluated at a single point
```python
class Jacobian:
	def __init__(self,vector_func):
		#The Jacobian class is initiated by a vector function

	def value(self):
    # Return the Jacobian value of the vector functions evaluated at a single point by looping through the vector function
    '''
      x, y = ad.Multi_AutoDiff_Creator(x=2, y=3).Vars
      #Define a vector function
      func = np.array([x+y, 2*x*y])
      Jacob = ad.Jacobian(func)
      print(Jacob.value()) # this class method output the full jacobian matrix as an np array
      '''
```

## Extension

**Description**

The implementation of reverse mode also lies in the same autodiff32 package with no additional extension requirements.

We implemented the reverse mode of automatic differentiation in which users can evaluate univariate and multivariate scalar/vector functions with series or single values. The alogorithm is based on the computational flow chart (graph) for reverse mode discussed in class, in which each node is recorded in sequence in a Graph for later use in value and derivative computation. In particular, each node stores its value, the graph it connects to, and the index it has in the graph. In order to backward compute the derivative for our root node, each node records the parent which produced it through some operations, and also the oepration type ("plus","sub",etc.) itself.

We find this more intuitive and pedagogical than using recursion, and potentially can be more computationally efficient since each node only has to remeber its direct children, but not the indirect ones. The graph and node can also be reused easily, which makes it simpler and more computationally efficient in our evaluation for multiple values of our variables.

The high level mathematical ideas will be similar to that of the forward mode, the only additional thing that will be helpful to keep in mind is the chain rule, which is:

\begin{aligned}
\frac{dz}{dx} = \frac{dz}{dy}\times\frac{dy}{dx}
\end{aligned}

This will help use understand how we can calculate the derivatives by starting to set seed value of 1 for our function, which is:
\begin{aligned}
\frac{dz}{df} = 1
\end{aligned}
For example, if we would like to calculate the derivative for $f = 2x$ backward,by letting $z=f$,we will have:

\begin{aligned}
\frac{dz}{dx}=\frac{dz}{df}\times\frac{df}{dx} = 1 \times 2 = 2
\end{aligned}

###Implementation Details for Reverse Mode

The reverse mode implementation will have the following classes:

1) A **Node** class which serve as single automatic differentiation object.All the node will be connected to the same graph for a given function.
```python
class Node:
  #INITIATOR
  '''
    The Node class is initiated with the paramters mentioned below. The node
    will connect to the graph as soon as it is created.
  '''
  def __init__(values,Graph,derivative,leftparentnode,rightparentnode,derivative = 0):

  def CheckConstant(x):
    #Check if x is a Node or not. If it is ,return it,
    # if not,return a new node with Node.value = x ,and connect it to the Graph.

  #OVERLOADING OPERATORS
  '''
  perform subtraction between nodes
  Return
  ======
  A new node which store the value,the graph it connects to, and self as its left nod and other as its right node.
  '''
  def __mul__(self,other):
  def __sub__(self,pther):
  def __truediv__(self,other):

```

2 A **ComputationalGraph** class which stores the nodes of a given function in sequence and compute the value and the gradient of the function and the root variables.

```python

class ComputationalGraph:

  def __init__(self):
    # the graph is initialized by an empty list
  def append(self):
    #append the node in sequence in the graph and record its index for later computation
  def ValidOp(self):
    # sturcturely store every valid operator in this reverse mode computation
     '''
    RETURN
    ======
    Valid operator code if valid otherwise raise error
    '''
  def ComputeValue(self):
    #Compute the Value of the function by a forward pass of th graph

  def ComputeGradient(self,lastindex  = -1):
    '''
    Backward propagate through the graph to calcutate the derivative of the  nodes and store it in each node by looping through the list in a reverse order and update the parent nodes using child nodes.
    INPUT : Last index is the seed : dz/df = 1

    RETURN
    ======
    NONE
    All the values after computation is stored in each node in the list
    '''
  def SeriesValues(self,args):
    #compute the value and derivatives for
    #a series of values for a function (illustration in detail in the How to use session)
    '''
    RETURN
    ===
    A two dimensional  Array for derivatives and one dimensional array for values
    Values for the funtion evaulated at different points
    Derivatives of root variables evaluated at different points
    '''

```

3) A class **ReverseVecFunc** which calculate the value and derivative for vector functions evaluated at different points
```python

class ReverseVecFunc:

  def __init__(self):
    #it stores the variables and the functions when the class is initialized.
  def value(self,Graph):
    #it computes the Jacobian and the value for vector functions for a given single value of variable
  def SeriesValues(self,values,dimension,Graph):
    '''
    INPUT
    =====
    value = values of the functions
    dimension : the number of variables
    Graph : the graph to be connected
    RETURN
    ======
    The value and the jacobian (both in 2D nparrays) of the function at a series of values

    '''
    #pseudocodes
    initialize a valuelist = []
    for each function in vection functions:
       calculate its gradients and values using Wrapper(args)
       append it to a valuelist

    def Wrapper(args):
      # A helper function to help calculate the values and derivative for a series values
      '''
      INPUT
      =====
      value = values of the functions
      dimension : the number of variables
      Graph : the graph to be connected
      Returns the derivatives and values of a single function evaluated different values
      '''

```
4) Additional **ElementaryReverse.py** which defines the elementary functions operations for the Node objects
```python

'''
NOTE
=====
In order to differentiate between reverse mode elementary functions and forward mode ones, all the elementary function in reverse mode will have a "r" as the last alphabet.

If user would like to use these function with a constant , please use built in functions in Numpy: such as : np.exp(2), np.sin(3) etc.

RETURN
=====
New Node object that stores the value after related computation and its relevant leftparent node (which is itself). The node returned wont have any rightparent node.

'''
def expr(x):
def logr(x):
def sqrtr(x):
  ...
```

## Future Features
As we have seen, automatic differentiation is an efficient and accurate way to compute derivatives, so it makes sense
to try to apply this where we can. One of the most popular methods that use the derivative is gradient descent.
Gradient descent is an optimization problem where we try to minimize some function. We do this by taking some starting point
and moving in the direction
of the steepest part of the function, or the most negative gradient. To find this minimum point, it makes sense that
want the gradient to be precise, so the point is precise, so automatic differentiation is the natural choice.

Building onto that idea, another use of this could be in neural networks. The entire concept of a neural network
is centered around optimizing its weights, and at each step and each time the node weights update, we need to calculate
lots of partial derivatives, so once again, it makes sense to use automatic differentiation.
Similarly, the concept of deep learning is essentially a large collection of neural networks, so the same utility
from automatic differentiation can be gained here.

Another interesting application can be found in statistics, namely, in a Markov chain Monte Carlo sampling method, called
the Hamiltonian Monte Carlo Algorithm. Without diving into deep statistical explanations, this is an algorithm to create a
random sample from a probability distribution that is difficult to get normally. Most Markov chain Monte Carlo algorithms
coverge quite slowly, and as a result explore the sampling slowly as well. The hamiltonian algorithm
does a better job with convergence, but at the cost of having to evaluate complicated gradients of probability models along the way.
However, with automatic differentiation, the user no longer has to manually derive the gradients, and this cost can be reduced significantly.

ref:http://jmlr.org/papers/volume18/17-468/17-468.pdf