$$
\text{This is the cutdown companion Jupyter notebook of Chapter 2, Postulates of Quantum Mechanics, of the book titled:}$$
$$\text{ "Quantum Chemistry and Computing for the Curious: Illustrated with Python and Qiskit® code" and with ISBN-13: 978-1803243900.}$$

The following MIT license only applies to the code, and not to the text and images. The authors are not granting a license to replicate or reuse the text and images in the companion Jupyter notebook.

# MIT License

Copyright (c) 2022 Packt

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

#2. Postulates of Quantum Mechanics

# Technical requirements

Install NumPy using the following command:

```python
# pip install numpy
```

Install SymPy using the following command:

```python
pip install sympy
```

Install Qiskit using the following command:

```python
pip install qiskit
```

Install Qiskit visualization support using the following command:

```python
pip install 'qiskit[visualization]'
```

Import the required functions and class methods. The array_to_latex function() returns a Latex representation of a complex array with dimension 1 or 2:

```python
from qiskit.visualization import array_to_latex
```

Import math libraries using the following commands:

```python
import cmath
import math
```

# 2.1. Postulate 1 - Wave functions

## 2.1.1. Spherical harmonic functions

### Importing NumPy, SciPy and Matplotlib Python modules

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from scipy.special import sph_harm
```

### Setting-up grids of polar (theta - $\theta$) and azimuthal (phi - $\phi$)

```python
def setup_grid(num=100):
  theta = np.linspace(0, np.pi, num)
  phi = np.linspace(0, 2*np.pi, num)
  # Create a 2D NumPy meshgrid from two 1D arrays of theta, phi coordinates
  theta, phi = np.meshgrid(theta, phi)
  # Compute cartesian coordinates with radius r = 1
  xyz = np.array([np.sin(theta) * np.sin(phi),
                np.sin(theta) * np.cos(phi),
                np.cos(theta)])
  return (theta, phi, xyz)
```

```python
(theta, phi, xyz) = setup_grid()
print("Shape of meshgrid arrays, theta: {}, phi: {}, xyz: {}".format(theta.shape, phi.shape, xyz.shape))
```

    Shape of meshgrid arrays, theta: (100, 100), phi: (100, 100), xyz: (3, 100, 100)

### Coloring the plotted surface of the real functions of the spherical harmonic function (Y)

```python
def colour_plot(ax, Y, Yx, Yy, Yz, cmap):
  # Colour the plotted surface according to the sign of Y.real
  # https://matplotlib.org/stable/gallery/mplot3d/surface3d.html?highlight=surface%20plots
  # https://matplotlib.org/stable/tutorials/colors/colormaps.html
  cmap = plt.cm.ScalarMappable(cmap=plt.get_cmap(cmap))
  cmap.set_clim(-0.5, 0.5)
  ax.plot_surface(Yx, Yy, Yz,
                  facecolors=cmap.to_rgba(Y.real),
                  rstride=2, cstride=2)
  return
```

### Drawing a set of $x$, $y$, $z$ axes for reference

```python
def draw_axes(ax, ax_lim, title):
  ax.plot([-ax_lim, ax_lim], [0,0], [0,0], c='0.5', lw=1, zorder=10)
  ax.plot([0,0], [-ax_lim, ax_lim], [0,0], c='0.5', lw=1, zorder=10)
  ax.plot([0,0], [0,0], [-ax_lim, ax_lim], c='0.5', lw=1, zorder=10)

  # Set the limits, set the title and then turn off the axes frame
  ax.set_title(title)
  ax.set_xlim(-ax_lim, ax_lim)
  ax.set_ylim(-ax_lim, ax_lim)
  ax.set_zlim(-ax_lim, ax_lim)
  ax.axis('off')
  return
```

### Defining a function which computes the real form of the spherical harmonic Y

```python
def comb_Y(l, m, theta, phi):
  Y = sph_harm(abs(m), l, phi, theta)
  if m < 0:
    Y = np.sqrt(2) * (-1)**m * Y.imag
  elif m > 0:
    Y = np.sqrt(2) * (-1)**m * Y.real
  return Y
```

### Defining a function that displays the spatial wave functions for a range of values of the angular momentum quantum number and the magnetic quantum number

```python
def plot_orbitals(k, cmap = 'autumn'):
  for l in range(0, k+1):
    for m in range(-l, l+1):
      fig = plt.figure(figsize=plt.figaspect(1.))
      (theta, phi, xyz) = setup_grid()
      ax = fig.add_subplot(projection='3d')
      Y = comb_Y(l, m, theta, phi)
      title = r'$l={{{}}}, m={{{}}}$'.format(l, m)
      Yx, Yy, Yz = np.abs(Y) * xyz
      colour_plot(ax, Y, Yx, Yy, Yz, cmap)
      draw_axes(ax, 0.5, title)
      fig_name = 'Hydrogen_l'+str(l)+'_m'+str(m)
      plt.savefig(fig_name)
      plt.show()
  return
```

## Spatial wave functions of the hydrogen atom
The result is shown in Figure 2.4.

```python
plot_orbitals(2)
```

## 2.1.2. Addition of momenta using Clebsch-Gordan coefficients

```python
import sympy
from sympy import S
from sympy.physics.quantum.cg import CG, cg_simp
```

### Defining a Clebsch-Gordan coefficient and evaluating its value

#### Fermionic spin pairing to the symmetric state $|s_{1,2}=1, m_{s_{1,2}}:

```python
CG(S(1)/2, S(1)/2, S(1)/2, -S(1)/2, 1, 0).doit()
```

$\displaystyle \frac{\sqrt{2}}{2}$

```python
CG(S(1)/2, -S(1)/2, S(1)/2, S(1)/2, 1, 0).doit()
```

$\displaystyle \frac{\sqrt{2}}{2}$

Figure 2.5 – Defining a Clebsch-Gordan coefficient and evaluating its value

#### Fermionic spin pairing to antisymmetric state $(s_{1,2}=0,m_{s_{1,2}}=0)$

```python
CG(S(1)/2, S(1)/2, S(1)/2, -S(1)/2, 0, 0).doit()
```

$\displaystyle \frac{\sqrt{2}}{2}$

```python
CG(S(1)/2, -S(1)/2, S(1)/2, S(1)/2, 0, 0).doit()
```

$\displaystyle - \frac{\sqrt{2}}{2}$

Figure 2.6 – Defining a Clebsch-Gordan coefficient and evaluating its value

#### Coupling spin and angular momentum

```python
CG(1, 0, S(1)/2, S(1)/2, S(1)/2, S(1)/2).doit()
```

$\displaystyle - \frac{\sqrt{3}}{3}$

```python
CG(1, 1, S(1)/2, -S(1)/2, S(1)/2, S(1)/2).doit()
```

$\displaystyle \frac{\sqrt{6}}{3}$

```python
CG(1, -1, S(1)/2, S(1)/2, S(1)/2, S(1)/2).doit()
```

$\displaystyle 0$

Figure 2.7 – Defining a Clebsch-Gordan coefficient and evaluating its value

### Spatial wave functions of different states of the nitrogen atom with 3 $p$ electrons

#### Spatial wave function of the ground state of the nitrogen atom with 3 $p$ electrons in $L=0, M=0$

#### Setting-up a dictionary of six configuration tuples

```python
T00 = {0: (1,-1, 1,0,  1,-1, 1,1,  0,0),
       1: (1,-1, 1,1,  1,0,  1,0,  0,0),
       2: (1,0,  1,-1, 1,-1, 1,1,  0,0),
       3: (1,0,  1,1,  1,1,  1,-1, 0,0),
       4: (1,1,  1,-1, 1,0,  1,0,  0,0),
       5: (1,1,  1,0,  1,1,  1,-1, 0,0)}
```

#### Defining a function which computes a product of Clebsch-Gordan coefficients

```python
def comp_CG(T, k, display = None):
  CGk = CG(*T[k][0:6]) * CG(*T[k][4:10])
  if display:
    print('CG(', *T[k][0:6], ') = ', CG(*T[k][0:6]).doit())
    print('CG(', *T[k][4:10], ') = ', CG(*T[k][4:10]).doit())
    print("CG{} =".format(k), 'CG(', *T[k][0:6], ') * CG(', *T[k][4:10], ') = ', CGk.doit())
  return CGk
```

```python
CG0 = comp_CG(T00, 0, display=True)
```

    CG( 1 -1 1 0 1 -1 ) =  -sqrt(2)/2
    CG( 1 -1 1 1 0 0 ) =  sqrt(3)/3
    CG0 = CG( 1 -1 1 0 1 -1 ) * CG( 1 -1 1 1 0 0 ) =  -sqrt(6)/6

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T00)):
  s = 'CG' + str(k) +' = comp_CG(T00, ' + str(k) + ')'
  exec(s)

s00 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}, CG5: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit(), CG5.doit())]

print(s00)
```

    ['CG0: -sqrt(6)/6, CG1: sqrt(6)/6, CG2: sqrt(6)/6, CG3: -sqrt(6)/6, CG4: -sqrt(6)/6, CG5: sqrt(6)/6']

#### Defining a set of spatial wave functions

```python
def Y_phase(theta, phi):

  Y10a = comb_Y(1, 0, theta, phi)
  Y11a = comb_Y(1, 1, theta, phi)
  Y1m1a = comb_Y(1, -1, theta, phi)

  Y10b = comb_Y(1, 0, theta, phi+1*np.pi/3)
  Y11b = comb_Y(1, 1, theta, phi+1*np.pi/3)
  Y1m1b = comb_Y(1, -1, theta, phi+1*np.pi/3)

  Y10c = comb_Y(1, 0, theta, phi+2*np.pi/3)
  Y11c = comb_Y(1, 1, theta, phi+2*np.pi/3)
  Y1m1c = comb_Y(1, -1, theta, phi+2*np.pi/3)

  return(Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c)
```

#### Computing the wave function of the ground state of the nitrogen atom with 3 $p$ electrons $(L=0, M=0)$

```python
def compute_00_Y(ax_lim, cmap, title,  fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y1m1a * Y10b * Y11c
  Y_01 = float(CG1.doit()) * Y1m1a * Y11b * Y10c
  Y_02 = float(CG2.doit()) * Y10a * Y1m1b * Y11c
  Y_03 = float(CG3.doit()) * Y10a * Y11b * Y1m1c
  Y_04 = float(CG4.doit()) * Y11a * Y1m1b * Y10c
  Y_05 = float(CG5.doit()) * Y11a * Y10b * Y1m1c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04 + Y_05

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the ground state of Nitrogen with 3 $p$ electrons $(L=0, M=0)$

```python
title = '$L=0,\ M=0$'
fig_name ='Nitrogen_3p_L0_M0.png'
compute_00_Y(0.01, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=1, M=0)$

### Setting-up a dictionary of configuration arrays

#### Defining a function which computes the Clebsch-Gordan coefficients

```python
T10 = {0: (1,1,  1,0,   2,1,  1,-1, 1,0),
       1: (1,0,  1,1,   2,1,  1,-1, 1,0),
       2: (1,1,  1,-1,  2,0,  1,0,  1,0),
       3: (1,-1, 1,1,   2,0,  1,0,  1,0),
       4: (1,1,  1,0,   1,1,  1,-1, 1,0),
       5: (1,0,  1,1,   1,1,  1,-1, 1,0),
       6: (1,-1, 1,0,   2,-1, 1,1,  1,0),
       7: (1,0,  1,-1,  2,-1, 1,1,  1,0),
       8: (1,-1, 1,0,   1,-1, 1,1,  1,0),
       9: (1,0,  1,-1,  1,-1, 1,1,  1,0),
       10:(1,1,  1,-1,  0,0,  1,0,  1,0)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T10)):
  s = 'CG' + str(k) +' = comp_CG(T10, ' + str(k) + ')'
  exec(s)

s10a = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit())]
print(s10a)

s10b = ["CG5: {}, CG6: {}, CG7: {}, CG8: {}, CG9: {}, CG10: {}".
     format(CG5.doit(), CG6.doit(), CG7.doit(), CG8.doit(), CG9.doit(), CG10.doit())]
print(s10b)
```

    ['CG0: sqrt(15)/10, CG1: sqrt(15)/10, CG2: -sqrt(15)/15, CG3: -sqrt(15)/15, CG4: 1/2']
    ['CG5: -1/2, CG6: sqrt(15)/10, CG7: sqrt(15)/10, CG8: 1/2, CG9: -1/2, CG10: sqrt(3)/3']

#### Computing the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=1, M=0)$

```python
def compute_10_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y10b * Y1m1c
  Y_01 = float(CG1.doit()) * Y10a * Y11b * Y1m1c
  Y_02 = float(CG2.doit()) * Y11a * Y1m1b * Y10c
  Y_03 = float(CG3.doit()) * Y1m1a * Y11b * Y10c
  Y_04 = float(CG4.doit()) * Y11a * Y10b * Y1m1c
  Y_05 = float(CG5.doit()) * Y10a * Y11b * Y1m1c
  Y_06 = float(CG6.doit()) * Y1m1a * Y10b * Y11c
  Y_07 = float(CG7.doit()) * Y10a * Y1m1b * Y11c
  Y_08 = float(CG8.doit()) * Y1m1a * Y10b * Y11c
  Y_09 = float(CG9.doit()) * Y10a * Y1m1b * Y11c
  Y_10 = float(CG10.doit()) * Y11a * Y1m1b * Y10c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04 + Y_05 + Y_06 + Y_07 + Y_08 + Y_09 + Y_10

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=1, M=0)$

```python
title = '$L=1,\ M=0$'
fig_name ='Nitrogen_3p_L1_M0.png'
compute_10_Y(0.030, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=2, M=0)$

#### Setting-up a dictionary of configuration arrays

```python
T20 = {0: (1,1,  1,0,   2,1,  1,-1, 2,0),
       1: (1,0,  1,1,   2,1,  1,-1, 2,0),
       2: (1,0,  1,-1,  2,-1, 1,1,  2,0),
       3: (1,-1, 1,0,   2,-1, 1,1,  2,0),
       4: (1,0,  1,1,   1,1,  1,-1, 2,0),
       5: (1,1,  1,0,   1,1,  1,-1, 2,0),
       6: (1,-1, 1,0,   1,-1, 1,1,  2,0),
       7: (1,0,  1,-1,  1,-1, 1,1,  2,0),
       8: (1,1,  1,-1,  1,0,  1,0,  2,0),
       9: (1,-1, 1,1,   1,0,  1,0,  2,0)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T20)):
  s = 'CG' + str(k) +' = comp_CG(T20, ' + str(k) + ')'
  exec(s)

s20 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}, CG5: {}, CG6: {}, CG7: {}, CG8: {}, CG9: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit(), CG5.doit(), CG6.doit(), CG7.doit(), CG8.doit(), CG9.doit())]
print(s20)
```

    ['CG0: 1/2, CG1: 1/2, CG2: -1/2, CG3: -1/2, CG4: -sqrt(3)/6, CG5: sqrt(3)/6, CG6: -sqrt(3)/6, CG7: sqrt(3)/6, CG8: sqrt(3)/3, CG9: -sqrt(3)/3']

#### Computing the wave function of the excited state of nitrogen with 3 $p$ electrons $(L=2, M=0)$

```python
def compute_20_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y10b * Y1m1c
  Y_01 = float(CG1.doit()) * Y10a * Y11b * Y1m1c
  Y_02 = float(CG2.doit()) * Y10a * Y1m1b * Y11c
  Y_03 = float(CG3.doit()) * Y1m1a * Y10b * Y11c
  Y_04 = float(CG4.doit()) * Y10a * Y11b * Y1m1c
  Y_05 = float(CG5.doit()) * Y11a * Y10b * Y1m1c
  Y_06 = float(CG6.doit()) * Y1m1a * Y10b * Y11c
  Y_07 = float(CG7.doit()) * Y10a * Y1m1b * Y11c
  Y_08 = float(CG8.doit()) * Y11a * Y1m1b * Y10c
  Y_09 = float(CG9.doit()) * Y1m1a * Y11b * Y10c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04 + Y_05 + Y_06 + Y_07 + Y_08 + Y_09

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=2, M=0)$

```python
title = '$L=2,\ M=0$'
fig_name ='Nitrogen_3p_L2_M0.png'
compute_20_Y(0.04, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=0)$

#### Setting-up a dictionary of configuration arrays

```python
T30 = {0: (1,1,  1,-1, 2,0,  1,0,  3,0),
       1: (1,0,  1,-1, 2,-1, 1,1,  3,0),
       2: (1,0,  1,1,  2,1,  1,-1, 3,0),
       3: (1,-1, 1,1,  2,0,  1,0,  3,0),
       4: (1,-1, 1,0,  2,-1, 1,1,  3,0),
       5: (1,1,  1,0,  2,1,  1,-1, 3,0)}
```

### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T30)):
  s = 'CG' + str(k) +' = comp_CG(T30, ' + str(k) + ')'
  exec(s)

s30 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}, CG5: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit(), CG5.doit())]
print(s30)
```

    ['CG0: sqrt(10)/10, CG1: sqrt(10)/10, CG2: sqrt(10)/10, CG3: sqrt(10)/10, CG4: sqrt(10)/10, CG5: sqrt(10)/10']

#### Computing the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=0)$

```python
def compute_30_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y1m1b * Y10c
  Y_01 = float(CG1.doit()) * Y10a * Y1m1b * Y11c
  Y_02 = float(CG2.doit()) * Y10a * Y11b * Y1m1c
  Y_03 = float(CG3.doit()) * Y1m1a * Y11b * Y10c
  Y_04 = float(CG4.doit()) * Y1m1a * Y10b * Y11c
  Y_05 = float(CG5.doit()) * Y11a * Y10b * Y1m1c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04 + Y_05

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=0)$

```python
title = '$L=3,\ M=0$'
fig_name ='Nitrogen_3p_L3_M0.png'
compute_30_Y(0.02, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=1, M=1)$

#### Setting-up a dictionary of configuration arrays

```python
T11 = {0: (1,1,  1,1,   2,2,  1,-1, 1,1),
       1: (1,1,  1,0,   2,1,  1,0,  1,1),
       2: (1,0,  1,0,   2,0,  1,1,  1,1),
       3: (1,1,  1,-1,  2,0,  1,1,  1,1),
       4: (1,-1, 1,1,   2,0,  1,1,  1,1),
       5: (1,0,  1,1,   2,1,  1,0,  1,1),
       6: (1,0,  1,1,   1,1,  1,0,  1,1),
       7: (1,1,  1,0,   1,1,  1,0,  1,1)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T11)):
  s = 'CG' + str(k) +' = comp_CG(T11, ' + str(k) + ')'
  exec(s)

s11 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}, CG5: {}, CG6: {}, CG7: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit(), CG5.doit(), CG6.doit(), CG7.doit())]
print(s11)
```

    ['CG0: sqrt(15)/5, CG1: -sqrt(15)/10, CG2: sqrt(15)/15, CG3: sqrt(15)/30, CG4: sqrt(15)/30, CG5: -sqrt(15)/10, CG6: -1/2, CG7: 1/2']

#### Computing the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=1, M=1)$

```python
def compute_11_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y11b * Y11c
  Y_01 = float(CG1.doit()) * Y10a * Y10b * Y11c
  Y_02 = float(CG2.doit()) * Y11a * Y10b * Y10c
  Y_03 = float(CG3.doit()) * Y11a * Y1m1b * Y11c
  Y_04 = float(CG4.doit()) * Y1m1a * Y11b * Y11c
  Y_05 = float(CG5.doit()) * Y10a * Y11b * Y10c
  Y_06 = float(CG6.doit()) * Y10a * Y11b * Y10c
  Y_07 = float(CG7.doit()) * Y11a * Y10b * Y10c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04 + Y_05 + Y_06 + Y_07

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=1, M=1)$

```python
title = '$L=1,\ M=1$'
fig_name ='Nitrogen_3p_L1_M1.png'
compute_11_Y(0.035, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of nitrogen with 3 $p$ electrons $(L=2, M=1)$

#### Setting-up a dictionary of configuration arrays

```python
T21 = {0: (1,1,  1,-1,  1,0,  1,1,  2,1),
       1: (1,-1, 1,1,   1,0,  1,1,  2,1),
       2: (1,1,  1,0,   1,1,  1,0,  2,1),
       3: (1,0,  1,1,   1,1,  1,0,  2,1),
       4: (1,1,  1,1,   2,2,  1,-1, 2,1),
       5: (1,1,  1,-1,  2,0,  1,1,  2,1),
       6: (1,-1, 1,1,   2,0,  1,1,  2,1),
       7: (1,1,  1,0,   2,1,  1,0,  2,1),
       8: (1,0,  1,1,   2,1,  1,0,  2,1)}
```

###" Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T21)):
  s = 'CG' + str(k) +' = comp_CG(T21, ' + str(k) + ')'
  exec(s)

s21 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}, CG5: {}, CG6: {}, CG7: {}, CG8: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit(), CG5.doit(), CG6.doit(), CG7.doit(), CG8.doit())]
print(s21)
```

    ['CG0: 1/2, CG1: -1/2, CG2: 1/2, CG3: -1/2, CG4: sqrt(3)/3, CG5: -sqrt(3)/6, CG6: -sqrt(3)/6, CG7: sqrt(3)/6, CG8: sqrt(3)/6']

#### Computing the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=2, M=1)$

```python
def compute_21_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y1m1b * Y11c
  Y_01 = float(CG1.doit()) * Y1m1a * Y11b * Y11c
  Y_02 = float(CG2.doit()) * Y11a * Y10b * Y10c
  Y_03 = float(CG3.doit()) * Y10a * Y11b * Y10c
  Y_04 = float(CG4.doit()) * Y11a * Y11b * Y1m1c
  Y_05 = float(CG5.doit()) * Y11a * Y1m1b * Y11c
  Y_06 = float(CG6.doit()) * Y1m1a * Y11b * Y11c
  Y_07 = float(CG7.doit()) * Y11a * Y10b * Y10c
  Y_08 = float(CG8.doit()) * Y10a * Y11b * Y10c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04 + Y_05 + Y_06 + Y_07 + Y_08

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=2, M=1)$

```python
print("We assume that at least one electron has been excited to a different energy level.")
title = '$L=2,\ M=1$'
fig_name ='Nitrogen_3p_L2_M1.png'
compute_21_Y(0.035, 'autumn', title, fig_name)
```

## Spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=1)$

#### Setting-up a dictionary of configuration arrays

```python
T31 = {0: (1,1,  1,0,   2,1,  1,0,  3,1),
       1: (1,0,  1,1,   2,1,  1,0,  3,1),
       2: (1,1,  1,-1,  2,0,  1,1,  3,1),
       3: (1,-1, 1,1,   2,0,  1,1,  3,1),
       4: (1,1,  1,1,   2,2,  1,-1, 3,1)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T31)):
  s = 'CG' + str(k) +' = comp_CG(T31, ' + str(k) + ')'
  exec(s)

s31 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit())]
print(s31)
```

    ['CG0: 2*sqrt(15)/15, CG1: 2*sqrt(15)/15, CG2: sqrt(15)/15, CG3: sqrt(15)/15, CG4: sqrt(15)/15']

#### Defining a function which computes the wave function of states of the nitrogen with 3 $p$ electrons

```python
def compute_31_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y10b * Y10c
  Y_01 = float(CG1.doit()) * Y10a * Y11b * Y10c
  Y_02 = float(CG2.doit()) * Y11a * Y1m1b * Y11c
  Y_03 = float(CG3.doit()) * Y1m1a * Y11b * Y11c
  Y_04 = float(CG4.doit()) * Y11a * Y11b * Y1m1c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=1)$

```python
title = '$L=3,\ M=1$'
fig_name ='Nitrogen_3p_L3_M1.png'
compute_31_Y(0.03, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=2, M=2)$

#### Setting-up a dictionary of configuration arrays

```python
T22 = {0: (1,1,  1,1,   2,2,  1,0,  2,2),
       1: (1,1,  1,0,   1,1,  1,1,  2,2),
       2: (1,0,  1,1,   1,1,  1,1,  2,2),
       3: (1,1,  1,0,   2,1,  1,1,  2,2),
       4: (1,0,  1,1,   2,1,  1,1,  2,2)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T22)):
  s = 'CG' + str(k) +' = comp_CG(T22, ' + str(k) + ')'
  exec(s)

s22 = ["CG0: {}, CG1: {}, CG2: {}, CG3: {}, CG4: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit(), CG3.doit(), CG4.doit())]
print(s22)
```

    ['CG0: sqrt(6)/3, CG1: sqrt(2)/2, CG2: -sqrt(2)/2, CG3: -sqrt(6)/6, CG4: -sqrt(6)/6']

#### Defining a function which computes the wave function of states of the nitrogen atom with 3 $p$ electrons

```python
def compute_22_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y11b * Y10c
  Y_01 = float(CG1.doit()) * Y11a * Y10b * Y11c
  Y_02 = float(CG2.doit()) * Y10a * Y11b * Y11c
  Y_03 = float(CG3.doit()) * Y11a * Y10b * Y11c
  Y_04 = float(CG4.doit()) * Y10a * Y11b * Y11c

  Y = Y_00 + Y_01 + Y_02 + Y_03 + Y_04

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=2, M=2)$

```python
title = '$L=2,\ M=2$'
fig_name ='Nitrogen_3p_L2_M2.png'
compute_22_Y(0.025, 'autumn', title, fig_name)
```

#### Spatial wave function of the excited state of nitrogen with 3 $p$ electrons $(L=3, M=2)$

#### Setting-up a dictionary of configuration arrays

```python
T32 = {0: (1,1,  1,0,   2,1,  1,1,  3,2),
       1: (1,0,  1,1,   2,1,  1,1,  3,2),
       2: (1,1,  1,1,   2,2,  1,0,  3,2)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
for k in range(0, len(T32)):
  s = 'CG' + str(k) +' = comp_CG(T32, ' + str(k) + ')'
  exec(s)

s32 = ["CG0: {}, CG1: {}, CG2: {}".
     format(CG0.doit(), CG1.doit(), CG2.doit())]
print(s32)
```

    ['CG0: sqrt(3)/3, CG1: sqrt(3)/3, CG2: sqrt(3)/3']

#### Defining a function which computes the wave function of states of the nitrogen atom with 3 $p$ electrons

```python
def compute_32_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y10b * Y11c
  Y_01 = float(CG1.doit()) * Y10a * Y11b * Y11c
  Y_02 = float(CG2.doit()) * Y11a * Y11b * Y10c

  Y = Y_00 + Y_01 + Y_02

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=2)$

```python
title = '$L=3,\ M=2$'
fig_name ='Nitrogen_3p_L3_M2.png'
compute_32_Y(0.02, 'autumn', title, fig_name)
```

#### Displaying the spatial wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=3)$

#### Setting-up a dictionary of configuration arrays

```python
T33 = {0: (1,1,  1,1,   2,2,  1,1,  3,3)}
```

#### Computing and printing the Clebsch-Gordan coefficients

```python
CG0 = comp_CG(T33, 0)

s33 = ["CG0: {}".format(CG0.doit())]
print(s33)
```

    ['CG0: 1']

#### Defining a function which computes the wave function of states of the nitrogen atom with 3 $p$ electrons

```python
def compute_33_Y(ax_lim, cmap, title, fig_name):
  fig = plt.figure(figsize=plt.figaspect(1.))
  (theta, phi, xyz) = setup_grid()
  ax = fig.add_subplot(projection='3d')

  (Y10a, Y11a, Y1m1a, Y10b, Y11b, Y1m1b, Y10c, Y11c, Y1m1c) = Y_phase(theta, phi)

  Y_00 = float(CG0.doit()) * Y11a * Y11b * Y11c

  Y = Y_00

  Yx, Yy, Yz = np.abs(Y) * xyz
  colour_plot(ax, Y, Yx, Yy, Yz, cmap)
  draw_axes(ax, ax_lim, title)
  plt.savefig(fig_name)
  plt.show()

  return
```

#### Displaying the wave function of the excited state of the nitrogen atom with 3 $p$ electrons $(L=3, M=3)$

```python
title = '$L=3,\ M=3$'
fig_name ='Nitrogen_3p_L3_M3.png'
compute_33_Y(0.02, 'autumn', title, fig_name)
```

## 2.1.3. General formulation of the Pauli exclusion principle

```python
def frame(mu):
  a = 0
  b = 0
  f = {}
  for k, v in mu.items():
    for c in range(v):
      f[a] = list(range(b+1, b+k+1))
      a += 1
      b += k
  return f
```

```python
print("F_21_10 =", frame({2: 1, 1:0}))
```

    F_21_10 = {0: [1, 2]}

```python
print("F_21_11 =", frame({2: 1, 1:1}))
```

    F_21_11 = {0: [1, 2], 1: [3]}

# 2.2. Postulate 2 – Probability amplitude

## 2.2.1. Computing the radial wavefunctions

```python
a0 = 5.29177210903e-11
```

```python
def comp_ak(n):
  n = max(n,1)
  # Create a dictionary with integer keys, starting with a0
  dict = {0: a0}
  for l in range (n-1):
    for k in range(n-l-1):
      ak = dict.get(k)
      #display("l: {}, k: {}, ak: {}".format(l, k, ak))
      dict[k+1] = ak*(k+l+1-n)/((k+1)*(k+2*l+2))
  return dict
```

```python
d = comp_ak(10)
for k, v in d.items():
  print("{}, {}".format(k,v))
```

    0, 5.29177210903e-11
    1, -2.9398733939055554e-12
    2, 1.9455044518492647e-13
    3, -1.5749321753065475e-14
    4, 1.615315051596459e-15
    5, -2.2027023430860805e-16
    6, 4.2830323337784895e-17
    7, -1.3257004842647707e-17
    8, 7.954202905588625e-18
    9, -1.458270532691248e-17

```python
from sympy.physics.hydrogen import R_nl
from sympy.abc import r, Z
```

```python
R_nl(1, 0, r, Z)
```

$\displaystyle 2 \sqrt{Z^{3}} e^{- Z r}$

```python
R_nl(2, 0, r, Z)
```

$\displaystyle \frac{\sqrt{2} \left(- Z r + 2\right) \sqrt{Z^{3}} e^{- \frac{Z r}{2}}}{4}$

```python
R_nl(2, 1, r, Z)
```

$\displaystyle \frac{\sqrt{6} Z r \sqrt{Z^{3}} e^{- \frac{Z r}{2}}}{12}$

## 2.2.2. Probability amplitude for a hydrogen anion $(H^-)$

```python
from sympy import symbols, integrate, exp, oo
x = symbols('x')
integrate(x**2 *exp(-2*x),(x,0,oo))
```

$\displaystyle \frac{1}{4}$

```python
from sympy import symbols, sin, pi
x = symbols('x')
integrate(sin(x),(x,0,pi))
```

$\displaystyle 2$

```python
integrate(1,(x,0,2*pi))
```

$\displaystyle 2 \pi$

# 2.3. Postulate 3 – Measurable quantities and operators

## 2.3.1. Hermitian operator

## 2.3.2. Unitary operator

## 2.3.3. Density matrix and mixed quantum states

## 2.3.4. Position operation

## 2.3.5. Momentum operation

```python
from sympy import symbols, sin, cos
x = symbols('x')
integrate(cos(x)*sin(x),(x,0,pi))
```

$\displaystyle 0$

## 2.3.6. Kinetic Energy operation

## 2.3.7. Potential Energy operation

```python
from sympy import symbols, integrate, exp, DiracDelta, oo
x, y = symbols('x y')
integrate(x**2 * exp(-2*x) * integrate(y**2 * exp(-2*y)*DiracDelta(x - y),(y,0,oo)),(x,0,oo))
```

$\displaystyle \frac{3}{128}$

```python
from sympy import symbols, integrate, exp, DiracDelta, oo
x, y = symbols('x y')
integrate(x**2 * exp(-2*x) * integrate(DiracDelta(x - y),(y,0,oo)),(x,0,oo))
```

$\displaystyle \frac{1}{4}$

## 2.3.8. Total Energy operation

# 2.4. Postulate 4 – Time independent stationary states

# 2.5. Postulate 5 – Time evolution dynamics
## Schrödinger’s equation

# Questions

1.	What quantum numbers do the total wave function depend on?

```python
#@title Enter your answer in lowercase or ? for a solution, then execute cell.
answer = "" #@param {type:"string"}
solution = "n, l, m_l, s, m_s"
if answer == solution:
  print("Correct")
elif answer == '?':
  print(solution)
else:
  print("Incorrect, please try again")
```

    Incorrect, please try again

2. What is the Clebsch-Gordan coefficient if we couple together $l_1 = 0$ and $m_1 = 0$ and $l_2 = 1$, $m_2 = 0$ to $L=1, M=0$

```python
#@title Enter your answer as an integer or 0 for a solution, then execute cell.
answer =  2#@param {type:"integer"}
solution = 1
if answer == solution:
  print("Correct")
elif answer == 0:
  print(solution)
else:
  print("Incorrect, please try again")
```

    Incorrect, please try again

3. What happens to the total wave function upon application of an antisymmetric operation? It is multiplied by?

```python
#@title Enter your answer in lowercase or ? for a solution, then execute cell.
answer = "" #@param {type:"string"}
solution = "-1"
if answer == solution:
  print("Correct")
elif answer == '?':
  print(solution)
else:
  print("Incorrect, please try again")
```

    Incorrect, please try again

4. For a pure fermionic state, is the Young frame horizontal or vertical?

```python
#@title Enter your answer in lowercase or ? for a solution, then execute cell.
answer = "" #@param {type:"string"}
solution = "vertical"
if answer == solution:
  print("Correct")
elif answer == '?':
  print(solution)
else:
  print("Incorrect, please try again")
```

    Incorrect, please try again

5. What is the position operator for the Z direction? Is it $\hat{\boldsymbol{r}}_{j_{z}} \psi\left(\boldsymbol{r}_{1}, \ldots, \boldsymbol{r}_{N}\right)=\mathrm{z}_{j} \psi\left(\boldsymbol{r}_{1}, \ldots, \boldsymbol{r}_{N}\right)$ ?

```python
#@title Enter your answer 'yes' or 'no' in lowercase or ? for a solution, then execute cell.
answer = "" #@param {type:"string"}
solution = "yes"
if answer == solution:
  print("Correct")
elif answer == '?':
  print(solution)
else:
  print("Incorrect, please try again")
```

    Incorrect, please try again

6. What is the sum of potential and kinetic energy?

```python
#@title Enter your answer in lowercase or ? for a solution, then execute cell.
answer = "" #@param {type:"string"}
solution = "total energy"
if answer == solution:
  print("Correct")
elif answer == '?':
  print(solution)
else:
  print("Incorrect, please try again")
```

    Incorrect, please try again


# Introduction

Since Newton invented calculus, differentiating a function has been essential to the advancement of humanity. Calculating the derivative of a function is crucial to finding the extrema for a function and determining zeros for a function, two operations that are central to optimization (1). Optimization is essential, especially due to the exploding popularity of machine learning, where minimizing a loss function is the essential task. Often, we can find the symbolic/analytical solution to the derivative of a function, however this has become increasingly complex and computationally expensive as our functions/equations have grown in size and complexity. Numerically solving differential equations forms a cornerstone of modern science and engineering and is intimately linked with machine learning; however this method suffers from rounding errors and numerical instability. Many of these issues can be solved using Automatic Differentiation (AD) because AD can calculate the exact derivative up to machine precision (2). The logic and processes behind AD enables it to be implemented using computer code, making it easily accessible for use by scientists and mathematicians. This python package will implement the forward mode of AD.

# How to use the package

## Installation

### Download from PyPI (for basic users)
Ideally, the user should have Anaconda installed (https://www.anaconda.com/download/).

The user can, but does not need to, create a virtual environment using `virtualenv`. If the user has Anaconda installed, then the user will already have `virtualenv` installed, but if they do not, the user can the user can run the following command in the terminal:

    sudo easy_install virtualenv
Then, to create a virtual environment, the user runs the following command in the terminal:

    virtualenv env
To activate the virtual environment, the user runs the following command in the terminal:

    source env/bin/activate
To deactivate the virtual environment, the user runs the following command in the terminal:

    deactivate
Whether or not the user uses a virtual environment, they will need to install the package.

    pip install autodiff-py
All the necessary dependencies will be installed along with `autodiff-py`. The user can then start using the package in their code with an import statement such as

    import autodiff as ad

### Download from Github repo (for developers)
Ideally, the a developer should have Anaconda installed (https://www.anaconda.com/download/).

To install the package, a developer will download or clone the repository onto their local machine. Then, the user can, but does not need to, create a virtual environment using `virtualenv`. If the user has Anaconda installed, then the user will already have `virtualenv` installed. If the user is not using Anaconda, he or she can run the following command in the terminal:

    sudo easy_install virtualenv
Then, to create a virtual environment, the user runs the following command in the terminal in the package directory:

    virtualenv env
To activate the virtual environment, the user runs the following command in the terminal in the package directory:

    source env/bin/activate
To deactivate the virtual environment, the user runs the following command in the terminal in the package directory:

    deactivate
Whether or not the user uses a virtual environment, he or she will need to have the necessary dependencies. The package directory includes a *requirements.txt* file that lists the necessary dependencies. The user can easily install all the dependencies with the following command:

    pip install -r requirements.txt
After installing the necessary dependencies, the user can then start using the package in their code with an import statement such as

    import autodiff as ad
To run the tests for the package, the user can run the following command in terminal in the package folder:

    pytest

```python
#install the package autodiff-py
!pip install autodiff-py
```

## Usage

For example, suppose $f(x, y) = x^2 + 2xy + y^2$, and $x = 1, y = 2$.

$
f(1, 2) = 9\\
\frac{\partial f}{\partial x} = 2x + 2y \implies \frac{\partial f}{\partial x}|_{x = 1, y = 2} = 6 \\
\frac{\partial f}{\partial y} = 2x + 2y \implies \frac{\partial f}{\partial y}|_{x = 1, y = 2} = 6
$

To solve this problem using our package, the user will run the following straightforward code.

```python
import autodiff as ad

# Define x and y scalars with variable name and value. The actual python variable names do not matter.
x = ad.Scalar('x', 1)
y = ad.Scalar('y', 2)
# Express f in terms of x and y. User does not need to define any more scalar objects besides the basic ones x and y above.
f = x ** 2 + 2 * x * y + y ** 2
```

```python
# Get value of f(1, 2)
f.getValue()
```

    9.0

```python
# Get partial derivative with respect to x. Pass in variable names as String in list.
f.getGradient(['x'])
```

    array([6.])

```python
# Get partial derivative with respect to y
f.getGradient(['y'])
```

    array([6.])

```python
# Get both partial derivatives
f.getGradient(['x', 'y'])
```

    array([6., 6.])

```python
# Get all partial derivatives for the function 'f'
f.getDeriv()
```

    {'y': 6.0, 'x': 6.0}

Also included in our package is an optimization library, `optimize`. The package includes Newton's method for root finding and also Quasi-Newton's methods and gradient descent for minimum finding. The benefit of this package is that the user does not need to use any functionality related to `autodiff-py` themselves; they just need to define the function (which only takes in one required argument) they want to optimize/find the root of and give the initial guess as a list/array.

For example, suppose we want to find the root of the function
$\vec{f}(x_1, x_2, x_3) = [2x_1 + 1, -2x_1 - x_2^2 + 3, -3 x_3 + 1]$. The roots of this function will be
$(x_1, x_2, x_3) = (-0.5, 2, 1/3)$ and $(x_1, x_2, x_3) = (-0.5, -2, 1/3)$. We can use the function `newtons_method` to solve for a root. The root found will change depending on the initial starting position. We can also specify the method used to calculate the step size for each iteration in `newtons_method`.

```python
import autodiff.optimize as optimize
import numpy as np

def f(x):
    x1 = x[0]
    x2 = x[1]
    x3 = x[2]
    return np.array([2 * x1 + 1, -2 * x1 - x2 ** 2 + 3, -3 * x3 + 1])
```

```python
#Returns two elements in a tuple. First element corresponds to position of root, the second the number of iterations
#it took to converge
#First root
optimize.newtons_method(f, [1, 1, 0])[0]
```

    array([-0.5       ,  2.        ,  0.33333333])

```python
#Second root
optimize.newtons_method(f, [1, -1, 0])[0]
```

    array([-0.5       , -2.        ,  0.33333333])

```python
#Use gmres action step update method
optimize.newtons_method(f, [1, 1, 0], method = 'gmres_action')[0]
```

    array([-0.5       ,  2.        ,  0.33333333])

```python
optimize.newtons_method(f, [1, -1, 0], method = 'gmres_action')[0]
```

    array([-0.5       , -2.        ,  0.33333333])

Suppose we want to find the minimum of the function $f(x_1, x_2) = 2(x_1-1)^2+3x_2^2$, which is found at $(x_1, x_2) = (1, 0)$. We can use the function `gradient_descent` or `quasi_newtons_method` depending on the method we want. `quasi_newtons_method` can solve for the minimum using one of the specified Quasi-Newton's methods.

```python
def f(x):
    return 2 * ((x[0] - 1) ** 2) + 3 * (x[1] ** 2)
```

```python
#Returns two elements in a tuple. First element corresponds to position of minimum, the second the number of iterations
#it took to converge
#Gradient descent
optimize.gradient_descent(f, [11, 41])[0]
```

    array([1.00000000e+000, 7.78609779e-268])

# Background

The following mathematical concepts and background are required for understanding automatic differentiation:

### 1. Differential calculus

Differential calculus is a subfield of calculus concerned with the study of the rates at which quantities change.
For example, given the function:
\begin{align}
 f\left(x\right) &=  {x^{2}}
 \end{align}

 Increment x by h:
 \begin{align}
 f\left(x+h\right) &=  {(x+h)^{2}}
 \end{align}

 Apply the finite difference approximation to calculate the slope:
  \begin{align}
 \frac{f\left(x+h\right) - f\left(x\right) }{h}
 \end{align}

Simplify the equation:
  \begin{align}
 &= \frac{x^{2}+2xh+h^{2}-x^{2} }{h}\\
 &= \frac{2xh+h^{2}}{h}\\
 &=2x+h
 \end{align}

 Set $h\rightarrow 0$:
   \begin{align}
 2x +0 &= 2x
  \end{align}

The derivative is then defined is:
\begin{align}
 \lim_{h\to0} \frac{f\left(x+h\right) - f\left(x\right) }{h}
 \end{align}

### 2. Elementary functions and their derivatives

|       Function $f(x)$                |       Derivative $f^{\prime}(x)$                |
| :-------------------:  | :------------------------------------------------------------------------------:  |
| ${c}$           | $0$         |
| ${x}$           | $1$         |
| ${x^{n}}$           | ${nx^{n-1}}$         |
| $\frac{1}{x}$     | $\frac{-1}{x^{2}}$     |
| $ln{x}$     | $\frac{1}{x}$     |
| $\sin(x)$           |   $\cos(x)$         |
| $\cos(x)$           |   $-\sin(x)$         |
| $\tan(x)$           |   $\dfrac{1}{\cos^2(x)}$         |
| $\exp(x)$           |   $\exp(x)$         |
| ${a^{x}}$           |   ${a^{x}\ln{a}}$         |

### 3. The chain rule$^{(1)}$

For a function $h(u(t))$, the derivative of $h$ with respect to $t$ can be expressed as:
$$\dfrac{\partial h}{\partial t} = \dfrac{\partial h}{\partial u}\dfrac{\partial u}{\partial t}.$$
If the function is expressed as a combination of multiple variables that are expressed in terms of t, i.e. $h(u(t), v(t))$, the the derivative of $h$ with respect to $t$ can be expressed as:
$$\frac{\partial h}{\partial t} = \frac{\partial h}{\partial u}\frac{\partial u}{\partial t} + \frac{\partial h}{\partial v}\frac{\partial v}{\partial t}$$

Note that we are only looking at scalar variables in this case, but this idea can be extended to vector variables as well.

  For any $h\left(y\left(x\right)\right)$ where $y\in\mathbb{R}^{n}$ and $x\in\mathbb{R}^{m}$,

  \begin{align}
    \nabla_{x}h = \sum_{i=1}^{n}{\frac{\partial h}{\partial y_{i}}\nabla y_{i}\left(x\right)}.
  \end{align}

### 4. The graph structure of calculations and forward accumulation

Forward accumulation is computing the derivative using the chain rule starting from the inner most derivative to the outer most derivative, where we assume the most basic variables have seed values. Using a graph helps visualize forward accumulation. For example,

\begin{align}
 f\left(x,y\right) &= \frac{x}{y} +cos(x)sin(y)\\
 x &= y = 1
\end{align}

| Trace | Elementary Function | Current Value | Elementary Function Derivative |     $\nabla_{x}$ Value          |     $\nabla_{y}$ Value
| :---: | :-----------------: | :-----------: | :----------------------------: | :-----------------: | :-----------------: | :-----------------: |
| $w_{1}$ | $1$ | $1$ | $\dot{w_1}$ | $1$ | $0$ |
| $w_{2}$ | $1$ | $1$ | $\dot{w_2}$ | $0$ | $1$ |
| $w_{3}$ | $cos{(w_1})$ | $cos{(1)}$ | $-sin{(w_1)}\dot{w_1}$ | $-sin(1)$ | $0$ |
| $w_{4}$ | $sin{(w_2})$ | $sin{(1)}$ | $cos{(w_2)}\dot{w_2}$ | $0$ | $cos{(1)}$ |
| $w_{5}$ | $w_3\dot w_4$ | $sin{(1)}cos{(1)}$ | $w_4\dot{w_3} + w_3\dot{w_4}$ | $-sin^2{(1)}$ | $cos^2{(1)}$ |
| $w_{6}$ | $w_1 / w_2$ | $1$ | $\dot{w_1}/w_2 - w_1 \dot{w_2}/ w_2^2$ | $1$ | $-1$ |
| $w_{7}$ | $w_5 + w_6$ | $sin{(1)}cos{(1)} + 1$ | $\dot{w_5} + \dot{w_6}$ | $-sin^2{(1)} + 1$ | $cos^2{(1)}-1$ |

### 5. Optimization

Often times, we want to optimize a convex function, which is a function where a line segment between any two points on the graph of the function lies above or on the graph. A differentiable function is at a critical point when the derivative/gradient is equal to $0$. For convex functions, there is a minimum at the critical point. Thus, we are interested in finding when the derivative/gradient of a function is equal to $0$.

It is important to understand that many optimization/root-finding methods are iterative in nature; in iteration $k$ the position is changed using step:

$x_{k+1} = x_{k} + \Delta x_k$.

In root-finding methods, we can solve for $\Delta x_k$ using the following equation:

$J_f\Delta x_k  = -f(x_k)$

where $J_f$ is the Jacobian, the matrix of first order derivatives, of $f(x)$ at iteration $k$.

Similarly, in most optimization methods, we can solve for $\Delta x$ using the following equation:

$B_k\Delta x_k = -\alpha \nabla f(x_k)$

where $B$ is the Hessian, the matrix of second order derivatives, of $f(x)$ at iteration $k$, $\alpha$ is found through backtracking line search, and $\nabla f(x)$ is the gradient of $f(x)$

In small cases, solving for the inverse of $J$ and $H = B^{-1}$ would not be a problem. However, as the dimension gets larger and larger, solving for the inverse becomes very computationally intensive. As such, one must find other ways of solving for $\Delta x$. One could use methods such as LU decomposition or the Generalized minimal residual method (Gmres) to solve for it.

For optimization, one could also use Quasi-Newton methods to estimate the inverse of $B$. At each iteration, the inverse of the Hessian can be approximated using an update equation. For example, for DFP, to estimate $H_k$, the inverse of $B_k$,

$
H_k = H_k + \frac{\Delta x_k \Delta x_k^T}{\Delta x_k^T y_k} - \frac{H_k y_k y_k^T H_k}{y_k^T H_k y_k}
$

where $y_k = \nabla f(x_{k+1}) -  \nabla f(x_{k})$

More update equations can be found on Wikipedia's Quasi-Newton method page (3).

# Software Organization

The directory structure is:

 ```
 autodiff\
          autodiff\
                    __init__.py
                    functions.py
                    scalar.py
                    optimize.py
                    vector.py
          tests\
                    test_composite.py
                    test_functions.py
                    test_scalar.py
                    test_optimize.py
                    test_vector.py
          docs\
                    milestone1.ipynb
                    milestone2.ipynb
                    Final_milestone.ipynb
          README.md
          requirements.txt
          setup.cfg
          LICENSE

 ```

**Basic modules and what they do?**

This module aims to compute forward automatic differentiation. The module we implemented can efficiently compute the derivatives of a function with automatic differentiation of a scalar input. The *scalar.py* file contains the objects which compute the value and the derivative of scalar variables. The dunder methods add, sub, mul, truediv, pow, iadd, isub, imul, idiv, ipow are implemented in this module. The *function.py* file contains sine, cosine, power, exponential, and other mathematical functions. Both *scalar.py* and *function.py* return the value and the derivative.

The *optimize.py* file contains functions for find the roots and minima of user-defined functions.

**Where do tests live? How are they run? How are they integrated?**

We are using both `TravisCI` and `Coveralls` to test our module. Each test function exists in the tests folder, and it utilizes the `pytest` package.

**How can someone install your package?**

Our package has been released in `PyPI` under the name `autodiff-py`. Therefore, our auto-differentiating software can be downloaded at https://github.com/cs207FinalProjectGroup/cs207-FinalProject.git.

# Implementation Details

## *Scalar* Class

This package relies on the *Scalar* class, which represents scalar variables. To initialize a *Scalar* class object, the user will pass in a string that represents the variable name (i.e. 'x', 'y', 'x1', etc.) and the value of the variable. The user also has the option to specify the starting value for the derivative; if a starting value for the derivative is not specified, then it defaults to $1$. A *Scalar* object holds two attributes: 1) the value of the variable `_val` at the current step and 2) a dictionary `_deriv` containing the derivative or partial derivatives (keys will be the names of the variables (i.e *x* and *y*) and the values will be the derivative value with respect to each variable). The user should utilize the methods `getValue()` and `getDeriv()` to obtain the value and derivative(s) of their Scalar object.

```python
import autodiff as ad

x = ad.Scalar('x',2);
print('Value of x: ', x.getValue()); #should be 2.0
print('Derivative of x:', x.getDeriv()['x']); #should be 1.0

y = ad.Scalar('y', val=5, deriv=3.2);
print('Value of y: ', y.getValue()); #should be 5.0
print('Derivative of y: ', y.getDeriv()['y']); #should be 3.2
```

    Value of x:  2.0
    Derivative of x: 1.0
    Value of y:  5.0
    Derivative of y:  3.2

Storing the partial derivatives with respect to each variable allows us to easily compute additional derivatives with respect to each variable when we are performing mathematical operations because we can update each partial derivative individually. When a *Scalar* object is initialized, by default `_deriv` will just be a dictionary with the only key being the string the user passes in with value 1. A user can access the value of a *Scalar* object using the *getValue()* method and access the derivative (or partial derivatives) for the object through the *getDeriv()* method. The user can also get the derivatives/partial derivatives as a numpy array with the *getGradient()* method, which takes in a list of strings, with each element representing the variable to take the derivative with respect to, as an argument.

The dunder methods __add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __itruediv__, __ipow__ (and the right equivalents for the ones that have one) have been overwritten so that they return a new *Scalar* object with an updated value and derivatives. Thus, the adding or substracting two *Scalars* or raising a *Scalar* to the power of another *Scalar* does not change the values or derivatives of the original *Scalar* objects. By overwriting these methods, we are implementing forward accumulation, as the orders of operation allows us to traverse the chainrule starting from the inside.

```python
#addition example
x,y = ad.Scalar('x', 2),ad.Scalar('y', 5);
val = x+y;
print(val.getValue()); #should be 7.0
print(val.getDeriv()['x']); #should be 1.0
print(val.getDeriv()['y']); #should be 1.0
print();
print(x.getValue(),',', y.getValue()); #x, y should retain original values
```

    7.0
    1.0
    1.0

    2.0 , 5.0

```python
#pow example
x=ad.Scalar('x', 2);
val = x**2;
print(val.getValue()) #should be 4.0
print(val.getDeriv()['x']) #should be 4.0
```

    4.0
    4.0

```python
#division example
x,y=ad.Scalar('x', 3),ad.Scalar('y', 2);
val = x/y;
print(val.getValue()) # should be 1.5
print(val.getDeriv()['x']); #should be 0.5
print(val.getDeriv()['y']); #should be -0.75
print();
print(x.getValue(),',', y.getValue()); #x, y should retain original values
```

    1.5
    0.5
    -0.75

    3.0 , 2.0

We have also implemented the functions **sin**, **cos**, **tan**, **arcsin**, **arccos**, **arctan**, **log**, **ln**, **power**, and **exp**. All of these functions can take in an `int`, `float`, or *Scalar* object. If only an `int`/`float` is provided, then a `float` is returned, but if *Scalar* object is provided, then a new *Scalar* object. No changes are made to the value or derivatives of the original *Scalar* object passed in. Within these functions, the *numpy* functions *sin*, *cos*, *tan*, *arcsin*, *arccos*, *arctan*, *log*, *ln* and *exp* are used to calculate the appropriate values.

The trigonometric functions  **sin(x)**, **cos(x)**, and **tan(x)** each take in an `int`, `float`, or *Scalar* object *x* and apply the respective trigonometric function to the *x*. If a *x* is a *Scalar* object, the trigonometric function is applied to the *x._val* attribute and the derivative(s) is updated accordingly.

```python
import autodiff as ad
import numpy as np

# Sine functions
x = ad.Scalar('x', 2.0) # float input
val = ad.sin(x)
print(val.getValue()); #shoud be 0.90929742682
print(val.getDeriv()['x']); #should be -0.41614683654

# Cosine functions
x = ad.Scalar('x', 0) # integer input
val = ad.cos(x)
print(val.getValue()); #should be 1.0
print(val.getDeriv()['x']); #should be 0.0

# tan function
x = ad.Scalar('x', 2.0); # float input
val = ad.tan(x);
print(val.getValue()); #shoudl be -2.185039863261519
print(val.getDeriv()['x']); #should be 5.7743992040419174
```

    0.9092974268256817
    -0.4161468365471424
    1.0
    -0.0
    -2.185039863261519
    5.774399204041917

The **power(x1, x2)** function raises *x1* to the power of *x2*. *x1* and *x2* can be any combination of `ints`, `floats`, or *Scalar* objects. If only ints and floats are provided, then **power** will return a `float` with value *x1* raised to the power of *x2*. If at least one *Scalar* object is provided, then **power** works just like using the **__pow__** operator and returns a new *Scalar* object without changing any values in the original *Scalar* object(s).

```python
#power example
x = 5.0
p = 3.0
print (ad.power(x, p)) # should be 125.0
```

    125.0

The **sqrt(x)** function raises *x* to the power of 0.5. *x1* can be a an `int`, `float`, or *Scalar* objects. If only an `int` or `float` is provided, then **sqrt** will return a `float` with value *x* raised to the power of 0.5. If a *Scalar* object is provided, then **sqrt** works just like using the **__pow__** operator and returns a new *Scalar* object without changing any values in the original *Scalar* object(s).

```python
# sqrt example

b = ad.Scalar('b', 4)
b_sqrt = ad.sqrt(b)
b = ad.Scalar('b', 4)
b_sqrt = ad.sqrt(b)
print (b_sqrt.getValue()); #should be 2.0
print (b_sqrt.getDeriv()['b']) #should be 0.25
```

    2.0
    0.25

The **exp(x)** function raises *e* to the power of *x* , where *x* can be an `int`, `float`, or *Scalar* object. If *x* is an `int` or `float`, then a `float` with value equal to *e* raised to the power of *x* is returned. If *x* is a Scalar object, then a new *Scalar* is returned with an updated value and derivative(s).

```python
#exponential example
x = 1.0
print (ad.exp(x)) # should be 2.718281828459045
```

    2.718281828459045

The **log(x, base)** function give the log of *x*, where *x* can be an int, float, or Scalar object. This function takes in an `int`, `float`, or *Scalar* object for *x* and an `int` or `float` for *base*. It applies log with base *base* to the to the value of *x* and updates the derivative appropriately. If the argument is an `int` or `float`, then the function returns a `float`. If the argument is a *Scalar* object, the function returns a new *Scalar* object with the updated value and derivative. The **ln(x)** function returns the log with base $e$ of *x* and behaves identically to  **log(x, base)**.

```python
# log functions

x = ad.log(ad.Scalar('x', 10), 10)
print (x.getValue()) # shoud be 1.0
print (x.getDeriv()['x']) # should be 0.043429448190325175

x = ad.ln(ad.Scalar('x', np.e))
print (x.getValue()) # should be 1.0
print (x.getDeriv()['x']) # should be 0.36787944117144233
```

    1.0
    0.043429448190325175
    1.0
    0.36787944117144233

The **logistic(x)** function takes in an `int`, `float`, or *Scalar* object and applies the logistic function to the value. If the argument is an `int` or `float`, then the function returns a `float`. If the argument is a *Scalar* object, the function returns a new *Scalar* object with the updated value and derivative.

```python
# logistic example

x = ad.Scalar('x', 8)
y = ad.logistic(x)
print (y.getValue())
print(y.getDeriv()['x'])

x = ad.Scalar('x', 0)
y = ad.logistic(x)
print(y.getValue())
print(y.getDeriv()['x'])
```

    0.9996646498695336
    0.0003352376707564743
    0.5
    0.25

We have also implemented the functions **sinh**, **cosh**, and **tanh**. All functions work with `np.sinh`,
`np.cosh`, `np.tanh` to calculate the appropriate values.

The hyperbolic functions **sinh(x)**, **cosh(x)**, and **tanh(x)** each take in an in `int`, `float`, or *Scalar* object *x* and apply the resprective hyperbolic functions to the *x*. If a *x* is a *Scalar* object, the hyperbolic function is applied to the *x._val* attribute and the derivative(s) is updated accordingly.

```python
# Sinh functions
x = ad.Scalar('x', 2.0) # float input
val = ad.sinh(x)
print(val.getValue());  # shoud be 3.6268604078470186
print(val.getDeriv()['x']); # should be 3.7621956910836314

# Cosh functions
x = ad.Scalar('x', 0) # integer input
val = ad.cosh(x)
print(val.getValue()); # should be 1.0
print(val.getDeriv()['x']); # should be 0.0

# tanh function
x = ad.Scalar('x', 2.0); # float input
val = ad.tanh(x);
print(val.getValue()); # shoudl be 0.9640275800758169
print(val.getDeriv()['x']); # should be 0.07065082485316443
```

    3.6268604078470186
    3.7621956910836314
    1.0
    0.0
    0.9640275800758169
    0.07065082485316443

The __eq__ and __ne__ dunder methods have been implemented to compare *Scalar* objects.
__eq__ checks if two *Scalar* objects have equivalent values for both the value and the derivative. __ne__ checks if two *Scalar* objects are not equal.

```python
# __eq__, __ne__ example

x = ad.Scalar('x', 1)
x2 = ad.Scalar('x', 1)
print (x == x2) # should be true
print (not (x != x2)) # should be true

x = ad.Scalar('x', 1)
x2 = ad.Scalar('x', 2.0)
print (x != x2) # should be true
print (not (x == x2)) # should be true
```

    True
    True
    True
    True

## *Vectors*

There is not an individual class for vectors/arrays. Users can create a vector using the `create_vector` function, which takes in a name for the vector and a list/array of starting values for each *Scalar* object in the vector. This function creates a `numpy` array of *Scalar* objects that is the same length as the list/array of starting values that is passed in. Each *Scalar* object in the vector is given a name equivalent to the name of the vector concatenated with the index of the object in the vector plus one (e.g. if a list `[1,2,3]` is passed in for the starting values and the vector is name `'v'`, then the *Scalar* objects are name `'v1'`, `'v2'`, and `'v3'`).  A user can also specify the starting derivative value for each *Scalar* object in the vector/array through the `seed_vector` argument.

```python
v = ad.create_vector('v', values=[1, 2], seed_vector=[3.1, 4.2]);
print(v[0].getValue()) # should print 1.0
print(v[0].getDeriv()['v1']); #should print 3.1

print(v[1].getValue()) # should print 2.0
print(v[1].getDeriv()['v2']); #should print 4.2
```

    1.0
    3.1
    2.0
    4.2

A user can use the function `get_value(vector)` to get the values of all of the *Scalar* objects in the vector. The function returns a `numpy` array of the values.

```python
ad.get_value(v)
```

    array([1., 2.])

To obtain the derivatives (or partial derivatives) for each *Scalar* object in the vector, the user can use the `get_deriv(vector)` function. This will return a `numpy` array of dictionaries containing the derivatives (or partial derivatives) for each *Scalar* object.

```python
ad.get_deriv(v)
```

    array([{'v1': 3.1}, {'v2': 4.2}], dtype=object)

Since a vector is just a `numpy` array of *Scalar* objects, all of the dunder methods implemented for the *Scalar* class (__add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __idiv__, __ipow__, and the right equivalents) will be used when the same operations are done on the vector. Similar to `numpy` methods, the operations are conducted element-wise (i.e. in an addition operation between a vector of *Scalar* objects and a float, the float is added to each *Scalar* object). Each operation generates a new vector of *Scalar* objects with updated values and derivatives, which importantly leaves the vector to which the operation was applied unchanged.

```python
#demonstration for addition with vectors of Scalar objects
w = (v + [1,4]) * 2;

print('Values for v: ', ad.get_value(v));
print('Derivatives for v: ', ad.get_deriv(v));

print('Values for w: ', ad.get_value(w));
print('Derivatives for w: ', ad.get_deriv(w));
```

    Values for v:  [1. 2.]
    Derivatives for v:  [{'v1': 3.1} {'v2': 4.2}]
    Values for w:  [ 4. 12.]
    Derivatives for w:  [{'v1': 6.2} {'v2': 8.4}]

The mathematical functions **sin**, **arccos**, **sqrt**, **power**, **ln**, and others can also be applied to a vector of *Scalar* objects. Like the dunder methods for *Scalar* objects, these functions will be applied element-wise and generate a new vector of updated *Scalar* objects.

```python
z = ad.sin(v);

print('Values for v: ', ad.get_value(v));
print('Derivatives for v: ', ad.get_deriv(v));

print('Values for z: ', ad.get_value(z));
print('Derivatives for z: ', ad.get_deriv(z));
```

    Values for v:  [1. 2.]
    Derivatives for v:  [{'v1': 3.1} {'v2': 4.2}]
    Values for z:  [0.84147098 0.90929743]
    Derivatives for z:  [{'v1': 1.6749371481912334} {'v2': -1.7478167134979983}]

A user may also easily obtain the Jacobian of the vector using the `get_jacobian(vector, variables)` method. `variables` is a list of the variable names in the order in which the partial derivatives should be organized in the resulting Jacobian matrix.

```python
x = ad.create_vector('x', [1, 2, 3])
y = ad.create_vector('y', [5, 8, -7])
z = x - y

jacobian = ad.get_jacobian(z, ['x1', 'y2', 't'])
print(jacobian); #notice that derivative with respect to 't' is all zeroes
```

    [[ 1.  0.  0.]
     [ 0. -1.  0.]
     [ 0.  0.  0.]]

The vector of *Scalar* objects is important to the implementation of our optimization methods below.

## Optimization Functions

As shown from the use case in the **Usage** section, this package is intended for use without the need for any knowledge regarding auto-differentiation. For each of the optimization methods, the user simply must pass in a function and a list of floats specifying the initial guess position. The user-defined function should only take one argument: a list/array of the same length as the initial guess list.

All of the optimization methods in the package rely on the functions `create_vector`, `get_jacobian`, and `get_value` in order to get the necessary values for calculating the step in each iteration. All the optimization methods have been faithfully implemented according to their algorithms.

We have implemented gradient descent and the quasi-Newton methods *DFP*, *BFGS*, and *Broyden* to solve optimization problems. They are all iterative algorithms that rely on an update of the form $x_{k+1} = x_k + \Delta x_k$.
* For gradient descent, in each iteration the step is defined by $\Delta x_k = - stepsize * \nabla f(x_k)$. The user can change the step size, max number of iterations, and tolerance.
* For the quasi-Newton methods,
$\Delta x_k = -\alpha_k H_k \nabla f(x_k)$.
The only difference between *DFP*, *BFGS*, and *Broyden* is the update formula for the estimate $H_k$ of $H_f(x_k)^{-1}$, the inverse of the Hessian. In order to solve for $\alpha_k$ in each iteration, we also implemented a basic backtracking line search that follows the algorithm (4). The user can also specify the max number of iterations and tolerance for these quasi-Newton methods.

We have also implemented Newton's method for root-finding using four different methods for determining the step size at each iteration. The step size $\Delta x_k$ at iteration $k$ can be determined by solving the equation $J_f(x_k) \Delta x_k = -f(x_k)$ which is of the form $Ax=b$ with $A$ being a given matrix and $b$ a given vector. $\Delta x_k$ can be calculated using:
* the inverse of $A$ (specified as `'inverse'`)
* `np.linalg.solve` which first factorizes $A$ using LU decomposition, then solves for $x$ using forward and backward substitution (specified as `'exact'`)
* `scipy`'s generalized minimal residual method `gmres`, passing the $A$ matrix in directly (specified as `'gmres'`)
* `gmres` from `scipy` and passing in the action of $A$, i.e. a function $f$ such that $f(x) = Ax$ (specified as `'gmres_action'`)

## Additional implementation

We did not implement the operators **__lt__**, **__gt__**, **__le__**, and **__ge__** for comparing two *Scalar* objects. These operators would most likely be implemented in the next release of this package.

# Future plans

In the future , we would also like to calculate the Hessian while doing auto-differentiation. Since manually calculating the Hessian during optimization methods can be computationally expensive, using auto-differentiation to calculate the Hessian would be more efficient and feasible. We can then implement Newton's method for minimization, since that requires the Hessian.

# Citations
1. Sondak, David. “Automatic Differentiation: The Basics.” CS207-Lecture9. Cambridge, MA. 2 October 2018.

2. Hoffman, Philipp H.W. “A Hitchhiker’s Guide to Automatic Differentiation.” *Numerical Algorithms*, 72, 24 October 2015, 775-811, *Springer Link*, DOI 10.1007/s11075-015-0067-6.

3. "Quasi-Newton method." https://en.wikipedia.org/wiki/Quasi-Newton_method. Wikipedia. 10 December 2018.

4. "Backtracking line search." https://en.wikipedia.org/wiki/Backtracking_line_search. Wikipedia. 10 December 2018.


## Sample Jupyter / iPython notebook

Louis Moresi
School of Earth Sciences
University of Melbourne
[louis.moresi@unimelb.edu.au](mailto:louis.moresi@unimelb.edu.au)
[www.moresi.info](http://www.moresi.info)

### Docker containers

This notebook

### Quick links

 - [Home](/) - the static web pages on this site
 - [This Page](/notebooks/Content/Notebooks/StartHere.ipynb)
 - [Browse](/notebooks/Content/Notebooks/) the jupyter filesystem, create and edit notebooks

### What is this ?

This is an example of the iPython / Jupyter notebook system. They are a form of literate programming in which we can mix textbook instruction and explanations with code (in this case, python) that can also be run and edited. The text and mathematics in the notebooks requires a little preliminary learning.

The notebook system also includes a [file browser](/notebooks/Content/Notebooks/) which also allows you to add your own notebook, add a text file or start a terminal on the machine running this notebook.

NOTE that this content is ephemeral - it will disappear with the container if you do not capture the output by mounting the volume or copying the data to your local machine.

### Markdown

You can document your iPython notebooks by making some cells into **Markdown** cells. Markdown is a way of formatting text that is supposed to be almost as readable un-rendered as when it is tidied up. You might argue that it looks equally bad either way, but that's tough because the notebooks use it and that's how I want you to produce nice looking output to hand in as an assignment !

If you look at the **Markdown** cells as source code (by double-clicking on them) you will see how the raw text looks. To get back to the pretty version of the text, hit shift-enter.

### Maths

In a browser, you can render beautiful equations using a javascript tool called **Mathjax** which is build into the iPython notebooks.

You can build in symbols to your text such as $\pi$ and $\epsilon$ if you use the \$ signs to indicate where your equations begin and end, and you know enough $\LaTeX$ [try it here !](http://www.codecogs.com/latex/eqneditor.php) to get by.

Equations in 'display' mode are written like this (again look at the source for this cell to see what is used)

\\[ e^{i\pi} + 1 = 0 \\]

or even like this

\begin{equation}
%%
    \nabla^4 \psi = \frac{\partial T}{\partial x}
%%
\end{equation}

Go back to the rendered form of the cell by 'running' it.

### Links

[Markdown Website](http://daringfireball.net/projects/markdown/)
[Mathjax Website](http://docs.mathjax.org)
[Jupyter Notebooks](http://www.jupyter.org)

```python
## This is a live notebook where you can execute python code

print "Hello world"

```

    Hello world

```sh
%%sh

## This cell is now running shell (bash) commands

ls -l
echo "---"

whoami
echo "---"

uname -a
echo "---"

```

    total 16
    -rw-r--r--  1 lmoresi  staff  5020 27 Jan 11:52 StartHere.ipynb
    ---
    lmoresi
    ---
    Darwin MU00011496 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64
    ---

```python
# A blank canvas

## Feel free to run your own code in this cell (then hit shift-return to execute it)

```