```python
from sympy import *

# Implementation of QuaternionBase<Derived>::toRotationMatrix(void).
# The quaternion q is given as a list [qw, qx, qy, qz].
def QuaternionToRotationMatrix(q):
  tx  = 2 * q[1]
  ty  = 2 * q[2]
  tz  = 2 * q[3]
  twx = tx * q[0]
  twy = ty * q[0]
  twz = tz * q[0]
  txx = tx * q[1]
  txy = ty * q[1]
  txz = tz * q[1]
  tyy = ty * q[2]
  tyz = tz * q[2]
  tzz = tz * q[3]
  return Matrix([[1 - (tyy + tzz), txy - twz, txz + twy],
                 [txy + twz, 1 - (txx + tzz), tyz - twx],
                 [txz - twy, tyz + twx, 1 - (txx + tyy)]])

# Implementation of SO3Group<Scalar> expAndTheta().
# Only implementing the first case (of very small rotation) since we take the Jacobian at zero.
def SO3exp(omega):
  theta = omega.norm()
  theta_sq = theta**2

  half_theta = theta / 2

  theta_po4 = theta_sq * theta_sq
  imag_factor = Rational(1, 2) - Rational(1, 48) * theta_sq + Rational(1, 3840) * theta_po4;
  real_factor = 1 - Rational(1, 2) * theta_sq + Rational(1, 384) * theta_po4;

  # return SO3Group<Scalar>(Eigen::Quaternion<Scalar>(
  #     real_factor, imag_factor * omega.x(), imag_factor * omega.y(),
  #     imag_factor * omega.z()));
  qw = real_factor
  qx = imag_factor * omega[0]
  qy = imag_factor * omega[1]
  qz = imag_factor * omega[2]

  return QuaternionToRotationMatrix([qw, qx, qy, qz])

# Implementation of SE3Group<Scalar> exp().
# Only implementing the first case (of small rotation) since we take the Jacobian at zero.
def SE3exp(tangent):
  omega = Matrix(tangent[3:6])
  V = SO3exp(omega)
  rotation = V
  translation = V * Matrix(tangent[0:3])
  return rotation.row_join(translation)

# Main
init_printing(use_unicode=True)

# Define the tangent vector with symbolic elements T_0 to T_5.
# (For a matrix, use: Matrix(3, 1, lambda i,j:var('S_%d%d' % (i,j))) )
T = Matrix(6, 1, lambda i,j:var('T_%d' % (i)))

# Compute transformation matrix from tangent vector.
T_matrix = SE3exp(T)

# Define the vector current_T * src:
S = Matrix(3, 1, lambda i,j:var('S_%d' % (i)))

# Matrix-vector multiplication with homogeneous vector:
result = T_matrix * S.col_join(Matrix([1]))

# Compute Jacobian:
# (Note: The transpose is needed for stacking the matrix columns (instead of rows) into a vector.)
jac = result.transpose().reshape(result.rows * result.cols, 1).jacobian(T)

# Take Jacobian at zero:
jac_subs = jac.subs([(T[0], 0), (T[1], 0), (T[2], 0), (T[3], 0), (T[4], 0), (T[5], 0)])

# Simplify and output:
jac_subs_simple = simplify(jac_subs)
pprint(jac_subs_simple)
```

    ⎡1  0  0   0   S₂   -S₁⎤
    ⎢                      ⎥
    ⎢0  1  0  -S₂   0   S₀ ⎥
    ⎢                      ⎥
    ⎣0  0  1  S₁   -S₀   0 ⎦

```python
# Treat the function of which we want to determine the derivative as a list of nested functions.
# This makes it easier to compute the derivative of each part, simplify it, and concatenate the results
# using the chain rule.

### Define the function of which the Jacobian shall be taken ###

# Matrix-vector multiplication with homogeneous vector:
def MatrixVectorMultiplyHomogeneous(matrix, vector):
  return matrix * vector.col_join(Matrix([1]))

# Define the vector current_T * src:
S = Matrix(3, 1, lambda i,j:var('S_%d' % (i)))

# The list of nested functions. They will be evaluated from right to left
# (this is to match the way they would be written in math: f(g(x)).)
functions = [lambda matrix : MatrixVectorMultiplyHomogeneous(matrix, S), SE3exp]

### Define the variables wrt. to take the Jacobian, and the position for evaluation ###

# Chain rule:
# d(f(g(x))) / dx = (df/dy)(g(x)) * dg/dx

# Define the parameter with respect to take the Jacobian, y in the formula above:
parameters = Matrix(6, 1, lambda i,j:var('T_%d' % (i)))

# Set the position at which to take the Jacobian, g(x) in the formula above:
parameter_values = zeros(6, 1)

### Automatic Jacobian calculation, no need to modify anything beyond this point ###

# Jacobian from previous step, dg/dx in the formula above:
previous_jacobian = 1

# TODO: Test whether this works with non-matrix functions.
def ComputeValueAndJacobian(function, parameters, parameter_values):
  # Evaluate the function.
  values = function(parameter_values)
  # Compute the Jacobian.
  symbolic_values = function(parameters)
  symbolic_values_vector = symbolic_values.transpose().reshape(symbolic_values.rows * symbolic_values.cols, 1)
  parameters_vector = parameters.transpose().reshape(parameters.rows * parameters.cols, 1)
  jacobian = symbolic_values_vector.jacobian(parameters_vector)
  # Set in the evaluation point.
  for row in range(0, parameters.rows):
    for col in range(0, parameters.cols):
      jacobian = jacobian.subs(parameters[row, col], parameter_values[row, col])
  # Simplify the jacobian.
  jacobian = simplify(jacobian)
  return (values, jacobian)

# Print info about initial state.
print('Taking the Jacobian of these functions (sorted from inner to outer):')
for i in range(len(functions) - 1, -1, -1):
  print(str(functions[i]))
print('with respect to:')
pprint(parameters)
print('at position:')
pprint(parameter_values)
print('')

# Loop over all functions:
for i in range(len(functions) - 1, -1, -1):
  # Compute value and Jacobian of this function.
  (values, jacobian) = ComputeValueAndJacobian(functions[i], parameters, parameter_values)

  # Update parameter_values
  parameter_values = values
  # Update parameters (create a new symbolic vector of the same size as parameter_values)
  parameters = Matrix(values.rows, values.cols, lambda i,j:var('T_%d%d' % (i,j)))
  # Concatenate this Jacobian with the previous one according to the chain rule:
  previous_jacobian = jacobian * previous_jacobian

  # Print intermediate result
  print('Intermediate step ' + str(len(functions) - i) + ', for ' + str(functions[i]))
  print('Position after function evaluation (function value):')
  pprint(parameter_values)
  print('Jacobian of this function wrt. its input only:')
  pprint(jacobian)
  print('Cumulative Jacobian wrt. the innermost parameter:')
  pprint(previous_jacobian)
  print('')

# Print final result
print('Final result:')
pprint(previous_jacobian)
```

    Taking the Jacobian of these functions (sorted from inner to outer):
    <function SE3exp at 0x7fdab798a840>
    <function <lambda> at 0x7fdab7639950>
    with respect to:
    ⎡T₀⎤
    ⎢  ⎥
    ⎢T₁⎥
    ⎢  ⎥
    ⎢T₂⎥
    ⎢  ⎥
    ⎢T₃⎥
    ⎢  ⎥
    ⎢T₄⎥
    ⎢  ⎥
    ⎣T₅⎦
    at position:
    ⎡0⎤
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎣0⎦

    Intermediate step 1, for <function SE3exp at 0x7fdab798a840>
    Position after function evaluation (function value):
    ⎡1  0  0  0⎤
    ⎢          ⎥
    ⎢0  1  0  0⎥
    ⎢          ⎥
    ⎣0  0  1  0⎦
    Jacobian of this function wrt. its input only:
    ⎡0  0  0  0   0   0 ⎤
    ⎢                   ⎥
    ⎢0  0  0  0   0   1 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   -1  0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   0   -1⎥
    ⎢                   ⎥
    ⎢0  0  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  1   0   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   1   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  -1  0   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎢1  0  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎢0  1  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎣0  0  1  0   0   0 ⎦
    Cumulative Jacobian wrt. the innermost parameter:
    ⎡0  0  0  0   0   0 ⎤
    ⎢                   ⎥
    ⎢0  0  0  0   0   1 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   -1  0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   0   -1⎥
    ⎢                   ⎥
    ⎢0  0  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  1   0   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   1   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  -1  0   0 ⎥
    ⎢                   ⎥
    ⎢0  0  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎢1  0  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎢0  1  0  0   0   0 ⎥
    ⎢                   ⎥
    ⎣0  0  1  0   0   0 ⎦

    Intermediate step 2, for <function <lambda> at 0x7fdab7639950>
    Position after function evaluation (function value):
    ⎡S₀⎤
    ⎢  ⎥
    ⎢S₁⎥
    ⎢  ⎥
    ⎣S₂⎦
    Jacobian of this function wrt. its input only:
    ⎡S₀  0   0   S₁  0   0   S₂  0   0   1  0  0⎤
    ⎢                                           ⎥
    ⎢0   S₀  0   0   S₁  0   0   S₂  0   0  1  0⎥
    ⎢                                           ⎥
    ⎣0   0   S₀  0   0   S₁  0   0   S₂  0  0  1⎦
    Cumulative Jacobian wrt. the innermost parameter:
    ⎡1  0  0   0   S₂   -S₁⎤
    ⎢                      ⎥
    ⎢0  1  0  -S₂   0   S₀ ⎥
    ⎢                      ⎥
    ⎣0  0  1  S₁   -S₀   0 ⎦

    Final result:
    ⎡1  0  0   0   S₂   -S₁⎤
    ⎢                      ⎥
    ⎢0  1  0  -S₂   0   S₀ ⎥
    ⎢                      ⎥
    ⎣0  0  1  S₁   -S₀   0 ⎦

libvis/scripts/LMOptimizer SE3Optimization Test Jacobian derivation.ipynb


**Exercise set 4**
==============

>The goal of this exercise is to perform **least-squares regression** and to see how we can estimate errors in the parameters we find.

**Exercise 4.1**

In this exercise we will use least-squares regression to investigate a physical phenomenon: the decay of beer froth with time. The file [erdinger.txt](Data/erdinger.txt) (located at 'Data/erdinger.txt') contains [measured heights](https://doi.org/10.1088/0143-0807/23/1/304) for beer froth as a function of time, along with the errors in the measured heights.

**(a)**  Use least-squares regression to create a linear model that predicts the beer froth height as a function of time. Plot your linear model together with the raw data.

```python
# Your code here
```

**Your answer to question 4.1(a):** *Double click here*

**(b)**  Obtain the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination), $R^2$, for your model.

```python
# Your code here
```

**Your answer to question 4.1(b):** *Double click here*

**(c)**  It is reasonable to assume that the change in the volume of froth is proportional
to the volume present at a given time. One can show that this leads
to exponential decay,

\begin{equation}
h(t) = h(0) \exp \left(-\frac{t}{\tau} \right),
\label{eq:hard}
\tag{1}\end{equation}

where $h(t)$ is the height of the froth as a function of time $t$, and $\tau$ is a parameter.
In the following, consider $h(0)$ as an unknown parameter to be determined. Show
how you can transform the equation above to a linear equation of the form,

\begin{equation}
y = a + b x,
\tag{2}\end{equation}

and give the relation(s) between the variables $h, h(0), t, \tau$ and
$a, b, x, y$.

**Your answer to question 4.1(c):** *Double click here*

**(d)**  Using the transformation found above, create a new linear model, and estimate $h(0)$ and $\tau$.
Further, calculate the coefficient of determination for this case, and compare the two
linear models you have found so far.

```python
# Your code here
```

**Your answer to question 4.1(d):** *Double click here*

**(e)**  From the analytical model Eq. (1), $h(0)$ is a known constant, equal to the height of
the froth at time zero. Thus, we can reformulate our model and fit it to just obtain
one parameter, $b$. Essentially, we are defining $y^\prime = y - a$ and using the model,

\begin{equation}
y^\prime = y - a = b x,
\tag{3}\end{equation}

that is, we have a linear model *without* the constant term.
Show that the least-squares solution for $b$ when fitting $y^\prime = bx$ is given by,

\begin{equation}
b = \frac{
\sum_{i=1}^n y_i^\prime x_i
}{\sum_{i=1}^n x_i^2
},
\label{eq:bexpr}
\tag{4}\end{equation}

where $n$ is the number of measurements and $x_i$ and $y_i^\prime$ are the
measured values.

**Your answer to question 4.1(e):** *Double click here*

**(f)**  Do the fitting a final time, but use Eq. (4)
to obtain the parameter $b$.
Calculate the coefficient of determination and compare the three linear models you have found.

```python
# Your code here
```

**Your answer to question 4.1(f):** *Double click here*

**Exercise 4.2**

In this exercise, we will consider a linear model where we have one variable:

\begin{equation}
y = a + bx,
\end{equation}

and we have determined $a$ and $b$ using the least-squares equations. We further have
$n$ data points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$ where the $x_i$'s do not have
any uncertainty, while the uncertainty in the $y_i$'s are all equal to $\sigma_y$.

> Our goal here is to find expressions for estimating
> the errors in the parameters $a$ and $b$,
> given the error in our measurements of $y$.

**Background information: Propagation of errors**

To be able to estimate the errors in $a$ and $b$, we will use [propagation of errors](https://en.wikipedia.org/wiki/Propagation_of_uncertainty).
For simplicity, consider a function, $f$, of two variables $u$ and $v$: $f = f(u, v)$.
By doing a Taylor expansion about the average values, $\bar{u}$ and $\bar{v}$, we can
show that the uncertainty (or "error") in the function $f$, $\sigma_f$, due to the uncertainties in $u$
and $v$ ($\sigma_u$ and $\sigma_v$, respectively) is given by:

\begin{equation}
\sigma_f^2 = \left(\frac{\partial f}{\partial u} \right)^2 \sigma_u^2 +
\left(\frac{\partial f}{\partial v} \right)^2 \sigma_v^2 +
2 \frac{\partial f}{\partial u} \frac{\partial f}{\partial v} \sigma_{uv} + \text{higher-order terms},
\end{equation}

where $\sigma_{uv}$ is the *covariance* between $u$ and $v$. Typically, the errors are "small"
and this motivates us to neglect the higher-order terms. Further, we will assume that the
variables $u$ and $v$ are *not* correlated: $\sigma_{uv} = 0$. We then arrive at the
(perhaps well-known) approximate propagation-of-errors-expression for the uncertainty in $f$:

\begin{equation}
\sigma_f^2 \approx \left(\frac{\partial f}{\partial u} \right)^2 \sigma_u^2 +
\left(\frac{\partial f}{\partial v} \right)^2 \sigma_v^2 .
\end{equation}

This can be generalized to $k$ variables, say $f=f(z_1, z_2, \ldots, z_k)$. The approximate
expression for the uncertainty in $f$, $\sigma_f$, due to the uncertainties
in the $z_i$'s, $\sigma_{z_{i}}$, is then:

\begin{equation}
\sigma_f^2 \approx \sum_{i=1}^{k} \left(\frac{\partial f}{\partial z_{i}} \right)^2 \sigma_{z_{i}}^2 .
\label{eq:errorp}
\tag{5}\end{equation}

We will use this expression to estimate the uncertainties in $a$ and $b$.

**Deriving expressions for the uncertainties in $a$ and $b$**

**(a)**  Show that the error in the $b$ parameter, $\sigma_b$,
is given by the following expression:

\begin{equation}
\sigma_b^2 = \frac{\sigma_y^2}{\sum_{i=1}^n \left(x_i - \bar{x}\right)^2},
\end{equation}

where $\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i$ is the average of $x$.

***Hint:*** Use the least-squares expression for $b$:

\begin{equation}
b = \frac{
\sum_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})
}{
\sum_{i=1}^n (x_i - \bar{x})^2
},
\end{equation}

together with the propagation-of-errors expression (Eq. (5)), and consider $b$ as a
function of the $y_i$'s: $b = f(y_1, y_2, \ldots, y_n)$. You might find it helpful to determine
$\frac{\partial b}{\partial y_j}$
as an intermediate step in your derivation.

**Your answer to question 4.2(a):** *Double click here*

**(b)**  Show that the error in the $a$ parameter, $\sigma_a$, is given by the following expression:

\begin{equation}
\sigma_a^2 = \frac{\sigma_y^2}{n} \times
\frac{
\sum_{i=1}^{n} x_i^2
}{
\sum_{i=1}^{n} (x_i - \bar{x})^2
} .
\end{equation}

***Hint:*** Use the least-squares expression for $a$:

\begin{equation}
a = \bar{y} - b \bar{x},
\end{equation}

together with the propagation-of-errors expression (Eq. (5)), and consider $a$ as a
function of the $y_i$'s *and* $b$: $a = f(y_1, y_2, \ldots, y_n,b)$. You might find it
helpful to determine
$\frac{\partial a}{\partial y_j}$ and $\frac{\partial a}{\partial b}$ as intermediate steps
in your derivation.

**Your answer to question 4.2(b):** *Double click here*


```python
# N-dim arrays
# four classes are available: dense/sparse + mutable/immutable
from sympy import *
#from sympy import Array
init_printing(use_unicode=True)
```

```python
A = Array([[1,2],[3,4]])
```

```python
A
```

```python
B = Array(range(12), (3,4))
```

```python
B
```

```python
C = Array(range(12), (2,2,3))
```

```python
C
```

```python
# rank of the Array
A.rank()
```

```python
B.rank()
```

```python
C.rank()
```

```python
A.tomatrix()
```

```python
A.tolist()
```

```python
B.tomatrix()
```

```python
C.tolist()
```

```python
# no default functionality for converting higher-order tensors into matrices
C.tomatrix()
```

```python
# tensor products: higher-order tensors from lower-order tensors

# unit vectors
##############

i = Array([1,0,0])
j = Array([0,1,0])
k = Array([0,0,1])
```

```python
ii = tensorproduct(i,i)
```

```python
ii
```

```python
ij = tensorproduct(i,j)
```

```python
ij

```

```python
# similarly
# ii, ij, ik
# ji, jj, jk
# ki, kj, kk

```

```python
i.rank()
```

```python
ii.rank()
```

```python
I2 = Array([[1,0,0], [0,1,0], [0,0,1]])
```

```python
I2
```

```python
# fourth-order tensor
# I4_1_ijkl = delta_ij delta_kl
I4_1 = tensorproduct(I2, I2)
```

```python
I4_1
```

```python
Matrix(I4_1)
```

```python
Matrix(I4_1).reshape(9,9)
```

```python
# I4_2_ijkl = delta_ik delta_jl
I4_2 = eye(9)
```

```python
I4_2
```

```python
# tensor contraction
A = Array([[1,2,3],[4,5,6],[7,8,9]])
```

```python
A
```

```python
# the matrix trace is equivalent to contaction of second-order tensor along axes 1 and 2
# since the starting index in Python is zero.
trA = tensorcontraction(A, (0,1))
```

```python
trA
```

```python
# matrix product is the contaction of fourth order tensor along axes 2 and 3
# of the fourth-order tensor formed as the tensor product of the matrices as rank-2 tensor
D = Array([[1,2],[3,4]])
```

```python
DtD = tensorproduct(D,D)
```

```python
DtD
```

```python
DtD.rank()
```

```python
tensorcontraction(DtD,(1,2))
```

```python
D.tomatrix()*D.tomatrix()
```

```python
#
```

```python
#
```

```python
# indexed objects
# A[i,j]
# A is the IndexedBase
# i and j are indices

A = IndexedBase('A')
i,j = symbols('i j', cls=Idx)
```

```python
A[i,j]
```

```python
A[i,j].shape
```

```python
i = Idx('i', 3)
j = Idx('j', 3)
```

```python
A[i,j].shape
```

```python
A[i,j].ranges
```

```python
i.lower
```

```python
i.upper
```

```python
# index with unbounded upper limit
k = Idx('k', oo)
```

```python
k.lower
```

```python
k.upper
```

```python
#
```

```python
#
```

```python
# Matrix expressions
from sympy import MatrixSymbol, Matrix
```

```python
F = MatrixSymbol('F',3,3)
```

```python
F.shape
```

```python
F
```

```python
Matrix(F)
```

```python
A = MatrixSymbol('A',3,3)
B = MatrixSymbol('B',3,3)
C = MatrixSymbol('C',3,3)
```

```python
Matrix(C)
```

```python
# MatrixAddition
Matrix(MatAdd(A,B,C))
```

```python
MatMul(A,B,C)
```

```python
Matrix(MatMul(A,B,C))
```

```python
Matrix(hadamard_product(A,B))
```

```python
# matrix inverse
Ainv = Inverse(A)
```

```python
Ainv
```

```python
IA = Trace(A)
```

```python
IA
```

```python
# tensor calculs

i = Idx('i', 3)
j = Idx('j', 3)
k = Idx('k', 3)
l = Idx('l', 3)

```

```python
A=MatrixSymbol('A',3,3)
```

```python
A
```

```python
print(diff(F[i,j],F[i,j]))
```

    1

```python
print(diff(F[k,l],F[i,j]))
```

    KroneckerDelta(i, k)*KroneckerDelta(j, l)

```python
from sympy.concrete.delta import _simplify_delta

_simplify_delta(diff(F[k,l],F[i,j]))
```

```python
C = F.T*F
```

```python
print(diff(C[i,j],F[k,l]))
_simplify_delta(diff(C[i,j],F[k,l]))
```

```python
simplify(diff(C[i,j],F[k,l]))
```

```python
Determinant(A)
```

```python
LeviCivita(i,j,k)
```

```python
Finv=Inverse(F)
```

```python
diff(Determinant(F),F[k,l])
```

```python

```


# AST 341 homework 2

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.6.2 (Python 3.8.5-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.6.2/

## 1.

a. We want to compute the pressure for a gas that obeys the Maxwell-Boltzmann distribution:

\begin{equation}
4\pi p^2 n(p) dp = \frac{n_I}{(2\pi m_I kT)^{3/2}} e^{-p^2/(2m_I kT)} 4 \pi p^2 dp
\end{equation}

Our pressure integral is:
\begin{equation}
P = \frac{1}{3} \int n(p) p v d^3 p = \frac{1}{3} \int_0^{\infty} 4\pi p^2 n(p) p v dp
\end{equation}
and for a non-relativistic gas, we can take $v = p/m_I$

Define the symbols we will use—make sure we tell SymPy which are positive and real, since we have a square root in our functions

```python
nI, m, k, T = symbols("n_I m_I k T", positive=True, real=True)
p = symbols("p")
```

Define the distribution function, $n(p)$

```python
n = nI/(2*pi*m*k*T)**Rational(3,2) * exp(-p**2/(2*m*k*T))
n
```

Now do the integral.  Note we indicate $\infty$ as `oo`

```python
P = Rational(1,3) * integrate(4*pi*p**2 * n * p * (p / m), (p, 0, oo))
P
```

Notice that we get the ideal gas law!
\begin{equation}
P = n_I k T
\end{equation}

b. Now we want to compute the energy

\begin{equation}
e = \frac{1}{\rho} \int_0^\infty 4\pi p^2 \mathcal{E}(p) n(p) dp
\end{equation}
where the kinetic energy for a single particle is just
\begin{equation}
\mathcal{E}(p) = \frac{p^2}{2 m_I}
\end{equation}
for a non-relativistic gas

```python
rho = symbols("rho", positive=True)
```

```python
e = rho**-1 * integrate(4*pi*p**2 * p**2 / (2*m) * n, (p, 0, oo))
e
```

So
\begin{equation}
\rho e = \frac{3}{2} n_I k T
\end{equation}

This shows us that
\begin{equation}
\rho e = \frac{3}{2} P
\end{equation}

```python

```


# Lesson 3 (Circular) Convolution

## 1D Convolution

A circulant matrix $C$ is a square matrix that's completely determined by the first column $c$. Other columns are just shifts of the first columns.

For example, the circulant matrix generated by $c=\begin{bmatrix}1\\2\\0\end{bmatrix}$ is $\qquad C=circ(c)=\begin{bmatrix}1&0&2\\2&1&0\\0&2&1\end{bmatrix}$.

```python
import numpy as np
from scipy.linalg import circulant
```

```python
circulant([1,2,0])
```

    array([[1, 0, 2],
           [2, 1, 0],
           [0, 2, 1]])

Let $c,x\in\mathcal{C}^n$. By definition, the convolution between $c$ and $x$ is
$$c*x(i)=\sum_{j=0}^{n-1}c(i-j)x(j)=\begin{bmatrix}c_i&c_{i-1}&\cdots&c_{i-(n-1)}\end{bmatrix}\begin{bmatrix}x_0\\x_1\\\vdots\\x_{n-1}\end{bmatrix},$$
which means that
$$c*x=\begin{bmatrix}c_0&c_{0-1}&\cdots&c_{0-(n-1)}\\
c_1&c_{1-1}&\cdots&c_{1-(n-1)}\\
\vdots&\vdots&\ddots&\vdots\\
c_{n-1}&c_{n-1-1}&\cdots&c_{n-1-(n-1)}\end{bmatrix}\begin{bmatrix}x_0\\x_1\\\vdots\\x_{n-1}\end{bmatrix}=circ(c)x.$$

Let $F_n$ be the $n\times n$ unitary discrete Fourier transform, as defined in Lesson 2. One fast way to compute convolution is to use FFT. It can be proven that
\begin{equation}F_n(c)\odot F_n(x)=F_n(c*x),\end{equation}
where $\odot$ is the coordinate-wise vector product, that is $[x\odot y](i)=x(i)y(i)$.

Let $diag(x)$ be the diagonal matrix whose diagonal is $x$, then the above equation can also be written as
$$diag(F_nc)F_nx=F_ncirc(c)x\Longleftrightarrow circ(c)=F_n^*diag(F_nc)F_n\Longleftrightarrow F_n circ(c)F_n^*=diag(F_nc).$$

The code below verifies that both ways of computing convolution give the same output.

```python
x = np.random.randn(3);print(x)
```

    [-0.57732409  0.33621261  2.05435539]

```python
circulant([1,2,0])@x
```

    array([ 3.5313867 , -0.81843557,  2.72678061])

```python
np.fft.ifft(np.fft.fft([1,2,0])*np.fft.fft(x))
```

    array([ 3.5313867 +0.j, -0.81843557+0.j,  2.72678061+0.j])

## to be continued

```python

```


```python
import sympy as sp
import pandas as pd
%matplotlib inline
```

```python
x, y = sp.symbols('x y')
f = sp.tan(y - 0.8 * x) + 0.8 * x * y - 0.3
g = x ** 2 + y ** 2 - 1.7
```

```python
plot = sp.plot_implicit(sp.Eq(f, 0), show=False, line_color='blue')
plot.extend(sp.plot_implicit(sp.Eq(g, 0), show=False, line_color='green'))
plot.show()
```

```python
def newton_system(x, y, f, g, x0, y0):
    dfdx = f.diff(x)
    dfdy = f.diff(y)
    dgdx = g.diff(x)
    dgdy = g.diff(y)

    d = dfdx * dgdy - dgdx * dfdy
    dx = f * dgdy - g * dfdy
    dy = dfdx * g - dgdx * f

    xs, ys, norms, fs, gs = [], [], [], [], []
    xk, yk = x0, y0
    while True:
        new_x = xk - (dx / d).evalf(subs={x: xk, y: yk})
        new_y = yk - (dy / d).evalf(subs={x: xk, y: yk})
        xs.append(new_x)
        ys.append(new_y)

        norm = sp.sqrt((new_x - xk) ** 2 + (new_y - yk) ** 2)
        norms.append(norm)
        fs.append(f.evalf(subs={x: new_x, y: new_y}))
        gs.append(g.evalf(subs={x: new_x, y: new_y}))

        xk, yk = new_x, new_y

        if norm < 1e-5 and len(xs) > 1:
            break
    return pd.DataFrame(list(zip(xs, ys, norms, fs, gs)), columns=['x', 'y', 'norm', 'f', 'g'])
```

```python
newton_system(x, y, f, g, 1, 1)
```

      x
      y
      norm
      f
      g

      0
      1.22761708889715
      0.622382911102848
      0.440912922225209
      -0.0648343503263468
      0.194404204985173

      1
      1.14149633521898
      0.636074171385597
      0.0872022638595296
      -0.00358082004103558
      0.00760423482222702

      2
      1.13755666993897
      0.637166805074043
      0.00408837509233626
      -8.94822418305518e-6
      1.67148108958352e-5

      3
      1.13754769473498
      0.637169712315441
      9.43431710133745e-6
      -5.15817799583199e-11
      8.90062924372405e-11

```python
newton_system(x, y, f, g, -1, -1)
```

      x
      y
      norm
      f
      g

      0
      -0.822059767940987
      -1.02794023205901
      0.180120467334300
      -0.0121769613176794
      0.0324433827527268

      1
      -0.824116762561558
      -1.01051444859597
      0.0175467705339115
      -0.000187332632752991
      0.000307889156169836

      2
      -0.824171430058251
      -1.01031752219007
      0.000204373541676120
      -3.26758676647519e-8
      4.17685447475136e-8

      3
      -0.824171440931074
      -1.01031749264951
      3.14779734395557e-8
      -7.59514473637850e-16
      8.46188933633038e-16


# Exercise 2: Markov Chains and Markov Decision Processes (MDP)

This exercise deals with the formal handling of Markov chains and Markov decision processes.

## 1) Markov Chain: State Transition
The graph shows the last beer problem.
The nodes show the states.
The arrows define the possible transitions to other states and the numbers besides the arrows define the propability of the corresponding transition.
If you are for example in the state "Inital Beer", with 30% propability you go to have a pizza, with 60% propability you meet friends and with 10% propability you end up sleeping.

Define the state transition probability matrix $\mathcal{P}_{xx'}$ of the graph shown in the figure below!

With $p_k = \begin{bmatrix}
\text{Pr}_k \lbrace \text{Inital Beer} \rbrace \\
\text{Pr}_k \lbrace \text{Meet Friends} \rbrace \\
\text{Pr}_k \lbrace \text{Pizza} \rbrace \\
\text{Pr}_k \lbrace \text{Another Beer} \rbrace \\
\text{Pr}_k \lbrace \text{"Last Beer"}\rbrace \\
\text{Pr}_k \lbrace \text{Sleep} \rbrace \\
\end{bmatrix}^\text{T}$

YOUR ANSWER HERE

## 2 ) Markov Chain: Stationary State
Using $p = p \mathcal{P}$, calculate the stationary state probability.

Please note that the sum of the state propabilities equals one for any specific point in time.

YOUR ANSWER HERE

## 3) Markov Reward Process: Evaluating States

In the following rewards for every state are defined.

Given the reward distribution $r_\mathcal{X}$, calculate the state-values $v_\mathcal{X}$.

The states are defined by:
$\mathcal{X} = \left\lbrace \begin{matrix}
\text{Inital Beer}\\
\text{Meet Friends}\\
\text{Pizza}\\
\text{Another Beer}\\
\text{"Last Beer"}\\
\text{Sleep}\\
\end{matrix}
\right\rbrace$

The rewards are defined by:
$r_\mathcal{X} = \begin{bmatrix}
+1\\
+1\\
+2\\
+1\\
-3\\
0\\
\end{bmatrix}$

The state-value is defined by the state-value Bellman equation: $v_\mathcal{X} = r_\mathcal{X} + \gamma \mathcal{P}_{xx'} v_\mathcal{X}$. Assume that $\gamma = 0.9$ and write a Python program to calculate $v_\mathcal{X}$. Which state is most promising? Why?

Which state is most promising when $\gamma = 0.1$?

YOUR ANSWER HERE

```python
import numpy as np

# define given parameters
gamma = 0.1 # discount factor

# YOUR CODE HERE
raise NotImplementedError()

print(v_X)

```

## 4) Markov Decision Process: State Transition

The graph shows an MDP.
The nodes are the states.
In every state you can choose between two actions (Lazy or Productive).
Taken actions impact the state transition probability to the next state.
If you for example have a "Hangover" and decide to be "Productive", there is a 30% chance for you to "Visit Lecture" and a 70% chance to stay in the "Hangover" state.

Define the lazy state transition probabilitiy $\mathcal{P}_{xx'}^{u=\text{Lazy}}$ and the productive state transition probability $\mathcal{P}_{xx'}^{u=\text{Productive}}$ of the graph shown in the figure below.

With $p_k = \begin{bmatrix}
\text{Pr}_k \lbrace \text{Hangover} \rbrace \\
\text{Pr}_k \lbrace \text{Sleep} \rbrace \\
\text{Pr}_k \lbrace \text{More Sleep} \rbrace \\
\text{Pr}_k \lbrace \text{Visit Lecture} \rbrace \\
\text{Pr}_k \lbrace \text{Study}\rbrace \\
\text{Pr}_k \lbrace \text{Pass the Exam} \rbrace \\
\end{bmatrix}^\text{T}$

## 4) Solution

\begin{align}
\mathcal{P}_{xx'}^{u=\text{Lazy}}&=\begin{bmatrix}
0 & 1 & 0 & 0 & 0   & 0\\
0 & 0 & 1 & 0 & 0   & 0\\
0 & 0 & 1 & 0 & 0   & 0\\
0 & 0 & 0 & 0 & 0.8 & 0.2\\
0 & 0 & 1 & 0 & 0   & 0\\
0 & 0 & 0 & 0 & 0   & 1\\
\end{bmatrix}\\
\mathcal{P}_{xx'}^{u=\text{Productive}}&=\begin{bmatrix}
0.7 & 0 & 0   & 0.3 & 0   & 0\\
0   & 0 & 0.4 & 0.6 & 0   & 0\\
0   & 0 & 0.5 & 0   & 0.5 & 0\\
0   & 0 & 0   & 0   & 1   & 0\\
0   & 0 & 0   & 0   & 0.1 & 0.9\\
0   & 0 & 0   & 0   & 0   & 1\\
\end{bmatrix}
\end{align}

## 5) Markov Decision Process: Trivial Policy Evaluation

The rewards for this problem are defined by:
$r_\mathcal{X} = r_\mathcal{X}^{u=\text{Productive}} = r_\mathcal{X}^{u=\text{Lazy}} = \begin{bmatrix}
-1\\
-1\\
-1\\
-1\\
-1\\
0\\
\end{bmatrix}$.

How can we interprete these rewards?
Evaluate both the lazy policy and the productive policy using $\gamma = 0.9$.

Bonus question: Can we evaluate the state-value of $\lbrace x=\text{More Sleep}, u=\text{Lazy}\rbrace$ for an infinite time horizon without the use of the Bellman equation?

YOUR ANSWER HERE

```python
import numpy as np

# YOUR CODE HERE
raise NotImplementedError()
```

```python
# Bonus question: Can we evaluate the state-value of {𝑥=More Sleep,𝑢=Lazy} for an infinite time horizon without the use of the Bellman equation?

# YOUR CODE HERE
raise NotImplementedError()
```

## 6) Action-Value Function Evalution

Now, the policy is defined by:
\begin{align}
\pi(u_k=\text{Productive} | x_k)&=\alpha,\\
\pi(u_k=\text{Lazy} | x_k)&=1-\alpha, \forall x_k \in \mathcal{X}
\end{align}

Calculate action-values for the problem as described using the 'fifty-fifty' policy ($\alpha = 0.5$) according to the Bellman Expectation Equation: $q_\pi(x_k, u_k) = \mathcal{R}^u_x + \gamma \sum_{x_{k+1} \in \mathcal{X}} p^u_{xx'} v_\pi(x_{k+1})$ $\forall x_k, u_k \in \mathcal{X}, \mathcal{U}$.

## 6) Solution

```python
import numpy as np

gamma = 0.9
alpha = 0.5
no_states = 6
no_actions = 2
r_X = np.array([-1, -1, -1, -1, -1, 0]).reshape(-1, 1)
q_XU = np.zeros([no_states, no_actions])
# YOUR CODE HERE
raise NotImplementedError()
```

## 7) Markov Decision Problem: Stochastic Policy Evalution

Plot the state-value of the states "Lecture" and "Study" for different $\alpha$. What do we see? Why?

## 7) Solution

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-talk')

n = 6 # dimension of state space
no_of_samples = 1000

alphas = np.linspace(0, 1, no_of_samples)
v_n_alpha = np.zeros([n, no_of_samples])

# YOUR CODE HERE
raise NotImplementedError()
```

```python
plt.figure(figsize=[10, 6])
states = ["Hangover", "Sleep", "More Sleep", "Visit Lecture", "Study", "Pass Exam"]
alphas = alphas.flatten()
for state, vnalp in zip(states, v_n_alpha):
    ls = '--' if state in ['Visit Lecture', 'Study'] else '-'
    plt.plot(alphas, vnalp, ls=ls, label=r"$x=${}".format(state))

plt.legend()
plt.xlabel(r"$\alpha$")
plt.ylabel(r"$v_\pi(x)$")
plt.xlim([0, 1])
plt.ylim([-10, 0])
```

YOUR ANSWER HERE


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.stats import norm
```

#Distribucion normal teórica

\begin{align}
P(X) = \frac{1}{\sigma \sqrt{2\pi}^{\left[ -\frac{1}{2}\left(\frac{X - 𝜇}{\sigma}\right)^2\right]}}
\end{align}

```python
def gaussian(x, mu, sigma):
  return 1/(sigma * np.sqrt(2*np.pi))*np.exp(-0.5 * pow((x-mu)/sigma, 2))
```

```python
#Lista de datos de entrada para la funcion gaussiana
x = np.arange(-4, 4, 0.1)
y = gaussian(x, 0.0, 1.0)
plt.plot(x, y)

```

#Usando scipy

```python
dist = norm(0, 1) #promedio, desviacion estandar
x = np.arange(-4, 4, 0.1)
y = [dist.pdf(value) for value in x]
plt.plot(x,y)
```

```python
dist = norm(0, 1)
x = np.arange(-4, 4, 0.1)
y = [dist.cdf(value) for value in x]
plt.plot(x, y)
```

#Distribución normal (gaussiana) a partir de los datos

Archivo [excel](https://seattlecentral.edu/qelp/sets/057/057.html)

```python
df = pd.read_excel('s057.xls')
arr = df['Normally Distributed Housefly Wing Lengths'].values[4:]
#frecuencia de los datos del array
values, dist = np.unique(arr, return_counts=True)
plt.bar(values, dist)
```

```python
#Estimacion parametrica de una distribucion
mu = arr.mean()
sigma = arr.std()
x = np.arange(30, 60, 0.1)
dist = norm(mu, sigma)
y = [dist.pdf(value) for value in x]
plt.plot(x,y)
values, dist = np.unique(arr, return_counts = True)
plt.bar(values, dist/len(arr))
```


# Non-uniform structured grid

In fluid mechanics and heat transfer, the solution often contains regions of large gradients and regions of small gradients. In cases where both regions are well localized, computational efficiency favors clustering computational nodes in regions of large gradients and mapping regions of small gradients with coarse meshes. This notebook described a common method to define a non-uniform grid for a symmetrical problem, with the finest resolution at the two ends of the domain, e.g. channel flow.

## Grid transformation

Consider a domain $[-H/2,+H/2]$ discretized with $N$ points. The location of computational nodes on a uniform grid is defined as:

$$
\tilde{y}_j = -h + j\Delta \text{ with }\Delta = \frac{2h}{N-1}\text{ and }j\in[0,N-1]
$$
where $h=H/2$.

The common approach to create a non-uniform grid is to operate a transform function over the uniform grid. For a channel flow, one such function is:

$$
y_j = h\frac{\tanh\gamma \tilde{y}_j}{\tanh\gamma h}
$$

The coefficient $\gamma$ controls the stretching of the grid, in this case the minimum mesh size, which is at the wall.

### Python set-up and useful functions

```python
%matplotlib inline
# plots graphs within the notebook

from IPython.display import display,Image, Latex
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')
from IPython.display import clear_output

import time

import numericaltools as numtools

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc
import h5py

import sympy as sym

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

font = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 16,
        }
fontlabel = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 16,
        }

from matplotlib.ticker import FormatStrFormatter
plt.rc('font', **font)

```

## Example

Create a grid for $H=2$, $N=33$, and $\Delta_{min}=0.001$

```python
H = 2
N = 33
Deltamin = 0.0001
gamma_guess = 2
y,gamma = numtools.stretched_mesh(H,N,Deltamin,gamma_guess)
print("Stretching coefficient gamma: %1.4e" %gamma)
plt.plot(y,'o')
plt.show()
plt.plot(0.5*(y[1:]+y[:-1]),y[1:]-y[:-1],'o')
plt.show()

```

## Grid generation at fixed $\gamma$

$H=2$, $N=257$, $\gamma = 2.6$

```python
H = 2.
h = H/2
N =257
gamma = 2.6
y_uni = np.linspace(-h,h,N)
y = h*np.tanh(gamma*y_uni)/np.tanh(gamma*h)
plt.plot(y,'o')
plt.show()
plt.plot(0.5*(y[1:]+y[:-1]),y[1:]-y[:-1],'o')
plt.show()
print("Deltamin = %1.4e" %(y[1]-y[0]))
```

## Generate a bunch of grids with different $N$ but constant $\Delta_{min}$

```python
H = 2
deltamin = 5e-4
gamma_guess = 2.
N_array = np.array([33,65,129,257,513,1025],dtype=int)
gamma_array = np.zeros(len(N_array))
j = 0
for N in N_array:
    y,gamma_array[j] = numtools.stretched_mesh(H,N,Deltamin,gamma_guess)
    gamma_guess = gamma_array[j]
    print("for N=%4i, gamma=%1.4f" %(N,gamma_array[j]))

    j += 1

print(gamma_array)

```

    for N=  33, gamma=4.8624
    for N=  65, gamma=4.3730
    for N= 129, gamma=3.9348
    for N= 257, gamma=3.5145
    for N= 513, gamma=3.0970
    for N=1025, gamma=2.6734
    [4.86238469 4.37303735 3.93484222 3.51452064 3.09698058 2.67343703]

## Generate a bunch of grids at constant $N$ and varying $\Delta_{min}$

```python
Deltamin_array = np.array([1e-2,5e-3,1e-3,5e-4,1e-4])
H = 2
N = 129
gamma_guess = 2
for Deltamin in Deltamin_array:
    y,gamma = numtools.stretched_mesh(H,N,Deltamin,gamma_guess)
    print("For Dmin=%1.1e, gamma=%1.4f" %(Deltamin,gamma))
```

    For Dmin=1.0e-02, gamma=0.8639
    For Dmin=5.0e-03, gamma=1.4658
    For Dmin=1.0e-03, gamma=2.5568
    For Dmin=5.0e-04, gamma=2.9842
    For Dmin=1.0e-04, gamma=3.9348

```python
2/129
```

$$0.015503875968992248$$

```python

```


```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt

```

```python
np.random.seed(1)
```

# Problem setup:

We're going to have a small number of (X, Y) data points.
We will see how the model misfit changes as we increase the number of parameters.
The type of model we'll use is a sum of sines and cosines of increasing frequency. The model parameters are the amplitudes of these sinusoids.
That is, given the m data points $(x_1, y_1) ,\ldots, (x_m, y_m)$ , we'll fit

\begin{align}
y_i = a_0 + a_1 \cos(x_i) + a_2 \cos(2x_i) + \ldots a_n \cos(nx_i) + b_1 \sin(x_i) + \ldots + b_n \sin(nx_i)
\end{align}

But before we work with this, we'll first show what happens with Numpy's `Polynomial.fit` function, which is often used to show the danger of high complexity model fitting.

# Make the truth data: all zeros, but there's one noise point at 1

For illustration purposes, the data generation process will just be all zeros. But we'll add noise to one single point to move it from 0 to 1.

An omniscient model would ignore the noise point and predict all zeros... but it will try to fit through that noise point at y=1.

```python
X_MAX = 2 * np.pi
def make_data(m):
    # Make Xs scattered on interval [0, 2pi]
    X = X_MAX * np.random.rand(m,)
    # Y's are all zero (1e-10 for plotting purposes)
    Y = 1e-9 + np.zeros((m,))
    # ...except for one noise point
    Y[m//2] = 1
    return X, Y

```

# Traditional model-fitting explanation: Use fewer parameters than data (n << m)

```python
from numpy.polynomial import Polynomial
m = 9
X, Y = make_data(m)

n = 4

p = Polynomial.fit(X, Y, n)
xx, yy = p.linspace(domain=[np.min(X), np.max(X)])
plt.figure()
plt.plot(xx, yy, label=f"poly fit {n = }")
plt.plot(X, Y, 'rx', label='sample points')
plt.legend()
```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x1187b2b80>

Here a polynomial won't be look great, but it's relatively stable in the region around the data.

However, the fit becomes unstable as the order increases up to the number of data points, and the errors go crazy goes crazy.

```python
n = 9
p = Polynomial.fit(X, Y, n)
xx, yy = p.linspace(domain=[np.min(X), np.max(X)])
plt.figure()
plt.plot(xx, yy, label=f"poly fit {n = }")
plt.plot(X, Y, 'rx', label='sample points')
plt.ylim(-10, 10)
plt.legend()
```

    /Users/scott/opt/anaconda3/envs/mapping/lib/python3.8/site-packages/numpy/polynomial/polynomial.py:1350: RankWarning: The fit may be poorly conditioned
      return pu._fit(polyvander, x, y, deg, rcond, full, w)

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x1188022e0>

The points are all interpolated here, but the fit inbetween is terrible. This is generally shown to be the reason for keeping model complexity low compared to the size of the data.

# Back to the sine/cosine model:

Now to show what happens as we increase the number of model parameters above and beyond the number of data points, we'll use the Fourier model:

\begin{align}
y_i = a_0 + a_1 \cos(x_i) + a_2 \cos(2x_i) + \ldots a_n \cos(nx_i) + b_1 \sin(x_i) + \ldots + b_n \sin(nx_i)
\end{align}

(Note that we're using this sine/cosine model to match [1], and because the `Polynomial.fit` fails for when the `deg`, polynomial order, exceeds `m`, the number of points).

Below are the functions for the making $\mathbf{A}$ matrix for fitting the system $\mathbf{y = Ax}$, along with `fit()` and `predict()` functions for the model:

```python
def form_A(X, order):
    # N is the highest order sin/cos, so num terms is n = 2order + 1
    # X has m data points
    m = X.shape[0]
    A = np.zeros((m, 2*order + 1))
    A[:, 0] = 1
    for k in range(1, order + 1):
        A[:, k] = np.cos(X * k)
        A[:, k + order] = np.sin(X * k)
    return A

def predict(coeffs, xs):
    """Take in the coefficients of the sin/cosine terms from the fitting,
    along with the x locations, and output the model predicted ys

    The `coeffs` vector are the a_i and b_i cos/sin coefficients"""
    order = (len(coeffs) - 1) // 2
    m = len(xs)
    idxs = np.arange(1, order+1).reshape((1, order))
    a0 = coeffs[0]
    cos_terms = np.sum(coeffs[1:(order+1)] * np.cos(xs.reshape((-1, 1)) * idxs), axis=1)
    sin_terms = np.sum(coeffs[order+1:] * np.sin(xs.reshape((-1, 1)) * idxs), axis=1)
    ys = a0 + cos_terms + sin_terms
    return ys

def calculate_avg_misfit(y_hat, y_truth = 0):
    # The real data should be all 0s (except that one noisy point), so a 0 line would be perfect
    return np.sum(np.abs(y_hat - y_truth)) / len(y_hat)

def fit(X, Y, order, x_test=None, x_max=X_MAX, print_misfit=True):
    """Fit the sin/cos model, return the [a_i, b_i] coefficients"""
    A = form_A(X, order)

    m = len(X)
    n = 2*order + 1
    assert A.shape == (m, n)
    # If overdetermined system (m > n, more data points (rows) than features (columns)),
    # this would be a least squares fitting of the model.
    # coeffs = np.linalg.lstsq(A, Y, rcond=None)[0]
    # For fat A (m < n), we'll use the pseudoinverse to fit
    coeffs = np.linalg.pinv(A) @ Y
    if x_test is None:
        x_test = np.linspace(0, X_MAX, 500)
    y_hat = predict(coeffs, x_test)

    misfit = calculate_avg_misfit(y_hat)
    if print_misfit:
        print(f"Average misfit: {misfit}")
    return x_test, y_hat, misfit

```

## Case: m > n, more data points than model parameters

The model does it's best to fit all points, but there are fewer parameters than data points, so we're doing a least-squares regression

```python
# m = Number of data points
# n = Number of parameters to fit
# m = 9
# X, Y = make_data(m)

order = 3
# n = 2*order + 1

x_test, y_hat, misfit = fit(X, Y, order)

plt.figure()
plt.plot(x_test, y_hat, 'b.-', label='predicted')
plt.plot(X, Y, 'rx', label='sample points')
plt.title("Predicted y vs x (linear scale)")
plt.legend()
```

    Average misfit: 1.9714393403195682

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x118838e80>

For a more "energy-like" view, we'll plot in a log scale. Any peaks above 0 can be considered "noise energy", as the truth would be all 0s.

```python
plt.figure()
plt.semilogy(x_test, y_hat, 'b.-', label='predicted')
plt.semilogy(X, Y, 'rx', label='sample points')
plt.title("Predicted y vs x (log-scale)")
plt.legend()
```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x118888610>

The model has `order` number of large humps, where large, badly-fitting spikes occur (where the humps might wrap around the `2 pi` point, since the sines/cosines are periodic there).

# Overfitting: m == n
The number of sin and cosine terms will now be equal to the number of data points to fit.

This is the **worst-case scenario** here, where we can interpolate the data (match the training points exactly), but in between those, the model goes wildly off the rails (just like with the polynomail fit).

```python
order = m // 2
n = 2*order + 1
assert n == m

x_test, y_hat, misfit = fit(X, Y, order)

plt.figure()
plt.semilogy(x_test, y_hat, 'b.-')
plt.semilogy(X, Y, 'rx')
plt.show()
```

    Average misfit: 20.355497441416627

    <IPython.core.display.Javascript object>

# Overfitting? Try n > m
Let's use twice as many parameters as data points

```python
order = m
# n = 2*order + 1

x_test, y_hat, misfit = fit(X, Y, order)

plt.figure()
plt.semilogy(x_test, y_hat, 'b.-')
plt.semilogy(X, Y, 'rx')
plt.show()
```

    Average misfit: 0.15200455330631488

    <IPython.core.display.Javascript object>

Now there are still badly fitting spikes, but there peaks start to all go down...
The "noise energy" is getting spread across higher frequencies, and the average misfit to the truth goes down.

# Harmless overfitting? n = 10m

Way more parameters: now use 10 times the number of parameters as data points

```python
order = 10*m

x_test, y_hat, misfit = fit(X, Y, order)

plt.figure()
plt.semilogy(x_test, y_hat, 'b.-')
plt.semilogy(X, Y, 'rx')
plt.show()

plt.figure()
plt.plot(x_test, y_hat, 'b.-', label='predicted')
plt.plot(X, Y, 'rx', label='sample points')
plt.title("Predicted y vs x (linear scale)")
plt.legend()
```

    Average misfit: 0.018012970909079603

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x11898b7c0>

# Show the misfit as a function of number of sin/cos terms

```python
orders = list(range(0, m)) + list(np.logspace(np.log10(m), np.log10(8*m), 10).astype(int))
ns = 2*np.array(orders) + 1
# print(ns)

misfits = []
for order in orders:
    x_test, y_hat, misfit = fit(X, Y, order, print_misfit=False)
    misfits.append(misfit)

```

```python
interp_thresh = m
plt.figure();
plt.semilogy(ns, misfits, '.-')
plt.semilogy(np.ones(5) * interp_thresh, np.linspace(0, np.max(misfits), 5), 'k-', label='interpolation threshold')
plt.ylabel("Average error")
plt.xlabel("Number of sin/cosine terms used to fit")
plt.legend()
```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x1189fb670>

### References
[1] Muthukumar, Vidya, et al. "Harmless interpolation of noisy data in regression." IEEE Journal on Selected Areas in Information Theory (2020).

The final figure is replicating one case from Figure 2: https://arxiv.org/pdf/1903.09139.pdf

# Random extra test: Fitting with a wide neural network

TODO: check similar "average error" graph as above with a NN of increasing width

```python
import torch
import torch.nn as nn
import torch.nn.functional as F

```

```python

nwide = 1000
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(1, nwide)
        self.fc2 = nn.Linear(nwide, nwide)
        self.fc3 = nn.Linear(nwide, 1)

    def forward(self, x):
#         x = F.relu(self.fc1(x))
#         x = F.relu(self.fc2(x))

#         x = self.fc1(x)
#         x = self.fc2(x)
#         x = torch.tanh(self.fc1(x))
#         x = torch.tanh(self.fc2(x))
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
print(net)

```

    Net(
      (fc1): Linear(in_features=1, out_features=1000, bias=True)
      (fc2): Linear(in_features=1000, out_features=1000, bias=True)
      (fc3): Linear(in_features=1000, out_features=1, bias=True)
    )

```python
import torch.optim as optim

inp = torch.Tensor(X.reshape(-1, 1))
target = torch.Tensor(Y).view(-1, 1)

criterion = nn.MSELoss()

# create your optimizer
optimizer = optim.SGD(net.parameters(), lr=0.01)

num_epochs = 5000
for epoch in range(num_epochs):
    # in your training loop:
    optimizer.zero_grad()   # zero the gradient buffers
    output = net(inp)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()    # Does the update
    if epoch % 500 == 0:
        print(f"{epoch = }, {loss.item() = }")

```

    epoch = 0, loss.item() = 0.15103742480278015
    epoch = 500, loss.item() = 0.05079585686326027
    epoch = 1000, loss.item() = 0.028149651363492012
    epoch = 1500, loss.item() = 0.008730195462703705
    epoch = 2000, loss.item() = 0.003026040503755212
    epoch = 2500, loss.item() = 0.000692488276399672
    epoch = 3000, loss.item() = 0.00022584182443097234
    epoch = 3500, loss.item() = 9.963271440938115e-05
    epoch = 4000, loss.item() = 4.930593786411919e-05
    epoch = 4500, loss.item() = 2.507205499568954e-05

```python
plt.figure()
x_test = np.linspace(0, X_MAX, 500).reshape(-1, 1)
y_test = net(torch.Tensor(x_test)).detach().numpy()

# xx, yy = inp.detach().numpy(), output.detach().numpy()
plt.plot(x_test, y_test, 'b.-')

plt.semilogy(x_test, y_test, 'b.-')
plt.semilogy(X, Y, 'rx')

plt.figure()
plt.plot(x_test, y_test, 'b.-')
plt.plot(X, Y, 'rx')
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x125158520>]


# Four ways to look at matrix multiplication

If $A$ is $m\times n$ and $B$ is $n\times p$, then $C=AB$ is an $m\times p$ matrix. We may regard as a definition the formula
\begin{equation}
C_{ij} = \sum_{k=1}^n A_{ik}B_{kj}.
\end{equation}

```julia
using LinearAlgebra
```

```julia
m,n,p = 4,5,3;
A = rand(-4:4,m,n);
B = rand(-4:4,n,p);
display(A), display(B);
```

    4×5 Array{Float64,2}:
     1.0  0.0  4.0  1.0  0.0
     3.0  3.0  4.0  3.0  2.0
     1.0  1.0  1.0  3.0  4.0
     4.0  3.0  4.0  1.0  0.0

    5×3 Array{Float64,2}:
     1.0  -1.0  -1.0
     0.0  -0.0   1.0
     0.0  -1.0   2.0
     1.0   1.0   1.0
     1.0   2.0   2.0

```julia
C = A*B
```

    4×3 Array{Float64,2}:
     2.0  -4.0   8.0
     8.0   0.0  15.0
     8.0   9.0  13.0
     5.0  -7.0   8.0

Here is a literal interpretation of the summation definition for the matrix product. Notice how in Julia, there are "implicit for" loops (aka generators) that can be enclosed in parentheses to generate a list for a command, or in brackets to generate a matrix or vector.

```julia
C_0 = [ sum(A[i,k]*B[k,j] for k=1:n) for i=1:m, j=1:p ]
```

    4×3 Array{Float64,2}:
     2.0  -4.0   8.0
     8.0   0.0  15.0
     8.0   9.0  13.0
     5.0  -7.0   8.0

## Inner products

If the matrices are real, then we can interpret each sum as the inner product between vectors that are of length $n$. In Julia we can use `dot` for the inner product, or the LaTeX symbol $\cdot$, which is entered as `\cdot` followed by the Tab key.

```julia
C_1 = [ A[i,:]⋅B[:,j] for i=1:m, j=1:p ]
```

    4×3 Array{Float64,2}:
     2.0  -4.0   8.0
     8.0   0.0  15.0
     8.0   9.0  13.0
     5.0  -7.0   8.0

Note that `A[i,:]` and `B[:,j]` extract one row and one column, respectively. In Julia, each result will be of type `Vector`, and the shape distinction is not preserved, as every vector is simply one-dimensional. (This is unlike MATLAB, where even vectors are regarded as having two dimensions, one with size 1.)

```julia
A[2,:]
```

    5-element Array{Float64,1}:
     3.0
     3.0
     4.0
     3.0
     2.0

```julia
size(ans)
```

    (5,)

## Linear combinations of columns

If we express $B$ columnwise, then the matrix product $AB$ can also be expressed columnwise, as
\begin{equation}
AB = \begin{bmatrix} A b_1 & A b_2 & \cdots & A b_p \end{bmatrix}.
\end{equation}

```julia
b1 = B[:,1]
```

    5-element Array{Float64,1}:
     1.0
     0.0
     0.0
     1.0
     1.0

```julia
display(C[:,1]), display(A*b1);
```

    4-element Array{Float64,1}:
     2.0
     8.0
     8.0
     5.0

    4-element Array{Float64,1}:
     2.0
     8.0
     8.0
     5.0

Furthermore, $A$ times a compatible vector is a linear combination of the columns of $A$:
$$
Av = v_1 a_1 + \cdots + v_n a_n.
$$

```julia
display(A*b1), display( sum(b1[k]*A[:,k] for k=1:n) );
```

    4-element Array{Float64,1}:
     2.0
     8.0
     8.0
     5.0

    4-element Array{Float64,1}:
     2.0
     8.0
     8.0
     5.0

Putting this all together, the full interpretation of $C=AB$ is

```julia
C_2 = hcat( ( sum(B[:,j][k]*A[:,k] for k=1:n) for j=1:p )... )
```

    4×3 Array{Float64,2}:
     2.0  -4.0   8.0
     8.0   0.0  15.0
     8.0   9.0  13.0
     5.0  -7.0   8.0

Of course, there is no reason to go through all that in practice.

## Linear combinations of rows

This is the dual of the previous version:

$$
AB = \begin{bmatrix} a_1^T B \\ \vdots \\ a_m^T B \end{bmatrix}
$$

We put the transposes in because we want to have all named vectors be column vectors. Thus, each row of $A$ has to have a transpose on it. Note also that transposing a Julia vector will create a "row vector":

```julia
a3T = A[3,:]'
```

    1×5 Adjoint{Float64,Array{Float64,1}}:
     1.0  1.0  1.0  3.0  4.0

Thus, in the third row of the product:

```julia
display(C[3,:]), display(a3T*B);
```

    3-element Array{Float64,1}:
      8.0
      9.0
     13.0

    1×3 Adjoint{Float64,Array{Float64,1}}:
     8.0  9.0  13.0

These are the same vector, although the second version thinks of it as having row shape. Furthermore, each such vector-matrix product is a linear combination of the rows of $B$,

```julia
display(a3T*B), display( sum(a3T[k]*B[k,:] for k=1:n) );
```

    1×3 Adjoint{Float64,Array{Float64,1}}:
     8.0  9.0  13.0

    3-element Array{Float64,1}:
      8.0
      9.0
     13.0

Finally, doing this for all the rows of $A$, we have another identity for the product $AB$.

```julia
C_3 = vcat( ( sum(A[i,:][k]*B[k,:]' for k=1:n) for i=1:m )... )
```

    4×3 Array{Float64,2}:
     2.0  -4.0   8.0
     8.0   0.0  15.0
     8.0   9.0  13.0
     5.0  -7.0   8.0

## Outer products

This form might be the most surprising, and it leads to some interesting perspectives. The outer product between two vectors is the matrix formed by all possible products of pairs of elements from them.

```julia
outer(u,v) = [ u[i]v[j] for i=1:length(u), j=1:length(v) ];
```

```julia
outer(A[:,2],B[4,:])
```

    4×3 Array{Float64,2}:
     0.0  0.0  0.0
     3.0  3.0  3.0
     1.0  1.0  1.0
     3.0  3.0  3.0

The outer product is consistent with the definition of the matrix product $uv^T$. Note this is a *column* times a *row*, which is the reverse of the inner product, nor do the vectors even need to have the same length.

```julia
A[:,2]*B[4,:]'
```

    4×3 Array{Float64,2}:
     0.0  0.0  0.0
     3.0  3.0  3.0
     1.0  1.0  1.0
     3.0  3.0  3.0

Because each column of $uv^T$ is a multiple of $u$ (or equivalently, each row is a multiple of $v^T$), any nonzero outer product has rank 1.

An interesting identity is that a general matrix product can be written as a sum of outer products:
$$
AB = \sum_{k=1}^n a_k b_k^T,
$$
where we are writing $A$ by its columns and $B$ by its rows.

```julia
C_4 = sum( A[:,k]*B[k,:]' for k=1:n )
```

    4×3 Array{Float64,2}:
     2.0  -4.0   8.0
     8.0   0.0  15.0
     8.0   9.0  13.0
     5.0  -7.0   8.0


# Simulating The Spread of Heat Through a Material
> Written by David Mascharka and Ryan Soklaski

## Understanding the Heat Propagation
In this problem, we will learn about a simple algorithm for numerically simulating the spread of heat through a material. We will want to use vectorization to write an efficient algorithm.

Imagine that we have a rectangular piece of steel. For now, let's treat this piece of steel as a 5x5 grid - we are only able to measure the average temperature of each of these 25 grid regions. Let's assume that steel starts off at a uniform 0-degrees. Thus, our temperature readout for each of its grid positions is:

```
                                0     0     0     0     0
                                0     0     0     0     0
                                0     0     0     0     0
                                0     0     0     0     0
                                0     0     0     0     0
```

Now, we will clamp hot contacts, which are always at a constant 100-degrees, along the outer edges of the steel. Upon clamping these contacts, our temperature readout at time-0 becomes:

```
                               100   100   100   100   100
                               100    0     0     0    100
                               100    0     0     0    100
                               100    0     0     0    100
                               100   100   100   100   100
```

We will adopt the same indexing scheme as a 2D NumPy array. That is, element (i,j) of this grid is row-i, column-j in the grid. The top-left corner is located at (0, 0), and has a temperature of 100.

Moving forward, we want to describe, numerically, how the heat from the contacts will spread through the material as time carries on. The heat equation is a partial differential equation that describes the flow of heat through space and time. In the following equation, the function $u(x, y, t)$ describes how much heat resides at the location $(x, y)$ at time $t$:

\begin{equation}
\frac{\partial u}{\partial t} - \alpha \left(\frac{\partial^{2} u}{\partial x^{2}} + \frac{\partial^{2} u}{\partial y^{2}} \right)= 0
\end{equation}

Do not worry if you have no clue what a partial differential equation is! You do not need to know anything about the heat equation, we are simply providing some background here.

What this equation ultimately says is that heat will spread such that a point will take on the average amount of heat among its neighboring points. Numerically, we can write this out as:

\begin{equation}
u^{(t)}_{ij} = \frac{u^{(t-1)}_{i+1,j} + u^{(t-1)}_{i-1,j} + u^{(t-1)}_{i,j+1} + u^{(t-1)}_{i,j-1}}{4}
\end{equation}

That is, $u^{(t)}_{ij}$ is the heat at grid-location $(i, j)$ at time-step $t$. It's value is given by the average of the heat of all four of its neighboring grid positions from time-step $t-1$. See that the right side of the equation averages the heat from above, below, left-of, and right-of $(i, j)$, at time-step $t-1$. This means of evolving the heat through our gridded material is known as the *finite difference method*.

Let's using the finite difference method to figure out what the distribution of heat looks like throughout our steel square at time-step 1. Keep in mind that we need not update any of the outer-edges of the steel - those positions are held at a fixed heat. We'll start at the upper-left corner and get

\begin{equation}
u^{t=1}_{1,1} = \frac{0 + 100 + 0 + 100}{4} = 50\\
u^{t=1}_{1,2} = \frac{0 + 100 + 0 + 0}{4} = 25\\
\end{equation}

and so on, yielding the heat distribution at timestep-1 of:
```
                                100   100   100   100   100
                                100    50    25    50   100
                                100    25     0    25   100
                                100    50    25    50   100
                                100   100   100   100   100
```

Repeating this process again will produce the heat distribution at timestep-2, and so on. After many iterations, we see the entire region becomes 100 degrees. This is because the heat from the edges flows inward until everything is the same temperature. This stabilized distribution of heat is known as the *steady state*. If we change the boundary conditions, i.e. change what we clamp to the edges of our steel, we will observe different steady states.

## Problem 1: A Simple Implementation of Finite Differences
Write a Python function that takes in a 2-dimensional numpy-array containing heat-values, and uses the finite difference method to produce the heat distribution for that material at the next time-step. Do this using simple for-loops to iterate over the values of the array.

Assume that the boundary-values of the array are fixed, so you need not update them. However, do *not* assume that the boundary values are all the same as one another, as they were in the preceding example.

Also, be careful not to change the content of the array that your function is given. You need to use the values in that array, unchanged, to compute the new heat distribution. Consider making use of `np.copy` to create a copy of the input array (so that your new array will have the appropriate boundary values).

```python
# make sure to execute this cell so that your function is defined
# you must re-run this cell any time you make a change to this function
import numpy as np
def evolve_heat_slow(u):
    """ Given a 2D array of heat-values (at fixed boundary), produces
        the new heat distribution after one iteration of the finite
        difference method.

        Parameters
        ----------
        u : numpy.ndarray shape=(M, N)
            An MxN array of heat values at time-step t-1.
            (M and N are both at least 2)

        Returns
        -------
        numpy.ndarray, shape=(M, N)
            An MxN array of heat values at time-step t.
        """
    new = np.copy(u)
    size = np.size(u)
    rows = np.size(u,0)
    columns = np.size(u,1)
    for r in range(1, rows - 1):
        for c in range(1, columns - 1):
            top = u[r-1, c]
            left = u[r, c-1]
            right = u[r, c+1]
            bottom = u[r+1, c]
            new[r,c] = (top+left+right+bottom)/4
    return new

    # student code goes here
```

```python
from bwsi_grader.python.heat_dispersion import grader
grader(evolve_heat_slow, grade_ver=1)
```

    ============================== ALL TESTS PASSED! ===============================
    Your submission code: bw4c328516b4c813fe5f8394f4f6b83799336f22f27044cb26efb23cf8
    ================================================================================

Armed with this function, we will find the steady state of a more finely-gridded sheet of steel, with a less trivial set of boundary heat-values.

We will create an 80x96 grid with the following boundary conditions:

- Along the top row, we linearly increase the heat from 0 to 300 degrees from left to right
- Along the bottom row, we fade from 0 to 80 degrees at the middle and back to 0 on the right
- Along the left side, we linearly increase from 0 degrees at the bottom to 90 at the top (note that the very corner point is 0 from the 0 -> 300 continuum above)
- Along the right side, we linearly increase the heat from 0 to 300 degrees from bottom to top

```python
# creating the 80x96-grid sheet with the non-trivial boundary conditions
# simply execute this cell; you need not change anything.

import numpy as np
# discretize our rectangle into an 80x96 grid
rows = 80
columns = 96
u = np.zeros((rows, columns))

# set up the boundary conditions
u[0] = np.linspace(0, 300, columns)                # top row runs 0 -> 300
u[1:,0] = np.linspace(90, 0, rows-1)               # left side goes 0 -> 90 bottom to top
u[-1,:columns//2] = np.linspace(0, 80, columns//2) # 0 (left) to 80 (middle) along the bottom
u[-1,columns//2:] = np.linspace(80, 0, columns//2) # 80 (middle) to 0 (left) along the bottom
u[:,-1] = np.linspace(300,0,rows)                  # 0 -> 300 bottom to top along the right
```

Let's plot the initial condition for this steel sheet. You should see a "hot spot" in the top-right corner, and varying amounts of heat elsewhere on the boundary. Check that this corresponds to the boundary conditions that we imposed.

```python
# execute this cell

# matplotlib is a Python library used for visualizing data
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
%matplotlib notebook

fig, ax = plt.subplots()
ax.imshow(u, cmap='hot');
```

    <IPython.core.display.Javascript object>

Now, we will make an animation of the heat spreading through this material. However, our current implementation is too slow - let's time the amount of time required to evolve the heat in the material for 5000 iterations. This should take 20 sec - 1 minute.

```python
import time
slow = u.copy()
start = time.time()
for _ in range(5000): # perform 5000 iterations to reach a steady state
    slow = evolve_heat_slow(slow)
t = round(time.time() - start, 1)
print("`evolve_heat_slow` took {} seconds to complete 5000 iterations".format(t))
```

    `evolve_heat_slow` took 45.3 seconds to complete 5000 iterations

# Problem 2: A Vectorized Version of Finite Differences
Use NumPy array arithmetic to vectorize the finite-difference method that you implemented in problem #1. Your code should not utilize any for-loops.

```python
import numpy as np
def evolve_heat_fast(u):
    """ Given a 2D array of heat-values (at fixed boundary), produces
        the new heat distribution after one iteration of the finite
        difference method.

        Parameters
        ----------
        u : numpy.ndarray shape=(M, N)
            An MxN array of heat values at time-step t-1.
            (M and N are both at least 2)

        Returns
        -------
        numpy.ndarray, shape=(M, N)
            An MxN array of heat values at time-step t.
        """
    old_temps = np.copy(u)
    new_temps = np.copy(u)
    new_temps[1:-1,1:-1] = (1/4) * (old_temps[2:,1:-1] + old_temps[:-2,1:-1] + old_temps[1:-1,:-2] + old_temps[1:-1,2:])
    return new_temps
```

```python
from bwsi_grader.python.heat_dispersion import grader
grader(evolve_heat_fast, grade_ver=2)
```

    ============================== ALL TESTS PASSED! ===============================
    Your submission code: bw95ca3a7cfec977141125d15af9059617240297927b952f8dfae97d2a
    ================================================================================

Now let's use our vectorized code to perform 5000 iterations to evolve the heat in our system. This should be nearly 100-times faster than our original implementation.

```python
# execute this cell

import time
fast = u.copy()
start = time.time()
all_frames = []
for _ in range(5000):
    all_frames.append(fast.copy())
    fast = evolve_heat_fast(fast)
t = round(time.time() - start, 1)
print("`evolve_heat_fast` took {} seconds to complete 5000 iterations".format(t))
```

Plotting the distribution of heat after 5000 time-steps of evolution.

```python
# execute this cell

fig, ax = plt.subplots()
ax.imshow(fast, cmap='hot');
```

Even better, let's plot an animation of the heat spreading through the steel sheet! The animation will loop back to the beginning after playing through.

```python
# execute this cell

fig = plt.figure()
t = u.copy()
im = plt.imshow(t, animated=True, cmap='hot')

def updatefig(*args):
    im.set_array(all_frames[args[0]])
    return im,

ani = FuncAnimation(fig, updatefig, range(5000), interval=1, blit=True, repeat=True,
                    repeat_delay=1000)
plt.show()
```

Try creating your own boundary conditions for the temperature distribution on this steel sheet. Reuse the code from above to animate how the heat spreads through it. Also congratulate yourself for numerically solving a fixed-boundary partial differential equation :)


# Cvxpylayers tutorial

```python
import cvxpy as cp
import matplotlib.pyplot as plt
import numpy as np
import torch
from cvxpylayers.torch import CvxpyLayer
torch.set_default_dtype(torch.double)

np.set_printoptions(precision=3, suppress=True)
```

## Parametrized convex optimization problem

$$
\begin{array}{ll} \mbox{minimize} & f_0(x;\theta)\\
\mbox{subject to} & f_i(x;\theta) \leq 0, \quad i=1, \ldots, m\\
& A(\theta)x=b(\theta),
\end{array}
$$
with variable $x \in \mathbf{R}^n$ and parameters $\theta\in\Theta\subseteq\mathbf{R}^p$

* objective and inequality constraints $f_0, \ldots, f_m$ are *convex* in $x$ for each $\theta$, *i.e.*, their graphs curve upward
* equality constraints are linear
* for a given value of $\theta$, find a value for $x$ that minimizes objective, while satisfying constraints
* we can efficiently solve these globally with near total reliability

## Solution map
* Solution $x^\star$ is an implicit function of $\theta$
* When unique, define solution map as function
$x^\star = \mathcal S(\theta)$
* Need to call numerical solver to evaluate
* This function is often differentiable
* In a series of papers we showed how to analytically differentiate this function, using the implicit function theorem
* Benefits of analytical differentiation: works with nonsmooth objective/constraints, low memory usage, don't compound errors

## CVXPY
* High level domain-specific language (DSL) for convex optimization
* Define variables, parameters, objective and constraints
* Synthesize into problem object, then call solve method
* We've added derivatives to CVXPY (forward and backward)

## CVXPYlayers

* Convert CVXPY problems into callable, differentiable Pytorch and Tensorflow modules in one line

## Applications
* learning convex optimization models (structured prediction): https://stanford.edu/~boyd/papers/learning_copt_models.html
* learning decision-making policies (reinforcement learning): https://stanford.edu/~boyd/papers/learning_cocps.html
* machine learning hyper-parameter tuning and feature engineering: https://stanford.edu/~boyd/papers/lsat.html
* repairing infeasible or unbounded optimization problems: https://stanford.edu/~boyd/papers/auto_repair_cvx.html
* as protection layers in neural networks: http://physbam.stanford.edu/~fedkiw/papers/stanford2019-10.pdf
* custom neural network layers (sparsemax, csoftmax, csparsemax, LML): https://locuslab.github.io/2019-10-28-cvxpylayers/
* and many more...

## Average example
Find the average of a vector:
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \sum_{i=1}^n (y_i - x)^2
\end{array}
\end{equation}
Variable $x$, parameters $y\in\mathbf{R}^n$

The solution map is clearly:
$$x=\sum_{i=1}^n y_i / n$$

```python
n = 7

# Define variables & parameters
x = cp.Variable()
y = cp.Parameter(n)

# Define objective and constraints
objective = cp.sum_squares(y - x)
constraints = []

# Synthesize problem
prob = cp.Problem(cp.Minimize(objective), constraints)

# Set parameter values
y.value = np.random.randn(n)

# Solve problem in one line
prob.solve(requires_grad=True)
print("solution:", "%.3f" % x.value)
print("analytical solution:", "%.3f" % np.mean(y.value))
```

    solution: -0.380
    analytical solution: -0.380

The gradient is simply:
$$\nabla_y x = (1/n)\mathbf{1}$$

```python
# Set gradient wrt x
x.gradient = np.array([1.])

# Differentiate in one line
prob.backward()
print("gradient:", y.gradient)
print("analytical gradient:", np.ones(y.size) / n)
```

    gradient: [0.143 0.143 0.143 0.143 0.143 0.143 0.143]
    analytical gradient: [0.143 0.143 0.143 0.143 0.143 0.143 0.143]

## Median example
Finding the median of a vector:
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \sum_{i=1}^n |y_i - x|,
\end{array}
\end{equation}
Variable $x$, parameters $y\in\mathbf{R}^n$

Solution:
$$x=\mathbf{median}(y)$$

Gradient (no duplicates):
$$(\nabla_y x)_i = \begin{cases}
1 & y_i = \mathbf{median}(y) \\
0 & \text{otherwise}.
\end{cases}$$

```python
n = 7

# Define variables & parameters
x = cp.Variable()
y = cp.Parameter(n)

# Define objective and constraints
objective = cp.norm1(y - x)
constraints = []

# Synthesize problem
prob = cp.Problem(cp.Minimize(objective), constraints)

# Set parameter values
y.value = np.random.randn(n)

# Solve problem in one line
prob.solve(requires_grad=True)
print("solution:", "%.3f" % x.value)
print("analytical solution:", "%.3f" % np.median(y.value))
```

    solution: -0.350
    analytical solution: -0.350

```python
# Set gradient wrt x
x.gradient = np.array([1.])

# Differentiate in one line
prob.backward()
print("gradient:", y.gradient)
g = np.zeros(y.size)
g[y.value == np.median(y.value)] = 1.
print("analytical gradient:", g)
```

    gradient: [-0.  0. -0.  1.  0. -0.  0.]
    analytical gradient: [0. 0. 0. 1. 0. 0. 0.]

## Elastic-net regression example
We are given training data $(x_i, y_i)_{i=1}^{N}$,
where $x_i\in\mathbf{R}$ are inputs and $y_i\in\mathbf{R}$ are outputs.
Suppose we fit a model for this regression problem by solving the elastic-net problem
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \frac{1}{N}\sum_{i=1}^N (ax_i + b - y_i)^2 + \lambda |a| + \alpha a^2,
\end{array}
\label{eq:trainlinear}
\end{equation}
where $\lambda,\alpha>0$ are hyper-parameters.

We hope that the test loss $\mathcal{L}^{\mathrm{test}}(a,b) =
\frac{1}{M}\sum_{i=1}^M (a\tilde x_i + b - \tilde y_i)^2$ is small, where
$(\tilde x_i, \tilde y_i)_{i=1}^{M}$ is our test set.

First, we set up our problem, where $\{x_i, y_i\}_{i=1}^N$, $\lambda$, and $\alpha$ are our parameters.

```python
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split

torch.manual_seed(0)
np.random.seed(0)
n = 2
N = 60
X, y = make_blobs(N, n, centers=np.array([[2, 2], [-2, -2]]), cluster_std=3)
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=.5)

Xtrain, Xtest, ytrain, ytest = map(
    torch.from_numpy, [Xtrain, Xtest, ytrain, ytest])
Xtrain.requires_grad_(True)
m = Xtrain.shape[0]

a = cp.Variable((n, 1))
b = cp.Variable((1, 1))
X = cp.Parameter((m, n))
Y = ytrain.numpy()[:, np.newaxis]

log_likelihood = (1. / m) * cp.sum(
    cp.multiply(Y, X @ a + b) - cp.logistic(X @ a + b)
)
regularization = - 0.1 * cp.norm(a, 1) - 0.1 * cp.sum_squares(a)
prob = cp.Problem(cp.Maximize(log_likelihood + regularization))
fit_logreg = CvxpyLayer(prob, [X], [a, b])

torch.manual_seed(0)
np.random.seed(0)
n = 1
N = 60
X = np.random.randn(N, n)
theta = np.random.randn(n)
y = X @ theta + .5 * np.random.randn(N)
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=.5)
Xtrain, Xtest, ytrain, ytest = map(
    torch.from_numpy, [Xtrain, Xtest, ytrain, ytest])
Xtrain.requires_grad_(True)

m = Xtrain.shape[0]

# set up variables and parameters
a = cp.Variable(n)
b = cp.Variable()
X = cp.Parameter((m, n))
Y = cp.Parameter(m)
lam = cp.Parameter(nonneg=True)
alpha = cp.Parameter(nonneg=True)

# set up objective
loss = (1/m)*cp.sum(cp.square(X @ a + b - Y))
reg = lam * cp.norm1(a) + alpha * cp.sum_squares(a)
objective = loss + reg

# set up constraints
constraints = []

prob = cp.Problem(cp.Minimize(objective), constraints)
```

```python
# convert into pytorch layer in one line
fit_lr = CvxpyLayer(prob, [X, Y, lam, alpha], [a, b])
# this object is now callable with pytorch tensors
fit_lr(Xtrain, ytrain, torch.zeros(1), torch.zeros(1))
```

    (tensor([[-0.6603]], grad_fn=<_CvxpyLayerFnFnBackward>),
     tensor([0.1356], grad_fn=<_CvxpyLayerFnFnBackward>))

```python
# sweep over values of alpha, holding lambda=0, evaluating the gradient along the way
alphas = np.logspace(-3, 2, 200)
test_losses = []
grads = []
for alpha_vals in alphas:
    alpha_tch = torch.tensor([alpha_vals], requires_grad=True)
    alpha_tch.grad = None
    a_tch, b_tch = fit_lr(Xtrain, ytrain, torch.zeros(1), alpha_tch)
    test_loss = (Xtest @ a_tch.flatten() + b_tch - ytest).pow(2).mean()
    test_loss.backward()
    test_losses.append(test_loss.item())
    grads.append(alpha_tch.grad.item())
```

```python
plt.semilogx()
plt.plot(alphas, test_losses, label='test loss')
plt.plot(alphas, grads, label='analytical gradient')
plt.plot(alphas[:-1], np.diff(test_losses) / np.diff(alphas), label='numerical gradient', linestyle='--')
plt.legend()
plt.xlabel("$\\alpha$")
plt.show()
```

```python
# sweep over values of lambda, holding alpha=0, evaluating the gradient along the way
lams = np.logspace(-3, 2, 200)
test_losses = []
grads = []
for lam_vals in lams:
    lam_tch = torch.tensor([lam_vals], requires_grad=True)
    lam_tch.grad = None
    a_tch, b_tch = fit_lr(Xtrain, ytrain, lam_tch, torch.zeros(1))
    test_loss = (Xtest @ a_tch.flatten() + b_tch - ytest).pow(2).mean()
    test_loss.backward()
    test_losses.append(test_loss.item())
    grads.append(lam_tch.grad.item())
```

```python
plt.semilogx()
plt.plot(lams, test_losses, label='test loss')
plt.plot(lams, grads, label='analytical gradient')
plt.plot(lams[:-1], np.diff(test_losses) / np.diff(lams), label='numerical gradient', linestyle='--')
plt.legend()
plt.xlabel("$\\lambda$")
plt.show()
```

```python
# compute the gradient of the test loss wrt all the training data points, and plot
plt.figure(figsize=(10, 6))
a_tch, b_tch = fit_lr(Xtrain, ytrain, torch.tensor([.05]), torch.tensor([.05]), solver_args={"eps": 1e-8})
test_loss = (Xtest @ a_tch.flatten() + b_tch - ytest).pow(2).mean()
test_loss.backward()
a_tch_test, b_tch_test = fit_lr(Xtest, ytest, torch.tensor([0.]), torch.tensor([0.]), solver_args={"eps": 1e-8})
plt.scatter(Xtrain.detach().numpy(), ytrain.numpy(), s=20)
plt.plot([-5, 5], [-3*a_tch.item() + b_tch.item(),3*a_tch.item() + b_tch.item()], label='train')
plt.plot([-5, 5], [-3*a_tch_test.item() + b_tch_test.item(), 3*a_tch_test.item() + b_tch_test.item()], label='test')
Xtrain_np = Xtrain.detach().numpy()
Xtrain_grad_np = Xtrain.grad.detach().numpy()
ytrain_np = ytrain.numpy()
for i in range(Xtrain_np.shape[0]):
    plt.arrow(Xtrain_np[i], ytrain_np[i],
              -.1 * Xtrain_grad_np[i][0], 0.)
plt.legend()
plt.show()
```

```python
# move the training data points in the direction of their gradients, and see the train line get closer to the test line
plt.figure(figsize=(10, 6))
Xtrain_new = torch.from_numpy(Xtrain_np - .15 * Xtrain_grad_np)
a_tch, b_tch = fit_lr(Xtrain_new, ytrain, torch.tensor([.05]), torch.tensor([.05]), solver_args={"eps": 1e-8})
plt.scatter(Xtrain_new.detach().numpy(), ytrain.numpy(), s=20)
plt.plot([-5, 5], [-3*a_tch.item() + b_tch.item(),3*a_tch.item() + b_tch.item()], label='train')
plt.plot([-5, 5], [-3*a_tch_test.item() + b_tch_test.item(), 3*a_tch_test.item() + b_tch_test.item()], label='test')
plt.legend()
plt.show()
```


# Automatic differentiation

Automatic differentiation (AD) is a set of techniques to calculate **exact** derivatives, numerically, in an automatic way. It is neither symbolic differentiation, nor something like finite differences.

There are two main methods: forward-mode AD and reverse-mode AD.
Each has its strengths and weaknesses. Forward mode is significantly easier to implement.

# Forward-mode AD

Let's start by thinking about univariate functions $f: \mathbb{R} \to \mathbb{R}$. We would like to calculate the derivative $f'(a)$ at some point $a \in \mathbb{R}$.

We know various rules about how to calculate such derivatives. For example, if we have already managed to calculate $f'(a)$ and $g'(a)$, we can calculate $(f+g)'(a)$ and $(f.g)'(a)$ as

\begin{align}
(f+g)'(a) &= f'(a) + g'(a)\\
(f.g)'(a) &= f'(a) \, g(a) + f(a) \, g'(a)
\end{align}

We also have the chain rule, which plays a crucial role:

$$(f \circ g)'(a) = f'(g(a)) \, g'(a)$$

We see that in general we will need, for each function $f$, both the value $f(a)$ and the derivative $f'(a)$, and this is the *only* information that we require in order to calculate the first derivative of any combination of functions.

## Jets

Formally, we can think of a first-order Taylor polynomial of $f$, called the
[**jet** of $f$](https://en.wikipedia.org/wiki/Jet_(mathematics) at $a$, denoted $J_a(f)$:

$$(J_a(f))(x) := f(a) + x f'(a)$$

[This can be thought of as representing the **set of all functions** with the same data $f(a)$ and $f'(a)$.]

Formally, it is common to think of this as a "dual number", $f + \epsilon f'$, that we can manipulate, following the rule that $\epsilon^2 = 0$. (Cf. complex numbers, which have the same structure, but with $\epsilon^2 = -1$.) E.g.

$$(f + \epsilon f') \times (g + \epsilon g') = f \, g + \epsilon (f' g + f g')$$

shows how to define the multiplication of two jets.

## Computer representation

As usual, we can represent a polynomial just by its degree and its coefficients, so we can define a Julia object as follows. We will leave the evaluation point $(a)$ as being implicit, although we could, of course, include it if desired.

```julia
immutable Jet{T} <: Real
    val::T  # value
    der::T  # derivative   # type \prime<TAB> to get ′
end
```

```julia
import Base: +, *, -, convert, promote_rule
```

```julia
+(f::Jet, g::Jet) = Jet(f.val + g.val, f.der + g.der)
-(f::Jet, g::Jet) = Jet(f.val - g.val, f.der - g.der)

*(f::Jet, g::Jet) = Jet(f.val*g.val, f.der*g.val + f.val*g.der)
```

    * (generic function with 150 methods)

We can now define `Jet`s and manipulate them:

```julia
f = Jet(3, 4)  # any function f such that f(a) = 3 and f'(a) = 4, or the set of all such functions
g = Jet(5, 6)  # any function g such that g(a) = 5 and g'(a) = 6

f + g   # calculate the value and derivative of (f + g) for any f and g in these sets
```

    Jet{Int64}(8,10)

```julia
f * g
```

    Jet{Int64}(15,38)

```julia
f * (g + g)
```

    Jet{Int64}(30,76)

## Performance

It seems like we must have introduced quite a lot of computational overhead by creating a relatively complex data structure, and associated methods, to manipulate pairs of numbers. Let's see how the performance is:

```julia
add(a1, a2, b1, b2) = (a1+b1, a2+b2)
```

    add (generic function with 1 method)

```julia
add(1, 2, 3, 4)
@time add(1, 2, 3, 4)
```

      0.000001 seconds (4 allocations: 176 bytes)

    (4,6)

```julia
a = Jet(1, 2)
b = Jet(3, 4)

add2(j1, j2) = j1 + j2
add2(a, b)
@time add2(a, b)
```

      0.000002 seconds (5 allocations: 192 bytes)

    WARNING: Method definition add2(Any, Any) in module Main at In[157]:4 overwritten at In[158]:4.

    Jet{Int64}(4,6)

```julia

```

```julia
@code_native add(1, 2, 3, 4)
```

    	.section	__TEXT,__text,regular,pure_instructions
    Filename: In[151]
    	pushq	%rbp
    	movq	%rsp, %rbp
    Source line: 1
    	addq	%rcx, %rsi
    	addq	%r8, %rdx
    	movq	%rsi, (%rdi)
    	movq	%rdx, 8(%rdi)
    	movq	%rdi, %rax
    	popq	%rbp
    	retq
    	nopw	%cs:(%rax,%rax)

```julia
@code_native add2(a, b)
```

    	.section	__TEXT,__text,regular,pure_instructions
    Filename: In[158]
    	pushq	%rbp
    	movq	%rsp, %rbp
    Source line: 4
    	movq	(%rdx), %rax
    	movq	8(%rdx), %rcx
    	addq	(%rsi), %rax
    	addq	8(%rsi), %rcx
    	movq	%rax, (%rdi)
    	movq	%rcx, 8(%rdi)
    	movq	%rdi, %rax
    	popq	%rbp
    	retq
    	nop

We see that there is only a slight overhead to do with moving the data around. The data structure itself has disappeared, and we basically have a standard Julia tuple.

## Functions on jets: chain rule

We can also define functions of these objects using the chain rule. For example, if `f` is a jet representing the function $f$, then we would like `exp(f)` to be a jet representing the function $\exp \circ f$, i.e. with value $\exp(f(a))$ and derivative $(\exp \circ f)'(a) = \exp(f(a)) \, f'(a)$:

```julia
import Base: exp
```

```julia
exp(f::Jet) = Jet(exp(f.val), exp(f.val) * f.der)
```

    exp (generic function with 12 methods)

```julia
f
```

    Jet{Int64}(3,4)

```julia
exp(f)
```

    Jet{Float64}(20.085536923187668,80.34214769275067)

## Conversion and promotion

However, we can't do e.g. the following:

```julia
# 3 * f
```

[Warning: In Julia 0.5, you may need to restart the kernel after doing this for the following to work correctly.]

In order to get this to work, we need to hook into Julia's type promotion and conversion machinery.
First, we specify how to promote a number and a `Jet`:

```julia
promote_rule{T<:Real,S}(::Type{Jet{S}}, ::Type{T}) = Jet{S}
```

    promote_rule (generic function with 102 methods)

Second, we specify how to `convert` a (constant) number to a `Jet`. By e.g. $g = f+3$, we mean the function such that $g(x) = f(x) + 3$ for all $x$, i.e. $g = f + 3.\mathbb{1}$, where $\mathbb{1}$ is the constant function $\mathbb{1}: x \mapsto 1$.

Thus we think of a constant $c$ as the constant function $c \, \mathbb{1}$, with $c(a) = c$ and $c'(a) = 0$, which we encode as the following conversion:

```julia
convert{T<:Union{AbstractFloat, Integer, Rational},S}(::Type{Jet{S}}, x::T) = Jet{S}(x, 0)
```

    convert (generic function with 600 methods)

```julia
convert(Jet{Float64}, 3.1)
```

    Jet{Float64}(3.1,0.0)

```julia
promote(Jet(1,2), 3.0)
```

    (Jet{Int64}(1,2),Jet{Int64}(3,0))

```julia
promote(Jet(1,2), 3.1)
```

```julia
convert(Jet{Float64}, 3.0)
```

    Jet{Float64}(3.0,0.0)

Julia's machinery now enables us to do what we wanted:

```julia
Jet(1.1, 2.3) + 3
```

    Jet{Float64}(4.1,2.3)

## Calculating derivatives of arbitrary functions

How can we use this to calculate the derivative of an arbitrary function? For example, we wish to differentiate the function

```julia
h(x) = x^2 - 2
```

    h (generic function with 1 method)

at $a = 3$.

We think of this as a function of $x$, which itself we think of as the identity function $\iota: x \mapsto x$, so that

$$h = \iota^2 - 2.\mathbb{1}$$

We represent the identity function as follows:

```julia
a = 3
x = Jet(a, 1)
```

    Jet{Int64}(3,1)

since $\iota'(a) = 1$ for any $a$.

Now we simply evaluate the function `h` at `x`:

```julia
h(x)
```

    Jet{Int64}(7,6)

The first component of the resulting `Jet` is the value $h(a)$, and the second component is the derivative, $h'(a)$.

We can codify this into a function as follows:

```julia
derivative(f, x) = f(Jet(x, one(x))).der
```

    WARNING: Method definition derivative(Any, Any) in module Main at In[32]:1 overwritten at In[49]:1.

    derivative (generic function with 1 method)

```julia
derivative(x -> 3x^5 + 2, 2)
```

    240

This is capable of differentiating any function that involves functions whose derivatives we have specified by defining corresponding rules on `Jet` objects. For example,

```julia
y = [1.,2]
k(x) = (y'* [x 2; 3 4] * y)[]
```

    WARNING: Method definition k(Any) in module Main at In[29]:2 overwritten at In[34]:2.

    k (generic function with 1 method)

```julia
k(3)
```

    29.0

```julia
derivative(x->k(x), 10)
```

    1

This works since Julia is constructing the following object:

```julia
[Jet(3.0, 1.0) 2; 3 4]
```

    2×2 Array{Jet{Float64},2}:
     Jet{Float64}(3.0,1.0)  Jet{Float64}(2.0,0.0)
     Jet{Float64}(3.0,0.0)  Jet{Float64}(4.0,0.0)

# Higher dimensions

How can we extend this to higher dimensions? For example, we wish to differentiate the following function $f: \mathbb{R}^2 \to \mathbb{R}$:

```julia
f1(x, y) = x^2 + x*y
```

    f1 (generic function with 1 method)

As we learn in calculus, the partial derivative $\partial f/\partial x$ is the function obtained by fixing $y$, thinking of the resulting function as a function only of $x$, and then differentiating.

Suppose that we wish to differentiate $f$ at $(a, b)$:

```julia
a, b = 3.0, 4.0

f1_x(x) = f1(x, b)  # single-variable function
```

    WARNING: Method definition f1_x(Any) in module Main at In[47]:3 overwritten at In[50]:3.

    f1_x (generic function with 1 method)

Since we now have a single-variable function, we can differentiate it:

```julia
derivative(f1_x, a)
```

    10.0

Under the hood this is doing

```julia
f1(Jet(a, one(a)), b)
```

    Jet{Float64}(21.0,10.0)

Similarly, we can differentiate with respect to $y$ by doing

```julia
f1(a, Jet(b, one(b)))
```

    Jet{Float64}(21.0,3.0)

Note that we must do **two separate calculations** to get the two partial derivatives. To calculate a gradient of a function $f:\mathbb{R}^n \to \mathbb{R}$ thus requires $n$ separate calculations.

Forward-mode AD is implemented in a clean and efficient way in the `ForwardDiff.jl` package.

# Syntax trees

## Forward-mode

To understand what forward-mode AD is doing, and its name, it is useful to think of an expression as a **syntax tree**; cf. [this notebook](Syntax trees in Julia.ipynb).

If we label the nodes in the tree as $v_i$, then forward differentiation fixes a variable, e.g. $y$, and calculates $\partial v_i / \partial y$ for each $i$. If e.g. $v_1 = v_2 + v_3$, then we have

$$\frac{\partial v_1}{\partial y} = \frac{\partial v_2}{\partial y} + \frac{\partial v_3}{\partial y}.$$

Denoting $v_1' := \frac{\partial v_1}{\partial y}$, we have $v_1' = v_2' + v_3'$, so we need to calculate the derivatives and nodes lower down in the graph first, and propagate the information up. We start at $v_x' = 0$, since $\frac{\partial x}{\partial y} = 0$, and $v_y' = 1$.

## Reverse mode

An alternative method to calculate derivatives is to fix not the variable with which to differentiate, but *what it is* that we differentiate, i.e. to calculate the **adjoint**, $\bar{v_i} := \frac{\partial f}{\partial v_i}$, for each $i$.

If $f = v_1 + v_2$, with $v_1 = v_3 + v_4$ and $v_2 = v_3 + v_5$, then

$$\frac{\partial f}{\partial v_3} = \frac{\partial f}{\partial v_1} \frac{\partial v_1}{\partial v_3} + \frac{\partial f}{\partial v_2} \frac{\partial v_2}{\partial v_3},$$

i.e.

$$\bar{v_3} = \alpha_{13} \, \bar{v_1} + \alpha_{2,3} \, \bar{v_2},$$

where $\alpha_{ij}$ are the coefficients specifying the relationship between the different terms. Thus, the adjoint information propagates **down** the graph, in **reverse** order, hence the name "reverse-mode".

For this reason, reverse mode is much harder to implement. However, it has the advantage that all derivatives $\partial f / \partial x_i$ are calculated in a *single pass* of the tree.

Julia has en efficient implementation of reverse-mode AD in https://github.com/JuliaDiff/ReverseDiff.jl

## Example of reverse mode

Reverse mode is difficult to implement in a general way, but easy to do by hand. e.g. consider the function

$$f(x,y,z) = x \, y - \sin(z)$$

We decompose this into its tree with labelled nodes, corresponding to the following sequence of elementary operations:

```julia
ff(x, y, z) = x*y - 2*sin(x*z)

x, y, z = 1, 2, 3

v₁ = x
v₂ = y
v₃ = z
v₄ = v₁ * v₂
v₅ = v₁ * v₃
v₆ = sin(v₅)
v₇ = v₄ - 2v₆  # f
```

    WARNING: Method definition ff(Any, Any, Any) in module Main at In[137]:1 overwritten at In[139]:1.

    1.7177599838802655

```julia
ff(x, y, z)
```

    1.7177599838802655

We have decomposed the **forward pass** into elementary operations. We now proceed to calculate the adjoints. The difficulty is to *find which variables depend on the current variable under question*.

```julia
v̄₇ = 1
v̄₆ = -2 # ∂f/∂v₆ = ∂v₇/∂v₆
v̄₅ = v̄₆ * cos(v₅)  # ∂v₇/∂v₆ * ∂v₆/∂v₅
v̄₄ = 1
v̄₃ = v̄₅ * v₁  # ∂f/∂v₃ = ∂f/∂v₅ . ∂v₅/∂v₃. # This gives ∂f/∂z
v̄₂ = v̄₄ * v₁
v̄₁ = v̄₅*v₃ + v̄₄*v₂
```

    7.939954979602673

Thus, in a single pass we have calculated the gradient $\nabla f(1, 2, 3)$:

```julia
(v̄₁, v̄₂, v̄₃)
```

    (7.939954979602673,1,1.9799849932008908)

Let's check that it's correct:

```julia
ForwardDiff.gradient(x->ff(x...), [x,y,z])
```

    3-element Array{Float64,1}:
     7.93995
     1.0
     1.97998

# Example: optimization

As an example of the use of AD, consider the following function that we wish to optimize:

```julia
x = rand(3)
y = rand(3)

distance(W) = W*x - y
```

    distance (generic function with 1 method)

```julia
using ForwardDiff
```

```julia
ForwardDiff.jacobian(distance, rand(3,3))
```

    3×9 Array{Float64,2}:
     0.889986  0.0       0.0       0.855784  …  0.659763  0.0       0.0
     0.0       0.889986  0.0       0.0          0.0       0.659763  0.0
     0.0       0.0       0.889986  0.0          0.0       0.0       0.659763

```julia
objective(W) = (a = distance(W); dot(a, a))
```

    WARNING: Method definition objective(Any) in module Main at In[60]:4 overwritten at In[66]:1.

    objective (generic function with 1 method)

```julia
W0 = rand(3, 3)
grad = ForwardDiff.gradient(objective, W0)
```

    3×3 Array{Float64,2}:
     2.14718   2.06467    1.59175
     0.023659  0.0227498  0.0175388
     2.13258   2.05063    1.58092

```julia
2*(W0*x-y)*x' == grad  # LHS is the analytical derivative
```

    true

# Example: Interval arithmetic

How can we find roots of a function?

```julia
f2(x) = x^2 - 2
```

    WARNING: Method definition f2(Any) in module Main at In[100]:1 overwritten at In[108]:1.

    f2 (generic function with 1 method)

## Exclusion of domains

An idea is to *exclude* regions of $\mathbb{R}$ by showing that they *cannot* contain a zero, by calculating the image (range) of the function over a given domain.

This is, in general, a difficult problem, but **interval arithmetic** provides a partial solution, by calculating an **enclosure** of the range, i.e. and interval that is guaranteed to contain the range.

```julia
using ValidatedNumerics
```

```julia
X = 3..4
```

    [3, 4]

```julia
typeof(X)
```

    ValidatedNumerics.Interval{Float64}

This is a representation of the set $X = [3, 4] := \{x\in \mathbb{R}: 3 \le x \le 4\}$.

We can evaluate a Julia function on an `Interval` object `X`. The result is a new `Interval`, which is **guaranteed to contain the true image** $\mathrm{range}(f; X) := \{f(x): x \in X \}$.  This is achieved by defining arithmetic operations on intervals in the correct way, e.g.

$$X + Y = [x_1, x_2] + [y_1, y_2] = [x_1+y_1, x_2+y_2].$$

```julia
f2(X)
```

    [7, 14]

Since this result does not contain $0$, we have *proved* that $f$ has no zero in the domain $[3,4]$. We can even use semi-infinite intervals:

```julia
X1 = 3..∞  # type \infty<TAB>
```

    [3, ∞]

```julia
f2(X1)
```

    [7, ∞]

```julia
X2 = -∞.. -3   # space is required
```

    [-∞, -3]

```julia
f2(X2)
```

    [7, ∞]

We have thus exclued two semi-infinite regions, and have proved that any root *must* lie in $[-3,3]$, by two simple calculations. However,

```julia
f2(-3..3)
```

    [-2, 7]

We cannot conclude anything from this, since the result is, in general, an over-estimate of the true range, which thus may or may not contain zero. We can proceed by bisecting the interval. E.g. after two bisections, we find

```julia
f2(-3.. -1.5)
```

    [0.25, 7]

so we have excluded another piece.

## Proving existence of roots

To prove that there *does* exist a root, we need a different approach. It is a standard method to evaluate the function at two end-points of an interval:

```julia
f2(1), f2(2)
```

    (-1,2)

Since there is a sign change, there exists at least one root $x^*$ in the interval $[1,2]$, i.e. a point such that $f(x^*) = 0$.

To prove that it is unique, one method is to prove that $f_2$ is *monotone* in that interval, i.e. that the derivative has a unique sign. To do so, we need to evaluate the derivative *at every point in the interval*, which seems impossible.

Again, however, interval arithmetic easily gives an *enclosure* of this image. To show this, we need to evaluate the derivative using interval arithmetic.

Thanks to Julia's parametric types, we get **composability for free**: we can just substitute in an interval to `ForwardDiff` or `Jet`, and it works:

```julia
ForwardDiff.derivative(f2, 1..2)
```

    [2, 4]

Again, the reason for this is that Julia creates the object

```julia
Jet(x, one(x))
```

    Jet{ValidatedNumerics.Interval{Float64}}([1, 2],[1, 1])

Since an enclosure of the derivative is the interval $[2, 4]$ (and, in fact, in this case this is the true image, but there is no way to know this other than with an analytical calculation), we have **proved** that the image of the derivative function $f'$ over the interval $X = [1,2]$ does *not* contain zero, and hence that the image is monotone.

To actually find the root within this interval, we can use the [Newton interval method](Interval Newton.ipynb). In general, we should not expect to be able to use intervals in standard numerical methods designed for floats; rather, we will need to modify the numerical method to take *advantage* of intervals.

The Newton interval method can find, in a guaranteed way, *all* roots of a function in a given interval (or tell you if when it is unable to to so, for example if there are double roots). Although people think that finding roots of a general function is difficult, this is basically a solved problem using these methods.

lectures/other/Automatic differentiation and applications.ipynb