# Estimation with sensor offset and drift

_This is based on a problem presented in EE 263 Linear Dynamical Systems at Stanford University_

### Background: Least Squares

Suppose that we are trying to estimate a vector $p\in\mathbf{R}^{n}$. We take $m$ scalar measurements, each a linear combination of elements of $p$. Each measurement, $y_i$, is modeled as

\begin{align}
y_i = a_i^T p + v_i,\quad\quad i=1,\ldots,m,
\end{align}

where

- $y_i\in\mathbf{R}$ is the $i^{\text{th}}$ measurement
- $p\in\mathbf{R}^n$ is vector we wish to measure
- $v_i$ is the sensor or measurement error of the ith measurement.

We assume that the $a_i$'s are given, i.e. we know the calibration values of the sensor for each measurement. Additionally, we assume that we have at least as many measurements as values we need to estimate $\left(m\geq n\right)$ and that the matrix A given by

\begin{align}
A = \left[\begin{matrix}
a_1^T \\ a_2^T \\ \vdots \\ a_m^T
\end{matrix}\right]
\end{align}

is full rank. For standard linear regression, the vector $x$ would be in $\mathbf{R}^2$ and would represent the slope and intercept parameters. In this context, the matrix $A$ would have the given values for the independent variable in the first column and all ones in the second column, and the vector $y\in\mathbf{R}^m$ would contain the values for the dependent variable. If the error terms, $v_i$, were small, random, and centered around zero (Gaussian white noise), then least squares gives the optimal estimation of x (i.e., minimizes the RMSE):

\begin{align}
\hat{p} = \left(A^TA\right)^{-1}A^T y = \underset{p}{\text{argmin}}\left\lVert y - Ap \right\rVert_2^2
\end{align}

Let's illustrate this with a quick example.

```python
import numpy as np
from numpy.linalg import inv
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set(context='talk', style='darkgrid', palette='colorblind')
```

```python
m = 2
b = -1
xs = np.random.uniform(0, 1, 10)
ys = m * xs + b + np.random.normal(0, .2, size=len(xs))
fig = plt.figure(figsize=(8,6))
plt.scatter(xs, ys)
plt.xlabel('x'); plt.ylabel('y')
plt.show()
```

Construct the matrix $A$.

```python
A = np.c_[xs, np.ones_like(xs)]
print A
```

    [[0.12997784 1.        ]
     [0.10977849 1.        ]
     [0.60488689 1.        ]
     [0.38266506 1.        ]
     [0.60065919 1.        ]
     [0.44154635 1.        ]
     [0.23412023 1.        ]
     [0.8084878  1.        ]
     [0.22136882 1.        ]
     [0.23757838 1.        ]]

Solve the least squares problem.

```python
p_hat = inv(A.T.dot(A)).dot(A.T).dot(ys)
```

Compared the estimated parameters to the real parameters.

```python
fig = plt.figure(figsize=(8,6))
plt.scatter(xs, ys, label='measured')
x2 = np.linspace(0, 1)
plt.plot(x2, m*x2 + b, linewidth=1, ls='--', label='actual')
plt.plot(x2, p_hat[0]*x2 + p_hat[1], linewidth=1, ls='--', label='estimated')
plt.xlabel('x'); plt.ylabel('y')
plt.xlim(0, 1); plt.ylim(-1, 1)
plt.legend(loc=4)
plt.show()
```

### Adding bias and drift to the error

For this problem, let us assume that the error contains some predictable terms in addition to white noise: a common offset term that is the same for all measurements and a drift term that grows linearly with each subsequent measurement. We model this situation as:

\begin{align}
v_i = \alpha + \beta i + w_i
\end{align}

We will use least squares to simultaneously the desired vector $x\in\mathbf{R}^n$, the bias term $\alpha\in\mathbf{R}$, and the drift term $\beta\in\mathbf{R}$. We begin by substituting our error model into the measurement model:

\begin{align}
y_i = a_i^T x + \alpha + \beta i + w_i,\quad\quad i=1,\ldots,m,
\end{align}

This induces the matrix equation:

\begin{align}
\left[\begin{matrix}
y_1 \\ y_2  \\ \vdots \\ y_m
\end{matrix}\right] &=
\left[\begin{matrix}
a_1^T & 1 & 1 \\
a_2^T & 1 & 2 \\
\vdots & \vdots & \vdots \\
a_m^T & 1 & m
\end{matrix}\right]
\left[\begin{matrix}
x \\ \alpha \\ \beta
\end{matrix}\right] +
\left[\begin{matrix}
w_1 \\ w_2 \\ \vdots \\ w_m
\end{matrix}\right] \\ \\
y &= \tilde{A}\tilde{x} + w
\end{align}

We can now use least squares to find all parameters:

\begin{align}
\hat{\tilde{x}} = \left[\begin{matrix}
\hat{x} \\ \hat{\alpha} \\ \hat{\beta}
\end{matrix}\right] = \left(\tilde{A}^T\tilde{A}\right)^{-1}\tilde{A}^T y
\end{align}

Alright! We have a closed-form expression to estimate $x$, $\alpha$, and $\beta$. A couple caveats: for this to work, the following much be true:

- $m\geq n +2$. We must have enough measurement to recover $n+2$ parameters.
- $\tilde{A}$ must be full rank. Note, even if $A$ is full rank (which is given), $\tilde{A}$ might not be. If some linear combination of the sensor signals looks like some linear combination of the offset and drift, then it is impossible to separate the offset and drift parameters from $x$.

Immediately, we see a potential problem! If we are using this to derive the parameters of a statistical model (as in the previous example), that model can't contain an offset term! If it does, then the second condition is violated.

Alright, that's enough theory. Now let's look at an application of this technique. Again, suppose we have some noisy measurements of a function. This time we'll make the function a bit more interesting: a cubic function with no offset. We'll start by again looking at the simpler case where the noise terms are i.i.d. Gaussian white noise, and then we'll add the offset and drift terms.

```python
def our_func(x, p1=1, p2=3, p3=-0.2):
    output = p1 * x + p2 * np.power(x, 2) + p3 * np.power(x, 3)
    return output
```

```python
np.random.seed(42)
```

```python
fig = plt.figure(figsize=(10, 8))
xs_ideal = np.linspace(-5, 15, 30)
xs_samples = np.random.uniform(-5, 15, 15)
ys_samples = our_func(xs_samples) + np.random.normal(0, 5, size=len(xs_samples))
plt.plot(xs_ideal, our_func(xs_ideal), linewidth=1, ls='--')
plt.scatter(xs_samples, ys_samples)
plt.legend(['Actual Function', 'Noisy Measurements'], loc=4)
plt.xlabel('x'); plt.ylabel('y')
_ = plt.title('Noisy Samples of a Cubic Function')
```

A cubic function with no offset is fully defined by three scalar coefficients:

\begin{align}
y = \mathcal{P}_3(x) &= p_1 x + p_2 x^2 + p_3 x^3 \\
&= \left[\begin{matrix} x & x^2 & x^3 \end{matrix}\right] \left[\begin{matrix} p_1 \\ p_2 \\ p_3 \end{matrix}\right]
\end{align}

So, given a set of noisy measurement, $\left\{\left(x_1, y_1\right),\left(x_2, y_2\right),\ldots, \left(x_m, y_m\right)\right\}$, we find $p=\left[\begin{matrix}p_1 & p_2 & p_3\end{matrix}\right]^T$ as follows. First, construct the following matrix:

\begin{align}
A &= \left[\begin{matrix}
x_1 & x_1^2 & x_1^3 \\
x_2 & x_2^2 & x_2^3 \\
\vdots & \vdots & \vdots \\
x_m & x_m^2 & x_m^3 \\
\end{matrix}\right]
\end{align}

Then, find $\hat{p}$ as before:

\begin{align}
\hat{p} = \left(A^TA\right)^{-1}A^T y = \underset{p}{\text{argmin}}\left\lVert y - Ap \right\rVert_2^2
\end{align}

Let's try it out on this cubic-fitting problem! As before, start by constructing matrix $A$.

```python
A = np.c_[xs_samples, np.power(xs_samples, 2), np.power(xs_samples, 3)]
np.set_printoptions(precision=2)
print A
```

    [[ 2.49e+00  6.20e+00  1.55e+01]
     [ 1.40e+01  1.96e+02  2.75e+03]
     [ 9.64e+00  9.29e+01  8.96e+02]
     [ 6.97e+00  4.86e+01  3.39e+02]
     [-1.88e+00  3.53e+00 -6.64e+00]
     [-1.88e+00  3.53e+00 -6.65e+00]
     [-3.84e+00  1.47e+01 -5.65e+01]
     [ 1.23e+01  1.52e+02  1.87e+03]
     [ 7.02e+00  4.93e+01  3.46e+02]
     [ 9.16e+00  8.39e+01  7.69e+02]
     [-4.59e+00  2.11e+01 -9.66e+01]
     [ 1.44e+01  2.07e+02  2.98e+03]
     [ 1.16e+01  1.36e+02  1.58e+03]
     [-7.53e-01  5.67e-01 -4.27e-01]
     [-1.36e+00  1.86e+00 -2.53e+00]]

Then, solve the least squares problem.

```python
p_hat = inv(A.T.dot(A)).dot(A.T).dot(ys_samples)
print p_hat
```

    [ 0.96  2.88 -0.19]

Recall, that the actual parameters were $\left[1, 3, -0.2\right]$. Let's compare the results.

```python
fig = plt.figure(figsize=(10, 8))
plt.plot(xs_ideal, our_func(xs_ideal), linewidth=1, ls='--', label='Actual Function')
plt.plot(xs_ideal, our_func(xs_ideal, p1=p_hat[0], p2=p_hat[1], p3=p_hat[2]),
         linewidth=1, ls='--', label='Estimated Function')
plt.scatter(xs_samples, ys_samples, label='Noisy Measurements')
plt.legend(loc=4)
plt.xlabel('x'); plt.ylabel('y')
_ = plt.title('Noisy Samples of a Cubic Function')
```

That's pretty good! Now let's make things interesting and include the offset and drift terms. The existing formulation already includes Gaussian white noise, so there's no need to add that again.

```python
alpha = -3
beta = 5
```

```python
ys_od = ys_samples + alpha + beta * np.arange(1, 1 + A.shape[0])
```

As a sanity check, here's what the measurements look like now, as compared to the actual function:

```python
fig = plt.figure(figsize=(10, 8))
plt.plot(xs_ideal, our_func(xs_ideal), linewidth=1, ls='--', label='Actual Function')
plt.scatter(xs_samples, ys_od, label='Noisy Measurements')
plt.legend(loc=4)
plt.xlabel('x'); plt.ylabel('y')
_ = plt.title('Noisy Samples of a Cubic Function')
```

Yikes! That looks like a mess. We might not think we would have any chance of recovering the original function from these measurements, but we'll see that correctly modeling the error saves the day.

First, we construct the augmented matrix $\tilde{A}$.

```python
A_tilde = np.concatenate([
    A,
    np.ones(A.shape[0])[:, None],
    np.arange(1, 1 + A.shape[0])[:, None]
], axis = 1)
print A_tilde
```

    [[ 2.49e+00  6.20e+00  1.55e+01  1.00e+00  1.00e+00]
     [ 1.40e+01  1.96e+02  2.75e+03  1.00e+00  2.00e+00]
     [ 9.64e+00  9.29e+01  8.96e+02  1.00e+00  3.00e+00]
     [ 6.97e+00  4.86e+01  3.39e+02  1.00e+00  4.00e+00]
     [-1.88e+00  3.53e+00 -6.64e+00  1.00e+00  5.00e+00]
     [-1.88e+00  3.53e+00 -6.65e+00  1.00e+00  6.00e+00]
     [-3.84e+00  1.47e+01 -5.65e+01  1.00e+00  7.00e+00]
     [ 1.23e+01  1.52e+02  1.87e+03  1.00e+00  8.00e+00]
     [ 7.02e+00  4.93e+01  3.46e+02  1.00e+00  9.00e+00]
     [ 9.16e+00  8.39e+01  7.69e+02  1.00e+00  1.00e+01]
     [-4.59e+00  2.11e+01 -9.66e+01  1.00e+00  1.10e+01]
     [ 1.44e+01  2.07e+02  2.98e+03  1.00e+00  1.20e+01]
     [ 1.16e+01  1.36e+02  1.58e+03  1.00e+00  1.30e+01]
     [-7.53e-01  5.67e-01 -4.27e-01  1.00e+00  1.40e+01]
     [-1.36e+00  1.86e+00 -2.53e+00  1.00e+00  1.50e+01]]

The standard least squared estimate:

```python
p_ls = inv(A.T.dot(A)).dot(A.T).dot(ys_od)
print p_ls
```

    [-3.5   4.51 -0.27]

The least squared estimate using the noise model:

```python
p_ls_plus_noise = inv(A_tilde.T.dot(A_tilde)).dot(A_tilde.T).dot(ys_od)
print p_ls_plus_noise[:3]
```

    [ 1.16  2.86 -0.19]

And we also recover the estimates of $\alpha$ and $\beta$.

```python
p_ls_plus_noise[3:]
```

    array([-7.56,  5.57])

So, we can see that the LS estimate that utilizes the noise model does a much better job of estimating the desired parameters. What about the error? Recall the quantity we are trying to minimize is

\begin{align}
\left\lVert y- Ap\right\rVert_2^2
\end{align}

So, there are really two errors to consider:

- The value of the cost function we are minimizing via the least squares method. In a real application, this is the only error we could consider
- The root-mean-square error between the estimated functions and the actual functions, evaluated at the sample points. We are only able to this because we know the function that generated the data

Let's begin by evaluating the cost function we've minimized.

```python
J_ls = np.sum(np.power(ys_od - A.dot(p_ls), 2))
J_ls_plus_noise = np.sum(np.power(ys_od - A_tilde.dot(p_ls_plus_noise), 2))
print 'Basic LS Error:        {:.3f}'.format(J_ls)
print 'Noise model LS Error:  {:.3f}'.format(J_ls_plus_noise)
```

    Basic LS Error:        13403.986
    Noise model LS Error:  469.325

That's great! By accounting for the error, we drive the cost down by an additional 32%. That certainly seems promising. Now, let's compare to the oracle (the actual function that generated the data).

```python
x = our_func(xs_samples)
x_ls = our_func(xs_samples, p1=p_ls[0], p2=p_ls[1], p3=p_ls[2])
x_ls_plus_noise = our_func(xs_samples, p1=p_ls_plus_noise[0], p2=p_ls_plus_noise[1], p3=p_ls_plus_noise[2])
err_ls = np.linalg.norm(x - x_ls)
err_ls_plus_noise = np.linalg.norm(x - x_ls_plus_noise)
```

```python
print 'Basic LS Error:        {:.3f}'.format(err_ls)
print 'Noise model LS Error:  {:.3f}'.format(err_ls_plus_noise)
```

    Basic LS Error:        120.583
    Noise model LS Error:  8.427

As expected, the method that correctly models the error gets much closer to the oracle.

And finally, visualize the results with another plot:

```python
fig = plt.figure(figsize=(10, 8))
plt.plot(xs_ideal, our_func(xs_ideal), linewidth=1, ls='--', label='Actual function')
plt.plot(xs_ideal, our_func(xs_ideal, p1=p_ls[0], p2=p_ls[1], p3=p_ls[2]), linewidth=1, ls='--', label='LS estimate')
plt.plot(xs_ideal, our_func(xs_ideal, p1=p_ls_plus_noise[0], p2=p_ls_plus_noise[1], p3=p_ls_plus_noise[2]),
         linewidth=1, ls='--', label='LS with error model estimate')
plt.scatter(xs_samples, ys_od, label='Noisy Measurements')
plt.legend(loc=4)
plt.xlabel('x'); plt.ylabel('y')
_ = plt.title('Noisy Samples of a Cubic Function')
```

As a final note, I implemented$\left(A^TA\right)^{-1}A^T$ directly to emphasize the underlying equation, but this is known as the pseudoinverse of $A$ and is implemented in the NumPy linear algebra package as [pinv](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.pinv.html).

```python
from numpy.linalg import pinv
```

```python
np.allclose(pinv(A), inv(A.T.dot(A)).dot(A.T))
```

    True


# Mathematics for Machine Learning: Linear Algebra
## Week3
## Module 3:

### Matrices, vectors, and solving simultaneous equation problems

#### Motivations for linear algebra

```python
from sympy import solve, Poly, Eq, Function, exp

from sympy.abc import x, y, z, a, b
```

$2a + 3b = 8$

$10a +1b = 13$

```python
solve((2 * a + 3* b- 8, 10 * a + b - 13), a, b)
```

    {a: 31/28, b: 27/14}

$\begin{pmatrix} 2 & 3 \\ 10 & 1 \end{pmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 8 \\ 13 \end{bmatrix}$

## How matrices transform space

```python
import numpy as np
m = np.array([[2 ,3],[10,1]])
v = np.array([[31/28], [27/14]])
```

```python
m@v
```

    array([[ 8.],
           [13.]])

```python
v.T
```

    array([1.10714286, 1.92857143])

```python
m = np.array([[7 ,-6],[12,8]])
```

```python
v1 = np.array([7, 12]); v2 = np.array([-6,8])
```

```python
5*v1 + 6 * v2
```

    array([ -1, 108])

```python
m @ [5,6]
```

    array([ -1, 108])

# Type of Matrix Transformation

```python
I = np.eye(2);I # Identity matrix
```

    array([[1., 0.],
           [0., 1.]])

```python
Scale = np.diag([3,2]); Scale # scale matrix
```

    array([[3, 0],
           [0, 2]])

```python
Scale2 = np.diag([-1,2]); Scale2 # scale matrix
```

    array([[-1,  0],
           [ 0,  2]])

### rotation matrix

```python
theta = np.pi/6
c, s = np.cos(theta), np.sin(theta)
R = np.array([[c, -s], [s, c]]);R
```

    array([[ 0.8660254, -0.5      ],
           [ 0.5      ,  0.8660254]])

### Composition or combination of matrix transformations

```python
A1 = R
```

```python
A2 = np.array([[-1, 0.0], [0.0, 1]]) #vertical mirror
```

```python
A21=A2@A1
np.where(abs(A21)>1e-15,A21,0).round()
```

    array([[ 0., -1.],
           [-1.,  0.]])

```python
A12 = A1@A2
np.where(abs(A12)>1e-15,A12,0).round()
```

    array([[0., 1.],
           [1., 0.]])

#### Matrix multipication is not commutative

#### but Matrix multipication is associative

# Using matrices to make transformations
### Practice Quiz

```python
A = np.array([[1/2, -1], [0, 3/4]]); A
```

    array([[ 0.5 , -1.  ],
           [ 0.  ,  0.75]])

```python
A@[-2, 4]
```

    array([-5.,  3.])

```python
M= np.array([[1, 0], [0, 8]])@ np.array([[1, 0], [-0.5, 1]]);M
```

    array([[ 1.,  0.],
           [-4.,  8.]])

# Solving the apples and bananas problem: Gaussian elimination

```python
M = np.array([[1,1,3],[1, 2, 4],[1, 1, 2]]); M
```

    array([[1, 1, 3],
           [1, 2, 4],
           [1, 1, 2]])

```python
from scipy.linalg import lu
pl, u = lu(M, permute_l=True); u
```

    array([[ 1.,  1.,  3.],
           [ 0.,  1.,  1.],
           [ 0.,  0., -1.]])

```python
pl@u
```

    array([[1., 1., 3.],
           [1., 2., 4.],
           [1., 1., 2.]])

```python
b = np.array([15, 21, 13])
x = np.linalg.solve(M, b);x
```

    array([5., 4., 2.])

```python
I1 = np.array([1, 0, 0])
b1 = np.linalg.solve(M, I1);b1
```

    array([ 0., -2.,  1.])

```python
I2 = np.array([0, 1, 0])
b2 = np.linalg.solve(M, I2);b2
```

    array([-1.,  1., -0.])

```python
I3 = np.array([0, 0, 1])
b3 = np.linalg.solve(M, I3);b3
```

    array([ 2.,  1., -1.])

```python
Minv = np.array([b1,b2,b3]).T;Minv
```

    array([[ 0., -1.,  2.],
           [-2.,  1.,  1.],
           [ 1., -0., -1.]])

```python
M@Minv
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.]])

```python
np.linalg.inv(M) == Minv
```

    array([[ True,  True,  True],
           [ True,  True,  True],
           [ True,  True,  True]])

### Practice Quiz

```python
M = np.array([[4,6,2],[3, 4, 1],[2, 8, 13]]); M
```

    array([[ 4,  6,  2],
           [ 3,  4,  1],
           [ 2,  8, 13]])

```python
b = np.array([9, 7, 2])
x = np.linalg.solve(M, b);x.round(1)
```

    array([ 3. , -0.5, -0. ])

5)

```python
A = np.array([[1, 1, 1],
             [3, 2, 1],
             [2, 1, 2]])
s = np.array([15, 28, 23])
p = np.linalg.solve(A, s);p.round(1)
```

    array([3., 7., 5.])

```python
np.linalg.inv(A).round(2)
```

    array([[-1.5,  0.5,  0.5],
           [ 2. ,  0. , -1. ],
           [ 0.5, -0.5,  0.5]])

# LAB

# Identifying special matrices
## Instructions
In this assignment, you shall write a function that will test if a 4×4 matrix is singular, i.e. to determine if an inverse exists, before calculating it.

You shall use the method of converting a matrix to echelon form, and testing if this fails by leaving zeros that can’t be removed on the leading diagonal.

Don't worry if you've not coded before, a framework for the function has already been written.
Look through the code, and you'll be instructed where to make changes.
We'll do the first two rows, and you can use this as a guide to do the last two.

### Matrices in Python
In the *numpy* package in Python, matrices are indexed using zero for the top-most column and left-most row.
I.e., the matrix structure looks like this:
```python
A[0, 0]  A[0, 1]  A[0, 2]  A[0, 3]
A[1, 0]  A[1, 1]  A[1, 2]  A[1, 3]
A[2, 0]  A[2, 1]  A[2, 2]  A[2, 3]
A[3, 0]  A[3, 1]  A[3, 2]  A[3, 3]
```
You can access the value of each element individually using,
```python
A[n, m]
```
which will give the n'th row and m'th column (starting with zero).
You can also access a whole row at a time using,
```python
A[n]
```
Which you will see will be useful when calculating linear combinations of rows.

A final note - Python is sensitive to indentation.
All the code you should complete will be at the same level of indentation as the instruction comment.

### How to submit
Edit the code in the cell below to complete the assignment.
Once you are finished and happy with it, press the *Submit Assignment* button at the top of this notebook.

Please don't change any of the function names, as these will be checked by the grading script.

If you have further questions about submissions or programming assignments, here is a [list](https://www.coursera.org/learn/linear-algebra-machine-learning/discussions/weeks/1/threads/jB4klkn5EeibtBIQyzFmQg) of Q&A. You can also raise an issue on the discussion forum. Good luck!

```python
# GRADED FUNCTION
import numpy as np

# Our function will go through the matrix replacing each row in order turning it into echelon form.
# If at any point it fails because it can't put a 1 in the leading diagonal,
# we will return the value True, otherwise, we will return False.
# There is no need to edit this function.
def isSingular(A) :
    B = np.array(A, dtype=np.float_) # Make B as a copy of A, since we're going to alter it's values.
    try:
        fixRowZero(B)
        fixRowOne(B)
        fixRowTwo(B)
        fixRowThree(B)
    except MatrixIsSingular:
        return True
    return False

# This next line defines our error flag. For when things go wrong if the matrix is singular.
# There is no need to edit this line.
class MatrixIsSingular(Exception): pass

# For Row Zero, all we require is the first element is equal to 1.
# We'll divide the row by the value of A[0, 0].
# This will get us in trouble though if A[0, 0] equals 0, so first we'll test for that,
# and if this is true, we'll add one of the lower rows to the first one before the division.
# We'll repeat the test going down each lower row until we can do the division.
# There is no need to edit this function.
def fixRowZero(A) :
    if A[0,0] == 0 :
        A[0] = A[0] + A[1]
    if A[0,0] == 0 :
        A[0] = A[0] + A[2]
    if A[0,0] == 0 :
        A[0] = A[0] + A[3]
    if A[0,0] == 0 :
        raise MatrixIsSingular()
    A[0] = A[0] / A[0,0]
    return A

# First we'll set the sub-diagonal elements to zero, i.e. A[1,0].
# Next we want the diagonal element to be equal to one.
# We'll divide the row by the value of A[1, 1].
# Again, we need to test if this is zero.
# If so, we'll add a lower row and repeat setting the sub-diagonal elements to zero.
# There is no need to edit this function.
def fixRowOne(A) :
    A[1] = A[1] - A[1,0] * A[0]
    if A[1,1] == 0 :
        A[1] = A[1] + A[2]
        A[1] = A[1] - A[1,0] * A[0]
    if A[1,1] == 0 :
        A[1] = A[1] + A[3]
        A[1] = A[1] - A[1,0] * A[0]
    if A[1,1] == 0 :
        raise MatrixIsSingular()
    A[1] = A[1] / A[1,1]
    return A

# This is the first function that you should complete.
# Follow the instructions inside the function at each comment.
def fixRowTwo(A) :
    # Insert code below to set the sub-diagonal elements of row two to zero (there are two of them).
    A[2] = A[2] - A[2,0] * A[0]
    A[2] = A[2] - A[2,1] * A[1]
    # Next we'll test that the diagonal element is not zero.
    if A[2,2] == 0 :
        # Insert code below that adds a lower row to row 2.
        A[2] = A[2] + A[3]
        # Now repeat your code which sets the sub-diagonal elements to zero.
        A[2] = A[2] - A[2,0] * A[0]
        A[2] = A[2] - A[2,1] * A[1]

    if A[2,2] == 0 :
        raise MatrixIsSingular()
    # Finally set the diagonal element to one by dividing the whole row by that element.
    A[2] = A[2] / A[2,2]
    return A

# You should also complete this function
# Follow the instructions inside the function at each comment.
def fixRowThree(A) :
    # Insert code below to set the sub-diagonal elements of row three to zero.
    A[3] = A[3] - A[3,0] * A[0]
    A[3] = A[3] - A[3,1] * A[1]
    A[3] = A[3] - A[3,2] * A[2]

    # Complete the if statement to test if the diagonal element is zero.
    if A[3,3] == 0 :
        raise MatrixIsSingular()
    # Transform the row to set the diagonal element to one.
    A[3] = A[3] / A[3,3]
    return A
```

## Test your code before submission
To test the code you've written above, run the cell (select the cell above, then press the play button [ ▶| ] or press shift-enter).
You can then use the code below to test out your function.
You don't need to submit this cell; you can edit and run it as much as you like.

Try out your code on tricky test cases!

```python
A = np.array([
        [2, 0, 0, 0],
        [0, 3, 0, 0],
        [0, 0, 4, 4],
        [0, 0, 5, 5]
    ], dtype=np.float_)
isSingular(A)
```

    True

```python
A = np.array([
        [0, 7, -5, 3],
        [2, 8, 0, 4],
        [3, 12, 0, 5],
        [1, 3, 1, 3]
    ], dtype=np.float_)
fixRowZero(A)
```

    array([[ 1. ,  7.5, -2.5,  3.5],
           [ 2. ,  8. ,  0. ,  4. ],
           [ 3. , 12. ,  0. ,  5. ],
           [ 1. ,  3. ,  1. ,  3. ]])

```python
fixRowOne(A)
```

    array([[ 1.        ,  7.5       , -2.5       ,  3.5       ],
           [-0.        ,  1.        , -0.71428571,  0.42857143],
           [ 3.        , 12.        ,  0.        ,  5.        ],
           [ 1.        ,  3.        ,  1.        ,  3.        ]])

```python
fixRowTwo(A)
```

    array([[ 1.        ,  7.5       , -2.5       ,  3.5       ],
           [-0.        ,  1.        , -0.71428571,  0.42857143],
           [ 0.        ,  0.        ,  1.        ,  1.5       ],
           [ 1.        ,  3.        ,  1.        ,  3.        ]])

```python
fixRowThree(A)
```

    array([[ 1.        ,  7.5       , -2.5       ,  3.5       ],
           [-0.        ,  1.        , -0.71428571,  0.42857143],
           [ 0.        ,  0.        ,  1.        ,  1.5       ],
           [ 0.        ,  0.        ,  0.        ,  1.        ]])

```python

```


# Logistic Regression

```{note}
Logistic Regression = Logistic model + binary cross entropy loss.<br/>
For multi-class classification problem, we can use softmax regression.
```

## Model

For binary classification problem where $x \in \mathbb{R}^{d}$, $y \in \left\{0, 1\right\}$, we could approach the classification problem using linear regression ignoring the fact that $y$ is discrete. Howerver, it is easy to construct examples that performs poorly, it doesn't make sense for $h(x)$ outside $\left[0, 1\right]$.

To fix this, we used the logistic function(sigmoid) to force the result in $\left[0, 1\right]$ after the affine transformation:

$$h(x) = \frac{1}{1 + \mbox{exp}(\theta^{T}x)} = \sigma(\theta^{T}x)$$

## Entropy

Self information $I(x)$ indicates the amount of information an event $x$ to happen that satisfies:

1. $I(x) \ge 0$
2. $\text{if }p(x_{1}) > p(x_{2}) \text{, then } I(x_{1}) < I(x_{2})$
3. $I(x_{1}, x_{2}) = I(x_{1}) + I(x_{2}) \text{ for independent }x_{1},x_{2}$

This leads to $I(x) = -\log_{r}p(x)$, for convenient $I(x) := -\log{p(x)}$.

While self-information measures the information of a event, entropy measures the information of a random variable:

$$
H(X) = E(I(x)) = E(-\log{p(x)}) = -\sum_{x \in \mathcal{X}}\log{p(x)}
$$

It is exactly the optimal encoding length of $X$.

Cross entropy $H(p, q)$ is the encoding length of $p$ by optimal encoding of $q$:

$$H(p,q)=E_{p}\left[-\log{q(x)}\right] = -\sum_{x}p(x)\log{q(x)}$$

Fix $p$, the closer $q$ is to $p$, the less is $H(p,q)$. We can use $H(p,q)$ to define the distance from $q$ to $p$.

## Loss

Using the definition of cross entropy above, we interpret label $y^{(i)}$ as the distribution $p(y^{(i)}|x^{(i)}) = 1, p(1 - y^{(i)}|x^{(i)})=0$.

In the same manner, interpret the hypothesis as $q(y=1|x^{(i)}) = h(x^{(i)}), q(y=0|x^{(i)}) = 1 - h(x^{(i)})$.

Cross Entropy Loss from $q$ to $p$ measures the distance from hypothesis to label:

$$l_{\theta}(x^{(i)}) = -y^{(i)}\log(h(x^{(i)})) - (1 - y^{(i)})\log(1 - h(x^{(i)}))$$

Sum them up derive the cross entropy loss logistic regression uses:

$$J(\theta) = \sum_{i=1}^{n}\left[-y^{(i)}\log(h(x^{(i)})) - (1 - y^{(i)})\log(1 - h(x^{(i)}))\right]$$

## Probabilistic Interpretation

As we supposes:

$$p(y|x) = h(x)^{y}\cdot(1 - h(x))^{1 - y} $$

Log likelihood of the dataset:

$$
\begin{equation}
\begin{split}
L(\theta) &= \log\prod_{i=1}^{n} h(x^{(i)})^{y^{(i)}}\cdot(1 - h(x^{(i)}))^{1 - y^{(i)}} \\
&= \sum_{i=1}^{n}y^{(i)}\log(h(x^{(i)})) + (1 - y^{(i)})\log(1 - h(x^{(i)}))
\end{split}
\end{equation}
$$

So logistic regression $\Leftrightarrow$ MLE if we see $h(x)$ as $p(y=1|x)$.

## Update Rule

Gradient of logistic regression:

$$
\begin{equation}
\begin{split}
\frac{\partial }{\partial \theta_{j}}J(\theta )
&= \sum_{i=1}^{n} \left (-y^{(i)}\frac{1}{\sigma(\theta^{T}x^{(i)})} + (1 - y^{(i)})\frac{1}{1 - \sigma(\theta^{T}x^{(i)})} \right )\frac{\partial }{\partial \theta_{j}}\sigma(\theta^{T}x^{(i)})\\
&=\sum_{i=1}^{n} \left (-y^{(i)}\frac{1}{\sigma(\theta^{T}x^{(i)})} + (1 - y^{(i)})\frac{1}{1 - \sigma(\theta^{T}x^{(i)})} \right )\sigma(\theta^{T}x^{(i)})(1-\sigma(\theta^{T}x^{(i)}))\frac{\partial }{\partial \theta_{j}}\theta^{T}x^{(i)} \\
&=\sum_{i=1}^{n}(h_{\theta}(x^{(i)}) - y^{(i)})x_{j}^{(i)}
\end{split}
\end{equation}
$$

Combine all dimensions:

$$\theta \to \theta - \alpha\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot{x}^{(i)} $$

Write in matrix form:

$$\theta \to \theta - \alpha{X}^{T}(\sigma({X}{\theta})-{y}) $$

where ${X} \in \mathbb{R}^{n\times{d}}, {y} \in \mathbb{R}^{n}$.

## Examples

```python
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
```

```python
from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(random_state=0, max_iter=5000)
clf.fit(X, y)
clf.predict(X[:2, :])
```

    array([0, 0])

```python
# score return the mean accuracy on the given test data and labels.
clf.predict_proba(X[:2, :]), clf.score(X, y)
```

    (array([[1.00000000e+00, 3.16211740e-14],
            [9.99996140e-01, 3.86002382e-06]]),
     0.9578207381370826)

## Softmax Regression

For multi-class classification, we start off with a simple image classification problem, each input consists of a $2\times{2}$ grayscale image, represent each pixel with a scalar, giving us features $\left\{x_{1},x_{2},x_{3}, x_{4}\right\}$. assume each image belong to one among the categories "cat", "chiken" and "dog".

We have a nice way to represent categorical data: the one-hot encoding, for our problem, "cat" represents by $(1,0,0)$, "chicken" by $(0, 1, 0)$, "dog" by $(0, 0, 1)$.

To estimate the conditional probabilities of all classes, we need a model with multiple outputs, one per class:

$$o_{1} = x_{1}w_{11} + x_{2}w_{12} + x_{3}w_{13} + x_{4}w_{14}$$
$$o_{2} = x_{1}w_{21} + x_{2}w_{22} + x_{3}w_{23} + x_{4}w_{24}$$
$$o_{3} = x_{1}w_{31} + x_{2}w_{32} + x_{3}w_{33} + x_{4}w_{34}$$

depict as:

We would like $\hat{y}_{j}$ to be interpreted as probability that a given item belong to class $j$, to transform our current outputs $\left\{o_{1},o_{2},o_{3},o_{4}\right\}$ to probability distribution $\left\{\hat{y}_{1},\hat{y}_{2},\hat{y}_{3},\hat{y}_{4}\right\}$, just use the softmax operation:

$$\hat{y}_{j} = \frac{\exp(o_{j})}{\sum_{k}\exp(o_{k})}$$

As logistic regression, we use the cross entropy loss:

$$H(y,\hat{y}) = -\sum_{k}y_{j}\log\hat{y}_{j} = -\log\hat{y}_{\text{category of y}}$$

Now complete the construction of softmax regression.

```python
"""multi-class classification problem"""
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
y
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

```python
"""
Set multi_class='multinomial' in LogisticRegression
"""
softmax_reg = LogisticRegression(multi_class="multinomial", solver="lbfgs", C=10, max_iter=1000)
softmax_reg.fit(X, y)
softmax_reg.predict(X[:3, :])
```

    array([0, 0, 0])


# Numerical Recipes Workshop 4
For the week of 14-18 October, 2019

This notebook will cover some basics of ODE solving and advanced plotting techniques.

```python
from matplotlib import pyplot as plt
%matplotlib inline
import numpy as np
```

```python
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['font.size'] = 14
```

## Solving ODEs
In this activity, you will implement and experiment with some ODE integration methods for initial value problems. You will then learn to use SciPy's [solve_ivp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) function for doing this more generally.

In each case, you will solve an ODE of the form:

$
\begin{align}
\frac{dy}{dt} = f(t, y)
\end{align}
$

A straightforward way to code this is to implement a Python function for $f(t, y)$. We will do this below for

$
\begin{align}
\frac{dy}{dt} = cos(t)
\end{align}
$

### Define the function
For reasons that will be clear later, the order of arguments should be $t$ first and $y$ second.

```python
def dy_dt(t, y):
    return np.cos(t)
```

### Define a step size
We will use $h$ to denote the step size. We will integrate from $t$ = 0 to 20. The `ts` and `ys` arrays will store the values of $t$ and the solution.

```python
h = 0.25
ts = np.arange(0, 20+h, h)
ys = np.empty(ts.size)
```

### Direct Euler

$
\begin{align}
y(t+h) = y(t) + h f(t, y)
\end{align}
$

This becomes:

$
\begin{align}
y_{n+1} = y_{n} + h f(t_{n}, y_{n})
\end{align}
$

```python
# Set the inital value
ys[0] = 0.5

# Now integrate
for i in range(1, ts.size):
    tn = ts[i-1]
    yn = ys[i-1]
    ys[i] = yn + h * dy_dt(tn, yn)
```

```python
plt.plot(ts, ys)
plt.xlabel('t')
plt.ylabel('y')
```

### Calculate the error
The above problem has the analytic solution

$
\begin{align}
y(t) = sin(t) + 0.5
\end{align}
$

We will compare to that to measure the error.

```python
yexact = np.sin(ts) + 0.5
```

```python
plt.plot(ts, yexact, label='exact')
plt.plot(ts, ys, label=f'h = {h}')
plt.legend(loc='best')
```

```python
err = np.abs(yexact - ys)
err_max = np.amax(err)
plt.plot(ts, err)
plt.xlabel('t')
plt.ylabel('error')
print(err_max)
```

```python
hs = np.arange(0.01, 1.0, 0.01)
err_array = np.array([])

for h in hs:
    ts = np.arange(0, 20+h, h)
    ys = np.empty(ts.size)

    # Set the inital value
    ys[0] = 0.5

    for i in range(1, ts.size):
        tn = ts[i-1]
        yn = ys[i-1]
        ys[i] = yn + h * dy_dt(tn, yn)

    yexact = np.sin(ts) + 0.5

    err = np.abs(yexact - ys)

    err_array = np.append(err_array, np.amax(err))

plt.plot(hs, err_array)
plt.xlabel('h')
plt.ylabel('error')
```

Record $h$ and the maximum value of the error and make a plot of error vs. step size. How does the error change with step size for the Direct Euler method?

### 2nd Order Runga-Kutta

$
\begin{align}
k_{1} = h f(t, y),\\
\end{align}
$

$
\begin{align}
k_{2} = h f(t + \frac{h}{2}, y + \frac{k_{1}}{2})
\end{align}
$

$
\begin{align}
y(t+h) = y(t) + k_{2}
\end{align}
$

```python
hs = np.arange(0.01, 1.0, 0.01)
err_array_RK2 = np.array([])

for h in hs:
    ts = np.arange(0, 20+h, h)
    ys = np.empty(ts.size)

    # Set the inital value
    ys[0] = 0.5

    # Now integrate
    for i in range(1, ts.size):
        tn = ts[i-1]
        yn = ys[i-1]
        k1 = h * dy_dt(tn, yn)
        k2 = h * dy_dt(tn + h/2, yn + k1/2)
        ys[i] = yn + k2

    yexact = np.sin(ts) + 0.5

    err = np.abs(yexact - ys)

    err_array_RK2 = np.append(err_array_RK2, np.amax(err))

plt.plot(ts, yexact, label='exact')
plt.plot(ts, ys, label=f'h = {h}')
plt.legend(loc='best')
```

```python
plt.plot(hs, err_array_RK2)
plt.xlabel('h')
plt.ylabel('error - RK2')
```

Plot the integrated and analytic solutions and calculate the error as a function of $h$. Compare this with the Direct Euler method.

### 4th Order Rung-Kutta
$
\begin{align}
k_{1} = h f(t, y),\\
\end{align}
$

$
\begin{align}
k_{2} = h f(t + \frac{h}{2}, y + \frac{k_{1}}{2})
\end{align}
$

$
\begin{align}
k_{3} = h f(t + \frac{h}{2}, y + \frac{k_{2}}{2})
\end{align}
$

$
\begin{align}
k_{4} = h f(t + h, y + k_{3})
\end{align}
$

$
\begin{align}
y(t+h) = y(t) + \frac{k_{1}}{6} + \frac{k_{2}}{3} + \frac{k_{3}}{3} + \frac{k_{4}}{6}
\end{align}
$

Implement this method yourself for the same initial value problem. Make the same plots and compare the error with the DE and RK23 methods.

```python
hs = np.arange(0.01, 1.0, 0.01)
err_array_RK4 = np.array([])

for h in hs:
    ts = np.arange(0, 20+h, h)
    ys = np.empty(ts.size)

    # Set the inital value
    ys[0] = 0.5

    # Now integrate
    for i in range(1, ts.size):
        tn = ts[i-1]
        yn = ys[i-1]
        k1 = h * dy_dt(tn, yn)
        k2 = h * dy_dt(tn + h/2, yn + k1/2)
        k3 = h * dy_dt(tn + h/2, yn + k2/2)
        k4 = h * dy_dt(tn + h, yn + k3)

        ys[i] = yn + k1/6 + k2/3 + k3/3 + k4/6

    yexact = np.sin(ts) + 0.5

    err = np.abs(yexact - ys)

    err_array_RK4 = np.append(err_array_RK4, np.amax(err))
```

```python
plt.plot(hs, err_array, label = 'Euler')
plt.plot(hs, err_array_RK2, label = 'RK2')
plt.plot(hs, err_array_RK4, label = 'RK4')
plt.legend()
plt.grid()
plt.xlabel('h')
plt.ylabel('error - RK4')
```

### The SciPy `solve_ivp` function
SciPy's [solve_ivp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) can solve IVPs using a variety of methods. The user must provide a function that accepts $t, y$ in that order as well an initial value, and range over which to solve.

```python
from scipy.integrate import solve_ivp
```

```python
h = 0.05
ts = np.arange(0, 50+h, h)
sol_range = (ts[0], ts[-1])
y0 = 1
```

```python
sol = solve_ivp(dy_dt, sol_range, [y0], t_eval=ts, method='RK45')
```

```python
plt.plot(sol.t, sol.y[0])
plt.xlabel('t')
plt.ylabel('y')
```

Now solve the following IVP:

$
\begin{align}
\large
\frac{dy}{dt} = y\ cos(t),\\
\end{align}
$

$
\begin{align}
\large
y(0) = 1
\end{align}
$

This has the analytic solution:

$
\begin{align}
\large
y(t) = e^{sin(t)}
\end{align}
$

Try a few different methods using the `method` and `max_step` keywords to experiment with different solvers and step sizes.

```python
def dy_dt_2(t,y):
    return y*np.cos(t)
h = 0.05
ts = np.arange(0, 50+h, h)
sol_range = (ts[0], ts[-1])
y0 = 1
yexact = np.exp(np.sin(ts))

sol = solve_ivp(dy_dt_2, sol_range, [y0], t_eval=ts, method='RK45')

plt.plot(sol.t, sol.y[0])
plt.xlabel('t')
plt.ylabel('y - solution to dy_dt_2')

```

## Solving a sytem of equations
The `solve_ivp` function can solve a system of ODEs just as easily as a single ODE. This is done by creating a function for `dy/dt` that returns multiple values.

Consider the problem of parabolic motion, i.e., a projectile launched into the air with an initial velocity, feeling only the force of gravity (no air resistance). We can define the follow system of equations:

$
\begin{align}
\large
\frac{dx}{dt} = v_{x}
\end{align}
$

$
\begin{align}
\large
\frac{dy}{dt} = v_{y}
\end{align}
$

$
\begin{align}
\large
\frac{dv_{x}}{dt} = 0
\end{align}
$

$
\begin{align}
\large
\frac{dv_{y}}{dt} = -g
\end{align}
$

We can then create a function to give to `solve_ivp` as follows:

```python
g = 9.80665 # m/s^2
def projectile_motion(t, f):
    """
    f0 = dx/dt  = vx
    f1 = dy/dt  = vy
    f2 = dvx/dt = 0
    f3 = dvy/dt = - g / m
    """

    vals = np.zeros(4)
    vals[0] = f[2]
    vals[1] = f[3]
    vals[2] = 0
    vals[3] = - g

    return vals
```

We must provide an initial value for each of the equations. Assume a starting position of (0, 0) with some initial velocity, $v_{i}$, with an angle, $\theta$, with the ground.

```python
vi = 100 # m/s
theta = 30 * np.pi / 180 # 30 degrees
vx = vi * np.cos(theta)
vy = vi * np.sin(theta)
fi = np.array([0, 0, vx, vy])
```

Additionally, we can provide a function to determine when the projectile hits the ground. The function below returns the value of the $y$ coordinate. A root-finding method will then determine the time when this function returns 0, i.e., the ball is on the ground.

The `terminal` attribute will instruct the solver to end there, even if the full time interval has not been evaluated. The `direction` attribute can be used to filter out solutions we don't want, i.e., the initial value.

```python
def object_lands(t, f):
    return f[1]
object_lands.terminal = True
object_lands.direction = -1
```

```python
# Set t values
ts = np.linspace(0, 20, 401)
trange = (ts[0], ts[-1])
```

```python
# Solve!
sol = solve_ivp(projectile_motion, trange, fi, events=(object_lands), t_eval=ts, dense_output=True)
```

Now, plot x position vs. y position. We could also plot $v_{x}$, $v_{y}$, or $t$.

```python
plt.plot(sol.y[0], sol.y[1])
plt.xlabel('x [m]')
plt.ylabel('y [m]')
```

```python

```

## 3D and Advanced Plotting
Lines and points are great, but often insufficient for visualizing multi-dimensional data. Matplotlib provides a [3D plotting toolkit](https://matplotlib.org/3.1.1/tutorials/toolkits/mplot3d.html) for a variety of 3D plots.

Let's return to the dust temperature example from Workshop 3. If you are not familiar with this, go back to the Workshop 3 notebook and read the short text on computing the temperature of interstellar dust grains.

The functions for calculating the dust temperature are given below. Previously, you solved for $T_{dust}$ for a range of densities ($n_{H}$) and a single gas temperature ($T_{gas}$). Now, let's do the same exercise for a range of gas temperatures as well. The code is given below.

```python
mh = 1.673735e-24 # g
# Stefan-Boltzmann constant
sigma_b = 5.670373e-5 # erg cm^−2 s^−1 K^−4

def gas_grain(Tgas):
    """
    Return gas/grain heat transfer rate coefficient.
    """

    grain_coef = 1.2e-31 * 1.0e3**-0.5 / mh
    gasgra = grain_coef * Tgas**0.5 * \
        (1.0 - (0.8 * np.exp(-75.0 / Tgas)))
    return gasgra

def kappa_grain(Tdust):
    """
    Return grain mean opacity.
    """

    kgr1   = 4.0e-4
    kgr200 = 16.0
    T_subl = 1500.

    Tdust = np.asarray(Tdust)
    kgr = np.zeros(Tdust.size)

    f1 = Tdust < 200
    if f1.any():
        kgr[f1] = kgr1 * Tdust[f1]**2

    kgr[(Tdust >= 200) & (Tdust < T_subl)] = kgr200

    f2 = Tdust >= T_subl
    if f2.any():
        kgr[f2] = kgr200 * (Tdust[f2] / T_subl)**-12

    return kgr

def gamma_isrf():
    """
    Interstellar radiation field heating rate coefficient.
    """

    return 4.154682e-22 / mh

def gamma_grain(Tdust, Tgas, nh, isrf=1.7, z=0):
    """
    Return the grain heating rate.

    Parameters
    ----------

    Tdust : float
        dust temperature in K
    Tgas : float
        gas temperature in K
    nh : float
        Hydrogen number density in cm^-3
    isrf : float, optional
        interstellar radiation field strengh in Habing units
        default: 1.7 (typical for local interstellar medium)
    z : float, optional
        current redshift, used to set the temperature of the
        Cosmic Microwave Background.
        default: 0
    """

    TCMB = 2.73 * (1 + z)
    my_isrf = isrf * gamma_isrf()

    return my_isrf + \
        4 * sigma_b * kappa_grain(Tdust) * (TCMB**4 - Tdust**4) + \
        (gas_grain(Tgas) * nh * (Tgas - Tdust))
```

```python
import scipy.optimize as opt
```

```python
# Create arrays of Tgas and nH
Tgas = np.logspace(1, 3.5, 61)
nH = np.logspace(0, 13, 131)
```

Now, calculate the dust temperature for all values of $n_{H}$ and $T_{dust}$. This will take a minute.

```python
Tdust = np.empty((nH.size, Tgas.size))
for i in range(nH.size):
    for j in range(Tgas.size):
        Tdust[i,j] = opt.brentq(gamma_grain, 1, 1e4, args=(Tgas[j], nH[i]))
```

### Plotting $T_{dust}$ as a surface

The dust temperature can be visualized in 3D as a surface in the space ($n_{H}$, $T_{gas}$). In order to give the surface some texture for a more intuitive appearance, we can also apply a colormap to color the surface according to the heigh in the z axis.

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
```

```python
fig = plt.figure(figsize=(16, 10))
ax = fig.gca(projection='3d')

# Make data.
# Put data in log-space to view the large dynamic range.
X, Y = np.meshgrid(np.log10(nH), np.log10(Tgas))

# Note, we transpose the Tdust array.
Z = np.log10(Tdust.T)

# Plot the surface.
surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)

# Customize the z axis.
# ax.set_zlim(-1.01, 1.01)
#ax.zaxis.set_major_locator(LinearLocator(10))
#ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
ax.xaxis.set_label_text('log ($n_{H} / cm^{-3}$)')
ax.xaxis.labelpad = 10
ax.yaxis.set_label_text('log ($T_{gas} / K$)')
ax.yaxis.labelpad = 10
ax.zaxis.set_label_text('log ($T_{dust} / K$)')
ax.zaxis.labelpad = 10

# Add a color bar which maps values to colors.
cb = fig.colorbar(surf, shrink=0.5, aspect=5, label='log ($T_{dust}$ / K)')
plt.show()
```

### Plotting $T_{dust}$ as a color mesh
There are obvious disadvantages to plotting $T_{dust}$ as a surface, i.e., not all points can be seen or apparent values can depend somewhat on perspective. Since the z height and the color are showing the same information, we can eliminate the height component and view the above plot from the top down using the [pcolormesh](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.pcolormesh.html#matplotlib.pyplot.pcolormesh) from matplotlib.

```python
plt.pcolormesh(X, Y, Z, cmap=cm.viridis) # change to cmap=cm.coolwarm to see same colormap as above
plt.colorbar(label='log ($T_{dust} / K$)')
plt.xlabel('log ($n_{H} / cm^{-3}$)')
plt.ylabel('log ($T_{gas} / K$)')
```

## Baseball: the objectively most interesting spectator sport
Baseball is undergoing a "launch angle revolution." Players have been traditionally coached to hit the ball low to the ground in order to get "line-drive" hits and to avoid hitting the ball into the air where it can be easily caught for an out. However, with the advent of technology to measure the speed of the batted ball off the bat (known as "exit-velocity"), analysis showed that many players hit the ball with enough power to be hitting home runs (the ball is hit out of the park) regularly. Thus, players are now adjusting their swings to hit the ball with a more upward trajectory.

Batted baseballs experience enough air resistance to alter their paths from a simple parabola. To compute the distance traveled by a baseball, we must consider the drag force, which is defined as

$
\begin{align}
F_{D} = \frac{1}{2} C_{D} A \rho v^{2},
\end{align}
$

where $C_{D}$ is the drag coefficient, $A$ is the cross-sectional area of the ball, $\rho$ is the density of air, and $v$ is the velocity of the ball.

Add the drag force to the projectile motion problem to compute the travel distance a batted ball for a given exit velocity and launch angle. The relevant constants are given below.

```python
# baseballs
m = 0.145 # kg
c = 23.2 # cm
r = c / 2 / np.pi
A = np.pi * (r)**2 / 10000 # m^2
Cd = 0.346
```

```python
# Earth-related constants
rhoE = 1.19657921 # kg/m^3
g = 9.80665 # m/s^s
```

```python
def baseball_flight(t, f):
    """
    f0 = dx/dt  = vx
    f1 = dy/dt  = vy
    f2 = dvx/dt = F_x / m
    f3 = dvy/dt = F_y / m - g
    """
    v_mag = np.sqrt(np.power(f[2],2)*np.power(f[3],2))
    Fx_hat = (-1/v_mag) * (f[2])
    Fy_hat = (-1/v_mag) * (f[3])
    F_x = .5 * Cd * A * rhoE * v_mag *Fx_hat
    F_y = .5 * Cd * A * rhoE * v_mag *Fy_hat

    vals = np.zeros(4)
    vals[0] = f[2]
    vals[1] = f[3]
    vals[2] = F_x / m
    vals[3] = F_y / m - g

    return vals
```

### Initial values
The average well-hit ball has an exit velocity of about 100 miles/hour, or about 45 m/s. Assume a launch angle of 30 degrees and that the ball is hit by the batter from a height of 1 m.

```python
vi = 45 # m/s
theta = 30 * np.pi / 180 # 30 degrees
vx = vi * np.cos(theta)
vy = vi * np.sin(theta)
fi = np.array([0, 1, vx, vy])
```

### Is it a home run?
The average outfield wall is about 100 m from the batter and about 2 m high. A ball must reach at least this point to be considered a homerun.

Baseballs will have a total flight time of less than 10 seconds.

Implement an event function to determine when the ball has returned to a height of 2 m.

```python
def ball_lands(t, f):
        return f[1]

ball_lands.terminal = True
ball_lands.direction = -1
```

```python
ts = np.linspace(0, 10, 201)
sol = solve_ivp(baseball_flight, (0, 10), fi, events=(ball_lands), t_eval=ts, dense_output=True)
```

Plot the trajectory of the batted ball. An outfield wall has been provided for you.

```python
plt.plot(sol.y[0], sol.y[1], label='baseball')
# Outfield wall
plt.plot([100, 100, 110], [0, 2, 2], label='outfield wall', color='red')
plt.legend(loc='best')
```

### Was it a homerun?
The `ball_lands` event will record the time for the ball to return to a height of 2 m. The `sol.sol` command can then be used to determine the values for each of the equations in the series at the given time.

```python
print (f'Flight time: {sol.t_events[0][0]} s.')
value = sol.sol(sol.t_events[0])
print (f'Distance: {value[0][0]} m.')
```

### Optimal launch angle
Plot the distance traveled in the x direction as a function of exit velocity and launch angle for exit velocities from 90 to 115 mph (make sure to convert to m/s). What is the optimal launch angle? What is the minimum exit velocity needed to hit a home run?

```python

```

### Chicago: the windy city
Wrigley Field, home of the Chicago Cubs, is well-known for fluctuating significantly between being friendly to hitters or pitchers, depending on the direction of the wind. Experiment with adding a 15 mph wind heading either in (against the direction of the batted ball) or out to see how this changes things for the problem above.

```python

```


# Path Planning for a Simple Car

## Required Installations

If run on Google Colab, it is necessary to install any needed solvers for each Colab session. The following cell tests if the notebook is run on Google Colab, then installs Pyomo and Ipopt if not already installed.

```python
try:
    import google.colab
    try:
        from pyomo.environ import *
    except:
        !pip install -q pyomo
    if not 'ipopt_executable' in vars():
        !wget -N -q "https://ampl.com/dl/open/ipopt/ipopt-linux64.zip"
        !unzip -o -q ipopt-linux64
        ipopt_executable = '/content/ipopt'
except:
    pass
```

## Kinematic Model

The following equations describe a simple model of a car

\begin{align}
\frac{dx}{dt} & = v \cos(\theta) \\
\frac{dy}{dt} & = v \sin(\theta) \\
\frac{d\theta}{dt} & = \frac{v}{L}\tan(\phi) \\
\end{align}

where $x$ and $y$ denote the position of the center of the rear axle, $\theta$ is the angle of the car axis to the horizontal, $v$ is velocity, and $\phi$ is the angle of the front steering wheels to the car axis. The length $L$ is the distance from the center of the rear axle to the center of the front axle.

The velocity $v$ is controlled by acceleration of the car, the position of the wheels is controlled by the rate limited steering input $v$.

\begin{align}
\frac{dv}{dt} & = a \\
\frac{d\phi}{dt} & = u
\end{align}

The state of the car is determined by the value of the five state variables $x$, $y$, $\theta$, $v$, and $\phi$.

The path planning problem is to find find values of the manipulable variables $a(t)$ and $u(t)$ on a time interval $0 \leq t \leq t_f$ to drive the car from an initial condition $\left[x(0), y(0), \theta(0), v(0), \phi(0)\right]$ to a specified final condition $\left[x(t_f), y(t_f), \theta(t_f), v(t_f), \phi(t_f)\right]$ that minimizes an objective function:

\begin{align}
J  = \min \int_0^{t_f} \left( \phi(t)^2 + \alpha a(t)^2 + \beta u(t)^2\right)\,dt
\end{align}

and which satisfy operational constraints

\begin{align*}
| u | & \leq u_{max}
\end{align*}

## Pyomo Model

```python
from pyomo.environ import *
from pyomo.dae import *

L = 2
tf = 50

# create a model object
m = ConcreteModel()

# define the independent variable
m.t = ContinuousSet(bounds=(0, tf))

# define control inputs
m.a = Var(m.t)
m.u = Var(m.t, domain=Reals, bounds=(-0.1,0.1))

# define the dependent variables
m.x = Var(m.t)
m.y = Var(m.t)
m.theta = Var(m.t)
m.v = Var(m.t)
m.phi = Var(m.t, domain=Reals, bounds=(-0.5,0.5))

m.xdot = DerivativeVar(m.x)
m.ydot = DerivativeVar(m.y)
m.thetadot = DerivativeVar(m.theta)
m.vdot = DerivativeVar(m.v)
m.phidot = DerivativeVar(m.phi)

# define the differential equation as a constraint
m.ode_x = Constraint(m.t, rule=lambda m, t: m.xdot[t] == m.v[t]*cos(m.theta[t]))
m.ode_y = Constraint(m.t, rule=lambda m, t: m.ydot[t] == m.v[t]*sin(m.theta[t]))
m.ode_t = Constraint(m.t, rule=lambda m, t: m.thetadot[t] == m.v[t]*tan(m.phi[t])/L)
m.ode_u = Constraint(m.t, rule=lambda m, t: m.vdot[t] == m.a[t])
m.ode_p = Constraint(m.t, rule=lambda m, t: m.phidot[t] == m.u[t])

# path constraints
m.path_x1 = Constraint(m.t, rule=lambda m, t: m.x[t] >= 0)
m.path_y1 = Constraint(m.t, rule=lambda m, t: m.y[t] >= 0)

# initial conditions
m.ic = ConstraintList()
m.ic.add(m.x[0]==0)
m.ic.add(m.y[0]==0)
m.ic.add(m.theta[0]==0)
m.ic.add(m.v[0]==0)
m.ic.add(m.phi[0]==0)

# final conditions
m.fc = ConstraintList()
m.fc.add(m.x[tf]==0)
m.fc.add(m.y[tf]==20)
m.fc.add(m.theta[tf]==0)
m.fc.add(m.v[tf]==0)
m.fc.add(m.phi[tf]==0)

# define the optimization objective
m.integral = Integral(m.t, wrt=m.t, rule=lambda m, t: 0.2*m.phi[t]**2 + m.a[t]**2 + m.u[t]**2)
m.obj = Objective(expr=m.integral)

# transform and solve
TransformationFactory('dae.collocation').apply_to(m, wrt=m.t, nfe=3, ncp=12, method='BACKWARD')
SolverFactory('ipopt', executable=ipopt_executable).solve(m).write()
```

## Accessing Solution Data

```python
# access the results
t= [t for t in m.t]

a = [m.a[t]() for t in m.t]
u = [m.u[t]() for t in m.t]

x = [m.x[t]() for t in m.t]
y = [m.y[t]() for t in m.t]
theta = [m.theta[t]() for t in m.t]
v = [m.v[t]() for t in m.t]
phi = [m.phi[t]() for t in m.t]
```

## Visualizing Car Path

```python
% matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

scl=0.3

def draw_car(x=0, y=0, theta=0, phi=0):
    R = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
    car = np.array([[0.2, 0.5], [-0.2, 0.5], [0, 0.5], [0, -0.5],
                    [0.2, -0.5], [-0.2, -0.5], [0, -0.5], [0, 0], [L, 0], [L, 0.5],
                    [L + 0.2*np.cos(phi), 0.5 + 0.2*np.sin(phi)],
                    [L - 0.2*np.cos(phi), 0.5 - 0.2*np.sin(phi)], [L, 0.5],[L, -0.5],
                    [L + 0.2*np.cos(phi), -0.5 + 0.2*np.sin(phi)],
                    [L - 0.2*np.cos(phi), -0.5 - 0.2*np.sin(phi)]])
    carz= scl*R.dot(car.T)
    plt.plot(x + carz[0], y + carz[1], 'k', lw=2)
    plt.plot(x, y, 'k.', ms=10)

plt.figure(figsize=(10,10))
for xs,ys,ts,ps in zip(x,y,theta,phi):
    draw_car(xs, ys, ts, scl*ps)
plt.plot(x, y, 'r--', lw=0.8)
plt.axis('square')
```

    UsageError: Line magic function `%` not found.

```python
plt.figure(figsize=(10,8))
plt.subplot(311)
plt.plot(t, a, t, u)
plt.legend(['Acceleration','Steering Input'])

plt.subplot(312)
plt.plot(t, phi, t, theta)
plt.legend(['Wheel Position','Car Direction'])

plt.subplot(313)
plt.plot(t, v)
plt.legend(['Velocity'])
```

```python

```


# 初始化环境

```python
from IPython.display import display, Math
from sympy import *
init_printing()

from helper import comparator_factory, comparator_eval_factory, comparator_method_factory

x,y,z = symbols('x y z')

comparator = comparator_factory('使用{}前:','使用后:')
method_comparator = comparator_method_factory('调用{}前:','调用后:')
eval_comparator = comparator_eval_factory('计算前:','计算后:')
```

# 微积分

这部分设计如何进行基本的微积分操作，比如微分，积分，求极限，级数展开。

## 微分
用```diff()```求微分

### 基本的微分
传入表达式和应用微分的符号变量。

```python
expr = sin(x)

expr_diff = diff(expr,x)

comparator(expr, diff, x)
```

也可以用方法调用的方式计算微分

```python
expr = sin(x)

method_comparator(expr, 'diff', x)
```

如果要创建一个未执行计算的微分， 可以使用```Derivative```类。 初始化的语法和```diff()```是一样的。

```python
expr = sin(x)

diff_expr = Derivative(expr,x)

print('Before evaluation:')
display(diff_expr)
```

然后调用```doit()```方法执行计算。

```python
print('After evaluation:')
display(diff_expr.doit())
```

为了避免冗余代码， 后面的笔记我们使用```eval_comparator```来进行比较。

### 高阶微分
如果要计算n次高阶微分， 传入符号n次或者在符号后面传入n。

```python
expr = Derivative(x**4,x,x,x)

eval_comparator(expr)
```

用第二种方式达到同样的效果。

```python
expr = Derivative(x**4,x,3)

eval_comparator(expr)
```

### 高阶偏微分

只要按照顺序传入符号就可以， 语法和一阶微分一样。

```python
expr = Derivative(exp(x*y*z),x, y, y, z, z, z, z)

eval_comparator(expr)
```

或者通过数字控制每个符号的微分阶数

```python
expr = Derivative(exp(x*y*z),x, y, 2, z, 4)

eval_comparator(expr)
```

# 积分

## 不定积分
和微分类似，积分可以通过```integral()```函数或者方法来实现。 如果要创建未计算的积分表达式，初始化一个```Integral```类然后调用```doit()```进行计算。

```python
expr = Integral(cos(x),x)

eval_comparator(expr)
```

### 定积分

传入一个包含符号，积分下限， 积分上线的tuple去进行定积分。

```python
expr = Integral(exp(-x),(x,0,oo))

eval_comparator(expr)
```

注意，在Sympy中$\infty$用oo表示(两个小写的'O')

### 多重积分
传入多个包含符号和积分限的tuple进行多重积分。

```python
expr = Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))

eval_comparator(expr)
```

如果Sympy无法计算一个表达式的积分，机会返回未计算的积分式。

```python
expr = Integral(x**x)

eval_comparator(expr)
```

# 极限

和微分一样， 极限可以用过```limit()```函数或者方法进行微分计算。 如果要创建未计算的表达式，初始化一个```Limit```类， 然后通过调用```doit()```方法完成计算。

默认情况下， ```dir = '+'```， 极限从右侧计算。

```python
expr = Limit(sin(x)/x, x, 0)

eval_comparator(expr)
```

如果要计算左侧极限， 可以传入```dir='-'```

```python
expr = Limit(sin(x)/x, x, 0, dir = '-')

eval_comparator(expr)
```

# 级数展开

通过调用```series()```方法， Sympy可以计算误差关于$O(x-x_0)^n$， 在点$x_0$处的渐进级数展开。

```python
expr = sin(x)

method_comparator(expr, 'series', x,0,6)
```

如果要去掉误差项，可以调用```removeO()```方法。

```python
print('Expanded expression without order term:')
expr.series(x,0,6).removeO()
```

# 参考资料
[Sympy Documentation](http://docs.sympy.org/latest/index.html)

# 相关文章
* [Sympy笔记I]({filename}0026_sympy_intro_1_ch.ipynb)
* [Sympy笔记II]({filename}0027_sympy_intro_2_ch.ipynb)
* [Sympy笔记III]({filename}0028_sympy_intro_3_ch.ipynb)
* [Sympy笔记IV]({filename}0029_sympy_intro_4_ch.ipynb)


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *
Image(url= "https://i.imgur.com/4HeGIe7.png")
```

```python
t = symbols("t")
Integral(sqrt(4*t+9))
```

$\displaystyle \int \sqrt{4 t + 9}\, dt$

```python
simplify(Integral(sqrt(4*t+9)).doit())

```

$\displaystyle \frac{\left(4 t + 9\right)^{\frac{3}{2}}}{6}$

```python
Image(url= "https://i.imgur.com/tAO2Ij3.png")

```

```python
Image(url= "https://i.imgur.com/kXlWcEb.png")
```

```python
x = symbols("x")
Integral(x**5*sin(x**6))
```

$\displaystyle \int x^{5} \sin{\left(x^{6} \right)}\, dx$

```python
print(Integral(x**5*sin(x**6)).doit())
```

    -cos(x**6)/6

```python
Image(url= "https://i.imgur.com/Mr3KaY2.png")
```

```python
Image(url= "https://i.imgur.com/RpUQmc0.png")
```

```python
from sympy import pi
Integral(cos(pi/x)/x**2)
```

$\displaystyle \int \frac{\cos{\left(\frac{\pi}{x} \right)}}{x^{2}}\, dx$

```python
print(Integral(cos(pi/x)/x**2).doit())
```

    -sin(pi/x)/pi

```python
Image(url= "https://i.imgur.com/Nj4pFIO.png")
```

```python
Image(url= "https://i.imgur.com/u7LsgcI.png")
```

```python
Integral(sin(x)**4*cos(x))
```

$\displaystyle \int \sin^{4}{\left(x \right)} \cos{\left(x \right)}\, dx$

```python
print(Integral(sin(x)**4*cos(x)).doit())
```

    sin(x)**5/5

```python
Image(url= "https://i.imgur.com/Bytu7tM.png")
```

```python
Image(url= "https://i.imgur.com/ZTp8zyi.png")
```

```python
Integral(x*cos(1-x**2))
```

$\displaystyle \int x \cos{\left(x^{2} - 1 \right)}\, dx$

```python
print(Integral(x*cos(1-x**2)).doit())
```

    sin(x**2 - 1)/2

```python
Image(url= "https://i.imgur.com/D72UOm8.png")
```

```python
Image(url= "https://i.imgur.com/Tt4wiJA.png")
```

```python
(Integral(cos(4*t)**15*sin(4*t)))
```

$\displaystyle \int \sin{\left(4 t \right)} \cos^{15}{\left(4 t \right)}\, dt$

```python
print(Integral(cos(4*t)**15*sin(4*t)).doit())
```

    -cos(4*t)**16/64

```python
Image(url= "https://i.imgur.com/fp90pIZ.png")
```

```python
Image(url= "https://i.imgur.com/87ipBxf.png")
```

```python
Integral(sec(5*x)*tan(5*x))
```

$\displaystyle \int \tan{\left(5 x \right)} \sec{\left(5 x \right)}\, dx$

```python
print(Integral(sec(5*x)*tan(5*x)).doit())
```

    1/(5*cos(5*x))

```python
Image(url= "https://i.imgur.com/p3TV8lD.png")
```

```python
Image(url= "https://i.imgur.com/VOZPhlq.png")
```

```python
Integral((2*x-5)**5,(x,3,5))
```

$\displaystyle \int\limits_{3}^{5} \left(2 x - 5\right)^{5}\, dx$

```python
print(Integral((2*x-5)**5,(x,3,5)).doit())
```

    1302

```python
Image(url= "https://i.imgur.com/IEjf5cG.png")
```

```python
Image(url= "https://i.imgur.com/SSN6ZNW.png")
```

```python
Integral(4/(sqrt(9-4*x)),(x,-4,-10))
```

$\displaystyle \int\limits_{-4}^{-10} \frac{4}{\sqrt{9 - 4 x}}\, dx$

```python
print(Integral(4/(sqrt(9-4*x)),(x,-4,-10)).doit())
```

    -4

```python
Image(url= "https://i.imgur.com/GpwBaxw.png")
```

```python
Image(url= "https://i.imgur.com/McCH6Kx.png")
```

```python
Integral(8*x**9 + 8*x**4)
```

$\displaystyle \int \left(8 x^{9} + 8 x^{4}\right)\, dx$

```python
print(Integral(8*x**9 + 8*x**4).doit())
```

    4*x**10/5 + 8*x**5/5

```python
Image(url= "https://i.imgur.com/Hziq0U9.png")
```

```python
Image(url= "https://i.imgur.com/cMQDWXn.png")
```

```python
from sympy import Function
f = Function('f')
def f(x):
    return x
Eq(Integral(f(x),(x,0,4)),8)
```

$\displaystyle \int\limits_{0}^{4} x\, dx = 8$

```python
Integral(f(2*x),(x,0,2)).doit()
```

$\displaystyle 4$

```python
Image(url= "https://i.imgur.com/Ir5yAmw.png")
```

```python
Image(url= "https://i.imgur.com/hIFtNsa.png")
```

```python
Integral(sqrt(5-2*x),(x,-4,2))
```

$\displaystyle \int\limits_{-4}^{2} \sqrt{5 - 2 x}\, dx$

```python
print(Integral(sqrt(5-2*x),(x,-4,2)).doit())
```

    -1/3 + 13*sqrt(13)/3

```python
Image(url= "https://i.imgur.com/RXi96fW.png")
```

```python
Image(url= "https://i.imgur.com/Kg571fy.png")
```

```python
e5 = sin(x**(Rational(1,4)))/x**(Rational(3,4))
e5
```

$\displaystyle \frac{\sin{\left(\sqrt[4]{x} \right)}}{x^{\frac{3}{4}}}$

```python
e6 = Integral(e5,(x,1,16)).doit()
e6
```

$\displaystyle - 4 \cos{\left(2 \right)} + 4 \cos{\left(1 \right)}$

```python
print(e6)
```

    -4*cos(2) + 4*cos(1)

```python
Image(url= "https://i.imgur.com/IZerZus.png")
```

```python
Image(url= "https://i.imgur.com/0JLEO9z.png")
```

```python
e7 = 3*sin(x)/cos(x)**2
e7
```

$\displaystyle \frac{3 \sin{\left(x \right)}}{\cos^{2}{\left(x \right)}}$

```python
Integral(e7,(x,0,pi/3))
```

$\displaystyle \int\limits_{0}^{\frac{\pi}{3}} \frac{3 \sin{\left(x \right)}}{\cos^{2}{\left(x \right)}}\, dx$

```python
Integral(e7,(x,0,pi/3)).doit()
```

$\displaystyle 3$

```python
Image(url= "https://i.imgur.com/ZN10AB4.png")
```

```python
Image(url= "https://i.imgur.com/mb5y8z8.png")
```

```python
Integral(cos(8*x)*sqrt(14-sin(8*x)))
```

$\displaystyle \int \sqrt{14 - \sin{\left(8 x \right)}} \cos{\left(8 x \right)}\, dx$

```python
print(Integral(cos(8*x)*sqrt(14-sin(8*x))).doit())
```

    sqrt(14 - sin(8*x))*sin(8*x)/12 - 7*sqrt(14 - sin(8*x))/6

```python
Image(url= "https://i.imgur.com/E2evsbD.png")
```

```python
Image(url= "https://i.imgur.com/iFHeHfX.png")
```

```python
Integral(1/x**5*sqrt(1+(1/x**4)),(x,1,4))
```

$\displaystyle \int\limits_{1}^{4} \frac{\sqrt{1 + \frac{1}{x^{4}}}}{x^{5}}\, dx$

```python
print(Integral(1/x**5*sqrt(1+(1/x**4)),(x,1,4)).doit())
```

    -257*sqrt(257)/24576 + sqrt(2)/3

```python
Image(url= "https://i.imgur.com/nceIFbp.png")
```

```python
Image(url= "https://i.imgur.com/NxrFbm7.png")
```

```python
Integral(x**2*(1+3*x**3)**5,(x,0,1))
```

$\displaystyle \int\limits_{0}^{1} x^{2} \left(3 x^{3} + 1\right)^{5}\, dx$

```python
print(Integral(x**2*(1+3*x**3)**5,(x,0,1)).doit())
```

    455/6

```python
Image(url= "https://i.imgur.com/RXX2ruO.png")
```

```python
Image(url= "https://i.imgur.com/8u9zXhU.png")
```

```python
Integral((4*x**3)/(x**4+1)**2,(x,2,4))
```

$\displaystyle \int\limits_{2}^{4} \frac{4 x^{3}}{\left(x^{4} + 1\right)^{2}}\, dx$

```python
print(Integral((4*x**3)/(x**4+1)**2,(x,2,4)).doit())

```

    240/4369

```python
Image(url= "https://i.imgur.com/pqFk6Hu.png")
```

```python
Image(url= "https://i.imgur.com/vFmjBoG.png")
```

```python
y = symbols('y')
Integral(y*sqrt(y+8))
```

$\displaystyle \int y \sqrt{y + 8}\, dy$

```python
print(Integral(y*sqrt(y+8)).doit())
```

    2*y**2*sqrt(y + 8)/5 + 16*y*sqrt(y + 8)/15 - 256*sqrt(y + 8)/15

```python
Image(url= "https://i.imgur.com/Ef9IGex.png")
```

```python
Image(url= "https://i.imgur.com/iniy2ix.png")
```

```python
f = 12/(4*x-7)**2
f
```

$\displaystyle \frac{12}{\left(4 x - 7\right)^{2}}$

```python
g = Integral(f)
c = symbols('c')
Eq(g.subs(x,2)+c,-6)
```

$\displaystyle c + \int\limits^{2} \frac{12}{\left(4 x - 7\right)^{2}}\, dx = -6$

```python
solve(Eq(g.subs(x,2)+c,-6),c)
```

    [-3]

```python
print(g.doit()-3)
```

    -3 - 12/(16*x - 28)

```python
Image(url= "https://i.imgur.com/K2tpqzf.png")
```


# Occupational Therapy — Notebook

For many practising engineers, their notebook plays a key role in helping them keep their thoughts on track when working through a problem.

Computational notebooks can play a similar role, with several additional benefits.

Firstly, as digital documents, they can be easily searched. Secondly, as *computational* notebooks, they can can be used to record, *and execute*, mathematical and computational models.

In this notebook, we'll provide a complement to Danny Nowlan's "Occupational Therapy" *Chassis Simulation* column from the July, 2020, edition of *Racecar Engineering* magazine to show how  to explore some of the models described using mathematical equations in that article interactively using computational notebooks.

Elsewhere, we'll show how you can explore actual data sets, using data rather than an mathematical models, using the computational notebook approach.

## Evaluating a Simple Equation

Over the years, the Pyhton programming language has been extended by many thousands of community generated "packages" that perform a wide variety of computational tasks, from data analysis and chart plotting, to finite element analysis to symbolic computation.

*Symbolic computaion*? What's that then...?

Maths... Programmes that do maths. If you've ever used Mathcad or Mathematica, that's symbolic computation.

In this notebook, I'll introduce a Python programming package called `sympy`, and show you how you can use it to "do maths" in a computational notebook.

One reason I'm doing it this way, in a *Jupyter* notebook, rather than Mathcad or Mathematica,  is that Jupyter notebooks, the Python programming language, and the `sympy` package are all free, open source software. Another reason is that Jupyter notebooks are increasingly widely used in academic teaching and research, as well as industry. This means they provide a good basis for being able to share your work with others, as well allowing you to directly benefit from code other people have put out there that you can learn from and crib from too..

So let's get started...

Danny Nowlan's article starts of with a simple equation that describes the traction circle radius ($TC_{RAD}$) as a function of tyre load ($F_z$) given a particular coefficient of friction ($k_a$) and coefficient drop off with load ($k_b$).

The basic equation takes the form:

$$TC_{RAD}=k_a(1-k_b.F_z)F_z$$

We can calculate this expression of a range of values for $F_z$, for example zero to one thousand.

*You may notice that equations in the notebook are presented in a stylised way. You can display all manner of mathematical equations in a notebook, and render the appropriately, by writing them using the LateX markup language.*

To start with, we declare that we want some symbolic variables:

```python
from sympy import symbols

TC_RAD, F_z, k_a, k_b = symbols('TC_RAD F_z k_a k_b')
```

We can assign particular values to symbolic quantities:

```python
k_a = 2
k_b = 5e-5
```

We can also write symbolic equations:

```python
TC_RAD = k_a *(1-k_b * F_z) * F_z
TC_RAD
```

$\displaystyle F_{z} \left(2 - 0.0001 F_{z}\right)$

Given a symbolic equation, we can plot it:

```python
from sympy.plotting import plot

# We aren't going to do symbolic sums using the min/max values
# so we don't need to define them as symbols in this case
# We can just declare normal Python variables
F_z_min = 0
F_z_max = 10000

plot(TC_RAD, (F_z, F_z_min, F_z_max), xlabel='$F_z$', ylabel='$TC_{RAD}$');
```

We can also make a interactive plot that allows us to explore the effect of changing the $k_a$ and $k_b$ parameter values:

```python
from ipywidgets import interact

# Unfortunately, the default slider widget does not show small numbers...
@interact(k_a=(0, 5, 0.1),
          k_b=(1e-5,1e-4, 5e-6))
def charter(k_a=2, k_b=5e-5):
    TC_RAD = k_a *(1-k_b * F_z) * F_z
    plot(TC_RAD, (F_z, F_z_min, F_z_max), xlabel='$F_z$', ylabel='$TC_{RAD}$',
         title=f'$k_b$={k_b}');

```

    interactive(children=(FloatSlider(value=2.0, description='k_a', max=5.0), FloatSlider(value=5e-05, description…

We can expliclty define a log slider plot that is a bit more sensitve in its display; note that by defining the plot, we can also control the label that is displayed:

```python
import ipywidgets as widgets

k_b_slider = widgets.FloatLogSlider(
    value=-5, # This is the power of the initial value
    base=10,
    min=-5, max=-4, # limits are powers
    step=0.01,
    description='$k_b$:'
)

interact(charter, k_b=k_b_slider);
```

    interactive(children=(IntSlider(value=2, description='k_a', max=6, min=-2), FloatLogSlider(value=1e-05, descri…

The article suggests differentiating the equation for $TC_{RAD}$ and then solving by setting the result to zero.

We can differentiate expression by asking *sympy* to differentiate the equation for us:

```python
from sympy import diff

# Redefine the symbols
TC_RAD, F_z, k_a, k_b = symbols('TC_RAD F_z k_a k_b')

# Redefine the equation
TC_RAD = k_a *(1 - k_b * F_z) * F_z

# Differentiate with respect to load, F_z
diff(TC_RAD, F_z)
```

$\displaystyle - F_{z} k_{a} k_{b} + k_{a} \left(- F_{z} k_{b} + 1\right)$

And then we can solve it automatically as well:

```python
from sympy import solve

solve(diff(TC_RAD, F_z))
```

    [{F_z: 1/(2*k_b)}, {k_a: 0}]

The result is a list of possible solutions (in Pyhton, a list represented by comma separated values in square brackets, `[...]`).

We can extract the first solution (list index `0`) and display it:

```python
L_P = symbols('L_P')

L_P = solve(diff(TC_RAD, F_z))[0][F_z]
L_P
```

$\displaystyle \frac{1}{2 k_{b}}$

which is the same as the expression for Equation 2 in the article:

$L_p=\frac{1}{2.k_b}$

The article then identifies the maximum traction circle radius as:

$TC_{RAD\_MAX}=\frac{k_a.L_P}{2}$

We can computationally derive this by substituting in our value of `L_P` into the equation for `TC_RAD`:

```python
TC_RAD_MAX = k_a *(1 - k_b * L_P) * L_P
TC_RAD_MAX
```

$\displaystyle \frac{k_{a}}{4 k_{b}}$

For $L_p=\frac{1}{2.k_b}$, this gives us the result of equation 3 in the article.

One of the major problems with maths notation used in Racecar Engineering magazine articles is that the symbols are often used inconsistently; for example, is $L_P$ of equation 3 intended to be the same as the previously derived $L_p$ from equation 2? As another example of notational ambiguity, in table 1, the value of $k_b$ is given as `5.0e-5 (1/N)`; the `(1/N)` component is presumably referring to the units rather than an unexplained part of the equation referring to some undeclared quantity `N`? (Many articles are all but impossible to follow without a lot of mental gymnastics trying to make sense of the garbled notation...)

Moving on in the article, equation 4 provides expressions for determining front and rear cornering speeds using quantities:

- $F_{xf}$: deduced front lateral force derived from equation 1;
- $F_{yr}$: deduced rear lateral force derived from equation 1;
- $wdf$: front weight distribution (per cent / 100);
- $m_t$: total car mass (kg);
- $iR$: peak corner curvature (1/m);
- $V_x$: cornering speed (m/s).

```python
F_xf, F_yr, wdf, m_t, iR, V_x = symbols('F_xf F_yr wdf m_t iR V_x')

F_xf = wdf * m_t * iR * V_x**2
F_yr = (1 - wdf) * m_t * iR * V_x**2
```

The article then gives a view of a whole host of other settings in an Excel worksheet but no formulas to show how they gerneate the claimed results also show in the worksheet.

In all the years I've read Racecar Engineer, I'm not sure I can recall a single article with a mathematical basis that I could replicate all the various elements myself... And this one is no different!


### Common Activation Functions

```python
import numpy as np
from math import tanh
from matplotlib import pyplot as plt
i = np.arange(-10, 10, .1)

def softplus(x):
    return np.log(1 + np.exp(x))

plt.plot(i, softplus(i))
plt.ylabel('softplus(x)')
plt.show()

def logistic_sigmoid(x):
    return 1 / (1 + np.exp(-x))

plt.plot(i, logistic_sigmoid(i))
plt.ylabel('logistic_sigmoid(x)')
plt.show()

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()

plt.plot(i, softmax(i))
plt.ylabel('softmax(x)')
plt.show()

def rectified_linear_activation_function(z):
    return max([0, z])

plt.plot(i, list(map(rectified_linear_activation_function, i)))
plt.ylabel('rectified_linear_activation_function(x)')
plt.show()

plt.plot(i, list(map(tanh, i)))
plt.ylabel('tanh(x)')
plt.show()

```

### XOR as a Neural Network in Numpy, Keras (TensorFlow), TensorFlow, and PyTorch

XOR with Numpy

```python
import numpy as np

X = np.matrix([[0, 0],
               [0, 1],
               [1, 0],
               [1, 1]])
W = np.matrix([[1, 1],
               [1, 1]])
c = np.array([0, -1])
w = np.matrix([[1],
               [-2]])

A_in = (X * W + c)
A_out = np.matrix([list(rectified_linear_activation_function(y) for y in x) for x in A_in.tolist()])
A_out * w
```

    matrix([[0],
            [1],
            [1],
            [0]])

XOR with Keras

```python
import keras

model = keras.models.Sequential()
model.add(keras.layers.core.Dense(2, activation='relu', input_shape=(2,)))
model.add(keras.layers.core.Dense(1)) # linear is the default activation (https://keras.io/activations/)
model.set_weights([
    np.array([[1 , 1], [1,  1]]),
    np.array([ 0, -1]),
    np.array([[1], [-2]]),
    np.array([ 0])
])
model.predict_classes(X)
```

    4/4 [==============================] - 0s

    array([[0],
           [1],
           [1],
           [0]], dtype=int32)

XOR with TensorFlow

```python
import tensorflow as tf

tfX = tf.constant(X, dtype='float64')
tfW = tf.Variable(W, dtype='float64')
tfc = tf.Variable(c, dtype='float64')
tfw = tf.Variable(w, dtype='float64')

node1 = tf.matmul(tfX, tfW)
node2 = tf.add(node1, tfc)
node3 = tf.nn.relu(node2)
node4 = tf.matmul(node3, tfw)

init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
sess.run(node4)
```

    array([[ 0.],
           [ 1.],
           [ 1.],
           [ 0.]])

XOR with PyTorch

```python
from torch import Tensor
from torch.autograd import Variable
from torch.nn import Module, Linear, Parameter
from torch.nn.functional import relu

class Net(Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = Linear(2, 2)
        self.fc2 = Linear(2, 1)
    def forward(self, x):
        x = relu(self.fc1(x))
        x = self.fc2(x)
        return x

net = Net()

net.fc1.weight = Parameter(Tensor([[1, 1],
                                   [1, 1]]))
net.fc1.bias = Parameter(Tensor([[0, -1]]))
net.fc2.weight = Parameter(Tensor([[1, -2]]))
net.fc2.bias = Parameter(Tensor([[0]]))

input = Variable(Tensor([[0, 0],
                         [0, 1],
                         [1, 0],
                         [1, 1]]))
out = net(input)
out
```

    Variable containing:
     0
     1
     1
     0
    [torch.FloatTensor of size 4x1]

## Neural Network w/ Loss Function, Gradient, and Optimization defined with Sympy

```python
import numpy as np
import sympy

x_data = np.array([1, 2, 3])
y_data = np.array([2, 4, 6])

x, y, w, y_pred = sympy.symbols('x y w y_pred')
y_pred = x * w
print('forward:', y_pred)
loss = (y_pred - y)**2
print('loss:', loss)
gradient = sympy.diff(loss, w)
print('gradient:', gradient)

w_in = np.random.randint(0,3)
print('w:', w_in)
learning_rate = 0.01
print('learning_rate:', learning_rate, '\n')
for epoch in range(10):
    for x_in, y_in in zip(x_data, y_data):
        print('x:', x_in, 'y:', y_in, 'w:', w_in)
        grad = gradient.evalf(subs={x: x_in, y: y_in, w: w_in})
        print('gradient:', grad)
        w_in = w_in - learning_rate * grad
        l = loss.evalf(subs={x: x_in, y: y_in, w: w_in})
    print('loss:', l, '\n')
```

    forward: w*x
    loss: (w*x - y)**2
    gradient: 2*x*(w*x - y)
    w: 2
    learning_rate: 0.01

    x: 1 y: 2 w: 2
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

    x: 1 y: 2 w: 2.00000000000000
    gradient: 0.e-141
    x: 2 y: 4 w: 2.00000000000000
    gradient: 0.e-140
    x: 3 y: 6 w: 2.00000000000000
    gradient: 0.e-140
    loss: 4.62129760221396e-274

```python

```


# Lab 3
## Introduction
In this lab we will analyse population dynamics under the logisitic model with managed harvesting.

First import the modules we need.

```python
from plotly.figure_factory import create_quiver
from plotly import graph_objs as go
from numpy import meshgrid, arange, sqrt, linspace
from scipy.integrate import odeint
```

## Harvesting of fish
A population of fish in a lake, left to its own devices, is modelled by the logistic differential equation
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y),
\end{align}
where the population $y$ is in units of thousands of fish and time $t$ is measured in years.

First define a function for $\mathrm{d}y/\mathrm{d}x$ in terms of $y$ and $x$.

```python
def diff_eq(y, x):
    return 4 * y * (1 - y)
```

Next define a function that creates a Plotly Figure object that contains a slope field and, optionally, a few solutions to initial value problems.

It automates a few things we did in the last lab.

- `diff_eq` is the differential equation to be plotted
- `x` and `y` should be outputs from `meshgrid`.
- `args` is any additional arguments to `diff_eq` (we will use that below).
- `initial_values` is a list (or array) of starting $y$ values from which approximate solutions will start. The corresponding $x$ value is the minimum element of `x`.

Note that the numerical solutions will plotted for the whole range of $x$ values in `x`, so if they blow up you will probably get a warning and less-than-useful plot.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
```

The slope field below should hopefully give you some idea for the fish population dynamics.

Note that we use `arange` rather than `linspace` this week so that we can carefully control the increments between our grid points. `arange(0, 1.1, 0.25)` returns an array that starts with 0 and increments by 0.25 until it exceeds 1.1.

The plot also contains the solution curves for
$y(0) = 1$ and $y(0) = 0.4$. Edit the cell to also include the solution curve for $y(0)=1.4$.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, initial_values=(0.4, 1))
fig.show('png')
```

### Equilibrium solutions
Looking back to our differential equation, $\mathrm{d}y/\mathrm{d}t = 0$ when $y(t) = 0$ or $y(t) = 1$. Looking at the slope field, we see that the equilibrium solution $y(t) = 1$ is stable (this is the carrying capacity here, corresponding to 1000 fish), whereas the equilibrium solution $y(t) = 0$ is unstable. Any non-zero initial population will eventually stabilise at 1000 fish.

### What will happen if harvesting is now commenced at a steady rate?
For the simplest harvesting model, assume that $H$ units (thousands) of fish are taken
continuously (smoothly) over the year, rather than at one instant each year.
Note that the units of $H$ are the same as those of $\mathrm{d}y/\mathrm{d}t$, thousands of fish per year, so we simply subtract $H$ from the RHS of our existing equation to give the DE with harvesting as
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y) - H.
\end{align}
Again, the (constant) equilibrium solutions are found by setting $\mathrm{d}y/\mathrm{d}t = 0$, giving from the quadratic formula (check this),
\begin{align}
y(t) = \frac{4\pm\sqrt{16-16H}}{8} = \frac{1\pm\sqrt{1-H}}{2}.
\end{align}
What happens after harvesting starts will depend on the equilibrium solutions, their
stability and the initial number of fish $y(0)$.

Start by redefining `diff_eq` to include the `H` parameter. Note that defining `diff_eq` again overides our original definition.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
```

Now set $H = 0.6$ and plot the slope field. This is done by setting `args=(0.6,)` when we call `create_slope_field`. This is exactly how you would pass additional arguments like this one to `odeint` if you were calling it directly.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,))
fig.show('png')
```

From the solutions to the quadratic equation above, the equilibrium solutions of the DE are found to be $y(t) \approx 0.184$ and $y(t) \approx 0.816$. The previous equilibrium solution with no harvesting at $y(t) = 0$ has moved up to $y(t) \approx 0.184$, while the previous equilibrium solution with no harvesting at $y(t) = 1$ has moved down to $y(t) \approx 0.816$.

From the slope field, we see that the equilibrium solution $y(t) \approx 0.184$ is unstable, whereas the equilibrium solution $y(t) \approx 0.816$ is stable. If the population ever falls below about 0.184, or 184 fish, it will then drop to 0. This is a new feature, introduced by harvesting.

In the cell below, use `create_slope_field` to experiment by plotting the solutions to the initial value problems $y(0)=0.183$ and $y(0)=0.25$. Extend the $x$ range of your slope field until the top line is close to equlibrium. Note that if you extend it too far you will break `odeint` (why?). You may also like to increase the increments in `arange` to make the plot clearer.

```python
x, y = meshgrid(arange(0, 2.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.25))
fig.show('png')
```

## Exercises

In this lab you will experiment with the population dynamics given by the logistic equation with harvesting that we started analysing in the lab.

This week the questions will be a combination of plots and written answers.

1. Assume that the harvest is 600 fish per year. **On the same figure,**
  a. plot the slope field,
  b. plot the equilibrium solutions that we found in above, and
  c. plot the solution curves for $y(0)=1$, $y(0)=0.3$, and $y(0)=0.15$.

```python
x, y = meshgrid(arange(0, 1.2, 0.1), arange(-0.4, 1.3, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(1, 0.3, 0.15))
fig.show('png')
```

1.  d. In the cell below, describe the behaviour of the fish population for each of these five initial numbers of fish.

If the initial value is y(0) = 1, the popuplation falls towards the stable equilirium at approx 0.8.
If the initial value is y(0) = 0.3, then the population tends out of the unstable equilibrium at approximately 0.2 and towards a stable equilibrium f If our initial value is at y(0) = 0.15, at after 0.6 years the population will be about 0. It zooms off to negative infinity as time goes on.

2. a. i. Assume that $H=0.8$. Plot the slope field and five solutions, one for each equilibrium solution and one for each region between, above, or below them. You can use the equation from the lab to calculate the equilibrium solutions.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.3, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.8,), initial_values=(1, 0.45, 0.2, 0.7236, 0.2764))
fig.show('png')
```

2. a. ii. In the cell below, describe the limiting behaviour of each line.

We can see limiting behaviours for lines with initial values (1, 0.45 and 0.7236) is towards the stable equilibrium found at approx 0.7236 in infinity time.

The limiting behaviour of the line at point 0.2764 stays at this point.

The bottom line (initial value of 0.2) tends to negative infinity.

2. b. i. Assume that $H=1$. Plot the slope field and three solutions for the equilibrium solution and the regions above and below it.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(1,), initial_values=(1, 0.46, 0.2764))
fig.show('png')
```

2. b. ii. Describe the limiting behaviour of each line.

Initial value of 1 declines and tends towards the equilibrium at approx. 0.46.

For any value below the equilibrium, it hits 0 and continues declining so it tends towards negative infinity.

2. c. i. Assume that  𝐻=1.2. Plot the slope field and two or three solutions.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(1.2,), initial_values=(1, 0.4,))
fig.show()
```

2. c. iii. Describe the limiting behaviour of the lines.

This situation demonstrates no critical points, the bottom line (initial value 0.4) will have a limiting behaviour that tends off to negative infinity.

Y will never become 0.

The fish population will always be declining. The species will inevitably die out.

3. Summarize what happens to the equilibrium solutions and their stability as $H$
is increased from 0 to beyond 1. Refer to your plots to support your answers.

The higher the H value the less equilibrium points exist. Beyond 1 there exists critical points so the populaiton will inevitably decline and remain declining to negative infinity.

4. What is a reasonable strategy for sustainable fishing in this case?
Don’t forget to allow qualitatively for minor catastrophes, such as disease or temporary overfishing.

A sustainable amount of fish to remove from the environment would be around 500.

We have a margin of error of around 500 fish, and by taking 500 from the population we are approximately half way through this margin of error (so as to incorporate catastrophes/disease/temporary overfishing)


# Linear Programming: Introduction

## Definition

Formally, a linear program is an optimzation problem of the form:

\begin{equation}
\min \vec{c}^\mathsf{T}\vec x\\
\textrm{subject to} \begin{cases}
\mathbf{A}\vec x=\vec b\\
\vec x\ge\vec 0
\end{cases}
\end{equation}

where $\vec c\in\mathbb R^n$, $\vec b\in\mathbb R^m$ and $\mathbf A \in \mathbb R^{m\times n}$. The vector inequality $\vec x\ge\vec 0$ means that each component of $\vec x$ is nonnegative. Several variations of this problem are possible; eg. instead of minimizing, we can maximize, or the constraints may be in the form of inequalities, such as $\mathbf A\vec x\ge \vec b$ or $\mathbf A\vec x\le\vec b$. We shall see later, these variations can all be rewritten into the standard form.

## Example

A manufacturer produces  four different  products  $X_1$, $X_2$, $X_3$ and $X_4$. There are three inputs to this production process:

- labor in man weeks,
- kilograms of raw material A, and
- boxes  of raw  material  B.

Each product has different input requirements. In determining each  week's production schedule, the manufacturer cannot use more than the available amounts of  manpower and the two raw  materials:

|Inputs|$X_1$|$X_2$|$X_3$|$X_4$|Availabilities|
|------|-----|-----|-----|-----|--------------|
|Person-weeks|1|2|1|2|20|
|Kilograms of material A|6|5|3|2|100|
|Boxes of material B|3|4|9|12|75|
|Production level|$x_1$|$x_2$|$x_3$|$x_4$| |

These constraints can be written in mathematical form

\begin{align}
x_1+2x_2+x_3+2x_4\le&20\\
6x_1+5x_2+3x_3+2x_4\le&100\\
3x_1+4x_2+9x_3+12x_4\le&75
\end{align}

Because negative production levels are not meaningful, we must impose the following nonnegativity constraints on the production levels:

\begin{equation}
x_i\ge0,\qquad i=1,2,3,4
\end{equation}

Now suppose that one unit of product $X_1$ sells for €6 and $X_2$, $X_3$ and $X_4$ sell for €4, €7 and €5, respectively. Then, the total revenue for any production decision $\left(x_1,x_2,x_3,x_4\right)$ is

\begin{equation}
f\left(x_1,x_2,x_3,x_4\right)=6x_1+4x_2+7x_3+5x_4
\end{equation}

The problem is then to maximize $f$ subject to the given constraints.

## Vector Notation

Using vector notation with

\begin{equation}
\vec x = \begin{pmatrix}
x_1\\x_2\\x_3\\x_4
\end{pmatrix}
\end{equation}

the problem can be written in the compact form
\begin{equation}
\max
\begin{pmatrix}6&4&7&5\end{pmatrix}
\begin{pmatrix}
x_1\\x_2\\x_3\\x_4
\end{pmatrix}\\
\textrm{subject to}
\begin{cases}
\begin{pmatrix}
1&2&1&2\\
6&5&3&2\\
3&4&9&12
\end{pmatrix}
\begin{pmatrix}
x_1\\x_2\\x_3\\x_4
\end{pmatrix}\le
\begin{pmatrix}
20\\100\\75
\end{pmatrix}\\
\begin{pmatrix}
x_1\\x_2\\x_3\\x_4
\end{pmatrix}\ge
\begin{pmatrix}
0\\0\\0\\0
\end{pmatrix}
\end{cases}
\end{equation}

## Two-dimensional Linear Program

Many fundamental concepts of linear programming are easily illustrated in two-dimensional space.

Consider the following linear program:

\begin{equation}
\max
\begin{pmatrix}
1&5
\end{pmatrix}
\begin{pmatrix}
x_1\\
x_2
\end{pmatrix}\\
\textrm{subject to}
\begin{cases}
\begin{pmatrix}
5&6\\
3&2
\end{pmatrix}
\begin{pmatrix}
x_1\\
x_2
\end{pmatrix}\le
\begin{pmatrix}
30\\
12
\end{pmatrix}\\
\begin{pmatrix}
x_1\\
x_2
\end{pmatrix}\ge
\begin{pmatrix}
0\\
0
\end{pmatrix}
\end{cases}
\end{equation}

```julia
#using Pkg
#pkg"add LaTeXStrings"

using Plots
using LaTeXStrings

x = -2:6
plot(x, (30 .- 5 .* x) ./ 6, linestyle=:dash, label=L"5x_1+6x_2=30")
plot!(x, (12 .- 3 .* x) ./ 2, linestyle=:dash, label=L"3x_1+2x_2=12")
plot!([0,4,1.5,0,0],[0,0,3.75,5,0], linewidth=2, label="constraints")
plot!(x, -x ./ 5, label=L"f\left(x_1,x_2\right)=x_1+5x_2=0")
plot!(x, (25 .- x) ./ 5, label=L"f\left(x_1,x_2\right)=x_1+5x_2=25")
```

## Slack Variables

Theorems and solution techniques are usually stated for problems in standard form. other forms of linear programs can be converted as the standard form. If a linear program is in the form

\begin{equation}
\min \vec{c}^\mathsf{T}\vec x\\
\textrm{subject to} \begin{cases}
\mathbf{A}\vec x\ge \vec b\\
\vec x\ge\vec 0
\end{cases}
\end{equation}

then by introducing _surplus variables_, we can convert the orginal problem into the standard form

\begin{equation}
\min \vec{c}^\mathsf{T}\vec x\\
\textrm{subject to} \begin{cases}
\mathbf{A}\vec x-\mathbf I\vec y = \vec b\\
\vec x\ge\vec 0\\
\vec y\ge\vec 0
\end{cases}
\end{equation}

where $\mathbf I$ is the $m\times m$ identity matrix.

If, on the other hand, the constraints have the form
\begin{equation}
\begin{cases}
\mathbf{A}\vec x\le b\\
\vec x\ge\vec 0
\end{cases}
\end{equation}

then we introduce the _slack variables_ to convert the constraints into the form

\begin{equation}
\begin{cases}
\mathbf{A}\vec x+\mathbf I\vec y = \vec b\\
\vec x\ge\vec 0\\
\vec y\ge\vec 0
\end{cases}
\end{equation}

Consider the following optimization problem

\begin{equation}
\max x_2-x_1\\
\textrm{subject to}
\begin{cases}
3x_1=x_2-5\\
\left|x_2\right|\le2\\
x_1\le0
\end{cases}
\end{equation}

To convert the problem into a standard form, we perform the following steps:

1. Change the objective function to:

\begin{equation}
\min x_1 - x_2
\end{equation}

2. Substitute $x_1=-x_1^\prime$.

3. Write $\left|x_2\right|\le2$ as $x_2\le 2$ and $-x_2\le 2$.

4. Introduce slack variables $y_1$ and $y_2$, and convert the inequalities above to

\begin{cases}
\hphantom{-}x_2 + y_1 =2\\
-x_2+y_2 =2
\end{cases}

5. Write $x_2=u-v$ with $u,v\ge0$.

Hence, we obtain

\begin{equation}
\min -x_1^\prime-u+v\\
\textrm{subject to}
\begin{cases}
3x_1^\prime+u-v=5\\
u-v+y_1=2\\
v-u+y_2=2\\
x_1^\prime,u,v,y_1,y_2\ge0
\end{cases}
\end{equation}

## Fundamental Theorem of Linear Programming

We consider the system of equalities

\begin{equation}
\mathbf{A}\vec x=\vec b
\end{equation}

where $\mathrm{rank}\,\mathbf A=m$.

Let $\mathbf B$ a square matrix whose columns are $m$ linearly independent columns of $\mathbf A$. If necessary, we reorder the columns of $\mathbf A$ so that the columns in $\mathbf B$ appear first: $\mathbf A$ has the form $\left(\mathbf B |\mathbf N\right)$.

The matrix is nonsingular, and thus we can solve the equation

\begin{equation}
\mathbf B\vec x_\mathbf B = \vec b
\end{equation}

The solution is $\vec x_\mathbf B = \mathbf B^{-1}\vec b$.

Let $\vec x$ be the vector whose first $m$ components are equal to $\vec x_\mathbf B$ and the remaining components are equal to zero. Then $\vec x$ is a solution to $\mathbf A\vec x=\vec b$. We call $\vec x$  a _basic solution_. Its components refering to the the components of $\vec x_\mathbf B$ are called _basic variables_.

- If some of the basic variables are zero, then the basic solution is _degenerate_.
- A vector $\vec x$ satisfying $\mathbf A\vec x=\vec b$, $\vec x \ge \vec 0$, is said to be a _feasible solution_.
- A feasible solution that is also basic is called a _basic feasible solution_.

The fundamental theorem of linear programming states that when solving a linear programming problem, we need only consider basic feasible solutions. This is because the optimal value (if it exists) is always achieved at a basic solution.