# Introduction to Programming for Engineers

## Python 3

# 05 Algebra with Sympy, Scipy and Numpy
## CLASS MATERIAL

 1. Finding Roots of Polynomials

 2. Root Finding using an Initial Estimate

 3. Systems of Coupled Equations

 4. Computation on Arrays : Broadcasting

 5. Symbolic Mathematics

6. Review Exercises

### Lesson Goal

In this class we will learn some useful techniques for making common algebraic operations faster and easier by solving them computationally.

This can help you to:
1. Quickly solve time-consuming mathematical problems
1. Solve "messy" problems (e.g. experimental data)
1. Check your answers to problems solved using hand calculations

### Fundamental programming concepts
 - Python `sympy`, `numpy` and `scipy` libraries
 - Manipulating arrays to use within functions

```python
import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline
```

## 1. Finding Roots of Polynomials

Many applications involve finding the roots of a function.

Example:  we want to find a value or values for $x$ such that $f(x)=0$.

This is used for problems including optimization.

__Optimization__ : Finding the maximal or minimal value of a dependent variable with respect to some independent variable(s).

To find the maxima and minima of some function:
- take the derivative(s) of the function with respect to the independent variable(s)
- find values of the function for which the derivative(s) are equal to zero

Previously, we studied how to fit a polynimal function through a series of data points.

For some functions, the roots are easy to find analytically:

$f(x) = 2x^3 - 8x^2 - 96x$

As each term is divisible by $2x$, we can easily factorise this function to:

$f(x)=(2x)(x + 4)(x - 8)$

The roots are 0, -4, and 8.

When working with real data, root finding is often not so straighforward.

- noisy experimental data
- higher order polynomials

```python
# Example data using example polynomial
x = np.linspace(-6, 10, 100) # 100 equally spaced points between -6 and 10

y = (2*x**3 - 8*x**2 - 96*x) # function y = 2x^3 - 8x^2 - 96x
```

We can use pseudo-random number generators to make the data more "messy" i.e. more like real experimental data.

This is an example of scope.

Notice that two random number libraries are used:

- `np.random` e.g. `np.random.random` : generates rando number in range [0.0, 1.0)
- `random` e.g `random.sample` : selects a random sample from data given

```python
# add noise : multiply each y value by random number
y_n = y * np.random.random(size=x.shape)

# Choose a random sample of 50 data points
x_e, y_e = zip(*random.sample(list(zip(x, y_n)), 50))

# sort into x-ascending order
tmp = sorted(zip(x_e, y_e))

# split into x and y
x_e = [t[0] for t in tmp]
y_e = [t[1] for t in tmp]
```

```python
# Fit a polynomial of order 3 to the data
coeffs = np.polyfit(x_e, y_e, 3)

yfit = np.poly1d(coeffs)(x_e)

# Round all coefficients to 3d.p.
coeffs = [round(c,3) for c in coeffs]
```

```python
# Plot the data
plt.plot(x, y,  label=f'2x^3 - 8x^2 - 96')      # orignal data
plt.plot(x_e, y_e, 'ro')                        # "experimental" data

plt.plot(x_e, yfit, label=f'{coeffs[0]}x^3 + ({coeffs[1]})x^2 + ({coeffs[2]})x + ({coeffs[3]})')      # fitted data

plt.legend(bbox_to_anchor=(1,1), loc="upper left")
```

We can see that the fitted polynomial does not factorise so cleanly.

This makes it difficult to solve the equation *analytically*.

- __Analytical solution__ : An exact solution can be found.
- __Numerical solution__ :  The solution is iteratively guessed until the answer is good enough enough to stop.

We can solve the problem numerically, for example using the bisection method.

__Biscetion method__:
For a given function F(x):
1. two values a and b are chosen for which F(a) > 0 and F(b) < 0 (or the other way around)
1. interval halving: a midpoint x=c is calculated as the arithmetic mean between a and b, c = (a + b) / 2
1. F(c) is evaluated at c
1. if F(c) = 0 the root of the function is c
1. if F(c) ≠ 0, check the sign of F(c):
    - if F(c) has the same sign as F(a), replace a with c (b stays the same)
    - if F(c) has the same sign as F(b), replace b with c (a stays the same)
1. go back to step 2.

The algorithm ends when the values of F(c) is less than a defined tolerance (e.g. 0.001).

When we import a Python function, a computer algorithm runs this or a similar numerical solver.

We can use the functions `roots` to find the roots of a polynomial function.

Appreciably, this is much faster than finding the roots numericlaly by hand or by writing the algorithm ourselves.

```python
r = np.roots(coeffs)
print(r)

```

    [ 9.15349497 -5.16123286  0.06450232]

We can plot the polynomial equation and the roots to check the solution:

```python
plt.plot(x_e, y_e, 'ro')                        # "experimental" data
plt.plot(x_e, yfit)                   # fitted data
plt.axhline(0, color='black', lw=0.5) # y=0 axis

z = np.zeros(len(r))                  # array of zeros
plt.plot(r, z, 'ks')                  # roots
```

For a polynomial function, we know that there will be as many roots as the order of the function.

e.g. the cubic function above will have three roots.

It is easy to verify this:
1. visually from the plot above.
1. in code:

```python
print(f'number of roots = {len(r)}')
```

    number of roots = 3

Another example showing the full process:

```python
X = np.linspace(-4, 4)       # x data
coeffs = [1, -1, -6, 2, 0]   # coefficients of polynomial
Y = np.poly1d(coeffs)(X)     # y data
R = np.roots(coeffs)         # roots
print (R)

# plot
plt.axhline(0, color='black', lw=0.5)
plt.plot(X, Y)               # x,y data
Z = np.zeros(len(R))
plt.plot(R, Z, 'ks')         # roots
```

# 2. Root Finding using an Initial Estimate

`roots` ony works for polynomials.

There are other python functions available for finding the roots of other functions.

For a polynomial function, we know that there will be as many roots as the order of the function.

For other functions, the number of roots may be:

- initially unknown
- infinite

The function `fsolve` finds the roots of a function, given a starting estimate.

```python
from scipy.optimize import fsolve
```

```python
# the function must be a python function
def f(t):
    return t**2 - 4

starting_guess = 3

print(fsolve(f, starting_guess))
```

    [2.]

We can plot the data to check that there is a root at 2.

```python
plt.plot(range(4), [f(r) for r in (range(4))])

plt.plot(2, f(2), 'o')

plt.axhline(0, color='black', lw=0.5)
```

A `lambda` function can be used to make the code shorter as we do not need to create a function definition with `def`.

```python
starting_guess = 3

print(fsolve(lambda t: t**2 - 4, starting_guess))
```

    [2.]

Let's plot the experimental data from earlier.

```python
plt.plot( x_e, y_e, 'ro')  # "experimental" data
plt.plot( x_e, yfit )      # fitted data
plt.plot( r, np.zeros(len(r)), 'ks')
plt.axhline(0, color='black', lw=0.5)
```

We can see that there are roots close to -6, 0 and 10

```python
# Fit a polynomial of order 3 to the data
coeffs = np.polyfit(x_e, y_e, 3)
```

```python
print(fsolve(lambda t: np.poly1d(coeffs)(t), -6))
print(fsolve(lambda t: np.poly1d(coeffs)(t), 0))
print(fsolve(lambda t: np.poly1d(coeffs)(t), 10))
```

    [-5.16171255]
    [0.06451022]
    [9.15520854]

These are the same as the roots we found earlier.

It is possible to automate this search by setting a tolerance for the proximity of a value to 0 at which the computer should check for a root.

As this depends on the resolution of your y data, it is advisable to plot the data in order to check your solution.

```python
def all_roots(tolerance):
    """
    Uses fsolve to find a root close a y value if the y value is within a given tolerance of the x axis
    """
    print(f'tolerance = {tolerance}')
    for guess in np.poly1d(coeffs)(x_e):    # for each y data point
        if abs(guess) < tolerance:          # if y is close to 0 within tolerance
            print(fsolve(lambda t: np.poly1d(coeffs)(t), guess))   # print a root
    print()

all_roots(0.1)
all_roots(1)
all_roots(10)
all_roots(11)
all_roots(20)
```

    tolerance = 0.1

    tolerance = 1

    tolerance = 10
    [0.06451022]
    [-5.16171255]

    tolerance = 11
    [0.06451022]
    [9.15520854]
    [-5.16171255]

    tolerance = 20
    [-5.16171255]
    [0.06451022]
    [9.15520854]
    [9.15520854]
    [-5.16171255]
    [-5.16171255]

# 3. Systems of Coupled Equations

 3.1 Systems of Linear Equations

 3.2. Systems of Non-Linear Equations

### 3.1 Systems of Linear Equations

A system of linear equations (or linear system) is a collection of two or more linear equations involving the same set of variables.

__Example :__ Given the equations below, find $x_1$, $x_2$ and $x_3$.

\begin{align*}
x_1 - x_2 + x_3 &= 0\\
10x_2 + 25x_3 &= 90\\
20x_1 + 10x_2 &= 80\\
\end{align*}

We have three equations and three unknown values so we can solve the equations by hand, for example using substitution or elimitaion.

As the number of variables or the number of equations increases, the calculation time and difficulty in finding the solution may also increase.

We can use the function `solve` from the Numpy linear algebra module.

`solve` solves equations of the form `ax = b` for x

To put the system of equations into this form, we must represent the system using matrices, and in code, `numpy array`s.

First, arrange the values from the right-hand-side and the coefficients from the left-hand-side as Numpy arrays.

```python
A = np.array([[1, -1, 1],
              [0, 10, 25],
              [20, 10, 0]])

B = np.array([0, 90, 80])
```

If you are familiar with matrices, this is equivalent to expressing the form:

\begin{equation*}
\underbrace{
\begin{bmatrix}
1 & -1 & 1  \\
0 & 10 & 25 \\
20 & 10 & 0 \\
\end{bmatrix}
}_{\mathbf{A}}
\cdot
\underbrace{
\begin{bmatrix}
x_1 \\
x_2 \\
x_3 \\
\end{bmatrix}
}_{\mathbf{x}}
=\underbrace{
\begin{bmatrix}
0 \\
90 \\
80 \\
\end{bmatrix}
}_{\mathbf{B}}
\end{equation*}

Notice that B appears in code as a horizontal array.

In Python, 1D arrays are ALWAYS represented horizontally.

The array is treated as a row or column vector depending on the equation in which it appears.

Solve for x:

```python
x = np.linalg.solve(A, B)
print(x)

```

    [2. 4. 2.]

$x_1 = 2$

$x_2 = 4$

$x_3 = 2$

To verify that this is correct, we can find the matrix product or dot product of $\mathbf{A}$ and $\mathbf{x}$.

```python
print(np.dot(A,x))
print(np.matmul(A,x))
```

    [ 0. 90. 80.]
    [ 0. 90. 80.]

The solution matches the original equation, which we can confim in code:

```python
print(np.dot(A,x) == B)
```

    [ True  True  True]

### Example Engineering Application: Statically Determinate Truss
A common problem in statics is the analysis of a statically determinate planar truss.

Trusses are characterized by linear members (beams) which are pinned together at their ends.

The pinned connections provide no resistance to rotation so there are no moments around the joints.

A planer truss is a truss arranged in a single plane; we consider forces in 2 dimensions only.

### Example Engineering Application: Statically Determinate Truss

A truss is said to be statically determinate if all:
- support reactions
- forces in each member

can be calculated using only the *equations of static equalibrium*.

#### Static Equilibrium
There is one force in each member.

The forces acting on each joint sum to zero.

The forces are resolved in the vertical and horizontal directions.

Therefore there are (number of joints x 2) equations of static equalibrium for a given truss.

A truss is statically determinate if:
> number of members + number of support reactions <= number of joints x 2

in other words:
> number of unknowns values <= number of equations

Consider a truss with:
- 5 members
- 4 joints
- 3 support reactions
- 1 external force

A truss is statically determinate if:

members + support reactions <= joints x 2

5 + 3 = 8

The truss is statically determinate.

__Assumption__ : Members 1, 2, 4, 5 are of equal length.

__Question : Find the support reaction forces R1-R3 and the forces in each member F1-F5__

#### Static Equilibrium

The sum of the forces at each node must equal zero.

There are two equations for each point, the horizontal equation and the vertical equation.

Let's find the horizontal (H) and vertical (V) equilibrium equation for each node:

1H : $ -F_1 - E = 0$

1V : $ -F_5 = 0$

2H : $ F_1 + cos \frac{\pi}{4}F_3 = 0$

2V : $ -F_2 - sin \frac{\pi}{4}F_3 = 0$

3H : $ F_4 = 0$

3V : $ F_2 + R_1 = 0$

4H : $ -cos \frac{\pi}{4}F_3 - F_4 + R_3 = 0$

4V : $ sin \frac{\pi}{4}F_3 + F_5 + R_2 = 0$

The equations above can be rewritten in matrix notation.
Notice  external force, $E$ is moved to the right hand side of the equation so it's sign is changed.

\begin{equation*}
\underbrace{
\begin{bmatrix}
-1  & 0   & 0    & 0   & 0    & 0   & 0  & 0  \\
0   & 0   & 0    & 0   & -1   & 0   & 0  & 0  \\
 1  & 0   & cos \frac{\pi}{4} & 0  & 0    & 0   & 0  & 0  \\
0   & -1  & -sin \frac{\pi}{4}& 0  & 0    & 0   & 0  & 0  \\
0   & 0   & 0    & 1   & 0    & 0   & 0  & 0  \\
0   & 1   & 0    & 0   & 0    & 1   & 0  & 0  \\
0   & 0   & -cos \frac{\pi}{4}& 0  & 0    & 0   & 0  & 0  \\
-1  & 0   & sin \frac{\pi}{4} & 0  & 1    & 0   & 1  & 0  \\
\end{bmatrix}
}_{\mathbf{coeffs}}
\cdot
\underbrace{
\begin{bmatrix}
F_1 \\
F_2 \\
F_3 \\
F_4 \\
F_5 \\
R_1 \\
R_2 \\
R_3 \\
\end{bmatrix}
}_{\mathbf{u}}
=\underbrace{
\begin{bmatrix}
1000 \\
0 \\
0 \\
0 \\
0 \\
0 \\
0 \\
0 \\
\end{bmatrix}
}_{\mathbf{k}}
\end{equation*}

The equations above can be rewritten in matrix notation.
Notice  external force, $E$ is moved to the right hand side of the equation so it's sign is changed.

\begin{equation*}
\underbrace{
\begin{bmatrix}
-1  & 0   & 0    & 0   & 0    & 0   & 0  & 0  \\
0   & 0   & 0    & 0   & -1   & 0   & 0  & 0  \\
 1  & 0   & \frac{\sqrt{2}}{2}      & 0  & 0    & 0   & 0  & 0  \\
0   & -1  & -\frac{\sqrt{2}}{2}     & 0  & 0    & 0   & 0  & 0  \\
0   & 0   & 0    & 1   & 0    & 0   & 0  & 0  \\
0   & 1   & 0    & 0   & 0    & 1   & 0  & 0  \\
0   & 0   & -\frac{\sqrt{2}}{2}     & 0  & 0    & 0   & 0  & 0  \\
-1  & 0   & \frac{\sqrt{2}}{2}      & 0  & 1    & 0   & 1  & 0  \\
\end{bmatrix}
}_{\mathbf{coeffs}}
\cdot
\underbrace{
\begin{bmatrix}
F_1 \\
F_2 \\
F_3 \\
F_4 \\
F_5 \\
R_1 \\
R_2 \\
R_3 \\
\end{bmatrix}
}_{\mathbf{u}}
=\underbrace{
\begin{bmatrix}
1000 \\
0 \\
0 \\
0 \\
0 \\
0 \\
0 \\
0 \\
\end{bmatrix}
}_{\mathbf{k}}
\end{equation*}

```python
print(np.sin(np.pi/4))
print(np.cos(np.pi/4))

print(2**(1/2)/2)
```

    0.7071067811865475
    0.7071067811865476
    0.7071067811865476

```python
# Arrays for the known values
coeffs = np.array([[-1,  0,   0,                  0,   0,  0,   0,   0 ],
                   [ 0,  0,   0,                  0,  -1,  0,   0,   0 ],
                   [ 1,  0,   np.cos(np.pi/4),    0,   0,  0,   0,   0 ],
                   [ 0, -1,   -np.sin(np.pi/4),    0,   0,  0,   0,   0 ],
                   [ 0,  0,   0,                  1,   0,  0,   0,   0 ],
                   [ 0,  1,   0,                  0,   0,  1,   0,   0 ],
                   [ 0,  0,   -np.cos(np.pi/4),    -1,   0,  0,   0,   1 ],
                   [ 0,  0,   np.sin(np.pi/4),     0,   1,  0,   1,   0 ]])

k = np.array([1000, 0, 0, 0, 0, 0, 0, 0])

# Solve for u
u = np.linalg.solve(coeffs, k)
print(u)
```

    [-1000.         -1000.          1414.21356237     0.
        -0.          1000.         -1000.          1000.        ]

```python
# Arrays for the known values

coeffs = np.array([[-1,  0,   0,             0,   0,  0,   0,   0 ],
                   [ 0,  0,   0,             0,  -1,  0,   0,   0 ],
                   [ 1,  0,   2**(1/2)/2,    0,   0,  0,   0,   0 ],
                   [ 0, -1,  -(2**(1/2)/2),  0,   0,  0,   0,   0 ],
                   [ 0,  0,   0,             1,   0,  0,   0,   0 ],
                   [ 0,  1,   0,             0,   0,  1,   0,   0 ],
                   [ 0,  0,  -(2**(1/2)/2), -1,   0,  0,   0,   1 ],
                   [ 0,  0,  2**(1/2)/2,     0,   1,  0,   1,   0 ]])

k = np.array([1000, 0, 0, 0, 0, 0, 0, 0])

u = np.linalg.solve(coeffs, k)
print(u)
```

    [-1000.         -1000.          1414.21356237     0.
        -0.          1000.         -1000.          1000.        ]

Check your solution.

The `@` symbol can be used as shorthand for `np.dot` to find the dot product.

```python
print(coeffs @ u)

print(np.around(coeffs @ u, 2))

```

    [ 1.00000000e+03  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00 -5.54631438e-14]
    [1000.    0.    0.    0.    0.    0.    0.   -0.]

##### Alternative Solution
We can find $\mathbf{u}$ by matrix multiplication:
\begin{align*}
\mathbf{coeffs}\cdot\mathbf{u}&=\mathbf{k}\\
\end{align*}

but we have to rearrange the equation.

\begin{align*}
\mathbf{u}  &=\frac{\mathbf{k}}{\mathbf{coeffs}}\\
            &=\mathbf{coeffs^{-1}}\cdot\mathbf{k}\\
\end{align*}

The Numpy linear algebra module can be used to find the inverse of matrix $\mathbf{k}$.

```python
coeffs_i = np.linalg.inv(coeffs)

u = np.dot(coeffs_i, k)

print(u)

```

    [-1000.         -1000.          1414.21356237     0.
         0.          1000.         -1000.          1000.        ]

## 3.2 Systems of Non-Linear Equations

The examples so far have been systems of linear equations.

We can also use `fsolve` to solve systems of non-linear equations.

__Example application__ : finding the coefficients of a quadratic polynomial function, using three points that lie on the curve of the function.

The equation for a parabola is $y=ax^2+bx+c$

The equation of a parabola can be fully defined using three points that lie on the curve (provided the curve is not a  straight line).

Given three points $(x_1,y_1)$, $(x_2,y_2)$, $(x_3,y_3)$,

three simultaneous equation may be written:

$$
\begin{split}
ax_1^2+bx_1+c&=y_1 \\
ax_2^2+bx_2+c&=y_2 \\
ax_3^2+bx_3+c&=y_3 \\
\end{split}
$$

__Example__

Given that:

$(x_1,y_1)=(-2,2)$

$(x_2,y_2)=(1,-1)$

$(x_3,y_3)=(4,4)$

find $a$,$b$ and $c$.

\begin{equation*}
\underbrace{
\begin{bmatrix}
x_1^2 & x_1 & 1 \\
x_2^2 & x_2 & 1 \\
x_3^2 & x_3 & 1 \\
\end{bmatrix}
}_{\mathbf{x}}
\cdot
\begin{bmatrix}
a \\
b \\
c \\
\end{bmatrix}
=\underbrace{
\begin{bmatrix}
y_1 \\
y_2 \\
y_3 \\
\end{bmatrix}
}_{\mathbf{y}}
\end{equation*}

\begin{equation*}
\underbrace{
\begin{bmatrix}
-2^2 & -2 & 1  \\
1^2 & 1 & 1 \\
4^2 & 4 & 1 \\
\end{bmatrix}
}_{\mathbf{x}}
\cdot
\begin{bmatrix}
a \\
b \\
c \\
\end{bmatrix}
=\underbrace{
\begin{bmatrix}
2 \\
-1 \\
4 \\
\end{bmatrix}
}_{\mathbf{y}}
\end{equation*}

```python
# 1D array, y
y = np.array([2, -1, 4])

# 2D array, x
x = np.array([[(-2**2), -2, 1],
              [(1**2),   1, 1],
              [(4**2),   4, 1]])

abc = np.linalg.solve(x, y)
print(abc)
```

    [ 0.8        -2.33333333  0.53333333]

# 4. Computation on Arrays: Broadcasting

 4.1 Broadcasting

 4.2 Rules of Broadcasting

 4.3 Example Uses of Broadcasting

Another means of vectorizing operations (making functions apply elementwise) is to use NumPy's *broadcasting* functionality.

Broadcasting is simply a set of rules for applying elementwise functions (e.g. addition, subtraction, multiplication...) on arrays of different sizes.

## Limitation with Array Arithmetic

For arrays of the *same size*, operations are performed on element-wise.

```python
import numpy as np
```

```python
a = np.array([0, 1, 2])
b = np.array([5, 5, 5])
a + b
```

    array([5, 6, 7])

## 4.1 Broadcasting

Broadcasting allows these types of binary operations to be performed on arrays of different sizes.

We can think of this like applying a scaler (using addition in this case) to each element of the array.

```python
a + 5
```

    array([5, 6, 7])

##### What does this code do?
This operation:
1. *stretches* or *duplicates* the value ``5`` into the array ``[5, 5, 5]``
2. and adds the results.

The advantage of NumPy's broadcasting is that we do not need to explicitly perform this duplication which can be especially helpful with large data sets.

##### Dimensions are compatible when the length of at least one of them is equal to 1.

In the dimension where `b` has size 1 and `a` has a size greater than 1 (i.e. 3), `b` behaves as if it were copied along that dimension.

This principle extends to arrays of higher dimension.

##### Dimensions are compatible when the length of at least one of them is equal to 1.

Example : Add a 1D array to a 2D array:

```python
a = np.array([0, 1, 2])
M = np.ones((3, 3))
print(M)
print(a)
```

    [[1. 1. 1.]
     [1. 1. 1.]
     [1. 1. 1.]]
    [0 1 2]

```python
M + a
```

    array([[1., 2., 3.],
           [1., 2., 3.],
           [1., 2., 3.]])

More complicated cases can involve broadcasting of both arrays.

The one-dimensional array ``a`` is stretched, or __broadcast__ across the second dimension in order to match the shape of ``M``.

```python
a = np.arange(3)

b = np.arange(3).reshape(3,1)
print(a)
print(b)
```

    [0 1 2]
    [[0]
     [1]
     [2]]

```python
a + b
```

    array([[0, 1, 2],
           [1, 2, 3],
           [2, 3, 4]])

Here we stretched *both* ``a`` and ``b`` to match a common shape, and the result is a two-dimensional array!

Summary figure adapted by ... from source published in the [astroML](http://astroml.org) documentation).

## 4.2 Rules of Broadcasting

Broadcasting in NumPy follows a strict set of rules to determine the interaction between the two arrays:

- __Rule 1:__ If the two arrays differ in their *number of dimensions*, the shape of the one with fewer dimensions is *padded* copied in the missing dimension.
- __Rule 2:__ If the shape of the two arrays does not match in any dimension, the array with shape equal to 1 in that dimension is *stretched* to match the other shape.
- __Rule 3:__ If in any dimension the sizes disagree *and* neither dimension is equal to 1, the arrays are incomatible (an error is raised).

### Broadcasting example 1

Let's look at adding a two-dimensional array to a one-dimensional array:

```python
M = np.ones((2, 3))
a = np.arange(3)
print(M)
print(a)
```

    [[1. 1. 1.]
     [1. 1. 1.]]
    [0 1 2]

Let's consider an operation on these two arrays. The shape of the arrays are

- ``M.shape = (2, 3) # 2D``
- ``a.shape = (3,) # 0D``

Rule 1 : array ``a`` has fewer dimensions --> copy in missing dimension:

- ``M.shape -> (2, 3) # 2D``
- ``a.shape -> (1, 3) # 1D``

Rule 2 : Number of columns matches --> stretch this dimension:

- ``M.shape -> (2, 3)``
- ``a.shape -> (2, 3)``

The shapes match, and we see that the final shape will be ``(2, 3)``:

```python
M = np.ones((2, 3))
a = np.arange(3)

M + a
```

    array([[1., 2., 3.],
           [1., 2., 3.]])

### Broadcasting example 2

Let's take a look at an example where *both* arrays need to be broadcast:

```python
a = np.arange(3)[:, np.newaxis]  # a = np.arange(3).reshape((3, 1))
b = np.arange(3)
print(a)
print(b)
```

    [[0]
     [1]
     [2]]
    [0 1 2]

Again, we'll start by writing out the shape of the arrays:

- ``a.shape = (3, 1)``
- ``b.shape = (3,)``

Rule 1 : array ``b`` has fewer dimensions --> copy in missing dimension

- ``a.shape -> (3, 1)``
- ``b.shape -> (1, 3)``

Rule 2 : Number of columns and rows mis-match. At least one is equal to 1 --> stretch this dimension:

- ``a.shape -> (3, 3)``
- ``b.shape -> (3, 3)``

Because the result matches, these shapes are compatible. We can see this here:

```python
a = np.arange(3)[:, np.newaxis]  # a = np.arange(3).reshape((3, 1))
b = np.arange(3)

a + b
```

    array([[0, 1, 2],
           [1, 2, 3],
           [2, 3, 4]])

### Broadcasting example 3

Now let's take a look at an example in which the two arrays are not compatible:

```python
M = np.ones((3, 2))
a = np.arange(3)
print(M)
print(a)
```

    [[1. 1.]
     [1. 1.]
     [1. 1.]]
    [0 1 2]

This is just a slightly different situation than in the first example: the matrix ``M`` is transposed.
How does this affect the calculation? The shape of the arrays are

- ``M.shape = (3, 2) # 2D``
- ``a.shape = (3,)   # 0D``

Rule 1 : array ``a`` has fewer dimensions --> copy in missing dimension:

- ``M.shape -> (3, 2)``
- ``a.shape -> (1, 3)``

Rule 3 : If in any dimension the sizes disagree *and* neither dimension is equal to 1 --> incompatibe.

- ``M.shape -> (3, 2)``
- ``a.shape -> (3, 3)``

Now we hit rule 3–the final shapes do not match, so these two arrays are incompatible, as we can observe by attempting this operation:

```python
M = np.ones((3, 2))
a = np.arange(3)

M + a
```

## 4.3 Example Uses of Broadcasting

### Broadcasting Example: Calorie Calculator

Let's say we have a large data set; each datum is a list of parameters.

Example datum: a type of food and the the amount of fat, protein and carbohydrate in a serving of that food.

Our data set of food nutrients might look something like the table below:

|Food (100g) |Fat(g)|Protein (g)|Carbohydrate (g)|
|------------|----- |-----------|----------------|
|Almonds     |    49|         21|              22|
|Peas        |     0|          5|              14|
|Avocado     |    15|          2|               9|
|Kale        |     1|          3|              10|

By applying the following sclaing factors, we can calculate the number of calories in a food type due to fat, protein and carbohydrate:
 -  fat: 9 cal/g
 -  protein: 4 cal/g
 -  carbohydrate 4 cal/g

Using what we have studied so far, we could convert the table to calories using a loop:

```python

nutrients = np.array([[49, 21, 22],
                      [0,   5, 14],
                      [15,  2,  9],
                      [ 1,  3, 10]])

cal_convert = np.array([9, 4, 4])

calories = np.empty((4,3))

for index, value in enumerate(nutrients):
    calories[index] = value * cal_convert

```

However, it is faster and more concise to broadcast the two arrays together:

```python
nutrients = np.array([[49, 21, 22],
                      [0,   5, 14],
                      [15,  2,  9],
                      [ 1,  3, 10]])

cal_convert = np.array([9, 4, 4])

calories = nutrients * cal_convert

print(calories)
```

### Broadcasting Example: Matrix Generation

Taking the example from earlier, where we solve a system of simultaneous equations

$$
\begin{split}
ax_1^2+bx_1+c&=y_1 \\
ax_2^2+bx_2+c&=y_2 \\
ax_3^2+bx_3+c&=y_3 \\
\end{split}
$$

we can write the equations in array or matrix form and solve for the missing values $a, b, c$

\begin{equation*}
\underbrace{
\begin{bmatrix}
-2^2 & -2 & 1  \\
1^2 & 1 & 1 \\
4^2 & 4 & 1 \\
\end{bmatrix}
}_{\mathbf{x}}
\cdot
\begin{bmatrix}
a \\
b \\
c \\
\end{bmatrix}
=\underbrace{
\begin{bmatrix}
2 \\
-1 \\
4 \\
\end{bmatrix}
}_{\mathbf{y}}
\end{equation*}

```python
# 1D array, y
y = np.array([2, -1, 4])

# 2D array, x
x = np.array([[(-2**2), -2, 1],
              [(1**2),   1, 1],
              [(4**2),   4, 1]])

abc = np.linalg.solve(x, y)
print(abc)
```

Typing this out, particularly matrix x is rather tedious.

Another way to create matrix x without writing it out explicitly is to use:
- broadcasting
- the numpy function, `power()` (raises first argument to power of second)

```python
# array
xs = np.array([-2, 1, 4])
#print(xs, end='\n\n')

#vertical 3x1
xs = xs[:, np.newaxis] # xs = xs.reshape((3,1))
print(xs, end='\n\n')

# horizontal array 0x3
power = np.array([2, 1, 0])
print(power, end='\n\n')

# 3x3
x = np.power(xs, power)
print(x, end='\n\n')
```

### Broadcasting Example: Vector Quantisation Algorithm
This is a simple algorithm used for catagorisation.

It determines which catagory a data point should belong to from its closest proximity to a set of values representing possible catagories.

Each value represents the mean of the corresponding catagory.

For example, colour quantisation is used in image processing reduces the number of distinct colors used in an image, while maintianing visual similarity to the original image.

CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1477223

In the plot below, each of the circles represents the mean height and weight of athletes grouped by type.

The square represents the height and weight of an athlete to be classified.

To find the closet point:
1. Use broadcasting to find the difference between the position of the __square__ and the position of each __circle__ in the x and y directions.

1. Find the euclidian distance, $d$ from the square, $s$ to each circle, $c$ using:
$d = \sqrt{(x_{c}-x_{s})^2 + (y_{c}-y_{s})^2}$

1. Choose the group corresponding to the minimum distance, $d_{min}$

```python
athlete = np.array([111.0,188.0])

categories = np.array([[102.0, 203.0],
                       [132.0, 193.0],
                       [45.0, 155.0],
                       [57.0, 173.0]])

# 1. broadcast
diff = categories - athlete

# 2. distance to each point (magnitude of values along axis 1 for each datum)
# dist = np.linalg.norm(diff,axis=1)
dist = np.sqrt(np.sum(diff**2,axis=1))

# 3. which group?
nearest = np.argmin(dist)
print(nearest)
```

The nearest group is index 0 of the array `catagories`.

Based on mean height and weight, the athlete is most likely to be a basketball player.

### Broadcasting Example : Plotting a two-dimensional function
One place that broadcasting is very useful is in displaying images based on two-dimensional functions.
If we want to define a function $z = f(x, y)$, broadcasting can be used to compute the function across the grid:

```python
# x and y have 50 steps from 0 to 5
x = np.linspace(0, 5, 50)
y = np.linspace(0, 5, 50)[:, np.newaxis]

z = np.sin(x) ** 10 + np.cos(10 + y * x) * np.cos(x)
```

```python
# We can use Matplotlib `imshow` to represent 3D data
%matplotlib inline
import matplotlib.pyplot as plt

plt.imshow(z, origin='lower', extent=[0, 5, 0, 5],
           cmap='viridis')
plt.colorbar();
```

## Vectorising Functions

Numpy functions applied to a single array, will be performed on each element in the array.

The function takes an array of values as an input argument.

```python
a = np.array([2, 3, 4])

print(np.sqrt(a))

print(a ** (1/2))
```

For example, we can apply trigonometric functions, elementwise, to arrays, and lists.

```python
x = np.array([0.0, np.pi/2, np.pi, 3*np.pi/2])
y = [0.0, np.pi/2, np.pi, 3*np.pi/2]
z = (0.0, np.pi/2, np.pi, 3*np.pi/2)

print(np.sin(x))
print(np.cos(y))
print(np.tan(z))
```

An array of values does not work as an input for all functions.

```python
def func(x):
    if x < 0:
        f = 2 * x
    else:
        f = 3 * x
    return f

x = np.array([2, -2])

#y = func(x) # Run this line after removing the # to see the error generated
```

This doesn't work because Python doesn't know what to do with the line

`if x < 0`

when `x` contains many values.

For some values of `x` the `if` statement may be `True`, for others it may be `False`.

A simple way around this problem is to vectorise the function.

We create a new function that is a *vectorized* form of the original function.

The new function and can be called with an array as an argument.

```python
funcvec = np.vectorize(func)

print(funcvec(x))
```

### Vectoried functions as function arguments

Functions that receive vectorised arguments are automatically applied to all elements of array inputs.

This is better exlpained with an example.

Recall the function `is_positive`:

```python
x = np.array([2, -2])

def is_positive(f, a):
    "Checks if the function value f(x) is positive"
    return f(a) > 0

# Apply is_positive to a non-vectorised function
print(is_positive(func, -3))
print()

# Apply is_positive to a vectorised function
print(x)
is_positive(funcvec, x)
```

# 5. Symbolic Mathematics

 5.1 Symbolic Representation

 5.2 Re-aranging using Symbolic Mathematics

It can be useful to express a solution in the form of a symbolic expression.

The `Sympy` package is designed for symbolic mathematics.

```python
import sympy
from sympy import solve, symbols, Symbol, Function, Eq, pprint
```

## 5.1 Symbolic Representation
To write a symbolic equation, we first create variables to store the symbols we want to use:

```python
a, b, c, x = symbols('a, b, c, x')
```

We then write an equation, using the variable names.

$$f(x) = ax^2 + bx + c$$

```python
f = a*x**2 + b*x + c
```

Let's *symbolically* find the roots of the function $f(x)$.

`sympy.solve()` works in the same way as `numpy.linalg.solve()` but for symbolic mathematics.

```python
solution = solve(f, x)

print(solution)
```

This equation takes the familar form of the __quadratic formula__ $$\frac{-b \pm \sqrt{b^2-4ac}}{2a}$$

However, it is difficult to tell from the way it is written.

`sympy.pprint` (pretty print) makes the solution more readable:

```python
from sympy import pprint
pprint(solution)
print(solution)
```

We can use `subs` to substitute in numerical values for the symbolic variables in the solution.

In this case, there are two solutions (returned as a list) so we address each index of the list by index:

```python
A, B, C = -1, 2, 3

x0 = solution[0].subs([(a, A),
                       (b, B),
                       (c, C)])

x1 = solution[1].subs([(a, A),
                       (b, B),
                       (c, C)])

display(x0, x1)
```

## 5.2 Re-aranging using Symbolic Mathematics
It can be useful to change the subject of an equation by rearranging.

Example, change the subject of the following equation to $P$.

```python
# create a symbolic representation of all values
ai, af, bi, bf, v , E , P , P_= sympy.symbols('ai af bi bf v  E  P P_')

# make a symbolic equation
bf_eq = sympy.Eq(bf, (af**3 / (bf**3 - af**3)) * ( (1 - 2*v) * bf + (1 + v) * bf**3 / (2*bf**2) ) * (P/E) + bi)
sympy.pprint(bf_eq)
```

##### What does this code do?
`bf_eq` is a Sympy __`Eq`__ (equation) object.

The arguments for __`Eq`__ are the left and right hand side of the equation to solve.

(__Note :__ `bf_eq` is different from the variable `bf`).

In other words `bf_eq` is the name given to the expression `bf = ...`

```python
# re-arrange for P using solve
pprint(sympy.solve(bf_eq, P))      # all solutions

P_eq = sympy.Eq(P, solve(bf_eq, P)[0]) # This be written as one line...

P_expr = solve(bf_eq, P)[0]            # ...or two lines
P_eq = sympy.Eq(P, P_expr)

sympy.pprint(P_eq)
```

Numerical values can then be substituted into the solution.

Notice we substitute into the SOLUTION not the equation.

```python
# substitute in numerical values
# notice we substitute into the SOLUTION not the equation

sol_1 = P_expr.subs(E, 3.48e-6).subs(ai,12700).subs(bi, 8031).subs(bf, 3).subs(v, 1000+273).subs(af, 2)

# substitute in initial condition
sol_2 = P_expr.subs([(E, 3.48e-6),       # E = 3.48e-6 --> subs 3.48e-6 for E
                     (ai,12700),
                     (bi, 8031),
                     (bf, 3),
                     (v, 1000+273),
                     (af, 2),
                     ])

# substitute in initial condition
sol_3 = P_expr.subs([(E, 3.48e-6)])       # E = 3.48e-6 --> subs 3.48e-6 for E

print(sol_1)
print(sol_2)
pprint(sol_3)
```

# Summary
- `numpy.roots()` finds all roots of a polynomial function.
- `scipy.optimize.fsolve()` finds a root of a function, given a starting estimate.
- `numpy.linalg.solve()` solves equations of the form `ax = b` (including matrices representing systems of equations e.g coupled linear and non-linear equations) for x
- `scipy.optimize.fsolve()` can also be used to solve couple nonlinear equations.
- `Sympy` package is designed for symbolic mathematics
- `sympy.solve()` works in the same way as `numpy.linalg.solve()` but for symbolic mathematics. It can also be used for re-arranging equations.

# 6. Review Exercises

Compete the exercises below.

## Review Exercise 1: Root finding

Fit a polynomial function to the data below.

Find the roots of the fitted polynomial.

```python
# Review Exercise 1: Root finding
x = [-6.0, -5.555555555555555, -5.444444444444445, -5.333333333333333, -5.0, -4.777777777777778, -4.555555555555555, -4.333333333333334, -4.111111111111111, -3.555555555555556, -3.3333333333333335, -3.2222222222222223, -2.2222222222222223, -1.8888888888888893, -1.7777777777777777, -1.5555555555555554, -1.4444444444444446, -1.333333333333334, -1.2222222222222223, -1.1111111111111116, -0.7777777777777777, -0.5555555555555562, -0.3333333333333339, -0.22222222222222232, 0.33333333333333304, 0.5555555555555554, 0.6666666666666661, 0.8888888888888884, 1.0, 1.2222222222222214, 1.333333333333333, 1.4444444444444438, 1.666666666666666, 1.7777777777777777, 2.1111111111111107, 2.333333333333332, 2.5555555555555554, 2.777777777777777, 2.8888888888888893, 3.1111111111111107, 3.2222222222222214, 3.5555555555555554, 3.777777777777777, 4.0, 4.111111111111111, 4.222222222222221, 4.333333333333332, 4.444444444444445, 4.8888888888888875, 5.0]
y = [-306.0670724099247, -273.4252575751447, -236.35910170243054, -2.147806809067588, -162.88428946693543, -72.0539258242078, -49.64195238514043, -75.05934686306523, -49.40805793483066, -15.803160491117433, -20.408192287721462, -34.04243919689319, -2.6008654388252075, -0.33819910212586596, 0.5967691522163541, 1.955165125544544, 0.754741501848223, 3.1485956879192134, 0.2736824650635393, 2.535463038423905, 2.0383401626385638, 0.8371085078493934, 0.27326740330999844, -0.14152399821562134, -0.15792222719404883, -1.357836647665497, -4.064496618469092, -2.2060777524379893, -6.716174537753252, -2.381049714701943, -0.8951333867263299, -3.703956978393335, -5.121504730336851, -1.4824097773484555, -0.0658532580151797, 2.5527247901789907, 9.310234512028755, 7.839090794578473, 0.8239015424106111, 27.801254862532222, 33.099581728518, 17.182186572769048, 63.28883410018085, 38.47325866392358, 74.26392095969987, 100.73153613329536, 119.19508682705471, 46.85235728093459, 175.63882495054517, 118.62483544333234]
```

```python

```

```python

```

## Review Exercise 2: Root Finding using an Initial Estimate

__Example:__ Find the root of the cosine function that is closest to -5.

```python
# Review Exercise 2: Finding the Closest Root to a Point
# Example Solution
from scipy.optimize import fsolve
print(fsolve(np.cos, -5))
```

## Review Exercise 3: Systems of Equations
### Example Engineering Application: An Electrical Circuit

#### Kirchhoff's Voltage Law
For a closed loop series path the algebraic sum of all the *voltages* and *voltage drops* around any closed loop in a circuit is equal to zero.

$\sum E - \sum V = 0 $

#### Electrical Elements Obey Ohm's Law
The current through a conductor (I, units amps) is the voltage measured across the conductor (V, units volts) divided by the resistance (R, units Ohms).

$$V = IR$$

Consider a three loop current network with five resistors and
two voltage sources.

Here we have three loops, hence we can write three equations
to use resitances R1, R2, R3, R4, R5 and voltages v1, v2, to solve for the three unknowns, the currents: i1, i2, i3.

We can use Kirchoff's voltage law to equate the voltage and voltage drop in each loop:

$\sum V = \sum E$

and Ohm's law : $V=IR$

__Loop 1:__   $ (R_1 + R_2) i_1 + i_2 R_2 = v_1$

__Loop 2:__   $ -R_2 i_1 + (R_2 + R_3 + R_4)i_2 - R_4 i_3 = 0$

__Loop 3:__   $ -R_4 i_2 + (R_4 + R_5) i_3 = -v_2$

Putting the equations in matrix form:

\begin{equation*}
\underbrace{
\begin{bmatrix}
(R_1 + R_2) & -R_2 & 0  \\
-R_2        & (R_2 + R_3 + R_4)  & -R_4  \\
0           & -R_4               & (R_4 + R_5)  \\
\end{bmatrix}
}_{\mathbf{R}}
\cdot
\underbrace{
\begin{bmatrix}
i_1 \\
i_2 \\
i_3 \\
\end{bmatrix}
}_{\mathbf{I}}
=\underbrace{
\begin{bmatrix}
v_1 \\
0 \\
-v_2 \\
\end{bmatrix}
}_{\mathbf{V}}
\end{equation*}

Given the following resitance and voltage values,  solve the system of equations to find the three unknown currents: i1, i2, i3.

$R1=1K\Omega$

$R2=300\Omega$

$R3=500\Omega$

$R4=1K\Omega$

$R5=300\Omega$

$v1 = 2V$

$v2 = 5V$

```python

```

```python
# Review Exercise 3: Systems of Equations
```

## Review Exercise 4: Symbolic math

$$ y = \frac{x^P}{4d} $$

Make $x$ the subject of the equation.

Using symbolic substitution, find the value of $x$ when:

$P = 12$

$d = 4$

$y = 2$

```python
# Review Exercise 4: Symbolic math
```

```python
import numpy as np

```

```python
x = np.pi
print(np.sin(x))
```

    1.2246467991473532e-16

```python

```


# Linear Regression - Sklearn, Statsmodel, TensorFlow, Keras, PyTorch

Author: Harry Yau

Date: September 22, 2019

Updated: September 24, 2019

The goal of this notebook is to compute a Linear Regression using different frameworks. The frameworks that will be investigated are:
- Sklearn
- Statsmodel
- TensorFlow 1.0
- Keras
- PyTorch
- TensorFlow 2.0

The Ordinary Least Squares equation is the following:

\begin{equation}
Y_i = \beta_0 + \beta_1 X_i + \epsilon_i
\end{equation}

The data will be created according to this equation.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
```

```python
def plot_output(X, y, predicted=None):
    '''
    Plots Regression Line Charts

    Arguments:
    X - Accepts a DataFrame or numpy array for the X data
    y - Accepts a DataFrame or numpy array for the y data
    predicted - Accepts a DataFrame or numpy array for the predicted y data. If None, this function only plots the scatter plot.
    '''

    plt.clf()
    plt.plot(X, y, 'go', label='True Data', alpha=0.5)
    if predicted is not None:
        plt.plot(X, predicted, '--', label='Predictions', alpha=0.5)
        plt.legend(loc='best')
    plt.show();
```

```python
np.random.seed(1) #Setting a seed to recreate experiment.
num_points = 250 #The number of data points

X = 2.5 * np.random.randn(num_points) + 1.5   # Array of 100 values with mean = 1.5, stddev = 2.5
res = 0.5 * np.random.randn(num_points)       # Generate 100 residual terms
y = 2 + 0.3 * X + res                  # Actual values of Y
```

```python
plot_output(X, y)
```

Reshaping X and Y necessary when the data has one column so that sklearn and pytorch will work properly. In addition, the numpy array is forced to have dtype np.float32 so that pytorch will accept these values.

```python
X = np.array(X, dtype=np.float32).reshape(-1,1)
y = np.array(y, dtype=np.float32).reshape(-1,1)
```

Scaling the data so that it helps with convergence when running gradient descent in TensorFlow, Keras and Pytorch.

```python
from sklearn.preprocessing import StandardScaler

X_scaler = StandardScaler()
X_scaled = X_scaler.fit_transform(X)

y_scaler = StandardScaler()
y_scaled = y_scaler.fit_transform(y)
```

```python
def plot_output(X, y, predicted=None):
    '''
    Plots Regression Line Charts

    Arguments:
    X - Accepts a DataFrame or numpy array for the X data
    y - Accepts a DataFrame or numpy array for the y data
    predicted - Accepts a DataFrame or numpy array for the predicted y data
    '''

    plt.clf()
    plt.plot(X, y, 'go', label='True Data', alpha=0.5)
    if predicted is not None:
        plt.plot(X, predicted, '--', label='Predictions', alpha=0.5)
        plt.legend(loc='best')
    plt.show();
```

### Sklearn

```python
from sklearn.linear_model import LinearRegression

lin_model = LinearRegression() #Set the model
lin_model.fit(X,y) #Fit the model
predicted = lin_model.predict(X)
plot_output(X, y, predicted)
```

### Statsmodel

```python
import statsmodels.api as sm # For constants
from statsmodels.regression import linear_model

#Compared to SKLearn, statsmodel doesn't include the bias term.
X_c = sm.add_constant(X)
linear_model_sm = linear_model.OLS(y, X_c).fit()  #Fit the model
predicted = linear_model_sm.predict(X_c)
plot_output(X, y, predicted)
```

### Standardized Settings for TensorFlow, Keras and PyTorch

```python
input_dim = 1        # takes variable 'x' 1 input variable.
output_dim = 1       # takes variable 'y' 1 output variable.
learning_rate = 0.01
epochs = 1000
```

### TensorFlow 1.0

Steps to calculate a linear regression using TensorFlow. The goal is to create the placeholders and variables to satisfy the following linear equation:
$$ y = X^TW + b$$

```python
import tensorflow as tf
print('Version:', tf.__version__)
```

    Version: 1.14.0

Create placeholders for X and y, because these values will be supplied in the opimization step using feed_dict.

```python
X_tf = tf.placeholder(dtype=tf.float32, shape=[None, input_dim])
y_tf = tf.placeholder(dtype=tf.float32, shape=[None, output_dim])
```

Create variables for W and b. Initializing W with Xavier/Glorot Initialization and b with zeros. W and B are updated during gradient descent so that its why they are variables instead of placeholders

```python
W_tf = tf.Variable(tf.glorot_uniform_initializer()((input_dim, output_dim)), dtype=tf.float32, name='W')
b_tf = tf.Variable(tf.zeros_initializer()((output_dim)), dtype=tf.float32, name='b')
```

    WARNING:tensorflow:From C:\Users\yauha\Anaconda3\envs\tf\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Call initializer instance with the dtype argument instead of passing it to the constructor

Define the y predicted equation

```python
#WX + b
y_pred_tf = tf.add(tf.matmul(X_tf, W_tf), b_tf)
```

Define the Cost Function.
In the case of Linear Regression, the Mean Squared Error is being minimized:
$$MSE = \frac{1}{n} \sum_{i=1}^n (\hat{y_i} - y_i)^2$$

```python
#The cost function
cost_tf = tf.losses.mean_squared_error(y_tf, y_pred_tf)
```

    WARNING:tensorflow:From C:\Users\yauha\Anaconda3\envs\tf\lib\site-packages\tensorflow\python\ops\losses\losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where

Set the optimizer.

```python
#The optimizer
optimizer_tf = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost_tf)
```

Set the Global Variables Initializer. This step is mandatory.

```python
#Global Variables Initilializer -- Mandatory
init = tf.global_variables_initializer()
```

Running the computation graph.

```python
#The Tensorflow Session
with tf.Session() as sess:
    sess.run(init)

    for epoch in range(epochs):

        #The dictionary to feed in the variables for hte optimizer
        feed_dict = {X_tf: X_scaled, y_tf: y_scaled}

        #Run the optimizer at every epoch
        #Also saves the weight and bias.
        _, weight, bias = sess.run([optimizer_tf, W_tf, b_tf], feed_dict)

        if (epoch + 1) % 50 == 0:
            #Calculate the cost at every epoch
            print("Epoch", (epoch + 1), ": cost =", cost_tf.eval(feed_dict))
```

    Epoch 50 : cost = 0.32126582
    Epoch 100 : cost = 0.31229258
    Epoch 150 : cost = 0.31110257
    Epoch 200 : cost = 0.31094474
    Epoch 250 : cost = 0.31092378
    Epoch 300 : cost = 0.310921
    Epoch 350 : cost = 0.31092066
    Epoch 400 : cost = 0.3109206
    Epoch 450 : cost = 0.3109206
    Epoch 500 : cost = 0.3109206
    Epoch 550 : cost = 0.31092063
    Epoch 600 : cost = 0.3109206
    Epoch 650 : cost = 0.3109206
    Epoch 700 : cost = 0.3109206
    Epoch 750 : cost = 0.3109206
    Epoch 800 : cost = 0.3109206
    Epoch 850 : cost = 0.3109206
    Epoch 900 : cost = 0.3109206
    Epoch 950 : cost = 0.3109206
    Epoch 1000 : cost = 0.3109206

8. Calculate the y predicted values. Note: Remember to unscale your data.

```python
predicted = weight * X_scaled + bias
predicted = y_scaler.inverse_transform(predicted)
```

```python
plot_output(X, y, predicted)
```

### Keras

Keras is a high-level API that uses TensorFlow as its backend. Creating a model is simpler with the Keras API. Essentially this is similar to putting together lego bricks.

```python
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
print('Version:', tf.keras.__version__)
```

    Version: 2.2.4-tf

Creating a Sequential model in Keras. A Sequential model is when layers are linearly stacked.
- For a Linear Regression, the activation layer will be 'linear' and the output dimension being 1 due to a value being a regression.
- The input shape needs to be defined.
- This layer is essentially y = Wx + b, with an activation of 'linear'. 'linear' means that no additional calculations will be done and will return the identity y = Wx + b.

```python
model = Sequential([
    Dense(output_dim, activation='linear',  input_shape=(input_dim,)),
])
```

Setting the loss function, optimizer and then compiling the model. This step configures the model for training

```python
optimizer = tf.keras.optimizers.SGD(learning_rate)

model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse'])
```

```python
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense (Dense)                (None, 1)                 2
    =================================================================
    Total params: 2
    Trainable params: 2
    Non-trainable params: 0
    _________________________________________________________________

Training the model using fit()

```python
history = model.fit(x = X_scaled, y = y_scaled, epochs = epochs, verbose=0)
```

Calculate the y predicted values using the predict() function on the model. Note: Remember to unscale your data.

```python
predicted = model.predict(X_scaled)
predicted = y_scaler.inverse_transform(predicted)
```

```python
plot_output(X, y, predicted)
```

### PyTorch

PyTorch is a framework created by Facebook. Implementing the model is slightly different. The most apparent differences is the way forward propagation is coded and GPU implementation is not seamless.

```python
import torch
print('Version:', torch.__version__)
```

    Version: 1.2.0

Create the model.
- If manually creating the tensors, must wrap with torch.nn.Parameter
- torch.nn.Linear is a layer that applies a linear transformation to y = Wx + b
- The forward propagation passes x to the linear transformation

Generally, the layer method is used in tutorial.

```python
class LinearRegressionModel_PyT(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressionModel_PyT, self).__init__()
        self.W = torch.nn.Parameter(torch.nn.init.xavier_uniform_(torch.empty(input_dim, output_dim)))
        self.b = torch.nn.Parameter(torch.zeros(output_dim))

    def forward(self, x):
        out = torch.matmul(x, self.W) + self.b
        return out

class LinearRegressionModel_PyT_Layers(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressionModel_PyT_Layers, self).__init__()
        self.linear = torch.nn.Linear(input_dim, output_dim)

    def forward(self, x):
        out = self.linear(x)
        return out
```

Initiate the model.

```python
model = LinearRegressionModel_PyT_Layers(input_dim, output_dim)

####### For GPU #######
if torch.cuda.is_available():
    model.cuda()
```

Initiate the cost function and the optimizer.

```python
criterion = torch.nn.MSELoss()
optimiser = torch.optim.SGD(model.parameters(), lr = learning_rate)
```

Converting the numpy data to torch tensors.

```python
# Converting X inputs and y labels to torch tensors
if torch.cuda.is_available():
    ####### For GPU #######
    X_data = torch.from_numpy(X_scaled).cuda()
    y_data = torch.from_numpy(y_scaled).cuda()
else:
    X_data = torch.from_numpy(X_scaled)
    y_data = torch.from_numpy(y_scaled)
```

Training the model.
 - Clear gradient buffer
 - Forward propagation
 - Get loss
 - Get gradients from backward propagation
 - Update parameters for gradient descent
 - Rinse and repeat

```python
for epoch in range(epochs):
    # Clear gradient buffers in the optimiser so that it doesn't carry forward
    optimiser.zero_grad()

    #The forward propagation step.
    y_pred = model(X_data)

    # Get loss for the predicted output
    loss = criterion(y_pred, y_data)

    # Get gradients w.r.t to parameters. This is the backward propagation step.
    loss.backward()

    # Update parameters for gradient descent
    optimiser.step()
    if (epoch + 1) % 50 == 0:
        print('epoch {}, loss {}'.format(epoch + 1, loss.item()))
```

    epoch 50, loss 0.3988889157772064
    epoch 100, loss 0.32258695363998413
    epoch 150, loss 0.3124677836894989
    epoch 200, loss 0.3111257553100586
    epoch 250, loss 0.31094780564308167
    epoch 300, loss 0.310924232006073
    epoch 350, loss 0.3109210729598999
    epoch 400, loss 0.31092068552970886
    epoch 450, loss 0.3109206259250641
    epoch 500, loss 0.3109205961227417
    epoch 550, loss 0.3109205961227417
    epoch 600, loss 0.3109206259250641
    epoch 650, loss 0.3109205961227417
    epoch 700, loss 0.3109205961227417
    epoch 750, loss 0.3109205961227417
    epoch 800, loss 0.3109205961227417
    epoch 850, loss 0.3109205961227417
    epoch 900, loss 0.3109205961227417
    epoch 950, loss 0.3109205961227417
    epoch 1000, loss 0.3109205961227417

Calculate the y predicted by passing in the data into the model.Note: Remember to unscale your data.

```python
with torch.no_grad():
    if torch.cuda.is_available():
        predicted = model(torch.from_numpy(X_scaled).cuda()).cpu().data.numpy()
    else:
        predicted = model(torch.from_numpy(X_scaled)).data.numpy()

predicted = y_scaler.inverse_transform(predicted)
```

```python
plot_output(X, y, predicted)
```

## Note: The environment has been switched at this point of the notebook

### Tensorflow 2.0

The API has changed drastically in TensorFlow 2.0. The structure is very similar to PyTorch.

```python
import tensorflow as tf
print('Version:', tf.__version__)
print('Version:', tf.keras.__version__)
```

    Version: 2.0.0-rc2
    Version: 2.2.4-tf

Creating the model.

Differences from TensorFlow 1.0:
- `tf.placeholders()`, `tf.Session()` has been removed
- `tf.glorot_uniform_initializer()` is gone, and has become `tf.keras.initializers.GlorotUniform()`

Alternatively, the model class can utilize the layers in Keras. This is analogous to PyTorch.

```python
class LinearRegressionModel_TF2(tf.keras.Model):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressionModel_TF2, self).__init__()
        self.W = tf.Variable(tf.keras.initializers.GlorotUniform()((input_dim, output_dim)), dtype=tf.float32, name='W')
        self.b = tf.Variable(tf.zeros_initializer()((output_dim)), dtype=tf.float32, name='b')

    #In PyTorch the function will be 'forward' instead.
    def call(self, x):
        out = tf.matmul(x, self.W) + self.b
        return out

class LinearRegressionModel_TF2_Layers(tf.keras.Model):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressionModel_TF2_Keras, self).__init__()
        self.d1 = tf.keras.layers.Dense(output_dim, activation='linear', input_dim=(input_dim,))

    def call(self, x):
        x = self.d1(x)
        return x
```

Initiate the model.

```python
model = LinearRegressionModel_TF2(input_dim, output_dim)
```

Initiate the loss function, gradient function, and optimzer.

```python
@tf.function
def loss(model, inputs, targets):
    loss_object = tf.keras.losses.MeanSquaredError()

    #Forward propagation
    predictions = model(inputs)

    #Get loss for the predicted output
    output = loss_object(targets, predictions)
    return output

@tf.function
def grad(model, inputs, targets):
    #For automatic differentiation
    with tf.GradientTape() as tape:
        loss_value = loss(model, inputs, targets)

    #This calculates the gradient.
    return tape.gradient(loss_value, model.trainable_variables)

optimizer = tf.keras.optimizers.SGD(learning_rate)
```

```python
for epoch in range(epochs):

    #This is the backward propagation step to get the parameters
    grads = grad(model, X_scaled, y_scaled)

    #Update parameters for gradient descent
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

    if (epoch+1) %  50 == 0:
        template = 'Epoch {}, Loss: {}'
        print(template.format(epoch+1, loss(model, X_scaled, y_scaled)))

```

    Epoch 50, Loss: 1.1609936952590942
    Epoch 100, Loss: 0.4236568808555603
    Epoch 150, Loss: 0.32587164640426636
    Epoch 200, Loss: 0.31290340423583984
    Epoch 250, Loss: 0.31118354201316833
    Epoch 300, Loss: 0.3109554648399353
    Epoch 350, Loss: 0.3109252452850342
    Epoch 400, Loss: 0.31092119216918945
    Epoch 450, Loss: 0.31092068552970886
    Epoch 500, Loss: 0.3109205961227417
    Epoch 550, Loss: 0.3109205961227417
    Epoch 600, Loss: 0.3109205961227417
    Epoch 650, Loss: 0.3109205961227417
    Epoch 700, Loss: 0.3109205961227417
    Epoch 750, Loss: 0.3109205961227417
    Epoch 800, Loss: 0.3109205961227417
    Epoch 850, Loss: 0.3109205961227417
    Epoch 900, Loss: 0.3109205961227417
    Epoch 950, Loss: 0.3109205961227417
    Epoch 1000, Loss: 0.3109205961227417

```python
predicted = y_scaler.inverse_transform(model(X_scaled))

plot_output(X, y, predicted)
```

Alternatively, the model can be ran through a .compile(), .fit(), and .predict() from the Keras API.

```python
model = LinearRegressionModel_TF2(input_dim, output_dim)
model.compile(loss=tf.keras.losses.MeanSquaredError(), optimizer=optimizer)
model.fit(x=X_scaled, y=y_scaled, epochs=epochs, verbose=0)
```

    <tensorflow.python.keras.callbacks.History at 0x1ab8d6e1a58>

```python
plot_output(X, y, y_scaler.inverse_transform(model.predict(X_scaled)))
```

Conclusion: The code for TensorFlow 2.0 is a lot cleaner than TensorFlow 1.0


# Model equations

$$ \dot{x}_i = \frac{f_i(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)} - x_i $$

where the number of offspring with genotype $i$ (i.e., fitness), f $f_i$, is an average of partial fecundities

$$ f_i(x; T, R, P, S, M, \epsilon) = x_1F_{i,1}(x; T, R, P, S, M, \epsilon) + x_2F_{i,2}(x; T, R, P, S, M, \epsilon) + x_3F_{i,3}(x; T, R, P, S, M, \epsilon) + (1 - x_1 - x_2 - x_3)F_{1,4}(x; T, R, P, S, M, \epsilon) $$

and the number of total offpspring across all genotypes, $F$, is the average of the total fecundities of the two females.

$$ F(x; T, R, P, S, M) = x_G F_G(x_A; T, R, P, S, M) + (1 - x_G)F_g(x_A; T, R, P, S, M) $$

# Partial derivatives

\begin{align}
    \frac{\partial \dot{x}_i}{\partial x_i} =& \frac{1}{F(x; T, R, P, S, M)}\left(\frac{\partial f_i(x; T, R, P, S, M, \epsilon)}{\partial x_i} - \frac{f_i(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_i}\right) - 1 \\
    \frac{\partial \dot{x}_i}{\partial x_j} =& \frac{1}{F(x; T, R, P, S, M)}\left(\frac{\partial f_i(x; T, R, P, S, M, \epsilon)}{\partial x_j} - \frac{f_i(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_j}\right)
\end{align}

## Jacobian

$$
\frac{1}{F(x; T, R, P, S, M)}
\begin{bmatrix}
    \left(\frac{\partial f_1(x; T, R, P, S, M, \epsilon)}{\partial x_1} - \frac{f_1(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_1}\right) & \left(\frac{\partial f_1(x; T, R, P, S, M, \epsilon)}{\partial x_2} - \frac{f_1(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_2}\right) & \left(\frac{\partial f_1(x; T, R, P, S, M, \epsilon)}{\partial x_3} - \frac{f_1(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_3}\right) \\
    \left(\frac{\partial f_2(x; T, R, P, S, M, \epsilon)}{\partial x_1} - \frac{f_2(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_1}\right) & \left(\frac{\partial f_2(x; T, R, P, S, M, \epsilon)}{\partial x_2} - \frac{f_2(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_2}\right) & \left(\frac{\partial f_2(x; T, R, P, S, M, \epsilon)}{\partial x_3} - \frac{f_2(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_3}\right) \\
    \left(\frac{\partial f_3(x; T, R, P, S, M, \epsilon)}{\partial x_1} - \frac{f_3(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_1}\right) & \left(\frac{\partial f_3(x; T, R, P, S, M, \epsilon)}{\partial x_2} - \frac{f_3(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_2}\right) & \left(\frac{\partial f_3(x; T, R, P, S, M, \epsilon)}{\partial x_3} - \frac{f_3(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\frac{\partial F(x; T, R, P, S, M)}{\partial x_3}\right)
\end{bmatrix}
-
I
$$

which we can further simplify as follows.

$$ \frac{1}{F(x; T, R, P, S, M)} \left( \nabla f(x; T, R, P, S, M, \epsilon) - \left[\frac{f(x; T, R, P, S, M, \epsilon)}{F(x; T, R, P, S, M)}\right]^T \nabla F(x; T, R, P, S, M) \right) - I $$

## Equilibrium

Note that in any equilibrium $x^*$ we have that.

$$ \frac{1}{F(x^*; T, R, P, S, M)} \bigg( \nabla f(x^*; T, R, P, S, M, \epsilon) - x^{*T} \nabla F(x^*; T, R, P, S, M) \bigg) - I $$

## Stability

Need the roots of the following polynomial to have negative real parts.

$$ \det \Bigg[\frac{1}{F(x^*; T, R, P, S, M)} \bigg( \nabla f(x^*; T, R, P, S, M, \epsilon) - x^{*T} \nabla F(x^*; T, R, P, S, M) \bigg) - \lambda I\Bigg] = 0 $$

### Trace

Computing the partial derivatives from above.

\begin{align}
    \frac{\partial f_1(x; T, R, P, S, M, \epsilon)}{\partial x_1} =& F_{1,1} - F_{1,4} + \left(x_1\left(\frac{\partial F_{1,1}(x; T, R, P, S, M, \epsilon)}{\partial x_1} - \frac{\partial F_{1,4}(x; T, R, P, S, M, \epsilon)}{\partial x_1}\right) + x_2\frac{\partial F_{1,2}(x; T, R, P, S, M, \epsilon)}{\partial x_1} + x_3\frac{\partial F_{1,3}(x; T, R, P, S, M, \epsilon)}{\partial x_1}\right) \\
    \frac{\partial f_2(x; T, R, P, S, M, \epsilon)}{\partial x_2} =& F_{2,2} - F_{2,4} + \left(x_1\frac{\partial F_{2,1}(x; T, R, P, S, M, \epsilon)}{\partial x_2}  + x_2\left(\frac{\partial F_{2,2}(x; T, R, P, S, M, \epsilon)}{\partial x_2} - \frac{\partial F_{2,4}(x; T, R, P, S, M, \epsilon)}{\partial x_2}\right) + x_3\frac{\partial F_{2,3}(x; T, R, P, S, M, \epsilon)}{\partial x_2}\right) \\
    \frac{\partial f_3(x; T, R, P, S, M, \epsilon)}{\partial x_3} =& F_{3,3} - F_{3,4} + \left(x_1\frac{\partial F_{3,1}(x; T, R, P, S, M, \epsilon)}{\partial x_3} + x_2\frac{\partial F_{3,2}(x; T, R, P, S, M, \epsilon)}{\partial x_3} + x_3\left(\frac{\partial F_{3,3}(x; T, R, P, S, M, \epsilon)}{\partial x_3} - \frac{\partial F_{3,4}(x; T, R, P, S, M, \epsilon)}{\partial x_3}\right)\right) \\
\end{align}

and

\begin{align}
    \frac{\partial F(x; T, R, P, S, M)}{\partial x_1} =& F_G(x_A; T, R, P, S, M) - F_g(x_A; T, R, P, S, M) + \left(x_G\frac{\partial F_G(x_A; T, R, P, S, M)}{\partial x_A} + (1 - x_G)\frac{\partial F_g(x_A; T, R, P, S, M)}{\partial x_A}\right) \\
    \frac{\partial F(x; T, R, P, S, M)}{\partial x_2} =& F_G(x_A; T, R, P, S, M) - F_g(x_A; T, R, P, S, M) \\
    \frac{\partial F(x; T, R, P, S, M)}{\partial x_3} =& \frac{\partial F(x; T, R, P, S, M)}{\partial x_1} - \frac{\partial F(x; T, R, P, S, M)}{\partial x_2} \\
\end{align}

We can compute the $Tr(J)$ as follow.

$$ Tr(J) = \frac{1}{F(x^*; T, R, P, S, M)}\Bigg[\left(F_{1,1} + F_{2,2} + F_{3,3} - \left(F_{1,4} + F_{2,4} + F_{3,4}\right) + \left(x_1\left(\frac{\partial F_{1,1}(x; T, R, P, S, M, \epsilon)}{\partial x_1} - \frac{\partial F_{1,4}(x; T, R, P, S, M, \epsilon)}{\partial x_1}\right) + x_2\frac{\partial F_{1,2}(x; T, R, P, S, M, \epsilon)}{\partial x_1} + x_3\frac{\partial F_{1,3}(x; T, R, P, S, M, \epsilon)}{\partial x_1}\right) + \left(x_1\frac{\partial F_{2,1}(x; T, R, P, S, M, \epsilon)}{\partial x_2}  + x_2\left(\frac{\partial F_{2,2}(x; T, R, P, S, M, \epsilon)}{\partial x_2} - \frac{\partial F_{2,4}(x; T, R, P, S, M, \epsilon)}{\partial x_2}\right) + x_3\frac{\partial F_{2,3}(x; T, R, P, S, M, \epsilon)}{\partial x_2}\right) + \left(x_1\frac{\partial F_{3,1}(x; T, R, P, S, M, \epsilon)}{\partial x_3} + x_2\frac{\partial F_{3,2}(x; T, R, P, S, M, \epsilon)}{\partial x_3} + x_3\left(\frac{\partial F_{3,3}(x; T, R, P, S, M, \epsilon)}{\partial x_3} - \frac{\partial F_{3,4}(x; T, R, P, S, M, \epsilon)}{\partial x_3}\right)\right)\right) - \left(x_G^*\left(2\big(F_G(x_A^*; T, R, P, S, M) - F_g(x_A^*; T, R, P, S, M)\big) + \left(x_G^*\frac{\partial F_G(x_A; T, R, P, S, M)}{\partial x_A} + (1 - x_G^*)\frac{\partial F_g(x_A; T, R, P, S, M)}{\partial x_A}\right)\right) - x_A^*\left(F_G(x_A^*; T, R, P, S, M) - F_g(x_A^*; T, R, P, S, M)\right)\right)\Bigg] - 3 $$

## Suppose that $x_2=1-x_1$, $x_3=0$, $U_G=1$

$$ Tr(J) = \frac{1}{2(R - M)}\Bigg[\left((1 - \epsilon)2(R - M) + \frac{1}{2}2(R - M) - \frac{1}{4}\epsilon F_g(x_A^*, T, R, P, S, M) - \frac{1}{4}\epsilon L_g(x_A^*, R, S, M) - \frac{1}{4}F_g(x_A^*, T, R, P, S, M) + \left(x_1\left(\frac{\partial F_{1,1}(x; T, R, P, S, M, \epsilon)}{\partial x_1} - \frac{\partial F_{1,4}(x; T, R, P, S, M, \epsilon)}{\partial x_1}\right) + (1 - x_1)\frac{\partial F_{1,2}(x; T, R, P, S, M, \epsilon)}{\partial x_1}\right) + \left(\frac{1}{4}\epsilon - \frac{1}{2}\right)(F_g(x_A^*, T, R, P, S, M) - L_g(x_A^*, R, S, M)) + \frac{1}{2}2(R - M)\frac{1}{x_1}\right) - \left(2\big(2(R - M) - F_g(x_A^*; T, R, P, S, M)\big) + \frac{\partial F_G(x_A; T, R, P, S, M)}{\partial x_A} - x_A^*\left(2(R - M) - F_g(x_A^*; T, R, P, S, M)\right)\right)\Bigg] - 3 $$

$$ (\frac{1}{4}\epsilon(F_g(x_A^*,T,R,P,S,M) - L_g(x_A^*, R,S,M)) + (x_1 - 1)(-\frac{1}{4}\epsilon(\frac{\partial F_g(x_A,T,R,P,S,M)}{\partial x_1} - \frac{\partial L_g(x_A, R, S, M)}{\partial x_1}) - (\epsilon - 1)\frac{\partial L_G(x_A, R, S, M)}{\partial x_1} + \frac{1}{2}\frac{\partial F_G(x_A,T,R,P,S,M)}{\partial x_1} - \frac{1}{2}\frac{\partial L_G(x_A, R, S, M)}{\partial x_1} - \frac{1}{4}\frac{\partial L_g(x_A, R, S, M)}{\partial x_1}))/(x_1 - 1) $$

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

Here we go...

\begin{align}
\det
\begin{bmatrix}
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) - \lambda & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) - \lambda & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) - \lambda
\end{bmatrix}
\end{align}

\begin{align}
   = & (-1)^{(2+1)}\frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) \det
\begin{bmatrix}
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) - \lambda
\end{bmatrix}
 + \\
 & (-1)^{(2+2)} \bigg(\frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) - \lambda\bigg) \det
 \begin{bmatrix}
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) - \lambda & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) - \lambda
\end{bmatrix}
+ \\
 & (-1)^{(2+1)}\frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \det
 \begin{bmatrix}
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) - \lambda & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right)
\end{bmatrix}
\end{align}

\begin{align}
   = & -\frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) \Bigg[\frac{1}{F(x^*; T, R, P, S, M)^2}\Bigg(\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right)\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) - \left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right)\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right)\Bigg) - \lambda\Bigg]
 + \\
 & (-1)^{(2+2)} \bigg(\frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) - \lambda\bigg) \det
 \begin{bmatrix}
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) - \lambda & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) - \lambda
\end{bmatrix}
+ \\
 & (-1)^{(2+1)}\frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_2(x^*; T, R, P, S, M, \epsilon)}{\partial x_3} - x_2^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_3}\right) \det
 \begin{bmatrix}
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) - \lambda & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_1(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_1^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right) \\
    \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_1} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_1}\right) & \frac{1}{F(x^*; T, R, P, S, M)}\left(\frac{\partial f_3(x^*; T, R, P, S, M, \epsilon)}{\partial x_2} - x_3^*\frac{\partial F(x^*; T, R, P, S, M)}{\partial x_2}\right)
\end{bmatrix}
\end{align}

```python

```


# ODE RLC circuit

This notebook shows how to solve numerically a second order ODE of a RLC series circuit.

We define

- $u(t)$ input voltage
- $x(t)$ voltage over the capacitor to the ground

```python
from scipy.integrate import solve_ivp
import numpy as np
import matplotlib.pyplot as plt
```

We have a second order ODE:

\begin{align}
LC \ddot{x} + RC \dot{x} + x = u(t)
\end{align}

In oder to calculate a numerical solution we have to rearrange the second order equation into a system of first order ODEs. A system of first order ODEs can be solved with Euler method.

We rewrite the ODE so that the highest order is the first - we substitute $v=\dot{x}$:

\begin{align}
LC \dot{v} + RC v + x = u(t)
\end{align}

The final ODE system is:

\begin{align}
\dot{v} = \frac{u(t) - RC v - x}{LC} \\
\dot{x} = v
\end{align}

```python
# imitates European line voltage
def input_signal(t):
    return 230*np.sin(2*np.pi*100*t)
```

```python
R = 100; C = 1e-3; L = 100e-3
```

```python
def ode_right_side(t, vars):
    v, x = vars[0], vars[1]
    vdot = (input_signal(t) - R*C*v - x)/(L*C)
    xdot = v
    return np.r_[vdot, xdot]
```

```python
initial_state = np.r_[0, 0]
```

```python
integral_range = [0, 1]
```

```python
eval_times = np.linspace(0, 1, 10000)
```

```python
result = solve_ivp(ode_right_side, integral_range, y0=initial_state, t_eval=eval_times)
```

```python
plt.figure(figsize=(15,5))
plt.plot(result.t, result.y[1, :], label="x(t)")
plt.xlabel("time [s]")
plt.ylabel("voltage [V]")
plt.legend()
plt.xlim([0, 0.5])
```

```python
# imitates step function from t > 0
def input_signal(t):
    return 230
```

```python
result = solve_ivp(ode_right_side, integral_range, y0=initial_state, t_eval=eval_times)
```

```python
plt.figure(figsize=(15,5))
plt.plot(result.t, result.y[1, :], label="x(t)")
plt.xlabel("time [s]")
plt.ylabel("voltage [V]")
plt.legend()
plt.xlim([0, 0.5])
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
```

```python
# create function
x = Symbol('x')
f = (x-10)**2

# derivative
df = f.diff(x)

# lambdify both
f = lambdify(x, f)
df = lambdify(x, df)

# evaluate
pts = np.linspace(0, 20)
plt.plot(pts, f(pts), label='f(x)')
plt.plot(pts, df(pts), label='df/dx')
plt.legend()
plt.show()
```

```python
max_iter = 300
lr = 0.05
```

```python
X0 = 30
cur_iter = 0
steps = [X0]

while True:
    # get closer to a local min
    X1 = X0 - lr * df(X0)
    # save step
    steps.append(X1)
    # calculate step size
    step_size = abs(X0 - X1)
    # save new step
    X0 = X1
    if cur_iter > max_iter:
        print("Max iter reached")
        break
    if step_size < 0.00001:
        print("Local min reached")
        break
    elif cur_iter%20 == 0:
        print("Iteration {} has x value {}".format(cur_iter, X1))
    cur_iter = cur_iter + 1

print("The local minimum occurs at", X1)
```

    Iteration 0 has x value 28.0
    Iteration 20 has x value 12.188379782630246
    Iteration 40 has x value 10.266055892945824
    Iteration 60 has x value 10.032346185398458
    Iteration 80 has x value 10.003932541009512
    Iteration 100 has x value 10.000478105179978
    Local min reached
    The local minimum occurs at 10.000088593855091

```python
pts = np.linspace(-10, 30)
plt.plot(pts, f(pts), label='f(x)')
plt.plot(pts, df(pts), label='df/dx')
# change color to see GD progress
color = [item/255 for item in list(range(len(steps)))]
plt.scatter(steps, [f(x) for x in steps], label='Gradient descent', c=color)
plt.legend()
plt.show()
```

```python

```


# __DATA 5600: Introduction to Regression and Machine Learning for Analytics__

## __Notes on the Bayesian Gamma-Poisson Conjugate Model__

Author:      Tyler J. Brough

Last Update: December 6, 2021

---

```python
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_style('darkgrid')
plt.rcParams['figure.figsize'] = [10, 5]
```

```python
np.random.seed(111111)
```

## __Introduction__

These notes provide an introduction to the Bayesian Gamma-Poisson conjugate model and are based in part on the following:

* https://vioshyvo.github.io/Bayesian_inference/conjugate-distributions.html

* https://web.ma.utexas.edu/users/pmueller/compbayes/

### __The Gamma-Poisson Model__

The Poisson distribution is a dicrete distribution which models count data (positive integer values). Examples:

* Points in a football game

* The number bicycles passing a point in the park during lunch hour

* The number of customers served at a check-out counter per hour

A Poisson random variable is denoted:

$$
Y \sim Poisson(\lambda)
$$

and has the following properties:

* $E[Y] = \lambda$

* $Var[Y] = \lambda$

* Therefore, $E[Y] = Var[Y]$ (proportional variance)

Let's simulate some observations from a $Poisson(3)$ distribution:

```python
n = 10
true_lambda = 3
y = np.random.poisson(lam=true_lambda, size=n)
```

```python
y
```

    array([3, 2, 6, 3, 2, 2, 5, 4, 2, 2])

### __The Likelihood__

We know that these data $\{y\}_{i=1}^{n=10}$ come from a $Poisson(3)$; but let's forget this for a minute and pretend that we are the Statistician who has obtained these data for the purpose of inference.

We will assume that the DGP (data generating process) for these data is the Poisson:

$$
Y_{1}, \ldots, Y_{n} \sim Poisson(\lambda)
$$

That is, in a Bayesian setup we will assume a Poisson likelihood function.

The likelihood can then be written as:

$$
p(y | \lambda) = \prod_{i=1}^{n} p(y_{i} | \lambda) = \prod_{i=1}^{n} \left(\lambda^{y_{i}} \frac{e^{-\lambda}}{y_{i}!} \right) \propto \lambda^{\sum_{i=1}^{n} y_{i}} e^{-n\lambda} = \lambda^{n \bar{y}} e^{-n \lambda}
$$

Where

$$
\bar{y} = \frac{1}{n} \sum_{i=1}^{n} y_{i}
$$

is the arithmetic mean.

### __The Prior__

A $Gamma(\alpha, \beta)$ is a convenient choice for a prior for $\lambda$ because it offers a conjugate relationship (i.e. the posterior distribution will also be in the $Gamma$ family).

So we can state the prior thus:

$$
\lambda \sim Gamma(\alpha, \beta)
$$

For now we will set the values of the hyperparameters to $\alpha = 1$ and $\beta = 1$.

```python
## Helper function to plot Beta priors
def plot_gamma(a=1, b=1):
    x = np.arange(0, 7, 0.01)
    y = stats.gamma.pdf(x, a=a, scale=1/b)
    plt.plot(x, y, lw = 2.0, color='darkblue', alpha=0.8)
    plt.fill_between(x, y, facecolor='orange', alpha=0.5)
    plt.title(f"Gamma({a},{b}) Prior Distribution")
    plt.show()
```

```python
plot_gamma()
```

Now, let's plot the posterior using the 10 observations from above.

```python
## Prior hyperparameters
alpha = 1
beta = 1

## Posterior hyperparameters
alpha_post = alpha + np.sum(y)
beta_post = beta + len(y)

(alpha_post, beta_post)
```

    (32, 11)

```python
## Plot the posterior
plot_gamma(a=alpha_post, b=beta_post)
```

Credit: https://github.com/tonyduan/conjugate-bayes/blob/master/conjugate_bayes/models.py

```python
class GammaPoisson(object):
    """
    Conjugate prior for a Poisson distribution with unknown rate λ.
    Parameters
    ----------
    alpha: prior for Gamma(alpha, beta) prior on parameter λ.
    beta:  prior for Gamma(alpha, beta) prior on parameter λ.
    """
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta

    def fit(self, x):
        self.alpha += np.sum(x)
        self.beta += len(x)

    def plot_posterior(self):
        x = np.arange(0, 7, 0.01)
        y = stats.gamma.pdf(x, a=self.alpha, scale=1/self.beta)
        plt.plot(x, y, lw = 2.0, color='darkblue', alpha=0.8)
        plt.fill_between(x, y, facecolor='orange', alpha=0.5)
        plt.title(f"Gamma({self.alpha},{self.beta}) Prior Distribution")
        plt.show()
```

```python
gp = GammaPoisson(1, 1)
```

```python
gp.fit(y)
```

```python
gp.alpha, gp.beta
```

    (32, 11)

```python
gp.plot_posterior()
```

```python
np.mean(y), np.var(y)
```

    (3.1, 1.8900000000000001)

### __A Larger Simulation Example__

* Set $n = 200$

* Set $\lambda = 3$

```python
n = 200
true_lambda = 3
y = np.random.poisson(lam=true_lambda, size=n)
```

```python
y_series = pd.Series(y)
y_series.describe()
```

    count    200.000000
    mean       3.180000
    std        1.741483
    min        0.000000
    25%        2.000000
    50%        3.000000
    75%        4.000000
    max        8.000000
    dtype: float64

```python
np.mean(y), np.var(y)
```

    (3.18, 3.0176)

```python
plt.hist(y, bins=8);
```

```python
jeep = GammaPoisson(1, 1)
jeep.fit(y)
```

```python
jeep.alpha, jeep.beta
```

    (637, 201)

```python
jeep.plot_posterior()
```

### __Frequentist MLE, Bootstrap, and Bayesian Comparisons__

#### __Frequentist__

```python
ybar = np.mean(y)
y_var = np.var(y)
ybar, y_var
```

    (3.18, 3.0176)

```python
y_se = np.std(y, ddof=1)/np.sqrt(n)
```

```python
ybar, y_se
```

    (3.18, 0.1231414596936279)

```python
conf_level = 0.05
lo = stats.t(n-1).ppf(conf_level/2.)
hi = stats.t(n-1).ppf(1. - (conf_level/2.))
CI = (ybar + y_se*lo, ybar + y_se*hi)
```

```python
CI
```

    (2.9371703926887274, 3.422829607311273)

Let's conduct a classical hypothesis test. The test is the following:

$$
\begin{align}
H_{0}: & \quad \lambda = 3 \\
& \\
H_{a}: & \quad \lambda \ne 3
\end{align}
$$

To do this of course we form the following test statistic:

$$
t = \frac{\bar{y} - 3}{s / \sqrt{n}}
$$

Which we can compute rather easily in Python:

```python
t = (ybar - 3.) / y_se
t
```

    1.4617335253929467

```python
## Two-tailed t-test with 5% confidence level
α = 0.05
t_crit = np.abs(stats.t(n-1).ppf(1. - (α / 2.)))
t_crit
```

    1.971956544249395

```python
t > t_crit
```

    False

We fail to reject the null hypothesis.

#### __Bootstrap__

```python
B = 10_000
y_bar_star = np.zeros(B)
t_star = np.zeros(B)

for b in range(B):
    y_star = np.random.choice(y, size=n, replace=True)
    y_bar_star[b] = np.mean(y_star)
    t_star[b] = y_bar_star[b] - 3.
```

```python
plt.hist(y_bar_star, bins=100);
```

```python
y_bar_star.sort()
```

```python
y_bar_star_se = np.std(y_bar_star, ddof=1)
```

```python
y_bar_star.mean(), y_bar_star_se
```

    (3.1815295000000003, 0.12279813136956436)

```python
α = 0.05
lb = int(B * α / 2.)
ub = int(B * (1. - α / 2.))
lb, ub
```

    (250, 9750)

```python
CI_star = y_bar_star[lb], y_bar_star[ub]
```

```python
CI_star
```

    (2.94, 3.425)

```python
t_star /= y_bar_star_se
```

```python
np.mean(t_star)
```

    1.4782757520444834

This is just to show that we can arrive at a standardized $t$-stat in much the same way. But the hypothesis test is rather trivial in actuality because we are working with the non-standardized bootstrap sampling distribution.

We would just compare the hypothesized value of $3$ to the $97.5$th quantile

```python
3. >= y_bar_star[ub]
```

    False

So again we fail to reject the null hypothesis.

```python

```


# Linear Inversion App

```python
from geoscilabs.inversion.LinearInversionDirect import LinearInversionDirectApp
from ipywidgets import interact, FloatSlider, ToggleButtons, IntSlider, FloatText, IntText
import matplotlib.pyplot as plt
import matplotlib
matplotlib.rcParams['font.size'] = 14
```

```python
app = LinearInversionDirectApp()
```

## Background

This app is based upon the inversion tutorial: "INVERSION FOR APPLIED GEOPHYSICS" by Oldenburg and Li (2005).

Douglas W. Oldenburg and Yaoguo Li (2005) 5. Inversion for Applied Geophysics: A Tutorial. Near-Surface Geophysics: pp. 89-150. eISBN: 978-1-56080-171-9 print ISBN: 978-1-56080-130-6 https://doi.org/10.1190/1.9781560801719.ch5

## Purpose

We illustrate how a generic linear inverse problem can be solved using a Tikhonov approach. The default parameters provided for the Forward and Inverse problems below generate a reasonable example for illustrating the inversion but the learning comes when these parameters are changed and outcomes are observed.

## Outline

The app is divided into two sections:

### Forward Problem

* Mathematical Background for the Forward Problem

* Step 1: Create a model, $\\mathbf{m}$.

* Step 2: Generate a sensitivity matrix $\\mathbf{G}$.

* Step 3: Simulate data ($\\mathbf{d} = \\mathbf{G}\\mathbf{m}$) and add noise.

These steps are explored individually but additional text given in [Linear Tikhonov Inversion](https://curvenote.com/@geosci/inversion-module/linear-tikhonov-inversion). For convenience, the widgets used to carry out all three steps are consolidated at the end of the section. A brief mathematical description is also provided.

### Inverse Problem

* Mathematical Background for the Inverse Problem

* Step 4: Invert the data, and explore the results

Here we provide widgets to adjust the parameters for the inverse problem. Some basic information is provided but details about the parameters are provided in the text [Linear Tikhonov Inversion](https://curvenote.com/@geosci/inversion-module/linear-tikhonov-inversion).

## Mathematical Background for the Forward Problem

Let $g_j(x)$ denote the kernel function for $j$th datum. With a given model $m(x)$, the $j$th datum can be computed by solving following integral equation:

$$d_j = \int_a^{b} g_j(x) m(x) dx$$

where

$$g_j(x) = e^{p_jx} cos (2 \pi q_jx)$$

is the $j^{th}$ kernel function. By integrating $g_j(x)$ over cells of width $\\Delta x$ and using the midpoint rule cell we obtain the sensitivities

$$\mathbf{g}_j(\mathbf{x}) = e^{p_j\mathbf{x}} cos (2 \pi q_j \mathbf{x}) \Delta x$$

where

* $\\mathbf{g}_j$: $j$th row vector for the sensitivty matrix ($1 \\times M$)

* $\\mathbf{x}$: model location ($1 \\times M$)

* $p_j$: decaying constant (<0)

* $q_j$: oscillating constant (>0)

By stacking multiple rows of $\\mathbf{g}_j$, we obtain sensitivity matrix, $\\mathbf{G}$:

$$\begin{aligned} \mathbf{G} = \begin{bmatrix} &\mathbf{g}1\ \\&\vdots\ \\& \mathbf{g}{N} \end{bmatrix} \end{aligned}$$

Here, the size of the matrix $\\mathbf{G}$ is $(N \\times M)$. Finally data, $\\mathbf{d}$, can be written as a linear equation:

$$\mathbf{d} = \mathbf{G}\mathbf{m}$$

where $\\mathbf{m}$ is an inversion model; this is a column vector ($M \\times 1$).

In real measurments, there will be various noise sources, and hence observation, $\\mathbf{d}^{obs}$, can be written as

$$\mathbf{d}^{obs} = \mathbf{G}\mathbf{m} + \mathbf{noise}$$

##  Step 1: Create a model, $\mathbf{m}$

The model $m$ is a function defined on the interval [0,1] and discretized into $M$ equal intervals. It is the sum of a: (a) background $m_{background}$, (b) box car $m1$ and (c) Gaussian $m2$.

- `m_background` : background value

The box car is defined by
- `m1` : amplitude
- `m1_center` : center
- `m1_width` : width

The Gaussian is defined by
- `m2` : amplitude
- `m2_center` : center
- `m2_sigma` : width of Gaussian (as defined by a standard deviation $\epsilon$)
- `M` : number of model parameters

```python
Q_model = app.interact_plot_model()
```

##  Step 2: Generate a sensitivity matrix, $\mathbf{G}$

By using the following app, we explore each row vector, $\mathbf{g}_j$, of the kernel or sensitivity matrix , $\mathbf{G}$. Parameters of the apps are:

- `M`: number of model parameters
- `N`: number of data
- `pmin`, `pmax`: minimum and maximum of the $M$-length range of decaying constant values (<0)
- `qmin`, `qmax`: minimum and maximum of the $M$-length range of oscillating constant values (>0)
- `ymin`, `ymax`: minimum and maximum of the y-axis

```python
Q_kernel = app.interact_plot_G()
```

## Step 3: Simulate data, $\mathbf{d}=\mathbf{Gm}$, and add noise

The $j$-th datum is the inner product of the $j$-th kernel $g_j(x)$ and the model $m(x)$. In discrete form it can be written as the dot product of the vector $\mathbf{g}_j$ and the model vector $\mathbf{m}$.

$$ d_j = \mathbf{g}_j \mathbf{m} $$

If there are $N$ data, these data can be written as a column vector, $\mathbf{d}$:

\begin{align}
    \mathbf{d} = \mathbf{G}\mathbf{m} =
    \begin{bmatrix}
        d_1\\
        \vdots\\
        d_{N}
    \end{bmatrix}
\end{align}

### Adding Noise

Observational data are always contaminated with noise. Here we add Gaussian noise $N(0,\epsilon)$ (zero mean and standard deviation $\epsilon$). Here we choose

$$ \epsilon = \% |d| + \text{floor} $$

```python
Q_data = app.interact_plot_data()
```

## Composite Widget for Forward Modelling

```python
app.reset_to_defaults()
app.interact_plot_all_three_together()
```

## Mathematical Background for the Inverse Problem

In the inverse problem we attempt to find the model $\mathbf{m}$ that gave rise to the observational data $\mathbf{d}^{obs}$. The inverse problem is formulated as an optimization problem:

$$\text{minimize} \ \ \ \phi(\mathbf{m}) = \phi_d(\mathbf{m}) + \beta \phi_m(\mathbf{m}) $$

where

- $\phi_d$: data misfit
- $\phi_m$: model regularization
- $\beta$: trade-off (Tikhonov) parameter  $0<\beta<\infty$

Data misfit is defined as

$$ \phi_d = \sum_{j=1}^{N}\Big(\frac{\mathbf{g}_j\mathbf{m}-d^{obs}_j}{\epsilon_j}\Big)^2$$

where $\epsilon_j$  is an estimate of the standard deviation of the $j$th datum.

The model regularization term, $\phi_m$, can be written as

$$ \phi_m(\mathbf{m}) = \alpha_s \int \left(\mathbf{m}-\mathbf{m}_{ref}\right)^2 dx + \alpha_x \int \left(\frac{d \mathbf{m}}{dx}\right)^2 dx$$

The first term is referred to as the "smallness" term. Minimizing this generates a model that is close to a reference model $\mathbf{m}_{ref}$. The second term penalizes roughness of the model. It is generically referred to as a "flattest" or "smoothness" term.

## Step 4: Invert the data, and explore the results

In the inverse problem we define parameters needed to evaluate the data misfit and the model regularization terms. We then deal with parameters associated with the inversion.

### Parameters

- `mode`: `Run` or `Explore`
    - `Run`: Each click of the app, will run `n_beta` inversions
    - `Explore`: Not running inversions, but explore result of the previously run inversions

#### Misfit
- `percent`: estiamte uncertainty as a percentage of the data (%)

- `floor`: estimate uncertainty floor

- `chifact`: chi factor for stopping criteria (when $\phi_d^{\ast}=N \rightarrow$ `chifact=1`)

#### Model norm
- `mref`: reference model

- `alpha_s`: $\alpha_s$ weight for smallness term

- `alpha_x`: $\alpha_x$ weight for smoothness term

#### Beta
- `beta_min`: minimum $\beta$

- `beta_max`: maximum $\beta$

- `n_beta`: the number of $\beta$

#### Plotting options

- `data`: `obs & pred` or `normalized misfit`
    - `obs & pred`: show observed and predicted data
    - `normalized misfit`: show normalized misfit

- `tikhonov`: `phi_d & phi_m` or `phi_d vs phi_m`
    - `phi_d & phi_m`: show $\phi_d$ and $\phi_m$ as a function of $\beta$
    - `phi_d vs phi_m`: show tikhonov curve

- `i_beta`: i-th $\beta$ value

- `scale`: `linear` or `log`
    - `linear`: linear scale for plotting the third panel
    - `log`: log scale for plotting the third panel

```python
app.interact_plot_inversion()
```