# Advanced topic: Solving the two-layer grey gas model analytically with sympy

____________

## 1. Introducing symbolic computation with sympy
____________

These notes elaborate on material in the [lecture on elementary greenhouse models](elementary-greenhouse.ipynb), demonstrating use of a computer algebra system to make precise calculations.

### Symbolic math with the sympy package

The two-layer grey gas model is simple enough that we can work out all the details algebraically. There are three temperatures to keep track of $(T_s, T_0, T_1)$, so we will have 3x3 matrix equations.

We all know how to work these things out with pencil and paper. But it can be tedious and error-prone.

Symbolic math software lets us use the computer to automate a lot of tedious algebra.

The [sympy](http://www.sympy.org/en/index.html) package is a powerful open-source symbolic math library that is well-integrated into the scientific Python ecosystem.

### Getting started with sympy

```python
import sympy
#  Allow sympy to produce nice looking equations as output
sympy.init_printing()
#  Define some symbols for mathematical quantities
#  Assume all quantities are positive (which will help simplify some expressions)
epsilon, T_e, T_s, T_0, T_1, sigma = \
    sympy.symbols('epsilon, T_e, T_s, T_0, T_1, sigma', positive=True)
#  So far we have just defined some symbols, e.g.
T_s
```

```python
#  We have hard-coded the assumption that the temperature is positive
sympy.ask(T_s>0)
```

    True

____________

## 2. Coding up the 2-layer grey gas model in sympy
____________

### Longwave emissions

Let's denote the emissions from each layer as
\begin{align}
E_s &= \sigma T_s^4 \\
E_0 &= \epsilon \sigma T_0^4 \\
E_1 &= \epsilon \sigma T_1^4
\end{align}

recognizing that $E_0$ and $E_1$ contribute to **both** the upwelling and downwelling beams.

```python
#  Define these operations as sympy symbols
#  And display as a column vector:
E_s = sigma*T_s**4
E_0 = epsilon*sigma*T_0**4
E_1 = epsilon*sigma*T_1**4
E = sympy.Matrix([E_s, E_0, E_1])
E
```

### Shortwave radiation

Since we have assumed the atmosphere is transparent to shortwave, the incident beam $Q$ passes unchanged from the top to the surface, where a fraction $\alpha$ is reflected upward out to space.

```python
#  Define some new symbols for shortwave radiation
Q, alpha = sympy.symbols('Q, alpha', positive=True)
#  Create a dictionary to hold our numerical values
tuned = {}
tuned[Q] = 341.3  #  global mean insolation in W/m2
tuned[alpha] = 101.9/Q.subs(tuned)  #  observed planetary albedo
tuned[sigma] = 5.67E-8  #  Stefan-Boltzmann constant in W/m2/K4
tuned
#  Numerical value for emission temperature
#T_e.subs(tuned)
```

### Tracing the upwelling beam of longwave radiation

Let $U$ be the upwelling flux of longwave radiation.

The upward flux **from the surface to layer 0** is

$$ U_0 = E_s $$

(just the emission from the suface).

```python
U_0 = E_s
U_0
```

Following this beam upward, we can write the upward flux from layer 0 to layer 1 as the sum of the transmitted component that originated below layer 0 and the new emissions from layer 0:

$$ U_1 = (1-\epsilon) U_0 + E_0 $$

```python
U_1 = (1-epsilon)*U_0 + E_0
U_1
```

Continuing to follow the same beam, the upwelling flux above layer 1 is
$$ U_2 = (1-\epsilon) U_1 + E_1 $$

```python
U_2 = (1-epsilon) * U_1 + E_1
```

Since there is no more atmosphere above layer 1, this upwelling flux is our Outgoing Longwave Radiation for this model:

$$ OLR = U_2 $$

```python
U_2
```

The three terms in the above expression represent the **contributions to the total OLR that originate from each of the three levels**.

Let's code this up explicitly for future reference:

```python
#  Define the contributions to OLR originating from each level
OLR_s = (1-epsilon)**2 *sigma*T_s**4
OLR_0 = epsilon*(1-epsilon)*sigma*T_0**4
OLR_1 = epsilon*sigma*T_1**4

OLR = OLR_s + OLR_0 + OLR_1

print( 'The expression for OLR is')
OLR
```

### Downwelling beam

Let $D$ be the downwelling longwave beam. Since there is no longwave radiation coming in from space, we begin with

```python
fromspace = 0
D_2 = fromspace
```

Between layer 1 and layer 0 the beam contains emissions from layer 1:

$$ D_1 = (1-\epsilon)D_2 + E_1 = E_1 $$

```python
D_1 = (1-epsilon)*D_2 + E_1
D_1
```

Finally between layer 0 and the surface the beam contains a transmitted component and the emissions from layer 0:

$$ D_0 = (1-\epsilon) D_1 + E_0 = \epsilon(1-\epsilon) \sigma T_1^4 + \epsilon \sigma T_0^4$$

```python
D_0 = (1-epsilon)*D_1 + E_0
D_0
```

This $D_0$ is what we call the **back radiation**, i.e. the longwave radiation from the atmosphere to the surface.

____________

## 3. Tuning the grey gas model to observations
____________

In building our new model we have introduced exactly one parameter, the absorptivity $\epsilon$. We need to choose a value for $\epsilon$.

We will tune our model so that it **reproduces the observed global mean OLR** given **observed global mean temperatures**.

To get appropriate temperatures for $T_s, T_0, T_1$, revisit the global, annual mean lapse rate plot from NCEP Reanalysis data we first encountered in the [Radiation notes](https://brian-rose.github.io/ClimateLaboratoryBook/courseware/radiation.html).

### Temperatures

First, we set
$$T_s = 288 \text{ K}  $$

From the lapse rate plot, an average temperature for the layer between 1000 and 500 hPa is

$$ T_0 = 275 \text{ K}$$

Defining an average temperature for the layer between 500 and 0 hPa is more ambiguous because of the lapse rate reversal at the tropopause. We will choose

$$ T_1 = 230 \text{ K}$$

From the graph, this is approximately the observed global mean temperature at 275 hPa or about 10 km.

```python
#  add to our dictionary of values:
tuned[T_s] = 288.
tuned[T_0] = 275.
tuned[T_1] = 230.
tuned
```

### OLR

From the [observed global energy budget](https://brian-rose.github.io/ClimateLaboratoryBook/courseware/models-budgets-fun.html#2.-The-observed-global-energy-budget) we set

$$ OLR = 238.5 \text{ W m}^{-2} $$

### Solving for $\epsilon$

We wrote down the expression for OLR as a function of temperatures and absorptivity in our model above.

We just need to equate this to the observed value and solve a **quadratic equation** for $\epsilon$.

This is where the real power of the symbolic math toolkit comes in.

Subsitute in the numerical values we are interested in:

```python
# the .subs() method for a sympy symbol means
#  substitute values in the expression using the supplied dictionary
#  Here we use observed values of Ts, T0, T1
OLR2 = OLR.subs(tuned)
OLR2
```

We have a quadratic equation for $\epsilon$.

Now use the `sympy.solve` function to solve the quadratic:

```python
#  The sympy.solve method takes an expression equal to zero
#  So in this case we subtract the tuned value of OLR from our expression
eps_solution = sympy.solve(OLR2 - 238.5, epsilon)
eps_solution
```

There are two roots, but the second one is unphysical since we must have $0 < \epsilon < 1$.

Just for fun, here is a simple of example of *filtering a list* using powerful Python *list comprehension* syntax:

```python
# Give me only the roots that are between zero and 1!
list_result = [eps for eps in eps_solution if 0<eps<1]
print( list_result)
# The result is a list with a single element.
#  We need to slice the list to get just the number:
eps_tuned = list_result[0]
print( eps_tuned)
```

    [0.586041150248834]
    0.586041150248834

We conclude that our tuned value is (to within 3 decimal places):

$$ \epsilon = 0.586 $$

This is the absorptivity that guarantees that our model reproduces the observed OLR given the observed tempertures.

```python
tuned[epsilon] = eps_tuned
tuned
```

____________

## 4. Level of emission
____________

Even in this very simple greenhouse model, there is **no single level** at which the OLR is generated.

The three terms in our formula for OLR tell us the contributions from each level.

```python
OLRterms = sympy.Matrix([OLR_s, OLR_0, OLR_1])
OLRterms
```

Now evaluate these expressions for our tuned temperature and absorptivity:

```python
OLRtuned = OLRterms.subs(tuned)
OLRtuned
```

So we are getting about 67 W m$^{-2}$ from the surface, 79 W m$^{-2}$ from layer 0, and 93 W m$^{-2}$ from the top layer.

In terms of fractional contributions to the total OLR, we have (limiting the output to two decimal places):

```python
sympy.N(OLRtuned / 238.5, 2)
```

Notice that the largest single contribution is coming from the top layer. This is in spite of the fact that the emissions from this layer are weak, because it is so cold.

Comparing to observations, the actual contribution to OLR from the surface is about 22 W m$^{-2}$ (or about 9% of the total), not 67 W m$^{-2}$. So we certainly don't have all the details worked out yet!

As we will see later, to really understand what sets that observed 22 W m$^{-2}$, we will need to start thinking about the spectral dependence of the longwave absorptivity.

____________

## 5. Radiative forcing in the 2-layer grey gas model
____________

Adding some extra greenhouse absorbers will mean that a greater fraction of incident longwave radiation is absorbed in each layer.

Thus **$\epsilon$ must increase** as we add greenhouse gases.

Suppose we have $\epsilon$ initially, and the absorptivity increases to $\epsilon_2 = \epsilon + \delta_\epsilon$.

Suppose further that this increase happens **abruptly** so that there is no time for the temperatures to respond to this change. **We hold the temperatures fixed** in the column and ask how the radiative fluxes change.

**Do you expect the OLR to increase or decrease?**

Let's use our two-layer leaky greenhouse model to investigate the answer.

The components of the OLR before the perturbation are

```python
OLRterms
```

After the perturbation we have

```python
delta_epsilon = sympy.symbols('delta_epsilon')
OLRterms_pert = OLRterms.subs(epsilon, epsilon+delta_epsilon)
OLRterms_pert
```

Let's take the difference

```python
deltaOLR = OLRterms_pert - OLRterms
deltaOLR
```

To make things simpler, we will neglect the terms in $\delta_\epsilon^2$. This is perfectly reasonably because we are dealing with **small perturbations** where $\delta_\epsilon << \epsilon$.

Telling `sympy` to set the quadratic terms to zero gives us

```python
deltaOLR_linear = sympy.expand(deltaOLR).subs(delta_epsilon**2, 0)
deltaOLR_linear
```

Recall that the three terms are the contributions to the OLR from the three different levels. In this case, the **changes** in those contributions after adding more absorbers.

Now let's divide through by $\delta_\epsilon$ to get the normalized change in OLR per unit change in absorptivity:

```python
deltaOLR_per_deltaepsilon = \
    sympy.simplify(deltaOLR_linear / delta_epsilon)
deltaOLR_per_deltaepsilon
```

Now look at the **sign** of each term. Recall that $0 < \epsilon < 1$. **Which terms in the OLR go up and which go down?**

**THIS IS VERY IMPORTANT, SO STOP AND THINK ABOUT IT.**

The contribution from the **surface** must **decrease**, while the contribution from the **top layer** must **increase**.

**When we add absorbers, the average level of emission goes up!**

### "Radiative forcing" is the change in radiative flux at TOA after adding absorbers

In this model, only the longwave flux can change, so we define the radiative forcing as

$$ R = - \delta OLR $$

(with the minus sign so that $R$ is positive when the climate system is gaining extra energy).

We just worked out that whenever we add some extra absorbers, the emissions to space (on average) will originate from higher levels in the atmosphere.

What does this mean for OLR? Will it increase or decrease?

To get the answer, we just have to sum up the three contributions we wrote above:

```python
R_per_deltaepsilon = -sum(deltaOLR_per_deltaepsilon)
R_per_deltaepsilon
```

Is this a positive or negative number? The key point is this:

**It depends on the temperatures, i.e. on the lapse rate.**

### Greenhouse effect for an isothermal atmosphere

Stop and think about this question:

If the **surface and atmosphere are all at the same temperature**, does the OLR go up or down when $\epsilon$ increases (i.e. we add more absorbers)?

Understanding this question is key to understanding how the greenhouse effect works.

#### Let's solve the isothermal case

We will just set $T_s = T_0 = T_1$ in the above expression for the radiative forcing.

```python
R_per_deltaepsilon.subs([(T_0, T_s), (T_1, T_s)])
```

which then simplifies to

```python
sympy.simplify(R_per_deltaepsilon.subs([(T_0, T_s), (T_1, T_s)]))
```

#### The answer is zero

For an isothermal atmosphere, there is **no change** in OLR when we add extra greenhouse absorbers. Hence, no radiative forcing and no greenhouse effect.

Why?

The level of emission still must go up. But since the temperature at the upper level is the **same** as everywhere else, the emissions are exactly the same.

### The radiative forcing (change in OLR) depends on the lapse rate!

For a more realistic example of radiative forcing due to an increase in greenhouse absorbers, we can substitute in our tuned values for temperature and $\epsilon$.

We'll express the answer in W m$^{-2}$ for a 2% increase in $\epsilon$.

```python
delta_epsilon = 0.02 * epsilon
delta_epsilon
```

The three components of the OLR change are

```python
(deltaOLR_per_deltaepsilon * delta_epsilon).subs(tuned)
```

And the net radiative forcing is

```python
(R_per_deltaepsilon*delta_epsilon).subs(tuned)
```

So in our example, **the OLR decreases by 2.6 W m$^{-2}$**, or equivalently, the radiative forcing is +2.6 W m$^{-2}$.

What we have just calculated is this:

*Given the observed lapse rates, a small increase in absorbers will cause a small decrease in OLR.*

The greenhouse effect thus gets stronger, and energy will begin to accumulate in the system -- which will eventually cause temperatures to increase as the system adjusts to a new equilibrium.

____________

## 6. Radiative equilibrium in the 2-layer grey gas model
____________

In the previous section we:

- made no assumptions about the processes that actually set the temperatures.
- used the model to calculate radiative fluxes, **given observed temperatures**.
- stressed the importance of knowing the lapse rates in order to know how an increase in emission level would affect the OLR, and thus determine the radiative forcing.

A key question in climate dynamics is therefore this:

**What sets the lapse rate?**

It turns out that lots of different physical processes contribute to setting the lapse rate.

Understanding how these processes acts together and how they change as the climate changes is one of the key reasons for which we need more complex climate models.

For now, we will use our prototype greenhouse model to do the most basic lapse rate calculation: the **radiative equilibrium temperature**.

We assume that

- the only exchange of energy between layers is longwave radiation
- equilibrium is achieved when the **net radiative flux convergence** in each layer is zero.

### Compute the radiative flux convergence

First, the **net upwelling flux** is just the difference between flux up and flux down:

```python
#  Upwelling and downwelling beams as matrices
U = sympy.Matrix([U_0, U_1, U_2])
D = sympy.Matrix([D_0, D_1, D_2])
# Net flux, positive up
F = U-D
F
```

####  Net absorption is the flux convergence in each layer

(difference between what's coming in the bottom and what's going out the top of each layer)

```python
# define a vector of absorbed radiation -- same size as emissions
A = E.copy()

#  absorbed radiation at surface
A[0] = F[0]
# Compute the convergence
for n in range(2):
    A[n+1] = -(F[n+1]-F[n])

A
```

#### Radiative equilibrium means net absorption is ZERO in the atmosphere

The only other heat source is the **shortwave heating** at the **surface**.

In matrix form, here is the system of equations to be solved:

```python
radeq = sympy.Equality(A, sympy.Matrix([(1-alpha)*Q, 0, 0]))
radeq
```

Just as we did for the 1-layer model, it is helpful to rewrite this system using the definition of the **emission temperture** $T_e$

$$ (1-\alpha) Q = \sigma T_e^4 $$

```python
radeq2 = radeq.subs([((1-alpha)*Q, sigma*T_e**4)])
radeq2
```

In this form we can see that we actually have a **linear system** of equations for a set of variables $T_s^4, T_0^4, T_1^4$.

We can solve this matrix problem to get these as functions of $T_e^4$.

```python
# Solve for radiative equilibrium
fourthpower = sympy.solve(radeq2, [T_s**4, T_1**4, T_0**4])
fourthpower
```

This produces a dictionary of solutions for the fourth power of the temperatures!

A little manipulation gets us the solutions for temperatures that we want:

```python
# need the symbolic fourth root operation
from sympy.simplify.simplify import nthroot

fourthpower_list = [fourthpower[key] for key in [T_s**4, T_0**4, T_1**4]]
solution = sympy.Matrix([nthroot(item,4) for item in fourthpower_list])
#  Display result as matrix equation!
T = sympy.Matrix([T_s, T_0, T_1])
sympy.Equality(T, solution)
```

In more familiar notation, the radiative equilibrium solution is thus

\begin{align}
T_s &= T_e \left( \frac{2+\epsilon}{2-\epsilon} \right)^{1/4} \\
T_0 &= T_e \left( \frac{1+\epsilon}{2-\epsilon} \right)^{1/4} \\
T_1 &= T_e \left( \frac{ 1}{2 - \epsilon} \right)^{1/4}
\end{align}

Plugging in the tuned value $\epsilon = 0.586$ gives

```python
Tsolution = solution.subs(tuned)
#  Display result as matrix equation!
sympy.Equality(T, Tsolution)
```

Now we just need to know the Earth's emission temperature $T_e$!

(Which we already know is about 255 K)

```python
# Here's how to calculate T_e from the observed values
sympy.solve(((1-alpha)*Q - sigma*T_e**4).subs(tuned), T_e)
```

```python
# Need to unpack the list
Te_value = sympy.solve(((1-alpha)*Q - sigma*T_e**4).subs(tuned), T_e)[0]
Te_value
```

#### Now we finally get our solution for radiative equilibrium

```python
#  Output 4 significant digits
Trad = sympy.N(Tsolution.subs([(T_e, Te_value)]), 4)
sympy.Equality(T, Trad)
```

Compare these to the values we derived from the **observed lapse rates**:

```python
sympy.Equality(T, T.subs(tuned))
```

The **radiative equilibrium** solution is substantially **warmer at the surface** and **colder in the lower troposphere** than reality.

This is a very general feature of radiative equilibrium, and we will see it again very soon in this course.

____________

## 7. Summary
____________

## Key physical lessons

- Putting a **layer of longwave absorbers** above the surface keeps the **surface substantially warmer**, because of the **backradiation** from the atmosphere (greenhouse effect).
- The **grey gas** model assumes that each layer absorbs and emits a fraction $\epsilon$ of its blackbody value, independent of wavelength.

- With **incomplete absorption** ($\epsilon < 1$), there are contributions to the OLR from every level and the surface (there is no single **level of emission**)
- Adding more absorbers means that **contributions to the OLR** from **upper levels** go **up**, while contributions from the surface go **down**.
- This upward shift in the weighting of different levels is what we mean when we say the **level of emission goes up**.

- The **radiative forcing** caused by an increase in absorbers **depends on the lapse rate**.
- For an **isothermal atmosphere** the radiative forcing is zero and there is **no greenhouse effect**
- The radiative forcing is positive for our atmosphere **because tropospheric temperatures tends to decrease with height**.
- Pure **radiative equilibrium** produces a **warm surface** and **cold lower troposphere**.
- This is unrealistic, and suggests that crucial heat transfer mechanisms are missing from our model.

### And on the Python side...

Did we need `sympy` to work all this out? No, of course not. We could have solved the 3x3 matrix problems by hand. But computer algebra can be very useful and save you a lot of time and error, so it's good to invest some effort into learning how to use it.

Hopefully these notes provide a useful starting point.

____________

## Credits

This notebook is part of [The Climate Laboratory](https://brian-rose.github.io/ClimateLaboratoryBook), an open-source textbook developed and maintained by [Brian E. J. Rose](http://www.atmos.albany.edu/facstaff/brose/index.html), University at Albany. It has been modified by [Nicole Feldl](http://nicolefeldl.com), UC Santa Cruz.

It is licensed for free and open consumption under the
[Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) license.

Development of these notes and the [climlab software](https://github.com/brian-rose/climlab) is partially supported by the National Science Foundation under award AGS-1455071 to Brian Rose. Any opinions, findings, conclusions or recommendations expressed here are mine and do not necessarily reflect the views of the National Science Foundation.
____________

```python

```


# Part 1: Linear Regression

```
# Execute this code block to install dependencies when running on colab
try:
    import torch
except:
    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision
```

## Getting started

At its heart, PyTorch is just a library for manipulating tensors. We're going to start learning how to use
PyTorch by looking at how we can implement simple linear regression.

Code speaks better than words, so lets start by looking at a bit of pytorch code to generate some 2d data to regress:

```
import torch

# Generate some data points on a straight line perturbed with Gaussian noise
N = 1000 # number of points
theta_true = torch.Tensor([[1.5], [2.0]]) # true parameters of the line

X = torch.rand(N, 2) #Returns a tensor filled with random numbers from a uniform distribution on the interval [0, 1)
X[:, 1] = 1.0  #the second column
print(X)
y = X @ theta_true + 0.1 * torch.randn(N, 1) # Note that just like in numpy '@' represents matrix multiplication and A@B is equivalent to torch.mm(A, B)
```

    tensor([[0.7577, 1.0000],
            [0.5942, 1.0000],
            [0.2248, 1.0000],
            ...,
            [0.2073, 1.0000],
            [0.8536, 1.0000],
            [0.6797, 1.0000]])

The above code generates $(x,y)$ data according to $y = 1.5x + 2$, with the $x$'s chosen from a uniform distribution. The $y$'s are additionally purturbed by adding an amount $0.1z$, where $z\sim \mathcal{N}(0,1)$ is a sample from a standard normal distribution.

Note that we represent our $x$'s as a two-dimensional (row) vector with a 1 in the second element so that the offset can be rolled into the matrix multiplication for efficiency:

\begin{align}
    y &= \mathbf{X}\begin{bmatrix}
           1.5 \\
           2
         \end{bmatrix}
  \end{align}

Let's use `matplotlib` to draw a scatter so we can be sure of what our data looks like:

```
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X[:,0].numpy(), y.numpy())
plt.show()
```

__Make sure you understand how the code above is generating data; feel free to change the parameters to see what effect they have.__

Now, lets consider the situation where we have been given the tensors $X$ and $y$ and wish to compute the regression parameters. Our model looks like $\mathbf{y} = \mathbf{X\theta}$, and we wish to recover the parameters $\theta$.

As the problem is both overcomplete (only two data pairs are required to find $\theta$), and the data is noisy, we can use the Moore-Penrose Pseudoinverse to find the least-squares solution to $\theta$: $\theta = \mathbf{X^+y}$. PyTorch has a built-in pseudoinverse method (`pinverse`) that can do all the work for us:

```
# direct solution using moore-penrose pseudo inverse
X_inv = torch.pinverse(X)
theta_pinv = torch.mm(X_inv, y)
print(theta_pinv)
```

    tensor([[1.5169],
            [1.9939]])

Running the above code should give you a solution vector for $\theta$ that is very similar to the true parameter vector (`theta_true`).

## Exercise: computing the pseudoinverse from the Singular Value Decomposition.

The standard way of computing the pseudoinverse is by using the Singular Value Decomposition (SVD). The SVD is defined as: $\mathbf{X} = \mathbf{U}\Sigma\mathbf{V}^\top$. The pseudoinverse is thus $\mathbf{X}^+ = \mathbf{V}\Sigma^{-1}\mathbf{U}^\top$ where $\Sigma^{-1}$ is a diagonal matrix in which the reciprocal of the corresponding non-zero elements in $\Sigma$ has been taken.

__Use the code block below to compute the parameter vector using the SVD directly rather than the through the `pinverse` method.__ You need to store your manually computed pseudoinverse in `X_inv_svd`. Useful methods will be `torch.svd()` to compute the SVD, `[Tensor].t()` to transpose a matrix and `torch.diag()` to form a diagonal matrix from a vector.

```
# YOUR CODE HERE
u,s,v=torch.svd(X)
print('U=',u)
print('S=',s)
print('V=',v)
s1=torch.diag(s)
print("S1=",s1)
X_inv_svd=v@torch.inverse(s1)@u.t()
print("X_inv_svd=",X_inv_svd)
# raise NotImplementedError()

theta_pinv_svd = torch.mm(X_inv_svd, y)
print("theta_pinv_svd=",theta_pinv_svd)
```

    U= tensor([[-0.0348,  0.0240],
            [-0.0326,  0.0063],
            [-0.0277, -0.0337],
            ...,
            [-0.0275, -0.0356],
            [-0.0361,  0.0343],
            [-0.0338,  0.0155]])
    S= tensor([35.6136,  8.1517])
    V= tensor([[-0.4725,  0.8813],
            [-0.8813, -0.4725]])
    S1= tensor([[35.6136,  0.0000],
            [ 0.0000,  8.1517]])
    X_inv_svd= tensor([[ 3.0518e-03,  1.1121e-03, -3.2711e-03,  ..., -3.4793e-03,
              4.1895e-03,  2.1268e-03],
            [-5.2742e-04,  4.4339e-04,  2.6372e-03,  ...,  2.7414e-03,
             -1.0968e-03, -6.4459e-05]])
    theta_pinv_svd= tensor([[1.5169],
            [1.9939]])

```
print(torch.abs(torch.add(theta_pinv, -theta_pinv_svd)))
assert(torch.all(torch.lt(torch.abs(torch.add(theta_pinv, -theta_pinv_svd)), 1e-6)))
```

    tensor([[1.1921e-07],
            [2.3842e-07]])

## Gradient based Linear Regression

Fundamentally, with linear regression we are trying to find a solution vector, $theta$ that minimises $f(\theta) = 0.5\|\mathbf{X}\theta - \mathbf{y}\|_2^2$.

We've already seen how this can be minimised directly using the pseudoinverse, but it could also be minimised by using gradient descent: $\theta \gets \theta - \alpha f'(\theta)$. (_Interesting aside_: SVD (and thus the pseudoinverse) can also be solved using gradient methods - in fact this becomes the only practical way for really large matrices.).

__Use the following block to derive and write down the gradient, $f'(\theta)$, of $f(\theta)$__. Note that you can insert latex code by wrapping expressions in dollar symbols.

- **Answer**:

**The deriviate of $f(\theta) = 0.5\|\mathbf{X}\theta - \mathbf{y}\|_2^2$ is $0.5* 2*\mathbf{X}^T(\mathbf{X}\theta-\mathbf{y})$**

__Now complete the following code block to implement your gradient as pytorch code:__

```
def linear_regression_loss_grad(theta, X, y):
    # theta, X and y have the same shape as used previously
    # YOUR CODE HERE
#     raise NotImplementedError()

    n, p = X.shape  # n rows, p columns actually n row 2 column
#     print(n,p)
#     grad1=0.5*(-2/n)*X.t()@((y-torch.mm(X,theta))) ###
    grad=0.5*(-2)*X.t()@((y-torch.mm(X,theta)))
    return grad
```

```
assert(linear_regression_loss_grad(torch.zeros(2,1), X, y).shape == (2,1))

```

Now we can plug that gradient function into a basic gradient descent solver and check that the solution is close to what we get with the pseudoinverse:

```
alpha = 0.001
theta = torch.Tensor([[0], [0]])
for e in range(0, 300):
    gr = linear_regression_loss_grad(theta, X, y)
    theta -= alpha * gr

print(theta)
```

    tensor([[1.5169],
            [1.9939]])

## Real data

Doing linear regression on synthetic data is a great way to understand how PyTorch works, but it isn't quite as satisfying as working with a real dataset. Let's now apply or understanding of computing linear regression parameters to a dataset of house prices in Boston.

We'll load the dataset using scikit-learn and perform some manipulations in the following code block:

```
from sklearn.datasets import load_boston

X, y = tuple(torch.Tensor(z) for z in load_boston(True)) #convert to pytorch Tensors
X = X[:, [2,5]] # We're just going to use features 2 and 5, rather than using all of of them  #from the 3th column to 5th column
X = torch.cat((X, torch.ones((X.shape[0], 1))), 1) # append a column of 1's to the X's

y = y.reshape(-1, 1) # reshape y into a column vector
print(X)
# print(X[:,2])
print('X:', X.shape)  #506*3
print('y:', y.shape)  #506*1

# We're also going to break the data into a training set for computing the regression parameters
# and a test set to evaluate the predictive ability of those parameters
perm = torch.randperm(y.shape[0]) #Returns a random permutation of integers from 0 to n - 1.
X_train = X[perm[0:253], :] #254*3
y_train = y[perm[0:253]]    #254*1
X_test = X[perm[253:], :]
y_test = y[perm[253:]]
```

    tensor([[ 2.3100,  6.5750,  1.0000],
            [ 7.0700,  6.4210,  1.0000],
            [ 7.0700,  7.1850,  1.0000],
            ...,
            [11.9300,  6.9760,  1.0000],
            [11.9300,  6.7940,  1.0000],
            [11.9300,  6.0300,  1.0000]])
    X: torch.Size([506, 3])
    y: torch.Size([506, 1])

__Use the following code block to compute the regression parameters using the training data in the variable `theta` by solving using the pseudoinverse directly:__

```
# compute the regression parameters in variable theta
# YOUR CODE HERE
# raise NotImplementedError()
X_inv = torch.pinverse(X_train)
theta= torch.mm(X_inv, y_train)
print(theta)
```

    tensor([[ -0.3223],
            [  7.4680],
            [-20.5415]])

```
assert(theta.shape == (3,1))

print("Theta: ", theta.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta, y_test))
```

    Theta:  tensor([[ -0.3223,   7.4680, -20.5415]])
    MSE of test data:  tensor(39.9052)

Now let's try using gradient descent:

```
alpha = 0.00001 #0.00001
theta_gd = torch.rand((X_train.shape[1], 1))  #initialize
for e in range(0, 10000):
    gr = linear_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr

print("Gradient Descent Theta: ", theta_gd.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))
```

    Gradient Descent Theta:  tensor([[-0.4533,  4.9514, -3.1055]])
    MSE of test data:  tensor(44.1014)

__Use the following block to note down any observations you can make about the choice of learning rate and number of iterations in the above code. What factors do you think influence the choice?__

```
alpha = 0.0001
theta_gd = torch.rand((X_train.shape[1], 1))  #initialize
for e in range(0, 10000):
    gr = linear_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr

print("Gradient Descent Theta: ", theta_gd.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))
```

    Gradient Descent Theta:  tensor([[nan, nan, nan]])
    MSE of test data:  tensor(nan)

```
import numpy as np
alphas = [10**(-7),0.000001,0.00001]
numiter_array=np.array([10**4,6*10**4,8*10**4,12*10**4])
for i,r in enumerate(alphas):
  for numiter in numiter_array:
    theta_gd = torch.rand((X_train.shape[1], 1))  #initialize
    for e in range(0, numiter):  #80000
        gr = linear_regression_loss_grad(theta_gd, X_train, y_train)
        theta_gd -= r * gr
    print("alpha is: ",r,"numiter is: ",numiter,"Gradient Descent Theta: ", theta_gd.t())
    print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))

```

    alpha is:  1e-07 numiter is:  10000 Gradient Descent Theta:  tensor([[-0.3959,  4.1427,  1.0744]])
    MSE of test data:  tensor(46.8674)
    alpha is:  1e-07 numiter is:  60000 Gradient Descent Theta:  tensor([[-0.4850,  4.3420,  1.1158]])
    MSE of test data:  tensor(45.9251)
    alpha is:  1e-07 numiter is:  80000 Gradient Descent Theta:  tensor([[-0.4786,  4.4641,  0.2695]])
    MSE of test data:  tensor(45.5343)
    alpha is:  1e-07 numiter is:  120000 Gradient Descent Theta:  tensor([[-0.4775,  4.4851,  0.1238]])
    MSE of test data:  tensor(45.4683)
    alpha is:  1e-06 numiter is:  10000 Gradient Descent Theta:  tensor([[-0.4836,  4.3690,  0.9293]])
    MSE of test data:  tensor(45.8378)
    alpha is:  1e-06 numiter is:  60000 Gradient Descent Theta:  tensor([[-0.4616,  4.7923, -2.0035]])
    MSE of test data:  tensor(44.5471)
    alpha is:  1e-06 numiter is:  80000 Gradient Descent Theta:  tensor([[-0.4597,  4.8288, -2.2563]])
    MSE of test data:  tensor(44.4430)
    alpha is:  1e-06 numiter is:  120000 Gradient Descent Theta:  tensor([[-0.4488,  5.0382, -3.7073]])
    MSE of test data:  tensor(43.8671)
    alpha is:  1e-05 numiter is:  10000 Gradient Descent Theta:  tensor([[-0.4502,  5.0118, -3.5239]])
    MSE of test data:  tensor(43.9378)
    alpha is:  1e-05 numiter is:  60000 Gradient Descent Theta:  tensor([[ -0.3659,   6.6300, -14.7352]])
    MSE of test data:  tensor(40.7059)
    alpha is:  1e-05 numiter is:  80000 Gradient Descent Theta:  tensor([[ -0.3500,   6.9359, -16.8546]])
    MSE of test data:  tensor(40.3446)
    alpha is:  1e-05 numiter is:  120000 Gradient Descent Theta:  tensor([[ -0.3335,   7.2530, -19.0517]])
    MSE of test data:  tensor(40.0538)

```
# YOUR CODE HERE
import numpy as np
# raise NotImplementedError()
def gradient_descent(X,y,theta_gd,alpha,numiter):
  theta_history = []
  mse_loss_history = []
  for i in range(numiter):
    mse_loss=torch.nn.functional.mse_loss(X@ theta_gd, y)
    theta_history.append(theta_gd)
    mse_loss_history.append(mse_loss)
    gr = linear_regression_loss_grad(theta_gd, X, y)
    theta_gd -= alpha * gr
  return theta_gd,theta_history,mse_loss_history

alphas = [10**(-7),0.000001,0.00001]
theta_gd_init = torch.rand((X_train.shape[1], 1))  #initialize
numiter_array=np.array([10**4,6*10**4,8*10**4])
for numiter in numiter_array:
  for i,r in enumerate(alphas):
    theta_gd,theta_history,mse_loss_history=gradient_descent(X_train,y_train,theta_gd_init,r,numiter)
    plt.plot(mse_loss_history,label=r)
    plt.legend()
  plt.show()

print("Gradient Descent Theta: ", theta_gd.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))

```

- Answer:

**From the above, it indicates that the smaller the learning rate, the slower it converges. On the other hand, the larger the learning rate, it is harder to converge. In terms of number of iterations, the more of number of iterations, the less of the  loss. **

**Finally, we choose learning rate = $0.00001$,  number of iterations=$180000$**

```
alpha = 0.00001
theta_gd = torch.rand((X_train.shape[1], 1))  #initialize
for e in range(0, 18*10**4):
    gr = linear_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr

print("Gradient Descent Theta: ", theta_gd.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))
```

    Gradient Descent Theta:  tensor([[ -0.3252,   7.4126, -20.1577]])
    MSE of test data:  tensor(39.9397)

Finally, just so we can visualise what our model has learned, we can plot the predicted house prices (from both the direct solution and from gradient descent) along with the true value for each of the houses in the test set (ordered by increasing true value):

```
perm = torch.argsort(y_test, dim=0)
plt.plot(y_test[perm[:,0]].numpy(), '.', label='True Prices')
plt.plot((X_test[perm[:,0]] @ theta).numpy(), '.', label='Predicted (pinv)')
plt.plot((X_test[perm[:,0]] @ theta_gd).numpy(), '.', label='Predicted (G.D.)')
plt.xlabel('House Number')
plt.ylabel('House Price ($,000s)')
plt.legend()
plt.show()
```

```

```


```python

```

Support the display of mathematical expressions typeset in LaTeX, which is rendered in the browser thanks to the `MathJax library`.

```python
from IPython.display import Math
Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx')
```

$\displaystyle F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx$

With the Latex class, you have to include the delimiters yourself. This allows you to use other LaTeX modes such as eqnarray:

```python
from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}""")
```

\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}

Or you can enter latex directly with the %%latex cell magic:

```latex
%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
```

\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

```python

```

To insert a mathematical formula we use the dollar symbol $, as follows:
Euler's identity: $ e^{i \pi} + 1 = 0 $
To isolate and center the formulas and enter in math display mode, we use 2 dollars symbol:
$$
...
$$
Euler's identity: $$ e^{i \pi} + 1 = 0 $$

```python

```


## Visualizing Attractors

An [attractor](https://en.wikipedia.org/wiki/Attractor#Strange_attractor) is a set of values to which a numerical system tends to evolve. An attractor is called a [strange attractor](https://en.wikipedia.org/wiki/Attractor#Strange_attractor) if the resulting pattern has a fractal structure. This notebook shows how to calculate and plot two-dimensional attractors of a variety of types, using code and parameters primarily from [Lázaro Alonso](https://lazarusa.github.io/Webpage/codepython2.html), [François Pacull](https://aetperf.github.io/2018/08/29/Plotting-Hopalong-attractor-with-Datashader-and-Numba.html), [Jason Rampe](https://softologyblog.wordpress.com/2017/03/04/2d-strange-attractors), [Paul Bourke](http://paulbourke.net/fractals/), and [James A. Bednar](http://github.io/jbednar).

## Clifford Attractors

For example, a [Clifford Attractor](http://paulbourke.net/fractals/clifford) is a strange attractor defined by two iterative equations that determine the _x,y_ locations of discrete steps in the path of a particle across a 2D space, given a starting point _(x0,y0)_ and the values of four parameters _(a,b,c,d)_:

\begin{equation}
x_{n +1} = \sin(a y_{n}) + c \cos(a x_{n})\\
y_{n +1} = \sin(b x_{n}) + d \cos(b y_{n})
\end{equation}

At each time step, the equations define the location for the following time step, and the accumulated locations show the areas of the 2D plane most commonly visited by the imaginary particle.

It's easy to calculate these values in Python using [Numba](http://numba.pydata.org). First, we define the iterative attractor equation:

```python
import numpy as np, pandas as pd, datashader as ds
from datashader import transfer_functions as tf
from datashader.colors import inferno, viridis
from numba import jit
from math import sin, cos, sqrt, fabs

@jit
def Clifford(x, y, a, b, c, d, *o):
    return sin(a * y) + c * cos(a * x), \
           sin(b * x) + d * cos(b * y)
```

We then evaluate this equation 10 million times, creating a set of _x,y_ coordinates visited. The `@jit` here and above is optional, but it makes the code 50x faster.

```python
n=10000000

@jit
def trajectory(fn, x0, y0, a, b=0, c=0, d=0, e=0, f=0, n=n):
    x, y = np.zeros(n), np.zeros(n)
    x[0], y[0] = x0, y0
    for i in np.arange(n-1):
        x[i+1], y[i+1] = fn(x[i], y[i], a, b, c, d, e, f)
    return pd.DataFrame(dict(x=x,y=y))
```

```python
%%time
df = trajectory(Clifford, 0, 0, -1.3, -1.3, -1.8, -1.9)
```

```python
df.tail()
```

We can now aggregate these 10,000,000 continuous coordinates into a discrete 2D rectangular grid with [Datashader](http://datashader.org), counting each time a point fell into that grid cell:

```python
%%time

cvs = ds.Canvas(plot_width = 700, plot_height = 700)
agg = cvs.points(df, 'x', 'y')
print(agg.values[190:195,190:195],"\n")
```

A small portion of that grid is shown above, but it's difficult to see the grid's structure from the numerical values.  To see the entire array at once, we can turn each grid cell into a pixel, using a greyscale value from white to black:

```python
ds.transfer_functions.Image.border=0

tf.shade(agg, cmap = ["white", "black"])
```

As you can see, the most-visited areas of the plane have an interesting structure for this set of parameters. To explore further, let's wrap up the above aggregation and shading commands into a function so we can apply them more easily:

```python
def dsplot(fn, vals, n=n, cmap=viridis, label=True):
    """Return a Datashader image by collecting `n` trajectory points for the given attractor `fn`"""
    lab = ("{}, "*(len(vals)-1)+" {}").format(*vals) if label else None
    df  = trajectory(fn, *vals, n=n)
    cvs = ds.Canvas(plot_width = 300, plot_height = 300)
    agg = cvs.points(df, 'x', 'y')
    img = tf.shade(agg, cmap=cmap, name=lab)
    return img
```

And let's load some colormaps that we can use for subsequent plots:

```python
from colorcet import palette
palette["viridis"]=viridis
palette["inferno"]=inferno
```

We can now use these colormaps with a pre-selected set of Clifford attractor parameter values (stored in a separate [YAML-format text file](https://raw.githubusercontent.com/pyviz/datashader/master/examples/topics/attractors.yml)) to show a wide variety of trajectories that these equations can form:

```python
import yaml
vals = yaml.load(open("attractors.yml","r"))

def args(name):
    """Return a list of available argument lists for the given type of attractor"""
    return [v[1:] for v in vals if v[0]==name]

def plot(fn, vals=None, **kw):
    """Plot the given attractor `fn` once per provided set of arguments."""
    vargs=args(fn.__name__) if vals is None else vals
    return tf.Images(*[dsplot(fn, v[1:], cmap=palette[v[0]][::-1], **kw) for v in vargs]).cols(4)
```

```python
plot(Clifford)
```

Here the values shown are the arguments for the first call to `Clifford(x, y, a, b, c, d)`, with each subsequent call using the _x,y_ location of the previous call.

Randomly sampling the parameter space typically yields much less dramatic patterns, such as all trajectory locations being on a small number of points:

```python
import numpy.random
numpy.random.seed(21)
num = 4

rvals=np.c_[np.zeros((num,2)), numpy.random.random((num,4))*4-2]
plot(Clifford, vals=[["kbc"]+list(rvals[i]) for i in range(len(rvals))], label=True)
```

If you wish, Datashader could easily be used to filter out such uninteresting examples, by applying a criterion to the aggregate array before shading and showing only those that remain (e.g. rejecting those where 80% of the pixel bins are empty).

## De Jong attractors

A variety of other sets of attractor equations have been proposed, such as these from [Peter de Jong](http://paulbourke.net/fractals/peterdejong):

```python
@jit
def De_Jong(x, y, a, b, c, d, *o):
    return sin(a * y) - cos(b * x), \
           sin(c * x) - cos(d * y)

plot(De_Jong)
```

## Svensson attractors

From [Johnny Svensson](http://paulbourke.net/fractals/peterdejong/):

```python
@jit
def Svensson(x, y, a, b, c, d, *o):
    return d * sin(a * x) - sin(b * y), \
           c * cos(a * x) + cos(b * y)

plot(Svensson)
```

## Bedhead Attractor

From [Ivan Emrich](https://www.deviantart.com/jaguarfacedman) and [Jason Rampe](https://softologyblog.wordpress.com/2017/03/04/2d-strange-attractors):

```python
@jit
def Bedhead(x, y, a, b, *o):
    return sin(x*y/b)*y + cos(a*x-y), \
           x + sin(y)/b

plot(Bedhead)
```

## Fractal Dream Attractor

From Clifford A. Pickover's book “Chaos In Wonderland”, with parameters from [Jason Rampe](https://softologyblog.wordpress.com/2017/03/04/2d-strange-attractors):

```python
@jit
def Fractal_Dream(x, y, a, b, c, d, *o):
    return sin(y*b)+c*sin(x*b), \
           sin(x*a)+d*sin(y*a)

plot(Fractal_Dream)
```

## Hopalong attractors

From Barry Martin, here with code for two variants from [François Pacull](https://aetperf.github.io/2018/08/29/Plotting-Hopalong-attractor-with-Datashader-and-Numba.html):

```python
@jit
def Hopalong1(x, y, a, b, c, *o):
    return y - sqrt(fabs(b * x - c)) * np.sign(x), \
           a - x
@jit
def Hopalong2(x, y, a, b, c, *o):
    return y - 1.0 - sqrt(fabs(b * x - 1.0 - c)) * np.sign(x - 1.0), \
           a - x - 1.0

plot(Hopalong1)
```

```python
plot(Hopalong2)
```

##  Gumowski-Mira Attractor

From [I. Gumowski and C. Mira](http://kgdawiec.bplaced.net/badania/pdf/cacs_2010.pdf), with code and parameters from [Jason Rampe](https://softologyblog.wordpress.com/2017/03/04/2d-strange-attractors) and [Lázaro Alonso](https://lazarusa.github.io/Webpage/codepython2.html):

```python
@jit
def G(x, mu):
    return mu * x + 2 * (1 - mu) * x**2 / (1.0 + x**2)

@jit
def Gumowski_Mira(x, y, a, b, mu, *o):
    xn = y + a*(1 - b*y**2)*y  +  G(x, mu)
    yn = -x + G(xn, mu)
    return xn, yn

plot(Gumowski_Mira)
```

##  Symmetric Icon Attractor

The Hopalong and Gumowski-Mira equations often result in symmetric patterns, but a different approach is to *force* the patterns to be symmetric, which is often pleasing. Examples from “Symmetry in Chaos” by Michael Field and Martin Golubitsky, with code and parameters from [Jason Rampe](https://softologyblog.wordpress.com/2017/03/04/2d-strange-attractors):

```python
@jit
def Symmetric_Icon(x, y, a, b, g, om, l, d, *o):
    zzbar = x*x + y*y
    p = a*zzbar + l
    zreal, zimag = x, y

    for i in range(1, d-1):
        za, zb = zreal * x - zimag * y, zimag * x + zreal * y
        zreal, zimag = za, zb

    zn = x*zreal - y*zimag
    p += b*zn

    return p*x + g*zreal - om*y, \
           p*y - g*zimag + om*x

plot(Symmetric_Icon)
```

## Interactive plotting

If you are running a live Python process, you can use Datashader with HoloViews and Bokeh to zoom in and see the individual steps in any of these calculations:

```python
import holoviews as hv
from holoviews.operation.datashader import datashade, dynspread
hv.extension('bokeh')

dynspread(datashade(hv.Points(trajectory(Clifford, *(args("Clifford")[5][1:]))),
                    cmap=viridis[::-1]).opts(width=400,height=400))
```

Each time you zoom in in a live process, the data will be reaggregated, which should take a small fraction of a second for 10 million points.  Eventually, once you zoom in enough you should see individual data points, as we are not connecting the points into a trajectory here.

You can also try "connecting the dots", which will reveal how the particle jumps discretely from one region of the space to another:

```python
dynspread(datashade(hv.Path([trajectory(Clifford, *(args("Clifford")[5][1:]))]),
                    cmap=viridis[::-1]).opts(width=400,height=400))
```

Again, if you zoom in on a live server, the plot will update so that you can see the individual traces involved.

On the live server, you can also explore to find your own parameter values that generate interesting patterns:

```python
def hv_clif(a,b,c,d,x0=0,y0=0,n=n):
    return datashade(hv.Points(trajectory(Clifford, x0, y0, a, b, c, d, n)),
                     cmap=inferno[::-1], dynamic=False)

x0,y0,a,b,c,d = args("Clifford")[6][1:]

dm = hv.DynamicMap(hv_clif, kdims=['a', 'b', 'c', 'd'])
dm = dm.redim.range(a=(-2.0, 2.0), b=(-2.0,2.0), c=(-2.0,2.0), d=(-2.0,2.0))
dm = dm.redim.default(a=a, b=b, c=c, d=d).opts(width=500,height=500)
dm
```

Although many of the regions of this four-dimensional parameter space generate uninteresting trajectories such as single points, you can find interesting regions by starting with one of the _a,b,c,d_ tuples of values in previous plots, then click on one slider and use the left and right arrow keys to see how the plot changes as that parameter changes.


# Tutorial

We will solve the following problem using a computer to using a programming
technique called **recursion**.

```{admonition} Problem

A sequence $a_1, a_2, a_3, …$ is defined by:

$$
    \left\{
    \begin{array}{l}
        a_1 = k,\\
        a_{n + 1} = 2a_n – 7, n \geq 1,
    \end{array}
    \right.
$$

where $k$ is a constant.

1. Write down an expression for $a_2$ in terms of $k$.
2. Show that $a_3 = 4k -21$
3. Given that $\sum_{r=1}^4 a_r = 43$ find the value of $k$.
```

We will use a Python to define a function that reproduces the mathematical
definition of $a_k$:

```python
def generate_a(k_value, n):
    """
    Uses recursion to return a_n for a given value of k:

    a_1 = k
    a_n = 2a_n - 7
    """
    if n == 1:
        return k_value
    return 2 * generate_a(k_value, n - 1) - 7
```

```{attention}
This is similar to the mathematical definition the Python definition of
the function refers to itself.
```

We can use this to compute $a_3$ for $k=4$:

```python
generate_a(k_value=4, n=3)
```

    -5

We can use this to compute $a_5$ for $k=1$:

```python
generate_a(k_value=1, n=5)
```

    -89

Finally it is also possible to pass a symbolic value to `k_value`. This allows
us to answer the first question:

```python
import sympy as sym

k = sym.Symbol("k")
generate_a(k_value=k, n=2)
```

$\displaystyle 2 k - 7$

Likewise for $a_3$:

```python
generate_a(k_value=k, n=3)
```

$\displaystyle 4 k - 21$

For the last question we start by computing the sum:

$$
    \sum_{r=1}^4 a_r = 43
$$

```python
sum_of_first_four_terms = sum(generate_a(k_value=k, n=r) for r in range(1, 5))
sum_of_first_four_terms
```

$\displaystyle 15 k - 77$

This allows us to create the given equation and solve it:

```python
equation = sym.Eq(sum_of_first_four_terms, 43)
sym.solveset(equation, k)
```

$\displaystyle \left\{8\right\}$

```{important}
In this tutorial we have

- Defined a function using recursion.
- Called this function using both numeric and symbolic values.
```


# 15.7. Analyzing a nonlinear differential system — Lotka-Volterra (predator-prey) equations

```python
from sympy import *
init_printing(pretty_print=True)

var('x y')
var('a b c d', positive=True)
```

```python
f = x * (a - b * y)
g = -y * (c - d * x)
```

```python
solve([f, g], (x, y))
```

```python
(x0, y0), (x1, y1) = _
```

```python
M = Matrix((f, g))
M
```

```python
J = M.jacobian((x, y))
J
```

```python
M0 = J.subs(x, x0).subs(y, y0)
M0
```

```python
M0.eigenvals()
```

```python
M1 = J.subs(x, x1).subs(y, y1)
M1
```

```python
M1.eigenvals()
```


# Solving Non Linear Systems using Newton's Method

We'll be finding the roots of the following system of equations:

$f(\bar{v},\theta)$  =  0.5$\bar{v}^{2}$  + $\sin(\theta)$ = 0

$g(\bar{v},\theta)$ =  0.5$\bar{v}^{2}$  - $\cos(\theta)$ = 0

```python
from numpy.linalg import inv
import numpy as np
import math
```

```python
def f(v_bar, theta):
    return 0.5*(v_bar**2) + math.sin(theta)

def g(v_bar, theta):
    return 0.5*(v_bar**2) - math.cos(theta)
```

The Jacobian Matrix is calculated as follows:
\begin{equation}
J =
\begin{bmatrix}
\frac{df(\bar{v},\theta)}{d\bar{v}} & \frac{df(\bar{v},\theta)}{d\theta}
ewline \\
 \frac{dg(\bar{v},\theta)}{d\bar{v}} & \frac{dg(\bar{v},\theta)}{d\theta}  \   \\
 \end{bmatrix}
\end{equation}

```python
def jacobian(v_bar,theta):
    array = np.array([[v_bar, math.cos(theta)],[v_bar, math.sin(theta)]])
    return array
```

Assuming the initial point as
\begin{equation}
x_{0} =
\begin{bmatrix}
0 \\
0 \\
\end{bmatrix}
\end{equation}

And the reiterating solution as
\begin{equation}
x_{n} =
\begin{bmatrix}
\bar{v}_{n} \\
\theta_{n} \\
\end{bmatrix}
\end{equation}

The next point is \begin{equation}
x_{n+1} =
x_{n} - J(\bar{v}_{n},\theta_{n})^{-1}F(\bar{v}_{n},\theta_{n})
\end{equation}

where \begin{equation}
F(\bar{v}_{n},\theta_{n}) =
\begin{bmatrix}
f(\bar{v}_{n},\theta_{n}) \\
g(\bar{v}_{n},\theta_{n}) \\
\end{bmatrix}
\end{equation}

```python
def next_x(old_x):

    inverse_jacobian = inv(jacobian(old_x[0][0],old_x[1][0]))
    F = np.array([[f(old_x[0][0],old_x[1][0])],[g(old_x[0][0],old_x[1][0])]])

    next_x = old_x - np.matmul(inverse_jacobian,F)
    return next_x
```

Running 10 iterations of the algorithm

```python
all_x = []

x = np.array([[1],[0]])
all_x.append(x)

for _ in range(10):
    x = next_x(x)
    all_x.append(x)
```

```python
final_x = np.array(all_x).reshape(11,2)
```

```python
print('Final Solution:',all_x[10][0][0],',',all_x[10][1][0])
```

    Final Solution: 1.189207115002721 , -0.7853981633974483

# Visualizing the convergence of the solution to the true solution

```python
import matplotlib.pyplot as plt

plt.figure(figsize=(12,6))
plt.plot(range(11),final_x[:,0])
plt.title('Convergence of the Value of v bar')
plt.axhline(1.18920712, color='red')
plt.show()
```

```python
import matplotlib.pyplot as plt

plt.figure(figsize=(12,6))
plt.plot(range(11),final_x[:,1])
plt.title('Convergence of the Value of theta')
plt.axhline(-0.78539816, color='red')
plt.show()
```


# Numpy & Scipy Review

I thought this would be the perfect opportunity to review some of the skills we've covered in Numpy and Scipy and introduce a few new skills.

This notebook will have you write a series of short functions that will perform deisred tasks. You may accomplish this with the Python `def` or `lambda` commands. Each one of your functions should be vector oriented (meaning I can pass in a numpy array and your function should still work). Most of what you'll be doing in this assignment is "wrapping" numpy and scipy functions, but this is designed to get you familiar with numerical processing in Python.

I'll go ahead and add code below for packages you may need. I'll also define several test data items below to refer to in the prompts.

I'll also lay out the function calls for you as well.

```python
# 3rd Part Imports
import numpy as np
import scipy.special
import scipy.linalg
from matplotlib import pyplot as plt

# Test Data
# 1D Data
a1d = np.arange(1, 4)
b1d = np.arange(4, 1, -1)
c1d = np.linspace(1, 2, 3)
x1dLarge = np.linspace(0, 5, 1001)
y1dLarge = np.linspace(0, 4, 501)

# Row Vectors
aRow = a1d[np.newaxis, :]
bRow = b1d[np.newaxis, :]

# Column Vectors
aCol = a1d[:, np.newaxis]
bCol = b1d[:, np.newaxis]

# Matrices
rng = np.random.default_rng(0)
A = rng.integers(0, 11, (3, 3))
B = rng.integers(0, 11, (3, 3))
```

## Example

### Element Wise Addition

An example prompt could read:

> Write a python function called `add` that takes two inputs, `a` and `b`, and adds the values together element-wise.

I'll show you two different ways this could be done here.

```python
# Add Function Definition 1
def add1(a, b):
    return a + b

# Add Function Definition 2
add2 = lambda a, b: a + b
```

```python
# Test the Functions here.
print(add1(A, B))
print(add2(A, B))
```

### Mesh Example

Here is another example of something I could ask similar to the previous example. It has to do with performing an operation over some kind of mesh. That is, consider a problem to be two dimensional, $x \times y$ for example, where $x$ is along the $\hat{i}$ dimension and $y$ is along the $\hat{j}$ direction. This can be solved numerically over a mesh of two 1D vectors `x` and `y`. A prompt may look like the following:

> Write a function that accepts two 1D Numpy vectors with size $m$ and $n$ respectively and returns the 2D value of $x \times y$. Call the function `mult2d`.

Here are my example solutions below.

```python
def mult2d(x, y):

    # Get the Gridded/Meshed x and y
    xM, yM = np.meshgrid(x, y)

    return xM * yM
```

```python
# My Test for the Code
z = mult2d(x1dLarge, y1dLarge)
_ = plt.contourf(x1dLarge, y1dLarge, z, 20)
_ = plt.xlabel('X')
_ = plt.ylabel('Y')
_ = plt.colorbar()
```

Numpy also has a series of functions called "outer" functions that can do this in a threaded basis, but computationally it is more complicated to understand.

```python
def mult2d(x, y):

    return np.multiply.outer(y, x)  # y has to come first since we want it along the rows
```

```python
# My Test for the Code
z = mult2d(x1dLarge, y1dLarge)
_ = plt.contourf(x1dLarge, y1dLarge, z, 20)
_ = plt.xlabel('X')
_ = plt.ylabel('Y')
_ = plt.colorbar()
```

## Assignment

Now, it is your turn take an opportunity to work the problems below.

### Area of a Trapezoid

> Write a function that accepts three values ($b_1$, $b_2$ and $h$) that calculates the area of a trapezoid. Call the function `trap`.

The equation of the trapezoid is

\begin{equation}
    \frac{b_1 + b_2}{2} h.
\end{equation}

```python
# Trapezoid Funcion
def trap(b1, b2, h):
    pass
```

```python
# You may test your function here

```

### Arc Length of Circle

> Write a function called `arc` that accepts the radius and interior angle (in degrees) of a circle which returns the arc length of the circle.

The formula for arc length of the circle is

\begin{equation}
    s = r \theta
\end{equation}

where $r$ is the radius and $\theta$ is the interior angle measured in *radians*. The Numpy [deg2rad](https://docs.scipy.org/doc/numpy/reference/generated/numpy.deg2rad.html) function may be useful.

```python
# Arc Length Function
def arc(r, th):
    pass
```

```python
# You may test your code here

```

### Sum Difference

> Write a Python function called `sumdiff` to calculate the difference between the squared sum of first n natural numbers and the sum of squared first n natural numbers.

The function should implement

\begin{equation}
    x = \left( \sum_{i=1}^N y_i \right)^2 - \sum_{i=1}^N y_i^2
\end{equation}

```python
def sumdiff(N):
    pass
```

```python
# You may test your code here

```

### Quadratic Roots

> Write a function called `quad` that accepts three arrays (`a`, `b` and `c`) and returns the two roots from the coefficients of the equation. If the inputs are of length $N$, the output should be of shape (N, 2).

Part of the solution is implemented below.

```python
def quad(a, b, c):

    # Store the plus sol here
    solP = []

    # Store the minus sol here
    solM = []

    return np.vstack((solP, solM))
```

```python
# You may test your code here

```

### Cross Product

> Write a function called `cross` that computes the cross product between two vectors.

```python
# Cross Product
def cross(a, b):
    pass
```

```python
# You May Test your code here

```

### Eigen Values

> Write a program to compute the eigen values of a matrix.

```python
# Eigen Value Function
def eig(A):
    pass
```

```python
# You may test your code here.

```

## Answer Checking

I will be checking your answers using the code below.

```python
print(trap(a1d, b1d, c1d))
print(arc(a1d, np.pi*c1d))
print(sumdiff(a1d))
print(quad(a1d, b1d, -c1d))
print(cross(a1d, b1d))
print(eig(A, B))
```


# Least square estimation of sinusoid parameters

The signal paramters for a function
$$x = o + g t + \hat{x} \sin \left( \omega t + \phi \right)$$
for a known frequency $\omega$ at discrete from a given series of data points $x_i$ sampled at discrete time intervals $t_i$ will be estimated.

The sunusoidal function can be expressed as
$$ \hat{x} \sin \left( \omega t + \phi \right) = a \sin \left( \omega t \right) + b \cos \left( \omega t \right) $$
with
$$ \phi = \text{arctan2} \left( b, a \right) \quad \text{and} \quad \hat{x} = \sqrt{a^2 + b^2} \text{.}$$

Therefore the problem can be rewritten as
$$x = c_0 + c_1 t + c_2 \sin \left( \omega t \right) + c_3 \cos \left( \omega t \right) \text{.}$$
Only the parameters $c_i$ have to be estimate, which means the problem can be solved using a linear least square.

## Linear least square

For a sinusoidal signal described above sampled at discreet time intervals the error term is
$$e_i = c_0 + c_1 t_i + c_2 \sin \left( \omega t_i \right) + c_3 \cos \left( \omega t_i \right) - x_i \text{.}$$

For the set of measured values the error term can be expressed in matrix notation
$$\mathbf{e} = \mathbf{V} \mathbf{c} - \mathbf{x}$$
with
$$\mathbf{e} = \begin{bmatrix}
    e_1 \\
    e_2 \\
    \vdots \\
    e_n \\
\end{bmatrix} \text{,} \quad
\mathbf{V} = \begin{bmatrix}
    1 & t_1 & \sin \left( \omega t_1 \right) & \cos \left( \omega t_1 \right) \\
    1 & t_2 & \sin \left( \omega t_2 \right) & \cos \left( \omega t_2 \right) \\
    \vdots & \vdots & \vdots & \vdots \\
    1 & t_n & \sin \left( \omega t_n \right) & \cos \left( \omega t_n \right) \\
\end{bmatrix} \text{,} \quad
\mathbf{c} = \begin{bmatrix}
    c_1 \\
    c_2 \\
    c_3 \\
    c_4 \\
\end{bmatrix} \text{,} \quad
\mathbf{x} = \begin{bmatrix}
    x_1 \\
    x_2 \\
    \vdots \\
    x_n \\
\end{bmatrix} \text{.}$$

The cost function of the least square algorithm is the squared error term (${e_i}^2$). In matrix notation the error term
\begin{equation}
\begin{split}
J = \mathbf{e}^\text{T} \mathbf{e} & = \left( \mathbf{c}^\text{T} \mathbf{V}^\text{T} - \mathbf{x}^\text{T} \right) \left( \mathbf{V} \mathbf{c} - \mathbf{x} \right) \\
& = \mathbf{c}^\text{T} \mathbf{V}^\text{T} \mathbf{V} \mathbf{c} - \mathbf{c}^\text{T} \mathbf{V}^\text{T}  \mathbf{x} -  \mathbf{x}^\text{T} \mathbf{V} \mathbf{c} \\
\end{split}
\end{equation}
has to be minimized. To find the minimum of $J \left( \mathbf{c} \right)$
$$ \frac{\text{d} J}{\text{d} \mathbf{c}} \stackrel{!}{=} 0 $$
is solved for $\mathbf{c}$.

The following equation has to be solved for $\mathbf{c}$:
$$\begin{split}
& \frac{\text{d} J}{\text{d} \mathbf{c}} = 0 \\
\Leftrightarrow \ &
    \underbrace{\frac{\text{d} \left( \mathbf{c}^\text{T} \mathbf{V}^\text{T} \mathbf{V} \mathbf{c} \right)}{\text{d} \mathbf{c}}}_{= \mathbf{V}^\text{T} \mathbf{V} \mathbf{c} + \mathbf{c}^\text{T} \mathbf{V}^\text{T} \mathbf{V}} -
    \underbrace{\frac{\mathbf{c}^\text{T} \mathbf{V}^\text{T}  \mathbf{x}}{\text{d} \mathbf{c}}}_{= \mathbf{V}^\text{T}  \mathbf{x}} -
    \underbrace{\frac{\mathbf{x}^\text{T} \mathbf{V} \mathbf{c}}{\text{d} \mathbf{c}}}_{= \mathbf{x}^\text{T} \mathbf{V}}
    = 0 \\
\Leftrightarrow \ & \mathbf{V}^\text{T} \mathbf{V} \mathbf{c} + \mathbf{c}^\text{T} \mathbf{V}^\text{T} \mathbf{V} - \mathbf{V}^\text{T}  \mathbf{x} - \mathbf{x}^\text{T} \mathbf{V} = 0 \\
\Leftrightarrow \ & 2 \mathbf{V}^\text{T} \mathbf{V} \mathbf{c} - 2 \mathbf{V}^\text{T}  \mathbf{x} = 0 \\
\Leftrightarrow \ & \mathbf{V}^\text{T} \mathbf{V} \mathbf{c} = \mathbf{V}^\text{T}  \mathbf{x} \\
\Leftrightarrow \ & \mathbf{c} = \left( \mathbf{V}^\text{T} \mathbf{V} \right)^{-1} \mathbf{V}^\text{T}  \mathbf{x} \\
\end{split}$$

## Python implementation of sine parameter estimator

The following python function implements a sine parameter estimator.

The frequency ($\omega$), the signal data ($x$) and the time vector ($t$) are passed to the function and the signal parameters offset ($o$), gradiant ($g$), amplitud ($\hat{x}$) and phase ($\phi$) are returned.

```python
import numpy as np

def estimate_sine(omega, data, t):
    func_mtrx_transp = np.array([
            len(data) * [1.],
            t,
            np.sin(omega * t),
            np.cos(omega * t),
        ])
    res = np.linalg.solve(np.dot(func_mtrx_transp, func_mtrx_transp.T),
                          np.dot(func_mtrx_transp, np.array([data]).T))
    amp = np.sqrt(res[2][0]**2. + res[3][0]**2.)
    phase = np.arctan2(res[3][0], res[2][0])
    return res[0][0], res[1][0], amp, phase
```

To test the function 'estimate_sine' a test signal is created and the paramters are estimated.

```python
import matplotlib.pyplot as plt

# Signal parameters
offset     = 7.5
gradient   = 2.25
omega      = 2. * np.pi * 2.
sine_amp   = 12.25
sine_phase = np.pi * 0.2
noise_amp  = 1.5

# Create test signal
t = np.arange(0., 2.0, 0.004)
x = offset + gradient * t + sine_amp * np.sin(omega * t + sine_phase) \
    + noise_amp * 2 * (np.random.random(len(t)) - 0.5)

# Estimate signal parameters
est_offset, est_grad, est_amp, est_phase = estimate_sine(omega, x, t)
x_est = est_offset + est_grad * t + est_amp * np.sin(omega * t + est_phase)

# Print results
print("            Original Estimated     Error")
print("Offset:    {orig_offs:>9f} {est_offs:>9f} {error:>9f}".format(
        orig_offs=offset, est_offs=est_offset, error=offset-est_offset))
print("Gradient:  {orig_grad:>9f} {est_grad:>9f} {error:>9f}".format(
        orig_grad=gradient, est_grad=est_grad, error=gradient-est_grad))
print("Amplitude: {orig_amp:>9f} {est_amp:>9f} {error:>9f}".format(
        orig_amp=sine_amp, est_amp=est_amp, error=sine_amp-est_amp))
print("Phase:     {orig_phase:>9f} {est_phase:>9f} {error:>9f}".format(
        orig_phase=sine_phase, est_phase=est_phase, error=sine_phase-est_phase))

# Plot results
plt.plot(t, x, 'b', linewidth=0.5, label="Signal")
plt.plot(t, x_est, 'r', linewidth=1.0, label="Estimated Sine")
plt.xlabel("time / s")
plt.ylabel("signal")
plt.legend(bbox_to_anchor=(1.4, 1.0))
plt.grid(True)
plt.show()
```


# Horner's Method

What is the best way to evaluate

\\[ f(x) = x^3 + 4x^2 - 10 \\]

at $ x = \dfrac{1}{2} $? The traditional and direct approach is

\\[ f(\dfrac{1}{2}) = \dfrac{1}{2} * \dfrac{1}{2} * \dfrac{1}{2} + 4 * \dfrac{1}{2} * \dfrac{1}{2} - 10 \\]

This procedure takes 4 multiplications and 2 additions where a subtraction can be interpreted as adding a negative number. All together it takes 6 operations to evaluate the function. Is there a way to reduce the number of operations? Rewrite the polynomial in such a way the variable $x$ is factored out:

\begin{align}
f(x) & = -10 + 4x^2 + x^3 \\
 & = -10 + x * (0 + 4x + x^2 ) \\
 & = -10 + x * (0 + x * (4 + x))
\end{align}

As you can see, it takes a total of 5 operations to evaluate the function; 3 additions and 2 multiplications. This method is called **Horner's Method**.
The point of this notebook is to check the absolute error and relative error and to test the efficiency of using Horner's Method for numerical computations. In this case, we explore its application to find the root of the equation using Bisection Method.

### Absolute and Relative Error
To calculate the error between the two functions we need to find the absolute and the relative error where $x^*$ is the approximate function and $x$ is the true function.

**Absolute Error** $= |x^* - x| $

**Relative Error** $= \dfrac{|x^* - x|}{|x|} $

```python
def error_analysis(true_fx, approx_fx):
    absolute_error = abs(approx_fx - true_fx)
    relative_error = absolute_error / abs(true_fx)
    return absolute_error, relative_error
```

#### True Function - Naive Method

```python
def f(x):
    return (x ** 3) + (4 * (x ** 2)) - 10
```

#### Approximate Function - Horner's Method

```python
def f_a(x):
    return -10 + 𝑥 * (0 + 𝑥 * (4 + 𝑥))
```

### Bisection Method
Bisection method is a root-finding algorithm based from the intermediate-value theorem from calculus.
You need to verify if a root exist by making sure the two endpoints of the interval $ [a,b] $ or $\{f(a),f(b)\}$ have different signs. If function $ f $ is continuous, then there will be a root $r$ between $a$ and $b$ such that $f(r) = 0$ and $a < r < b$.

```python
import numpy as np
```

```python
def bisection_method(a,b):
    if f(a) * f(b) < 0:
        roots = []
        while (b - a) / 2 > 1e-7:
            c = (a + b) / 2
            if f(a) * f(c) < 0:
                b = c
            else:
                a = c
            roots.append(c)
    return np.array(roots)
```

```python
%%timeit
roots = bisection_method(1,2)
```

    24.7 µs ± 1.49 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
roots = bisection_method(1,2)
```

```python
def bisection_method_hm(a,b):
    if f_a(a) * f_a(b) < 0:
        roots = []
        while (b - a) / 2 > 1e-7:
            c = (a + b) / 2
            if f_a(a) * f_a(c) < 0:
                b = c
            else:
                a = c
            roots.append(c)
    return np.array(roots)
```

```python
%%timeit
roots_a = bisection_method_hm(1,2)
```

    20 µs ± 4.59 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
roots_a = bisection_method(1,2)
```

According to the benchmark an implementation of the Horner's Method perform faster than the naive method. Now let us look at the error analysis to find out whether results are different on a precision level.

```python
absolute_error, relative_error = error_analysis(roots, roots_a)
```

```python
absolute_error
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0.])

```python
relative_error
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0.])

Luckily errors were not found in this example, however polynomials should always be expressed in the nested form before performing an evaluation, because this form minimizes the number of arithmetic calculations, as a result, the errors are reduced. The example \begin{align}
f(x) & = -10 + 4x^2 + x^3 \\
 & = -10 + x * (0 + 4x + x^2 ) \\
 & = -10 + x * (0 + x * (4 + x))
\end{align}

has already placed the coefficients and the variables in the rightful place. Notice we placed an additional coefficient $0$ to fill-in the missing degree.

### Evaluation using Horner's Method

As an engineer we can write an algorithm to evaluate `x` given only the `degree` (or the number of terms in a polynomial) and its `coefficients`. This way we don't have to rewrite a function everytime a new polynomial is introduced.

Consider the following polynomial $ P(x) = a_{k}x^{k} + a_{k-1}x^{k-1} + a_{k-2}x^{k-2} + ... + a_{1}x + a_{0} $ we can evaluate `x` using this algorithm below.

```python
def evaluate(x, k, c, b=None):
    '''
    Evaluate `x` from a polynomial using Horner's Method.

    Parameters
    ----------
    x : int or float
            The value of x
    k : int
            degrees or the number of terms
    c : int or float
            coefficients
    b : int or float
            base points

    Returns
    -------
    y : int or float
            The output of the polynomial
    '''
    y = c[0]
    if b is None:
        for i in range(1, k):
            y = c[i] + (x * y)
    else:
        for i in range(1, k):
            y = (y * (x - b[i])) + c[i]
    return y
```

```python
%%timeit
evaluate(2, 4, [1, 4, 0, -10])
```

    684 ns ± 9.99 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

```python
%%timeit
f_a(2)
```

    160 ns ± 2.02 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

```python
%%timeit
f(2)
```

    581 ns ± 9.06 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Although Horner's method is slower in comparison with functions whose polynomials are laid out, this method still is efficient as it saves time to rewrite equations and evaluate. We shall look more into the applications of Horner's Method once we get into Interpolation.

### Reference

- R.L. Burden and J.D. Faires. *Numerical Analysis*. Brooks/Cole, Cengage Learning, Boston, 9th edition, 2010.
- Timothy Sauer. 2018. *Numerical Analysis (3rd. ed.)*. Pearson, UK.
- Justin Solomon. 2015. *Numerical Algorithms: Methods for Computer Vision, Machine Learning, and Graphics*. A. K. Peters, Ltd., USA.

```python

```


```python
import numpy as np
```

### Matrices

```python
matrix_01 = np.matrix("1, 2, 3; 4, 5, 6"); matrix_01
```

    matrix([[1, 2, 3],
            [4, 5, 6]])

```python
matrix_02 = np.matrix([[1, 2, 3], [4, 5, 6]]); matrix_02
```

    matrix([[1, 2, 3],
            [4, 5, 6]])

### Math Operations with Arrays and Matrices

```python
array_01 = np.array([[1, 2], [3, 4]]); array_01
```

    array([[1, 2],
           [3, 4]])

```python
type(array_01)
```

    numpy.ndarray

```python
array_01 * array_01
```

    array([[ 1,  4],
           [ 9, 16]])

```python
matrix_01 = np.mat(array_01); matrix_01
```

    matrix([[1, 2],
            [3, 4]])

```python
type(matrix_01)
```

    numpy.matrix

```python
matrix_01 * matrix_01
```

    matrix([[ 7, 10],
            [15, 22]])

### The multiplications results between arrays and matrices are different. The math is the following:

## $$  \boxed{ \begin{align} \begin{pmatrix} 1 & 2    \\ 3 &  4   \end{pmatrix} & \begin{pmatrix} 1 & 2    \\ 3 &  4   \end{pmatrix}  = \begin{pmatrix} 7 & 10    \\ 15 &  22   \end{pmatrix} \end{align} }$$

### A matrix apply the multiplication from lines to columns, resulting in the following operations:

```python
from IPython.display import Image
Image('aux/images/matrix-multiplication.png')
```

### To do the same operation with an array object:

```python
array_01 = np.array([[1, 2], [3, 4]]); array_01
```

    array([[1, 2],
           [3, 4]])

```python
np.dot(array_01, array_01)
```

    array([[ 7, 10],
           [15, 22]])

### Conversions

```python
array_01 = np.array([[1, 2], [3, 4]]); array_01
```

    array([[1, 2],
           [3, 4]])

```python
# Array to Matrix
matrix_01 = np.asmatrix(array_01); matrix_01
```

    matrix([[1, 2],
            [3, 4]])

```python
# Matrix to Array
array_02 = np.asarray(matrix_01); array_02
```

    array([[1, 2],
           [3, 4]])


# Functions as Real-valued Circuits

## Base Case: Single Gate in the Circuit

### Single Multiply Gate

The first thing we are going to compute is a simple function of two variables $F = f(X,Y)=X*Y$.

The circuit takes two real-valued inputs $X$ and $Y$, computes the product $X*Y$ defined by the function $f$ and stores it in the variable $F$.

```python
def f(X,Y):
    return X*Y
```

```python
F = f(-2,3)
print (f'The output F is: {F}')
```

    The output F is: -6

### *How should one tweak the inputs $(X,Y)$ slightly to increase the output of the function __f__?*

### Strategy #1 - Numerical Gradient

The partial derivative of the function $f$ with respect to $X$ can be computed as:

\begin{align}
\frac{\partial f (X,Y)}{\partial X} = \lim_{h\to 0} \frac{f(X+h,Y)-f(X,Y)}{h} \\
\end{align}

This can be simulated in code by choosing $h$ to be a very small number:

```python
h = 0.0001
```

#### Computing a partial derivative with respect to $X$

```python
X = -2; Y = 3
```

```python
X_derivative = (f(X+h, Y)-f(X, Y))/h
print (f'the derivative in respect to X is: {X_derivative}')
```

    the derivative in respect to X is: 3.00000000000189

Positive derivative value indicates that $X$ should be increased in order to increase $f$, let's try that:

```python
print (f'the original output is: {f(-2,3)}')
# increase X for a small value, for example 0.2
print (f'the new output is: {f(-1.8,3)}')
```

    the original output is: -6
    the new output is: -5.4

$-5.4$ is larger than $-6$, it works!

#### compute a partial derivative with respect to $Y$

```python
Y_derivative = (f(X, Y+h)-f(X, Y))/h
print (f'the derivative in respect to Y is: {Y_derivative}')
```

    the derivative in respect to Y is: -2.0000000000042206

Negative derivative value indicates that $Y$ should be decreased in order to increase $f$, let's try that:

```python
print (f'the original output is: {f(-2,3)}')
# decrease Y for a small value, for example 0.1
print (f'the new output is: {f(-2,2.9)}')
```

    the original output is: -6
    the new output is: -5.8

$-5.8$ is larger than $-6$, it works!

The __gradient__ of a function is made up of all the partial derivatives of this function concatenated in a vector

\begin{align}
\nabla f(X,Y)=\left[\frac{\partial f (X,Y)}{\partial X},\frac{\partial f (X,Y)}{\partial Y}\right]
\end{align}

#### Gradually minimizing a function by using derivatives

In order to gradually maximize the function $f$ (step-by-step) towards our desired result, we need to update our parameters. This is done by adding to the parameter's value the value of its partial derivative. To achieve this gradually in small steps, we multiply the value of the partial derivative by a a small number (step).

```python
step_size = 0.01
F = f(X,Y)
print (f'The original output F is: {F}')
```

    The original output F is: -6

```python
X = X + step_size * X_derivative
Y = Y + step_size * Y_derivative
print (f'X is: {X}, Y is: {Y}')
```

    X is: -1.969999999999981, Y is: 2.979999999999958

```python
F_new = f(X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -6
    new output: -5.87059999999986

The new output is larger than the old, thanks to partial derivatives.

This approach, however, is still expensive because we need to compute the circuit’s output as we tweak every input value independently a small amount.

### Strategy #2 - Analytic Gradient

We can use calculus to compute partial derivatives of the function $f(X,Y)$.

\begin{align}
\frac{\partial F}{\partial X}=\frac{\partial f (X,Y)}{\partial X} = \lim_{h\to 0} \frac{f(X+h,Y)-f(X,Y)}{h} \\
\end{align}

A partial derivative of $F=f(X,Y)$ in respect to $X$ is:

\begin{align*}
\frac{\partial F}{\partial X}=\frac{\partial f (X,Y)}{\partial X} &= \lim_{h\to 0} \frac{f(X+h,Y)-f(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+h)Y -XY}{h} \\
&=\lim_{h\to 0}\frac{XY+Yh-XY}{h} \\
&=\lim_{h\to 0}\frac{Yh}{h} \\
\frac{\partial F}{\partial X}=\frac{\partial f (X,Y)}{\partial X}&=Y
\end{align*}

A partial derivative of $F=f(X,Y)$ in respect to $Y$ is:

\begin{align*}
\frac{\partial F}{\partial Y}=\frac{\partial f (X,Y)}{\partial Y} &= \lim_{h\to 0} \frac{f(X,Y+h)-f(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{X(Y+h) -XY}{h} \\
&=\lim_{h\to 0}\frac{XY+Xh-XY}{h} \\
&=\lim_{h\to 0}\frac{Xh}{h} \\
\frac{\partial F}{\partial Y}=\frac{\partial f (X,Y)}{\partial Y} &=X
\end{align*}

Here are both partial derivatives:

\begin{align*}
\frac{\partial F}{\partial X}=Y ; \frac{\partial F}{\partial Y}=X \\
\end{align*}

We can represent this as a gradient:

\begin{align}
\nabla f(X,Y)=\left[Y,X\right]
\end{align}

We can now use this information to increase the output of the function __f__:

```python
X = -2; Y = 3
F = f(X,Y)
print (f'the output F is: {F}')
```

    the output F is: -6

```python
X_gradient = Y
Y_gradient = X
print (f'X-gradient: {X_gradient} \nY-gradient: {Y_gradient}')
```

    X-gradient: 3
    Y-gradient: -2

```python
step_size = 0.001
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
print (f'X is now: {X}, \nY is now: {Y}')
```

    X is now: -1.997,
    Y is now: 2.998

```python
F_new = f(X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -6
    new output: -5.987006000000001

The new output $-5.8706$ is larger than the old: $-6$.

***

### Single Add Gate

The second function we're going to compute is  $G=g(X,Y)=X+Y.$

The circuit takes two real-valued inputs $X$ and $Y$ and computes the sum $X+Y$.

```python
def g(X,Y):
    return X+Y
```

```python
X = -2; Y = 3
G = g(X,Y)
print (f'The output is: {G}')
```

    The output is: 1

As we did before, we can use calculus to compute partial derivatives for the function $G=g(X,Y)$:

\begin{align}
\frac{\partial G}{\partial X}=\frac{\partial g (X,Y)}{\partial X} = \lim_{h\to 0} \frac{g(X+h,Y)-g(X,Y)}{h} \\
\end{align}

A partial derivative of $G=g(X,Y)$ in respect to $X$ is:

\begin{align*}
\frac{\partial G}{\partial X}=\frac{\partial g (X,Y)}{\partial X} &= \lim_{h\to 0} \frac{g(X+h,Y)-g(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{X+h+Y -X-Y}{h} \\
\frac{\partial G}{\partial X}=\frac{\partial g (X,Y)}{\partial X}&=\lim_{h\to 0}\frac{h}{h} =1 \\
\end{align*}

A partial derivative of $G=g(X,Y)$ in respect to $Y$ is:

\begin{align*}
\frac{\partial G}{\partial Y}=\frac{\partial g (X,Y)}{\partial Y} &= \lim_{h\to 0} \frac{g(X,Y+h)-g(X,Y)}{h} \\\\
&=\lim_{h\to 0}\frac{X+Y+h -X-Y}{h} \\
\frac{\partial G}{\partial Y}=\frac{\partial g (X,Y)}{\partial Y}&=\lim_{h\to 0}\frac{h}{h} =1 \\
\end{align*}

Both partial derivatives $\frac{\partial G}{\partial X}$ and $\frac{\partial G}{\partial Y}$ in this case are equal to $1$.

We can use this information to maximize the function __g__:

```python
X_gradient = 1
Y_gradient = 1
print (f'X-gradient: {X_gradient} \nY-gradient: {Y_gradient}')
```

    X-gradient: 1
    Y-gradient: 1

```python
step_size = 0.01
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
print (f'X is: {X} \nY is: {Y}')
```

    X is: -1.99
    Y is: 3.01

```python
F_new = g(X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -6
    new output: 1.0199999999999998

## Recursive Case: Circuits with Multiple Gates

The expression we are computing now is $M = m(X,Y,Z)=(X+Y)*Z$.

```python
def m(X,Y,Z):
    return (X+Y)*Z
```

```python
X = -2; Y = 5; Z = -4
```

this is equal to $M=m(-2,5,-4)=(-2+5)-4=3*-4=-12$

```python
M = m(X,Y,Z)
print (f'the output M is: {M}')
```

    the output M is: -12

As we did before, we can use calculus to derive partial derivatives for the function $M=m(X,Y,Z)$.

\begin{align}
\frac{\partial M}{\partial X}=\frac{\partial m (X,Y,Z)}{\partial X} = \lim_{h\to 0} \frac{m(X+h,Y,Z)-m(X,Y,Z)}{h} \\
\end{align}

A partial derivative of $M=m(X,Y,Z)$ in respect to $X$ is:

\begin{align*}
\frac{\partial M}{\partial X}=\frac{\partial m(X,Y,Z)}{\partial X} &= \lim_{h\to 0} \frac{f(X+h,Y,Z)-f(X,Y,Z)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+h+Y)*Z -(X+Y)*Z}{h} \\
&=\lim_{h\to 0}\frac{ZX+Zh+ZY-ZX-ZY}{h} \\
\frac{\partial M}{\partial X}=\frac{\partial m(X,Y,Z)}{\partial X}&=\lim_{h\to 0}\frac{Zh}{h} =Z \\
\end{align*}

Similarly, partial derivative of $M=m(X,Y,Z)$ in respect to $Y$ is:

\begin{align*}
\frac{\partial M}{\partial Y}=\frac{\partial m(X,Y,Z)}{\partial Y} &= \lim_{h\to 0} \frac{f(X,Y+h,Z)-f(X,Y,Z)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+Y+h)*Z -(X+Y)*Z}{h} \\
&=\lim_{h\to 0}\frac{ZX+ZY+Zh-ZX-ZY}{h} \\
\frac{\partial M}{\partial Y}=\frac{\partial m(X,Y,Z)}{\partial Y}&=\lim_{h\to 0}\frac{Zh}{h} =Z \\
\end{align*}

A partial derivative of $M=m(X,Y,Z)$ in respect to $Z$ is:

\begin{align*}
\frac{\partial M}{\partial Z}=\frac{\partial m(X,Y,Z)}{\partial Z} &= \lim_{h\to 0} \frac{f(X,Y,Z+h)-f(X,Y,Z)}{h} \\\\
&=\lim_{h\to 0}\frac{(X+Y)*(Z+h) -(X+Y)*Z}{h} \\
&=\lim_{h\to 0}\frac{XZ+Xh+YZ+Yh-XZ-YZ}{h} \\
&=\lim_{h\to 0}\frac{Xh+Yh}{h} \\
&=\lim_{h\to 0}\frac{h(X+Y)}{h}\\
\frac{\partial M}{\partial Z}=\frac{\partial m(X,Y,Z)}{\partial Z}&=X+Y \\
\end{align*}

Here are all three partial derivatives:

\begin{align*}
\frac{\partial M}{\partial X}=Z ; \frac{\partial M}{\partial Y}=Z ;\frac{\partial M}{\partial Z}=X+Y \\
\end{align*}

We can represent this as a gradient:

\begin{align}
\nabla m(X,Y,Z)=\left[Z,Z,X+Y\right]
\end{align}

We can now use this information to maximize the output of the function __m__:

```python
X_gradient = Z
Y_gradient = Z
Z_gradient = X+Y
print (f'X-gradient: {X_gradient} \nY-gradient: {Y_gradient} \nZ-gradient: {Z_gradient}')
```

    X-gradient: -4
    Y-gradient: -4
    Z-gradient: 3

```python
step_size = 0.01
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
Z = Z + step_size * Z_gradient
print (f'X is: {X} \nY is: {Y} \nZ is: {Z}')
```

    X is: -2.04
    Y is: 4.96
    Z is: -3.97

```python
M_new = m(X,Y,Z)
print (f'old output: {M}\nnew output: {M_new}')
```

    old output: -12
    new output: -11.5924

***

### Backpropagation

Instead of working with the function $m(X,Y,Z)=(X+Y)*Z$, we can simplify the computation by composing two new simpler functions:
$G=g(X,Y)=X+Y$, and
$F=f(G,Z)=G*Z$, into:
$F=f(g(X,Y),Z)$

Here, we can apply the chain rule for derivation, because $F$ is a function of $G$ and $G$ is a function of $X$ and $Y$.

So instead of computing $\frac{\partial M}{\partial X}$, $\frac{\partial M}{\partial Y}$  and $\frac{\partial M}{\partial Z}$ which gets more complicated to compute with more complex expressions, we compute instead $\frac{\partial F}{\partial X}$, $\frac{\partial F}{\partial Y}$ and $\frac{\partial F}{\partial Z}$, which can be decomposed:

\begin{align*}
\frac{\partial F}{\partial X}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial X}
\end{align*}

Here $\frac{\partial F}{\partial G}$ is a simple multiplication gate, whose derivate we have already computed:$\frac{\partial F}{\partial G}$ =$\frac{\partial f(G,Z)}{\partial G}=Z$

Also $\frac{\partial G}{\partial X}$ is a simple addition gate, whose derivate we have already computed: $\frac{\partial G}{\partial X}$ =$\frac{\partial g(X,Y)}{\partial X}=1$, thus:

\begin{align*}
\frac{\partial F}{\partial X}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial X} = Z*1 = Z
\end{align*}

***

The same applies when computing the partial $\frac{\partial F}{\partial Y}$:

\begin{align*}
\frac{\partial F}{\partial Y}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial Y}
\end{align*}

\begin{align*}
\frac{\partial F}{\partial Y}=\frac{\partial F}{\partial G}\frac{\partial G}{\partial Y} = Z*1 = Z
\end{align*}

***

The partial derivative $\frac{\partial F}{\partial Z}$ does not need to be decomposed:

\begin{align*}
\frac{\partial F}{\partial Z}=G=X+Y
\end{align*}

We can now use this information to maximize the output of the function __m_decomposed__:

```python
def m_decomposed(X,Y,Z):
    G = g(X,Y)
    F = f(G,Z)
    return F
```

```python
X = -2; Y = 5; Z = -4
```

```python
F = m_decomposed(X,Y,Z)
print (f'the output is: {F}')
```

    the output is: -12

```python
X_gradient = Z
Y_gradient = Z
Z_gradient = X+Y
print (f'X-gradient: {X_gradient}, \nY-gradient: {Y_gradient}, \nZ-gradient: {Z_gradient}.')
```

    X-gradient: -4,
    Y-gradient: -4,
    Z-gradient: 3.

Here, we can observe, that in the backward pass, the multiplication gate switches the values of outputs: what used to be (3,-4) in the forward pass, it becomes (-4,3) in the backward pass. The addition gate, on the other hand just passes its input value to the ouput without changing it.

```python
step_size = 0.01
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
Z = Z + step_size * Z_gradient
print (f'X is: {X}\nY is: {Y}\nZ is: {Z}')
```

    X is: -2.04
    Y is: 4.96
    Z is: -3.97

```python
F_new = m(X,Y,Z)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: -12
    new output: -11.5924

***

### Example: More complex functions

Here is a seemingly complicated function:

\begin{align}
l(A,B,C,X,Y)&= \frac{1}{1+e^{-(AX+BY+C)}}\\
&or \\
l(A,B,C,X,Y)&= \sigma (AX+BY+C)\\
\end{align}

The function $\sigma$ is called a *sigmoid function*:

\begin{align}
\sigma&= \frac{1}{1+e^{-x}}\\
\end{align}

and it was used a lot in machine learning before.

The derivative of the sigmoid function is:

\begin{align}
\frac{d\sigma(x)}{dx}= \sigma(x) * (1-\sigma(x))
\end{align}

which means that once we compute the final activation $F=\sigma(AX+BY+C)$, we can simply calculate the derivative as $F*(1-F)$.

We can compute the forward pass of this function without a problem, however, directly computing the partial derivatives $\frac{\partial L}{\partial A}$, $\frac{\partial L}{\partial B}$, $\frac{\partial L}{\partial C}$, ... could be tricky. It is much better to use the chain rule and compose multiple functions togeher.

We can create 4 simple functions:

\begin{align*}
G=g(A,X)&=A*X \\
H=h(B,Y)&=B*Y \\
K=k(G,H,C)&=G+H+C \\
F=f(K)&= \frac{1}{1+e^{-x}}\\
\end{align*}

and compose them as:

\begin{align*}
F=f(k(g(A,X),h(B,Y),C))
\end{align*}

This looks much simpler on a diagram:

Let's compute all the partial derivatives by using the chain rule:

\begin{align*}
\frac{\partial F}{\partial A}&=\frac{\partial F}{\partial K}*\frac{\partial K}{\partial G}*\frac{\partial G}{\partial A} \\
\frac{\partial F}{\partial A}&= F(1-F)*1*X \\
\frac{\partial F}{\partial A}&= XF(1-F)
\end{align*}

\begin{align*}
\frac{\partial F}{\partial X}&=\frac{\partial F}{\partial K}*\frac{\partial K}{\partial G}*\frac{\partial G}{\partial X} \\
\frac{\partial F}{\partial X}&= F(1-F)*1*A \\
\frac{\partial F}{\partial X}&= AF(1-F)
\end{align*}

By following the exact same procedure for $B$, $Y$, and $C$ we get:

\begin{align*}
\frac{\partial F}{\partial B}&=YF(1-F)\\
\end{align*}

\begin{align*}
\frac{\partial F}{\partial Y}&=BF(1-F)\\
\end{align*}

\begin{align*}
\frac{\partial F}{\partial C}&=F(1-F)\\
\end{align*}

We can represent this as a gradient:

\begin{align}
\nabla l(A,B,C,X,Y)=\left[XF(1-F),YF(1-F),F(1-F),AF(1-F),BF(1-F) \right]
\end{align}

We can now use this information to maximize the output of the function __l__:

```python
import numpy as np
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

```python
def l(A,B,C,X,Y):
    G = f(A,X)
    H = f(B,Y)
    K = G + H + C
    F = sigmoid(K)
    return F
```

```python
A = 1.0; B = 2.0; C = -3.0; X = -1.0; Y = 3.0
```

```python
F = l(A,B,C,X,Y)
print (f'the output F is: {F}')
```

    the output F is: 0.8807970779778823

Since every partial derivative involves $F(1-F)$, we will compute it first as `F_K`

```python
gradient_end = 1
F_K = (F * (1 - F)) * gradient_end
print (f'F_K = {F_K}')
```

    F_K = 0.10499358540350662

```python
A_gradient = X*F_K
B_gradient = Y*F_K
C_gradient = F_K
X_gradient = A*F_K
Y_gradient = B*F_K

print (f'A-gradient: {A_gradient} \nX-gradient: {X_gradient} \nB-gradient: {B_gradient} \nY-gradient: {Y_gradient}\nC-gradient: {C_gradient}')
```

    A-gradient: -0.10499358540350662
    X-gradient: 0.10499358540350662
    B-gradient: 0.31498075621051985
    Y-gradient: 0.20998717080701323
    C-gradient: 0.10499358540350662

```python
step_size = 0.01
A = A + step_size * A_gradient
B = B + step_size * B_gradient
C = C + step_size * C_gradient
X = X + step_size * X_gradient
Y = Y + step_size * Y_gradient
print (f'A is: {A}, \nB is: {B}, \nC is: {C}, \nX is: {X}, \nY is: {Y}')
```

    A is: 0.998950064145965,
    B is: 2.0031498075621053,
    C is: -2.9989500641459648,
    X is: -0.998950064145965,
    Y is: 3.00209987170807

```python
F_new = l(A,B,C,X,Y)
print (f'old output: {F}\nnew output: {F_new}')
```

    old output: 0.8807970779778823
    new output: 0.8825501816218984

The new output is higher than the old one!

04. Functions as Real Valued Circuits.ipynb


```python
from sympy import symbols, diff
# initialize x and y to be symbols to use in a function
x, y = symbols('x y', real=True)
f = (x**2)/y
# Find the partial derivatives of x and y
fx = diff(f, x, evaluate=True)
fy = diff(f, y, evaluate=True)
print(fx)
print(fy)
# print(f.evalf(subs={x: 2, y: 1}))
print(fx.evalf(subs={x: 2, y: 1}))
print(fy.evalf(subs={x: 2, y: 1}))
```

```python

def f(x, y):
    return x**2/y

eps = 1e-6
x = 2
y = 1
print((f(x + eps, y) - f(x, y)) / eps)
print((f(x, y + eps) - f(x, y)) / eps)
```

```python

```


# First Steps

- [DifferentialEquations.jl docs](https://diffeq.sciml.ai/dev/index.html)

## Building an ODE model

How to define your model, state variables, initial (and/or boundary) conditions, and parameters.

As a simple example, the concentration of a decaying nuclear isotope could be described as an exponential decay:

$$
\frac{d}{dt}C(t) = - \lambda C(t)
$$

**State variable(s)**
- $C(t)$: The concentration of a decaying nuclear isotope.

**Parameter(s)**
- $\lambda$: The rate constant of decay. The half-life $t_{\frac{1}{2}} = \frac{ln2}{\lambda}$

For our ODE model to be compatible to the `DifferentialEquations.jl` ecosystem, the function for the right hand side should follow one of the following two formats:

- Out-of-place form: `f(u, p, t)` where `u` is the state variable(s), `p` is the parameter(s), and `t` is the independent variable (usually time). The output is the right hand side (RHS) of the differential equation system.
- In-place form: `f!(du, u, p, t)`, where the output is saved to `du`. The rest is the same as the out of place form. The in-place form has potential performance benefits since it allocates less arrays than the out-of-place form.

```julia
using DifferentialEquations

# The Exponential decay ODE model
expdecay(u, p, t) = p * u

p = -1.0 # Parameter
u0 = 1.0 # Initial condition
tspan = (0.0, 2.0) # Simulation start and end time points
prob = ODEProblem(expdecay, u0, tspan, p) # Define the problem
sol = solve(prob) # Solve the problem
```

We can then use `Plots.jl` to visualized the solution. `DifferentialEquations.jl` has defined a plot recipe so that one can call `plot(sol)` directly.

```julia
using Plots
plot(sol) # Visualize the solution
```

## The SIR model

A more complicated example is the [SIR model](https://www.maa.org/press/periodicals/loci/joma/the-sir-model-for-spread-of-disease-the-differential-equation-model) describing infectious disease spreading. There are more state variables and parameters.

$$
\begin{align}
\frac{d}{dt}S(t) &= - \beta S(t)I(t)  \\
\frac{d}{dt}I(t) &= \beta S(t)I(t)  - \gamma I(t)  \\
\frac{d}{dt}R(t) &= \gamma I(t)
\end{align}
$$

**State variable(s)**

- $S(t)$ : the fraction of susceptible people
- $I(t)$ : the fraction of infectious people
- $R(t)$ : the fraction of recovered (or removed) people

**Parameter(s)**

- $\beta$ : the rate of infection when susceptible and infectious people meet
- $\gamma$ : the rate of recovery of infectious people

```julia
using DifferentialEquations
using Plots

# SIR model
# In-place form
function sir!(du, u, p ,t)
	s, i, r = u
	β, γ = p
	v1 = β * s * i
	v2 = γ * i
    du[1] = -v1
    du[2] = v1 - v2
    du[3] = v2
	return nothing
end

# Parameters of the SIR model
p = (β = 1.0, γ = 0.3)
u0 = [0.99, 0.01, 0.00]  # s, i, r
tspan = (0.0, 20.0)

# Define a problem
prob = ODEProblem(sir!, u0, tspan, p)

# Solve the problem
sol = solve(prob)

# Visualize the solution
plot(sol, label=["S" "I" "R"], legend=:right)
```