### Example 2: Nonlinear convection in 2D

Following the initial convection tutorial with a single state variable $u$, we will now look at non-linear convection (step 6 in the original). This brings one new crucial challenge: computing a pair of coupled equations and thus updating two time-dependent variables $u$ and $v$.

The full set of coupled equations is now

\begin{aligned}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = 0 \\
\\
\frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = 0\\
\end{aligned}

and rearranging the discretized version gives us an expression for the update of both variables

\begin{aligned}
u_{i,j}^{n+1} &= u_{i,j}^n - u_{i,j}^n \frac{\Delta t}{\Delta x} (u_{i,j}^n-u_{i-1,j}^n) - v_{i,j}^n \frac{\Delta t}{\Delta y} (u_{i,j}^n-u_{i,j-1}^n) \\
\\
v_{i,j}^{n+1} &= v_{i,j}^n - u_{i,j}^n \frac{\Delta t}{\Delta x} (v_{i,j}^n-v_{i-1,j}^n) - v_{i,j}^n \frac{\Delta t}{\Delta y} (v_{i,j}^n-v_{i,j-1}^n)
\end{aligned}

So, for starters we will re-create the original example run in pure NumPy array notation, before demonstrating
the Devito version. Let's start again with some utilities and parameters:

```python
from examples.cfd import plot_field, init_hat
import numpy as np
import sympy
%matplotlib inline

# Some variable declarations
nx = 101
ny = 101
nt = 80
c = 1.
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .2
dt = sigma * dx
```

Let's re-create the initial setup with a 2D "hat function", but this time for two state variables.

```python
#NBVAL_IGNORE_OUTPUT

# Allocate fields and assign initial conditions
u = np.empty((nx, ny))
v = np.empty((nx, ny))

init_hat(field=u, dx=dx, dy=dy, value=2.)
init_hat(field=v, dx=dx, dy=dy, value=2.)

plot_field(u)
```

Now we can create the two stencil expression for our two coupled equations according to the discretized equation above. We again use some simple Dirichlet boundary conditions to keep the values on all sides constant.

```python
#NBVAL_IGNORE_OUTPUT
for n in range(nt + 1): ##loop across number of time steps
    un = u.copy()
    vn = v.copy()
    u[1:, 1:] = (un[1:, 1:] -
                 (un[1:, 1:] * c * dt / dy * (un[1:, 1:] - un[1:, :-1])) -
                  vn[1:, 1:] * c * dt / dx * (un[1:, 1:] - un[:-1, 1:]))
    v[1:, 1:] = (vn[1:, 1:] -
                 (un[1:, 1:] * c * dt / dy * (vn[1:, 1:] - vn[1:, :-1])) -
                 vn[1:, 1:] * c * dt / dx * (vn[1:, 1:] - vn[:-1, 1:]))

    u[0, :] = 1
    u[-1, :] = 1
    u[:, 0] = 1
    u[:, -1] = 1

    v[0, :] = 1
    v[-1, :] = 1
    v[:, 0] = 1
    v[:, -1] = 1

plot_field(u)
```

Excellent, we again get a wave that resembles the one from the oiginal examples.

Now we can set up our coupled problem in Devito. Let's start by creating two initial state variables $u$ and $v$, as before, and initialising them with our "hat function.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Grid, TimeFunction

# First we need two time-dependent data fields, both initialized with the hat function
grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)

v = TimeFunction(name='v', grid=grid)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

plot_field(u.data[0])
```

Using the two `TimeFunction` objects we can again derive our discretized equation, rearrange for the forward stencil point in time and define our variable update expression - only we have to do everything twice now! We again use forward differences for time via `u.dt` and backward differences in space via `u.dxl` and `u.dyl` to match the original tutorial.

```python
from devito import Eq, solve

eq_u = Eq(u.dt + u*u.dxl + v*u.dyl)
eq_v = Eq(v.dt + u*v.dxl + v*v.dyl)

# We can use the same SymPy trick to generate two
# stencil expressions, one for each field update.
stencil_u = solve(eq_u, u.forward)
stencil_v = solve(eq_v, v.forward)
update_u = Eq(u.forward, stencil_u, subdomain=grid.interior)
update_v = Eq(v.forward, stencil_v, subdomain=grid.interior)

print("U update:\n%s\n" % update_u)
print("V update:\n%s\n" % update_v)
```

    U update:
    Eq(u(t + dt, x, y), dt*(-u(t, x, y)*Derivative(u(t, x, y), x) - v(t, x, y)*Derivative(u(t, x, y), y) + u(t, x, y)/dt))

    V update:
    Eq(v(t + dt, x, y), dt*(-u(t, x, y)*Derivative(v(t, x, y), x) - v(t, x, y)*Derivative(v(t, x, y), y) + v(t, x, y)/dt))

We then set Dirichlet boundary conditions at all sides of the domain to $1$.

```python
x, y = grid.dimensions
t = grid.stepping_dim
bc_u = [Eq(u[t+1, 0, y], 1.)]  # left
bc_u += [Eq(u[t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(u[t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(u[t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(v[t+1, 0, y], 1.)]  # left
bc_v += [Eq(v[t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(v[t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(v[t+1, x, 0], 1.)]  # bottom
```

And finally we can put it all together to build an operator and solve our coupled problem.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our data field and ICs
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

op = Operator([update_u, update_v] + bc_u + bc_v)
op(time=nt, dt=dt)

plot_field(u.data[0])
```

Excellent, we have now a scalar implementation of a convection problem, but this can be written as a single vectorial equation:

$\frac{d U}{dt} + \nabla(U)U = 0$

Let's now use devito vectorial utilities and implement the vectorial equation

```python
from devito import VectorTimeFunction, grad

U = VectorTimeFunction(name='U', grid=grid)
init_hat(field=U[0].data[0], dx=dx, dy=dy, value=2.)
init_hat(field=U[1].data[0], dx=dx, dy=dy, value=2.)

plot_field(U[1].data[0])

eq_u = Eq(U.dt + grad(U)*U)
```

We now have a vectorial equation. Unlike in the previous case, we do not need to play with left/right derivatives
as the automated staggering of the vectorial function takes care of this.

```python
eq_u
```

$\displaystyle \left[\begin{matrix}\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \frac{\partial}{\partial t} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}\\\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\partial}{\partial t} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}\end{matrix}\right] = 0$

Then we set the nboundary conditions

```python
x, y = grid.dimensions
t = grid.stepping_dim
bc_u = [Eq(U[0][t+1, 0, y], 1.)]  # left
bc_u += [Eq(U[0][t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(U[0][t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(U[0][t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(U[1][t+1, 0, y], 1.)]  # left
bc_v += [Eq(U[1][t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(U[1][t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(U[1][t+1, x, 0], 1.)]  # bottom
```

```python
# We can use the same SymPy trick to generate two
# stencil expressions, one for each field update.
stencil_U = solve(eq_u, U.forward)
update_U = Eq(U.forward, stencil_U, subdomain=grid.interior)
```

And we have the updated (stencil) as a vectorial equation once again

```python
update_U
```

$\displaystyle \left[\begin{matrix}\operatorname{U_{x}}{\left(t + dt,x + \frac{h_{x}}{2},y \right)}\\\operatorname{U_{y}}{\left(t + dt,x,y + \frac{h_{y}}{2} \right)}\end{matrix}\right] = \left[\begin{matrix}dt \left(- \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} - \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \frac{\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}}{dt}\right)\\dt \left(- \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} - \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}}{dt}\right)\end{matrix}\right]$

We finally run the operator

```python
#NBVAL_IGNORE_OUTPUT
op = Operator([update_U] + bc_u + bc_v)
op(time=nt, dt=dt)

# The result is indeed the expected one.
plot_field(U[0].data[0])
```

```python
from devito import norm
assert np.isclose(norm(u), norm(U[0]), rtol=1e-5, atol=0)
```


**Estimation Using Maximum Likelihood (MLE)** is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.
https://en.wikipedia.org/wiki/Maximum_likelihood_estimation

```python
import numpy as np
import scipy.stats
import sympy
import math
```

```python
# sample count
N = 1000
x = sympy.symbols('x', positive= True)
```

```python
def parameterEstimate(p_dist, df, param_true,v):
  print("True parameter value:", param_true)
  # Experiment and samples
  samples = p_dist.rvs(N)
  #likelihood function
  L = np.product([df.subs(x,sample) for sample in samples])
  print("Likelihood Function:", L)
  # solve for Likelihood
  #Note: log of the likelihood fincton makes the maximization problem tractable.

  Log_L = sympy.expand_log(sympy.log(L))
  param_est, = sympy.solve(sympy.diff(Log_L, v),v)

  print("Estimated parameter value:", float(param_est))
```

```python
print("Bernoulli Distribution")
p_true = 0.7
p_dist = scipy.stats.bernoulli(p_true)
# variables using sympy
p = sympy.symbols('p', positive= True)
#probability function
df = p**x * (1-p)**(1-x)
parameterEstimate(p_dist, df, p_true,p)
```

    Bernoulli Distribution
    True parameter value: 0.7
    Likelihood Function: p**741*(1 - p)**259
    Estimated parameter value: 0.741

```python
print("Exponential Distribution")
p_dist = scipy.stats.expon()
p_true = p_dist.mean()
# variables using sympy
t = sympy.symbols('t', positive= True)

#probability function
df = (1/t) *  math.e ** (-x/t)

parameterEstimate(p_dist, df, p_true,t)

```

    Exponential Distribution
    True parameter value: 1.0
    Likelihood Function: 2.71828182845905**(-1022.71172198597/t)/t**1000
    Estimated parameter value: 1.02271172198597

```python
print("Normal Distribution")
mu_true = 2
sigma_true=0
p_dist = scipy.stats.norm(mu_true,sigma_true)
# variables using sympy
mu, sigma = sympy.symbols('mu, sigma', positive= True)

#probability function
df =  ((2 * math.pi* sigma*sigma) ** (-0.5)) * (math.e ** (-0.5 * (x-mu)* (x-mu)/(sigma * sigma)))
parameterEstimate(p_dist, df, mu_true,mu)

```

    Normal Distribution
    True parameter value: 2
    Likelihood Function: 8.12953716728196e-400*2.71828182845905**(1000*(2.0 - mu)*(0.5*mu - 1.0)/sigma**2)*sigma**(-1000.0)
    Estimated parameter value: 2.0

```python
print("Poisson Distribution")
k_true= 2
p_dist = scipy.stats.poisson(k_true)
k = sympy.symbols('k', positive= True)
df = k**x*sympy.exp(-k)/sympy.factorial(x)
parameterEstimate(p_dist, df, k_true,k)

```

    Poisson Distribution
    True parameter value: 2
    Likelihood Function: k**1997*exp(-1000*k)/142220199860881056663191396650367732478138060843772563671397683466569187541722759507915845631107969632545273741424688952787500633027625617630113037289315761090010517971364415451864150528618267221053183525752106609291883404681346679086957134357390537918919070304662927052260585891725831988996973817758666931146942020746209214900436818491150673157476389615461350353676686715592963879195841207050750154875803762161535927975936000000000000000000000000000000000000000000000
    Estimated parameter value: 1.997


## Chemical kinetics
In chemistry one is often interested in how fast a chemical process proceeds. Chemical reactions (when viewed as single events on a molecular scale) are probabilitic. However, most reactive systems of interest involve very large numbers of molecules (a few grams of a simple substance containts on the order of $10^{23}$ molecules. The sheer number allows us to describe this inherently stochastic process deterministically.

### Law of mass action
In order to describe chemical reactions as as system of ODEs in terms of concentrations ($c_i$) and time ($t$), one can use the [law of mass action](https://en.wikipedia.org/wiki/Law_of_mass_action):

$$
\frac{dc_i}{dt} = \sum_j S_{ij} r_j
$$
where $r_j$ is given by:
$$
r_j = k_j\prod_l c_l^{R_{jl}}
$$

and $S$ is a matrix with the overall net stoichiometric coefficients (positive for net production, negative for net consumption), and $R$ is a matrix with the multiplicities of each reactant for each equation.

### Example: Nitrosylbromide
We will now look at the following (bi-directional) chemical reaction:

$$
\mathrm{2\,NO + Br_2 \leftrightarrow 2\,NOBr}
$$

which describes the equilibrium between nitrogen monoxide (NO) and bromine (Br$_2$) and nitrosyl bromide (NOBr). It can be represented as a set of two uni-directional reactions (**f**orward and **b**ackward):

$$
\mathrm{2\,NO + Br_2 \overset{k_f}{\rightarrow} 2\,NOBr} \\
\mathrm{2\,NOBr \overset{k_b}{\rightarrow} 2\,NO + Br_2}
$$

The law of mass action tells us that the rate of the first process (forward) is proportional to the concentration Br$_2$ and the square of the concentration of NO. The rate of the second reaction (the backward process) is in analogy proportional to the square of the concentration of NOBr. Using the proportionality constants $k_f$ and $k_b$ we can formulate our system of nonlinear ordinary differential equations as follows:

$$
\frac{dc_1}{dt} = 2(k_b c_3^2 - k_f c_2 c_1^2) \\
\frac{dc_2}{dt} = k_b c_3^2 - k_f c_2 c_1^2 \\
\frac{dc_3}{dt} = 2(k_f c_2 c_1^2 - k_b c_3^2)
$$

where we have denoted the concentration of NO, Br$_2$, NOBr with $c_1,\ c_2,\ c_3$ respectively.

This ODE system corresponds to the following two matrices:

$$
S = \begin{bmatrix}
-2 & 2 \\
-1 & 1 \\
2 & -2
\end{bmatrix}
$$

$$
R = \begin{bmatrix}
2 & 1 & 0 \\
0 & 0 & 2
\end{bmatrix}
$$

### Solving the initial value problem numerically
We will now integrate this system of ordinary differential equations numerically as an initial value problem (IVP) using the ``odeint`` solver provided by ``scipy``:

```python
import numpy as np
from scipy.integrate import odeint
```

By looking at the [documentation](https://docs.scipy.org/doc/scipy-0.19.0/reference/generated/scipy.integrate.odeint.html) of odeint we see that we need to provide a function which computes a vector of derivatives ($\dot{\mathbf{y}} = [\frac{dy_1}{dt}, \frac{dy_2}{dt}, \frac{dy_3}{dt}]$). The expected signature of this function is:

    f(y: array[float64], t: float64, *args: arbitrary constants) -> dydt: array[float64]

in our case we can write it as:

```python
def rhs(y, t, kf, kb):
    rf = kf * y[0]**2 * y[1]
    rb = kb * y[2]**2
    return [2*(rb - rf), rb - rf, 2*(rf - rb)]
```

```python
%load_ext scipy2017codegen.exercise
```

Replace **???** by the proper arguments for ``odeint``, you can write ``odeint?`` to read its documentaiton.

```python
%exercise exercise_odeint.py
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.plot(tout, yout)
_ = plt.legend(['NO', 'Br$_2$', 'NOBr'])
```

Writing the ``rhs`` function by hand for larger reaction systems quickly becomes tedious. Ideally we would like to construct it from a symbolic representation (having a symbolic representation of the problem opens up many possibilities as we will soon see). But at the same time, we need the ``rhs`` function to be fast. Which means that we want to produce a fast function from our symbolic representation. Generating a function from our symbolic representation is achieved through *code generation*.

In summary we will need to:

1. Construct a symbolic representation from some domain specific representation using SymPy.
2. Have SymPy generate a function with an appropriate signature (or multiple thereof), which we pass on to the solver.

We will achieve (1) by using SymPy symbols (and functions if needed). For (2) we will use a function in SymPy called ``lambdify``―it takes a symbolic expressions and returns a function. In a later notebook, we will look at (1), for now we will just use ``rhs`` which we've already written:

```python
import sympy as sym
sym.init_printing()
```

```python
y, k = sym.symbols('y:3'), sym.symbols('kf kb')
ydot = rhs(y, None, *k)
ydot
```

## Exercise
Now assume that we had constructed ``ydot`` above by applying the more general law of mass action, instead of hard-coding the rate expressions in ``rhs``. Then we could have created a function corresponding to ``rhs`` using ``lambdify``:

```python
%exercise exercise_lambdify.py
```

```python
plt.plot(tout, odeint(f, y0, tout, k_vals))
_ = plt.legend(['NO', 'Br$_2$', 'NOBr'])
```

In this example the gains of using a symbolic representation are arguably limited. However, it is quite common that the numerical solver will need another function which calculates the [Jacobian](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant) of $\dot{\mathbf{y}}$ (given as Dfun in the case of ``odeint``). Writing that by hand is both tedious and error prone. But SymPy solves both of those issues:

```python
sym.Matrix(ydot).jacobian(y)
```

In the next notebook we will look at an example where providing this as a function is beneficial for performance.


# Linear Regression of a Nonlinear Model
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

## Synopsis
In this lesson, you will exercise your knowledge on how to perform linear least-squares regression on a nonlinear exponential model.

## Problem Statement

Chemical reacting systems consist of different substances reacting with each other. These reactants have different reaction rates. A reaction rate is the speed at which reactants are converted into products. A popular model for the reaction rate is known as the Arrhenius reaction rate model given by
\begin{equation}
k(T) = A e^{-\frac{E_\text{a}}{RT}}
\label{eq:arrhenius}
\end{equation}
where $k$ is known as the reaction rate constant, $T$ is the absolute temperature (in Kelvin), $A$ is the pre-exponential factor and is a measure of the frequency of molecular collisions, $E_\text{a}$ is the activation energy or the energy required to get the reaction started, and $R$ is the universal gas constant.

We can typically measure $k$ for a given temperature and chemical reaction. Our goal is to find $A$ and $E_\text{a}$ given those data.

|T (K)| k (1/s) |
|:-------:|:-------:|
|313|0.00043|
|319|0.00103|
|323|0.00180|
|328|0.00355|
|333|0.00717|

First, let's get some boiler plate out of the way

```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
import numpy as np
```

Now input the data from the table into numpy arrays

```python
R=8.314
T=np.array([313, 319, 323, 328, 333])
k=np.array([0.00043, 0.00103, 0.00180, 0.00355, 0.00717])
```

As usual, you should generate a plot to get an idea of how the data looks like

```python
plt.plot(T,k,'o')
plt.xlabel('Temperature (K)')
plt.ylabel('k (1/s)')
plt.title('Arrhenius reaction rate')
plt.grid()
```

The data do indeed look like they follow an exponential trend. We will use linear regression now to fit this model. We will first take the logarithm of the Arrhenius equation
\begin{equation}
\ln k = \ln A + - \frac{1}{RT} E_\text{a}
\end{equation}
where we will choose our new y-values as $y_\text{new} = \ln (k)$ and the new x-values as $x_\text{new} = - \frac{1}{RT}$. We will also call $a_0 = \ln{A}$ and $a_1=E_\text{a}$ so that our new model fit is
\begin{equation}
y_\text{new} = a_0 + a_1 x_\text{new}
\end{equation}

To program this, we will first create arrays for $y_\text{new}$ and $x_\text{new}$

```python
ynew = np.log(k)
xnew = -1.0/R/T
```

Let's plot the transformed data to see how they look like

```python
plt.plot(xnew,ynew,'o')
plt.xlabel('-1/RT')
plt.ylabel('log(k)')
plt.title('Arrhenius reaction rate')
plt.grid()
```

As expected, the transformed data do follow a linear trend. Now we can apply the usual linear least-squares regression to these transformed data.

Let's use the normal equations. In this case, we have

```python
N = len(ynew)
A = np.array([np.ones(N), xnew]).T
ATA = A.T @ A
coefs = np.linalg.solve(ATA, A.T @ ynew)
print(coefs)
```

    [3.89245804e+01 1.21481509e+05]

The variable `coefs` contains the coefficients of the straight line fit. We can untangle that using `poly1d`

```python
p = np.poly1d(np.flip(coefs,0)) # must reverse array for poly1d to work
```

Finally, we can plot the fit over the transformed data

```python
plt.plot(xnew, p(xnew), label='best fit')
plt.plot(xnew, ynew, 'o', label='original data')
plt.xlabel('-1/RT')
plt.ylabel('log(k)')
plt.title('Arrhenius reaction rate')
plt.grid()
```

we can also calculate the $R^2$ value. Recall that the $R^2$ value can be computed as:
\begin{equation}R^2 = 1 - \frac{\sum{(y_i - f_i)^2}}{\sum{(y_i - \bar{y})^2}}\end{equation}
where
\begin{equation}
\bar{y} = \frac{1}{N}\sum{y_i}
\end{equation}

```python
ybar = np.average(ynew)
fnew = p(xnew)
rsq = 1 - np.sum( (ynew - fnew)**2 )/np.sum( (ynew - ybar)**2 )
print(rsq)
```

    0.9998570770329075

This is an unbelievable fit! But it is correct since the data was actually setup to produce such a high R2 value.

Now we need to compute the original coefficients of the model, $A$ and $E_\text{a}$. This can be done by inverting the original transfomration
\begin{equation}
A = e^{a_0}
\end{equation}
and
\begin{equation}
E_\text{a} = a_1
\end{equation}

```python
a0 = coefs[0]
a1 = coefs[1]
A = np.exp(a0)
Ea = a1
```

```python
# construct a function  for the regressed line
y_fit = lambda x: A * np.exp(-Ea/R/x)

# To produce a nice smooth plot for the fit, generate points between Tmin and Tmax and use those as the x data
Tpoints = np.linspace(min(T), max(T))
plt.plot(Tpoints, y_fit(Tpoints), label='Best fit' )

# now plot the original input data
plt.plot(T, k, 'o', label='Original data')

plt.xlabel('Temperature (K)')
plt.ylabel('Reaction rate, k (1/s)')
plt.title('Arrhenius reaction rate')
plt.legend()
plt.grid()
```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


[ADS entry](http://adsabs.harvard.edu/abs/1985Natur.317...44C)

```python
import sympy
sympy.init_printing()
```

Equation 1

```python
rho = sympy.Symbol('rho') # Density
u = sympy.Symbol('u') # Velocity
r = sympy.Symbol('r', positive=True) # Radius
q = sympy.Symbol('q') # Mass injection rate
eqn_1 = sympy.Eq(sympy.Derivative(rho*u*r**2,r)/r**2,q)
eqn_1
```

Equation 2

```python
p = sympy.Symbol('p') # Pressure
eqn_2 = sympy.Eq(rho*u*sympy.Derivative(u,r),-sympy.Derivative(p,r)-q*u)
eqn_2
```

Equation 3

```python
Q = sympy.Symbol('Q', positive=True) # Energy injection rate
gamma = sympy.Symbol('gamma', positive=True) # Adiabatic index
eqn_3 = sympy.Eq(sympy.Derivative(rho*u*r**2*(u**2/2+gamma*p/rho/(gamma+1)),r)/r**2,Q)
eqn_3
```

Mass conservation can be integrated

```python
_ = q*r**2
_ = sympy.integrate(_,r)
int_mass_cons = sympy.Eq(rho*u,_/r**2)
int_mass_cons
```

Energy conservation can be integrated

```python
_ = Q*r**2
_ = sympy.integrate(_,r)
_ = _/r**2
_ = _/ int_mass_cons.rhs
int_energy_cons = sympy.Eq(gamma*p/(gamma-1)/rho+u**2/2,_)
int_energy_cons
```

Solution without injection

```python
m = sympy.Symbol('m', positive=True) # Mach number
c = sympy.Symbol('c') # Speed of sound
dot_M = sympy.Symbol(r'\dot{M}', positive=True)
B = sympy.Symbol('B',positive=True)
S = sympy.Symbol('S', positive=True)
xi = sympy.Symbol('xi', positive=True)
_ = [sympy.Eq(rho*u*r**2,dot_M),
    sympy.Eq(u**2/2+gamma*p/rho/(gamma-1),B),
    sympy.Eq(p/rho**gamma,S)]
_ = [itm.subs(u,m*c) for itm in _]
_ = [itm.subs(p,rho*c**2/gamma) for itm in _]
_ = [itm.subs(sympy.solve(_[0],rho,dict=True)[0]) for itm in _]
_ = [itm.subs(sympy.solve(_[1],c,dict=True)[1]) for itm in _]
_ = _[2]
_ = sympy.expand_power_base(_,force=True)
_ = _.simplify()
_ = _.subs(m**2,xi**2/(gamma-1)).simplify().subs(xi,m*sympy.sqrt(gamma-1)).simplify()
_ = _.subs(gamma,xi+1).simplify().subs(xi,gamma-1).simplify()
outside_solution = _
outside_solution
```

```python
z = sympy.Symbol('z') # Speed of sound squared
n = sympy.Symbol('n', positive=True) # 1/m**2
xi = sympy.Symbol('xi', positive=True) # Auxiliary variable
_ = [int_mass_cons, int_energy_cons, eqn_2]
_ = [itm.subs(p,rho*z/gamma) for itm in _]
_ = [itm.subs(u,m*sympy.sqrt(z)) for itm in _]
_ = [itm.subs(sympy.solve(_[1],z,dict=True)[0]) for itm in _]
_ = [itm.subs(sympy.solve(_[0],rho,dict=True)[0]) for itm in _]
_ = _[2]
_ = _.subs(m,1/sympy.sqrt(n(r)))
_ = _.doit()
_ = sympy.solve(_,n(r).diff(r))[0]
_ = _.subs(n(r),n)
_ = 1/_/r
_ = sympy.integrate(_,n)
_ = sympy.logcombine(_,force=True)
_ = sympy.exp(_)
_ = _.factor(n)
_ = _.subs(n,1/m**2)
inside_solution = _
inside_solution
```

Both cases ($r<R$ and $r>R$) have singularities at $m = 1$. Hence, the flow in each region is either supersonic or subsonic, and therefore the sonic point must be at $r = R$.

equation 4

```python
R = sympy.Symbol('R') # Radius of the wind emitting region
_ = inside_solution/inside_solution.subs(m,1)
eqn_4 = sympy.Eq(r/R,sympy.powsimp(_))
eqn_4
```

Equation 5

```python
_ = sympy.solve(outside_solution,r)[0]
_ = _/(_.subs(m,1))
_ = sympy.expand_power_base(_,force=True)
_ = _.simplify()
_ = _.subs(gamma,xi+1).simplify().subs(xi,gamma-1)
eqn_5 = sympy.Eq(r/R,_)
eqn_5
```

```python

```


# [Day 13](https://adventofcode.com/2020/day/13): Shuttle Search

```python
with open("../data/13.txt", "r") as f:
    t0, times = (l.strip() for l in f.readlines())

t0 = int(t0)
times = times.split(",")
times = [(o, int(p)) for o, p in enumerate(times) if p != "x"]
offsets, periods = zip(*times)
```

## Part 1

```python
wait, period = min(zip((-t0 % p for p in periods), periods))
assert 2092 == wait * period
```

## Part 2

The earliest timestamp ($t$) satisfies the system of congruences
$$
\eqalign{
t+T_0&\equiv0\pmod{P_0},\cr
t+T_1&\equiv0\pmod{P_1},\cr
&\dots\cr}
$$
where $T_n$, $P_n$ are the offsets and periods. The [Chinese Remainder Theorem](https://en.wikipedia.org/wiki/Chinese_remainder_theorem) yields a solution.

```python
from sympy import gcd
from sympy.ntheory.modular import crt
from itertools import combinations

# Verify that we can apply the Chinese Remainder Theorem
for m, n in combinations(periods, 2):
    assert gcd(m, n) == 1

timestamp, _ = crt(periods, (-o for o in offsets))
assert 702970661767766 == int(timestamp)
```


# METOD algorithm - Sum of Gaussians

# 1. Import libraries

The following libraries are required to run the METOD Algorithm.

```python
import numpy as np
from numpy import linalg as LA
import pandas as pd

import metod_alg as mt
from metod_alg import objective_functions as mt_obj
```

# 2. Define function and gradient

Weighted Sum of Gaussians objectve function:

\begin{equation}
\label{eq:funct1}
f(x_n^{(k)})= -\sum_{p=1}^{P} c_p\exp \Bigg\{ {-\frac{1}{2 \sigma^2}(x_n^{(k)}-x_{0p})^T A_p^T \Sigma_p A_p(x_n^{(k)}-x_{0p})}\Bigg\}\, .
\end{equation}
where $x_n^{(k)}$ is the $n$-th point after $k$ iterations of anti-gradient descent, $P$ is the number of Gaussian densities; $A_p$ is a random rotation matrix of size $d\times d$; $\Sigma_p$ is a diagonal positive definite matrix of size $d\times d$ with smallest and largest eigenvalues $\lambda_{min}$ and $\lambda_{max}$ respectively;  $x_{0p} \in \mathfrak{X}=[0,1]^d$ (centers of the Gaussian densities); $c_p$ is a fixed constant and $p=1,...,P$.

Note that anti-gradient descent iterations are terminated at the smallest $k=K_n$ such that $\nabla f(x_n^{(k)}) < \delta$, where $\delta$ is some small positive constant.

```python
f = mt_obj.sog_function
g = mt_obj.sog_gradient
```

# 3. Defining parameters

The following parameters are required in order to derive $A_p$ ($p=1,...,P$) ; $\Sigma_p$ ($p=1,...,P$); $x_{0p}$ ($p=1,...,P$) and $c_p$ ($p=1,...,P$).

•d: dimension

•P: number of minima

•lambda_1: smallest eigenvalue of $\Sigma_p$ ($p=1,...,P$)

•lambda_2: largest eigenvalue of $\Sigma_p$ ($p=1,...,P$)

•sigma_sq: value for $\sigma^2$

In order to replicate results, we will control the pseudo-random number generator seed, so that the same random objective function and random starting points $x_n^{(0)}$ $(n=1,...,1000)$ will be generated each time the code is run. The random seed number will be set to 90.

```python
d = 20
P = 10
lambda_1 = 1
lambda_2 = 10
sigma_sq = 0.8
seed = 90
```

```python
np.random.seed(seed)
store_x0, matrix_combined, store_c = mt_obj.function_parameters_sog(P, d, lambda_1, lambda_2)
```

Where,

•store_x0: $x_{0p}$ ($p=1,...,P$)

•matrix_combined: $A_p^T \Sigma_p A_p$ ($p=1,...,P$)

•store_c: $c_p$ ($p=1,...,P$)

```python
args = P, sigma_sq, store_x0, matrix_combined, store_c
```

# 4. Run METOD Algorithm

```python
(discovered_minimizers,
 number_minimizers,
 func_vals_of_minimizers,
 excessive_no_descents,
 starting_points,
 grad_evals) = mt.metod(f, g, args, d)
```

# 5. Results of the METOD Algorithm

Total number of minimizers found:

```python
number_minimizers
```

Positions of minimizers:

```python
discovered_minimizers
```

Function values of minimizers:

```python
func_vals_of_minimizers
```

Total number of excessive descents:

```python
excessive_no_descents
```

Number of gradient evaluations for each starting point

```python
grad_evals
```

# 6. Save results to csv file (optional)

The below csv files will be saved to the same folder which contains the METOD Algorithm - Sum of Gaussians notebook.

Rows in discovered_minimizers_d_%s_p_%s_sog.csv represent discovered minimizers. The total number of rows will be the same as the value of number_minimizers.

```python
np.savetxt('discovered_minimizers_d_%s_p_%s_sog.csv' % (d, P), discovered_minimizers, delimiter=",")
```

Each row in func_vals_discovered_minimizers_d_%s_p_%s_sog.csv represents the function value of each discovered minimizer. The total number of rows will be the same as the value for number_minimizers.

```python
np.savetxt('func_vals_discovered_minimizers_d_%s_p_%s_sog.csv' % (d, P), func_vals_of_minimizers, delimiter=",")
```

summary_table_d_%s_p_%s_sog.csv will contain the total number of minimizers discovered and the total number of extra descents.

```python
summary_table = pd.DataFrame({
"Total number of unique minimizers": [number_minimizers],
"Extra descents": [excessive_no_descents]})
summary_table.to_csv('summary_table_d_%s_p_%s_sog.csv' % (d, P))
```

# 7. Test results (optional)

This test can only be used for the Sum of Gaussians function.

To check each discovered minimizer is unique, we do the following:

For each minimizer $x_l^{(K_l)}$ ($l=1,...,L$)

\begin{equation}
p_l = {\rm argmin}_{1\le p \le P} \|x_l^{(K_l)} - x_{0p}\|
\end{equation}

For each $p_l$ found, it is ensured that $\|x_l^{(K_l)} - x_{0p_l}\| \text{  is small}$.

If all $p_l$ is different for each l=$(1,...,L)$ and $\|x_l^{(K_l)} - x_{0p_l}\|$ is small for each $p_l$, then all discovered minimizers are unique.

```python
def calc_minimizer(point, p, store_x0):
    """Returns the index p_1 and also the distance between the minimizer discovered by METOD and x_{0p_1}"""
    dist = np.zeros((p))
    for i in range(p):
        dist[i] = LA.norm(point - store_x0[i])
    return np.argmin(dist), np.min(dist)
```

```python
"""Store values from calc_minimizer function"""
norms_with_minimizer = np.zeros((number_minimizers))
pos_list = np.zeros((number_minimizers))
for j in range(number_minimizers):
    pos, min_dist = calc_minimizer(discovered_minimizers[j], P, store_x0)
    pos_list[j] = pos
    norms_with_minimizer[j] = min_dist
```

${\max}_{1\le l \le L}  \|x_l^{(K_l)}-x_{0p_l}\|$ should be small

```python
np.max(norms_with_minimizer)
```

Ensure that the number of unique minimizers is $L$

```python
np.unique(pos_list).shape[0] == number_minimizers
```

```python

```

Examples/Notebooks/METOD Algorithm - Sum of Gaussians.ipynb


# Saving Memory

Running operations can cause new memory to be allocated to host results.

1. First, we do not want to run around allocating memory unnecessarily all the time. In machine learning, we might have hundreds of megabytes of parameters and update all of them multiple times per second. Typically, we will want to perform these updates **in place**.
2. we might point at the same parameters from multiple variables. If we do not update in place, other references will still point to the old memory location, making it possible for parts of our code to inadvertently reference stale parameters.

```python
import torch
y = torch.tensor([3.5])
x = 1.2
print(id(y))
y = 1.2 + y
print(id(y))
```

    2232877748616
    2232877751896

```python
# Fortunately, performing in-place operations is easy. We can assign the result of an operation
# to a previously allocated array with slice notation
Z = torch.zeros_like(y)
print('id(Z):', id(Z))
Z[:] = x + y
print('id(Z):', id(Z))
```

    id(Z): 2232877750936
    id(Z): 2232877750936

```python
print(id(y))
y[:] = 1.2 + y
print(id(y))
```

    2232877751896
    2232877751896

```python
print(id(y))
y += x
print(id(y))
```

    2232877751896
    2232877751896

# Linear Algebra

## Scalar, Vector, Matrix and Tensor
1. Formally, we call values consisting of just one numerical quantity **scalars** --> $x \in \mathbb{R}$
\begin{align}
x = 2.5
\end{align}
2. You can think of a **vector** as simply a list of scalar values. We call these values the elements (entries or components) of the vector. We work with vectors via one-dimensional tensors. In general tensors can have arbitrary lengths, subject to the memory limits of your machine. Extensive literature considers column vectors to be the default orientation of vectors, so does this book.  --> $X \in \mathbb{R}^n$ represents a vector x consists of n real-valued scalars.
\begin{equation}
  X = \begin{bmatrix}
x_1 \\
x_2 \\
\vdots \\
x_n
\end{bmatrix}
\end{equation}
3. Just as vectors generalize scalars from order zero to order one, **matrices** generalize vectors from order one to order two. $A \in \mathbb{R}^{m \times n}$ to express that the matrix A consists ofmrows and n columns of real-valued scalars
\begin{equation}
A = \begin{bmatrix}
a_{11} & a_{12} & \ldots & a_{1n} \\
a_{21} & a_{22} & \ldots & a_{2n} \\
\vdots & \vdots & \ddots &\vdots \\
a_{m1} & a_{m2} & \ldots & a_{mn}
\end{bmatrix}
\end{equation}
- when a matrix has the same number of rows and columns, its shape becomes a square; thus, it is called a **square matrix**.
- Sometimes, we want to flip the axes. When we exchange a matrixʼs rows and columns, the result is called the **transpose** of the matrix, which is denoted by $A^T$
- a **symmetric matrix** A is equal to its transpose: $A = A^T$

Matrices are useful data structures: they allow us to organize data that have **different modalities of variation**. For example, rows in our matrix might correspond to different houses (data examples), while columns might correspond to different attributes.

Thus, although the default orientation of a single vector is a column vector, in a matrix that represents a tabular dataset, **it is more conventional to treat each data example as a row vector in the matrix.** And, as we will see in later chapters, this convention will enable common deep learning practices

4. Just as vectors generalize scalars, and matrices generalize vectors, we can build data structures with even more axes. **Tensors** (“tensors” in this subsection refer to algebraic objects) give us a generic way of describing **n-dimensional arrays** with an arbitrary number of axes.
 - Vectors, for example, are **first-order** tensors, and matrices are **second-order** tensors.

## Basic Properties of Tensor Arithmetic

### Hadamard product $A*B$

<!-- $\odot$ \quad $\oplus$ \quad $\otimes$ \quad $\ominus$ \quad $\oslash$ -->
Specifically, elementwise multiplication of two matrices is called their Hadamard product (math notation $\odot$). For example, $A, B \in \mathbb{R}^{m \times n}$:
\begin{equation}
B = \begin{bmatrix}
b_{11} & b_{12} & \ldots & b_{1n} \\
b_{21} & b_{22} & \ldots & b_{2n} \\
\vdots & \vdots & \ddots &\vdots \\
b_{m1} & b_{m2} & \ldots & b_{mn}
\end{bmatrix}
\end{equation}

\begin{equation}
A \odot B = A * B= \begin{bmatrix}
a_{11}b_{11} &  a_{12}b_{12} & \ldots & a_{1n}b_{1n} \\
a_{21}b_{21} & a_{22}b_{22} & \ldots & a_{2n}b_{2n} \\
\vdots & \vdots & \ddots &\vdots \\
a_{m1}b_{m1} & a_{m2}b_{m2} & \ldots & a_{mn}b_{mn}
\end{bmatrix}
\end{equation}

### Dot Products $torch.dot(x, y)$

Given two vectors $x, y \in \mathbb{R}^d$, their **dot product** $x^Ty$ or ($<x, y>$) is a sum over the products of the elements at the same position:
\begin{equation}
x^Ty = \sum_{i = 1}^d x_iy_i
\end{equation}

```python
import torch
y = torch.ones(4, dtype=torch.float32)
y
```

    tensor([1., 1., 1., 1.])

```python
x = torch.tensor([0., 1., 2., 3.])
x
```

    tensor([0., 1., 2., 3.])

```python
torch.dot(x,y)
```

    tensor(6.)

```python
x*y
```

    tensor([0., 1., 2., 3.])

```python
torch.sum(x * y)
```

    tensor(6.)

**Usage: weighted average** Dot products are useful in a wide range of contexts. For example, given some set of values, denoted by a vector $x \in \mathbb{R}^d$ and a set of weights denoted by $w \in \mathbb{R}^d$, the weighted sum of the values in $x$ according to the weights $w$ could be expressed as the dot product $x^Tw$. When the weights are non-negative and sum to one (i.e., $\sum_{i=1}^{d}w_i = 1$), the dot product expresses a **weighted average**. After normalizing two vectors to have the unit length, the dot products express the cosine of the angle between them.

### Matrix-Vector Products $torch.mv(A, x)$

Recall the matrix $A \in \mathbb{R}^{m \times n}$ and the vector $x \in \mathbb{R}^n$. The matrix-vector product $Ax$ is simply a column vector of length $m$, whose $i^{th}$ element is the dot product $a_i^Tx$

### Matrix-Matrix Multiplication $torch.mm(A, B)$

Say that we have two matrices $A \in \mathbb{R}^{m x k}$ and $B \in \mathbb{R}^{k x n}$, $C = AB \in mathbb{R}^{m x n}$

### Norms

Informally, the norm of a vector tells us how big a vector is. The notion of **size** under consideration here concerns not dimensionality but rather the magnitude of the components.

In linear algebra, a vector norm is a function $f$ that maps a vector ($x, Y\in \mathbb{R}^n$) to a scalar $f: \mathbb{R}^n \rightarrow \mathbb{R}$, satisfying a handful of properties:
1. its norm also scales by the absolute value $\alpha$ of the same constant factor: $f(\alpha x) = |\alpha| f(x)$.
2. Triangle inequality: $f(x + y) \leq f(x) + f(y)$
3. the norm must be non-negative: $f(x) \geq 0$

You might notice that norms sound a lot like measures of distance:

1. $L_2$ norm (Euclidean distance) of $x$ is the square root of the sum of the squares of the vector elements:
\begin{equation}
L_2(x) = \parallel x \parallel_2 = \sqrt{\sum_{i=1}^{n}x_i^2}
\end{equation}
In deep learning, we work more often with the squared L2 norm.

```python
# In code, we can calculate the L2 norm of a vector as follows.
u = torch.tensor([3.0, -4.0])
torch.norm(u)
```

    tensor(5.)

2. $L_1$ norm is expressed as the sum of the absolute values of the vector elements:
\begin{equation}
L_1(x) = \parallel x \parallel_1 = \sum_{i=1}^{n}|x_i|
\end{equation}

```python
# To calculate the L1 norm, we compose the absolute value function with a sum over the elements.
torch.abs(u).sum()
```

    tensor(7.)

As compared with the L2 norm, it is less influenced by outliers.  Both the $L_2$ norm and the $L_1$ norm are special cases of the more general $L_p$ norm:
\begin{equation}L_p(x) = \parallel x \parallel_p = (\sum_{i=1}^{n}|x_i|^p)^{1/p}\end{equation}

Analogous to $L_2$ norms of vectors, the Frobenius norm of a matrix $X \in \mathbb{R}^{m x n}$ is the square root of the sum of the squares of the matrix elements:
\begin{equation}
\parallel X \parallel_F = \sqrt{\sum_{i=1}^m\sum_{j=1}^nx_{ij}^2}
\end{equation}

```python
# Invoking the following function will calculate the Frobenius norm of a matrix.
torch.norm(torch.ones((4, 9)))
```

    tensor(6.)

```python

```

deep_learning/Geometry and Linear Algebra.ipynb


```python
import matplotlib.pyplot as plt
import numpy as np
```

(a) At the start of section 4.1 it was shown that the likelihood for a set of n independent Binomial trials is given by:

\begin{align}
L(\theta) & = \prod_{i=1}^{n} \theta^{x_i} (1-\theta)^{1-x_i} \\
& = \theta^{\sum_{i=1}^{n} x_i} (1 - \theta)^{n - \sum_{i=1}^{n} x_i}
\end{align}

or in terms of the log-likelihood:
\begin{align}
logL(\theta) & = log \theta \sum_{i=1}^{n} x_i + \log (1-\theta) \Bigg( {n- \sum_{i=1}^{n} x_i} \Bigg)
\end{align}

(b) The likelihood of $\theta$ is given by

```python
k = np.array([i for i in range(7)])
n_k_2 = np.array([42860, 89213, 47819, 0, 0, 0, 0])
n_k_6 = np.array([1096, 6233, 15700, 22221, 17332, 7908, 1579])
x = n_k_2 + n_k_6
print(repr(k))
print(repr(x))
```

```python
def log_likelihood(k: np.ndarray, x: np.ndarray, theta: np.ndarray) -> np.ndarray:
    n = np.sum(k * x)
    sum_x = np.sum(x)
    log_like = np.log(theta) * sum_x + np.log(1 - theta) * (n - sum_x)
    #like = np.exp(log_like - np.max(log_like))  # tiny values
    return log_like
```

```python
theta = np.linspace(0.01, 0.99, num=100)
log_like = log_likelihood(k, x, theta)
plt.plot(theta, log_like)
plt.xlabel(r'$\theta$')
plt.ylabel('Log likelihood')
plt.title(r'Log likelihood of $\theta$');
```

The MLE of $\theta$ is given by the solution to the score equation which is:

$$
\hat{\theta} = \frac{1}{n} \sum_{i=1}^{n} x_i
$$

```python
def mle(k: np.ndarray, x: np.ndarray) -> float:
    return np.sum(x) / np.sum(k * x)
```

```python
theta_hat = mle(k, x)
print(f'MLE of theta = {np.round(theta_hat, 2)}')
```

The standard error of $\theta$ is given by:

\begin{align}
se(\hat{\theta}) & = \sqrt{var \Bigg(\frac{1}{n} \sum_{i=1}^{n} x_i  \Bigg)} \\
& = \sqrt{\frac{\hat{\theta} (1 - \hat{\theta})}{n}}
\end{align}

```python
se_theta_hat = np.sqrt((theta_hat * (1 - theta_hat)) / np.sum(k * x))
np.round(se_theta_hat, 4)
```

(c) The binomial model is not a good fit to the data as the MLE is larger than would be expected. There is not equal sampling probability amongst the familes. Extra-binomial variance is observed since the sampling is done from a mixed population of families of size 2 and families of size 6.


# Task One : Fuandamentals of Qunatum Computation

## Basic Arithmatic Operations & Complex Numbers

```python

```

```python

```

Complex numbers are always of the form

\begin{align}
\alpha = a + bi
\end{align}

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## Complex conjugate

```python

```

```python

```

## Norms/Absolute Values

\begin{align} ||z|| &= \sqrt{zz^*} = \sqrt{|z|^2},\\ ||w|| &= \sqrt{ww^*} = \sqrt{|w|^2}, \end{align}

```python

```

```python

```

## Row Vectors, Column Vectors, and Bra-Ket Notation

\begin{align} \text{Column Vector:} \ \begin{pmatrix}
a_1 \\ a_2 \\ \vdots \\ a_n
\end{pmatrix}
\quad \quad \text{Row Vector:} \ \begin{pmatrix}
a_1, & a_2, & \cdots, & a_n
\end{pmatrix} \end{align}

```python

```

```python

```

Row vectors in quantum mechanics are also called **bra-vectors**, and are denoted as follows:

\begin{align} \langle A| = \begin{pmatrix}
a_1, & a_2, \cdots, & a_n
\end{pmatrix} \end{align}

Column vectors are also called **ket-vectors** in quantum mechanics denoted as follows:

\begin{align} |B\rangle = \begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix} \end{align}

In general, if we have a column vector, i.e. a ket-vector:

\begin{align} |A\rangle = \begin{pmatrix}
a_1 \\ a_2 \\ \vdots \\ a_n
\end{pmatrix} \end{align}

the corresponding bra-vector:

\begin{align} \langle A| = \begin{pmatrix}
a_1^*, & a_2^*, & \cdots, & a_n^*
\end{pmatrix} \end{align}

```python

```

```python

```

## Inner Product

\begin{align} \langle A| = \begin{pmatrix}
a_1, & a_2, & \cdots, & a_n
\end{pmatrix}, \quad \quad
|B\rangle = \begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix} \end{align}

Taking the inner product of $\langle A|$ and $|B\rangle$ gives the following:

\begin{align} \langle A| B \rangle &= \begin{pmatrix}
a_1, & a_2, & \cdots, & a_n
\end{pmatrix}
\begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix}\\
&=
a_1b_1 + a_2b_2 + \cdots + a_nb_n\\
&= \sum_{i=1}^n a_ib_i
\end{align}

```python
# Define the 4x1 matrix version of a column vector (instead of using the np.array() version):
```

```python
# Define B as a 1x4 matrix

```

```python
# Compute <B|A>

```

## Matrices

\begin{align}
M = \begin{pmatrix}
2-i & -3 \\
-5i & 2
\end{pmatrix}
\end{align}

```python

```

```python

```

Hermitian conjugates are given by taking the conjugate transpose of the matrix

```python

```

## Tensor Products of Matrices

\begin{align}
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix} \otimes
\begin{pmatrix}
x & y \\
z & w
\end{pmatrix} =
\begin{pmatrix}
a \begin{pmatrix}
x & y \\
z & w
\end{pmatrix} & b \begin{pmatrix}
x & y \\
z & w
\end{pmatrix} \\
c \begin{pmatrix}
x & y \\
z & w
\end{pmatrix} & d \begin{pmatrix}
x & y \\
z & w
\end{pmatrix}
\end{pmatrix} =
\begin{pmatrix}
ax & ay & bx & by \\
az & aw & bz & bw \\
cx & cy & dx & dy \\
cz & cw & dz & dw
\end{pmatrix}
\end{align}

```python

```

# Task Two: Qubits, Bloch Sphere and Basis States

```python

```

\begin{align}
|\psi \rangle = \begin{pmatrix}
\alpha \\ \beta
\end{pmatrix}, \quad \text{where } \sqrt{\langle \psi | \psi \rangle} = 1.
\end{align}

Think of Qubit as an Electron:

\begin{align}
\text{spin-up}: \ |0\rangle &= \begin{pmatrix} 1\\0 \end{pmatrix} \\
\text{spin-down}: \ |1\rangle & = \begin{pmatrix} 0\\1 \end{pmatrix}
\end{align}

```python

```

Another representation is via Bloch Sphere:

```python

```

## Spin + / - :

\begin{align}
\text{spin +}: \ |+\rangle &= \begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix} = \frac{1}{\sqrt{2}} \left(|0\rangle + |1\rangle\right) \\
\text{spin -}: \ |-\rangle & = \begin{pmatrix} 1/\sqrt{2} \\ -1/\sqrt{2} \end{pmatrix} = \frac{1}{\sqrt{2}} \left(|0\rangle - |1\rangle\right)
\end{align}

```python

```

## Basis States

\begin{align}
 |0\rangle &= \begin{pmatrix} 1\\0 \end{pmatrix} \\
 |1\rangle & = \begin{pmatrix} 0\\1 \end{pmatrix}
\end{align}
Preapring other states from Basis States:
\begin{align}
|00 \rangle &= |0\rangle \otimes |0\rangle = \begin{pmatrix} 1\\0 \end{pmatrix} \otimes \begin{pmatrix} 1\\0 \end{pmatrix} = \begin{pmatrix} 1\\0\\0\\0 \end{pmatrix} \\
|01 \rangle &= |0\rangle \otimes |1\rangle = \begin{pmatrix} 1\\0 \end{pmatrix} \otimes \begin{pmatrix} 0\\1 \end{pmatrix} = \begin{pmatrix} 0\\1\\0\\0 \end{pmatrix} \\
|10 \rangle &= |1\rangle \otimes |0\rangle = \begin{pmatrix} 0\\1 \end{pmatrix} \otimes \begin{pmatrix} 1\\0 \end{pmatrix} = \begin{pmatrix} 0\\0\\1\\0 \end{pmatrix} \\
|11 \rangle &= |1\rangle \otimes |1\rangle = \begin{pmatrix} 0\\1 \end{pmatrix} \otimes \begin{pmatrix} 0\\1 \end{pmatrix} = \begin{pmatrix} 0\\0\\0\\1 \end{pmatrix}
\end{align}

```python

```

```python

```

# Task Three: Qunatum Gates and Circuits

```python
from qiskit import *
from qiskit.visualization import plot_bloch_multivector
```

## Pauli Matrices

\begin{align}
I = \begin{pmatrix} 1&0 \\ 0&1 \end{pmatrix}, \quad
X = \begin{pmatrix} 0&1 \\ 1&0 \end{pmatrix}, \quad
Y = \begin{pmatrix} 0&i \\ -i&0 \end{pmatrix}, \quad
Z = \begin{pmatrix} 1&0 \\ 0&-1 \end{pmatrix} \quad
\end{align}

## X-gate

The X-gate is represented by the Pauli-X matrix:

$$ X = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} = |0\rangle\langle1| + |1\rangle\langle0| $$

Effect a gate has on a qubit:

$$ X|0\rangle = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix} = |1\rangle$$

```python
# Let's do an X-gate on a |0> qubit

```

```python
# Let's see the result

```

## Z & Y-Gate

$$ Y = \begin{bmatrix} 0 & -i \\ i & 0 \end{bmatrix} \quad\quad\quad\quad Z = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} $$

$$ Y = -i|0\rangle\langle1| + i|1\rangle\langle0| \quad\quad Z = |0\rangle\langle0| - |1\rangle\langle1| $$

```python
# Do Y-gate on qubit 0

# Do Z-gate on qubit 0

```

## Hadamard Gate

$$ H = \tfrac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} $$

We can see that this performs the transformations below:

$$ H|0\rangle = |+\rangle $$

$$ H|1\rangle = |-\rangle $$

```python
#create circuit with three qubit

# Apply H-gate to each qubit:

# See the circuit:

```

## Identity Gate

$$
I = \begin{bmatrix} 1 & 0 \\ 0 & 1\end{bmatrix}
$$

$$ I = XX $$

```python

```

 ** Other Gates: S-gate , T-gate, U-gate

# Task Four : Multiple Qubits, Entanglement

## Multiple Qubits

The state of two qubits :

$$ |psi\rangle = a_{00}|00\rangle + a_{01}|01\rangle + a_{10}|10\rangle + a_{11}|11\rangle = \begin{bmatrix} a_{00} \\ a_{01} \\ a_{10} \\ a_{11} \end{bmatrix} $$

```python
from qiskit import *
```

```python

# Apply H-gate to each qubit:

# See the circuit:

```

Each qubit is in the state $|+\rangle$, so we should see the vector:

$$
|{+++}\rangle = \frac{1}{\sqrt{8}}\begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \\
                              1 \\ 1 \\ 1 \\ 1 \\
              \end{bmatrix}
$$

```python
# Let's see the result

```

```python

```

$$
X|q_1\rangle \otimes H|q_0\rangle = (X\otimes H)|q_1 q_0\rangle
$$

The operation looks like this:

$$
X\otimes H = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \otimes \tfrac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} = \frac{1}{\sqrt{2}}
\begin{bmatrix} 0 & 0 & 1 & 1 \\
                0 & 0 & 1 & -1 \\
                1 & 1 & 0 & 0 \\
                1 & -1 & 0 & 0 \\
\end{bmatrix}
$$

Which we can then apply to our 4D statevector $|q_1 q_0\rangle$. You will often see the clearer notation:

$$
X\otimes H =
\begin{bmatrix} 0 & H \\
               H & 0\\
\end{bmatrix}
$$

## C-Not Gate

```python
#create circuit with two qubit

# Apply CNOT

# See the circuit:

```

Classical truth table of C-Not gate:

| Input (t,c) | Output (t,c) |
|:-----------:|:------------:|
| 00          | 00           |
| 01          | 11           |
| 10          | 10           |
| 11          | 01           |

## Entanglement

```python
#create two qubit circuit

# Apply H-gate to the first:

```

```python
# Let's see the result:

```

Quantum System Sate is:

$$
|0{+}\rangle = \tfrac{1}{\sqrt{2}}(|00\rangle + |01\rangle)
$$

```python

# Apply H-gate to the first:

# Apply a CNOT:

```

```python
# Let's see the result:

```

We see we have this final state (Bell State):

$$
\text{CNOT}|0{+}\rangle = \tfrac{1}{\sqrt{2}}(|00\rangle + |11\rangle)
$$

Other Bell States:

# Task Five: Bernstein-Vazirani Algorithm

A black-box function $f$, which takes as input a string of bits ($x$), and returns either $0$ or $1$, that is:

$$f(\{x_0,x_1,x_2,...\}) \rightarrow 0 \textrm{ or } 1 \textrm{ where } x_n \textrm{ is }0 \textrm{ or } 1  $$

The function is guaranteed to return the bitwise product of the input with some string, $s$.

In other words, given an input $x$, $f(x) = s \cdot x \, \text{(mod 2)} =\ x_0 * s_0+x_1*s_1+x_2*s_2+...\ $ mod 2

The quantum Bernstein-Vazirani Oracle:

1. Initialise the inputs qubits to the $|0\rangle^{\otimes n}$ state, and output qubit to $|{-}\rangle$.
2. Apply Hadamard gates to the input register
3. Query the oracle
4. Apply Hadamard gates to the input register
5. Measure

## Example Two Qubits:

<ol>
    <li> The register of two qubits is initialized to zero:

$$\lvert \psi_0 \rangle = \lvert 0 0 \rangle$$

   </li>

   <li> Apply a Hadamard gate to both qubits:

$$\lvert \psi_1 \rangle = \frac{1}{2} \left( \lvert 0 0 \rangle + \lvert 0 1 \rangle + \lvert 1 0 \rangle + \lvert 1 1 \rangle \right) $$

   </li>

   <li> For the string $s=11$, the quantum oracle performs the operation:
$$
|x \rangle \xrightarrow{f_s} (-1)^{x\cdot 11} |x \rangle.
$$

$$\lvert \psi_2 \rangle = \frac{1}{2} \left( (-1)^{00\cdot 11}|00\rangle + (-1)^{01\cdot 11}|01\rangle + (-1)^{10\cdot 11}|10\rangle + (-1)^{11\cdot 11}|11\rangle \right)$$

$$\lvert \psi_2 \rangle = \frac{1}{2} \left( \lvert 0 0 \rangle - \lvert 0 1 \rangle - \lvert 1 0 \rangle + \lvert 1 1 \rangle \right)$$

   </li>

   <li> Apply a Hadamard gate to both qubits:

$$\lvert \psi_3 \rangle = \lvert 1 1 \rangle$$

   </li>

   <li> Measure to find the secret string $s=11$
   </li>

</ol>

```python
from qiskit import *
%matplotlib inline
from qiskit.tools.visualization import plot_histogram
```

```python

```

```python

```

```python

```

```python

```

```python

```


# Sympy

Anaconda Python comes with a package called [Sympy](https://www.sympy.org/en/index.html) which is a shortened term for *Symbolic Python*. Most simply, this package allows the user to analytically solve problems that he might otherwise have to solve numerically in Python. Generally, in the past, most analytical work was done in what is known as a Computer Algebra System (CAS) such as *Mathematica*. While *Mathematica* still remains at the peak of most symbolic representations, tools like Sympy can be a powerful asset in the Physicist's toolbelt.

We'll begin below with a few examples to get an understanding of how Sympy works. Considering I have never used Sympy before, I will be closely following the [Dynamics and Controls in Jupyter Notebooks Tutorial](https://dynamics-and-control.readthedocs.io/en/latest/index.html)

## Imports

```python
# Python
from IPython.display import display

# 3rd Party Imports
import numpy as np
import sympy as sy
```

## Notebook Setup

It appears that Sympy, be default, does not try to "pretty print" math output. But, Jupyter notebooks can display LaTeX and MathJax equations so we will turn on this sympy feature now.

```python
# Turn on Pretty Print
sy.init_printing()
```

## Symbol / Variable Creation

Python behaves the same way with Sympy as it does with other aspects of the language. For example, if you attempted to run the line

```python
print(x)
```

without first defining `x`, Python would throw a name error. Therefore, we would have to assign a value to `x` then call the `print` function like the example below.

```python
x = 5
print(x)
```

With the above syntax, we are telling Python to first assign the value of 5 to `x` then recall what `x` represents. Sympy works the same way; however, we will use Sympy to tell the Python interpretter that `x` represents a Symbol. Although, you do not have to know this to know how Sympy works, it may be useful to know that Symbol is simply a class in Sympy and we are simply instantiating an object of that class.

The next cell shows how we might instantiate a Symbol object `x` and print its value.

```python
x = sy.Symbol('x')
y = sy.Symbol('z')
display(x)
display(y)
```

Now, there are a few things to note from the cell above. On the first line, we initialized a Sympy Symbol called $x$ and stored it into the variable `x`. On the second line, we initialized a Sympy Symbol but $z$ but stored it into a variable called `y`. Note how Python does not care what the variable name is. We could have called the Symbol $z$ `joe` but Python would still interpret that variable `joe` as $z$.

One last thing to notice. Above, I call the IPython function `display` rather than the built in Python function `print`. This is simply bacause pretty printing is not a Python capability. Rather, this is an IPython capability. However, if (like below) you only have to print one thing per cell, IPython will automatically `display` the value for you.

```python
x**2
```

## Higher Level Examples

We'll next study Sympy's options by looking at the simply mathematical construct: the polynomial. Let's first create and display a polynomial to work with.

```python
# Create the Polynomial
pol = (3*x**2 - x + 1)**2
pol
```

We can then perform higher level operations on the polynomial by operating directly on the `pol` object.

```python
# Display the Expanded Polynomial
pol.expand()
```

```python
# Get the First Derivative
pol.diff()
```

```python
# Get the Second Derivative
pol.diff(x, 2)  # Arg 1: wrt; Arg 2: Second deriv
```

```python
# Get the Indefinite Integral
pol.integrate()
```

```python
# Get the Definite Integral from -1 to 1
display(pol.integrate((x, -1, 1)))
display(sy.N(pol.integrate((x, -1, 1))))  # As a Decimal Number
```

We can even use Sympy to get the Taylor Series expansion of expressions.

```python
display(sy.series(sy.exp(x), x, 0, 5))  # Expansion of e^x at x=0
display(sy.series(sy.sin(x), x, 0, 5))  # Expansion of sin(x) at x=0
display(sy.series(sy.cos(x), x, 0, 5))  # Expansion of cos(x) at x=0
```

### Solving a System of Equations

If you know what you are doing, solving a system of equations using a computer is just as easy to do it numerically as it is symbolically. However, it is sometimes nice to solve a system of equations with any arbitrary variables. For example, below we have a system of equations with four unknowns, but we are interested in solving the equations for $x$ and $y$.

One advantage (at least in Python) of solving the system numerically with Numpy is that Numpy is much faster than Sympy. For small systems of equations, this is not of great importance, but if you had many variables then this could become a problem very quickly. See the [Dynamics and Controls - Linear Systems Example](https://dynamics-and-control.readthedocs.io/en/latest/1_Dynamics/1_Modelling/Equation%20solving%20tools.html#Special-case:-linear-systems) for details on the speed of Sympy vs. the speed of Numpy.

```python
# Assign Symbols
x, y, a, b = sy.symbols('x, y, a, b')

# Solve the system of equations
sy.solve(
    [
        a*x - 3*y + 4, # = 0
        2*x + b*y - 1  # = 0
    ],
    [x, y]
)
```

This same concept can be extended to solving for a differential equation. Below, we express $x$ as some unknown function of $t$, setup the differential equation and solve it.

```python
# Create the Variables
t = sy.Symbol('t', postive=True)
w = sy.Symbol('omega', positive=True)

# Create the Position function
x = sy.Function('x', real=True)

# Create and Print the Differential equation
de = x(t).diff(t, 2) + w**2 * x(t)  # = 0
de
```

```python
# Get the Solution
sy.dsolve(de)
```

### The Laplace Transform

To avoid re-inventing the wheel, I will point you to the [Dynamics and Controls - Laplace Transform Introduction](https://dynamics-and-control.readthedocs.io/en/latest/1_Dynamics/3_Linear_systems/Laplace%20transforms.html#Laplace-transforms-in-SymPy) to begin this section. Instead, I will recreate the first few lines of a [Table of Laplace Transforms](http://tutorial.math.lamar.edu/Classes/DE/Laplace_Table.aspx).

Now, I do not want to type out `sy.laplace_transform` and `sy.inverse_laplace_transform` everytime, so I will just import them below with a shorter, simpler name.

```python
# Import the Laplace Transforms
from sympy import laplace_transform as L, inverse_laplace_transform as invL

# Define the Symbols needed
s, t = sy.symbols('s t')
a    = sy.Symbol('a', real=True, positive=True)
```

```python
# Do the Laplace Transform of 1
display(L(1, t, s))

# Display the same without conditions
display(L(1, t, s, noconds=True))
```

```python
# Do the Laplace Transform of exp(a t)
L(sy.exp(a*t), t, s, noconds=True)
```

```python
# Do the Laplace Transform of t^n
n = sy.Symbol('n', integer=True, positive=True)
L(t**n, t, s, noconds=True)
```

```python
# Do the Laplace Transform of t^p, p > -1
p = sy.Symbol('p', real=True)
L(t**p, t, s)
```

```python
# Do the Laplace Transform of cos(a t)
L(sy.cos(a*t), t, s, noconds=True)
```

```python
# Show the integral of sinc(a x) = sin(a x)/(a x) directly
f = sy.sin(t)/(t)
display(f)

# Show the Definite integral of f from 0 to inf
display(f.integrate((t, 0, sy.oo)))

# Get the Laplace Transform
F = L(f, t, s, noconds=True)
display(F)

# Do the Integral with the Laplace Transform
display(sy.lambdify((s), F, 'numpy')(0))
```

## Assignment

Your assignment is to take a problem from another class for which you had to use the Laplace Transform, describe it in the text cell below, then get the Laplace Transform of the equation using Sympy in the cell below that.

```python

```


# 2021-09-13 Variable coefficients

## Last time
* Conditioning and stability of polynomial interpolation
* Runge Phenomenon
* Chebyshev differentiation
* Chebyshev-based solver for the Poisson problem

## Today
* Variable coefficients
* Conservative/divergence form vs non-divergence forms
* Verification with discontinuities

```julia
using Plots
using LinearAlgebra

function vander(x, k=nothing)
    if k === nothing
        k = length(x)
    end
    V = ones(length(x), k)
    for j = 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function fdstencil(source, target, k)
    "kth derivative stencil from source to target"
    x = source .- target
    V = vander(x)
    rhs = zero(x)'
    rhs[k+1] = factorial(k)
    rhs / V
end

function poisson(x, spoints, forcing; left=(0, zero), right=(0, zero))
    n = length(x)
    L = zeros(n, n)
    rhs = forcing.(x)
    for i in 2:n-1
        jleft = min(max(1, i-spoints÷2), n-spoints+1)
        js = jleft : jleft + spoints - 1
        L[i, js] = -fdstencil(x[js], x[i], 2)
    end
    L[1,1:spoints] = fdstencil(x[1:spoints], x[1], left[1])
    L[n,n-spoints+1:n] = fdstencil(x[n-spoints+1:n], x[n], right[1])
    rhs[1] = left[2](x[1])
    rhs[n] = right[2](x[n])
    L, rhs
end

CosRange(a, b, n) = (a + b)/2 .+ (b - a)/2 * cos.(LinRange(-pi, 0, n))

function vander_chebyshev(x, n=nothing)
    if isnothing(n)
        n = length(x) # Square by default
    end
    m = length(x)
    T = ones(m, n)
    if n > 1
        T[:, 2] = x
    end
    for k in 3:n
        T[:, k] = 2 * x .* T[:,k-1] - T[:, k-2]
    end
    T
end

function chebdiff(x, n=nothing)
    T = vander_chebyshev(x, n)
    m, n = size(T)
    dT = zero(T)
    dT[:,2:3] = [one.(x) 4*x]
    for j in 3:n-1
        dT[:,j+1] = j * (2 * T[:,j] + dT[:,j-1] / (j-2))
    end
    ddT = zero(T)
    ddT[:,3] .= 4
    for j in 3:n-1
        ddT[:,j+1] = j * (2 * dT[:,j] + ddT[:,j-1] / (j-2))
    end
    T, dT, ddT
end
```

    chebdiff (generic function with 2 methods)

# Lagrange interpolating polynomials on `CosRange`

"nodal" --- "modal"

```julia
n = 5
x = CosRange(-1, 1, n)
xx = LinRange(-1, 1, 100)
Tx = vander_chebyshev(x)
Txx, dTxx, ddTxx = chebdiff(xx, n)
plot(xx, Txx / Tx)
scatter!(x, [one.(x) zero.(x)])
```

```julia
plot(xx, Txx)
```

# Solving a BVP on Chebyshev nodes

```julia
function poisson_cheb(n, rhsfunc, leftbc=(0, zero), rightbc=(0, zero))
    x = CosRange(-1, 1, n)
    T, dT, ddT = chebdiff(x)
    dT /= T
    ddT /= T
    T /= T
    L = -ddT
    rhs = rhsfunc.(x)
    for (index, deriv, func) in
            [(1, leftbc...), (n, rightbc...)]
        L[index,:] = (T, dT)[deriv+1][index,:]
        rhs[index] = func(x[index])
    end
    x, L, rhs
end
```

    poisson_cheb (generic function with 3 methods)

```julia
manufactured(x) = tanh(2x)
d_manufactured(x) = 2*cosh(2x)^-2
mdd_manufactured(x) = 8 * tanh(2x) / cosh(2x)^2
x, A, rhs = poisson_cheb(12, mdd_manufactured,
    (0, manufactured), (1, d_manufactured))
plot(x, A \ rhs, marker=:auto)
plot!(manufactured, legend=:bottomright)
```

# "spectral" (exponential) convergence

```julia
function poisson_error(n)
    x, A, rhs = poisson_cheb(n, mdd_manufactured, (0, manufactured), (1, d_manufactured))
    u = A \ rhs
    norm(u - manufactured.(x), Inf)
end

ns = 3:20
plot(ns, abs.(poisson_error.(ns)), marker=:auto, yscale=:log10)
ps = [1 2 3]
plot!([n -> n^-p for p in ps], label=map(p -> "\$n^{-$p}\$", ps))
```

# Variable coefficients

* Heat conduction: steel, brick, wood, foam
* Electrical conductivity: copper, rubber, air
* Elasticity: steel, rubber, concrete, rock
* Linearization of nonlinear materials
  * ketchup, glacier ice, rocks (mantle/lithosphere)

```julia
kappa_step(x) = .1 + .9 * (x > 0)
kappa_smooth(x) = .55 + .45 * sin(pi*x/2)
plot([kappa_step, kappa_smooth], xlims=(-1, 1), ylims=(0, 1), label="κ")
```

\begin{align}
-(\kappa u_x)_x &= 0 & u(-1) &= 0 & \kappa u_x(1) &= 1
\end{align}

* What physical scenario could this represent?
* Qualitatively, what would a solution look like?

# A naive finite difference solver

| Conservative (divergence) form | Non-divergence form |
| ------------------------------ | ------------------- |
| $-(\kappa u_x)_x = 0$           | $-\kappa u_{xx} - \kappa_x u_x = 0$ |

```julia
function poisson_nondivergence(x, spoints, kappa, forcing; leftbc=(0, zero), rightbc=(0, zero))
    n = length(x)
    L = zeros(n, n)
    rhs = forcing.(x)
    kappax = kappa.(x)
    for i in 2:n-1
        jleft = min(max(1, i-spoints÷2), n-spoints+1)
        js = jleft : jleft + spoints - 1
        kappa_x = fdstencil(x[js], x[i], 1) * kappax[js]
        L[i, js] = -fdstencil(x[js], x[i], 2) .* kappax[i] - fdstencil(x[js], x[i], 1) * kappa_x
    end
    L[1,1:spoints] = fdstencil(x[1:spoints], x[1], leftbc[1])
    if leftbc[1] == 1
        L[1, :] *= kappax[1]
    end
    L[n,n-spoints+1:n] = fdstencil(x[n-spoints+1:n], x[n], rightbc[1])
    if rightbc[1] == 1
        L[n, :] *= kappax[n]
    end
    rhs[1] = leftbc[2](x[1])
    rhs[n] = rightbc[2](x[n])
    L, rhs
end
```

    poisson_nondivergence (generic function with 1 method)

# Try it

```julia
x = LinRange(-1, 1, 30)
L, rhs = poisson_nondivergence(x, 3, kappa_smooth, zero, rightbc=(1, one))
u = L \ rhs
plot(x, u)
plot!(x -> 5*kappa_smooth(x))
```

# Manufactured solutions for variable coefficients

```julia
manufactured(x) = tanh(2x)
d_manufactured(x) = 2*cosh(2x)^-2
d_kappa_smooth(x) = .45*pi/2 * cos(pi*x/2)
flux_manufactured_kappa_smooth(x) = kappa_smooth(x) * d_manufactured(x)
function forcing_manufactured_kappa_smooth(x)
    8 * tanh(2x) / cosh(2x)^2 * kappa_smooth(x) -
     d_kappa_smooth(x) * d_manufactured(x)
end
x = LinRange(-1, 1, 20)
L, rhs = poisson_nondivergence(x, 3, kappa_smooth,
    forcing_manufactured_kappa_smooth,
    leftbc=(0, manufactured), rightbc=(1, flux_manufactured_kappa_smooth))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)

plot!([manufactured flux_manufactured_kappa_smooth forcing_manufactured_kappa_smooth kappa_smooth])
```

# Convergence

```julia
function poisson_error(n, spoints=3)
    x = LinRange(-1, 1, n)
    L, rhs = poisson_nondivergence(x, spoints, kappa_smooth,
        forcing_manufactured_kappa_smooth,
        leftbc=(0, manufactured), rightbc=(1, flux_manufactured_kappa_smooth))
    u = L \ rhs
    norm(u - manufactured.(x), Inf)
end
ns = 2 .^ (3:10)
plot(ns, poisson_error.(ns, 5), marker=:auto, xscale=:log10, yscale=:log10)
plot!([n -> n^-2, n -> n^-4], label=["\$1/n^2\$" "\$1/n^4\$"])
```

#  ✅ Verified!

# Let's try with the discontinuous coefficients

```julia
x = LinRange(-1, 1, 20)
L, rhs = poisson_nondivergence(x, 3, kappa_step,
    zero,
    leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

# Discretizing in conservative form

| Conservative (divergence) form | Non-divergence form |
| ------------------------------ | ------------------- |
| $-(\kappa u_x)_x = 0$           | $-\kappa u_{xx} - \kappa_x u_x = 0$ |

```julia
function poisson_conservative(n, kappa, forcing; leftbc=(0, zero), rightbc=(0, zero))
    x = LinRange(-1, 1, n)
    xstag = (x[1:end-1] + x[2:end]) / 2
    L = zeros(n, n)
    rhs = forcing.(x)
    kappa_stag = kappa.(xstag)
    for i in 2:n-1
        flux_L = kappa_stag[i-1] * fdstencil(x[i-1:i], xstag[i-1], 1)
        flux_R = kappa_stag[i] * fdstencil(x[i:i+1], xstag[i], 1)
        js = i-1:i+1
        weights = -fdstencil(xstag[i-1:i], x[i], 1)
        L[i, i-1:i+1] = weights[1] * [flux_L..., 0] + weights[2] * [0, flux_R...]
    end
    if leftbc[1] == 0
        L[1, 1] = 1
        rhs[1] = leftbc[2](x[1])
        rhs[2:end] -= L[2:end, 1] * rhs[1]
        L[2:end, 1] .= 0
    end
    if rightbc[1] == 0
        L[end,end] = 1
        rhs[end] = rightbc[2](x[end])
        rhs[1:end-1] -= L[1:end-1,end] * rhs[end]
        L[1:end-1,end] .= 0
    end
    x, L, rhs
end
```

    poisson_conservative (generic function with 1 method)

# Compare conservative vs non-divergence forms

```julia
forcing = zero # one
x, L, rhs = poisson_conservative(20, kappa_step,
    forcing, leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

```julia
x = LinRange(-1, 1, 200)
L, rhs = poisson_nondivergence(x, 3, kappa_step,
    forcing, leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

# Continuity of flux

```julia
forcing = zero
x, L, rhs = poisson_conservative(20, kappa_step,
    forcing, leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

```julia
xstag = (x[1:end-1] + x[2:end]) ./ 2
du = (u[1:end-1] - u[2:end]) ./ diff(x)
plot(xstag, [du .* kappa_step.(xstag)], marker=:auto, ylims=[-1, 1])
```

# Manufactured solutions with discontinuous coefficients

* We need to be able to evaluate derivatives of the flux $-\kappa u_x$.
* A physically-realizable solution would have continuous flux, but we we'd have to be making a physical solution to have that in verification.
* Idea: replace the discontinuous function with a continuous one with a rapid transition.

```julia
kappa_tanh(x, epsilon=.1) = .55 + .45 * tanh(x / epsilon)
d_kappa_tanh(x, epsilon=.1) = .45/epsilon * cosh(x/epsilon)^-2
plot([kappa_tanh])
```

# Solving with the smoothed step $\kappa$

```julia
kappa_tanh(x, epsilon=.01) = .55 + .45 * tanh(x / epsilon)
d_kappa_tanh(x, epsilon=.01) = .45/epsilon * cosh(x/epsilon)^-2
flux_manufactured_kappa_tanh(x) = kappa_tanh(x) * d_manufactured(x)
function forcing_manufactured_kappa_tanh(x)
    8 * tanh(2x) / cosh(2x)^2 * kappa_tanh(x) -
    d_kappa_tanh(x) * d_manufactured(x)
end
x, L, rhs = poisson_conservative(300, kappa_tanh,
    forcing_manufactured_kappa_tanh,
    leftbc=(0, manufactured), rightbc=(0, manufactured))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright, title="Error $(norm(u - manufactured.(x), Inf))")
plot!([manufactured flux_manufactured_kappa_tanh forcing_manufactured_kappa_tanh kappa_tanh])
```

# Convergence

```julia
function poisson_error(n, spoints=3)
    x, L, rhs = poisson_conservative(n, kappa_tanh,
        forcing_manufactured_kappa_tanh,
        leftbc=(0, manufactured), rightbc=(0, manufactured))
    u = L \ rhs
    norm(u - manufactured.(x), Inf)
end
ns = 2 .^ (3:10)
plot(ns, poisson_error.(ns, 3), marker=:auto, xscale=:log10, yscale=:log10)
plot!(n -> n^-2, label="\$1/n^2\$")
```


Integration on manifolds exercise in Sympy!
===========================================
Here I compute the scale factor $J(t)$ for the integration over the faces of a "cube mapped" sphere. That is
$$ F = (u,v,1)^t/\sqrt{u^2 + v^2 + 1}$$

Explanation:
------------
Following http://www.owlnet.rice.edu/~fjones/chap11.pdf

Let $M \in R^n$ be a manifold defined by map $F: A \rightarrow M$, where $A \in R^m, m<n$.
If $g$ is a real-valued function defined on $M$, its integral over $M$ is given by
$$ I = \int_M g(x) dx = \int_A g(F(t)) J(t) dt $$
with
$$ J(t) = \sqrt{det \{(DF)^t (DF)\}}$$
and the Jacobian matrix
$$ DF_{ij} = \partial F_i / \partial t_j$$

For probability densities we have
$$ I = \int_M p_X(x) dx = \int_A p_X(F(t)) J(t) dt = \int_A p_T(t) dt $$
Hence
$$ p_X = \frac{p_T}{J} $$

```python
from sympy import *
import sympy
print(sympy.__version__)
```

```python
u, v = Symbol('u'), Symbol('v')
```

```python
n = sqrt(u*u + v*v + 1.)
```

```python
print(n)
```

```python
x = u/n
y = v/n
z = 1/n
```

```python
import numpy as np
```

```python
DF = np.zeros((3,2), np.object)
for i, a in enumerate([x, y, z]):
    for j, b in enumerate([u, v]):
        DF[i,j] = simplify(diff(a, b))
DF = Matrix(DF)
DF
```

```python
DF.T
```

```python
J = sqrt((DF.T*DF).det())
```

```python
J = simplify(J)
```

```python
J
```

```python
J.subs([(u, 0), (v, 0)]).evalf()
```

```python
q = np.linspace(-1., 1., 21)
print(q)
```

```python
# let's check if the integration works by computing the
# area of a cube mapped face of the unit sphere.
# Of course the area should come out as 1/6 of the area of the full sphere.
du = (q[1]-q[0])
dv = du
def f(s, t):
    return J.subs([(u, q[s]), (v, q[t])]).evalf()
I = 0.
for s in range(len(q)-1):
    for t in range(len(q)-1):
        # 2D trapezoid quadrature.
        vol = (f(s, t) + f(s+1, t) + f(s,t+1) + f(s+1, t+1))*0.25*du*dv
        I += vol
```

```python
import math
print('Relative to exact area:', I/(4.*math.pi)*6.)
```

```python
from sympy.utilities.codegen import codegen
```

```python
x_, y_, z_ = symbols('x_ y_ z_')
[(c_name, c_code), (h_name, c_header)] = codegen(
    [("cube_map_J", J), ('cube_map_F', [Eq(x_,x), Eq(y_,y), Eq(z_,z)])], "C", "test", header=False, empty=False)
```

```python
print(c_code)
```

```python

```


# Computational Astrophysics
## Ordinary Differential Equations.
## Application 01. Motion of a Comet

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we solve the equations of motion for a comet around the Sun using some integration methods.

`A. Garcia. Numerical Methods for Physics. (1999). Chapter 3 `

---

## A Comet-Sun System

In this application problem, we will describe the motion of a comet orbiting around the Earth. We will consider that the comet has a mass $m$ very small compared to Sun's mass $M$, i.e. $m \ll M$. This assumption implies that Sun will remain static as the comet orbits under the influence of its gravity. For example, the mass of Halley's comet is $m \approx 2.2 \times 10^{14}$ kg which is a very small quantity compared with Sun's mass $M \approx 1.9 \times 10^{30}$ kg.

---
The equations of motion of the comet, in cartesian coordinates with origin on the Sun's center, are

\begin{align}
\frac{d^2 x}{dt^2} = &- \frac{GM}{(x^2 + y^2)^{3/2}} x \\
\frac{d^2 y}{dt^2} = &- \frac{GM}{(x^2 + y^2)^{3/2}} y ,
\end{align}

where we restricted the problem to the $xy$-plane. In order to solve this ODEs system, we will transform it into a first-order differential system by introducing two new functions,

\begin{align}
\frac{d x}{dt} = &u \\
\frac{d y}{dt} = &v \\
\frac{du}{dt} = &- \frac{GM}{(x^2 + y^2)^{3/2}} x \\
\frac{dv}{dt} = &- \frac{GM}{(x^2 + y^2)^{3/2}} y
\end{align}

An important consideration to take into account involves the big numbers involved in the problem such as the Sun's mass and the distances, which will be of the order of $1$ au. Hence, it is convenient to use a system of units with

- unit of mass: Solar mass
- units of distance: au
- units of time: years

Hence, the Newtonian gravitational constant will have the value $G = 4\pi^2$,

```python
import astropy.constants as const
import astropy.units as units

G = const.G.to('au^3/(M_sun year^2)')
G
```

$39.476926 \; \mathrm{\frac{AU^{3}}{M_{\odot}\,yr^{2}}}$

The ODEs system is implemented by defining the function

```python
import numpy as np

# Newtonian Gravitational Constant
G = 4.*np.pi**2
M = 1.

def ODE(t, q0):
    '''
    ------------------------------------------
    ODE(t,q0)
    ------------------------------------------
    ODEs system for the motion of a comet
    around the Sun using cartesian coordinates
    in the orbital plane.
    Arguments:
    t: time parameter (not necessary for
       Newtonian problem)
    q0: numpy array with the initial condition
        data:
        q0[0] = x
        q0[1] = y
        q0[2] = dx/dt
        q0[3] = dy/dt
    ------------------------------------------
    '''
    x = q0[0]
    y = q0[1]
    u = q0[2]
    v = q0[3]
    f = np.zeros(4)
    f[0] = u
    f[1] = v
    f[2] = - G*M*x/(x**2 + y**2)**(3/2)
    f[3] = - G*M*y/(x**2 + y**2)**(3/2)
    return f

```

---
## Initial Condition

The initial condition is chosen so that at $t=0$, the coordinates will be $x = 1 $ au and $y=0$. We will chose an arbitrary initial velocity for this first attempt, say $u=0$ and $v=1$ au/yr.

The time grid to apply the integration method will be defined from $t_0 = 0$ to $t_f = 10$ years.

## Soving the ODEs System

Using the Forward Euler and the RK4 algorithms to solve the differential problem, we obtain

```python
from RK4 import *

def FEuler(h, t0, q0):
    '''
    ------------------------------------------
    FEuler(h, t0, q0)
    ------------------------------------------
    Forward Euler's method for solving a ODEs
    system.
    Arguments:
    h: stepsize for the iteration
    t0: independent parameter initial value
    q0: numpy array with the initial values of
        the functions in the ODEs system
    ------------------------------------------
    '''
    f = ODE(t0, q0)
    q1 = q0 + h*f
    return q1

# Creation of the time grid (in years)
t_0 = 0.
t_f = 20.

# Number of steps in the grid
n = 100000

# Constant stepsize defined by the number of steps in the grid
h = (t_f - t_0)/n

# Arrays to store the solution
t = np.linspace(t_0, t_f, n) # Time information
QE = np.zeros([4,n]) # Euler's Method information
QR = np.zeros([4,n]) # RK4's Method information

# Initial Conditions
QE[0,0] = 1.
QE[1,0] = 0.
QE[2,0] = 0.
QE[3,0] = 3.

QR[0,0] = 1.
QR[1,0] = 0.
QR[2,0] = 0.
QR[3,0] = 3.

# Main loops for solving the problem
for i in range(1,n):
    q0 = QE[:,i-1]
    qf = FEuler(h, 0, q0)
    QE[:,i] = qf[:]

for i in range(1,n):
    q0 = QR[:,i-1]
    qf = RK4(ODE, h, 0, q0)
    QR[:,i] = qf[:]

```

```python
import matplotlib.pyplot as plt
%matplotlib inline

fig, ax = plt.subplots(1,2, figsize=(14,5))
ax[0].plot(QE[0,:], QE[1,:], color='crimson', label=f'$h=$ {h:.2e}')
ax[0].set_title('Forward Euler Method')
ax[0].set_xlabel(r'$x$')
ax[0].set_ylabel(r'$y$')
ax[0].legend()

ax[1].plot(QR[0,:], QR[1,:], color='cornflowerblue', label=f'$h=$ {h:.2e}')
ax[1].set_title('RK4 Method')
ax[1].set_xlabel(r'$x$')
ax[1].set_ylabel(r'$y$')
ax[1].legend()
plt.show()
```

It is clear that the forward Euler's method is unstable with the chosen parameters. With a smaller value of the stepsize, the behavior of the Euler's method solution is improved,

```python
# Creation of the time grid (in years)
t_0 = 0.
t_f = 5.

# Number of steps in the grid (1 millon points!)
nI = 1000000

# Constant stepsize defined by the number of steps in the grid
hI = (t_f - t_0)/nI

# Arrays to store the solution
tI = np.linspace(t_0, t_f, nI) # Time information
QEI = np.zeros([4,nI]) # Euler's Method information

# Initial Conditions
QEI[0,0] = 1.
QEI[1,0] = 0.
QEI[2,0] = 0.
QEI[3,0] = 3.

# Main loops for solving the problem
for i in range(1,nI):
    q0 = QEI[:,i-1]
    qf = FEuler(hI, 0, q0)
    QEI[:,i] = qf[:]

```

```python
fig, ax = plt.subplots()
ax.plot(QEI[0,:], QEI[1,:], color='crimson', label=f'$h=$ {hI:.5e}')
ax.set_title('Forward Euler Method')
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'$y$')
ax.legend()

plt.show()
```

---

The bad behavior for the forward Euler's method can be appreciated by calculating the conserved quantities of the problem (in this case energy and angular momentum). A plot is useful to show this behavior:

```python
EnergyE = np.zeros(n)
AngMomE = np.zeros(n)

EnergyR = np.zeros(n)
AngMomR = np.zeros(n)

for i in range(n):
    speed2 = QE[2,i]**2 + QE[3,i]**2
    r = np.sqrt(QE[0,i]**2 + QE[1,i]**2)
    EnergyE[i] = speed2/2 - G*M/r
    AngMomE[i] = QE[0,i]*QE[3,i] - QE[1,i]*QE[2,i]

for i in range(n):
    speed2 = QR[2,i]**2 + QR[3,i]**2
    r = np.sqrt(QR[0,i]**2 + QR[1,i]**2)
    EnergyR[i] = speed2/2 - G*M/r
    AngMomR[i] = QR[0,i]*QR[3,i] - QR[1,i]*QR[2,i]

fig, ax = plt.subplots(2,2, figsize=(14,10))
ax[0,0].plot(QE[0,:], QE[1,:], color='crimson', label=f'$h=$ {h:.2e}')
ax[0,0].set_title('Forward Euler Method')
ax[0,0].set_xlabel(r'$x$')
ax[0,0].set_ylabel(r'$y$')
ax[0,0].legend()

ax[0,1].plot(QR[0,:], QR[1,:], color='cornflowerblue', label=f'$h=$ {h:.2e}')
ax[0,1].set_title('RK4 Method')
ax[0,1].set_xlabel(r'$x$')
ax[0,1].set_ylabel(r'$y$')
ax[0,1].legend()

ax[1,0].plot(t, EnergyE, color='mediumslateblue', label=f'Energy')
ax[1,0].plot(t, AngMomE, color='steelblue', label=f'Angular Momentum')
ax[1,0].set_title('Forward Euler Method')
ax[1,0].set_xlabel(r'$t$')
ax[1,0].set_ylabel(r'$E, \ell$')
ax[1,0].legend()

ax[1,1].plot(t, EnergyR, color='mediumslateblue', label=f'Energy')
ax[1,1].plot(t, AngMomR, color='steelblue', label=f'Angular Momentum')
ax[1,1].set_title('RK4 Method')
ax[1,1].set_xlabel(r'$t$')
ax[1,1].set_ylabel(r'$E, \ell$')
ax[1,1].legend()

plt.show()
```

These plots show clearly that the forward Euler's method does not satisfies the conservation of energy (and produces a small increase in the angular momentum), while the RK4 do have a very good behavior.

```python
print('The change in energy for Euler\'s method is :', np.abs(EnergyE[n-1] - EnergyE[0]))
print('\nThe change in energy for RK4 method is :', np.abs(EnergyR[n-1] - EnergyR[0]))
```

    The change in energy for Euler's method is : 34.01224429334252

    The change in energy for RK4 method is : 1.1176754846076165e-05

Now we will concentrate in the RK4 algorithm. Using a grid with $100000$ the computation time is

```python
from RK4 import *

# Creation of the time grid (in years)
t_0 = 0.
t_f = 20.

# Number of steps in the grid
n = 100000

# Constant stepsize defined by the number of steps in the grid
h = (t_f - t_0)/n

# Arrays to store the solution
t = np.linspace(t_0, t_f, n) # Time information
Q = np.zeros([4,n]) # RK4's Method information

# Initial Conditions
Q[0,0] = 1.
Q[1,0] = 0.
Q[2,0] = 0.
Q[3,0] = 3.

# Computation time
import time
start = time.time()

# Main loops for solving the problem
for i in range(1,n):
    q0 = Q[:,i-1]
    qf = RK4(ODE, h, 0, q0)
    Q[:,i] = qf[:]

end = time.time()
print('The elapsed time was:', end - start)
```

    The elapsed time was: 5.354465961456299

```python
Energy = np.zeros(n)
AngMom = np.zeros(n)

for i in range(n):
    speed2 = Q[2,i]**2 + Q[3,i]**2
    r = np.sqrt(Q[0,i]**2 + Q[1,i]**2)
    Energy[i] = speed2/2 - G*M/r
    AngMom[i] = Q[0,i]*Q[3,i] - Q[1,i]*Q[2,i]

fig, ax = plt.subplots(1,2, figsize=(14,5))

ax[0].plot(Q[0,:], Q[1,:], color='cornflowerblue', label=f'$h=$ {h:.2e}')
ax[0].set_title('RK4 Method')
ax[0].set_xlabel(r'$x$')
ax[0].set_ylabel(r'$y$')
ax[0].legend()

ax[1].plot(t, Energy, color='mediumslateblue', label=f'Energy')
ax[1].set_title('RK4 Method')
ax[1].set_xlabel(r'$t$')
ax[1].set_ylabel(r'$E, \ell$')
ax[1].legend()

plt.show()
```

Note that the energy is not a constant, but shows a small continuos change that may accumulate to an important value over long intervals of time. For this time interval, the difference between final and initial values of energy is

```python
np.abs(Energy[n-1]- Energy[0])
```

    1.1176754846076165e-05

## Adaptative Rung-Kutta

In order to improve the solution, giving a smaller change in the energy of the system, we will implement an adaptative RK algorithm. This method will change the stepsize according to a given error tolerance.

```python
from AdaptativeRK import *

# Creation of the time grid (in years)
t_0 = 0.
t_f = 20.

# Initial stepsize (we take the value used in the RK4 method above)
h = 1E-4

# Arrays to store the solution
t = np.zeros(1) # Time information
Q = np.zeros([4,1]) # RK4's Method information

# Initial Conditions
t[0] = t_0
Q[0,0] = 1.
Q[1,0] = 0.
Q[2,0] = 0.
Q[3,0] = 3.

# Computation time
import time
start = time.time()

# Counter to display the advance in calculations
Tpercentage = 0.1

# Main loops for solving the problem
i = 1
while t[len(t)-1]<t_f:
    q0 = Q[:,i-1]
    h, qf = ARK(ODE, h, 0., q0)
    Q = np.append(Q, np.reshape(qf, (4,1)), axis=1)
    t = np.append(t, t[i-1]+ h)
    if t[i]>(t_f*Tpercentage):
        print(Tpercentage*100, '% completed')
        Tpercentage +=0.1
    i=i+1

end = time.time()
print('The elapsed time was:', end - start)
```

    10.0 % completed
    20.0 % completed
    30.000000000000004 % completed
    40.0 % completed
    50.0 % completed
    60.0 % completed
    70.0 % completed
    80.0 % completed
    89.99999999999999 % completed
    99.99999999999999 % completed
    The elapsed time was: 878.3249571323395

```python
n = len(t)
Energy = np.zeros(n)
AngMom = np.zeros(n)

for i in range(n):
    speed2 = Q[2,i]**2 + Q[3,i]**2
    r = np.sqrt(Q[0,i]**2 + Q[1,i]**2)
    Energy[i] = speed2/2 - G*M/r
    AngMom[i] = Q[0,i]*Q[3,i] - Q[1,i]*Q[2,i]

fig, ax = plt.subplots(1,2, figsize=(14,5))

ax[0].plot(Q[0,:], Q[1,:], color='cornflowerblue', label=f'$h=$ {h:.2e}')
ax[0].set_title('RK4 Method')
ax[0].set_xlabel(r'$x$')
ax[0].set_ylabel(r'$y$')
ax[0].legend()

ax[1].plot(t, Energy, color='mediumslateblue', label=f'Energy')
ax[1].set_title('RK4 Method')
ax[1].set_xlabel(r'$t$')
ax[1].set_ylabel(r'$E, \ell$')
ax[1].legend()

plt.show()
```

```python
np.abs(Energy[n-1] - Energy[0])
```

    3.2153850852978394e-06

Note that the conservation of energy is improved but the computation time increases considerably.

The RK4 algorithm evaluates $100000$ grid points in $5.71$ s., with change in the energy of $1.12\times 10^{-05}$.

The adaptive RK algorithm, using the error tolerance $\epsilon = 1\times 10^{-11}$ and a fudge factor $S=0.999$, evaluated $326089$ points with a variation of the energy between the initial and final values of $3.22\times 10^{-6}$. However, the computation time increased to $935.43$ s. or approximately $15.6$ minutes.

```python

```


# First Order Initial Value Problem

The more general form of a first order Ordinary Differential Equation is:
\begin{equation}
y^{'}=f(t,y).
\end{equation}
This can be solved analytically by integrating both sides but this is not straight forward for most problems.
Numerical methods can be used to approximate the solution at discrete points.

## Euler method

The simplest one step numerical method is the Euler Method named after the most prolific of mathematicians [Leonhard Euler](https://en.wikipedia.org/wiki/Leonhard_Euler) (15 April 1707 – 18 September 1783) .

The general Euler formula for the first order differential equation
\begin{equation}
y^{'} = f(t,y),
\end{equation}

approximates the derivative at time point $t_i$,

\begin{equation}
y^{'}(t_i) \approx \frac{w_{i+1}-w_i}{t_{i+1}-t_{i}},
\end{equation}

where $w_i$ is the approximate solution of $y$ at time $t_i$.

This substitution changes the differential equation  into a __difference__ equation of the form

\begin{equation}
\frac{w_{i+1}-w_i}{t_{i+1}-t_{i}}=f(t_i,w_i).
\end{equation}

Assuming uniform stepsize $t_{i+1}-t_{i}$ is replaced by $h$, re-arranging the equation gives
\begin{equation}
w_{i+1}=w_i+hf(t_i,w_i).
\end{equation}
 This can be read as the future $w_{i+1}$ can be approximated by the present $w_i$ and the addition of the input to the system $f(t,y)$ times the time step.

```python
## Library
import numpy as np
import math
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

## Population growth

The general form of the population growth differential equation is:
\begin{equation}
y^{'}=\epsilon y
\end{equation}
where $\epsilon$ is the growth rate. The initial population at time $a$ is
\begin{equation}
y(a)=A,
 \end{equation}
\begin{equation}
 a\leq t \leq b.
\end{equation}
Integrating gives  the general analytic (exact) solution:
\begin{equation}
 y=Ae^{\epsilon x}.
\end{equation}
We will use this equation to illustrate the application of the Euler method.

## Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ intervals by a constant stepsize
\begin{equation}
h=\frac{b-a}{N}.
\end{equation}
Here the interval is $0\leq t \leq 2$ is discretised into $20$ intervals with stepsize
\begin{equation}
 h=\frac{2-0}{20}=0.1,
\end{equation}
this gives the 21 discrete points:
\begin{equation}
 t_0=0, \ t_1=0.1, \ ... t_{20}=2.
\end{equation}
This is generalised to
\begin{equation}
t_i=0+i0.1, \ \ \ i=0,1,...,20.
\end{equation}
The plot below shows the discrete time steps.

```python
### Setting up time
t_end=2.0
t_start=0
N=20
h=(t_end-t_start)/(N)
time=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(time,0*time,'o:',color='red')
plt.xlim((0,2))
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.plot();
```

## Initial Condition
To get a specify solution to a first order initial value problem, an __initial condition__ is required.

For our population problem the intial condition is:
\begin{equation}
y(0)=10.
\end{equation}
This gives the analytic solution
\begin{equation}
y=10e^{\epsilon t}.
\end{equation}
### Growth rate
Let the growth rate
\begin{equation}
\epsilon=0.5
\end{equation}
giving the analytic solution.
\begin{equation}
y=10e^{0.5 t}.
\end{equation}
The plot below shows the exact solution on the discrete time steps.

```python
## Analytic Solution y
y=10*np.exp(0.5*time)

fig = plt.figure(figsize=(10,4))
plt.plot(time,y,'o:',color='black')
plt.xlim((0,2))
plt.xlabel('time')
plt.ylabel('y')
plt.title('Analytic (Exact) solution')
plt.plot();
```

## Numerical approximation of Population growth
The differential equation is transformed using the Euler method into a difference equation of the form
\begin{equation} w_{i+1}=w_{i}+h \epsilon w_i. \end{equation}
This approximates a series of of values $w_0, \ w_1, \ ..., w_{N}$.
For the specific example of the population equation the difference equation is
     \begin{equation} w_{i+1}=w_{i}+h 0.5 w_i. \end{equation}
where $w_0=10$. From this initial condition the series is approximated.
The plot below shows the exact solution $y$ in black circles and Euler approximation $w$ in blue squares.

```python
w=np.zeros(N+1)
w[0]=10
for i in range (0,N):
    w[i+1]=w[i]+h*(0.5)*w[i]

fig = plt.figure(figsize=(10,4))
plt.plot(time,y,'o:',color='black',label='exact')
plt.plot(time,w,'s:',color='blue',label='Euler')
plt.xlim((0,2))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Analytic and Euler solution')
plt.plot();
```

## Numerical Error
With a numerical solution there are two types of error:
* local truncation error at one time step;
* global error which is the propagation of local error.

### Derivation of  Euler Local truncation error
The left hand side of a initial value problem $\frac{dy}{dt}$ is approximated by __Taylors theorem__ expand about a point $t_0$ giving:
\begin{equation}
y(t_1) = y(t_0)+(t_1-t_0)y^{'}(t_0) + \frac{(t_1-t_0)^2}{2!}y^{''}(\xi), \ \ \ \ \ \ \xi \in [t_0,t_1].
\end{equation}
Rearranging and letting $h=t_1-t_0$ the equation becomes
\begin{equation}
y^{'}(t_0)=\frac{y(t_1)-y(t_0)}{h}-\frac{h}{2}y^{''}(\xi).
\end{equation}
From this the local truncation error is
\begin{equation}
\tau \leq \frac{h}{2}M  \sim O(h),
\end{equation}
where $y^{''}(t) \leq M $.
#### Derivation of  Euler Local truncation error for the Population Growth
In most cases $y$ is unknown but in our example problem there is an exact solution which can be used to estimate the local truncation
\begin{equation}
y'(t)=5e^{0.5 t},
\end{equation}
\begin{equation}
y''(t)=2.5e^{0.5 t}.
\end{equation}
From this a maximum upper limit can be calculated for $y^{''} $ on the interval $[t_0,t_1]=[0,0.1]$
\begin{equation}
y''(0.1)=2.5e^{0.1\times 0.5}=2.63=M,
\end{equation}
\begin{equation}
\tau=\frac{h}{2}2.63=0.1315.
\end{equation}
The plot below shows the exact local truncation error $|y-w|$ (red triangle) and the upper limit of the Truncation error (black v) for the first two time points $t_0$ and $t_1$.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time[0:2],np.abs(w[0:2]-y[0:2]),'^:'
         ,color='red',label='Error |y-w|')
plt.plot(time[0:2],0.1*2.63/2*np.ones(2),'v:'
         ,color='black',label='Upper Local Truncation')
plt.xlim((0,.15))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Local Truncation Error')
plt.plot();
```

## Global Error
The error does not stay constant accross the time this is illustrated in the figure below for the population growth equation. The actual error (red triangles) increases over time while the local truncation error (black v) remains constant.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time,np.abs(w-y),'^:'
         ,color='red',label='Error |y-w|')
plt.plot(time,0.1*2.63/2*np.ones(N+1),'v:'
         ,color='black',label='Upper Local Truncation')
plt.xlim((0,2))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Why Local Truncation does not extend to global')
plt.plot();
```

## Theorems
The theorem below proves an upper limit of the global truncation error.
### Euler Global Error
__Theorem Global Error__

Suppose $f$ is continuous and satisfies a Lipschitz Condition with constant
L on $D=\{(t,y)|a\leq t \leq b, -\infty < y < \infty \}$ and that a constant M
exists with the property that
\begin{equation}
|y^{''}(t)|\leq M.
\end{equation}
Let $y(t)$ denote the unique solution of the Initial Value Problem

\begin{equation}
y^{'}=f(t,y) \ \ \ a\leq t \leq b, \ \ \ y(a)=\alpha,
\end{equation}
and $w_0,w_1,...,w_N$ be the approx generated by the Euler method for some
positive integer N.  Then for $i=0,1,...,N$
\begin{equation}
|y(t_i)-w_i| \leq \frac{Mh}{2L}|e^{L(t_i-a)}-1|.
\end{equation}

### Theorems about Ordinary Differential Equations
__Definition__

A function $f(t,y)$ is said to satisfy a __Lipschitz Condition__ in the variable $y$ on
the set $D \subset R^2$ if a constant $L>0$ exist with the property that
\begin{equation}
|f(t,y_1)-f(t,y_2)| < L|y_1-y_2|
\end{equation}
whenever $(t,y_1),(t,y_2) \in D$.  The constant L is call the Lipschitz Condition
of $f$.

__Theorem__
Suppose $f(t,y)$ is defined on a convex set $D \subset R^2$. If a constant
$L>0$ exists with
\begin{equation}
\left|\frac{\partial f(t,y)}{\partial y}\right|\leq L,
\end{equation}
then $f$ satisfies a Lipschitz Condition an $D$ in the variable $y$ with
Lipschitz constant L.

### Global truncation error for the population equation
For the population equation specific values $L$ and $M$ can be calculated.

In this case $f(t,y)=\epsilon y$ is continuous and satisfies a Lipschitz Condition with constant
\begin{equation} \left|\frac{\partial f(t,y)}{\partial y}\right|\leq L, \end{equation}
\begin{equation} \left|\frac{\partial \epsilon y}{\partial y}\right|\leq \epsilon=0.5=L, \end{equation}

on $D=\{(t,y)|0\leq t \leq 2, 10 < y < 30 \}$ and that a constant $M$
exists with the property that
\begin{equation} |y^{''}(t)|\leq M. \end{equation}
\begin{equation} |y^{''}(t)|=2.5e^{0.5\times 2} \leq 2.5 e=6.8. \end{equation}

__Specific Theorem Global Error__

Let $y(t)$ denote the unique solution of the Initial Value Problem
\begin{equation} y^{'}=0.5 y, \ \ \ 0\leq t \leq 2, \ \ \ y(0)=10, \end{equation}
and $w_0,w_1,...,w_N$ be the approx generated by the Euler method for some
positive integer N.  Then for $i=0,1,...,N$
\begin{equation} |y(t_i)-w_i| \leq \frac{6.8 h}{2\times 0.5}|e^{0.5(t_i-0)}-1| \end{equation}

The figure below shows the exact error $y-w$ in red triangles and the upper global error in black x's.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time,np.abs(w-y),'^:'
         ,color='red',label='Error |y-w|')
plt.plot(time,0.1*6.8*(np.exp(0.5*time)-1),'x:'
         ,color='black',label='Upper Global Truncation')
plt.xlim((0,2))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Global Truncation Error')
plt.plot();
```

### Table
The table below shows the iteration $i$, the discrete time point t[i], the Euler approximation w[i] of the solution $y$, the exact error $|y-w|$ and the upper limit of the global error for the linear population equation.

```python

d = {'time t_i': time[0:10],    'Euler (w_i) ':w[0:10],'Exact (y)':y[0:10],'Exact Error (|y_i-w_i|)':np.round(np.abs(w[0:10]-y[0:10]),10),r'Global Error ':np.round(0.1*6.8*(np.exp(0.5*time[0:10])-1),20)}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Euler (w_i)
      Exact (y)
      Exact Error (|y_i-w_i|)
      Global Error

      0
      0.0
      10.000000
      10.000000
      0.000000
      0.000000

      1
      0.1
      10.500000
      10.512711
      0.012711
      0.034864

      2
      0.2
      11.025000
      11.051709
      0.026709
      0.071516

      3
      0.3
      11.576250
      11.618342
      0.042092
      0.110047

      4
      0.4
      12.155062
      12.214028
      0.058965
      0.150554

      5
      0.5
      12.762816
      12.840254
      0.077439
      0.193137

      6
      0.6
      13.400956
      13.498588
      0.097632
      0.237904

      7
      0.7
      14.071004
      14.190675
      0.119671
      0.284966

      8
      0.8
      14.774554
      14.918247
      0.143693
      0.334441

      9
      0.9
      15.513282
      15.683122
      0.169840
      0.386452

```python

```

Chapter 01 - Euler Methods/101_Euler_method_with_Theorems_Growth_function.ipynb