# DiffEqBiological Tutorial I: Introduction
In this tutorial we'll provide an introduction to using DiffEqBiological to specify chemical reaction networks, and then to solve ODE, jump, tau-leaping and SDE models for them. Let's start by using the DiffEqBiological `reaction_network` DSL to specify a simply chemical reaction network; the well-known Repressilator.

We first import the basic packages we'll need, and use PyPlot for making figures:

```julia
using DifferentialEquations, DiffEqBiological, Plots, Latexify;
pyplot();
```

We next construct the reaction network. The basic types of arrows and predefined rate laws one can use are discussed in the DiffEqBiological [documentation](http://docs.juliadiffeq.org/latest/models/biological.html). Here we use a mix of first order, zero order and repressive Hill function rate laws. Note, $\varnothing$ corresponds to the empty state, and is used for zeroth order production and first order degradation reactions:

```julia
repressilator = @reaction_network begin
    hillr(P₃,α,K,n), ∅ --> m₁
    hillr(P₁,α,K,n), ∅ --> m₂
    hillr(P₂,α,K,n), ∅ --> m₃
    (δ,γ), m₁ ↔ ∅
    (δ,γ), m₂ ↔ ∅
    (δ,γ), m₃ ↔ ∅
    β, m₁ --> m₁ + P₁
    β, m₂ --> m₂ + P₂
    β, m₃ --> m₃ + P₃
    μ, P₁ --> ∅
    μ, P₂ --> ∅
    μ, P₃ --> ∅
end α K n δ γ β μ;
```

We can use Latexify to look at the corresponding reactions

```julia
latexify(repressilator; env=:chemical)
```

\begin{align}
\require{mhchem}
\ce{ \varnothing &->[\frac{\alpha \cdot K^{n}}{K^{n} + P_3^{n}}] m_{1}}\\
\ce{ \varnothing &->[\frac{\alpha \cdot K^{n}}{K^{n} + P_1^{n}}] m_{2}}\\
\ce{ \varnothing &->[\frac{\alpha \cdot K^{n}}{K^{n} + P_2^{n}}] m_{3}}\\
\ce{ m_{1} &<=>[\delta][\gamma] \varnothing}\\
\ce{ m_{2} &<=>[\delta][\gamma] \varnothing}\\
\ce{ m_{3} &<=>[\delta][\gamma] \varnothing}\\
\ce{ m_{1} &->[\beta] m_{1} + P_{1}}\\
\ce{ m_{2} &->[\beta] m_{2} + P_{2}}\\
\ce{ m_{3} &->[\beta] m_{3} + P_{3}}\\
\ce{ P_{1} &->[\mu] \varnothing}\\
\ce{ P_{2} &->[\mu] \varnothing}\\
\ce{ P_{3} &->[\mu] \varnothing}
\end{align}

or to look at the corresponding ODEs

```julia
latexify(repressilator)
```

\begin{align}
\frac{dm_1}{dt} =& \frac{\alpha \cdot K^{n}}{K^{n} + P_3^{n}} - \delta \cdot m_1 + \gamma + 0 \\
\frac{dm_2}{dt} =& \frac{\alpha \cdot K^{n}}{K^{n} + P_1^{n}} - \delta \cdot m_2 + \gamma + 0 \\
\frac{dm_3}{dt} =& \frac{\alpha \cdot K^{n}}{K^{n} + P_2^{n}} - \delta \cdot m_3 + \gamma + 0 \\
\frac{dP_1}{dt} =& \beta \cdot m_1 - \mu \cdot P_1 \\
\frac{dP_2}{dt} =& \beta \cdot m_2 - \mu \cdot P_2 \\
\frac{dP_3}{dt} =& \beta \cdot m_3 - \mu \cdot P_3 \\
\end{align}

To solve the ODEs we need to specify the values of the parameters in the model, the initial condition, and the time interval to solve the model on. To do this it helps to know the orderings of the parameters and the species. Parameters are ordered in the same order they appear after the `end` statement in the `@reaction_network` macro. Species are ordered in the order they first appear within the `@reaction_network` macro. We can see these orderings using the `speciesmap` and `paramsmap` functions:

```julia
speciesmap(repressilator)
```

    OrderedCollections.OrderedDict{Symbol,Int64} with 6 entries:
      :m₁ => 1
      :m₂ => 2
      :m₃ => 3
      :P₁ => 4
      :P₂ => 5
      :P₃ => 6

```julia
paramsmap(repressilator)
```

    OrderedCollections.OrderedDict{Symbol,Int64} with 7 entries:
      :α => 1
      :K => 2
      :n => 3
      :δ => 4
      :γ => 5
      :β => 6
      :μ => 7

## Solving the ODEs:
Knowing these orderings, we can create parameter and initial condition vectors, and setup the `ODEProblem` we want to solve:

```julia
# parameters [α,K,n,δ,γ,β,μ]
p = (.5, 40, 2, log(2)/120, 5e-3, 20*log(2)/120, log(2)/60)

# initial condition [m₁,m₂,m₃,P₁,P₂,P₃]
u0 = [0.,0.,0.,20.,0.,0.]

# time to solve over
tspan = (0., 10000.)

# create the ODEProblem we want to solve
oprob = ODEProblem(repressilator, u0, tspan, p)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 10000.0)
    u0: [0.0, 0.0, 0.0, 20.0, 0.0, 0.0]

At this point we are all setup to solve the ODEs. We can now use any ODE solver from within the DiffEq package. We'll just use the default DifferentialEquations solver for now, and then plot the solutions:

```julia
sol = solve(oprob, saveat=10.)
plot(sol, fmt=:svg)
```

So we see the well-known oscillatory behavior of the repressilator! For more on choices of ODE solvers, see the JuliaDiffEq [documentation]().

---

## Stochastic Simulation Algorithms (SSAs) for Stochastic Chemical Kinetics
Let's now look at a stochastic version of the repressilator, modeling it with jump processes. Here we will construct a `JumpProblem` that uses Gillespie's Direct method, and then solve it to generate one realization of the jump process:

```julia
# first we redefine the initial condition to be integer valued
u0 = [0,0,0,20,0,0]

# first we create a discrete problem to encode that our species are integer valued:
dprob = DiscreteProblem(repressilator, u0, tspan, p)

# now we create a JumpProblem, and specify Gillespie's Direct Method as the solver:
jprob = JumpProblem(dprob, Direct(), repressilator, save_positions=(false,false))

# now let's solve and plot the jump process:
sol = solve(jprob, SSAStepper(), saveat=10.)
plot(sol, fmt=:svg)
```

Here we see that oscillations remain, but become much noiser. Note, in constructing the `JumpProblem` we could have used any of the SSAs that are part of DiffEqJump instead of the `Direct` method.

---
## $\tau$-leaping Methods:
While SSAs generate exact realizations for stochastic chemical kinetics jump process models, [$\tau$-leaping](https://en.wikipedia.org/wiki/Tau-leaping) methods offer a performant alternative by discretizing in time the underlying time-change representation of the stochastic process. The DiffEqJump package has limited support for $\tau$-leaping methods in the form of the basic Euler's method type approximation proposed by Gillespie. We can simulate the repressilator by constructing a `JumpProblem` from the `reaction_network.regular_jumps` field:

```julia
lprob = JumpProblem(dprob, Direct(), repressilator.regular_jumps)
lsol = solve(lprob, SimpleTauLeaping(), dt=.1)
plot(lsol, plotdensity=1000, fmt=:svg)
```

---
## Chemical Langevin Equation (CLE) Stochastic Differential Equation (SDE) Models:
At an intermediary physical scale between macroscopic ODE models and microscopic stochastic chemical kinetic models lies the CLE, a SDE version of the model. The SDEs add to each ODE above a noise term. As the repressilator has species that get very close to zero in size, it is not a good candidate to model with the CLE (where solutions can then go negative and become unphysical). Let's create a simpler reaction network for a birth-death process that will stay non-negative:

```julia
bdp = @reaction_network begin
  c₁, X --> 2X
  c₂, X --> 0
  c₃, 0 --> X
end c₁ c₂ c₃
p = (1.0,2.0,50.)
u₀ = [5.]
tspan = (0.,4.);
```

The corresponding Chemical Langevin Equation SDE is then
```math
dX_t = \left(c₁ X - c₂ X + c₃ \right) dt + \left( \sqrt{c₁ X} - \sqrt{c₂ X} + \sqrt{c₃} \right)dW_t,
```
where $W_t$ denotes a standard Brownian Motion. We can solve the CLE SDE model by creating an SDEProblem and solving it similar to what we did for ODEs above:

```julia
# SDEProblem for CLE
sprob = SDEProblem(bdp, u₀, tspan, p)

# solve and plot
sol = solve(sprob, saveat=.004)
plot(sol, fmt=:svg)
```

---
## What information the `reaction_network` provides:
The generated `reaction_network` contains a lot of basic information about the network. For example
- `repressilator.f` is a function `f(du,u,p,t)` that given the current state vector `u` and time `t` returns the a vector of the time derivatives for the ODEs within `du`.
- `repressilator.jac` is a function `jac(J,u,p,t)` that evaluates and returns the Jacobian of the ODEs. The corresponding expressions are stored in the `symjac` field:

```julia
latexify(repressilator.symjac)
```

\begin{equation}
\left[
\begin{array}{cccccc}
1 \cdot  - \delta & 1 \cdot 0 & 1 \cdot 0 & 1 \cdot 0 & 1 \cdot 0 & 1 \cdot \frac{ - K^{n} \cdot n \cdot \alpha \cdot P_3^{-1 + n}}{\left( K^{n} + P_3^{n} \right)^{2}} \\
1 \cdot 0 & 1 \cdot  - \delta & 1 \cdot 0 & 1 \cdot \frac{ - K^{n} \cdot n \cdot \alpha \cdot P_1^{-1 + n}}{\left( K^{n} + P_1^{n} \right)^{2}} & 1 \cdot 0 & 1 \cdot 0 \\
1 \cdot 0 & 1 \cdot 0 & 1 \cdot  - \delta & 1 \cdot 0 & 1 \cdot \frac{ - K^{n} \cdot n \cdot \alpha \cdot P_2^{-1 + n}}{\left( K^{n} + P_2^{n} \right)^{2}} & 1 \cdot 0 \\
1 \cdot \beta & 1 \cdot 0 & 1 \cdot 0 & 1 \cdot  - \mu & 1 \cdot 0 & 1 \cdot 0 \\
1 \cdot 0 & 1 \cdot \beta & 1 \cdot 0 & 1 \cdot 0 & 1 \cdot  - \mu & 1 \cdot 0 \\
1 \cdot 0 & 1 \cdot 0 & 1 \cdot \beta & 1 \cdot 0 & 1 \cdot 0 & 1 \cdot  - \mu \\
\end{array}
\right]
\end{equation}

- `repressilator.jac` is a function `pjac(J,u,p,t)` that evaluates and returns the Jacobian, `J`, of the ODEs *with respect to the parameters*. This allows `reaction_network`s to be used in the DiffEqs parameter estimation packages.

See the [DiffEqBiological documentation](http://docs.juliadiffeq.org/latest/models/biological.html) for a thorough description of the different fields within the generated network.

```julia

```


Construir o gráfico e encontrar o foco e uma equação da diretriz.

2. $y^2 = 6x$

$2p = 6$,logo

$p = 3$

Calculando o foco

$F = \frac{p}{2}$

$F = \frac{3}{2}$

$F(\frac{3}{2},0)$

Calculando a diretriz

$d = -\frac{p}{2}$

$d = -\frac{3}{2}$

$d : x = - \frac{3}{2}$

$V(0,0)$

$F(\frac{3}{2},0)$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y-0)**2, 6*(x+0)), (x,-5,10), (y,-10,10),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/02.ipynb


## 0. Import Packages

```python
import sympy
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## 1. Gradient Vector Field and Contour

```python
#define symbolic vars, function
x, y = sympy.symbols('x y')
fun = (x-1)**2+(y-1)**2

#take the gradient symbolically
gradfun = [sympy.diff(fun,var) for var in (x,y)]

#turn into a bivariate lambda for numpy
numgradfun = sympy.lambdify([x,y],gradfun)

x = np.linspace(-4, 6, 15)
y = np.linspace(-4, 6, 15)
x1 = np.linspace(-4, 6, 100)
y1 = np.linspace(-4, 6, 100)

X,Y = np.meshgrid(x, y)
X1, Y1 = np.meshgrid(x1, y1)
graddat = numgradfun(X,Y)
Z = (X1-1)**2 + (Y1-1)**2

plt.figure()
plt.axes().set_aspect('equal', 'datalim')
plt.quiver(X,Y,graddat[0],graddat[1])
plt.contour(x1, y1, Z, cmap='seismic')

plt.show()
```

## 2. Linear Regression
- x = np.array([1, 2, 3])
- y = np.array([4.1, 6.9, 9.9])
- Hypothesis H = wx+b

```python
x = np.array([1, 2, 3])
y = np.array([4.1, 6.9, 9.9])
w = 0
b = 0
a = 0.05      # learning rate

H = w*x + b

for i in range(20):
    dw = 2*sum(x*(H-y))/len(y)
    db = 2*sum(H-y)/len(y)
    w = w - a*dw
    b = b - a*db

    H = w*x+b
    c = sum((H-y)*(H-y))/len(y)

    plt.plot(x, y, "ro")
    plt.plot(x, w*x+b, "b-")

    print("w: ", w, "b: ", b, "Cost: ", c)

print("Prediction:", w*4+b)
```

- x = np.array([1, 2, 3])
- y = np.array([4.1, 6.9, 9.9])
- Hypothesis H = wx

```python
x = np.array([1, 2, 3])
y = np.array([4.1, 6.9, 9.9])
w = 0
a = 0.05      # learning rate

H = w*x

for i in range(20):
    g = 2*sum(x*(H-y))/len(y)
    w = w - a*g
    H = w*x
    c = sum((H-y)*(H-y))/len(y)
    plt.plot(x, y, "ro")
    plt.plot(x, w*x, "b-")

    print("w: ", w, "Cost: ", c)

print("Prediction:", w*4)
```

- x = np.array([30, 20, 40])
- y = np.array([6.1, 3.1, 6.8])
- Hypothesis H = w*x + b

```python
x = np.array([30, 20, 40])
y = np.array([6.1, 3.1, 6.8])
w = 0
b = 0
a = 0.001      # learning rate

H = w*x + b

for i in range(1000):
    dw = 2*sum(x*(H-y))/len(y)
    db = 2*sum(H-y)/len(y)
    w = w - a*dw
    b = b - a*db

    H = w*x+b
    c = sum((H-y)*(H-y))/len(y)
    if i % 100 == 0:
        print("w: ", w, "b: ", b, "Cost: ", c)

print("Prediction:", w*25+b)
```

    w:  0.3446666666666667 b:  0.010666666666666666 Cost:  27.16708562962964
    w:  0.1784041700910588 b:  0.0024629078261677556 Cost:  0.2972409816719897
    w:  0.17829247047790894 b:  -0.0005424250099560584 Cost:  0.2971102467195272
    w:  0.1783840228514668 b:  -0.0035003652573958647 Cost:  0.2970226733833154
    w:  0.17847457052784338 b:  -0.006417814659193166 Cost:  0.29693748080630333
    w:  0.17856387925085382 b:  -0.009295335080892234 Cost:  0.2968546042070466
    w:  0.17865196567185065 b:  -0.012133473010365266 Cost:  0.29677398062521215
    w:  0.1787388465192008 b:  -0.01493276744664607 Cost:  0.2966955488120573
    w:  0.1788245382926983 b:  -0.017693750011875956 Cost:  0.29661924918389987
    w:  0.17890905726631662 b:  -0.02041694505227746 Cost:  0.29654502377685077
    Prediction: 4.451713595271376

## 2. Multi-Variable Linear regression

```python
X = np.matrix([[30, 20, 40], [1, 2, 3], [5, 10, 15]])
Y = np.matrix([6.1, 3.1, 6.8])

W = np.matrix([0, 0, 0])
b = 0

a = 1e-4 # learning rate
H = W*X + b

for t in range(3000):
    dW = 2*((H-Y)*X.T)/Y.shape[1] # gradient
    db = 2*np.mean(H-Y)
    W = W - a*dW
    b = b - a*db
    c = (H-Y)*(H-Y).T/Y.shape[1]
    H = W*X+b
    if t % 100 == 0:
        print('Iteration:', t, 'Cost:', c[0,0])

print('')
X_test = np.matrix([[25], [2], [10]])
print("Price Prediction:", (W*X_test+b)[0,0])
```

    Iteration: 0 Cost: 31.019999999999996
    Iteration: 100 Cost: 0.34082133458696035
    Iteration: 200 Cost: 0.21074274888050268
    Iteration: 300 Cost: 0.1305638504780756
    Iteration: 400 Cost: 0.08114184956251845
    Iteration: 500 Cost: 0.05067761941048248
    Iteration: 600 Cost: 0.031898479662662904
    Iteration: 700 Cost: 0.02032173588706472
    Iteration: 800 Cost: 0.013184369717335368
    Iteration: 900 Cost: 0.008783327191182105
    Iteration: 1000 Cost: 0.00606888933209325
    Iteration: 1100 Cost: 0.004394036161773092
    Iteration: 1200 Cost: 0.003359961078985585
    Iteration: 1300 Cost: 0.0027208498684453456
    Iteration: 1400 Cost: 0.0023251889139681518
    Iteration: 1500 Cost: 0.0020795890801245617
    Iteration: 1600 Cost: 0.001926487676877947
    Iteration: 1700 Cost: 0.0018304045201090295
    Iteration: 1800 Cost: 0.0017694705183611975
    Iteration: 1900 Cost: 0.001730206110082214
    Iteration: 2000 Cost: 0.0017043028257053155
    Iteration: 2100 Cost: 0.0016866395247191208
    Iteration: 2200 Cost: 0.0016740596548655451
    Iteration: 2300 Cost: 0.0016646175859395386
    Iteration: 2400 Cost: 0.0016571140691929865
    Iteration: 2500 Cost: 0.0016508099063763692
    Iteration: 2600 Cost: 0.0016452494626485812
    Iteration: 2700 Cost: 0.0016401518844429645
    Iteration: 2800 Cost: 0.0016353440489375395
    Iteration: 2900 Cost: 0.0016307192358216817

    Price Prediction: 4.185693634972486


```python
import sympy as sym
from IPython.display import display,Math

# Please enter the value as 3*x**2 instead of 3x^2

# Bokijonov Mukhsinjon
# 52336
# Task 4
# 27 points

```

```python
from sympy.abc import x
```

```python
def Polynomial():
    print("Welcome to the polynomial calculator.")
    # Accept user input as Polynomial
    # Note: Please enter the valuse as 3*x**2 instead of 3x^2

    input1 = sym.Poly(input("Please enter the first Polynomial: "))
    input2 = sym.Poly(input("Please enter the second Polynomial: "))
    p1= input1
    p2 = input2

    # Accept operation type
    user_input = input("Please, choose Operation (+, -, *, /): ")
    # Print the result
    if user_input == '+':
        print("Result = ")
        display(Math('(%s) + (%s) =%s'%(sym.latex(p1),sym.latex(p2),sym.latex(p1 + p2))))
    elif user_input == '-':
        print("Result = ")
        display(Math('(%s) - (%s) =%s'%(sym.latex(p1),sym.latex(p2),sym.latex(p1 - p2))))
    elif user_input == '*':
        print("Result = ")
        display(Math('(%s) * (%s) =%s'%(sym.latex(p1),sym.latex(p2),sym.latex(p1 * p2))))
    elif user_input == '/':
        print("Result = ")
        display(Math('(%s) / (%s) =%s'%(sym.latex(p1),sym.latex(p2),sym.latex(p1 / p2))))
    else:
        print("Unsuppoerted operator. Restart the code! ")
        Polynomial()

```

```python
def Repeat():
    repeat = input("Choose: n to enter new arguments, x to exit: ")
    if repeat == 'n'.lower():
        return Polynomial()
    elif repeat  == 'x':
        print("Bye!")

```

```python
while True:
    Polynomial()
    Repeat()
    break

```

    Welcome to the polynomial calculator.
    Please enter the first Polynomial: x**2
    Please enter the second Polynomial: x**2
    Please, choose Operation (+, -, *, /): +
    Result =

$\displaystyle (\operatorname{Poly}{\left( x^{2}, x, domain=\mathbb{Z} \right)}) + (\operatorname{Poly}{\left( x^{2}, x, domain=\mathbb{Z} \right)}) =\operatorname{Poly}{\left( 2 x^{2}, x, domain=\mathbb{Z} \right)}$

    Choose: n to enter new arguments, x to exit: n
    Welcome to the polynomial calculator.

```python

```


# MaxEnt for deriving probability distributions

Today we will derive some standard probability distribution by maximizing the entropy subject to constraints derived from information that we have.

## Example 1: the Gaussian

First we take the constraint
\begin{equation}
\int_{-\infty}^\infty (x-\mu)^2 p(x) dx=\sigma^2.
\end{equation}

We also have the normalization constraint:
\begin{equation}
\int_{-\infty}^\infty p(x) dx=1.
\end{equation}

So we maximize:

\begin{equation}
Q(p;\lambda_0,\lambda_1)=-\int p(x)\ln\left(\frac{p(x)}{m(x)}\right) \, dx + \lambda_0 \left(1-\int p(x) dx\right)+ \lambda_1\left(\sigma^2 - \int p(x) (x-\mu)^2 dx\right)
\end{equation}

We will assume a uniform m(x).

Step 1: differentiate with respect to p(x). What do you get?

```python

```

Step 2: set the functional derivative equal to 0. Show that the solution is:

\begin{equation}
p(x)={\cal N} \exp(-\lambda_1 (x-\mu)^2),
\end{equation}

where ${\cal N}=e^{-1-\lambda_0}$.

```python

```

Step 3a: Now, we impose the constraints. First, use the fact that $\int_{-\infty}^{\infty} \exp(-y^2) \, dy=\sqrt{\pi}$ to fix ${\cal N}$.

```python

```

Step 3b: Second, compute $\int_{-\infty}^{\infty} y^2 \exp(-y^2) \, dy$, and use the results to show $\lambda=\frac{1}{2 \sigma^2}$.

## Example 2: the Poisson distribution

Now we will take a constraint on the mean (first moment):
\begin{equation}
\int_{0}^\infty x p(x) dx=\mu.
\end{equation}

As usual, we also have the normalization constraint:
\begin{equation}
\int_{0}^\infty p(x) dx=1.
\end{equation}

So we maximize:

\begin{equation}
Q(p;\lambda_0,\lambda_1)=-\int p(x)\ln\left(\frac{p(x)}{m(x)}\right) \, dx + \lambda_0 \left(1-\int p(x) dx\right)+ \lambda_1\left(\mu - \int p(x) x dx\right)
\end{equation}

We will again assume a uniform m(x).

Go through the steps as you did in the first example.

```python

```

 You should obtain the Poisson distribution:

\begin{equation}
p(x)=\frac{1}{\mu} \exp\left(-\frac{x}{\mu}\right)
\end{equation}

## Third example: log normal distribution

Suppose the constraint is on the variance of $\ln x$, i.e.,
\begin{equation}
\int p(x)\left[\log\left(\frac{x}{x_0}\right)\right]^2 dx=\sigma^2
\end{equation}

Change variables to $y=\log(x/x_0)$. What is the constraint in terms of $y$?

Now maximize the entropy, subject to this constraint, and, of course, the normalization constraint.

```python

```

You should obtain the log-normal distribution:

\begin{equation}
p(x)=\frac{1}{\sqrt{2 \pi} x \sigma} \exp\left[-\frac{\ln^2(x/x_0)}{2 \sigma^2}\right].
\end{equation}

When do you think it would make sense to say that we know the variance of $\log(x)$, rather than the variance of $x$ itself?

```python

```

## Fourth example: l1-norm

Finally, we take the constraint on the mean absolute value of $x-\mu$: $\langle |x-\mu| \rangle=\epsilon$.

This constraint is written as:

\begin{equation}
\int p(x) \, |x - \mu| \, dx=\epsilon.
\end{equation}

Use the uniform measure, and go through the steps once again, to show that:
\begin{equation}
p(x)=\frac{1}{2 \epsilon} \exp\left(-\frac{|x-\mu|}{\epsilon}\right).
\end{equation}

```python

```


# Spectral Estimation of Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## The Welch Method

In the previous section it has been shown that the [periodogram](periodogram.ipynb) as a non-parametric estimator of the power spectral density (PSD) $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of a random signal $x[k]$ is not consistent. This is due to the fact that its variance does not converge towards zero even when the length of the random signal is increased towards infinity. In order to overcome this problem, the [Bartlett method](https://en.wikipedia.org/wiki/Bartlett's_method) and [Welch method](https://en.wikipedia.org/wiki/Welch's_method)

1. split the random signal into segments,
2. estimate the PSD for each segment, and
3. average over these local estimates.

The averaging reduces the variance of the estimated PSD. While Barlett's method uses non-overlapping segments, Welch's is a generalization using windowed overlapping segments. For the discussion of Welch's method we assume a wide-sense ergodic real-valued random process.

### Derivation

The random signal $x[k]$ is split into into $L$ overlapping segments of length $N$, starting at multiples of the step size $M \in {1,2, \dots, N}$. These segments are windowed by the window $w[k]$ of length $N$, resulting in a windowed $l$-th segment $x_l[k]$ with $0\leq l\leq L-1$. The discrete-time Fourier transformation (DTFT) $X_l(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the windowed $l$-th segment is then given as

\begin{equation}
X_l(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k = 0}^{N-1} x[k + l \cdot M] \, w[k] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k}
\end{equation}

where the window $w[k]$ defined within $0\leq k\leq N-1$ should be normalized as $\frac{1}{N} \sum\limits_{k=0}^{N-1} | w[k] |^2 = 1$. The stepsize $M$ determines the overlap between the segments. In general, $N-M$ number of samples overlap between adjacent segments. For $M = N$ no overlap occurs. The overlap is sometimes given as ratio $\frac{N-M}{N}\cdot 100\%$.

Introducing $X_l(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ into the definition of the periodogram yields the periodogram of the $l$-th segment

\begin{equation}
\hat{\Phi}_{xx,l}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{N} \,| X_l(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

The estimated PSD is then given by averaging over the segment's periodograms $\hat{\Phi}_{xx,l}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$

\begin{equation}
\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{L} \sum_{l = 0}^{L-1} \hat{\Phi}_{xx,l}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

Note, that the total number $L$ of segments has to be chosen such that the last required sample $(L-1)\cdot M + N - 1$ does not exceed the total length of the random signal. Otherwise the last segment $x_{L-1}[k]$ may be zeropadded towards length $N$.

The Bartlett method uses a rectangular window $w[k] = \text{rect}_N[k]$ and non-overlapping segments $M=N$. The Welch method uses overlapping segments and a window that must be chosen according to the intended spectral analysis task.

### Example

The following example is equivalent to the previous [periodogram example](periodogram.ipynb#Example---Periodogram). We aim at estimating the PSD of a random process which draws samples from normally distributed white noise with zero-mean and unit variance. The true PSD is consequently given as $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 128  # length of segment
M = 64  # stepsize
L = 100  # total number of segments

# generate random signal
np.random.seed(5)
x = np.random.normal(size=L*M)

# compute periodogram by Welch's method
nf, Pxx = sig.welch(x, window='hamming', nperseg=N, noverlap=(N-M))
Pxx = .5*Pxx  # due to normalization in scipy.signal
Om = 2*np.pi*nf

# plot results
plt.figure(figsize=(10,4))
plt.stem(Om, Pxx, 'b', label=r'$\hat{\Phi}_{xx}(e^{j \Omega})$', basefmt=' ')
plt.plot(Om, np.ones_like(Pxx), 'r', label=r'$\Phi_{xx}(e^{j \Omega})$')
plt.title('Estimated and true PSD')
plt.xlabel(r'$\Omega$')
plt.axis([0, np.pi, 0, 2])
plt.legend()

# compute mean value of the periodogram
print('Mean value of the periodogram: %f' %np.mean(np.abs(Pxx)))
```

**Exercise**

* Compare the results to the periodogram example. Is the variance of the estimator lower?
* Change the number of segments `L`. What changes?
* Change the segment length `N` and stepsize `M`. What changes?

When comparing both the estimates of the PSD in the previous periodogram and above example, it is obvious that the variance of the Welch estimator is lower. Increasing the number of segments `L` lowers the variance further. Increasing the segment length `N` increases the total number of discrete frequencies in the estimated PSD. Since in above example the total number of segments is kept constant, the variance increases. Lowering the stepsize `M` has the same result, since the total number of samples is reduced for a fixed number of segments.

### Evaluation

It is shown in [[Stoica et al.](../index.ipynb#Literature)] that Welch's method is asymptotically unbiased. Under the assumption of a wide-sense stationary (WSS) random process, the periodograms $\hat{\Phi}_{xx,l}(e^{j \Omega})$ of the segments can be assumed to be approximately uncorrelated. Hence, averaging over these reduces the overall variance of the estimator. It can be shown formally that in the limiting case of an infinitely number of segments (infintely long signal) the variance tends towards zero. As a result Welch's method is an asymptotically consistent estimator of the PSD.

Note, that for a finite segment length $N$ the properties of the estimated PSD $\hat{\Phi}_{xx}(e^{j \Omega})$ depend on the length $N$ of the segments and the window function $w[k]$ due to the [leakage effect](../spectral_analysis_deterministic_signals/leakage_effect.ipynb).

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# Module 1: Setting up the problem

Before we bgin, import SimPEG into ipython notebook as follows:

```
from SimPEG import *
from IPython.html.widgets import interactive
```

    Efficiency Warning: Interpolation will be slow, use setup.py!

                python setup.py build_ext --inplace

**Introduction**

Every geophysical survey consists of a similar basic framework. An energy source is delivered into the earth, and an array of recievers pick up a signal at the surface...

**Module summary:**

(1) Start with an expression that relates a kernel function with the continuous distribution of a physical property.

(2) Discretize this expression.

(3) Define a mesh that organizes the data.

(4) Build up the matrix equation $d = Gm$.

**Step 1: Physical property distribution and the kernel function.**

1.1 define kernel function (...add description)

1.2 define the model. (...add description)

Each datum can be expressed as the inner product of the kernel function and the model:

*--the integral needs a physical description--*

\begin{equation}d_j = \int_a^b g(x) m(x) dx \end{equation}

...each data point is a measure of a physical property of the entire earth at that point. Consequently it is the sum total (and is therefore the integral) of the influence of the material property specified by the model,

**Step 2: Discretize the function for each datum.**

\begin{equation}d_i = \sum_{j=1}^N g_i (x_j) m_i \Delta x\end{equation}

We can then gather all the data (the $d^i$s) into a column vector (and say we have $M$ data). Similarly we assemble each kernel function as a row in a matrix, $\widetilde{G}$, and our model parameters $m$ into a column vector of length $N$. For the spacing increments, we require that we obtain a single output for each $\Delta x$ in the kernel function. It follows that this is best represented as a diagonal matrix, with a $\Delta x$ on every diagonal entry. Combining all of the aforemention yields the following vector equation:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\vdots\\
d_M
\end{array}
\right]
=
\begin{bmatrix}
    g_1(x_j)    & g_1(x_j) &  g_1(x_j)  & \dots & g_1(x_N)  \\
    g_2(x_j)    & g_2(x_j) &  g_2(x_j)   & \dots  & g_2(x_N)  \\
    \vdots       &  \vdots    &  \vdots     & \vdots   &  \vdots \\
    g_M(x_j)    & g_M(x_j) &  g_M(x_j)   & \dots  & g_M(x_N)  \\
\end{bmatrix}
\begin{bmatrix}
    \Delta x_1    & 0    & \dots &0   \\
        0         & \Delta x_2   & \dots  & 0  \\
    \vdots        &      \vdots   & \ddots  & \vdots\\
    0             & 0     & \dots  & \Delta x_N  \\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
\vdots\\
m_N
\end{array}
\right]
\end{equation}

Or put more succinctly: $d = \widetilde{G} diag(\Delta x) m$.

Next, set up "toy problem" with $d_1$, $d_2$ and $j=1:5$.

**Step 3: Define a mesh.**

While the previous section describes how the data is converted from a continuous function in physical space into discreet data "chunks," what it does not address is the manner in which it is to be represented on a computer so that it is available for processing.

3.1 The physical property of interest lies in the cell centers. (...add sketch)

3.2 The kernel functions reside on the nodes (...add sketch)

**Step 4: Build up the matrix equation $d=Gm$.**

Note again that the kernel functions "live" on the nodes and the model values in the cell centers. To achieve the inner product of the kernel function with the model parameters we need to refomulate the $G$ matrix to obtain values for the kernel functions at the cell centers.

4.1 Define the matrix $G_n$ (as in "n" for "nodes") as the matrix containing the values for the kernel functions on the nodes. Given that we have $M$ data points and seek $N$ model parameters, it follows that the dimensions of $G_n$ will be $M \times (N+1)$. Schematically, $G_n$ will appear like this:

\begin{equation}
G_n =
\begin{bmatrix}
    g_{n_1}^1    &  g_{n_2}^1 &  g_{n_3}^1  & \dots & g_{n_{N+1}}^1  \\
    g_{n_1}^2    &  g_{n_2}^2 &  g_{n_3}^2  & \dots  & g_{n_{N+1}}^2  \\
    \vdots       &  \vdots    &  \vdots     & \vdots   &  \vdots   \\
    g_{n_1}^M    &  g_{n_2}^M &  g_{n_3}^M  & \dots  & g_{n_{N+1}}^M
\end{bmatrix}
\end{equation}

In order to evaluate the kernel functions on the cell centers, we will employ the trapezoidal rule, which states:

```
import numpy as np
```

```
m=5
n=6
w=n-1
s = (m,n)
A = np.zeros(s)

for i in range(m):
    j=i
    k=i+1
    A[i,j] = 0.5
    A[i,k] = 0.5
print A
```

    [[ 0.5  0.5  0.   0.   0.   0. ]
     [ 0.   0.5  0.5  0.   0.   0. ]
     [ 0.   0.   0.5  0.5  0.   0. ]
     [ 0.   0.   0.   0.5  0.5  0. ]
     [ 0.   0.   0.   0.   0.5  0.5]]

```
g = lambda x, k, p, q: np.exp(-p*k*x)*np.cos(2*np.pi*q*k*x)
x=np.linspace(0,1,6)
#x = np.array([0., 0.2, 0.4, 0.6, 0.8, 1.])
p = 0.01
q = 0.1
k = np.array([1, 2, 3, 4, 5, 6])
Gn = np.zeros((len(x), len(k)))

for i in range(len(k)):
    f = g(x,k[i],p,q)
    Gn[:,i] = f
#print f
print Gn

```

    [[ 1.          1.          1.          1.          1.          1.        ]
     [ 0.99013245  0.96471657  0.92421453  0.86932419  0.80096714  0.72027328]
     [ 0.96471657  0.86932419  0.72027328  0.52732179  0.30289805  0.06130149]
     [ 0.92421453  0.72027328  0.41818383  0.06130149 -0.29988416 -0.61488486]
     [ 0.86932419  0.52732179  0.06130149 -0.41237005 -0.77729498 -0.94561804]
     [ 0.80096714  0.30289805 -0.29988416 -0.77729498 -0.95122942 -0.76190351]]

```

```

```
 f = lambda x, y : x + y
 f(1,1)
```

    2

```
# make the delta x array
deltax = 0.2*np.ones(m)
V = np.diag(deltax)
print V

```

    [[ 0.2  0.   0. ]
     [ 0.   0.2  0. ]
     [ 0.   0.   0.2]]

```

```


### Using PyMC3 for Bayesian Analysis
I will use PyMC3 to do some Bayesian analysis. I will follow the book by Davidson-Pilon.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
%matplotlib inline
```

```python
txt_data = np.loadtxt('data/txtdata.csv')
```

```python
n_txt_data = len(txt_data)
days = np.arange(n_txt_data)
```

```python
fig, ax = plt.subplots(1, 1, figsize = (25,9))
ax.bar(days, txt_data, color = 'xkcd:violet')
ax.set_xlabel(r'No. of Days', fontsize = 20)
ax.set_ylabel('No. of text messages', fontsize = 20)
ax.tick_params(axis = 'both', labelsize = 20 )
ax.set_facecolor('0.80')
ax.grid(which = 'both', axis = 'both', color = 'xkcd:white')
ax.set_axisbelow(True)

```

We want to model the texting behaviour of the author over various days. Since we are counting text messages and appropriate distribution function to consider would be the Poisson distribution. However since we want to check for texting behaviour we might want to consider two different Poisson distributions to get a better understanding.

Let us denote the text message count on the $i^{th}$ day as $C_i$. Since we are couunting here we can furthermore write:

$C_i = Poisson\left(\lambda\right)$

If we assume that $\lambda$ might change after somedayy, then we can assume two values $\lambda_1$ and say after $\tau$ days we have $\lambda_2$. The specific time $\tau$ is known as the *switchpoint*.

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

Now in order to estimate $\lambda_1$ and $\lambda_2$ using Bayesian inference we need to have a prior distribution. We know that we can use an exponential distribution function to describe non-discrete positive numbers. This exponential distribution function will have a parameter of its own. let us call this parameter $\alpha$

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

Since the data is noisy it would be hard to decide a prior for $\tau$. We can thus consider a uniform prior in this case:

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

#### Using PyMC3

```python
import pymc3 as pm
import theano.tensor as tt
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
    C:\Users\nathd\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

```python
with pm.Model() as model:
    alpha = 1.0/txt_data.mean()  # Recall count_data is the
                                   # variable that holds our txt counts
    lambda_1 = pm.Exponential("lambda_1", alpha)
    lambda_2 = pm.Exponential("lambda_2", alpha)

    tau = pm.DiscreteUniform("tau", lower=0, upper=n_txt_data - 1)
```

We can now create a function lambda_ which will allow us to switch between $\lambda_1$ and $\lambda_2$ after time $\tau$.

```python
with model:
    idx = np.arange(n_txt_data) # Index
    lambda_ = pm.math.switch(tau > idx, lambda_1, lambda_2)
```

All the parameters $\lambda_1, \lambda_2, \text{ and } \tau$ are random number and so is lambda_ .

```python
with model:
    observation = pm.Poisson("obs", lambda_, observed=txt_data)
```

The variable `observation` combines our data, `count_data`, with our proposed data-generation scheme, given by the variable `lambda_`, through the `observed` keyword.

The code below will be explained in Chapter 3, but I show it here so you can see where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Markov Chain Monte Carlo* (MCMC), which I also delay explaining until Chapter 3. This technique returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distributions look like. Below, we collect the samples (called *traces* in the MCMC literature) into histograms.

```python
with model:
    step = pm.Metropolis()
    trace = pm.sample(10000, tune=5000,step=step)
```

    Multiprocess sampling (2 chains in 2 jobs)
    CompoundStep
    >Metropolis: [tau]
    >Metropolis: [lambda_2_log__]
    >Metropolis: [lambda_1_log__]
    The number of effective samples is smaller than 25% for some parameters.

```python
lambda_1_samples = trace['lambda_1']
lambda_2_samples = trace['lambda_2']
tau_samples = trace['tau']
```

```python
fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize = (15, 15))
ax1.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.6,
         label="posterior of $\lambda_1$", color="#A60628", normed=True)
ax1.legend(loc = 'best')
ax1.set_xlim([15, 30])
ax1.set_xlabel("$\lambda_1$")
ax1.set_facecolor('0.80')
ax1.grid(which = 'both', axis = 'both', color = 'xkcd:white', linestyle='dashed')
ax1.set_axisbelow(True)
ax2.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.5,
         label="posterior of $\lambda_2$", color="#7A68A6", normed=True)
ax2.legend(loc='best')
ax2.set_xlim([15, 30])
ax2.set_xlabel("$\lambda_2$")
ax2.set_facecolor('0.80')
ax2.grid(which = 'both', axis = 'both', color = 'xkcd:white', linestyle='dashed')
ax2.set_axisbelow(True)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
ax3.hist(tau_samples, bins=n_txt_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
ax3.legend(loc='best')
ax3.set_xlim([35, len(txt_data)-20])
ax3.set_xlabel("Days")
ax3.set_ylabel('Probability')
ax3.set_facecolor('0.80')
ax3.grid(which = 'both', axis = 'both', color = 'xkcd:white', linestyle='dashed')
ax3.set_axisbelow(True)
```

We see that the values of $\lambda_1$ and $\lambda_2$ is around 18 and 23 respectively. Additionally we see that the change in text messages received takes place over a span of 4 days, with the likelihood that the change in the number of text takes place on the 45th day is around 50 %. We would like to see how this change of number of text messages received takes place over time.

```python
fig, ax = plt.subplots(1,1, figsize = (15, 5))

# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_txt_data)
for day in range(n_txt_data):
    #ix is the boolean index of all tay samples corresponding to the switch point occuring prior to the value of day
    ix = day<tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N
ax.plot(range(n_txt_data), expected_texts_per_day, '-', color = 'xkcd:magenta', linewidth = 5, alpha = 0.7, label = 'Expected Text Per Day')
ax.bar(days, txt_data, color = 'xkcd:violet', label = 'Observed texts per day', alpha = 0.7)
ax.legend(loc = 'best')
ax.set_xlabel(r'Days', fontsize = 20)
ax.set_ylabel('No. of text messages', fontsize = 20)
ax.tick_params(axis = 'both', labelsize = 20 )
ax.set_facecolor('0.80')
ax.grid(which = 'both', axis = 'both', color = 'xkcd:white')
ax.set_axisbelow(True)
```

```python

```

Markov Chain Monte Carlo/Bayesian Statistics using PyMC3.ipynb


# 1. Kernel Ridge Regression

\begin{align*}
\text{Primal form:}~&\Vert Ax - y \Vert_2^2 + \lambda \Vert x \Vert_2^2\\
\textbf{Closed-form solution:}~&x = \left(A^TA + \lambda I_n\right)^{-1}A^Ty\\
\end{align*}

From the identity $\left(A^TA + \lambda I_n\right)^{-1}A^T = A^T\left(AA^T + \lambda I_m\right)^{-1}$, we can obtain a closed-form solution with the alternate form:
\begin{align*}
\textbf{Alternate closed-form solution:}~x = A^T\left(AA^T + \lambda I_m\right)^{-1}y
\end{align*}

Let $A$ have $m$ rows and $n$ columns, such that each row is a data-point (or sample) and each column is a feature. In typical applications of kernel ridge regression, $m >> n$. This means that computing $\alpha$ directly requires computing the $m \times m$ matrix, $AA^T$. For very large $m$ this can be an expensive $m^2n$ operation and, in the extreme, we many not even have space to store $AA^T$.

The Kernelized variant of ridge regression replaces the $m \times n$ matrix $A$ with a high-dimensional feature space for each sample $\Phi(A)$. $\Phi(A)$ is computational-expensive to form, especially when the chosen kernel function has infinite-dimensional feature space (e.g. Radial Basis Function). However, the $m \times m$ kernel matrix $K$ can be computed since the kernel functions are defined by $K_{i,j} = \sum k(a_i, a_j)$, where $a_i$ is the $i$-th row of $A$ and $k(...)$ is a user-defined kernel function.

One example of such a kernel function, is the polynomial kernel: $k(a_i, a_j) = (a_ia_j^T + r)^d$ s.t. $r \geq 0$ and $d \geq 1$. *When $r = 0$ and $d = 1$, we obtain a simple dot-product (or linear kernel).* In replacing the dot-products with a kernel function, we obtain a feature-space that can have very high dimensions (possibly infinite-dimensions). This means that computing the solution, $x$, is often very expensive. So instead, we will focus on obtaining the solution:
\begin{align*}
\bf
\text{Alternate solution:}~&\alpha = \left(\frac{1}{\lambda}K + I_m\right)^{-1}y\\
\text{and}~&w = \frac{1}{\lambda}\Phi(A)^T\alpha
\end{align*}

# 2. Subspace Iteration for Kernel Ridge Regression

Note that the kernel matrix, $K$, is more expensive to compute than $AA^T$ (due to the kernel function), so we will instead compute the solution to $\alpha$ iteratively using ***subspace iteration*** by selecting, and solving for a few samples (row of $A$) every iteration.
\begin{align*}
\text{Dual form:}~&\Vert \alpha - y \Vert_2^2 + \frac{1}{\lambda} \Vert \Phi(A)^T\alpha \Vert_2^2\\

\bf\text{Iterative update:}~&\alpha_h = \alpha_{h-1} + \mathbb{I}_h^T {u_h}\\
\end{align*}
The matrix $\mathbb{I}_h$ has $b$ rows and $m$ columns such that rows of $\mathbb{I}_h$ are sub-sampled rows of the $m$-dimensional identity matrix, $I_m$.

By substituting the iterative update into the dual form, we obtain the following minimization problem:
\begin{align*}
{\arg\min}~\left\Vert \alpha_{h-1} + \mathbb{I}_h^Tu_h - y \right\Vert_2^2 + \frac{1}{\lambda} \left\Vert \Phi(A)^T\alpha_{h-1} + \Phi(A)^T\mathbb{I}_h^T u_h\right\Vert_2^2
\end{align*}

By taking the derivative w.r.t $u_h$ we can obtain the gradient for just the $b$ rows of $A$ that we are sampling at every iteration:
\begin{align*}
0 &= \mathbb{I}_h\alpha_{h-1} + u_h - \mathbb{I}_hy + \frac{1}{\lambda} \mathbb{I}_h \Phi(A)\Phi(A)^T\alpha_{h-1} + \frac{1}{\lambda}\mathbb{I}_h\Phi(A)\Phi(A)^T\mathbb{I}_h^T u_h\\
 \textbf{Gradient w.r.t chosen samples: }u_h &= \left(\frac{1}{\lambda} \mathbb{I}_h\Phi(A)\Phi(A)^T\mathbb{I}_h^T + I_b \right)^{-1}\left(\mathbb{I}_h y - \mathbb{I}_h \alpha_{h-1} - \frac{1}{\lambda}\mathbb{I}_h \Phi(A)\Phi(A)^T\alpha_{h-1}\right)
\end{align*}

Notice that the term $\mathbb{I}_h \Phi(A)\Phi(A)^T \alpha_{h-1}$ does not permit the use of an auxiliary vector to represent $\Phi(A)^T \alpha_{h-1}$ due to the kernelization. This means that we will have to explicitly compute $\mathbb{I}_h\Phi(A)\Phi(A)^T$ at every iteration. Additionally, the quantity $\mathbb{I}_h\Phi(A)\Phi(A)^T\mathbb{I}_h^T$ does not need to be computed at every iteration because we can simply select the relevant columns (namely the columns defined by right-multiplying by $\mathbb{I}_h^T)$ from the quantity $\mathbb{I}_h\Phi(A)\Phi(A)^T$.

# 3. Communication-Avoiding Subspace Iteration for Kernel Ridge Regression

In this section, we will derive the communication-avoiding variant of subspace iteration for solving the Kernel Ridge Regression problem. Lets start by re-stating the subspace iteration's update step:
\begin{align*}
    u_{h+1} &= \left(\frac{1}{\lambda} \mathbb{I}_{h+1}\Phi(A)\Phi(A)^T\mathbb{I}_{h+1}^T + I_b \right)^{-1}\left(\mathbb{I}_{h+1} y - \mathbb{I}_{h+1} \alpha_{h} - \frac{1}{\lambda}\mathbb{I}_{h+1} \Phi(A)\Phi(A)^T\alpha_{h}\right)\\
    \alpha_{h+1} &= \alpha_{h} + \mathbb{I}_{h+1}^T u_{h+1}
\end{align*}
Given this two-line recurrence, we can write the next update for iteration $h+2$ as:
\begin{align}
    u_{h+2} &= \left(\frac{1}{\lambda} \mathbb{I}_{h+2}\Phi(A)\Phi(A)^T\mathbb{I}_{h+2}^T + I_b \right)^{-1}\left(\mathbb{I}_{h+2} y - \mathbb{I}_{h+2} \alpha_{h+1} - \frac{1}{\lambda}\mathbb{I}_{h+2} \Phi(A)\Phi(A)^T\alpha_{h+1}\right)\tag{1}\\
    \alpha_{h+2} &= \alpha_{h+1} + \mathbb{I}_{h+2}^T u_{h+2}.\tag{2}
\end{align}
Notice that in the update step above, we can replace all instances of $\alpha_{h}$ with $\alpha_{h-1} + \mathbb{I}_h^T u_h$. By doing this, we can write the update steps for $u_{h+1}$ and $\alpha_{h+1}$ in terms of $\alpha_{h-1}$ and $u_h$, instead of $\alpha_h$:

\begin{align*}
    u_{h+2} &= \left(\frac{1}{\lambda} \mathbb{I}_{h+2}\Phi(A)\Phi(A)^T\mathbb{I}_{h+2}^T + I_b \right)^{-1}\left(\mathbb{I}_{h+2} y \color{blue}{- \mathbb{I}_{h+2} \alpha_{h} - \frac{1}{\lambda}\mathbb{I}_{h+2} \Phi(A)\Phi(A)^T\alpha_{h}} \color{red}{- \mathbb{I}_{h+2}\mathbb{I}_{h+1}^T u_{h+1} - \frac{1}{\lambda}\mathbb{I}_{h+2} \Phi(A)\Phi(A)^T\mathbb{I}_{h+1}^T u_{h+1}}\right)\tag{3}\\
    \alpha_{h+2} &= \alpha_{h} \color{red}{+ \mathbb{I}_{h+1}^Tu_{h+1} + \mathbb{I}_{h+2}^T u_{h+2}}.
\end{align*}
Comparing eq. (3) and (1), we can see that the terms in blue differ in that they depend on $\alpha_{h}$ instead of $\alpha_{h+1}$. The terms in red in eq. (3) are additional corrections which depend on the previous iteration's gradient, $u_{h+1}$, which also depend on $\alpha_{h}$. Finally, we can update $\alpha_{h+2}$ by summing the gradients $u_{h+1}$ and $u_{h+2}$.

The example above performs a recurrence unrolling of two iterations (i.e. computing $\alpha_{h+2}$ w.r.t $\alpha_h$), but we can unroll for an arbitrary number of iterations. We will use the variable $s$ to represent the number of iterations we unroll the recurrences. Notice that we still compute all of the gradients, $\{u_h, u_{h+1}, \ldots, u_{h+s}\}$, but only need $\alpha_h$ in order to compute $\alpha_{h+s}$. This means that we can defer updates to $\alpha$ for $s$ iterations.
\begin{align*}
u_{h + j} &= \left(\frac{1}{\lambda} \mathbb{I}_{h + j}\Phi(A)\Phi(A)^T\mathbb{I}_{h + j}^T + I_b\right)^{-1}\left(\mathbb{I}_{h+j} y \color{blue}{- \mathbb{I}_{h+j}\alpha_{h} - \frac{1}{\lambda}\mathbb{I}_{h+j}\Phi(A)\Phi(A)^T\alpha_{h}} \color{red}{- \sum_{k = 1}^{j-1}\mathbb{I}_{h + j}\mathbb{I}_{h + k}^T u_{h + k} - \sum_{k = 1}^{j-1} \frac{1}{\lambda}\mathbb{I}_{h+j}\Phi(A)\Phi(A)^T\mathbb{I}_{h + k}^T u_{h + k}}\right)\\
\alpha_{h+j} &= \alpha_{h} \color{red}{+ \sum_{k = 1}^{j} \mathbb{I}_{h + k}^T u_{h + k}}.\\
\text{for } j &= \{1, 2, \ldots, s\} \text{ where $s$ is a free parameter.}
\end{align*}

Notice that in the summation $\sum_{k = 1}^{j-1} \mathbb{I}_{h+k}\mathbb{I}_{k + j}^T u_k$, the computation $\mathbb{I}_{h+k}\mathbb{I}_k^T$ is computing a $b \times b$ identity-like matrix. This matrix is essentially computing the intersection between the sampled indices from iteration $h + j$ and iterations $h + k$ (where $k$ goes from $1$ to $j - 1$). This is crucial to ensure that gradient updates computed at iterations $h + k$ are incorporated into the iteration $h + j$.

```python
import numpy as np
import scipy as sp
from sklearn.metrics import pairwise_distances
import matplotlib.pyplot as plt

class Kernel_Ridge_Regression(object):
    def __init__(self, data, labels, kernel_params={'kernel_type': 'gaussian', 'sigma': 1e-1}):
        self.kernel_params = kernel_params
        self.nsamples, self.nfeatures = data.shape
        if self.nsamples != len(labels):
            raise ValueError('len(labels) != data.shape[1].')
        self.data = data
        self.labels = labels
        self._set_kernel_function(kernel_params['kernel_type'])
        self.alpha = np.zeros(self.nsamples)

    def _set_kernel_function(self, kernel_type):
        kernel_type = kernel_type.lower()
        if kernel_type == 'gaussian':
             self.kernel_function = self._gaussian_kernel
        elif kernel_type == 'polynomial':
            self.kernel_function = self._polynomial_kernel
        else:
            raise ValueError("Invalid kernel choice: {}".format(kernel_type))

    def _gaussian_kernel(self, subsamples, samples):
        kernel_matrix = pairwise_distances(subsamples, samples, metric='euclidean')**2
        kernel_matrix = np.exp(-kernel_matrix/self.kernel_params['sigma']**2)
        return kernel_matrix

    def _polynomial_kernel(self, subsamples, samples):
        kernel_matrix = (subsamples.dot(samples.T) + self.kernel_params['shift']) ** self.kernel_params['dim']
        return kernel_matrix

    def compute_objective(self):
        obj_val = np.linalg.norm(self.alpha - self.labels)**2
        obj_val += 1/self.C * (self.alpha.T.dot(self.kernel_function(self.data, self.data).dot(self.alpha)))
        return obj_val

    def block_coordinate_descent(self, idxs):
        blkrow_kernel_matrix = self.kernel_function(self.data[idxs, :], self.data)
        residual = self.labels[idxs] - self.alpha[idxs] - (1/self.C)*blkrow_kernel_matrix.dot(self.alpha)
        blkdiag_kernel_matrix = ((1/self.C) * blkrow_kernel_matrix[:, idxs]) + 1
        gradient = np.linalg.solve(blkdiag_kernel_matrix, residual)
        self.alpha[idxs] += gradient

    def commavoid_block_coordinate_descent(self, idxs):
        gradient = np.zeros(self.s*self.block_size)
        blkrow_kernel_matrix = self.kernel_function(self.data[idxs, :], self.data)
        residual = self.labels[idxs] - self.alpha[idxs] - (1/self.C)*blkrow_kernel_matrix.dot(self.alpha)
        # compute 's' gradients.
        blk_residual = residual[0:self.block_size]
        blkdiag_kernel_matrix = ((1/self.C) * blkrow_kernel_matrix[0:self.block_size, idxs[0:self.block_size]]) + 1
        gradient[0:self.block_size] = np.linalg.solve(blkdiag_kernel_matrix, blk_residual)
        for i in range(1, self.s):
            residual_correction = np.zeros(self.block_size)
            blk_idxs = range(i*self.block_size, (i+1)*self.block_size)
            blk_residual = residual[blk_idxs]
            blkdiag_kernel_matrix = ((1/self.C) * blkrow_kernel_matrix[blk_idxs, idxs[blk_idxs]]) + 1
            for j in range(i):
                blk_gradient = gradient[j*self.block_size:(j+1)*self.block_size]
                innerblk_idxs = idxs[j*self.block_size, (j+1)*self.block_size]
                commonvals, idxof_outerblk, idxof_innerblk = np.intersect1d(blk_idxs, innerblk_idxs, assume_unique=True, return_indices=True)
                residual_correction[idxof_outerblk] = blk_gradient[idxof_innerblk]
                residual_correction += (1/self.C) * (blkrow_kernel_matrix[blk_idxs, j*self.block_size:(j+1)*self.block_size].dot(blk_gradient))
            blk_residual -= residual_correction
            gradient[blk_idxs] = np.linalg.solve(blkdiag_kernel_matrix, blk_residual)
        self.alpha[idxs] += gradient

    def train(self, epochs=100, optimizer='bcd', optimizer_params={'C': 1e-5, 'block_size': 1, 'shuffle': True}):
        if optimizer.lower() == 'bcd':
            optimizer = self.block_coordinate_descent
        elif optimizer.lower() == 'cabcd':
            optimizer = self.commavoid_block_coordinate_descent
            self.s = optimizer_params['s']
        else:
            raise ValueError('Invalid optimizer choice: {}'.format(optimizer))
        self.C = optimizer_params['C']
        self.block_size = optimizer_params['block_size']
        self.shuffle = optimizer_params['shuffle']
        loss_history = []
        for epoch in range(1,epochs+1):
            if self.shuffle:
                idxs = np.random.choice(self.nsamples, size=self.nsamples, replace=False)
            else:
                idxs = np.arange(self.nsamples)
            nblocks = len(idxs)/self.block_size
            for block in np.array_split(idxs, nblocks):
                if len(block) < 1:
                    break
                optimizer(block)
            loss_history.append(self.compute_objective())
            epoch_strlen = len(str(epochs))
            print('[{}] loss {:.16f}.'.format(str(epoch).zfill(epoch_strlen), loss_history[-1]))
        plt.plot(loss_history)

    def test(self):
        raise NotImplementedError()

```

```python
from sklearn.datasets import load_svmlight_file

(data, labels) = load_svmlight_file('../data/abalone')
krr = Kernel_Ridge_Regression(data, labels)
krr.train(epochs=100)
```


```python
from IPython.display import Math

from sympy import Symbol
from sympy.core.containers import Tuple
from sympy import pi, cos, sin, exp
from sympy import expand

from sympde.core import Constant
from sympde.calculus import grad, dot, inner, outer, cross, rot, curl, div
from sympde.calculus import laplace, hessian, bracket, convect, D, conv
from sympde.calculus import ArgumentTypeError
from sympde.calculus import jump, avg, Dn, minus, plus
from sympde.topology import Domain, Boundary
from sympde.topology import ScalarFunctionSpace, VectorFunctionSpace
from sympde.topology import ProductSpace
from sympde.topology import H1Space, HcurlSpace, HdivSpace, L2Space, UndefinedSpace
from sympde.topology import TestFunction, ScalarTestFunction, VectorTestFunction
from sympde.topology import ScalarField, VectorField
from sympde.topology import element_of, elements_of
from sympde.expr     import LinearForm
from sympde.expr     import integral
from sympde.expr     import linearize
from sympde.printing.latex import latex
```

# Nonlinear problems

### Nonlinear poisson

Let us consider the weak formulation $l(v)=0$, where

\begin{align}
  l(v) = \int_{\Omega} \nabla u \cdot \nabla v + e^{-u}v ~d\Omega
  \label{eq:nonlinear-example-wf}
\end{align}

The linearization of such a linear form means that we replace $u$ by an expansion $u + \epsilon \delta u$ then we compute the Taylor expansion with respect to $\epsilon$.
The linearisation leads to the construction of a bilinear form that has arguments $\left( \delta u, v \right)$, as shown in the follozing Python code:

```python
domain = Domain('Omega', dim=2)
Gamma_N = Boundary('Gamma_N', domain.interior)
x,y = domain.coordinates

V = ScalarFunctionSpace('V', domain)

u, du,v = elements_of(V, names='u, delta_u, v')

int_0 = lambda expr: integral(domain, expr)

expr = int_0(dot(grad(v), grad(u)) -4.*exp(-u)*v)
l = LinearForm(v, expr )
```

```python
# linearising l around u, using du
a = linearize(l, u, trials=du)
```

```python
Math(latex(a))
```

$\displaystyle \left( \delta_{u}, v \right) \mapsto \int_{\Omega} 4.0 \delta_{u} v e^{- u} ~d\mathbf{x} + \int_{\Omega} \nabla{\delta_{u}} \cdot \nabla{v} ~d\mathbf{x}$

```python

```


### Import sympy parametric plot modules

```python
%reset -f
from sympy import *
from sympy.plotting import plot_parametric
from sympy.plotting import plot3d_parametric_line
import math

init_printing()        # for nice math output

# uncomment for plot in separate window
%matplotlib qt

# uncomment for plot inline --- DEFAULT
#%matplotlib inline
```

### Declare symbolic variables

```python
t = symbols('t')
```

```python
# 2d - Example 1
x = cos(t)
y = sin(t)
plot_parametric(x, y, (t, 0, 2*pi))
```

```python
# 2d - Example 2
x = t + 0.5 * cos(10*t)
y = t + 0.5 * sin(10*t)
plot_parametric(x, y, (t, -2*pi, 2*pi))
```

```python
# 2d - Example 3 - Cycloid
x = t - sin(t)
y = 1 - cos(t)
plot_parametric(x, y, (t, -2*pi, 2*pi))
```

```python
# 2d - Example 4
x = 0.1*t*cos(t)
y = 0.1*t*sin(t)
plot_parametric(x, y, (t, 0, 10*pi))
```

```python
# 3d - Example 1
import math
x = sin(t/10)*cos(t)
y = sin(t/10)*sin(t)
z = t

# uncomment for plot in separate window
%matplotlib qt

plot3d_parametric_line(x, y, z, (t, -10*math.pi, 10*math.pi))

```

    <sympy.plotting.plot.Plot at 0x7f433c50d490>

```python
# 3d - Example 2
x = t*cos(t)
y = t*sin(t)
z = t

plot3d_parametric_line(x, y, z, (t, 0, 6*math.pi))
```

```python

```


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
import sympy as sym
from approx1D import least_squares_orth, comparison_plot
import matplotlib.pyplot as plt

x = sym.Symbol('x')

# Naive approach: (not utilizing the fact that i+1 computations can
# make use of i computations)
def naive(f, s, Omega, N=10):
    psi = []
    for i in range(N+1):
        psi.append(sym.sin((2*i+1)*x))
        u, c = least_squares_orth(f, psi, Omega, symbolic=False)
        comparison_plot(f, u, Omega, 'tmp_sin%02dx' % i,
                        legend_loc='upper left', show=True)

# Efficient approach: compute just the matrix diagonal
def efficient(f, s, Omega, N=10):
    u = 0
    for i in range(N+1):
        psi = [sym.sin((2*i+1)*x)]
        next_term, c = least_squares_orth(f, psi, Omega, False)
        u = u + next_term
        comparison_plot(f, u, Omega, 'tmp_sin%02dx' % i,
                        legend_loc='upper left', show=False,
                        plot_title='s=%g, i=%d' % (s, i))

if __name__ == '__main__':
    s = 20  # steepness
    f = sym.tanh(s*(x-sym.pi))
    from math import pi
    Omega = [0, 2*pi]  # sym.pi did not work here
    efficient(f, s, Omega, N=10)
    # Make movie
    # avconv/ffmpeg skips frames, use convert instead (few files)
    cmd = 'convert -delay 200 tmp_sin*.png tanh_sines_approx.gif'
    os.system(cmd)
    # Make static plots, 3 figures on 2 lines
    for ext in 'pdf', 'png':
        cmd = 'doconce combine_images %s -3 ' % ext
        cmd += 'tmp_sin00x tmp_sin01x tmp_sin02x tmp_sin04x '
        cmd += 'tmp_sin07x tmp_sin10x tanh_sines_approx'
        os.system(cmd)
    plt.show()

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/24_TANH_SINES.ipynb


# Quantum Harmonic Osscilator
### Christina Lee
### Physics Prerequisites: Quantum Mechanics
### Quantum Time Evolution Series
* [Quantum Harmonic Osscillator](../Prerequisites/QHO.ipynb)
* [Quantum Time Evolution](../Prerequisites/Time-Evolution.ipynb)
* [Quantum Ground States: Imaginary Time](../Prerequisites/Imaginary-Time-Evolution.ipynb)

Slinkies. They started out as toys.  I still have one to play with on my desk.
Rubber bands What was once something useful, is now a wonderful projectile weapon.
 Swings I still love them, but people seem to not make them in adult sizes for some reason.

A person's perception of these objects start to change as they enter their first physics class.  Even in that beginning classical mechanics, the problems are filled with harmonic osscilators, like slinkies, rubber bands, or swings, which exert a force proportional to their displacement
\begin{equation}
F=-kx
\end{equation}
and therefore a quadratic potential
\begin{equation}
V(x)=\frac{1}{2} k x^2
\end{equation}

This is all extremely fun and useful in the classical regime, but we add Quantum Mechanics to the mix, and LOW AND BEHOLD! we have one of the few exactly solvable models in Quantum Mechanics. Moreso, this solution demonstrates some extremely important properties of quantum mechanical systems.

##### The Hamiltonian
\begin{equation}
    {\cal H}= \frac{p^2}{2 m} + \frac{1}{2} m \omega ^2 x^2
\end{equation}

##### The Solution
\begin{equation}
    \Psi (x) = \frac{1}{\sqrt{2^n n!}} \left(\frac{m \omega}{\hbar \pi}\right)^{1/4} \mathrm{e}^{-m \omega x^2/2 \hbar}  H_n \left( \sqrt{\frac{m \omega}{\hbar}} x \right)
\end{equation}
Today, I just intend to present the form of the solution, calculate this equation numerically, and visualize the results.  If you wish to know how the equation is derived, you can find a standard quantum mechanics textbook, or stay tuned till I manage to write it up.

### Physicists' Hermite Polynomials
Note: These are not the same as the "probabilists' Hermite Polynomial".  The two functions differ by scaling factors.

Physicists' Hermite polynomials are defined as eigenfunctions for the differential equation
\begin{equation}
u^{\prime \prime}-2xu^{\prime} = -2 \lambda u
\end{equation}

\begin{equation}
H_n(x) = (-1)^n \mathrm{e}^{x^2} \frac{\mathrm{d}^n}{\mathrm{d}x^n}
\left( e^{-x^2} \right)
\end{equation}

I leave it as an exercise to the reader (muahahahaha) to
* demonstrate othogonality with respect to the measure $e^{-x^2}$, ie
\begin{equation}
\int_{-\infty}^{\infty} H_m(x) H_n(x) e^{-x^2} \mathrm{d}x = \sqrt{\pi} 2^n n! \delta_{mn}
\end{equation}
* demonstrate completeness.  This means we can describe every function by a linear combination of Hermite polynomials, provided it is suitably well behaved.

Though a formula exists for calculating a function at $n$ directly, the most efficient method at low $n$ for calculating polynomials relies on recurrence relationships.  These recurrence relationships will also be quite handy if you ever need to show orthogonality, or expectation values.
##### Recurrence Relations
\begin{equation}
H_{n+1}(x) = 2xH_n(x) - H^{\prime}_n(x)
\end{equation}
\begin{equation}
H^{\prime}_n(x) = 2n H_{n-1}(x)
\end{equation}
\begin{equation}
H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x)
\end{equation}

```julia
using Roots;
using Plots
pyplot()
```

    Plots.PyPlotBackend()

#### Programming Tip!
Since Hermite polynomials are generated recursively, I wanted to generate and save all the functions up to a designated value at once.  In order to do so, I created an array, whose values are anonymous functions.

```julia
function GenerateHermite(n)
    Hermite=Function[]

    push!(Hermite,x->1);
    push!(Hermite,x->2*x);

    for ni in 3:n
        push!(Hermite,x->2*x.*Hermite[ni-1](x)-2*n.*Hermite[ni-2](x))
    end
    return Hermite
end
```

    GenerateHermite (generic function with 1 method)

So let's generate some Hermite polynomials and look at them.
 Make sure you don't call a Hermite you haven't generated yet!

```julia
Hermite=GenerateHermite(5);
```

#### Programming Tip!
Since the Hermite polynomials, and the wavefunctions after them, are composed on anonymous functions, we need to use `map(f,x)` in order to map the function `f` onto the array `x`.  Otherwise our polynomials only work on single values.

```julia
x=collect(-2:.01:2);
plot(ylims=(-50,50))
for j in 1:5
    plot!(x,map(Hermite[j],x),label="H_$j (x)")
end
plot!(xlabel="x",ylabel="y",title="Hermite Polynomials")
```

```julia
# Let's make our life easy and set all units to 1
m=1
ω=1
ħ=1

#Finally, we define Ψ
Ψ(n,x)=1/sqrt(factorial(n)*2^n)*(m*ω/(ħ*π))^(1/4)*exp(-m*ω*x^2/(2*ħ))*Hermite[n](sqrt(m*ω/ħ)*x)
```

    Ψ (generic function with 1 method)

### Finding Zeros
The eigenvalue maps to the number of zeros in the wavefunction.  Below, I use Julia's roots package to identify roots on the interval from -3 to 3.

```julia
zeds=Array{Array{Float64}}(undef,1)
zeds[1]=[] #ground state has no zeros
for j in 2:4
    push!(zeds,fzeros(y->Ψ(j,y),-3,3))
end
```

```julia
zeds[3]
```

    2-element Array{Float64,1}:
     -1.5811388300841895
      1.5811388300841895

```julia
# AHHHHH! So Much code!
# Don't worry; it's all just plotting
x=collect(-3:.01:3)  #Set some good axes

plot(xlim=(-3,3),ylim=(-.5,4.5))
for j in 1:4    #how many do you want to view?
    plot!(x,map.(y->Ψ(j,y),x).+j.-1,label="| $j >")
    plot!(x,(j-1)*ones(length(x)),
        color="black",label="")
    scatter!(zeds[j],(j-1)*ones(length(zeds[j])),
        label="")
end

plot!(x,.5*m*ω^2*x.^2,label="Potential")

scatter!([],[],label="Zeros")

plot!(xlabel="x",ylabel="Ψ+n",legend=:topright,
    title="Eigenstates of a Harmonic Osscilator")
```

## Example Result

## More to come
This barely scratched the surface into the richness that can be seen in the quantum harmonic osscilator.  Here, just we developed a way for calculating the functions, and visualized the results.  Stay tuned to hear here about ground state energy, ladder operators, and atomic trapping.

```julia

```


## **Numerical Method of First Derivative using Taylor Series Approximation**

#### This exercise covers the following aspects:

* Calculation of numerical first derivative
* Comparison with analytical solution
* Dependence of error on space increment

```python
# Import Libraries
import numpy as np
from math import *
import matplotlib.pyplot as plt
```

We initialize a space-dependent *sin* function

\begin{equation}
f(x)= \sin (k x)
\end{equation}

where the wavenumber k is

\begin{equation}
k = \dfrac{2 \pi}{\lambda}
\end{equation}

and $\lambda$ is wavelength.

```python
# Initial parameters
xmax = 10.0                     # physical domain (m)
nx = 200                        # number of samples
dx = xmax/(nx-1)                # grid increment dx (m)
x  = np.linspace(0,xmax,nx)     # space coordinates

# Initialization of sin function
l = 20*dx     # wavelength
k = 2*pi/l    # wavenumber
f = np.sin(k*x)
```

```python
# Plot sin function
plt.figure(figsize=(10,6))
plt.plot(x, f)
plt.title('Sin function')
plt.xlabel('x, m')
plt.ylabel('Amplitude')
plt.xlim((0, xmax))
plt.grid()
plt.show()
```

In the cell below we calculate the central finite-difference derivative of f(x) using two points

\begin{equation}
f^{\prime}(x)=\dfrac{f(x+dx)-f(x-dx)}{2dx}
\end{equation}

and  compare with the analytical derivative

\begin{equation}
f^{\prime}(x) = k \cos(k x)
\end{equation}

```python
# First derivative with two points

# Initiation of numerical and analytical derivatives
nder=np.zeros(nx)          # numerical derivative
ader=np.zeros(nx)          # analytical derivative

# Numerical derivative of the given function
for i in range (1, nx-1):
    nder[i]=(f[i+1]-f[i-1])/(2*dx)

# Analytical derivative of the given function
ader= k * np.cos(k*x)
# Exclude boundaries
ader[0]=0.
ader[nx-1]=0.

# Error (rms)
rms = np.sqrt(np.mean(nder-ader)**2)
```

```python
# Plotting
# ----------------------------------------------------------------
plt.figure(figsize=(10,6))
plt.plot (x, nder,label="Numerical Derivative, 2 points", marker='+', color="blue")
plt.plot (x, ader, label="Analytical Derivative", lw=2, ls="-",color="black")
plt.plot (x, nder-ader, label="Difference", lw=2, ls=":")
plt.title("First derivative, Err (rms) = %.6f " % (rms) )
plt.xlabel('x, m')
plt.ylabel('Amplitude')
plt.legend(loc='lower left')
plt.grid()
plt.show()
```

#### The concept of number of points per wavelength

\begin{equation}
n_\lambda = \dfrac{\lambda}{dx}
\end{equation}

How does the error of the numerical derivative change with the number of points per wavelength?

```python
# Plotting number of points per wavelength
plt.figure(figsize=(10,6))
plt.plot (x, nder,label="Numerical Derivative, 2 points", marker='+', color="blue")
plt.title("First derivative, Error = %.6f, $n_\lambda$ = %.2f " % ( rms, l/dx) )
plt.xlabel('x, m')
plt.ylabel('Amplitude')
plt.legend(loc='lower left')
plt.xlim((xmax/2-l,xmax/2+l))
plt.grid()
plt.show()
```

#### Investigate the error as a function of grid points per wavelength

```python
# Define a range of number of points per wavelength: [nmin=4,5,6 ... ,nmax=40]
# Loop over points, calculate corresponding wavelength and calculate error

# Initialize vectors
nmin=3
nmax=16
na =  np.zeros(nmax-nmin)    # Vector with number of points per wavelength
err = np.zeros(nmax-nmin)    # Vector with error

j = -1  # array index

# Loop through finite-difference derivative calculation
for n in range (nmin,nmax):

    j = j+1   # array index
    na[j] = n

    # Initialize sin function
    l = na[j]*dx  # wavelength
    k = 2*pi/l    # wavenumber
    f = np.sin(k*x)

    # Numerical derivative of the sin function
    for i in range (1, nx-1):
        nder[i]=(f[i+1]-f[i-1])/(2*dx)

    # Analytical derivative of the sin function
    ader= k * np.cos(k*x)
    # Exclude boundaries
    ader[0]=0.
    ader[nx-1]=0.

    i0 = np.int(nx/2)
    # Error (rms)
    err[j] = (nder[i0]-ader[i0])**2/ader[i0]**2 * 100
```

```python
# ----------------------------------------------------------------
# Plotting error as function of number of points per wavelength
plt.figure(figsize=(10,6))
plt.plot(na,err, ls='-', color="blue")
plt.title('Error as a function of $n_\lambda$ ')
plt.xlabel('n$_\lambda$')
plt.ylabel('rms ')
plt.grid()
plt.show()
```

### Conclusions

* 2-point finite-difference approximations can provide  estimates of the 1st derivative of a function
* The accuracy depends on the "number of points per wavelength", i.e., how well we sample the original function
* The more points we use the more accurate is the derivative approximation


# Intro to Mathematics with Code

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Set seed for reproducibility
np.random.seed(10)
# Set max rows displayed for readability
pd.set_option('display.max_rows', 6)
# Plot settings
plt.style.use('plot_style.txt')
```

In this chapter, you'll learn about doing mathematics with code, including solving equations both in the abstract and numerically.

This chapter uses the **numpy**, **scipy**, and **sympy** packages. If you're running this code, you may need to install these packages using, for example, `pip install packagename` on your computer's command line. (If you're not sure what a command line or terminal is, take a quick look at the basics of coding chapter.)

## Symbolic mathematics

```python
from myst_nb import glue
import sympy
a = 8
glue('sqrt', 2*np.sqrt(a))
glue('symsqrt', sympy.sqrt(a))
```

When using computers to do mathematics, we're most often performing numerical computations such as $\sqrt{8} = ${glue:}`sqrt`. Although we have the answer, it's only useful for the one special case. Symbolic mathematics allows us to use coding to solve equations in the general case, which can often be more illuminating. As an example, if we evaluate this in symbolic mathematics we get $\sqrt{8} = ${glue:}`symsqrt`.

The Python package for symbolic mathemtics is [**sympy**](https://www.sympy.org/en/index.html), which provides some features of a computer algebra system.

To define *symbolic* variables, we use sympy's symbols function. For ease, we'll import the entire sympy library into the namespace by using `from sympy import *`.

```python
from sympy import *
x, t, α, β = symbols(r'x t \alpha \beta')
```

```{note}
The leading 'r' in some strings tells Python to treat the string literally so that backslashes are not treated as instructions--otherwise, combinations like `\n` would begin a newline.
```

Having created these symbolic variables, we can refer to and see them just like normal variables--though they're not very interesting *because* they are just symbols (for now):

```python
α
```

Things get much more interesting when we start to do maths on them. Let's see some integration, for example, say we want to evaluate

```python
Integral(log(x), x)
```

(note that the symbols are printed as latex equations) we simply call

```python
integrate(log(x), x)
```

We can differentiate too:

```python
diff(sin(x)*exp(x), x)
```

and even take limits!

```python
limit(sin(x)/x, x, 0)
```

It is also possible to solve equations using **sympy**. The solve function tries to find the roots of $f(x)$ and has syntax `solve(f(x)=0, x)`. Here's an example:

```python
solve(x*5 - 2, x)
```

There are also solvers for differential equations (`dsolve`), continued fractions, simplifications, and more.

Another really important thing to know about symbolic mathematics is that you can 'cash in' at any time by substituting in an actual value. For example,

```python
expr = 1 - 2*sin(x)**2
expr.subs(x, np.pi/2)
```

But you don't have to substitute in a real value; you can just as well substitute in a different symbolic variable:

```python
expr = 1 - 2*sin(x)**2
simplify(expr.subs(x, t/2))
```

I snuck in a simplify here too!

### Symbolic mathematics for economics

The library does a lot, so let's focus on a few features that are likely to be useful for economics in particular.

#### Series expansion

The first is performing **Taylor series expansions**. These come up all the time in macroeconomic modelling, where models are frequently log-linearised. Let's see an example of a couple of expansions together:

```python
expr = log(sin(α))

expr.series(α, 0, 4)
```

This is a 3rd order expansion around $\alpha=0$.

#### Symbolic linear algebra

The support for **matrices** can also come in handy for economic applications. Here's a matrix,

```python
M = Matrix([[1, 0, x], [α, -t, 3], [4, β, 2]])
M
```

and its determinant:

```python
M.det()
```

I can hardly go to a talk in economics that involves matrices that doesn't see those matrices get diagonalised: there's a function for that too.

```python
P, D = Matrix([[1, 0], [α, -t]]).diagonalize()
D
```

#### Lagrangians

Function optimisation using Lagrangians is about as prevalent in economics as any bit of maths: let's see how it's done symbolically.

We're going to find the minimum over x, y of the function $f(x,y)$, subject to $g(x,y)=0$, where $f(x,y) = 4xy - 2x^2 + y^2$ and $g(x,y) = 3x+y-5$.

First we need to specify the problem, and the Lagrangian for it, in code

```python
x, y, λ = symbols(r'x y \lambda', real=True)
f = 4*x*y - 2*x**2 + y**2
g = 3*x+y-5

ℒ = f - λ*g
ℒ
```

The Karush-Kuhn-Tucker (KKT) conditions tell us whether any solutions we find will be optimal. Simply, the constaint is that a solution vector is a saddle point of the Lagrangian, $\nabla \mathcal{L} = 0$. Let's solve this.

```python
gradL = [diff(ℒ, c) for c in [x, y]]
KKT_eqns = gradL + [g]
```

```python
KKT_eqns = gradL + [g]
glue('kkt_0', KKT_eqns[0])
glue('kkt_1', KKT_eqns[1])
glue('kkt_2', KKT_eqns[2])
```

This gives 3 equations from the KKT conditions:  {glue:}`kkt_0`,   {glue:}`kkt_1`, and  {glue:}`kkt_2`. (The symbolic manipulation is now over: we solved for the conditions in terms of algebra--now we're looking for real values.) Now we look for the values of $x, y$ that minimise $f$ given that $g=0$ by solving these equations over $x$, $y$, and $\lambda$.

```python
stationary_pts = solve(KKT_eqns, [x, y, λ], dict=True)
stationary_pts
```

Now, we can substitute these in to find the (first--and in this case only) point that minimises our function:

```python
stationary_pts[0][x], stationary_pts[0][y], f.subs(stationary_pts[0])
```

#### Exporting to latex

To turn any equation, for example `diff(sin(x)*exp(x), x)`, into latex and export it to a file that can be included in a paper, use

```python
eqn_to_export = latex(diff(sin(x)*exp(x), x), mode='equation')
open('latex_equation.tex', 'w').write(eqn_to_export)
```

which creates a file called 'latex_equation.tex' that has a single line in it: '\begin{equation}\int \log{\left(x \right)}\, dx\end{equation}'. There are a range of options for exporting to latex, `mode='equation*'` produces an unnumbered equation, 'inline' produces an inline equation, and so on. To include these in your latex paper, use '\input{latex_equation.tex}'.

### Why coding symbolic mathematics is useful

1. Accuracy--using a computer to solve the equations means you're less likely to make a mistake. At the very least, it's a useful check on your by-hand working.

2. Consistency--by making your code export the equations you're solving to your write-up, you can ensure that the equations are consistent across both *and* you only have to type them once.

## Numerical Mathematics

For much of the time, you'll be dealing with numbers rather than symbols. The workhorses of numerical mathematics are the two packages **numpy** and **scipy**. Both have excellent documentation, where you can find out more. In this section, we'll look at how to use them in some standard mathematical operations that arise in economics.

The most basic object is an array, which can be defined as follows:

```python
import numpy as np
a = np.array([0, 1, 2, 3], dtype='int64')
a
```

Arrays are very memory efficient and fast objects that you should use in preference to lists for any heavy duty numerical operation.

To demonstrate this, let's do a time race between lists and arrays for squaring all elements of an array:

Lists:

```python
a_list = range(1000)
%timeit [i**2 for i in a_list]
```

Arrays:

```python
a = np.arange(1000)
%timeit a**2
```

Using arrays was *two orders of magnitude** faster! Okay, so we should use arrays for numerical works. How do we make them? You can specify an array explicitly as we did above to create a vector. This manual approach works for other dimensions too:

```python
mat = np.array([[0, 1, 2], [3, 4, 5], [6, 7, 8]])
mat
```

To find out about matrix properties, we use `.shape`

```python
mat.shape
```

We already saw how `np.arange(start, stop, step)` produces a vector; `np.linspace(start, stop, number)` produces a vector of length `number` by equally dividing the space between `start` and `stop`.

Three really useful arrays are `np.ones(shape)`, for example,

```python
np.ones((3, 3))
```

`np.diag` for diagnoal arrays,

```python
np.diag(np.array([1, 2, 3, 4]))
```

and `np.zeros` for empty arrays:

```python
np.zeros((2, 2))
```

Random numbers are supplied by `np.random.rand()` for a uniform distribution in [0, 1], and `np.random.randn()` for numbers drawn from a standard normal distribution.

You can, of course, specify a function to create an array:

```python
c = np.fromfunction(lambda i, j: i**2+j**2, (4, 5))
c
```

To access values in an array, you can use all of the by-position slicing methods that you've seen already in data analysis and with lists. The figure gives an example of some common slicing operations:

Arrays can also be sliced and diced based on boolean indexing, just like a dataframe.

For example, using the array defined above, we can create a boolean array of true and false values from a condition such as `c > 6` and use that to only access some elements of an array (it doesn't have to be the same array, though it usually is):

```python
c[c > 6]
```

As with dataframes, arrays can be combined. The main command to remember is `np.concatenate`, which has an `axis` keyword option.

```python
x = np.eye(3)
np.concatenate([x, x], axis=0)
```

Splitting is performed with `np.split(array, splits, axis=)`, for example

```python
np.split(x, [3], axis=0)
```

Aggregation operations are very similar to those found in dataframes: `x.sum(i)` to sum across the $i$th dimension of the array; similarly for standard deviation, and so on.

As with dataframes, you can (and often should) specify the datatype of an array when you create it by passing a `dtype=` keyword, eg `c = np.array([1, 2, 3], dtype=float)`. To find out the data type of an array that already exists, use `c.dtype`.

Finally, numpy does a lot of smart broadcasting of arrays. Broadcasting is what means that summing two arrays gives you a third array that has elements that are each the sum of the relevant elements in the two original arrays. Put another way, it's what causes `x + y = z` (for arrays x and y with the same shape) to result in an array z for which $z_{ij} = x_{ij} + y_{ij}$.

Summing two arrays of the same shape is a pretty obvious example, but it also applies to cases that are *not* completely matched. For example, multiplication by a scalar is broadcast across all elements of an array:

```python
x = np.ones(shape=(3, 3))
x*3
```

Similarly, numpy functions are broadcast across elements of an array:

```python
np.exp(x)
```

### Numeric linear algebra

The transpose of an array `x` is given by `x.T`.

Matrix multiplation is performed using the `@` operator. Here we perform $ M_{il} = \sum_{k} x_{ik} * (x^T)_{kl}$, where $x^T$ is the transpose of $x$.

```python
x @ x.T
```

To multiply two arrays element wise, ie to do $ M_{ij} = x_{ij} y_{ij}$, it's the usual multiplication operator `*`.

Inverting matrices:

```python
a = np.random.randint(9, size=(3, 3), dtype='int')
b = a @ np.linalg.inv(a)
b
```

Computing the trace:

```python
b.trace()
```

Determinant:

```python
np.linalg.det(a)
```

Computing a Cholesky decomposition, i.e. finding lower triangular matrix $C$ such that $C C' = \Sigma$ for $\Sigma$ a 2-dimensional positive definite matrix.

```python
Σ = np.array([[4, 1], [1, 3]])
c = np.linalg.cholesky(Σ)

c @ c.T - Σ
```

#### Solving systems of linear equations

Say we have a system of equations, $4x + 3y + 2z = 25$,
$-2x + 2y + 3z = -10$, and $3x -5y + 2z = -4$. We can solve these three equations for the three unknowns, x, y, and z, using the `solve` method. First, remember that this equation can be written in matrix form as

$$
M\cdot \vec{x} = \vec{c}
$$

We can solve this by multiplying by the matrix inverse of $M$:

$$
M^{-1} M \cdot \vec{x} = I \cdot \vec{x} = M^{-1} \cdot \vec{c}
$$

which could be called by running `x = la.inv(M).dot(c)`. There's a convenience function in **numpy** called solve that does the same thing: here it finds the real values of the vector $\vec{x}$.

```python
M = np.array([[4, 3, 2], [-2, 2, 3], [3, -5, 2]])
c = np.array([25, -10, -4])
np.linalg.solve(M, c)
```

Finally, eigenvalues and eigenvectors can be found from:

```python
import scipy.linalg as la
eigvals, eigvecs = la.eig(M)
eigvals
```

### Interpolation

This section draws on the **scipy** documentation. There are built-in **pandas** methods for interpolation in dataframes, but **scipy** also has a range of functions for this including for univariate data `interp1d`, multidimensional interpolation on a grid `interpn`, `griddata` for unstructured data. Let's see a simple example with interpolation between a regular grid of integers.

```python
import matplotlib.pyplot as plt
from scipy import interpolate
x = np.arange(0, 10)
y = np.exp(-x/3.0)
f = interpolate.interp1d(x, y, kind='cubic')
# Create a finer grid to interpolation function f
xnew = np.arange(0, 9, 0.1)
ynew = f(xnew)
plt.plot(x, y, 'o', xnew, ynew, '-')
plt.show()
```

What about unstructured data? Let's create a Cobb-Douglas function on a detailed grid but then only retain a random set of the established points.

```python
from scipy.interpolate import griddata

def cobb_doug(x, y):
    alpha = 0.8
    return x**(alpha)*y**(alpha-1)

# Take some random points of the Cobb-Douglas function
points = np.random.rand(1000, 2)
values = cobb_doug(points[:,0], points[:,1])

# Create a grid
grid_x, grid_y = np.mgrid[0.01:1:200j, 0.01:1:200j]

# Interpolate the points we have onto the grid
interp_data = griddata(points, values, (grid_x, grid_y), method='cubic')

# Plot results
fig, axes = plt.subplots(1, 2)
# Plot function & scatter of random points
axes[0].imshow(cobb_doug(grid_x, grid_y).T,
               extent=(0, 1, 0, 1),
               origin='lower', cmap='plasma_r',
               vmin=0, vmax=1)
axes[0].plot(points[:, 0], points[:, 1], 'r.', ms=1.2)
axes[0].set_title('Original + points')
# Interpolation of random points
axes[1].imshow(interp_data.T, extent=(0, 1, 0, 1),
               origin='lower', cmap='plasma_r',
               vmin=0, vmax=1)
axes[1].set_title('Cubic interpolation');
```

### Optimisation

**scipy** has functions for minimising scalar functions, minimising multivariate functions with complex surfaces, and root-finding. Let's see an example of finding the minimum of a scalar function.

```python
from scipy import optimize

def f(x):
    return x**2 + 10*np.sin(x) - 1.2

result = optimize.minimize(f, x0=0)
result
```

The result of the optimisation is in the 'x' attribute of `result`. Let's see this:

```python
x = np.arange(-10, 10, 0.1)
fig, ax = plt.subplots()
ax.plot(x, f(x))
ax.scatter(result.x, f(result.x), s=150, color='k')
ax.set_xlabel('x')
ax.set_ylabel('f(x)', rotation=90)
plt.show()
```

In higher dimensions, the minimisation works in much the same way, with the same function `optimize.minimize`. There are a LOT of minimisation options that you can pass to the `method=` keyword; the default is intelligently chosen from BFGS, L-BFGS-B, or SLSQP, depending upon whether you supply constraints or bounds.

Root finding, aka solving equations of the form $f(x)=0$, is also catered for by **scipy**, through `optimize.root`. It works in much the same way as `optimizer.minimize`.

In both of these cases, be warned that multiple roots and multiple minima can be hard to detect, and you may need to carefully specify the bounds or the starting positions in order to find the root you're looking for. Also, both of these methods can accept the Jacobian of the function you're working with as an argument, which is likely to improve performance with some solvers.

### Numerical Integration

**scipy** provides routines to numerically evaluate integrals in `scipy.integrate`, which you can find the documentation for [here](https://docs.scipy.org/doc/scipy/reference/integrate.html). Let's see an example using the 'vanilla' integration method, `quad`, to solve a known function between given (numerical) limits:

$$
\displaystyle\int_0^{\pi} \sin(x) d x
$$

```python
from scipy.integrate import quad
res, err = quad(np.sin, 0, np.pi)
res
```

What if we just have data samples? In that case, there are several routines that perform purely numerical integration:

```python
from scipy.integrate import simps
x = np.arange(0, 10)
f_of_x = np.arange(0, 10)

simps(f_of_x, x) - 9**2/2
```

Even with just 10 evenly spaced points, the composite Simpson’s rule integration given by `simps` is able to accurately find the answer as $\left( x^2/2\right) |_{0}^{9}$.

## Advanced: Composable Function Transformations

In recent years, there have been great developments in the ability of Python to easily carry out numerical 'composable function transformations'. What this means is that, if you can dream up an arbitrary numerical operations -- including differentiation, linear algebra, and optimisation -- you can write code that will execute it quickly and automatically on CPUs, GPUs, or TPUs as you like.

Here we'll look at one library that does this, **jax**, developed by Google {cite}`jax2018github`. It can automatically differentiate native Python and **numpy** functions, including when they are in loops, branches, or subject to recursion, and it can take derivatives of derivatives of derivatives. It supports reverse-mode differentiation (a.k.a. backpropagation) via grad as well as forward-mode differentiation, and the two can be composed arbitrarily to any order.

To do these at speed, it uses just-in-time compilation. If you don't know what that is, don't worry: the details aren't important. It's just a way of getting close to C++ or Fortran speeds while still being able to write code in *much* more user friendly Python!

### Auto-differentiation

Let's see an example of auto-differentiation an arbitrary function. We'll write the definition of $\tanh(x)$ as a function and evaluate it. Because we already imported a (symbolic) `tanh` function from Sympy above, we'll call the function below `tanh_num`.

```python
from jax import grad
import jax.numpy as jnp

def tanh_num(θ):  # Define a function
  y = jnp.exp(-2.0 * θ)
  return (1.0 - y) / (1.0 + y)

grad_tanh = grad(tanh_num)  # Obtain its gradient function
grad_tanh(1.0)   # Evaluate it at x = 1.0
```

```bash
DeviceArray(0.4199743, dtype=float32)
```

You can differentiate to any order using grad:

```python
grad(grad(grad(tanh_num)))(1.0)
```

```bash
DeviceArray(0.6216266, dtype=float32)
```

Let's check this using symbolic mathematics:

```python
θ = Symbol(r'\theta')
triple_deriv = diff(diff(diff(tanh(θ), θ)))
triple_deriv
```

```python
symp_est = triple_deriv.subs(θ, 1.)
glue('symp_est', f'{symp_est:.3f}')
```

If we evaluate this at $\theta=1$, we get {glue:}`symp_est`. This was a simple example that had a (relatively) simple mathematical expression. But imagine if we had lots of branches (eg if, else statements), and/or a really complicated function: **jax**'s grad would still work. It's designed for really complex derivatives of the kind encountered in machine learning.

### Just-in-time compilation

The other nice feature of **jax** is the ability to do just-in-time (JIT) compilation. Because they do not compile their code into machine-code before running, high-level languages like Python and R are not as fast as the same code written in C++ or Fortran (the benefit is that it takes you less time to write the code in the first place). Much of the time, there are pre-composed functions that call C++ under the hood to do these things--but only for those operations that people have already taken the time to code up in a lower level language. JIT compilation offers a compromise: you can code more or less as you like in the high-level language but it will be compiled just-in-time to give you a speed-up!

**jax** is certainly not the only Python package that does this, and if you're not doing anything like differentiating or propagating, **numba** is a more mature alternative. But here we'll see the time difference for JIT compilation on an otherwise slow operation: element wise multiplication and addition.

```python
from jax import jit

def slow_f(x):
  """Slow, element-wise function"""
  return x * x + x * 2.0

x = jnp.ones((5000, 5000))
fast_f = jit(slow_f)
```

Now let's see how fast the 'slow' version goes:

```python
%timeit -n15 -r3 slow_f(x)
```

```bash
60.1 ms ± 3.67 ms per loop (mean ± std. dev. of 3 runs, 15 loops each)
```

what about with the JIT compilation?

```python
%timeit -n15 -r3 fast_f(x)
```

```bash
17.7 ms ± 434 µs per loop (mean ± std. dev. of 3 runs, 15 loops each)
```

This short introduction has barely scratched the surface of **jax** and what you can do with it. For more, see the [official documentation](https://jax.readthedocs.io/en/latest/).

## Set theory

Set theory is a surprisingly useful tool in research (and invaluable in spatial analysis). Here are some really useful bits of set theory inspired by examples in {cite}`sheppard2012introduction`.

Sets are first class citizens in Python in the same way that lists are. We can define and view a set like this:

```python
x = set(['Ada Lovelace', 'Sadie Alexander',
         'Charles Babbage', 'Ada Lovelace',
         'Adam Smith', 'Sadie Alexander'])
x
```

Notice that a couple of entries appeared twice in the list but only once in the set: that's because a set contains only unique elements. Let's define a second set in order to demonstrate some of the operations we can perform on sets.

```python
y = set(['Grace Hopper', 'Jean Bartik',
         'Janet Yellen', 'Joan Robinson',
         'Adam Smith', 'Ada Lovelace'])
y
```

```python
from myst_nb import glue
inters = x.intersection(y)
differ = x.difference(y)
union = x.union(y)
glue("inters", inters)
glue("differ", differ)
glue("union", union)
```

Now we have two sets we can look at to demonstrate some of the basic functions you can call on the set object type. `x.intersection(y)` gives, in this example, {glue:}`inters`, `x.difference(y)` gives {glue:}`differ`, and `x.union(y)` gives {glue:}`union`.

**numpy** also has functions that use set theory. `np.unique` returns only the unique entries of an input array or list:

```python
np.unique(['Lovelace', 'Hopper', 'Alexander', 'Hopper', 45, 27, 45])
```

We can also ask which of a second set is a repeat of a first:

```python
x = np.arange(10)
y = np.arange(5, 10)
np.in1d(x, y)
```

And we have the numpy equivalents of intersection, `np.intersect1d(x, y)`, difference, `np.setdiff1d(x, y)`, and union, `np.union1d(x, y)`. Additionally, there is the exclusive-or (that I like to call 'xor'). This effectively returns the two arrays with their union removed:

```python
a = np.array([1, 2, 3, 2, 4])
b = np.array([2, 3, 5, 7, 5])
np.setxor1d(a,b)
```

## Review

In this chapter, you should have:

- ✅ seen how to use symbolic algebra with code, including Lagrangrians and linear algebra;
- ✅ seen how to code numerical mathematics, including linear algebra and optimisation; and
- ✅ found out about using set theory via the `set` object type and set-oriented functions.


# SPH

### Kernel function

Cubic spline kernel function

\begin{equation}
W(r,h)=\frac{C}{{{h}^{d}}}\left( \begin{matrix}
   1-\frac{3}{2}{{q}^{2}}+\frac{3}{4}{{q}^{3}} & {} & 0\le q<1  \\
   \frac{1}{4}{{(2-q)}^{3}} & {} & 1\le q<2  \\
   0 & {} & q\ge 2  \\
\end{matrix} \right.
\end{equation}

Where $q=|r|/h$  ,Where C is a scaling factor which assures the compliance with the kernel function properties and depends on the dimension of the problem (e.g. d=1 for 1D):

\begin{equation}
C=\left( \begin{matrix}
   \frac{2}{3} & d=1  \\
   \frac{10}{7\pi } & d=2  \\
   \frac{1}{\pi } & d=3  \\
\end{matrix} \right.
\end{equation}

The 1-order and 2-order derivative of kernal function can be expressed as follows:

\begin{equation}
{W}'(r,h)=\frac{C}{{{h}^{d+1}}}\left( \begin{matrix}
   -3q+\frac{9}{4}{{q}^{2}} & 0\le q<1  \\
   -\frac{3}{4}{{(2-q)}^{2}} & 1\le q<2  \\
   0 & q\ge 2  \\
\end{matrix} \right.
\end{equation}

\begin{equation}
{W}''(r,h)=\frac{C}{{{h}^{d+2}}}\left( \begin{matrix}
   -3+\frac{9}{2}q & 0\le q<1  \\
   \frac{3}{2}(2-q) & 1\le q<2  \\
   0 & q\ge 2  \\
\end{matrix} \right.
\end{equation}

Intergral approximation of 2-order dispersion derivation:
$$F({{r}_{i}}-{{r}_{j}})=F(r,h)=\frac{1}{|r|}{W}'(r,h)$$

### Piecewise function input

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def kernelfun (h, r,d):
    q=abs(r/(h**d))
    if  d  == 1: C=2/3
    elif d == 2: C=10/7/np.pi
    elif d == 3: C=1/np.pi
    if 0 <= q < 1:
        return (1-1.5*q*q+0.75*q*q*q) * C
    elif 1 <= q < 2:
        return (0.25*(2-q)**3) * C
    return 0.0

kfun = np.vectorize(kernelfun)

def dkernelfun (h, r,d):
    q=abs(r/(h**(d+1)))
    if   d == 1: C=2/3
    elif d == 2: C=10/7/np.pi
    elif d == 3: C=1/np.pi
    if 0 <= q < 1:
        return (-3*q+2.25*q*q) * C
    elif 1 <= q < 2:
        return (-0.75*(2-q)**2) * C
    return 0.0

dkfun = np.vectorize(dkernelfun)

def ddkernelfun (h, r,d):
    q=abs(r/(h**(d+2)))
    if   d == 1: C=2/3
    elif d == 2: C=10/7/np.pi
    elif d == 3: C=1/np.pi
    if 0 <= q < 1:
        return (-3+4.5*q) * C
    elif 1 <= q < 2:
        return (1.5*(2-q)) * C
    return 0.0

ddkfun = np.vectorize(ddkernelfun)

def Ffun (h, r,d):
    q=abs(r/(h**(d+1)))
    if   d == 1: C=2/3
    elif d == 2: C=10/7/np.pi
    elif d == 3: C=1/np.pi
    if 0 <= q < 1:
        return (1/abs(r))*(-3*q+2.25*q*q) * C
    elif 1 <= q < 2:
        return (1/abs(r))*(-0.75*(2-q)**2) * C
    return 0.0

Fkfun = np.vectorize(Ffun)
```

### Plot the result

```python
# Four axes, returned as a 2-d array
plt.figure(figsize=(16, 16))

x = np.linspace(-3, 3, 1000)
y1 = kfun(1, x,1)
y2 = dkfun(1, x,1)
y3 = ddkfun(1, x,1)
y4 = Fkfun(1, x,1)

plt.subplot(4, 2, 1)
plt.plot(x, y1, '--',lw=2)
plt.xlim(-3, 3)
plt.ylim(0, 0.8)
plt.grid(True)
plt.xlabel('r/h', fontsize=16)
plt.ylabel('W(r/h)', fontsize=16)
plt.tick_params(labelsize=16)

plt.subplot(4, 2, 2)
plt.plot(x, y2, '--',lw=2)
plt.xlim(-3, 3)
plt.ylim(-0.8, 0.005)
plt.grid(True)
plt.xlabel('r/h', fontsize=16)
plt.ylabel('W prime (r/h)', fontsize=16)
plt.tick_params(labelsize=16)

plt.subplot(4, 2, 3)
plt.plot(x, y3, '--',lw=2)
plt.xlim(-3, 3)
plt.ylim(-2, 1)
plt.grid(True)
plt.xlabel('r/h', fontsize=16)
plt.ylabel('dW(r/h)/dr', fontsize=16)
plt.tick_params(labelsize=16)

plt.subplot(4, 2, 4)
plt.plot(x, y4, '--',lw=2)
plt.xlim(-3, 3)
plt.ylim(-2, 0.01)
plt.grid(True)
plt.xlabel('r/h', fontsize=16)
plt.ylabel('F(r/h)', fontsize=16)
plt.tick_params(labelsize=16)

plt.show()
```

Results in Paulo's paper

```python

```

```python

```

```python

```

```python

```


## Linear algebra in NumPy notes

```python
import numpy as np
import sympy as sym
```

```python
a11, a12, a21, a22, b11, b12, b21, b22 = sym.symbols('a11 a12 a21 a22 b11 b12 b21 b22')
A = sym.Matrix([[a11, a12], [a21, a22]])
B = sym.Matrix([[b11, b12], [b21, b22]])
print(A,B)

#x = np.array([[1,2,3,4]])
#y = np.array([[4,5,6,7]]).T
#print(x.shape,y.shape)
#print(np.dot(x,y))

```

    Matrix([[a11, a12], [a21, a22]]) Matrix([[b11, b12], [b21, b22]])

```python

```