## ECON622 Final Project
### Thomas Chan

#### Project Outline

In Guerre, Perrigne and Vuong (2000), a two step kernel based estimator was derived to estimate the latent density of private values using observed bids in first price sealed-bid auctions. They showed that this estimator achieves the best rate for uniform convergence.

In Ma, Marmer and Shneyerov (2019), the asymptotic normality is established using results from empirical processes and V-statistics. They also proposed a consistent estimator for the asymptotic variance as well as showed the validity of bootstrap. One of the key assumptions is that the same bandwidth is used in the first stage, where the density of bids is estimated, and in the second stage where the density of private values is estimated using the inferred pseudo-values from first stage.

In this project, I plan to replicate the Monte-Carlo results in GPV, and some results from Ma, Marmer and Shneyreov (2019).

```julia
using Pkg
Pkg.add(["Distributions","QuadGK","Statistics","Plots","NLsolve","ForwardDiff","StatsBase","StatsPlots"])
```

```julia
using Distributions, QuadGK, Statistics, Plots, NLsolve, ForwardDiff, StatsBase, StatsPlots
```

### Replicating results from Guerre, Perrigne and Vuong (2000)

We begin by replicating Monte Carlo results in Guerre, Perrigne and Vuong(2000). They used L = 200 auctions, each with I = 5 bidders so that there is 1000 observed bids. Private values are drawn from a log normal distribution with parameters (0,1), and truncated at 0.055 and 2.5.

Using the private values, bids are generated using the Bayesian Nash equilibrium which is

\begin{align}
 b_i & = s(v_i,F,I,p_0)  \\
     & = v_i - \frac{1}{( F(v_i) )^{I-1}} \int^{v_i}_{p_0}(F(u))^{I-1}du \\
\end{align}

where $p_0$ is set equal to $\underline{v} = 0.055. $

```julia
d = truncated(LogNormal(0,1), 0.055, 2.5)
L = 200
I = 5
div = 20
F(x) = cdf(d,x)
f(x) = pdf(d,x)

function GPVbid(v;I=I, F=F, p₀ = 0.055)
    b = v - 1/((F(v))^(I-1)) * quadgk(u->F(u)^(I-1),p₀,v)[1]
    return b
end

function generateGPVbids(I,L)
    V = rand(d, I, L)
    B = GPVbid.(V)
end

B = generateGPVbids(I,L)
```

    5×200 Array{Float64,2}:
     0.143876  1.48393   0.971869  1.02013   …  0.320881  1.11943   0.221473
     0.809255  0.657708  0.806348  0.394685     0.643557  1.23468   0.50507
     0.400871  1.26797   0.480246  0.314797     0.384658  1.31628   0.430655
     0.598635  0.43845   1.46369   0.124238     1.08607   0.670787  0.343074
     0.382256  0.549899  0.265235  0.885578     0.85133   0.554553  1.28013

The Monte Carlo experiment in GPV uses the following setup for the estimation procedure:

In the first step, we estimate the CDF and PDF of bids. They estimate the CDF of the bids by

\begin{align}
    \tilde{G}(b) = \frac{1}{IL} \underset{l=1}{\overset{L}{\sum}} \underset{p=1}{\overset{I}{\sum}} 1 \{ B_{pl} \leq b \}
\end{align}

They also estimate the pdf using kernel regression

\begin{align}
    \tilde{g}(b) = \frac{1}{ILh_g} \underset{l=1}{\overset{L}{\sum}} \underset{p=1}{\overset{I}{\sum}} K_g(\frac{b-B_{pl}}{h_g})
\end{align}

where $h_g$ is some bandwidth and $K_g$ is a kernel function with comapct support. In particular, they used the triweight kernel

\begin{align}
       K_g(u) = \frac{35}{32}(1-u^2)^3 1(|u| \leq 1)
\end{align}

Bandwidth selection is $h_g = 1.06\hat{\sigma_b}(IL)^{-\frac{1}{5}}$ where $\hat{\sigma_b}$ is the estimated standard deviation of observed bids. Note that this kernel has support of length $\rho_g =2 $.

```julia
function Gtilde(b, B)
    G = mean(B .<= b)
    return G
end

function gtilde(b, B, hg; Kg = GPVKg)
    g = 1/hg*mean(Kg.((b.-B)./hg))
    return g
end

GPVKg(u) = 35/32 * (1-u^2)^3 * (abs(u)<=1)

ρg = 2
```

    2

We replicate the result for 5% percentile, mean and 95% percentile of the inverse bidding strategy. Note that a coarser division of the support is chosen here compared to the paper.

```julia
function onesimulation(v,I,L,genbid; Kg = Kg)
    B = genbid(I,L)
    σbhat = sqrt(var(B))
    hg = 1.06 * σbhat * (I*L)^(-1/5)
    G1 = b -> Gtilde(b, B)
    g1 = b-> gtilde(b, B, hg; Kg=Kg)
    ξ(b) = b .+ (1/(I-1) * (G1.(b) ./ g1.(b) )).-v

    b = nlsolve(ξ,zeros(div).+0.1, autodiff = :forward).zero
    return b, hg
end

function bidsmontecarlo(k, genbid;v=v,I=I,L=L, Kg=Kg)
    results = zeros(div)'
    for i = 1:k
        b,hg = onesimulation(v,I,L, genbid; Kg=Kg)
        results = vcat(results,b')
    end
    results = results[2:k+1,:]
    return results
end

## This is for plotting confidence intervals
function estimatequantiles(estimates,p)
    k = size(estimates)[2]
    quantiles = 0
    for i = 1:k
        q = quantile(estimates[:,i],p)
        quantiles = hcat(quantiles,q)
    end
    quantiles = quantiles[2:k+1]
end
```

    estimatequantiles (generic function with 1 method)

```julia
div = 10
v = range(0.055,2.5,length=div)[:,1]

@time results = bidsmontecarlo(1000, generateGPVbids; Kg=GPVKg)
pc5 = estimatequantiles(results,0.05)
pc95 = estimatequantiles(results,0.95)
m = mean(results,dims=1)[1,:]

plot()
plot(v,pc5, xlabel = "Private values", ylabel = "Bids", label = "5 percentile", legend = :topleft)
plot!(v,m, label = "Mean")
plot!(v,pc95, label = "95 percentile")
plot!(v,GPVbid.(v), label = "True bid")
```

     83.012304 seconds (365.38 M allocations: 61.727 GiB, 9.78% gc time)

The estimated CDF and PDFs are used to construct pseudo private values $\hat{V_{pl}}$ which requires some truncation in order to tackle the issue that $\tilde{g}$ is asymptotically biased for points close to the boundaries, where closeness is related to length of the kernel's support $\rho_g$ and bandwidth selection $h_g$.

$$
    \hat{V}_{pl} = \begin{cases}
    B_{pl} + \frac{1}{I-1}\frac{\tilde{G}(B_{pl})}{\tilde{g}(B_{pl})} \quad \text{if} \quad B_{min}+\frac{\rho_g h_g}{2} \leq B_{pl} \leq B_{max} - \frac{\rho_g h_g}{2},\\
    \infty \quad \text{otherwise}
    \end{cases}
$$

```julia
function Vhatestimate(b,B,hg; ρg=2, Kg=Kg)
    Bmin = minimum(B)
    Bmax = maximum(B)
    I = size(B)[1]
    L = size(B)[2]
    inc = (Bmin + ρg*hg/2 <= b) * (b <= Bmax - ρg*hg/2)
    if inc == 1
        V = b + 1/(I-1)*Gtilde(b,B)/gtilde(b,B,hg; Kg=Kg)
    else
        V = Inf
    end
    return V
end
```

    Vhatestimate (generic function with 1 method)

The pseudo-sample of private values is then used to estimate the density of private values using

\begin{align}
    \tilde{f}(v) = \frac{1}{ILh_f} \underset{l=1}{\overset{L}{\sum}} \underset{p=1}{\overset{I}{\sum}} K_f(\frac{v-\hat{V}_{pl}}{h_f})
\end{align}

where kernel and bandwidth selection is similar to before with $K_f=K_g$ and $h_f = 1.06\hat{\sigma_v}(IL)^{-\frac{1}{5}}$ where $\hat{\sigma_v}$ is the estimated standard deviation of trimmed pseudo-values.

```julia
GPVKf(u) = 35/32 * (1-u^2)^3 * (abs(u)<=1)

function fhatestimate(v, Vhat, hf; Kf=Kf)
    f = 1/hf * mean( Kf.( (v.-Vhat) ./hf) )
    return f
end
```

    fhatestimate (generic function with 1 method)

We now do 1000 simulations, and compare the mean, 5% percentile and 95% percentile for the estimates of the private values.

```julia
function GPVfsbandwidth(B,I,L)
    σbhat = sqrt(var(B))
    hg = 1.06 * σbhat * (I*L)^(-1/5)
    return hg
end

function GPVssbandwidth(Vhat,I,L)
    trimVhat = Vhat[Vhat.<Inf]
    σvhat = sqrt(var(trimVhat))
    ILt = length(trimVhat)
    hf = 1.06 * σvhat * (ILt)^(-1/5)
end

function densityestimation(v,B,fsbandwidth,ssbandwidth,Kg,Kf)
    I = size(B)[1]
    L = size(B)[2]
    σbhat = sqrt(var(B))
    hg = fsbandwidth(B,I,L)
    Vestimate = b-> Vhatestimate(b,B,hg;Kg=Kg)
    Vhat = Vestimate.(B)
    hf = ssbandwidth(Vhat,I,L)
    festimate = v-> fhatestimate(v,Vhat,hf; Kf=Kf)
    fhat = festimate.(v)
    return fhat, hf
end

function densitysimulation(v,I,L, genbid, fsbandwidth, ssbandwidth, Kg, Kf)
    B = genbid(I,L)
    fhat, hf = densityestimation(v,B,fsbandwidth,ssbandwidth,Kg,Kf)
    return fhat, hf
end

function densitymontecarlo(k,genbid,fsbandwidth,ssbandwidth,Kg,Kf; v=v, I=I, L=L)
    div = length(v)
    result = zeros(div)'
    bandwidths = 0

    for i = 1:k
        fhat, hf = densitysimulation(v,I,L, genbid, fsbandwidth, ssbandwidth, Kg, Kf)
        result = vcat(result,fhat')
        bandwidths = vcat(bandwidths, hf)
    end

    result = result[2:k+1,:]
    bandwidths = bandwidths[2:k+1]

    return result, bandwidths
end
```

    densitymontecarlo (generic function with 1 method)

```julia
div = 500
v = range(0.055,2.5,length=div)[:,1]

densityresult, bandwidths = densitymontecarlo(1000,generateGPVbids,GPVfsbandwidth, GPVssbandwidth, GPVKg,GPVKf)
pc5 = estimatequantiles(densityresult,0.05)
pc95 = estimatequantiles(densityresult,0.95)
m = mean(densityresult,dims=1)[1,:]
meanbandwidth = mean(bandwidths)

@show meanbandwidth

plot(v,pc5, title = "GPV density estimate",xlabel = "Private Value",
    ylabel = "Density/ Density Estimate", label = "5 percentile")
plot!(v,m, label = "mean")
plot!(v,pc95, label = "95 percentile")
plot!(v,f.(v), label = "True density")
```

    meanbandwidth = 0.14726034858119774

We have the same result as GPV where the mean almost coincides with the true density perfectly for points far enough from the boundaries 0.055 and 2.5.

### Results from Ma, Marmer and Shneyerov (2019)

In Ma, Marmer and Shneyerov, they assumed that bids follow a distribution characterized by the cdf

\begin{align}
    F_\theta(v) = v^\theta\cdot 1(0\leq v \leq1)
\end{align}

with pdf

\begin{align}
    f_\theta(v) = \theta v^{\theta-1} \cdot 1(0\leq v \leq1)
\end{align}

```julia
MMScdf(x,θ) = x^θ*(x≤1)*(x≥0)
MMSpdf(x,θ) = θ*x^(θ-1)*(x≤1)*(x≥0)
MMSquantile(x,θ) = exp(log(x)/θ)

function generateMMSvalues(I,L;θ=θ)
    U = rand(I,L)
    inversecdf = x-> MMSquantile(x,θ)
    sample = inversecdf.(U)
end
```

    generateMMSvalues (generic function with 1 method)

The corresponding BNE bidding strategy is

\begin{align}
    s(v) = (1-\frac{1}{\theta(I-1)+1})v
\end{align}

The special property of this specification is that the inverse bidding strategy has an analytic form

$$ \xi(b) = b + \frac{1}{I-1}\frac{G(b)}{g(b)} $$

We will define the estimated inverse bidding strategy as

$$ \hat{\xi}(b) = b + \frac{1}{I-1}\frac{\hat{G}(b)}{\hat{g}(b)} $$

```julia
function MMSbid(v,θ;I=I)
    b = (1-(θ*(I-1)+1)^(-1))*v
    return b
end

function generateMMSbids(I,L;θ=θ)
    V = generateMMSvalues(I,L;θ=θ)
    f = x->MMSbid(x,θ;I=I)
    return f.(V)
end

θ = 3
generateMMSbids(5,200)
```

    5×200 Array{Float64,2}:
     0.820108  0.581365  0.844222  0.854909  …  0.778212  0.921784   0.424836
     0.620776  0.881553  0.460267  0.711483     0.812613  0.753715   0.510624
     0.723872  0.305518  0.76788   0.849582     0.695944  0.773091   0.804168
     0.656387  0.651768  0.334781  0.60853      0.779482  0.836033   0.60314
     0.780711  0.433966  0.694147  0.86427      0.675543  0.0869814  0.692992

They considered θ∈{1，2} and drew samples from $f_\theta$. In the first step, the fourth-order triweight kernel is chosen and in the second step, the second order triweight kernel is chosen. In terms of bandwidth, the first stage bandwidth is

\begin{align}
 h_g = 3.72 \cdot \hat{\sigma_b} \cdot (IL)^{-1/5}
\end{align}

and
$$ h_f = 3.15 \cdot \hat{\sigma_v} \cdot ((I\cdot L)_\mathbb{T})^{-1/5}$$

where $(I\cdot L)_\mathbb{T}$ is the number of bids remaining after the trimming.

```julia
MMSKf(u) = 35/32 * (1-u^2)^3 * (abs(u)<=1)
MMSKg(u) = 27/16*(1-11/3*u^2)*MMSKf(u)
MMSdensity = v->MMSpdf(v,θ)

function MMSfsbandwidth(B,I,L)
    σbhat = sqrt(var(B))
    hg = 3.72 * σbhat * (I*L)^(-1/5)
    return hg
end

function MMSssbandwidth(Vhat,I,L)
    trimVhat = Vhat[Vhat.<Inf]
    σvhat = sqrt(var(trimVhat))
    ILt = length(trimVhat)
    hf = 3.15 * σvhat * (ILt)^(-1/5)
end
```

    MMSssbandwidth (generic function with 1 method)

```julia
div = 500
v = range(-0.5,1.5,length=div)[:,1]
θ = 1

densityresult, bandwidths = densitymontecarlo(1000,generateMMSbids,MMSfsbandwidth, MMSssbandwidth, MMSKg, MMSKf)
pc5 = estimatequantiles(densityresult,0.05)
pc95 = estimatequantiles(densityresult,0.95)
m = mean(densityresult,dims=1)[1,:]
meanbandwidth = mean(bandwidths)

@show meanbandwidth

plot(v,pc5, title = "MMS density estimate for θ=1",xlabel = "Private Value",
    ylabel = "Density/ Density Estimate", label = "5 percentile")
plot!(v,m, label = "mean")
plot!(v,pc95, label = "95 percentile")
plot!(v,MMSdensity.(v), label = "True density")
```

    meanbandwidth = 0.12264524339503591

```julia
div = 500
v = range(-0.5,1.5,length=div)[:,1]
θ = 2

densityresult, bandwidths = densitymontecarlo(1000,generateMMSbids,MMSfsbandwidth, MMSssbandwidth, MMSKg, MMSKf)
pc5 = estimatequantiles(densityresult,0.05)
pc95 = estimatequantiles(densityresult,0.95)
m = mean(densityresult,dims=1)[1,:]
meanbandwidth = mean(bandwidths)
MMSdensity = v->MMSpdf(v,θ)
@show meanbandwidth

plot(v,pc5, title = "MMS density estimate for θ=2",xlabel = "Private Value",
    ylabel = "Density/ Density Estimate", label = "5 percentile")
plot!(v,m, label = "mean")
plot!(v,pc95, label = "95 percentile")
plot!(v,MMSdensity.(v), label = "True density")
```

    meanbandwidth = 0.1306534014625496

Again, for points away from boundary, the mean coincides nearly perfectly with the true density. Now, we look at the distribution estimates at several points that are sufficiently far from the boundaries.

```julia
h1 = density(densityresult[:,250], label = "Density at v=0.5",
    ylabel = "Distribution of estimates", xlabel = "Density", title="Distribution plot")
vline!([MMSdensity.(v)[250]], label = "True density at v=0.5")

h2 = density!(densityresult[:,275], label = "Density at v=0.6")
vline!([MMSdensity.(v)[275]], label = "True density at v=0.6")

h3 = density!(densityresult[:,225], label = "Density at v=0.4")
vline!([MMSdensity.(v)[225]], label = "True density at v=0.4")

```

We compute the standard errors and uniform confidence bands using the bootstrap based method proposed in MMS. The process is:

First we draw $B^*$ an $I \times L$ bootstrap sample from original B. We construct $\hat{G}^*$ and $\hat{g}^*$ which are bootstrap analogues of $\hat{G}$ and $\hat{g}$. Let $\hat{Xi}^*$ be the bootstrap analogue of  $\hat{\xi}$  defined using $\hat{G}^*$ and $\hat{g}^*$.

We then generate bootstrap samples of pseudo values as $\hat{V}^*_{il} = \hat{\xi}^*(B^*_{il})$

Finally we construct a bootstrap analogue of $\hat{f}_{GPV}(v)$:

$$ \hat{f} _{GPV} ^*(v) = \frac{1}{IL} \sum{\mathbb{T}^*_{il}}\frac{1}{h_f}K_f(\frac{\hat{V}^*_{il}-v}{h_f}) $$

where $\mathbb{T}^*_{il}=1(Bmin + h_g \leq B^*_{il} \leq Bmax-h_g)$

The percentile bootstrap CI is

$$ CI^*(v) = [\hat{f} _{GPV}(v)+\frac{s^*_\frac{\alpha}{2}(v)}{\sqrt{(Lh^3)}} , \hat{f} _{GPV}(v)+\frac{s^*_{1-\frac{\alpha}{2}}(v)}{\sqrt{(Lh^3)}}] $$

where $s^*_\tau(v)$ is the $\tau$ quantile of the conditional distribution of

$$S^*(v) = (Lh^3)^\frac{1}{2}(\hat{f}^*_{GPV}(v)-\hat{f}_{GPV}(v))$$

We shall compute this quantity using Monte Carlo methods.

```julia
function Vtrim(b,Bmax,Bmin,hg)
    inc = (b<=Bmax-hg)*(b>=Bmin+hg)
    if inc == 1
        Vtrim = 1
    else
        Vtrim = 0
    end
end

function xistar(b, B, fsbandwidth; Kg = GPVKg)
    I = size(B)[1]
    Ghat = x->Gtilde(x,B)
    hg = fsbandwidth(B,I,L)
    ghat = x->gtilde(x,B,hg;Kg=Kg)
    xi = b +1/(I-1)*Ghat(b)/ghat(b)
end

function fstarestimate(v,B,fsbandwidth,ssbandwidth,Kg,Kf)
    I = size(B)[1]
    L = size(B)[2]
    Bstar = reshape(sample(B,I*L),(I,L))
    Bmax = maximum(Bstar)
    Bmin = minimum(Bstar)
    Ghatstar = x->Gtilde(x,Bstar)
    hg = fsbandwidth(Bstar,I,L)
    ghatstar = x->giltde(x,Bstar,hg;Kg = Kg)
    xihat = b->xistar(b,B,fsbandwidth)
    Vhatstar = xihat.(Bstar)
    trim = x-> Vtrim(x,Bmax,Bmin,hg)
    Bstartrim = trim.(Bstar)
    hf = ssbandwidth(Vhatstar,I,L)
    fhatstar = 1/(I*L)*(1/hf)*sum(Bstartrim.*Kf.((Vhatstar.-v)/hf))
    return fhatstar
end

fstarestimate(1.0,B,MMSfsbandwidth,MMSssbandwidth,MMSKg,MMSKf)
```

    0.4572159299123322

```julia
function confidenceinterval(v,B,fsbandwidth,ssbandwidth,Kg,Kf,α; k=1000)
    fhat,h = densityestimation(v,B,fsbandwidth,ssbandwidth,Kg,Kf)
    fstardraws = 0
    for i = 1:k
        fhatstar = fstarestimate(v,B,fsbandwidth,ssbandwidth,Kg,Kf)
        fstardraws = vcat(fstardraws,fhatstar)
    end
    fstardraws = fstardraws[2:k+1]
    L = size(B)[2]
    Sstar = (L*h^3)^(1/2).*(fstardraws.-fhat)
    a = quantile(Sstar,α/2)
    b = quantile(Sstar,1-α/2)
    LB = fhat + a/sqrt(L*h^3)
    RB = fhat + b/sqrt(L*h^3)
    return LB,RB
end
```

    confidenceinterval (generic function with 1 method)

```julia
θ = 2
B = generateMMSbids(5,200)
LB,RB = confidenceinterval(0.5,B,MMSfsbandwidth,MMSssbandwidth,MMSKg,MMSKf,0.05)
```

    (0.8638354494915652, 1.0823724760596236)

Now we check the coverage probability of the confidence interval.

```julia
function coverageprobability(v,I,L,fsbandwidth,ssbandwidth,Kg,Kf,α;θ=θ,M=500,k= 500)
    actual = MMSdensity(v)
    check = 0
    for i = 1:M
        B = generateMMSbids(I,L)
        LB,RB = confidenceinterval(v,B,fsbandwidth,ssbandwidth,Kg,Kf,α; k=k)
        inCI = ((actual<=RB)*(actual>=LB)).*1
        check = vcat(check,inCI)
    end
    check = check[2:M+1]
    mean(check)
end
```

    coverageprobability (generic function with 1 method)

Ideally, M and K should be large. However, on my laptop, this takes too long to run, so I am picking smaller numbers for k and M just to demonstrate its functionality.

```julia
θ = 2
@time coverageprobability(0.5,5,200,MMSfsbandwidth,MMSssbandwidth,MMSKg,MMSKf,0.05, M=20, k=100)
```

     29.430384 seconds (34.88 M allocations: 24.361 GiB, 8.60% gc time)

    0.95


# Mathematical equations

This page should have a ton of mathematical equations on it across cells etc.

## Inline equations

Here's an inline equation: $\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$. It
should be properly rendered within the block of text.

## Many equations in once cell

Here's a collection of equations:

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

$$\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$$

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

$$\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$$

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

$$\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$$

$$\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$$

They should be rendered properly

## Equations across multiple cells w/ mixed content

$$
\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}
$$

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

$$\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$$

$$\sum_{i=0}^n i^2 = \frac{(n^2+n)(2n+1)}{6}$$

## Latex-style equations

\begin{align}
\sqrt{37} & = \sqrt{\frac{73^2-1}{12^2}} \\
 & = \sqrt{\frac{73^2}{12^2}\cdot\frac{73^2-1}{73^2}} \\
 & = \sqrt{\frac{73^2}{12^2}}\sqrt{\frac{73^2-1}{73^2}} \\
 & = \frac{73}{12}\sqrt{1 - \frac{1}{73^2}} \\
 & \approx \frac{73}{12}\left(1 - \frac{1}{2\cdot73^2}\right)
\end{align}

\begin{align}
\sqrt{37} & = \sqrt{\frac{73^2-1}{12^2}} \\
 & = \sqrt{\frac{73^2}{12^2}\cdot\frac{73^2-1}{73^2}} \\
 & = \sqrt{\frac{73^2}{12^2}}\sqrt{\frac{73^2-1}{73^2}} \\
 & = \frac{73}{12}\sqrt{1 - \frac{1}{73^2}} \\
 & \approx \frac{73}{12}\left(1 - \frac{1}{2\cdot73^2}\right)
\end{align}

\begin{align}
\sqrt{37} & = \sqrt{\frac{73^2-1}{12^2}} \\
 & = \sqrt{\frac{73^2}{12^2}\cdot\frac{73^2-1}{73^2}} \\
 & = \sqrt{\frac{73^2}{12^2}}\sqrt{\frac{73^2-1}{73^2}} \\
 & = \frac{73}{12}\sqrt{1 - \frac{1}{73^2}} \\
 & \approx \frac{73}{12}\left(1 - \frac{1}{2\cdot73^2}\right)
\end{align}


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth.

# Relax and hold steady

Ready for more relaxing? This is the third lesson of **Module 5** of the course, exploring solutions to elliptic PDEs.
In [Lesson 1](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_01_2D.Laplace.Equation.ipynb) and [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_02_2D.Poisson.Equation.ipynb) of this module we used the Jacobi method (a relaxation scheme) to iteratively find solutions to Laplace and Poisson equations.

And it worked, so why are we still talking about it?  Because the Jacobi method is slow, very slow to converge.  It might not have seemed that way in the first two notebooks because we were using small grids, but we did need more than 3,000 iterations to reach the exit criterion while solving the Poisson equation on a $41\times 41$ grid.

You can confirm this below: using `nx,ny=` $128$ on the Laplace problem of Lesson 1, the Jacobi method requires nearly *20,000* iterations before we reach $10^{-8}$ for the L2-norm of the difference between two iterates. That's a *lot* of iterations!

Now, consider this application:  an incompressible Navier-Stokes solver has to ensure that the velocity field is divergence-free at every timestep.  One of the most common ways to ensure this is to solve a Poisson equation for the pressure field. In fact, the pressure Poisson equation is responsible for the majority of the computational expense of an incompressible Navier-Stokes solver. Imagine having to do 20,000 Jacobi iterations for *every* time step in a fluid-flow problem with many thousands or perhaps millions of grid points!

The Jacobi method is the slowest of all relaxation schemes, so let's learn how to improve on it. In this lesson, we'll study the Gauss-Seidel method—twice as fast as Jacobi, in theory—and the successive over-relaxation (SOR) method. We also have some neat Python tricks lined up for you to get to the solution even faster. Let's go!

### Test problem

Let's use the same example problem as in [Lesson 1](./05_01_2D.Laplace.Equation.ipynb): Laplace's equation with boundary conditions

\begin{equation}
  \begin{gathered}
p=0 \text{ at } x=0\\
\frac{\partial p}{\partial x} = 0 \text{ at } x = L\\
p = 0 \text{ at }y = 0 \\
p = \sin \left(  \frac{\frac{3}{2}\pi x}{L} \right) \text{ at } y = H
  \end{gathered}
\end{equation}

We import our favorite Python libraries, and also some custom functions that we wrote in [Lesson 1](./05_01_2D.Laplace.Equation.ipynb), which we have saved in a 'helper' Python file for re-use.

```python
import numpy
from matplotlib import pyplot, cm
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```python
from laplace_helper import p_analytical, plot_3D, L2_rel_error
```

We now have the analytical solution in the array `p_analytical`, and we have the functions `plot_3D` and `L2_rel_error` in our namespace.  If you can't remember how they work, just use `help()` and take advantage of the docstrings.  It's a good habit to always write docstrings in your functions, and now you see why!

In this notebook, we are going to use larger grids than before, to better illustrate the speed increases we achieve with different iterative methods. Let's create a $128\times128$ grid and initialize.

```python
nx = 128
ny = 128

L = 5
H = 5

x = numpy.linspace(0,L,nx)
y = numpy.linspace(0,H,ny)

dx = L/(nx-1)
dy = H/(ny-1)

p0 = numpy.zeros((ny, nx))

p0[-1,:] = numpy.sin(1.5*numpy.pi*x/x[-1])
```

We said above that the Jacobi method takes nearly 20,000 iterations before it satisfies our exit criterion of $10^{-8}$ (L2-norm difference between two consecutive iterations). You'll just have to confirm that now. Have a seat!

```python
def laplace2d(p, l2_target):
    '''Solves the Laplace equation using the Jacobi method
    with a 5-point stencil

    Parameters:
    ----------
    p: 2D array of float
        Initial potential distribution
    l2_target: float
        Stopping criterion

    Returns:
    -------
    p: 2D array of float
        Potential distribution after relaxation
    '''

    l2norm = 1
    pn = numpy.empty_like(p)
    iterations = 0

    while l2norm > l2_target:
        pn = p.copy()
        p[1:-1,1:-1] = .25 * (pn[1:-1,2:] + pn[1:-1,:-2] +\
                              pn[2:,1:-1] + pn[:-2,1:-1])

        ##Neumann B.C. along x = L
        p[1:-1,-1] = .25 * (2*pn[1:-1,-2] + pn[2:,-1] + pn[:-2, -1])

        l2norm = numpy.sqrt(numpy.sum((p - pn)**2)/numpy.sum(pn**2))
        iterations += 1

    return p, iterations
```

```python
l2_target = 1e-8
p, iterations = laplace2d(p0.copy(), l2_target)

print ("Jacobi method took {} iterations at tolerance {}".\
        format(iterations, l2_target))
```

    Jacobi method took 19993 iterations at tolerance 1e-08

Would we lie to you?  19,993 iterations before we reach the exit criterion of $10^{-8}$.  Yikes!

We can also time how long the Jacobi method takes using the `%%timeit` cell-magic.  Go make some tea, because this can take a while—the `%%timeit` magic runs the function a few times and then averages their runtimes to give a more accurate result.

- - -
##### Notes

1. When using `%%timeit`, the return values of a function (`p` and `iterations` in this case) *won't* be saved.

2. We document our timings below, but your timings can vary quite a lot, depending on your hardware. In fact, you may not even see the same trends (some recent hardware can play some fancy tricks with optimizations that you have no control over).
- - -

With those caveats, let's give it a shot:

```python
%%timeit
laplace2d(p0.copy(), l2_target)
```

    1 loops, best of 3: 5.92 s per loop

The printed result above (and others to come later) is from a mid-2007 Mac Pro, powered by two 3-GHz quad-core Intel Xeon X5364 (Clovertown). We tried also on more modern machines, and got conflicting results—like the Gauss-Seidel method being slightly slower than Jacobi, even though it required fewer iterations. Don't get too hung up on this: the hardware optimizations applied by more modern CPUs are varied and make a big difference sometimes.

Meanwhile, let's check the overall accuracy of the numerical calculation by comparing it to the analytical solution.

```python
pan = p_analytical(x,y)
```

```python
L2_rel_error(p,pan)
```

    6.1735513352884566e-05

That's a pretty small error.  Let's assume it is good enough and focus on speeding up the process.

## Gauss-Seidel

You will recall from [Lesson 1](./05_01_2D.Laplace_Equation.ipynb) that a single Jacobi iteration is written as:

\begin{equation}
p^{k+1}_{i,j} = \frac{1}{4} \left(p^{k}_{i,j-1} + p^k_{i,j+1} + p^{k}_{i-1,j} + p^k_{i+1,j} \right)
\end{equation}

The Gauss-Seidel method is a simple tweak to this idea: use updated values of the solution as soon as they are available, instead of waiting for the values in the whole grid to be updated.

If you imagine that we progress through the grid points in the order shown by the arrow in Figure 1, then you can see that the updated values $p^{k+1}_{i-1,j}$ and $p^{k+1}_{i,j-1}$ can be used to calculate $p^{k+1}_{i,j}$.

#### Figure 1. Assumed order of updates on a grid.

The iteration formula for Gauss-Seidel is thus:

\begin{equation}
p^{k+1}_{i,j} = \frac{1}{4} \left(p^{k+1}_{i,j-1} + p^k_{i,j+1} + p^{k+1}_{i-1,j} + p^k_{i+1,j} \right)
\end{equation}

There's now a problem for the Python implementation.  You can no longer use NumPy's array operations to evaluate the solution updates.  Since Gauss-Seidel requires using values immediately after they're updated, we have to abandon our beloved array operations and return to nested `for` loops. Ugh.

We don't like it, but if it saves us a bunch of time, then we can manage. But does it?

Here's a function to compute the Gauss-Seidel updates using a double loop.

```python
def laplace2d_gauss_seidel(p, nx, ny, l2_target):

    iterations = 0
    iter_diff = l2_target+1 #init iter_diff to be larger than l2_target

    while iter_diff > l2_target:
        pn = p.copy()
        iter_diff = 0.0
        for j in range(1,ny-1):
            for i in range(1,nx-1):
                p[j,i] = .25 * (p[j,i-1] + p[j,i+1] + p[j-1,i] + p[j+1,i])
                iter_diff += (p[j,i] - pn[j,i])**2

        #Neumann 2nd-order BC
        for j in range(1,ny-1):
            p[j,-1] = .25 * (2*p[j,-2] + p[j+1,-1] + p[j-1, -1])

        iter_diff = numpy.sqrt(iter_diff/numpy.sum(pn**2))
        iterations += 1

    return p, iterations
```

We would then run this with the following function call:

```Python
p, iterations = laplace2d_gauss_seidel(p,1e-8)
```

But **don't do it**.  We did it so that you don't have to!

The solution of our test problem with the Gauss-Seidel method required several thousand fewer iterations than the Jacobi method, but it took nearly *10 minutes* to run on our machine.

##### What happened?

If you think back to the far off days when you first learned about array operations, you might recall that we discovered that NumPy array operations could drastically improve code performance compared with nested `for` loops.  NumPy operations are written in C and pre-compiled, so they are *much* faster than vanilla Python.

But the Jacobi method is not algorithmically optimal, giving slow convergence.  We want to take advantage of the faster-converging iterative methods, yet unpacking the array operations into nested loops destroys performance. *What can we do?*

## Use Numba!

[Numba](http://numba.pydata.org) is an open-source optimizing compiler for Python. It works by reading Python functions that you give it, and generating a compiled version for you—also called Just-In-Time (JIT) compilation.  You can then use the function at performance levels that are close to what you can get with compiled languages (like C, C++ and fortran).

It can massively speed up performance, especially when dealing with loops.  Plus, it's pretty easy to use. Like we overheard at a conference: [*Numba is a Big Deal.*](http://twitter.com/lorenaabarba/status/625383941453656065)

##### Caveat

We encourage everyone following the course to use the [Anaconda Python](https://www.continuum.io/downloads) distribution because it's well put-together and simple to use.  If you *haven't* been using Anaconda, that's fine, but let us **strongly** suggest that you take the plunge now.  Numba is great and easy to use, but it is **not** easy to install without help.  Those of you using Anaconda can install it by running

`conda install numba`

If you *really* don't want to use Anaconda, you will have to [compile all of Numba's dependencies](https://pypi.python.org/pypi/numba).

- - -

### Intro to Numba

Let's dive in!  Numba is great and easy to use.  We're going to first walk you through a simple example to give you a taste of Numba's abilities.

After installing Numba (see above), we can use it by adding a line to `import numba` and another to `import autojit` (more on this in a bit).

```python
import numba
from numba import jit
```

You tell Numba which functions you want to accelerate by using a [Python decorator](http://www.learnpython.org/en/Decorators), a special type of command that tells the Python interpreter to modify a callable object (like a function). For example, let's write a quick function to calculate the $n^{\text{th}}$ number in the Fibonacci sequence:

```python
def fib_it(n):
    a = 1
    b = 1
    for i in range(n-2):
        a, b = b, a+b

    return b
```

There are several faster ways to program the Fibonacci sequence, but that's not a concern right now (but if you're curious, [check them out](http://mathworld.wolfram.com/BinetsFibonacciNumberFormula.html)).  Let's use `%%timeit` and see how long this simple function takes to find the 500,000-th Fibonacci number.

```python
%%timeit
fib_it(500000)
```

    1 loops, best of 3: 4.22 s per loop

Now let's try Numba!  Just add the `@jit` decorator above the function name and let's see what happens!

```python
@jit
def fib_it(n):
    a = 1
    b = 1
    for i in range(n-2):
        a, b = b, a+b

    return b
```

```python
%%timeit
fib_it(500000)
```

    The slowest run took 138.50 times longer than the fastest. This could mean that an intermediate result is being cached
    1000 loops, best of 3: 499 µs per loop

*Holy cow!* In our machine, that's more than 8,000 times faster!

That warning from `%%timeit` is due to the compilation overhead for Numba.  The very first time that it executes the function, it has to compile it, then it caches that code for reuse without extra compiling.  That's the 'Just-In-Time' bit.  You'll see it disappear if we run `%%timeit` again.

```python
%%timeit
fib_it(500000)
```

    1000 loops, best of 3: 499 µs per loop

We would agree if you think that this is a rather artificial example, but the speed-up is very impressive indeed. Just adding the one-word decorator!

##### Running in `nopython` mode

Numba is very clever, but it can't optimize everything.  When it can't, rather than failing to run, it will fall back to the regular Python, resulting in poor performance again.  This can be confusing and frustrating, since you might not know ahead of time which bits of code will speed up and which bits won't.

To avoid this particular annoyance, you can tell Numba to use `nopython` mode. In this case, your code will simply fail if the "jitted" function can't be optimized. It's simply an option to give you "fast or nothing."

Use `nopython` mode by adding the following line above the function that you want to JIT-compile:

```Python
@jit(nopython=True)
```

- - -

##### Numba version check

In these examples, we are using the latest (as of publication) version of Numba: 0.22.1.  Make sure to upgrade or some of the code examples below may not run.

- - -

```python
print(numba.__version__)
```

    0.22.1

## Back to Jacobi

We want to compare the performance of different iterative methods under the same conditions. Because the Gauss-Seidel method forces us to unpack the array operations into nested loops (which are very slow in Python), we use Numba to get the code to perform well. Thus, we need to write a new Jacobi method using for-loops and Numba (instead of NumPy), so we can make meaningful comparisons.

Let's write a "jitted" Jacobi with loops.

```python
@jit(nopython=True)
def laplace2d_jacobi(p, pn, l2_target):
    '''Solves the Laplace equation using the Jacobi method
    with a 5-point stencil

    Parameters:
    ----------
    p: 2D array of float
        Initial potential distribution
    pn: 2D array of float
        Allocated array for previous potential distribution
    l2_target: float
        Stopping criterion

    Returns:
    -------
    p: 2D array of float
        Potential distribution after relaxation
    '''

    iterations = 0
    iter_diff = l2_target+1 #init iter_diff to be larger than l2_target
    denominator = 0.0
    ny, nx = p.shape
    l2_diff = numpy.zeros(20000)

    while iter_diff > l2_target:
        for j in range(ny):
            for i in range(nx):
                pn[j,i] = p[j,i]

        iter_diff = 0.0
        denominator = 0.0

        for j in range(1,ny-1):
            for i in range(1,nx-1):
                p[j,i] = .25 * (pn[j,i-1] + pn[j,i+1] + pn[j-1,i] + pn[j+1,i])

        #Neumann 2nd-order BC
        for j in range(1,ny-1):
            p[j,-1] = .25 * (2*pn[j,-2] + pn[j+1,-1] + pn[j-1, -1])

        for j in range(ny):
            for i in range(nx):
                iter_diff += (p[j,i] - pn[j,i])**2
                denominator += (pn[j,i]*pn[j,i])

        iter_diff /= denominator
        iter_diff = iter_diff**0.5
        l2_diff[iterations] = iter_diff
        iterations += 1

    return p, iterations, l2_diff
```

```python
p, iterations, l2_diffJ = laplace2d_jacobi(p0.copy(), p0.copy(), 1e-8)

print("Numba Jacobi method took {} iterations at tolerance {}".format(iterations, l2_target))
```

    Numba Jacobi method took 19993 iterations at tolerance 1e-08

```python
%%timeit
laplace2d_jacobi(p0.copy(), p0.copy(), 1e-8)
```

    1 loops, best of 3: 2.41 s per loop

In our old machine, that's faster than the NumPy version of Jacobi, but on some newer machines it might not be. Don't obsess over this: there is much hardware black magic that we cannot control.

Remember that NumPy is a highly optimized library.  The fact that we can get competitive execution times with this JIT-compiled code is kind of amazing.  Plus(!) now we get to try out those techniques that aren't possible with NumPy array operations.

##### Note

We're also saving the history of the L2-norm of the difference between consecutive iterations.  We'll take a look at that once we have a few more methods to compare.
- - -

##### Another Note

Why did we use

```Python
l2_diff = numpy.zeros(20000)
```

Where did the `20000` come from?

We cheated a little bit.  Numba doesn't handle _mutable_ objects well in `nopython` mode, which means we can't use a *list* and append each iteration's value of the L2-norm.  So we need to define an array big enough to hold all of them and we know from the first run that Jacobi converges in fewer than 20,000 iterations.

- - -

##### Challenge task

It is possible to get a good estimate of the number of iterations needed by the Jacobi method to reduce the initial error by a factor $10^{-m}$, for given $m$. The formula depends on the largest eigenvalue of the coefficient matrix, which is known for the discrete Poisson problem on a square domain. See Parviz Moin, *"Fundamentals of Engineering Numerical Analysis"* (2nd ed., pp.141–143).

* Find the estimated number of iterations to reduce the initial error by $10^{-8}$ when using the grids listed below, in the section on grid convergence, with $11$, $21$, $41$ and $81$ grid points on each coordinate axis.

## Back to Gauss-Seidel

If you recall, the reason we got into this Numba sidetrack was to try out Gauss-Seidel and compare the performance with Jacobi.  Recall from above that the formula for Gauss-Seidel is as follows:

\begin{equation}
p^{k+1}_{i,j} = \frac{1}{4} \left(p^{k+1}_{i,j-1} + p^k_{i,j+1} + p^{k+1}_{i-1,j} + p^k_{i+1,j} \right)
\end{equation}

We only need to slightly tweak the Jacobi function to get one for Gauss-Seidel.  Instead of updating `p` in terms of `pn`, we just update `p` using `p`!

```python
@jit(nopython=True)
def laplace2d_gauss_seidel(p, pn, l2_target):
    '''Solves the Laplace equation using Gauss-Seidel method
    with a 5-point stencil

    Parameters:
    ----------
    p: 2D array of float
        Initial potential distribution
    pn: 2D array of float
        Allocated array for previous potential distribution
    l2_target: float
        Stopping criterion

    Returns:
    -------
    p: 2D array of float
        Potential distribution after relaxation
    '''

    iterations = 0
    iter_diff = l2_target + 1 #initialize iter_diff to be larger than l2_target
    denominator = 0.0
    ny, nx = p.shape
    l2_diff = numpy.zeros(20000)

    while iter_diff > l2_target:
        for j in range(ny):
            for i in range(nx):
                pn[j,i] = p[j,i]

        iter_diff = 0.0
        denominator = 0.0

        for j in range(1,ny-1):
            for i in range(1,nx-1):
                p[j,i] = .25 * (p[j,i-1] + p[j,i+1] + p[j-1,i] + p[j+1,i])

        #Neumann 2nd-order BC
        for j in range(1,ny-1):
            p[j,-1] = .25 * (2*p[j,-2] + p[j+1,-1] + p[j-1, -1])

        for j in range(ny):
            for i in range(nx):
                iter_diff += (p[j,i] - pn[j,i])**2
                denominator += (pn[j,i]*pn[j,i])

        iter_diff /= denominator
        iter_diff = iter_diff**0.5
        l2_diff[iterations] = iter_diff
        iterations += 1

    return p, iterations, l2_diff
```

```python
p, iterations, l2_diffGS = laplace2d_gauss_seidel(p0.copy(), p0.copy(), 1e-8)

print("Numba Gauss-Seidel method took {} iterations at tolerance {}".format(iterations, l2_target))
```

    Numba Gauss-Seidel method took 13939 iterations at tolerance 1e-08

Cool!  Using the most recently updated values of the solution in the Gauss-Seidel method saved 6,000 iterations!  Now we can see how much faster than Jacobi this is, because both methods are implemented the same way:

```python
%%timeit
laplace2d_gauss_seidel(p0.copy(), p0.copy(), 1e-8)
```

    1 loops, best of 3: 2.09 s per loop

We get some speed-up over the Numba version of Jacobi, but not a lot. And you may see quite different results—on some of the machines we tried, we could still not beat the NumPy version of Jacobi. This can be confusing, and hard to explain without getting into the nitty grity of hardware optimizations.

Don't lose hope!  We have another trick up our sleeve!

## Successive Over-Relaxation (SOR)

Successive over-relaxation is able to improve on the Gauss-Seidel method by using in the update a linear combination of the previous and the current solution, as follows:

\begin{equation}
p^{k+1}_{i,j} = (1 - \omega)p^k_{i,j} + \frac{\omega}{4} \left(p^{k+1}_{i,j-1} + p^k_{i,j+1} + p^{k+1}_{i-1,j} + p^k_{i+1,j} \right)
\end{equation}

The relaxation parameter $\omega$ will determine how much faster SOR will be than Gauss-Seidel. SOR iterations are only stable for $0 < \omega < 2$.  Note that for $\omega = 1$, SOR reduces to the Gauss-Seidel method.

If $\omega < 1$, that is technically an "under-relaxation" and it will be slower than Gauss-Seidel.

If $\omega > 1$, that's the over-relaxation and it should converge faster than Gauss-Seidel.

Let's write a function for SOR iterations of the Laplace equation, using Numba to get high performance.

```python
@jit(nopython=True)
def laplace2d_SOR(p, pn, l2_target, omega):
    '''Solves the Laplace equation using SOR with a 5-point stencil

    Parameters:
    ----------
    p: 2D array of float
        Initial potential distribution
    pn: 2D array of float
        Allocated array for previous potential distribution
    l2_target: float
        Stopping criterion
    omega: float
        Relaxation parameter

    Returns:
    -------
    p: 2D array of float
        Potential distribution after relaxation
    '''

    iterations = 0
    iter_diff = l2_target + 1 #initialize iter_diff to be larger than l2_target
    denominator = 0.0
    ny, nx = p.shape
    l2_diff = numpy.zeros(20000)

    while iter_diff > l2_target:
        for j in range(ny):
            for i in range(nx):
                pn[j,i] = p[j,i]

        iter_diff = 0.0
        denominator = 0.0

        for j in range(1,ny-1):
            for i in range(1,nx-1):
                p[j,i] = (1-omega)*p[j,i] + omega*.25 * (p[j,i-1] + p[j,i+1] + p[j-1,i] + p[j+1,i])

        #Neumann 2nd-order BC
        for j in range(1,ny-1):
            p[j,-1] = .25 * (2*p[j,-2] + p[j+1,-1] + p[j-1, -1])

        for j in range(ny):
            for i in range(nx):
                iter_diff += (p[j,i] - pn[j,i])**2
                denominator += (pn[j,i]*pn[j,i])

        iter_diff /= denominator
        iter_diff = iter_diff**0.5
        l2_diff[iterations] = iter_diff
        iterations += 1

    return p, iterations, l2_diff
```

That wasn't too bad at all.  Let's try this out first with $\omega = 1$ and check that it matches the Gauss-Seidel results from above.

```python
l2_target = 1e-8
omega = 1
p, iterations, l2_diffSOR = laplace2d_SOR(p0.copy(), p0.copy(), l2_target, omega)

print("Numba SOR method took {} iterations\
 at tolerance {} with omega = {}".format(iterations, l2_target, omega))
```

    Numba SOR method took 13939 iterations at tolerance 1e-08 with omega = 1

We have the exact same number of iterations as Gauss-Seidel.  That's a good sign that things are working as expected.

Now let's try to over-relax the solution and see what happens.  To start, let's try $\omega = 1.5$.

```python
l2_target = 1e-8
omega = 1.5
p, iterations, l2_diffSOR = laplace2d_SOR(p0.copy(), p0.copy(), l2_target, omega)

print("Numba SOR method took {} iterations\
 at tolerance {} with omega = {}".format(iterations, l2_target, omega))
```

    Numba SOR method took 7108 iterations at tolerance 1e-08 with omega = 1.5

Wow!  That really did the trick!  We dropped from 13939 iterations down to 7108.  Now we're really cooking!  Let's try `%%timeit` on SOR.

```python
%%timeit
laplace2d_SOR(p0.copy(), p0.copy(), l2_target, omega)
```

    1 loops, best of 3: 1.18 s per loop

Things continue to speed up. But we can do even better!

### Tuned SOR

Above, we picked $\omega=1.5$ arbitrarily, but we would like to over-relax the solution as much as possible without introducing instability, as that will result in the fewest number of iterations.

For square domains, it turns out that the ideal factor $\omega$ can be computed as a function of the number of nodes in one direction, e.g., `nx`.

\begin{equation}
\omega \approx \frac{2}{1+\frac{\pi}{nx}}
\end{equation}

This is not some arbitrary formula, but its derivation lies outside the scope of this course.  (If you're curious and have some serious math chops, you can check out Reference 3 for more information).  For now, let's try it out and see how it works.

```python
l2_target = 1e-8
omega = 2./(1 + numpy.pi/nx)
p, iterations, l2_diffSORopt = laplace2d_SOR(p0.copy(), p0.copy(), l2_target, omega)

print("Numba SOR method took {} iterations\
 at tolerance {} with omega = {:.4f}".format(iterations, l2_target, omega))
```

    Numba SOR method took 1110 iterations at tolerance 1e-08 with omega = 1.9521

Wow!  That's *very* fast.  Also, $\omega$ is very close to the upper limit of 2.  SOR tends to work fastest when $\omega$ approaches 2, but don't be tempted to push it.  Set $\omega = 2$ and the walls will come crumbling down.

Let's see what `%%timeit` has for us now.

```python
%%timeit
laplace2d_SOR(p0.copy(), p0.copy(), l2_target, omega)
```

    10 loops, best of 3: 184 ms per loop

Regardless of the hardware in which we tried this, the tuned SOR gave *big* speed-ups, compared to the Jacobi method (whether implemented with NumPy or Numba). Now you know why we told you at the end of [Lesson 1](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/05_relax/05_01_2D.Laplace.Equation.ipynb) that the Jacobi method is the *worst* iterative solver and almost never used.

Just to convince ourselves that everything is OK, let's check the error after the 1,110 iterations of tuned SOR:

```python
L2_rel_error(p,pan)
```

    7.7927433550683514e-05

Looking very good, indeed.

We didn't explain it in any detail, but notice the very interesting implication of Equation $(6)$: the ideal relaxation factor is a function of the grid size.
Also keep in mind that the formula only works for square domains with uniform grids. If your problem has an irregular geometry, you will need to find a good value of $\omega$ by numerical experiments.

## Decay of the difference between iterates

In the [Poisson Equation notebook](./05_02_2D.Poisson.Equation.ipynb), we noticed how the norm of the difference between consecutive iterations first dropped quite fast, then settled for a more moderate decay rate. With Gauss-Seidel, SOR and tuned SOR, we reduced the number of iterations required to reach the stopping criterion. Let's see how that reflects on the time history of the difference between consecutive solutions.

```python
pyplot.figure(figsize=(8,8))
pyplot.xlabel(r'iterations', fontsize=18)
pyplot.ylabel(r'$L_2$-norm', fontsize=18)
pyplot.semilogy(numpy.trim_zeros(l2_diffJ,'b'),
                'k-', lw=2, label='Jacobi')
pyplot.semilogy(numpy.trim_zeros(l2_diffGS,'b'),
                'k--', lw=2, label='Gauss-Seidel')
pyplot.semilogy(numpy.trim_zeros(l2_diffSOR,'b'),
                'g-', lw=2, label='SOR')
pyplot.semilogy(numpy.trim_zeros(l2_diffSORopt,'b'),
                'g--', lw=2, label='Optimized SOR')
pyplot.legend(fontsize=16);
```

The Jacobi method starts out with very fast convergence, but then it settles into a slower rate. Gauss-Seidel shows a faster rate in the first few thousand iterations, but it seems to be slowing down towards the end. SOR is a lot faster to converge, though, and optimized SOR just plunges down!

## References

1.  [Gonsalves, Richard J.  Computational Physics I. State University of New York, Buffalo: (2011): Section 3.1 ](http://www.physics.buffalo.edu/phy410-505/2011/index.html)

2. Moin, Parviz, "Fundamentals of Engineering Numerical Analysis," Cambridge University Press, 2nd edition (2010).

3. Young, David M. "A bound for the optimum relaxation factor for the successive overrelaxation method." Numerische Mathematik 16.5 (1971): 408-413.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>


```

%%HTML

```

# Neural Ordinary Differential Equations

## Summary

NeurIPS is the largest AI conference in the world. 4,854 papers were submitted. 4 received "Best paper" award. This is one of them. The basic idea is that neural networks are made up of stacked layers of simple computation nodes that work together to approximate a function. If we re-frame a neural network as an "Ordinary Differential Equation", we can use existing ODE solvers (like Euler's method) to approximate a function. This means no discrete layers, instead the network is a continous function. No more specifying the # of layers beforehand, instead specify the desired accuracy, it will learn how to train itself within that margin of error. It's still early stages, but this could be as big a breakthrough as GANs!

## Demo
An ODENet approximated this spiral function better than a Recurrent Network.

## Why Does this matter?

1. Faster testing time than recurrent networks, but slower training time. Perfect for low power edge computing! (precision vs speed)
2. More accurate results for time series predictions (!!) i.e continous-time models
3. Opens up a whole new realm of mathematics for optimizing neural networks (Diff Equation Solvers, 100+ years of theory)
4, Compute gradients with constant memory cost

## Concepts we'll learn about in this video
1. Basic neural network theory
2. "Residual" neural network theory
3. Ordinary Differential Equations (ODEs)
4. ODE Networks
5. Euler's Method to Optimize an ODENet
6. Adjoint Method for ODENet Optimization
7. ODENet's Applied to time series data
8. Future Applications of ODENets

## 1 Basic Neural Network Theory

-  Neural Networks are a popular type of ML model
- Neural Networks are built with linear algebra (matrices & matrix operations) & optimized using Calculus (gradient descent & other algorithms)
- Neural networks consist of a series of "layers", which are just matrix operations
- Each layer introduces a little bit of error that compounds through the network

#### Basic Neural Network Diagram

#### More Detailed Neural Network Diagram

## Basic Neural Network Example

```
import numpy as np

# compute sigmoid nonlinearity
def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

# convert output of sigmoid function to its derivative
def sigmoid_output_to_derivative(output):
    return output*(1-output)

# input dataset
X = np.array([  [0,1],
                [0,1],
                [1,0],
                [1,0] ])

# output dataset
y = np.array([[0,0,1,1]]).T

# seed random numbers to make calculation
# deterministic (just a good practice)
np.random.seed(1)

# initialize weights randomly with mean 0
synapse_0 = 2*np.random.random((2,1)) - 1

for iter in range(10000):

    # forward propagation
    layer_0 = X
    layer_1 = sigmoid(np.dot(layer_0,synapse_0))

    # how much did we miss?
    layer_1_error = layer_1 - y

    # multiply how much we missed by the
    # slope of the sigmoid at the values in l1
    layer_1_delta = layer_1_error * sigmoid_output_to_derivative(layer_1)
    synapse_0_derivative = np.dot(layer_0.T,layer_1_delta)

    # update weights
    synapse_0 -= synapse_0_derivative

print("Output After Training:")
print(layer_1)

```

    Output After Training:
    [[0.00505119]
     [0.00505119]
     [0.99494905]
     [0.99494905]]

## Stack More Layers?

- To reduce compounded error, add more layers!

- The # of layers & # of neurons deeply affec the output of the network
- Too few layers could cause underfitting
- Too many layers coudl cause overfitting + long training time

#### There are many rule-of-thumb methods for determining the correct number of neurons to use in the hidden layers, such as the following:

- The number of hidden neurons should be between the size of the input layer and the size of the output layer.
- The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
- The number of hidden neurons should be less than twice the size of the input layer

## Example: Long-Short Term Memory Neural Network with many layers

```

from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np

data_dim = 16
timesteps = 8
num_classes = 10

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32

model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

# Generate dummy training data
x_train = np.random.random((1000, timesteps, data_dim))
y_train = np.random.random((1000, num_classes))

# Generate dummy validation data
x_val = np.random.random((100, timesteps, data_dim))
y_val = np.random.random((100, num_classes))

model.fit(x_train, y_train,
          batch_size=64, epochs=5,
          validation_data=(x_val, y_val))
```

## 2 Residual Neural Network Theory

A solution to this was proposed by Microsoft for the 2015 ImageNet competiton (residual networks)
- In December of  2015, Microsoft proposed "Residual networks" as a solution to the ImageNet Classification Competition
- ResNets had the best accuracy in the competition
- ResNets utilize "skip-connections" between layers, which increases accuracy.
- They were able to train networks of up to 1000 layers deep while avoiding vanishing gradients (lower accuracy)
- 6 months later, their publicatio already had more than 200 references.

### How do ResNets work?

- Instead of hoping each stack of layers directly fits a desired underlying mapping, we explicitly let these layers fit a residual mapping.
- The original mapping is recast into F(x)+x.
- Residual neural networks do this by utilizing skip connections or short-cuts to jump over some layers.
- The residual layer adds the output of the activation function to the input of the layer.
- This seemingly minor change has led to a rethinking of how neural network layers are designed.
- In its limit as ResNets it will only skip over a single layer
- With an additional weight matrix to learn the skip weights it is referred to as HighwayNets
- With several parallel skips it is referred to as DenseNets

The residual layer is actually quite simple: add the output of the activation function to the original input to the layer. As a formula, the k+1th layer has the formula:

\begin{equation} x_{k+1} = x_{k} + F(x_{k})\end{equation}

where F is the function of the kth layer and its activation. For example, F might represent a convolutional layer with a relu activation. This simple formula is a special case of the formula:

\begin{equation} x_{k+1} = x_{k} + h F(x_k),\end{equation}

which is the formula for the Euler method for solving ordinary differential equations (ODEs) when h=1

### Wait, WTF is Euler's method? What does differential equations have to do with anything? Hold that thought, look at this code first.

```
#normal convolutional layer

def Unit(x,filters):

    out = BatchNormalization()(x)
    out = Activation("relu")(out)
    out = Conv2D(filters=filters, kernel_size=[3, 3], strides=[1, 1], padding="same")(out)

    out = BatchNormalization()(out)
    out = Activation("relu")(out)
    out = Conv2D(filters=filters, kernel_size=[3, 3], strides=[1, 1], padding="same")(out)

    return out
```

```
#residual convolutional layer

def Unit(x,filters):
    res = x
    out = BatchNormalization()(x)
    out = Activation("relu")(out)
    out = Conv2D(filters=filters, kernel_size=[3, 3], strides=[1, 1], padding="same")(out)

    out = BatchNormalization()(out)
    out = Activation("relu")(out)
    out = Conv2D(filters=filters, kernel_size=[3, 3], strides=[1, 1], padding="same")(out)

    out = keras.layers.add([res,out])

    return out
```

### Awesome, ok back to the question. What is the significance between residual networks and Ordinary Differential Equations?

## 3 Ordinary Differential Equations

- A "differential equation" is an equation that just tells us the slope without specifying the original function whose derivative we are taking

### Example Differential Equation

```
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# function dy/dx = x + y/5.
func = lambda y,x : x + y/5.
# Initial condition
y0 = -3  # at x=0
# values at which to compute the solution (needs to start at x=0)
x = np.linspace(0, 4, 101)
# solution
y = odeint(func, y0, x)
# plot the solution, note that y is a column vector
plt.plot(x, y[:,0])
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

## 4 ODE (Ordinary Differential Equation) Networks

Consider a simplified ODE from physics: we want to model the position x of a marble. Assume we can calculate its velocity x′ (the derivative of position) at any position x. We know that the marble starts at rest x(0)=0 and that its velocity at time t depends on its position through the formula:

\begin{equation} x^\prime(t) = f(x) \end{equation}

The Euler method solves this problem by following the physical intuition: my position at a time very close to the present depends on my current velocity and position. For example, if you are travelling at a velocity of 5 meters per second, and you travel 1 second, your position changes by 5 meters. If we travel h seconds, we will have travelled 5h meters. As a formula, we said:

\begin{equation}x(t+h) = x(t) + h x^\prime(t),\end{equation}

but since we know

\begin{equation} x^\prime(t) = f(x) \end{equation}

we can rewrite this as

\begin{equation} x(t+h) = x(t) + h f(x).\end{equation}

If you squint at this formula for the Euler method, you can see it looks just like the formula for residual layers!

This observation has meant three things for designing neural networks:

- New neural network layers can be created through different numerical approaches to solving ODEs
- The possibility of arbitrarily deep neural networks
- Training of a deep network can be improved by considering the so-called stability of the underlying ODE and its numerical discretization

### 2 more points

- To create arbitrarily deep networks with a finite memory footprint, design neural networks based on stable ODEs and numerical discretizations.
- Gradient descent can be viewed as applying Euler's method for solving ordinary differential equation to gradient flow.

## What Does an ODENet Look like?

*An ODE is a function that usually describes the change of some system through time. In this setting, time is a continuous variable. Now imagine a neural network is that system, and time is really something more like the depth of the network. Note that there are usually a discrete number of layers in an ANN. This is a notion of continuous number of layers.*

- The team didn't use Euler's method, they computed the exact ODE solution (within a small error tolerance) using adaptive solvers (faster)
- The dynamics change smoothly with depth. You can think of this either as having weights that are a function of depth, or as having shared weights across layers but adding the depth as an extra input to f.
- Anywhere you can put a resnet you can put an ODEnet.
- Each ODEBlock can be used to replace a whole stack of ResBlocks.
- In their MNIST example, each ODEBlock replaces 6 ResBlocks.

### Traditional Deep Nets

```
h1 = f1(x)
h2 = f2(h1)
h3 = f3(h2)
h4 = f3(h3)
y  = f5(h4)
```

### ResNets

```
h1 = f1(x)  + x
h2 = f2(h1) + h1
h3 = f3(h2) + h2
h4 = f4(h3) + h3
y  = f5(h4) + h4
```

- Where f1, f2, etc are neural net layers.
- The idea is that it's easier to model a small change to an almost-correct answer than to output the whole improved answer at once.
-This looks like a primitive ODE solver (Euler's method) that solves the trajectory of a system by just taking small steps in the direction of the system dynamics and adding them up.
-They connection allows for better training methods.
- What if we define a deep net as a continuously evolving system?
- Instead of updating the hidden units layer by layer, we define their derivative with respect to depth instead
- We can use off-the-shelf adaptive ODE solvers to compute the final state of these dynamics, and call that the output of the neural network.

## 5 Euler's Method

- We want to recover the blue curve, but all we have is an initial point A_0 (think inputs to the network) and a differential equation.
- From the differential equation, we can calculate the tangent line. If we take a small step along the tangent line, we arrive at A_1, which will be close to the desired blue line if the step is small enough.
- Repeat this process to uncover a polygonal curve A_{0}A_{1}A_{2}...A_{n}.

Many neural networks have a composition that looks exactly like the steps of Euler’s method. We start with an initial state \mathbf{z}_0, and apply successive transformations over time (layers):

…

### In the limit, we parameterize the continuous dynamics of hidden units using an ordinary differential equation (ODE) specified by a neural network:

![alt text](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle+%5Cfrac%7Bd%5Cmathbf%7Bz%7D%28t%29%7D%7Bdt%7D+%3D+f%28%5Cmathbf%7Bz%7D%28t%29%2C+t%2C+%5Ctheta%29&bg=ffffff&fg=333333&s=0
)

### The equivalent of having T layers in the network, is finding the solution to this ODE at time T.

- Euler’s method is perhaps the simplest method for solving ODEs.
- There since been more than 120 years of development of efficient and accurate ODE solvers.
- Modern ODE solvers provide guarantees about the growth of approximation error, monitor the level of error, and adapt their evaluation strategy on the fly to achieve the requested level of accuracy.
- This allows the cost of evaluating a model to scale with problem complexity.

### We’ve seen how to feed-forward, but how do you efficiently train a network defined as a differential equation? The answer lies in the adjoint method (which dates back to 1962). Think of the adjoint as the instantaneous analog of the chain rule.

## 6 The Adjoint Method

- This approach computes gradients by solving a second, augmented ODE backwards in time, and is applicable to all ODE solvers.
- This approach scales linearly with problem size, has low memory cost, and explicitly controls numerical error.
- The adjoint captures how the loss function L changes with respect to the hidden state.
- Starting from the output of the network, we can recompute the hidden state backwards in time together with the adjoint.

- A third integral then tells us how the loss changes with the parameters \theta ( dL/d\theta).
- All three of these integrals can be computed in a single call to an ODE solver, which concatenates the original state, the adjoint, and the other partial derivatives into a single vector.
- Algorithm 1 shows how to construct the necessary dynamics, and call an ODE solver to compute all gradients at once.

## ODE Net Example

```
## Import the Adjoint Method (ODE Solver)
from torchdiffeq import odeint_adjoint as odeint

## Normal Residual Block Example

class ResBlock(nn.Module):

    #init a block - Convolve, pool, activate, repeat
    def __init__(self, inplanes, planes, stride=1, downsample=None):
        super(ResBlock, self).__init__()
        self.norm1 = norm(inplanes)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.norm2 = norm(planes)
        self.conv2 = conv3x3(planes, planes)

    #Forward pass - pass output of one layer to the input of the next
    def forward(self, x):
        shortcut = x
        out = self.relu(self.norm1(x))
        out = self.conv1(out)
        out = self.norm2(out)
        out = self.relu(out)
        out = self.conv2(out)

        return out + shortcut

## Ordinary Differential Equation Definition

class ODEfunc(nn.Module):

    # init ODE variables
    def __init__(self, dim):
        super(ODEfunc, self).__init__()
        self.norm1 = norm(dim)
        self.relu = nn.ReLU(inplace=True)
        self.conv1 = conv3x3(dim, dim)
        self.norm2 = norm(dim)
        self.conv2 = conv3x3(dim, dim)
        self.norm3 = norm(dim)
        self.nfe = 0

    # init ODE operations
    def forward(self, t, x):
      #nfe = number of function evaluations per timestep
        self.nfe += 1
        out = self.norm1(x)
        out = self.relu(out)
        out = self.conv1(out)
        out = self.norm2(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.norm3(out)
        return out

 ## ODE block
class ODEBlock(nn.Module):

    #initialized as an ODE Function
    #count the time
    def __init__(self, odefunc):
        super(ODEBlock, self).__init__()
        self.odefunc = odefunc
        self.integration_time = torch.tensor([0, 1]).float()

    #foorward pass
    #input the ODE function and input data into the ODE Solver (adjoint method)
    # to compute a forward pass
    def forward(self, x):
        self.integration_time = self.integration_time.type_as(x)
        out = odeint(self.odefunc, x, self.integration_time, rtol=args.tol, atol=args.tol)
        return out[1]

    @property
    def nfe(self):
        return self.odefunc.nfe

    @nfe.setter
    def nfe(self, value):
        self.odefunc.nfe = value

## Main Method

if __name__ == '__main__':

    #Add Pooling
    downsampling_layers = [
         nn.Conv2d(1, 64, 3, 1),
         ResBlock(64, 64, stride=2, downsample=conv1x1(64, 64, 2)),
         ResBlock(64, 64, stride=2, downsample=conv1x1(64, 64, 2)),
     ]

    # Initialize the network as 1 ODE Block
    feature_layers = [ODEBlock(ODEfunc(64))]
    # Fully connected Layer at the end
    fc_layers = [norm(64), nn.ReLU(inplace=True), nn.AdaptiveAvgPool2d((1, 1)), Flatten(), nn.Linear(64, 10)]

    #The Model consists of an ODE Block, pooling, and a fully connected block at the end
    model = nn.Sequential(*downsampling_layers, *feature_layers, *fc_layers).to(device)

    #Declare Gradient Descent Optimizer
    optimizer = torch.optim.SGD(model.parameters(), lr=args.lr, momentum=0.9)

    #Training Loop
    for itr in range(args.nepochs * batches_per_epoch):

        #init the optimizer
        optimizer.zero_grad()

        #Generate training data
        x, y = data_gen()
        #Input Training data to model, get Prediction
        logits = model(x)
        #Compute Error using Prediction vs Actual Label
        loss = CrossEntropyLoss(logits, y)

        #Backpropagate
        loss.backward()
        optimizer.step()
```

## Closing Summary  of Neural Ordinary Differential Equations

- A neural network is a popular type of machine learning model
- Neural Networks are built with linear algebra and optimized using Calculus

- Neural networks consist of a series of "layers", which are just matrix operations
- Each layer introduces a little bit of error that compounds through the network
- The way to reduce that error is to add more and more layers
- The problem is that we see a drop off in performance after a certain # of layers
- A solution to this was proposed by Microsoft for the 2015 ImageNet competiton (residual networks)
- Residual Networks connect the output of previous layers to the output of new layers
- Prof. Duvenaud's team at University of Toronto noticed that that ResNets are similar to a primative "Ordinary Differential Equation" Solver called "Euler's Method"
- Ordinary Differential Equations involve one or more ordinary derivatives of unkown functions. 1 independent variable.
- Partial Differential Equations involve one or more partial derivatives of unknown functions. 2 or more independnet variables.
- Eulers method is a numerical method to sovle 1st order differential equations
- More efficient than Eulers method is the adjoint method. And this acts as our optimization strategy
- The result? No need to specificy # of layers beforehand, now just specifiy accuracy. it will train itself.
- No more discrete layers, instead a continous computational block

Applications
-Irregular time series data (medical history recorded at random times) , discrete layers are bad at this
-Memory Efficiency (constant memory), slower training time, faster test time,