```python
%matplotlib inline
import numpy as np
import sympy as sym
from sympy import init_printing
init_printing(use_latex=True)
from sympy.utilities.lambdify import lambdify
import matplotlib.pyplot as plt
```

Burgers' equation in one dimension looks like:

$$
\dfrac{\partial u}{\partial t} + u \dfrac{\partial u}{\partial x} = \nu \dfrac{\partial^2 u}{\partial x^2}
$$

Rearranging the discretizations used before, the equation looks like:

$$
\dfrac{u^{n+1}_{i}-u^{n}_{i}}{\Delta t}+u\dfrac{u^{n}_{i}-u^{n}_{i-1}}{\Delta x} =\nu \dfrac{u^{n}_{i+1}-2u^{n}_{i}+u^{n}_{i-1}}{\Delta x^2}
$$

The only unknown in this problem is $u_i^{n+1}$. Solving for it:

$$
u^{n+1}_{i}=u^n_i-u^n_i\dfrac{\Delta t}{\Delta x} \left( u^{n}_{i}-u^{n}_{i-1} \right) + \dfrac{\nu  \Delta t}{\Delta x^2} \left( {u^{n}_{i+1}-2u^{n}_{i}+u^{n}_{i-1}} \right)
$$

The initial condition for this case will be:

$$
u = - \dfrac{2\nu}{\phi}\dfrac{\partial \phi}{\partial x}+4 \quad \text{ where } \phi = \exp \left( -\dfrac{x^2}{4\nu} \right) + \exp \left( -\dfrac{(x-2\pi)^2}{4\nu} \right)
$$

The boundary condition is given by the periodic relation:

$$
u(0) = u(2\pi)
$$

The initial condition given has an analytical solution, given by:

$$
u = - \dfrac{2\nu}{\phi}\dfrac{\partial \phi}{\partial x}+4 \quad \text{ where } \phi = \exp \left( -\dfrac{(x-4t)^2}{4\nu(t+1)} \right) + \exp \left( -\dfrac{(x-4t-2\pi)^2}{4\nu(t+1)} \right)
$$

```python
x, nu, t = sym.symbols('x nu t')
phi = (sym.exp(-(x-4*t)**2/(4*nu*(t+1)))+sym.exp(-(x-4*t-2*np.pi)**2/(4*nu*(t+1))))
phi
```

```python
phiprime = phi.diff(x)
phiprime
```

```python
u = -2 * nu * (phiprime/phi) + 4
ufunc = lambdify((t,x,nu),u)
print(ufunc(1,4,3))
```

    3.49170664206

```python
nx = 101
nt = 100
dx = 2*np.pi/(nx-1)
nu = 0.07
dt = dx*nu

x = np.linspace(0,2*np.pi,nx)
un = np.zeros(nx)
t = 0

u = np.asarray([ufunc(t,x0,nu) for x0 in x])

plt.figure(figsize=(11,7), dpi=100)
plt.plot(x, u, 'bo-', lw=2)
plt.xlim([0,2*np.pi])
plt.ylim([0,10])
plt.show()
```

```python
for n in range(nt):
    un = u.copy()
    for i in range(1,nx-1):
        u[i] = un[i] - un[i] *dt/dx*(un[i]-un[i-1]) + nu*dt/dx**2*(un[i+1]-2*un[i]+un[i-1])
    u[0] = un[0] - un[0] *dt/dx*(un[0]-un[-2]) + nu*dt/dx**2*(un[1]-2*un[0]+un[-2])
    u[-1] = u[0]

u_analytical = np.asarray([ufunc(nt*dt,xi,nu) for xi in x])

plt.figure(figsize=(11,7), dpi=100)
plt.plot(x,u, 'bo-', lw=2, label='Numerical')
plt.plot(x,u_analytical, 'g', label='Analytical')
plt.xlim([0,2*np.pi])
plt.ylim([0,10])
plt.legend()
```


# The Lyapunov spectrum of the Lorenz system

Here, we present a calculation of the Lyapunov spectrum of the Lorenz system, using `TaylorIntegration.jl`. Our calculation involves evaluating the 1st order variational equations $\dot \xi = J \cdot \xi$ for this system, where $J = \operatorname{D}f$ is the Jacobian. By default, the numerical value of the Jacobian is computed using automatic differentiation techniques implemented in `TaylorSeries.jl`. Automatic differentation helps us to avoid writing down manually the Jacobian; instead, `TaylorIntegration.jl` does that for us! For complex problems, or when the number of dependent variables is large, this may help save typos, which sometimes are hard to trace. Otherwise, if performance is critical, then the user may provide the Jacobian function, as will be shown below.

The Lorenz system is the ODE defined as:

$$
\begin{align}
    \begin{split}
    \dot x_1 &= \sigma(x_2-x_1) \\
    \dot x_2 &= x_1(\rho-x_3)-x_2 \\
    \dot x_3 &= x_1x_2-\beta x_3
    \end{split}
\end{align}
$$

where $\sigma$, $\rho$ and $\beta$ are constant parameters.

## The setup

First, we write a Julia function which evaluates (in-place) the Lorenz system equations:

```julia
#Lorenz system ODE:
function lorenz!(t, x, dx)
    dx[1] = σ*(x[2]-x[1])
    dx[2] = x[1]*(ρ-x[3])-x[2]
    dx[3] = x[1]*x[2]-β*x[3]
    nothing
end
```

    lorenz! (generic function with 1 method)

Next, we define the parameters:

```julia
#Lorenz system parameters
#we use the `const` prefix in order to help the compiler speed things up
const σ = 16.0
const β = 4.0
const ρ = 45.92
```

    45.92

The initial conditions, the initial and final time are:

```julia
const x0 = [19.0, 20.0, 50.0] #the initial condition
const t0 = 0.0 #the initial time
const tmax = 100.0 #final time of integration
```

    100.0

We know that the sum of the Lyapunov spectrum has to be equal to the trace of the Jacobian of the equations of motion. We will calculate this trace using `TaylorSeries.jl`, and after the numerical integration, we will come back to this value to check that this is indeed the case:

```julia
# Note that TaylorSeries.jl exported variables and methods are @reexport-ed by TaylorIntegration.jl
#Calculate trace of Lorenz system Jacobian via TaylorSeries.jacobian:
import LinearAlgebra: tr
using TaylorIntegration
xi = set_variables("δ", order=1, numvars=length(x0))
x0TN = [ x0[1]+xi[1], x0[2]+xi[2], x0[3]+xi[3] ]
dx0TN = similar(x0TN)
lorenz!(t0, x0TN, dx0TN)
lorenztr = tr(TaylorSeries.jacobian(dx0TN)) #trace of Lorenz system Jacobian matrix
```

    -21.0

# The integration

Now, we are ready to perform the integration, using `TaylorIntegration.jl`. Usually, we would use the `taylorinteg` method in order to integrate the equations of motion; but, since we are interested in evaluating the Lyapunov spectrum, we will use the `lyap_taylorinteg` method, which calculates the Lyapunov spectrum via the variational equations and Oseledets' theorem. The expansion order will be $28$; the local absolute tolerance will be $10^{-20}$. `lyap_taylorinteg` will return three arrays: one with the evaluation times, one with the value of the dependent variables at the time of evaluation, and another one with the vaues of the Lyapunov spectrum at each time of the evaluation.

```julia
@time tv, xv, λv = lyap_taylorinteg(lorenz!, x0, t0, tmax, 28, 1e-20; maxsteps=2000000);
```

      3.038651 seconds (15.28 M allocations: 1.371 GiB, 13.00% gc time)

As explained above, instead of using automatic differentiation to compute the numerical value of the Jacobian, the user may provide a function which computes the Jacobian in-place, as follows:

```julia
#Lorenz system Jacobian (in-place):
function lorenz_jac!(jac, t, x)
    jac[1,1] = -σ+zero(x[1]); jac[1,2] = σ+zero(x[1]); jac[1,3] = zero(x[1])
    jac[2,1] = ρ-x[3]; jac[2,2] = -1.0+zero(x[1]); jac[2,3] = -x[1]
    jac[3,1] = x[2]; jac[3,2] = x[1]; jac[3,3] = -β+zero(x[1])
    nothing
end
```

    lorenz_jac! (generic function with 1 method)

Then, we may perform the same integration as before using `lorenz_jac!`:

```julia
@time tv_, xv_, λv_ = lyap_taylorinteg(lorenz!, x0, t0, tmax, 28, 1e-20, lorenz_jac!; maxsteps=2000000);
```

      0.856854 seconds (4.35 M allocations: 575.767 MiB, 20.75% gc time)

In general, we may check the consistency of our integration checking that both solutions are either equal, or at least approximately equal with differences comparable to roundoff errors. In the particular case of the integrations performed above, we obtain the same solution, *exactly*:

```julia
tv == tv_, xv == xv_, λv == λv_
```

    (true, true, true)

The number of steps taken is:

```julia
length(tv)
```

    4093

Is the final time actually the requested value?

```julia
tv[end] == tmax
```

    true

What is the minimum, maximum and mean time-step?

```julia
import Statistics: mean
minimum(diff(tv)), maximum(diff(tv)), mean(diff(tv))
```

    (0.009471209659452029, 0.0608034582365633, 0.024437927663734114)

What is the standard deviation of the time-step size distribution?

```julia
import Statistics: std
std(diff(tv))
```

    0.007428238447533755

What is the final value for each of the exponents?

```julia
λv[end,:]
```

    3-element Array{Float64,1}:
       1.4856676067102397
      -0.0012149610701265275
     -22.484452645640157

What is the value of the sum of the value we obtained for the spectrum?

```julia
sum(λv[end,:])
```

    -21.000000000000043

Now, is the sum of the exponents exactly equal to the trace of the Jacobian?

```julia
sum(λv[end,:]) == lorenztr
```

    false

So the sum is not *exactly* equal, but is it approximately equal? What is the relative error in the computed vs the expected value?

```julia
rel_error = (sum(λv[end,:])-lorenztr)/lorenztr
```

    2.0301221021717147e-15

How does this error compare to the machine epsilon?

```julia
rel_error/eps()
```

    9.142857142857142

So the relative error in our computation for the sum of the Lyapunov spectrum is comparable to the machine epsilon.

# Visualization

Finally, we plot our results:

```julia
using Plots
#gr()
#using LaTeXStrings
#plotlyjs()
```

### Lyapunov exponents vs time plot

```julia
plot(tv, λv[:,1], label="L_1")
plot!(tv, λv[:,2], label="L_2")
plot!(tv, λv[:,3], label="L_3")
xlabel!("time")
ylabel!("L_i, i=1,2,3")
title!("Lyapunov exponents vs time")
```

### Lyapunov exponents vs time plot (semi-log)

```julia
plot(tv[2:end], λv[:,1][2:end], xscale=:log10, label="L_1")
plot!(tv[2:end], λv[:,2][2:end], label="L_2")
plot!(tv[2:end], λv[:,3][2:end], label="L_3")
xlabel!("time")
ylabel!("L_i, i=1,2,3")
title!("Lyapunov exponents vs time (semi-log)")
```

### Convergence of Lyapunov exponents vs time plot (log-log)

```julia
plot(tv[2:end], abs.(diff(λv[:,1])), yscale=:log10, xscale=:log10, label="dL_1")
plot!(tv[2:end], abs.(diff(λv[:,2])), label="dL_2")
plot!(tv[2:end], abs.(diff(λv[:,3])), label="dL_3")
xlabel!("time")
ylabel!("dL_i, i=1,2,3")
title!("Convergence of Lyapunov exponents vs time (log-log)")
```

### Absolute difference wrt trace of Jacobian vs time plot (log-log)

```julia
plot(tv[2:end], log10.(abs.(λv[2:end,1]+λv[2:end,2]+λv[2:end,3].-lorenztr)), xscale=:log10, leg=false)
#ylims!(-15,-12)
xlabel!("time (log10)")
ylabel!("Absolute difference wrt Tr(J) (log10)")
```

## The Lorenz attractor

```julia
plot(xv[:,1], xv[:,2], xv[:,3], leg=false)
```

```julia

```


#### MIT License (c) 2018 by Andrew Lyasoff

#### Jupyter notebook written in Python 3. It illustrates the use of SymPy to compute the distribution function of the Gaussian law and its inverse, which is then used to transform a uniform Monte Carlo sample into a Gaussian Monte Carlo sample.

First, compute the distribution function and its inverse in symbolic form:

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import *
from sympy import *
x,y,n,u=symbols('x y n u')
init_printing()
```

```python
integrate((1/sqrt(2*pi))*exp(-u**2/2),(u,-oo,+oo))
```

```python
AA=integrate(sqrt(1/(2*pi))*exp(-u**2/2),(u,-oo,x))
AA
```

```python
(1+erf(sqrt(2)*x/2))/2-y
```

```python
BB=solve((1+erf(sqrt(2)*x/2))/2-y,x)[0]
BB
```

```python
sqrt(2)
```

```python
sqrt(2.)
```

```python
def F(xxx):
    return erf(sqrt(2)*xxx/2)/2 + 1/2

def Finv(yyy):
    return sqrt(2)*erfinv(2*yyy - 1)

```

```python
F(Finv(0.8))
```

```python
Finv(F(5))
```

```python
Finv(F(5.))
```

Now plot the distribution function for the standard normal ${\cal N}(0,1)$ distribution law.

```python
t = arange(-3, 3, 0.01)
s=[F(u) for u in t]
fig, ax = plt.subplots()
ax.plot(t, s)
plt.show()
```

```python
t = arange(.001, 1, 0.001)
s=[Finv(u) for u in t]
fig, ax = plt.subplots()
ax.plot(t, s)
plt.show()
```

Plot the inverse:

Compute the "exact" probability for falling outside the interval $[-3\sigma,3\sigma]$.

```python
2*(1-F(3.0))
```

```python
1-(_)
```

Now try to compute the above probability by using Monte Carlo simulation and sampling from the uniform distribution in $[0,1]$.

```python
2*(1-F(3.0))-0.002641
```

```python
Finv(F(8))
```

```python
1+1
```

```python

```


# Neural networks from scratch with NumPy

Neural networks are very popular function approximators used in a wide variety of fields nowadays and coming in all kinds of flavors, so there are countless frameworks that allow us to train and use them without knowing what is going on behind the scenes. So I set out to *reinvent the wheel* and decided to write a post deriving the math for backpropagation from the maximum likelihood principle, as well as implementing it with Python and NumPy. While doing this, I wanted to

- be as careful as possible with the notation, which can get messy quite fast when moving between layers, weight parameters and their gradients,
- think in terms of matrix multiplications that could easily be translated to vectorized code.

By writing down a derivation in my own words, I could reinforce and further understand what I had learned in the past about neural networks; by implementing it from scratch I was able to realize important and interesting details that are often hidden by deep learning frameworks, such as the log-sum-exp trick. It took longer than I expected, but it was definitely worth the time, and I recommed it as a way to gain a deeper understanding of neural networks.

## The theory

Consider a classification problem with $K$ classes and input vector $\mathbf{x}$. Neural networks can be used as discriminative models for the probability $p(\mathcal{C}_i\mid \mathbf{x}, \boldsymbol{\theta})$ of a target class $\mathcal{C}_i$ given the input vector $\mathbf{x}$, where $\boldsymbol{\theta}$ denotes all the parameters of the network. In a neural network with $L$ layers, this probability is modeled in the last layer using the softmax function:

$$
p(\mathcal{C}_i\mid \mathbf{x}, \mathbf{\theta}) = a_{iL} = f_L(z_{iL}) = \frac{\exp(z_{iL})}{\sum_{j=1}^K\exp(z_{jL})}
$$

Here $f_L(\cdot)$ corresponds to the so-called *activation function* of the last layer, which in this case is the softmax[1](#1).

We can put all the probabilities $a_{iL}$ (with $i = 1, ..., K$) in a column vector, so that the complete output of the network is a vector $\mathbf{a}_L$ with $K$ elements, containing probabilities for each class. In general, we will call $\mathbf{a}_l$ the activation output of the $l$-th layer.

Note that there is one value of $z_{iL}$ for $i = 1, ..., K$. Each one corresponds to a "unit" in the output layer, and is the result of a linear combination of the activations of the previous layer $\mathbf{a}_{L-1}$ via a set of weights $\mathbf{w}_{iL}$ and a bias (constant term) $b_{iL}$:

$$
z_{iL} = \mathbf{w}_{iL}^T\mathbf{a}_{L-1} + b_{iL}
$$

The weights $\mathbf{w}_{iL}$ are column vectors with a number of elements equal to the number of output units in the previous layer. If we put these in the rows of a matrix $\mathbf{W}_L$, and all the biases in a column vector $\mathbf{b}_L$, we can write all the $z_{iL}$ in a single column vector $\mathbf{z}_L$ given by the following matrix operation:

$$
\mathbf{z}_L = \mathbf{W}_L\mathbf{a}_{L-1} + \mathbf{b}_L
$$

These equations show how the neural network builds a function composition, from inputs to outputs, in order to model the class probabilities. We can think of the input vector $\mathbf{x}$ as the activation of layer zero, $\mathbf{a}_{0}$, which allows us to write down the equations to perform a forward pass on the neural network:

$$
\begin{align}
\mathbf{z}_1 &= \mathbf{W}_1\mathbf{x} + \mathbf{b}_1\\
\mathbf{a}_1 &= f_1(\mathbf{z}_1)\\
\mathbf{z}_2 &= \mathbf{W}_2\mathbf{a}_1 + \mathbf{b}_2\\
\mathbf{a}_2 &= f_2(\mathbf{z}_2)\\
&\vdots\\
\mathbf{z}_L &= \mathbf{W}_L\mathbf{a}_{L-1} + \mathbf{b}_L\\
\mathbf{a}_L &= f_L(\mathbf{z}_L)
\end{align}
$$

When we say that we "train" the neural network, we aim to find the right weights and biases (which we have collectively called $\boldsymbol{\theta}$) in order to minimize a certain loss function. If we have a dataset $\mathbf{D}$ of input vectors and their corresponding class labels, we can compute the likelihood $p(\mathbf{D}\mid\boldsymbol{\theta})$ of observing the dataset as a function of a particular value of the parameters, which is known as the *likelihood function*, and obtain the set of parameters that maximize it. By doing so, we hope to obtain parameters that not only fit well to the observed data, but that also generalize to unseen data.

For mathematical convenience (such as converting products into sums that simplify differentiation), it is often preferred to work with the logarithm of the likelihood. If we define the task in terms of minimization, we end up with the goal of finding the parameters that minimize the negative log-likelihood[2](#2):

$$
\arg\min_{\boldsymbol{\theta}}\lbrace-\log p(\mathbf{D}\mid\boldsymbol{\theta})\rbrace
$$

We will solve this optimization problem using gradient descent, by iteratively adjusting the parameters in the direction of decreasing loss. We obtain this direction by calculating the gradient with respect to the parameters.

### Deriving the gradients

To specify the label for an input vector we will use a vector with a one hot encoding, so that the vector has $K$ elements, all zero, except for the position of the correct label, which is one. Let's say we observe a target vector $\mathbf{t}$ for class $m$ (so that the element at position $k$ is one and the rest are zero) for an input vector $\mathbf{x}$, then the negative log-likelihood with this single observation is

$$
\begin{align}
\mathcal{L}(\boldsymbol{\theta}) = -\log p(\mathbf{t}|\mathbf{x}, \boldsymbol{\theta}) &= -\log\prod_{i=1}^K p(\mathcal{C}_i\mid \mathbf{x}, \boldsymbol{\theta})^{t_i}\\
&= -\sum_{i=1}^K t_i\log p(\mathcal{C}_i\mid \mathbf{x}, \boldsymbol{\theta})\\
&= -\sum_{i=1}^K t_i\log\left(
\frac{\exp(z_{iL})}{\sum_{j=1}^K\exp(z_{jL})}
\right)\\
&= -\sum_{i=1}^K t_i\left(
z_{iL} - \log\sum_{j=1}^K\exp(z_{jL}))
\right)\\
&= -z_{kL} + \log\sum_{j=1}^K\exp(z_{jL})\qquad\star\\
&= -z_{kL} + \log Z
\end{align}
$$

In the starred step we have made use of the fact that the target vector $\mathbf{t}$ is zero everywhere except at position $k$, so we end up with only the term of the sum corresponding to $i=k$. We also have defined

$$
Z = \sum_{j=1}^K\exp(z_{jL})
$$

We now have to calculate the gradients of the loss with respect to every parameter of the neural network. This process consists of a careful application of the chain rule, as we will now see.

The loss $\mathcal{L}$ is a function of the quantity $z_{iL}$, which is a function of the weights $\mathbf{w}_{iL}$ of unit $i$ in the last layer. Therefore, to obtain the gradient of the loss with respect to the weights $\mathbf{w}_{iL}$, we must use the chain rule and calculate

$$
\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{iL}} = \frac{\partial \mathcal{L}}{\partial z_{iL}}\frac{\partial z_{iL}}{\partial\mathbf{w}_{iL}}
$$

For the first derivative in this expression we have

$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial z_{iL}} &= \frac{\partial}{\partial z_{iL}}[-z_{kL} + \log Z]\\
&= \frac{\partial}{\partial z_{iL}}[-z_{kL}] + \frac{\partial}{\partial z_{iL}}[\log Z]\\
&= \frac{\partial}{\partial z_{iL}}[-z_{kL}] + \frac{1}{Z}\frac{\partial}{\partial z_{iL}}[Z]\\
&= \frac{\partial}{\partial z_{iL}}[-z_{kL}] + \frac{1}{Z}\frac{\partial}{\partial z_{iL}}
\left[
\sum_{j=1}^K\exp(z_{jL})
\right]\\
&= \frac{\partial}{\partial z_{iL}}[-z_{kL}] + \frac{1}{Z}\frac{\partial}{\partial z_{iL}}[\exp(z_{iL})]\\
&= \frac{\partial}{\partial z_{iL}}[-z_{kL}] + \frac{1}{Z}\exp(z_{iL})\\
&= \left\{
\begin{matrix}
-1 + \frac{1}{Z}\exp(z_{iL}) \text{ if } i = k\\
\frac{1}{Z}\exp(z_{iL}) \text{ if } i \neq k
\end{matrix}
\right.
\end{align}
$$

The second derivative is computed as follows:

$$
\frac{\partial z_{iL}}{\partial\mathbf{w}_{iL}} = \frac{\partial}{\partial \mathbf{w}_{iL}}[\mathbf{w}_{iL}^T\mathbf{a}_{L-1} + b_{iL}] = \mathbf{a}_{L-1}
$$

We therefore have

$$
\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{iL}} = \frac{\partial \mathcal{L}}{\partial z_{iL}}\mathbf{a}_{L-1}
$$

This means that for all weight vectors $\mathbf{w}_{iL}$, the gradient is obtained by multiplying the activation vector of the previous layer by the derivative $\partial L/\partial z_{iL}$ (a scalar). The result is a new vector with the same number of elements of $\mathbf{w}_{iL}$. If we put all these in the rows of a matrix, we end up with a gradient matrix of the same size of $\mathbf{W}_{L}$. Furthermore, if we put the derivatives $\partial L/\partial z_{iL}$, for $i=1,...,K$, in a column vector $\partial L/\partial\mathbf{z}_L$, then this gradient matrix can be calculated in a compact way using an outer product of vectors:

$$
\nabla_{\mathbf{W}_L}\mathcal{L} = \frac{\partial \mathcal{L}}{\partial\mathbf{z}_L} \mathbf{a}_{L-1}^T =
\begin{pmatrix}
\partial \mathcal{L}/\partial z_{1L}\\
\partial \mathcal{L}/\partial z_{2L}\\
\vdots\\
\partial \mathcal{L}/\partial z_{KL}
\end{pmatrix}
\begin{pmatrix}
-\;\mathbf{a}_{L-1}\;-
\end{pmatrix}
=
\begin{pmatrix}
-\;\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{1L}}\;-\\
-\;\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{2L}}\;-\\
\vdots\\
-\;\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{KL}}\; -
\end{pmatrix}
$$

Having calculated the derivative $\partial \mathcal{L}/\partial z_{iL}$, the gradient with respect to the biases of the last layers is straightforward:

$$
\frac{\partial \mathcal{L}}{\partial b_{iL}} = \frac{\partial \mathcal{L}}{\partial z_{iL}}\frac{\partial z_{iL}}{\partial b_{iL}} = \frac{\partial \mathcal{L}}{\partial z_{iL}}\frac{\partial}{\partial b_{iL}}[\mathbf{w}_{iL}^T\mathbf{a}_{L-1} + b_{iL}] = \frac{\partial \mathcal{L}}{\partial z_{iL}}\cdot 1 = \frac{\partial \mathcal{L}}{\partial z_{iL}}
$$

If we put all $\partial \mathcal{L}/\partial b_{iL}$ in a column vector, we get the gradient of the loss with respect to the bias vector $\mathbf{b}_L$:

$$
\nabla_{\mathbf{b}_L}\mathcal{L} = \frac{\partial \mathcal{L}}{\partial\mathbf{z}_L} =
\begin{pmatrix}
\partial \mathcal{L}/\partial z_{1L}\\
\partial \mathcal{L}/\partial z_{2L}\\
\vdots\\
\partial \mathcal{L}/\partial z_{KL}
\end{pmatrix}
$$

We have derived the gradients of the loss with respect to all parameters in the last layer. What about the other layers? Let's think, in particular, of the gradients of layer $L-1$. The loss $\mathcal{L}$ is a function of the quantity $z_{jL-1}$, which is a function of the weights $\mathbf{w}_{jL-1}$ of unit $j$ in layer $L-1$. Therefore, similarly as before we have

$$
\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{jL-1}} = \frac{\partial \mathcal{L}}{\partial z_{jL-1}}\frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}
$$

However, since we have fully connected layers, the loss depends on $z_{jL-1}$ through the values of the next layer: $z_{1L}$, $z_{2L}$, and so on, up to $z_{1L}$. As a result, again due to the chain rule, the gradient can be rewritten as a sum, one for each $z_{iL}$ in the next layer on which $z_{jL-1}$ has an influence:

$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{jL-1}}
&= \frac{\partial \mathcal{L}}{\partial z_{jL-1}}\frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}\\
&= \left(\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}\frac{\partial z_{iL}}{\partial z_{jL-1}} \right) \frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}\\
&= \left(\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}\frac{\partial z_{iL}}{\partial a_{jL-1}} \frac{\partial a_{jL-1}}{\partial z_{jL-1}}\right) \frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}\\
&= \frac{\partial a_{jL-1}}{\partial z_{jL-1}} \left(\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}\frac{\partial z_{iL}}{\partial a_{jL-1}} \right) \frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}\\
\end{align}
$$

Note that the first term inside the summation, $\partial\mathcal{L}/\partial z_{iL}$, was already calculated when computing the gradients of layer $L$. It appears now in the calculation of the gradients of layer $L-1$, that is, **the gradients are being "backpropagated"**. The second term inside the summation is derived as follows:

$$
\begin{align}
\frac{\partial z_{iL}}{\partial a_{jL-1}}
&=
\frac{\partial}{\partial a_{jL-1}}[\mathbf{w}_{iL}^T\mathbf{a}_{L-1} + b_{iL}]\\
&=
\frac{\partial}{\partial a_{jL-1}}[w_{iLj}a_{jL-1} + b_{iL}]\\
&=
w_{iLj}
\end{align}
$$

Furthermore, we have

$$
\frac{\partial a_{jL-1}}{\partial z_{jL-1}} = \frac{\partial }{\partial z_{jL-1}}[f_{L-1}(z_{jL-1})] = f_{L-1}^\prime(z_{L-1})
$$

This corresponds to the derivative of the activation function of layer $L-1$. For instance, if we use a [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), we have

$$
\begin{align}
f(x) &= \max(0, x)\\
f^\prime(x) &= \left\{
\begin{matrix}
0 \text{ if } x \leq 0\\
1 \text{ if } x > 0
\end{matrix}
\right.
\end{align}
$$

Finally, from the previous results, we can readily see that

$$
\frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}} = \mathbf{a}_{L-2}
$$

Putting these results together, we obtain

$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{jL-1}}
&=
\frac{\partial \mathcal{L}}{\partial z_{jL-1}}\frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}\\
&=
\frac{\partial a_{jL-1}}{\partial z_{jL-1}} \left(\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}\frac{\partial z_{iL}}{\partial a_{jL-1}} \right) \frac{\partial z_{jL-1}}{\partial\mathbf{w}_{jL-1}}\\
&=
f^\prime_{L-1}(z_{jL-1})\left(\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}w_{iLj} \right) \mathbf{a}_{L-2}\\
\end{align}
$$

From the chain rule, the term inside the parentheses is equal to the scalar $\partial \mathcal{L}/\partial a_{jL-1}$, which for all $j$ we can arrange into a column vector:

$$
\begin{align}
\frac{\partial\mathcal{L}}{\partial \mathbf{a}_{L-1}}
= \begin{pmatrix}
\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}w_{iL1}\\
\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}w_{iL2}\\
\vdots\\
\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}w_{iLK^\prime }
\end{pmatrix}
\end{align}
$$

where $K^\prime$ is the number of units in layer $L-1$. This allows us to write

$$
\frac{\partial \mathcal{L}}{\partial\mathbf{z}_{L-1}}
=
f^\prime_{L-1}(\mathbf{z}_{L-1})\odot \frac{\partial\mathcal{L}}{\partial \mathbf{a}_{L-1}}
$$

where $\odot$ denotes element-wise multiplication.

Note that in $\partial\mathcal{L}/\partial\mathbf{a}_{L-1}$ every element of the form

$$
\sum_{i=1}^K \frac{\partial\mathcal{L}}{\partial z_{iL}}w_{iLj}
$$

sums a product between the scalars $\partial\mathcal{L}/\partial z_{iL}$ and the $j$-th element of every weight vector of layer $L$. Taking the $j$-th element of every weight vector of layer $L$ is equivalent to taking the $j$-th column of $\mathbf{W}_L$. Equivalently, the sum is then a dot product between the rows of $\mathbf{W}_L^T$ and the vector $\partial L/\partial \mathbf{z}_L$. This allows us to write $\partial\mathcal{L}/\partial \mathbf{a}_{L-1}$ as

$$
\frac{\partial\mathcal{L}}{\partial \mathbf{a}_{L-1}}
=
\mathbf{W}_L^T \frac{\partial L}{\partial \mathbf{z}_L}
$$

We can conclude that

$$
\frac{\partial \mathcal{L}}{\partial\mathbf{w}_{jL-1}} = \frac{\partial \mathcal{L}}{\partial z_{jL-1}}\mathbf{a}_{L-2}
$$

which, as before, we can put in the rows of a matrix gradient that can be expressed as an outer product:

$$
\nabla_{\mathbf{W}_L-1}\mathcal{L} = \frac{\partial \mathcal{L}}{\partial\mathbf{z}_{L-1}} \mathbf{a}_{L-2}^T
$$

In a similar way, with respect to the biases of layer $L-1$ we have

$$
\frac{\partial \mathcal{L}}{\partial b_{jL-1}}
= \frac{\partial \mathcal{L}}{\partial z_{jL-1}}\frac{\partial z_{jL-1}}{\partial b_{jL-1}} = \frac{\partial \mathcal{L}}{\partial z_{jL-1}}
$$

Therefore, the gradient is

$$
\nabla_{\mathbf{b}_{L-1}}\mathcal{L} = \frac{\partial \mathcal{L}}{\partial\mathbf{z}_{L-1}}
$$

And we are done! We can see that the expressions for the gradients are the same for every layer, and the only difference in the last layer comes in the calculation of the term $\partial\mathcal{L}/\partial\mathbf{z}_L$. These equations give us the algorithm to "train a neural network" using gradient descent, that is, to iteratively adjust the parameters of the model in the direction that minimizes the negative log-likelihood for an observed data point.

Usually, instead of doing gradient descent for a single point, we do this with mini-batches. In this case, we have an input matrix $\textbf{X}$ and activation matrices $\mathbf{A}_l$ for each layer, with a number of columns equal to the mini-batch size. The gradients are calculated for multiple data points in a mini-batch, adding them together and dividing by the number of data points to obtain an average gradient, which is then used to update the parameters. In the algorithm below, we denote the average of elements in an array through the mini-batch dimension as $\texttt{mean}. The result can be a number, if the array is one dimensional, or a vector, if it is two dimensional. In the case of outer products, such as the one needed to calculate $\nabla_{\mathbf{W}_L-1}\mathcal{L}$, the sum across the mini-batch dimension is performed by the matrix multiplication, so we only need to divide by the number of samples in the mini-batch to obtain the average gradient.

## The algorithm

**Inputs**

- $\mathbf{X}$, each column is an input vector, with $n$ columns corresponding to the mini-batch size.

- $\mathbf{T}$, each column is a one-hot target vector where for each sample, position $k$ is one for the assigned label.

**Forward pass**

$\mathbf{A}_0 = \mathbf{X}$

For $l=1,...,L-1$:

- $\mathbf{Z}_l = \mathbf{W}_l\mathbf{A}_{l-1} + \mathbf{b}_l$
- $\mathbf{A}_l = f_l(\mathbf{Z}_l)$

$\mathbf{Z}_L = \mathbf{W}_L\mathbf{A}_{L-1} + \mathbf{b}_L$

$\mathbf{Z} =$ vector with elements

$\qquad\sum_{j=1}^K\exp(z_{jL})$ per sample

$\mathcal{L} =$ vector with elements

$\qquad -z_{kL} + \log\sum_{j=1}^K\exp(z_{jL})$ per sample

and calculate $\texttt{mean}[\mathcal{L}]$ to obtain the mini-batch loss.

**Backward pass**

For $l=L,...,1$:

- $\frac{\partial\mathcal{L}}{\partial\mathbf{Z}_l} = \left\{
\begin{aligned}
&\frac{1}{\mathbf{Z}}\odot\exp(\mathbf{Z}_L) - \mathbf{T}&\text{if } l = L\\
&f^\prime_{l}(\mathbf{z}_{l})\odot \frac{\partial\mathcal{L}}{\partial \mathbf{a}_{l}}&\text{if } l < L
\end{aligned}
\right.$

- $\nabla_{\mathbf{W}_l}\mathcal{L} = \frac{1}{n}\left[\frac{\partial\mathcal{L}}{\partial\mathbf{Z}_{l}} \mathbf{A}_{l-1}^T\right]$

- $\nabla_{\mathbf{b}_{l}}\mathcal{L} = \texttt{mean}\left[\frac{\partial\mathcal{L}}{\partial\mathbf{Z}_{l}}\right]$

- $\frac{\partial\mathcal{L}}{\partial\mathbf{a}_{l-1}} = \mathbf{W}_l^T\frac{\partial L}{\partial\mathbf{z}_l}$

**Parameter updates with Gradient Descent**

For $l = 1, ..., L$:

- $\mathbf{W}_l = \mathbf{W}_l - \alpha\nabla_{\mathbf{W}_l}\mathcal{L}$
- $\mathbf{b}_L = \mathbf{b}_L - \alpha\nabla_{\mathbf{b}_{l}}\mathcal{L}$

where $\alpha$ is the learning rate.

## The code

We can define functions and classes that wrap the operations we have defined. First, let's create a class that represents a linear operation of the form $\mathbf{W}\mathbf{X} + \mathbf{b}$, which will store the parameters, the gradients and the values needed for backpropagation. We will use the so-called *He initialization* [1] to initialize the weights $\mathbf{W}$.

```python
import numpy as np

class Linear:
    """ A linear layer that performs the operation
    W X + b
    Args:
        - n_inputs (int): number of inputs
        - n_outputs (int): number of outputs
    """
    def __init__(self, n_inputs, n_outputs):
        self.W = np.random.standard_normal((n_outputs, n_inputs)) * np.sqrt(2/n_inputs)
        self.b = np.zeros((n_outputs, 1))
        self.dW = None
        self.db = None
        self.A_prev = None
        self.Zl = None

    def __call__(self, X, cache=False):
        """ Performs a forward pass on the layer given X.
        Args:
            - X (n_inputs, n): input array
            - cache (bool): whether or not to save intermediate
                values. Used during backpropagation.
        Returns: the result of the operation of the layer on X.
        """
        Zl = self.W @ X + self.b
        if cache:
            self.A_prev = X.copy()
            self.Zl = Zl.copy()
        return Zl
```

For the activation function we will use ReLU.

```python
def relu(X):
    """ ReLU activation function """
    return np.where(X > 0, X, 0)

def d_relu(X):
    """ Derivative of the ReLU activation function """
    return np.where(X > 0, 1, 0)

# Let's visualize the actions of these functions
import matplotlib.pyplot as plt
%matplotlib inline

x = np.linspace(-1, 1, num=100)
plt.plot(x, relu(x), label='f(x)')
plt.plot(x, d_relu(x), label='df/dx')
plt.title('f = ReLU')
plt.xlabel('x')
plt.legend();
```

We will define an API for the model that accepts arrays of integer labels for each sample. However, to keep compatibility with our equations, we will internaly make use of one hot vectors for some operations. The following function does this transformation.

```python
def one_hot_convert(t, n_classes):
    """ Convert arrays of integer labels to one-hot encoded vectors.
    Args:
        - t (n,): array containing n labels
        - n_classes (int): the number of possible classes
    Returns: (n_classes, n) binary array containing one-hot
        vectors in its columns.
    """
    if np.min(t) < 0 or np.max(t) >= n_classes:
        raise ValueError("Elements in array must be in the interval [0, {:d})".format(n_classes))
    T = np.zeros((n_classes, len(t)), dtype=int)
    T[t, np.arange(len(t))] = 1
    return T
```

Next, we define the `NNClassifier` class, which contains methods to initialize the architecture, and methods for forward and backward passes as derived above. For convenience, I've defined different forward passes, which can be used for different situations:

- `NNClassifier.forward` performs a forward pass up to but not including the softmax. These values are not probabilities, as they are merely the result of the linear operation in the last layer. They are also known as the *logits*. When we want to pick the most likely class given an input, it is enough to get the index of the maximum logit, avoiding the calculation of probabilities.
- `NNClassifier.forward_probs` performs a complete forward pass, including the last softmax layer. This results in actual probabilities in the interval $(0, 1)$.

As we saw during the derivations, the gradients with respect to the parameters of the layer require information of the input and output of the layer. This means that when doing a forward pass, this information needs to be stored if a backward pass is required later. The `forward` function has a `cache` argument that determines this. The `loss` function computes the loss given input vectors `X` and targets `t`, by calling `forward` with `cache=True` and saving also the gradient $\partial\mathcal{L}/\partial\mathbf{Z}_L$, so that all the information is ready to perform a backward pass to calculate all the gradients. Using this API, an example to calculate the gradients for a given input would go as follows:

```python
# Instantiate a model
model = NNClassifier(units)
# Get the loss with a forward pass
# (this also stores intermediate values needed for backpropagation)
loss = model.loss(X, t)
# Calculate the gradients with backpropagation
model.backward()
```

The class is defined below.

```python
class NNClassifier:
    """ A neural network classifier.
    Args:
        - units (iterable): contains the number of units (int)
            for each layer, from the input layer up to the
            output layer.
    """
    def __init__(self, units):
        self.units = units
        self.n_classes = units[-1]
        self.layers = []
        for i in range(1, len(units)):
            self.layers.append(Linear(units[i - 1], units[i]))
        self.dZL = None

    def forward(self, X, cache=False):
        A_prev = X
        for layer in self.layers:
            Zl = layer(A_prev, cache)
            A_prev = relu(Zl)

        return Zl

    def _log_normalizer(self, ZL):
        max_ZL = np.max(ZL, axis=0, keepdims=True)
        log_Z = max_ZL + np.log(np.sum(np.exp(ZL - max_ZL), axis=0, keepdims=True))
        return log_Z

    def loss(self, X, t):
        ZL = self.forward(X, cache=True)
        log_Z = self._log_normalizer(ZL)
        log_probs = ZL - log_Z

        T = one_hot_convert(t, self.n_classes)
        Z = np.exp(log_Z)
        self.dZL = np.exp(ZL)/Z - T

        return -np.mean(log_probs[T == 1])

    def forward_probs(self, X):
        ZL = self.forward(X)
        log_Z = self._log_normalizer(ZL)
        log_probs = ZL - log_Z
        return np.exp(log_probs)

    def backward(self):
        # Number of samples in mini-batch
        n = self.dZL.shape[1]
        for i, layer in enumerate(reversed(self.layers)):
            if i == 0:
                dZl = self.dZL
            else:
                dZl = d_relu(layer.Zl) * dAl
            layer.dW = 1/n * (dZl @ layer.A_prev.T)
            layer.db = np.mean(dZl, axis=1, keepdims=True)
            dAl = layer.W.T @ dZl
```

Lastly, we implement Stochastic Gradient Descent (SGD) to optimize the parameters of the network.

```python
class SGD:
    """ Stochastic Gradient Descent optimizer.
    Args:
        - model (NNClassifier): model with parameters to optimize
        - lr (float): learning rate
    """
    def __init__(self, model, lr):
        self.lr = lr
        self.model = model

    def step(self):
        for layer in self.model.layers:
            layer.W = layer.W - self.lr * layer.dW
            layer.b = layer.b - self.lr * layer.db
```

## Testing the implementation

Given the rather general implementation, we are free to create different architectures. If we let the input vector to be one dimensional and build a binary classifier with no hidden layers, we end up with a Logistic Regression model. We thus should expect the model to be able to model the sigmoid function. Let us test this first.

```python
import matplotlib.pyplot as plt
from scipy.special import expit # i.e. sigmoid function
%matplotlib inline

# Define a binary classifier over a one-dimensional variable:
# 1 input unit, 2 output units, no hidden layers
model = NNClassifier([1, 2])
# Set the weights so that the model outputs the sigmoid
model.layers[0].W = np.array([[1], [0]])

# Visualize for different values of the input
x = np.linspace(-2, 2).reshape((1, -1))
y = model.forward_probs(x)
plt.plot(x.T, y[0, :], label='model(x)')
plt.plot(x.T, expit(x.T), '.', label=r'$\sigma$(x)')
plt.legend();
```

As we can see, the network can model the sigmoid function. For bigger networks, we can check that the output probabilities sum to one.

```python
# Define an arbitrary network
inputs = 500
outputs = 10
model = NNClassifier([inputs, 100, 50, 40, outputs])

# Define data
n_samples = 1
X = np.random.rand(inputs, n_samples)

# Sum output probabilities
print(np.sum(model.forward_probs(X)))
```

    1.0000000000000002

Close enough!

There is one last and very important test that we need, which concerns the implementation of backpropagation. If it is correct, all the gradients of the loss with respect to the parameters could be approximated up to a small error using a finite difference approximation. To check this we can use scipy's `check_grad` function, which according to the documentation, returns

> The square root of the sum of squares (i.e. the 2-norm) of the difference between grad(...) and the finite difference approximation of grad.

We will use this function to print the sum of squares for each layer, for each parameter (weights and bias) in the network.

```python
from scipy.optimize import check_grad

# Create some random targets to evaluate the loss
t = np.random.randint(0, high=outputs, size=(1, n_samples))

for l, layer in enumerate(model.layers):
    print('Layer {:d}'.format(l + 1))
    for param in ['W', 'b']:
        def func(x):
            # Set the parameter and evaluate the loss
            setattr(layer, param, np.reshape(x, getattr(layer, param).shape))
            return model.loss(X, t)

        def grad(x):
            # Set the parameter and evaluate the gradient
            setattr(layer, param, np.reshape(x, getattr(layer, param).shape))
            model.loss(X, t)
            model.backward()
            return np.ravel(getattr(layer, 'd' + param))

        x = np.ravel(getattr(layer, param))
        print('d{}: {}'.format(param, check_grad(func, grad, x)))
```

    Layer 1
    dW: 2.830136936348668e-06
    db: 1.1719993320136231e-07
    Layer 2
    dW: 5.290619237012628e-07
    db: 1.3910441647589437e-07
    Layer 3
    dW: 5.194495178455115e-07
    db: 6.909870175421613e-08
    Layer 4
    dW: 2.013252965205644e-07
    db: 9.237907776608362e-08

We can deem these low enough to trust the implementation of backpropagation. We can proceed to train the model with some toy data, just to see if the model is able to overfit to it.

```python
from sklearn.datasets import make_moons

X, Y = make_moons(noise=0.1)
plt.scatter(X[Y==0, 0], X[Y==0, 1])
plt.scatter(X[Y==1, 0], X[Y==1, 1]);
```

A linear model with no hidden units would not perform well since the classes are not linearly separable. We will train a model with one hidden layer, with 10 units in it.

```python
model = NNClassifier([2, 10, 2])
optimizer = SGD(model, lr=0.5)

n_epochs = 300
loss_history = np.zeros(n_epochs)

for epoch in range(n_epochs):
    # Forward pass
    loss_history[epoch] = model.loss(X.T, Y)
    # Backward pass
    model.backward()
    # Parameter updates
    optimizer.step()

plt.plot(loss_history)
plt.title('Final loss: {:.3f}'.format(np.min(loss_history)))
plt.xlabel('Epochs');
```

The optimizer seems to be doing its job! The loss decreases monotonically and gets to a low value, confirming that we can overfit to the training data. As in a [previous post](https://dfdazac.github.io/04-fisher-example.html), we can visualize the decision boundary created by the model after training:

```python
# Create a grid
N = 200
x1 = np.linspace(-2, 3, N)
x2 = np.linspace(-1, 2, N)
X1, X2 = np.meshgrid(x1, x2)
X_flat = np.column_stack((X1.flatten(), X2.flatten()))

# Evaluate model on grid
Y_pred = model.forward_probs(X_flat.T)[0, :].reshape(X1.shape) < 0.5

plt.contourf(X1, X2, Y_pred, cmap='bone', alpha=0.1)
plt.scatter(X[Y == 0, 0], X[Y == 0, 1])
plt.scatter(X[Y == 1, 0], X[Y == 1, 1]);
```

It's interesting to see how this decision boundary is much more irregular compared to that obtained by Fisher's discriminant in a previous post. The neural network is a more complex model, so it can come up with more elaborate functions of the input, as we observe here.

As we have seen, the implementation from scratch can be challenging but also very rewarding. Knowing how a neural network works and is trained can be useful when debugging problems such as overfitting and exploding and vanishing gradients. On the other hand, we are lucky to have tools such as automatic differentiation, which enables us to apply backpropagation to bigger and more complex models easily.

In future posts I hope to use the implementation created here in problems such as the classic MNIST dataset, and important aspects such as improved gradient descent algorithmgs, regularization techniques and hyperameter tuning.

**References**

[1] He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." *Proceedings of the IEEE international conference on computer vision*. 2015.

---

1. But why this "softmax"? For binary classification problems we use the *logistic sigmoid*, and the fact that these functions are used is related to a class of models known as Generalized Linear Models. When working with binary classification problems, we are trying to model a Bernoulli distribution, for which we can find what is known as a *link function* from which the sigmoid can be derived. Similarly, the softmax corresponds to the Categorical distribution. [This](http://blog.shakirm.com/2015/01/a-statistical-view-of-deep-learning-i-recursive-glms/) is an interesting discussion that frames neural networks as hierarchical generalized linear models.[^](#1b)

2. In practice we don't necessarily want to find the minimum for this objective. We could minimize it as much as we can by choosing an appropriately complex model and training for long enough, but we might end up overfitting to the training set and with a model that does not generalize well to unseen data. In the end, we want our model to perform well both on training and test data, and with neural networks there are multiple ways to achieve this, such as weight regularization, dropout, data augmentation, among other techniques. [^](#2b)


# Aula 4

# Soluções Numéricas

Frequentemente, em situações reais, os problemas de engenharia são muito complexos para que uma solução analítica (uma equação explícita). Nesses casos geralmente utilizamos soluções numéricas. De forma geral essas soluções consistem em dividir um problema real em pequenas partes (discretização) e aplicar os princípios físicos que regem o problema às essas pequenas partes. Nessa aula construiremos soluções numéricas para problemas de magnetismo.

```python
import numpy as np
import matplotlib.pyplot as plt

mi0 = 4*np.pi*1e-7 #[H/m]
```

Trataremos do problema de um dipolo magnético, conforme a seção 8.4 do livro Física Básica - Eletromagnetismo de Alaor Chaves. Esse problema consiste no cálculo do campo magnético ao longo do eixo Z gerado por uma espira de raio $R$ onde circula uma corrente $I$, conforme figura abaixo:

Para esse caso específico, uma solução analítica é possível e é dada por:

\begin{equation}
B = \frac{\mu_0}{2} \frac{IR^2}{(R^2+z^2)^{3/2}}
\end{equation}

Que podemos implementar em Python facilmente:

```python
def CalcularB(I, R, z):
    return (mi0*I*np.power(R,2))/(2.0*np.power(np.power(R, 2.0) + np.power(z, 2.0),3.0/2.0))
```

E podemos testá-la com valores arbitrários

```python
R = 5 # [m] Raio da espira condutora
I = 1 # [A] corrente que circula na espira
z_pos = 1 # [m] posicao em Z para o qual o valor de campo magnetico sera calculado

CalcularB(I,R,z_pos)
```

    1.1848404026933503e-07

Lembrando que o valor que essa função retorna é referente a componente ao longo do eixo Z. AS componentes $î$ e $ĵ$ são nulas.

Para desenvolver um método numérico que resolve o mesmo tipo de problema, precisamos primeiramente discretizar o sistema físico. Começaremos representando o círculo por um conjunto de pontos igualmente espaçados ao longo de seu perímetro.

```python
num_pts = 30 # numero de pontos para discretizacao da espira
circle = np.zeros((num_pts,3)) # matriz de coordenadas x,y,z dos pontos que representam a espira
angles = np.linspace(start=0, stop=2*np.pi, num=num_pts) # os angulos para os quais calcularemos os pontos

for idx, angle in enumerate(angles):
    circle[idx,0] = R*np.cos(angle)
    circle[idx,1] = R*np.sin(angle)

```

No código acima nos criamos uma matriz nula de tamanho $num\_pts\times 3$. Em seguida é gerada uma lista de ângulos para os quais queremos calcular a posição $(x,y)$ do cículo, que correspondem às colunas 0 e 1, respectivamente. Para isso utilizamos as seguintes equações:

\begin{equation}
    x = R cos(\theta)\\
    y = R sin(\theta)\\
    z = 0
\end{equation}

```python
plt.figure(figsize=(10,10))
plt.scatter(circle[:,0], circle[:,1]) # Plot do tipo scatter para visualisarmos os pontos
plt.xlabel('Eixo X [m]')
plt.ylabel('Eixo Y [m]')
plt.show()
```

Com essa estrutura já podemos, por exemplo, calcular o perímetro da espira numericamente. Nesse caso devemos implementar a seguinte integral:

\begin{equation}
        P = \int_{C}|d\overrightarrow{L}|
\end{equation}

No domínio discreto, a integral acima vira um simples somatório:

\begin{equation}
        P = \sum_{n=1}^{k-1} |c[n+1]-c[n]|
\end{equation}

Onde $c[n]$ é o ponto na n-ésima linha da matriz de pontos do círculo, $k$ é o número de pontos utilizados para a discretização e $c[n+1]-c[n]$ é a aproximação discreta para  $d\overrightarrow{L}$.

```python
perimeter = 0 # Acumulador para o perimetro
dL = np.zeros(circle.shape) # guardaremos os dLs para uso futuro
for idx in range(num_pts-1):
    dl = circle[idx+1,:] - circle[idx,:]
    dL[idx,:] = dl # armazena na matriz de dLs
    perimeter = perimeter + np.linalg.norm(dl) # acumula a norma de dl
print("O perimetro calculado eh %.7f e o valor teorico eh de %.7f " % (perimeter, 2*np.pi*R))

```

    O perimetro calculado eh 31.3545153 e o valor teorico eh de 31.4159265

Lembrando que quanto maior o número de pontos utilizados na discretização, melhor será a aproximação. Altere a variável $num\_pts$ e veja o impacto no valor da aproximação.

Agora que temos os pontos da espira e uma matriz com todos os $dL$s que precisamos, conforme figura abaixo, podemos voltar ao problema inicial.

```python
plt.figure(figsize=(10,10))
plt.scatter(circle[:,0], circle[:,1])
plt.quiver(circle[:,0], circle[:,1], dL[:,0], dL[:,1],units='inches')
plt.show()

```

A integral que deve ser resolvida é:

\begin{equation}
    \overrightarrow{B} = \frac{\mu_0 I}{4\pi} \int \frac{d\overrightarrow{L} \times \overrightarrow{r}}{|\overrightarrow{r}|^3}
\end{equation}

```python
B_resultante = np.zeros((1,3))
z_vector = np.array([0,0,z_pos])
K = (mi0/(4*np.pi))*I # Representamos todos os termos constantes apenas por K para facilidade

for c, dL_vec in zip(circle, dL): # Loop para percorrer pontos e seus dLs associados
    r_vec = z_vector - c
    dem = np.linalg.norm(r_vec)**3.0
    B_resultante = B_resultante + np.cross(dL_vec, r_vec)/dem

B_resultante = B_resultante*K
valor_analitico = CalcularB(I,R,z_pos)
print ("O valor teorico do campo na direcao Z eh %.5E e o valor calculado numericamente eh %.5E" % (valor_analitico, B_resultante[0,2]))
print ("O erro da aproximacao numerica eh %f%%" % ((valor_analitico - B_resultante[0,2])*100/valor_analitico))
```

    O valor teorico do campo na direcao Z eh 1.18484E-07 e o valor calculado numericamente eh 1.17559E-07
    O erro da aproximacao numerica eh 0.780536%

Utilizando a solução numérica temos acesso ainda a todas as componentes de $\overrightarrow{B}$:

```python
print(B_resultante)
```

    [[-3.46944695e-25 -1.01915004e-24  1.17559229e-07]]

Pode-se notar que as componentes X e Y não são exatamente 0, porém muito próximos, por conta da precisão da simulação. Novamente, quanto mais pontos forem utilizados, mais precisa é a simulação numérica.

## Exercício 1

Refazer a simulação substituindo o círculo de raio $R$ por um quadrado de lado $L=1m$ centrado na origem, mantendo o cálculo do campo no mesmo ponto.

Dica: basta gerar a lista de pontos que representa o quadrado, o restante do problema é identico ao problema do círculo.

```python

```

## Exercício 2

Dada a simulação do Ex. 1, agora calcular o módulo do campo magnético $\overrightarrow{B}$ ao longo de uma reta que passa pelo ponto $(0,0,0.1)$ e tem direção $î$ e plotar o resultado no intervalo de $-2 m$ à $2 m$.

Dica: Você terá que dividir a reta em diversos pontos e calcular o campo em cada um deles.

```python

```


# COURSE: A deep understanding of deep learning
## SECTION: Math prerequisites
### LECTURE: Derivatives: product and chain rules
#### TEACHER: Mike X Cohen, sincxpress.com
##### COURSE URL: udemy.com/course/dudl/?couponCode=202201

```python
# import libraries
import numpy as np
import sympy as sym

# make the equations look nicer
from IPython.display import display
```

```python
# create symbolic variables in sympy
x = sym.symbols('x')

# create two functions
fx = 2*x**2
gx = 4*x**3 - 3*x**4

# compute their individual derivatives
df = sym.diff(fx)
dg = sym.diff(gx)

# apply the product rule "manually"
manual = df*gx + fx*dg
thewrongway = df*dg

# via sympy
viasympy = sym.diff( fx*gx )

# print everything
print('The functions:')
display(fx)
display(gx)
print(' ')

print('Their derivatives:')
display(df)
display(dg)
print(' ')

print('Manual product rule:')
display(manual)
print(' ')

print('Via sympy:')
display(viasympy)
print(' ')

print('The wrong way:')
display(thewrongway)
```

```python
# repeat with chain rule
gx = x**2 + 4*x**3
fx = ( gx )**5

print('The function:')
display(fx)
print(' ')

print('Its derivative:')
display(sym.diff(fx))
```

```python

```


```python
import numpy as np
from sympy import Symbol, Eq, simplify
from sympy import sqrt, cosh, exp
from sympy.solvers import solve
from sympy import init_printing
init_printing()
```

```python
# Relevant variables
eta = Symbol(r'\eta')
x = Symbol('x')

# Kinematics
Q = Symbol('Q', positive=True)
qT = Symbol(r'|\mathbf{q}_T|', positive=True)
y = Symbol('y')
M = Symbol('M') #sqrt(Q**2+qT**2)

# Cut parameters
etamax = Symbol(r'\eta_{\rm max}', positive=True)
pTmin = Symbol(r'p_{T,\rm min}', positive=True)

# Definitions
ch = Symbol('\cosh(\eta-y)') #cosh(eta - y)
pTo = Q**2 / 2 / ( M * ch - qT * x)
```

```python
# xpTmin1
f = Eq(pTo, pTmin)
solve(f, x)
```

```python
# xpTmin2
f = Eq(qT**2 + pTo * ( pTo - 2 * qT * x), pTmin**2)
solve(f, x)
```

```python
f = Eq( ( M * exp(y) - pTo * exp(eta) ) / ( M * exp(-y) - pTo * exp(-eta) ) , exp(2*etamax))
solve(f, x)
```

```python
che = Symbol('\cosh(\eta)')
she = sqrt(che**2 - 1)
f = Eq(( che * ( Q**2 - 2 * pTmin**2 ) - Q * sqrt( Q**2 * she**2 + pTmin**2) ) / ( Q**2 - pTmin**2 ), 0)
solve(f, Q)
```

```python

```


# Derivatives and Integrals

## Basic Derivative Formula $$\frac{d}{dx} x^n = n \times x^{n-1}$$

# Basic Integral Formula $$\int x^n dx = \frac{1}{n+1} x^{n+1} + C$$

```python
# Load sympy module
from sympy import *
```

```python
# Define variables
x, y = symbols('x y')
```

## Define the function $$f(x) = x^3 - 2x^2 + x - 1$$

```python
# Function f(x) = x^3 - 2x^2 + x - 1
def f(x):
    return x**3 - 2*x**2 + x - 1
```

## Calculate $$D_x f = 3x^2 - 4x + 1$$

```python
# Derivative of f(x)
d_x = diff(f(x), x)
d_x
```

$\displaystyle 3 x^{2} - 4 x + 1$

```python
# Substitute x = 2
number_subs = diff(f(x), x).subs(x, 2)
print(number_subs)
```

    5

## Calculate $$\int f(x) dx$$

```python
# Integral of f(x)
integral = Integral(f(x), x)
integral
```

$\displaystyle \int \left(x^{3} - 2 x^{2} + x - 1\right)\, dx$

## The Result is $$\frac{x^4}{4} - \frac{2x^3}{3} + \frac{x^2}{2} - x$$

```python
# Do the integral of f(x)
integral.doit()
```

$\displaystyle \frac{x^{4}}{4} - \frac{2 x^{3}}{3} + \frac{x^{2}}{2} - x$

## Subtitute $$x = 2$$

```python
# Substitute x = 2
integ = integrate(f(x), x).subs(x, 2)
print(integ)
```

    -4/3

## Define the function $$g(x) = x \times y$$

```python
# Function g(x, y) = x * y
def g(x, y):
    return x * y
```

## Calculate $$\displaystyle \int_{0}^{2} \int_{0}^{2} f(x) dy dx = 4$$

```python
# Integral of g(x, y)
print(integrate(g(x, y), (x, 0, 2), (y, 0, 2)))
```

    4


$$
\sqrt{2}+\sqrt{3}=\sqrt{\left(\sqrt{2}+\sqrt{3}\right)^2}=\sqrt{2\sqrt{6}+5}
=\sqrt{\sqrt{\left(2\sqrt{6}+5\right)^2}} = \sqrt{\sqrt{20\sqrt{6}+49}}
$$

```python
import sympy as S
S.init_printing()
a = S.sqrt( S.sqrt(49+20*S.sqrt(6)))

a
```

```python
S.sqrtdenest(a)
```

$$
\sqrt{2}+\sqrt{3}=\sqrt{2\sqrt{6}+5}=\sqrt{1+(2\sqrt{6}+4)}
=\sqrt{1+\sqrt{\left(2\sqrt{6}+4\right)^2}} = \sqrt{1+\sqrt{16\sqrt{6}+40}}
$$

```python
b = S.sqrt(1+S.sqrt(40+16*S.sqrt(6)))

b
```

```python
S.sqrtdenest(b)
```


## Import packages, Environment Setting

```python
import numpy as np
import tensorflow as tf

from tensorflow.keras import activations

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)
```

## Gradient descent (1 dimensional)

Suppose we have a one variable nonlinear function of a single feature: $\mathbf{y}=\frac{1}{1000}(10\mathbf{x})^3 + \frac{1}{20}(10\mathbf{x})^2 + \frac{1}{100}(10\mathbf{x})$. Now we would like to construct a model to fit the above-mentioned function: $\frac{1}{1000}(w\mathbf{x})^3 + \frac{1}{20}(w\mathbf{x})^2 + \frac{1}{100}(w\mathbf{x})$. It is trivial to see that $w=10$ gives us the exact function of $\mathbf{y}$. Here, we will use a __gradient descent__ algorithm to approximate the solution.

To do so, we need to first define the __objective function__ between $\mathbf{y}$ and $\mathbf{\hat{y}}$, we use mean square error here:
$$
\begin{align}
L(\mathbf{y}, \mathbf{\hat{y}})&=\sum\limits_{i=1}^{m}(y_i - \hat{y}_i)^2\\
&=\sum\limits_{i=1}^{m}[y_i - (\frac{w^3x_{i}^3}{1000} + \frac{w^2x_{i}^2}{20} + \frac{wx_{i}}{100})]^2
\end{align}
$$
We can then compute the gradient of $L$ with respect to $w$:
$$
\nabla_{\mathbf{w}}L=2(\frac{3w^2\mathbf{x}^3}{1000} + \frac{w\mathbf{x}^2}{10} + \frac{\mathbf{x}}{100})^T(\mathbf{\hat{y}} - \mathbf{y})
$$
The derived gradient will always point towards the opposite direction from the weight that yield local minimum of the loss surface. Therefore, we can use this property to update the weight to make sure it is closer to the local minimum:
$$
\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)} - \eta\nabla_{\mathbf{w}}L(\mathbf{\hat{y}})
$$
where $\eta$ is a hyperparameter called __learning rate__, which is used to control the magnitude of each updates.

```python
num_samples = 10
tf.random.set_seed(0)
x = tf.random.normal((num_samples, 1), mean=0, stddev=1)
y = x ** 3 + 5 * x ** 2 + 0.1 * x
w = tf.Variable(0.5)

mse = tf.keras.losses.MeanSquaredError()

eta = 0.5
for i in range(0, 10):
    with tf.GradientTape() as g:
        g.watch(w)
        y_hat = ((w * x) ** 3 + 50 * (w * x) ** 2 + 10 * w * x) / 1000
        loss = mse(y, y_hat)
    gradient = g.gradient(loss, w)
    w = w.assign(w - eta * gradient)
    print(f'iteration:{i+1:>3}{"":>4}weight:{w.numpy():>7.3f}{"":>4}loss:{loss.numpy():>7.3f}')
```

    iteration:  1    weight:  0.838    loss: 36.317
    iteration:  2    weight:  1.390    loss: 36.016
    iteration:  3    weight:  2.291    loss: 35.213
    iteration:  4    weight:  3.746    loss: 33.086
    iteration:  5    weight:  5.979    loss: 27.677
    iteration:  6    weight:  8.818    loss: 15.992
    iteration:  7    weight: 10.401    loss:  2.022
    iteration:  8    weight:  9.678    loss:  0.284
    iteration:  9    weight: 10.187    loss:  0.168
    iteration: 10    weight:  9.862    loss:  0.060

## Gradient descent (2 dimensional)
Now let's look into multiple variables regression. Given a nonlinear transformatin of $\mathbf{X}$: $\mathbf{y}=ReLU(5.0\mathbf{x}_1 + 5.0\mathbf{x}_2)$, we would like to find $\hat{y}=ReLU(w_1\mathbf{x}_1+w_2\mathbf{x}_2)$:
$$
\mathbf{\hat{y}} = ReLU(\mathbf{X}\mathbf{w}) = ReLU(
\begin{bmatrix}
    x_{1,1} & x_{1,2}\\
    x_{2,1} & x_{2,2}\\
    \vdots  & \vdots \\
    x_{m,1} & x_{m,2}\\
\end{bmatrix}\begin{bmatrix}
    w_{1}   \\
    w_{2}   \\
\end{bmatrix}
)
$$
that approximate this function. The square loss function then become:
$$
\begin{align}
L&=\sum\limits_{i=1}^{m}(y_i - \hat{y}_i)^2\\
 &=\sum\limits_{i=1}^{m}[y_i - ReLU(\sum\limits_{j=1}^{2}w_jx_{ij})]^2
\end{align}
$$
The gradient of $L$ with respect to $\mathbf{w}$ is:
$$
\nabla_{\mathbf{w}}L(\mathbf{\hat{y}})=2\frac{\partial(\text{ReLU}(\mathbf{X}))}{\partial\mathbf{w}}(\mathbf{\hat{y}}-\mathbf{y})
$$

Although we only show the example of two dimensional feature matrix. The algorithm can be easily generalized to higher dimension.

```python
eta = 1
num_samples = 10

tf.random.set_seed(0)
x = tf.random.normal((num_samples, 2), mean=0, stddev=1)
w = tf.Variable([[8.], [10.]])
y = activations.relu(tf.matmul(x, w))

w = tf.Variable([[-15.], [-15.]])

for i in range(0, 10):
    with tf.GradientTape() as g:
        g.watch(w)
        y_hat = activations.relu(tf.matmul(x, w))
        loss = tf.reduce_mean(tf.pow(y - y_hat, 2))
    gradient = g.gradient(loss, w)
    w = w.assign(w - eta * gradient)
    message = ''.join((
        f'iteration:{i+1:>3}{"":>4}weight 1:{w.numpy()[0, 0]:>7.3f}{"":>4}',
        f'weight 2:{w.numpy()[1, 0]:>7.3f}{"":>4}loss:{loss.numpy():>7.3f}'
    ))
    print(message)
```

    iteration:  1    weight 1: -1.922    weight 2: -4.658    loss:233.694
    iteration:  2    weight 1:  0.084    weight 2: -1.810    loss: 66.604
    iteration:  3    weight 1:  0.454    weight 2: -0.885    loss: 58.866
    iteration:  4    weight 1:  5.278    weight 2:  0.828    loss: 57.196
    iteration:  5    weight 1:  8.291    weight 2:  6.033    loss: 27.748
    iteration:  6    weight 1:  8.661    weight 2:  8.062    loss:  3.822
    iteration:  7    weight 1:  8.543    weight 2:  8.947    loss:  0.819
    iteration:  8    weight 1:  8.401    weight 2:  9.367    loss:  0.226
    iteration:  9    weight 1:  8.305    weight 2:  9.575    loss:  0.077
    iteration: 10    weight 1:  8.235    weight 2:  9.695    loss:  0.036

## Epoch and Batch
When we compute the loss, we can either consider every sample or a subset of samples at a time. We call the iteration that every sample in the dataset has been used to update the parameters an __epoch__. In each epoch, we can split the samples into different __batches__.
* __Mini-batch gradient descent__: updates the parameters consider a batch of samples at a time.
* __Stochastic gradient descent (SGD)__: update the parameters consider one sample at a time).

```python
def mini_batch_gradient_descent(x, y, w, num_epochs, batch_size, eta=0.03):
    num_samples = x.shape[0]

    for i in range(0, num_epochs):
        indices = tf.range(start=0, limit=num_samples, dtype=tf.int32)
        tf.random.set_seed(i)
        shuffled_indices = tf.random.shuffle(indices)

        shuffled_x = tf.gather(x, shuffled_indices)
        shuffled_y = tf.gather(y, shuffled_indices)

        batches = [batch_size] * (num_samples // batch_size)
        if num_samples % batch_size != 0:
            batches.extend([num_samples % batch_size])

        batches_x = tf.split(shuffled_x, batches, axis=0)
        batches_y = tf.split(shuffled_y, batches, axis=0)

        for j, (batch_x, batch_y) in enumerate(zip(batches_x, batches_y)):
            with tf.GradientTape() as g:
                g.watch(w)
                y_hat = activations.relu(tf.matmul(batch_x, w))
                loss = tf.reduce_mean(tf.pow(batch_y - y_hat, 2))
            gradient = g.gradient(loss, w)
            w = w.assign(w - eta * gradient)
            message = ''.join((
                f'epoch:{i+1:>3}{"":>4}batch:{j+1:>3}{"":>4}weight 1:{w.numpy()[0, 0]:>7.3f}',
                f'{"":>4}weight 2:{w.numpy()[1, 0]:>7.3f}{"":>4}loss:{loss.numpy():>8.3f}'
            ))
            print(message)

    return
```

```python
eta = 0.5
batch_size = 5
num_epochs = 5
num_samples = 20

tf.random.set_seed(0)
x = tf.random.normal((num_samples, 2), mean=0, stddev=1)
w = tf.Variable([[8.], [10.]])
y = activations.relu(tf.matmul(x, w))

w = tf.Variable([[-12.90], [-10.10]])

mini_batch_gradient_descent(x, y, w, num_epochs, batch_size, eta)
```

    epoch:  1    batch:  1    weight 1: -8.620    weight 2:-11.302    loss: 104.110
    epoch:  1    batch:  2    weight 1:  2.346    weight 2: -9.223    loss: 176.833
    epoch:  1    batch:  3    weight 1:  2.168    weight 2: -7.583    loss: 161.436
    epoch:  1    batch:  4    weight 1:  1.886    weight 2: -5.014    loss:  43.494
    epoch:  2    batch:  1    weight 1:  1.244    weight 2: -4.225    loss:  16.860
    epoch:  2    batch:  2    weight 1:  6.825    weight 2: -2.493    loss:  78.891
    epoch:  2    batch:  3    weight 1:  9.963    weight 2:  1.797    loss: 102.498
    epoch:  2    batch:  4    weight 1: 10.698    weight 2:  6.697    loss:  38.748
    epoch:  3    batch:  1    weight 1: 10.968    weight 2:  8.526    loss:   5.312
    epoch:  3    batch:  2    weight 1:  7.207    weight 2:  9.451    loss:  12.181
    epoch:  3    batch:  3    weight 1:  7.223    weight 2:  9.569    loss:   0.077
    epoch:  3    batch:  4    weight 1:  7.456    weight 2: 10.010    loss:   0.371
    epoch:  4    batch:  1    weight 1:  7.586    weight 2: 10.173    loss:   0.069
    epoch:  4    batch:  2    weight 1:  7.876    weight 2:  9.987    loss:   0.152
    epoch:  4    batch:  3    weight 1:  7.937    weight 2: 10.011    loss:   0.008
    epoch:  4    batch:  4    weight 1:  7.937    weight 2: 10.007    loss:   0.000
    epoch:  5    batch:  1    weight 1:  7.965    weight 2: 10.015    loss:   0.002
    epoch:  5    batch:  2    weight 1:  7.967    weight 2: 10.013    loss:   0.000
    epoch:  5    batch:  3    weight 1:  7.969    weight 2: 10.007    loss:   0.000
    epoch:  5    batch:  4    weight 1:  7.990    weight 2:  9.999    loss:   0.001

## Instability
The gradient descent algorithm might not be able to find the optimal parameters that globally minimize the loss function for many reasons. For instance, if the learning rate is set to be too large, the loss function will be unstable and might lead to non-optimal approximation. This problem can be identified by inspecting the training loss in each epoch:

```python
eta = 1.5
num_samples = 10
tf.random.set_seed(0)
x = tf.random.normal((num_samples, 1), mean=0, stddev=1)
y = x ** 3 + 5 * x ** 2 + 0.1 * x

w = tf.Variable(0.5)

for i in range(0, 10):
    with tf.GradientTape() as g:
        g.watch(w)
        y_hat = ((w * x) ** 3 + 50 * (w * x) ** 2 + 10 * w * x) / 1000
        loss = tf.reduce_mean(tf.pow(y - y_hat, 2))
    gradient = g.gradient(loss, w)
    w = w.assign(w - eta * gradient)
    print(f'iteration:{i+1:>3}{"":>4}weight:{w.numpy():>7.3f}{"":>4}loss:{loss.numpy():>7.3f}')
```

    iteration:  1    weight:  1.515    loss: 36.317
    iteration:  2    weight:  4.452    loss: 34.980
    iteration:  3    weight: 12.023    loss: 24.310
    iteration:  4    weight: -2.383    loss:  8.710
    iteration:  5    weight: -6.264    loss: 33.324
    iteration:  6    weight:-12.476    loss: 18.812
    iteration:  7    weight: -2.094    loss: 16.372
    iteration:  8    weight: -5.565    loss: 34.031
    iteration:  9    weight:-11.888    loss: 21.742
    iteration: 10    weight: -4.579    loss: 12.917

However, if the learning rate is set to be too small. The algorithm might stuck in local minimum. We usually solve this by proper weight initialization or applying the optimizers, which we will explain in more details in other exercise:

```python
num_samples = 10
tf.random.set_seed(0)
x = tf.random.normal((num_samples, 1), mean=0, stddev=1)
y = x ** 3 + 5 * x ** 2 + 0.1 * x

w = tf.Variable(-5.)

eta = 0.2
for i in range(0, 10):
    with tf.GradientTape() as g:
        g.watch(w)
        y_hat = ((w * x) ** 3 + 50 * (w * x) ** 2 + 10 * w * x) / 1000
        loss = tf.reduce_mean(tf.pow(y - y_hat, 2))
    gradient = g.gradient(loss, w)
    w = w.assign(w - eta * gradient)
    print(f'iteration:{i+1:>3}{"":>4}weight:{w.numpy():>7.3f}{"":>4}loss:{loss.numpy():>7.3f}')
```

    iteration:  1    weight: -5.829    loss: 24.109
    iteration:  2    weight: -6.671    loss: 20.628
    iteration:  3    weight: -7.473    loss: 17.149
    iteration:  4    weight: -8.184    loss: 14.095
    iteration:  5    weight: -8.766    loss: 11.784
    iteration:  6    weight: -9.206    loss: 10.289
    iteration:  7    weight: -9.516    loss:  9.460
    iteration:  8    weight: -9.721    loss:  9.060
    iteration:  9    weight: -9.852    loss:  8.887
    iteration: 10    weight: -9.932    loss:  8.818

```python

```


We consider the IL of an LP who contributes $\Delta r_\alpha$ of liquidity in asset $i$ to Omnipool through a price of of $p_\alpha \to p_i^Q$.

Per the spec, when the LP contributed $\Delta r_\alpha (< 0)$ shares, they received
$$
s_i = \Delta s_\alpha = S_{i0} \frac{-\Delta r_\alpha}{R_{i0}}\\
$$

(Note that the sign of $\Delta$ variables will indicate flow to the agent or AMM depending on the variable. When they agent contributes liquidity to the AMM, $\Delta r_\alpha < 0$, since it is an agent variable.)

We now consider what the LP is entitled to withdraw.

Per the spec, the LP receives $\Delta r_i$ of asset $i$ and $\Delta q_i$ of LHDX, with
$$
\begin{align}
\Delta B_i &= max\left(\frac{p_\alpha - p_i^Q}{p_i^Q + p_\alpha}s_i, -B_i\right)\\
\Delta r_i &= \frac{R_i}{S_i} (s_i - \Delta B_i) \\\\
\Delta q_i &= p_i^Q\left(\frac{2p_i^Q}{p_i^Q + p_\alpha} \frac{s_i}{S_i}R_i - \Delta r_i\right)
\end{align}
$$

We consider 2 cases.

### $\frac{p_\alpha - p_i^Q}{p_i^Q + p_\alpha}s_i > -B_i$

$$
\Delta B_i = \frac{p_\alpha - p_i^Q}{p_i^Q + p_\alpha}s_i\\
\Delta r_i = \frac{R_i}{S_i} (s_i - \frac{p_\alpha - p_i^Q}{p_i^Q + p_\alpha}s_i) = \frac{R_i}{S_i}\frac{2p_i^Q}{p_i^Q + p_\alpha}s_i\\
\Delta q_i = 0
$$

### $\frac{p_\alpha - p_i^Q}{p_i^Q + p_\alpha}s_i < -B_i$

$$
\begin{align}
\Delta B_i &= -B_i\\
\Delta r_i &= \frac{R_i}{S_i} (s_i + B_i)\\
\Delta q_i &= p_i^Q\left(\frac{2p_i^Q}{p_i^Q + p_\alpha} \frac{s_i}{S_i}R_i - \frac{R_i}{S_i} (s_i + B_i)\right)\\
\end{align}
$$

Note that in either of these cases, the payout has value
$$
p_i^Q\Delta r_i + \Delta q_i = p_i^Q \frac{R_i}{S_i}\frac{2p_i^Q}{p_i^Q + p_\alpha}s_i
$$

For the IL calculation, this is $val_{pool}$, while $val_{hold} = -p_i^Q\Delta r_\alpha$.

Thus
$$
\begin{align}
IL &= \frac{val_{pool}}{val_{hold}} - 1\\
&= p_i^Q \frac{R_i}{S_i}\frac{2p_i^Q}{p_i^Q + p_\alpha}s_i \frac{1}{-p_i^Q\Delta r_\alpha} - 1\\
&= \frac{R_{i0} + \Delta r_\alpha}{S_{i0} + s_i}\frac{S_{i0}}{R_{i0}}\frac{2p_i^Q}{p_i^Q + p_\alpha} - 1\\
&= \frac{R_{i0} - \Delta r_\alpha}{R_{i0}S_{i0} - S_{i0}\Delta r_\alpha}\frac{S_{i0}}{1}\frac{2p_i^Q}{p_i^Q + p_\alpha} - 1\\
&= \frac{2p_i^Q}{p_i^Q + p_\alpha} - 1\\
\end{align}
$$

This is identical to the IL of an $xy = k$ pool between the asset $i$ and LHDX.


# Variablen

Wenn Sie ein neues Jupyter Notebook erstellen, wählen Sie `Python 3.6` als Typ des Notebooks aus.

Innerhalb des Notebooks arbeiten Sie dann mit Python in der Version 3.6. Um zu verstehen, welche Bedeutung Variablen haben, müssen Sie also Variablen in Python 3.6 verstehen.

In Python ist eine Variable ein Kurzname für einen Speicherbereich, in den Daten abgelegt werden können. Auf diese Daten kann dann unter dem Namen der Variablen zugegriffen werden. Dies haben wir prinzipiell bereits in den ersten Übungen durchgeführt. Beispiele:

`d_i = 20 # Innendurchmesser eines Rohres`

Den Name einer Variablen muss eindeutig sein. Er darf nicht mit einer Ziffer beginnen und darf keine Operationszeichen wie `'+', '-', '*', ':', '^'` oder `'#'` enthalten. Soll einer Variablen ein Wert zugewiesen werden, so geschieht das mittels des Gleichheitszeichens, siehe oben.

Das `'#'`-Zeichen leitet einen Kommentar ein. Alles, was nach diesem Zeichen folgt, hat einen rein informativen Charakter aber wird von Python ignoriert.

Sobald eine Variable angelegt ist, kann diese in Berechnungen verwendet werden, z.B.

```python
import math
```

```python
d_i = 20e-3 # Innendurchmesser eines Rohres in m

A_i = math.pi*d_i**2/4 # Lichter Querschnitt in m**2
```

In einer Zelle dürfen mehrere Operationen stehen, siehe oben. Eine Zelle kann eine Ausgabe haben. Das ist dann immer das Ergebnis der Letzten Operation.

In der oben stehenden Zelle steht als letztes eine Wertzuweisung `A_i = ...`, die keine Ausgabe erzeugt.

Um interaktiv arbeiten zu können, sollte man nicht zu lange Berechnungen in einzelnen Zellen durchführen. Statt dessen sollte man nach wenigen sinnvollen Schritten Zwischenergebnisse anzeigen, um den Gang der Arbeit nachvollziehen zu können.

Stellt man einen Fehler fest, so können Werte geändert und die Zelle erneut ausgeführt werden.

Um das Ergebnis der oben durchgeführten Berechnung anzuzeigen, kann man die zuletzt erzeugte Variable aufrufen. Die Zelle sähe dann so aus:

```python
d_i = 20e-3 # Innendurchmesser eines Rohres in m

A_i = math.pi*d_i**2/4 # Lichter Querschnitt in m**2
A_i
```

Variablen haben einen Typ, den man sich durch die Funktion `type(variable)` anzeigen kann, z.B.:

```python
type(A_i)
```

# Aufgabe

Untersuchen Sie, welchen Typ die Variablen

`a=1`

`x=5.0`

`Name = 'Bernd'`

und

`Punkt = (3,4)`

haben.

Untersuchen Sie, welche Auswirkung der Befehl

`2*Name`

mit dem oben definierten Namen hat. Welche Auswirkung hat analog

`2*Punkt`?

Stellen Sie eine Vermutung auf, welchen Typ das Produkt `a*x` und die Summe `a+x` mit den oben festgelegten Werten von `a` und `x` haben und überprüfen Sie diese.

Um die Anwendungen in der Mathematik nicht aus den Augen zu verlieren, bearbeiten Sie die folgende Aufgabe:

# Aufgabe

Berechnen Sie das Gewicht von 250m Kupferrohr CU15$\times$1 in kg. Entnehmen Sie dafür die Dichte $\varrho$ von Kupfer ihrem Tabellenbuch. Die Zusammenhänge sind durch die folgenden Formeln gegeben:

\begin{align}
  A &= \dfrac{\pi\,(d_a^2 - d_i^2)}{4}
  \\[2ex]
  V &= A\, l
  \\[2ex]
  m &= \varrho\, V
\end{align}

```python
import math
```

```python
# Ihre Lösung beginnt hier
```


# Random Operators: examples of random states and channels

```python
import numpy as np
from sympy.combinatorics import Permutation
import forest_benchmarking.random_operators as rand_ops
import numpy.linalg
```

## Complex Ginibre ensemble

```python
rand_ops.ginibre_matrix_complex(2,2)
```

## Haar random unitary

```python
U = rand_ops.haar_rand_unitary(2)
print(np.around(U.dot(np.transpose(np.conj(U))),decimals=15))
print(np.around(U.dot(np.transpose(np.conj(U))),decimals=16))
# only good to 16 decimal places...
```

## Haar random pure state

```python
psi = rand_ops.haar_rand_state(2)
print(psi)
```

## Ginibre State (mixed state with rank K)

```python
# mixed single qubit state
print(np.around(rand_ops.ginibre_state_matrix(2,2),4))
print("----------------------")
# mixed two qubit state
print(np.around(rand_ops.ginibre_state_matrix(4,2),4))
```

```python
# you cant have Rank > Hilbert space Dim
rand_ops.ginibre_state_matrix(2,3)
```

## State from Bures measure

```python
rand_ops.bures_measure_state_matrix(2)
```

## Uniform ensemble of CPTP maps

```python
# random quantum channel on one qubit in Choi form
choi = rand_ops.rand_map_with_BCSZ_dist(2,2)
choi
```

```python
choi.shape
```

## Permutations of operators on tensor product Hilbert spaces

```python
# pick a hilbert space dimension
D = 2
# pick a way you want to permute the operators
perm =[1,2,0]
# Note: the number of elements in the permutation determines
# the number of Hilbert spaces you are considering.
```

^^ here the Identity permutation is P = [0,1,2] which maps (a,b,c) to (a,b,c).
The permutation P = [1,2,0] maps (a,b,c) to (b,c,a).

### Create the basis states in the Hilbert space

```python
basis = list(range(0,D))
states = []
for jdx in basis:
    emptyvec = np.zeros((D,1))
    emptyvec[jdx] =1
    states.append(emptyvec)
```

### Create inital state and answer after applying the permutation [1,2,0]

```python
inital_vector = np.kron(np.kron(states[0],states[0]),states[1]) # before permuting anything
perm_vector = np.kron(np.kron(states[0],states[1]),states[0]) # apply the permutation by hand
```

### create permutation operator

```python
P_120 = rand_ops.permute_tensor_factors(D, perm)
```

### check the permutation operator applied to the intial vector gives the correct answer

```python
answer = np.matmul(P_120,inital_vector)
```

```python
np.matmul(perm_vector.T,answer)
```

```python

```


# Example for calculating the fn-coefficients

In the following, a simple example on how to manually evaluate the fn-coefficients is given.
The ground is hereby defined as a Lambertian-surface and the covering layer is assumed to consist of Rayleigh-particles. Thus, we have: ($R_0$ hereby denotes the diffuse albedo of the surface)

Within the following calculation, the symbols and functions are defined as presented in the Theory-chapter of the documentation.

### Definition of the BRDF of the ground surface:

- $BRDF(\theta, \phi, \theta_{ex},\phi_{ex}) = \frac{R_0}{\pi}$

### Definition of the scattering phase-function of the covering layer:

- $p(\theta, \phi, \theta_{ex},\phi_{ex}) = \frac{3}{16\pi} (1+\cos(\Theta)^2) \quad$ with $\mbox{}\quad$ $\cos(\Theta) = \cos(\theta)\cos(\theta_{ex}) + \sin(\theta)\sin(\theta_{ex})\cos(\phi - \phi_{ex})$

### Manual Calculation of the fn-coefficients:

$INT := \int_0^{2\pi} p(\theta_0, \phi_0, \theta,\phi) * BRDF(\pi-\theta, \phi, \theta_{ex},\phi_{ex}) d\phi $

$\phantom{INT} = \frac{3 R_0}{16 \pi^2} \int_{0}^{2\pi}  (1+[\cos(\theta_0)\cos(\theta) + \sin(\theta_0)\sin(\theta)\cos(\phi_0 - \phi)]^2) d\phi$

$\phantom{INT} = \frac{3 R_0}{16 \pi^2} \int_0^{2\pi} (1+ \mu_0^2 \mu^2 + 2 \mu_0 \mu \sin(\theta_0) \sin(\theta) \cos(\phi_0 - \phi) + (1-\mu_0)^2(1-\mu)^2 \cos(\phi_0 - \phi)^2 d\phi$

where the shorthand-notation $\mu_x = \cos(\theta_x)$ has been introduced.
The above integral can now easily be solved by noticing:

$\int_0^{2\pi} \cos(\phi_0 - \phi)^n d\phi = \left\lbrace \begin{matrix} 2 \pi & \textrm{for } n=0 \\ 0 & \textrm{for } n=1 \\ \pi  & \textrm{for } n=2 \end{matrix} \right.$

Inserting the above solution and using some algebraic manipulations we therefore find:

$INT = \frac{3 R_0}{16\pi} \Big[ (3-\mu_0^2) + (3 \mu_0 -1) \mu^2 \Big] := \sum_{n=0}^2 f_n \, \mu^n $

$\Rightarrow \quad f_0 = \frac{3 R_0}{16\pi}(3-\mu_0^2) \qquad f_1 = 0 \qquad f_2 = \frac{3 R_0}{16\pi}(3 \mu_0 -1) \qquad f_n = 0 \, \forall \, n>2
$

## Evaluation of the fn-coefficients using the RT1-module

```python
# imports
from rt1.rt1 import RT1
from rt1.volume import Rayleigh
from rt1.surface import Isotropic

import sympy as sp
# enable printing sympy equations via latex-equation-rendering
sp.init_printing(use_latex='mathjax')
```

```python
# definition of volume and surface
# (the values of tau and omega have no effect on the calculated fn-coefficients)
V = Rayleigh(tau=0.7, omega=0.3)

R0 = 1.    # set the diffuse reflectance
SRF = Isotropic(NormBRDF=R0)
```

```python
I0 = 1.   # set incident intensity
R = RT1(I0, 0., 0., 0., 0., V=V, SRF=SRF, geometry='mono')
# setting geometry='vvvv' ensures that both incident- and scattering-angles
# are treated symbolically
```

```python
fn = R.fn #calculate fn-coefficients

# in order to see that the calculated coefficients are actually equal, we have
# to apply some trigonometric simplifications:

# apply basic trigonometric simplifications
fn = [sp.trigsimp(i) for i in R.fn]
# replace appearing sin^2 with 1-cos^2
fn = [i.xreplace({sp.sin('theta_0')**2 : 1.-sp.cos('theta_0')**2}) for i in fn]
# convert floats to rationals and factor out
fn = [i.nsimplify().factor() for i in fn]
```

    evaluating fn-coefficients...

```python
# print calculated fn-coefficients
fn
```

$\displaystyle \left[ - \frac{3 \left(\cos^{2}{\left(\theta_{0} \right)} - 3\right)}{16 \pi}, \  0, \  \frac{3 \left(3 \cos^{2}{\left(\theta_{0} \right)} - 1\right)}{16 \pi}\right]$

```python

```


Traçar um esboço do gráfico e obter uma equação da parábola que satisfaça as condições dadas.

25. Vértice: $V(4,-3)$, eixo paralelo ao eixo dos x, passando pelo ponto $P(2,1)$

Se a parábola é paralela ao eixo dos $x$, temos que sua equação é dada por $(y-k)^2 = 2p(x-h)$

Descobrindo o valor de $p$ substituindo $V(4,-3)$ e $P(2,1)$

$(-3-3)^2 = 2p(4-2)$

$(-4)^2 = 2p(2)$

$16 = 4p$

$p$ precisa ser negativo para passar pelo ponto $P(2,1)$

$-4 = p$

Substituindo os pontos na equação da parábola

$(y-(-3))^2 = 2\cdot -4 \cdot (x-4)$

$(y+3)^2 = -8(x-4)$

$y^2 + 6y + 9 = -8x + 32$

$y^2 + 6y + 8x - 23 = 0$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y+3)**2, -8*(x-4)), (x,-20,20), (y,-20,20),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/25.ipynb


```python
import numpy as np
import matplotlib.pyplot as plt

```

Fourier Analysis of a Plucked String
------------------------------------

Let's assume we're studying a plucked string with a shape like this:

\begin{eqnarray}
f(x) & = & 2 A \frac{x}{L}  &  (x<L/2) \\
\\
f(x) & = & 2 A \left(\frac{L-x}{L}\right) & (x >= L/2)
\end{eqnarray}

Let's graph that and see what it looks like:

```python
L=1.0
N=500  # make sure N is even for simpson's rule
A=1.0

def fLeft(x):
    return 2*A*x/L

def fRight(x):
    return 2*A*(L-x)/L

def fa_vec(x):
    """
    vector version
    'where(cond, A, B)', returns A when cond is true and B when cond is false.
    """
    return np.where(x<L/2, fLeft(x), fRight(x))

x=np.linspace(0,L,N)  # define the 'x' array
h=x[1]-x[0]        # get x spacing
y=fa_vec(x)
plt.title("Vertical Displacement of a Plucked String at t=0")
plt.xlabel("x")
plt.ylabel("y")
plt.plot(x,y)
plt.grid()
```

Let's define the basis functions:

\begin{equation}
|n\rangle = b_n(x) = \sqrt{\frac{2}{L}} \sin(n \pi x/L)
\end{equation}

Let's look at a few of those. In python well use the function `basis(x,n)` for $|n\rangle$:

```python
def basis(x, n):
    return np.sqrt(2/L)*np.sin(n*np.pi*x/L)

for n in range(1,5):
    plt.plot(x,basis(x,n),label="n=%d"%n)

plt.legend(loc=3)
```

If we guess that we can express $f(x)$ as a superposition of $b_n(x)$ then we have:

\begin{equation}
f(x) = c_1 b_1(x) + c_2 b_2(x) + c_3 b_3(x) + \cdots = \sum_{n=1}^\infty c_n b_n(x)
\end{equation}

What happens if we multiply $f(x)$ with $b_m(x)$ and then integrate from $x=0$ to $x=L$?

\begin{equation}
\int_0^L f(x) b_m(x) dx = c_1 \int_0^L b_m(x) b_1(x)dx + c_2 \int_0^L b_m(x) b_2(x)dx + c_3 \int_0^L b_m(x) b_3(x)dx + \cdots
\end{equation}

or more compactly in dirac notation:

\begin{equation}
\langle m|f\rangle = c_1 \langle m|1\rangle + c_2 \langle m|2\rangle + c_3 \langle m|3\rangle + \cdots
\end{equation}

or equivalently:

\begin{equation}
\langle m|f\rangle = \sum_{n=1}^\infty c_n \langle m|n\rangle
\end{equation}

Remember that the $b_n(x)$ are *orthonormal* so that means:

$$\langle n|m \rangle = \int_0^L b_n(x) b_m(x)dx = \delta_{nm}$$

where $\delta_{nm}$ is defined as 0 if $n<>m$ and 1 if $n=m$.

So:

$$\sum_{n=1}^\infty c_n \langle m|n\rangle = c_m$$

or in other words:

$$c_m = \langle m|f\rangle = \int_{0}^{L} b_m(x) f(x)  dx $$

Yeah! Let's do that integral for this case. Note that the function $f(x)$ is symmetric about the midpoint, just like $b_m$ when $m$ is odd. When $m$ is even, the integral is zero. SO, for *odd* m we can write:

$$ c_m = \int_{0}^{L} b_m(x) f(x)  dx = 2 \int_{0}^{L/2} b_m(x) f(x)  dx $$

(when $m$ is odd) or:

$$ c_m = 2 \int_0^{L/2}  \sqrt{\frac{2}{L}} \sin(\frac{m\pi x}{L}) 2A \frac{x}{L} dx $$

$$ c_m = \frac{4A}{L} \sqrt{\frac{2}{L}} \int_0^{L/2} x\sin(\frac{m\pi x}{L}) dx $$

$$ c_m = \frac{4A}{L} \sqrt{\frac{2}{L}} \frac{L^2}{\pi^2 m^2} (-1)^{\frac{m-1}{2}}$$

Or simplifying:

$$ c_m = \frac{4A \sqrt{2L}}{\pi^2 m^2} (-1)^{\frac{m-1}{2}}$$

```python
def simpson_array(f, h):
    """

    Use Simpson's Rule to estimate an integral of an array of
    function samples

    f: function samples (already in an array format)
    h: spacing in "x" between sample points

    The array is assumed to have an even number of elements.

    """
    if len(f)%2 != 0:
        raise ValueError("Sorry, f must be an array with an even number of elements.")

    evens =  f[2:-2:2]
    odds = f[1:-1:2]
    return (f[0] + f[-1] + 2*odds.sum() + 4*evens.sum())*h/3.0

def braket(n):
    """
    Evaluate <n|f>
    """
    return simpson_array(basis(x,n)*fa_vec(x),h)

M=20
coefs = [0]
coefs_th = [0]
ys = [[]]
sup = np.zeros(N)
for n in range(1,M):
    coefs.append(braket(n))   # do numerical integral

    if n%2==0:
        coefs_th.append(0.0)
    else:
        coefs_th.append(4*A*np.sqrt(2*L)*(-1)**((n-1)/2.0)/(np.pi**2*n**2))  # compare theory

    ys.append(coefs[n]*basis(x,n))
    sup += ys[n]
    plt.plot(x,sup)

print("%10s\t%10s\t%10s" % ('n', 'coef','coef(theory)'))
print("%10s\t%10s\t%10s" % ('---','-----','------------'))

for n in range(1,M):
    print("%10d\t%10.5f\t%10.5f" % (n, coefs[n],coefs_th[n]))

```

Project 11
============

Pick your own function and compute it's fourier coefficients analytically. Then, check your answer both graphically and numerically using simpson's rule.

```python

```


# Linear Advection with Finite Differences
## CH EN 6355 - Computational Fluid Dynamics
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

We will be solving the linear advection equation
\begin{equation}
\frac{\partial u}{\partial t} = - c \frac{\partial u}{\partial x}
\end{equation}
subject to the initial condition $u(x,0)\equiv u_0(x)$ on the domain $[0,L]$. The exact solution for this equation is given by
\begin{equation}
u(x,t) = u_0(x-ct)
\end{equation}

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
import matplotlib.animation as animation
plt.rcParams['animation.html'] = 'html5' # this is used to display animations in jupyter notebooks
```

Let's start by defining an initial condition - let's take
\begin{equation}
    u_0(x) = \sin(\omega x)
\end{equation}

```python
n = 41
L = 1.0
dx = L/(n-1)
x = np.linspace(0,L,n)
u0 = np.sin(2.0*np.pi*x)
plt.plot(x,u0,'o')
plt.grid()
```

## Forward in Time, Central in Space (FTCS)
Using a forward Euler method for the time derivative and a central discretization for the spatial derivative, the FTCS scheme results in
\begin{equation}
u_i^{n + 1} = u_i^n + c\Delta t\frac{u_{i + 1}^n - u_{i - 1}^n}{2\Delta x}
\end{equation}

```python
# the following implementation does not use ghost cells
dt = 0.001  # s
tend = 2.0 # s
c = 10.0 # m/s - wave speed
cfl = c*dt/2.0/dx

sol = []
sol.append(u0)

t = 0.0
while t < tend:
    un = sol[-1]
    unew = un.copy()
    unew[1:-1] = un[1:-1] - cfl * (un[2:] - un[:-2])
    unew[-1] = un[-1] - cfl*(un[1] - un[-2]) # compute last point on the right using periodicity
    unew[0] = unew[-1] # set periodic boundary on the left
    sol.append(unew)
    t += dt
```

```python
ims = []
fig = plt.figure(figsize=[4,3])
plt.grid()

i = 0
for solution in sol:
    if (i%10==0): # output frequency for frames
        im = plt.plot(x,solution,'k-',animated=True)
        plt.ylim(-1,1)
        ims.append(im)
    i+=1
ani = animation.ArtistAnimation(fig, ims, interval=35, blit=True,
                                repeat_delay=1000)
# ani.save('ftcs.mp4')
ani
```

## Forward in Time, Backward (Upwind) in Space (FTUS)

We have shown in class that the previous FTCS scheme is unconditionally unstable. Here we will implement the upwind scheme in space

```python
tend = 1.0 # s
c = 1.0 # m/s - wave speed
dt = dx/c  # run just at the CFL condition
cfl = c*dt/dx

sol = []
sol.append(u0)

t = 0.0
while t < tend:
    un = sol[-1]
    unew = un.copy()
    unew[1:-1] = un[1:-1] - cfl * (un[1:-1] - un[:-2])
    unew[-1] = un[-1] - cfl*(un[-1] - un[-2]) # compute last point on the right using periodicity
    unew[0] = unew[-1] # set periodic boundary on the left
    sol.append(unew)
    t += dt
```

```python
ims = []
fig = plt.figure(figsize=[5,4],dpi = 150)
plt.grid()

i = 0
for solution in sol:
    if (i%2==0):
        im = plt.plot(x,solution,'k-',animated=True)
        plt.ylim(-1,1)
        ims.append(im)
    i+=1
ani = animation.ArtistAnimation(fig, ims, interval=35, blit=True,
                                repeat_delay=1000)
# ani.save('ftus.mp4')
ani
```

```python
import urllib
import requests
from IPython.core.display import HTML
def css_styling():
    styles = requests.get("https://raw.githubusercontent.com/saadtony/NumericalMethods/master/styles/custom.css")
    return HTML(styles.text)
css_styling()

```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```

1D Linear Advection Equation - FDM.ipynb