# 2021-09-03 Convergence and Fourier analysis

## Last time
* Method of manufactured solutions
* More on boundary conditions and symmetry

## Today

* Investigate convergence with manufactured solutions
* Fourier methods for analyzing finite difference methods
* Discussion and introduce/start boundary conditions activity

```julia
using Plots
using LinearAlgebra

function vander(x, k=nothing)
    if k === nothing
        k = length(x)
    end
    V = ones(length(x), k)
    for j = 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function fdstencil(source, target, k)
    "kth derivative stencil from source to target"
    x = source .- target
    V = vander(x)
    rhs = zero(x)'
    rhs[k+1] = factorial(k)
    rhs / V
end

function poisson(x, spoints, forcing; left=(0, zero), right=(0, zero))
    n = length(x)
    L = zeros(n, n)
    rhs = forcing.(x)
    for i in 2:n-1
        jleft = min(max(1, i-spoints÷2), n-spoints+1)
        js = jleft : jleft + spoints - 1
        L[i, js] = -fdstencil(x[js], x[i], 2)
    end
    L[1,1:spoints] = fdstencil(x[1:spoints], x[1], left[1])
    L[n,n-spoints+1:n] = fdstencil(x[n-spoints+1:n], x[n], right[1])
    rhs[1] = left[2](x[1])
    rhs[n] = right[2](x[n])
    L, rhs
end
```

# Method of manufactured solutions

## Problem: analytic solutions to PDEs are hard to find

Let's choose a smooth function with rich derivatives,
$$ u(x) = \tanh(x) . $$
Then $$ u'(x) = \cosh^{-2}(x) $$ and $$ u''(x) = -2 \tanh(x) \cosh^{-2}(x) . $$

* This works for nonlinear too.

```julia
x = LinRange(-2, 2, 5)
L, rhs = poisson(x, 3,
    x -> 2 * tanh(x) / cosh(x)^2,
    left=(0, tanh),
    right=(1, x -> cosh(x)^-2))
u = L \ rhs
plot(x, u, marker=:circle, legend=:topleft)
plot!(tanh)
```

# Compare MMS error with different number of stencil points (Neumann)

```julia
x = LinRange(-2, 2, 5)
L, rhs = poisson(x, 3,
    x -> 2 * tanh(x) / cosh(x)^2,
    left=(0, tanh),
    right=(1, x -> cosh(x)^-2))
u = L \ rhs
plot(x, u, marker=:circle, legend=:topleft)
plot!(tanh)
```

```julia
x = LinRange(-2, 2, 5)
L, rhs = poisson(x, 5,
    x -> 2 * tanh(x) / cosh(x)^2,
    left=(0, tanh),
    right=(1, x -> cosh(x)^-2))
u = L \ rhs
plot(x, u, marker=:circle, legend=:topleft)
plot!(tanh)
```

# Compare MMS error with different number of stencil points (Dirichlet)

```julia
x = LinRange(-2, 2, 5)
L, rhs = poisson(x, 3,
    x -> 2 * tanh(x) / cosh(x)^2,
    left=(0, tanh),
    right=(0, tanh))
u = L \ rhs
plot(x, u, marker=:circle, legend=:topleft)
plot!(tanh)
```

```julia
x = LinRange(-2, 2, 5)
L, rhs = poisson(x, 5,
    x -> 2 * tanh(x) / cosh(x)^2,
    left=(0, tanh),
    right=(0, tanh))
u = L \ rhs
plot(x, u, marker=:circle, legend=:topleft)
plot!(tanh)
```

# Convergence rate

```julia
ns = [5, 6, 8, 10, 15, 30, 100]
hs = 1 ./ ns
function poisson_error(n, spoints)
    x = LinRange(-2, 2, n)
    L, rhs = poisson(x, spoints, x -> 2 * tanh(x) / cosh(x)^2,
        left = (0, tanh),
        right = (1, x -> cosh(x)^-2))
    u = L \ rhs
    norm(u - tanh.(x), Inf)
end
```

    poisson_error (generic function with 1 method)

```julia
plot(hs, [poisson_error(n, 5) for n in ns], marker=:circle)
plot!(hs, [poisson_error(n, 3) for n in ns], marker=:square)
plot!([h -> h^2, h -> h^4], xscale=:log10, yscale=:log10, legend=:topleft)
```

## Discuss

* asymptotic vs preasmptotic regime
* errors on coarse grids (here especially) may be dominated by the boundary conditions
* leading error coeffficient in Taylor expansion is bigger near endpoints
* model can be considered **verified** when it converges at the design order of accuracy for one "rich" manufactured solution
  * manufactured solution needs to exercise every aspect of the model
  * all types/cases of boundary conditions
  * a reliable tool only for smooth problems
  * for more: [Roache, Code Verification by the Method of Manufactured Solutions](https://doi.org/10.1115/1.1436090) (7 pages)

# Symmetry in boundary conditions: Dirichlet

We have implemented Dirichlet conditions by modifying the first row of the matrix,
$$ \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ \\ & & A_{1:,:} & & \\ \\ \end{bmatrix} \begin{bmatrix} u_0 \\ \\ u_{1:} \\ \\ \end{bmatrix} = \begin{bmatrix} a \\ \\ f_{1:} \\ \\ \end{bmatrix} . $$

* This matrix is not symmetric even if $A$ is.
* We can eliminate $u_0$ and create a reduced system for $u_{1:}$.
* Generalize: consider a $2\times 2$ block system
$$ \begin{bmatrix} I & 0 \\ A_{10} & A_{11} \end{bmatrix} \begin{bmatrix} u_0 \\ u_1 \end{bmatrix} = \begin{bmatrix} f_0 \\ f_1 \end{bmatrix} .$$

We can rearrange as
$$ A_{11} u_1 = f_1 - A_{10} f_0, $$
which is symmetric if $A_{11}$ is.
* This is called "lifting" and is often done implicitly in the mathematics literature.  It is convenient for linear solvers and eigenvalue solvers, but inconvenient for IO and postprocessing, as well as some nonlinear problems.
* Convenient alternative: write
$$ \begin{bmatrix} I & 0 \\ 0 & A_{11} \end{bmatrix} \begin{bmatrix} u_0 \\ u_1 \end{bmatrix} = \begin{bmatrix} f_0 \\ f_1 - A_{10} f_0 \end{bmatrix}, $$
which is symmetric and decouples the degrees of freedom associated with the boundary. This method applies cleanly to nonlinear problems.
* Optionally scale the identity by some scalar related to the norm of $A_{11}$.

# Symmetry in boundary conditions: Neumann

Consider FD discretization of the Neumann boundary condition
$$ \frac{du}{dx}(1) = b . $$
1. Use a one-sided difference formula as in
$$ \frac{u_n - u_{n-1}}{h} = b . $$
  * an extra discretization choice
  * may reduce order of accuracy compared to interior discretization, lose symmetry.
2. Temporarily introduce a ghost value $u_{n+1} = u(x_{n+1} = 1 + h)$ (possibly more) and define it to be a reflection of the values from inside the domain.  In the case $b=0$, this reflection is $u_{n+i} = u_{n-i}$.  More generally,
$$ u_{n+i} = u_{n-i} + 2b(x_n - x_{n-i}) . $$

After this definition of ghost values, we apply the interior discretization at the boundary. For our reference equation, we would write

$$ \frac{-u_{n-1} + 2 u_n - u_{n+1}}{h^2} = f(x_n) $$

which simplifies to $$ \frac{u_n - u_{n-1}}{h^2} = f(x_n)/2 + b/h $$
after dividing by 2 and moving the boundary term to the right hand side.

# Fourier analysis of stencils

Consider the plane waves $\phi(x, \theta) = e^{i\theta x}$.

Sample $\phi$ on a discrete grid $x = \mathbb Z$ and apply the stencil
\begin{align}
S \phi(x, \theta) &= s_{-1} \phi(x-1, \theta) + s_{0} \phi(x, \theta) + s_1 \phi(x+1,\theta) \\
&= \Big( s_{-1} e^{-i\theta} + s_0 + s_{1} e^{i\theta} \Big) \phi(x, \theta)
\end{align}
With $S = \begin{bmatrix} -1 & 2 & -1 \end{bmatrix}$, we get
$$S \phi(x, \theta) = \underbrace{(2 - 2 \cos\theta)}_{\hat S(\theta)} \phi(x, \theta)$$
We call $\hat S(\theta)$ the *symbol* of the operator.
What is the symbol of the continuous second derivative?

```julia
plot(theta -> 2 - 2*cos(theta), xlims=(-pi, pi))
plot!(theta -> theta^2)
```

# Numerically computing symbols

```julia
function symbol(S, theta)
    if length(S) % 2 != 1
        error("Length of stencil must be odd")
    end
    w = length(S) ÷ 2
    phi = exp.(1im * (-w:w) * theta')
    vec(S * phi) # not! (S * phi)'
end

symbol([-.5 0 .5], LinRange(-pi, pi, 10))
```

    10-element Vector{ComplexF64}:
     0.0 - 1.2246467991473532e-16im
     0.0 - 0.6427876096865395im
     0.0 - 0.9848077530122081im
     0.0 - 0.8660254037844388im
     0.0 - 0.34202014332566877im
     0.0 + 0.34202014332566877im
     0.0 + 0.8660254037844385im
     0.0 + 0.9848077530122081im
     0.0 + 0.6427876096865395im
     0.0 + 1.2246467991473532e-16im

```julia
function plot_symbol(S, deriv, n_theta=30)
    theta = LinRange(-pi, pi, n_theta)
    sym = symbol(S, theta)
    rsym = real.((-1im)^deriv * sym)
    fig = plot(theta, rsym, marker=:circle, label="stencil")
    plot!(fig, th -> th^deriv, label="exact")
    fig
end

plot_symbol([-.5 0 .5], 1)
#plot_symbol([1 -2 1], 2)
```

# Stencils of high-order operators

```julia
x = -5:5
plot_symbol(fdstencil(x, 0, 1), 1)
```

```julia
x = -5:5
plot_symbol(fdstencil(x, 0, 2), 2)
```

# Outlook on Fourier methods

* the Fourier modes $e^{i\theta x}$ and their multi-dimensional extensions are eigenvectors of **all** stencil-type operations
* "high frequencies" $[-\pi, \pi) \setminus [-\pi/2, \pi/2)$ are generally poorly resolved so we need to use a grid fine enough that important features are at low frequencies $[-\pi/2, \pi/2)$
* same technique can be used to study the inverse (and approximations thereof), as with multigrid and multilevel domain decomposition methods (later in the course)
* these methods can also be framed within the theory of (block) Toeplitz matrices


# Numbers

The `numpy` array is the foundation of essentially all numerical computing in Python, so it is important to understand the array and how to use it well.

## Learning objectives

1. Attributes of an array
2. How to create vectors, matrices, tensors
3. How to index and slice arrays
4. Generating random arrays and sampling
5. Universal functions, vectorization and matrix multiplication
6. Array axes and marginal calculations
7. Broadcasting
8. Masking
9. Combining and splitting arrays
10. Vectorizing loops

```python
import numpy as np
```

## The `ndarray`: Vectors, matrices and tenosrs

dtype, shape, strides

### Vector

```python
x = np.array([1,2,3])
x
```

    array([1, 2, 3])

```python
type(x)
```

    numpy.ndarray

```python
x.dtype
```

    dtype('int64')

```python
x.shape
```

    (3,)

```python
x.strides
```

    (8,)

### Matrix

```python
x = np.array([[1,2,3], [4,5,6]], dtype=np.int32)
x
```

    array([[1, 2, 3],
           [4, 5, 6]], dtype=int32)

```python
x.dtype
```

    dtype('int32')

```python
x.shape
```

    (2, 3)

```python
x.strides
```

    (12, 4)

### Tensor

```python
x = np.arange(24).reshape((2,3,4))
```

```python
x
```

    array([[[ 0,  1,  2,  3],
            [ 4,  5,  6,  7],
            [ 8,  9, 10, 11]],

           [[12, 13, 14, 15],
            [16, 17, 18, 19],
            [20, 21, 22, 23]]])

## Creating `ndarray`s

### From a file

```python
%%file numbers.txt
a,b,c # can also skip headers
1,2,3
4,5,6
```

    Writing numbers.txt

```python
np.loadtxt('numbers.txt', dtype='int', delimiter=',',
           skiprows=1, comments='#')
```

    array([[1, 2, 3],
           [4, 5, 6]])

### From Python lists or tuples

```python
np.array([
    [1,2,3],
    [4,5,6]
])
```

    array([[1, 2, 3],
           [4, 5, 6]])

### From ranges

arange, linspace, logspace

```python
np.arange(1, 7).reshape((2,3))
```

    array([[1, 2, 3],
           [4, 5, 6]])

```python
np.linspace(1, 10, 4)
```

    array([ 1.,  4.,  7., 10.])

```python
np.logspace(0, 4, 5, dtype='int')
```

    array([    1,    10,   100,  1000, 10000])

### From a function

`fromfunciton`

```python
np.fromfunction(lambda i, j: i*3 + j + 1, (2,3))
```

    array([[1., 2., 3.],
           [4., 5., 6.]])

```python
np.fromfunction(lambda i, j: (i-2)**2 + (j-2)**2, (5,5), dtype='int')
```

    array([[8, 5, 4, 5, 8],
           [5, 2, 1, 2, 5],
           [4, 1, 0, 1, 4],
           [5, 2, 1, 2, 5],
           [8, 5, 4, 5, 8]])

#### How to visualize `fromfunction`

```python
j = np.repeat([np.arange(5)], 5, axis=0)
i = j.T
```

```python
i
```

    array([[0, 0, 0, 0, 0],
           [1, 1, 1, 1, 1],
           [2, 2, 2, 2, 2],
           [3, 3, 3, 3, 3],
           [4, 4, 4, 4, 4]])

```python
j
```

    array([[0, 1, 2, 3, 4],
           [0, 1, 2, 3, 4],
           [0, 1, 2, 3, 4],
           [0, 1, 2, 3, 4],
           [0, 1, 2, 3, 4]])

```python
(i-2)**2 + (j-2)**2
```

    array([[8, 5, 4, 5, 8],
           [5, 2, 1, 2, 5],
           [4, 1, 0, 1, 4],
           [5, 2, 1, 2, 5],
           [8, 5, 4, 5, 8]])

#### Using element-wise functions in `fromfunction`

```python
np.fromfunction(lambda i, j: np.where(i==j,0, -1), (5,5))
```

    array([[ 0, -1, -1, -1, -1],
           [-1,  0, -1, -1, -1],
           [-1, -1,  0, -1, -1],
           [-1, -1, -1,  0, -1],
           [-1, -1, -1, -1,  0]])

```python
np.fromfunction(lambda i, j: np.where(i<j, 1, np.where(i==j,0, -1)), (5,5))
```

    array([[ 0,  1,  1,  1,  1],
           [-1,  0,  1,  1,  1],
           [-1, -1,  0,  1,  1],
           [-1, -1, -1,  0,  1],
           [-1, -1, -1, -1,  0]])

```python
np.fromfunction(lambda i, j: np.minimum(i,j), (5,5), dtype='int')
```

    array([[0, 0, 0, 0, 0],
           [0, 1, 1, 1, 1],
           [0, 1, 2, 2, 2],
           [0, 1, 2, 3, 3],
           [0, 1, 2, 3, 4]])

```python
np.fromfunction(lambda i, j: np.maximum(i,j), (5,5), dtype='int')
```

    array([[0, 1, 2, 3, 4],
           [1, 1, 2, 3, 4],
           [2, 2, 2, 3, 4],
           [3, 3, 3, 3, 4],
           [4, 4, 4, 4, 4]])

### From special constructors

zeros, ones, eye, diag

```python
np.zeros((2,3))
```

    array([[0., 0., 0.],
           [0., 0., 0.]])

```python
np.ones((2,3))
```

    array([[1., 1., 1.],
           [1., 1., 1.]])

```python
np.eye(3)
```

    array([[1., 0., 0.],
           [0., 1., 0.],
           [0., 0., 1.]])

```python
np.eye(3, 4)
```

    array([[1., 0., 0., 0.],
           [0., 1., 0., 0.],
           [0., 0., 1., 0.]])

```python
np.eye(4, k=-1)
```

    array([[0., 0., 0., 0.],
           [1., 0., 0., 0.],
           [0., 1., 0., 0.],
           [0., 0., 1., 0.]])

```python
np.diag([1,2,3,4])
```

    array([[1, 0, 0, 0],
           [0, 2, 0, 0],
           [0, 0, 3, 0],
           [0, 0, 0, 4]])

```python
np.diag([1,2,3,4], k=1)
```

    array([[0, 1, 0, 0, 0],
           [0, 0, 2, 0, 0],
           [0, 0, 0, 3, 0],
           [0, 0, 0, 0, 4],
           [0, 0, 0, 0, 0]])

### From random variables

#### Convenience functions

rand, randn

```python
np.random.rand(2,3)
```

    array([[0.96634527, 0.19003724, 0.64836211],
           [0.16154332, 0.37613518, 0.56387698]])

```python
np.random.randn(2,3)
```

    array([[-0.68305569,  1.02565577,  1.58315507],
           [ 0.40577645,  0.35897126, -1.26921157]])

#### Distributions

uniform, normal, randint, poisson, multinomial, multivariate_ normal

```python
np.random.uniform(0, 1, (2,3))
```

    array([[0.57022181, 0.27519768, 0.61746728],
           [0.46176807, 0.07558838, 0.49391296]])

```python
np.random.normal(0, 1, (2,3))
```

    array([[-0.86289848,  0.22753939, -0.84960146],
           [-1.28280192,  0.538776  , -0.6454793 ]])

```python
np.random.randint(0, 10, (4,5))
```

    array([[5, 1, 6, 1, 1],
           [3, 2, 4, 2, 3],
           [1, 4, 1, 8, 7],
           [6, 2, 6, 2, 2]])

```python
np.random.poisson(10, (4,5))
```

    array([[ 5, 12, 11, 10, 15],
           [15,  7,  9, 11, 13],
           [13,  6, 11,  7, 15],
           [12,  9, 17, 10,  9]])

```python
np.random.multinomial(n=5, pvals=np.ones(5)/5, size=8)
```

    array([[0, 1, 2, 1, 1],
           [1, 1, 0, 3, 0],
           [1, 1, 1, 1, 1],
           [1, 2, 0, 0, 2],
           [2, 0, 0, 2, 1],
           [3, 0, 1, 1, 0],
           [1, 0, 1, 2, 1],
           [2, 1, 2, 0, 0]])

```python
np.random.multivariate_normal(mean=[10,20,30], cov=np.eye(3), size=4)
```

    array([[10.17827907, 21.16944403, 29.31747505],
           [ 9.93282935, 18.76152496, 31.14436234],
           [ 9.45692559, 19.1736249 , 32.1318146 ],
           [ 9.63006312, 20.2535023 , 29.52264229]])

### Sampling using `choice`

Works much like the R `sample` function.

```python
x = np.random.permutation(list('ABCDEF'))
```

```python
x
```

    array(['E', 'F', 'A', 'B', 'D', 'C'], dtype='<U1')

```python
np.random.choice(x, 3)
```

    array(['E', 'F', 'E'], dtype='<U1')

```python
np.random.choice(x, 10)
```

    array(['A', 'B', 'C', 'F', 'D', 'E', 'E', 'D', 'F', 'D'], dtype='<U1')

```python
try:
    np.random.choice(x, 10, replace=False)
except ValueError as e:
    print(e)
```

    Cannot take a larger sample than population when 'replace=False'

## Indexing

```python
x = np.arange(20).reshape((4,5))
x
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19]])

### Extracing a scalar

```python
x[1,1]
```

    6

### Extracting a vector

```python
x[1]
```

    array([5, 6, 7, 8, 9])

### Using slices

```python
x[1,:]
```

    array([5, 6, 7, 8, 9])

```python
x[:,1]
```

    array([ 1,  6, 11, 16])

```python
x[1:3,1:3]
```

    array([[ 6,  7],
           [11, 12]])

### Using slices with strides

```python
x[::2,::2]
```

    array([[ 0,  2,  4],
           [10, 12, 14]])

### Extrcting blocks with arbitrary row and column lists (fancy indexing)

`np.ix_`

```python
x[:, [0,3]]
```

    array([[ 0,  3],
           [ 5,  8],
           [10, 13],
           [15, 18]])

Warning: Fancy indexing can only be used for 1 dimension at a time.

In the example below, `numpy` treats the arguments as *paired* coordinates, and returns the values at (0,0) and (2,3).

```python
x[[0,2],[0,3]]
```

    array([ 0, 13])

Use the helper `np.ix_` to extract arbitrary blocks.

```python
x[np.ix_([0,2], [0,3])]
```

    array([[ 0,  3],
           [10, 13]])

### A slice is a view, not a copy

**Warning**

```python
b = a[:]
```

makes a copy if `a` is a list but not if `a` is a numpy array

```python
a1 = list(range(3))
a2 = np.arange(3)
```

```python
b = a1[:]
b[1] = 9
a1
```

    [0, 1, 2]

```python
b = a2[:]
b[1] = 9
a2
```

    array([0, 9, 2])

```python
x
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19]])

```python
y = x[1:-1, 1:-1]
y
```

    array([[ 6,  7,  8],
           [11, 12, 13]])

```python
y *= 10
```

```python
y
```

    array([[ 60,  70,  80],
           [110, 120, 130]])

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

Use the copy method to convert a view to a copy

```python
z = x[1:-1, 1:-1].copy()
```

```python
z
```

    array([[ 60,  70,  80],
           [110, 120, 130]])

```python
z[:] = 0
```

```python
z
```

    array([[0, 0, 0],
           [0, 0, 0]])

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

### Boolean indexing

```python
x[x % 2 == 0]
```

    array([  0,   2,   4,  60,  70,  80,  10, 110, 120, 130,  14,  16,  18])

```python
x [x > 3]
```

    array([  4,   5,  60,  70,  80,   9,  10, 110, 120, 130,  14,  15,  16,
            17,  18,  19])

### Functions that return indexes

```python
idx = np.nonzero(x)
idx
```

    (array([0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3]),
     array([1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4]))

```python
x[idx]
```

    array([  1,   2,   3,   4,   5,  60,  70,  80,   9,  10, 110, 120, 130,
            14,  15,  16,  17,  18,  19])

```python
idx = np.where(x > 3)
idx
```

    (array([0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3]),
     array([4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4]))

```python
x[idx]
```

    array([  4,   5,  60,  70,  80,   9,  10, 110, 120, 130,  14,  15,  16,
            17,  18,  19])

## Universal functions

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

Operations

```python
x + x
```

    array([[  0,   2,   4,   6,   8],
           [ 10, 120, 140, 160,  18],
           [ 20, 220, 240, 260,  28],
           [ 30,  32,  34,  36,  38]])

Element-wise functions

```python
np.log1p(x)
```

    array([[0.        , 0.69314718, 1.09861229, 1.38629436, 1.60943791],
           [1.79175947, 4.11087386, 4.26267988, 4.39444915, 2.30258509],
           [2.39789527, 4.7095302 , 4.79579055, 4.87519732, 2.7080502 ],
           [2.77258872, 2.83321334, 2.89037176, 2.94443898, 2.99573227]])

```python
x.clip(10, 100)
```

    array([[ 10,  10,  10,  10,  10],
           [ 10,  60,  70,  80,  10],
           [ 10, 100, 100, 100,  14],
           [ 15,  16,  17,  18,  19]])

Scans

```python
np.cumsum(x, axis=1)
```

    array([[  0,   1,   3,   6,  10],
           [  5,  65, 135, 215, 224],
           [ 10, 120, 240, 370, 384],
           [ 15,  31,  48,  66,  85]])

Reductions

```python
np.sum(x)
```

    703

```python
x.prod()
```

    0

## Margins and the `axis` argument

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

The 0th axis has 4 items, the 1st axis has 5 items.

```python
x.shape
```

    (4, 5)

```python
x.mean()
```

    35.15

### Marginalizing out the 0th axis = column summaries

```python
x.mean(axis=0)
```

    array([ 7.5 , 46.75, 52.25, 57.75, 11.5 ])

### Marginalizing out the 1st axis = row summaries

```python
x.mean(axis=1)
```

    array([ 2. , 44.8, 76.8, 17. ])

Note marginalizing out the last axis is a common default.

```python
x.mean(axis=-1)
```

    array([ 2. , 44.8, 76.8, 17. ])

### Marginalization works for higher dimensions in the same way

```python
x = np.random.random((2,3,4))
x
```

    array([[[0.53438657, 0.9188977 , 0.63964362, 0.98093193],
            [0.85858239, 0.15679935, 0.20372358, 0.60655892],
            [0.65779681, 0.84197462, 0.85369405, 0.96133565]],

           [[0.95375215, 0.95867659, 0.74715273, 0.4698344 ],
            [0.98018627, 0.07420097, 0.5479176 , 0.14725169],
            [0.05951145, 0.86547273, 0.38129592, 0.96394799]]])

```python
x.shape
```

    (2, 3, 4)

```python
x.mean(axis=0).shape
```

    (3, 4)

```python
x.mean(axis=1).shape
```

    (2, 4)

```python
x.mean(axis=2).shape
```

    (2, 3)

```python
x.mean(axis=(0,1)).shape
```

    (4,)

```python
x.mean(axis=(0,2)).shape
```

    (3,)

```python
x.mean(axis=(1,2)).shape
```

    (2,)

## Broadcasting

Broadcasting is what happens when `numpy` tries to perform binary operations on two arrays with different shapes. In general, shapes are *promoted* to make the arrays compatible using the following rule

- For each axis from highest to lowest
    - If both dimensions are the same, do nothing
    - If one of the dimensions is 1 or None and the other is $k$, promote to $k$
    - Otherwise print error message

```python
x = np.zeros((3,2))
x.shape
```

    (3, 2)

```python
x
```

    array([[0., 0.],
           [0., 0.],
           [0., 0.]])

Shapes are compatible

```python
y = np.ones(2)
y.shape
```

    (2,)

```python
x + y
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

Shapes are compatible

```python
y = np.ones((1,2))
y.shape
```

    (1, 2)

```python
x + y
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

Shapes are incompatible but can be made compaible by adding empty dimension

```python
y = np.ones(3)
y.shape
```

    (3,)

```python
try:
    x + y
except ValueError as e:
    print(e)
```

    operands could not be broadcast together with shapes (3,2) (3,)

```python
y[:, None].shape
```

    (3, 1)

```python
x + y[:, None]
```

    array([[1., 1.],
           [1., 1.],
           [1., 1.]])

Shapes are incompatible

```python
y = np.ones((2,2))
y.shape
```

    (2, 2)

```python
try:
    x + y
except ValueError as e:
    print(e)
```

    operands could not be broadcast together with shapes (3,2) (2,2)

### More examples of broadcasting

```python
x1 = np.arange(12)
```

```python
x1
```

    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])

```python
x1 * 10
```

    array([  0,  10,  20,  30,  40,  50,  60,  70,  80,  90, 100, 110])

```python
x2 = np.random.randint(0,10,(3,4))
```

```python
x2
```

    array([[1, 1, 0, 6],
           [6, 9, 3, 7],
           [5, 4, 7, 5]])

```python
x2 * 10
```

    array([[10, 10,  0, 60],
           [60, 90, 30, 70],
           [50, 40, 70, 50]])

```python
x2.shape
```

    (3, 4)

### Column-wise broadcasting

```python
mu = np.mean(x2, axis=0)
mu.shape
```

    (4,)

```python
x2 - mu
```

```python
(x2 - mu).mean(axis=0)
```

### Row wise broadcasting

```python
mu = np.mean(x2, axis=1)
mu.shape
```

```python
try:
    x2 - mu
except ValueError as e:
    print(e)
```

### We can add a "dummy" axis using None or `np.newaxis`

```python
mu[:, None].shape
```

```python
x2 - mu[:, None]
```

```python
x2 - mu[:, np.newaxis]
```

```python
np.mean(x2 - mu[:, None], axis=1)
```

#### Reshaping works too

```python
x2 - mu.reshape((-1,1))
```

#### Exercise in broadcasting

Creating a 12 by 12 multiplication table

```python
x = np.arange(1, 13)
x[:,None] * x[None,:]
```

Scaling to have zero mean and unit standard devation for each feature.

```python
x = np.random.normal(10, 5,(3,4))
x
```

Scaling column-wise

```python
(x - x.mean(axis=0))/x.std(axis=0)
```

Scaling row-wise

```python
(x - x.mean(axis=1)[:, None])/x.std(axis=1)[:, None]
```

## Masking

- [Ref](https://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html)

```python
import numpy.ma as ma
```

```python
x = np.arange(20).reshape(4,5)
```

```python
x
```

```python
mask = x % 2 == 0
mask
```

- Note that values that are True in the mask are not used in the array
- Values that are False are *not* masked and so remain
- So the above mask keeps only the *odd* numbers in the array `x`

```python
m = ma.masked_array(x, mask)
```

```python
m
```

```python
m.data
```

```python
m.mask
```

```python
m.sum(axis=0).data
```

```python
m.sum(axis=1).data
```

### Often used with missing value sentinels

```python
import warnings

with warnings.catch_warnings():
    warnings.simplefilter('ignore', RuntimeWarning)
    x1 = x / mask
```

```python
x1
```

```python
x1.sum()
```

```python
x2 = ma.masked_invalid(x1)
x2
```

```python
x2.data
```

```python
x2.mask
```

```python
x2.sum()
```

```python
x2.filled(0)
```

## Combining `ndarray`s

```python
x1 = np.zeros((3,4))
x2 = np.ones((3,5))
x3 = np.eye(4)
```

```python
x1
```

```python
x2
```

```python
x3
```

### Binding rows when number of columns is the same

```python
np.r_[x1, x3]
```

### Binding columns when number of rows is the same

```python
np.c_[x1, x2]
```

### You can combine more than 2 at a time

```python
np.c_[x1, x2, x1]
```

### Stacking

```python
np.vstack([x1, x3])
```

```python
np.hstack([x1, x2])
```

```python
np.dstack([x2, 2*x2, 3*x2])
```

### Generic stack with axis argument

```python
np.stack([x2, 2*x2, 3*x2], axis=0)
```

```python
np.stack([x2, 2*x2, 3*x2], axis=1)
```

```python
np.stack([x2, 2*x2, 3*x2], axis=2)
```

### Repetition and tiling

#### For a vector

```python
x = np.array([1,2,3])
```

```python
np.repeat(x, 3)
```

```python
np.tile(x, 3)
```

#### For a matrix

```python
x = np.arange(6).reshape((2,3))
x
```

```python
np.repeat(x, 3)
```

```python
np.repeat(x, 3, axis=0)
```

```python
np.repeat(x, 3, axis=1)
```

```python
np.tile(x, (3,2))
```

## Splitting `ndarray`s

```python
x = np.arange(32).reshape((4,8))
```

```python
x
```

```python
np.split(x, 4)
```

```python
np.split(x, 4, axis=1)
```

## Saving and loading arrays

```python
x = np.arange(16).reshape(4,4)
y = np.arange(20).reshape(-1,4)
```

```python
np.save('x.npy', x)
```

```python
np.load('x.npy')
```

```python
np.savez('xy.npz', x=x, y=y)
```

```python
arr = np.load('xy.npz')
```

```python
arr['x']
```

```python
arr['y']
```

```python
import os
```

```python
os.remove('x.npy')
```

```python
os.remove('xy.npz')
```

## Vectorization

### Example 1

The operators and functions (ufuncs) in Python are vectorized, and will work element-wise over all entries in an `ndarray`.

```python
xs = np.zeros(10, dtype='int')
for i in range(10):
    xs[i] = i**2
xs
```

```python
xs = np.arange(10)**2
xs
```

Using ufuncs

```python
np.sqrt(xs)
```

```python
np.log1p(xs)
```

### Example 2

Scalar product.

```python
n = 10

xs = np.random.rand(n)
ys = np.random.rand(n)

s = 0
for i in range(n):
    s += xs[i] * ys[i]
s
```

```python
np.dot(xs, ys)
```

```python
xs @ ys
```

### Example 3

\begin{align}
y_0 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
y_1 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
y_2 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
\end{align}

```python
m = 3
n = 2

alpha = np.random.rand(1)
betas = np.random.rand(n,1)
xs = np.random.rand(m,n)
```

```python
alpha
```

```python
betas
```

```python
xs
```

### Using loops

```python
ys = np.zeros((m,1))
for i in range(m):
    ys[i] = alpha
    for j in range(n):
        ys[i] += betas[j] * xs[i,j]
ys
```

### Removing inner loop

```python
ys = np.zeros((m,1))
for i in range(m):
    ys[i] = alpha + xs[i,:].T @ betas
ys
```

### Removing all loops

```python
ys = alpha + xs @ betas
ys
```

### Alternative approach

The calculaiton with explicit intercepts and coefficients is common in deep learning, where $\alpha$ is called the bias ($b$) and $\beta$ are called the weights ($w$), and each equation is $y[i] = b + w[i]*x[i]$.

It is common in statisiics to use an augmented matrix in which the first column is all ones, so that all that is needed is a single matrix multiplicaiotn.

```python
X = np.c_[np.ones(m), xs]
X
```

```python
alpha
```

```python
betas
```

```python
betas_ = np.concatenate([[alpha], betas])
betas_
```

```python
ys = X @ betas_
ys
```

### Simulating diffusion

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
w = 100
h = 100
x = np.zeros((w+2,h+2), dtype='float')
x[(w//2-1):(w//2+2), (h//2-1):(h//2+2)] = 1

wts = np.ones(5)/5

for i in range(41):
    if i % 10 == 0:
        plt.figure()
        plt.imshow(x[1:-1, 1:-1], interpolation='nearest')

    center = x[1:-1, 1:-1]
    left = x[:-2, 1:-1]
    right = x[2:, 1:-1]
    bottom = x[1:-1, :-2]
    top = x[1:-1, 2:]
    nbrs = np.dstack([center, left, right, bottom, top])
    x = np.sum(wts * nbrs, axis=-1)
```

```python

```


```python
# coding: utf-8
from __future__ import print_function
from sympy import *
from sympy.abc import psi,phi,eta,delta,zeta

init_printing()
f,ck,c = symbols('f ck c', real=True)
sigma, kappa = symbols(r'\sigma \kappa', real=True)
kx, ky, kh =  symbols('k_x k_y k_h', real=True)
u,v =  symbols('u v')

def subsigma(M):
    M = M.subs(f**2 + ck**2,  sigma**2)
    M = M.subs(f**2 + c**2*kappa**2, sigma**2)
    return M

def simplifysigma(M):
    M = M.subs(sigma, sqrt(f**2 + ck**2) )
    M = simplify(M)
    return subsigma(M)

def subkappa(M):
    M = M.subs(kx**2 + ky**2,  kappa**2)
    M = M.subs(kh*kh.conjugate(), kappa**2)
    M = M.subs(kx*kx.conjugate() + ky*ky.conjugate(), kappa**2)
    return M

def simplifykappa(M):
    M = M.subs(kappa, sqrt(kx**2 + ky**2))
    M = M.subs(kh, sqrt(kx**2 + ky**2))
    M = simplify(M)
    return subkappa(M)

def subck(M):
    return M.subs(ck,c*kappa)
```

```python
A = Matrix(( [0, 1j*f,0],[-1j*f, 0, -1j*ck],[0,1j*ck,0]))
X = A.eigenvects()
eigenvals = list()
eigenvects = list()
for i in xrange(3):
    eigval, multi, eigvect =  X[i]
    eigenvals.append(eigval)
    eigenvects.append(eigvect[0])
```

### Eigenvectors and eigenvalues

```python
eigenvals
```

### Normal mode decomposition

```python
XX = A.diagonalize()[0]
XX = subsigma(XX)
sig = sqrt(f**2 + ck**2)

NORM = eye(3)
NORM[0,0] = f/sig
NORM[1,1] = ck/(sig*sqrt(2))
NORM[2,2] = ck/(sig*sqrt(2))

# Eigen vector matrix - normalization
XN = subsigma(XX*NORM)
XNCT = XN.conjugate().transpose()
```

```python
XN
```

```python
"""
Transformation matrices to normal mode
W1 --[T21]--> W2 --[T32]--> W3 --[TN3]---> N
"""
W1 = Matrix( [u,v,eta] )
T21 = Matrix(( [-1j*ky, 1j*kx, 0],[1j*kx,1j*ky,0],[0,0,ck] ))
W2 = T21 * W1
T32 = Matrix(( [1, 0, -f/ck],[f,0,ck],[0,1,0] ))
W3 = T32 * W2
TN3 = Matrix([ [1,0,0],[0,1,-1j*sigma],[0,1,1j*sigma]]) / (sqrt(2) * sigma)
N = simplify(subck((TN3 * W3).subs(sigma, sqrt(f**2 + ck**2))))
```

```python
W1, W2, W3, N
```

```python
(N * N.transpose())
```

```python

```

Jupyter/q,a,d versus q,a+,a- orthogonality.ipynb


# Bayesian Logistic Regression

Contact: mihaelacr@google.com

Aim of this colab:
Logistic regression with a twist! We will use variational inference to learn a distribution over the learned parameters.

Througout this work we will be focusing on a binary classification task, on
a small dataset (the UCD breast cancer dataset).

Check out the slides for this colab [here](https://github.com/eeml2019/PracticalSessions/tree/master/unsupervised).

## Logistic regression

Learn the weights which best classify the $(x, y)$ pairs in the dataset.
The weights are learned via the following maximization problem:

\begin{equation}
\mathbb{E}_{p^*(x, y)} \log p(y|x, w) = \\
 \mathbb{E}_{p^*(x, y)} \left[y \log \sigma(w x + b) + (1 - y) \log(1- \sigma(w x + b)\right]
\end{equation}

Here $\sigma$ denotes the sigmoid function.

## Bayesian logistic regression

Learn a distribution over the weights which best classify the $(x, y)$ pairs in the dataset.

We now want to learn the distributional parameters of $q(w)$ in order to maximize:

\begin{equation}
\mathbb{E}_{p^*(x, y)} \log p_w(y| x) = \\
\mathbb{E}_{p^*(x, y)} \log \int p(y|x, w) p(w) \delta w = \\
\mathbb{E}_{p^*(x, y)} \log \int p(y|x, w) p(w) \frac{q(w)}{q(w)} \delta w \ge \\
\mathbb{E}_{p^*(x, y)} \mathbb{E}_{q(w)} \log \left[p(y| x, w) \frac{p(w)}{q(w)} \right]= \\
\mathbb{E}_{q(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)  -  KL(q(w)||p(w)) = \\
 \mathbb{E}_{q(w)}\mathbb{E}_{p^*(x, y)} \left[y \log \sigma(w x + b) + (1 - y) \log(1- \sigma(w x + b)\right] -  KL(q(w)||p(w))
\end{equation}

In Bayesian Logistic Regression, we thus learn a distribution over parameters
$w$ which can explain the data, while staying close to a chosen prior $p(w)$.

Throughout this lab, we will use Gaussian Distributions for $p(w)$ and $q(w)$.

For more details, see
[this paper](https://pdfs.semanticscholar.org/e407/ea7fda6d152d2186f4b5e27aa04ec2d32dcd.pdf) or
[this tutorial](https://www.ece.rice.edu/~vc3/elec633/logistic.pdf).

### Gradient estimation

In Bayesian Logistic Regression, we aim to learn the parameters of the distribution $q(w)$. We will call these parameters $\theta$. In the case of a
Gaussian distribution, these will be the mean and covariance matrix of a
multivariate distribution. Since we will use stochastic gradient descent to learn $\theta$, we need to be able to compute the gradients with respect to
$\theta$ of our objective. Since our objective contains an expectation with respect to $q(w)$, this can be challenging. To illustrate this, from now on we will denote $q(w)$ as $q_{\theta}(w)$.

Specifically, we are interested in:
\begin{equation}
  \nabla_{\theta} \left[\mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)  -  KL(q_{\theta}(w)||p(w))\right]
\end{equation}

Since in we will be using Gaussian distributions for $q_{\theta}(w)$ and $p(w)$
$KL(q_{\theta}(w)||p(w))$ can be computed in closed form and we can rely on
TensorFlow's automatic differentiation to compute
$\nabla_{\theta} KL(q_{\theta}(w)||p(w))$.

Let's now turn attention to computing $\nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)$. Unlike the KL, the integral
$\mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)$
is not tractable, and thus cannot be computed in closed form. Thus, we are interested in other approaches to compute the gradient.

### Reinforce gradient estimation

A first approach to compute an estimate of the gradient is to use the
REINFORCE gradient estimator:

\begin{equation}
  \nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  = \\
  \nabla_{\theta} \int q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \delta w = \\
  \int \nabla_{\theta} \left[q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right] \delta w = \\
 \int \nabla_{\theta} \left[q_{\theta}(w)\right] \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \delta w = \\
 \int q_{\theta}(w) \nabla_{\theta} \left[\log q_{\theta}(w)\right] \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  \delta w = \\
 \mathbb{E}_{q_{\theta}(w)} \left[ \nabla_{\theta} \log q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right]
\end{equation}

We can use samples from $q_{\theta}(w)$ to compute the Monte Carlo estimate of
the last integral, to get an unbiased estimator of the true gradient. The more
samples we use to estimate the integral, the more accurate the gradient estimate
will be.

### Pathwise gradient estimation (reparametrization)

In the REINFORCE estimator, we did not use any knowledge of the variational
distribution $q_{\theta}(w)$.
For a Gaussian distribution, we can use the reparametrization trick to obtain
an unbiased gradient estimator of  $\nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)$.

To do so, we will use the fact that
\begin{equation}
    z \sim N(\mu, \sigma), z = \mu + \epsilon \sigma, {\text {with }} \epsilon \sim N(0, 1)
\end{equation}

Namely:
\begin{equation}
  \nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  = \\
  \nabla_{\theta} \mathbb{E}_{p(\epsilon)} \mathbb{E}_{p^*(x, y)} \log p(y| x, \mu + \epsilon \sigma)  = \\
  \mathbb{E}_{p(\epsilon)} \nabla_{\theta} \mathbb{E}_{p^*(x, y)} \log p(y| x, \mu + \epsilon \sigma)
\end{equation}

Note that we were able to move the gradient inside the integral since
$p(\epsilon)$ does not depend on $\theta$.
To estimate the last integral, we can use a Monte Carlo estimate using samples
of $p(\epsilon)$.

```
import enum
import numpy as np

import math
from sklearn import datasets
from sklearn import linear_model
import collections

from matplotlib import pyplot as plt
import tensorflow as tf
import tensorflow_probability as tfp
import seaborn as sns

tfd = tfp.distributions

```

```
sns.set(rc={"lines.linewidth": 2.8}, font_scale=2)
sns.set_style("whitegrid")
```

## Get the dataset and visualize it

```
def get_data(normalize=True):
    data = datasets.load_breast_cancer()

    print(data.target_names)
    print(data.feature_names)

    features = np.array(data.data, dtype=np.float32)
    targets = np.array(data.target, dtype=np.float32)

    if normalize:
        # Note: the data dimensions have very different scales.
        # We normalize by the mean and scale of the entire dataset.
        features = features - features.mean(axis=0)
        features = features / features.std(axis=0)

    # Add a dimension of 1 (bias).
    features = np.concatenate((features, np.ones((features.shape[0], 1))), axis=1)
    features = np.array(features, dtype=np.float32)

    return features, targets, data.feature_names
```

### Dataset

This datasets has meaningful features: we are trying to classify based on these features whether someone has breast cancer or not. We can look at the
features we are using to classify the data below.

```
features, targets, feature_names = get_data()
```

    ['malignant' 'benign']
    ['mean radius' 'mean texture' 'mean perimeter' 'mean area'
     'mean smoothness' 'mean compactness' 'mean concavity'
     'mean concave points' 'mean symmetry' 'mean fractal dimension'
     'radius error' 'texture error' 'perimeter error' 'area error'
     'smoothness error' 'compactness error' 'concavity error'
     'concave points error' 'symmetry error' 'fractal dimension error'
     'worst radius' 'worst texture' 'worst perimeter' 'worst area'
     'worst smoothness' 'worst compactness' 'worst concavity'
     'worst concave points' 'worst symmetry' 'worst fractal dimension']

### Trivial baseline

How well can we do with a constant classifier?

If we look at the average label, we can see how many 1s are in the data, so we know that our classifier has to get more than that accuracy

```
np.mean(targets)
```

    0.6274165

```
features.shape
```

    (569, 31)

```
dataset_size = features.shape[0]
data_dims = features.shape[-1]
```

## Logistic regression baseline - how well can we do on this dataset?

Let's look at a simple, non Bayesian classification approach to get an idea of how well we can do on the dataset.

```
lr = linear_model.LogisticRegression()
```

```
lr.fit(features, targets)
lr.score(features, targets)
```

    0.9876977152899824

## Utilities

```
def get_shape_list(tensor):
  return tensor.shape.as_list()
```

```
def get_sklearn_data_as_tensors(
      features, targets, batch_size, dataset_name='breast_cancer'):
  """Read sklearn datasets as tf.Tensors.

  Args:
    batch_size: Integer or None. If None, the entire dataset is used.
    dataset_name: A string, the name of the dataset.

  Returns:
    A tuple of size two containing two tensors of rank 2 `[B, F]`, the data
    features and targets.
  """
  dataset = tf.data.Dataset.from_tensor_slices((features, targets))
  if batch_size:
    # Shuffle, repeat, and batch the examples.
    batched_dataset = dataset.shuffle(1000).repeat().batch(batch_size)
  else:
    batch_size = features.shape[0]
    batched_dataset = dataset.repeat().batch(batch_size)

  iterator = batched_dataset.make_one_shot_iterator()
  batch_features, batch_targets = iterator.get_next()

  data_dim = features.shape[1]
  batch_features.set_shape([batch_size, data_dim])
  batch_targets.set_shape([batch_size])

  return batch_features, batch_targets
```

## Bayesian Logistic Regression - define the prior and the posterior

We will learn a distribution over parameters, so we will first define prior distribution, and then a posterior distribution over parameters which we
will learn.

Both of these distibutions will be Gaussian.

```
def multi_normal(loc, log_scale):
  scale = tf.exp(log_scale)
  return tfd.MultivariateNormalDiag(loc=loc, scale_diag=scale)

def diagonal_gaussian_posterior(data_dims):
  mean = tf.Variable(
      tf.zeros(shape=(data_dims), dtype=tf.float32), name='mean')
  log_scale = tf.Variable(
      tf.zeros(shape=(data_dims), dtype=tf.float32),
      name='log_scale')
  learned_vars = [mean, log_scale]
  return multi_normal(loc=mean, log_scale=log_scale), learned_vars
```

## Bayesian Logistic Regression - the model

### Hypers

```
BATCH_SIZE = 500
NUM_POSTERIOR_SAMPLES = 100
```

```
prior = multi_normal(loc=tf.zeros(data_dims), log_scale=tf.zeros(data_dims))
posterior, learned_vars  = diagonal_gaussian_posterior(data_dims)
```

```
features, targets = get_sklearn_data_as_tensors(features, targets, BATCH_SIZE)
```

```
def _predictions(logits):
  return tf.to_float(logits >= 0)

def _accuracy(targets, predictions):
  # `predictions` have rank 2: batch_size, num_posterior samples.
  # We expand dims the targets to compute the accuracy per sample.
  targets = tf.expand_dims(targets, axis=1)
  return tf.reduce_mean(tf.to_float(tf.equal(targets, predictions)))

def linear_model(data, targets, posterior_samples):
  num_posterior_samples = tf.shape(posterior_samples)[0]
  logits = tf.matmul(data, posterior_samples, transpose_b=True)
  # Make targets [B, 1] to use broadcasting.
  targets = tf.expand_dims(targets, axis=1)
  targets = targets * tf.ones([1, num_posterior_samples])

  log_probs = - tf.nn.sigmoid_cross_entropy_with_logits(
      labels=targets, logits=logits)
  return log_probs, logits
```

```
def run_model(features, targets, prior, posterior, num_samples):
  posterior_samples = posterior.sample(num_samples)

  # Compute the log probs of the data under all the models we have sampled.
  log_probs, logits = linear_model(features, targets, posterior_samples)
  # Compute the KL.
  kl = posterior.kl_divergence(prior)

  # Sum over data.
  param_log_probs = tf.reduce_mean(log_probs, axis=0) * dataset_size
  param_log_probs.shape.assert_is_compatible_with([num_samples])

  return param_log_probs, kl, posterior_samples, logits
```

```
param_log_probs, kl, posterior_samples, logits = run_model(
      features, targets, prior, posterior, NUM_POSTERIOR_SAMPLES)
predictions = _predictions(logits)
accuracy = _accuracy(targets=targets, predictions=predictions)

accuracy = tf.reduce_mean(accuracy)
per_sample_elbo = param_log_probs - kl
elbo = tf.reduce_mean(per_sample_elbo)
```

## Training loop

## Training the model via reparametrization

```
# Stochastic loss depends on the output of a sampling operation  (posterior samples)
# but tensorflow implements reparametrization by default.
per_sample_reparametrization_loss = - per_sample_elbo
reparametrization_loss = tf.reduce_mean(per_sample_reparametrization_loss)
```

```
optimizer = tf.train.GradientDescentOptimizer(0.0001)
reparam_min_op = optimizer.minimize(reparametrization_loss)
```

```
NUM_ITERATIONS = 1000
```

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

reparam_accuracies = []
reparam_kls = []
reparam_elbos = []

for i in xrange(NUM_ITERATIONS):
  sess.run(reparam_min_op)

  if i % 10 == 0:
    reparam_acc, reparam_kl, reparam_elbo = sess.run([accuracy, kl, elbo])
    reparam_accuracies += [reparam_acc]
    reparam_kls += [reparam_kl]
    reparam_elbos += [reparam_elbo]

    print('Iteration {}. Elbo {}. KL {}'.format(
        i, reparam_elbo, reparam_kl))

reparam_learned_mean, reparam_learned_log_scale = sess.run(learned_vars)
```

    Iteration 0. Elbo -1206.31604004. KL 0.00301633775234
    Iteration 10. Elbo -642.610412598. KL 0.293668031693
    Iteration 20. Elbo -577.772949219. KL 0.783629059792
    Iteration 30. Elbo -392.721923828. KL 1.2917201519
    Iteration 40. Elbo -351.349578857. KL 1.781883955
    Iteration 50. Elbo -306.663360596. KL 2.20695495605
    Iteration 60. Elbo -228.974685669. KL 2.57792901993
    Iteration 70. Elbo -234.82182312. KL 2.95425152779
    Iteration 80. Elbo -228.548690796. KL 3.29743432999
    Iteration 90. Elbo -196.769805908. KL 3.61361837387
    Iteration 100. Elbo -190.871459961. KL 3.9075319767
    Iteration 110. Elbo -220.950912476. KL 4.18836021423
    Iteration 120. Elbo -181.642272949. KL 4.45673751831
    Iteration 130. Elbo -163.075805664. KL 4.70559310913
    Iteration 140. Elbo -166.205795288. KL 4.96141529083
    Iteration 150. Elbo -161.878341675. KL 5.22007894516
    Iteration 160. Elbo -165.926605225. KL 5.44898462296
    Iteration 170. Elbo -158.645767212. KL 5.66910552979
    Iteration 180. Elbo -164.979400635. KL 5.88050985336
    Iteration 190. Elbo -144.492584229. KL 6.12092161179
    Iteration 200. Elbo -151.752105713. KL 6.33854484558
    Iteration 210. Elbo -165.249649048. KL 6.54486370087
    Iteration 220. Elbo -127.59765625. KL 6.73881149292
    Iteration 230. Elbo -119.381202698. KL 6.92593240738
    Iteration 240. Elbo -111.081077576. KL 7.11052560806
    Iteration 250. Elbo -115.483856201. KL 7.28692436218
    Iteration 260. Elbo -135.568496704. KL 7.45468378067
    Iteration 270. Elbo -126.830368042. KL 7.62879562378
    Iteration 280. Elbo -121.736724854. KL 7.79468870163
    Iteration 290. Elbo -120.828201294. KL 7.97476053238
    Iteration 300. Elbo -123.977989197. KL 8.13904094696
    Iteration 310. Elbo -127.896560669. KL 8.29942989349
    Iteration 320. Elbo -130.98664856. KL 8.4574213028
    Iteration 330. Elbo -107.724006653. KL 8.59646511078
    Iteration 340. Elbo -96.2202224731. KL 8.7516078949
    Iteration 350. Elbo -99.8892974854. KL 8.88774490356
    Iteration 360. Elbo -110.625091553. KL 9.02900981903
    Iteration 370. Elbo -110.705986023. KL 9.16001415253
    Iteration 380. Elbo -118.763916016. KL 9.29595279694
    Iteration 390. Elbo -101.959465027. KL 9.42527866364
    Iteration 400. Elbo -108.359840393. KL 9.55098152161
    Iteration 410. Elbo -94.4869918823. KL 9.68084907532
    Iteration 420. Elbo -88.2326431274. KL 9.8120470047
    Iteration 430. Elbo -104.121643066. KL 9.92966651917
    Iteration 440. Elbo -112.153770447. KL 10.0524768829
    Iteration 450. Elbo -102.999305725. KL 10.1651391983
    Iteration 460. Elbo -110.671699524. KL 10.2773513794
    Iteration 470. Elbo -99.833732605. KL 10.3971281052
    Iteration 480. Elbo -101.300125122. KL 10.5014190674
    Iteration 490. Elbo -106.583534241. KL 10.6122703552
    Iteration 500. Elbo -100.561035156. KL 10.7189235687
    Iteration 510. Elbo -99.5090408325. KL 10.8257017136
    Iteration 520. Elbo -94.6386489868. KL 10.9309864044
    Iteration 530. Elbo -92.851524353. KL 11.0329875946
    Iteration 540. Elbo -83.7906646729. KL 11.1409406662
    Iteration 550. Elbo -100.541503906. KL 11.242099762
    Iteration 560. Elbo -84.2009353638. KL 11.3463287354
    Iteration 570. Elbo -99.4818344116. KL 11.4444103241
    Iteration 580. Elbo -87.810798645. KL 11.5469789505
    Iteration 590. Elbo -90.5667190552. KL 11.6429700851
    Iteration 600. Elbo -89.6919326782. KL 11.7347326279
    Iteration 610. Elbo -89.1579208374. KL 11.8316688538
    Iteration 620. Elbo -95.3815689087. KL 11.924328804
    Iteration 630. Elbo -97.5506820679. KL 12.0179824829
    Iteration 640. Elbo -89.58984375. KL 12.1033706665
    Iteration 650. Elbo -87.0892181396. KL 12.1903495789
    Iteration 660. Elbo -83.8239746094. KL 12.2757358551
    Iteration 670. Elbo -98.5659866333. KL 12.3601484299
    Iteration 680. Elbo -85.6225814819. KL 12.454410553
    Iteration 690. Elbo -85.4044265747. KL 12.5378456116
    Iteration 700. Elbo -81.7902679443. KL 12.6191778183
    Iteration 710. Elbo -90.9003829956. KL 12.701210022
    Iteration 720. Elbo -86.0762786865. KL 12.7857275009
    Iteration 730. Elbo -81.9525756836. KL 12.8606281281
    Iteration 740. Elbo -82.8061523438. KL 12.9407939911
    Iteration 750. Elbo -75.7044754028. KL 13.014670372
    Iteration 760. Elbo -85.7485656738. KL 13.0864629745
    Iteration 770. Elbo -84.7195129395. KL 13.1637172699
    Iteration 780. Elbo -81.5038986206. KL 13.2388887405
    Iteration 790. Elbo -91.4418334961. KL 13.305138588
    Iteration 800. Elbo -91.3380737305. KL 13.378786087
    Iteration 810. Elbo -79.8998031616. KL 13.4527482986
    Iteration 820. Elbo -82.5501861572. KL 13.5232639313
    Iteration 830. Elbo -73.904045105. KL 13.5923223495
    Iteration 840. Elbo -94.8883132935. KL 13.6631288528
    Iteration 850. Elbo -82.671081543. KL 13.7367839813
    Iteration 860. Elbo -89.1314620972. KL 13.8053359985
    Iteration 870. Elbo -89.9466018677. KL 13.8741493225
    Iteration 880. Elbo -72.7582550049. KL 13.9450321198
    Iteration 890. Elbo -86.2754516602. KL 14.0125989914
    Iteration 900. Elbo -75.2065505981. KL 14.0771455765
    Iteration 910. Elbo -76.9281082153. KL 14.1364145279
    Iteration 920. Elbo -76.7940979004. KL 14.2024030685
    Iteration 930. Elbo -71.4757919312. KL 14.2663583755
    Iteration 940. Elbo -78.8379974365. KL 14.3307580948
    Iteration 950. Elbo -69.5632553101. KL 14.3894805908
    Iteration 960. Elbo -84.2538375854. KL 14.4518518448
    Iteration 970. Elbo -93.6064071655. KL 14.5128087997
    Iteration 980. Elbo -83.2115631104. KL 14.5737915039
    Iteration 990. Elbo -82.0588302612. KL 14.636089325

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(reparam_elbos, label='ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].set_ylim((-500, -50))
axes[0].legend()

axes[1].plot(reparam_kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(reparam_accuracies, label='Accuracy')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

## Training the model via reinforce

```
# Important: the multiplication has to be done elementwise.
# We have to train the model using a surrogate loss.
# Since TensorFlow does automatic differentation, we have to use
# stop_gradient on the loss, as well as the samples!
per_sample_reinforce_loss =  - posterior.log_prob(tf.stop_gradient(posterior_samples)) * tf.stop_gradient(param_log_probs)
reinforce_loss = tf.reduce_mean(per_sample_reinforce_loss) + kl
```

```
optimizer = tf.train.GradientDescentOptimizer(0.0001)
reinforce_min_op = optimizer.minimize(reinforce_loss)
```

```
NUM_ITERATIONS = 1000
```

```
reinforce_accuracies = []
reinforce_kls = []
reinforce_elbos = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reinforce_min_op)

  if i % 10 == 0:
    reinforce_acc, reinforce_kl, reinforce_elbo = sess.run([accuracy, kl, elbo])
    reinforce_accuracies += [reinforce_acc]
    reinforce_kls += [reinforce_kl]
    reinforce_elbos += [reinforce_elbo]

    print('Iteration {}. Elbo {}. KL {}'.format(
        i, reinforce_elbo, reinforce_kl))

reinforce_learned_mean, reinforce_learned_log_scale = sess.run(learned_vars)
```

    Iteration 0. Elbo -1231.48754883. KL 0.0220593810081
    Iteration 10. Elbo -808.592407227. KL 0.338543713093
    Iteration 20. Elbo -524.581542969. KL 0.909575939178
    Iteration 30. Elbo -379.652740479. KL 1.41321325302
    Iteration 40. Elbo -301.671539307. KL 1.8704212904
    Iteration 50. Elbo -279.019744873. KL 2.25260615349
    Iteration 60. Elbo -287.911437988. KL 2.66793394089
    Iteration 70. Elbo -248.627319336. KL 3.03274250031
    Iteration 80. Elbo -274.209838867. KL 3.39950537682
    Iteration 90. Elbo -238.783615112. KL 3.7056684494
    Iteration 100. Elbo -180.144882202. KL 4.10226154327
    Iteration 110. Elbo -216.213302612. KL 4.40311670303
    Iteration 120. Elbo -183.531951904. KL 4.69419574738
    Iteration 130. Elbo -186.911499023. KL 4.96079969406
    Iteration 140. Elbo -173.515960693. KL 5.19095230103
    Iteration 150. Elbo -157.491165161. KL 5.43379211426
    Iteration 160. Elbo -158.108306885. KL 5.6262254715
    Iteration 170. Elbo -163.221206665. KL 5.8356795311
    Iteration 180. Elbo -146.565139771. KL 6.06200742722
    Iteration 190. Elbo -190.070983887. KL 6.17145061493
    Iteration 200. Elbo -153.055160522. KL 6.33472442627
    Iteration 210. Elbo -161.657592773. KL 6.56980705261
    Iteration 220. Elbo -116.549758911. KL 6.77828407288
    Iteration 230. Elbo -125.225143433. KL 7.03187656403
    Iteration 240. Elbo -147.91607666. KL 7.22355747223
    Iteration 250. Elbo -133.264282227. KL 7.41294574738
    Iteration 260. Elbo -126.578613281. KL 7.57341766357
    Iteration 270. Elbo -119.080780029. KL 7.75994729996
    Iteration 280. Elbo -109.886116028. KL 7.95444965363
    Iteration 290. Elbo -121.945098877. KL 8.08462715149
    Iteration 300. Elbo -124.018943787. KL 8.18129253387
    Iteration 310. Elbo -106.144706726. KL 8.37997722626
    Iteration 320. Elbo -121.718406677. KL 8.57028007507
    Iteration 330. Elbo -125.367019653. KL 8.72685909271
    Iteration 340. Elbo -118.693809509. KL 8.87324142456
    Iteration 350. Elbo -101.319114685. KL 9.01533222198
    Iteration 360. Elbo -114.103240967. KL 9.16397762299
    Iteration 370. Elbo -91.875869751. KL 9.27885913849
    Iteration 380. Elbo -114.098571777. KL 9.42788505554
    Iteration 390. Elbo -114.865745544. KL 9.54022693634
    Iteration 400. Elbo -94.1702957153. KL 9.65817832947
    Iteration 410. Elbo -109.974090576. KL 9.79271125793
    Iteration 420. Elbo -109.752044678. KL 9.85466575623
    Iteration 430. Elbo -110.549911499. KL 10.0302639008
    Iteration 440. Elbo -108.423713684. KL 10.163107872
    Iteration 450. Elbo -80.6597824097. KL 10.3107624054
    Iteration 460. Elbo -103.389984131. KL 10.4190883636
    Iteration 470. Elbo -101.75. KL 10.5393753052
    Iteration 480. Elbo -112.833908081. KL 10.6717405319
    Iteration 490. Elbo -94.2823638916. KL 10.7516059875
    Iteration 500. Elbo -102.636833191. KL 10.8633699417
    Iteration 510. Elbo -90.9130401611. KL 10.9743413925
    Iteration 520. Elbo -76.0304870605. KL 11.0320749283
    Iteration 530. Elbo -102.61164856. KL 11.130033493
    Iteration 540. Elbo -97.4822235107. KL 11.2614850998
    Iteration 550. Elbo -80.1745529175. KL 11.3975610733
    Iteration 560. Elbo -92.6554031372. KL 11.5362567902
    Iteration 570. Elbo -87.7685241699. KL 11.6449260712
    Iteration 580. Elbo -113.672859192. KL 11.7816791534
    Iteration 590. Elbo -89.4435577393. KL 11.870429039
    Iteration 600. Elbo -98.1239089966. KL 11.971988678
    Iteration 610. Elbo -86.9800796509. KL 12.0661430359
    Iteration 620. Elbo -105.8777771. KL 12.1732368469
    Iteration 630. Elbo -90.7897491455. KL 12.2917041779
    Iteration 640. Elbo -94.3220977783. KL 12.3896751404
    Iteration 650. Elbo -88.0863800049. KL 12.4850358963
    Iteration 660. Elbo -84.8598632812. KL 12.6095962524
    Iteration 670. Elbo -90.5580291748. KL 12.6718521118
    Iteration 680. Elbo -79.3591690063. KL 12.7729558945
    Iteration 690. Elbo -90.6020584106. KL 12.8668231964
    Iteration 700. Elbo -80.9606170654. KL 12.9421491623
    Iteration 710. Elbo -98.1525802612. KL 13.0128526688
    Iteration 720. Elbo -89.2859802246. KL 13.1471347809
    Iteration 730. Elbo -84.0707321167. KL 13.1970977783
    Iteration 740. Elbo -89.5597915649. KL 13.2572488785
    Iteration 750. Elbo -99.1416320801. KL 13.3330011368
    Iteration 760. Elbo -92.910118103. KL 13.4171209335
    Iteration 770. Elbo -76.833732605. KL 13.5153656006
    Iteration 780. Elbo -77.1132965088. KL 13.5576887131
    Iteration 790. Elbo -93.1914291382. KL 13.6402273178
    Iteration 800. Elbo -88.9732894897. KL 13.7351942062
    Iteration 810. Elbo -101.056121826. KL 13.8010444641
    Iteration 820. Elbo -99.5751037598. KL 13.8656616211
    Iteration 830. Elbo -91.9339675903. KL 13.9571743011
    Iteration 840. Elbo -70.7454147339. KL 14.0105953217
    Iteration 850. Elbo -91.7919006348. KL 14.0381126404
    Iteration 860. Elbo -82.0033187866. KL 14.1070785522
    Iteration 870. Elbo -81.9294433594. KL 14.1275062561
    Iteration 880. Elbo -103.577262878. KL 14.1831588745
    Iteration 890. Elbo -88.1100387573. KL 14.2206459045
    Iteration 900. Elbo -95.5690231323. KL 14.2572431564
    Iteration 910. Elbo -68.3534698486. KL 14.3253746033
    Iteration 920. Elbo -86.1409835815. KL 14.3638477325
    Iteration 930. Elbo -76.2172088623. KL 14.3877239227
    Iteration 940. Elbo -67.9566345215. KL 14.4778957367
    Iteration 950. Elbo -78.4613800049. KL 14.5488395691
    Iteration 960. Elbo -76.8335723877. KL 14.587480545
    Iteration 970. Elbo -75.7317123413. KL 14.6351766586
    Iteration 980. Elbo -85.0177459717. KL 14.7029514313
    Iteration 990. Elbo -96.061920166. KL 14.7556276321

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(reinforce_elbos, label='ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].set_ylim((-500, -50))
axes[0].legend()

axes[1].plot(reinforce_kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(reinforce_accuracies, label='Accuracy')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

## Analysis

* How sensitive are the different estimators to the number of posterior samples used? Try 1, 10, 100.
* How sensitive are the different estimators to the learning rate used? Try 1e-4, 1e-3 and 1e-2 for both estimators.
* How sensitive are the different estimators to batch size? Try batch size 10 for both estimators.

# Answer
* Reparametriztion less sensitive to changes in posterior samples, learning rates, batch size.

# Visualize learned parameters

* Let's visualize the learned parameters, as well as check the different uncertanties around different parameters
* Can we now see which parameters are most important in the classification problem?

```
learned_mean = reparam_learned_mean
learned_scale = np.exp(reparam_learned_log_scale)
```

```
PARAM_INDEX = 19
```

```
def gaussian(x, mu, sig):
    return np.exp(-np.power(x - mu, 2.) / (2 * np.power(sig, 2.)))
```

```
mean = learned_mean[PARAM_INDEX]
scale = learned_scale[PARAM_INDEX]
x_values = np.linspace(-3, 3, num=1000)
y_values = [gaussian(x, mean, scale) for x in x_values]

plt.figure(figsize=(7,5))
plt.plot(x_values, y_values)
plt.vlines(
      learned_mean[PARAM_INDEX],
      ymin=0,
      ymax=gaussian(mean, mean, scale),
      colors='g',
      alpha=0.8)

x_std_values = np.linspace(mean - scale, mean + scale, num=1000)
y_std_values = [gaussian(x, mean, scale) for x in x_std_values]
plt.fill_between(x_std_values, y_std_values, 0, alpha=0.3, color='b')

plt.title('Parameter name= {}'.format(feature_names[PARAM_INDEX]))
```

## Visualize gradient variance - which estimator has lower variance?

* Let's assess the gradient variance per of each estimator, by looking at how much gradients vary for each estimators.

```
def per_batch_grads(ys, xs):
  """Computes the gradients of ys[i] wrt xs."""
  grads = []

  for y in tf.unstack(ys, axis=0):
    grads.append(tf.squeeze(tf.gradients(y, xs)[0]))
  return tf.stack(grads, axis=0)
```

```
def compute_grad_var(grads, param_index):
  return np.mean(grads[:, param_index]), np.std(grads[:, param_index])**2
```

```
per_batch_reinforce_grads = per_batch_grads(per_sample_reinforce_loss, learned_vars[0])
per_batch_reinforce_grads.shape
```

    TensorShape([Dimension(100), Dimension(31)])

```
per_batch_reparam_grads = per_batch_grads(per_sample_reparametrization_loss, learned_vars[0])
per_batch_reparam_grads.shape
```

    TensorShape([Dimension(100), Dimension(31)])

```
reinforce_grads = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reinforce_min_op)

  if i % 10 == 0:
    reinforce_grads.append(sess.run(per_batch_reinforce_grads))
```

```
reparam_grads = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reparam_min_op)

  if i % 10 == 0:
    reparam_grads.append(sess.run(per_batch_reparam_grads))
```

```
PARAM_INDEX = 19
```

```
reinforce_grad_vars = []
for timestep_grad in reinforce_grads:
  _, reinforce_grad_var = compute_grad_var(timestep_grad, PARAM_INDEX)
  reinforce_grad_vars.append(reinforce_grad_var)
```

```
reparam_grad_vars = []
for timestep_grad in reparam_grads:
  _, reparam_grad_var = compute_grad_var(timestep_grad, PARAM_INDEX)
  reparam_grad_vars.append(reparam_grad_var)
```

```
plt.figure(figsize=(7,5))
plt.plot(reinforce_grad_vars, label='reinforce')
plt.plot(reparam_grad_vars, label='reparametrization')
plt.yscale('log')
plt.xlabel('It')
plt.ylabel('Gradient variance')
plt.legend()
```

### Analysis

* What do you notice about gradient variance as training progresses?
* What do you notice about the variance of the different estimators?
* How do batch size and number of posterior samples affect the variance of the different estimators?
* How does gradient variance affect convergence?
* Which gradient estimator would you use in practice for this problem?

# Answer
 * Gradient variance goes down as training progresses regardless of estimator.
 * Reinforce = bad.
 * Higher batch size and number of samples = lower variance.
 * Learning rate: lower learning rate is a way to counter balance high gradient variance.
 * Unless you reduce learning rates in training, you might oscilate too much, which slows down learning.
 * Reparametrization.


# Taylor integration of the Kepler problem

Here, we try to reproduce __exactly__ the [Kepler problem integration example](http://nbviewer.jupyter.org/github/JuliaDiff/TaylorSeries.jl/blob/master/examples/1-KeplerProblem.ipynb) made by Luis Benet in [JuliaDiff/TaylorSeries.jl](https://github.com/JuliaDiff/TaylorSeries.jl).

The Kepler problem is the basis of planetary motion; it describes the motion of a secondary body (e.g., a planet, asteroid, comet, etc.), around a primary body (e.g., the Sun). In cartesian coordinates over the orbital plane, the Hamiltonian for the Kepler problem reads:

$$
\begin{align}
H_{\mathrm{Kepler}} &= \frac{1}{2\mu}(p_x^2+p_y^2)-\frac{\mu}{\sqrt{x^2+y^2}}
\end{align}
$$

where $\mu=G(m_1+m_2)$, $G$ is the gravitational constant, $m_1$ is the mass of the primary body and $m_2$ is the mass of the secondary body. If we write $\vec r_1 = (x_1,y_1)$ and $\vec r_2 = (x_2,y_2)$, respectively, for the position of the primary and secondary body, then the vector $\vec r = \vec r_2-\vec r_1$, with coordinates $\vec r = (x, y)=(x_2-x_1,y_2-y_1)$, represents the position of the secondary body relative to the primary body; i.e., $\vec r$ represents the so-called relative coordinates. In terms of the vector $\vec{r}$, the position $\vec{r}=0$ corresponds to the position of the primary body.

Using Hamilton equations, we can obtain the equations of motion for the Kepler problem:

$$
\begin{align}
\dot x &= u \\
\dot y &= v \\
\dot u &= -\frac{\mu x}{(x^2+y^2)^{3/2}}\\
\dot v &= -\frac{\mu y}{(x^2+y^2)^{3/2}}
\end{align}
$$

Note that the canonical momenta are $p_x$ and $p_y$, while $u$ and $v$ are, respectively, the  $x$ and $y$ components of the velocity; i.e., $p_x = \mu u$ and $p_y = \mu v$.

First of all, we shall include all relevant packages:

```julia
using TaylorIntegration, Plots, LaTeXStrings
pyplot()
```

    Fontconfig warning: ignoring UTF-8: not a valid region tag

    Plots.PyPlotBackend()

Some parameters necessary for the integration:

+ $\mu$: the gravitational parameter
+ `q0`: the initial condition (we will select an initial condition which corresponds to elliptical motion)
+ `order`: the order of the Taylor expansion
+ `t_max`: the final time of the integration
+ `abs_tol`: the absolute tolerance
+ `n_iter`: the number of time-steps

```julia
const μ = 1.0
const q0 = [0.19999999999999996, 0.0, 0.0, 3.0] # a initial condition for elliptical motion
const order = 28
const t0 = 0.0
const t_max = 10000*(2π) # we are just taking a wild guess about the period ;)
const abs_tol = 1.0E-20
const steps = 500000
```

    500000

As usual, we write down the equations of motion into a `function`, which here we will name `kepler!`. `q` represents the system state; i.e., the set of values of the dynamical variables at a given instant; `dq` represents the time derivatives of the components of `q`.

```julia
#an auxiliary array which helps with optimization:
const r_p3d2 = Array{TaylorSeries.Taylor1{Float64}}(1)

#the equations of motion for the Kepler problem:
function kepler!(t, q, dq)
    r_p3d2[1] = (q[1]^2+q[2]^2)^(3/2)

    dq[1] = q[3]
    dq[2] = q[4]
    dq[3] = -μ*q[1]/r_p3d2[1]
    dq[4] = -μ*q[2]/r_p3d2[1]

    nothing
end
```

    kepler! (generic function with 1 method)

The Taylor integration:

```julia
t, q = taylorinteg(kepler!, q0, t0, 0.01, order, abs_tol, maxsteps=2); #warm-up lap
@time t, q = taylorinteg(kepler!, q0, t0, t_max, order, abs_tol, maxsteps=steps);
```

     43.060857 seconds (464.94 M allocations: 39.523 GB, 12.87% gc time)

The final state:

```julia
t[end]/(2pi), q[end,:]
```

    (10000.0,[0.2,2.72854e-9,-2.27378e-8,3.0])

Let's extract the values of $x$, $y$, $u$ and $v$ for each time-step:

```julia
x, y, u, v = view(q,:,1), view(q,:,2), view(q,:,3), view(q,:,4);
```

## Orbital motion

The initial conditions we selected correspond to a elliptical orbit with a relatively high eccentricity: $e=0.8$ for the initial condition `q0=[0.19999999999999996, 0.0, 0.0, 3.0]`. How does the motion of the planet/asteroid/comet looks like? Well, let's plot its orbit over the $x-y$ plane (the orbit is shown in blue; the position of the primary body is shown as a yellow dot):

```julia
scatter(

[0.0], [0.0],
label = L"\mathrm{Center\; of\; attraction}",
ms=10

)
scatter!(

x[1:75:end], y[1:75:end],
title = L"\mathrm{Orbital\;\;motion}",
xaxis = (L"x", (-2.0, 0.5), -2.0:0.5:2.0),
yaxis = (L"y", (-0.8, 0.8), -2.0:0.5:2.0),
label = L"\mathrm{Keplerian\; ellipse}",
ms=0.5

)
```

## Energy conservation

Below, we write the energy function of the Kepler problem; i.e., the Hamiltonian $H_{\mathrm{Kepler}}$ in terms of $x$, $y$, $u$ and $v$:

```julia
H(x_, y_, u_, v_) = 0.5*(u_^2+v_^2)-μ/sqrt(x_^2+y_^2)
```

    H (generic function with 1 method)

Now, using the function `H` defined above, we calculate the energy during each time-step:

```julia
E = H.(x, y, u, v);
```

$E_0=H_{\mathrm{Kepler}}(x_0,y_0,u_0,v_0)$, the initial value of the energy, is:

```julia
E0=E[1]
```

    -0.5000000000000009

We define $\delta E$ as the relative error in the energy; i.e.:

$$
\begin{align}
\delta E(t) &= \frac{E(t)-E_0}{E_0}
\end{align}
$$

```julia
δE = (E-E0)/(E0)
```

    436153-element Array{Float64,1}:
     -0.0
      1.77636e-15
      1.77636e-15
      8.88178e-16
      8.88178e-16
      2.66454e-15
      3.55271e-15
      2.66454e-15
      3.55271e-15
      2.66454e-15
      3.9968e-15
      2.66454e-15
      3.10862e-15
      ⋮
     -7.12763e-14
     -7.19425e-14
     -7.14984e-14
     -7.10543e-14
     -7.19425e-14
     -7.10543e-14
     -7.10543e-14
     -6.92779e-14
     -7.01661e-14
     -6.92779e-14
     -6.92779e-14
     -7.10543e-14

The analytical solution preserves the energy; i.e., $\delta E(t)=0$ for the analytical solution. Thus, we expect our solution to be close to zero. So, even if the $\delta E$ is not perfectly zero during the whole integration, it is comparable to Julia's `Float64` machine-epsilon.

Below, we plot $\delta E$ in units of Julia's `Float64` machine-epsilon as a function of time, $t$. In Julia, the machine-epsilon for the `Float64` type has a value

```julia
eps(Float64)
```

    2.220446049250313e-16

```julia
plot(

t/(2π), δE/eps(Float64),
title = L"\mathrm{Energy\;\; relative\;\; error\;\; vs\;\; time}",
xaxis = (L"t\mathrm{\,(orbital\;\;periods)}"),
yaxis = (L"\delta E \mathrm{\;\;(machine\;\;epsilons)}"),
label = L"\mathrm{Energy\;\; relative\;\; error\;\; vs\;\; time}"

)
```

Now, how does the energy error distribute around zero?

```julia
histogram(

δE/eps(Float64),
title = L"\mathrm{Distribution\;\;of\;\;energy\;\;relative\;\;error}",
xaxis = (L"\delta E"),
yaxis = (L"\mathrm{Number\;\;of\;\;}\delta E\mathrm{\;\;values\;\;within\;\;bin\;\;range}"),
leg = false,
nbins=20

)
```

    INFO: binning = 20

The energy error behaves roughly as a random walk, which means that the numerical error in the energy is dominated by rounding errors.

## Angular momentum conservation

Analogously to the energy, we now focus on the angular momentum, which is preserved by the analytical solution too. The value of the angular momentum is given by
$$
\begin{align}
L &= xv-yu
\end{align}
$$
We write the angular momentum function as:

```julia
ang_mom(x_, y_, u_, v_) = x_.*v_-y.*u_
```

    ang_mom (generic function with 1 method)

So the angular momentum during each time-step is:

```julia
L = ang_mom(x, y, u, v);
```

$L_0=x_0v_0-y_0u_0$, the initial value of the angular momentum, is:

```julia
L0 = L[1]
```

    0.5999999999999999

We define $\delta L$ as the relative error in the angular momentum; i.e.:

$$
\begin{align}
\delta L(t) &= \frac{L(t)-L_0}{L_0}
\end{align}
$$

```julia
δL = (L-L0)/L0;
```

Just as we did for the energy, we now plot $\delta L$ in units of `eps(Float64)` vs $t$:

```julia
plot(

t/(2π), δL/eps(Float64),
title = L"\mathrm{Angular\;\;momentum\;\; relative\;\; error\;\; vs\;\; time}",
xaxis = (L"t\mathrm{\,(orbital\;\;periods)}"),
yaxis = (L"\delta L \mathrm{\;\;(machine\;\;epsilons)}"),
label = L"\mathrm{Angular\;\;momentum\;\; relative\;\; error\;\; vs\;\; time}",
leg=false,
color=:orange

)
```

Again, we see that the angular momentum relative error is comparable to `eps(Float64)`; the maximum variation is ~$150$ machine epsilons. How does the distribution of this error look like?

```julia
histogram(

δL/eps(Float64),
title = L"\mathrm{Distribution\;\;of\;\;angular\;\;momentum\;\;relative\;\;error}",
xaxis = (L"\delta L"),
yaxis = (L"\mathrm{Number\;\;of\;\;}\delta L\mathrm{\;\;values\;\;within\;\;bin\;\;range}"),
leg = false,
nbins=20,
color=:orange

)

```

    INFO: binning = 20

The distribution of $\delta L$ shows a peak near zero and roughly symmetrical around this value. This means that the error in the angular momentum is dominated also by rounding errors of floating-point arithmetic.

Lastly, we reproduce __exactly__ the last plot shown in the original version of this example, authored by Luis Benet and included in `JuliaDiff/TaylorSeries.jl`'s [Kepler problem integration example](http://nbviewer.jupyter.org/github/JuliaDiff/TaylorSeries.jl/blob/master/examples/1-KeplerProblem.ipynb) jupyter notebook.

A $\delta E$, $\delta L$ plot vs $t$:

```julia
plot(

[t/(2π), t/(2π)], [δE/eps(Float64), δL/eps(Float64)],
title = L"\delta E\mathrm{\;\;(blue),\;\;}\delta L\mathrm{\;\;(green)\;\;vs\;\;time}",
xaxis = (L"t\mathrm{\,(orbital\;\;periods)}"),
yaxis = (L"\delta E\mathrm{,\;\;}\delta L\;\;\mathrm{(machine\;\;epsilons)}"),
leg=false,

)
```

```julia

```


# Periodic Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Spectrum

Periodic signals are an import class of signals. Many practical signals can be approximated reasonably well as periodic functions. This holds especially when considering only a limited time-interval. Examples of periodic signals are superpositions of harmonic signals, signals captured from vibrating structures or rotating machinery, as well as speech signals or signals from musical instruments. The spectrum of a periodic signal exhibits specific properties which are discussed in the following.

### Representation

A [periodic signal](https://en.wikipedia.org/wiki/Periodic_function) $x(t)$ is a signal that repeats its values in regular periods. It has to fulfill

\begin{equation}
x(t) = x(t + n \cdot T_\text{p})
\end{equation}

for $n \in \mathbb{Z}$ where its period is denoted by $T_\text{p} > 0$. A signal is termed *aperiodic* if is not periodic. One period $x_0(t)$ of a periodic signal is given as

\begin{equation}
x_0(t) = \begin{cases}
x(t) & \text{for } 0 \leq t < T_\text{p} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

A periodic signal can be represented by [periodic summation](https://en.wikipedia.org/wiki/Periodic_summation) of shifted copies of one period $x_0(t)$

\begin{equation}
x(t) = \sum_{\mu = - \infty}^{\infty} x_0(t - \mu T_\text{p})
\end{equation}

which can be rewritten as convolution

\begin{equation}
x(t) = \sum_{\mu = - \infty}^{\infty} x_0(t) * \delta(t - \mu T_\text{p}) = x_0(t) * \sum_{\mu = - \infty}^{\infty} \delta(t - \mu T_\text{p})
\end{equation}

using the sifting property of the Dirac impulse. It can be concluded that a periodic signal can be represented by one period $x_0(t)$ of the signal convolved with a series of Dirac impulses.

**Example**

The cosine signal $x(t) = \cos (\omega_0 t)$ has a periodicity of $T_\text{p} = \frac{2 \pi}{\omega_0}$. One period is given as

\begin{equation}
x_0(t) = \cos (\omega_0 t) \cdot \text{rect} \left( \frac{t}{T_\text{p}} - \frac{T_\text{p}}{2} \right)
\end{equation}

Introduced into above representation of a periodic signal yields

\begin{align}
x(t) &= \cos (\omega_0 t) \cdot \text{rect} \left( \frac{t}{T_\text{p}} - \frac{T_\text{p}}{2} \right) * \sum_{\mu = - \infty}^{\infty} \delta(t - \mu T_\text{p}) \\
&= \cos (\omega_0 t) \sum_{\mu = - \infty}^{\infty} \text{rect} \left( \frac{t}{T_\text{p}} - \frac{T_\text{p}}{2} - \mu T_\text{p} \right) \\
&= \cos (\omega_0 t)
\end{align}

since the sum over the shifted rectangular signals is equal to one.

### The Dirac Comb

The sum of shifted Dirac impulses, as used above to represent a periodic signal, is known as [*Dirac comb*](https://en.wikipedia.org/wiki/Dirac_comb). The Dirac comb is defined as

\begin{equation}
{\bot \!\! \bot \!\! \bot}(t) = \sum_{\mu = - \infty}^{\infty} \delta(t - \mu)
\end{equation}

It is used for the representation of periodic signals and for the modeling of ideal sampling. In order to compute the spectrum of a periodic signal, the Fourier transform of the Dirac comb $\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \}$ is derived in the following. Fourier transformation of the left- and right-hand side of above definition yields

\begin{equation}
\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \} = \sum_{\mu = - \infty}^{\infty} e^{-j \mu \omega}
\end{equation}

The exponential function $e^{-j \mu \omega}$ for $\mu \in \mathbb{Z}$ is periodic with a period of $2 \pi$. Hence, the Fourier transform of the Dirac comb is also periodic with a period of $2 \pi$.

In order to gain further insight, the following convolution of a [rectangular signal](../notebooks/continuous_signals/standard_signals.ipynb#Rectangular-Signal) with a Dirac comb is considered

\begin{equation}
{\bot \!\! \bot \!\! \bot}(t) * \text{rect}(t) = 1
\end{equation}

The right hand side follows from the fact that the rectangular signals equals one for $-\frac{1}{2} < t < \frac{1}{2}$ which is then periodically summed up with a period of one. Fourier transform of the left- and right-hand side yields

\begin{equation}
\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \} \cdot \text{sinc}\left(\frac{\omega}{2}\right) = 2 \pi \delta(\omega)
\end{equation}

For $\text{sinc}( \frac{\omega}{2} ) \neq 0$, which is equal to $\omega \neq 2 n \cdot \pi$ with $n \in \mathbb{Z} \setminus \{0\}$, this can be rearranged to

\begin{equation}
\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \} = 2 \pi \, \delta(\omega) \cdot \frac{1}{\text{sinc}\left(\frac{\omega}{2}\right)} = 2 \pi \, \delta(\omega)
\end{equation}

Note that the [multiplication property](../continuous_signals/standard_signals.ipynb#Dirac-Impulse) of the Dirac impulse and $\text{sinc}(0) = 1$ has been used to derive the last equality. The Fourier transform is now known in the interval $-2 \pi < \omega < 2 \pi$. It has already been concluded that the Fourier transform is periodic with a period of $2 \pi$. Hence, the Fourier transformation of the Dirac comb can be derived by periodic continuation

\begin{equation}
\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \} = \sum_{\mu = - \infty}^{\infty} 2 \pi \, \delta(\omega - 2 \pi \mu) = \sum_{\mu = - \infty}^{\infty} 2 \pi \, \left( \frac{\omega}{2 \pi} - \mu \right)
\end{equation}

The last equality follows from the scaling property of the Dirac impulse. Using the definition of the Dirac comb, the Fourier transform can now be rewritten in terms of the Dirac comb

\begin{equation}
\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \} = {\bot \!\! \bot \!\! \bot} \left( \frac{\omega}{2 \pi} \right)
\end{equation}

The Fourier transform of a Dirac comb with unit distance between the Dirac impulses is a Dirac comb with a distance of $2 \pi$ between the Dirac impulses which are weighted by $2 \pi$.

**Example**

The following example computes the truncated series

\begin{equation}
X(j \omega) = \sum_{\mu = -M}^{M} e^{-j \mu \omega}
\end{equation}

as approximation of the Fourier transform $\mathcal{F} \{ {\bot \!\! \bot \!\! \bot}(t) \}$ of the Dirac comb. For this purpose the sum is defined and plotted in `SymPy`.

```python
import sympy as sym
sym.init_printing()

mu = sym.symbols('mu', integer=True)
w = sym.symbols('omega', real=True)
M = 20

X = sym.Sum(sym.exp(-sym.I*mu*w), (mu, -M, M)).doit()
sym.plot(X, xlabel='$\omega$', ylabel='$X(j \omega)$',
         adaptive=False, nb_of_points=1000);
```

**Exercise**

* Change the summation limit $M$. How does the approximation change? Note: Increasing $M$ above a certain threshold may lead to numerical instabilities.

### Fourier-Transform

In order to derive the Fourier transform $X(j \omega) = \mathcal{F} \{ x(t) \}$ of a periodic signal $x(t)$ with period $T_\text{p}$, the signal is represented by one period $x_0(t)$ and the Dirac comb. Rewriting above representation of a periodic signal in terms of a sum of Dirac impulses by noting that $\delta(t - \mu T_\text{p}) = \frac{1}{T_\text{p}} \delta(\frac{t}{T_\text{p}} - \mu)$ yields

\begin{equation}
x(t) = x_0(t) * \frac{1}{T_\text{p}} {\bot \!\! \bot \!\! \bot} \left( \frac{t}{T_\text{p}} \right)
\end{equation}

The Fourier transform is derived by application of the [convolution theorem](../fourier_transform/theorems.ipynb#Convolution-Theorem)

\begin{align}
X(j \omega) &= X_0(j \omega) \cdot {\bot \!\! \bot \!\! \bot} \left( \frac{\omega T_\text{p}}{2 \pi} \right) \\
&= \frac{2 \pi}{T_\text{p}} \sum_{\mu = - \infty}^{\infty} X_0 \left( j \, \mu \frac{2 \pi}{T_\text{p}} \right) \cdot
\delta \left( \omega - \mu \frac{2 \pi}{T_\text{p}} \right)
\end{align}

where $X_0(j \omega) = \mathcal{F} \{ x_0(t) \}$ denotes the Fourier transform of one period of the periodic signal. From the last equality it can be concluded that the Fourier transform of a periodic signal consists of a series of weighted Dirac impulses. These Dirac impulse are equally distributed on the frequency axis $\omega$ at an interval of $\frac{2 \pi}{T_\text{p}}$. The weights of the Dirac impulse are given by the values of the spectrum $X_0(j \omega)$ of one period at the locations $\omega = \mu \frac{2 \pi}{T_\text{p}}$. Such a spectrum is termed *line spectrum*.

### Parseval's Theorem

[Parseval's theorem](../fourier_transform/theorems.ipynb#Parseval%27s-Theorem) relates the energy of a signal in the time domain to its spectrum. The energy of a periodic signal is in general not defined. This is due to the fact that its energy is unlimited, if the energy of one period is non-zero. As alternative, the average power of a periodic signal $x(t)$ is used. It is defined as

\begin{equation}
P = \frac{1}{T_\text{p}} \int_{0}^{T_\text{p}} |x(t)|^2 \; dt
\end{equation}

Introducing the Fourier transform of a periodic signal into [Parseval's theorem](../fourier_transform/theorems.ipynb#Parseval%27s-Theorem) yields

\begin{equation}
\frac{1}{T_\text{p}} \int_{0}^{T_\text{p}} |x(t)|^2 \; dt = \frac{1}{T_\text{p}} \sum_{\mu = - \infty}^{\infty} \left| X_0 \left( j \, \mu \frac{2 \pi}{T_\text{p}} \right) \right|^2
\end{equation}

The average power of a periodic signal can be calculated in the time-domain by integrating over the squared magnitude of one period or in the frequency domain by summing up the squared magnitude weights of the coefficients of the Dirac impulses of its Fourier transform.

### Fourier Transform of the Pulse Train

The [pulse train](https://en.wikipedia.org/wiki/Pulse_wave) is commonly used for power control using [pulse-width modulation (PWM)](https://en.wikipedia.org/wiki/Pulse-width_modulation). It is constructed from a periodic summation of a rectangular signal $x_0(t) = \text{rect} (\frac{t}{T} - \frac{T}{2})$

\begin{equation}
x(t) = \text{rect} \left( \frac{t}{T} - \frac{T}{2} \right) * \frac{1}{T_\text{p}} {\bot \!\! \bot \!\! \bot} \left( \frac{t}{T_\text{p}} \right)
\end{equation}

where $0 < T < T_\text{p}$ denotes the width of the pulse and $T_\text{p}$ its periodicity. Its usage for power control becomes evident when calculating the average power of the pulse train

\begin{equation}
P = \frac{1}{T_\text{p}} \int_{0}^{T_\text{p}} | x(t) |^2 dt = \frac{T}{T_\text{p}}
\end{equation}

The Fourier transform of one period $X_0(j \omega) = \mathcal{F} \{ x_0(t) \}$ is derived by applying the scaling and shift theorem of the Fourier transform to the [Fourier transform of the retangular signal](../fourier_transform/definition.ipynb#Transformation-of-the-Rectangular-Signal)

\begin{equation}
X_0(j \omega) = e^{-j \omega \frac{T}{2}} \cdot T \, \text{sinc} \left( \frac{\omega T}{2} \right)
\end{equation}

from which the spectrum of the pulse train follows by application of above formula for the Fourier transform of a periodic signal

\begin{equation}
X(j \omega) = 2 \pi \frac{1}{T_\text{p}} \sum_{\mu = - \infty}^{\infty} e^{-j \mu \pi \frac{T}{T_\text{p}}} \cdot T \, \text{sinc} \left( \mu \pi \frac{T}{T_\text{p}} \right) \cdot \delta \left( \omega - \mu \frac{2 \pi}{T_\text{p}} \right)
\end{equation}

**Example**

The pulse train and its spectrum are illustrated by the subsequent computational example. First the pulse train is defined and plotted in `SymPy`

```python
mu = sym.symbols('mu', integer=True)
t = sym.symbols('t', real=True)
T = 2
Tp = 5

def pulse_train(T, Tp):
    n = sym.symbols('n', integer=True)
    x0 = sym.Piecewise((0, t < 0), (1, t < T), (0, True))
    return sym.summation(x0.subs(t, t+n*Tp), (n, -10, 10))

sym.plot(pulse_train(T, Tp), (t, -5, 20), xlabel=r'$t$', ylabel=r'$x(t)$');
```

The weights of the Dirac impulses are defined for fixed values $T$ and $T_\text{p}$

```python
X_mu = sym.exp(-sym.I * mu * sym.pi * T/Tp) * T * sym.sinc(mu * sym.pi * T/Tp)
X_mu
```

The weights of the Dirac impulses are plotted with [`matplotlib`](http://matplotlib.org/index.html), a Python plotting library. The library expects the values of the function to be plotted at a series of sampling points. In order to create these, the function [`sympy.lambdify`](http://docs.sympy.org/latest/modules/utilities/lambdify.html?highlight=lambdify#sympy.utilities.lambdify) is used which numerically evaluates a symbolic function at given sampling points. The resulting plot illustrates the positions and weights of the Dirac impulses.

```python
import numpy as np
import matplotlib.pyplot as plt

Xn = sym.lambdify(mu, sym.Abs(X_mu), 'numpy')

n = np.arange(-15, 15)
plt.stem(n*2*np.pi/Tp, Xn(n))
plt.xlabel('$\omega$')
plt.ylabel('$|X(j \omega)|$');
```

**Exercise**

* Change the ratio $\frac{T}{T_\text{p}}$. How does the spectrum of the pulse train change?
* Can you derive the periodicity $T_\text{p}$ of the signal from its spectrum?
* Calculate the average power of the pulse train in the frequency domain by applying Parseval's theorem.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Information Theory Lab 04: Source Coding,  Part I: Encoding

## About

This file is designed to be viewed and run online in a browser.

This file is a Jupyter Notebook file usign `xeus-cling`, a Jupyter kernel for C++ based on the `cling` C++ interpreter and the `xeus` native implementation of the Jupyter protocol, xeus.

- GitHub repository: https://github.com/jupyter-xeus/xeus-cling/
- Online documentation: https://xeus-cling.readthedocs.io/

## Usage

To run a selected code cell:

- Ctrl  + Enter = Run cell and remain at current cell
- Shift + Enter = Run cell and advance to next cell

## 1. Objective

Understand linear block codes by implementing a basic encoding application.

## 2. Practical considerations, Part I: Working with bits

### Read individual bits in a variable

In C we can read and modify **individual bits** in a variable by applying a AND / OR / XOR mask.

Consider a number $x$ with binary representation `11101101`. To read the value of bit number 3 (from right to left, LSB to MSB), we apply the following **binary mask** with a single 1 on the position of the bit we want to read:
$$
\begin{align}
1110b101&&  \\
    \&  \\
00001000&&  \\
    =   &&  \\
0000b000&&
\end{align}
$$

The result is 0 (e.g False) if that bit in $x$ has value 0, and is a non-zero value (e.g True) if that bit is 1.

We can do this in C as follows:

```c++
int x = 78745;
int bit = ((x & (1U << 0)) != 0);  // Read bit from position 3 (right to left)

// Display the result
bit
```

    1

Explained:
- x is the number
- `1U << 3` means `00000001` (number 1, unsigned) shifted to the left with 3 positions, which produces the mask `00001000`
- we have a bitwise AND (operator `&`) between `x` and this mask `00001000`
- we obtain 0 or non-zero (but not necessarily 1); we compare it with 0 so that in the end we get either a False (0) or a True (1).

To use it multiple times, we can encode this operation with a **macro** defined as:

```c++
#define READ_BIT(x,i)       (int)(((x) & (1U << (i))) != 0)

// Let's use this macro:
READ_BIT(78740, 8)          // read bit 3 from number 78745
```

    1

#### **Exercise**: convert number to binary
Find the binary representation of number 212 (run READ_BIT 8 times and note down the results).

```c++
// TODO: run READ_BIT
READ_BIT(212, 0)
```

    0

### Set individual bits in a variable (= make 1)

We can change a certain bit in a variable to 1 ("set the bit") by using a bitwise OR with a similar mask:
$$
\begin{align}
bbbbbbbb&&  \\
    |  \\
00010000&&  \\
    =   &&  \\
bbb1bbbb&&
\end{align}
$$

Every (bit OR 0) leaves that bit value unchanged, but when we make OR with 1 the result is 1.

We can package this as a macro as well. The macro does not return anything, but it changes bit number $i$ from variable $x$ to 1.

```c++
// Set to 1 the bit number i from value x
#define SET_BIT(x,i)        ((x) = (x) | (1U << (i)))

int x = 15;
SET_BIT(x, 7)   // set to 1 the bit number 7. Will print the result
```

    143

#### **Exercise**: convert binary to base-10 number
Find the value defined in binary as `10100000`. Start from `00000000` and use SET_BIT to set the corresponding bit positions.

```c++
int x = 0;

// TODO: set bits

// Display the value
x
```

    0

### Clear individual bits in a variable (= make 0)

We can change a certain bit in a variable to 0 ("clear the bit") by using a bitwise AND with an inverted mask:
$$
\begin{align}
bbbbbbbb&&  \\
    \&  \\
11101111&&  \\
    =   &&  \\
bbb0bbbb&&
\end{align}
$$

Every (bit AND 1) leaves that bit value unchanged, but when we make AND with 0 the result is 0.

We can package this as a macro as well.

```c++
// Clear (make 0) the bit number i from value x
#define CLEAR_BIT(x,i)      ((x) = (x) & ~(1U << (i)))     // note the bineary negation operator ~ which inverts every bit of the mask

int x = 240;
CLEAR_BIT(x, 7)   // clear bit number 7. Will print the result
```

    112

### Writing individual bits in a variable (= set them to a given value)

We can combine SET_BIT and CLEAR_BIT into a common macro WRITE_BIT which takes the desired value as an argument. The macro checks the desired value, and decides:
- if we want a 1, it calls SET_BIT
- if we want a 0, it calls CLEAR_BIT

```c++
#define WRITE_BIT(x,i,val)  ((val) ? SET_BIT((x),(i)) : CLEAR_BIT((x),(i)))

int x = 150;
WRITE_BIT(x, 5, 0);    // Make bit number 5 from x equal to 0
WRITE_BIT(x, 6, 1);    // And then make bit number 6 from x equal to 1
```

    214

### Toggle individual bits in a variable (= make opposite)

We can toggle a certain bit in a variable by using a bitwise XOR with a similar mask:
$$
\begin{align}
bbbbbbbb&&  \\
    \hat{}  \\
00010000&&  \\
    =   &&  \\
bbb\overline{b}bbbb&&
\end{align}
$$

Every (bit XOR 0) leaves that bit value unchanged, but when we make XOR with 1 the result is the opposite value $\overline{b}$ to the original $b$.

We can package this as a macro as well.

```c++
#define TOGGLE_BIT(x,i)     ((x) = (x) ^ (1U << (i)))

int x = 150;
TOGGLE_BIT(x, 0);    // Toggle bit number 0 from x
```

    151

### Working with bit vectors

We'll often have to work with very long sequences of bits. We cannot use a single variable for holding all of them. Instead, we will usually define a **vector of bytes** (unsigned char). Let's define a vector with 1000 bytes, capable of holding 8000 individual bits.

```c++
unsigned char vector[1000];   // 1000 bytes = 8000 bits
```

We need a quick way to read and write but values from such a vector.

How do we read the bit on position 14 from this vector? We figure out on which element (byte) in the vector falls bit number 14, and which of the 8 bits of that element it is:
- $14 \; / \; 8 = 1.75$, so we know position 2166 is somewhere in the second element of the vector, e.g. in `vector[1]`, i.e. `vector[position/8]`
- $14 \; \% \; 8 = 6$, so we know it's the 6th bit inside `vector[1]`

Thus, in general, we can access bit number $i$ from the whole vector by accessing bit number `i%8` from element `vector[i/8]`. We can use READ_BIT and WRITE_BIT to read and write at this position.

We define two macros VECREAD_BIT and VECWRITE_BIT to read and write bit values **in a whole vector**, as follows:

```c++
// Read bit number i from vector v
#define VECREAD_BIT(v,i)       (READ_BIT((v[(i)/8]),(i)%8))

// Write value val in bit number i from vector v
#define VECWRITE_BIT(v,i,val)  (WRITE_BIT((v[(i)/8]),((i)%8),val))
```

#### **Exercise**: read / write bits in a vector

Consider a vector with 20 bytes (160 bits). Make the first 100 bit values all equal to 1, and the remaining 60 bits all equal to 0

```c++
unsigned char vec[20];

// TODO: write here
for (int i = 0; i < 100; i++)
    VECWRITE_BIT(vec, i, 1);

for (int i = 100; i < 160; i++)
    VECWRITE_BIT(vec, i, 0);

//-------------------------
// Check: display vec[11], vec[12], vec[13]
// Should be 255, 15 and 0
printf("vec[11] = %d \nvec[12] = %d \nvec[13] = %d", vec[11], vec[12], vec[13]);
```

    vec[11] = 255
    vec[12] = 15
    vec[13] = 0

## 2. Practical considerations, Part II: Encoding

Encoding a certain sequence of messages with a code means replacing each message $s_i$ with its codeword $c_i$.

In this lab, we will replace **each character** with a specially crafted code. We will build this code in Lab 6.

### Codewords

In these labs, a codeword is defined using the following structure data type:
```
typedef struct
{
    int len;                /* length of code, in bits */
    unsigned long code;     /* the first "len" bits are the codeword */
} CODE32BIT;
```

All the codewords are available as a vector in file `codero.dat`. We can read such a vector with `fread()`, as follows

```c++
// Define the structure type
typedef struct
{
    int len;               /* length of code, in bits */
    unsigned int code;     /* the first "len" bits are the codeword */
} CODE32BIT;

// Define a vector of 256 codewords, one for each ASCII character
CODE32BIT cb[256];

// Open the file
FILE* f = fopen("codero.dat", "rb");

// Read the vector from the file
fread(&cb[0], sizeof(CODE32BIT), 256, f);   // Read from f, 256 elements, each of size "sizeof(CODE32BIT)" bytes, and place them in cb

fclose(f);
//sizeof(unsigned int)
```

    0

From the structure definition we see that each codeword element has two components:
- `len`: length of the codeword (number of bits)
- `code`: the actual bits of the codeword (only the first `len` bits are written)

Let's check the codeword for letter `a`, which is `cb[97]` (ASCII code of 'a is 97).

The codeword has length:

```c++
cb['a'].len    // instead of its code 97 we can use directly char 'a'
```

    3

The bits in `a`'s codeword  are:

```c++
READ_BIT(cb['a'].code, 0)  // First bit
```

    0

```c++
READ_BIT(cb['a'].code, 1)  // Second bit
```

    0

```c++
READ_BIT(cb['a'].code, 2) // Third bit
```

    1

#### **Exercise**: compare codeword lengths

Check the codeword lengths for `a`, `b`, and `x`. Can you think of a reason why they would be different?

```c++
// TODO: write here
cb['z'].len
//READ_BIT(cb['b'].code, 1)
```

    8

#### **Exercise**: print one codeword

Write all the bits of the codeword of `z` into the vector `vec`, one by one, using a for loop:

```c++
unsigned char vec[20];  // vector of 20 bytes, has space for 160 bits

// TODO: write below
for (int i=0; i < cb['z'].len; i++)
{
    int bit = READ_BIT(cb['z'].code, i);  // Read every bit from codeword of 'z'
    VECWRITE_BIT(vec, i, bit);            //  and save it into vector 'vec'
}

// Display vector
vec
```

    { '}', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00', '0x00' }

### Encoding procedure

For encoding a sequence of letters, we simply write the codeword of every letter into an output vector.

#### **Exercise**: encode a sentence

Encode the sentence *"Humpty-Dumpty sat on a wall"*, and write the binary output sequence in the vector `vec`.

Questions:
  - how many bits did we use?
  - how many bits would be used if we encode the letters with ASCII code (8 bits / letter)?
  - what is the compression ration achieved with this code?

```c++
const char* s = "Humpty-Dumpty sat on a wall";          // the input sequence
unsigned char vec[1000];                                // output vector for holding the bits

// TODO: write here

```

#### **Exercise**: encode a file

Read a text file and encode every byte (character), writing the binary output sequence in the vector `vec`. Reuse the code from the previous labs in order to open the file and read every byte.

```c++
const char* filename = "textro.txt";    // the text file to encode
unsigned char vec[1000000];                 // one milion bytes, holds up to 1MB of data

// TODO: write here

```

### Writing the encoded bit sequence to an output file

Finally, once we have all the bits in the vector `vec`, let's write the data into an output file.

From the encoding above, we need **the total number of bits written**. Let's call this `len`. We save the data as follows:
1. Open the output file with `fopen()`, for writing in binary mode
2. Write the integer `len`, using `fwrite()`
3. Write the vector `vec` using `fwrite()`, but only the bytes actually written (number of bytes to write = `ceil (len/8)`)
4. Close the file

```c++
//int len = 5;

FILE* f = fopen("textroencoded.enc", "wb");                      // open file
fwrite(&len, sizeof(len), 1, f);                           // write len
fwrite(vec, sizeof(unsigned char), ceil(len/8), f);        // write the encoded bitstring, only the actual written bytes ceil(len/8)
fclose(f);                                                 // close file
```

## 3. Final Exercises

1. Put everything into a dedicated program `encode.c`, to encode every byte from a given data file.

   The program shall be called as follows:

   `encode.exe codero.dat input.txt output.txt`

   The arguments are:
      - `code.dat`: a file containing the code to be used (known as the "codebook" file)
      - `input.txt`: the file to encode
      - `output.txt`: the output (encoded) file

   The codebook file contains a vector of 256 elements of the following structure type:

        typedef struct
        {
            int len;                /* length of code, in bits */
            unsigned long code;     /* the first "len" bits are the codeword */
        } CODE32BIT;

   The program will follow the following steps:

    - Allocate an array named `out` of `unsigned char` of max size 10MB (i.e. 10000000 bytes);
    - Open and read the full vector from the codebook file;
    - Then, open the input file and read every byte in a loop. For each byte do the following:

        - Write the codeword for the byte, bit by bit, in the `out` vector. Use the `VECWRITE_BIT()` macro

        - Keep track of the number of bits written, in order to continue writing from where the previous codeword stopped.

    - Write the array `out` to the output file, as follows:

        - Open the output file for writing

        - Write first the total number of bits
        - Write afterwards the vector `out`, but only the number of bytes actually used for coding
        - *Note: when decoding the file, we will read back the data in the same order*.

2. Encode the file `textro.txt` with the provided codebook `codero.dat`. Check the size of the output file and compute the compression ratio.

3. Repeat 2. for `texten.txt` with codebook `codeen.dat`.

4. Encode a file with the codebook from the other language. Check the size of the output file and compute the compression ratio. Compare
with the one using the same language codebook. Which case is better?

## 4. Final questions

1. TBD