# Mathematical intuition of gradient descent
> Understanding and coding gradient descent from scratch

- toc: true
- badges: true
- comments: true
- author: Gieun Kwak
- categories: [optimization, gradient descent]

```python
# collapse

import matplotlib.pyplot as plt
import numpy as np
from sympy import *
# from IPython.display import set_matplotlib_formats
# import matplotlib_inline.backend_inline
# matplotlib_inline.backend_inline.set_matplotlib_formats('png')
```

### 1. How gradient descent looks like

Gradient descent is commonly used in ML and AI. Looking at the image below, we encounter from lectures that the gradient descent is a 'process of finding the most lowest point of the function'. While this is a very concise and accurate explanation of gradient descent, understanding it in a mathematical way would enable a deeper understanding.

### 2. How gradient descent works: simple maths

The gradient descent image above goes through a line until it reaches the 'bottom' of the function. This 'bottom' of the function is called 'minimum'. We want to reach the minimum because it will give us the best values for our problem. One of the common use of gradient is when we estimate the parameters of a ML or AI model. We want to find the paramaters that has the smalles loss value (closes to the actual observation).

Lets consider a function $y = (x-5)^2$. We want to find the minimum value of this function by following a sequence of steps.
1. Start from a certain point
2. Move to a new point that gives a better objective value (here, smaller values)
3. Repeat step 2

```python
# collapse

# x = np.arange(2, 8, 0.01)
# def y(x):
#     return (x-5)**2

# fig, ax = plt.subplots()
# ax.plot(x, y(x))
# ax.set_xlabel("x")
# ax.set_ylabel("y")
# ax.plot([5], y(5), marker='o', color='black')

# fig.savefig('my_icons/03_/one.png');
```

One thing most explanations exclude is the constraint. Actually, for every optimization problem, there is an objective function and a constraint. The objective function is $y = (x-5)^2$. And the constraint is hidden, which is $x \in R$, $y \in R$ ($R$ means real numbers).

The optimization problem changes along the constraints, but for now lets say the constraint is $y \geq (x-5)^2$. This means the set of potential solutions for this problem is all the points above the function line.

The process of gradient descent will only happen inside the blue area (points that satisfy the constraints)

```python
# collapse

# x = np.arange(2, 8, 0.01)
# def y(x):
#     return (x-5)**2

# fig, ax = plt.subplots()
# ax.plot(x, y(x))
# ax.set_xlabel("x")
# ax.set_ylabel("y")

# ax.fill_between(x, y(x), 9, color='blue', alpha=.1)
# ax.plot([5], y(5), marker='o', color='black')

# fig.savefig('my_icons/03_/two.png')

```

Now lets say we start at a point where $x=7$ and $y=4$

```python
# collapse

# x = np.arange(2, 8, 0.01)
# def y(x):
#     return (x-5)**2

# fig, ax = plt.subplots()
# ax.plot(x, y(x))
# ax.set_xlabel("x")
# ax.set_ylabel("y")
# ax.fill_between(x, y(x), 9, color='blue', alpha=.1)

# ax.plot([7], y(7), marker='o', color='red')
# ax.plot([5], y(5), marker='o', color='black')

# fig.savefig('my_icons/03_/three.png')
```

This red dot can go anywhere as long as it is in the blue area. Lets say this dot moves to $(6,6)$. This point is still in the blue area and satisfies the constraint we have set. This is called feasibility.

- __The process of gradient descent ONLY considers the solutions that are feasible (satisfies constraints)__

```python
# collapse

# x = np.arange(2, 8, 0.01)
# def y(x):
#     return (x-5)**2

# fig, ax = plt.subplots()
# ax.plot(x, y(x))
# ax.set_xlabel("x")
# ax.set_ylabel("y")
# ax.fill_between(x, y(x), 9, color='blue', alpha=.1)

# ax.plot([7], y(7), marker='o', color='red')
# ax.plot([6], 6, marker='o', color='green')
# ax.plot([6], 2, marker='o', color='blue')
# ax.plot([5], y(5), marker='o', color='black')

# ax.arrow(7, y(7), 6-7+0.1, 2-y(7)+0.1, head_width=0.1, head_length=0.1);

# fig.savefig('my_icons/03_/four.png')
```

While the point $(6,6)$ is feasible, does it go to the optimal value? The optimal value we want to find is the smallest point of $y=(x-5)^2$.

Y value when x=7 : 4

Y value when x=6 : 6

The value of the function rather increased, and it means that this is not the right direction.

If the red dot goes to the direction of blue dot, the objective function decreases. This then would be the right direction we want to go. And the 'amount' we want to move is the learning rate($\lambda$) which is a small step towards the direction.

- __The solution for each step is updated by $x^1 = x^0 + \lambda d^0$, and the direction is where the optimal value gets closer to the global minimum.__

### 3. How is gradient calculated?

There are two aspects in calculating gradient. First is the 'direction', and the second is the 'distance'.
 Lets say $d$ is for firection and $\alpha$ for step size ('distance'). Then this could be formulated as the following: $f(x^k + \alpha d^k) < f(x^k)$, which means the new value is smaller than the current function value.

Using Taylor's expanson, it could be formulated as $f(x^k + \alpha d^k) \approx f(x^k) + \alpha\nabla f(x^k)^T d_k$

Since $f(x^k + \alpha d^k) < f(x^k)$, we want $\alpha\nabla f(x^k)^T d_k$ to be smaller than zero. The steepest direction $d_k$ would be $-\nabla f(x^k)$ since it is the opposite direction from $\nabla f(x^k)^T$

Then, the new point is updated every step by $x^{k+1} \leftarrow x_k - \alpha \nabla f(x_k)$

This process will continue until $||\nabla f(x_k)|| \leq \epsilon$ where $\epsilon$ is a very small number. This is the point where the gradient is small enough or vanishes. This means that there is no more 'movement'.

```python
# collapse

# x = np.arange(2, 8, 0.01)
# def y(x):
#     return (x-5)**2

# fig, ax = plt.subplots()
# ax.plot(x, y(x))
# ax.set_xlabel("x")
# ax.set_ylabel("y")
# ax.fill_between(x, y(x), 9, color='blue', alpha=.1)

# ax.plot([7], y(7), marker='o', color='red')
# ax.plot([5], y(5), marker='o', color='black')

# def line1(x):
#     return 4*x - 24
# def line2(x):
#     return x*0

# ax.plot(np.arange(6, 8, 0.01), line1(np.arange(6, 8, 0.01)), color='red')
# ax.plot(x, line2(x), color='black');
# fig.savefig('my_icons/03_/five.png')
```

The red dot has a negative gradient, and it moves the point towards left. If the point reaches the black dot, the gradient is zero (vanishes), meaning there is no further direction for the point to move, indicating the end of the search.

### 4. Newton's method from scratch

One of the methods of gradient descent is Newton's method.

Let $x^k$ be the current state and consider a second order Taylor approximation of $f(x)$.

$g(x) = f(x^k)+\nabla f(x^k)^T (x-x^k) + \frac{1}{2}(x-x^k)\nabla^2f(x^k)(x-x^k)$

The next step would be to find the point that minimizes the function $g(x)$. That point would be where the first derivative equals to zero.
$\nabla g(x) = \nabla f(x^k)+\nabla^2f(x^k)(x-x^k) = 0$

Solving this equation for $x$, this gives us

$x = x^k - [\nabla^2f(x_k)]^{-1} \nabla f(x^k)$ which will be the new $x^{k+1}$

To formulate this into the code, here is the psuedo code.

#### Parameters

- $k$ is the indicator for current step
- $\epsilon$ is the minimum value for stopping the search
- $\alpha$ is the step size. $\alpha$ changes via line search. This means that we want to find a step size that actually leads to minimizing the function value
- $\rho$ and $c$ are scalars used to derive $\alpha$ during line search. They are between 0 and 1

#### Code walkthrough

- Set the initial values for $\alpha$, $d$, $k$. Initial starting point $x^0$ will be provided via input
- The search ends only if the first derivative of $f(x)$ is smaller than $\epsilon$. Here, it is actually the norm value of the vector.
- Inside the search, first determine perform line search to find $\alpha$
    - If $\alpha$ is found, update $x^{k+1}$ as well as $d^{k+1}$
- Continue until vanishing gradient or when it is small enough

#### Coding - from scratch

Lets solve the optimzal minimum value for the following function.
$$y = 100(x_2-x_1^2)^2 + (1-x_1)^2$$

```python
#collapse

def plot_contour_2d(X1, X2, y):
    fig, ax = plt.subplots(figsize=(12,7))

    c_plot = ax.contour(X1, X2, y)
    ax.clabel(c_plot, inline=True,
              fontsize=10);
    ax.set_title('2D contour plot');

def plot_contour_3d(X1, X2, y, save_dir):
    fig = plt.figure(figsize=(12,7))
    ax = plt.axes(projection='3d')
    ax.contour3D(X1, X2, y, 50, cmap='binary')
    ax.set_xlabel("x1")
    ax.set_ylabel("x2")
    ax.set_zlabel("y")
    ax.set_title('3D contour plot');
```

```python
# collapse

# X1 = np.arange(-50, 50, 0.1)
# X2 = np.arange(-50, 20, 0.1)
# X1, X2 = np.meshgrid(X1, X2)
# y = 100 * (X2 - X1**2)**2 + (1-X1)**2

# plot_contour_3d(X1, X2, y, 'my_icons/03_/3D_contour.png')
```

```python
# collapse

# define x and y
x1 = Symbol('x1')
x2 = Symbol('x2')
y = 100 * (x2 - x1**2)**2 + (1-x1)**2
```

```python
# collapse

print("Function Y")
y
```

    Function Y

$\displaystyle \left(1 - x_{1}\right)^{2} + 100 \left(- x_{1}^{2} + x_{2}\right)^{2}$

```python
# collapse

# in order for the function to be calculated with x input, it has to be 'lambdafied'
# for more information, see the package sympy
# it allows for place holders for the function, making it easy for us to read and understand

def f(x1_value, x2_value, x1=x1, x2=x2, function=y):
    """
    Input the values for x1 and x2, to the function
    """
    working_function = lambdify((x1,x2), function, 'numpy')
    return working_function(x1_value, x2_value)

def hessian_matrix(x1_value, x2_value, x1, x2, y):
    """
    Calculates the hessian matrix for 2 variable function.
    """
    matrix = np.zeros((2,2))
    for i, der1 in enumerate([x1,x2]):
        for j, der2 in enumerate([x1,x2]):
            matrix[i,j] = f(x1_value, x2_value, x1, x2, y.diff(der1).diff(der2))
    return matrix

def first_order_matrix(x1_value, x2_value, x1, x2, y):
    matrix = np.zeros((2,1))
    for i, der in enumerate([x1,x2]):
        matrix[i] = f(x1_value, x2_value, x1, x2, y.diff(der))
    return matrix
```

```python
# collapse

print("Partial derivative x1")
y.diff(x1)
```

    Partial derivative x1

$\displaystyle - 400 x_{1} \left(- x_{1}^{2} + x_{2}\right) + 2 x_{1} - 2$

```python
# collapse

print("Partial derivative x2")
y.diff(x2)
```

    Partial derivative x2

$\displaystyle - 200 x_{1}^{2} + 200 x_{2}$

```python
# collapse

print("Partial derivative x1, x2")
y.diff(x1).diff(x2)
```

    Partial derivative x1, x2

$\displaystyle - 400 x_{1}$

```python
# collapse

print("Partial derivative x2, x1")
y.diff(x2).diff(x1)
```

    Partial derivative x2, x1

$\displaystyle - 400 x_{1}$

Newton's method implemented

```python
def newton_iteration(x, x1, x2, y, verbose=False):
    """
    x: initial value
    """
    # initialization
    k = 0
    e = 10**(-4)
    # calculate hessian matrix and first derivative matrix
    hessian_m = hessian_matrix(x[0], x[1], x1, x2, y)
    first_order_m = first_order_matrix(x[0], x[1], x1, x2, y)
    # use them to calculate the gradient
    d = -1 * np.dot(np.linalg.inv(hessian_m), first_order_m)
    # calculate the l2 norm of gradient. used as a stopping criteria
    gradient_norm = np.linalg.norm(first_order_m)

    # initial value for alpha and parameters for line search
    alpha = 1
    rho = 0.2
    c = 0.4

    if verbose:
        print("Initialized algorithm.")
        print("step: {}\nd: {}\ngradient: {}\nthreshold: {}".format(k, d, gradient_norm, e))
        print("-"*40)

    # run until gradient is small enough
    while gradient_norm > e:
        # perform line search -> find appropriate alpha
        while f(x[0]+alpha*d[0], x[1]+alpha*d[1]) > f(x[0], x[1]) + c*alpha*np.dot(first_order_matrix(x[0], x[1], x1, x2, y).T, d):
            alpha *= rho
        # update x
        x = x + alpha*d
        # calculate updated hessian and first order matrix
        hessian_m = hessian_matrix(x[0], x[1], x1, x2, y)
        first_order_m = first_order_matrix(x[0], x[1], x1, x2, y)
        # update d
        d = -1 * np.dot(np.linalg.inv(hessian_m), first_order_m)
        # update norm value of gradient
        gradient_norm = np.linalg.norm(first_order_m)
        k += 1
        function_value = f(x[0], x[1])

        if verbose:
            print("step: {}\nalpha: {}\nx: {}\nupdated function value: {}".format(k, alpha, x, function_value[0]))
            print("d: {}\ngradient norm: {}\nthreshold: {}".format(d, gradient_norm, e))
            print()
    return x,k
```

```python
# collapse

x, k = newton_iteration(np.array([[1.1],[1.1]]), x1, x2, y, False)

print("Function converged at step {}. \nx = {}".format(k, x))
```

    Function converged at step 44.
    x = [[1.00000816]
     [1.00001616]]

If the starting point is $[x1,x2]^T=[1.2, 1.2]^T$, the function converges to $[x1,x2]^T\approx[1, 1]^T$ at step 44.

```python
# collapse

x, k = newton_iteration(np.array([[6],[4]]), x1, x2, y, False)

print("Function converged at step {}. \nx = {}".format(k, x))
```

    Function converged at step 491.
    x = [[1.00001367]
     [1.00002717]]

If the starting point is $[6, 4]^T$, it takes more iterations (491 steps), but the $x$ value is similar $[x1,x2]^T\approx[1, 1]^T$

Since the gradient is an iterative process, and optimizing a function get exponentially hard as the function gets complicated, the exact number would differ depending on where the iteration starts. That is also why there are local and global minimuns.

The gradient descent process is printed out for your reference. You can see the gradient decreasing as the function value decreases, while $x$ converges towards 1

```python
# collapse

x, k = newton_iteration(np.array([[1.1],[1.1]]), x1, x2, y, True)
```

    Initialized algorithm.
    step: 0
    d: [[-0.00434783]
     [ 0.10043478]]
    gradient: 53.34753977457635
    threshold: 0.0001
    ----------------------------------------
    step: 1
    alpha: 1
    x: [[1.09565217]
     [1.20043478]]
    updated function value: 0.009149374108868819
    d: [[-0.0952919 ]
     [-0.20879466]]
    gradient norm: 0.19962485729758325
    threshold: 0.0001

    step: 2
    alpha: 0.2
    x: [[1.07659379]
     [1.15867585]]
    updated function value: 0.0058809236765559915
    d: [[-0.07120573]
     [-0.15294095]]
    gradient norm: 0.3250473248156841
    threshold: 0.0001

    step: 3
    alpha: 0.2
    x: [[1.06235265]
     [1.12808766]]
    updated function value: 0.003913404243300558
    d: [[-0.05662774]
     [-0.11981177]]
    gradient norm: 0.35423959760700396
    threshold: 0.0001

    step: 4
    alpha: 0.2
    x: [[1.0510271 ]
     [1.10412531]]
    updated function value: 0.0026321371824487164
    d: [[-0.04611446]
     [-0.09640244]]
    gradient norm: 0.3429543838077912
    threshold: 0.0001

    step: 5
    alpha: 0.2
    x: [[1.04180421]
     [1.08484482]]
    updated function value: 0.0017737229394667728
    d: [[-0.03792668]
     [-0.07851316]]
    gradient norm: 0.3137556000113143
    threshold: 0.0001

    step: 6
    alpha: 0.2
    x: [[1.03421887]
     [1.06914219]]
    updated function value: 0.0011926921395593723
    d: [[-0.03129878]
     [-0.06427308]]
    gradient norm: 0.27756728506417017
    threshold: 0.0001

    step: 7
    alpha: 0.2
    x: [[1.02795912]
     [1.05628757]]
    updated function value: 0.000798717397382519
    d: [[-0.02582888]
     [-0.0526897 ]]
    gradient norm: 0.2400900376255633
    threshold: 0.0001

    step: 8
    alpha: 0.2
    x: [[1.02279334]
     [1.04574963]]
    updated function value: 0.0005322515681819315
    d: [[-0.021276  ]
     [-0.04316532]]
    gradient norm: 0.20432202337886463
    threshold: 0.0001

    step: 9
    alpha: 0.2
    x: [[1.01853814]
     [1.03711657]]
    updated function value: 0.0003528662051710774
    d: [[-0.01747768]
     [-0.0353    ]]
    gradient norm: 0.17175012620431226
    threshold: 0.0001

    step: 10
    alpha: 0.2
    x: [[1.0150426 ]
     [1.03005657]]
    updated function value: 0.0002327782263645213
    d: [[-0.01431288]
     [-0.02880145]]
    gradient norm: 0.14298481269502508
    threshold: 0.0001

    step: 11
    alpha: 0.2
    x: [[1.01218003]
     [1.02429628]]
    updated function value: 0.0001528529201920002
    d: [[-0.01168431]
     [-0.02344112]]
    gradient norm: 0.11812679939146666
    threshold: 0.0001

    step: 12
    alpha: 0.2
    x: [[1.00984316]
     [1.01960805]]
    updated function value: 9.995613546068292e-05
    d: [[-0.00951   ]
     [-0.01903206]]
    gradient norm: 0.09698963848128284
    threshold: 0.0001

    step: 13
    alpha: 0.2
    x: [[1.00794116]
     [1.01580164]]
    updated function value: 6.512846188240299e-05
    d: [[-0.00771924]
     [-0.01541733]]
    gradient norm: 0.07923808656277329
    threshold: 0.0001

    step: 14
    alpha: 0.2
    x: [[1.00639732]
     [1.01271818]]
    updated function value: 4.2303526271417644e-05
    d: [[-0.00625057]
     [-0.01246374]]
    gradient norm: 0.06447413557389624
    threshold: 0.0001

    step: 15
    alpha: 0.2
    x: [[1.0051472 ]
     [1.01022543]]
    updated function value: 2.7405114468215976e-05
    d: [[-0.00505076]
     [-0.01005805]]
    gradient norm: 0.05228928155837711
    threshold: 0.0001

    step: 16
    alpha: 0.2
    x: [[1.00413705]
     [1.00821382]]
    updated function value: 1.7714179820000433e-05
    d: [[-0.00407399]
     [-0.00810429]]
    gradient norm: 0.042294696053979584
    threshold: 0.0001

    step: 17
    alpha: 0.2
    x: [[1.00332225]
     [1.00659296]]
    updated function value: 1.1428983950133293e-05
    d: [[-0.00328118]
     [-0.00652159]]
    gradient norm: 0.03413701016380426
    threshold: 0.0001

    step: 18
    alpha: 0.2
    x: [[1.00266601]
     [1.00528864]]
    updated function value: 7.362607422700623e-06
    d: [[-0.00263936]
     [-0.0052423 ]]
    gradient norm: 0.027504938021446976
    threshold: 0.0001

    step: 19
    alpha: 0.2
    x: [[1.00213814]
     [1.00424018]]
    updated function value: 4.737096082738663e-06
    d: [[-0.00212089]
     [-0.00421017]]
    gradient norm: 0.022130294612251602
    threshold: 0.0001

    step: 20
    alpha: 0.2
    x: [[1.00171396]
     [1.00339815]]
    updated function value: 3.0447323568498954e-06
    d: [[-0.00170282]
     [-0.00337876]]
    gradient norm: 0.017785795014394316
    threshold: 0.0001

    step: 21
    alpha: 0.2
    x: [[1.0013734]
     [1.0027224]]
    updated function value: 1.955353864088912e-06
    d: [[-0.00136622]
     [-0.00270989]]
    gradient norm: 0.014281198855098794
    threshold: 0.0001

    step: 22
    alpha: 0.2
    x: [[1.00110016]
     [1.00218042]]
    updated function value: 1.2549004612078538e-06
    d: [[-0.00109553]
     [-0.00217237]]
    gradient norm: 0.011458789044168765
    threshold: 0.0001

    step: 23
    alpha: 0.2
    x: [[1.00088105]
     [1.00174594]]
    updated function value: 8.049277112761418e-07
    d: [[-0.00087808]
     [-0.00174077]]
    gradient norm: 0.009188780777784835
    threshold: 0.0001

    step: 24
    alpha: 0.2
    x: [[1.00070544]
     [1.00139779]]
    updated function value: 5.16076208111751e-07
    d: [[-0.00070352]
     [-0.00139446]]
    gradient norm: 0.00736499520209676
    threshold: 0.0001

    step: 25
    alpha: 0.2
    x: [[1.00056473]
     [1.0011189 ]]
    updated function value: 3.307632542718407e-07
    d: [[-0.0005635 ]
     [-0.00111676]]
    gradient norm: 0.005900963590270291
    threshold: 0.0001

    step: 26
    alpha: 0.2
    x: [[1.00045203]
     [1.00089554]]
    updated function value: 2.119323136229432e-07
    d: [[-0.00045124]
     [-0.00089417]]
    gradient norm: 0.00472652355749282
    threshold: 0.0001

    step: 27
    alpha: 0.2
    x: [[1.00036178]
     [1.00071671]]
    updated function value: 1.3576189126097238e-07
    d: [[-0.00036128]
     [-0.00071583]]
    gradient norm: 0.0037849075285548606
    threshold: 0.0001

    step: 28
    alpha: 0.2
    x: [[1.00028953]
     [1.00057354]]
    updated function value: 8.695187002111232e-08
    d: [[-0.0002892 ]
     [-0.00057298]]
    gradient norm: 0.003030290360051668
    threshold: 0.0001

    step: 29
    alpha: 0.2
    x: [[1.00023169]
     [1.00045895]]
    updated function value: 5.568216012573998e-08
    d: [[-0.00023148]
     [-0.00045859]]
    gradient norm: 0.002425747303054435
    threshold: 0.0001

    step: 30
    alpha: 0.2
    x: [[1.00018539]
     [1.00036723]]
    updated function value: 3.565348531745216e-08
    d: [[-0.00018526]
     [-0.000367  ]]
    gradient norm: 0.001941568396278841
    threshold: 0.0001

    step: 31
    alpha: 0.2
    x: [[1.00014834]
     [1.00029383]]
    updated function value: 2.2826895374249215e-08
    d: [[-0.00014825]
     [-0.00029368]]
    gradient norm: 0.0015538763558939346
    threshold: 0.0001

    step: 32
    alpha: 0.2
    x: [[1.00011869]
     [1.00023509]]
    updated function value: 1.4613653716375122e-08
    d: [[-0.00011863]
     [-0.000235  ]]
    gradient norm: 0.001243499182237761
    threshold: 0.0001

    step: 33
    alpha: 0.2
    x: [[1.00009496]
     [1.00018809]]
    updated function value: 9.355013775390774e-09
    d: [[-9.49261392e-05]
     [-1.88033408e-04]]
    gradient norm: 0.0009950542554677025
    threshold: 0.0001

    step: 34
    alpha: 0.2
    x: [[1.00007598]
     [1.00015049]]
    updated function value: 5.9883745449197485e-09
    d: [[-7.59534568e-05]
     [-1.50448575e-04]]
    gradient norm: 0.0007962066119685916
    threshold: 0.0001

    step: 35
    alpha: 0.2
    x: [[1.00006079]
     [1.0001204 ]]
    updated function value: 3.833156859157997e-09
    d: [[-6.07707990e-05]
     [-1.20372852e-04]]
    gradient norm: 0.000637069775853806
    threshold: 0.0001

    step: 36
    alpha: 0.2
    x: [[1.00004863]
     [1.00009632]]
    updated function value: 2.4535262533249835e-09
    d: [[-4.86217830e-05]
     [-9.63072398e-05]]
    gradient norm: 0.0005097227089664632
    threshold: 0.0001

    step: 37
    alpha: 0.2
    x: [[1.00003891]
     [1.00007706]]
    updated function value: 1.5704134538484076e-09
    d: [[-3.89007196e-05]
     [-7.70515274e-05]]
    gradient norm: 0.0004078209843882173
    threshold: 0.0001

    step: 38
    alpha: 0.2
    x: [[1.00003113]
     [1.00006165]]
    updated function value: 1.0051448367064757e-09
    d: [[-3.11226839e-05]
     [-6.16448937e-05]]
    gradient norm: 0.00032628419496853685
    threshold: 0.0001

    step: 39
    alpha: 0.2
    x: [[1.0000249 ]
     [1.00004932]]
    updated function value: 6.433337797516271e-10
    d: [[-2.48994967e-05]
     [-4.93182655e-05]]
    gradient norm: 0.0002610448991252154
    threshold: 0.0001

    step: 40
    alpha: 0.2
    x: [[1.00001992]
     [1.00003946]]
    updated function value: 4.1175465760644115e-10
    d: [[-1.99204612e-05]
     [-3.94561170e-05]]
    gradient norm: 0.0002088471481323487
    threshold: 0.0001

    step: 41
    alpha: 0.2
    x: [[1.00001594]
     [1.00003157]]
    updated function value: 2.635337540940221e-10
    d: [[-1.59369220e-05]
     [-3.15658567e-05]]
    gradient norm: 0.0001670849055650914
    threshold: 0.0001

    step: 42
    alpha: 0.2
    x: [[1.00001275]
     [1.00002525]]
    updated function value: 1.6866711905022221e-10
    d: [[-1.27498915e-05]
     [-2.52533019e-05]]
    gradient norm: 0.0001336725245054508
    threshold: 0.0001

    step: 43
    alpha: 0.2
    x: [[1.0000102]
     [1.0000202]]
    updated function value: 1.0794978077835293e-10
    d: [[-1.02001398e-05]
     [-2.02030360e-05]]
    gradient norm: 0.00010694096373611484
    threshold: 0.0001

    step: 44
    alpha: 0.2
    x: [[1.00000816]
     [1.00001616]]
    updated function value: 6.908930602668813e-11
    d: [[-8.16025679e-06]
     [-1.61626813e-05]]
    gradient norm: 8.555465541454276e-05
    threshold: 0.0001

---
END


created by Zsolt Elter (2020)

in case any mistakes are found, please let me know.

# 2nd tutorial

Some words on scattering reactions in general. We mostly work with elastic scattering, thus the kinetic energy and the linear momentum stay constant during the collision.

One can derive that

$$E_l'=\frac{1}{2}E_l[(1+\alpha)+(1-\alpha)cos\theta]$$

where

$$\alpha=\big(\frac{A-1}{A+1}\big)^2$$

and $A=M/m$

```python
import numpy as np
import matplotlib.pyplot as plt
theta=np.linspace(0,360,361)*np.pi/180
A=1
alpha=((A-1)/(A+1))**2

plt.figure()
plt.plot(theta*180/np.pi,(1+alpha)+(1-alpha)*np.cos(theta))
plt.xlabel('angle (deg)')
plt.show()
```

Thus $E_l'$ is bound between $\alpha E_l$ (for $\theta=\pm\pi$) and $E_l$ (for $\theta = 0$).

If one assumes isotropic scattering in the COM frame. one can arrive to the pdf

\begin{equation}
    P(E_l \rightarrow E_l') =
    \begin{cases}
      \frac{1}{E_l(1-\alpha)} & \text{if $\alpha E_l < E_l' < E_l$} \\
      0 & \text{if $0<E_l'<\alpha E_l$}
    \end{cases}
\end{equation}

Also one can see that the energy loss $ E_l-E_l'$ depends on $E_l$, however the ratio $E_l/E_l'$ is independent of energy. Thus it might be interesting to look at ratios (look up the lethargy variable).

The slowing down decrement (or average increase of lethargy per collision):

$$\xi=\int_{\alpha E_l}^{E_l}\ln \Big(\frac{E_l}{E_l'}\Big)P(E_l \rightarrow E_l')dE_l'$$

Thus increase of elastic and isotropic scattering in COM.

$$\xi=1-\frac{(A-1)^2}{2A}\ln\Big(\frac{A+1}{A-1}\Big)=1+\frac{\alpha}{1-\alpha}\ln\alpha$$

## 1st exercise

Calculate the slowing down decrement $\xi$ for natural hydrogen and the minimum number of elastic scattering collisions required to slow a neutron down from 2.2 MeV to 0.1 eV.

The slowing down decrement is easy to see from the previous definition:

$$\xi_H=1$$

Also one can notice that $\alpha=0$, thus the neutron can loose all of its energy in a head-on hit. Which also means that it can slow down to 0.1eV in one collision, the collision angle for that is

$$cos \theta= 2\frac{E_l'}{E_l}-1$$

```python
Ei=2.2
Ef=0.1e-6

theta=np.arccos(2*Ef/Ei-1)

print(theta/np.pi*180)
```

    179.97556899734053

## 2nd exercise Slowing down time

The slowing-down time for neutrons can be calculated by describing the process in terms of a deterministic sequence instead of a stochastic one. It is assumed that the path length $\lambda$ between one collision and the next is always the same.

   a, On the basis of this model, find an expression giving the average time taken for the energy of a neutron to become $E_f$.

$$\xi = \ln \Big(\frac{E_{n-1}}{E_{n}}\Big)$$

Thus

$$E_{n} = E_{n-1}e^{-\xi}$$

and

$$E_{n-1} = E_{n-2}e^{-\xi}$$.

and

$$E_{n} = E_{n-2}e^{-2\xi}$$

and the general form can be written as:

$$E_{f} = E_{i}e^{-n\xi}$$

The velocity of a non relativistic neutron is $v = \sqrt{\frac{2E}{m}}$, therefore is possible to express the same in terms of the velocity of the neutron before and after the collisions:

$$v_{f} = v_{i}e^{-n\xi/2}$$

The time to travel through one mean free path ($\lambda = \frac{1}{\Sigma} = \frac{1}{\Sigma(E)}$) is changing as the neutron slows (from now on $v_i=v_0$):

$$t_0=\frac{\lambda}{v_0}$$

then

$$t_1=\frac{\lambda}{v_1}=\frac{\lambda}{v_0}e^{\frac{\xi}{2}}$$

and

$$t_2=\frac{\lambda}{v_2}=\frac{\lambda}{v_0}e^{\frac{2\xi}{2}}$$

up to after the $n$th collision

$$t_n=\frac{\lambda}{v_n}=\frac{\lambda}{v_0}e^{\frac{n\xi}{2}}$$

and the average time to slow down from $v_i$ to $v_f$

$$T=t_0+t_1+t_2+...=\sum_{n=0}^{N}\frac{\lambda}{v_0}e^{\frac{n\xi}{2}}=\sum_{n=0}^{N}\frac{\lambda}{v_0}\Big(e^{\frac{\xi}{2}}\Big)^n=\frac{\lambda}{v_0}\frac{1-e^{\frac{(N+1)\xi}{2}}}{1-e^{\frac{\xi}{2}}}$$

remember $\sum_{n=0}^{N}ar^n=a\frac{1-r^{N+1}}{1-r}$

## 3rd exercise

- Estimate the average number of collisions required to reduce fast fission neutrons of initial energy 2 MeV to thermal energy (0.025 eV) in graphite moderator;
- Calculate the corresponding slowing-down time given that $\Sigma_s$ = 0.385 cm$^{-1}$.

The average number of collisions comes from

$$n=\frac{1}{\xi}\ln\frac{E_i}{E_f}$$

remember in case of heads-on, it would be

$$n=\frac{1}{\ln\alpha}\ln\frac{E_f}{E_i}$$

```python
Ei=2.0 #MeV
Ef=0.025e-6 #MeV
A=12
alpha=((A-1)/(A+1))**2
xi=1+alpha/(1-alpha)*np.log(alpha)

nave=np.log(Ei/Ef)/xi
nheads=np.log(Ef/Ei)/np.log(alpha)
print(xi)
print(nave)
print(nheads)
```

    0.15776898982320375
    115.34292773903384
    54.46600491490566

For the slowing down time

$$T=\frac{\lambda}{v_0}\frac{1-e^{\frac{(N+1)\xi}{2}}}{1-e^{\frac{\xi}{2}}}$$

```python
Sig=0.385
lam=1/Sig/100 #m
eV_to_J=1.60217662e-19
E0=2*1e6*eV_to_J #ev to J
m=1.675E-27 #kg
v0=np.sqrt(2*E0/m) #m/s
print(v0)
N=nave
T=(lam/v0)*((1-np.exp((N+1)*xi/2))/(1-np.exp(xi/2)))
print(T)
```

    19560402.401012927
    0.00015656219687178042

If we wouldn't know how to solve geometric series, we still could have programmed our way out by calculating the expected energy after each collision.

```python
eV_to_J=1.60217662e-19
m=1.675E-27 #kg
E=2.0e6
t=0
N=0
Es=[]
while E>0.025:
    v=np.sqrt(2*E*eV_to_J/m) #m/s
    t=t+lam/v
    Es.append(E)
    E=E*np.exp(-xi)
    N=N+1
print(N-1)
print(t)
plt.figure()
plt.plot(Es)
plt.show()
```

## 4th exercise

```python
import matplotlib.pyplot as plt
import matplotlib.patches as p
fig, ax = plt.subplots(figsize=(15,10))

fuel = plt.Circle((40, 0), 30,facecolor='white',edgecolor='black',hatch='/')
ax.add_artist(fuel)
thFiss1 = plt.Rectangle((20,20), 40, 20,facecolor='white',edgecolor='black')
thFiss2 = p.Ellipse((40,30), 20, 10,facecolor='white',edgecolor='black')
resEsc1 = plt.Rectangle((100,20), 40, 20,facecolor='white',edgecolor='black')
resEsc2 = p.Ellipse((120,30), 20, 10,facecolor='white',edgecolor='black')
thermalized1 = plt.Rectangle((100,-40), 40, 20,facecolor='white',edgecolor='black')
thermalized2 = p.Ellipse((120,-30), 20, 10,facecolor='white',edgecolor='black')
absorbed1 = plt.Rectangle((20,-40), 40, 20,facecolor='white',edgecolor='black')
absorbed2 = p.Ellipse((40,-30), 20, 10,facecolor='white',edgecolor='black')
ax.add_artist(thFiss1)
ax.add_artist(thFiss2)
ax.add_artist(absorbed1)
ax.add_artist(absorbed2)
ax.add_artist(resEsc1)
ax.add_artist(resEsc2)
ax.add_artist(thermalized1)
ax.add_artist(thermalized2)
plt.arrow(60, 30, 38.5, 0,linewidth=5,head_width=1, head_length=1,color='black')
plt.arrow(120, 20, 0, -38.5,linewidth=5,head_width=1, head_length=1,color='black')
plt.arrow(100, -30, -38.5, 0,linewidth=5,head_width=1, head_length=1,color='black')
plt.arrow(40, -20, 0, 38.5,linewidth=5,head_width=1, head_length=1,color='black')
c1=plt.Circle((80,30),5,facecolor='white',edgecolor='black')
c2=plt.Circle((120,0),5,facecolor='white',edgecolor='black')
c3=plt.Circle((80,-30),5,facecolor='white',edgecolor='black')
c4=plt.Circle((40,0),5,facecolor='white',edgecolor='black')
ax.add_artist(c1)
ax.add_artist(c2)
ax.add_artist(c3)
ax.add_artist(c4)
plt.text(39,0,r'$\eta$',fontsize=18)
plt.text(79,29,r'$\epsilon$',fontsize=18)
plt.text(69,36,'fast neutrons',fontsize=18)
#################
plt.arrow(50,20,10,-10,linewidth=3,head_width=1, head_length=1,color='black')
plt.arrow(60,10,19,14,linewidth=3,head_width=1, head_length=1,color='black')
plt.text(70,15,'fast fission: 26',fontsize=13)
plt.arrow(80,-25,0,15,linewidth=3,head_width=1, head_length=1,color='black')
plt.text(81,-18,'parasitic capture: 72',fontsize=13)
plt.arrow(35,0,-10,0,linewidth=3,head_width=1, head_length=1,color='black')
plt.text(25,1,'capture',fontsize=13)
plt.text(22,17,'thermal fission',fontsize=13)
##########
plt.text(119,-1,r'$p$',fontsize=18)
plt.text(126,-5,'resonance \n energy \n neutrons',fontsize=18)
plt.arrow(115, 0, -55, 0,linewidth=3,head_width=1, head_length=1,color='black')
plt.text(80,1,'resonance capture: 96',fontsize=13)

plt.text(79,-31,r'$f$',fontsize=18)
plt.text(69,-38,'thermal neutrons',fontsize=18)

plt.text(33,29,'N=1000',fontsize=18)
plt.text(28,36,'Thermal fissions',fontsize=18)
plt.text(28,22,'Born at 2 MeV',fontsize=18)

plt.text(33+80,29,'N=1026',fontsize=18)
plt.text(28+80,36,'Neutrons reach',fontsize=18)
plt.text(26+80,22,'Resonance energy',fontsize=18)

plt.text(34+80,29-60,'N=930',fontsize=18)
plt.text(30+80,36-60,'Neutrons',fontsize=18)
plt.text(28+80,22-60,'Thermalized',fontsize=18)

plt.text(33,29-60,'N=858',fontsize=18)
plt.text(26,36-60,'Neutrons absorbed',fontsize=18)
plt.text(32,22-60,'in fuel',fontsize=18)

plt.text(45,-10,'FUEL', fontsize=20)
plt.text(45,-45,'MODERATOR', fontsize=20)
#ax.add_artist(polygon)
plt.xlim(0,150)
plt.ylim(-50,50)
#plt.xticks([-200,0,200], ('-a/2',0,'a/2'))
ax.yaxis.set_visible(False) # Hide only x axis
plt.show()
```

**Fast fission factor**: ratio between the numberr of fission neutrons induced by both thermal and fast neutrons and the number of fission neutrons induced by thermal neutrons only

$$\epsilon=\frac{\int_{V_{f}} \int_0^\infty \nu(E)\Sigma_f(r,E)\phi(r,E)dVdE}{\int_{V_{f}} \int_0^{\sim 5kT} \nu(E)\Sigma_f(r,E)\phi(r,E)dVdE}$$

In the current case fission neutrons have enough energy to induce fast fission in U-238 (threshold is around 1.2MeV).

$$\epsilon = 1026/26=1.026$$

**Resonance escape probability**: probaility for fast neutrons to slow-down to thermal energies without being absorbed in the resonance region of absorbers

$$p=1-\frac{\int_{V_{f}} \int_{\sim 5kT}^\infty \Sigma_a(r,E)\phi(r,E)dVdE}{\int_{V_{f}} \int_0^{\infty} \nu(E)\Sigma_f(r,E)\phi(r,E)dVdE}$$

In the current case the neutrons slow to resonance energies and some of them are scattered to fuel regions, if a neutron at this energy enters fuel will be almost certainly captured.

$$p=\frac{930}{1026}=1-\frac{96}{1026}=0.906$$

**Thermal utilization factor**: the ratio of the number of thermal neutrons absorbed in the fuel and the number of thermal neutrons absorbed in all materials

$$f=\frac{\int_{V_{f}} \int_0^{\sim 5kT} \Sigma_a(r,E)\phi(r,E)dVdE}{\int_{V_{t}} \int_0^{\sim 5kT} \Sigma_a(r,E)\phi(r,E)dVdE}$$

In the current case some neutrons will be absorbed by the moderator (or other non-fuel material), this is parasitic capture.

$$f=858/930=0.923$$

**Thermal fission factor**: the average number of fission neutrons emitted per thermal neutron absorbed in the fuel

$$\eta=\frac{\int_{V_{f}} \int_0^{\sim 5kT} \nu(E)\Sigma_f(r,E)\phi(r,E)dVdE}{\int_{V_{f}} \int_0^{\sim 5kT} \Sigma_a(r,E)\phi(r,E)dVdE}$$

In the current case the number of thermal neutrons absorbed in the fuel is $fp\epsilon N_0$.. Some of these will cause fission (and return neutrons) and some of them will be captured.

$$\eta=1000/858=1.66$$

Thus

$$k_{\infty}=\eta f p \epsilon=1.0$$

which is not surprising since the ratio between the number of neutrons in the consecutive cycles is 1.0.

```python
eps=1026/1000
p=930/1026
f=858/930
eta=1000/858
kinf=eta*f*p*eps
print(kinf)
```

    1.0000000000000002

## 5th exercise: thermal utilization factor

Calculate the thermal utilization factor for a heterogeneous lattice made up of
cylindrical uranium rods of diameter 3 cm and pitch 18 cm placed in graphite. Take the flux ratio $\phi_M/\phi_U$ as 1.6.

Densities: Uranium 18700 kg m$^{-3}$, Graphite = 1620 kg m$^{-3}$. \\

Absorption cross-sections $\sigma_{a,U}$=7.68 b; $\sigma_{a,M}$ = 4.5 $\cdot$ 10$^{-3}$ b.

The absorption rate in the fuel is

$$\Sigma_{a,U}\phi_U V_U$$

and the absorption rate in the moderator is

$$\Sigma_{a,M}\phi_M V_M$$

where $V_M$ and $V_U$ are the volumes. We do not know the height, but as we will see only the volume ratio will matter:

$$\frac{V_M}{V_U}=\frac{A_Mh}{A_Uh}=\frac{p^2-r^2\pi}{r^2\pi}$$

where $A_U$ and $A_M$ are the cross sectional areas, $r$ is the fuel radius and $p$ is the pitch of the cell

The fraction of thermal neutrons absorbed by the Uranium fuel as compared to the total number of thermal neutron absorptions in the assembly is known as the thermal utilization factor $f$ and is given by

$$f = \frac{\Sigma_{a,U}\phi_U V_U}{\Sigma_{a,U}\phi_U V_U + \Sigma_{a,M}\phi_M V_M }    = \frac{1}{1 + \frac{\Sigma_{a,M}\phi_M V_M }{\Sigma_{a,U}\phi_U V_U}} $$

```python
r=3/2 #cm
p=18 #cm
sig_a_U=7.68 #b
sig_a_M=4.5e-3
flux_MpU=1.6
rho_U=18.7 #g/cm3
rho_M=1.62 #g/cm3

M_U=238 #approx, we dont know enrichment
M_C=12

NA=6.022e23

N_U=NA*rho_U/M_U #approximate...
N_M=NA*rho_M/M_C #/cm3

Sig_a_U = N_U*sig_a_U
Sig_a_M = N_M*sig_a_M
Sig_MpU=Sig_a_M/Sig_a_U

A_U=r**2*np.pi
A_M=p**2-A_U
V_MpU=A_M/A_U

f=1/(1+Sig_MpU*flux_MpU*V_MpU)

print(f)

```

    0.9326421327798471

```python

```


```
%matplotlib inline
```

```
import sympy
sympy.init_printing(use_latex=True)
```

```
from sympy.abc import x, y, alpha, beta
from sympy import latex
```

```
from IPython.display import HTML
```

```
def association_table(assocs):
    latexed = {}
    for key, value in assocs.items():
        latexed[key] = latex(value)
    latexed['s_total'] = latex(assocs['t_s'] + assocs['f_s'])
    latexed['ns_total'] = latex(assocs['t_ns'] + assocs['f_ns'])
    return """
    <TABLE>
    <TR><TH><TH>Significant<TH>Not significant
    <TR><TH>True association<TD>${t_s}$<TD>${t_ns}$
    <TR><TH>No association<TD>${f_s}$<TD>${f_ns}$
    <TR><TH>Total<TD>${s_total}$<TD>${ns_total}$
    </TABLE>
    """.format(**latexed)
```

```
assocs = dict(t_s = x,
              t_ns = alpha,
              f_s = beta,
              f_ns = y)
```

```
HTML(association_table(assocs))
```

SignificantNot significant

True association$x$$\alpha$

No association$\beta$$y$

Total$\beta + x$$\alpha + y$


 # Defining Functions

Matlab allows you to define inline functions, known as function handles or anonymous functions. You may want to do this because you didn't want to use a separate code block to define a single function and it is more clear to be able to "see" all of the implementation in a single script. In Matlab, ODE routines often take these function handles as inputs, which makes defining anonymous functions commonplace. In MatLab, to define the function,

\begin{equation}
f(x) = \frac{1}{x^2}
\end{equation}

inline would look something like the following:

```
f = @(x) 1. / x.^2
```

where the dot operators would make the function *vectorized*. We can do the same thing in Python, but there are several ways we can implement the same behavior. The imperative approach would be to define a function, which in Python looks like the following:

```python
def f(x):
    """Returns the inverse of the square of x."""
    return 1.0 / x**2
```

We can then use this function in any of the Cells of our notebook, but only after we Run the cell defining the function. Notice how `In [ ]:` in the left margin shows the execution order of the cells. To use our function,

```python
print(f(5))
```

    0.04

Python provides an excellent set of tools for inspection. The most obvious and useful is `help`. We can use it like this.

```python
help(f)
```

    Help on function f in module __main__:

    f(x)
        Returns the inverse of the square of x.

You can also define a function as a *lambda* expression, just like how anonymous function are defined in Matlab. Anonymous functions, lambda functions, lambda expressions, function literals are all the same thing. They orginate from the concepts of lambda calculus, which defines a framework for encoding any computation in a function. In Python, we use the key word `lambda`

```python
one_over_xsquared = lambda x: 1 / x**2
one_over_xsquared(5)
```

    0.04

While there can be instances where lambda expressions are useful, there are usually *better*, more Pythonic ways of doing the same thing. Generator expressions and list comprehensions are often more expressive. In fact, the style guide for Python code defined by PEP 8 states the following:

PEP 8 : Always use a def statement instead of an assignment statement that binds a lambda expression directly to an    identifier. [Source](https://www.python.org/dev/peps/pep-0008/#programming-recommendations)

Consider these motivating examples.

- To filter out the even numbers in a sequence

```python
even = lambda x: x%2 == 0
list(filter(even, range(11)))
```

    [0, 2, 4, 6, 8, 10]

```python
[x for x in range(11) if x%2 == 0]
```

    [0, 2, 4, 6, 8, 10]

- To capitalize the first letter in a list of words

```python
list(map(lambda x: x.capitalize(), ['cat', 'dog', 'cow']))
```

    ['Cat', 'Dog', 'Cow']

```python
[x.capitalize() for x in ['cat', 'dog', 'cow']]
```

    ['Cat', 'Dog', 'Cow']

 # Linear Algebra and NumPy

```python
import numpy as np
```

Python has many built-in data structures, such as lists, tuples, dictionaries, etc. These data structures, however, are not effecient for computing numerical operations. Instead, we need to use NumPy arrays. They may look similar, but be aware they are very different.

```python
a_list = [1,2,3,4]
print(f'This is a list: {a_list}. It has type: {type(a_list)}')
```

    This is a list: [1, 2, 3, 4]. It has type: <class 'list'>

```python
a = np.array([1,2,3,4])
print(f'This is a numpy array: {a}. It has type: {type(a)}')
```

    This is a numpy array: [1 2 3 4]. It has type: <class 'numpy.ndarray'>

```python
# We define a matrix  like this
b = np.array([[0,1],[1,2],[2,3],[3,4]])

# We can print the shape of the Numpy arrays
print(f'The shape of a is: {a.shape}')
print(f'The shape of b is: {b.shape}')
```

    The shape of a is: (4,)
    The shape of b is: (4, 2)

Arrays of different sizes cannot be added or subtracted, and for multiplication we need the shapes of the two arrays to conform to the rules of inner/outer products. There are often times when we would like to work around these requirements. NumPy does this by broadcasting.

    The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.

NumPy does broadcasting in a memory and computationally efficient way. This concept is often used in machine learning libraries like Theano or Tensorflow.

```python
# Let's define a scalar
c = 5
```

What doe we expect to happen, when we add `c` to the array `a`?

```python
print(f'a = {a}')
print(f'c = {c}\n')

a_plus_c = a + c
print(f'The shape of the sum of the numpy array, "a", and the scalar, `c`, is: {a_plus_c.shape}')
print(f'a + c = {a_plus_c}')
```

    a = [1 2 3 4]
    c = 5

    The shape of the sum of the numpy array, "a", and the scalar, `c`, is: (4,)
    a + c = [6 7 8 9]

What about the sum of a two-dimensional array and a scalar?

```python
print(f'b = {b}')
print(f'c = {c}\n')

print(f'b + c = {b + c}')
```

    b = [[0 1]
     [1 2]
     [2 3]
     [3 4]]
    c = 5

    b + c = [[5 6]
     [6 7]
     [7 8]
     [8 9]]

Broadcasting can only be done if the shape of the arrays are equal or one of the arrays has the dimension size of 1. If the shapes on the two arrays do not match, then Python first "pads" the inner dimension, and then stretches that dimension to match the other array.

```python
M = np.ones((2,3))
a = np.arange(3)

# Equivalently, we could have used the function `np.linspace` to
# create a, which gives you further control over the creation
# of the array.
#
# a = np.linspace(0,2,3, dtype=np.int)
#

print(f'M is a matrix of shape: {M.shape}')
print(f'a is a vector of shape: {a.shape}\n')

print(f'M = {M}')
print(f'a = {a}')
```

    M is a matrix of shape: (2, 3)
    a is a vector of shape: (3,)

    M = [[1. 1. 1.]
     [1. 1. 1.]]
    a = [0 1 2]

To sum the matrix and vector, the first step is the pad the inner dimension

- `M.shape => (2,3)`
- `a.shape => (1,3)`

Next, the inner dimension is stretched so that the matrix and vector have the same shape:

- `M.shape => (2,3)`
- `a.shape => (2,3)`

As a result the sum of the matrix, `M`, and the vector, `a` is

```python
print(M + a)
print(f'The shape of M + a is: {(M+a).shape}')
```

    [[1. 2. 3.]
     [1. 2. 3.]]
    The shape of M + a is: (2, 3)

```python
print(a.shape)
print(a)
```

    (3,)
    [0 1 2]

Think about what happens in the following example. Let `M` be a column vector.

\begin{equation}
M = \begin{bmatrix} 0 \\ 1 \\ 2 \end{bmatrix}
\end{equation}

and let `a` be a row vector

\begin{equation}
a = \begin{bmatrix} 0 & 1 & 2 \end{bmatrix}
\end{equation}

What is the sum of `M` and `a`?

```python
M = np.arange(3).reshape((3, 1))
a = np.arange(3)

print(f'M is a matrix of shape: {M.shape}')
print(f'a is a vector of shape: {a.shape}\n')

print(f'M = {M}')
print(f'a = {a}')
```

    M is a matrix of shape: (3, 1)
    a is a vector of shape: (3,)

    M = [[0]
     [1]
     [2]]
    a = [0 1 2]

```python
M_plus_a = M + a

print(f'The shape is {M_plus_a.shape} with the result: \n {M_plus_a}')
```

    The shape is (3, 3) with the result:
     [[0 1 2]
     [1 2 3]
     [2 3 4]]


**Exercise set 12**
==============

>In this exercise, we will process signals and you will learn how you
>can remove noise, obtain numerical derivatives of noisy signals, and
>correct near-infrared spectra by removing unwanted scattering effects.

**Exercise 12.1**

In this exercise, we will test out the Savitzky–Golay filter for smoothing
and numerical differentiation. We will use a test signal which has been generated
from the following analytical function,

\begin{equation}
y(t) = \sin 8t - 1.8t^2 + 0.5t^3.
\label{eq:signal}
\tag{1}\end{equation}

The signal is available in the file `Data/signal.txt`.
In addition, we will investigate a test signal which is generated from the same
analytical function, but with noise added. This signal is available in the file
`Data/signal_noise.txt`.
In `scipy` a Savitzky–Golay filter can be created by using
the method `savgol_filter` from `scipy.signal`.(https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.savgol_filter.html)

**(a)**  Create a Savitzky–Golay smoothing filter with a window of size $5$
and polynomial order of $3$. Apply this filter to the signal without noise
and compare the filtered signal with the original signal.

```python
# Your code here
```

**Your answer to question 12.1(a):** *Double click here*

**(b)**  Create a Savitzky–Golay smoothing filter with a window of size $7$
and polynomial order of $3$. Apply this filter to the signal without noise
and compare the filtered signal with the original signal.

```python
# Your code here
```

**Your answer to question 12.1(b):** *Double click here*

**(c)**  Create a Savitzky–Golay filter for first-order differentiation with a
window size of $5$ and polynomial order of $3$. Apply this to the signal
without noise and compare the differentiated signal with the analytical
derivative of Eq. 1.

**Note:** In order to obtain the derivative,
you will have to supply the spacing between your points using the
`delta` parameter of the `savgol_filter` method.

```python
# Your code here
```

**Your answer to question 12.1(c):** *Double click here*

**(d)**  Create a Savitzky–Golay filter for first-order differentiation with a
window size of $7$ and polynomial order of $3$. Apply this to the signal
without noise and compare the differentiated signal with the analytical
derivative of Eq. 1.

```python
# Your code here
```

**Your answer to question 12.1(d):** *Double click here*

**(e)**  Create a Savitzky–Golay filter for smoothing of the signal with noise.
Here, you have to experiment with the window size to use. Use a polynomial
order of $3$, and compare your smoothed signal with the original signal, and
the signal without noise.

```python
# Your code here
```

**Your answer to question 12.1(e):** *Double click here*

**(f)**  Create a Savitzky–Golay filter for first-order differentiation of the signal
with noise. Here, you have to experiment with the window size to use.
Use a polynomial order of $3$ and compare your differentiated signal
with the analytical derivative of Eq. 1.

```python
# Your code here
```

**Your answer to question 12.1(f):** *Double click here*

**Exercise 12.2**

In this exercise, we will try to smooth by performing convolution.
We will attempt to smooth the signal given in Eq. 1
with added noise (data file: `Data/signal_noise.txt`). A short example
on performing convolution with `numpy`:
```python
import numpy as np

signal = np.loadtxt('Data/signal_noise.txt')[:, 1]  # Load signal.
window = np.bartlett(21)  # Create a triangular window.
window /= window.sum()  # Normalize the window.
conv = np.convolve(signal, window, mode='same')  # Calculate convolution.
```

**(a)**  Select a windowing function(A list of the available windowing
functions in `numpy` can be found at [here](https://docs.scipy.org/doc/numpy/reference/routines.window.html)
and use this to smooth the signal by convolution. Compare your
smoothed signal with the raw data and the analytical expression
without noise (Eq. 1).

```python
# Your code here
```

**Your answer to question 12.2(a):** *Double click here*

**(b)**  Create a windowing function which is "rectangular": constant for a small and
finite region, and zero otherwise. Use this function to smooth the signal, and
compare it with the raw data and Eq. 1.

```python
# Your code here
```

**Your answer to question 12.2(b):** *Double click here*

**Exercise 12.3**

We will now investigate a noise signal that contains an artificially added
trend. We will try to remove this trend and locate the peaks present in the signal.

**(a)**  Consider the signal given in `Data/peaks.txt`.
Use a third-order polynomial to estimate the trend in the data.
Subtract this trend from the raw data and plot the resulting signal.

```python
# Your code here
```

**Your answer to question 12.3(a):** *Double click here*

**(b)** Smooth the signal with the trend removed (either by
convolution or by using a Savitzky–Golay
filter) and plot the smoothed signal. Identify the $6$ main peaks present
in the signal.

```python
# Your code here
```

**Your answer to question 12.3(b):** *Double click here*

**Exercise 12.4**

You have measured the signal given in the file `Data/spike.txt`. Unfortunately,
the signal contains a prominent spike that you would like to remove.

**(a)**  Process the signal by convolution. Use a window of your choice,
for instance, the [Bartlett](https://en.wikipedia.org/wiki/Window_function#Triangular_window) window.
Are you able to remove the spike?

```python
# Your code here
```

**Your answer to question 12.4(a):** *Double click here*

**(b)**  Process the signal by implementing a *median* filter.
The median filter
returns the median value of the signal in a given window.
Unfortunately, this filter can not
be expressed as a simple convolution and you will thus have to create
a new method that processes the signal.
Create such a method, and process the signal. Are you able to remove the
spike with this filter?

```python
# Your code here
```

**Your answer to question 12.4(b):** *Double click here*

**(c)**  Compare the median filter you have created with the convolution results.

```python
# Your code here
```

**Your answer to question 12.4(c):** *Double click here*

**Exercise 12.5**

Multiple Scatter Correction (MSC) is one approach to remove non-linear
effects in near-infrared (NIR) spectra. Such effects may arise as a result
of scattering effects in a sample.
This will generate additional variance which is not related to the
chemical interesting information contained in the spectrum, and we would,
therefore, like to remove it.

The file `Data/nir_msc.txt` contains $222$ spectra which have been
sampled at $121$ wavelengths. Each row contains a spectrum, and each
column represents a single wavelength. You will now apply MSC to
correct these spectra.

**(a)**  Find a representative spectrum by taking the mean of the $222$ spectra.
We will refer to this representative spectrum as $f(x)$ in the following.

```python
# Your code here
```

**Your answer to question 12.5(a):** *Double click here*

**(b)**  Correct each spectrum, $h_i(x)$, by first fitting it to a linear
equation,
\begin{equation}
h_i(x) = a_i f(x) + b_i ,
\end{equation}

and then remove the scattering effects by taking,
\begin{equation}
h_{i, \text{corrected}}(x) = \frac{h_i(x) -b_i}{a_i} .
\end{equation}

```python
# Your code here
```

**Your answer to question 12.5(b):** *Double click here*

**(c)**  Plot the corrected and uncorrected spectra.
Does this look like what you would expect?

```python
# Your code here
```

**Your answer to question 12.5(c):** *Double click here*

**(d)**  Quantify the effect of the MSC by calculating the sum of squares, $SS_0$,
of the *centered* spectra,
\begin{equation}
SS_0 = \sum_i \sum_j (x_{ij} - \overline{x}_j)^2,
\end{equation}

where $x_{ij}$ is the intensity for spectrum $i$ at wavelength $j$, and
$\overline{x}_j$ is the mean of all spectra for wavelength $j$.
Calculate $SS_0$ for both the corrected and uncorrected spectra.
Are these values as you would expect?

```python
# Your code here
```

**Your answer to question 12.5(d):** *Double click here*

**(e)**  A simple alternative to MSC is to "auto-scale" each spectrum,
\begin{equation}
h_{i, \text{auto-scale}}(x) = \frac{h_i(x) - \overline{h}_i}{\sigma_i},
\end{equation}

where $h_i(x)$ is the original spectrum, $\overline{h}_i$ its average, and
$\sigma_i$ the standard deviation of the spectrum. Apply this to the
original spectra, and compare with the uncorrected spectra and the MSC spectra.

```python
# Your code here
```

**Your answer to question 12.5(e):** *Double click here*


```python
# Boilerplate imports
from numpy import array as ary; from numpy import log as ln
from numpy import cos, sin, pi, sqrt, exp, arccos, arcsin
tau = 2*pi
import numpy as np;
from matplotlib import pyplot as plt
# Linear algebra functions
from numpy.linalg import inv, pinv, det, eig, eigh, eigvals
from matplotlib.patches import Ellipse # for plotting ellipse
from collections import namedtuple
```

```python
# program to generate a covariance matrix where the variance values are fixed at [2,2].
def generate_cov(off_diag):
    cov = ary([[2.0, off_diag], [off_diag, 2.0]])
    return cov

Dots = namedtuple('Dots', ['points', 'area'])
def get_encircled_dots(covariance_matrix, bounds=[-10, 10], resolution=300):
    pt_list = np.linspace(*bounds, resolution)
    unit_square_size = ((bounds[1]-bounds[0])/resolution)**2
    points = ary(np.meshgrid(pt_list, pt_list)).T.flatten().reshape([-1,2])

    chi2_level = ((points @ inv(covariance_matrix)) * points).sum(axis=1)
    mask = chi2_level <= 1 # choose only points within the error ellipse
    area = sum(mask)*unit_square_size

    return Dots(points[mask], area)

PLOT_CIRCLE = True
# if PLOT_CIRCLE: plot the error ellipse using 11 differernt covariance values;
# else: plot plot the variation of the error ellipse area wrt. the covariance value.
if not PLOT_CIRCLE:
    determinant_cov, determinant_inv_cov, size = [], [], []
```

Setup:

- We have a point cloud with a center at (0,0).
- 1 sigma of the points (68% of them) lies within the error ellipse. (We won't be plotting the remaining 32%)
- The top left element of the covariance matrix describes the width of this ellipse, and the bottom right describes the height of the ellipse.
- Therefore, varying the covariance value (i.e. the symmetric off-diagonal terms) should only make the ellipse into a thinner ellipse that is leaning left/right.

```python
# plot the error ellipse
cov_range = np.linspace(-1.98, 1.98, 11)
for i in cov_range:
    cov = generate_cov(i)
    (minor, major) , eig_mat = eig(inv(cov) * det(cov))
    mat = inv(eig_mat)
    orientation = np.mean([arcsin(mat[0,1]), -arcsin(mat[1,0])])*np.sign(mat[0,0])
    fig, ax = plt.subplots()
    ellipse = Ellipse([0,0], # centered at the origin
                2*sqrt(major),
                2*sqrt(minor),
                np.rad2deg(orientation)
                )
    # DUUUUDE I got the width=2*sqrt(major)/sqrt(det(inv(cov))) equation by trial and error LMAO
    ax.add_patch(ellipse)
    ax.scatter(*(get_encircled_dots(cov).points.T), marker='+', alpha=0.4, color='C1', zorder=10) # scatter plot approach
    plt.show()

```

Using the code block above we can verify that we have correctly plotted the error ellipse correctly: All points with $\chi^2 \le 1$ are plotted; and it concides exactly with the error ellipse.
The parameters about the ellipse is closely related to the following matrix:

\begin{equation}
M = S^{-1} \dot det(S)
\end{equation}

where S is the covariance matrix

The major radius is equal to sqrt(the larger eigen value of $M$), minor radius is equal to sqrt(the smaller eigen value of $M$).
To draw the ellipse on our graph, we first draw an ellipse with those specified radii (major axis in the horizontal direction), then apply the rotation matrix as described by the eigenvector matrix of $M$.

We can also show that the covariance ellipse area is equal to the expression $det(S)$ (i.e. determinant of the covariance matrix).

The ellipse area was empirically calculated by counting the number of points that the ellipse covers when spread over an evenly spaced grid.

```python
determinant_cov, size = [], []
for i in np.linspace(-0, 1.98, 30):
    cov = generate_cov(i)
    determinant_cov.append(det(cov))
    size.append(get_encircled_dots(cov).area)

plt.plot(cov_range, sqrt(determinant_cov)*pi, label='cov determinant')
plt.xlabel('covariance (off-diagonal elements) value')
plt.ylabel('area/area prediction/other quantities')
plt.plot(cov_range, size, label='ellipse size')
plt.legend()
plt.show()

print("Notice that these two lines overlap very well.")
print("In fact, they would be exactly the same if the number of samples we take approaches infinity.")
```

The important conclusion from this project is that, if we fix the variance values, increasing the absolute value of the covariance will make the error ellipse thinner. And the specific algorithm required to plot the error ellipse is also found.


# "An Overview of Initialization for Deep Neural Networks, Part 1"

> "Without appropriate choice of activation function and/or smart initialization for it neural networks, especially deep ones, tend to refuse training at all. We will consider sigmoid and tanh activation functions and two initialization methods most suitable for them - LeCun and Xavier (Glorot) initialization."

- toc: false
- branch: master
- badges: false
- comments: false
- categories: [fastpages, jupyter]
- use_math: true

For the past few years deep neural networks have proven themselves to be able to solve various AI-level tasks effectively. In some very specific tasks, such as image classification, game of Go, autonomous driving, and many others they already perform better than humans. These achievements were not possible without tremendous research invested in study of difficulties that occur during training of deep networks, and methods to overcome them. A part of this progress is due to better non-linearities and initialization methods.

## What Is the Choice of Activation Function and Initialization and Why Is It Important?

Neural networks are typically trained via stochastic gradient descent or one of its more powerful variants, such as RMSprop or Adam, that use gradients computed by the back propagation algorithm. The back propagation algorithm essentially recursively applies the chain rule of calculus to compute gradients efficiently. Just as a reminder, the chain rule of calculus look like this

\begin{equation}
    \frac{\partial}{\partial x}g(f(x)) = \frac{\partial g}{\partial f} \frac{\partial f}{\partial x}
\end{equation}

In earlier years folks used sigmoid activation function, defined as

\begin{equation}
    \sigma(x) = \frac{1}{1 + e^{-x}}
\end{equation}

However, soon it became evident that such nets are hard to train, because of some properties of sigmoid. One such property is the effect of saturation. Look at the graph of sigmoid.

```
def sigmoid(x):
    return 1. / (1. + np.exp(-x))

def sigmoid_derivative(x):
    return sigmoid(x) * (1. - sigmoid(x))

def plot_activation_and_derivative(activation, derivative, inputs, titles):
    fig, (ax_f, ax_d) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
    ax_f.plot(inputs, activation(inputs))
    ax_f.set_title(titles[0])
    ax_d.plot(inputs, derivative(inputs))
    ax_d.set_title(titles[1])
    plt.show()

values = np.arange(-6.0, 6.0, 0.01)
plot_activation_and_derivative(sigmoid, sigmoid_derivative, values,
                               ('Sigmoid activation', 'Derivative of sigmoid'))
```

When input values become large enough (whether positive or negative), output of sigmoid remains roughly unchanged and goes to 1 in case of positive inputs, or to 0 in case of negative. This saturation of sigmoid causes gradient vanishing. Since output values are roughly equal in those saturated tails, the derivative of sigmoid becomes very close to zero. As back propagation progresses towards input layers, gradients get multiplied, and if at least one of multiples becomes close to zero, the whole product is likely to become close to zero as well. Hence it can require too many gradient updates to tweak parameters in right way. Now look at the graph of the derivative of sigmoid more closely. You can note that its maximum value is 0.25 when input value is 0. So, in the best case, gradient gets four times smaller after sigmoid activation during back propagation.

There is another saturating activation function that, however, suffers less from vanishing gradient problems. It is called hyperbolic tangent, or tanh for short.

\begin{equation}
    \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}
\end{equation}

Actually it can be shown from their definitions that tanh can be expressed in terms of sigmoid as follows

\begin{equation}
    \tanh(x) = 2\sigma(2x) - 1
\end{equation}

In spite of that, tanh has two advantages over the sigmoid. First, it is symmetric in the origin, which makes it easier to learn identity function, and second, maximum value of its derivative is 1.

```
def tanh(x):
    return np.tanh(x)

def tanh_derivative(x):
    return 1. / np.cosh(x) ** 2

plot_activation_and_derivative(tanh, tanh_derivative, values,
                               ('Tanh activation', 'Tanh derivative'))
```

Tanh is widely used in recurrent neural networks today.

Nowadays we have many ways to address vanishing gradient and similar exploding gradient issues. And one of these is better initialization. Initialization is also important because gradient-based methods can stuck in bad local minima, not reaching the global minimum, and take long to converge because of plateus in the cost function surface. Better initialization reduces chances to start from "unlucky" points.

## LeCun Initialization

This is one of the first heuristics for initialization. It was proposed in 1998 by Yann LeCun et al. in the paper called "[Efficient BackProp](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)". LeCun initialization suggests that weights of network are drawn from normal distribution of the following form

\begin{equation}
    W \sim \mathcal{N}\Bigg(0, \frac{1}{fan_{in}}\Bigg)
\end{equation}

where $fan_{in}$ is the number of connections feeding into the node. This is also known as LeCun normal initialization, while in case of LeCun uniform initialization weights are drawn from uniform distribution

\begin{equation}
    W \sim \mathcal{U}\Bigg(-\frac{1}{\sqrt{fan_{in}}}, \frac{1}{\sqrt{fan_{in}}}\Bigg)
\end{equation}

LeCun initialization was designed specifically for sigmoid activation. But it has more applications than that. It was proposed to be used with the [SELU activation function and self-normalizing networks](https://arxiv.org/pdf/1706.02515.pdf).

Note that regardless the method of initialization, weights are usually drawn either from normal or uniform distribution, and there's connection between them. If we initialize from a normal distribution with $\mathrm{Var}[W] = \sigma^2$, we can easily switch to uniform distribution by computing $a = \sqrt{3\sigma^2}$ and sampling weights from $\mathcal{U}(-a, a)$. Because of this connection I will only consider normal distribution and specify suggested variance for it.

### Derivation of LeCun Initialization

The reasoning behind LeCun initialization is fairly simple. We don't want weights to be very large, because they will lead to saturation and cause gradient vanishing. We also don't want weights to be very small, because they will make gradient small as well and make the whole optimization slow. What we want is that initial weights map inputs into linear region of sigmoid (roughly between -2 and 2), so the gradient is large enough and network learns easier linear part first.

Consider a sigmoid layer and let $x$ denote input example and $y$ linearly transformed input, so

\begin{equation}
    y = xW \\
    y_i = \sum_{j = 1}^{fan_{in}}{x_{j}w_{ji}}
\end{equation}

We can think of bias term as weight that always receives 1 as input and omit it. To achieve the above requirements we first demand $x$ to have zero mean and variance of 1. This is satisfied by normalizing inputs. Now we want initialize weights so that they ensure $y$ also has zero mean and unit variance. Let's write down expression for the variance of $y_i$

\begin{equation}
    \mathrm{Var}[y_i] = \mathrm{Var}\Bigg[\sum_{j = 1}^{fan_{in}}{x_{j}w_{ji}}\Bigg] = \sum_{j = 1}^{fan_{in}} w_{ji}^2\mathrm{Var}[x_j] + \sum_{j \neq k}w_{ji}w_{ki}\mathrm{Cov}[x_j, x_k]
\end{equation}

Assuming uncorrelated $x$, we have $\mathrm{Cov}[x_j, x_k] = 0$ for all pairs $j \neq k$. Recall we assumed all $\mathrm{Var}[x_i] = 1$. Hence

\begin{equation}
    \mathrm{Var}[y_i] = \sum_{j = 1}^{fan_{in}} w_{ji}^2 \Rightarrow\ \mathrm{Var}[w] = \frac{\mathrm{Var}[y]}{fan_{in}} = \frac{1}{fan_{in}}
\end{equation}

And that's it! One thing that should be mentioned here is that original paper also proposed a variant of sigmoid that ensures $\sigma(y)$ preserves zero mean and unit variance of $y$.

## Xavier (Glorot) Initialization

The next method was proposed in paper "[Understanding the Difficulty of Training Deep Feedforward Neural Networks](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)" by Xavier Glorot and Yoshua Bengio in 2010. I don't know who decides whether we should call an invention after author's first name or last name, but this technique is widely known under both Xavier initialization and Glorot initialization.

If you have skimmed through the derivation of LeCun initialization above or read it in original paper, you might have noticed that it was motivated entirely from the perspective of the forward pass. In their paper Glorot and Bengio additionly require preserving variance of gradient during the backward pass. From one hand, the forward pass requires, as previously

\begin{equation}
    \mathrm{Var}[W] = \frac{1}{fan_{in}}
\end{equation}

They show that the backward pass from the other hand requires

\begin{equation}
    \mathrm{Var}[W] = \frac{1}{fan_{out}}
\end{equation}

where $fan_{out}$ is the number of connections going out of a unit.

Both constraints can be satisfied if and only if $fan_{in} = fan_{out}$, which isn't always the case. As a compromise authors suggest to set variance to

\begin{equation}
    \mathrm{Var}[W] = \frac{1}{fan_{avg}} = \frac{2}{fan_{in} + fan_{out}}
\end{equation}

Observe also that when $fan_{in} = fan_{out}$ Xavier initialization and LeCun initialization are equivalent.

The paper compares performance of several networks of the same architecture that only differ in activation function and initialization method. Those networks were trained on different data sets using the same procedure. The final test errors for networks using tanh activation are represented in the following table

TYPE | Shapeset | MNIST | CIFAR-10 | Small ImageNet
--- | --- | --- | --- | ---
Tanh | 27.15 | 1.76 | 55.9 | 70.58
Tanh N | 15.60 | 1.64 | 52.92 | 68.57

Here tanh stands for the network with tanh activations and LeCun uniform initialization, while tanh N - for the network with tanh activations and Xavier uniform initialization. As can be seen, on all data sets Xavier initialization gives little to significant boost in performance when compared to LeCun initialization.

A trouble with Xavier initialization is that it assumes linear units. This is a valid assumption when inputs are mapped to linear parts of non-linearities, but invalid in general. Nevertheless it still works pretty well. The following histograms of activation values and gradients from the paper show that Xavier initialization in experimental settings works as intended. Check out details in original paper!

### Derivation of Xavier Initialization

Consider a neural network with $L$ consequent layers. We begin with writing down equations for both forward and backward propagations. We make the same assumptions about input data as in the derivation of LeCun initialization. The forward pass is given by

\begin{equation}
    Z^{[i]}_k = \varphi^{[i]}(Z^{[i - 1]}_k) W^{[i]}_{:, k}
\end{equation}

And the backward pass by

\begin{equation}
    \frac{\partial\mathscr{L}}{\partial Z^{[i]}_k} = \varphi^{[i]\prime}(Z^{[i]}_k)W^{[i]}_{k, :}\frac{\partial\mathscr{L}}{\partial Z^{[i + 1]}} \\
    \frac{\partial\mathscr{L}}{\partial W^{[i]}_{lk}} = Z^{[i]}_l \frac{\partial\mathscr{L}}{\partial Z^{[i]}_k}
\end{equation}

Then for a layer number $i$, assuming that $\varphi^{[i]\prime}(z^{[i]}_k) = 1$

\begin{equation}
    \mathrm{Var}[Z^{[i]}] = \mathrm{Var}[X]\prod_{j=0}^{i-1}n^{[j]}\mathrm{Var}[W^{[j]}]\\
    \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg] = \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[L]}}\Bigg]\prod_{j = i}^{L - 1}n^{[j + 1]}\mathrm{Var}[W^{[j]}]\\
    \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial W^{[i]}}\Bigg] = \mathrm{Var}[Z^{[i]}]\mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg]
\end{equation}

From the perspective of the forward pass we wish

\begin{equation}
    \forall\ (i, j):\ \mathrm{Var}[Z^{[i]}] = \mathrm{Var}[Z^{[j]}]
\end{equation}

And from the back prop point of view we want

\begin{equation}
    \forall\ (i, j):\ \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg] = \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[j]}}\Bigg]
\end{equation}

These two constraints can be easily satisfied if

\begin{equation}
    \forall\ i:\ n^{[i]}\mathrm{Var}[W^{[i]}] = \mathrm{Var}[X] = 1\\
    \forall\ i:\ n^{[i + 1]}\mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[i]}}\Bigg] = \mathrm{Var}\Bigg[\frac{\partial\mathscr{L}}{\partial Z^{[L]}}\Bigg] = 1
\end{equation}

From these two equations we finally get

\begin{equation}
    \mathrm{Var}[W^{[i]}] = \frac{1}{n^{[i]}} = \frac{1}{fan_{in}} \\
    \mathrm{Var}[W^{[i]}] = \frac{1}{n^{[i + 1]}} = \frac{1}{fan_{out}}
\end{equation}

As a compromise for cases when $fan_{in} \neq fan_{out}$

\begin{equation}
    \mathrm{Var}[W^{[i]}] = \frac{2}{fan_{in} + fan_{out}}
\end{equation}

## Conclusions

Without appropriate choice of activation function and/or smart initialization for it neural networks, especially deep ones, tend to refuse training at all. Better activation functions and better initialization strategies are tightly related, so that they complement each other and evolve together. Once popular sigmoid non-linearity turned out to be impractical to use in hidden layers, and now is only used in the output layer for binary classification, while tanh that can be expressed in terms of sigmoid, is still useful for hidden layers. At the first glance initialization strategies seem some wisdom sent from heaven, but at least in case of LeCun and Xavier initialization their derivations are in fact straightforward. All you need to know is where to start and what assumptions to make.


# Advection using the k-Scheme
## CH EN 6355 - Computational Fluid Dynamics
**Prof. Tony Saad (www.tsaad.net)

slides at: www.tsaad.net

Department of Chemical Engineering

University of Utah**

Here, we will implement the k-scheme or kappa-schemes for advection. It is easiest to implement this scheme since for different values of k, we recover all sorts of high-order flux approximations. We will assume a positive advecting velocity for illustration purposes.

We are solving the constant speed advection equation given by
\begin{equation}
u_t = - c u_x = - F_x;\quad F = cu
\end{equation}
We will use a simple Forward Euler explicit method. Using a finite volume integration, we get
\begin{equation}
u_i^{n+1} = u_i^n - \frac{\Delta t}{\Delta x} (F_{i+\tfrac{1}{2}}^n - F_{i-\tfrac{1}{2}}^n)
\end{equation}
For constant grid spacing, the k-Scheme is given by
\begin{equation}
{\phi _f} = {\phi _{\rm{C}}} + \frac{{1 - k}}{4}({\phi _{\rm{C}}} - {\phi _{\rm{U}}}) + \frac{{1 + k}}{4}({\phi _{\rm{D}}} - {\phi _{\rm{C}}})
\end{equation}
which, for a positive advecting velocity, gives us
\begin{equation}
F_{i + {\textstyle{1 \over 2}}}^n = c\phi _{i + {\textstyle{1 \over 2}}}^n = c{\phi _i} + c\frac{{1 - k}}{4}({\phi _i} - {\phi _{i - 1}}) + c\frac{{1 + k}}{4}({\phi _{i + 1}} - {\phi _i})
\end{equation}

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
import matplotlib.animation as animation
plt.rcParams['animation.html'] = 'html5'
from matplotlib import cm
```

```python
def step(x,x0):
    x0 = 0.6
    x1 = 0.8
    result = x - x0
    result[x-x1<x1] = 1.0
    result[x<x0] = 0.0
    result[x>x1] = 0.0
    return result

def gaussian(x,x0):
    s = 0.08
    s = s*s
    result = np.exp( -(x-x0)**2/s)
    return result
```

```python
L = 1.0
n = 128 # cells
dx = L/n # n intervals
x = np.linspace(-3*dx/2, L + 3*dx/2, n+4) # include ghost cells - we will include 2 ghost cells on each side for high order schemes

# create arrays
phi = np.zeros(n+4) # cell centered quantity
f = np.zeros(n+4+1) # flux
u = np.ones(n+4+1) # velocity field - assumed to live on faces same as flux

x0 = 0.3
# u0 = np.zeros(N + 2)
# u0[1:-1] = np.sin(2*np.pi*x)
# u0 = np.zeros(N)
# phi0 = np.sin(np.pi*x)
phi0 = gaussian(x,x0) + step(x,x0)
# u0 = triangle(x,0.5,0.75,1)
# u0[0:N//2] = 1.0
plt.plot(x,phi0)
```

    [<matplotlib.lines.Line2D at 0x1199f8630>]

```python
cfl =0.5
c = 1.0 # use a negative value for left traveling waves
dt = cfl*dx/abs(c)
print('dt=',dt)
print('dx=',dx)
```

    dt= 0.00390625
    dx= 0.0078125

```python
# the k scheme
k = 0.5
# finite volume implementation with arrays for fluxes
t = 0
tend= L/abs(c)

sol = []
sol.append(phi0)
ims = []

fig = plt.figure(figsize=[5,3],dpi=200)
plt.rcParams["font.family"] = "serif"
plt.rcParams["font.size"] = 10
plt.rc('text', usetex=True)

# plt.grid()
plt.xlim([0.,L])
plt.ylim([-0.25,1.25])
plt.xlabel('$x$')
plt.ylabel('$\phi$')
plt.tight_layout()
# plot initial condition
plt.plot(x,phi0,'darkred',animated=True)

i = 0
while t < tend:
    phin = sol[-1]

#     if (i%16==0):
#         shift = int(np.ceil(c*(t-dt)/dx))
#         im = plt.plot(x[2:-2], np.roll(phin[2:-2], -shift) ,'k-o',markevery=2,markersize=3.5,markerfacecolor='deepskyblue',
#              markeredgewidth=0.25, markeredgecolor='k',linewidth=0.45, animated=True)
#         ims.append(im)

    # impose periodic conditions
    phin[-2] = phin[2]
    phin[-1] = phin[3]
    phin[0] = phin[-4]
    phin[1] = phin[-3]

    phi = np.zeros_like(phi0)

    # predictor - take half a step and use upwind
    # du/dt = -c*du/dx
    if c >= 0:
        ϕc = phin[1:-2] # phi upwind
    else:
        ϕc = phin[2:-1] # phi upwind

    f[2:-2] = c*ϕc
    phi[2:-2] = phin[2:-2] - dt/2.0/dx*(f[3:-2] - f[2:-3])
    phi[-2] = phi[2]
    phi[-1] = phi[3]
    phi[0] = phi[-4]
    phi[1] = phi[-3]

    # du/dt = -c*du/dx
    if c >= 0:
        ϕc = phi[1:-2] # phi upwind
        ϕu = phi[:-3]  # phi far upwind
        ϕd = phi[2:-1] # phi downwind
    else:
        ϕc = phi[2:-1] # phi upwind
        ϕu = phi[3:]  # phi far upwind
        ϕd = phi[1:-2] # phi downwind

    f[2:-2] = ϕc + (1-k)/4.0*(ϕc - ϕu) + (1+k)/4.0*(ϕd - ϕc)
    f = c*f # multiply the flux by the velocity
    # advect
    phi[2:-2] = phin[2:-2] - c * dt/dx*(f[3:-2] - f[2:-3]) #+ dt/dx/dx*diffusion
    t += dt
    i+=1
    sol.append(phi)

# plt.annotate('k = '+ str(k), xy=(0.5, 0.8), xytext=(0.015, 0.9),fontsize=8)
# plt.legend(('exact','numerical'),loc='upper left',fontsize=7)
# ani = animation.ArtistAnimation(fig, ims, interval=100, blit=True,
#                                 repeat_delay=1000)

# ani.save('k-scheme-'+str(k)+'.mp4',dpi=300,fps=24)
```

```python
plt.plot(sol[0], label='initial condition')
plt.plot(sol[-1], label='one residence time')
plt.legend()
plt.grid()
```

# Create Animation in Moving Reference Frame

```python
"""
Create Animation in Moving Reference Frame
"""
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
matplotlib.use("Agg")
fig, ax = plt.subplots(figsize=(4,3),dpi=150)
ax.grid(True)
f0 = sol[0]
line0, = ax.plot(x[2:-2], f0[2:-2] ,'r-',linewidth=0.75, animated=True)
line1, = ax.plot(x[2:-2], f0[2:-2] ,'k-o',markevery=2,markersize=3.5,markerfacecolor='deepskyblue',
             markeredgewidth=0.25, markeredgecolor='k',linewidth=0.45, animated=True)

ann = ax.annotate('time ='+str(round(t,3))+' s', xy=(2, 1), xytext=(40, 200),xycoords='figure points')
ax.annotate('k ='+str(k) + ' (k-scheme)', xy=(2, 1), xytext=(40, 190),xycoords='figure points')
plt.tight_layout()

def animate_moving(i):
    print('time=',i*dt)
    t = i*dt
    xt = x + i*1.1*c*dt
    line0.set_xdata(xt[2:-2])
    line1.set_xdata(xt[2:-2])
    ax.axes.set_xlim(xt[0],0.0*dx + xt[-1])
    f = sol[i]
    ax.axes.set_ylim(1.1*min(f) - 0.1,1.1*max(f))
    ann.set_text('time ='+str(round(t,4))+'s (' + str(i)+ ').')
    shift =int(np.ceil(i*c*dt/dx))
    line1.set_ydata(np.roll(f[2:-2], -shift))

    f0 = sol[0]
    line0.set_ydata(f0[2:-2])
    return line0,line1

# Init only required for blitting to give a clean slate.
def init():
    line0.set_ydata(np.ma.array(x[2:-2], mask=True))
    line1.set_ydata(np.ma.array(x[2:-2], mask=True))
    return line0,line1

ani = animation.FuncAnimation(fig, animate_moving, np.arange(0,len(sol),2*int(1/cfl)), init_func=init,
                              interval=20, blit=False)
print('done!')
```

    done!

```python
ani.save('__k-scheme_' + str(k)+'.mp4',fps=24,dpi=300)
```

```python
import urllib
import requests
from IPython.core.display import HTML
def css_styling():
    styles = requests.get("https://raw.githubusercontent.com/saadtony/NumericalMethods/master/styles/custom.css")
    return HTML(styles.text)
css_styling()
```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


```
from sympy.interactive import printing
printing.init_printing()

from sympy import var, simplify, collect, expand, Matrix, eye, inverse_laplace_transform, exp
```

```
var("s, t")
```

La matriz exponencial se define asi:

$$
\exp{(At)} = \sum_{i=0}^{\infty} \frac{1}{i!} (At)^i
$$

sin embargo, es completamente impractico calcularla de esta manera, principalmente porque es una serie infinita...

Dicho esto, hay otras maneras de calcularla, aqui exploraremos algunas de ellas.

# Calculo de matriz exponencial por la transformada de Laplace

El primer método esta definido por la siguiente ecuación:

$$
exp(At) = \mathcal{L}^{-1} \left\{ (sI - A)^{-1} \right\}
$$

por lo que podemos calcular rapidamente diferentes matrices exponenciales como ejemplo:

```
A1 = Matrix([[0, 1], [-1, -2]])
A1
```

```
(s*eye(2) - A1).inv()
```

```
[inverse_laplace_transform((s*eye(2) - A1).inv()[i], s, t) for i in range(4)]
```

En donde $\theta(t)$, es la función del escalon unitario de [Heavyside](http://en.wikipedia.org/wiki/Heaviside_step_function). Por el momento podemos ignorarla; digamos que se convierte en lo siguiente:

$$
\begin{bmatrix}
t+1 & t \\
-t & -t+1
\end{bmatrix} e^{-t}
$$

Podemos hacer otros ejemplos:

```
A2 = Matrix([[0, 1], [0, 0]])
A2
```

```
(s*eye(2) - A2).inv()
```

```
[inverse_laplace_transform((s*eye(2) - A2).inv()[i], s, t) for i in range(4)]
```

es decir:

$$
\begin{bmatrix}
1 & t \\
0 & 1
\end{bmatrix}
$$

```
A3 = Matrix([[0, 1],[-1, 0]])
A3
```

```
(s*eye(2) - A3).inv()
```

```
[inverse_laplace_transform((s*eye(2) - A3).inv()[i], s, t) for i in range(4)]
```

es decir:

$$
\begin{bmatrix}
\cos{t} & \sin{t} \\
-\sin{t} & \cos{t}
\end{bmatrix}
$$

# Calculo de la matriz exponencial por su forma de Jordan

Si bien el método anterior es una manera de calcular la matriz exponencial que funciona bien en la mayoria de los casos, impone restricciones de tiempo cuando la matriz es excesivamente grande, y el calculo de su inversa sería demasiado costoso. Por esta razon, existe un metodo mas elegante de calcular la matriz exponencial, que toma en cuenta algunas de las propiedades de esta matriz.

Consiste en notar que la representacion de [Jordan](http://en.wikipedia.org/wiki/Jordan_matrix) $J$ de una matriz cualquiera $A$, no es mas que la suma de dos matrices mas simples, un compuesta por los eigenvalores de $A$, $D$, y otra con el resto de los elementos, que invariablemente será [nilpotente](http://es.wikipedia.org/wiki/Nilpotente) $N$.

$$
T^{-1} A T = J = D + N
$$

por lo que si aplicamos la función exponencial, primero, a los elementos de la derecha, tendremos:

$$
\exp{(Jt)} = \exp{((D + N)t)} = \exp{(Dt)} \cdot \exp{(Nt)}
$$

donde:

$$
D =
\begin{bmatrix}
\lambda_1 & 0 & 0 & \dots & 0 \\
0 & \lambda_2 & 0 & \dots & 0 \\
0 & 0 & \lambda_3 & \dots & 0 \\
\vdots & \vdots & \vdots && \vdots \\
0 & 0 & 0 & \dots & \lambda_k
\end{bmatrix}
$$

$$
N =
\begin{bmatrix}
0 & * & * & \dots & * \\
0 & 0 & * & \dots & * \\
0 & 0 & 0 & \dots & * \\
\vdots & \vdots & \vdots && \vdots \\
0 & 0 & 0 & \dots & 0
\end{bmatrix}
$$

y por lo tanto:

$$
\exp{(Dt)} =
\begin{bmatrix}
e^{\lambda_1 t} & 0 & 0 & \dots & 0 \\
0 & e^{\lambda_2 t} & 0 & \dots & 0 \\
0 & 0 & e^{\lambda_3 t} & \dots & 0 \\
\vdots & \vdots & \vdots && \vdots \\
0 & 0 & 0 & \dots & e^{\lambda_k t}
\end{bmatrix}
$$

$$
\exp{(Nt)} = \sum_{j = 0}^{\infty} \frac{1}{j!} (Nt)^j = \sum_{j = 0}^{\gamma - 1} \frac{1}{j!} (Nt)^j
\quad \text{con}
\quad \gamma = \max{\left\{ \gamma_1, \gamma_2, \dots, \gamma_k \right\} }
$$

es decir, que para la matriz de eigenvalores, su exponencial esta compuesta por la exponencial de cada eigenvalor, y para la matriz nilpotente, solo tenemos que calcular la expansion de Taylor hasta $\gamma - 1$, siendo $\gamma$ a lo mas, el rango de la matriz.

Tomando el primer ejemplo de arriba:

```
T1, J1 = A1.jordan_form()
J1
```

donde podemos observar que las matrices $D$ y $N$ son:

```
D1 = Matrix([[-1, 0], [0, -1]])
D1
```

```
N1 = Matrix([[0, 1], [0, 0]])
N1
```

Teniendo estas matrices, tan solo tenemos que sacar la exponencial de cada una:

```
eD1 = (D1*t).exp()
eD1
```

y para $N$ tenemos que hacer la expansión de Taylor hasta $\gamma = \operatorname{rank} A - 1$, pero el rango de $A$ es 2, por lo que:

```
eN1 = eye(2) + N1*t
eN1
```

Si ahora regresamos a nuestra ecuación para la forma de Jordan, podemos ver que teniamos:

$$
T^{-1} A T = D + N
$$

de donde podemos concluir que:

$$
\exp{(T^{-1} \cdot At \cdot T)} = \exp{(Dt + Nt)}
$$

$$
T^{-1} \cdot \exp{(At)} \cdot T = \exp{(Dt)} \cdot \exp{(Nt)}
$$

$$
\exp{(At)} = T \cdot \exp{(Dt)} \cdot \exp{(Nt)} \cdot T^{-1}
$$

por lo que ahora solo tenemos que multiplicar cada una de estas matrices que ya hemos calculado:

```
eA1 = T1*eD1*eN1*T1.inv()
eA1
```

De donde facilmente se puede ver que:

$$
\begin{bmatrix}
t+1 & t \\
-t & -t+1
\end{bmatrix} e^{-t}
$$

Haciendo los ejemplos anteriores por este metodo:

```
T2, J2 = A2.jordan_form()
J2
```

```
D2 = Matrix([[0, 0], [0, 0]])
N2 = Matrix([[0, 1], [0, 0]])
```

```
eD2 = (D2*t).exp()
eD2
```

```
eN2 = eye(2) + N2*t
eN2
```

```
eA2 = T2*eD2*eN2*T2.inv()
eA2
```

```
T3, J3 = A3.jordan_form()
J3
```

```
D3 = J3
N3 = Matrix([[0, 0], [0, 0]])
```

```
eD3 = simplify((D3*t).exp())
eD3
```

```
eN3 = eye(2) + N3*t
eN3
```

```
eA3 = T3*eD3*eN3*T3.inv()
simplify(eA3)
```

Lo cual es:

$$
\begin{bmatrix}
\cos{(t)} & \sin{(t)} \\
-\sin{(t)} & \cos{(t)}
\end{bmatrix}
$$

### Nota
Cabe aclarar, que a pesar de que estos calculos fueron hechos a traves de software de calculo simbolico, estos métodos son principalmente para realizarse sin el uso de computadora, principalmente porque en lugar de hacer todo esto, lo más facil sería hacer:

```
(A1*t).exp()
```

```
(A2*t).exp()
```

```
simplify((A3*t).exp())
```

pero los métodos descritos anteriormente, son una buena opción cuando el uso de la computadora no es permitido.

IPythonNotebooks/Teoria de Control I/Matrix Exponencial.ipynb


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Cumulative Distribution Functions

A random process can be characterized by the statistical properties of its amplitude values. [Cumulative distribution functions](https://en.wikipedia.org/wiki/Cumulative_distribution_function) (CDFs) are one possibility to do so.

### Univariate Cumulative Distribution Function

The univariate CDF $P_x(\theta, k)$ of a continuous-amplitude real-valued random signal $x[k]$ is defined as

\begin{equation}
P_x(\theta, k) := \Pr \{ x[k] \leq \theta\}
\end{equation}

where $\Pr \{ \cdot \}$ denotes the probability that the given condition holds. The univariate CDF quantifies the probability that for the entire ensemble and for a fixed time index $k$ the amplitude $x[k]$ is smaller or equal to $\theta$. The term '*univariate*' reflects the fact that only one random process is considered.

The CDF shows the following properties which can be concluded directly from its definition

\begin{equation}
\lim_{\theta \to -\infty} P_x(\theta, k) = 0
\end{equation}

and

\begin{equation}
\lim_{\theta \to \infty} P_x(\theta, k) = 1
\end{equation}

The former property results from the fact that all amplitude values $x[k]$ are larger than $- \infty$, the latter from the fact that all amplitude vales lie within $- \infty$ and $\infty$. The univariate CDF $P_x(\theta, k)$ is furthermore a non-decreasing function.

The probability that $\theta_1 < x[k] \leq \theta_2$ is given as

\begin{equation}
\Pr \{\theta_1 < x[k] \leq \theta_2\} = P_x(\theta_2, k) - P_x(\theta_1, k)
\end{equation}

Hence, the probability that a continuous-amplitude random signal takes a specific value $x[k]=\theta$ is zero when calculated by means of the CDF. This motivates the definition of probability density functions introduced later.

### Bivariate Cumulative Distribution Function

The statistical dependencies between two signals are frequently of interest in statistical signal processing. The bivariate or joint CDF $P_{xy}(\theta_x, \theta_y, k_x, k_y)$ of two continuous-amplitude real-valued random signals $x[k]$ and $y[k]$ is defined as

\begin{equation}
P_{xy}(\theta_x, \theta_y, k_x, k_y) := \Pr \{ x[k_x] \leq \theta_x  \wedge y[k_y] \leq \theta_y  \}
\end{equation}

The joint CDF quantifies the probability for the entire ensemble of sample functions that for a fixed $k_x$ the amplitude value $x[k_x]$ is smaller or equal to $\theta_x$ and that for a fixed $k_y$ the amplitude value $y[k_y]$ is smaller or equal to $\theta_y$. The term '*bivariate*' reflects the fact that two random processes are considered. The bivariate CDF can also be used to characterize the statistical properties of one random signal $x[k]$ at two different time-instants $k_x$ and $k_y$ by setting $y[k] = x[k]$.

The definition of the bivariate CDF can be extended straightforward to the case of more than two random variables. The resulting CDF is termed as multivariate CDF.

## Probability Density Functions

[Probability density functions](https://en.wikipedia.org/wiki/Probability_density_function) (PDFs) describe the probability for one or multiple random signals to take on a specific value. First the univariate case is discussed.

### Univariate Probability Density Function

The univariate PDF $p_x(\theta, k)$ of a continuous-amplitude real-valued random signal $x[k]$ is defined as the derivative of the univariate CDF

\begin{equation}
p_x(\theta, k) = \frac{\partial}{\partial \theta} P_x(\theta, k)
\end{equation}

Due to the properties of the CDF and the definition of the PDF, it shows the following properties

\begin{equation}
p_x(\theta, k) \geq 0
\end{equation}

and

\begin{equation}
\int\limits_{-\infty}^{\infty} p_x(\theta, k) \, \mathrm{d}\theta = P_x(\infty, k) = 1
\end{equation}

The univariate PDF has only positive values and the area below the PDF is equal to one.

The CDF can be derived from the PDF by integration

\begin{equation}
P_x(\theta, k) = \int\limits_{-\infty}^{\theta} p_x(\theta, k) \, \mathrm{d}\theta
\end{equation}

due to its definition as derivative of the CDF.

#### Example - Estimate of an univariate PDF by the histogram

In the process of calculating a [histogram](https://en.wikipedia.org/wiki/Histogram), the entire range of amplitude values of a random signal is split into a series of intervals (bins). For a given random signal the number of samples is counted which fall into one of these intervals. This is repeated for all intervals. This process constitutes a numerical approximation of the PDF.

In the following example the histogram of an ensemble of random signals is computed for each time index $k$. The CDF is computed by taking the cumulative sum over the histogram bins. This constitutes a numerical approximation of above integral.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

K = 32  # number of temporal samples
N = 10000  # number of sample functions
bins = 100  # number of bins for the histogram

# draw sample functions from a random process
np.random.seed(2)
x = np.random.normal(size=(N, K))
x += np.tile(np.cos(2*np.pi/K*np.arange(K)), [N, 1])

# compute the histogram
px = np.zeros((bins, K))
for n in range(K):
    px[:, n], edges = np.histogram(x[:, n], bins=bins, range=(-4,4), density=True)

# compute the CDF
Px = np.cumsum(px, axis=0) * 8/bins

# plot the PDF
plt.figure(figsize=(10,6))
plt.pcolor(np.arange(K), edges, px)
plt.title(r'Estimated PDF $\hat{p}_x(\theta, k)$')
plt.xlabel(r'$k$')
plt.ylabel(r'$\theta$')
plt.colorbar()
plt.autoscale(tight=True)

# plot the CDF
plt.figure(figsize=(10,6))
plt.pcolor(np.arange(K), edges, Px, vmin=0, vmax=1)
plt.title(r'Estimated CDF $\hat{P}_x(\theta, k)$')
plt.xlabel(r'$k$')
plt.ylabel(r'$\theta$')
plt.colorbar()
plt.autoscale(tight=True)
```

**Exercise**

* Change the number of sample functions `N` or/and the number of `bins` and rerun the examples. What changes? Why?

In numerical simulations of random processes only a finite number of sample functions and temporal samples can be considered. This holds also for the number of intervals (bins) used for the histogram. As a result, numerical approximations of the CDF/PDF will be subject to statistical uncertainties that typically will become smaller if the number of sample functions `N` is increased.

### Bivariate Probability Density Function

The bivariate or joint PDF $p_{xy}(\theta_x, \theta_y, k_x, k_y)$ of two continuous-amplitude real-valued random signals $x[k]$ and $y[k]$ is defined as

\begin{equation}
p_{xy}(\theta_x, \theta_y, k_x, k_y) := \frac{\partial^2}{\partial \theta_x \partial \theta_y} P_{xy}(\theta_x, \theta_y, k_x, k_y)
\end{equation}

The bivariate PDF quantifies the joint probability that $x[k]$ takes the value $\theta_x$ and that $y[k]$ takes the value $\theta_y$ for the entire ensemble of sample functions.

If $x[k] = y[k]$ the bivariate PDF $p_{xx}(\theta_1, \theta_2, k_1, k_2)$ describes the probability that the random signal $x[k]$ takes the value $\theta_1$ at time instance $k_1$ and the value $\theta_2$ at time instance $k_2$. Hence, $p_{xx}(\theta_1, \theta_2, k_1, k_2)$ provides insights into the temporal dependencies of a random signal $x[k]$.

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016*.


Creating bell like shape using numpy

```python
import numpy as np
```

```python
import matplotlib.pyplot as plt
```

```python
import sympy
```

```python
df = 300
db = 16
r = 800
```

```python
a,b,c,x = sympy.symbols('a b c x')
```

```python
p = 0.7
```

```python
func= a/(x+b)**p + c
```

```python
eq1 = sympy.Eq(func.subs(x,0),df)
```

```python
eq2 = sympy.Eq(func.subs(x,120),60)
```

```python
eq3 = sympy.Eq(func.subs(x,r),db)
```

```python

```

```python
sympy.mpmath.findroot([eq1,eq2,eq3],[a,b,c])
```

```python
xx = np.linspace(0,r,801,endpoint = True)
```

```python
yy = np.zeros(len(xx),dtype=float)
```

```python
len(yy)
```

    801

```python
p = 0.75
for i in range(len(xx)):
    yy[i] = bell(xx[i],0.5)
```

```python
plt.plot(xx,yy)
```

```python
# reverse data
rx = xx[::-1]
ry = yy[::-1]
```

```python
# export data as a bell.xmen
f = open('../sample/bell.xmen','w')
for i in range(len(rx)-1):
    f.write('{0},{1},{2}\n'.format(ry[i],ry[i+1],-rx[i+1]+rx[i]))
f.close()
```

```python

```