# Integrals and derivatives

One of the most basic but also most important application of computers in physics is the evaluation of integrals and derivatives. Numerical evaluation of integrals is a particularly crucial topic, because integrals occur widely in physics and, while some integrals can be done analytically in closed form, most cannot. They can, however, almost always be solved on a computer. Conversely, derivatives can almost always be performed analytically, if a close analytic expression of the expression is available. However, we may not always be interested in an analytic expression of the derivative, since the expression could be too complicated or because its evaluation is more involved than a numerical derivative. In this chapter we examine a number of different techniques for evaluating integrals and derivatives, as well as taking a brief look at the related operation of interpolation.

## Integration

In integration our objective is to find the value of a definite integral with fixed bounds from $x=a$ to $x=b$ $$ I(a,b)=\int_a^b f(x)dx.$$ Note that this is a much simpler problem than finding the antiderivative (or the indefinite integral) $F(y)=\int_0^y f(x)dx$. The simplest and most naive way to solve the definite integral is to discretize the sum and to assign a rectangle to each segment $$I(a,b)\approx\sum_{i=k}^N f(x_k)h.$$ To discretize the continuous variable x, we take the interval $[a,b]$ and divide it into $N$ equally sized portions $h=(b-a)/N$. The sequence of $x$-points then becomes $x_k=a+(k-\frac{1}{2})h$.

Discretization is a core concept in computational science. An example of the simple rectangular integration is shown in panel a) of the following figure.

We can now write a little python program that evaluates the sum for the integral. We do this for the function $f(x)=x^4-2x+1$ from $a=0$ to $b=2$. We know the correct answer for this integral and compare the result of our program to it $$\int_0^2 (x^4-2x+1)dx = \left[\frac{1}{5}x^5-x^2+x\right]_0^2=4.4 .$$

```python
def f(x):
    return x**4 - 2*x + 1

N = 10
a = 0.0
b = 2.0
h = (b-a)/N

s = 0.0
for k in range(1,N):
    s += f(a+(k-0.5)*h)

print(h*s)
```

    2.3003400000000007

With only 10 points, the value of the integral is not very good, because the constant value of the function that we assume in each interval does not reflect the curvature of the function $f(x)$ very well. We could improve the accuracy of our integration by using more points (i.e. by making the interval smaller and smaller). This strategy will eventually lead to success, but it might require a large number of points. Maybe we can do something smarter.

### Trapezoidal rule

Instead of replacing the function in each interval with a horizontal line, we could represent the function by a line that runs through the function values at the end point of each interval. Such sloped lines, would capture more rapidly varying functions better. The idea is shown in panel b) of the previous figure.

The two end-points of each segment are $a+(k-1)h$ and $a+kh$. Following the $\textit{trapezoidal rule}$ this gives the simple formula for the area of the segment $$A_k=\frac{1}{2}h\left[ f(a+(k-1)h)+f(a+k h)\right].$$ All we have to do now is sum up the segments again to obtain the integral $$\begin{align} I(a,b)\approx \sum_{k=1}^N A_k &=\frac{1}{2}h\sum_{k=1}^N \left[ f(a+(k-1)h)+f(a+kh)\right]  \\ &=h \left[ \frac{1}{2}f(a)+\frac{1}{2}f(b)+\sum_{k=1}^{N-1} f(a+kh) \right]\end{align}$$

We can easily modify the code for the rectangular integration to incorporate the trapezoidal rule:

```python
def f(x):
    return x**4 - 2*x + 1

N = 10
a = 0.0
b = 2.0
h = (b-a)/N

s = 0.5*f(a) + 0.5*f(b)
for k in range(1,N):
    s += f(a+k*h)

print(h*s)
```

    4.50656

We see that with still only 10 discretization points, we immediately achieve a better integration accuracy. With a value of 4.507, we are now $\sim$2\% off.

### Simpson's rule

The trapezoidal rule approximated the integrand with linear segments. To improve on the trapezoidal rule, we make an attempt to represent the function better. We do this by a powerlaw expansion. The simplest expansion beyond linear is quadratic. To fit a quadratic piece to our function requires 3 points, which implies that we now need two discretized segments for each fit. This is schematically shown in the figure.

Denoting our 3 points $-h$, $0$ and $h$, we fit a quadratic function $Ax^2+Bx+C$ through these points. At the 3 points the function assumes the following values $$\begin{gather} f(-h)=Ah^2-Bh+C, & f(0)=C, & f(h)=Ah^2+Bh+C\end{gather}.$$ Solving this system of equations provides the desired expressions for the 3 coefficients $$\begin{gather} A=\frac{1}{h^2}\left[\frac{1}{2}f(-h)-f(0)+\frac{1}{2}f(h)\right], & B=\frac{1}{2h}\left[f(h)-f(-h)\right], & C=f(0)\end{gather}.$$ These expressions we insert into the integral of the quadratic approximation to the integral under the curve of the first two segments: $$\int_{-h}^{h} (Ax^2+Bx+C)dx=\frac{2}{3}Ah^3+2Ch=\frac{1}{3}\left[f(-h)+4f(0)+f(h)\right] .$$

Now we have to generalize this expression to incorporate also the remaining segments. We do this by sliding the procedure along in pairs of segments and summing up the results. In general, the three points of our first pair of segments are at $a$, $a+h$ and $a+2h$. The three points for the next pair of segments lie at $a+2h$, $a+3h$ and $a+4h$, and so forth. The approximate value of the integral then becomes $$\begin{align}I(a,b)\approx & \frac{h}{3}\left[ f(a)+4f(a+h)+f(a+2h)\right] \\ & +\frac{h}{3} \left[f(a+2h)+4f(a+3h)+f(a+4h) \right] + \ldots \\ & +\frac{h}{3} \left[f(a+(N-2)h+4f(a+(N-1)h)+f(b)\right] \end{align}$$ Note that the total number of slices must be even for this to work. Collecting terms, we now have $$\begin{align} I(a,b)\approx & \frac{h}{3} \left[f(a)+4f(a+h)+2f(a+2h)+4f(a+3h)+\ldots + f(b)\right] \\ &=\frac{h}{3} \left[f(a)+f(b)+4\!\!\!\!\sum_{\: \: k \: \textrm{odd} \\ 1\ldots N-1} \!\! f(a+kh)+2 \!\!\!\!\sum_{\: \: k \: \textrm{even} \\ 2\ldots N-2} \!\! f(a+kh) \right] \end{align} $$ This formula is called the $\textit{extended Simpson's rule}$.

The sums over odd and even values of $k$ can be conveniently accomplished in Python by using a for loop of the form "$\texttt{for k in the range(1,N,2)}$" for the odd terms and "$\texttt{for k in the range(2,N,2)}$" for the even terms. Alternatively, we could rewrite Simpson's rule as $$ I(a,b)\approx \frac{h}{3} \left[f(a)+f(b)+4\sum_{k=1}^{N/2} f(a+(2k-1)h)+2\sum_{k=1}^{N/2-1}f(a+2kh) \right] $$ and use an ordinary for loop.

With this insight, we modify our integration code once more to adapt it to the Simpson's rule:

```python
def f(x):
    return x**4 - 2*x + 1

N = 10
a = 0.0
b = 2.0
h = (b-a)/N

s = f(a) + f(b)
for k in range(1,N,2):
    s += 4*f(a+k*h)

for k in range(2,N,2):
    s += 2*f(a+k*h)

print(h*s/3.0)
```

    4.400426666666668

In summary, we have seen three increasingly more sophisticated integration algorithms. Here is an overview of their performance for 10 integration points: $$ \begin{array}{llrr} \text{Method} & & \text{Value} & \text{Error} \\\hline  \text{Flat} & : & 2.3003 & 52.20 \% \\ \text{Trapezoidal} & : & 4.5066 & 2.40\% \\ \text{Simpson} & : & 4.4004 & 0.01\% \\ \text{Exact} & : & 4.4000 \\  \end{array} $$

# Higher-order integration methods

In the previous section, we have increased the sophistication of approximating the integrand in each method. We went from constants (0th order) to lines (1st order (trapezoidal)) to quadratic curves (2nd order (Simpson's rule)). In principle, we could keep going to higher and higher orders. The general expression for the integral is $$\int_a^b f(x)dx\approx \sum_{k=1}^{N}w_kf(x_k),$$ where $x_k$ are the positions at which we evaluate the integrand and $w_k$ are a set of weights given by the integration method. Since we used homogenuous grids (i.e. fixed grid spacing), we can split off the grid spacing $h$ from the weights $w_k=c_k h$. The different integration methods then only differ in the set of coefficients that make up the integration weights. Different coefficient sets are summarized in the table below.  $$ \begin{array}{lll} \text{Order} & \text{Polynomial} & \text{Coefficients} \\\hline \text{0 (naive)} & \text{constant} & 1,1,1,\ldots,1 \\ \text{1 (trapezoidal rule)} & \text{straight line} & \frac{1}{2}, 1, 1, \ldots, 1, \frac{1}{2} \\ \text{2 (Simpson's rule)} & \text{quadratic} & \frac{1}{3},\frac{4}{3},\frac{2}{3},\frac{4}{3},\ldots,\frac{4}{3},\frac{1}{3} \\ \text{3} & \text{cubic} & \frac{3}{8},\frac{9}{8},\frac{9}{8},\frac{3}{4},\frac{9}{8},\frac{9}{8},\frac{3}{4},\ldots, \frac{9}{8},\frac{3}{8}\\ \text{4} & \text{quartic} & \frac{14}{45}, \frac{64}{45},\frac{8}{15},\frac{64}{45},\frac{28}{45},\frac{64}{45}, \frac{8}{15},\frac{64}{45}, \ldots, \frac{64}{45},\frac{14}{45} \end{array}$$

We can now view integration as a simple sum over integration $\textit{weights}$ and the integrand evaluated at certain discretized $\text{grid}$ points. The weights depend on the integration method and give the weighting of the integrand at a corresponding grid point in the sum. With this new perspective we can rethink our integration program and make it more modular. If we create a function that prefills the weights, we can compute integrals for different methods in the same code framework. Here is an example of our Simpson rule integration adapted to a sum over a grid with the corresponding weights.

```python
from numpy import empty,array
def f(x):
    return x**4 - 2*x + 1

# variable definitions
N = 10
a = 0.0
b = 2.0
h = (b-a)/N
s = 0.0
weights = empty(N+1,float)
grid = empty(N+1,float)

# grid specification (this step is in principle redundant, but we use it to illustrate the concept)
for k in range (0,N+1):
    grid[k]=a+k*h

# weights specification (only this part would need to be changed for a new method)
for k in range(1,N,2):
    weights[k]=h*4.0/3.0

for k in range(2,N,2):
    weights[k]=h*2.0/3.0

weights[0]=h/3.0
weights[N]=h/3.0

# actual integration
for k in range(0,N+1):
    s += weights[k]*f(grid[k])

print(s)
```

    4.400426666666667

The program gives the same value as its orginal version. We can now, however, easily upgrade it by modifying the section that specifies the weights. In principle, we could even outsource the weight definitions into subroutines.

# Gaussian quadrature

Equipped with our new perspective of integration as looping over a grid with weighted integration points, we now consider non-uniform grids. We will choose the position of the grid points such that they are optimal for the integration. The integration weights are then derived correspondingly.

For simplicity, we restrict the domain of integration to $[-1,1]$ and will later scale it back to $[a,b]$. Our objective is then to find the grid points $x_k$ and weights $w_k$ for the integral $$\int_{-1}^{1}f(x)dx\approx\sum_{k=1}^Nw_kf(x_k).$$ Let us assume that our function $f(x)$ is a polynomial in $x$ of degree $2N-1$. We can then use the properties of the Legendre polynomials for our integration method:
 - The legendre polynomial $P_N(x)$ is orthogonal to every polynomial of lower degree, i.e. $ \int_{-1}^{1}x^k P_N(x)dx=0$ for all integers $k$ in the range $0 \leq k \lt N$.
 - For all $N$, $P_N(x)$ has $N$ real roots that all lie in the interval from $-1$ to $1$.

If we now divide $f(x)$ by the Legendre polynomial $P_N(x)$, we get $$f(x)=q(x)P_N(x)+r(x),$$ where $q(x)$ and $r(x)$ are both polynomials of degree $N-1$ or less. This simplifies our original integral $$\int_{-1}^{1} f(x)dx=\int_{-1}^{1} q(x)P_N(x)dx+\int_{-1}^{1}r(x)dx=\int_{-1}^{1}r(x)dx, $$ because $P_N(x)$ is orthogonal to $q(x)$. This transformation has not gained us anything, since we do not know the function $r(x)$. However, we can make the same substitution in the summation over the grid and weights: $$ \sum_{k=1}^N w_kf(x_k)=\sum_{k=1}^Nq(x_k)P_N(x_k)+\sum_{k=1}^Nw_kr(x_k)$$
We now make use of the second property of the Legendre polynomials, namely, that $P_N(x)$ has $N$ many zeros between $-1$ and $1$. We take these zero positions (roots) as the grid points $x_k$ so that $P_N(x_k)=0$ for all $k$. This simplifies our sum to $$\sum_{k=1}^N w_k f(x_k) = \sum_{k=1}^N w_k r(x_k)=\int_{-1}^{1} r(x) dx = \int_{-1}^{1} f(x) dx .$$ The second equality holds only, because we assumed that $f(x)$ is a polynomial of degree $2N-1$, which makes $r(x)$ a polynomial of degree $N-1$ or less.

To proceed, we have to do the following:

 - Find the roots of $P_N(x)$: this is possible, but tedious. The derviation can be found e.g. in (Abramowitz & Stegun 1972, p. 887).

 - Given a set of $x_k$ grid points (corresponding to the roots of $P_N(x)$) compute the corresponding weights.

For the second point, we assume that we can find a single polynomial of degree $N-1$ to fit the function $f(x)$ or $r(x)$. For this, we use the _method of interpolating polynomials_. Our _interpolating polynomial_ is $$ \begin{align} \phi_k(x) & =\prod_{m=1\ldots N \\ \:\:\: m\neq k}\frac{(x-x_m)}{(x_k-x_m)} \\ & = \frac{(x-x_1)}{(x_k-x_1)} \times \ldots \times \frac{(x-x_{k-1})}{(x_k-x_{k-1})} \frac{(x-x_{k+1})}{(x_k-x_{k+1})} \times \ldots \times  \frac{(x-x_N)}{(x_k-x_N)} \end{align}$$ Note that the numerator contains one factor for each sample point except the point $x_k$. $\phi_k(x)$ is thus a polynomial of degree $N-1$. It has the property $$\phi_k(x_m)=\delta_{km}.$$ We use this property to define a new function $\Phi(x)$: $$\Phi(x)=\sum_{k=1}^Nf(x_k)\phi_k(x),$$ which is also a polynomial of degree $N-1$. By definition $\Phi(x)$ fits $f(x)$ exactly at $x_m$: $$\Phi(x_m)=\sum_{k=1}^Nf(x_k)\phi_k(x_m)=\sum_{k=1}^Nf(x_k)\delta_{km}=f(x_m).$$

We now insert $\Phi(x)$ into our integral $$\int_{-1}^{1}f(x)dx\approx\int_{-1}^{1}\Phi(x)dx=\int_{-1}^{1}\sum_{k=1}^Nf(x_k)\phi_k(x)dx=\sum_{k=1}^Nf(x_k)\int_{-1}^{1}\phi_k(x)dx=\sum_{k=1}^Nf(x_k)w_k.$$ In other words, for an arbitrary set of points $x_k$, the integration weights are given by $$w_k=\int_{-1}^{1}\phi_k(x)dx.$$

This is a general expression for finding integration weights for a given set of grid points. For the Gauss Legendre quadrature, we insert the roots of $P_N(x)$ as $x_k$ into the interpolating polynomial and then carry out the integral to find the corresponding weights $w_k$. For small $N$, analytic expressions have been derived. In general, however, roots and weights are calculated numerically. We have done this in the accompanying program gaussxw.py. The figure below shows the positions of the grid points and the values of the corresponding weights for $N=10$ and $N=100$. The grid points are not evenly spaced. Their density increases towards the edges of the integration interval. Concomitantly, the weights are lowest at the interval edges and rise in towards the middle of the interval, where the point density is lowest.

The calculation of the grid points and weights in Gaussian quadrature takes a bit of effort. In return, Gaussian quadrature exactly integrates functions that are polynomials of degree $2N-1$ with only $N$ grid points. This is due to the properties of the Legendre polynomials and the clever grid choice. As an example, we will consider our test function $x^4-2x+1$. This time, we will integrate it with only $N=3$ grid points using Gaussian quadrature.

Before we perform the integration, we need to briefly consider how to change the integration interval from $[-1,1]$ to $[a,b]$. Since the area under a curve does not depend on where that curve is along the $x$ axis, the sample points can be freely slid up and down the $x$ axis _en masse_. If the desired domain is wider or narrower than the interval from $-1$ to $1$ then we also need to spread the points out or squeeze them together. The stretching or squeezing operation that accomplishes this is $$x_k'=\frac{1}{2}(b-a)x_k+\frac{1}{2}(b+a).$$ Similarly, the weights do not change, if we are simply sliding the sample points up and down the $x$ axis, but if the width of the integration domain changes then the value of the integral will increase or decrease by a corresponding factor. The weights have to be rescaled accordingly $$w_k'=\frac{1}{2}(b-a)w_k.$$

Here, finally, is then then the Python program that integrates $x^4-2x+1$ with only 3 grid points using Gaussian quadrature.

```python
from gaussxw import gaussxw

def f(x):
    return x**4 - 2*x + 1

N = 3
a = 0.0
b = 2.0

# Calculate the sample points and weights, then map them
# to the required integration domain
x,w = gaussxw(N)
xp = 0.5*(b-a)*x + 0.5*(b+a)
wp = 0.5*(b-a)*w

# Perform the integration
s = 0.0
for k in range(N):
    s += wp[k]*f(xp[k])

print(s)
```

    4.4000000000000075

# Choosing an integration method

Which integration method should you use in practice? There is no general answer, because the performance of the methods depends on the nature of the integrand. If in doubt, try several methods. Below is a brief table of advantages and disadvantes of each method. $$ \begin{array}{lll} \text{Method} & \text{Advantage} & \text{Disadvantage} \\\hline  \text{Trapezoidal} & \text{simple and versatile; works for pathological integrands and noisy data} & \text{limited accuracy} \\ \text{Simpson} & \text{simple with good accuracy} & \text{less suitable for pathological integrands and noisy data}  \\ \text{Gaussian} & \text{very good accuracy} & \text{less suitable for pathological integrands and noisy data} \\  \end{array} $$

# Integrals over infinite ranges

Often in physics we encounter integrals over infinite ranges, like $\int_0^\infty f(x)dx$. The techniques we have used so far will not work for such integrals, because we would need an infinite number of integration points. The solution to this problem is to change variables. For an integral over the range from $0$ to $\infty$ the standard change of variables is $$z=\frac{x}{1+x} \quad \text{or equivalently} \quad x=\frac{z}{1-z} .$$ With $dx=dz/(1-z)^2$ we obtain $$\int_0^\infty f(x)dx=\int_0^1 \frac{1}{(1-z)^2} f(\frac{z}{1-z})dz,$$ which can be done using any of the techniques covered earlier in the chapter.

For integrals over a range from a nonzero value $a$ to $\infty$, we have to make two changes of variables. First we change $y=x-a$, which shifts the start of the integration range to $0$, and then apply the previous substitution, but this time in $y$: $z=y/(1+y)$. Or we combine both changes into a single one: $$ z=\frac{x-a}{1+x-a} \quad \text{or equivalently} \quad x=\frac{z}{1-z}+a .$$ Again $dx=dz/(1-z)^2$, so that we end up with $$\int_a^\infty f(x)dx=\int_0^1 \frac{1}{(1-z)^2} f(\frac{z}{1-z}+a)dz .$$

Integrals from $-\infty$ to $a$ can be done the same way by substituting $z \rightarrow -z$ and integrals from $-\infty$ to $\infty$ can be broken up into to integrals from $-\infty$ to $0$ and from $0$ to $\infty$. Alternatively, we could use a single change of variables, such as $$x=\frac{z}{1-z^2}, \quad dx=\frac{1+z^2}{(1-z^2)^2}dz,$$ which would give $$\int_{-\infty}^\infty f(x)dx=\int_{-1}^1 \frac{1+z^2}{(1-z)^2} f(\frac{z}{1-z^2})dz.$$

As an example, we will calculate the value of the following integral using Gaussian quadrature: $$I=\int_0^\infty e^{-t^2}dt$$ We make the change of variables $z=t/(1+t)$ and the integral becomes $$I=\int_0^1 \frac{e^{-z^2/(1-z)^2}}{(1-z)^2}dz .$$ We modify our program for the function $x^4-2x+1$ and perform the integration with $N=50$ grid points.

```python
from gaussxw import gaussxwab
from math import exp

def f(z):
    return exp(-z**2/(1-z)**2)/(1-z)**2

N = 50
a = 0.0
b = 1.0
x,w = gaussxwab(N,a,b)
s = 0.0
for k in range(N):
    s += w[k]*f(x[k])
print(s)
```

    0.8862269254528349

The value of this integral is known analytically: $I=\frac{1}{2}\sqrt{\pi}=0.886226925453\ldots$. Again we see the impressive accuracy of the Gaussian quadrature method. With just 50 sample points, we have calculated an estimate of the integral that is correct to the limits of machine precision.

# Multiple integrals

Integrals over more than one variable are common in physics problems and can be tackled using generalizations of the methods we have already seen. Consider for instance the integral $$I=\int_0^1\int_0^1f(x,y) dx dy.$$ We can rewrite this by defining a function $F(y)$ thus $$F(y)=\int_0^1f(x,y)dx.$$ Then our integral becomes $$I=\int_0^1 F(y) dy.$$ We can thus do multiple integrals numerically by first integrating in one variable and then in the others. For instance, if we do the integrals by Gaussian quadrature with the same number $N$ of points for both $x$ and $y$ integrals, we have $$F(y)\approx \sum_{i=1}^N w_i f(x_i,y) \quad \text{and} \quad I\approx \sum_{j=1}^N w_j F(y_j) .$$ Substituting the first into the second sum not surprisingly gives us the following double sum for the integral $$ I\approx\sum_{i=1}^N\sum_{j=1}^N w_i w_j f(x_i,y_j). $$ This expression has a form similar to the standard integration formula for single integrals with a sum over values of the function $f(x,y)$ at a set of grid points, multiplied by appropriate weights. Now the points are distributed on a two dimensional grid. The approach can be generalized to arbitrary dimensions. The figure below shows the points on a two dimensional Gaussian quadradture grid.

# Derivatives

## Forward and backward differences

The standard definition of a derivative, the one you see in calculus books, is $$\frac{df}{dx}=\lim_{h\rightarrow 0}\frac{f(x+h)-f(x)}{h} .$$ The basic method for calculating numerical derivatives is precisely an implementation of this formula. We cannot take the limit $h \rightarrow 0$ in practice, but we can make $h$ very small and then calculate $$\frac{df}{dx} \approx \frac{f(x+h)-f(x)}{h} .$$ This is called the _forward difference_, because it is measured in the forward (i.e. positive) direction from the point of interest $x$. Analogously, we can define the _backward difference_ $$\frac{df}{dx} \approx \frac{f(x)-f(x-h)}{h} .$$ Both are shown in the figure below. They usually give the same value, provided $h$ is small enough and the function is not pathological.

## Errors

Forward and backward difference are usually not the most accurate. To understand this, we Taylor expand $f(x)$ around $x$: $$f(x+h)=f(x)+hf'(x)+\frac{1}{2}h^2f''(x)+\ldots ,$$ where $f'$ and $f''$ denote the first and second derivatives of $f$, respectively. Rearranging gives us $$f'(x)=\frac{f(x+h)-f(x)}{h}-\frac{1}{2}hf''(x)+\dots .$$ The first part is our numeric forward difference. The following terms we omit in our numeric difference. They contribute to the error and are proportional to $h$ in leading order ($\frac{1}{2}h|f''(x)|$). It would seem that if we make $h$ smaller, we would reduce the error. However, we are bound by machine precision and that affects the accuracy of the numeric difference, which means we have a lower bound for $h$. Since $f(x+h)$ and $f(x)$ are typically close in value the total round error on $f(x+h)-f(x)$ will be $2C|f(x)|$, where $C$ is machine precision (typically $10^{-16}$ in Phython). The total error for our derivative is thus $$\epsilon =\frac{2C|f(x)|}{h} + \frac{1}{2}h|f''(x)|.$$

We want to find the value of $h$ that minimizes this error, so we differentiate with respect to $h$ and set the result equal to zero, which gives $$-\frac{2C|f(x)|}{h^2}+\frac{1}{2}|f''(x)|=0, \quad \text{or equivalently} \quad h=\sqrt{4C\left|\frac{f(x)}{f''(x)}\right|}  .$$ Substituting this back into our expression for the error $\epsilon$, we find that the lowest error on our deivative is $$\epsilon=\sqrt{4C|f(x)f''(x)}.$$ If both $f(x)$ and $f''(x)$ are of order $1$, we should choose $h$ to be of order $\sqrt{C}$, which will typically be $10^{-8}$ and our final error will also be of order $\sqrt{C}$, i.e. $10^{-8}$. In many cases, this might be ok, but it is significantly less accurate than what we have seen so far and we can do better.

## Central differences

A simple improvement on the forward and backward difference is the _central difference_: $$\frac{df}{dx}\approx\frac{f((x+h/2)-f(x-h/2)}{h}.$$ The central difference is similar to the forward and the backward difference, approximating the derivative using the difference between two values of $f(x)$ at points a distance $h$ apart. What has changed is that the two points are now placed symmetrically around $x$, one at a distance $\frac{1}{2}h$ in the forward and one at $-\frac{1}{2}h$ in the backward direction.

To calculate the approximation error on the central difference we write two Taylor expansions: $$\begin {align} & f(x+h/2)=f(x)+\frac{1}{2}hf'(x)+\frac{1}{8}h^2f''(x)+\frac{1}{48}h^3f'''(x)+\ldots \\ &  f(x-h/2)=f(x)-\frac{1}{2}hf'(x)+\frac{1}{8}h^2f''(x)-\frac{1}{48}h^3f'''(x)+\ldots\end{align}$$ Subtracting the second expression from the first and rearranging for $f'(x)$, we get $$f'(x)=\frac{f((x+h/2)-f(x-h/2)}{h}-\frac{1}{24}h^2f'''(x)+\ldots.$$ To leading order the magitude of the error is now $\frac{1}{24}h^2|f'''(x)|$, which is one order in $h$ higher than before. The size of the rounding error remains unchanged, so the total error on our derivative is $$\epsilon =\frac{2C|f(x)|}{h} + \frac{1}{24}h^2|f'''(x)|.$$ Differentiating to find the minimum and rearranging, we find that the optimal value of $h$ is $$h=\left(24C\left|\frac{f(x)}{f'''(x)}\right| \right)^{\frac{1}{3}}.$$ Substituting this back into the error itself, we find the optimal error to be $$\epsilon=(\frac{9}{8}[f(x)]^2|f'''(x)|)^\frac{1}{3}.$$ If we again assume that $f(x)$ and $f'''(x)$ are of order 1, the ideal $h$ is now of order $C^\frac{1}{3}$, which is typically $10^{-5}$, but the error is of order $C^\frac{2}{3}$, which will be $10^{-10}$.

Thus, the central difference is about a factor $100$ more accurate than the forward or backwards difference. And we achieve this with a larger value of $h$, which is a bonus.

## Second derivatives

We can also derive numerical approximations for the second derivative of a function $f(x)$. The second derivative is, by definition, the derivative of the first derivative, so we can calculate it by applying our first-derivative formulas twice. If we do this for the central difference formula we can write expressions for the first derivative at $x+h/2$ and $x-h/2$: $$ f'(x+h/2)\approx \frac{f(x+h)-f(x)}{h} \quad \text{and} \quad  f'(x-h/2)\approx \frac{f(x)-f(x-h)}{h} .$$  This gives us for the second derivative: $$\begin{align} f''(x) & \approx \frac{f'(x+h/2)-f'(x-h/2)}{h} \\ & = \frac{(f(x+h)-f(x))-(f(x)-f(x-h))}{h^2} \\ &  = \frac{f(x+h)-2f(x)+f(x-h)}{h^2} \end{align}$$ This is the simplest approximation for the second derivative and we will use it extensively for solving second-order differential equations.

The error of the second derivative can be estimated analogously to before by Taylor expanding $f(x+h)$ and $f(x-h)$. We obtain $$\epsilon=\frac{4Cf(x)}{h^2}+\frac{1}{12}h^2|f''''(x)|.$$ The optimum values of $h$ and $\epsilon$ are $$h=\left(48C\left|\frac{f(x)}{f''''(x)} \right| \right)^\frac{1}{4} \quad \text{and} \quad \epsilon=(\frac{4}{3}C|f(x)f''''(x))^\frac{1}{2} .$$ This means that the error in the 2nd derivative is of order $\sqrt{C}$, which is typically around $10^{-8}$. Our 2nd derivative therefore has about the same error as the forward or backward difference for the first derivative.

```python

```

Lecture 2 - Integrals and Derivatives/Integrals_and_Derivatives.ipynb


# Numerical Integration

## Equally Spaced Points

### Trapezoidal Rule
Assuming $n$ equally spaced points $x_0=x_l, x_1, \ldots , x_{n-2}, x_{n-1}=x_u$ with $h = x_i - x_{i-1}$ and corresponding values of the function stored in an array $y_0, y_1, \ldots , y_{n-2}, y_{n-1}$, the approximation to the integral is
$$
I = \int_{x_l}^{x_{u}} y \ dx \approx \frac{h}{2} \left[ y_0 + 2 \left( y_1 + y_2 + \cdots + y_{n-3} + y_{n-2} \right) + y_{n-1} \right]
$$
The above expresion presumes that the function $y = f(x)$ has been evaluated at $n$ equally spaced points prior to computing the integral

In this procedure, we take the following approach:

1. Generate $n$ equally spaced data points $x_i, i = 0, 1, 2, \ldots , n-1$
2. Compute the value of the function for each of the data points $y_i = f(x_i)$
3. Carry out numerical integration using Trapezoidal rule

```python
from __future__ import print_function, division
import numpy as np
from matplotlib import pyplot as plt

%matplotlib inline

def trap(h, y):
    n = len(y)
    s = y[0] + 2 * sum(y[1:-1]) + y[-1]
    return s * h / 2.0

x1 = 0.0
x2 = np.pi
n = 500
x = np.linspace(x1, x2, n+1)
y = np.sin(x)
h = (x2 - x1) / float(n)
I = trap(h, y)
%timeit trap(h, y)
print(I)
plt.fill(x, y)
plt.axis([x1, 2*x2, -1.1, 1.1])
plt.grid()
plt.show()
```

```python
def f(x):
    return np.sin(x)

def trap2(f, x1, x2, n):
    h = (x2 - x1) / n
    x = np.linspace(x1, x2, n+1)
    y = f(x)
    s = y[0] + 2 * sum(y[1:-1]) + y[-1]
    return s * h / 2

I = trap2(f, 0, np.pi, 500)
print(I)
%timeit trap2(f, 0, np.pi, 500)
```

    1.99999342026
    10000 loops, best of 3: 105 µs per loop

```python
def trap3(f, x1, x2, n):
    h = (x2 - x1) / n
    s = (f(x1) + f(x2)) / 2
    x = x1 + h
    while x <= x2:
        s += f(x)
        x += h
    return s * h

I = trap3(f, 0, np.pi, 500)
print(I)
%timeit trap3(f, 0, np.pi, 500)
```

    1.99999342026
    1000 loops, best of 3: 851 µs per loop

### Simpson's 1/3 Rule

$$
I = \int_{x_1}^{x_2} y \, dx \approx \frac{h}{3} \left[ y_0 + 4 \left( y_1 + y_3 + \cdots + y_{n-2} \right) + 2 \left( y_2 + y_4 + \cdots + y_{n-3} \right) + y_{n-1} \right]
$$

```python
def simp(f, x1, x2, n):
    h = (x2 - x1) / n
    x = np.linspace(x1, x2, n+1)
    y = f(x)
    s = y[0] + y[-1]
    s += 4 * sum(y[1:-1:2])
    s += 2 * sum(y[2:-2:2])
    return s * h / 3

I = simp(f, 0, np.pi, 500)
print(I)
%timeit simp(f, 0, np.pi, 500)
```

    2.00000000002
    10000 loops, best of 3: 106 µs per loop

```python
from sympy import *
x = symbols('x')
integrate(sin(x), (x, 0, pi))
```

    2


### Principal Component Analysis

```python
import scipy.io as sio
import numpy as np
import matplotlib.pyplot as plt
mat_contents = sio.loadmat('mnistAll.mat')
```

#### Load data

```python
train_img = mat_contents['mnist'][0][0][0]
train_label = mat_contents['mnist'][0][0][2]
idx3 = np.where(mat_contents['mnist'][0][0][2]==3)
ndx = idx3[0][:1000]; d = 28*28;
n = len(ndx)
X = np.double(train_img[:,:,ndx].reshape(d,n))
```

```python
plt.subplot(1,2,1)
plt.imshow(X[:,0].reshape(28,28))
plt.subplot(1,2,2)
mu = np.mean(X,axis=1)
plt.imshow(mu.reshape(28,28))
```

#### processing

```python
def pcaPmtk(x):
    n,d=x.shape
    K = np.linalg.matrix_rank(x)x
    mu = np.mean(x,axis=1)
    xc = x-np.transpose(np.tile(mu,(d,1)))
    w,v = np.linalg.eigh(np.cov(xc))
    wsort = np.sort(w)[::-1]
    ind = np.argsort(w)[::-1]
    B = v[:,ind[:K]]
    z = np.matmul(np.transpose(xc),B)
    xrecon = np.matmul(z,np.transpose(B))+np.tile(mu,(d,1))
    return B,z,w
```

```python
v,z,evals = pcaPmtk(X)
```

#### Principal Axis

```python
for i in range(4):
    plt.subplot(2,2,i+1)
    plt.imshow(v[:,i].reshape(28,28))
```

#### Reconstructed Image

```python
# number of principal axes
Ks = [5,10,20,np.linalg.matrix_rank(X)]
for j in range(4):
    xre = np.matmul(z[124,:Ks[j]],np.transpose(v[:,:Ks[j]]))+mu
    plt.subplot(2,2,j+1)
    plt.imshow(xre.reshape(28,28))
```

### Proof

#### Cost function
$$J(w,z_{i1})=\frac{1}{N}\sum^N_{i=1}(x_i-z_{i1}w_1)^\top(x_i-z_{i1}w_1)=\frac{1}{N}\sum^N_{i=1}(x_i^\top x_i-2x_i^\top z_{i1}w_1+z_{i1}^2w^\top_1 w_1)$$

#### Minimizing the Cost function,
Since $w_1$ is orthnormal, $w_1^\top w_1 = 1$

\begin{align*}
\frac{\partial J}{\partial z_{j1}} & = \frac{1}{N}\sum^N_{i=1}(-2x_i^\top \frac{\partial z_{i1}}{\partial z_{j1}}w_1+2z_i\frac{\partial z_{i1}}{\partial z_{j1}}w_1^\top w_1) = \frac{1}{N}\sum^N_{i=1} \delta_{ij}(-2)(x_i^\top w_1-z_{i1}w_1^\top w_1)
\\
&= \frac{1}{N}(x_j^\top w_1-z_{j1}w_1^\top w_1) = 0
\qquad\text{  Therefore,   } \qquad z_{j1}=w_1^\top x_j
\end{align*}

#### Substitute back to the Cost function,
\begin{align*}
J(z_{i1}) &= \frac{1}{N}\sum^N_{i=1}(x^\top_i x_i - 2z_{i1}w_1^\top x_i + z_{i1}^2) = \frac{1}{N}\sum^N_{i=1}(x_i^\top x_i - 2z_{i1}^2+z_{i1}^2) = \frac{1}{N}\sum^N_{i=1}(x_i^\top x_i-z_{i1}^2) = constant - \frac{1}{N}\sum^N_{i=1}z^2_{i1}
\end{align*}

#### Covariance matrix
Consider $z_{i1} = w_1^\top x_i$,

$$
\frac{1}{N}\sum^N_{i=1}z_{i1}^2 = \frac{1}{N}\sum^N_{i=1}w_1^\top x_ix_i^\top w_1=w_1^\top \bigg(\frac{1}{N}\sum^N_{i=1}x_ix_i^\top \bigg)w_1 = w_1^\top \hat{\Sigma}w_1
$$

To minimize the cost function $J(z_{i1})$ is equivalent to maximize $\frac{1}{N}\sum^N_{i=1}z_{i1}^2$, so we can rewrite the cost function as $\tilde{J}(w_1)$ with a constraint of $w^\top_1w_1=1$ using Lagrange multiplier.
\begin{equation}
\tilde{J}(w_1) = w^\top_1\hat{\Sigma}w_1+\lambda_1(w^\top_1w_1-1)
\end{equation}
After minimizing the new cost function, obtain the eigenvector
$$\frac{\partial \tilde{J}}{\partial w_1} = 2\hat{\Sigma}w_1-2\lambda_1w_1 = 0 \qquad \longrightarrow \qquad  \hat{\Sigma}w_1=\lambda_1w_1$$

```python

```

```python

```

```python

```

```python

```

```python

```

Machine Learning A Probabilistic Perspective/12LatentLinearModels/F12.5/.ipynb_checkpoints/12.5pcaImageDemo-checkpoint.ipynb


```python
%matplotlib inline

'''
grad_descent.py

Use gradient descent to find the minimum value of a
single variable function. This also checks for the existence
of a solution for the equation, f'(x)=0 and plots the intermediate
points traversed.
'''

from sympy import Derivative, Symbol, sympify, solve
import matplotlib.pyplot as plt

def grad_descent(x0, f1x, x):
    # check if f1x=0 has a solution
    if not solve(f1x):
        print('Cannot continue, solution for {0}=0 does not exist'.format(f1x))
        return None
    epsilon =  1e-6
    step_size = 1e-4
    x_old = x0
    x_new = x_old - step_size*f1x.subs({x:x_old}).evalf()

    # list to store the X values traversed
    X_traversed = []
    while abs(x_old - x_new) > epsilon:
        X_traversed.append(x_new)
        x_old = x_new
        x_new = x_old-step_size*f1x.subs({x:x_old}).evalf()

    return x_new, X_traversed

def frange(start, final, interval):

    numbers = []
    while start < final:
        numbers.append(start)
        start = start + interval

    return numbers

def create_plot(X_traversed, f, var):
    # First create the graph of the function itself
    x_val = frange(-1, 1, 0.01)
    f_val = [f.subs({var:x}) for x in x_val]
    plt.plot(x_val, f_val, 'bo')
    # calculate the function value at each of the intermediate
    # points traversed
    f_traversed = [f.subs({var:x}) for x in X_traversed]
    plt.plot(X_traversed, f_traversed, 'r.')
    plt.legend(['Function', 'Intermediate points'], loc='best')
    plt.show()

if __name__ == '__main__':

    f = input('Enter a function in one variable: ')
    var = input('Enter the variable to differentiate with respect to: ')
    var0 = float(input('Enter the initial value of the variable: '))
    try:
        f = sympify(f)
    except SympifyError:
        print('Invalid function entered')
    else:
        var = Symbol(var)
        d = Derivative(f, var).doit()
        var_min, X_traversed = grad_descent(var0, d, var)
        if var_min:
            print('{0}: {1}'.format(var.name, var_min))
            print('Minimum value: {0}'.format(f.subs({var:var_min})))
            create_plot(X_traversed, f, var)

```


# Introduction to orthogonal coordinates

In $\mathbb{R}^3$, we can think that each point is given by the
intersection of three surfaces. Thus, we have three families of curved
surfaces that intersect each other at right angles. These surfaces are
orthogonal locally, but not (necessarily) globally, and  are
defined by

$$u_1 = f_1(x, y, z)\, ,\quad u_2 = f_2(x, y, z)\, ,\quad u_3=f_3(x, y, z) \, .$$

These functions should be invertible, at least locally, and we can also write

$$x = x(u_1, u_2, u_3)\, ,\quad y = y(u_1, u_2, u_3)\, ,\quad z = z(u_1, u_2, u_3)\, ,$$

where $x, y, z$ are the usual Cartesian coordinates. The curve defined by the intersection of two of the surfaces gives us
one of the coordinate curves.

## Scale factors

Since we are interested in how these surface intersect each other locally,
we want to express differential vectors in terms of the coordinates. Thus,
the differential for the position vector ($\mathbf{r}$) is given by

$$\mathrm{d}\mathbf{r} = \frac{\partial\mathbf{r}}{\partial u_1}\mathrm{d}u_1
+ \frac{\partial\mathbf{r}}{\partial u_2}\mathrm{d}u_2
+ \frac{\partial\mathbf{r}}{\partial u_3}\mathrm{d}u_3\, ,
$$

or

$$\mathrm{d}\mathbf{r} = \sum_{i=1}^3 \frac{\partial\mathbf{r}}{\partial u_i}\mathrm{d}u_i\, .$$

The factor $\partial \mathbf{r}/\partial u_i$ is a non-unitary vector that takes
into account the variation of $\mathbf{r}$ in the direction of $u_i$, and is then
tangent to the coordinate curve $u_i$. We can define a normalized basis $\hat{\mathbf{e}}_i$
using

$$\frac{\partial\mathbf{r}}{\partial u_i} = h_i \hat{\mathbf{e}}_i\, .$$

The coefficients $h_i$ are functions of $u_i$ and we call them _scale factors_. They
are really important since they allow us to _measure_ distances while we move along
our coordinates. We would need them to define vector operators in orthogonal coordinates.
When the coordinates are not orthogonal we would need to use the [metric tensor](https://en.wikipedia.org/wiki/Metric_tensor), but we are going to restrict ourselves to orthogonal systems.

Hence, we have the following

$$\begin{align}
&h_i = \left|\frac{\partial\mathbf{r}}{\partial u_i}\right|\, ,\\
&\hat{\mathbf{e}}_i = \frac{1}{h_i} \frac{\partial \mathbf{r}}{\partial u_i}\, .
\end{align}$$

## Curvilinear coordinates available

The following coordinate systems are available:

- Cartesian;

- Cylindrical;

- Spherical;

- Parabolic cylindrical;

- Parabolic;

- Paraboloidal;

- Elliptic cylindrical;

- Oblate spheroidal;

- Prolate spheroidal;

- Ellipsoidal;

- Bipolar cylindrical;

- Toroidal;

- Bispherical; and

- Conical.

To obtain the transformation for a given coordinate system we can use
the function `transform_coords` in the `vector` module.

```python
import sympy as sym
from continuum_mechanics import vector
```

First, we define the variables for the coordinates $(u, v, w)$.

```python
sym.init_printing()
u, v, w = sym.symbols("u v w")
```

And, we compute the coordinates for the **parabolic** system using ``transform_coords``.
The first parameter is a string defining the coordinate system and the second is
a tuple with the coordinates.

```python
vector.transform_coords("parabolic", (u, v, w))
```

The scale factors for the coordinate systems mentioned above are availabe.
We can compute them for bipolar cylindrical coordinates. The coordinates
are defined by

$$\begin{align}
&x = a \frac{\sinh\tau}{\cosh\tau - \cos\sigma}\, ,\\
&y = a \frac{\sin\sigma}{\cosh\tau - \cos\sigma}\, ,\\
&z = z\, ,
\end{align}$$

and have the following scale factors

$$h_\sigma = h_\tau = \frac{a}{\cosh\tau - \cos\sigma}\, ,$$

and $h_z = 1$.

```python
sigma, tau, z, a = sym.symbols("sigma tau z a")
z = sym.symbols("z")
scale = vector.scale_coeff_coords("bipolar_cylindrical", (sigma, tau, z), a=a)
scale
```

Finally, we can compute vector operators for different coordinates.

The Laplace operator for the bipolar cylindrical system is given by

$$
\nabla^2 \phi =
\frac{1}{a^2} \left( \cosh \tau - \cos\sigma \right)^{2}
\left(
\frac{\partial^2 \phi}{\partial \sigma^2} +
\frac{\partial^2 \phi}{\partial \tau^2}
\right) +
\frac{\partial^2 \phi}{\partial z^2}\, ,$$

and we can compute it using the function ``lap``. For this function,
the first parameter is the expression that we want to compute the
Laplacian for, the second parameter is a tuple with the coordinates
and the third parameter is a tuple with the scale factors.

```python
phi = sym.symbols("phi", cls=sym.Function)
lap = vector.lap(phi(sigma, tau, z), coords=(sigma, tau, z), h_vec=scale)
sym.simplify(lap)
```


# Basis for grayscale images

## Introduction

Consider the set of real-valued matrices of size $M\times N$; we can turn this into a vector space by defining addition and scalar multiplication in the usual way:

\begin{align}
\mathbf{A} + \mathbf{B} &=
    \left[
        \begin{array}{ccc}
            a_{0,0} & \dots & a_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    +
    \left[
        \begin{array}{ccc}
            b_{0,0} & \dots & b_{0,N-1} \\
            \vdots & & \vdots \\
            b_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    \\
    &=
    \left[
        \begin{array}{ccc}
            a_{0,0}+b_{0,0} & \dots & a_{0,N-1}+b_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0}+b_{M-1,0} & \dots & a_{M-1,N-1}+b_{M-1,N-1}
        \end{array}
    \right]
    \\ \\ \\
\beta\mathbf{A} &=
    \left[
        \begin{array}{ccc}
            \beta a_{0,0} & \dots & \beta a_{0,N-1} \\
            \vdots & & \vdots \\
            \beta a_{M-1,0} & \dots & \beta a_{M-1,N-1}
        \end{array}
    \right]
\end{align}

As a matter of fact, the space of real-valued $M\times N$ matrices is completely equivalent to $\mathbb{R}^{MN}$ and we can always "unroll" a matrix into a vector. Assume we proceed column by column; then the matrix becomes

$$
    \mathbf{a} = \mathbf{A}[:] = [
        \begin{array}{ccccccc}
            a_{0,0} & \dots & a_{M-1,0} & a_{0,1} & \dots & a_{M-1,1} & \ldots & a_{0, N-1} & \dots & a_{M-1,N-1}
        \end{array}]^T
$$

Although the matrix and vector forms represent exactly the same data, the matrix form allows us to display the data in the form of an image. Assume each value in the matrix is a grayscale intensity, where zero is black and 255 is white; for example we can create a checkerboard pattern of any size with the following function:

```python
# usual pyton bookkeeping...
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import IPython
from IPython.display import Image
import math
```

```python
# ensure all images will be grayscale
plt.gray();
```

    <Figure size 432x288 with 0 Axes>

```python
# let's create a checkerboard pattern
SIZE = 4
img = np.zeros((SIZE, SIZE))
for n in range(0, SIZE):
    for m in range(0, SIZE):
        if (n & 0x1) ^ (m & 0x1):
            img[n, m] = 255

# now display the matrix as an image
plt.matshow(img);
```

Given the equivalence between the space of $M\times N$ matrices and $\mathbb{R}^{MN}$ we can easily define the inner product between two matrices in the usual way:

$$
\langle \mathbf{A}, \mathbf{B} \rangle = \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} a_{m,n} b_{m, n}
$$

(where we have neglected the conjugation since we'll only deal with real-valued matrices); in other words, we can take the inner product between two matrices as the standard inner product of their unrolled versions. The inner product allows us to define orthogonality between images and this is rather useful since we're going to explore a couple of bases for this space.

## Actual images

Conveniently, using IPython, we can read images from disk in any given format and convert them to numpy arrays; let's load and display for instance a JPEG image:

```python
img = np.array(plt.imread('_inputs/20190919/cameraman.jpg'), dtype=int)
plt.matshow(img);
```

The image is a $64\times 64$ low-resolution version of the famous "cameraman" test picture. Out of curiosity, we can look at the first column of this image, which is is a $64×1$ vector:

```python
img[:,0]
```

    array([156, 157, 157, 152, 154, 155, 151, 157, 152, 155, 158, 159, 159,
           160, 160, 161, 155, 160, 161, 161, 164, 162, 160, 162, 158, 160,
           158, 157, 160, 160, 159, 158, 163, 162, 162, 157, 160, 114, 114,
           103,  88,  62, 109,  82, 108, 128, 138, 140, 136, 128, 122, 137,
           147, 114, 114, 144, 112, 115, 117, 131, 112, 141,  99,  97])

The values are integers between zero and 255, meaning that each pixel is encoded over 8 bits (or 256 gray levels).

## The canonical basis

The canonical basis for any matrix space $\mathbb{R}^{M\times N}$ is the set of "delta" matrices where only one element equals to one while all the others are 0. Let's call them $\mathbf{E}_n$ with $0 \leq n < MN$. Here is a function to create the canonical basis vector given its index:

```python
def canonical(n, M=5, N=10):
    e = np.zeros((M, N))
    e[(n % M), int(n / M)] = 1
    return e
```

Here are some basis vectors: look for the position of white pixel, which differentiates them and note that we enumerate pixels column-wise:

```python
plt.matshow(canonical(0));
plt.matshow(canonical(1));
plt.matshow(canonical(49));
```

## Transmitting images

Suppose we want to transmit the "cameraman" image over a communication channel. The intuitive way to do so is to send the  pixel values one by one, which corresponds to sending the coefficients of the decomposition of the image over the canonical basis. So far, nothing complicated: to send the cameraman image, for instance, we will send $64\times 64 = 4096$ coefficients in a row.

Now suppose that a communication failure takes place after the first half of the pixels have been sent. The received data will allow us to display an approximation of the original image only. If we replace the missing data with zeros, here is what we would see, which is not very pretty:

```python
# unrolling of the image for transmission (we go column by column, hence "F")
tx_img = np.ravel(img, "F")

# oops, we lose half the data
tx_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.reshape(tx_img, (64, 64), "F")
plt.matshow(rx_img);
```

Can we come up with a trasmission scheme that is more robust in the face of channel loss? Interestingly, the answer is yes, and it involves a different, more versatile basis for the space of images. What we will do is the following:

* describe the Haar basis, a new basis for the image space
* project the image in the new basis
* transmit the projection coefficients
* rebuild the image using the basis vectors

We know a few things: if we choose an orthonormal basis, the analysis and synthesis formulas will be super easy (a simple inner product and a scalar multiplication respectively). The trick is to find a basis that will be robust to the loss of some coefficients.

One such basis is the **Haar basis**. We cannot go into too many details in this notebook but, for the curious, a good starting point is [here](https://chengtsolin.wordpress.com/2015/04/15/real-time-2d-discrete-wavelet-transform-using-opengl-compute-shader/). Mathematical formulas aside, the Haar basis works by encoding the information in a *hierarchical* way: the first basis vectors encode the broad information and the higher coefficients encode the detail. Let's have a look.

First of all, to keep things simple, we will remain in the space of square matrices whose size is a power of two. The code to generate the Haar basis matrices is the following: first we generate a 1D Haar vector and then we obtain the basis matrices by taking the outer product of all possible 1D vectors (don't worry if it's not clear, the results are what's important):

```python
def haar1D(n, SIZE):
    # check power of two
    if math.floor(math.log(SIZE) / math.log(2)) != math.log(SIZE) / math.log(2):
        print("Haar defined only for lengths that are a power of two")
        return None
    if n >= SIZE or n < 0:
        print("invalid Haar index")
        return None

    # zero basis vector
    if n == 0:
        return np.ones(SIZE)

    # express n > 1 as 2^p + q with p as large as possible;
    # then k = SIZE/2^p is the length of the support
    # and s = qk is the shift
    p = math.floor(math.log(n) / math.log(2))
    pp = int(pow(2, p))
    k = SIZE / pp
    s = (n - pp) * k

    h = np.zeros(SIZE)
    h[int(s):int(s+k/2)] = 1
    h[int(s+k/2):int(s+k)] = -1
    # these are not normalized
    return h

def haar2D(n, SIZE=8):
    # get horizontal and vertical indices
    hr = haar1D(n % SIZE, SIZE)
    hv = haar1D(int(n / SIZE), SIZE)
    # 2D Haar basis matrix is separable, so we can
    #  just take the column-row product
    H = np.outer(hr, hv)
    H = H / math.sqrt(np.sum(H * H))
    return H
```

First of all, let's look at a few basis matrices; note that the matrices have positive and negative values, so that the value of zero will be represented as gray:

```python
plt.matshow(haar2D(0));
plt.matshow(haar2D(1));
plt.matshow(haar2D(10));
plt.matshow(haar2D(63));
```

We can notice two key properties

* each basis matrix has positive and negative values in some symmetric patter: this means that the basis matrix will implicitly compute the difference between image areas
* low-index basis matrices take differences between large areas, while high-index ones take differences in smaller **localized** areas of the image

We can immediately verify that the Haar matrices are orthogonal:

```python
# let's use an 8x8 space; there will be 64 basis vectors
# compute all possible inner product and only print the nonzero results
for m in range(0,64):
    for n in range(0,64):
        r = np.sum(haar2D(m, 8) * haar2D(n, 8))
        if r != 0:
            print("[%dx%d -> %f] " % (m, n, r), end="")
```

    [0x0 -> 1.000000] [1x1 -> 1.000000] [2x2 -> 1.000000] [3x3 -> 1.000000] [4x4 -> 1.000000] [5x5 -> 1.000000] [6x6 -> 1.000000] [7x7 -> 1.000000] [8x8 -> 1.000000] [9x9 -> 1.000000] [10x10 -> 1.000000] [11x11 -> 1.000000] [12x12 -> 1.000000] [13x13 -> 1.000000] [14x14 -> 1.000000] [15x15 -> 1.000000] [16x16 -> 1.000000] [16x17 -> -0.000000] [17x16 -> -0.000000] [17x17 -> 1.000000] [18x18 -> 1.000000] [19x19 -> 1.000000] [20x20 -> 1.000000] [21x21 -> 1.000000] [22x22 -> 1.000000] [23x23 -> 1.000000] [24x24 -> 1.000000] [24x25 -> -0.000000] [25x24 -> -0.000000] [25x25 -> 1.000000] [26x26 -> 1.000000] [27x27 -> 1.000000] [28x28 -> 1.000000] [29x29 -> 1.000000] [30x30 -> 1.000000] [31x31 -> 1.000000] [32x32 -> 1.000000] [33x33 -> 1.000000] [34x34 -> 1.000000] [35x35 -> 1.000000] [36x36 -> 1.000000] [37x37 -> 1.000000] [38x38 -> 1.000000] [39x39 -> 1.000000] [40x40 -> 1.000000] [41x41 -> 1.000000] [42x42 -> 1.000000] [43x43 -> 1.000000] [44x44 -> 1.000000] [45x45 -> 1.000000] [46x46 -> 1.000000] [47x47 -> 1.000000] [48x48 -> 1.000000] [49x49 -> 1.000000] [50x50 -> 1.000000] [51x51 -> 1.000000] [52x52 -> 1.000000] [53x53 -> 1.000000] [54x54 -> 1.000000] [55x55 -> 1.000000] [56x56 -> 1.000000] [57x57 -> 1.000000] [58x58 -> 1.000000] [59x59 -> 1.000000] [60x60 -> 1.000000] [61x61 -> 1.000000] [62x62 -> 1.000000] [63x63 -> 1.000000]

OK! Everything's fine. Now let's transmit the "cameraman" image: first, let's verify that it works

```python
# project the image onto the Haar basis, obtaining a vector of 4096 coefficients
# this is simply the analysis formula for the vector space with an orthogonal basis
tx_img = np.zeros(64*64)
for k in range(0, (64*64)):
    tx_img[k] = np.sum(img * haar2D(k, 64))

# now rebuild the image with the synthesis formula; since the basis is orthonormal
#  we just need to scale the basis matrices by the projection coefficients
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += tx_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

Cool, it works! Now let's see what happens if we lose the second half of the coefficients:

```python
# oops, we lose half the data
lossy_img = np.copy(tx_img);
lossy_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

That's quite remarkable, no? We've lost the same amount of information as before but the image is still acceptable. This is because we lost the coefficients associated to the fine details of the image but we retained the "broad strokes" encoded by the first half.

Note that if we lose the first half of the coefficients the result is markedly different:

```python
lossy_img = np.copy(tx_img);
lossy_img[0:int(len(tx_img)/2)] = 0

rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

In fact, schemes like this one are used in *progressive encoding*: send the most important information first and add details if the channel permits it. You may have experienced this while browsing the interned over a slow connection.

All in all, a great application of a change of basis!

```python

```

```python

```


```python
import numpy as np                     # Load the library

a = np.linspace(-np.pi, np.pi, 100)    # Create even grid from -π to π
b = np.cos(a)                          # Apply cosine to each element of a
c = np.sin(a)                          # Apply sin to each element of a
```

```python
b @ c
```

    3.3306690738754696e-16

```python
from scipy.stats import norm
from scipy.integrate import quad

ϕ = norm()
value, error = quad(ϕ.pdf, -2, 2)  # Integrate using Gaussian quadrature
value
```

    0.9544997361036417

```python
from sympy import Symbol

x, y = Symbol('x'), Symbol('y')  # Treat 'x' and 'y' as algebraic symbols
x + x + x + y
```

$\displaystyle 3 x + y$

```python
expression = (x + y)**2
expression.expand()
```

$\displaystyle x^{2} + 2 x y + y^{2}$

```python
from sympy import solve

solve(x**2 + x + 2)
```

    [-1/2 - sqrt(7)*I/2, -1/2 + sqrt(7)*I/2]

```python
from sympy import limit, sin, diff

limit(1 / x, x, 0)
```

$\displaystyle \infty$

```python
limit(sin(x) / x, x, 0)
```

$\displaystyle 1$

```python
diff(sin(x), x)
```

$\displaystyle \cos{\left(x \right)}$

```python
import pandas as pd
np.random.seed(1234)

data = np.random.randn(5, 2)  # 5x2 matrix of N(0, 1) random draws
dates = pd.date_range('28/12/2010', periods=5)

df = pd.DataFrame(data, columns=('price', 'weight'), index=dates)
print(df)
```

                   price    weight
    2010-12-28  0.471435 -1.190976
    2010-12-29  1.432707 -0.312652
    2010-12-30 -0.720589  0.887163
    2010-12-31  0.859588 -0.636524
    2011-01-01  0.015696 -2.242685

```python
df.mean()
```

    price     0.411768
    weight   -0.699135
    dtype: float64


## TNK

Tanaka suggested the following two-variable problem:

**Definition**

\begin{equation}
\newcommand{\boldx}{\mathbf{x}}
\begin{array}
\mbox{Minimize} & f_1(\boldx) = x_1, \\
\mbox{Minimize} & f_2(\boldx) = x_2, \\
\mbox{subject to} & C_1(\boldx) \equiv x_1^2 + x_2^2 - 1 -
0.1\cos \left(16\arctan \frac{x_1}{x_2}\right) \geq 0, \\
& C_2(\boldx) \equiv (x_1-0.5)^2 + (x_2-0.5)^2 \leq 0.5,\\
& 0 \leq x_1 \leq \pi, \\
& 0 \leq x_2 \leq \pi.
\end{array}
\end{equation}

**Optimum**

Since $f_1=x_1$ and $f_2=x_2$, the feasible objective space is also
the same as the feasible decision variable space. The unconstrained
decision variable space consists of all solutions in the square
$0\leq (x_1,x_2)\leq \pi$. Thus, the only unconstrained Pareto-optimal
solution is $x_1^{\ast}=x_2^{\ast}=0$.
However, the inclusion of the first constraint makes this solution
infeasible. The constrained Pareto-optimal solutions lie on the boundary
of the first constraint. Since the constraint function is periodic and
the second constraint function must also be satisfied,
not all solutions on the boundary of the first constraint are Pareto-optimal. The
Pareto-optimal set is disconnected.
Since the Pareto-optimal
solutions lie on a nonlinear constraint surface, an optimization
algorithm may have difficulty in finding a good spread of solutions across
all of the discontinuous Pareto-optimal sets.

**Plot**

```python
from pymoo.factory import get_problem
from pymoo.util.plotting import plot

problem = get_problem("tnk")
plot(problem.pareto_front(), no_fill=True)
```


```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
```

# Superposition of two waves in perpendicular direction
\begin{equation}
x = a \sin (2\pi f_1 t)\\
y=b \sin (2\pi f_2 t - \phi)
\end{equation}

```python
a = [10,30]             # amplitude of first wave
f1 = [1,2,4,8,12,16]    # frequency of first wave
b=[10,30]               # amplitude of second wave
f2=[1,2,4,8,12,16]      # frequency of second wave
phi=[0,np.pi/4,np.pi/2] # phase angle(0,30,45,60,90)
t = np.arange(0,8.0,0.01) # time
```

## Example

### 1.same amplitude,phase is zero ,different frequency
 Lissajous figure

```python
plt.figure(figsize = [6,36])
for i in range(len(f2)):
    plt.subplot(len(f2),1,i+1)
    ax = plt.gca()
    ax.set_facecolor('k') # backgound color
    ax.grid(color='xkcd:sky blue') # grid color
    x = a[0]*np.sin(2*np.pi*f1[2]*t)
    y = b[0]*np.sin(2*np.pi*f2[i]*t-phi[0])
    plt.plot(x,y, color ='g',label='f1='+str(f1[2])+',f2='+str(f2[i]))
    plt.xlabel("x",color='r',fontsize=14)
    plt.ylabel("y",color='r',fontsize=14)
    ax.xaxis.set_minor_locator(AutoMinorLocator()) ##
    ax.yaxis.set_minor_locator(AutoMinorLocator()) ###
    ax.tick_params(which='both', width=2)
    ax.tick_params(which='major', length=9)
    ax.tick_params(which='minor', length=4)
    plt.legend()
    plt.subplots_adjust(wspace = 0.5, hspace = 0.5)
plt.show()
```

### 2.same frequency and amplitude,different phase

```python
plt.figure(figsize = [6,24])
for i in range(len(phi)):
    plt.subplot(len(phi),1,i+1)
    ax = plt.gca()
    ax.set_facecolor('xkcd:sky blue')
    ax.grid(color='g')
    x = a[0]*np.sin(2*np.pi*f1[2]*t)
    y = b[0]*np.sin(2*np.pi*f2[2]*t-phi[i])
    plt.plot(x,y, color ='purple',label='phase='+str(phi[i]*180/np.pi))
    plt.xlabel("x",color='g',fontsize=14)
    plt.ylabel("y",color='g',fontsize=14)
    plt.legend()
    plt.subplots_adjust(wspace = 0.5, hspace = 0.5)
plt.show()
```

### 3.same frequency ,different phase and amplitude

```python
plt.figure(figsize = [8,24])
for i in range(len(phi)):
    plt.subplot(len(phi),1,i+1)
    ax = plt.gca()
    ax.grid(color='tab:brown')
    x = a[0]*np.sin(2*np.pi*f1[2]*t)
    y = b[1]*np.sin(2*np.pi*f2[2]*t-phi[i])
    plt.plot(x,y, color ='r',label='phase='+str(phi[i]*180/np.pi))
    plt.xlabel("x",color='r',fontsize=14)
    plt.ylabel("y",color='r',fontsize=14)
    plt.legend()
    plt.subplots_adjust(wspace = 0.5, hspace = 0.5)
plt.show()
```

# Superposition of two waves in same direction
\begin{equation}
y_1 = a \sin (2\pi f_1 t)\\
y_2=b \sin (2\pi f_2 t - \phi)
\end{equation}

## Example

### 1.same amplitude and phase,different frequency

```python
f1=[1,2,3,4,5]
f2=[1,2,3,4,5]
plt.figure(figsize = [12,16])
for i in range(len(f2)):
    plt.subplot(len(f2),1,i+1)
    ax = plt.gca() #graphic current axis
    ax.set_facecolor('k')
    ax.grid(False)
    y1 = a[0]*np.sin(2*np.pi*f1[0]*t)
    y2 = a[0]*np.sin(2*np.pi*f2[i]*t-phi[0])
    y=y1+y2
    plt.plot(t,y,color ='tab:olive',label='f1='+str(f1[0])+',f2='+str(f2[i]))
    plt.xlabel("t",color='r',fontsize=14)
    plt.ylabel("y",color='r',fontsize=14)
    plt.legend()
    plt.subplots_adjust(wspace = 0.5, hspace = 0.5)
plt.show()
```

```python

```


# Coin toss problem

## What is the problem?

Let's pick up a random coin (not necessarily a fair one with equal probability of head and tail). We did the coin toss experiment $n$ times and gathered the observed data $D$ as a set of outcomes (e.g. $\{H, T, T, ...\}$). Now, we are interested in predicting the probability of heads $p(H)=\theta_{best}$ for our coin.

## Applying Bayes rule

In this problem, we will {ref}`model the distribution of parameters <parameters-framework>`.

\begin{equation}
\underbrace{p(\theta|D)}_{\text{Posterior}} = \frac{\overbrace{p(D|\theta)}^{\text{Likelihood}}}{\underbrace{p(D)}_{\text{Evidence}}}\underbrace{p(\theta)}_{\text{Prior}}
\end{equation}
\begin{equation}
p(D) = \int_{\theta}p(D|\theta)p(\theta)d\theta
\end{equation}

We are interested in $p(\theta|D)$ and to derive that, we need prior, likelihood and evidence terms. Let us look at them one by one.

### Prior

What is our prior belief about the coin's probability of head $p($H$)$? Yes, that's exactly the question. A most simple way is to assume equal probability of heads and tails. However, we can represent our prior belief in terms of a distribution. Let's assume a beta distribution over the probability of heads $p(H) = \theta$ (we will see in later sections why beta and not Gaussian or uniform or something else?). So, our prior distibution $p(\theta)$ is:

$$
p(\theta|\alpha, \beta) = \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}, \alpha,\beta>0\\
B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\\
\Gamma(\alpha) = (\alpha-1)!
$$

Here, $\alpha$ and $\beta$ are the hyperparameters of the beta distrubution. $B$ is Beta function. You may play with [this interactive demo](https://huggingface.co/spaces/Zeel/Beta_distribution) to see how pdf changes with $\alpha$ and $\beta$. In our modeling, we can assume that $\alpha$ and $\beta$ are already known. There are methods of assuming distributions over the $\alpha$ and $\beta$ as well but that's out of the scope for now.

### Likelihood

Likelihood is probability of observing the data $D$ given $\theta$. From, $n$ number of experiments, if we received heads $h$ times, then $p(D|\theta)$ follows a Bernoulli distribution. We can also arrive at this formula by following the basic probability rules for independent events:

$$
p(D|\theta) = \theta^h(1-\theta)^{n-h}
$$

### Maximum likelihood estimation (MLE)

In cases, where prior is not available, we can use likelihood to get the best estimate of $\theta$. Let us find the optimal theta by differentiating likelihood $p(D|\theta)$ w.r.t $\theta$.

\begin{align}
p(D|\theta) &= (\theta)^h(1-\theta)^{n-h}\\
\text{taking log both sides to simplify things,}\\
\log p(D|\theta) &= h\log(\theta)+(n-h)\log(1-\theta)\\
\frac{d}{d\theta}\log p(D|\theta) &= \frac{h}{\theta} - \frac{n-h}{1-\theta} = 0\\
&= h(1-\theta)-(n-h)\theta = 0\\
&= h - h\theta - n\theta + h\theta = 0\\
\therefore \theta_{MLE} = \frac{h}{n}
\end{align}

How can we know if optima at $\theta_{MLE}$ is a maxima? well, it is a maxima if $\frac{d^2}{d\theta^2}\log p(D|\theta)$ is negative [(check here if not convinced)](https://www.khanacademy.org/math/multivariable-calculus/applications-of-multivariable-derivatives/optimizing-multivariable-functions/a/second-partial-derivative-test):

\begin{align}
\frac{d}{d\theta}\log p(D|\theta) &= \frac{h}{\theta} - \frac{n-h}{1-\theta}\\
\frac{d^2}{d\theta^2}\log p(D|\theta) &= -\frac{h}{\theta^2}-\frac{n-h}{(1-\theta)^2}
\end{align}

After a bit of thinking, one can see that above value is always negative and thus our optima is a maxima.

### Maximum a posteriori estimation (MAP)

We know that posterior is given by the following formula:

\begin{equation}
\underbrace{p(\theta|D)}_{\text{Posterior}} = \frac{\overbrace{p(D|\theta)}^{\text{Likelihood}}}{\underbrace{p(D)}_{\text{Evidence}}}\underbrace{p(\theta)}_{\text{Prior}}
\end{equation}

If we are only interested in maximum probable value of $\theta$ in the posterior (point estimate in other words), we can differentiate the posterior w.r.t. $\theta$. However, we have not yet derived the evidence but it does not depend on $\theta$. So, we can claim that the following is true:

$$
\arg \max_{\theta} p(\theta|D) = \arg \max_{\theta} p(D|\theta)p(\theta)
$$

Now, differentiating $p(D|\theta)p(\theta)$ w.r.t $\theta$:

\begin{align}
p(D|\theta)p(\theta) &= \theta^h(1-\theta)^{N-h}\cdot\frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha, \beta)}\\
                     &= \frac{\theta^{h+\alpha-1}(1-\theta)^{N-h+\beta-1}}{B(\alpha, \beta)}\\
\text{Taking log for simplification}\\
\log p(\theta|D)p(\theta) &= (h+\alpha-1)\log(\theta) + (N-h+\beta-1)\log(1-\theta) - \log(B(\alpha, \beta))\\
\\
\frac{d}{d\theta} \log p(\theta|D)p(\theta) &= \frac{h+\alpha-1}{\theta} - \frac{N-h+\beta-1}{1-\theta} = 0\\
\\
\therefore \theta_{MAP} = \frac{h+(\alpha-1)}{N+(\alpha-1)+(\beta-1)}
\end{align}

Now, we have the maximum probable value of $\theta$ from the posterior but if we are interested in the posterior distribution, we must get the evidence!

### Evidence

The formula for computing the evidence is the following:

$$
p(D) = \int\limits_{\theta}p(D|\theta)p(\theta)d\theta
$$

Substituting the values and deriving the formula:

\begin{align}
p(D) &= \int\limits_{0}^{1}p(D|\theta)p(\theta)d\theta\\
     &= \int\limits_{0}^{1}(\theta)^h(1-\theta)^{N-h}\frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}d\theta\\
     &= \frac{1}{B(\alpha,\beta)}\int\limits_{0}^{1}(\theta)^{h+\alpha-1}(1-\theta)^{N-h+\beta-1}d\theta\\
     &= \frac{1}{B(\alpha,\beta)}B(h+\alpha, N-h+\beta)\\
     \therefore p(D) = \frac{B(h+\alpha, N-h+\beta)}{B(\alpha,\beta)}
\end{align}

The last step follows from definition of [the Beta function](https://en.wikipedia.org/wiki/Beta_function).

### Posterior

Now, we have all the required terms to compute the posterior $p(\theta|D)$.

\begin{align}
p(\theta|D) &= \frac{p(D|\theta)}{p(D)}p(\theta)\\
&= \theta^h(1-\theta)^{n-h} \cdot \frac{B(\alpha,\beta)}{B(h+\alpha, N-h+\beta)} \cdot \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}\\
&= \frac{\theta^{h+\alpha-1}(1-\theta)^{N-h+\beta-1}}{B(h+\alpha, N-h+\beta)}
\\
\therefore p(\theta|D) = Beta(h+\alpha, N-h+\beta)
\end{align}

We have successfully derived the posterior and it follows a Beta distribution.

## MAP is not the expected value of the posterior

From [Wikipedia](https://en.wikipedia.org/wiki/Beta_distribution), expected value of our posterior is:

$$
\mathbb{E}_{\theta}(p(\theta|D)) = \frac{h+\alpha}{N + \alpha + \beta}
$$

We derived the MAP as:

$$
\theta_{MAP} = \frac{h+(\alpha-1)}{N+(\alpha-1)+(\beta-1)}
$$

We can see that both values are clearly different.


# Symbolic math introduction

Symbolic mathematics is a maturing technology that lets a computer do maths using symbolic manipulation rather than numerical computation.  Python has support for symbolic computation via the "sympy" package.

The sympy documentation can be found at http://docs.sympy.org/latest/index.html.  The PDF version is just under 2000 pages long, which is quite frightening.

Nonetheless, this notebook will introduce some of the basics of symbolic math using Python.  In particular we will see how to define functions of symbolic variables, and differentiate and integrate them symbolically.

Some other good examples of sympy in use are at http://www.cfm.brown.edu/people/dobrush/am33/SymPy/index.html and https://github.com/sympy/sympy/wiki/Quick-examples.  The first one in particular deals with differential equations, which will be used in subsequent computer assignments.

## Basic differentiation

The cell below imports the symbolic math package, and defines two symbolic variables `x` and `y`.  A symbolic function $f(x,y) = (x^2-2x+3)/y$ is then defined and printed.

```python
import sympy as sp #import the sympy library under the name 'sp'
#?sp.integrate()

x, y = sp.symbols('x y');
f = (x**2 - 2*x + 3)/y;
print(f);
```

    (x**2 - 2*x + 3)/y

```python
# protip: you can make your outputs look sexy with LaTex:
from IPython.display import display
sp.init_printing() # initializes pretty printing. Only needs to be run once.

display(f)
```

Note that `f` here is a symbol representing a function.  It would be nice if the notation made it explicit that it's actually a function of $x$ and $y$, namely `f(x,y)`, but that's not how it works.  However, we can query the free variables:

```python
f.free_symbols
```

We can get sympy to find a symbolic expression for the partial derivative of $f(x,y)$ with respect to $y$:

```python
fpy = sp.diff(f, y)
fpy
```

To evaluate this derivative at some particular values $x=\pi$ and $y=2$ we can substitute into the symbolic expression:

```python
fpyv = fpy.subs([(x, sp.pi), (y, 2)])
fpyv
```

Notice though that this is still a symbolic expression.  It can be evaluated using the "evalf" method, which finally returns a number:

```python
fpyv.evalf()
```

## More advanced differentiation

Symbolic expressions can be manipulated.  For example we can define $g(t) = f(x(t), y(t))$, which in this case given above means
$$g(t) = (x(t)^2-2x(t)+3)/y(t),$$
and find its derivative with respect to time.

```python
t = sp.symbols('t');
#xt, yt = sp.symbols('xt yt', cls=sp.Function);
xt = sp.Function("x")(t);  # x(t)
yt = sp.Function("y")(t)  # y(t)

g = f.subs([(x,xt),(y,yt)]);
gp = sp.diff(g,t);
print(g);
print(gp);
```

    (x(t)**2 - 2*x(t) + 3)/y(t)
    (2*x(t)*Derivative(x(t), t) - 2*Derivative(x(t), t))/y(t) - (x(t)**2 - 2*x(t) + 3)*Derivative(y(t), t)/y(t)**2

## Plotting symbolic functions

The sympy module has a `plot` method that knows how to plot symbolic functions of a single variable.  The function `g` above with $x(t) = \sin(t)$ and $y(t) = \cos(2t)$ is a function of a single time variable `t`, and can be visualised as follows:

```python
gs = g.subs([(xt,sp.sin(t)), (yt,sp.cos(2*t))]);
print(gs);
sp.plot(gs, (t,1,2));
```

    (sin(t)**2 - 2*sin(t) + 3)/cos(2*t)

A roughly equivalent plot could be obtained numerically by creating a lambda function for the expression, evaluating it for a closely-spaced set of values of `t` over the required range, and using standard numerical plotting functions that draw straight lines between the calulated points.  If you increase the number of calculated points over the interval then the  approximation in the above graph becomes more accurate.

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib notebook

tv = np.linspace(1, 2, 10);
gs_h = sp.lambdify(t, gs, modules=['numpy']);
gstv = gs_h(tv);
plt.plot(tv, gstv);

# lambda functions are also called anonymous functions. You can think of creating one as just another way of defining a function. For example:
# For example: say you had a symbolic expression, f = x^2
# writing my_func = sp.lambdify(x,f);
# is the same as going
# def my_func(x):
#     x^2
```

    <IPython.core.display.Javascript object>

The sympy plot function is quite fragile, and might not always work.  Symbolic math packages are amazing, but they're difficult to implement and are sometimes not robust:  you'll find various postings on the internet that give instances of very good symbolic math engines giving a wrong result.  In short, they are useful but you should be careful when using them.

One other nice thing about Jupyter notebooks is that a pretty-print method exists for symbolic expressions.  After the appropriate setup, note the difference in output between the `print` and `display` methods below:

```python
from IPython.display import display
sp.init_printing()  # pretty printing

print(gs);
display(gs);
```

## Symbolic integration

Integration is also a standard function in sympy, so we can find for example the integral
$$y(t) = \int_{-10}^t x(\lambda) d\lambda$$
for $x(t) = e^{-t/10} \cos(t)$:

```python
xt = sp.exp(-t/10)*sp.cos(t);  # x(t)
lamb = sp.symbols('lamb');
xl = xt.subs(t,lamb);  # x(lamb)

yt = sp.integrate(xl, (lamb, -10, t));  # indefinite integral
yt

# to get a definite integral over the range, say -10 to 0, you'd go yt = sp.integrate(xl, (lamb, -10, 0));
# This would give a numeric value.
# NOTE: don't forget about your initial conditions. The definite integral just gives the change in the variable over the
# interval, so you need to add its initial state to this value get the true final state.
```

## Tasks

These tasks involve writing code, or modifying existing code, to meet the objectives described.

1.  Define the expression $y(t) = v_0 t - \frac{1}{2} g t^2$ for some symbolic values of $v_0$ and $g$ using sympy.  You should recognise this as the "altitude" of a particle moving under the influence of gravity, given that the initial velocity at time $t=0$ is $v_0$.  Make a plot of the particle height in meters for $v_0 = 22.5m/s$ given $g = 9.8 m/s^2$, over the range $t=0$ to $t=5s$.

2.  Use symbolic math and the `roots` method to find an expression for the zeros of the expression $y(t)$ above for the same set of conditions.  Substitute to find the nonzero numerical value of $t$ for which your plot in the previous task crosses the x-axis.

For help on the `roots` method, type `?sp.roots()` in an empty code cell and run it.  The method takes a symbolic expression as input, and returns a Python dictionary object as the output.  The roots are contained in the tags of the dictionary.  Note that for a d

3.  Use symbolic differentiation to find the vertical velocity of the particle in the previous task as a function of time, given the same conditions.  Make a plot of this velocity over the same time range.

4.  Suppose the acceleration of a particle is given by $a(t) = 0.2 + \cos(t)$ for positive time.  Use symbolic methods to find and plot the velocity $v(t)$ of the particle over the range $t=0$ to $t=5$ given the initial condition $v(0) = -0.3$.  Then find and plot the position $s(t)$ of the particle over the same time period, given the additional auxiliary condition $s(0) = 0.1$.

```python
?sp.roots
```


# Neural Networks

**[1. Neural Networks](#1.Neural_Networks)**
 * [1.1. Perceptron](#1.1.Perceptron)
 * [1.2. Sigmoid](#1.2.Sigmoid)

**[2. Neural Networks Architecture](#2.Neural_Networks_Architecture)**

**[3. Training Neural Network](#3.Training_Neural_Network)**
 * [3.1. Forward Propagation](#3.1.Forward_Propagation)
 * [3.2. Compute Error](#3.2.Compute_Error)
 * [3.3. Back Propagation](#3.3.Back_Propagation)
 * [3.4. Gradient Descent](#3.4.Gradient_Descent)
 * [3.5. Computational Graph](#3.5.Computational_Graph)
 * [3.6. Gradient_Checking](#3.6.Gradient_Checking)
 * [3.7. Parameter Update](#3.7.Parameter_Update)
 * [3.8. Learning Rate](#3.8.Learning_Rate)

# 1. Neural Networks

Neural networks (NN) are a broad family of algorithms that have formed the basis for the recent resurgence in the computational field called deep learning. Early work on neural networks actually began in the 1950s and 60s. And just recently, neural network has experienced a resurgence of interest, as deep learning has achieved impressive state-of-the-art results.

Neural network is basically a mathematical model built from simple functions with changing parameters. Just like a biological neuron has dendrites to receive signals, a cell body to process them, and an axon to send signals out to other neurons, an artificial neuron has a number of input channels, a processing stage, and one output that can branch out to multiple other artificial neurons. Neurons are interconnected and pass message to each other.

To understand neural networks, Let's get started with **Perceptron**.

## 1.1. Perceptron

A perceptron takes several binary inputs, $x_1,x_2,…,x_n$, and produces a single binary output.

The example above shows a perceptron taking three inputs $x_1, x_2, x_3$. Each input is given a $weight$ $W \in \mathbb{R}$ and it serves to express the importance of its corresponding input in the computation of output for that perceptron. The perceptron output, 0 or 1, is determined by the weighted sum $\sum_i w_ix_i$ with respect to a $threshold$ value as follows:

\begin{equation}
    output = \left\{
            \begin{array}{rl}
            0 & \text{if } \sum_iw_ix_i \leq \text{threshold}\\
            1 & \text{if } \sum_iw_ix_i > \text{threshold}
            \end{array} \right.
\end{equation}

The weighted sum can be categorically defined as a dot product between $w$ and $x$ as follows:

$$\sum_i w_ix_i \equiv w \cdot x$$

where $w$ and $x$ are vectors corresponding to weights and inputs respectively. Introducing a bias term $b \equiv -threshold$ results in

\begin{equation}
    output = \left\{
            \begin{array}{rl}
            0 & \text{if } w \cdot x + b \leq 0\\
            1 & \text{if } w \cdot x + b > 0
            \end{array} \right.
\end{equation}

You can think of the $bias$ as a measure of how easy it is to get the perceptron to output 1. For a perceptron with a high positive $bias$, it is extremely easy for the perceptron to output 1. In constrast, if the $bias$ is relatively  a negative value, it is difficult for the perceptron to output 1.

A way to think about the perceptron is that it is a device that makes **decisions** by weighing up evidence.

```python
import numpy as np
X = np.array([0, 1, 1])
W = np.array([5, 1, -3])
b=5

def perceptron_neuron(X, W, b):
    return int(X.dot(W)+b > 0)

perceptron_neuron(X,W,b)
```

    1

## 1.2. Sigmoid

Small changes to $weights$ and $bias$ of any perceptron in a network can cause the output to flip from 0 to 1 or 1 to 0. This flip can cause the behaviour of the rest of the network to change in a complicated way.

```python
x = np.array([100])
b = np.array([9])
w1 = np.array([-0.08])
w2 = np.array([-0.09])

print(perceptron_neuron(x,w1,b))
print(perceptron_neuron(x,w2,b))

```

    1
    0

The problem above can be overcome by using a Sigmoid neuron. It functions similarly to a Perceptron but modified such that small changes in $weights$ and $bias$ cause only a small change in the output.

As with a Perceptron, a Sigmoid neuron also computes $w \cdot x + b $, but now with the Sigmoid function being incorporated as follows:

\begin{equation}
    z = w \cdot x + b \\
    \sigma(z) = \frac{1}{1+e^{-z}}
\end{equation}

A Sigmoid function produces output between 0 and 1, and the figure below shows  the function. If $z$ is large and positive, the output of a sigmoid neuron approximates to 1, just as it would for a perceptron. Alternatively if $z$ is highly negative, the output approxiates to 0.

```python
def sigmoid(x):
    return 1/(1 + np.exp(-x))

def sigmoid_neuron(X, W, b):
    z = X.dot(W)+b
    return sigmoid(z)

print(sigmoid_neuron(x,w1,b))
print(sigmoid_neuron(x,w2,b))
```

    [0.73105858]
    [0.5]

Click here to go back [Table of Content](#Table_of_Content).

# 2. Neural Networks Architecture

A neural network can take many forms. A typical architecture consists of an input layer (leftmost), an output layer (rightmost), and a middle layer (hidden layer). Each layer can have multiple neurons while the number of neurons in the output layer is dependent on the number of classes.

Click here to go back [Table of Content](#Table_of_Content).

```python
# Create dataset
from sklearn.datasets import make_moons, make_circles
import matplotlib.pyplot as plt
import pandas as pd

seed = 123

np.random.seed(seed)
X, y = make_circles(n_samples=1000, factor=.5, noise=.1, random_state=seed)
```

```python
colors = {0:'red', 1:'blue'}
df = pd.DataFrame(dict(x=X[:,0], y=X[:,1], label=y))

fig, ax = plt.subplots()
grouped = df.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
plt.show()
```

## 3. Training Neural Network

Previously we have learnt that $weights$ express the importance of variables, and $bias$ is a threshold to control the behaviour of neurons. So, how can we determine these $weights$ and $bias$?

Consider these steps:
1. Since we do not know the ideal $weights$ and $bias$, we initialize them using random numbers **(parameters initialization).**
2. Let the data flow through the network with these initialized $weights$ and $bias$ to get a predicted output. This process is known as **forward propagation**.
3. Compare the predicted output with the actual output. An error is computed if there is a difference between them. A high error thus indicates that current $weights$ and $bias$ do not give an accurate prediction. **(compute error)**
4. To fix these $weights$ and $bias$, a backward computation is carried out by finding the partial derivative of error with respect to each $weight$ and $bias$ and then updating their values accordingly. This process is known as **backpropagation**.
5. Repeat steps (2) to (4) until the error is below a pre-defined threshold to obtain the optimized $weights$ and $bias$.

### 3.1. Forward Propagation

The model above has two neurons in the input layer, four neurons (also known as **activation units**) in the hidden layer, and one neuron in the output layer.

$X = [x_1, x_2]$ is the input matrix

$W^{j+1} =$ matrix of $weights$ controlling function mapping from layer $j$ to layer $j + 1$

\begin{align}
     W^{j+1} \equiv
\begin{bmatrix}
w_{11} & w_{12} \\
w_{21} & w_{22} \\
w_{31} & w_{32} \\
w_{41} & w_{42} \\
\end{bmatrix}^{j+1}
\end{align}
$W^{j+1}_{kl}$, $k$ is node in layer $j+1$, $l$ is node in layer $j$

$B^{j+1} = $ matrix of $bias$ controlling function mapping from layer $j$ to layer $j + 1$

\begin{align}
     B^{j+1} \equiv
\begin{bmatrix}
b_{1} & b_{2} & b_{3} & b_{4}
\end{bmatrix}^{j+1}
\end{align}

$B^{j+1}_k$, $k$ is node in layer $j + 1$

the activation units can be label as $a_i^j =$ "activation" of unit $i$ in layer $j$.

if $j=0,\quad a_i^j, $ is equivalent to input layer.

Finally the activation function in layer 1 can be denode as,

\begin{align}
     a_1^1 = \sigma(W_{11}^1x_1+W_{12}^1x_2+B^1_{1}) \\
     a_2^1 = \sigma(W_{21}^1x_1+W_{22}^1x_2+B^1_{2}) \\
     a_3^1 = \sigma(W_{31}^1x_1+W_{32}^1x_2+B^1_{3}) \\
     a_4^1 = \sigma(W_{41}^1x_1+W_{42}^1x_2+B^1_{4})
\end{align}

Simplified using vectorization,

\begin{align}
     a^1 = \sigma(X \cdot W^{1T}+B^1) \\
\end{align}

\begin{align}
     output = \sigma( a^1 \cdot W^{2T}+B^2) \\
\end{align}

Implement forward propagation and feed in the generated data

Tips:
- Using numpy function *random.randn()* to generate a Gaussian distribution with mean 0, and variance 1.

```python
#step 1: parameters initialization
def initialize_params():
    params = {
        'W1': np.random.randn(4,2),
        'B1': np.random.randn(1,4),
        'W2': np.random.randn(1,4),
        'B2': np.random.randn(1,1),
    }
    return params
```

```python
#step 2: forward propagation
x_ = np.array([X[0]])
y_ = np.array([y[0]])

np.random.seed(0)
params = initialize_params()

def forward(X, params):
    a1 = sigmoid(X.dot(params['W1'].T)+params['B1'])
    output = sigmoid(a1.dot(params['W2'].T)+params['B2'])
    cache={'a1':a1, 'params': params}
    return output, cache

output, cache = forward(x_, params)
print('Actual output: ', y_)
print('Predicted output: ',output)
```

    Actual output:  [0]
    Predicted output:  [[0.91620003]]

### 3.2. Compute Error

Error is also known as Loss or Cost.

**Loss function** is usually a function defined on a data point, prediction and label, and measures the penalty.

**Cost function** is usually more general. It might be a sum of loss functions over your training set plus some model complexity penalty (regularization).

To compute the error, we should first define a $cost$ $function$. For simplicity, we will use **One Half Mean Squared Error** as our cost function. The equation is listed below:

\begin{equation}
    MSE = \frac{1}{2n} \sum (\hat y - y)^2
\end{equation}

where $n$ is the number of training samples, $\hat y$ is the predicted output, and $y$ is the actual output. A low cost results is returned if the predicted output is close to the actual output, which indicates a good measure of accuracy.

```python
#step 3: cost function
def mse(yhat, y):
    n = yhat.shape[0]
    return (1/(2*n)) * np.sum(np.square(yhat-y))

mse(output, y_)
```

    0.4197112460136526

### 3.3. Back Propagation

Now we know that to get a good prediction, the cost should be as low/small as possible. To minimize the cost, we have to tune the $weights$ and $bias$, but how can we do that? Do we go with random trial and error or is there a better way to do it? Fortunately, there is a better way and it is called **Gradient Descent**.

### 3.4. Gradient Descent

Gradient descent is an optimization algorithm that iteratively looks for optimal $weights$ and $bias$ so that the cost gets smaller and eventually equals zero.

In the interative process, the gradient (of the cost function with respect to $weights$ and $bias$) is computed. The gradient is the change in cost when $weights$ and $bias$ are changed. This helps us update $weights$ and $bias$ in the direction in which the cost is minimized.

Let's recall the forward propagation equation:

\begin{align}
     a^1 = \sigma(X \cdot W^{1T}+B^1) \\
     output = \sigma( a^1 \cdot W^{2T}+B^2) \\
     cost = \frac{1}{2n} \sum (output - y)^2
\end{align}

Arrange them into a single equation, and $cost$, $L$ can be defined as follows:

\begin{align}
     L = \frac{1}{2n} \sum (\sigma( \sigma(X \cdot W^{1T}+B^1) \cdot W^{2T}+B^2)  - y)^2
\end{align}

From the equation, we want to find the gradient or derivative of $L$ with respect to $W^1, W^2, B^1, B^2$.
\begin{align}
      \frac{\partial L}{\partial W^1}, \frac{\partial L}{\partial W^2}, \frac{\partial L}{\partial B^1}, \frac{\partial L}{\partial B^2}
\end{align}

Computation of partial derivatives of $L$ with respect to the $weights$ and $bias$ can become very complex if the layer of the network grows. To make it simple, we can actually break the equation into smaller compenents and use **chain rule** to derive the partial derivative.

### 3.5. Computational Graph
Eventually, we can think of the forward propagation equation as a computational graph.

#### Scalar Example
\begin{align}
     a_1^1 = \sigma(W_{11}^1x_1+W_{12}^1x_2+B^1_{1}) \quad \equiv \quad \frac{1}{1+\exp^{-(W_1x_1+W_2x_2+B_1)}}
\end{align}

#### Note:
\begin{align}
   L \quad &\rightarrow \quad \frac{\partial L}{\partial L} = 1 \\
   L = \frac{1}{2n} \sum (output - y)^2 \quad &\rightarrow \quad \frac{\partial L}{\partial output} = \frac{1}{n} (output - y) \\
     f(x) = e^x \quad &\rightarrow \quad  \frac{\partial f}{\partial x} = e^x \\
     f(x) = xy \quad &\rightarrow \quad  \frac{\partial f}{\partial x} = y, \quad \frac{\partial f}{\partial y} = x \\
     f(x) = 1/x \quad &\rightarrow \quad  \frac{\partial f}{\partial x} = -1/x^2 \\
     f(x) = x+c \quad &\rightarrow \quad  \frac{\partial f}{\partial x} = 1 \\
     \sigma(x) = \frac{1}{1+e^{-x}} \quad &\rightarrow \quad \frac{\partial f}{\partial \sigma} = \sigma(1-\sigma)
\end{align}

```python
print('X: ', X[0])
print('W11: ', params['W1'][0])
print('B11: ', params['B1'][0][0])

# how is the graph look like in our case?
# calculate forward and backward flows
```

    X:  [-0.08769568  1.08597835]
    W11:  [1.76405235 0.40015721]
    B11:  -0.10321885179355784

#### A Vectorized Example

\begin{align}
     a^1 = \sigma(X \cdot W^{1T}+B^1) \\
\end{align}

#### Note:
\begin{align}
    q = X\cdot(W^T) \quad &\rightarrow \quad \frac{\partial f}{\partial X} = \frac{\partial f}{\partial q} \cdot W , \quad \frac{\partial f}{\partial W} = X^T \cdot \frac{\partial f}{\partial q} \\
    l = q+B \quad &\rightarrow \quad \frac{\partial f}{\partial B} = \begin{bmatrix}1 & 1 \end{bmatrix} \cdot \frac{\partial f}{\partial l} , \quad \frac{\partial f}{\partial q} = \frac{\partial f}{\partial l}
\end{align}

```python
def dmse(output, y):
    return (output - y)/output.shape[0]

def backward(X, output, y, cache):
    grads={}
    a1=cache['a1']
    params=cache['params']

    dloss = dmse(output, y)

    doutput = output*(1-output)*dloss

    #compute gradient of B2 and W2
    dW2 = a1.T.dot(doutput)
    dB2 = np.sum(doutput, axis=0, keepdims=True)

    dX2 = doutput.dot(params['W2'])
    da1 = a1*(1-a1)*dX2

    #compute gradient of B1 and W1
    dW1 = X.T.dot(da1)
    dB1 = np.sum(da1, axis=0, keepdims=True)

    grads['W1'] = dW1.T
    grads['W2'] = dW2.T
    grads['B1'] = dB1
    grads['B2'] = dB2

    return grads

```

```python
X_ = X[:3]
Y_ = y[:3].reshape(-1,1)

def step(X,y,params):

    output, cache = forward(X, params)

    cost = mse(output, y)

    grads = backward(X, output, y, cache)

    return (cost, grads)

np.random.seed(0)
params = initialize_params()

cost, grads = step(X_, Y_, params)

print(cost)
print(grads)
```

    0.4184302966572367
    {'W1': array([[-0.00208634,  0.0076568 ],
           [-0.00053028,  0.00068688],
           [-0.0004186 ,  0.00346495],
           [-0.00170289,  0.00300797]]), 'W2': array([[0.03212358, 0.05777044, 0.02222057, 0.05217427]]), 'B1': array([[0.01009638, 0.00114439, 0.00470392, 0.00425563]]), 'B2': array([[0.07033491]])}

### 3.6. Gradient Checking
We can use Numerical Gradient to evaluate the gradient (Analytical gradient) that we have calculated.

Consider the image above, where the red line is our function, the blue line is the gradient derived from the point $x$, the green line is the approximated gradient from the point of $x$, and $h$ is the step size. It can then be shown that:

$$ \frac{\partial f}{\partial x} \approx \frac{Y_C-Y_B}{X_C-X_B} \quad = \quad \frac{f(x+h) - f(x-h)}{(x+h)-(x-h)} \quad = \quad \frac{f(x+h) - f(x-h)}{2h} $$

##### EXAMPLE

```python
w1 = 3; x1 = 1; w2 = 2; x2 = -2; b1 = 2

h = 1e-4

def f(w1, x1, w2, x2, b1):
    linear = (w1*x1)+(w2*x2)+b1
    return 1/(1+np.exp(-linear))

num_grad_x1 = (f(w1, x1+h, w2, x2, b1) - f(w1, x1-h, w2, x2, b1))/(2*h)
print(num_grad_x1)

```

    0.5898357981348745

```python
# vectorized gradient checking
def gradient_check(f, x, h=0.00001):
    grad = np.zeros_like(x)
  # iterate over all indexes in x
    it = np.nditer(x, flags=['multi_index'])
    while not it.finished:
    # evaluate function at x+h
        ix = it.multi_index
        oldval = x[ix]
        x[ix] = oldval + h # increment by h
        fxph = f(x) # evalute f(x + h)
        x[ix] = oldval - h
        fxnh = f(x) # evaluate f(x - h)
        x[ix] = oldval # restore

        # compute the partial derivative with centered formula
        grad[ix] = ((fxph - fxnh) / (2 * h)).sum() # the slope
        it.iternext() # step to next dimension

    return grad

def rel_error(x, y):
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))
```

```python
for param_name in grads:
    f = lambda W: step(X_, Y_, params)[0]

    param_grad_num = gradient_check(f, params[param_name])
    print('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))

```

    W1 max relative error: 3.437088e-09
    W2 max relative error: 5.328225e-11
    B1 max relative error: 2.182159e-09
    B2 max relative error: 3.440171e-11

### 3.7. Parameter Update
After getting the gradient, parameters are updated depend on the gradients; if it is positive, then the updated parameters reduces in value, and if it is negative, then the updated parameter increases in value. Regardless of the gradient, the main goal is to reach the global minimum.

### 3.8. Learning Rate
Large or small updates are controlled by the learning rate known as $\alpha$. Hence, gradient descent equations are as follows:

\begin{align}
    W^1 &= W^1 - \alpha * \frac{\partial L}{\partial W^1} \\
    B^1 &= B^1 - \alpha * \frac{\partial L}{\partial B^1} \\
    W^2 &= W^2 - \alpha * \frac{\partial L}{\partial W^2} \\
    B^2 &= B^2 - \alpha * \frac{\partial L}{\partial B^2} \\
\end{align}

```python
def update_parameter(params, grads, learning_rate):
    params['W1'] += -learning_rate * grads['W1']
    params['B1'] += -learning_rate * grads['B1']
    params['W2'] += -learning_rate * grads['W2']
    params['B2'] += -learning_rate * grads['B2']

```

```python
params = initialize_params()

def train(X,y,learning_rate=0.1,num_iters=30000,batch_size=256):
    num_train = X.shape[0]
    costs = []
    for it in range(num_iters):
        random_indices = np.random.choice(num_train, batch_size)
        X_batch = X[random_indices]
        y_batch = y[random_indices]

        cost, grads = step(X_batch, y_batch, params)
        costs.append(cost)

        # update parameters
        update_parameter(params, grads, learning_rate)

    return costs

costs = train(X,y.reshape(-1,1))
plt.plot(costs)
```

```python
def predict(X, params):
    W1 = params['W1']
    B1 = params['B1']
    W2 = params['W2']
    B2 = params['B2']

    output, _ = forward(X, params)
    return output
```

```python
# test on training samples
y_pred = []
for i in range(len(X)):
    pred = np.squeeze(predict(X[i],params)).round()
    y_pred.append(pred)

plt.scatter(X[:,0], X[:,1], c=y_pred, linewidths=0, s=20);
```

```python
# test on new samples
X_new, _ = make_circles(n_samples=1000, factor=.5, noise=.1)
y_pred = []

for i in range(len(X_new)):
    pred = np.squeeze(predict(X_new[i],params)).round()
    y_pred.append(pred)

plt.scatter(X_new[:,0], X_new[:,1], c=y_pred, linewidths=0, s=20);
```

# References:

- http://neuralnetworksanddeeplearning.com/

- http://cs231n.github.io/optimization-2/

- http://kineticmaths.com/index.php?title=Numerical_Differentiation

- https://google-developers.appspot.com/machine-learning/crash-course/backprop-scroll/

Click here to go back [Table of Content](#Table_of_Content).