Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

**NB. Do not add new or remove/cut cells in the notebook. Additionally, change the filename of this notebook to your_name-student_number.ipynb before submitting.**

Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your student number below:

```python
STUDENT_NUMBER = "141927"
```

---

# Exercises Lecture 1

## Exercise 1.1: Another ball dropped from a tower (2 points)

A ball is dropped from a tower of height $h$ with initial velocity zero.  Write a program that asks the user to enter the height in meters of the tower and then calculates and prints the time the ball takes until it hits the ground, ignoring air resistance.  Use your program to calculate the time for a ball dropped from a $100\,$m high tower.

```python
def time_from_h(s):
    """calculates time taken for a mass dropped from height
    s to hit the ground
    Args:
        s (float): height in meters
    Returns:
        float: time to reach the ground"""
    # constants
    a = 9.8  # acceleration due to gravity (m/s^2)
    t = 0  # time (s)

    # YOUR CODE HERE
    dt = 0.0001  # time step (s)
    v = 0  # velocity (m/s)

    # iterate time steps until s = 0
    while s > 0:
        v += (-a) * dt  # update velocity downward
        s += v * dt  # update position
        t += dt  # update time passed
    # raise NotImplementedError()
    return t
```

```python
# validation
assert time_from_h(0) == 0.0
assert abs(time_from_h(10) - 1.4285714) < 1e-4
print("Time for a ball dropped from 100m high tower: {} s"
      "".format(time_from_h(100)))
```

    Time for a ball dropped from 100m high tower: 4.51750000000281 s

## Exercise 1.2: Quantum potential step (2 points)

A well-known quantum mechanics problem involves a particle of mass $m$ that encounters a one-dimensional potential step, like this:

<!--  -->

The particle with initial kinetic energy $E$ and wavevector $k_1=\sqrt{2mE}/\hbar$ enters from the left and encounters a
sudden jump in potential energy of height $V$ at position $x=0$.  By solving the Schroedinger equation, one can show that when $E>V$ the particle may either (a) pass the step, in which case it has a lower kinetic energy of $E-V$ on the other side and a correspondingly smaller wavevector of $k_2=\sqrt{2m(E-V)}/\hbar$, or (b) it may be reflected, keeping all of
its kinetic energy and an unchanged wavevector but moving in the opposite direction.  The probabilities $T$ and $R$ for transmission and reflection are given by

$$T = {4k_1k_2\over(k_1+k_2)^2}\,,\qquad
R = \biggl( {k_1-k_2\over k_1+k_2} \biggr)^2.$$

Suppose we have a particle with mass equal to the electron mass $m=9.11\times10^{-31}\,$kg and energy $10\,$eV encountering a potential step of height $9\,$eV.  Write a Python program to compute and print out the transmission and reflection probabilities using the formulas above.

```python
def get_tnr_prob(m, e, v):
    """get transmission and reflection probabilities
    Args:
        m (float): mass of the object in Kg
        e (float): energy of the object in eV
        v (float): height of the potential step in eV

    Retruns:
        float: transmission values
        float: reflection values
    """
    # constants
    hbar = 6.58211956e-16  # eVs
    t, r = 0, 0

    # YOUR CODE HERE
    k1 = (2 * m * e)**0.5 / hbar  # initial wavevector
    k2 = (2 * m * (e - v))**0.5 / hbar  # wavevector after the step

    t = (4 * k1 * k2) / ((k1 + k2)**2)  # transmission probability
    r = ((k1 - k2) / (k1 + k2))**2  # reflection probability
    # raise NotImplementedError()
    return t, r
```

```python
# validation
t, r = get_tnr_prob(8e-30, 20, 15)
assert abs(t - 0.88888888) < 1e-4, "bad transmission probability"
assert abs(r - 0.11111111) < 1e-4, "bad reflection probability"

# ex
m = 9.11e-31  # Kg
e = 10  # eV
v = 9  # eV
t, r = get_tnr_prob(m, e, v)
print("With: M {} Kg, E {} eV, and V {} eV\n"
      "Transmission probability: {}\n"
      "Reflection probability: {}".format(m, e, v, t, r))
```

    With: M 9.11e-31 Kg, E 10 eV, and V 9 eV
    Transmission probability: 0.7301261363877616
    Reflection probability: 0.26987386361223836

## Exercise 1.3: Catalan numbers (2 points)

The Catalan numbers $C_n$ are a sequence of integers 1, 1, 2, 5, 14, 42, 132 $\dots$ that play an important role in quantum mechanics and the theory of disordered systems.  (They were central to Eugene Wigner's proof
of the so-called semicircle law.)  They are given by

$$
C_0 = 1,\qquad C_{n+1} = {4n+2\over n+2}\,C_n.
$$

Write a program that prints in increasing order all Catalan numbers less than or equal to one billion.

```python
def get_catalan_nos():
    """ gets catalan numbers upto 1 billion
    Returns:
        list: list of catalan numbers upto 1 billion"""
    cat_nos = []

    # YOUR CODE HERE
    C = lambda n: (4*(n-1) + 2) / ((n-1) + 2) * cat_nos[n-1] if (n >= 1) else 1  # lambda function to get the n'th catalan number
    cat_nos.append(1)  # append C0 = 1
    n = 1  # start indexing at 1 (C0 has an index 0)

    while True:
        cat = int(C(n))  # get new cat number
        if cat >= 1e9:  # iterate as long as cat number is under 1 billion
            break
        cat_nos.append(cat)  # append to list
        n += 1

    for cat_n in cat_nos:
        print(cat_n, end=' ')  # print the cat numbers
    #raise NotImplementedError()

    return cat_nos
```

```python
# validation
cat_nos = get_catalan_nos()
assert cat_nos[0] == 1
assert cat_nos[1] == 1
assert cat_nos[2] == 2
assert cat_nos[3] == 5
assert cat_nos[-1] <= 1e9
```

    1 1 2 5 14 42 132 429 1430 4862 16796 58786 208012 742900 2674440 9694845 35357670 129644790 477638700

## Exercise 1.4: The Madelung constant (2 points)

In condensed matter physics the Madelung constant gives the total electric potential felt by an atom in a solid.  It depends on the charges on the other atoms nearby and their locations.  Consider for instance solid sodium chloride---table salt.  The sodium chloride crystal has atoms arranged on a cubic lattice, but with alternating sodium and chlorine atoms, the sodium ones having a single positive charge $+e$ and the chlorine ones a single negative charge $-e$, where $e$ is the charge on the electron.  If we label each position on the lattice by three integer
coordinates $(i,j,k)$, then the sodium atoms fall at positions where $i+j+k$ is even, and the chlorine atoms at positions where $i+j+k$ is odd.

Consider a sodium atom at the origin, $i=j=k=0$, and let us calculate the Madelung constant.  If the spacing of atoms on the lattice is $a$, then the distance from the origin to the atom at position $(i,j,k)$ is

$$
\sqrt{(ia)^2 + (ja)^2 + (ka)^2} = a \sqrt{i^2+j^2+k^2},
$$

and the potential at the origin created by such an atom is

$$
V(i,j,k) = \pm {e\over4\pi\epsilon_0 a\sqrt{i^2+j^2+k^2}},
$$

with $\epsilon_0$ being the permittivity of the vacuum and the sign of the expression depending on whether $i+j+k$ is even or odd.  The total potential felt by the sodium atom is then the sum of this quantity over all other atoms.  Let us assume a cubic box around the sodium at the origin, with $L$ atoms in all directions.  Then

$$
V_\textrm{total} = \sum_{\substack{i,j,k=-L\\ \textrm{not }i=j=k=0}}^L
                   \hspace{-0.5em} V(i,j,k)
                 = {e\over4\pi\epsilon_0 a}\,M,
$$

where $M$ is the Madelung constant, at least approximately---technically the Madelung constant is the value of $M$ when $L\to\infty$, but one can get a good approximation just by using a large value of $L$.

Write a program to calculate and print the Madelung constant for sodium chloride.  Use as large a value of $L$ as you can, while still having your program run in reasonable time---say in a minute or less.

```python
def get_madelung_constant(L):
    """gets madelung constant
    Args:
        L (int): number of atoms in all directions
    Returns:
        float: madelungs constant"""
    M = 0
    # YOUR CODE HERE
    # Equation: M = 1 / Sum (i,j,k = -L -> L) +/- [ sqrt(i^2 + j^2 + k^2) ]
    for i in range(-L, L+1):
        for j in range(-L, L+1):
            for k in range(-L, L+1):
                if i == 0 and j == 0 and k == 0:  # skip i = j = k = 0
                    continue
                if (i + j + k) % 2 == 0:
                    M += 1 / (i**2 + j**2 + k**2)**0.5  # even: sodium
                else:
                    M -= 1 / (i**2 + j**2 + k**2)**0.5  # odd: chlorine
    #raise NotImplementedError()
    return M
```

```python
# validation
import time
t = time.time()
M = get_madelung_constant(200)
print("Run time: {} s\n"
      "Madelung constant: {}".format(time.time() - t, M))
assert abs(M + 1.747565) < 1e-2, "wrong value"
```

    Run time: 98.90437960624695 s
    Madelung constant: -1.7446850421707747

## Exercise 1.5: Recursion (3 points)

A useful feature of user-defined functions is *recursion*, the ability of a function to call itself.  For example, consider the following definition of the factorial $n!$ of a positive integer $n$:

$$
n! = \biggl\lbrace\begin{array}{ll}
  1 & \qquad\mbox{if $n=1$,} \\
  n\times(n-1)! & \qquad\mbox{if $n>1$.}
\end{array}
$$

This constitutes a complete definition of the factorial which allows us to calculate the value of $n!$ for any positive integer.  We can employ this definition directly to create a Python function for factorials, like this:

```
def factorial(n):
    if n==1:
        return 1
    else:
        return n*factorial(n-1)
```

Note how, if $n$ is not equal to 1, the function calls itself to calculate the factorial of $n-1$.  This is recursion.  If we now say `print(factorial(5))` the computer will correctly print the answer 120.

(a) We encountered the Catalan numbers $C_n$ previously in Exercise 1.3.  With just a little rearrangement, the definition given there can be rewritten in the form

$$
C_n = \left\lbrace\begin{array}{ll}
  \rule[-9pt]{0pt}{10pt}1 & \qquad\mbox{if $n=0$,} \\
  \dfrac{4n-2}{n+1}\,C_{n-1} & \qquad\mbox{if $n>0$.}
\end{array}\right.
$$

Write a Python function, using recursion, that calculates $C_n$.  Use your function to calculate and print $C_{100}$.

(b) Euclid showed that the greatest common divisor $g(m,n)$ of two nonnegative integers $m$ and $n$ satisfies

$$
g(m,n) = \biggl\lbrace\begin{array}{ll}
  m & \qquad\mbox{if $n=0$,} \\
  g(n,m\>\textrm{mod}\>n) & \qquad\mbox{if $n>0$.}
\end{array}
$$

Write a Python function `g(m,n)` that employs recursion to calculate the greatest common divisor of $m$ and $n$ using this formula.  Use your function to calculate and print the greatest common divisor of 108 and 192.

Comparing the calculation of the Catalan numbers in part (a) above with that of Exercise 1.3, we see that it's possible to do the calculation two ways, either directly or using recursion.  In most cases, if a quantity can
be calculated *without* recursion, then it will be faster to do so, and we normally recommend taking this route if possible.  There are some calculations, however, that are essentially impossible (or at least much
more difficult) without recursion.  We will see some examples later in this book.

```python
def get_catalan_n(n):
    """ gets nth catalan number with recursion
    Args:
        n (int): the index of catalan number needed
    Returns:
        int: nth catalan number
    Raises:
        AssertionError: if n is not integer"""
    cn = 1
    assert isinstance(n, int), "n should be int"

    # YOUR CODE HERE
    C = lambda x: (4 * x - 2) / (x + 1) * C(x - 1) if (x > 0) else 1  # recursive function for C(n)
    cn = C(n)
    #raise NotImplementedError()
    return cn
```

```python
# validation
# catalan numbers
c100 = get_catalan_n(100)
assert get_catalan_n(0) == 1, "bad get_catalan_n func"
assert get_catalan_n(1) == 1, "bad get_catalan_n func"
assert get_catalan_n(2) == 2, "bad get_catalan_n func"
assert get_catalan_n(20) == 6564120420, "bad get_catalan_n func"
print("100th catalan number = {}".format(c100))
```

    100th catalan number = 8.965199470901317e+56

```python
def get_gcd(m, n):
    """gets greatest common divisor of two non negative integers
    Args:
        m (int): first integer
        n (int): second integer
    Returns:
        int: greatest common divisor of m and n
    Raises:
        AssertionError: if m or n are not non negative integers"""
    gcd = 0

    # YOUR CODE HERE
    # special cases m = n, m = 0, n = 0
    if m == n:
        gcd = m
    elif m == 0:
        gcd = n
    elif n == 0:
        gcd = m
    else:
        m_divisors = []  # collect the divisors of m to a list
        n_divisors = []  # collect the divisors of n to a list
        limit = round(max(m, n) / 2) + 1  # upper limit for iteration
        for i in range(1, limit):  # iterate and find divisors until reached the midpoint of max(m, n)
            if m % i == 0:
                m_divisors.append(i)  # append divisors of m
            if n % i == 0:
                n_divisors.append(i)  # append divisors of n
        common_divisors = [value for value in m_divisors if value in n_divisors]  # get the union of the divisor lists
        gcd = max(common_divisors)  # gcd is the largest value in the list
    # raise NotImplementedError()

    return gcd
```

```python
# gcd
assert get_gcd(2, 0) == 2, "bad get_gcd func"
assert get_gcd(2, 10) == 2, "bad get_gcd func"
assert get_gcd(12, 16) == 4, "bad get_gcd func"
print("gcd of {} and {} = {}".format(108, 192, get_gcd(108, 192)))
```

    gcd of 108 and 192 = 12

## Exercise 1.6: Deterministic chaos and the Feigenbaum plot (2 points)

One of the most famous examples of the phenomenon of chaos is the *logistic map*, defined by the equation

$$
x' = rx(1-x).
$$

For a given value of the constant $r$ you take a value of $x$ --- say $x=\frac{1}{2}$ --- and you feed it into the right-hand side of this equation, which gives you a value of $x'$.  Then you take that value and feed it back in on the right-hand side again, which gives you another value, and so forth.  This is a *iterative map*.  You keep doing the same
operation over and over on your value of $x$, and one of three things happens:

1. The value settles down to a fixed number and stays there.  This is called a *fixed point*.  For instance, $x=0$ is always a fixed point of the logistic map.  (You put $x=0$ on the right-hand side and you get $x'=0$ on the left.)

2. It doesn't settle down to a single value, but it settles down into a periodic pattern, rotating around a set of values, such as say four values, repeating them in sequence over and over.  This is called a *limit cycle*.

3. It goes crazy.  It generates a seemingly random sequence of numbers that appear to have no rhyme or reason to them at all.  This is *deterministic chaos*.  "Chaos" because it really does look chaotic, and "deterministic" because even though the values look random, they're not.  They're clearly entirely predictable, because they are given to you by one simple equation.  The behavior is *determined*, although it may not look like it.

Write a program that calculates and displays the behavior of the logistic map.  Here's what you need to do.  For a given value of $r$, start with $x=\frac{1}{2}$, and iterate the logistic map equation a thousand times.  That will give it a chance to settle down to a fixed point or limit cycle if it's going to.  Then run for another thousand iterations and plot the points $(r,x)$ on a graph where the horizontal axis is $r$ and the vertical axis is $x$.  You can either use the ``plot`` function with the options ``"ko"`` or ``"k."`` to draw a graph with dots, one for each point, you can use the ``scatter`` function to draw a scatter plot (which always uses dots).  Repeat the whole calculation for values of $r$ from 1 to 4 in steps of 0.01, plotting the dots for all values of $r$ on the same figure and then finally using the function ``show`` once to display the complete figure.

Your program should generate a distinctive plot that looks like a tree bent over onto its side.  This famous picture is called the *Feigenbaum plot*, after its discoverer Mitchell Feigenbaum, or sometimes the *figtree plot*, a play on the fact that it looks like a tree and Feigenbaum means "figtree" in German.

```python
def logistic_map(r, x_init, n_settle, n_prod):
    """returns a list of values generated through the logistic map
    x' = r * x * (1 - x)
    Args:
        r (float): r value
        x_init (float): initial x value
        n_settle (int): number of values to make the map settle
        n_prod (int): number of values to include in the final map
    Returns:
        list: list of values generated through the logistic map"""
    map_values = []

    # YOUR CODE HERE
    x = x_init  # start with the given value
    for i in range(n_settle):  # iterate n_settle times to settle the value of x
        x = r * x * (1 - x)  # calculate new x
    for i in range(n_prod):  # iterate to produce n_prod new values for x
        x = r * x * (1 - x)  # calculate new x
        map_values.append(x)  # collect values for plotting
    #raise NotImplementedError()

    return map_values
```

```python
# validation
import numpy as np
l_map = logistic_map(2, 0.5, 1000, 1000)
assert np.all([abs(x - 0.5) < 1e-4 for x in l_map]) == True, "bad function"
assert len(l_map) == 1000, "len mismatch"
```

```python
# plotting
from matplotlib import pyplot as plt
r = [x / 20 for x in range(20, 101)]  # 1, 1.05, 1.10, 1.15 ... 5

# YOUR CODE HERE
for i in range(len(r)):  # calculate for every value of r
    l_map = logistic_map(r[i], 0.5, 1000, 100)  # for every r, get a list of values x
    for j in range(len(l_map)):
        plt.plot(r[i], l_map[j], 'k.')  # plot every x with the corresponding value of r

#raise NotImplementedError()
plt.xlabel('r')
plt.ylabel('x')
plt.show()
```

Give answers to the following questions:

(a) For a given value of $r$ what would a fixed point look like on the Feigenbaum plot?  How about a limit cycle?  And what would chaos look like?

(b) Based on your plot, at what value of $r$ does the system move from orderly behavior (fixed points or limit cycles) to chaotic behavior? This point is sometimes called the "edge of chaos."

The logistic map is a very simple mathematical system, but deterministic chaos is seen in many more complex physical systems also, including especially fluid dynamics and the weather.  Because of its apparently random nature, the behavior of chaotic systems is difficult to predict and strongly affected by small perturbations in outside conditions.  You've probably heard of the classic exemplar of chaos in weather systems, the *butterfly effect*, which was popularized by physicist Edward Lorenz in 1972 when he gave a lecture to the American Association for the
Advancement of Science entitled, "Does the flap of a butterfly's wings in Brazil set off a tornado in Texas?"  (Although arguably the first person to suggest the butterfly effect was not a physicist at all, but the science
fiction writer Ray Bradbury in his famous 1952 short story *A Sound of Thunder*, in which a time traveler's careless destruction of a butterfly during a tourist trip to the Jurassic era changes the course of history.)

**Comment:** There is another approach for computing the Feigenbaum plot, which is neater and faster, making use of Python's ability to perform arithmetic with entire arrays.  You could create an array ``r`` with one element containing each distinct value of $r$ you want to investigate: ``[1.0, 1.01, 1.02, ... ]``.  Then create another
array ``x`` of the same size to hold the corresponding values of $x$, which should all be initially set to $0.5$.  Then an iteration of the logistic map can be performed for all values of $r$ at once with a statement of the form ``x = r*x*(1-x)``.  Because of the speed with which Python can perform calculations on arrays, this method should be
significantly faster than the more basic method above.

### Explanation (double click)

(a) Values of x are fixed to a point for all r lower than 3. A limit cycle can be seen, e.g., for value of r between 3 and 3.5, where x cycles between two values. From r values higher than 3.6, x behaves chaotically, except for some small areas of order.

(b) Around 3.6.

## Exercise 1.7: Least-squares fitting and the photoelectric effect (3 points)

It's a common situation in physics that an experiment produces data that lies roughly on a straight line, like the dots in this figure:

The solid line here represents the underlying straight-line form, which we usually don't know, and the points representing the measured data lie roughly along the line but don't fall exactly on it, typically because of measurement error.

The straight line can be represented in the familiar form $y=mx+c$ and a frequent question is what the appropriate values of the slope $m$ and intercept $c$ are that correspond to the measured data.  Since the data don't fall perfectly on a straight line, there is no perfect answer to such a question, but we can find the straight line that gives the best compromise fit to the data.  The standard technique for doing this is the *method of least squares*.

Suppose we make some guess about the parameters $m$ and $c$ for the straight line.  We then calculate the vertical distances between the data points and that line, as represented by the short vertical lines in the figure, then we calculate the sum of the squares of those distances, which we denote $\chi^2$.  If we have $N$ data points with coordinates $(x_i,y_i)$, then $\chi^2$ is given by

$$
\chi^2 = \sum_{i=1}^N (mx_i+c-y_i)^2.
$$

The least-squares fit of the straight line to the data is the straight line that minimizes this total squared distance from data to line.  We find the minimum by differentiating with respect to both $m$ and $c$ and setting the derivatives to zero, which gives

\begin{align*}
m \sum_{i=1}^N x_i^2 + c \sum_{i=1}^N x_i - \sum_{i=1}^N x_iy_i &= 0, \\
m \sum_{i=1}^N x_i + cN - \sum_{i=1}^N y_i &= 0.
\end{align*}

For convenience, let us define the following quantities:
$$
E_x = {1\over N} \sum_{i=1}^N x_i,\qquad
E_y = {1\over N} \sum_{i=1}^N y_i,\qquad
E_{xx} = {1\over N} \sum_{i=1}^N x_i^2,\qquad
E_{xy} = {1\over N} \sum_{i=1}^N x_iy_i,
$$
in terms of which our equations can be written
\begin{align*}
mE_{xx} + cE_x &= E_{xy}\,, \\
mE_x + c &= E_y\,.
\end{align*}

Solving these equations simultaneously for $m$ and $c$ now gives

\begin{equation}
m = {E_{xy}-E_x E_y\over E_{xx} - E_x^2},\qquad
c = {E_{xx}E_y-E_x E_{xy}\over E_{xx} - E_x^2}.
\end{equation}

These are the equations for the least-squares fit of a straight line to $N$ data points.  They tell you the values of $m$ and $c$ for the line that best fits the given data.

(a) In the folder that you obtained this notebook you will find a file called ``millikan.txt``.  The file contains two columns of numbers, giving the $x$ and $y$ coordinates of a set of data points.  Write a program to read these data points and make a graph with one dot or circle for each point.

(b) Add code to your program, before the part that makes the graph, to calculate the quantities $E_x$, $E_y$, $E_{xx}$, and $E_{xy}$ defined above, and from them calculate and print out the slope $m$ and intercept $c$ of the best-fit line.

(c) Now write code that goes through each of the data points in turn and evaluates the quantity $mx_i+c$ using the values of $m$ and $c$ that you calculated.  Store these values in a new array or list, and then graph this new array, as a solid line, on the same plot as the original data. You should end up with a plot of the data points plus a straight line that runs through them.

(d) The data in the file ``millikan.txt`` are taken from a historic experiment by Robert Millikan that measured the *photoelectric effect*.  When light of an appropriate wavelength is shone on the surface of a metal, the photons in the light can strike conduction electrons in the metal and, sometimes, eject them from the surface into the free space above.  The energy of an ejected electron is equal to the energy of the photon that struck it minus a small amount $\phi$ called the *work function* of the surface, which represents the energy needed to remove an electron from the surface.  The energy of a photon is $h\nu$, where $h$ is Planck's constant and $\nu$ is the frequency of the light, and we can measure the energy of an ejected electron by measuring the voltage $V$ that is just sufficient to stop the electron moving.  Then the voltage, frequency, and work function are related by the equation

\begin{equation}
V = {h\over e}\nu - \phi,
\end{equation}

where $e$ is the charge on the electron.  This equation was first given by Albert Einstein in 1905.

The data in the file ``millikan.txt`` represent frequencies $\nu$ in hertz (first column) and voltages $V$ in volts (second column) from photoelectric measurements of this kind.  Using the equation above and the program you wrote, and given that the charge on the electron is $1.602\times10^{-19}\,$C, calculate from Millikan's experimental data a value for Planck's constant.  Compare your value with the accepted value of the constant, which you can find in books or on-line.  You should get a result within a couple of percent of the accepted value.

This calculation is essentially the same as the one that Millikan himself used to determine of the value of Planck's constant, although, lacking a computer, he fitted his straight line to the data by eye.  In part for this work, Millikan was awarded the Nobel prize in physics in 1923.

```python
# data fitting
def fit(x, y):
    """Fits straight line to the data
    Args:
        x (list): list of x values
        y (list): list of y values
    Returns
        float: m slope of the line fit
        float: c constant of the line fit"""
    m, c = 0, 0

    # YOUR CODE HERE
    #x_millikan = []
    #y_millikan = []
    #with open("millikan.txt", "r") as file:  # read file
    #    for line in file.readlines():
    #        row = line.split(' ')  # split lines by whitespace
    #        x_millikan.append(float(row[0]))  # read x
    #        y_millikan.append(float(row[1]))  # and y values
    #x = x_millikan
    #y = y_millikan

    N = len(x)  # number of data points
    # calculate the quantities Ex, Ey, Exx, and Exy
    Ex = Ey = Exx = Exy = 0  # initialize all quantities as zero
    for i in range(N):
            Ex += x[i]
            Ey += y[i]
            Exx += x[i] * x[i]
            Exy += x[i] * y[i]

    # divide all quantities by N
    Ex /= N
    Ey /= N
    Exx /= N
    Exy /= N

    # calculate the slope m and intercept c
    m = (Exy - Ex * Ey) / (Exx - Ex * Ex)
    c = (Exx * Ey - Ex * Exy) / (Exx - Ex * Ex)

    print("The slope m is:", m)
    print("The intercept c is:", c)

    # calculate the best fit line data points, ŷ = mx + c
    y_hat = []
    for i in range(N):
        y_hat.append(m * x[i] + c)

    plt.plot(y, x, 'ko')  # data points
    plt.plot(y_hat, x, 'r-')  # best fit line
    plt.show()
    #raise NotImplementedError()

    return m, c
```

```python
# validation
m, c = fit([0, 1], [0, 1])
assert m == 1, "bad fit"
assert c == 0, "bad fit"
```

```python
def get_plank_const(filename):
    """returns planks constant from millikan experiment
    Args:
        filename (str): name of file with millikan data
    Returns:
        float: plank's constant"""
    # constants
    m = 0 # slope of fit, h/e
    c = 0 # fitting constant, \phi, work funtion
    e = 1.602e-19 # charge on electron

    # YOUR CODE HERE
    x = []
    y = []
    with open(filename, "r") as file:  # read the file
        for line in file.readlines():
            row = line.split(' ')  # split lines by whitespace
            x.append(float(row[0]))  # read x
            y.append(float(row[1]))  # and y values

    m, c = fit(x, y)

    # calculate the Planck's constant
    #h = m * e  # m2 kg / s
    #h_real = 6.626068e-34  # m2 kg / s
    #print("Planck's constant:", h)
    #print("Literature value:", h_real)
    #print("Error: {:.2f}%".format(abs(h_real - h) / h_real * 100))
    #raise NotImplementedError()

    return m * e
```

```python
h = get_plank_const('millikan.txt')
print("plank's constant calculated = {} m^2Kg/s\n"
      "plank's constant = 6.626068e-34 m^2Kg/s".format(h))
assert abs(h - 6.626068e-34) < 1e-35, "error too large"
```

## Exercise 1.8 (2 points)

Write a program to calculate and print the factorial of a number entered by the user. Write your program so that it calculates the factorial using *integer* variables, not floating-point ones.  Use your program to calculate the factorial of 200.

Now modify your program to use floating-point variables instead and again calculate the factorial of 200.  What do you find?  Explain.

```python
def factorial_int(n):
    """factorial of a number with integer variables
    Args:
        n (int): number to calculate factorial of
    Returns:
        int: the factorial of n"""
    # YOUR CODE HERE
    factorial = lambda n: int(n) * factorial(int(n) - 1) if n > 1 else 1
    fact_n = int(factorial(n))
    #raise NotImplementedError()
    return fact_n

def factorial_float(n):
    """factorial of a number with floating variables
    Args:
        n (float): number to calculate factorial of
    Returns:
        int: the factorial of n"""
    # YOUR CODE HERE
    factorial = lambda n: float(n) * factorial(float(n) - 1) if n > 1 else 1
    fact_n = float(factorial(n))
    #raise NotImplementedError()
    return fact_n
```

```python
# validation
print("Factorial of {}\nWith int variables: {}\nWith float variables: {}"
      "".format(200, factorial_int(200), factorial_float(200)))
assert factorial_float(4) == 24
assert factorial_int(4) == 24
assert isinstance(factorial_int(0), int)
assert isinstance(factorial_float(0), float)
```

    Factorial of 200
    With int variables: 788657867364790503552363213932185062295135977687173263294742533244359449963403342920304284011984623904177212138919638830257642790242637105061926624952829931113462857270763317237396988943922445621451664240254033291864131227428294853277524242407573903240321257405579568660226031904170324062351700858796178922222789623703897374720000000000000000000000000000000000000000000000000
    With float variables: inf

### Explanation (double click to edit)

Factorial with int values is a very large number, but the same with floats goes to infinity. This is probably because in floating point arithmetic there is some small errors in very low decimal places, which build up in multiplication to produce a too high value.

## Exercise 1.9: Calculating integrals (2 points)

Suppose we want to calculate the value of the integral

\begin{equation}
I = \int_{-1}^1 \sqrt{1-x^2} \>d x.
\end{equation}

The integrand looks like a semicircle of radius 1:

and hence the value of the integral---the area under the curve---must be $\frac{1}{2}\pi=1.57079632679\ldots$

Alternatively, we can evaluate the integral on the computer by dividing the domain of integration into a large number $N$ of slices of width $h=2/N$ each and then using the Riemann definition of the integral:

\begin{equation}
I = \lim_{N\to\infty} \sum_{k=1}^N hy_k\,,
\end{equation}

where

\begin{equation}
y_k = \sqrt{1 - x_k^2}\qquad\mbox{and}\qquad
x_k = -1 + hk.
\end{equation}

We cannot in practice take the limit $N\to\infty$, but we can make a reasonable approximation by just making $N$ large.

(a) Write a program to evaluate the integral above with $N=100$ and compare the result with the exact value.  The two will not agree very well, because $N=100$ is not a sufficiently large number of slices.

(b) Increase the value of $N$ to get a more accurate value for the integral.  If we require that the program runs in about one second or less, how accurate a value can you get?

Evaluating integrals is a common task in computational physics calculations.  We will study techniques for doing integrals in detail in the next chapter.  As we will see, there are substantially quicker and more accurate methods than the simple one we have used here.

```python
def semicircle_area(N=100):
    """Calculates the area of a semi circle with unit radius using
    the riemann definition of an integral
    Args:
        N (int): number of slices to compute area
    Retruns:
        float: area of the semi circle"""
    N = 100
    area = 0

    # YOUR CODE HERE
    h = 2/N
    area = 0
    for k in range(1, N+1):
        xk = -1 + h*k
        yk = (1 - xk**2)**0.5
        area += h * yk
    #raise NotImplementedError()

    return area
```

```python
# validation
N = 1000

import time
t = time.time()
area = semicircle_area(N)
print("Runtime = {} s\nArea with N {} = {} sq.m.\n"
      "Error = {} %".format((time.time() - t), N, area,
                               abs(area / 1.57079632679 - 1) * 100))

assert abs(area - 1.57079632679) < 1e-2, "error too large"
```

    Runtime = 0.00016355514526367188 s
    Area with N 1000 = 1.5691342555492505 sq.m.
    Error = 0.10581074149480507 %

### Explanation (double click to edit)

YOUR ANSWER HERE


# Demystifying `meshgrid()`

We use the Numpy [meshgrid()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) function to plot 2D arrays but it can be a bit mysterious as to what the "meshgrid" actually does or look like.

The docs say:

_Return coordinate matrices from coordinate vectors._

_Make N-D coordinate arrays for vectorized evaluations of N-D scalar/vector fields over N-D grids, given one-dimensional coordinate arrays x1, x2,..., xn._

## Visualization with a Gaussian
Testing with the initial condition, an isotropic 2D [Gaussian](http://mathworld.wolfram.com/GaussianFunction.html)

$$
g_2(x, y) = \frac{1}{2\pi\sigma^2} \exp\left[-\frac{(x-x_0)^2 + (y-y_0)^2}{2\sigma^2}\right]
$$

Note that the 2D Gaussian can be written as a product of two 1D Gaussians

\begin{align}
g_2(x, y) &= \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(x-x_0)^2}{2\sigma^2}\right] \cdot
          \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(y-y_0)^2}{2\sigma^2}\right]\\
        &= g_1(x) \cdot g_1(y)
\end{align}

Plot $g_2(x, y)$ by (1) constructing it from the product of $g_1$s and (2) from the explicit formula. Use [`np.meshgrid`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) to evaluate the function on a grid.

Use an *asymmetric* grid ($L_x=0.5, N_x=50$ and $L_y=1, N_y=100$) to clearly see the shape of the underlying arrays.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
```

```python
L = 0.5
Lx, Ly = 0.5, 1
Nx, Ny = 50, 100

x = np.linspace(0, Lx, Nx+1)  # need N+1!
y = np.linspace(0, Ly, Ny+1)
X, Y = np.meshgrid(x, y)

def gaussian2D(x, y, u0=0.05, x0=None, y0=None, sigma=0.1*L):
    x0 = np.mean(x) if x0 is None else x0
    y0 = np.mean(y) if y0 is None else y0
    return u0/(2*np.pi*sigma) * np.exp(-((x-x0)**2 + (y-y0)**2)/(2*sigma**2))

def gaussian(x, u0=0.05, x0=None, sigma=0.1*L):
    x0 = np.mean(x) if x0 is None else x0
    return u0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2 / (2*sigma**2))
```

#### Meshgrid

```python
print(x.shape, y.shape)
print(X.shape, Y.shape)
print(X)
print(Y)
```

    (51,) (101,)
    (101, 51) (101, 51)
    [[ 0.    0.01  0.02 ...,  0.48  0.49  0.5 ]
     [ 0.    0.01  0.02 ...,  0.48  0.49  0.5 ]
     [ 0.    0.01  0.02 ...,  0.48  0.49  0.5 ]
     ...,
     [ 0.    0.01  0.02 ...,  0.48  0.49  0.5 ]
     [ 0.    0.01  0.02 ...,  0.48  0.49  0.5 ]
     [ 0.    0.01  0.02 ...,  0.48  0.49  0.5 ]]
    [[ 0.    0.    0.   ...,  0.    0.    0.  ]
     [ 0.01  0.01  0.01 ...,  0.01  0.01  0.01]
     [ 0.02  0.02  0.02 ...,  0.02  0.02  0.02]
     ...,
     [ 0.98  0.98  0.98 ...,  0.98  0.98  0.98]
     [ 0.99  0.99  0.99 ...,  0.99  0.99  0.99]
     [ 1.    1.    1.   ...,  1.    1.    1.  ]]

#### Plot 1D Gaussians
Plot the 1D Gaussians using the meshgrid components.

```python
gaussian(X).shape
```

    (101, 51)

```python
fig = plt.figure()
ax = fig.add_subplot(111)
ax.contourf(X, Y, gaussian(X, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
plt.show()
```

```python
gaussian(Y).shape
```

    (101, 51)

```python
ax = plt.subplot(131)
ax.contourf(X, Y, gaussian(X, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
ax = plt.subplot(132)
ax.contourf(X, Y, gaussian(Y, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
ax = plt.subplot(133)
ax.contourf(X, Y, gaussian(X, sigma=0.1) * gaussian(Y, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
```

#### 2D gaussian

```python
gaussian2D(X, Y).shape
```

    (101, 51)

```python
ax = plt.subplot(111)
ax.contourf(X, Y, gaussian2D(X, Y, sigma=0.1), 30, cmap=plt.cm.viridis)
ax.set_aspect(1)
```

```python

```


# Symbolic Computation with SymPy (Incomplete)

As humans we manipulate and compute with mathematical expressions fluidly, but you may realize that abstract manipulation is not so simple for computers. People have built some very complex computer algebra systems but imagine just how involved the system that can rectify

$$\cos x - \sqrt{3}\sin x  = 2\sin(x+2\pi/3)$$

Such an open source system is called [SymPy](https://www.sympy.org/en/index.html). Feel free to explore their [tutorial](https://docs.sympy.org/latest/tutorial/index.html#tutorial) if you want to dive deeper than this brief intro.

```python
# WARNING! Importing everything with * from a python module is generally a BAD idea.
# You may not realize what you are overwriting. We're just playing around in this notebook so it is ok.

from sympy import *
```

First up, we need to tell sympy what variables we're using. The `var` function does this.

```python
# error if executed before declaration
x
```

$\displaystyle x$

```python
x,y,z,t,r,th = var("x y z t r theta")
```

```python
type(x)
```

    sympy.core.symbol.Symbol

SymPy knows how to assemble ex

```python
x**2
```

$\displaystyle x^{2}$

```python
sin(th)
```

$\displaystyle \sin{\left(\theta \right)}$

```python
type(sin)
```

    sympy.core.function.FunctionClass

```python
sqrt(0-1)
```

$\displaystyle i$

```python
1j
```

    1j

```python

```

tutorial/MVC with Jupyter - Symbolic computation.ipynb


```
# Standard IPython notebook imports
%matplotlib inline

import os

from io import StringIO
import itertools
import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook

import scipy.stats as sstats
import scipy.optimize as sopt

import seaborn as sns
import sklearn.tree
import sklearn.ensemble

import graphviz
from numpy.linalg import inv
```

# Logistic Regression

Linear regression is suitable for problems, where
the forecasted values are real numbers. We use logistic regression,
when we want to label the data with $0$ and $1$.

Let $x\in \mathbb{R}^n$ be a vector of $n$ real numbers, and
$y\in \{0,1\}$ the given class label. Similarly to what was shown
during the lecture, we add an additional element $x_0=1$
to vector $x$, to account for the bias term (and simplify the equations).

Similarly to linear regression, vector
$\Theta\in \mathbb{R}^{n+1}$ parametrizes the model
($n$ coefficients describes the data, the remaining one is the intercept).
In logistic regression, we model conditional probability that
sample $x$ belongs to class $1$ as:

\begin{equation}
p(\text{class}=1|x, \Theta)=h_\Theta(x) = \sigma\left(\sum_{j=0}^n \Theta_j x_j\right)
= \sigma\left(\Theta^T x \right),
\end{equation}

where $\sigma(a) = \frac{1}{1+\exp(-a)}$ is being called the logistic sigmoid
(a function, which plot is s-curved).

An unknown sample $x$ is being labeled $1$ if
$h_\Theta(x)\geq 0.5$, or equivalently, $\Theta^T x \geq 0$.

Classification mismatch between the forecasted values and
the data is being measured most of the time with cross-entropy:

\begin{equation}
    J(\Theta) = - \sum_{i=1}^m y^{(i)} \log \left(h_\Theta (x^{(i)})\right) + (1-y^{(i)}) \log \left(1-h_\Theta (x^{(i)})\right),
\end{equation}

assuming $0\log(0)=0$.

Use logistic regression to distinguish
  _Versicolor_ and _Virginica_ irises. Use only the
  `petal length` and `petal width` features. Use either
  Gradient Descent, or L-BFGS to solve for the optimal $\Theta$.
  Prepare the scatterplot of the data and plot the class separation
  boundary found by logistic regression.

```
from sklearn import datasets
iris = datasets.load_iris()
print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)
petal_length = iris.data[:,iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]
```

    Features:  ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
    Targets:  ['setosa' 'versicolor' 'virginica']

```
#Extract the petal_length and petal_width of versicolors and virginicas

IrisX = np.vstack([np.ones_like(petal_length), petal_length, petal_width])
IrisX = IrisX[:, iris.target!=0]

# Set versicolor=0 and virginia=1
IrisY = (iris.target[iris.target!=0]-1).reshape(1,-1).astype(np.float64)

plt.scatter(IrisX[1,:], IrisX[2,:], c=IrisY.ravel(), cmap='winter')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
```

```
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def logreg_loss(Theta, X, Y):
    #
    # Write a logistic regression cost suitable for use with fmin_l_bfgs
    #

    #reshape Theta
    ThetaR = Theta.reshape(X.shape[0],1)
    Z = Theta.T.dot(X)

    nll = -np.sum([
                    y * np.log2(sigmoid(Z)) \
                    + (1-y) * np.log2(1 - sigmoid(Z)) \
                    for y in Y
                    ])

    grad = X.dot((sigmoid(Z) - Y).T)

    #reshape grad into the shape of Theta, for fmin_l_bfsgb to work
    return nll, grad.reshape(Theta.shape)
```

```
Theta0 = np.zeros((3))

#
# Call a solver
#
ThetaOpt = sopt.fmin_l_bfgs_b(lambda Theta: logreg_loss(Theta, IrisX, IrisY), np.array(Theta0))[0]

#
# Now plot the found separation line
#

plt.scatter(IrisX[1,:], IrisX[2,:], c=IrisY.ravel(), cmap='winter')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
pl_min, pl_max = plt.xlim()
pl = np.linspace(pl_min, pl_max, 1000)
plt.plot(pl, -(ThetaOpt[0]+ThetaOpt[1]*pl)/ThetaOpt[2])
plt.xlim(pl_min, pl_max)
```

```

```


# Chapter 1: Using Neural Nets to Recognize Handwritten Digits

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" }}
});
MathJax.Hub.Queue(
  ["resetEquationNumbers", MathJax.InputJax.TeX],
  ["PreProcess", MathJax.Hub],
  ["Reprocess", MathJax.Hub]
);
```

    <IPython.core.display.Javascript object>

**Perceptron:** is a type of artificial neuron. It takes several binary inputs $x_j$ (inputs that are either 0 or 1) and produces a single binary output.

For each input, there is a weight $w_j$ associated with it, real numbers that express the importance of each input.

The neuron's output, 0 or 1, is determined by whether the weighted sum $\sum_{j} w_j x_j$ is less than or greater than a threshold value, which is a real number.

To simplify this expression, we can write weights and inputs as vectors and take their scalar product to represent the sum. The thresholds can be replaced by a bias $b = -\text{thresholds}$ so that the perceptron rule can be written as:

\begin{equation}
    \text{output} =
    \begin{cases}
        0 & \text{if $\mathbf{w} \cdot \mathbf{x} + b \leq 0$} \\
        1 & \text{if $\mathbf{w} \cdot \mathbf{x} + b > 0$}
    \end{cases}
\end{equation}

The bias can be thought of as a measure of how easy it is for the perceptron to "fire" or output a 1. The larger the $b$, the easier it is to fire.

The problem with a perceptron is its binary output. When we make a small change in weights or biases, we want a small change in output. However, for the perceptron, a small change in $\mathbf{w}$ or $b$ may cause the output to flip discontinuously from 0 to 1 (or vice versa), affecting the rest of the network as well.

**Sigmoid Neuron:** is another type of artificial neuron. It takes in real number inputs $x_j$ between $[0,1]$, unlike the perceptron. It also has weights $w_j$ associated with each input and a bias $b$.

The output of the sigmoid is determined by the sigmoid/logistic function $\sigma (z)$, which is defined by:

\begin{equation}
    \sigma (z) = \frac{1}{1 + e^{-z}},
\end{equation}

where $z = \mathbf{w} \cdot \mathbf{x} + b$. The logistic/sigmoid function ensures that the output is a real number between $0$ and $1$, inclusive. This is in contrary to the output of the perceptron, which is simply a step function.

Such a function that governs the output of the neuron is called an **activation function**. The output value of an activation function is called the **activation value**.

The leftmost layer (first layer) of a neural network is called the **input layer** and the neurons within the layer are called **input neurons**.

The rightmost layer (last layer) of a neural network is called the **output layer** and the neurons within the layer are called **output neurons**.

The middle layers are called **hidden layers** simply because they are not input or output layers.

Neural networks in which the output from one layer is used as input for the next layer are called **feedforward neural networks**. There are no loops in such networks because information is always fed forward, never fed back.

Neural networs in which feedback loops are possible are called **recurrent neural networks (RNN)**. This is more closer to how our brains work.

**Segmentation problem:** splitting an image of many digits into a sequence of separate images, each containing a single digit.

To recognize digits, we use a three-layer neural network.

The input layer contains neurons encoding the value of the input pixels of an image.

Each scanned handwritten digit image consists of $28 \times 28 = 784$ pixels, so the input layer contains $784$ input neurons.

Each input pixel is grayscale, with $0$ representing white and $1$ representing black, and various shades of gray in between $(0,1)$.

The second layer of the network is a hidden layer with an arbitrary number of neurons, which will be fine-tuned later on.

The output layer of the network contains 10 neurons to represent the digits from 0 to 9. When classifying digits, the neural network decides on the output neuron with the highest activation value. For instance, if the 3rd output neuron (from 0th to 9th) has the highest activation value, much larger than the others, then it can confidently say that the input image is a 3.

The **MNIST** dataset contains 60000 training images and 10000 test images, each consisting of 784 grayscale pixels.

We denote a training input vector as $\mathbf{x}$, a 784-dimensional column vector. Each entry of the vector represents the grey value of a single pixel in the image.

The desired output vector is denoted by $\mathbf{y} = \mathbf{y} (\mathbf{x})$, which is a 10-dimensional column vector. For example, if an image $\mathbf{x}$ represents a 6, then $\mathbf{y} (\mathbf{x})$ would be

$$
\mathbf{y} (\mathbf{x}) = (0,0,0,0,0,0,1,0,0,0)^T
$$

To obtain a measure of how much deviation the network's activation value predictions are from the actual value, we define the **quadratic cost/loss/objective function** $C(\mathbf{W}, \mathbf{b})$, summed over all the images/training inputs $\mathbf{x}$ (also known as the **mean squared error MSE**):

\begin{equation}
C(\mathbf{W}, \mathbf{b}) = \frac{1}{2n} \sum_{x} \| \mathbf{y}(\mathbf{x}) - \mathbf{a}(\mathbf{x}, \mathbf{W}, \mathbf{b}) \|^{2}
\end{equation}

The $1/2$ is introduced because we want the derivative of $C$ to look pretty, without trailing a $2$ in front, and the factor $1/n$ is introduced to normalize/scale  the total cost by taking the average.

The cost function is non-negative because of the squares, and tends to $0$ (becomes small) as the actual output vectors $\mathbf{a}$ tends to the desired output vectors $\mathbf{y}$.

Hence, the aim of our training algorithm will be to minimize the cost as a function of weights and biases.

We minimize the cost function instead of maximizing the number of images classified correctly because the cost function is a continuous function of the weights and biases and the number of images is not a continuous/smooth function.

We choose to use the MSE and minimize it because it is a standard measurement of error.

One way to minimize a function is to find the first derivative and set it to 0. This analytical method is VERY slow for large numbers of weights and biases. Another technique is called **gradient descent**, which is described as follows.

Suppose we have a function of two variables $C(v_1, v_2)$. A small change in $C$ can be brought about by small changes in $v_1$ and $v_2$:

\begin{equation}
\Delta C \approx \frac{\partial C}{\partial v_1} \Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2
\end{equation}

We would like $\Delta C$ to be negative because we want to descend along a path which brings $C$ to a minimum. That is, the updated $C'$ should be smaller than the previous $C$:

\begin{equation}
C \to C' = C + \Delta C
\end{equation}

The path of descent is chosen by choosing the values of $\Delta v_1$ and $\Delta v_2$ such that $\Delta C$ is negative

We define the vector of changes in $v_i$ to be

$$
\mathbf{\Delta v} = (\Delta v_1, \Delta v_2)^{T}
$$

and the gradient of $C$ to be

\begin{equation}
\nabla C = \bigg( \frac{\partial C}{\partial v_1}, \frac{\partial C}{\partial v_2} \bigg)^{T}.
\end{equation}

Hence, we can rewrite (4) as

\begin{equation}
    \Delta C \approx \nabla C \cdot \Delta \mathbf{v}.
\end{equation}

If we want $\Delta C$ to be negative, we choose

\begin{equation}
    \Delta \mathbf{v} = - \eta \nabla C,
\end{equation}

where $\eta$ is a positive parameter known as the **learning rate**, chosen within the limits of the approximation in (5) to ensure that $\Delta C$ always stays negative: $\Delta C \approx - \eta \| \nabla C \|^{2}$. We use (6) to compute a value for $\Delta \mathbf{v}$ and update the vector $\mathbf{v}$ by that incremental amount:

\begin{equation}
\mathbf{v} \to \mathbf{v'} = \mathbf{v} - \eta \nabla C
\end{equation}

Incrementally updating the vector like this draws us a path of descent for $C$ and hence, $C$ keeps decreasing until we reach a **global minimum** (or a **local minimum** if we are not so lucky).

In summary, the gradient descent algorithm is a procedure to repeatedly compute the gradient $\nabla C$, and move in the negative/opposite direction of it, "falling down" the slope of a valley. The whole argument above can be applied to a cost function with any number of variables.

If we try to mimic a rolling ball for gradient descent, we run into second partial derivatives, which require exponentially more computations, making it impractical to use. However, there are other methods that avoid this problem. (Perhaps Hamiltonian mechanics could be used?)

The problem with gradient descent is that we try to compute the cost fuction gradient separately for each training input $\mathbf{x}$ and average them, which is slow when there are a large number of training inputs.

To overcome this problem, we use the method of **stochastic gradient descent**. The idea is to randomly pick out a small sample of training inputs, compute the average cost gradient, update the variable vector and incrementally decrease the cost, and repeat the procedure with another small sample until we have exhausted all the training data. This is said to complete an **epoch** of training. At this point, we start over with a new training epoch.

Stochastic gradient descent speeds up training because we are not calculating the average cost gradient for the full batch size of $n = 60000$ inputs and then updating our variable vector and cost afterwards, and repeating. Instead, we calculate the average cost gradient for a small mini-batch of $m = 10$ inputs, update the variable vector and cost, and repeating. This speeds up gradient estimation by a factor of 6000.

Think about gradient descent like a sober man walking in careful steps on the shortest path to his destination. Compare this to stochastic gradient descent, which is like a drunk man randomly taking steps towards the general direction of his destination. The exact computation of the gradient is not important, we need only to move in a general direction towards the minima to decrease the cost $C$.

# Chapter 2: How the Backpropagation Algorithm Works

**Backpropagation:** is a fast algorithm for computing the gradient of the cost function. This allows our neural network to learn faster from the gradient descent algorithm. Backpropagation is about understanding how changing the weights and biases in a network changes the cost function.

The weight of the connection from the $k^\text{th}$ neuron in the $(l - 1)^\text{th}$ layer to the $j^\text{th}$ neuron in the $l^\text{th}$ layer is denoted by $w^{l}_{jk}$. Bias of the $j^\text{th}$ neuron in the $l^\text{th}$ layer is denoted by $b_{j}^{l}$. Activation of the $j^\text{th}$ neuron in the $l^\text{th}$ layer is denoted by $a_{j}^{l}$.

$$
\text{ weight } w_{jk}^{l} \text{ : connects }
k^\text{th} \text{ neuron in } (l - 1)^\text{th} \text{ layer }
\to j^\text{th} \text{ neuron in } l^\text{th} \text{ layer }
$$

$$
\text{bias } b_{j}^{l} \text{ : bias of } j^\text{th} \text{ neuron in } l^\text{th} \text{ layer }
$$

$$
\text{activation } a_{j}^{l} \text{ : activation of }
j^\text{th} \text{ neuron in } l^\text{th} \text{ layer }
$$

With this notation, we can write the activation $a_{j}^{l}$ of the $j^\text{th}$ neuron in the $l^\text{th}$ layer to be related to the activation $a_{k}^{l-1}$ of the $k^\text{th}$ neuron in the $(l - 1)^\text{th}$ layer through the equation:

\begin{equation}
a_{j}^{l} = \sigma \bigg( \sum_{k} w_{jk}^{l} a_{k}^{l - 1} + b_{j}^{l} \bigg)
\end{equation}

Equation (10) is written in components, which represent the activation and bias vectors and the weight matrix. We write it in a compact, vectorized form, without indices, also defining the **weighted input** $z_{j}^{l} = \mathbf{W}^{l} \mathbf{a}^{l-1} + \mathbf{b}^{l} = \sum_{k} w_{jk}^{l} a_{k}^{l - 1} + b_{j}^{l}$ to the activation function for neuron $j$ in layer $l$:

\begin{equation}
\mathbf{a}^{l} = \sigma ( \mathbf{W}^{l} \mathbf{a}^{l-1} + \mathbf{b}^{l} )
\end{equation}

$$
\mathbf{a}^{l} = \sigma ( \mathbf{z}^{l} )
$$

The goal of backpropagation is to compute the partial derivatives $\partial C / \partial w$ and $\partial C / \partial b$ of the cost function $C$ with respect to any weight $w$ or bias $b$ in the network.

For backpropagation to work, we need to make two main assumptions about the form of the cost function. The cost function is written as an average:

\begin{equation}
C = \frac{1}{2n} \sum_{\mathbf{x}} \| \mathbf{y(\mathbf{x})} - \mathbf{a}^{L}(\mathbf{x}) \|^{2},
\end{equation}

where $n$ is the total number of training inputs, the sum is over individual training inputs $\mathbf{x}$, $\mathbf{y} = \mathbf{y(\mathbf{x})}$ is the corresponding desired output, $L$ is the number of layers in the network (so the $L^\text{th}$ layer is the last layer in the network), and $\mathbf{a} = \mathbf{a}^L(\mathbf{x})$ is the output activation vector of the last layer in the network when $\mathbf{x}$ is input.

The first assumption about the cost function is that it can be written as an average $C = \frac{1}{n} \sum_{\mathbf{x}} C_{\mathbf{x}}$ over cost functions $C_{\mathbf{x}}$ for individual training inputs $\mathbf{x}$.

The reason for this assumption is because we want to compute the partial derivatives $\partial C_{\mathbf{x}} / \partial w$ and $\partial C_{\mathbf{x}} / \partial b$ for singular training inputs and then recover $\partial C / \partial w$ and $\partial C / \partial b$ by averaging over training inputs.

The second assumption about the cost function is that it can be written as a function of the activation output values from the last layer in the neural network. For instance, for a single training input $\mathbf{x}$, the cost function can be written as:

\begin{equation}
C_{\mathbf{x}} = \frac{1}{2} \| \mathbf{y} - \mathbf{a}^{L} \|^{2} =
\frac{1}{2} \sum_{i} ( y_{i} - a_{i}^{L} )^{2},
\end{equation}

where $i$ stands for the index number of neurons in the last $L^\text{th}$ layer of the network.

**Hadamard product:** is the elementwise product of two vectors of the same length/dimension. For instance, for vectors $\mathbf{s}$ and $\mathbf{t}$, the Hadamard product between them is written as:

\begin{equation}
\mathbf{s} \odot \mathbf{t} = ( s \odot t )_{j} = s_{j} t_{j}
\end{equation}

We introduce an intermediate quantity $\delta_{j}^{l}$, which is the **error** of the $j^\text{th}$ neuron in the $l^\text{th}$ layer. This error is defined in the following way:

As the weighted input $z_{j}^{l}$ comes in to this neuron, instead of outputting $\sigma ( z_{j}^{l} )$, the neuron instead outputs $\sigma ( z_{j}^{l} + \Delta z_{j}^{l} )$. This change propagates throughout the network and so the overall change in cost is given by:

$$
\Delta C = \frac{\partial C}{\partial z_{j}^{l}} \Delta z_{j}^{l},
$$

where $\Delta z_{j}^{l}$ is the change in weighted input. This change is such that in updating $C \to C' = C + \Delta C$, the updated cost is smaller than the previous cost, so $\Delta z_{j}^{l}$ has the opposite sign to $\partial C / \partial z_{j}^{l}$. The closer $\partial C / \partial z_{j}^{l}$ is to $0$, the smaller the perturbation $\Delta z_{j}^{l}$ needs to be, so a good measure of the error in the neuron can be given by:

\begin{equation}
\delta_{j}^{l} = \frac{\partial C}{\partial z_{j}^{l}}
\end{equation}

The vector $\delta^{l}$ denotes the vector of errors for all neurons associated with layer $l$.

**Equation for the error in the output layer:**

\begin{equation}
\delta_{j}^{L} = \frac{\partial C}{\partial a_{j}^{L}} \sigma' ( z_{j}^{L} )
\end{equation}

$$
\delta_{j}^{L} = \frac{\partial C}{\partial a_{j}^{L}}
\frac{\partial a_{j}^{L}}{\partial z_{j}^{L}}
$$

The first term $\partial C / \partial a_{j}^{L}$ measures how fast the cost is changing as a function of the $j^\text{th}$ neuron's output activation in the last $L^\text{th}$ layer. The second term $\sigma' ( z_{j}^{L} )$ measures how fast the activation function $\sigma$ is changing at $z_{j}^{L}$. Equation (15) is a componentwise expression, so we can rewrite it in matrix-based form:

\begin{equation}
\delta^{L} = \nabla_{a} C \odot \sigma' ( \mathbf{z}^{L} ),
\end{equation}

where $\nabla_{a} C$ is defined to be a vector whose components are the partial derivatives $\partial C / \partial a_{j}^{L}$, which is elementwise multiplied with the sigmoid derivative. We may also write it in matrix form without using the Hadamard product:

\begin{equation}
\delta^{L} = \mathbf{S}^{L} \nabla_{a} C
\end{equation}

$$
\delta_{j}^{L} = \text{diag} ( \sigma' ( z_{j}^{L} ) ) \nabla_{a} C,
$$

where $\mathbf{S} (\mathbf{z}^{L})$ is a $j \times j$ diagonal square matrix, whose diagonal entries are the values $\sigma' ( z_{j}^{L} )$, and the off-diagonal entries are $0$. The reason the off-diagonal terms are $0$ is because the activation value of neuron $j$ in output layer $L$ does not depend on the weighted input to another neuron in the same layer.

**Equation for the error $\delta^{l}$ in terms of the error in the next layer $\delta^{l+1}$:**

\begin{equation}
\delta^{l} = ((\mathbf{W}^{l+1})^{T} \delta^{l+1}) \odot \sigma' (\mathbf{z}^{l}),
\end{equation}

where $(\mathbf{W}^{l+1})^\text{T}$ is the transpose of the weight matrix $\mathbf{W}^{l+1}$ for the $(l+1)^\text{th}$ layer. Similarly, this equation can be written without the Hadamard product:

\begin{equation}
\delta^{l} = \mathbf{S}^{l} (\mathbf{W}^{l+1})^\text{T} \delta^{l+1}
\end{equation}

$$
\delta_{j}^{l} = \text{diag}(\sigma' (z_{j}^{l})) (w_{jk}^{l+1})^\text{T}
\delta_{k}^{l+1}
$$

Hence, by combining Equations (17) and (19) in iterative fashion, we can derive:

\begin{equation}
\delta^{l} = \mathbf{S}^{l} (\mathbf{W}^{l+1})^\text{T} \mathbf{S}^{l+1} (\mathbf{W}^{l+2})^\text{T} \ldots \mathbf{S}^{L-1} (\mathbf{W}^{L})^\text{T}
\mathbf{S}^{L} \nabla_{a} C
\end{equation}

**Equation for the rate of change in cost with respect to any bias in the network:**

\begin{equation}
\frac{\partial C}{\partial b_{j}^{l}} = \delta_{j}^{l},
\end{equation}

that is, the error is $\delta_{j}^{l}$ is exactly equal to the rate of change $\partial C / \partial b_{j}^{l}$.

**Equation for the rate of change in cost with respect to any weight in the network:**

\begin{equation}
\frac{\partial C}{\partial w_{jk}^{l}} = a_{k}^{l-1} \delta_{j}^{l},
\end{equation}

where $a_{k}^{l-1}$ is the input activation value that comes with the weight $w_{jk}^{l}$ as input to the $j^\text{th}$ neuron in the $l^\text{th}$ layer and $\delta_{j}^{l}$ is the output error of that neuron. Hence, a weight will learn slowly if either the input neuron is low-activation or if the output neuron has saturated, which means $\sigma' (z_{j}^{l}) \approx 0$, giving rise to a small $\delta_{j}^{l}$.

$$
\textbf{Summary: the equations of backpropagation}
$$

$$
\delta^{L} = \nabla_a C \odot \sigma'(\mathbf{z}^{L})
$$

$$
\delta^{l} = ((\mathbf{W}^{l+1})^\text{T} \delta^{l+1}) \odot \sigma' (\mathbf{z}^{l})
$$

$$
\frac{\partial C}{\partial b_{j}^{l}} = \delta_{j}^{l}
$$

$$
\frac{\partial C}{\partial w_{jk}^{l}} = a_{k}^{l-1} \delta_{j}^{l}
$$

We can think of backpropagation as a way of computing the gradient of the cost function by systematically applying the chain rule from multi-variable calculus.

Einstein summation convention will be used in some of the sections below.

### The Backpropagation Algorithm:

**1. Input $\mathbf{x}$:** Provide a training input $x_{k}$ for each neuron $k$ in the first input layer. The corresponding activation of this input layer is set as $a_{j}^{1} = \sigma ( w_{jk}^{1} x_{k} )$.

$$
\mathbf{a}^{1} = \sigma ( \mathbf{W}^{1} \mathbf{x} )
$$

**2. Feedforward:** For each layer $l = 2, 3, \ldots, L$ $(l \neq 1)$, compute the weighted inputs $z_{j}^{l} = w_{jk}^{l} a_{k}^{l-1} + b_{j}^{l}$ and the activation values of each neuron in each layer $a_{j}^{l} = \sigma ( z_{j}^{l} )$.

$$
\mathbf{z}^{l} = \mathbf{W}^{l} \mathbf{a}^{l-1} + \mathbf{b}^{l} \quad{\text{and}} \quad{} \mathbf{a}^{l} = \sigma ( \mathbf{z}^{l} )
$$

**3. Output error $\delta^{L}$:** Compute the error vector of the last output layer $\delta_{j}^{L} = (\partial C / \partial a_{j}^{L}) \odot (\partial a_{j}^{L} / \partial z_{j}^{L})$.

$$
\delta^{L} = \nabla_a C \odot \sigma' ( \mathbf{z}^{L} )
$$

**4. Backpropagate the error:** For each layer $l = L-1, L-2, \ldots, 2$ $(l \neq 1)$, compute the error vectors $\delta_{j}^{l} = ((w_{kj}^{l+1})^\text{T} \delta_{k}^{l+1}) \odot \sigma' (z_{j}^{l})$.

$$
\delta^{l} = ((\mathbf{W}^{l+1})^\text{T}\delta^{l+1}) \odot \sigma' ( \mathbf{z}^{l} )
$$

**5. Output gradient:** The gradient of the cost function can then be calculated by:

$$
\frac{\partial C}{\partial w_{jk}^{l}} = a_{k}^{l-1} \delta_{j}^{l} \quad \text{and} \quad
\frac{\partial C}{\partial b_{j}^{l}} = \delta_{j}^{l}
$$

From inspecting the recipe, the algorithm is called backpropagation because we compute the error vectors $\delta^{l}$ backwards, starting from the final layer. This is a consequence of the fact that the cost is a function of outputs from the network, so we need to start from the final output layer and work backwards through the layers to obtain useful expressions.

### Mini-Batch Stochastic Gradient Descent  Algorithm with Backpropagation:

**1. Input mini-batch $m$:** Provide a mini-batch of $m$ training inputs for the first input layer of the neural network.

**2. For each training input $\mathbf{x}$ in the mini-batch $m$:** Set the corresponding input activation $a^{\mathbf{x},1}$, and perform the backpropagation algorithm from Steps 2 to 4:

**$\quad{}$ Feedforward:**
$$
\mathbf{z}^{\mathbf{x}, l} = \mathbf{W}^{l} \mathbf{a}^{\mathbf{x}, l-1}
+ \mathbf{b}^{l} \quad{} \text{and} \quad{} \mathbf{a}^{\mathbf{x},l} = \sigma ( \mathbf{z}^{\mathbf{x}, l} )
$$

**$\quad{}$ Output Error:**
$$
\delta^{\mathbf{x},L} = \nabla_a C_{\mathbf{x}} \odot \sigma' (\mathbf{z}^{\mathbf{x},L})
$$

**$\quad{}$ Backpropagate the Error:**
$$
\delta^{\mathbf{x},l} = ((\mathbf{W}^{l+1})^\text{T}\delta^{\mathbf{x},l+1}) \odot \sigma' ( \mathbf{z}^{\mathbf{x},l} )
$$

**3. Gradient Descent:** For each layer $l = L, L-1,\ldots,2$ update the weights and biases for all training inputs in the mini-batch according to the rules:

\begin{equation}
\mathbf{W}^{l} \to \mathbf{W}^{l} - \frac{\eta}{m} \sum_{\mathbf{x}} \delta^{\mathbf{x},l} (\mathbf{a}^{\mathbf{x},l-1})^\text{T}
\end{equation}

\begin{equation}
\mathbf{b}^{l} \to \mathbf{b}^{l} - \frac{\eta}{m} \sum_{\mathbf{x}}
\delta^{\mathbf{x}, l}
\end{equation}

These are simply the rules for updating the variable vector $\mathbf{v} \to \mathbf{v'} = \mathbf{v} - \eta \nabla C$ from Equation (9), where the variable vector includes all weights and biases in the network, and the gradient of the cost function computed using backpropagation. The updates are averaged over a mini-batch $m$, because the gradient is computed using that mini-batch.

# Chapter 3: Improving the Way Neural Networks Learn

By using the sigmoid as an activation function, the neuron learns much more slowly even though it's badly wrong about an outcome. This is quite different from how humans learn, because humans learn fastest when they are very wrong about something.

Inspecting Equations (22) and (23), the gradient of cost depends on the errors $\delta^l$, which in turn depends on the derivative of the sigmoid. The sigmoid derivative becomes very small as it approaches the outputs $0$ or $1$, which means the cost gradients also become very small. Since the neuron learns by changing the weights and biases at a rate determined by the cost gradient, small cost gradients mean that the neuron learns slowly.

To avoid this slowdown in learning, we use an alternative cost function (not quadratic) called the **cross-entropy cost function**:

\begin{equation}
C = - \frac{1}{n} \sum_{x} \sum_{j} y_{j} \ln{a_{j}^{L}} + (1-y_{j}) \ln{(1-a_{j}^{L})},
\end{equation}

where $n$ is the total number of training inputs, the x-sum is over all training inputs $x$, the j-sum is over all final layer neurons, $a_j$ are the actual output values of final layer neurons and $y_j$ is the corresponding desired output for each final layer neuron $j$. Similar to the quadratic function, the cost-entropy is positive, and tends toward zero as the neuron gets better at computing the desired output $y$ for all inputs $x$.

Unlike the quadratic function though, the cost-entropy doesn't slow down because the cost gradients do not have the sigmoid derivative term in it.

The cost-entropy is almost always the better choice, provided the output neurons are sigmoid neurons. However, if the output neurons are linear neurons, i.e. $a_{j}^{L} = z_{j}^{L}$, then the quadratic cost is an appropriate function to use because there is no slowdown in learning.

We can derive the cross-entropy cost function by integrating the cost gradients obtained from the quadratic cost function $\partial C / \partial w_{jk}^{L}$ and $\partial C / \partial b_{j}^{L}$, with the quadratic gradients divided by the sigmoid derivative $\sigma' (z_{j}^{L})$. In other words, we want a cost function such that there is no sigmoid derivative when we take the gradient.

```python

```


## Zakharov

The Zakharov function has no local minima except the global one. It is shown here in its two-dimensional form.

**Definition**

\begin{align}
\begin{split}
f(x) &=& \sum\limits_{i=1}^n {x_i^2} + \bigg( \frac{1}{2} \sum\limits_{i=1}^n {ix_i} \bigg)^2 + \bigg( \frac{1}{2} \sum\limits_{i=1}^n {ix_i} \bigg)^4, \\[2mm]
&& -10 \leq x_i \leq 10 \quad i=1,\ldots,n
\end{split}
\end{align}

**Optimum**

$$f(x^*) = 0 \; \text{at} \; x^* = (0,\ldots,0) $$

**Contour**

```python
import numpy as np
from pymoo.factory import get_problem, get_visualization

problem = get_problem("zakharov", n_var=2)
get_visualization("fitness-landscape", problem, angle=(45, 45), _type="surface").show()
get_visualization("fitness-landscape", problem, _type="contour", contour_levels = 200, colorbar=True).show()
```

```python

```


# 09 Differentiation – Theory (Part 1)
See *Computational Physics* (Landau, Páez, Bordeianu), Chapter 5.1 – 5.6.

These slides include material from  *Computational Physics. eTextBook Python 3rd Edition.* Copyright © 2012 Landau, Rubin, Páez. Used under the Creative-Commons Attribution-NonCommerical-ShareAlike 3.0 Unported License.

Taking numerical *derivatives* is based on the elementary definition

$$
\frac{dy(t)}{dt} := \lim_{h\rightarrow 0} \frac{y(t+h) - y(t)}{h}
$$

Problem $h\rightarrow 0$:
* subtractive cancelations in numerator
* overflow `number / epsilon_m`

## Forward difference
Expand $y(t)$ in Taylor series

$$
y(t+h) = y(t) + h y^{(1)}(t) + \frac{h^2}{2!} y^{(2)}(t) + \frac{h^3}{3!} y^{(3)}(t) + \cdots
$$

(with the derivatives $y^{(n)} \equiv \frac{d^{n}y}{dt^n}$)

$$
\frac{y(t+h) - y(t)}{h} = y^{(1)}(t) + \frac{h}{2!} y^{(2)}(t) + \frac{h^2}{3!} y^{(3)}(t) + \cdots
$$

### Forward difference algorithm

Neglect $\mathcal{O}(h^2)$ terms:

$$
\frac{dy(t)}{dt} \simeq D_\text{fd} y(t) \equiv \frac{y(t+h) - y(t)}{h}
$$

$$
D_\text{fd} y(t) \equiv \frac{y(t+h) - y(t)}{h}
$$

Image from Computational Physics. eTextBook Python 3rd Edition. Copyright © 2012 Landau, Rubin, Páez. Used under the Creative-Commons Attribution-NonCommerical-ShareAlike 3.0 Unported License.

### Approximation error in the forward difference algorithm
Substitute the Taylor series into $D_\text{fd} y(t)$

$$
D_\text{fd} y(t) = y^{(1)}(t) - \frac{h}{2} y^{(2)}(t) + \cdots
$$
so the error is **first order in $h$**:
$$
y^{(1)}(t) - D_\text{fd} y(t) = \frac{1}{2} h y^{(2)}(t) + \mathcal{O}(h^2)
$$

### Example for $D_\text{fd}$

Quadratic function
$$
y(t) = a + bt^2 \quad \text{and}\quad y^{(1)}(t) = 2bt
$$
Forward difference approximation
$$
D_\text{fd} y(t) = \frac{y(t+h) - y(t)}{h} = 2bt + bh
$$
is only good for small $h \ll \frac{1}{b}$.

## Central difference
Improve over the (rather bad) error in the *forward difference algorithm* by using the *central difference*.

Image from Computational Physics. eTextBook Python 3rd Edition. Copyright © 2012 Landau, Rubin, Páez. Used under the Creative-Commons Attribution-NonCommerical-ShareAlike 3.0 Unported License.

### Central difference algorithm
$$
D_\text{cd} y(t) \equiv \frac{y\Big(t + \frac{h}{2}\Big) - y\Big(t - \frac{h}{2}\Big)}{h}
$$

Applied to the parabola $y(t) = a + bt^2$ yields the exact result:
$$
D_\text{cd} y(t)= 2bt
$$

### Approximation error in $D_\text{cd}$
Insert Taylor expansion...

$$
y^{(1)}(t) - D_\text{cd} y(t) = \frac{1}{24} h^2 y^{(3)}(t) + \mathcal{O}(h^4)
$$

The error is **second order in $h$**.

This is generally *much better* than the forward difference.

"Generally" means "when $y(t)$ is sufficiently smooth", i.e.
\begin{align}
y^{(3)} \frac{h^2}{24} &\ll y^{(2)} \frac{h}{2} \quad\text{or}\\
12\frac{y^{(2)}}{y^{(3)}} &\gg h.
\end{align}

Other difference algorithms exist with even better error behavior (see homework) but all have problems with data that are not smooth. In this case you might have to smooth the data in advance by *interpolation* or *kernel density estimates* (KDE).

## Error assessment
$$
\epsilon_\text{tot} = \epsilon_\text{app} + \epsilon_\text{ro}
$$

**Round-off error of differentiation**: difference of two similar numbers cannot be better than machine precision, always divided by small number $h$:

$$
\epsilon_\text{ro} \approx \frac{\epsilon_m}{h}
$$

**Forward difference algorithmic error**
$$
\epsilon_\text{app}^\text{fd} = y^{(2)} \frac{h}{2}
$$

What is the *most accurate value of $h$* that we can choose for $D_\text{fd}$?

Cross-over from approximation error to round-off error around

$$
\epsilon_\text{app} = \epsilon_\text{ro}
$$

For the forward difference
$$
y^{(2)} \frac{h_\text{fd}}{2} = \frac{\epsilon_m}{h_\text{fd}}\\
h_\text{fd}^2 = \frac{2\epsilon_m}{y^{(2)}}
$$

For rough estimate, assume $y^{(1)} \approx y^{(2)} \approx y^{(3)} \approx 1$:

```python
import numpy as np
def h_fd(eps_m, y2=1):
    return np.sqrt(2*eps_m/y2)
def eps_fd(eps_m, y2=1.):
    h = h_fd(eps_m, y2=y2)
    return 2*eps_m/h
```

```python
eps_m = 1e-15
print("h_fd = {0:.2e}".format(h_fd(eps_m, y2=1)))
print("eps_fd = {0:.2e}".format(eps_fd(eps_m, y2=1)))
```

    h_fd = 4.47e-08
    eps_fd = 4.47e-08

For double precision ($\epsilon_m = 10^{-15}$)
$$
h_\text{fd} \approx 4\times 10^{-8}\\
\epsilon_\text{tot}^\text{fd} \approx 2\epsilon_\text{ro} = \frac{2\epsilon_m}{h_\text{fd}} =  h_\text{fd} = 4 \times 10^{-8}
$$

(Note: simplified because $y^{(2)} \approx 1$.)

Do you expect the "best" $h_\text{cd}$ for the *central difference algorithm* to be larger or smaller than $h_\text{fd}$?

$$
h_\text{cd}^3 = \frac{24\epsilon_m}{y^{(3)}}
$$

Generally $h_\text{cd} > h_\text{fd}$ because the better algorithm allows you to use a bigger step size for a smaller error.

# Student Problem: Finite Difference Operators (Part 2)
- implement the three finite difference operators as functions
- test implementations
- `git pull` the course resources and work with the notebook **[09_differentiation/09-differentiation-students.ipynb](http://nbviewer.jupyter.org/github/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/09_differentiation/09-differentiation-students.ipynb)**.

## Implementation in Python

\begin{align}
D_\text{fd} y(t) &\equiv \frac{y(t+h) - y(t)}{h} \\
D_\text{cd} y(t) &\equiv \frac{y\Big(t + \frac{h}{2}\Big) - y\Big(t - \frac{h}{2}\Big)}{h}
\end{align}

and also the *extended difference algorithm*

\begin{align}
D_\text{ed} y(t) &\equiv \frac{4 D_\text{cd}y(t, h/2) - D_\text{cd}y(t, h)}{3} \\
  &= \frac{8\big(y(t+h/4) - y(t-h/4)\big) - \big(y(t+h/2) - y(t-h/2)\big)}{3h}
\end{align}

```python
def D_fd(y, t, h):
    """Forward difference"""
    return (y(t + h) - y(t))/h

def D_cd(y, t, h):
    """Central difference"""
    # implement

def D_ed(y, t, h):
    """Extended difference"""
    # implement
```

### Test your implementations
Test function: $y(t) = \cos t$
1. What is the analytical derivative $\frac{d\cos(t)}{dt}$?
1. Calculate the derivative of $y(t) = \cos t$ at $t=0.1, 1, 100$.
1. Print derivative and relative error $E = \frac{D y(t) - y^{(1)}(t)}{y^{(1)}(t)}$ (finite difference value $D y(t)$ compared to the analystical value $y^{(1)}(t)$– use numpy functions for "exact" values) as function of $h$.
1. Reduce $h$ until you reach machine precision, $h \approx \epsilon_m$
1. Plot $\log_{10} |E(h)|$ against $\log_{10} h$.

Try to do the above for all three algorithms

#### Function definitions

```python
def D_fd(y, t, h):
    """Forward difference"""
    return (y(t + h) - y(t))/h

def D_cd(y, t, h):
    """Central difference"""
    return (y(t + h/2) - y(t - h/2))/h

def D_ed(y, t, h):
    """Extended difference"""
    return (8*(y(t + h/4) - y(t - h/4)) - (y(t + h/2) - y(t - h/2)))/(3*h)
```

```python
import numpy as np
# test function: np.cos

# Analytical derivative
def y1(t):
    return -np.sin(t)

t_values = np.array([0.1, 1, 100], dtype=np.float64)
```

Use numpy functions for everything because then you can operate on all `t_values` at once.

#### Evaluate the finite difference derivatives
Note that we pass *a function* `y` to the forward difference function `D_fd` and we can also pass a whole array of `t_values`!

```python
D_fd(np.cos, t_values, 0.1)
```

    array([-0.14937587, -0.86706184,  0.46244209])

```python
D_fd(np.cos, t_values, 1e-12)
```

    array([-0.09980905, -0.84154905,  0.50359716])

#### Evaluate the exact derivatives
Compute the exact derivatives (again, operate on all $t$ together... start thinking in numpy arrays!)

```python
y1(t_values)
```

    array([-0.09983342, -0.84147098,  0.50636564])

Calculation of the **absolute error**: subtract the two arrays that you got previously:

```python
D_fd(np.cos, t_values, 0.1) - y1(t_values)
```

    array([-0.04954246, -0.02559086, -0.04392355])

#### Calculate the relative error $E$

```python
def error(Dxx, y, y1, t, h):
    """Relative error.

    Note: will fail for exact 0.
    """
    y1_val = y1(t)
    # return (Dxx(y, t, h) - y1_val/y1_val
    return Dxx(y, t, h)/y1_val - 1
```

Note that we pass again a general function for the difference operator so that we can use `error()` with `D_fd()`, `D_cd()` and `D_ep()`.

```python
error(D_fd, np.cos, y1, t_values, 0.1)
```

    array([ 0.49625125,  0.03041205, -0.08674275])

```python
error(D_fd, np.cos, y1, t_values, 1e-12)
```

    array([ -2.44073917e-04,   9.27754603e-05,  -5.46734793e-03])

#### Plot $|E|$
Plot $\log_{10} |E(h)|$ against $\log_{10} h$.

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')
```

```python
h_values = 10**(np.arange(-15, -1, 0.1))
abs_errors = np.abs(error(D_fd, np.cos, y1, 0.1, h_values))
```

```python
plt.loglog(h_values, abs_errors, label="t=0.1")
```

##### Forward Difference $D_\text{fd}$
Plot the three different $t$ values together in one plot:

```python
for t in t_values:
    abs_errors = np.abs(error(D_fd, np.cos, y1, t, h_values))
    plt.loglog(h_values, abs_errors, label=r"$t={}$".format(t))
ax = plt.gca()
ax.legend(loc="best")
ax.set_xlabel(r"$h$")
ax.set_ylabel(r"$|E_\mathrm{fd}|$");
```

* error behavior depends on $t$ and on cancellation of errors (e.g. for $t=1$
* algorithmic error decreases for decreasing $h$ until the round of error starts dominating

##### Central Difference $D_\text{cd}$
Plot the three different $t$ values together in one plot:

```python
for t in t_values:
    abs_errors = np.abs(error(D_cd, np.cos, y1, t, h_values))
    plt.loglog(h_values, abs_errors, label=r"$t={}$".format(t))
ax = plt.gca()
ax.legend(loc="best")
ax.set_xlabel(r"$h$")
ax.set_ylabel(r"$|E_\mathrm{cd}|$");
```

##### Extended Difference $D_\text{ed}$
Plot the three different $t$ values together in one plot:

```python
for t in t_values:
    abs_errors = np.abs(error(D_ed, np.cos, y1, t, h_values))
    plt.loglog(h_values, abs_errors, label=r"$t={}$".format(t))
ax = plt.gca()
ax.legend(loc="best")
ax.set_xlabel(r"$h$")
ax.set_ylabel(r"$|E_\mathrm{ed}|$");
```

```python

```