 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function
from __future__ import absolute_import

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Sources of Error

Error can come from many sources when using applying a numerical method:
 - Model/Data Error
 - Truncation Error
 - Floating Point Error

**Goal:** Categorize and understand each type of error and explore some simple approaches to analyzing error.

## Model and Data Error

Errors in fundamental formulation
 - Lotka-Volterra - fractional rabbits, no extinctions, etc.
 - Data Error - Inaccuracy in measurement or uncertainties in parameters

Unfortunatley we cannot control model and data error directly but we can use methods that may be more robust in the presense of these types of errors.

## Truncation Error

Errors arising from approximating a function with a simpler function (e.g. $sin(x) \approx x$ for $|x| \approx 0$.

## Floating Point Error

Errors arising from approximating real numbers with finite-precision numbers and arithmetic.

## Basic Definitions

Given a true value of a function $f$ and an approximate solution $F$ define:

Absolute Error:
$$
    e = | f - F |
$$

Relative Error:
$$
    r = \frac{e}{|f|} = \frac{|f - F|}{|f|}
$$

Decimal precision $p$ is defined as the minimum value that satisfies

$$x = \text{round}(10^{-n} \cdot x) \cdot 10^n$$

where

$$n = \text{floor}(\log_{10} x) + 1 - p$$

Note that if we are asking the decimal precision of the approximation $F$ of $f$ then we need to use the absolute error to determine the precision.  To find the decimal precision in this case look at the magnitude of the absolute error and deterimine the place of the first error.  Combine this with the number of "correct" digits and you will get the decimal precision of the approximation.

## Truncation Error and Taylor's Theorem

**Taylor's Theorem:**  Let $f(x) \in C^{m+1}[a,b]$ and $x_0 \in [a,b]$, then for all $x \in (a,b)$ there exists a number $c = c(x)$ that lies between $x_0$ and $x$ such that

$$ f(x) = T_N(x) + R_N(x)$$

where $T_N(x)$ is the Taylor polynomial approximation

$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot(x-x_0)^n}{n!}$$

and $R_N(x)$ is the residual (the part of the series we left off)

$$R_N(x) = \frac{f^{(n+1)}(c) \cdot (x - x_0)^{n+1}}{(n+1)!}$$

Another way to think about these results involves replacing $x - x_0$ with $\Delta x$.  The primary idea here is that the residual $R_N(x)$ becomes smaller as $\Delta x \rightarrow 0$.

$$
    T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot\Delta x^n}{n!}
$$

and $R_N(x)$ is the residual (the part of the series we left off)

$$
    R_N(x) = \frac{f^{(n+1)}(c) \cdot \Delta x^{n+1}}{(n+1)!} \leq M \Delta x^{n+1}
$$

#### Example 1

$f(x) = e^x$ with $x_0 = 0$

Using this we can find expressions for the relative and absolute error as a function of $x$ assuming $N=2$.

Derivatives:
$$\begin{aligned}
    f'(x) &= e^x \\
    f''(x) &= e^x \\
    f^{(n)}(x) &= e^x
\end{aligned}$$

Taylor polynomials:
$$\begin{aligned}
    T_N(x) &= \sum^N_{n=0} e^0 \frac{x^n}{n!} \Rightarrow \\
    T_2(x) &= 1 + x + \frac{x^2}{2}
\end{aligned}$$

Remainders:
$$\begin{aligned}
    R_N(x) &= e^c \frac{x^{n+1}}{(n+1)!} = e^c \cdot \frac{x^3}{6} \quad \Rightarrow \\
    R_2(x) &\leq \frac{e^1}{6} \approx 0.5
\end{aligned}$$

Accuracy:
$$
    e^1 = 2.718\ldots \\
    T_2(1) = 2.5 \Rightarrow e \approx 0.2 ~~ r \approx 0.1
$$

We can also use the package sympy which has the ability to calculate Taylor polynomials built-in!

```python
import sympy
x = sympy.symbols('x')
f = sympy.symbols('f', cls=sympy.Function)

f = sympy.exp(x)
f.series(x0=0, n=6)
```

Lets plot this numerically for a section of $x$.

```python
x = numpy.linspace(-1, 1, 100)
T_N = 1.0 + x + x**2 / 2.0
R_N = numpy.exp(1) * x**3 / 6.0

plt.plot(x, T_N, 'r', x, numpy.exp(x), 'k', x, R_N, 'b')
plt.plot(0.0, 1.0, 'o', markersize=10)
plt.grid(True)
plt.xlabel("x")
plt.ylabel("$f(x)$, $T_N(x)$, $R_N(x)$")
plt.legend(["$T_N(x)$", "$f(x)$", "$R_N(x)$"], loc=2)
plt.show()
```

#### Example 2

Approximate
$$
    f(x) = \frac{1}{x} \quad x_0  = 1,
$$
using $x_0 = 1$ to the 3rd Taylor series term.

$$\begin{aligned}
    f'(x) &= -\frac{1}{x^2} \\
    f''(x) &= \frac{2}{x^3} \\
    f^{(n)}(x) &= \frac{(-1)^n n!}{x^{n+1}}
\end{aligned}$$

$$\begin{aligned}
    T_N(x) &= \sum^N_{n=0} (-1)^n (x-1)^n \Rightarrow \\
    T_2(x) &= 1 - (x - 1) + (x - 1)^2
\end{aligned}$$

$$\begin{aligned}
    R_N(x) &= \frac{(-1)^{n+1}(x - 1)^{n+1}}{c^{n+2}} \Rightarrow \\
    R_2(x) &= \frac{-(x - 1)^{3}}{c^{4}}
\end{aligned}$$

```python
x = numpy.linspace(0.8, 2, 100)
T_N = 1.0 - (x-1) + (x-1)**2
R_N = -(x-1.0)**3 / (1.1**4)

plt.plot(x, T_N, 'r', x, 1.0 / x, 'k', x, R_N, 'b')
plt.plot(1.0, 1.0, 'o', markersize=10)
plt.grid(True)
plt.xlabel("x")
plt.ylabel("$f(x)$, $T_N(x)$, $R_N(x)$")

plt.legend(["$T_N(x)$", "$f(x)$", "$R_N(x)$"], loc=8)
plt.show()
```

## Big-O Notation

Big-O notation:
$$
    f(x) = \mathcal{O}(g(x)) \quad \text{as} \quad x \rightarrow a
$$
if and only if
$$
    |f(x)| \leq M |g(x)| \quad \text{as}\quad  |x - a| < \delta \quad \text{where} \quad M,a > 0.
$$

In practice we use Big-O notation to say something about how the terms we may have left out of a series might behave.  We saw an example earlier of this with the Taylor's series approximations:

#### Example:
$f(x) = \sin x$ with $x_0 = 0$ then

$$T_N(x) = \sum^N_{n=0} (-1)^{n} \frac{x^{2n+1}}{(2n+1)!}$$

We can actually write $f(x)$ then as

$$f(x) = x - \frac{x^3}{6} + \frac{x^5}{120} + \mathcal{O}(x^7)$$

This becomes more useful when we look at this as we did before with $\Delta x$:

$$f(x) = \Delta x - \frac{\Delta x^3}{6} + \frac{\Delta x^5}{120} + \mathcal{O}(\Delta x^7)$$

**We can also develop rules for error propagation based on Big-O notation:**

In general, there are two theorems that do not need proof and hold when the value of x is large:

Let
$$\begin{aligned}
    f(x) &= p(x) + \mathcal{O}(x^n) \\
    g(x) &= q(x) + \mathcal{O}(x^m) \\
    k &= \max(n, m)
\end{aligned}$$
then
$$
    f+g = p + q + \mathcal{O}(x^k)
$$
and
\begin{align}
    f \cdot g &= p \cdot q + p \mathcal{O}(x^m) + q \mathcal{O}(x^n) + O(x^{n + m}) \\
    &= p \cdot q + \mathcal{O}(x^{n+m})
\end{align}

On the other hand, if we are interested in small values of x, say $\Delta x$, the above expressions can be modified as follows:

\begin{align}
    f(\Delta x) &= p(\Delta x) + \mathcal{O}(\Delta x^n) \\
    g(\Delta x) &= q(\Delta x) + \mathcal{O}(\Delta x^m) \\
    r &= \min(n, m)
\end{align}
then
$$
    f+g = p + q + O(\Delta x^r)
$$
and
\begin{align}
    f \cdot g &= p \cdot q + p \cdot \mathcal{O}(\Delta x^m) + q \cdot \mathcal{O}(\Delta x^n) + \mathcal{O}(\Delta x^{n+m}) \\
    &= p \cdot q + \mathcal{O}(\Delta x^r)
\end{align}

**Note 1:** In this case we suppose that at least the polynomial with $k = \max(n, m)$ has the following form:

$$
    p(\Delta x) = 1 + p_1 \Delta x + p_2 \Delta x^2 + \ldots
$$
or
$$
    q(\Delta x) = 1 + q_1 \Delta x + q_2 \Delta x^2 + \ldots
$$

so that there is an $\mathcal{O}(1)$ term that guarantees the existence of $\mathcal{O}(\Delta x^r)$ in the final product.

To get a sense of why we care most about the power on $\Delta x$ when considering convergence the following figure shows how different powers on the convergence rate can effect how quickly we converge to our solution.  Note that here we are plotting the same data two different ways.  Plotting the error as a function of $\Delta x$ is a common way to show that a numerical method is doing what we expect and exhibits the correct convergence behavior.  Since errors can get small quickly it is very common to plot these sorts of plots on a log-log scale to easily visualize the results.  Note that if a method was truly of the order $n$ that they will be a linear function in log-log space with slope $n$.

```python
dx = numpy.linspace(1.0, 1e-4, 100)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2.0)
axes = []
axes.append(fig.add_subplot(1, 2, 1))
axes.append(fig.add_subplot(1, 2, 2))

for n in range(1, 5):
    axes[0].plot(dx, dx**n, label="$\Delta x^%s$" % n)
    axes[1].loglog(dx, dx**n, label="$\Delta x^%s$" % n)

axes[0].legend(loc=2)
axes[1].set_xticks([10.0**(-n) for n in range(5)])
axes[1].set_yticks([10.0**(-n) for n in range(16)])
axes[1].legend(loc=4)
for n in range(2):
    axes[n].set_title("Growth of Error vs. $\Delta x^n$")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Estimated Error")
    axes[n].set_title("Growth of different")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Estimated Error")

plt.show()
```

## Operation Counting

### Example 1: Matrix-Vector Multiplication

Let $A, B \in \mathbb{R}^{N \times N}$ and $x \in \mathbb{R}^N$.

1. Count the approximate number of operations it will take to compute $A x$.
2. Do the same for $A B$.

Matrix-vector product:  Defining $[A]_i$ as the $i$th row of $A$ and $A_{ij}$ as the $i$, $j$th entry then
$$
    A x = \sum^N_{i=1} [A]_i \cdot x = \sum^N_{i=1} \sum^N_{j=1} A_{ij} x_j
$$

Take a explicit case, say $N = 3$, then the operation count is
$$
    A x = [A]_1 \cdot v + [A]_2 \cdot v + [A]_3 \cdot v = \begin{bmatrix}
        A_{11} \times v_1 + A_{12} \times v_2 + A_{13} \times v_3 \\
        A_{21} \times v_1 + A_{22} \times v_2 + A_{23} \times v_3 \\
        A_{31} \times v_1 + A_{32} \times v_2 + A_{33} \times v_3
    \end{bmatrix}
$$

This leads to 15 operations (6 additions and 9 multiplications).

Take another case, say $N = 4$, then the operation count is
$$
    A x = [A]_1 \cdot v + [A]_2 \cdot v + [A]_3 \cdot v = \begin{bmatrix}
        A_{11} \times v_1 + A_{12} \times v_2 + A_{13} \times v_3 + A_{14} \times v_4 \\
        A_{21} \times v_1 + A_{22} \times v_2 + A_{23} \times v_3 + A_{24} \times v_4 \\
        A_{31} \times v_1 + A_{32} \times v_2 + A_{33} \times v_3 + A_{34} \times v_4 \\
        A_{41} \times v_1 + A_{42} \times v_2 + A_{43} \times v_3 + A_{44} \times v_4 \\
    \end{bmatrix}
$$

This leads to 28 operations (12 additions and 16 multiplications).

Generalizing this there are $N^2$ multiplications and $N (N -1)$ additions for a total of

$$
    \text{operations} = N (N - 1) + N^2 = \mathcal{O}(N^2).
$$

Matrix-Matrix product ($AB$):  Defining $[B]_j$ as the $j$th column of $B$ then
$$
    (A B)_{ij} = \sum^N_{i=1} \sum^N_{j=1} [A]_i \cdot [B]_j
$$
The inner product of two vectors is represented by
$$
    a \cdot b = \sum^N_{i=1} a_i b_i
$$
leading to $\mathcal{O}(3N)$ operations.  Since there are $N^2$ entries in the resulting matrix then we would have $\mathcal{O}(N^3)$ operations.

There are methods for performing matrix-matrix multiplication faster.  In the following figure we see a collection of algorithms over time that have been able to bound the number of operations in certain circumstances.  Here
$$
    \mathcal{O}(N^\omega)
$$

### Example 2:  Horner's Method for Evaluating Polynomials

Given

$$P_N(x) = a_0 + a_1 x + a_2 x^2 + \ldots + a_N x^N$$

or

$$P_N(x) = p_1 x^N + p_2 x^{N-1} + p_3 x^{N-2} + \ldots + p_{N+1}$$

want to find best way to evaluate $P_N(x)$.

First consider two ways to write $P_3$:

$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$

and using nested multiplication:

$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$

Consider how many operations it takes for each...

$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$

$$P_3(x) = \overbrace{p_1 \cdot x \cdot x \cdot x}^3 + \overbrace{p_2 \cdot x \cdot x}^2 + \overbrace{p_3 \cdot x}^1 + p_4$$

Adding up all the operations we can in general think of this as a pyramid

We can estimate this way that the algorithm written this way will take approximately $O(N^2 / 2)$ operations to complete.

Looking at our other means of evaluation:

$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$

Here we find that the method is $O(N)$ (the 2 is usually ignored in these cases).  The important thing is that the first evaluation is $O(N^2)$ and the second $O(N)$!

#### Algorithm

Fill in the function and implement Horner's method:
```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """
    pass
```

```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """
    ### ADD CODE HERE
    pass
```

```python
# Scalar version
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """

    y = p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y

# Vectorized version
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x can by a NumPy ndarray.
    """

    y = numpy.ones(x.shape) * p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y

p = [1, -3, 10, 4, 5, 5]
x = numpy.linspace(-10, 10, 100)
plt.plot(x, eval_poly(p, x))
plt.show()
```

## Truncation Error vs. Floating Point Error

Truncation error:  Errors arising from approximation of a function, truncation of a series...

$$\sin x \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} + O(x^7)$$

Floating-point Error:  Errors arising from approximating real numbers with finite-precision numbers

$$\pi \approx 3.14$$

or $\frac{1}{3} \approx 0.333333333$ in decimal, results form finitely number of registers to represent each number.

### Floating Point Systems

Numbers in floating point systems are represented as a series of bits that represent different pieces of a number.  In *normalized floating point systems* there are some standard conventions for what these bits are used for.  In general the numbers are stored by breaking them down into the form

$$F = \pm d_1 . d_2 d_3 d_4 \ldots d_p \times \beta^E$$

$$F = \pm d_1 . d_2 d_3 d_4 \ldots d_p \times \beta^E$$

where
1. $\pm$ is a single bit and of course represents the sign of the number
2. $d_1 . d_2 d_3 d_4 \ldots d_p$ is called the *mantissa*.  Note that technically the decimal could be moved but generally, using scientific notation, the decimal can always be placed at this location.  The digits $d_2 d_3 d_4 \ldots d_p$ are called the *fraction* with $p$ digits of precision.  Normalized systems specifically put the decimal point in the front like we have and assume $d_1 \neq 0$ unless the number is exactly $0$.
3. $\beta$ is the *base*.  For binary $\beta = 2$, for decimal $\beta = 10$, etc.
4. $E$ is the *exponent*, an integer in the range $[E_{\min}, E_{\max}]$

The important points on any floating point system is that
1. There exist a discrete and finite set of representable numbers
2. These representable numbers are not evenly distributed on the real line
3. Arithmetic in floating point systems yield different results from infinite precision arithmetic (i.e. "real" math)

#### Properties of Floating Point Systems
All floating-point systems are characterized by several important numbers
 - Smalled normalized number (underflow if below - related to subnormal numbers around zero)
 - Largest normalized number (overflow if above)
 - Zero
 - Machine $\epsilon$ or $\epsilon_{\text{machine}}$
 - `inf` and `nan`, infinity and **N**ot **a** **N**umber respectively

##### Example:  Toy System
Consider the toy 2-digit precision decimal system (normalized)
$$f = \pm d_1 . d_2 \times 10^E$$
with $E \in [-2, 0]$.

**Number and distribution of numbers**
1. How many numbers can we represent with this system?

2. What is the distribution on the real line?

3. What is the underflow and overflow limits?

How many numbers can we represent with this system?

$$
    f = \pm d_1 . d_2 \times 10^E ~~~ \text{with} E \in [-2, 0]
$$

$$
    2 \times 9 \times 10 \times 3 + 1 = 541
$$

What is the distribution on the real line?

```python
d_1_values = [1, 2, 3, 4, 5, 6, 7, 8, 9]
d_2_values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
E_values = [0, -1, -2]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, '+', markersize=20)
axes.plot([-10.0, 10.0], [0.0, 0.0], 'k')

axes.set_title("Distribution of Values")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-0.1, 0.1])
plt.show()
```

What is the underflow and overflow limits?

Smallest number that can be represented is the underflow:  $1.0 \times 10^{-2} = 0.01$
Largest number that can be represented is the overflow:  $9.9 \times 10^0 = 9.9$

### Binary Systems
Consider the 2-digit precision base 2 system:

$$
    f=\pm d_1 . d_2 \times 2^E \quad \text{with} \quad E \in [-1, 1]
$$

#### Number and distribution of numbers
1. How many numbers can we represent with this system?

2. What is the distribution on the real line?

3. What is the underflow and overflow limits?

How many numbers can we represent with this system?

$$f=\pm d_1 . d_2 \times 2^E ~~~~ \text{with} ~~~~ E \in [-1, 1]$$

$$ 2 \times 1 \times 2 \times 3 + 1 = 13$$

What is the distribution on the real line?

```python
d_1_values = [1]
d_2_values = [0, 1]
E_values = [1, 0, -1]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.5) * 2**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.5) * 2**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, 'r+', markersize=20)
axes.plot([-4.5, 4.5], [0.0, 0.0], 'k')

axes.set_title("Distribution of Values")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-3.5, 3.5])
plt.show()
```

Smallest number that can be represented is the underflow:  $1.0 \times 2^{-1} = 0.5$
Largest number that can be represented is the overflow:  $1.1 \times 2^1 = 3$

### Real Systems - IEEE 754 Binary Floating Point Systems

#### Single Precision
 - Total storage alloted is 32 bits
 - Exponent is 8 bits $\Rightarrow E \in [-126, 127]$
 - Fraction 23 bits ($p = 24$)

```
s EEEEEEEE FFFFFFFFFFFFFFFFFFFFFFF
0 1      8 9                     31
```
Overflow $= 2^{127} \approx 3.4 \times 10^{38}$

Underflow $= 2^{-126} \approx 1.2 \times 10^{-38}$

$\epsilon_{\text{machine}} = 2^{-23} \approx 1.2 \times 10^{-7}$

#### Double Precision
 - Total storage alloted is 64 bits
 - Exponent is 11 bits $\Rightarrow E \in [-1022, 1024]$
 - Fraction 52 bits ($p = 53$)

```
s EEEEEEEEEE FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FF
0 1       11 12                                                      63
```
Overflow $= 2^{1024} \approx 1.8 \times 10^{308}$

Underflow $= 2^{-1022} \approx 2.2 \times 10^{-308}$

$\epsilon_{\text{machine}} = 2^{-52} \approx 2.2 \times 10^{-16}$

### Python Access to IEEE Numbers

Access many important parameters, such as machine epsilon:

```python
import numpy
numpy.finfo(float).eps
```

```python
print(numpy.finfo(numpy.float16))
print(numpy.finfo(numpy.float32))
print(numpy.finfo(float))
print(numpy.finfo(numpy.float128))
```

## Why should we care about this?

 - Floating point arithmetic is not commutative or associative
 - Floating point errors compound, do not assume even double precision is enough!
 - Mixing precision is very dangerous

### Example 1: Simple Arithmetic

Simple arithmetic $\delta < \epsilon_{\text{machine}}$

   $$(1+\delta) - 1 = 1 - 1 = 0$$

   $$1 - 1 + \delta = \delta$$

### Example 2: Catastrophic cancellation

Let us examine what happens when we add two numbers $x$ and $y$ where $x + y \neq 0$.  We can actually estimate these bounds by doing some error analysis.  Here we need to introduce the idea that each floating point operation introduces an error such that

$$
    \text{fl}(x ~\text{op}~ y) = (x ~\text{op}~ y) (1 + \delta)
$$

where $\text{fl}(\cdot)$ is a function that returns the floating point representation of the expression enclosed, $\text{op}$ is some operation (e.g. $+, -, \times, /$), and $\delta$ is the floating point error due to $\text{op}$.

Back to our problem at hand.  The floating point error due to addition is

$$
    \text{fl}(x + y) = (x + y) (1 + \delta).
$$

Comparing this to the true solution using a relative error we have

$$\begin{aligned}
    \frac{(x + y) - \text{fl}(x + y)}{x + y} &= \frac{(x + y) - (x + y) (1 + \delta)}{x + y} = \delta.
\end{aligned}$$

so that if $\delta = \mathcal{O}(\epsilon_{\text{machine}})$ we are not too concerned.

What if instead we consider a floating point error on the representations of $x$ and $y$, $x \neq y$, and say $\delta_x$ and $\delta_y$ are the magnitude of the errors in their representation.  Here we will assume this constitutes the floating point error rather than being associated with the operation itself.

Given all of this we then would have

$$\begin{aligned}
    \text{fl}(x + y) &= x (1 + \delta_x) + y (1 + \delta_y) \\
    &= x + y + x \delta_x + y \delta_y \\
    &= (x + y) \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right)
\end{aligned}$$

Again computing the relative error we then have

$$\begin{aligned}
    \frac{x + y - (x + y) \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right)}{x + y} &= 1 - \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right) \\
    &= \frac{x}{x + y} \delta_x + \frac{y}{x + y} \delta_y \\
    &= \frac{1}{x + y} (x \delta_x + y \delta_y)
\end{aligned}$$

The important distinction here is that now the error is dependent on the values of $x$ and $y$ and more importantly, there sum.  Of particular concern is the relative size of $x + y$.  As it approaches zero relative to the magnitudes of $x$ and $y$ the error could be arbitrarily large.  This is known as **catastrophic cancellation**.

```python
dx = numpy.array([10**(-n) for n in range(1, 16)])
x = 1.0 + dx
y = -numpy.ones(x.shape)
error = numpy.abs(x + y - dx) / (dx)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1)
axes.loglog(dx, x + y, 'o-')
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$x + y$")
axes.set_title("$\Delta x$ vs. $x+y$")

axes = fig.add_subplot(1, 2, 2)
axes.loglog(dx, error, 'o-')
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$|x + y - \Delta x| / \Delta x$")
axes.set_title("Difference between $x$ and $y$ vs. Relative Error")

plt.show()
```

### Example 3: Function Evaluation

Consider the function
$$
    f(x) = \frac{1 - \cos x}{x^2}
$$
with $x\in[-10^{-4}, 10^{-4}]$.

Taking the limit as $x \rightarrow 0$ we can see what behavior we would expect to see from evaluating this function:
$$
    \lim_{x \rightarrow 0} \frac{1 - \cos x}{x^2} = \lim_{x \rightarrow 0} \frac{\sin x}{2 x} = \lim_{x \rightarrow 0} \frac{\cos x}{2} = \frac{1}{2}.
$$

What does floating point representation do?

```python
x = numpy.linspace(-1e-3, 1e-3, 100, dtype=numpy.float32)
error = (0.5 - (1.0 - numpy.cos(x)) / x**2) / 0.5

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error, 'o')
axes.set_xlabel("x")
axes.set_ylabel("Relative Error")
```

### Example 4: Evaluation of a Polynomial

   $$f(x) = x^7 - 7x^6 + 21 x^5 - 35 x^4 + 35x^3-21x^2 + 7x - 1$$

```python
x = numpy.linspace(0.988, 1.012, 1000, dtype=numpy.float16)
y = x**7 - 7.0 * x**6 + 21.0 * x**5 - 35.0 * x**4 + 35.0 * x**3 - 21.0 * x**2 + 7.0 * x - 1.0

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, y, 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
plt.show()
```

### Example 5: Rational Function Evaluation
Compute $f(x) = x + 1$ by the function $$F(x) = \frac{x^2 - 1}{x - 1}$$.

Do you expect there to be issues?

```python
x = numpy.linspace(0.5, 1.5, 101, dtype=numpy.float16)
f_hat = (x**2 - 1.0) / (x - 1.0)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.abs(f_hat - (x + 1.0)))
axes.set_xlabel("$x$")
axes.set_ylabel("Absolute Error")
plt.show()
```

## Combination of Error

In general we need to concern ourselves with the combination of both truncation error and floating point error.

### Example 1

Consider the finite difference approximation where $f(x) = e^x$ and we are evaluating at $x=1$?

$$f'(x) \approx \frac{f(x + \Delta x) - f(x)}{\Delta x}$$

Compare the error between decreasing $\Delta x$ and the true solution $f'(1) = e$.

```python
delta_x = numpy.linspace(1e-20, 5.0, 100)
delta_x = numpy.array([2.0**(-n) for n in range(1, 60)])
x = 1.0
f_hat_1 = (numpy.exp(x + delta_x) - numpy.exp(x)) / (delta_x)
f_hat_2 = (numpy.exp(x + delta_x) - numpy.exp(x - delta_x)) / (2.0 * delta_x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.loglog(delta_x, numpy.abs(f_hat_1 - numpy.exp(1)), 'o-', label="One-Sided")
axes.loglog(delta_x, numpy.abs(f_hat_2 - numpy.exp(1)), 's-', label="Centered")
axes.legend(loc=3)
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("Absolute Error")
plt.show()
```

### Example 2

Evaluate $e^x$ with its Taylor series.

$$e^x = \sum^\infty_{n=0} \frac{x^n}{n!}$$

Can we pick $N < \infty$ that can approximate $e^x$ over a give range $x \in [a,b]$ such that the relative error $E$ satisfies $E < 8 \cdot \varepsilon_{\text{machine}}$?

What might be a better way than simply evaluating the Taylor polynomial directly for various $N$?

```python
import scipy.misc

def my_exp(x, N=10):
    value = 0.0
    for n in range(N + 1):
        value += x**n / scipy.misc.factorial(n)

    return value

x = numpy.linspace(-2, 2, 100, dtype=numpy.float32)
for N in range(1, 50):
    error = numpy.abs((numpy.exp(x) - my_exp(x, N=N)) / numpy.exp(x))
    if numpy.all(error < 8.0 * numpy.finfo(float).eps):
        break

print(N)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error)
axes.set_xlabel("x")
axes.set_ylabel("Relative Error")
plt.show()
```

### Example 3 - Relative Error

Say we wanted to compute the relative error between two values $x$ and $y$ using $x$ as the normalizing value.  Algebraically the forms
$$
    E = \frac{x - y}{x}
$$
and
$$
    E = 1 - \frac{y}{x}
$$
are equivalent.  In finite precision what form might be expected to be more accurate and why?

*Example based on a [blog](https://nickhigham.wordpress.com/2017/08/14/how-and-how-not-to-compute-a-relative-error/) post by Nick Higham*

Using this model the original definition contains two floating point operations such that
$$\begin{aligned}
    E_1 = \text{fl}\left(\frac{x - y}{x}\right) &= \text{fl}(\text{fl}(x - y) / x) \\
    &= \left[ \frac{(x - y) (1 + \delta_+)}{x} \right ] (1 + \delta_/) \\
    &= \frac{x - y}{x}  (1 + \delta_+) (1 + \delta_/)
\end{aligned}$$

For the other formulation we have
$$\begin{aligned}
    E_2 = \text{fl}\left( 1 - \frac{y}{x} \right ) &= \text{fl}\left(1 - \text{fl}\left(\frac{y}{x}\right) \right) \\
    &= \left(1 - \frac{y}{x} (1 + \delta_/) \right) (1 + \delta_-)
\end{aligned}$$

If we assume that all $\text{op}$s have similar error magnitudes then we can simplify things by letting
$$
    |\delta_\ast| \le \epsilon.
$$

To compare the two formulations we again use the relative error between the true relative error $e_i$ and our computed versions $E_i$.

Original definition:
$$\begin{aligned}
    \frac{e - E_1}{e} &= \frac{\frac{x - y}{x} - \frac{x - y}{x}  (1 + \delta_+) (1 + \delta_/)}{\frac{x - y}{x}} \\
    &\le 1 - (1 + \epsilon) (1 + \epsilon) = 2 \epsilon + \epsilon^2
\end{aligned}$$

Manipulated definition:

$$\begin{aligned}
    \frac{e - E_2}{e} &= \frac{e - \left[1 - \frac{y}{x}(1 + \delta_/) \right] (1 + \delta_-)}{e} \\
    &= \frac{e - \left[e - \frac{y}{x} \delta_/) \right] (1 + \delta_-)}{e} \\
    &= \frac{e - \left[e + e\delta_- - \frac{y}{x} \delta_/ - \frac{y}{x} \delta_/ \delta_-)) \right] }{e} \\
    &= - \delta_- + \frac{1}{e} \frac{y}{x} \left(\delta_/ + \delta_/ \delta_- \right) \\
    &= - \delta_- + \frac{1 -e}{e} \left(\delta_/ + \delta_/ \delta_- \right) \\
    &\le \epsilon + \left |\frac{1 - e}{e}\right | (\epsilon + \epsilon^2)
\end{aligned}$$

We see then that our floating point error will be dependent on the relative magnitude of $e$

```python
# Based on the code by Nick Higham
# https://gist.github.com/higham/6f2ce1cdde0aae83697bca8577d22a6e
# Compares relative error formulations using single precision and compared to double precision

N = 501    # Note: Use 501 instead of 500 to avoid the zero value
d = numpy.finfo(numpy.float32).eps * 1e4
a = 3.0
x = a * numpy.ones(N, dtype=numpy.float32)
y = [x[i] + numpy.multiply((i - numpy.divide(N, 2.0, dtype=numpy.float32)), d, dtype=numpy.float32) for i in range(N)]

# Compute errors and "true" error
relative_error = numpy.empty((2, N), dtype=numpy.float32)
relative_error[0, :] = numpy.abs(x - y) / x
relative_error[1, :] = numpy.abs(1.0 - y / x)
exact = numpy.abs( (numpy.float64(x) - numpy.float64(y)) / numpy.float64(x))

# Compute differences between error calculations
error = numpy.empty((2, N))
for i in range(2):
    error[i, :] = numpy.abs((relative_error[i, :] - exact) / numpy.abs(exact))

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(y, error[0, :], '.', markersize=10, label="$|x-y|/|x|$")
axes.semilogy(y, error[1, :], '.', markersize=10, label="$|1-y/x|$")

axes.grid(True)
axes.set_xlabel("y")
axes.set_ylabel("Relative Error")
axes.set_xlim((numpy.min(y), numpy.max(y)))
axes.set_ylim((5e-9, numpy.max(error[1, :])))
axes.set_title("Relative Error Comparison")
axes.legend()
plt.show()
```

Some other links that might be helpful regarding IEEE Floating Point:
 - [What Every Computer Scientist Should Know About Floating-Point Arithmetic](http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)
 - [IEEE 754 Floating Point Calculator](http://babbage.cs.qc.edu/courses/cs341/IEEE-754.html)
 - [Numerical Computing with IEEE Floating Point Arithmetic](http://epubs.siam.org/doi/book/10.1137/1.9780898718072)


# Introduction to Numpy (tutorial)
This notebook is about Numpy, Python's core library for numeric computing. Knowing how to use Numpy is *essential* in domains such as machine learning, neuroimaging analysis, and image processing.

## Contents
1. The *nd*array
2. Indexing
3. Array math
4. Broadcasting

Let's start by importing numpy, which is commonly done as follows:

```python
import numpy as np
```

## The *ndarray*
The most important feature of Numpy is it's core data structure: the numpy *ndarray* (which stands for *n*-*d*imensional array, referring to the fact that the array may be of any dimension: 1D, 2D, 3D, 180D ... *n*D). This is, just like basic lists/dictionaries/integers/tuples, a data structure with its own syntax, operations, and methods, as we will explain below.

### Python lists vs. numpy arrays
Basically, numpy arrays are a lot like Python lists. The major difference, however, is that *numpy arrays may contain only a single data-type*, while Python lists may contain different data-types within the same list.

Let check this out:

```python
# Python lists may contain mixed data-types: an integer, a float, a string, a list
python_list = [1, 2.5, "whatever", [3, 4, 5]]

for value in python_list:

    print("%s is a: %s" % (str(value), type(value)))
```

Unlike Python lists, numpy only allows entries of the same data-type. This difference between Python lists and numpy arrays is basically the same as R lists (allow multiple data-types) versus R matrices/arrays (only allow one data type), and is also the same as MATLAB cells (allow multiple data-types) versus MATLAB matrices (only allow one data type).

In fact, if you try to make a numpy array with different data-types, numpy will force the entries into the same data-type (in a smart way), as is shown in the example below:

```python
# Importantly, you often specify your arrays as Python lists first, and then convert them to numpy
to_convert_to_numpy = [1, 2, 3.5]               # specify python list ...
numpy_array = np.array(to_convert_to_numpy)     # ... and convert ('cast') it to numpy

for entry in numpy_array:

    print(entry)
    print('this is a: %s \n' % type(entry))
```

As you can see, Numpy converted our original list (to_convert_to_numpy), which contained both integers and floats, to an array with only floats! You might think that such a data structure that only allows one single data type is not ideal. However, the very fact that it only contains a single data-type makes operations on numpy arrays extremely fast. For example, loops over numpy arrays are often way faster than loops over python lists. This is because, internally, Python has to check the data-type of each loop entry before doing something with that entry. Because numpy arrays one allow a single data-type, it only has to check for the entries' data type **once**. If you imagine looping over an array or list of length 100,000, you probably understand that the numpy loop is way faster.

Let's check out the speed difference between Python list operations and numpy array operations:

```python
# %timeit is a cool 'feature' that you can use in Notebooks (no need to understand how it works)
# it basically performs a computation that you specify a couple of times and prints how long it took on average
results_python = %timeit -o [x * 2 for x in range(0, 100000)]
```

And now let's do the same with numpy:

```python
results_numpy = %timeit -o np.arange(0, 10000) * 2

ratio = results_python.average / results_numpy.average
print("Numpy is %i times faster than base Python!" % ratio)
```

You see that Numpy *much* faster! This really matters when you start doing more complex operations or large arrays!

### Creating numpy arrays
As shown an earlier example, numpy arrays can be created as follows:

1. Define a Python list, e.g. `my_list = [0, 1, 2]`
2. Convert the list to a numpy array, e.g. `numpy_array = np.array(my_list)`

Importantly, a simple Python list will be converted to a 1D numpy array, but a nested Python list will be converted to a 2D (or even higher-dimensional array). Nesting is simply combining different lists, separated by commans, as is shown here:

```python
my_list = [1, 2, 3]
my_array = np.array(my_list)

print("A 1D (or 'flat') array:")
print(my_array, '\n')

my_nested_list = [[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]]

my_2D_array = np.array(my_nested_list)
print("A 2D array:")
print(my_2D_array)
```

    ToDo: Create the following 1D array:

\begin{align}
\begin{bmatrix}
5 & 3 & 2 & 8 & 9
\end{bmatrix}
\end{align}

and store it in a variable named <tt>vec</tt>.

```python
""" Implement the ToDo here. """
### BEGIN SOLUTION
vec = np.array([5, 3, 2, 8, 9])
### END SOLUTION
```

```python
""" Tests the above ToDo. """
if 'vec' not in dir():
    raise ValueError("You haven't created the variable 'vec' yet!")

if type(vec) != np.ndarray:
    raise ValueError("vec is not a numpy array!")

if vec.ndim != 1:
    raise ValueError("vec is not a 1D array!")

if vec.tolist() != [5, 3, 2, 8, 9]:
    raise ValueError("vec does not have the right contents ...")

print("Yay!")
```

    ToDo:

Create the following matrix (2D array):

\begin{align}
\begin{bmatrix}
5 & 2 & 8 \\
8 & 2 & 1 \\
4 & 4 & 4 \\
1 & 2 & 3
\end{bmatrix}
\end{align}

and store it in a variable named <tt>arr_2d</tt>. Hint: start by creating a nested python list (like we did with the <tt>my_nested_list</tt> variable) and then convert it to numpy.

```python
""" Implement the ToDo here. """
### BEGIN SOLUTION
arr_2d = np.array([
    [5, 2, 8],
    [8, 2, 1],
    [4, 4, 4],
    [1, 2, 3]
])
### END SOLUTION
```

```python
""" Tests the above ToDo. """
if type(arr_2d) != np.ndarray:
    raise ValueError("arr_2d is not a numpy array!")

if arr_2d.shape != (4, 3):
    raise ValueError("arr_2d does not have the right shape!")

print("Yes! Well done!")
```

    ToDo (optional): Let's try something more difficult! Create a 3D array with dimensions $2 \times 2 \times 2$ with consecutive numbers starting at 1 and store this in a variable named <tt>arr_3d</tt>. So <tt>arr_3d[0, 0, 0]</tt> should evaluate to 1, <tt>arr_3d[0, 0, 1]</tt> should evaluate to 2, <tt>arr_3d[0, 1, 0]</tt> evaluates to 3, etc.

```python
""" Implement the ToDo here. """
### BEGIN SOLUTION
arr_3d = np.array([
    [
        [1, 2],
        [3, 4]
    ],
    [
        [5, 6],
        [7, 8]
    ]
])
### END SOLUTION
```

```python
""" Tests the above ToDo. """
if arr_3d.shape != (2, 2, 2):
    raise ValueError("The array does not have the right dimensions!")

if arr_3d[0, 0, 0] != 1:
    raise ValueError("Hmm, something went wrong ...")

if arr_3d[0, 0, 1] != 2:
    raise ValueError("Hmm, something went wrong ...")

if arr_3d[1, 1, 1] != 8:
    raise ValueError("Hmm, something went wrong ...")
```

As you can imagine, creating numpy arrays from nested lists becomes cumbersome if you want to create (large) arrays with more than 2 dimensions. There are, fortunately, a lot of other ways to create ('initialize') large, high-dimensional numpy arrays. One often-used method is to create an array with zeros using the numpy function `np.zeros`. This function takes one (mandatory) argument, which is a tuple with the dimensions of your desired array:

```python
my_desired_dimensions = (2, 5) # suppose I want to create a matrix with zeros of size 2 by 5
my_array = np.zeros(my_desired_dimensions)

print(my_array)
```

    ToDo: Using the <tt>np.zeros</tt> function, create a five dimensional array of shape $2 \times 3 \times 5 \times 3 \times 7$ and store it in a variable named <tt>array_5d</tt>.

```python
""" Implement the ToDo here. """
### BEGIN SOLUTION
array_5d = np.zeros((2, 3, 5, 3, 7))
### END SOLUTION
```

```python
""" Tests the above ToDo. """
from tests import test_create_array_with_zeros
test_create_array_with_zeros(array_5d)
```

Using arrays with zeros is often used in what is called 'pre-allocation', in which you create an 'empty' array with only zeros and for example, 'fill' that array in a loop.

Below, we show you an example in which we pre-allocate an array with 5 zeros, and fill that in a for-loop with the squares of 1 - 5. Try to understand how it works! (You have to do something similar in the next ToDo!)

```python
my_array = np.zeros(5)

print('Original zeros-array')
print(my_array)

for i in range(5):  # notice the range function here! This loop now iterates over [0, 1, 2, 3, 4]
    number_to_calculate_the_square_of = i + 1
    my_array[i] = number_to_calculate_the_square_of ** 2

print('\nFilled array')
print(my_array)
```

ToDo: Write a loop in which you fill each value in the variable <tt>my_array2</tt> with the value 1 divided by the current index plus one. So, suppose the index (usually <tt>i</tt>, like the last example) is 3, then you should fill <tt>my_array2</tt> at index <tt>i</tt> with <tt>1 / (3 + 1)</tt>.

```python
""" Implement the ToDo here. """
my_array2 = np.zeros(8)

### BEGIN SOLUTION
for i in range(8):
    my_array2[i] = 1 / (i + 1)
### END SOLUTION
```

```python
""" Tests the above ToDo. """
from tests import test_fill_array_with_complement
test_fill_array_with_complement(my_array2)
```

In addition to `np.zeros`, you can create numpy arrays using other functions, like `np.ones` and `random` from the `np.random` module:

```python
ones = np.ones((5, 10)) # create an array with ones
print(ones, '\n')

rndom = np.random.random((5, 10)) # Create an array filled with random values (0 - 1 uniform)
print(rndom)
```

## Numpy indexing
Indexing (extracting a single value of an array) and slicing (extracting multiple values - a subset - from an array) of numpy arrays is largely the same as with regular Python lists and data structures from other scientific computing languages such as R and MATLAB. Let's check out a couple of examples of a 1D array:

```python
my_array = np.arange(10, 21)  # numpy equivalent of list(range(10, 21))
print('Full array:')
print(my_array, '\n')
```

```python
print("Index the first element:")
print(my_array[0])
```

```python
print("Index the second-to-last element:")
print(my_array[-2])
```

```python
print("Slice from 5 until (not including!) 8")
print(my_array[5:8])
```

```python
print("Slice from beginning until 4")
print(my_array[:4])
```

Setting values in numpy arrays works the same way as lists:

```python
my_array = np.arange(10, 21)
my_array[0] = 10000
print(my_array)

my_array[5:7] = 0
print(my_array)
```

    ToDo (1 point): In the array <tt>my_array</tt> below, set all odd indices (i.e., the first element, the third element, the fifth element, etc.) to the value <tt>0.0</tt> in a single statement using a slice (i.e., not a for-loop).

```python
""" Implement the ToDo here. """
my_array = np.arange(3, 25)

### BEGIN SOLUTION
my_array[1::2] = 0.0
### END SOLUTION
```

```python
''' Tests the above ToDo. '''
from tests import test_set_odd_indices_to_zero
test_set_odd_indices_to_zero(my_array)
```

### Multidimensional indexing
Often, instead of working on and indexing 1D array, we'll work with multi-dimensional (>1D) arrays. Indexing multi-dimensional arrays is, again, quite similar to other scientific computing languages.

Like indexing Python lists, indexing multidimensional numpy arrays is done with square brackets `[]`, in which you can put as many comma-delimited numbers as there are dimensions in your array.

For example, suppose you have a 2D array of shape $3 \times 3$ and you want to index the value in the first row and first column. You would do this as follows:

```python
my_array = np.zeros((3, 3)) # 3 by 3 array with zeros
indexed_value = my_array[0, 0]
print("Value of first row and first column: %.1f" % indexed_value)
```

ToDo: Using multidimensional indexing, set the value in the last row and last column of the 2D array <tt>my_array</tt> to <tt>1.0</tt>. In other words, set the value in the lower-right "corner" of the 2D array to 1.0.

```python
""" Implement the ToDo here. """
my_array = np.zeros((3, 3))

print(my_array)

# ToDo: set the value in the lower right corner to 1

### BEGIN SOLUTION
my_array[-1, -1] = 1.0
### END SOLUTION
```

```python
""" Tests the ToDo above. """
from tests import test_set_lower_right_value_to_one
test_set_lower_right_value_to_one(my_array)
```

In addition to setting specific slices to specific values, you can also extract sub-arrays using slicing/indexing. An important construct here is that you can use a single colon `:` to select all values from a particular dimension. For example, if you want to select all column-values (second dimension) from only the first row (first dimension), do this:

```
some_2d_arr[0, :]
```

We'll show you some examples below:

```python
my_array = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]])

print(my_array, '\n')

all_column_values_from_first_row = my_array[0, :]
print('First row')
print(all_column_values_from_first_row, '\n')

all_row_values_from_first_col = my_array[:, 0]
print('First column')
print(all_row_values_from_first_col)
```

So far, we only talked about 2D arrays, which are relatively easy to 'understand'. In neuroimaging, however, we usually work with 3D, 4D, or even higher-dimensional arrays. These arrays are great for organizing data, but they are somewhat unintuitive. This takes some time to get used to!

To get you used to thinking in more than 2 (or 3) dimensions, consider the following scenario. Suppose that a researcher wants to test the efficacy of a particular medicine against high blood pressure. To do so, she/he measures the (average systolic) blood pressure of a group of twenty subjects every hour for 30 days when they're not on the medication. The same subjects are then again measured for another 30 days, but then when they're on medication.

After this period of data collection, the researcher has $20\ \mathrm{(subjects)}\ \times\ 24\ \mathrm{(hours)}\ \times\ 30\ \mathrm{(days)}\ \times\ 2\ \mathrm{(conditions: off/on)} = 28800$ measurements! We can then organize the data in a 4D array, in which each factor (subjects/hours/days/conditions) represents a separate dimension (also called "axis") of our array!\*

---
\* Note that this type of data is perhaps best represented in a "table-like" structure, such as a Dataframe in R or a Pandas DataFrame, the Python-equivalent of R's dataframe). But you can ignore that for the sake of this example.

So, let's generate some random data (from a normal distribution to generate 'realistic' blood pressure data) that could represent this blood pressure dataset:

```python
np.random.seed(42)  # this is not important for now
bp_data = np.random.normal(loc=100, scale=5, size=(20, 24, 30, 2))
```

(Note that we're not printing the `bp_data` variable here, because it's way too much to visualize/interpret at once anyway!)

Now, suppose I would want to extract the blood pressure of subject 5 at 09.00 (AM) in the morning at day 21 when he/she was *not* on medication. In that case, I would do:

```python
# Note that I count midnight as the first measurement
this_particular_datapoint = bp_data[4, 8, 20, 0]
print("Blood pressure of participant 5 at 09.00 (AM) on day 21 in "
      "the no-medication (off) condition is %.3f" % this_particular_datapoint)

# Also, remember that Python is zero-based, so e.g. participant 5 is indexed by index 4!
```

Try to remember this: *in multidimensional arrays, each dimension (axis) represents a different "attribute" of your data (e.g. subjects, time, condition, etc.)*. This concept is very important to understand in, for example, functional MRI analysis (which revolves around 4D data: 3 spatial dimensions and 1 time dimension)!

ToDo: Consider the blood-pressure dataset again. In the code-cell below, extract from all participants all data from day 18 in the medication ("on") condition. You'll have to use slices with the single <tt>:</tt>! Store the results of your index-operation in the variable <tt>day_18_condition_on</tt>.

```python
""" Implement the ToDo here. """
### BEGIN SOLUTION
day_18_condition_on = bp_data[:, :, 17, 1]
### END SOLUTION
```

```python
""" Tests the above ToDo. """
from tests import test_bloodpressure_index
test_bloodpressure_index(day_18_condition_on)
```

Multidimensional indexing using slices (especially the single colon `:` slice, i.e., selecting everything) is very common in scientific programming such as neuroimaging analyses. There is, however, yet another way of (multidimensional) indexing called "boolean indexing".

In this type of indexing, you index an array with a boolean array (i.e. array with True and False values) of the same shape. Basically, when you're indexing the array `my_array` with boolean array `bool_array`, you're saying: "give me all values in `my_array` that are `True` at the same location in `bool_array`!"

Let's look at an example:

```python
my_array = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]])

print("The original array:\n")
print(my_array, '\n')

bool_array = np.array([[True, False, True],
                       [False, True, False],
                       [True, False, True]])

print("The boolean array:\n")
print(bool_array, '\n')

print('Result of indexing my_array with bool_array:\n')
print(my_array[bool_array])
```

Usually, you do not write out the boolean array in full (as we did above), but you base it on the data itself to "filter" it according to some criterion formalized as a logical statement (i.e., using the boolean operators >, <, ==, or !=). For example, suppose I want to extract only the values above 6 from the `my_array` variable in the above example.

To do so, I could do the following:

```python
my_array = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]])

print("The original array:\n")
print(my_array, '\n')

bool_array = my_array > 6

print("The boolean array:\n")
print(bool_array, '\n')

print('Result of indexing my_array with bool_array:\n')
print(my_array[bool_array])
```

Easy, right? Now try it yourself in the next ToDo!

ToDo: Use a boolean index to extract all values whose square (i.e. $x^2$) is larger than 4 from the array (<tt>my_array</tt>) below. Store the results in a variable with the name <tt>square_is_larger_than_4</tt>.

```python
""" Implement the ToDo here. """
my_array = np.array([[0, 1, -1, -2],
                     [2, -5, 1, 4],
                     [10, -2, -4, 20]])

# Make a new boolean array below (name it whatever you want) ...

# And use it to index my_array and store it in a variable `square_is_larger_than_4`.

### BEGIN SOLUTION
square_is_larger_than_4 = my_array[my_array ** 2 > 4]
### END SOLUTION
```

```python
""" Tests the above ToDo. """
from tests import test_boolean_indexing
test_boolean_indexing(square_is_larger_than_4)
```

Again, it's very important to understand how to (effectively) index multidimensional numpy arrays using slices and boolean indexing, as we'll use it a lot in this course!

## Data-types
Every numpy array is a grid of values of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy guesses the datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype.

Here are a couple of examples:

```python
x1 = np.array([1, 2])  # Let numpy choose the datatype (here: int)
x2 = np.array([1.0, 2.0])  # Let numpy choose the datatype (here: float)
x3 = np.array([1, 2], dtype=np.float64)  # Force a particular datatype (input: int, but converted to 64bit float)
x4 = np.array([-1, 0, 1, 2], dtype=bool)  # Convert ints to booleans! 0 -> False, everthing else -> True

print('%r (%s)' % (x1, type(x1[0])))
print('%r (%s)' % (x2, type(x2[0])))
print('%r (%s)' % (x3, type(x3[0])))
print('%r (%s)' % (x4, type(x4[0])))
```

Note that, after creating the array, you can still change the datatype by using the numpy method `astype()`!

```python
x = np.array([1, 2, 3, 4])
print(type(x[0]))

# Now convert it to floasts!
x = x.astype(float)  # or astype(np.float)
print(type(x[0]))
```

    ToDo: Below, we've defined a boolean array named <tt>bool_array</tt>. Convert it to an array with integers and store it in a new array called <tt>bool2int_array</tt>. Check out the values of the new array!

```python
""" Implement the ToDo here. """
bool_array = np.array([True, True, False, False, True])
### BEGIN SOLUTION
bool2int_array = bool_array.astype(int)
print(bool2int_array)
### END SOLUTION
```

```python
""" Tests the above ToDo (not for points). """
np.testing.assert_array_equal(bool2int_array, np.array([1, 1, 0, 0, 1]))
print("Well done!")
```

## Methods vs. functions
In the previous notebooks, you've learned that, in addition to functions, 'methods' exist that are like functions of an object. You've seen examples of list methods, e.g. `my_list.append(1)`, and string methods, e.g. `my_string.replace('a', 'b')`.

Like lists and strings, numpy arrays have a lot of convenient methods that you can call (like the `astype` method we saw earlier). Again, this is just like a function, but then applied to itself. Often, numpy provides both a function and method for simple operations.

Let's look at an example:

```python
my_array = np.arange(10)  # creates a numpy array from 0 until (excluding!) 10
print(my_array, '\n')

mean_array = np.mean(my_array)
print('The mean of the array is: %f' % mean_array)

mean_array2 = my_array.mean()
print('The mean of the array (computed by its corresponding method) is: %f' % mean_array2)

print('Is the results from the numpy function the same as '
      'the corresponding method? Answer: %s' % str(mean_array == mean_array2))
```

If there is both a function and a method for the operation you want to apply to the array, it really doesn't matter what you choose! Let's look at some more (often used) methods of numpy ndarrays:

```python
my_array = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]])

std_my_array = my_array.std()  # same as np.std(array)
print("Standard deviation of my_array: %.3f" % std_my_array, '\n')

transpose_my_array = my_array.T  # same as np.transpose(array)
print("Transpose of my_array:\n%r" % transpose_my_array, '\n')

min_my_array = my_array.min()  # same as np.min(array)
print("Minimum of my_array: %i" % my_array.min(), '\n')

max_my_array = my_array.max()  # same as np.max(array)
print("Maximum of my_array: %i" % max_my_array, '\n')

sum_my_array = my_array.sum()  # same as np.sum(array)
print("Sum of my_array: %i" % sum_my_array, '\n')
```

Importantly, a method may or may not take arguments (input).
If no arguments are given, it just looks like "object.method()", i.e. two enclosing brackets with nothing in between.
However, a method may take one or more arguments (like the my_list.append(1) method)!
This argument may be named or unnamed - doesn't matter. An example:

```python
my_array2 = np.random.random((3, 3))
print('Original array:')
print(my_array2, '\n')

print('Use the round() method with the argument 3:')
print(my_array2.round(3), '\n')

print('Use the round() method with the named argument 5:')
print(my_array2.round(decimals=5), '\n')
```

In addition to the methods listed above, you'll probably see the following methods a lot in the code of others.

Reshaping arrays:

```python
my_array = np.arange(10)
print(my_array.reshape((5, 2))) # reshape to desired shape
```

Ravel ("flatten") an array:

```python
temporary = my_array.reshape((5, 2))
print("Initial shape: %s" % (temporary.shape,))
print(temporary.ravel()) # unroll multi-dimensional array to single 1D array
print("Shape after ravel(): %s" % (temporary.ravel().shape,))
```

```python
# .dot() does matrix multiplication (dot product: https://en.wikipedia.org/wiki/Dot_product)
# This linear algebra operation is used very often in neuroimaging research
# (which depends heavily on the General Linear Model!)
array1 = np.array([0, 1, 2, 3])
array2 = np.array([4, 5, 6, 7])

dot_product = array1.dot(array2)
print(dot_product)
```

ToDo (1 point): Let's practice writing functions some more. Complete the function below, named <tt>calculate_range</tt>. This function takes  a single input-argument — a 1D numpy array — and subsequently calculates the range of the array. The range is the difference between the maximum and minimum of any given array (vector) $x$:

\begin{align}
\mathrm{range}(x) = \max(x) - \min(x)
\end{align}

You may use the corresponding numpy min/max methods or functions in your custom function, doesn't matter. Don't forget to explicitly return the value of the range!

Note: this custom function that implements the *mathematical* formula for a vector's range is completely unrelated to the Python function <tt>range</tt> that is often used in for-loops!

```python
""" Implement the ToDo here by completing the function. """
def calculate_range(arr):
    ''' Calculate the range of an array.

    Parameters
    ----------
    arr : a 1D numpy array

    Returns
    -------
    The range of the input arr
    '''

    ### BEGIN SOLUTION
    return arr.max() - arr.min()
    ### END SOLUTION
```

```python
""" Tests the above ToDo. """

outp = calculate_range(np.array([0, 1, 2, 3]))
if outp is None:
    raise ValueError("Didn't get any output! Did you explicitly return the range?")

assert(outp == 3)
assert(calculate_range(np.array([-1, 0, 1, 2])) == 3)
assert(calculate_range(np.array([0, 0, 0, 0])) == 0)

print("Great job!")
```

Alright, by now, if you see a variable followed by a word ending with enclosed brackets, e.g. `my_array.mean()`, you'll know that it's a method! But sometimes you might see something similar, but **without** the brackets, such as `my_array.size`. As you know, this `.size` is called an **attribute** of the variable `my_array`. The attribute may be of any data-type, like a string, integer, tuple, an array itself. Let's look at a numpy-specific example:

```python
my_array = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]])

print(my_array, '\n')
print('The size (number of element) in the array is:')
print(my_array.size, '\n')
print('The .size attribute is of data-type: %s' % type(my_array.size))
```

Again, you might not use attributes a lot during this course, but you'll definitely see them around in the code of the tutorials. Below, some of the common ndarray attributes are listed:

```python
my_array = np.array([[1, 2, 3],
                     [4, 5, 6],
                     [7, 8, 9]])

print('Size (number of elements) of array:')
print(my_array.size, '\n') # returns an integer

print('Shape of array:')
print(my_array.shape, '\n') # this is a tuple!

print('Number of dimensions:')
print(my_array.ndim) # this is an integer
```

    ToDo: Let's try another one. Complete the function below, named <tt>compute_one_sample_ttest</tt>. This function takes two input arguments:

- arr : a 1D numpy array
- h0 : a scalar value (single number) that represents the value representing the null-hypothesis

The function should compute the one-sample t-test, which tests whether the mean of an array is significantly different from a given value representing the null-hypothesis. Formally, for any given array $x$ and null-hypothesis $h_{0}$:

\begin{align}
t = \frac{\bar{x} - h_{0}}{s\ / \sqrt{N}}
\end{align}

Here: $\bar{x}$ represents the mean of $x$, $s$ represents the standard deviation of $x$, and $N$ represents the length ('size') of x. So, in slightly less mathematical notation:

\begin{align}
t = \frac{\mathrm{mean}(x) - h_{0}}{\mathrm{std}(x)\ / \sqrt{\mathrm{length}(x) - 1}}
\end{align}

Make sure to return the t-value!

**Hint 1**: to compute $N$, you can use the <tt>.size</tt> attribute of an array ...

**Hint 2**: use the function <tt>np.sqrt(some_number)</tt> to calculate the square root ...

```python
""" Implement the ToDo here. """
def compute_one_sample_ttest(arr, h0):
    ''' Computes the one-sample t-test for any array and h0.

    Parameters
    ----------
    arr : a 1D numpy array
    h0 : an int or float

    Returns
    -------
    A single value representing the t-value
    '''

    ### BEGIN SOLUTION
    t_val = (arr.mean() - h0) / (arr.std() / np.sqrt(arr.size - 1))
    return t_val
    ### END SOLUTION
```

```python
""" Tests the ToDo above. """
from tests import test_tvalue_computation

arr = np.random.randn(100)
outp = compute_one_sample_ttest(arr , 0)

if outp is None:
    raise ValueError("Your function didn't return anything! Did you forget the return statement?")

print("Going to do 3 tests ...")
test_tvalue_computation(arr, 0, outp)

outp = compute_one_sample_ttest(arr, 5)
test_tvalue_computation(arr, 5, outp)

outp = compute_one_sample_ttest(arr, -3)
test_tvalue_computation(arr, -3, outp)
```

## Numpy: array math
Now you know all the numpy basics necessary to do neuroimaging analysis! As you'll see in the last section (Working with nifti-images), we'll work with 3D (structural MRI images) or 4D (functional MRI images) numpy arrays a lot. Given that you know how the basics about numpy in general and numpy ndarrays in particular, we can utilize some of numpy's best features: (very fast) array math.

Basic mathematical functions operate elementwise on arrays, which means that the operation (e.g. addition) is applied onto each element in the array. So, let's initialize a 1D array with ten zeros and let's add 1 to it:

```python
x = np.zeros(10)
print(x, '\n')
x += 1 # remember: this the same as x = x + 1
print(x)
```

Additionally, you can also sum two arrays together in an elementwise manner by simply writing: `array_1 + array_2`, given that these two (or more) arrays are of the same shape! Let's look at an example:

```python
x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.float64)

print("x: \n%r" % x, '\n')
print("y: \n%r" % y, '\n')
print("x+y: \n%r" % (x + y), '\n')
```

Often, there exist function-equivalents of the mathematical operators. For example, `x + y` is the same as `np.add(x, y)`. However, it is recommended to use the operators wherever possible to improve readability of your code. See below for an example:

```python
print(x + y, '\n')
print(np.add(x, y))
```

Next to addition, we can also do elementwise subtraction, multiplication, divison, square root, and exponentiation:

```python
# Elementwise difference; both produce the array
print(x - y, '\n')
print(np.subtract(x, y))  # function-equivalent of above
```

```python
# Elementwise product; both produce the array
print(x * y, '\n')
print(np.multiply(x, y))
```

```python
# Elementwise division; both produce the array
# [[ 0.2         0.33333333]
#  [ 0.42857143  0.5       ]]
print(x / y, '\n')
print(np.divide(x, y))
```

```python
# Elementwise square root; there is no operator-equivalent!
print(np.sqrt(x))
```

```python
# Elementwise exponentiation
print(x ** y, '\n')
print(np.power(x, y))
```

ToDo: Do an elementwise product between the two variables defined below (<tt>arr_A</tt> and <tt>arr_B</tt>) and subsequently add 5 to each element; store the result in a new variable called <tt>result_product_and_sum</tt>.

```python
""" Implement the ToDo here. """
arr_A = np.arange(10).reshape((5, 2))
arr_B = np.arange(10, 20).reshape((5, 2))

### BEGIN SOLUTION
result_product_and_sum = (arr_A * arr_B) + 5
### END SOLUTION
```

```python
""" Tests the above ToDo. """
from tests import test_array_product_and_sum
test_array_product_and_sum(result_product_and_sum)
```

Note that unlike Matlab, `*` is elementwise multiplication, not matrix multiplication. We instead use the dot function (or method!) to  compute matrix operations like inner products of vectors, multiplication of a vector by a matrix, and multiplication of two matrices.

These matrix operations are quite important in this course, because they are used a lot in neuroimaging methods (such as the General Linear Model, the topic of next week!). You're not expected to fully understand these matrix operations, but you'll see them several times in this course, so make sure you're familiar with its implementation in Python/Numpy.

An example of the inner product ("dot product") of two vectors in both the function-format and the method-format:

```python
v = np.array([9,10])
w = np.array([11, 12])

# Inner product of vectors; both produce 219
print(v.dot(w))
print(np.dot(v, w))
```

Additionally, in Python 3.6 (and above), you can use the `@` character as an operator for matrix multiplication of numpy arrays!

```python
print(v @ w)
```

Probably the most used functions in numpy are the sum() and mean() fuctions (or equivalent methods!). A nice feature is that they can operate on the entire array (this is the default) or they can be applied per dimension (or, in numpy lingo, per "axis").

Applying functions along axes is very common in scientific computing! Let's look at an example in which we apply the `sum` function/method across rows and columns:

```python
x = np.array([[1, 2],[3, 4], [5, 6]])

print('Original array:')
print(x, '\n')

print('Sum over ALL elements of x:')
print(np.sum(x), '\n')

print('Sum across rows of x:')
print(np.sum(x, axis=0), '\n')

print('Sum across columns of x:')
print(x.sum(axis=1)) # this is the method form! Is exactly the same as np.sum(x, axis=1)
```

Importantly, application of functions across axes is much quicker (and more concise) than writing for-loops! Let's look at the speed difference between a for-loop (implemented as a list comprehension) and the numpy-style application of the `sum` function across axes:

```python
arr = np.random.random((1000, 100))
loop_style = %timeit -o [arr[i, :].sum() for i in range(arr.shape[0])]
axis_style = %timeit -o arr.sum(axis=1)
print("Using the axis-argument is %.3f times faster than a for-loop!" % (loop_style.average / axis_style.average))
```

This application of functions on entire arrays (in an elementwise manner, like `arr_1 + arr_2`) and application of functions across a certain axis (like `np.sum(arr_1, axis=1)`) is often called **vectorization**. This is an incredibly useful concept and something that is used in many data analysis tools! Make sure you understand this.

Let's practice this a bit.

ToDo: Remember the "range" function from before? Below, we started writing a function, called <tt>calculate_range_vectorized</tt>, that calculates, for any 2D array, the range of each column (i.e. calculates range across rows for each column) in a vectorized manner (so without any for loops!).

Complete the function, and make sure to return the result, which should be a 1D vector which values represents the ranges for all columns. So, if the input array would be of shape <tt>(100, 10)</tt>, then the shape of your returned array should be <tt>(10,)</tt>.

```python
""" Implement the ToDo here (by completing the function below). """
def compute_range_vectorized(arr):
    ''' Compute the range across rows for each column in a 2D array.

    Parameters
    ----------
    arr : a 2D numpy array

    Returns
    -------
    A 1D vector with the ranges for each column
    '''

    ### BEGIN SOLUTION
    return arr.max(axis=0) - arr.min(axis=0)
    ### END SOLUTION
```

```python
""" Tests the above ToDo. """
from tests import test_compute_range_vectorized

np.random.seed(42)
test_arr = np.random.random((100, 10))
outp = compute_range_vectorized(test_arr)

if outp is None:
    raise ValueError("Output is None! Did you forget the Return statement in your function?")

test_compute_range_vectorized(test_arr, outp)
```

## Broadcasting
Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.

For example, suppose that we want to add a vector to each row of a matrix. We could do it like this:

```python
# We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])

print('x array is of shape: %r' % (x.shape,))
print(x, '\n')

v = np.array([1, 0, 1])
print('v vector is of shape: %r (different shape than x!)' % (v.shape,))
print(v, '\n')

y = np.zeros(x.shape)   # Create an empty (zeros) matrix with the same shape as x
print('Shape of (pre-allocated) y-matrix: %r' % (y.shape,))

# Add the vector v to each row of the matrix x with an explicit loop
for i in range(x.shape[0]): # see how the shape attributes comes in handy in creating loops?
    y[i, :] = x[i, :] + v

print('The result of adding v to each row of x, as stored in y:')
print(y)
```

This works; however when the matrix `x` is very large, computing an explicit for-loop in Python could be slow. Note that adding the vector v to each row of the matrix `x` is equivalent to forming a matrix `vv` by stacking multiple copies of `v` vertically, like this `[[1 0 1], [1 0 1], [1 0 1], [1 0 1]]`, and subsequently elementwise addition of `x + vv`:

```python
vv = np.tile(v, (4, 1)) # i.e. expand vector 'v' 4 times along the row dimension (similar to MATLAB's repmat function)
y = x + vv  # Add x and vv elementwise
print(y)
```

Numpy **broadcasting** allows us to perform this computation without actually creating multiple copies of v. Consider this version, using broadcasting:

```python
# We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])
y = x + v  # Add v to each row of x using broadcasting
print(y)
```

The line `y = x + v` works even though `x` has shape `(4, 3)` and `v` has shape `(3,)` due to broadcasting; this line works as if v actually had shape `(4, 3)`, where each row was a copy of `v`, and the sum was performed elementwise.

This broadcasting function is really useful, as it prevents us from writing unnessary and by definition slower explicit for-loops. Additionally, it's way easier to read and write than explicit for-loops (which need pre-allocation). Functions that support broadcasting are known as universal functions. You can find the list of all universal functions in the [documentation](http://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs).

    Tip: The video below (which renders after running the cell) explains broadcasting very nicely — check it out if you want to see a visual representation of how broadcasting works!

```python
from IPython.display import YouTubeVideo
YouTubeVideo('K96OoAWbhBE', width=800, height=300)
```

Here are some applications of broadcasting using different functions:

```python
x = np.array([[1, 2],[3, 4], [5, 6]], dtype=np.float)
x_sum = x.sum(axis=0)

print(x / x_sum, '\n')

x_mean = x.mean(axis=0)
print(x / x_mean)
```

    ToDo: Below, we started writing a function called <tt>standardize_columns</tt>, which takes a single input-argument — a 2D numpy array — and should subsequently standardize each column such that each column has a mean of zero and a standard deviation of 1. In other words, standardization subtracts the mean (across rows) of each column from the values in that column, and then divides those value with the standard deviation of that column. Formally, for each column $j$ in any 2D array, standardization entails:

\begin{align}
x_{j}\ \mathrm{standardized} = \frac{(x_{j} - \bar{x_{j}})}{\mathrm{std}(x_{j})}
\end{align}

Standardization, which is also oftend called "z-transformation", is often done in statistics. So, in the function below, make sure that it is able to standardize each column in any 2D array. Make sure to use vectorized array computations (i.e, the <tt>np.mean</tt> and <tt>np.std</tt> functions/methods across rows) and broadcasting, so no for-loops!

```python
""" Implement the ToDo here (by completing the function below). """
def standardize_columns(arr):
    ''' Standardize each column of a 2D input-array.

    Parameters
    ----------
    arr : a 2D numpy array

    Returns
    -------
    The input-array with standardized columns (should have the same shape as the input-array!)
    '''

    ### BEGIN SOLUTION
    return (arr - arr.mean(axis=0)) / arr.std(axis=0)
    ### END SOLUTION
```

```python
""" Tests the above ToDo. """

test_arr = np.random.normal(5, 2, size=(100, 10))
outp = standardize_columns(test_arr)

if outp is None:
    raise ValueError("The output from your function is None; did you forget the Return statement?")

try:
    np.testing.assert_array_almost_equal(outp.mean(axis=0), np.zeros(test_arr.shape[1]))
except AssertionError as e:
    print("The mean of the columns of your standardized array are not 0!")
    raise(e)

try:
    np.testing.assert_array_almost_equal(outp.std(axis=0), np.ones(test_arr.shape[1]))
except AssertionError as e:
    print("The std of the columns of your standardized array are not 1!")
    raise(e)

print("A-MA-ZING!")
```

Now, you know the most important numpy concepts and functionality that is necessary to start using it. Surely, there is a lot more to the numpy package that what we've covered here, but that's for another time 😃


```
from sympy import *
init_printing(use_latex='mathjax')
x, y, z = symbols('x,y,z')
```

# Probability Modeling

```
from sympy.stats import *
```

```
X = Normal('X', 0, 1)
X
```

```
# Density
```

```
# Density of expressions
```

```
# Probability
```

```
# Sampling
```

```

```

## Dice

```
A = Die('A', 6)
B = Die('B', 6)

density(A+B)
```

```

```

### Evaluate = False

```

```

```

```

```

```

## More distributions

```
from sympy.stats import Gamma
Gamma?
```

```
k = Symbol('k', positive=True)
theta = Symbol('theta', positive=True)
A = Gamma('A', k, theta)
A
```

```
density(A**2)(x)
```


```python
import sympy as sm
```

# Limit
> ## $ \lim \limits_{x \to \pi} sin(\frac{x}{2}+sin(x))$

```python
x = sm.symbols('x')
expr = sm.sin(x/2 + sm.sin(x))
sm.limit(expr,x,sm.pi)
```

$\displaystyle 1$

## $ \lim \limits_{x\to0^{+}} \frac{2e^{1/x}}{e^{1/x}+1} $

```python
sm.limit(2*sm.exp(1/x) / (sm.exp(1/x)+1),x,0,dir='+')
```

$\displaystyle 2$

## $ \lim\limits_{x \to \infty} \frac{cos(x)-1}{x}$

```python
sm.limit( (sm.cos(x)-1)/x, x, sm.oo)
```

$\displaystyle 0$

# Derivatvies
## $ \frac{d}{dx}(\frac{1+sin\,x}{1-cos\, x})^2 $

```python

```