Create python definitions of the following 5 functions and their derivatives:

$f(x) = ln(x^3)$

$f(x) = exp(x^{1/3})$

$f(x) = x^2 sin(x)$

$f(x) = \frac{\sqrt{2}}{sin(x) cos(x)}$

$f(x) = \sigma(x)$ // this is the sigmoid function, important historically for neural nets

```
# LAMBDA SCHOOL
#
# MACHINE LEARNING
#
# MIT LICENSE

# Your code goes here...
```

Plot the above five functions each on separate plots.

```
import numpy as np
import matplotlib.pyplot as plt

# Your code goes here

def f_1(x):
    return np.log(np.power(x, 3))

def f_2(x):
    return np.exp(np.cbrt(x))

def f_3(x):
    return x**2 * np.sin(x)

def f_4(x):
    return np.sqrt(2) / (np.sin(x) * np.cos(x))

def f_5(x):
    return 1 / (1 + np.exp(-x))

x = np.linspace(-10, 10, 5000)

y = f_1(x)
plt.plot(x, y);
plt.show()

y = f_2(x)
plt.plot(x, y);
plt.show()

y = f_3(x)
plt.plot(x, y);
plt.show()

y = f_4(x)
upper_lim = 100
lower_lim = -100
y[y>upper_lim] = np.inf
y[y<lower_lim] = -np.inf
plt.plot(x, y);
plt.ylim((-15, 15))
plt.show()

y = f_5(x)
plt.plot(x, y);
plt.show()
```

Where is the derivative of each function equal to zero?

Visually, where the plot of the function is flat, such as in a local minimum or maximum, or another point of zero slope such as the point (0,0) in  $ f(x) = x^3 $

By inspection, we can see that f_1, f_2, and f_5 have no such places where the derivative equals zero, f_4 seems to have infinitely many that show up in a periodic fashion, and f_3 also has many, though it is not clear from the plot if there are infinitely many.

---

Analytically:

$ \frac{df_1(x)}{dx} = \frac{1}{x^3} * \frac{d}{dx}(x^3) $

$ = \frac{1}{x^3} * 3x^2 $

$ = \frac{3}{x}, x>0 $

While the derivative of f_1(x) approaches zero as x approaches infinity, it is never equal to zero.

---

$ \frac{df_2(x)}{dx} = exp(x^{1/3}) * \frac{d}{dx}(x^{1/3}) $

$ = exp(x^{1/3}) * \frac{x^{-2/3}}{3} $

$ = \frac{exp(x^{1/3})}{3x^{2/3}}, x \neq 0 $

The derivative of f_2(x) is never equal to zero.

---

$ \frac{df_3(x)}{dx} = x^2 \frac{d}{dx}(sin(x)) + sin(x)\frac{d}{dx}(x^2) $

$ = x^2 cos(x) + 2xsin(x) $

This derivative function has infinitely many zeros. One at x = 0, and then more whenever $x^2 cos(x) =  2xsin(x)$, or in other terms, $\frac{tan(x)}{x} = \frac{1}{2}$ Since the tangent function is periodic and cycles through all the real numbers, there are infinitely many instances of this.

---

$ \frac{df_4(x)}{dx} = \sqrt{2} \frac{-1}{sin^2(x)cos^2(x)}\frac{d}{dx}(sin(x)cos(x)) $

$ = \sqrt{2} \frac{-1}{sin^2(x)cos^2(x)} (-sin^2(x) + cos^2(x)) $

$ = \sqrt{2} \frac{sin^2(x) - cos^2(x)}{sin^2(x)cos^2(x)} $

$ = \sqrt{2}(sec^2(x) - csc^2(x)) $

This derivative is equal to zero whenever $ sec^2(x) = cos^2(x) $, which is equivalent to whenever $ |sin(x)| = |cos(x)| $.
This happens whenever $ x = n\pi - \frac{\pi}{4}, x = n\pi - \frac{3\pi}{4} $ so there are infinitely many zeros.

---

$ \frac{df_4(x)}{dx} = \frac{-1}{(1 + exp(-x))^2} \frac{d}{dx}(1 + exp(-x))$

$ = \frac{-1}{(1 + exp(-x))^2} (-exp(-x)) $

$ = \frac{exp(-x)}{(1 + exp(-x))^2} $

The sigmoid function has derivatives that approach zero as x approaches infinity, and as x approaches negative infinity, but nowhere does the derivative equal exactly zero.

```
import sympy as sym

def my_ln_prime(a):
    x = sym.Symbol('x')
    y = sym.log(x**3)
    yprime = y.diff(x)

    f = sym.lambdify(x, yprime)
    return f(a)

my_ln_prime(1)
```

    3.0

Week 01 Mathematical Foundations/Function Optimization.ipynb


# Transformations, Eigenvectors, and Eigenvalues

Matrices and vectors are used together to manipulate spatial dimensions. This has a lot of applications, including the mathematical generation of 3D computer graphics, geometric modeling, and the training and optimization of machine learning algorithms. We're not going to cover the subject exhaustively here; but we'll focus on a few key concepts that are useful to know when you plan to work with machine learning.

## Linear Transformations
You can manipulate a vector by multiplying it with a matrix. The matrix acts a function that operates on an input vector to produce a vector output. Specifically, matrix multiplications of vectors are *linear transformations* that transform the input vector into the output vector.

For example, consider this matrix ***A*** and vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

We can define a transformation ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

To perform this transformation, we simply calculate the dot product by applying the *RC* rule; multiplying each row of the matrix by the single column of the vector:

$$\begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\end{bmatrix}$$

Here's the calculation in Python:

```python
import numpy as np

v = np.array([1,2])
A = np.array([[2,3],
              [5,2]])

t = A@v
print (t)
```

    [8 9]

In this case, both the input vector and the output vector have 2 components - in other words, the transformation takes a 2-dimensional vector and produces a new 2-dimensional vector; which we can indicate like this:

$$ T: \rm I\!R^{2} \to \rm I\!R^{2} $$

Note that the output vector may have a different number of dimensions from the input vector; so the matrix function might transform the vector from one space to another - or in notation, ${\rm I\!R}$n -> ${\rm I\!R}$m.

For example, let's redefine matrix ***A***, while retaining our original definition of vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

Now if we once again define ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

We apply the transformation like this:

$$\begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\\3\end{bmatrix}$$

So now, our transformation transforms the vector from 2-dimensional space to 3-dimensional space:

$$ T: \rm I\!R^{2} \to \rm I\!R^{3} $$

Here it is in Python:

```python
import numpy as np
v = np.array([1,2])
A = np.array([[2,3],
              [5,2],
              [1,1]])

t = A@v
print (t)
```

    [8 9 3]

```python
import numpy as np
v = np.array([1,2])
A = np.array([[1,2],
              [2,1]])

t = A@v
print (t)
```

    [5 4]

## Transformations of Magnitude and Amplitude

When you multiply a vector by a matrix, you transform it in at least one of the following two ways:
* Scale the length (*magnitude*) of the matrix to make it longer or shorter
* Change the direction (*amplitude*) of the matrix

For example consider the following matrix and vector:

$$ A = \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\0\end{bmatrix}$$

As before, we transform the vector ***v*** by multiplying it with the matrix ***A***:

\begin{equation}\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}\end{equation}

In this case, the resulting vector has changed in length (*magnitude*), but has not changed its direction (*amplitude*).

Let's visualize that in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([t,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v, color=['blue'], scale=10)
plt.quiver(*origin, *t, color=['orange'], scale=10)
plt.show()
```

The original vector ***v*** is shown in orange, and the transformed vector ***t*** is shown in blue - note that ***t*** has the same direction (*amplitude*) as ***v*** but a greater length (*magnitude*).

Now let's use a different matrix to transform the vector ***v***:
\begin{equation}\begin{bmatrix}0 & -1\\1 & 0\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}0\\1\end{bmatrix}\end{equation}

This time, the resulting vector has been changed to a different amplitude, but has the same magnitude.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[0,-1],
              [1,0]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t, color=['blue'], scale=10)
plt.show()
```

Now let's see change the matrix one more time:
\begin{equation}\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\1\end{bmatrix}\end{equation}

Now our resulting vector has been transformed to a new amplitude *and* magnitude - the transformation has affected both direction and scale.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,1],
              [1,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t, color=['blue'], scale=10)
plt.show()
```

### Afine Transformations
An Afine transformation multiplies a vector by a matrix and adds an offset vector, sometimes referred to as *bias*; like this:

$$T(\vec{v}) = A\vec{v} + \vec{b}$$

For example:

\begin{equation}\begin{bmatrix}5 & 2\\3 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\1\end{bmatrix} + \begin{bmatrix}-2\\-6\end{bmatrix} = \begin{bmatrix}5\\-2\end{bmatrix}\end{equation}

This kind of transformation is actually the basis of linear regression, which is a core foundation for machine learning. The matrix defines the *features*, the first vector is the *coefficients*, and the bias vector is the *intercept*.

here's an example of an Afine transformation in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,1])
A = np.array([[5,2],
              [3,1]])
b = np.array([-2,-6])

t = A@v + b
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=15)
plt.quiver(*origin, *t, color=['blue'], scale=15)
plt.show()
```

## Eigenvectors and Eigenvalues
So we can see that when you transform a vector using a matrix, we change its direction, length, or both. When the transformation only affects scale (in other words, the output vector has a different magnitude but the same amplitude as the input vector), the matrix multiplication for the transformation is the equivalent operation as some scalar multiplication of the vector.

For example, earlier we examined the following transformation that dot-mulitplies a vector by a matrix:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

You can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

The following python performs both of these calculation and shows the results, which are identical.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t1 = A@v
print (t1)
t2 = 2*v
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t1, color=['blue'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t2, color=['blue'], scale=10)
plt.show()
```

In cases like these, where a matrix transformation is the equivelent of a scalar-vector multiplication, the scalar-vector pairs that correspond to the matrix are known respectively as eigenvalues and eigenvectors. We generally indicate eigenvalues using the Greek letter lambda (λ), and the formula that defines eigenvalues and eigenvectors with respect to a transformation is:

$$ T(\vec{v}) = \lambda\vec{v}$$

Where the vector ***v*** is an eigenvector and the value ***λ*** is an eigenvalue for transformation ***T***.

When the transformation ***T*** is represented as a matrix multiplication, as in this case where the transformation is represented by matrix ***A***:

$$ T(\vec{v}) = A\vec{v} = \lambda\vec{v}$$

Then  ***v*** is an eigenvector and ***λ*** is an eigenvalue of ***A***.

A matrix can have multiple eigenvector-eigenvalue pairs, and you can calculate them manually. However, it's generally easier to use a tool or programming language. For example, in Python you can use the ***linalg.eig*** function, which returns an array of eigenvalues and a matrix of the corresponding eigenvectors for the specified matrix.

Here's an example that returns the eigenvalue and eigenvector pairs for the following matrix:

$$A=\begin{bmatrix}2 & 0\\0 & 3\end{bmatrix}$$

```python
import numpy as np
A = np.array([[2,0],
              [0,3]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [2. 3.]
    [[1. 0.]
     [0. 1.]]

So there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 3, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

So far so good. Now let's check the second pair:

$$ 3 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix} $$

So our eigenvalue-eigenvector scalar multiplications do indeed correspond to our matrix-eigenvector dot-product transformations.

Here's the equivalent code in Python, using the ***eVals*** and ***eVecs*** variables you generated in the previous code cell:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))
```

    Matrix A:
    [[2 0]
     [0 3]]
    -------
    lam1: 2.0
    v1: [1. 0.]
    Av1: [2. 0.]
    lam1 x v1: [2. 0.]
    -------
    lam2: 3.0
    v2: [0. 1.]
    Av2: [0. 3.]
    lam2 x v2: [0. 3.]

You can use the following code to visualize these transformations:

```python
t1 = lam1*vec1
print (t1)
t2 = lam2*vec2
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t1, color=['blue'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t2, color=['blue'], scale=10)
plt.show()
```

Similarly, earlier we examined the following matrix transformation:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

And we saw that you can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

This works because the scalar value 2 and the vector (1,0) are an eigenvalue-eigenvector pair for this matrix.

Let's use Python to determine the eigenvalue-eigenvector pairs for this matrix:

```python
import numpy as np
A = np.array([[2,0],
              [0,2]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [2. 2.]
    [[1. 0.]
     [0. 1.]]

So once again, there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 2, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

Well, we already knew that. Now let's check the second pair:

$$ 2 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix} $$

Now let's use Pythonto verify and plot these transformations:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the resulting vectors
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t1, color=['blue'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t2, color=['blue'], scale=10)
plt.show()
```

Let's take a look at one more, slightly more complex example. Here's our matrix:

$$\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}$$

Let's get the eigenvalue and eigenvector pairs:

```python
import numpy as np

A = np.array([[2,1],
              [1,2]])

eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [3. 1.]
    [[ 0.70710678 -0.70710678]
     [ 0.70710678  0.70710678]]

This time the eigenvalue-eigenvector pairs are:

$$ \lambda_{1} = 3, \vec{v_{1}} = \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 1, \vec{v_{2}} = \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} $$

So let's check the first pair:

$$ 3 \times \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix} $$

Now let's check the second pair:

$$ 1 \times \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix} $$

With more complex examples like this, it's generally easier to do it with Python:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the results
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t1, color=['blue'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t2, color=['blue'], scale=10)
plt.show()
```

## Eigendecomposition
So we've learned a little about eigenvalues and eigenvectors; but you may be wondering what use they are. Well, one use for them is to help decompose transformation matrices.

Recall that previously we found that a matrix transformation of a vector changes its magnitude, amplitude, or both. Without getting too technical about it, we need to remember that vectors can exist in any spatial orientation, or *basis*; and the same transformation can be applied in different *bases*.

We can decompose a matrix using the following formula:

$$A = Q \Lambda Q^{-1}$$

Where ***A*** is a trasformation that can be applied to a vector in its current base, ***Q*** is a matrix of eigenvectors that defines a change of basis, and ***Λ*** is a matrix with eigenvalues on the diagonal that defines the same linear transformation as ***A*** in the base defined by ***Q***.

Let's look at these in some more detail. Consider this matrix:

$$A=\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix}$$

***Q*** is a matrix in which each column is an eigenvector of ***A***; which as we've seen previously, we can calculate using Python:

```python
import numpy as np

A = np.array([[3,2],
              [1,0]])

l, Q = np.linalg.eig(A)
print(Q)
```

    [[ 0.96276969 -0.48963374]
     [ 0.27032301  0.87192821]]

So for matrix ***A***, ***Q*** is the following matrix:

$$Q=\begin{bmatrix}0.96276969 & -0.48963374\\0.27032301 & 0.87192821\end{bmatrix}$$

***Λ*** is a matrix that contains the eigenvalues for ***A*** on the diagonal, with zeros in all other elements; so for a 2x2 matrix, Λ will look like this:

$$\Lambda=\begin{bmatrix}\lambda_{1} & 0\\0 & \lambda_{2}\end{bmatrix}$$

In our Python code, we've already used the ***linalg.eig*** function to return the array of eigenvalues for ***A*** into the variable ***l***, so now we just need to format that as a matrix:

```python
L = np.diag(l)
print (L)
```

    [[ 3.56155281  0.        ]
     [ 0.         -0.56155281]]

So ***Λ*** is the following matrix:

$$\Lambda=\begin{bmatrix}3.56155281 & 0\\0 & -0.56155281\end{bmatrix}$$

Now we just need to find ***Q-1***, which is the inverse of ***Q***:

```python
Qinv = np.linalg.inv(Q)
print(Qinv)
```

    [[ 0.89720673  0.50382896]
     [-0.27816009  0.99068183]]

The inverse of ***Q*** then, is:

$$Q^{-1}=\begin{bmatrix}0.89720673 & 0.50382896\\-0.27816009 & 0.99068183\end{bmatrix}$$

So what does that mean? Well, it means that we can decompose the transformation of *any* vector multiplied by matrix ***A*** into the separate operations ***QΛQ-1***:

$$A\vec{v} = Q \Lambda Q^{-1}\vec{v}$$

To prove this, let's take vector ***v***:

$$\vec{v} = \begin{bmatrix}1\\3\end{bmatrix} $$

Our matrix transformation using ***A*** is:

$$\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix} \cdot \begin{bmatrix}1\\3\end{bmatrix} $$

So let's show the results of that using Python:

```python
v = np.array([1,3])
t = A@v

print(t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t, color=['blue'], scale=10)
plt.show()
```

And now, let's do the same thing using the ***QΛQ-1*** sequence of operations:

```python
import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t = (Q@(L@(Qinv)))@v

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=10)
plt.quiver(*origin, *t, color=['blue'], scale=10)
plt.show()
```

So ***A*** and ***QΛQ-1*** are equivalent.

If we view the intermediary stages of the decomposed transformation, you can see the transformation using ***A*** in the original base for ***v*** (orange to blue) and the transformation using ***Λ*** in the change of basis decribed by ***Q*** (red to magenta):

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t1 = Qinv@v
t2 = L@t1
t3 = Q@t2

# Plot the transformations
vecs = np.array([v,t1, t2, t3])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, *v,  color=['orange'], scale=20)
plt.quiver(*origin, *t1, color=['blue'], scale=20)
plt.quiver(*origin, *t2, color=['red'], scale=20)
plt.quiver(*origin, *t3, color=['magenta'], scale=20)
plt.show()
```

So from this visualization, it should be apparent that the transformation ***Av*** can be performed by changing the basis for ***v*** using ***Q*** (from orange to red in the above plot) applying the equivalent linear transformation in that base using ***Λ*** (red to magenta), and switching back to the original base using ***Q-1*** (magenta to blue).

## Rank of a Matrix

The **rank** of a square matrix is the number of non-zero eigenvalues of the matrix. A **full rank** matrix has the same number of non-zero eigenvalues as the dimension of the matrix. A **rank-deficient** matrix has fewer non-zero eigenvalues as dimensions. The inverse of a rank deficient matrix is singular and so does not exist (this is why in a previous notebook we noted that some matrices have no inverse).

Consider the following matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find its eigenvalues (***Λ***):

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])
l, Q = np.linalg.eig(A)
L = np.diag(l)
print(L)
```

    [[-1.  0.]
     [ 0.  5.]]

$$\Lambda=\begin{bmatrix}-1 & 0\\0 & 5\end{bmatrix}$$

This matrix has full rank. The dimensions of the matrix is 2. There are two non-zero eigenvalues.

Now consider this matrix:

$$B=\begin{bmatrix}3 & -3 & 6\\2 & -2 & 4\\1 & -1 & 2\end{bmatrix}$$

Note that the second and third columns are just scalar multiples of the first column.

Let's examine it's eigenvalues:

```python
B = np.array([[3,-3,6],
              [2,-2,4],
              [1,-1,2]])
lb, Qb = np.linalg.eig(B)
Lb = np.diag(lb)
print(Lb)
```

    [[ 3.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00 -6.00567308e-17  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  3.57375398e-16]]

$$\Lambda=\begin{bmatrix}3 & 0& 0\\0 & -6\times10^{-17} & 0\\0 & 0 & 3.6\times10^{-16}\end{bmatrix}$$

Note that matrix has only 1 non-zero eigenvalue. The other two eigenvalues are so extremely small as to be effectively zero. This is an example of a rank-deficient matrix; and as such, it has no inverse.

## Inverse of a Square Full Rank Matrix
You can calculate the inverse of a square full rank matrix by using the following formula:

$$A^{-1} = Q \Lambda^{-1} Q^{-1}$$

Let's apply this to matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find the matrices for ***Q***, ***Λ-1***, and ***Q-1***:

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])

l, Q = np.linalg.eig(A)
L = np.diag(l)
print(Q)
Linv = np.linalg.inv(L)
Qinv = np.linalg.inv(Q)
print(Linv)
print(Qinv)
```

    [[-0.70710678 -0.4472136 ]
     [ 0.70710678 -0.89442719]]
    [[-1.  -0. ]
     [ 0.   0.2]]
    [[-0.94280904  0.47140452]
     [-0.74535599 -0.74535599]]

So:

$$A^{-1}=\begin{bmatrix}-0.70710678 & -0.4472136\\0.70710678 & -0.89442719\end{bmatrix}\cdot\begin{bmatrix}-1 & -0\\0 & 0.2\end{bmatrix}\cdot\begin{bmatrix}-0.94280904 & 0.47140452\\-0.74535599 & -0.74535599\end{bmatrix}$$

Let's calculate that in Python:

```python
Ainv = (Q@(Linv@(Qinv)))
print(Ainv)
```

    [[-0.6  0.4]
     [ 0.8 -0.2]]

That gives us the result:

$$A^{-1}=\begin{bmatrix}-0.6 & 0.4\\0.8 & -0.2\end{bmatrix}$$

We can apply the ***np.linalg.inv*** function directly to ***A*** to verify this:

```python
print(np.linalg.inv(A))
```

    [[-0.6  0.4]
     [ 0.8 -0.2]]


### Solution

```python
# Special command for plotting in jupyter notebooks
%matplotlib inline
```

First import the required modules - `numpy` and `matplotlib`.

```python
import numpy as np
import matplotlib.pyplot as plt
```

Initially, it is good to have a look at our reference solution. Based on the given formula for the trajectory of the partical, we can work out the formula for its velocity as:

\begin{align}
v_x &= -\dfrac{1}{T_1}\exp \left( -\dfrac{t}{T_1} \right)
\\
v_y &= -\dfrac{2}{T_2} \sin \left(\dfrac{t}{T_2}\right)
\end{align}

and acceleration as:

\begin{align}
a_x &= \dfrac{1}{T^2_1} \exp \left( -\dfrac{t}{T_1} \right)
\\
a_y &= -\dfrac{2}{T^2_2} \cos\left(\dfrac{t}{T_2}\right)
\end{align}

We declare our constants as followed.

```python
# Declare the time constants
T1 = 1.0
T2 = 1.0/3.0
```

Then initialise the array to store our reference results. Here, our positions, velocities and accelerations of the particles in time are stored as 2D array of vectors - the first column corresponds to the $x$ component, and the second the $y$ component.

```python
# Initialise array for position, velocity and acceleration
num_points = 500
r_ref = np.zeros((num_points, 2))
v_ref = np.zeros((num_points, 2))
a_ref = np.zeros((num_points, 2))

# Generate the reference time vector
t_ref = np.linspace(0, 2*np.pi, num_points)
```

The results can be calculated using the formula provided above.

```python
# Calculate the x and y coordinates of the particle
r_ref[:, 0] = np.exp(-t_ref/T1)
r_ref[:, 1] = 2.0*np.cos(t_ref/T2)

# Exact solution for velocity
v_ref[:, 0] = -(1.0/T1)*np.exp(-t_ref/T1)
v_ref[:, 1] = -(2.0/T2)*np.sin(t_ref/T2)

# Exact solution for acceleration
a_ref[:, 0] = (1.0/(T1**2))*np.exp(-t_ref/T1)
a_ref[:, 1] = -(2.0/(T2**2))*np.cos(t_ref/T2)
```

We can now plot the postion, velocity and acceleration fields.

```python
# Plot the solutions
# Initilise the figure
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(24, 6))

# Plot the position
ax1.plot(r_ref[:, 0], r_ref[:, 1], linewidth=2.0)
ax1.set_title('Position')
ax1.set_xlabel('x')
ax1.set_ylabel('y')

# Plot the velocity
ax2.plot(v_ref[:, 0], v_ref[:, 1], linewidth=2.0)
ax2.set_title('Velocity')
ax2.set_xlabel('x')
ax2.set_ylabel('y')

# Plot the acceleration
ax3.plot(a_ref[:, 0], a_ref[:, 1], linewidth=2.0)
ax3.set_title('Acceleration')
ax3.set_xlabel('x')
ax3.set_ylabel('y');
```

### Numerical differentiation

#### Forward difference method

Now we assume that we only have a finite number of points for the location of the particle, `r`, in time. For this solution, we assume that we only have 20 data points, i.e. `num_points = 20`, for the particle's trajectory in time interval $t = 0 - 2 \pi$s.

```python
# Generate our finite time vector, with 20 points
num_points = 20
t = np.linspace(0, 2*np.pi, num_points)

# Calculate the position used for the numerical term
r_num = np.zeros((num_points, 2))
r_num[:, 0] = np.exp(-t/T1)
r_num[:, 1] = 2.0*np.cos(t/T2)
```

To calculate the particles velocity and acceleration we need to calculate the first and second derivatives of its position, `r`, with respect to time. Reminds ourselve that here we only have discrete number of points, rather than a continuous function. Hence, a numerical scheme is required.

Python offers the `np.diff()` comand to calculate the difference between consecutive elements in a `numpy` array. The resulting vector is one element shorter than the original, e.g. if `x=[1, 4, 3, 5]`, then `np.diff(x)` returns `[3, -1, 2]`.

With `dx = np.diff(x)` and `dt = np.diff(t)` the division `dx/dt` gives an approximation of the particle’s velocity in the x direction. The second derivative can be calculated using:

    np.diff(x,2)/(np.power(dt[0:num_points - 2], 2)

where `np.diff(x,2)` is the second order difference of `x` – using the above example `np.diff(x,2)` returns `[-4, 3]`.

For our case, we need to apply the differentiation for the $x$ and $y$ direction of the position, which are stored in the first and the second column of `r`, respectively.

```python
# Initialise array for velocity and acceleration
# calculated using forward difference method
v_f = np.zeros((num_points-1, 2))
a_f = np.zeros((num_points-2, 2))

# Forward difference method to calculate velocity
v_f[:, 0] = np.diff(r_num[:, 0]) / np.diff(t)
v_f[:, 1] = np.diff(r_num[:, 1]) / np.diff(t)
# And then the acceleration
# Calculate acceleration
a_f[:, 0] = np.diff(r_num[:, 0], 2) / (np.power(np.diff(t)[0:num_points-2], 2))
a_f[:, 1] = np.diff(r_num[:, 1], 2) / (np.power(np.diff(t)[0:num_points-2], 2))
```

We can look at our forward difference scheme, with 20 data points, compared to the reference - 'smooth' - result.

```python
# Generate the subplots
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(24, 6))

# Plot the position
ax1.plot(r_num[:, 0], r_num[:, 1], linewidth=2.0, label='Forward')
ax1.plot(r_ref[:, 0], r_ref[:, 1], linewidth=2.0, label='Reference')
ax1.set_title('Position')
ax1.legend()

# Plot the velocity
ax2.plot(v_f[:, 0], v_f[:, 1], linewidth=2.0, label='Forward')
ax2.plot(v_ref[:, 0], v_ref[:, 1], linewidth=2.0, label='Reference')
ax2.set_title('Velocity')
ax2.legend(loc=2)

# Plot the acceleration
ax3.plot(a_f[:, 0], a_f[:, 1], linewidth=2.0, label='Forward')
ax3.plot(a_ref[:, 0], a_ref[:, 1], linewidth=2.0, label='Reference')
ax3.set_title('Acceleration')
ax3.legend();
```

#### Central difference method

In the forward difference method, the current velocity is calculated using the current and the following position. The current acceleration is calculated using the previous and next velocity. This leads to a *mismatch* between velocity on one hand and position and acceleration on the other by one timestep. The centre difference method allows to calculate velocities and accelerations at the same positions over the interval `[1, N-1]`. This method is implemented below, and full understanding is not required.

```python
# Initialise array for velocity and acceleration
# calculated using central difference method
v_c = np.zeros((num_points-1, 2))
a_c = np.zeros((num_points-2, 2))

# Positions and time difference between adjacent
# elements of position and time. The argument axis=0
# indicates that we carry out np.diff on column-wise
# of r
dr = np.diff(r_num, axis=0)
dt = np.diff(t)

# Centre difference method
for i in range(1, num_points-1):
    v_c[i-1, :] = (dr[i, :] + dr[i-1, :])/(dt[i] + dt[i-1])
    a_c[i-1, :] = (dr[i, :]/dt[i] - dr[i-1, :]/dt[i-1])/(0.5*(dt[i] + dt[i-1]))
```

We can quantitatively examine the difference between the forward and central difference scheme to calculate the velocity and the acceleration. Again, here we stick to only 20 data points as before.

```python
# Generate the subplots
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))

# Plot the velocity
ax1.plot(v_ref[:, 0], v_ref[:, 1], label='Reference', lw=3.0, ls='--')
ax1.plot(v_f[:, 0], v_f[:, 1], label='Forward')
ax1.plot(v_c[:, 0], v_c[:, 1], label='Central')
ax1.set_title('Velocity')
ax1.legend(loc=2)

# Plot the acceleration
ax2.plot(a_ref[:, 0], a_ref[:, 1], label='Reference', lw=3.0, ls='--')
ax2.plot(a_f[:, 0], a_f[:, 1], label='Forward')
ax2.plot(a_c[:, 0], a_c[:, 1], label='Central')
ax2.set_title('Acceleration')
ax2.legend();
```

#### A programmer approach - with interactive!

We can define functions that do our calculations for the forward and central difference method as followed.

```python
def forward_method(r, t):
    n = len(t)

    # Initialise array for velocity and acceleration
    # calculated using forward difference method
    v_for = np.zeros((n-1, 2))
    a_for = np.zeros((n-2, 2))

    # Forward difference method to calculate velocity
    v_for[:, 0] = np.diff(r[:, 0]) / np.diff(t)
    v_for[:, 1] = np.diff(r[:, 1]) / np.diff(t)

    # And then the acceleration
    # Calculate acceleration
    a_for[:, 0] = np.diff(r[:, 0], 2) / (np.power(np.diff(t)[0:n-2], 2))
    a_for[:, 1] = np.diff(r[:, 1], 2) / (np.power(np.diff(t)[0:n-2], 2))

    return v_for, a_for
```

```python
def central_method(r, t):
    # Initialise array for velocity and acceleration
    # calculated using central difference method
    n = len(t)
    v_cen = np.zeros((n, 2))
    a_cen = np.zeros((n, 2))

    # Positions and time difference between adjacent
    # elements of position and time
    dr = np.diff(r, axis=0)
    dt = np.diff(t)

    # Centre difference method
    for i in range(1, n-2):
        v_cen[i-1, :] = (dr[i, :] + dr[i-1, :])/(dt[i] + dt[i-1])
        a_cen[i-1, :] = (dr[i, :]/dt[i] - dr[i-1, :]/dt[i-1]) / (0.5*(dt[i] + dt[i-1]))

    return v_cen, a_cen
```

By doing so, we can easily calculate the results achieved from either scheme. We can call these functions for the interactive widget, which illustrates the dependence of these two methods on the number of available data points. Full understanding of the code is not required, yet curious students are encouraged to explore.

```python
# Interactive notebook to compare the two methods
from ipywidgets import *
```

```python
def compare_difference(num_points):
    t = np.linspace(0, 2*np.pi, num_points)

    # Calculate the position used for the numerical term
    r_num = np.zeros((num_points, 2))
    r_num[:, 0] = np.exp(-t/T1)
    r_num[:, 1] = 2.0*np.cos(t/T2)

    # Forward difference method
    v_for, a_for = forward_method(r_num, t)

    # Central difference metho
    v_cen, a_cen = central_method(r_num, t)

    # Generate the subplots
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))

    # Plot the velocity
    ax1.plot(v_ref[:, 0], v_ref[:, 1], label='Reference', lw=3.0, ls='--')
    ax1.plot(v_for[:, 0], v_for[:, 1], label='Forward')
    ax1.plot(v_cen[:, 0], v_cen[:, 1], label='Central')
    ax1.set_title('Velocity')
    ax1.legend(loc=2)

    # Plot the acceleration
    ax2.plot(a_ref[:, 0], a_ref[:, 1], label='Reference', lw=3.0, ls='--')
    ax2.plot(a_for[:, 0], a_for[:, 1], label='Forward')
    ax2.plot(a_cen[:, 0], a_cen[:, 1], label='Central')
    ax2.set_title('Acceleration')
    ax2.legend()
```

```python
interact(compare_difference, num_points=IntSlider(
        min=10, max=200, step=10, value=20, description='No. points'));
```

# Lecture 1: solving ordinary differential equations

This lecture introduces ordinary differential equations, and some techniques for solving first order equations. This notebook uses computer algebra via [Sympy](<http://sympy.org/>) to solve some ODE examples from the lecture notes.

# Importing SymPy

To use Sympy, we first need to import it and call `init_printing()` to get nicely typeset equations:

```python
import sympy
from sympy import symbols, Eq, Derivative, init_printing, Function, dsolve, exp, classify_ode, checkodesol

# This initialises pretty printing
init_printing()
from IPython.display import display

# Support for interactive plots
from ipywidgets import interact

# This command makes plots appear inside the browser window
%matplotlib inline
```

# Example: car breaking

During braking a car’s velocity is given by $v = v_{0} e^{−t/\tau}$. Calculate the distance travelled.

We first define the symbols in the equation ($t$, $\tau$ and $v_{0}$), and the function ($x$, for the displacement):

```python

```

Next, we define the differential equation, and print it to the screen for checking:

```python
eqn = Eq(Derivative(x(t), t), v0*exp(-t/(tau)))
display(eqn)
```

The `dsolve` function solves the differential equation symbolically:

```python
x = dsolve(eqn, x(t))
display(x)
```

where $C_{1}$ is a constant. As expected for a first-order equation, there is one constant.

SymPy is not yet very good at eliminating constants from initial conditions, so we will do this manually assuming that $x = 0$ and $t = 0$:

```python
x = x.subs('C1', v0*tau)
display(x)
```

Specifying a value for $v_{0}$, we create an interactive plot of $x$ as a function of the parameter $\tau$:

```python
x = x.subs(v0, 100)

def plot(τ=1.0):
    x1 = x.subs(tau, τ)

    # Plot position vs time
    sympy.plot(x1.args[1], (t, 0.0, 10.0), xlabel="time", ylabel="position");

interact(plot, τ=(0.0, 10, 0.2));
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

# Classification

We can ask SymPy to classify our ODE, e.g. show that it is first order):

```python
classify_ode(eqn)
```

    ('separable',
     '1st_exact',
     '1st_linear',
     'Bernoulli',
     '1st_power_series',
     'lie_group',
     'nth_linear_constant_coeff_undetermined_coefficients',
     'nth_linear_constant_coeff_variation_of_parameters',
     'separable_Integral',
     '1st_exact_Integral',
     '1st_linear_Integral',
     'Bernoulli_Integral',
     'nth_linear_constant_coeff_variation_of_parameters_Integral')

# Parachutist

Find the variation of speed with time of a parachutist subject to a drag force of $kv^{2}$.

The equations to solve is

$$
\frac{m}{k} \frac{dv}{dt} = \alpha^{2} - v^{2}
$$

where $m$ is mass, $k$ is a prescribed constant, $v$ is the velocity, $t$ is time and $\alpha^{2} = mg/k$ ($g$ is acceleration due to gravity).

We specify the symbols, unknown function $v$ and the differential equation

```python
t, m, k, alpha = symbols("t m k alpha")
v = Function("v")
eqn = Eq((m/k)*Derivative(v(t), t), alpha*alpha - v(t)*v(t))
display(eqn)
```

First, let's classify the ODE:

```python
classify_ode(eqn)
```

    ('separable', '1st_power_series', 'lie_group', 'separable_Integral')

We see that it is not linear, but it is separable. Using `dsolve` again,

```python
v = dsolve(eqn, v(t))
display(v)
```

SymPy can verify that an expression is a solution to an ODE:

```python
print("Is v a solution to the ODE: {}".format(checkodesol(eqn, v)))
```

    Is v a solution to the ODE: (True, 0)

Try adding the code to plot velocity $v$ against time $t$.

```python

```

```python

```


## Minimum of a set of Exponentially Distributed Random Variables

Let $\{X_i\}$ be a set of exponentially distributed random variables. Let $y=min\{X_i\}$ be the minimum value of that set.

The cumulative distribution function that $y$ should be smaller or equal all of $\{X_i\}$ is given by:

\begin{equation}
p(y\leq\{X_i\}) = \prod_i p(X_i \geq y) = \lambda^n \exp-n\lambda y
\end{equation}

The cumulative probability depends on the probability that $y$ assumes a *specific* value:

\begin{equation}
p(y\leq\{X_i\}) = \int_0^\infty p(y)
\end{equation}

So that $p(y)$ can be found by taking the derivative:

\begin{equation}
p(y) = n\lambda^n\exp-n\lambda y
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt

scale = 1
n = 5
samples = 5000

X = np.random.exponential(scale=scale,size=[n,samples])
minX = np.min(X,axis=0)

def pdf_x(x):
    return (1/scale)*np.exp(-x/scale)

def pdf_min(y):
    return n*(1/scale)**n * np.exp(-n*y/scale)
```

```python
x = np.linspace(0,10,1000)

plt.figure(figsize=(12,8))

_ = plt.hist(X.flatten(),bins = x,density=True)
_ = plt.hist(minX,bins = x,density=True)

plt.plot(x,pdf_x(x),linewidth=4)
plt.plot(x,pdf_min(x),linewidth=4)
```

## Maximum of a set of Exponentially Distributed Random Variables

Similarly,

\begin{equation}
p(y\geq\{X_i\}) = \prod_i p(Y>X_i = (1-\lambda\exp-\lambda y)^n
\end{equation}

Implying:

\begin{equation}
p(y) = n\lambda(1-\exp-\lambda y)^{n-1} \exp-\lambda y
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt

scale = 1
n = 5
samples = 5000

X = np.random.exponential(scale=scale,size=[n,samples])
maxX = np.max(X,axis=0)

def pdf_x(x):
    return (1/scale)*np.exp(-x/scale)

def pdf_max(y):
    return n * (1/scale) * (1-np.exp(-y/scale))**(n-1) * np.exp(-y/scale)
```

```python
x = np.linspace(0,10,1000)

plt.figure(figsize=(12,8))

_ = plt.hist(X.flatten(),bins = x,density=True)
_ = plt.hist(maxX,bins = x,density=True)

plt.plot(x,pdf_x(x),linewidth=4)
plt.plot(x,pdf_max(x),linewidth=4)
```

Probability - Minimum and Maximum of Exponentially Distributed Random Variables.ipynb


## Calculation of exponent function using Maclaurin Series for x = 1
\begin{align}
e^x = \sum\limits_{n=0}^{\infty}\frac{x^n}{n!}
\end{align}

```python
# importing dependency functions
from math import exp as ideal_exp
from matplotlib import pyplot as plt

# initial guess for iteration number
iter_num = 25

# implementation of factorial function
def custom_factorial(n):
#     base case to stop recursion
    if n < 1: return 1
#     general case to compute factorial
    return n * custom_factorial(n-1)

# implementation of power function
def custom_power(base, degree):
#     base case to stop recursion
    if degree < 1: return 1
#     general case to compute power
    return base * custom_power(base, degree - 1)

# recursive implementation of exponential function
def custom_exp_recursive(x, counter=0, limit = iter_num):
#     base case to stop recursion
    if counter == limit - 1: return custom_power(x, counter)/custom_factorial(counter)
#     general case to compute exponent
    return custom_power(x, counter) / custom_factorial(counter) + custom_exp_recursive(x, counter + 1, limit)

# loop implementation of exponential function to analyze error function
def custom_exp_analysis(x, iter_num=iter_num):
#     computing math library exponent as reference
    ideal = ideal_exp(x)
#     initialization of final result and error vector
    result = 0
    error_vec = [ideal]
#     Maclaurin iteration loop
    for i in range(iter_num):
        result += custom_power(x, i)/custom_factorial(i)
        error_vec.append((ideal - result) / ideal)
    return result, error_vec

print("Calculated exponent value is {} using {} iterations of Maclaurin series".format(custom_exp_recursive(1), iter_num))
```

    Calculated exponent value is 2.718281828459045 using 25 iterations of Maclaurin series

## True relative error analysis.

After each iteration term is added, the relative error is computed and plotted

```python
# plotting true error function
plt.plot(custom_exp_analysis(1)[1])
plt.ylabel('True error')
plt.xlabel('# of iterations')
plt.grid(color='b', linestyle='--', linewidth=0.5)
plt.show()
```

## Precision test

Suppose that $10^-15$ is a principal precision for custom exponent function. How many Maclaurin terms is needed in order to reach the given precision?

```python
# zoom plotting to explore precision
plt.plot(custom_exp_analysis(1)[1])
plt.ylabel('True relative error')
plt.xlabel('# of iterations')
plt.xlim(15, 25)
plt.ylim(-1e-15, 1e-15)
plt.grid(color='b', linestyle='--', linewidth=0.5)
plt.show()
```

From above plot it is clearly seen that 17 iterations are enough to reach e-15 precision, since error function fits into e-15 precision boundaries.

## Truncation error test

After 19th iteration, result is not improving anymore. Why?

Increasing number of iterations has no any effect to error function anymore. To understand causes let's analyze 19th cycle process. 19th Maclaurin element is calculated as 1/19! and added to global result. Let's see what we get by dividing 1 to 19!.

```python
from decimal import Decimal
print(Decimal(1/custom_factorial(19)))
```

    8.2206352466243294955370400408296422011147285715637438030523043153152684681117534637451171875E-18

It is not 0, so cause is not from this term.

```python
small_term = 1/custom_factorial(19)
added_something = 0.99 + small_term
print(Decimal(added_something))
```

    0.9899999999999999911182158029987476766109466552734375

Something strange happens when we add 0.99 to 19th Maclauring series. Resulting value becomes less than 0.99.

```python
added_something = 2599999999999999990000.999999 + small_term
print(Decimal(added_something))
```

    2600000000000000000000

Even 10000 difference is rounded here to obtain number without fractions.

```python
print(type(small_term))
```

    <class 'float'>

In Python there is 53 bits of precision available for floating numbers.

### Precision tests using x = 2 and x = 10

```python
# plotting true error function
plt.plot(custom_exp_analysis(2)[1])
plt.ylabel('True relative error')
plt.xlabel('# of iterations')
plt.grid(color='b', linestyle='--', linewidth=0.5)
plt.show()
```

```python
# zoom plotting to explore precision
plt.plot(custom_exp_analysis(2, iter_num=25)[1])
plt.ylabel('True relative error')
plt.xlabel('# of iterations')
plt.xlim(15, 25)
plt.ylim(-1e-15, 1e-15)
plt.grid(color='b', linestyle='--', linewidth=0.5)
plt.show()
```

This time, we could reach e-15 accuracy in 22 iterations.

```python
# plotting true error function
plt.plot(custom_exp_analysis(10)[1])
plt.ylabel('True relative error')
plt.xlabel('# of iterations')
plt.grid(color='b', linestyle='--', linewidth=0.5)
plt.show()
```

```python
# zoom plotting to explore precision
plt.plot(custom_exp_analysis(10, iter_num=50)[1])
plt.ylabel('True error')
plt.xlabel('# of iterations')
plt.xlim(40, 50)
plt.ylim(-1e-15, 1e-15)
plt.grid(color='b', linestyle='--', linewidth=0.5)
plt.show()
```

In the case when x = 10, error function converged to e-15 precision at 45th Maclaurin iteration.

### Conclusion about approximations

Most decimal fractions cannot be represented exactly as binary fractions. Therefore, in general, the decimal floating-point numbers are only approximated by the binary floating-point numbers actually stored in the machine. For instance 1/3 fraction will be stored in bits in following way 0.0001100110011001100110011001100110011001100110011... and it consumes all bits allocated for current variable. To save space, Python stops its binary iterations at some step to approximate value and save bits. This may be a reason of such strange behaviour of floating number operations and early approximations.

```python

```


# Number theory and a Google recruitment puzzle

## Find the first 10-digit prime in the decimal expansion of 17π

Description: The first 5 digits in the decimal expansion of π are 14159. The first 4-digit prime in the decimal expansion of π are 4159. You are asked to find the first 10-digit prime in the decimal expansion of 17π.

There are three main steps to this question.
    1) Generate an arbitrary large expansion of a mathematical expression
    2) Check if a number is prime
    3) Generate sliding windows of a specified width from a long iterable

We will start with the first step. The goal is to return an arbitrary expansion (expansion after the decimal) of a mathematical expression. The user inputs will be the mathematical expression along with the multiplier (can be 1) and the number of digits of the expansion.

I first tried using the decimal library to expand the mathematical expression "pi", but I realized I was approximating pi by using the expression "355/113" and I could not find a more accurate way to expand pi, so I opted to use the sympy library. I used the decimal library to expand "e".

One important thing to note is that a multiplier of the mathematical expression can be an input like "2e" or "3pi". The multiplier must be a separate input from the expression.

```python
from decimal import *
from sympy import *

def expansion(multiplier, expression, n):
    """
    Input: mathematical expression, a multiplier of the expression, and number of digits of prime number
    Behavior: expands mathematical expression
    Output: an arbitrary expansion (specified by user) of a mathematical expressions pi or e
    """
    if expression == "pi":
        # N function captures number of digits, so use log formula to capture number of specified digits after decimal
        x = N(multiplier*pi, n + log(multiplier*pi, 10) + 1)
        # obtain digits after decimal and return as integer
        # integers that start with 0 will not capture the 0
        before, after = str(x).split('.')
        return int(after)

    if expression == "e":
        # set number of digits for expansion
        getcontext().prec = n
        x = (Decimal(1).exp())*multiplier
        before, after = str(x).split('.')
        return int(after)

    # can only generate expansions of pi and e
    else:
        return "Can not solve"
```

The next step is to determine if a number is prime. To make this function efficient, we can loop through divisors in the range from 2 to the square root of the number rounded up using the ceiling function from the math module and check whether that number is a factor. The mathematical intution behind this is that a number that is not prime will have at least one factor that is less than its square root (and one factor greater). So, if a number has no factors less than its square root, it must be prime. For example, let's take the number 11. The square root of 11 rounded up is 4, so the loop will run from 2 to 4. If there are no factors in that range, that 11 is prime, which is the case. I used this mathematical intuition because looping through all numbers in the range to the specified number is very inefficient, especially because we are working with such large numbers in this question.

```python
from math import sqrt, ceil

def prime_num(n):
    """
    Input: number
    Behavior: returns true if number is prime
    Output: true or false
    """
    # prime numbers have to be greater than 1
    if n > 1:
        # loop through numbers in range 2 to the ceiling of the square root of the specified number
        for number in range(2, ceil(sqrt(n))):
            # return to outer for loop and increment by one if input number is not divisible by number
            if (n % number) == 0:
                # number is not prime (divisible by a number other than one and itself)
                return False
        else:
            # number is prime, so return true
            return True
    else:
        # return false if number is not greater than 1
        return False
```

The next step is to write the final helper function. The goal is to generate sliding windows of a specified width from a long iterable.

I tried a lot of different things to generate sliding windows including using itertools, but decided that using list comprehensions would be equally succinct in accomplishing this task.

First, we will make the iterable (input number) into a list of its digits as strings. For example, 1234 becomes ['1','2','3','4']. Then, we will create a window of a specified size for each element (digit) in the range of the number until a window of the specified size cannot be made. For example, if we want a window of size 3, we will loop through all the elements in range of the length of the list of digits minus the size of the window plus one. So, using our example, the loop will run from '1' to '2'. For each iteration, the window will be appeneded as a list to a list of windows. So, from our example, [['1','2','3'],['2','3','4']] will be the output from the loop. Next, we want to return a list of integers, so we will combine the digits into one number and convert to an integer. From our example, the output would be [123, 234].

```python
def sliding_window(iterable, size):
    """
    Input: integer and size of window
    Behavior: generates sliding windows of a specified width from a long iterable
    Output: list of windows
    """
    window_list = []

    # length of number has to be greater than the window size
    if len(str(iterable)) >= size:
        # make iterable into list of its digits
        it_list = [str(x) for x in str(iterable)]

        # create window as a list by grabbing elements in range of current element through the elements in the window size
        windows = [it_list[x:x+size] for x in range(len(it_list) - size + 1)]

        # convert list of digits in window to a single integer and append to list
        for window in windows:
            a_string = "".join(window)
            an_integer = int(a_string)
            window_list.append(an_integer)

        # returns list of integers
        return window_list
    else:
        return "Size of window bigger than iterable"
```

The final step is to create a function with these helper functions to return a specified digit-length prime in the decimal expansion of a mathematical expression like "pi". First, we will create the decimal expansion using the expansion helper function. Then, we will create the list of sliding windows given the size of the prime number we want using the sliding windows helper function. Finally, we will check whether each number in each window is prime using the prime helper function and return the first prime number.

```python
def prime_expanded_expression(size, multiplier, expression, digits):
    """
    Input: size of window, multiplier of expression, mathematical expression, and number of digits to expand
    Behavior: returns a specified digit-length prime in the decimal expansion of a mathematical expression
    Output: number
    """
    # create decimal expansion of mathematical expression
    expanded = expansion(multiplier, expression, digits)
    # create list of sliding windows of decimal expansion given size of window
    expansion_window = sliding_window(expanded, size)
    # check whether each window (number) is a prime and if it is, return that number
    for number in expansion_window:
        # numbers that are less than a length of 10 start with 0, so cannot be prime
        if len(str(number)) == 10:
            # check whether number is prime
            if prime_num(number):
                return number
```

Now, we will write unit tests for each function to assert that they are returning the correct values and to check edge cases. Two tests are written for each function.

```python
import unittest

class TestNotebook(unittest.TestCase):

    def test_expansion(self):
        """test expansion."""
        self.assertEqual(expansion(1, "pi", 5), 14159)
        self.assertEqual(expansion(17, "pi", 9), 407075111)

    def test_prime_num(self):
        """test prime_num."""
        self.assertFalse(prime_num(1))
        self.assertTrue(prime_num(4159))

    def test_sliding_window(self):
        """test sliding_window."""
        self.assertEqual(sliding_window(407075111, 4), [4070, 707, 7075, 751, 7511, 5111])
        self.assertEqual(sliding_window(1234, 5), "Size of window bigger than iterable")

    def prime_expanded_expression(self):
        """test prime_expanded_expression."""
        self.assertEqual(prime_expanded_expression(10, 1, "e", 110), 7427466391)
        self.assertEqual(prime_expanded_expression(4, 1, "pi", 110), 4159)

unittest.main(argv=[''], verbosity=2, exit=False)
```

    test_expansion (__main__.TestNotebook)
    test expansion. ... ok
    test_prime_num (__main__.TestNotebook)
    test prime_num. ... ok
    test_sliding_window (__main__.TestNotebook)
    test sliding_window. ... ok

    ----------------------------------------------------------------------
    Ran 3 tests in 0.015s

    OK

    <unittest.main.TestProgram at 0x7f210dc81650>

Finally, let's solve the problem! The first 10-digit prime in the decimal expansion of 17π is 8649375157.

```python
print(prime_expanded_expression(10, 17, "pi", 110))
```

    8649375157


# Poisson's problem

## General formulation

Here I provide the "Hello, world!" of FEM. The Poisson's problem subject to Dirichlet boundary conditions is stated as:

Find $u \in \mathbb{C}^2(\overline{\Omega})$ in a closed domain $\overline{\Omega}\subset \mathbb{R}^n\,(n=1,2,3)$  such that:

\begin{equation}
\left\{
\begin{aligned}
- &\Delta u = f(x), \quad \forall x \in \Omega \\
&\left. u \right|_{\partial \Omega} = g(x)
\end{aligned}
\right.
\end{equation}
where $\overline{\Omega} := \Omega \cup \partial \Omega$ denotes the closed domain, $\Omega$ is the open domain and $\partial \Omega$ is the domain's boundary (the closure). Note that $\Omega \cap \partial \Omega = \emptyset$. The solution is a scalar field such that $u: \overline{\Omega} \to \mathbb{R}$. Also, we have the "source term" that is a scalar field of the same form, $f: \overline{\Omega} \to \mathbb{R}$, but this function is a given data to the (direct) problem. The $g(x)$ is a prescribed function applied to the boundary, which is also a given data to the problem.

P.S.: I will not discuss the ill problems formulation, neither others mathematical requirements to assert the well-posedness of the problem.

## 1D simplification

In some scenarios, the mathematical modeling approach can be simplified to the 1D case, which is written as follows:

Find $u \in \mathbb{C}^2$ such that:

\begin{equation}
\left\{
\begin{aligned}
- &u'' = f(x), \quad \forall x \in ((x_1, x_2) \subset \mathbb{R}) \\
&u(x_1) = u_1 \\
&u(x_2) = u_2
\end{aligned}
\right.
\end{equation}

where $u_1, u_2 \in \mathbb{R}$ are the prescribed boundary condition values, $u'(x) \equiv \dfrac{d u}{d x}$ and thus $u''(x) \equiv \dfrac{d^2 u}{d x^2}$. For the sake of simplicity, I will omit the argument of $u$ function. Such aforementioned problems are known as Two-Point Boundary Value problems.

### Variational formulation and Galerkin approximation

One can find the derivation of the weak form, but I will just give its result below.

Given the space of admissible solution:

\begin{equation}
\mathcal{U} (\overline{\Omega}) := \left\{ u \in H^1 (\overline{\Omega}) \left| \,u(x_1) = u_1, u(x_2) = u_2 \right. \right\}
\end{equation}

and the space of suitable variations as

\begin{equation}
\mathcal{V} (\overline{\Omega}) := \left\{ v \in H^1 (\overline{\Omega}) \left| \,u(x_1) = u(x_2) = 0 \right. \right\}
\end{equation}

The discretization of the above spaces is performed by Galerkin approximation with Lagrangean function space $\mathbb{P}$ of order $k$ defined over $\overline{\Omega}$. Thus we define here:

\begin{equation}
            \mathcal{S}_h^k(\overline{\Omega}) := \left\{ \varphi_h \in \mathcal{C}(\Omega^e): \left.\varphi\right|_{\Omega^e}
            \in \mathbb{P}_k(\Omega^e), \forall \Omega^e \in \mathcal{T}_h \right\}
\end{equation}

the space of Lagrange polynomials subject to a domain partition $\mathcal{T}_h:=\cup \Omega^e \approx \overline{\Omega}$ and $\cap \Omega^e = \emptyset$ (non-overlapping elements). Thus

\begin{equation}
    \mathcal{U}_h := \mathcal{U} \cap \mathcal{S}_h^k \quad \text{and} \quad
    \mathcal{V}_h := \mathcal{V} \cap \mathcal{S}_h^k
\end{equation}

we have the discretized spaces.

Now, we can stated out our "discretized" Variational Formulation as:

Find $u_h \in \mathcal{U}_h$ such that

\begin{equation}
a(u_h, v_h) = F(v_h), \quad \forall v_h \in \mathcal{V}_h
\end{equation}

where

\begin{align}
a(u, v) &:= \int_{\Omega} u' v' dx \\
F(v) &:= \int_{\Omega} f v dx
\end{align}

and the domain is $\overline{\Omega} \equiv [x_1, x_2]$.

### A practical FEniCS example

Solve this in FEniCS is very straightforward compared to classical FEM codes. The FEniCS framework provides a high-level interface which eases the pain at most.

Just for learning purpose, let the given data be setted as:

\begin{align*}
&f(x) \equiv f = 1 \\
&u_1 = u_2 = 0 \\
&x_1 = 0, \quad x_2 = 1
\end{align*}

With the source term as $f(x) = 1$, the exact solution can be easily obtained:

\begin{equation}
u(x) = \frac{1}{2}\left(x_1^2 - x^2\right) + \left[\frac{1}{2}(x_1 + x_2) + \frac{u_1 - u_2}{x_1 - x_2} \right] (x - x_1) + u_1
\end{equation}

So, how to solve the problem with FEniCS? We will construct the procedures stepwisely.

* Importing all the libs we'll need.

```python
from fenics import *  # all the FEniCS namespace (not a recommended Python practice)
import matplotlib.pyplot as plt
from matplotlib import rc
import numpy as np
```

* Defining the domain and related mesh

```python
x_left = 0.0
x_right = 1.0
numel = 15
mesh = IntervalMesh(numel, x_left, x_right)  # IntervalMesh(num_of_elements, inf_interval, sup_interval)
mesh_ref = IntervalMesh(100, x_left, x_right)
```

* Setting the degree of the functions in the Continuous Galerkin method and the variation space. Additionaly, we define here a space to be employed in the projection of the reference analytical solution

```python
p = 1
V = FunctionSpace(mesh, "CG", p)  # "CG" stands for Continuous Galerkin, p is the degree
Vref = FunctionSpace(mesh_ref, "CG", p)
```

* Defining a Python function which marks the boundaries

```python
def left(x, on_boundary):
    return x < 0+DOLFIN_EPS
def right(x, on_boundary):
    return x > 1-DOLFIN_EPS
```

* Setting the prescribed boundary values

```python
u1, u2 = 0.0, 0.0
g_left = Constant(u1)
g_right = Constant(u2)
```

* Now we modify the spaces, as expected

```python
bc_left = DirichletBC(V, g_left, left)
bc_right = DirichletBC(V, g_right, right)
dirichlet_condition = [bc_left, bc_right]
```

* Here we define the source function over the domain

```python
f = Constant(1)
```

* Now comes the good part. We set up the Trial and Test functions from the admissible space

```python
u_h = TrialFunction(V)
v_h = TestFunction(V)
```

* Then we write the bilinear form, very much like it is written in mathematical form

\begin{equation}
a(u_h, v_h) \equiv \left(u_h', v_h'\right) = \int_{\Omega} u_h' v_h' dx
\end{equation}

```python
a = inner(grad(v_h), grad(u_h))*dx
```

* Also we define the associated linear form

\begin{equation}
L(v) := (f, v_h) = \int_{\Omega} f v_h dx
\end{equation}

```python
L = f*v_h*dx
```

* Now we declare the solution variable, which means that `u_sol` is a function over the space $\mathcal{V}$

```python
u_sol = Function(V)
```

* Thus we set the discretized variational problem to be solved. The problem is declared with the FEniCS function `LinearVariationalProblem`, that has as arguments the LHS, the RHS, the variable where the solution will be stored and computed and the essential boundary conditions to be considered

```python
problem = LinearVariationalProblem(a, L, u_sol, dirichlet_condition)
```

* And we go further and solve it with no difficulties!

```python
solver = LinearVariationalSolver(problem)
solver.solve()
```

* But we will need to check if the solution is "good enough". So we compare with the available exact solution

```python
sol_exact = Expression(
    "(1. / 2.) * (pow(x_1,2.0) - pow(x[0],2.0)) + \
    ((1./2.) * (x_1 + x_2) + (u_1 - u_2) / (x_1 - x_2)) * (x[0] - x_1) + u_1",
    degree=p+1,
    u_1=u1,
    u_2=u2,
    x_1=x_left,
    x_2=x_right
)
u_e = interpolate(sol_exact, Vref)
```

* Now, lets plot our results!

```python
plot(u_sol, marker='x', label='Approx')
plot(u_e, label='Exact')
# Setting the font
plt.rc('text',usetex=True)
plt.rc('font', size=14)
# Plotting
plt.xlim(x_left, x_right)   # Limites do eixo x
plt.ylim(np.min(u_sol.vector().get_local()), 1.02*np.max(u_e.vector().get_local()))  # Limites do eixo y
plt.grid(True, linestyle='--')  # Ativa o grid do grafico
plt.xlabel(r'$x$')  # Legenda do eixo x
plt.ylabel(r'$u(x)$')   # Legenda do eixo y
plt.legend(loc='best',borderpad=0.5)    # Ativa legenda no grafico e diz para se posicionar na melhor localizacao detectada
plt.show()  # Exibe o grafico em tela
```


# Scientific Computing with Python

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

```python
# what is this line all about?
%matplotlib inline
import matplotlib.pyplot as plt
```

## Numpy and Scipy
### Introduction
The numpy package (module) is used in almost all numerical computation using Python. It is a package that provide high-performance vector, matrix and higher-dimensional data structures for Python. It is implemented in C and Fortran so when calculations are vectorized (formulated with vectors and matrices), performance is very good.

The SciPy framework builds on top of the low-level NumPy framework for multidimensional arrays, and provides a large number of higher-level scientific algorithms.

### Fitting to polynomial

```python
import numpy as np
```

```python
np.random.seed(12)

x = np.linspace(0, 1, 20)
y = np.cos(x) + 0.3*np.random.rand(20)
p = np.poly1d(np.polyfit(x, y, 16))

t = np.linspace(0, 1, 200)
plt.plot(x, y, 'o', t, p(t), '-')
plt.show()
```

### Fit in a Chebyshev basis

```python
np.random.seed(0)

x = np.linspace(-1, 1, 2000)
y = np.cos(x) + 0.3*np.random.rand(2000)
p = np.polynomial.Chebyshev.fit(x, y, 90)

t = np.linspace(-1, 1, 200)
plt.plot(x, y, 'r.')
plt.plot(t, p(t), 'k-', lw=3)
plt.show()
```

### A demo of 1D interpolation

```python
np.random.seed(0)
measured_time = np.linspace(0, 1, 10)
noise = 1e-1 * (np.random.random(10)*2 - 1)
measures = np.sin(2 * np.pi * measured_time) + noise

# Interpolate it to new time points
from scipy.interpolate import interp1d
linear_interp = interp1d(measured_time, measures)
interpolation_time = np.linspace(0, 1, 50)
linear_results = linear_interp(interpolation_time)
cubic_interp = interp1d(measured_time, measures, kind='cubic')
cubic_results = cubic_interp(interpolation_time)

# Plot the data and the interpolation
from matplotlib import pyplot as plt
plt.figure(figsize=(6, 4))
plt.plot(measured_time, measures, 'o', ms=6, label='measures')
plt.plot(interpolation_time, linear_results, label='linear interp')
plt.plot(interpolation_time, cubic_results, label='cubic interp')
plt.legend()
plt.show()
```

### Minima and roots of a function
\begin{equation}
    f(x) = x^2 + 10\sin(x)
\end{equation}

**(1) find minima**

```python
def f(x):
    return x**2 + 10*np.sin(x)

from scipy import optimize

# Global optimization
grid = (-10, 10, 0.1)
xmin_global = optimize.brute(f, (grid, ))
print("Global minima found %s" % xmin_global)

# Constrain optimization
xmin_local = optimize.fminbound(f, 0, 10)
print("Local minimum found %s" % xmin_local)
```

    Global minima found [-1.30641113]
    Local minimum found 3.8374671194983834

**(2) root finding**

```python
root = optimize.root(f, 1)  # our initial guess is 1
print("First root found %s" % root.x)
root2 = optimize.root(f, -2.5)
print("Second root found %s" % root2.x)
```

    First root found [0.]
    Second root found [-2.47948183]

**(3) Plot function, minima, and roots**

```python
fig = plt.figure(figsize=(6, 4))
ax = fig.add_subplot(111)

x = np.arange(-10, 10, 0.1)
# Plot the function
ax.plot(x, f(x), 'b-', label="f(x)")

# Plot the minima
xmins = np.array([xmin_global[0], xmin_local])
ax.plot(xmins, f(xmins), 'go', label="Minima")

# Plot the roots
roots = np.array([root.x, root2.x])
ax.plot(roots, f(roots), 'kv', label="Roots")

# Decorate the figure
ax.legend(loc='best')
ax.set_xlabel('x')
ax.set_ylabel('f(x)')
ax.axhline(0, color='gray')
plt.show()
```

## Matplotlib
### Introduction
Matplotlib is an excellent 2D and 3D graphics library for generating scientific figures.

### Reading and writing a panda

**(1) original figure**

```python
plt.figure()
img = plt.imread('../data/panda.jpg')
plt.imshow(img)
plt.imsave("original.jpg",img)

print (np.shape(img))
```

**(2) red channel displayed in grey**

```python
plt.figure()
img_red = img[:, :, 0]
plt.imshow(img_red, cmap=plt.cm.gray)
```

**(3) lower resolution (compression)**

```python
plt.figure()
img_tiny = img[::8, ::8]
plt.imshow(img_tiny, interpolation='nearest')
#plt.savefig("compressed.jpg")
plt.imsave("compressed.jpg",img_tiny)
```

### Mandlebrot Set (Mandelbrot fractal)

```python
def compute_mandelbrot(N_max, some_threshold, nx, ny):
    # A grid of c-values
    x = np.linspace(-2, 1, nx)
    y = np.linspace(-1.5, 1.5, ny)

    c = x[:,np.newaxis] + 1j*y[np.newaxis,:]

    # Mandelbrot iteration

    z = c
    for j in range(N_max):
        z = z**2 + c

    mandelbrot_set = (abs(z) < some_threshold)

    return mandelbrot_set

mandelbrot_set = compute_mandelbrot(50, 50., 601, 401)

plt.imshow(mandelbrot_set.T, extent=[-2, 1, -1.5, 1.5])
plt.gray()
plt.show()
```

### A simple example of 3D plotting
$$ z = \sin(\sqrt{x^2 + y^2}) $$

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = Axes3D(fig)
X = np.arange(-4, 4, 0.25)
Y = np.arange(-4, 4, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X ** 2 + Y ** 2)
Z = np.sin(R)

ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=plt.cm.hot)
ax.contourf(X, Y, Z, zdir='z', offset=-2, cmap=plt.cm.hot)
ax.set_zlim(-2, 2)

plt.show()
```

### An example displaying the contours of a function
$$ f(x,y) = \left(1 - \frac{x}{2} + x^5 + y^3\right)e^{-x^2-y^2}.$$

```python
def f(x,y):
    return (1 - x / 2 + x**5 + y**3) * np.exp(-x**2 -y**2)

n = 256
x = np.linspace(-3, 3, n)
y = np.linspace(-3, 3, n)
X,Y = np.meshgrid(x, y)

plt.axes([0.025, 0.025, 0.95, 0.95])

plt.contourf(X, Y, f(X, Y), 8, alpha=.75, cmap=plt.cm.hot)
C = plt.contour(X, Y, f(X, Y), 8, colors='black')
plt.clabel(C, inline=1, fontsize=10)

plt.xticks(())
plt.yticks(())
plt.show()
```

## Sympy - Symbolic algebra in Python

### Introduction
There are two notable Computer Algebra Systems (CAS) for Python:

* [SymPy](http://sympy.org/en/index.html) - A python module that can be used in any Python program, or in an IPython session, that provides powerful CAS features.
* [Sage](http://www.sagemath.org/) - Sage is a full-featured and very powerful CAS enviroment that aims to provide an open source system that competes with Mathematica and Maple. Sage is not a regular Python module, but rather a CAS environment that uses Python as its programming language.

Sage is in some aspects more powerful than SymPy, but both offer very comprehensive CAS functionality. The advantage of SymPy is that it is a regular Python module and integrates well with the Jupyter notebook.

```python
from sympy import *

init_printing()
```

### Expand, factor and simplify

```python
x, y = symbols('x y')

(x+1)*(x+2)*(x+3)*(x+4)*(x+5)
```

```python
expand((x+1)*(x+2)*(x+3)*(x+4)*(x+5))
```

```python
sin(x+y)
```

```python
expand(sin(x+y), trig=True)
```

```python
expand((x+y)**8)
```

```python
x**3 + 6 * x**2 + 11*x + 6
```

```python
factor(x**3 + 6 * x**2 + 11*x + 6)
```

```python
sin(x)**2 + cos(x)**2
```

```python
simplify(sin(x)**2 + cos(x)**2)
```

```python
cos(x)/sin(x)
```

```python
simplify(cos(x)/sin(x))
```

### Calculus
**(1) differentiation and integration**

$f(x) = (x+1)^2$

```python
f = (x+1)**2
f
```

Computing $\frac{d f}{dx}$, $\frac{d f^2}{dx}$

```python
diff(f,x)
```

```python
diff(f**2,x)
```

Computing $\frac{d \sin(f)}{dx}$, $\frac{d^2 \sin(f)}{dx^2}$

```python
diff(sin(f),x)
```

```python
diff(sin(f),x,2)
```

```python
diff(sin(f),x,4)
```

$$ f(x,y) = \sin(xy) + \cos(xy),$$
computing
$$ \frac{\partial^3 f}{\partial x \partial y^2},\quad \int f(x,y)\,dx,\quad \int_{-1}^{1}f(x,y)\,dx$$

```python
f = sin(x*y) + cos(y*x)
f
```

```python
diff(f, x, 1, y, 2)
```

```python
integrate(f, x)
```

```python
integrate(f, (x, -1, 1))
```

Computing $\int_{-\infty}^\infty e^{-x^2}\,dx$

```python
integrate(exp(-x**2), (x, -oo, oo))
```

**(2) limits**
$$ \lim\limits_{x\rightarrow 0}\frac{\sin(x)}{x},\quad \lim\limits_{x\rightarrow 0^{+}}\frac{1}{x},\quad \lim\limits_{x\rightarrow 0^{-}}\frac{1}{x}$$

```python
limit(sin(x)/x, x, 0)
```

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

**(3) series**

```python
exp(x)
```

```python
series(exp(x), x)
```

```python
series(exp(x), x, 1)
```

```python
series(sin(x), x, 0, 12)
```

```python
series(sin(x)*cos(x), x, 0, 8)
```

```python
series(sin(x)*cos(x)*exp(x), x, 0, 12)
```

### Linear algebra: Matrices

```python
m11, m12, m21, m22 = symbols("m11, m12, m21, m22")
b1, b2 = symbols("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]])
A
```

```python
b = Matrix([[b1], [b2]])
b
```

```python
A**2
```

```python
A**5
```

```python
A * b
```

```python
A.det()
```

```python
A.inv()
```

### Solving equations
Solving
$$ x^2 - 1 = 0,\quad x^4 - x^2 - 1 = 0$$

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 + x**3 - x**2 - 1, x)
```

Solving systems:
$$ x + y - 1 = 0,\quad x - y - 1 = 0,$$
and
$$ x + y - a = 0,\quad x - y - b = 0.$$

```python
solve([x + y - 1, x - y - 1], [x,y])
```

```python
a, b = symbols('a, b')
solve([x + y - a, x - y - b], [x,y])
```

```python

```


# Appendix B.2 $\quad$ Complex Numbers in Linear Algebra

### Example 1

Solve the following linear system
\begin{eqnarray*}
% \nonumber to remove numbering (before each equation)
  (1+i)x_1 + (2+i)x_2 &=& 5, \\
  (2-2i)x_1 +ix_2 &=& 1+2i.
\end{eqnarray*}

```python
from sympy import *

x1, x2 = symbols('x1 x2');
Eq1 = (1+1j)*x1 + (2+1j)*x2 - 5;
Eq2 = (2-2j)*x1 + 1j*x2 - (1+2j);

solve([Eq1, Eq2], (x1, x2))
```

    {x2: 2.0 - 1.0*I, x1: 0.0}

### Example 2

Find the determinant of the coefficient matrix in Example 1.

```python
from sympy import *

A = Matrix([[1+1j, 2+1j], [2-2j, 1j]]);

A.det()
```

    -7.0 + 3.0*I

### Example 3

Find the eigenvalues and eigenvector of
\begin{equation*}
  A =
  \left[
    \begin{array}{cc}
      1 & 1 \\
      -1 & 1 \\
    \end{array}
  \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[1, 1], [-1, 1]]);

A.eigenvects()
```

    [(1 - I, 1, [Matrix([
       [I],
       [1]])]), (1 + I, 1, [Matrix([
       [-I],
       [ 1]])])]

### Example 4

Find the eigenvalues and eigenvector of
\begin{equation*}
  A = \left[
        \begin{array}{ccc}
          2 & 0 & 0\\
          0 & 2 & i \\
          0 & -i & 2\\
        \end{array}
      \right]
\end{equation*}

```python
from sympy import *

A = Matrix([[2, 0, 0], [0, 2, 1j], [0, -1j, 2]]);

A.eigenvects()
```

    [(1.00000000000000, 1, [Matrix([
       [     0],
       [-1.0*I],
       [   1.0]])]), (2.00000000000000, 1, [Matrix([
       [1.0],
       [  0],
       [  0]])]), (3.00000000000000, 1, [Matrix([
       [    0],
       [1.0*I],
       [  1.0]])])]

**Remark:** If $A$ is a Hermitian matrix (i.e., $\overline{A^T} = A$), then


## Maximum Likelihood Estimation

Maximum likelihood estimation is one of the key techniques employed in statistical signal processing for a wide variety of applications from signal detection to parameter estimation. In the following, we consider a simple experiment and work through the details of maximum likelihood estimation to ensure that we understand the concept in one of its simplest applications.

### Setting up the Coin Flipping Experiment

Suppose we have coin and want to estimate the probability of heads ($p$) for it. The coin is Bernoulli distributed:

$$ \phi(x)= p^x (1-p)^{(1-x)} $$

where $x$ is the outcome, *1* for heads and *0* for tails. The $n$ independent flips, we have the likelihood:

$$ \mathcal{L}(p|\mathbf{x})= \prod_{i=1}^n p^{ x_i }(1-p)^{1-x_i} $$

This is basically notation. We have just substituted everything into $ \phi(x)$ under the independent-trials assumption.

The idea of *maximum likelihood* is to maximize this as the function of $p$ after plugging in all of the $x_i$ data. This means that our estimator, $\hat{p}$ , is a function of the observed $x_i$ data, and as such, is a random variable with its own distribution.

### Simulating the Experiment

We need the following code to simulate coin flipping.

```
%matplotlib inline
from __future__ import division
from scipy.stats import bernoulli
import numpy as np

p_true=1/2 # this is the value we will try to estimate from the observed data
fp=bernoulli(p_true)

def sample(n=10):
    'simulate coin flipping'
    return fp.rvs(n)# flip it n times

xs = sample(100) # generate some samples
```

Now, we can write out the likelihood function using `sympy`

```
import sympy
from sympy.abc import x, z
p=sympy.symbols('p',positive=True)

L=p**x*(1-p)**(1-x)
J=np.prod([L.subs(x,i) for i in xs]) # objective function to maximize
```

Below, we find the maximum using basic calculus. Note that taking the `log` of $J$ makes the maximization problem tractable but doesn't change the extrema.

```
logJ=sympy.expand_log(sympy.log(J))
sol=sympy.solve(sympy.diff(logJ,p),p)[0]

x=linspace(0,1,100)
plot(x,map(sympy.lambdify(p,logJ,'numpy'),x),sol,logJ.subs(p,sol),'o',
                                          p_true,logJ.subs(p,p_true),'s',)
xlabel('$p$',fontsize=18)
ylabel('Likelihood',fontsize=18)
title('Estimate not equal to true value',fontsize=18)
```

Note that our estimator $\hat{p}$ (red circle) is not equal to the true value of $p$ (green square), but it is at the maximum of the likelihood function. This may sound disturbing, but keep in mind this estimate is a function of the random data; and since that data can change, the ultimate estimate can likewise change. I invite you to run this notebook a few times to observe  this. Remember that the estimator is a *function* of the data and is thus also a *random variable*, just like the data is.

Let's write some code to empirically examine the behavior of the maximum likelihood estimator using a simulation of multiple trials. All we're doing here is combining the last few blocks of code.

```
def estimator_gen(niter=10,ns=100):
    'generate data to estimate distribution of maximum likelihood estimator'
    out=[]
    x=sympy.symbols('x',real=True)
    L=   p**x*(1-p)**(1-x)
    for i in range(niter):
        xs = sample(ns) # generate some samples from the experiment
        J=np.prod([L.subs(x,i) for i in xs]) # objective function to maximize
        logJ=sympy.expand_log(sympy.log(J))
        sol=sympy.solve(sympy.diff(logJ,p),p)[0]
        out.append(float(sol.evalf()))
    return out if len(out)>1 else out[0] # return scalar if list contains only 1 term

etries = estimator_gen(100) # this may take awhile, depending on how much data you want to generate
hist(etries) # histogram of maximum likelihood estimator
title('$\mu=%3.3f,\sigma=%3.3f$'%(mean(etries),std(etries)),fontsize=18)
```

Note that the mean of the estimator ($\mu$) is pretty close to the true value, but looks can be deceiving. The only way to know for sure is to check if the estimator is unbiased, namely, if

$$ \mathbb{E}(\hat{p}) = p $$

Because this problem is simple, we can solve for this in general noting that since $x=0$ or $x=1$, the terms in the product of $\mathcal{L}$ above are either $p$, if $x_i=1$ or $1-p$ if $x_i=0$. This means that we can write

$$ \mathcal{L}(p|\mathbf{x})= p^{\sum_{i=1}^n x_i}(1-p)^{n-\sum_{i=1}^n x_i} $$

with corresponding log as

$$ J=\log(\mathcal{L}(p|\mathbf{x})) =  \log(p)  \sum_{i=1}^n x_i +   \log(1-p) \left(n-\sum_{i=1}^n x_i\right)$$

Taking the derivative of this gives:

$$  \frac{dJ}{dp} = \frac{1}{p}\sum_{i=1}^n x_i + \frac{(n-\sum_{i=1}^n x_i)}{p-1} $$

and solving this leads to

$$  \hat{p} = \frac{1}{ n} \sum_{i=1}^n x_i $$

This is our *estimator* for $p$. Up til now, we have been using `sympy` to solve for this based on the data $x_i$ but now we have it generally and don't have to solve for it again. To check if this estimator is biased, we compute its expectation:

$$ \mathbb{E}\left(\hat{p}\right) =\frac{1}{n}\sum_i^n \mathbb{E}(x_i) = \frac{1}{n} n \mathbb{E}(x_i) $$

by linearity of the expectation and where

$$\mathbb{E}(x_i)  = p$$

Therefore,

$$ \mathbb{E}\left(\hat{p}\right) =p $$

This means that the esimator is unbiased. This is good news. We almost always want our estimators to be unbiased. Similarly,

$$ \mathbb{E}\left(\hat{p}^2\right) = \frac{1}{n^2} \mathbb{E}\left[\left(  \sum_{i=1}^n x_i \right)^2 \right]$$

and where

$$ \mathbb{E}\left(x_i^2\right) =p$$

and by the independence assumption,

$$ \mathbb{E}\left(x_i x_j\right) =\mathbb{E}(x_i)\mathbb{E}( x_j) =p^2$$

Thus,

$$ \mathbb{E}\left(\hat{p}^2\right) =\left(\frac{1}{n^2}\right) n
\left[
p+(n-1)p^2
\right]
$$

So, the variance of the estimator, $\hat{p}$ is the following:

$$ \sigma_\hat{p}^2 = \mathbb{E}\left(\hat{p}^2\right)- \mathbb{E}\left(\hat{p}\right)^2  = \frac{p(1-p)}{n} $$

Note that the $n$ in the denominator means that the variance asymptotically goes to zero as $n$ increases (i.e. we consider more and more samples). This is good news also because it means that more and more coin flips leads to a better estimate of the underlying $p$.

Unfortunately, this formula for the variance is practically useless because we have to know $p$ to compute it and $p$ is the parameter we are trying to estimate in the first place! But, looking at $ \sigma_\hat{p}^2 $, we can immediately notice that if $p=0$, then there is no estimator variance because the outcomes are guaranteed to be tails. Also, the maximum of this variance, for whatever $n$, happens at $p=1/2$. This is our worst case scenario and the only way to compensate is with more samples (i.e. larger $n$).

All we have computed is the mean and variance of the estimator. In general, this is insufficient to characterize the underlying probability density of $\hat{p}$, except if we somehow knew that  $\hat{p}$ were normally distributed. This is where the powerful [*central limit theorem*](http://mathworld.wolfram.com/CentralLimitTheorem.html) comes in. The form of the estimator, which is just a mean estimator, implies that we can apply this theorem and conclude that  $\hat{p}$ is normally distributed. However, there's a wrinkle here: the theorem tells us that  $\hat{p}$ is asymptotically normal, it doesn't quantify how many samples $n$ we need to approach this asymptotic paradise. In our simulation this is no problem since we can generate as much data as we like, but in the real world, with a costly experiment, each sample may be precious. In the following, we won't apply this theorem and instead proceed analytically.

### Probability Density for the Estimator

To write out the full density for $\hat{p}$, we first have to ask what is the probability that the estimator will equal a specific value and the tally up all the ways that could happen with their corresponding probabilities. For example, what is the probability that

$$  \hat{p} = \frac{1}{n}\sum_{i=1}^n x_i  = 0 $$

This can only happen one way: when $x_i=0 \hspace{0.5em} \forall i$. The probability of this happening can be computed from the density

$$ f(\mathbf{x},p)= \prod_{i=1}^n \left(p^{x_i} (1-p)^{1-x_i}  \right) $$

$$ f\left(\sum_{i=1}^n x_i  = 0,p\right)= \left(1-p\right)^n  $$

Likewise, if $\lbrace x_i \rbrace$ has one $i^{th}$  value equal to one, then

$$ f\left(\sum_{i=1}^n x_i  = 1,p\right)= n p \prod_{i=1}^{n-1} \left(1-p\right)$$

where the $n$ comes from the $n$ ways to pick one value equal to one from the $n$ elements $x_i$. Continuing this way, we can construct the entire density as

$$ f\left(\sum_{i=1}^n x_i  = k,p\right)= \binom{n}{k} p^k  (1-p)^{n-k}  $$

where the term on the left is the binomial coefficient of $n$ things taken $k$ at a time. This is the binomial distribution and it's not the density for $\hat{p}$, but rather for $n\hat{p}$. We'll leave this as-is because it's easier to work with below. We just have to remember to keep track of the $n$ factor.

#### Confidence Intervals

Now that we have the full density for $\hat{p}$, we are ready to ask some meaningful questions. For example,

$$ \mathbb{P}\left( | \hat{p}-p | \le \epsilon p \right) $$

Or, in words, what is the probability we can get within $\epsilon$ percent of the true value of $p$. Rewriting,

$$ \mathbb{P}\left(  p - \epsilon p \lt \hat{p} \lt p + \epsilon p \right) = \mathbb{P}\left(  n p - n \epsilon p \lt \sum_{i=1}^n x_i   \lt n p + n \epsilon p \right)$$

Let's plug in some live numbers here for our worst case scenario where $p=1/2$. Then, if $\epsilon = 1/100$, we have

$$ \mathbb{P}\left( \frac{99 n}{100} \lt \sum_{i=1}^n x_i   \lt \frac{101 n}{100} \right)$$

Since the sum in integer-valued, we need $n> 100$ to even compute this. Thus, if $n=101$ we have

$$ \mathbb{P}\left( \frac{9999}{200} \lt \sum_{i=1}^{101} x_i   \lt \frac{10201}{200} \right) = f\left(\sum_{i=1}^{101} x_i  = 50,p\right)= \binom{101}{50} (1/2)^{50}  (1-1/2)^{101-50} = 0.079$$

This means that in the worst-case scenario for $p=1/2$, given $n=101$ trials, we will only get within 1% of the actual $p=1/2$ about 8% of the time. If you feel disappointed, that only means you've been paying attention. What if the coin was really heavy and it was costly to repeat this 101 times? Then, we would be within 1% of the actual value only 8% of the time. Those odds are terrible.

Let's come at this another way: given I could only flip the coin 100 times, how close could I come to the true underlying value with high probability (say, 95%)? In this case we are seeking to solve for $\epsilon$. Plugging in gives,

$$ \mathbb{P}\left(  50 - 50 \epsilon \lt \sum_{i=1}^{100} x_i   \lt 50 + 50 \epsilon  \right) = 0.95$$

which we have to solve for $\epsilon$. Fortunately, all the tools we need to solve for this are already in `scipy`.

```
import scipy.stats

b=scipy.stats.binom(100,.5) # n=100, p = 0.5, distribution of the estimator \hat{p}

f,ax= subplots()
ax.stem(arange(0,101),b.pmf(arange(0,101))) # heres the density of the sum of x_i

g = lambda i:b.pmf(arange(-i,i)+50).sum() # symmetric sum the probability around the mean
print 'this is pretty close to 0.95:%r'%g(10)
ax.vlines( [50+10,50-10],0 ,ax.get_ylim()[1] ,color='r',lw=3.)

```

The two vertical lines in the plot show how far out from the mean we have to go to accumulate 95% of the probability. Now, we can solve this as

$$ 50 + 50 \epsilon = 60 $$

which makes $\epsilon=1/5$ or 20%. So, flipping 100 times means I can only get within 20% of the real $p$ 95% of the time in the worst case scenario (i.e. $p=1/2$).

```
b=scipy.stats.bernoulli(.5) # coin distribution
xs = b.rvs(100) # flip it 100 times
phat = mean(xs) # estimated p

print abs(phat-0.5) < 0.5*0.20 # did I make it w/in interval 95% of the time?
```

    True

Let's keep doing this and see if we can get within this interval 95% of the time.

```
out=[]
b=scipy.stats.bernoulli(.5) # coin distribution
for i in range(500): # number of tries
    xs = b.rvs(100) # flip it 100 times
    phat = mean(xs) # estimated p
    out.append(abs(phat-0.5) < 0.5*0.20 ) # within 20%

print 'Percentage of tries within 20 interval = %3.2f'%(100*sum(out)/float(len(out) ))
```

    Percentage of tries within 20 interval = 97.20

Well, that seems to work. Now we have a way to get at the quality of the estimator, $\hat{p}$.

## Summary

In this section, we explored the concept of maximum likelihood estimation using a coin flipping experiment both analytically and numerically with the  scientific Python tool chain. There are two key points to remember. First, maximum likelihood estimation produces a function of the data that is itself a random variable, with its own statistics and distribution. Second, it's worth considering how to analytically derive the density function of the estimator rather than relying on canned packages to compute confidence intervals wherever possible. This is especially true when data is hard to come by and the approximations made in the central limit theorem are therefore harder to justify.

### References

This [IPython notebook](www.ipython.org) is available for [download](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Maximum_likelihood.ipynb). I urge you to experiment with the calculations for different parameters. As always, corrections and comments are welcome!