```
import numpy as np
import scipy as sp
import scipy.signal
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

Shannon's sampling theorem tells us the signal $x(t)$ can be exactly and uniquely reconstructed for all time from its samples $x(nT_S)$ by bandlimited interpolation:

\begin{equation}
\hat x(t) = \sum_{n=-\infty}^{+\infty} x(nT_S)h_S(t-nT_S) = (x \ast h_S)(t) \equiv x(t)
\end{equation}

where
$$
h_S(t) = \text{sinc}(tf_S)
$$

```
Ts = 1
Fs = 1/Ts

res = 10 # resolution parameter to fake continuous time
sc_zc = 10 # number of uni-directional zero crossing taken into account

t = np.arange(-sc_zc*Ts,+sc_zc*Ts,Ts/res)
h_S = np.sinc(t*Fs)

fig, ax = plt.subplots(1, 1, figsize=(13,4))
ax.plot(t, h_S)

ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['left'].set_position('zero')
ax.spines['bottom'].set_position('zero')

n = np.arange(-10,10,1)
ax.set_xticks(Ts * n)
labels = ["$%d T_S$" % y for y in n[0:]]
labels[10] = "0"
ax.set_xticklabels(labels, fontsize=12)
ax.grid()
ax.set_xlim([-7, 7]);

```

This function is the impulse response of the ideal LPF with cut off frequency set at $F_S/2$. The convolution can be interpreted as a superpositon of shifted and scaled $\text{sinc}$ functions.

```
Ts = 1
Fs = 1/Ts
res = 10 # resolution parameter to fake continuous time
sc_zc = 10 # number of uni-directional zero crossing taken into account

t = np.arange(-sc_zc*Ts,+sc_zc*Ts,Ts/res)
h_S = np.sinc(t*Fs)

# my discrete signal
x = np.array([0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0])
# x = np.sin(np.arange(10)*2*np.pi/10)
# x = np.array([0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0])

# interpolation
sampled_t = np.arange(0,len(x)*Ts,Ts)
continuous_t = np.arange(0,(len(x)-1)*Ts + Ts/res,Ts/res)
x_interp = np.zeros((len(continuous_t),))
x_interp[0::res] = x

#convolve
x_hat = np.convolve(x_interp, h_S)
x_hat = x_hat[(len(h_S)/2):-len(h_S)/2+1]

# plot
fig, axes = plt.subplots(1, 2, figsize=(16,6))
axes[0].stem(sampled_t, x)
axes[1].plot(continuous_t, x_hat)

for ax in axes:
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['left'].set_position('zero')
    ax.spines['bottom'].set_position('zero')

    n = np.arange(0,len(x),1)
    ax.set_xticks(Ts * n)
    labels = ["$%d T_S$" % y for y in n[0:]]
    labels[0] = "0"
    ax.set_xticklabels(labels, fontsize=12)
    ax.grid()
    ax.set_xlim([0, continuous_t[-1]+Ts]);
    ax.set_ylim([np.min((0,np.min((np.min(x),np.min(x_hat)))))-0.1,np.max((np.max(x),np.max(x_hat)))+0.1])
```


## Linear Algebra

Linear algebra refers to the study of linear relationships. In this class, we will cover some basic concepts of linear algebra that are needed to understand some more advanced and *practical* concepts and definitions. If you are interested in the concepts related to linear algebra and application, there is an excellent online series that covers these topics in detail

https://github.com/fastai/numerical-linear-algebra

Linear algebra is a fundamental component of machine learning, so if you are interested in using machine learning in the future go and check that class.

### Vectors

A vector is a collection of numbers. Vectors can be **row vectors** or **column vectors** depending on their orientation. In general, you can assume that a vector is a **column vector** unless otherwise stated.

```python
import numpy as np
vector_row = np.array([[1, -5, 3, 2, 4]])
vector_column = np.array([[1],
                          [2],
                          [3],
                          [4]])
print(vector_row.shape)
print(vector_column.shape)
```

    (1, 5)
    (4, 1)

The transpose ($T$) of a vector is an operation that transform a column vector into a row vector and a row vector into a column vector. If $v$ is a vector, then $v^{T}$ is the transpose.

```python
vector_row, vector_row.T
```

    (array([[ 1, -5,  3,  2,  4]]),
     array([[ 1],
            [-5],
            [ 3],
            [ 2],
            [ 4]]))

The norm of a vector is a measure of its lenght. There are many ways to measure lenght and you can use different definitions depending on the application. The most common norm is the $L_2$ norm, if $v$ is a vector, then the $L_2$ norm ($\Vert v \Vert_{2}$) is

$$
\Vert v \Vert_{2} = \sqrt{\sum_i v_i^2}
$$

This is also known as the Euclidian norm.

Others well known norms are the $L_1$ norm (or Manhattan Distance), and the $L_\infty$ norm (or infinity norm) equal to the maximum absolut value of the vector

```python
from numpy.linalg import norm
new_vector = vector_row.T
norm_1 = norm(new_vector, 1)
norm_2 = norm(new_vector, 2)
norm_inf = norm(new_vector, np.inf)
print('L_1 is: %.1f'%norm_1)
print('L_2 is: %.1f'%norm_2)
print('L_inf is: %.1f'%norm_inf)
```

    L_1 is: 15.0
    L_2 is: 7.4
    L_inf is: 5.0

The **dot product**  of two vectors is the sum of the product of the respective elements in each vector and is denoted by $\cdot$. If $v$ and $w$ are vectors, then the dot product is defined as
$$
d = v \cdot w= \sum_{i = 1}^{n} v_iw_i
$$

alternatively, the dot product can be computed as

$$
v \cdot w = \Vert v \Vert_{2} \Vert w \Vert_{2} \cos{\theta}
$$

where $\theta$ is the angle between the vectors. In the same way, the angle between two vector can be computed as

$$
\theta = cos^{-1}\left[\frac{v \cdot w }{\Vert v \Vert_{2} \Vert w \Vert_{2}}\right]
$$

```python
#lets take two vectors that are on the same direction but have different lenghts
from numpy import arccos, dot
v = np.array([[1,2]])
w = np.array([[5,10]])
theta = arccos(v.dot(w.T)/(norm(v)*norm(w)))
theta*(180/pi) #arcos return gradients, we are convering to degrees
```

    array([[8.53773646e-07]])

```python
#lets take two vectors that are on opposite directions
from numpy import arccos, dot, pi
v = np.array([[1,2]])
w = np.array([[-1,-2]])
theta = arccos(v.dot(w.T)/(norm(v)*norm(w)))
theta*(180/pi) #arcos return gradients, we are convering to degrees
```

    array([[179.99999879]])

```python
#lets take two vectors that are on orthogonal to eachother
from numpy import arccos, dot, pi
v = np.array([[1,1]])
w = np.array([[-1,1]])
theta = arccos(v.dot(w.T)/(norm(v)*norm(w)))
theta*(180/pi) #arcos return gradients, we are convering to degrees
```

    array([[90.]])

The **cross product** between two vectors, $v$ and $w$, is written $v\times w$. It is defined by

$$
v \times w = \Vert v \Vert_{2}\Vert w \Vert_{2}\sin{(\theta)}
$$

where $θ$ is the angle between the $v$ and $w$.

The geometric interpretation of the cross product is a vector perpendicular to both $v$ and $w$ with length (as measured by $L_2$) equal to the area enclosed by the parallelogram created by the two vectors.

```python
v = np.array([[0, 2, 0]])
w = np.array([[3, 0, 0]])
cross = np.cross(v, w)
print(cross)
```

    [[ 0  0 -6]]

```python
arccos(v.dot(cross.T)/(norm(v)*norm(cross)))*(180/pi)
```

    array([[90.]])

```python
arccos(w.dot(cross.T)/(norm(w)*norm(cross)))*(180/pi)
```

    array([[90.]])

### Matrices

An $n \times m $ matrix is a rectangular table of numbers consisting of $m$ rows and $n$ columns.

The norm of a matrix can be consider as a kind of vector norm by alingming the $n * m$ elements of the matrix into a single vector
$$
\Vert M \Vert_{p} = \sqrt[p]{(\sum_i^m \sum_j^n |a_{ij}|^p)}
$$

where $p$ defines the norm order ($p=0, 1, 2,...$)

**Matrix multiplication** between two matrices, $P$ and $Q$, is defined when $P$ is an $m \timed p$ matrix and $Q$ is a $p \times n$ matrix. The result of $M=PQ$ is a matrix $M$ that is $m \times n$. The dimension with size $p$ is called the inner matrix dimension, and the inner matrix dimensions must match (i.e., the number of columns in $P$ and the number of rows in $Q$ must be the same) for matrix multiplication. The dimensions $m$ and $n$ are called the outer matrix dimensions. Formally,  $M=PQ$ is defined as
$$
M_{ij} = \sum_{k=1}^p P_{ik}Q_{kj}
$$

```python
P = np.array([[1, 7], [2, 3], [5, 0]])
Q = np.array([[2, 6, 3, 1], [1, 2, 3, 4]])
print(P)
print(f'The dimensions of P are: {P.shape}')
print(Q, Q.shape)
print(f'The dimensions of Q are: {Q.shape}')
print(np.dot(P, Q))
print(f'The dimensions of PxQ are: {np.dot(P, Q).shape}')
```

    [[1 7]
     [2 3]
     [5 0]]
    The dimensions of P are: (3, 2)
    [[2 6 3 1]
     [1 2 3 4]] (2, 4)
    The dimensions of Q are: (2, 4)
    [[ 9 20 24 29]
     [ 7 18 15 14]
     [10 30 15  5]]
    The dimensions of PxQ are: (3, 4)

```python
#what will happend here?
np.dot(P, Q)
```

The **determinant** is an important property of square matrices (same number of rows and columns). The determinant is denoted by $\det(M)$ or $|M|$.

In the case of $2 \times 2$ matrices, the determinant is
$$
\begin{split}
|M| = \begin{bmatrix}
a & b \\
c & d\\
\end{bmatrix} = ad - bc\end{split}
$$

In the case of $3 \times 3$ matrices, the determinant is
$$
\begin{split}
\begin{eqnarray*}
|M| = \begin{bmatrix}
a & b & c \\
d & e & f \\
g & h & i \\
\end{bmatrix} & = & a\begin{bmatrix}
\Box &\Box  &\Box  \\
\Box & e & f \\
\Box & h & i \\
\end{bmatrix} - b\begin{bmatrix}
\Box &\Box  &\Box  \\
d & \Box & f \\
g & \Box & i \\
\end{bmatrix}+c\begin{bmatrix}
\Box &\Box  &\Box  \\
d & e & \Box \\
g & h & \Box \\
\end{bmatrix} \\
&&\\
& = & a\begin{bmatrix}
e & f \\
h & i \\
\end{bmatrix} - b\begin{bmatrix}
d & f \\
g & i \\
\end{bmatrix}+c\begin{bmatrix}
d & e \\
g & h \\
\end{bmatrix} \\
&&\\
& = & aei + bfg + cdh - ceg - bdi - afh
\end{eqnarray*}\end{split}
$$

Computing the determinant or larger matrices is cumbersome. However, the process can be easily automated and always reduced to computing the determinant of $2 \time 2$ matrices. Numpy includes an efficient method to compute the determinant of a matrix

```python
from numpy.linalg import det

M = np.array([[0,2,1,3],
             [3,2,8,1],
             [1,0,0,3],
             [0,3,2,1]])
print(f'M: {M}')

print(f'Determinant: {det(M):0.2f}') #note that the :0.2f limits the number of decimals printed!

```

    M: [[0 2 1 3]
     [3 2 8 1]
     [1 0 0 3]
     [0 3 2 1]]
    Determinant: -38.00

The inverse of a square matrix $M$ is a matrix of the same size, $N$, such that $M \bullet N=I$, Where $I$ is a matrix with only ones in its diagonal (unity matrix). The inverse of a matrix $M$ is denoted as $M^{-1}$. For a $2 \times 2$ matrix, the inverse is defined as

$$
\begin{split}
M^{-1} = \begin{bmatrix}
a & b \\
c & d\\
\end{bmatrix}^{-1} = \frac{1}{|M|}\begin{bmatrix}
d & -b \\
-c & a\\
\end{bmatrix}\end{split}
$$

calculating the inverse of a matrix is a complex process; however, it is an important step in many calculations and several *easier* approaches have been developed.

if the determinant of a matrix is zero, then the matrix doesn't have an inverse.

```python
from numpy.linalg import inv

M = np.array([[0,2,1,3],
             [3,2,8,1],
             [1,0,0,3],
             [0,3,2,1]])
print(f'M: {M}')

print(f'Inverse: {inv(M)}') #note that the :0.2f limits the number of decimals printed!

print(f'M x inv(M) = {np.dot(M,inv(M))}')

```

    M: [[0 2 1 3]
     [3 2 8 1]
     [1 0 0 3]
     [0 3 2 1]]
    Inverse: [[-1.57894737 -0.07894737  1.23684211  1.10526316]
     [-0.63157895 -0.13157895  0.39473684  0.84210526]
     [ 0.68421053  0.18421053 -0.55263158 -0.57894737]
     [ 0.52631579  0.02631579 -0.07894737 -0.36842105]]
    M x inv(M) = [[ 1.00000000e+00 -3.46944695e-18  5.55111512e-17  1.11022302e-16]
     [ 0.00000000e+00  1.00000000e+00  4.99600361e-16 -1.11022302e-16]
     [ 2.22044605e-16  5.20417043e-17  1.00000000e+00 -3.33066907e-16]
     [ 0.00000000e+00  1.73472348e-17  5.55111512e-17  1.00000000e+00]]

A matrix that is close to being singular (i.e., the determinant is close to 0) is called **ill-conditioned**. Although ill-conditioned matrices have inverses, they are problematic numerically in the same way that dividing a number by a very, very small number is problematic.
The **condition number** is a measure of how ill-conditioned a matrix is, and it can be computed using Numpy’s function cond from linalg. The higher the condition number, the closer the matrix is to being singular.

The **rank** of an $m \times n$ matrix $A$ is the number of linearly independent columns or rows of $A$ (that is, you cannot write a row or column as a linear combination of other rows or columns), and is denoted by **rank(A)**. It can be shown that the number of linearly independent rows is always equal to the number of linearly independent columns for any matrix. A matrix is called full rank. if **rank (A)=min(m,n)**. The matrix, $A$, is also full rank if all of its columns are linearly independent.

```python
from numpy.linalg import cond, matrix_rank

A = np.array([[1,1,0],
              [0,1,0],
              [1,0,1]])

print(f'Condition number: {cond(A)}')
print(f'Rank: {matrix_rank(A)}')
```

    Condition number: 4.048917339522305
    Rank: 3

if you append a new columns (or row) to a matrix, the rank will increase if the new columns add new information (that is, the new column cannot be explained by a linear combinantion of existing columns)

```python
y = np.array([[1], [2], [1]])
A_y = np.concatenate((A, y), axis = 1)
print(f'Augmented matrix: \n {A_y}')
print(f'Rank of augmented matrix: {matrix_rank(A_y)} ')
```

    Augmented matrix:
     [[1 1 0 1]
     [0 1 0 2]
     [1 0 1 1]]
    Rank of augmented matrix: 3

### Linear Transformations

You can transform a vector by applying linear operations to it, for examples

- Sum with a scalar
- Multiplication with a scalar
- Sum with another vector
- Multiplication with another vector
- Multiplication with a matrix

The last operation is one of the most important operation in linear algebra and has many applications.

Example, **Vector Rotation**

```python
import numpy as np
import matplotlib.pyplot as plt
V = np.array([[3],[1]])
origin = np.array([[0], [0]]) # origin point

plt.quiver(0,0,*V, color=['r'], scale=21)
plt.plot([-1,1],[0,0], lw=0.5, color = 'k')
plt.plot([0,0],[-1,1], lw=0.5, color = 'k')
plt.show()
```

To rotate a vector by an angle $\theta$, you have to multiply it by a rotation matrix given by
$$
R = \begin{bmatrix}
\cos(\theta) & -\sin(\theta) \\
\sin(\theta) & \cos(\theta)
\end{bmatrix}
$$

```python
#Rotate the vector by 45 degress
theta = 45 * (np.pi/180)
Rot_Matrix = np.array([[np.cos(theta), -np.sin(theta)],[np.sin(theta), np.cos(theta)]])

rot_V = Rot_Matrix @ V

#(2x2) @ (2x1) -> (2x1)
plt.quiver(0,0,*V, color=['r'], scale=21)
plt.quiver(0,0,*rot_V, color=['tab:green'], scale=21)
plt.plot([-1,1],[0,0], lw=0.5, color = 'k')
plt.plot([0,0],[-1,1], lw=0.5, color = 'k')
plt.show()
```

## Exercise
Try it yourself, rotate the vector
$$
R = \begin{bmatrix}
5 \\
3
\end{bmatrix}
$$

by 50 degrees. Verify the result of the operation

```python
import numpy as np

def rot_matrix(theta):

    return np.array([[np.cos(theta), -np.sin(theta)],[np.sin(theta), np.cos(theta)]])

a = np.array([[5,3]]).T
angle = 50 * (np.pi/180)
R = rot_matrix(angle)

a_rot = R.dot(a)

print(np.arccos(((a.T).dot(a_rot))/(np.linalg.norm(a)*np.linalg.norm(a_rot)))*(180/np.pi))
```

    [[50.]]

Linear transformation are **inversible**, you can recover the original vector by multiplying by the inverse of the rotation matrix

```python
V = np.array([[3],[1]])

theta = 45 * (np.pi/180)
Rot_Matrix = np.array([[np.cos(theta), -np.sin(theta)],[np.sin(theta), np.cos(theta)]])

rot_V = Rot_Matrix @ V

rec_V = np.linalg.inv(Rot_Matrix)@rot_V

print(f'The original Vector is: \n {V}')
print(f'The recovered Vector is : \n {rec_V}')
```

    The original Vector is:
     [[3]
     [1]]
    The recovered Vector is :
     [[3.]
     [1.]]

### System of linear equations

A system of linear equations is a set of linear equations that share the same variables. Consider the following system of linear equations:

$$
\begin{eqnarray*}
\begin{array}{rcrcccccrcc}
a_{1,1} x_1 &+& a_{1,2} x_2 &+& {\ldots}& +& a_{1,n-1} x_{n-1} &+&a_{1,n} x_n &=& y_1,\\
a_{2,1} x_1 &+& a_{2,2} x_2 &+&{\ldots}& +& a_{2,n-1} x_{n-1} &+& a_{2,n} x_n &=& y_2, \\
&&&&{\ldots} &&{\ldots}&&&& \\
a_{m-1,1}x_1 &+& a_{m-1,2}x_2&+ &{\ldots}& +& a_{m-1,n-1} x_{n-1} &+& a_{m-1,n} x_n &=& y_{m-1},\\
a_{m,1} x_1 &+& a_{m,2}x_2 &+ &{\ldots}& +& a_{m,n-1} x_{n-1} &+& a_{m,n} x_n &=& y_{m}.
\end{array}
\end{eqnarray*}$$

The matrix form of a system of linear equations is $\textbf{A}x = y$ where $\textbf{A}$ is a m×n matrix, $y$ is a vector, and $x$ is an unknown vector:
$$
\begin{split}\begin{bmatrix}
a_{1,1} & a_{1,2} & ... & a_{1,n}\\
a_{2,1} & a_{2,2} & ... & a_{2,n}\\
... & ... & ... & ... \\
a_{m,1} & a_{m,2} & ... & a_{m,n}
\end{bmatrix}\left[\begin{array}{c} x_1 \\x_2 \\ ... \\x_n \end{array}\right] =
\left[\begin{array}{c} y_1 \\y_2 \\ ... \\y_m \end{array}\right]\end{split}
$$

For example, the system of linear equations

$$
\begin{eqnarray*}
4x + 3y - 5z &=& 2 \\
-2x - 4y + 5z &=& 5 \\
7x + 8y   &=& -3 \\
x   + 2z &=& 1  \\
9 + y - 6z &=& 6 \\
\end{eqnarray*}
$$

can be written as
$$
\begin{split}\begin{bmatrix}
4 & 3 & -5\\
-2 & -4 & 5\\
7 & 8 & 0\\
1 & 0 & 2\\
9 & 1 & -6
\end{bmatrix}\left[\begin{array}{c} x \\y \\z \end{array}\right] =
\left[\begin{array}{c} 2 \\5 \\-3 \\1 \\6 \end{array}\right]\end{split}
$$

#### Solutions to Systems of Linear Equations

The objective is to find a set of scalars ($x$, $y$, and $z$) that allow us to write the vector $y$ as a linear combination of the columns of $\textbf{A}$.

If the rank of the augmented matriz $[\textbf{A},y]$ is equal to the rank of $[\textbf{A}]$, this solution exist. Otherwise, the solution doesn't exists.

Moreover, if $\textbf{A}$ is not full rank (i.e, the $rank(\textbf{A})$ is smaller than the number of columns), then not all the columns of $\textbf{A}$ are independent and the system will have infinite number of solutions.

There are many methods that can be used to solve a system of linear equations. Most methods were designed to simplify manual calculations, however, we are mostly interested in computer based methods
1) Direct matrix inversion
In this method, we multiply by the inverse of the matrix $\textbf{A}$ in both sides of the equation
$$
\begin{align}
\textbf{A} x &= y \\
\textbf{A}^{-1}\textbf{A}x &= \textbf{A}^{-1} y \\
x &= \textbf{A}^{-1}y
\end{align}
$$

```python
A = np.array([[8, 8, 0],
              [-2, -4, 5],
             [4, 3, -5] ])
y = np.array([2, 5, -3])

x = np.linalg.inv(A)@y
print(x)
```

    [ 0.75 -0.5   0.9 ]

```python
#Verify the results
print(f'Ax = {A@x}')
```

    Ax = [ 2.  5. -3.]

That methods works fine unless the matrix $\textbf{A}$ is close to be singular. In that case, we can use other methods that avoid finding the inverse of the matrix. The most common method is called $LU$ decomposition, , where a matrix $\textbf{A}$ is expressed as
$$
\textbf{A} = \textbf{L}\textbf{U}
$$

with $\textbf{L}$ a lower diagonal matrix and $\textbf{U}$ a upper diagonal matrix

$$
\begin{split}Ax = y \rightarrow LUx=y\rightarrow
\begin{bmatrix}
l_{1,1} & 0 & 0 & 0\\
l_{2,1} & l_{2,2} & 0 & 0\\
l_{3,1} & l_{3,2} & l_{3,3} & 0 \\
l_{4,1} & l_{4,2} & l_{4,3} & l_{4,4}
\end{bmatrix}
\begin{bmatrix}
u_{1,1} & u_{1,2} & u_{1,3} & u_{1,4}\\
0 & u_{2,2} & u_{2,3} & u_{2,4}\\
0 & 0 & u_{3,3} & u_{3,4} \\
0 & 0 & 0 & u_{4,4}
\end{bmatrix}\left[\begin{array}{c} x_1 \\x_2 \\ x_3 \\x_4 \end{array}\right] =
\left[\begin{array}{c} y_1 \\y_2 \\ y_3 \\y_4 \end{array}\right]\end{split}
$$

we can now split this problem into two simpler problems
$$
\begin{split}
\begin{bmatrix}
u_{1,1} & u_{1,2} & u_{1,3} & u_{1,4}\\
0 & u_{2,2} & u_{2,3} & u_{2,4}\\
0 & 0 & u_{3,3} & u_{3,4} \\
0 & 0 & 0 & u_{4,4}
\end{bmatrix}\left[\begin{array}{c} x_1 \\x_2 \\ x_3 \\x_4 \end{array}\right] =
\left[\begin{array}{c} m_1 \\m_2 \\ m_3 \\m_4 \end{array}\right]\end{split}
$$

and

$$
\begin{split}
\begin{bmatrix}
l_{1,1} & 0 & 0 & 0\\
l_{2,1} & l_{2,2} & 0 & 0\\
l_{3,1} & l_{3,2} & l_{3,3} & 0 \\
l_{4,1} & l_{4,2} & l_{4,3} & l_{4,4}
\end{bmatrix}
\left[\begin{array}{c} m_1 \\m_2 \\ m_3 \\m_4 \end{array}\right] =
\left[\begin{array}{c} y_1 \\y_2 \\ y_3 \\y_4 \end{array}\right]\end{split}
$$

Note that if $\textbf{A}$ is full rank, then the matrices $\textbf{L}$ and $\textbf{U}$ exist, and its inverse is easy to find and the determinant is equal to the mutiplication of the elements in the diagonal

```python
from scipy.linalg import lu #note that we are not using numpy
A = np.array([[8, 8, 0],
              [-2, -4, 5],
             [4, 3, -5] ])
y = np.array([2, 5, -3])

P,L,U = lu(A)
print(L)
print(U)
```

    [[ 1.    0.    0.  ]
     [-0.25  1.    0.  ]
     [ 0.5   0.5   1.  ]]
    [[ 8.   8.   0. ]
     [ 0.  -2.   5. ]
     [ 0.   0.  -7.5]]

```python
#compute m using L and y
m = np.linalg.inv(L)@y
```

```python
#compute x using U and m
x = np.linalg.inv(U)@m
print(x)
```

    [ 0.75 -0.5   0.9 ]

```python
#Verify the results
print(f'Ax = {A@x}')
```

    Ax = [ 2.  5. -3.]

```python
#numpy does the same in its own function to solve linear system
from numpy.linalg import solve

x = solve(A,y)
print(x)
```

    [ 0.75 -0.5   0.9 ]


## Calculus
### limits

Note: for interactive work it is useful to run `init_session()`, which defines some basic symbols

```python
from sympy import *
init_session()
%matplotlib inline
```

    IPython console for SymPy 1.3 (Python 3.6.6-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.3/

```python
limit(sin(x)/x, x, 0)
```

```python
limit(1/x, x, 0)
```

Limits are from the right by default. It can be changed with `dir` argument:

```python
limit(1/x, x, 0, dir="-")
```

```python
limit(1/x, x, 0, dir="+-")
```

We may also create an expression with the limit without evaluating it

```python
Limit(1/x, x, 0, dir="+")
```

and evaluate later if needed

```python
_.doit()
```

#### Differentiation

```python
diff(sin(x)/x, x)
```

Multiple and multivariate derivatives are possible

```python
diff(sin(x)/x, x, x)
```

```python
diff(sin(x)/x, x, 2)
```

```python
diff(sin(x)/y, x, 2, y, 2)
```

Unevaluated derivatives are useful for expressing differential equations

```python
Derivative(sin(x)/y, x, 2, y, 2)
```

#### Series expansion

```python
exp(x).series()
```

```python
expr = exp(sin(x**2+pi))
expr.series(x)
```

```python
expr.series(x, 1)
```

Lets compare the original function with its series expansion visually

### Plotting side-note $\to$

```python
p1 = plot(expr, expr.series(x, 1, 7).removeO(), (x, -4, 7),
    ylim = (-1,3),
    show = False,
    legend = True
)
p1[1].line_color="r"
p1[1].label="series(6)"
p1.show()
```

### ODE solving

```python
f, g = symbols('f, g', cls=Function)
```

```python
eqn = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
eqn
```

```python
sol = dsolve(eqn, f(x))
sol
```

We can test that the solution is valid

```python
eqn.subs(f(x), sol.args[1]).doit()
```

Systems of equations are also supported

```python
alpha = symbols("alpha", positive=True)
eqn = Eq(f(x).diff(x), alpha*g(x)), Eq(g(x).diff(x), -alpha*g(x))
eqn
```

```python
dsolve(eqn, (f(x), g(x)))
```


# Transformations, Eigenvectors, and Eigenvalues

Matrices and vectors are used together to manipulate spatial dimensions. This has a lot of applications, including the mathematical generation of 3D computer graphics, geometric modeling, and the training and optimization of machine learning algorithms. We're not going to cover the subject exhaustively here; but we'll focus on a few key concepts that are useful to know when you plan to work with machine learning.

## Linear Transformations
You can manipulate a vector by multiplying it with a matrix. The matrix acts a function that operates on an input vector to produce a vector output. Specifically, matrix multiplications of vectors are *linear transformations* that transform the input vector into the output vector.

For example, consider this matrix ***A*** and vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

We can define a transformation ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

To perform this transformation, we simply calculate the dot product by applying the *RC* rule; multiplying each row of the matrix by the single column of the vector:

$$\begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\end{bmatrix}$$

Here's the calculation in Python:

```python
import numpy as np

v = np.array([1,2])
A = np.array([[2,3],
              [5,2]])

t = A@v
print (t)
```

    [8 9]

In this case, both the input vector and the output vector have 2 components - in other words, the transformation takes a 2-dimensional vector and produces a new 2-dimensional vector; which we can indicate like this:

$$ T: \rm I\!R^{2} \to \rm I\!R^{2} $$

Note that the output vector may have a different number of dimensions from the input vector; so the matrix function might transform the vector from one space to another - or in notation, ${\rm I\!R}$n -> ${\rm I\!R}$m.

For example, let's redefine matrix ***A***, while retaining our original definition of vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

Now if we once again define ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

We apply the transformation like this:

$$\begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\\3\end{bmatrix}$$

So now, our transformation transforms the vector from 2-dimensional space to 3-dimensional space:

$$ T: \rm I\!R^{2} \to \rm I\!R^{3} $$

Here it is in Python:

```python
import numpy as np
v = np.array([1,2])
A = np.array([[2,3],
              [5,2],
              [1,1]])

t = A@v
print (t)
```

    [8 9 3]

```python
import numpy as np
v = np.array([1,2])
A = np.array([[1,2],
              [2,1]])

t = A@v
print (t)
```

    [5 4]

## Transformations of Magnitude and Amplitude

When you multiply a vector by a matrix, you transform it in at least one of the following two ways:
* Scale the length (*magnitude*) of the matrix to make it longer or shorter
* Change the direction (*amplitude*) of the matrix

For example consider the following matrix and vector:

$$ A = \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\0\end{bmatrix}$$

As before, we transform the vector ***v*** by multiplying it with the matrix ***A***:

\begin{equation}\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}\end{equation}

In this case, the resulting vector has changed in length (*magnitude*), but has not changed its direction (*amplitude*).

Let's visualize that in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([t,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

The original vector ***v*** is shown in orange, and the transformed vector ***t*** is shown in blue - note that ***t*** has the same direction (*amplitude*) as ***v*** but a greater length (*magnitude*).

Now let's use a different matrix to transform the vector ***v***:
\begin{equation}\begin{bmatrix}0 & -1\\1 & 0\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}0\\1\end{bmatrix}\end{equation}

This time, the resulting vector has been changed to a different amplitude, but has the same magnitude.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[0,-1],
              [1,0]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=10)
plt.show()
```

Now let's see change the matrix one more time:
\begin{equation}\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\1\end{bmatrix}\end{equation}

Now our resulting vector has been transformed to a new amplitude *and* magnitude - the transformation has affected both direction and scale.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,1],
              [1,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=10)
plt.show()
```

### Afine Transformations
An Afine transformation multiplies a vector by a matrix and adds an offset vector, sometimes referred to as *bias*; like this:

$$T(\vec{v}) = A\vec{v} + \vec{b}$$

For example:

\begin{equation}\begin{bmatrix}5 & 2\\3 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\1\end{bmatrix} + \begin{bmatrix}-2\\-6\end{bmatrix} = \begin{bmatrix}5\\-2\end{bmatrix}\end{equation}

This kind of transformation is actually the basis of linear regression, which is a core foundation for machine learning. The matrix defines the *features*, the first vector is the *coefficients*, and the bias vector is the *intercept*.

here's an example of an Afine transformation in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,1])
A = np.array([[5,2],
              [3,1]])
b = np.array([-2,-6])

t = A@v + b
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=15)
plt.show()
```

## Eigenvectors and Eigenvalues
So we can see that when you transform a vector using a matrix, we change its direction, length, or both. When the transformation only affects scale (in other words, the output vector has a different magnitude but the same amplitude as the input vector), the matrix multiplication for the transformation is the equivalent operation as some scalar multiplication of the vector.

For example, earlier we examined the following transformation that dot-mulitplies a vector by a matrix:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

You can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

The following python performs both of these calculation and shows the results, which are identical.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t1 = A@v
print (t1)
t2 = 2*v
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

In cases like these, where a matrix transformation is the equivelent of a scalar-vector multiplication, the scalar-vector pairs that correspond to the matrix are known respectively as eigenvalues and eigenvectors. We generally indicate eigenvalues using the Greek letter lambda (λ), and the formula that defines eigenvalues and eigenvectors with respect to a transformation is:

$$ T(\vec{v}) = \lambda\vec{v}$$

Where the vector ***v*** is an eigenvector and the value ***λ*** is an eigenvalue for transformation ***T***.

When the transformation ***T*** is represented as a matrix multiplication, as in this case where the transformation is represented by matrix ***A***:

$$ T(\vec{v}) = A\vec{v} = \lambda\vec{v}$$

Then  ***v*** is an eigenvector and ***λ*** is an eigenvalue of ***A***.

A matrix can have multiple eigenvector-eigenvalue pairs, and you can calculate them manually. However, it's generally easier to use a tool or programming language. For example, in Python you can use the ***linalg.eig*** function, which returns an array of eigenvalues and a matrix of the corresponding eigenvectors for the specified matrix.

Here's an example that returns the eigenvalue and eigenvector pairs for the following matrix:

$$A=\begin{bmatrix}2 & 0\\0 & 3\end{bmatrix}$$

```python
import numpy as np
A = np.array([[2,0],
              [0,3]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [2. 3.]
    [[1. 0.]
     [0. 1.]]

So there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 3, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

So far so good. Now let's check the second pair:

$$ 3 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix} $$

So our eigenvalue-eigenvector scalar multiplications do indeed correspond to our matrix-eigenvector dot-product transformations.

Here's the equivalent code in Python, using the ***eVals*** and ***eVecs*** variables you generated in the previous code cell:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))
```

    Matrix A:
    [[2 0]
     [0 3]]
    -------
    lam1: 2.0
    v1: [1. 0.]
    Av1: [2. 0.]
    lam1 x v1: [2. 0.]
    -------
    lam2: 3.0
    v2: [0. 1.]
    Av2: [0. 3.]
    lam2 x v2: [0. 3.]

You can use the following code to visualize these transformations:

```python
t1 = lam1*vec1
print (t1)
t2 = lam2*vec2
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

Similarly, earlier we examined the following matrix transformation:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

And we saw that you can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

This works because the scalar value 2 and the vector (1,0) are an eigenvalue-eigenvector pair for this matrix.

Let's use Python to determine the eigenvalue-eigenvector pairs for this matrix:

```python
import numpy as np
A = np.array([[2,0],
              [0,2]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

So once again, there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 2, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

Well, we already knew that. Now let's check the second pair:

$$ 2 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix} $$

Now let's use Pythonto verify and plot these transformations:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the resulting vectors
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

Let's take a look at one more, slightly more complex example. Here's our matrix:

$$\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}$$

Let's get the eigenvalue and eigenvector pairs:

```python
import numpy as np

A = np.array([[2,1],
              [1,2]])

eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

    [3. 1.]
    [[ 0.70710678 -0.70710678]
     [ 0.70710678  0.70710678]]

This time the eigenvalue-eigenvector pairs are:

$$ \lambda_{1} = 3, \vec{v_{1}} = \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 1, \vec{v_{2}} = \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} $$

So let's check the first pair:

$$ 3 \times \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix} $$

Now let's check the second pair:

$$ 1 \times \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix} $$

With more complex examples like this, it's generally easier to do it with Python:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the results
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

## Eigendecomposition
So we've learned a little about eigenvalues and eigenvectors; but you may be wondering what use they are. Well, one use for them is to help decompose transformation matrices.

Recall that previously we found that a matrix transformation of a vector changes its magnitude, amplitude, or both. Without getting too technical about it, we need to remember that vectors can exist in any spatial orientation, or *basis*; and the same transformation can be applied in different *bases*.

We can decompose a matrix using the following formula:

$$A = Q \Lambda Q^{-1}$$

Where ***A*** is a trasformation that can be applied to a vector in its current base, ***Q*** is a matrix of eigenvectors that defines a change of basis, and ***Λ*** is a matrix with eigenvalues on the diagonal that defines the same linear transformation as ***A*** in the base defined by ***Q***.

Let's look at these in some more detail. Consider this matrix:

$$A=\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix}$$

***Q*** is a matrix in which each column is an eigenvector of ***A***; which as we've seen previously, we can calculate using Python:

```python
import numpy as np

A = np.array([[3,2],
              [1,0]])

l, Q = np.linalg.eig(A)
print(Q)
```

So for matrix ***A***, ***Q*** is the following matrix:

$$Q=\begin{bmatrix}0.96276969 & -0.48963374\\0.27032301 & 0.87192821\end{bmatrix}$$

***Λ*** is a matrix that contains the eigenvalues for ***A*** on the diagonal, with zeros in all other elements; so for a 2x2 matrix, Λ will look like this:

$$\Lambda=\begin{bmatrix}\lambda_{1} & 0\\0 & \lambda_{2}\end{bmatrix}$$

In our Python code, we've already used the ***linalg.eig*** function to return the array of eigenvalues for ***A*** into the variable ***l***, so now we just need to format that as a matrix:

```python
L = np.diag(l)
print (L)
```

So ***Λ*** is the following matrix:

$$\Lambda=\begin{bmatrix}3.56155281 & 0\\0 & -0.56155281\end{bmatrix}$$

Now we just need to find ***Q-1***, which is the inverse of ***Q***:

```python
Qinv = np.linalg.inv(Q)
print(Qinv)
```

The inverse of ***Q*** then, is:

$$Q^{-1}=\begin{bmatrix}0.89720673 & 0.50382896\\-0.27816009 & 0.99068183\end{bmatrix}$$

So what does that mean? Well, it means that we can decompose the transformation of *any* vector multiplied by matrix ***A*** into the separate operations ***QΛQ-1***:

$$A\vec{v} = Q \Lambda Q^{-1}\vec{v}$$

To prove this, let's take vector ***v***:

$$\vec{v} = \begin{bmatrix}1\\3\end{bmatrix} $$

Our matrix transformation using ***A*** is:

$$\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix} \cdot \begin{bmatrix}1\\3\end{bmatrix} $$

So let's show the results of that using Python:

```python
v = np.array([1,3])
t = A@v

print(t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'b'], scale=20)
plt.show()
```

And now, let's do the same thing using the ***QΛQ-1*** sequence of operations:

```python
import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t = (Q@(L@(Qinv)))@v

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'b'], scale=20)
plt.show()
```

So ***A*** and ***QΛQ-1*** are equivalent.

If we view the intermediary stages of the decomposed transformation, you can see the transformation using ***A*** in the original base for ***v*** (orange to blue) and the transformation using ***Λ*** in the change of basis decribed by ***Q*** (red to magenta):

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t1 = Qinv@v
t2 = L@t1
t3 = Q@t2

# Plot the transformations
vecs = np.array([v,t1, t2, t3])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'red', 'magenta', 'blue'], scale=20)
plt.show()
```

So from this visualization, it should be apparent that the transformation ***Av*** can be performed by changing the basis for ***v*** using ***Q*** (from orange to red in the above plot) applying the equivalent linear transformation in that base using ***Λ*** (red to magenta), and switching back to the original base using ***Q-1*** (magenta to blue).

## Rank of a Matrix

The **rank** of a square matrix is the number of non-zero eigenvalues of the matrix. A **full rank** matrix has the same number of non-zero eigenvalues as the dimension of the matrix. A **rank-deficient** matrix has fewer non-zero eigenvalues as dimensions. The inverse of a rank deficient matrix is singular and so does not exist (this is why in a previous notebook we noted that some matrices have no inverse).

Consider the following matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find its eigenvalues (***Λ***):

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])
l, Q = np.linalg.eig(A)
L = np.diag(l)
print(L)
```

    [[-1.  0.]
     [ 0.  5.]]

$$\Lambda=\begin{bmatrix}-1 & 0\\0 & 5\end{bmatrix}$$

This matrix has full rank. The dimensions of the matrix is 2. There are two non-zero eigenvalues.

Now consider this matrix:

$$B=\begin{bmatrix}3 & -3 & 6\\2 & -2 & 4\\1 & -1 & 2\end{bmatrix}$$

Note that the second and third columns are just scalar multiples of the first column.

Let's examine it's eigenvalues:

```python
B = np.array([[3,-3,6],
              [2,-2,4],
              [1,-1,2]])
lb, Qb = np.linalg.eig(B)
Lb = np.diag(lb)
print(Lb)
```

    [[3.00000000e+00 0.00000000e+00 0.00000000e+00]
     [0.00000000e+00 5.23364153e-16 0.00000000e+00]
     [0.00000000e+00 0.00000000e+00 0.00000000e+00]]

$$\Lambda=\begin{bmatrix}3 & 0& 0\\0 & -6\times10^{-17} & 0\\0 & 0 & 3.6\times10^{-16}\end{bmatrix}$$

Note that matrix has only 1 non-zero eigenvalue. The other two eigenvalues are so extremely small as to be effectively zero. This is an example of a rank-deficient matrix; and as such, it has no inverse.

## Inverse of a Square Full Rank Matrix
You can calculate the inverse of a square full rank matrix by using the following formula:

$$A^{-1} = Q \Lambda^{-1} Q^{-1}$$

Let's apply this to matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find the matrices for ***Q***, ***Λ-1***, and ***Q-1***:

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])

l, Q = np.linalg.eig(A)
L = np.diag(l)
print(Q)
Linv = np.linalg.inv(L)
Qinv = np.linalg.inv(Q)
print(Linv)
print(Qinv)
```

So:

$$A^{-1}=\begin{bmatrix}-0.70710678 & -0.4472136\\0.70710678 & -0.89442719\end{bmatrix}\cdot\begin{bmatrix}-1 & -0\\0 & 0.2\end{bmatrix}\cdot\begin{bmatrix}-0.94280904 & 0.47140452\\-0.74535599 & -0.74535599\end{bmatrix}$$

Let's calculate that in Python:

```python
Ainv = (Q@(Linv@(Qinv)))
print(Ainv)
```

That gives us the result:

$$A^{-1}=\begin{bmatrix}-0.6 & 0.4\\0.8 & -0.2\end{bmatrix}$$

We can apply the ***np.linalg.inv*** function directly to ***A*** to verify this:

```python
print(np.linalg.inv(A))
```

MathsToML/Module03-Vectors and Matrices/03-05-Transformations Eigenvectors and Eigenvalues.ipynb


# Logistic Regression Model

## 1. Cost function

We assume that we have a training set $\{(\boldsymbol{x}^{(1)}, y^{(1)}), (\boldsymbol{x}^{(2)}, y^{(2)}), \dots, (\boldsymbol{x}^{(m)}, y^{(m)})\}$ with $m$ examples, where

$$x=\left[
\begin{array}{c}
x_0 \\
x_1 \\
\vdots \\
x_n
\end{array}
\right]
$$

with $x_0 = 1$ and $y\in\{0, 1\}.$

The hypothesis function is:

$$
h_{\theta}(\boldsymbol{x}) = \frac{1}{1 + e^{-\boldsymbol{\theta}^T\boldsymbol{x}}} = \frac{1}{1 + e^{-\boldsymbol{x}^T\boldsymbol{\theta}}}
$$

How can we choose the parameters $\theta$? Does the linear regression cost works?

Recall that the linear regression cost could be expressed as:

$$
J(\boldsymbol{\theta}) = \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(\boldsymbol{x}^{(i)}) - y^{(i)})^2.
$$

It turns out that for this hypothesis, this cost function is not convex.

Now, let's consider the **logistic regression cost function**:

$$
J(\boldsymbol{\theta}) = \frac{1}{m}\sum_{i=1}^{m} \mathrm{cost}(h_{\theta}(\boldsymbol{x}^{(i)}), y^{(i)}),
$$

where

$$
\mathrm{cost}(h_{\theta}(\boldsymbol{x}), y) = \left\lbrace
\begin{array}{ccc}
-\log(h_{\theta}(\boldsymbol{x})) & \mathrm{if} & y=1 \\
-\log(1-h_{\theta}(\boldsymbol{x})) & \mathrm{if} & y=0.
\end{array}
\right.
$$

Note that $\mathrm{cost}(h_{\theta}(\boldsymbol{x}), y) = 0$:
- If $y=1$ and $h_{\theta}(\boldsymbol{x})=1$, or
- If $y=0$ and $h_{\theta}(\boldsymbol{x})=0$.

Otherwise, if for example $h_{\theta}(\boldsymbol{x})\to 0$ and $y=1$, then $\mathrm{cost}(h_{\theta}(\boldsymbol{x}), y)\to \infty$.

Then, this cost function captures the desired behavior.

## 2. Simplified Cost Function and Gradient Descent

Note that the term $\mathrm{cost}(h_{\theta}(\boldsymbol{x}), y)$ can be written in only one expression as:

$$
\mathrm{cost}(h_{\theta}(\boldsymbol{x}), y) = \left\lbrace
\begin{array}{ccc}
-\log(h_{\theta}(\boldsymbol{x})) & \mathrm{if} & y=1 \\
-\log(1-h_{\theta}(\boldsymbol{x})) & \mathrm{if} & y=0.
\end{array}
\right. = -y\log(h_{\theta}(\boldsymbol{x})) - (1-y)\log(1-h_{\theta}(\boldsymbol{x})).
$$

Thus, the logistic regression cost function can be rewritten as:

$$
J(\boldsymbol{\theta}) = -\frac{1}{m}\sum_{i=1}^{m} \left[y^{(i)}\log(h_{\theta}(\boldsymbol{x}^{(i)})) + (1-y^{(i)})\log(1-h_{\theta}(\boldsymbol{x}^{(i)}))\right],
$$

Now, recalling that

$$
h_{\theta}(\boldsymbol{x}) = \frac{1}{1 + e^{-\boldsymbol{\theta}^T\boldsymbol{x}}} = \frac{1}{1 + e^{-\boldsymbol{x}^T\boldsymbol{\theta}}},
$$

we can write this cost function in a vectorized form as:

$$
J(\boldsymbol{\theta}) = -\frac{1}{m} \left[y^T \log\left(\frac{1}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right) + (1- y)^T \log\left(\frac{e^{-\boldsymbol{X}\boldsymbol{\theta}}}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right)\right],
$$

where

$$
\boldsymbol{X} = \left[
\begin{array}{c}
\boldsymbol{x}^{(1)} \ ^T \\
\boldsymbol{x}^{(2)} \ ^T \\
\vdots                    \\
\boldsymbol{x}^{(n)} \ ^T
\end{array}
\right] = \left[
\begin{array}{ccccc}
x_0^{(1)} & x_1^{(1)} & x_2^{(1)} & \dots  & x_n^{(1)} \\
x_0^{(2)} & x_1^{(2)} & x_2^{(2)} & \dots  & x_n^{(2)} \\
\vdots    & \vdots    & \vdots    & \ddots & \vdots    \\
x_0^{(m)} & x_1^{(m)} & x_2^{(m)} & \dots  & x_n^{(m)}
\end{array}
\right] = \left[
\begin{array}{ccccc}
1         & x_1^{(1)} & x_2^{(1)} & \dots  & x_n^{(1)} \\
1         & x_1^{(2)} & x_2^{(2)} & \dots  & x_n^{(2)} \\
\vdots    & \vdots    & \vdots    & \ddots & \vdots    \\
1         & x_1^{(m)} & x_2^{(m)} & \dots  & x_n^{(m)}
\end{array}
\right] \in \mathbb{R}^{m \times (n+1)}
$$

is the matrix of all the training examples, and the functions $e^{(\cdot)}$ and $\log{(\cdot)}$ are understood as the componentwise application of the exponential and logarithm functions.

With the above vectorization, the gradient of the cost function is:

\begin{align}
\frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta}) ^T &= - \frac{1}{m} \left[y^T \left(1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}\right) - (1- y)^T \left(\frac{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}{e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right)\right] \frac{\partial}{\partial \boldsymbol{\theta}}\left(\frac{1}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right)^T \\
&= - \frac{1}{m} \left[y^T \left(1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}\right) - (1- y)^T \left(\frac{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}{e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right)\right] \left(\frac{e^{-\boldsymbol{X}\boldsymbol{\theta}}}{\left(1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}\right)^2}\right)^T \boldsymbol{X}\\
&= - \frac{1}{m} \left[y^T \left(1 - \frac{1}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right) - (1- y)^T \left(\frac{1}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right)\right] \boldsymbol{X}\\
&= \frac{1}{m} \left[\left(\frac{1}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right)^T - y^T\right] \boldsymbol{X}
\end{align}

This is:

$$
\frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta}) = \frac{1}{m} \boldsymbol{X}^T \left[\left(\frac{1}{1 + e^{-\boldsymbol{X}\boldsymbol{\theta}}}\right) - y\right].
$$

Having defined the gradient, we can apply some numerical optimization method to minimize $J(\boldsymbol{\theta})$ and find the parameters.

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez. Based on the content of the Machine Learning course offered through coursera by Prof. Andrew Ng.
</footer>


# Introducción

En esta sección se aprenderá a hacer lo siguiente:

* Importar *Sympy* y configurar la impresión "bonita"
* Usar operaciones matemáticas como `sqrt` y `sin`
* Crear símbolos *Sympy*
* Hacer derivadas de expresiones

## Preámbulo

Al igual que NumPy y Pandas reemplazan funciones como `sin`,` cos`, `exp` y `log` con implementaciones numéricas potentes, SymPy reemplaza `sin`, `cos`, `exp` y `log` con potentes implementaciones matemáticas.

```python
from sympy import *
init_printing()  # configura impresión elegante
```

```python
import math
math.sqrt(2)
```

```python
sqrt(2)  # Este `sqrt` proviene de SymPy
```

### Ejercicio

Usa la función `acos` en `-1` para encontrar cuando el coseno es igual a `-1`. Prueba esta misma función con la biblioteca *math*. ¿Obtienes el mismo resultado?

```python
# Usa a acos en -1 para encontrar en qué parte del círculo la coordenada x es igual a -1

```

```python
# Usa a `math.acos` en -1 para encontrar el mismo resultado usando el módulo math.
# ¿Es el mismo resultado?
# ¿Que te da la función `numpy.arccos`?

```

## Symbols

Al igual que `ndarray` de NumPy  o `DataFrame` de Pandas, SymPy tiene `Symbol`, que representa una variable matemática.

Creamos símbolos usando la función `symbols`. Las operaciones en estos símbolos no hacen un trabajo numérico como con NumPy o Pandas, sino que construyen expresiones matemáticas.

```python
x, y, z = symbols('x,y,z')
alpha, beta, gamma = symbols('alpha,beta,gamma')
```

```python
x + 1
```

```python
log(alpha**beta) + gamma
```

```python
sin(x)**2 + cos(x)**2
```

### Ejercicio

Usa `symbols` para crear dos variables, `mu` y `sigma`.

```python
?, ? = symbols('?')
```

### Ejercicio

Usa `exp`, `sqrt` y operadores aritméticos de Python como `+, -, *, **` para crear la curva de campana estándar con objetos SymPy

$$ e^{-\frac{(x - \mu)^2}{ \sigma^2}} $$

```python
exp(?)
```

## Derivadas

Una de las operaciones más solicitadas en SymPy es la derivada. Para tomar la derivada de una expresión, usa el método `diff`

```python
(x**2).diff(x)
```

```python
sin(x).diff(x)
```

```python
(x**2 + x*y + y**2).diff(x)
```

```python
diff(x**2 + x*y + y**2, y) # diff también está disponible como una función
```

### Ejercicio

En la última sección hiciste una distribución normal

```python
mu, sigma = symbols('mu,sigma')
```

```python
bell = exp(-(x - mu)**2 / sigma**2)
bell
```

Toma la derivada de esta expresión con respecto a $x$

```python
?.diff(?)
```

### Ejercicio

Hay tres símbolos en esa expresión. Normalmente estamos interesados en la derivada con respecto a `x`, pero podríamos  pedir la derivada con respecto a `sigma`. Prueba esto ahora

```python
# Derivada de la curva de campana con respecto a sigma

```

### Ejercicio

La segunda derivada de una expresión es solo la derivada de la derivada. Encadena llamadas `.diff( )` para encontrar la segunda y tercera derivada de tu expresión

```python
# Encuentra la segunda y tercera derivada de `bell`

```

## Funciones

*SymPy* tiene varias rutinas para manipular expresiones. La función más utilizada es `simplify`.

```python
expr = sin(x)**2 + cos(x)**2
expr
```

```python
simplify(expr)
```

### Ejercicio

En el ejercicio anterior, encontraste la tercera derivada de la curva de campana

```python
bell.diff(x).diff(x).diff(x)
```

Puedes notar que esta expresión tiene mucha estructura compartida. Podemos factorizar algunos términos para simplificar esta expresión.

Llama a `simplify` en esta expresión y observa el resultado.

```python
# Llama simplify en la tercera derivada de la curva de campana

```

## Sympify

La función `sympify` transforma objetos Python (ints, floats, strings) en objetos Sympy (Integers, Reals, Symbols).
*nota la diferencia entre `sympify` y `simplify`. Estas no son las mismas funciones.*

```python
sympify('r * cos(theta)^2')
```

Es útil cuando interactúas con el mundo real, o para copiar y pegar rápidamente una expresión de una fuente externa.


The Pebble Game

Let's imagine right now, that rather than being on this zoom call we were all transported to the beaches of Monte Carlo. You can see it now, can't you? The beautiful clear blue sky and crystal clear water, and maybe even a cold brew. All of a sudden someone in the group has an idea... All excited, he/she grabs a stick and quickly draws a large square in the sand, and this person have a keen eye for measurements makes each side 2 meters.... The person says 'lets play a game...' OKay so we're going to play a game, a geeky game, but a game nonetheless. The game that's proposed, let's call it the 'pebble game', asks if we estimate the value of \pi by randomly throwing pebbles into the circle inscribed square in the sand? The rules of the game are - Pebbles are randomly thrown into the enclosed area (let's blindfold the gineu pig). The ratio of stones in the circle and the total number of thrown stones is a simple or direct sampling statistical estimate for the area of the pond.

Let there be a circle with radius 1 and centre (0,0).

Surrounding the circle is a square with sides of length 2.
<br>

\begin{align}
A_{circle} = \pi r^2 \\
\pi = \frac{A_{circle}}{r^2} \tag{1}\\
A_{square} = (2r)^2 = 4r^2\\
r=\sqrt{\frac{A_{square}}{4}} \tag{2}\\
\text{substitute eq. 4 into eq. 2:}\\
\boxed{\pi = 4\left(\frac{A_{circle}}{A_{square}}\right)} \tag{3}\\
\end{align}

We can estimate the ratio $\frac{A_{circle}}{A_{square}}$ by generating random points
on the plane between (-1, 1). We count the number of points that land
within the circle and the number of points outside the circle.
Then, $\frac{A_{circle}}{A_{square}} \approx \frac{Trials_{\text{hits}}}{Trials_{total}}$.

So, $\pi \approx 4 \cdot \frac{Trials_{\text{hits}}}{Trials_{total}}$

```python

import random
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
import matplotlib.animation as animation
from IPython.display import Video
from random import choice
import pandas as pd
import math
%matplotlib inline

def MonteCarloSim(num_trials, decimalplaces = 5):
    '''returns dataframe with all the results of num_trials = trials of the Monte Carlo algo'''
    df = pd.DataFrame(columns=['x', 'y', 'r', 'Location', 'piEstimate','Error','Color'])
    df['x'] = 2*(np.random.rand(num_trials)-0.5)
    df['y'] = 2*(np.random.rand(num_trials)-0.5)
    df['r'] = np.sqrt(df['x']**2 + df['y']**2)
    df.loc[df['r'] <= 1, 'Location'] = 'Inside'
    df.loc[df['r'] > 1, 'Location'] = 'Outside'
    df.loc[df['r'] <= 1, 'Color'] = 'green'
    df.loc[df['r'] > 1, 'Color'] = 'red'
    df['piEstimate'] = 4*(df['Location'] == 'Inside').cumsum()/(df.index-1)
    df['Error'] = 4*(df['Location'] == 'Inside').cumsum()/(df.index-1) - math.pi
    return df
```

```python

def MonteCarloAnimation(Ntrials):
    data=MonteCarloSim(Ntrials)

    plt.rcParams['axes.facecolor'] = plt.rcParamsDefault['axes.facecolor']
    plt.rcParams['axes.edgecolor'] = plt.rcParamsDefault['axes.edgecolor']
    plt.rcParams['axes.grid'] = plt.rcParamsDefault['axes.grid']
    plt.rcParams['grid.alpha'] = plt.rcParamsDefault['grid.alpha']
    plt.rcParams['grid.color'] = plt.rcParamsDefault['grid.color']

    circle = plt.Circle((0, 0), 1.0, color='g', linewidth=1, fill=False) ##define circle with built in circle object

    plotfunc = plt.figure(figsize=(10,10))
    plotfunc.subplots_adjust(top=0.8, wspace=0.025)
    plotfunc.tight_layout()

    ax1 = plt.subplot(111, xlim=(-1.0, 1), ylim=(-1.0, 1.0))
    ax1.add_patch(circle)

    scat = ax1.scatter(x=0, y=0, s=70, marker='o')

    # Animation update function
    def animationUpdate(k):
        x=list(data['x'])[:k]
        y = list(data['y'])[:k]
        scat.set_offsets(np.c_[x,y])
        scat.set_color(data['Color'][:k])
        return scat,

    # function for creating animation
    anim = FuncAnimation(plotfunc, animationUpdate, frames=Ntrials, interval=400, blit=True)

    # Set up formatting for the movie files
    Writer = animation.writers['ffmpeg']
    writer = Writer(fps=1, bitrate=10000)
    anim.save('animatedMonteCarlo.mp4', writer=writer)

    Video('animatedMonteCarlo.mp4')
```

```python
MonteCarloAnimation(30)
```

```python
!pwd
```

    /Users/okara/Desktop/pythonMC_visual/Onur-MonteCarlo

```python

```

Python/6. Simulations and Modeling/Monte-Carlo-Visualizations/MonteCarlo_develop.ipynb


Probability is associated with at least one event. E.g.: rolling a die or pulling a coloured ball out of a bag

Outcome of the event is random (can’t be sure of the outcome of the die will).

Hence the variable that represents the outcome of these events is called a random variable

# A. 3 Types of Probability

## Marginal Probability

Marginal Probability is the probability of event A occurring, P(A). Rigorously written as P(X=A).

Example: A pack of playing cards, the probability that a card drawn from a pack is red: P(X=red) = 0.5

## Joint Probability

Joint Probability is the probability ofntersection of two or more events. Written as $ P(A \cap B) , P(A,B), P(A\ and\ B)$

Example: Probability that a card drawn from a pack is red and 4 is P(red and 4) = 2/52 = 1/26

<b>NOTE: Joint probability is symmetrical P(A,B) = P(B,A)</b>

## Conditional Probability

Conditional probability is the probability of an event(s) occuring given that we know other events have already occurred. If A and B are two events then the conditional probability of A occurring given that B has occurred is written as P(A|B)

Example: Probability that a card is a four given that we have drawn a red card is P(4|red) = 2/26 = 1/13

# B. Linking the 3 types of Probability

This is the equation linking the 3 probabilities
\begin{equation}
\begin{aligned}
P(A | B) =  \dfrac{P(A \cap B)}{P(B)}
\end{aligned}
\tag{Equation B.1}
\end{equation}

### Intutition

We start with. This is logical.
\begin{equation}
\begin{aligned}
P(A \cap B) =  P(B)\,P(A \mid B)
\end{aligned}
\tag{Equation B.2}
\end{equation}

Rearranging eq B.2, we get the following

\begin{equation}
\begin{aligned}
P(A \cap B) =  P(A \mid B)\,P(B)
\end{aligned}
\tag{Equation B.3}
\end{equation}
\begin{align}
\end{align}

Rearranging further we get the first equation B.1

# C. Special Case of Joint Probabilities for Independent Events

When two events A and B are independent, P(A | B) = P(A)
Using this in equation B.3 we get

\begin{equation}
\begin{aligned}
P(A \cap B) =  P(A)\,P(B)
\end{aligned}
\tag{Equation C.1}
\end{equation}
\begin{align}
\end{align}

# D. Additive rule of Probability

Addition is used in OR scenario we have to add the individual probabilities and subtract the intersection

\begin{equation}
\begin{aligned}
P(A \cup B) =  P(A) + P(B) - P(A \cap B)
\end{aligned}
\tag{Equation D.1}
\end{equation}
\begin{align}
\end{align}

If you visualize Venn Diagram, then adding P(A) and P(B) accounts for adding the intersection twice. One of this has to be removed. Hence the equation

## D.1 Special case of Mutually exclusive events
In case of mutually exclusive events $ P(A \cap B) = 0 $ and the above equation simplifies as

\begin{equation}
\begin{aligned}
P(A \cup B) =  P(A) + P(B)
\end{aligned}
\tag{Equation D.2}
\end{equation}
\begin{align}
\end{align}

# E. Bayes Theorem

Bayes theorem provides a mechanism to utilize prior knowledge to improve probability estimation

E.g:
\begin{align*}
\text{Initially: P(Person has Cancer) = Percent of population with cancer}
\end{align*}

\begin{align*}
\text{Prior evidence: Person is smoker}
\end{align*}

\begin{align*}
\text{Modification: } P(\text{Person has Cancer}\, \mid \text{Person is smoker})\, > \,  P(\text{Person has Cancer)}
\end{align*}

\begin{equation}
\begin{aligned}
P(A \mid B) =  \dfrac{P(B \mid A)\,P(A)}{P(B)}
\end{aligned}
\tag{Equation E.1}
\end{equation}

It is also used when calculating one conditional propbability say P(A|B) is hard, but the converse conditional probability P(B|A) is easily found

E.g.:
1. Suppose there are 2 bowls - M and N.
2. Bowl M has 5 Oranges and 8 Apples.
3. Bowl N has 12 Oranges and 4 Apples

What is probability of having picked from M, given than the picking was Apple ?
\begin{align*}
\text{Modification: } P(M \mid Apple)\, = \, \dfrac{P(Apple \mid N)\,P(M)}{P(Apple)}
\end{align*}

Here, the P(LHS) is hard by itself. But all P() on RHS can be easily calculated. Hence its utlity

## E.1 Derivation

P(A,B) is equal to P(B,A). Combining b.2 and B.3
\begin{equation}
P(B)\,P(A \mid B) =  P(A \mid B)\,P(B)
\end{equation}

Re arranging, we get Bayes Theorem forumla shown in E.1

## E.2 Bayesian Terminology

1. P(A|B) is called the posterior; this is what we are trying to estimate. In the above example, this would be the “probability of having cancer given that the person is a smoker”.
2. P(B|A) is called the likelihood; this is the probability of observing the new evidence, given our initial hypothesis. In the above example, this would be the “probability of being a smoker given that the person has cancer”.
3. P(A) is called the prior; this is the probability of our hypothesis without any additional prior information. In the above example, this would be the “probability of having cancer”.
4. P(B) is called the marginal likelihood; this is the total probability of observing the evidence. In the above example, this would be the “probability of being a smoker”. In many applications of Bayes Rule, this is ignored, as it mainly serves as normalization.

\begin{align*}
P(A \mid B)\, \propto \,P(B \mid A)\,P(A)
\end{align*}

# Acknowledgements
1. Probability Basics Explanation https://towardsdatascience.com/probability-concepts-explained-introduction-a7c0316de465
2. https://machinelearningmastery.com/joint-marginal-and-conditional-probability-for-machine-learning/
3. https://towardsdatascience.com/what-is-bayes-rule-bb6598d8a2fd
4. https://machinelearningmastery.com/bayes-theorem-for-machine-learning/
5. https://towardsdatascience.com/maximum-likelihood-vs-bayesian-estimation-dd2eb4dfda8a

```python

```


# Optimization Exercise 1

## Imports

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy.optimize as opt
```

## Hat potential

The following potential is often used in Physics and other fields to describe symmetry breaking and is often known as the "hat potential":

$$ V(x) = -a x^2 + b x^4 $$

Write a function `hat(x,a,b)` that returns the value of this function:

```python
def hat(x,a,b):
    return -1*a*(x**2) + b*(x**4)
```

```python
assert hat(0.0, 1.0, 1.0)==0.0
assert hat(0.0, 1.0, 1.0)==0.0
assert hat(1.0, 10.0, 1.0)==-9.0
```

Plot this function over the range $x\in\left[-3,3\right]$ with $b=1.0$ and $a=5.0$:

```python
a = 5.0
b = 1.0
```

```python
x = np.linspace(-3,3,100)
v = hat(x,a,b)
graph = plt.plot(x,v)

```

```python
assert True # leave this to grade the plot
```

Write code that finds the two local minima of this function for $b=1.0$ and $a=5.0$.

* Use `scipy.optimize.minimize` to find the minima. You will have to think carefully about how to get this function to find both minima.
* Print the x values of the minima.
* Plot the function as a blue line.
* On the same axes, show the minima as red circles.
* Customize your visualization to make it beatiful and effective.

```python
f = lambda g: hat(g,a,b)
x1 = float(opt.minimize(f,-2 ).x)
x2 = float(opt.minimize(f,2 ).x)
print(x1)
print(x2)
graph = plt.plot(x,v)
plt.plot([x1,x2],[f(x1),f(x2)],'ro')
```

```python
assert True # leave this for grading the plot
```

To check your numerical results, find the locations of the minima analytically. Show and describe the steps in your derivation using LaTeX equations. Evaluate the location of the minima using the above parameters.

\begin{align}
V(x) = -a x^2 + b x^4 \\
x(4bx^2-2a) = 0 \\
4bx^2=2a \\
x = \pm \sqrt{\frac{a}{2b}} \\
x = \pm \sqrt{\frac{5}{2(1)}} \approx \pm 1.5811388
\end{align}


# Solving Multiple Equations/Multiple Unknowns in SymPy

Joseph C. Slater, Oct. 2018

```python
# import symbolic capability to Python
from sympy import *

# print things all pretty. Not as important in this notebook.
from sympy.abc import *
init_printing()
```

```python
# Need to define the variables as symbolic for sympy to use them.
x, y = symbols("x, y", real = True)
```

All of this is based on using the [general symbolic solvers](https://docs.sympy.org/latest/modules/solvers/solvers.html). They are not as robust as putting things in matrix form. A function to do that wouldn't be too hard, but would take a couple hours to make.

Let's do a simple test case.

$$ x^4 = 4$$

Note that we have to solve it to be equal to zero, so we use the form

$$x^4-4=0$$

```python
equation = x**2 - 4
solve(equation, x)
```

That worked. Let's make some equation. I'm going to store them in variable to make this easier to keep track of.

Here we have

$$3x + 4y - 5 = 0$$

```python
equation1 = 3*x + 4*y - 5
equation1
```

Now we have

$$
6x+7y-8=0
$$

```python
equation2 = 6*x + 7*y - 8
equation2
```

``solve`` would like to be told which variables to solve for, but it can usually make a reasonable guess.

The (seemingly) extra parentheses are because there is only a single argument being sent in, the list of equations: ``(equation1, equation2)``.

```python
solve((equation1, equation2))
```

Below I'm more explicit in sending it two arguments (two lists), the first is a list of equations, the second is a list of variables to solve for.

```python
solve((equation1, equation2), (x, y))
```

```python
checksol(equation1, {x: -1, y: 2})
```

    True

If the second equation had been nonlinear, such as

$$6x+7y^2-8=0$$

```python
equation2 = 6*x + 7*y**2 - 8
equation2
```

We now get multiple solutions:

```python
ans = solve((equation1, equation2))
ans
```

Engineers like decimals in most cases, so, if any number given were a float, we wouldn't get the square roots and all.

```python
equation2 = 6*x + 7*y**2 - 8.0
equation2
```

```python
solve((equation1, equation2))
```

We can create an equation with two sides by using the ``Eq`` command as below.

```python
equation2 = Eq(6*x + 7*y**2 , 8.0)
equation2
```

```python
solve((equation1, equation2))
```


# Generating Fractals

Here we will be generating fractals and learning about a few things as well

1. Complex numbers (very breifly)
2. Root finding (this is a good primer for gradient decent for those of you with ML on the mind)
3. Thinking iteratively ( very important nor numerical mathematics)

## Complex Numbers

Before we start talking about fractals, we should probably introduce the idea of a complex number before hand, as the fractas we will be generating rely heavily on them. I'll note that this may get kind of deep into some mathematics, and if you want to skip this section you totally can. There are som quirks to complex numbers, but luckily Python will handle them for you, and realistically they will behave like any other number in Python (for our purposes).

First and foremost, let us introduce the complex number $i$
$$
\begin{equation}
i  = \sqrt{-1} \implies i^2 = -1
\end{equation}
$$

where $i$ is known as the imaginary number, or a number that when multiplied by itself, returns a negative number, an impossible situation in the real numbers (the set of numbers including integers, natural numbers, rational and irrational numbers). At first this might be concerning, but we must remember that when it comes to math, there is no concensus on whether math is discovered or invented. People thought [mathematicians who suggested working with complex numbers were crazy](https://medium.com/i-math/imaginary-numbers-explained-e5aa63bdb7ae), only for time to reveal that complex number were applicable in many areas, including [electrical engineering](https://www.electronics-tutorials.ws/accircuits/complex-numbers.html).\
More often than not, we will write a complex number as $z$, which is written as the sum of an imaginary numers real and complex parts,

$$
z = a + ib
$$
where $a$ is the real part of the complex number $z$, and $b$ is another real number, multiplied by the imaginary unit $i$ representing the imaginary part of the number.

If you're curious, you can get more details about complex numbers using the drop down below. For our purposes however, all that you really need to know is the following:

1. A complex number $a + ib$ can be thought of as the classical $x,y$ ordered pair we're used to. In this case our $x$ axis is the real part of the number, and our $y$ axis is the imaginary part of the number

2. These $(a, b)$ ordered pairs can be used to define the "complex plane", which we will use to plot our fractals

 More details on complex numbers (not required)

There are a lot of very useful properties that come from imaginary numbers, and they're used all the time in fiels such as pure mathematics, signal processing, physics, chemistry, fluid dynamics... in principle we could go on forever. The largest reason for this is due to something known as [eulers formula](https://en.wikipedia.org/wiki/Euler%27s_formula), where we instead think of complex numbers on the "complex plane", with the real part of the function $a$ representing our $x$ axis, and the imaginary part of our function $b$ on the $y$ axis. Euler's formula states that we can write the exponential of a complex number as follows

$$
e^{ix} = r (\cos \theta + i\sin \theta)
$$
where $r$ is the radius of tihs circle on the complex plane (more on this later).
For those of you mathematically inclined and want to prove this, the easiest way is to write out the Taylor series for each $e^{ix}, i\sin \theta$ and $\cos \theta$, and you may be surprised what you see.

Long story short, using Euler's formula, we can essentially write any complex number in terms of sine and cosine, this is useful for a whole lot of reasons, but for fractals specificially, this implies periodic boundary conditions -- we expect to see repeating patterns. Indeed, as cosine and sine have $N$ roots repeating every $\pi$, we expect that our complex functions may also have (up to) that many roots!

### Finding Roots Of Complex Polynomials

We all remember polynomial equations of real numbers, for example the quadratic function

$$
x^2 - C = 0
$$
where we can all read quite readily that the solution to this equation is $x = \pm \sqrt{C}$. But what about if we have some complex polynomial like

$$
z^2 + C = 0,
$$

To be honest, I pulled a fast one on you. It' just as easy! In this case, we have two roots which are $\pm i \sqrt{C} $, where we just bring our friend the imaginary unit along for the ride. In principle what we do is we once again factor this into the real and imaginary part of the solution, but often it's easier to think of this (in a way that will make mathmaticians cry) we simply get rid of the part we don't like, and call it $i$. For quadratic complex polynomial equations, we can simply use the quadratic formula and sprinkle in the imaginary unit where ever we need it. This is a bit of an over simplification, but for our purposes it should be fine.

Where this can get a little spicier is when the roots aren't obvious enough to be read off. For example, the equation

$$
z^3 = 1,
$$

is cubic, which means we have three distinct roots which satisfy this equation. In this case, it is easier to look at our friend Euler's formula to find these roots. So let's rewrite our complex equation above using euler's formula. First, the left hand side

$$
z = r e^{ix} \implies z^3 = r^3 e^{3ix}
$$

and the right hand side:

$$
1 = re^{ix} = r (\cos \theta + i\sin \theta)
$$

Where, as one has no complex component, we know that this must be one. Therefore, $r$ is equal to one in this equation, angles are those where cosine is one and sine is zero, or  $\theta = 2\pi k$ where $k$ is an integer. Therefore, we have

$$
e^{3ix} = e^{2\pi k}
$$

or by taking the natural logarithm of each side,

$$
ix = \frac{2 \pi k}{3}
$$

And going back to our original equation:

$$
z = e^{ix} = e^{\frac{2 \pi k}{3}}
$$

Where we can take our first three roots as $k = -1, 0, 1$ and obtain

$$
z = 1, e^{2\pi i/3}, e^{-2\pi i/3}
$$

We also note we have periodic roots at integer values of $k$, but we won't worry about those. Knowing these roots are useful in terms of understanding the behaviour of our fractals. We may expect that different roots may cause different basins of convergence, or result in rotations of our fractal. This is also important with respect to establishing the domains in which our fractals may exist.

# Root Finding With Complex Numbers

If you were crazy enough to read the drop down menu, you may have noticed that finding roots to complex equations were more work that the quadratic formula. And if you take anything away from these notebooks it should be that we Data Scientists (and regular scientists) are _super_ lazy. Wouldn't it be nice if we could use our computer to solve these for us? Luckily, the answer is a resounding yes! And even more better is that our Newton Raphson formula from before generalizes to the complex domain without us having to do anything. Convenient!

One thing to be aware of however in Python is we will need to define complex numbers. It's quite simple. For a complex number $ a + ib$, we can define that in python like

```python3
a = b = 1
complex_number = complex(a, b)
```

and we can then throw that number at our root finding routines as we did before and find ourselves solutions.

## My First Fractal: Mandelbrot

Before we go into root finding for fractals, let's start with a relatively simple one to generate the Mandelbrot.  Rather than have me drone on for a year, here's a YouTube video that does a better job than I could explaining that set.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('NGMRB4O922I')
```

# Your Task

Write two functions, the first function will be to calculate the Mandelbrot set as follows

### Mandelbrot Function
1. Initialize $z$ and the number of iterations $n$ as zero,
    - Also intialize the maximum iterations as 80
2. while `abs(z) <= 2 and n < max_iter` do the following
    * $z = z^2 + c$
    * n += 1
3. On exit, return the number of iterations $n$

```python
# note the **kwargs is not strictly necessary, and won'tdo anything here, but will be useful
# if you want to use some of the provided functions later.

def mandelbrot(c, max_iter = 80, **kwargs):
    '''
    here c is any complex number, and max_iter is the maximum numberof iterations through your loop
    you want to go
    '''
    z = # YOUR CODE HERE
    n = # YOUR CODE HERE
    while CONDITION: # YOUR CODE HERE
        z = SOMETHING #YOUR CODE HERE
        n += 1

    return n

```

### Iteration Function

You will also need a function which iterates across the complex plane to see what pixel you should generate there. This function itself will require three functions

1. A scale function to convert pixel location into complex coordinates
2. A scale function to convert the number of itereations in your `mandelbrot` function into an RGB color scale
3. A function which iterates over pixels with a given height and width for your image.

In principle we will have the following pseudocode for point 3 which will encompass the other functions

---
```python
def CreateImage(mandelbrot, height, width, domain):

    ImageMap = np.zeros([width, height])

    for x in range(0, width):
        for y in range(0, height):

            c = scale_function_coordinate(x, y, width, height, real_max, real_min, complex_max, complex_min)

            m = mandelbrot(c)

            color = color_function(m)

            X[x,y] = color

   return ImageMap

```
---
Where let's outline those functions explicitly

##### Pixel Scale Function

Given a pixel coordinate $x$ and $y$, we need to convert this location into a complex number within our domain. the formula for this is as follows

$$
R = R_{min} + \frac{x}{\text{Image Width}} \times (R_{max} - R_{min})
$$

Where $R$ is your value in the real coordinate, $R_{min}$ is the smallest value in the real domain, and $R_{max}$ is the largest value in the real domain, $x$ is the current pixel, and Image Width is the width of the image in pixels you want to create. A similar formula for the complex domain is as follows

$$
C = C_{min} + \frac{x}{\text{Image Height}} \times (C_{max} - C_{min})
$$

Where $C$ is your value in the complex coordinate, $C_{min}$ is the smallest value in the complex domain,  $C_{max}$ is the largest value in the complex domain, $x$ is the current pixel, and Image Height is the height of the image in pixels you want to create.

In this case, if we have a known image size in advance, and we know the domain in which our fractal will exist, we can calculate the complex value $c$ at this place in our image. Please fill in the function below

```python
import numpy as np
def scale_function_coordinate(x, y, width, height, r_max, r_min, c_max, c_min):
    '''
    x --> x coordnate of pixel
    y --> y coordinate of pixel

    width --> width of image
    height --> height of image

    r_max, r_min --> maximum and minimum numbers on thereal axis
    c_max, c_min --> maximum and minimum numbers on the complex axis
    '''

    R = None # YOUR CODE HERE
    C = None # YOUR CODE HERE

    return complex(R, C) # complex is a built in function for complex numbers

```

##### Color Function

Now we need to be able to convert the number of iterations to an RGB coordinate. RGB colors can take values between 0 and 255, so we need to find a way to scale our number of iterations to become some pretty colors so we can observe the changes. Rather than boring you with this, I'll just provide the function

```python
def color(number_of_iterations):
    return 255 - int(m * 255/max_iter)

```

## Putting it All Together

If that worked out well for you, you should be able to fill in the following to create your image functions!

```python
# Boundaries for the mandelbrot function
bounds = [-2, 1, -1, 1]

def CreateImage(function, width, height, bounds):
    r_max, r_min, c_max, c_min = bounds

    if width > 1000:
        print(f'width of {width} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    if height > 1000:
        print(f'height of {height} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    X = np.zeros(width, height)

    for x in range(0, width):
        for y in range(0, height):
            c = scale_function_coordinate(x, y, width, height, r_max, r_min, c_max, c_min)

            # Note here for changes later for root finding
            m = function(c)
            color = color(m)
            X[width, height] = color

    return X

# When you're ready uncomment this line to see if it worked

# plt.imshow(X)
```

If all that worked out, running the above cell should produce what you see below!

```python
import sys
sys.path.append('scripts/')
import fractalfuncs as FF
import matplotlib.pyplot as plt

bounds = [-2, 1, -1, 1]
X = FF.CreateImageMap(function = FF.mandelbrot, function_args = {}, bounds = bounds)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

# Using Rootfinding

Now, rather than using the mandelbrot set, let's try and use our root finding techniques to find roots instead! If we're in a stable region,it should be pretty easy! If not, it will get spicy and diverge. We will use that divergence to create our fractals instead. Here the fractal properties not only come from the mathematical formulation of our complex set, but also the convergence properties of our root finder: different root finding techniques will result in different fractals.

## Your Task

Copy and paste your NewtonRaphson root finder from the Root Finding portion of this, and use it in this assignment. **NOTE** instead of returning the root, you will have to modify your NewtonRaphson function to return the number of iterations it took.

To use root finding, we will do exactly what we did for the mandelbrot set above, however, we will now modify your image generation function to use $c$ as an initial guess at your solution, and see if your NewtonRaphson root finder can find a solution or not. You will need to modify the cell below for use with your own function. Remember that you will also need to pass the derivative and the function you are evaluating (Hint: `**kwargs` can be handy here)

```python
def CreateImageRootFinding(YOUR ARGUMENTS HERE, width, height, bounds):
    r_max, r_min, c_max, c_min = bounds

    if width > 1000:
        print(f'width of {width} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    if height > 1000:
        print(f'height of {height} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    X = np.zeros(width, height)

    for x in range(0, width):
        for y in range(0, height):
            INITIAL_GUESS = scale_function_coordinate(x, y, width, height, r_max, r_min, c_max, c_min)

            # Note you may need to wrap this in try/except to prevent accidental zero division/other nastiness
            m = MY_ROOT_FINDER(INITIAL_GUESS)
            color = color(m)
            X[width, height] = color

```

## First Fractal With Root Finding

The mandelbrot set works well for what it is, but alas, if you try to use thet function in root finding, you will find that your fractal is dreadfully boring. A more interesting function is

$$
f(z) = z^3 - 1
$$

Whose derivative is

$$
f^\prime(z) = 3z^2
$$

### Sanity Check

See if you can reproduce the image below with your own function

```python
import importlib
importlib.reload(FF)
def function(z):
    # All this extra stuff on top is if you want
    # to use this with numerical derivatives as shown below
    rlist = False
    if isinstance(z, list):
        rlist = True
        z= z[0]
    if rlist:
        return [z**3 - 1]
    # If you're using analytical derivatives, you just need this line at the bottom.
    return z**3 - 1

def derivative(z):
    ans = FF.nderiv(function, [z])

    return ans#3 * z ** 2

bounds = [-1,1, -1, 1]

newton_args = dict(fprime = derivative, f = function, max_iter = 50, prec = 1e-5)
X = FF.CreateImageMap(FF.NewtonRaphsonFact, newton_args, bounds, height=200, width=200)

plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

Using the function above, try playing around with the following:

1. Different powers of $z$
2. Change the constant (-1) term. Larger/Smaller positive/negative. What if this term is complex?

What do you observe about the fractal at higher powers and different values of the constant

# Other Functions To Try

Once you've got that working, you should try these functions as well and see what fractals you observe!

$$
\begin{aligned}
f(z) &= \sin(z), x \in \left[-\frac{\pi}{2} - \frac{1}{2}, -\frac{\pi}{2} + \frac{1}{2}\right], y\in \left[-0.3, 0.3\right] \\
f(z) &= \cosh(z) - 1, x \in \left[-0.2, 0.2\right], y \in \left[-\pi, -\pi -\frac{\pi}{8}\right]\\
f(z) &= z^3 - 3^z, x\in [-10, 10], y\in[-10, 10]
\end{aligned}
$$

Note that if you don't know how to calculate a derivative, that's okay, you can use wolfrapmalpha, or alternatively, you can take them numerically with a function i've provided. It can be used as follows

```python
# Only if you haven't imported it already
import sys
sys.path.append('scripts/')
import fractalfuncs as FF
def myfunction(z):
    return z**2 # for example

def myderivative(z):
    return FF.nderiv(myfunction, z)
```
I note that numerical derivatives are always worse than analytic ones, but that's okay for now. If anyone is interested I can talk about how that works later as well.

## Bored of that?

If you're bored, you can also try other root finding techniques instead of your newton solver! Here are some suggestions

1. [Secant Method](https://en.wikipedia.org/wiki/Secant_method#:~:text=In%20numerical%20analysis%2C%20the%20secant,difference%20approximation%20of%20Newton's%20method.)
2. [Halley's Method](https://en.wikipedia.org/wiki/Halley%27s_method#:~:text=In%20numerical%20analysis%2C%20Halley's%20method,Householder's%20methods%2C%20after%20Newton's%20method.)
3. [Schroder's Method](https://mathworld.wolfram.com/SchroedersMethod.html)

Note that these fractals are getting created with based on convergence properties of the above solvers. For example, the cells below outline the same function we used originially, just with their different methods of solving!

```python
def function(z):
    return z**3 - 1

def derivative(z):
    return 3 * z ** 2

def secondder(z):
    return 6 * z

bounds = [-1,1,-1,1]
secant_args  = dict(function = function, mult = 0.5)
X = FF.CreateImageMap(FF.secantfact, secant_args, bounds, height=100, width=100)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

```python
bounds = [-1,1,-1,1]
schroder_args = dict(derivative = derivative, function = function,
                     secondder=secondder, prec = 1e-6, max_iter = 50)
X = FF.CreateImageMap(FF.schroderfact, schroder_args, bounds, height=100, width=100)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

```python
bounds = [-1,1,-1,1]
halley_args = dict(derivative = derivative, function = function,
                     seconder=secondder, prec = 1e-6, max_iter = 50)
X = FF.CreateImageMap(FF.halleyfact, halley_args, bounds, height=100, width=100)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

Where you'll notice each root finding technique has different convergence criteria, so the fractals generated are all slightly different. If you try the other fractals listed, you'll notice tht their fractal patterns will show even more variation