## Ejercicios Big O()
### 1. Calcule el costo T(n) para la siguiente función.
        A. Grafique la función junto con otros O() clásicos.
        B. Analice los cambios a medida que el n aumenta.
        C. T(n) esta incluido en la familia de O(n)? Justifique utilizando definición de O().
        D. El costo obtenido es aceptable?

A continuación un ayuda memoria de $O()$ en operaciones sobre Python:

```python
from IPython.display import IFrame
IFrame('https://wiki.python.org/moin/TimeComplexity', width=900, height=350)

```

```python
def dummy_func(lst, n):
    print("primer elemento", lst[0])

    midpoint = int(n / 2)

    for val in lst[:midpoint]:
        print(val)

    for x in range(10):
        print('o_O')
```

```python
dummy_func([1, 2, 3, 4], 4)
```

    primer elemento 1
    1
    2
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O

1.A)
Analizando las lineas de la función dada podemos ver que:
- En linea 2 tenemos $O(1)$
- En linea 4 tenemos $O(1)$
- En linea 6 y 7 tenemos $O(n/2)$
- En linea 9 y 10 tenemos $O(10)$

Por lo que nuestro $O(n/2 + 12)$

1.B) Ahora graficaremos $O(n/2 + 12)$ junto con $O(n)$, $O(n \log n)$, $O(n^2)$, $O(2^n)$

```python
from math import log
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('default')

# Set up runtime comparisons

n = np.linspace(1, 100, 1000)
labels = ['Lineal', 'Lineal Log', 'Cuadrática','Exponencial', 'Mi T(n)']
big_o = [n, n* np.log(n), n**2, 2**n, n/2 + 12]

# Plot setup
plt.figure(figsize=(12,10))
plt.ylim(0, 700)
plt.xlim(1, 100)

for i in range(len(big_o)):
    plt.plot(n,big_o[i],label = labels[i])

plt.legend(loc=0)
plt.ylabel('Tiempo relativo de ejecución')
plt.xlabel('n')
```

 1.C) Podemos observar que nuestra función con $n_0 = 24, c=1$ $ \rightarrow T(n) \in O(n)\; \forall\; n \geq 24$:

 \begin{equation}
 n/2 + 12 \leq c.n \; \forall\; n \geq 24 \\
 n/2 + 12 \leq n \; \forall\; n \geq 24 \\
 \end{equation}

1.D)

Imagen de http://bigocheatsheet.com/

### 2. Desarrolle dos funciones en Python para encontrar el número más chico en una lista. La primer función debería comparar cada número entre si ($O(n^2)$). La segunda función debería ser lineal ($O(n)$).
        A. Grafique ambas funciones.
        B. Con que implementación se quedaría?
        C. Hay correspondencia entre el benchmarking y $O()$.

```python
def min_value_n2(lst):
    min_value = lst[0] # O(1)
    for i in lst:      # O(n)
        for j in lst:  # O(n)
            if i < j:  # O(1)
                min_value = i   # O(1)
            elif j < i:         # O(1)
                min_value = j   # O(1)
    return min_value            # O(1)

min_value_n2([3, 2, 6, 1])
# O(4*n^2 + 2)
```

    1

```python
def min_value_n(lst):
    min_value = lst[0]        # O(1)
    for i in lst:     # O(n)
        if i < min_value:     # O(1)
            min_value = i     # O(1)
    return min_value          # O(1)

min_value_n([3, 2, 6, 1])
# O(2*n + 2)
```

    1

2.A)

```python
from math import log
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('default')

# Set up runtime comparisons

n = np.linspace(1, 100, 1000)
labels = ['min_value_n', 'min_value_n2']
big_o = [2*n + 2, 4*n**2 + 2]

# Plot setup
plt.figure(figsize=(12,10))
plt.ylim(0, 700)
plt.xlim(1, 100)

for i in range(len(big_o)):
    plt.plot(n,big_o[i],label = labels[i])

plt.legend(loc=0)
plt.ylabel('Tiempo relativo de ejecución')
plt.xlabel('n')
```

2.C)

```python
assert(min_value_n([3, 2, 6, 1]) == min_value_n2([3, 2, 6, 1]))
```

```python
%timeit min_value_n(range(10000))
```

    1000 loops, best of 3: 362 µs per loop

```python
%timeit min_value_n2(range(10000))
```

    1 loop, best of 3: 5.51 s per loop


## RIHAD VARIAWA, Data Scientist - Who has fun LEARNING, EXPLORING & GROWING
## Linear Equations
The equations in the previous lab included one variable, for which you solved the equation to find its value. Now let's look at equations with multiple variables. For reasons that will become apparent, equations with two variables are known as linear equations.

## Solving a Linear Equation
Consider the following equation:

\begin{equation}2y + 3 = 3x - 1 \end{equation}

This equation includes two different variables, **x** and **y**. These variables depend on one another; the value of x is determined in part by the value of y and vice-versa; so we can't solve the equation and find absolute values for both x and y. However, we *can* solve the equation for one of the variables and obtain a result that describes a relative relationship between the variables.

For example, let's solve this equation for y. First, we'll get rid of the constant on the right by adding 1 to both sides:

\begin{equation}2y + 4 = 3x \end{equation}

Then we'll use the same technique to move the constant on the left to the right to isolate the y term by subtracting 4 from both sides:

\begin{equation}2y = 3x - 4 \end{equation}

Now we can deal with the coefficient for y by dividing both sides by 2:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Our equation is now solved. We've isolated **y** and defined it as 3x-4/2

While we can't express **y** as a particular value, we can calculate it for any value of **x**. For example, if **x** has a value of 6, then **y** can be calculated as:

\begin{equation}y = \frac{3\cdot6 - 4}{2} \end{equation}

This gives the result 14/2 which can be simplified to 7.

You can view the values of **y** for a range of **x** values by applying the equation to them using the following Python code:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the solved equation to x
df['y'] = (3*df['x'] - 4) / 2

#Display the dataframe
df
```

      x
      y

      0
      -10
      -17.0

      1
      -9
      -15.5

      2
      -8
      -14.0

      3
      -7
      -12.5

      4
      -6
      -11.0

      5
      -5
      -9.5

      6
      -4
      -8.0

      7
      -3
      -6.5

      8
      -2
      -5.0

      9
      -1
      -3.5

      10
      0
      -2.0

      11
      1
      -0.5

      12
      2
      1.0

      13
      3
      2.5

      14
      4
      4.0

      15
      5
      5.5

      16
      6
      7.0

      17
      7
      8.5

      18
      8
      10.0

      19
      9
      11.5

      20
      10
      13.0

We can also plot these values to visualize the relationship between x and y as a line. For this reason, equations that describe a relative relationship between two variables are known as *linear equations*:

```python
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey", marker = "o")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.show()
```

In a linear equation, a valid solution is described by an ordered pair of x and y values. For example, valid solutions to the linear equation above include:
- (-10, -17)
- (0, -2)
- (9, 11.5)

The cool thing about linear equations is that we can plot the points for some specific ordered pair solutions to create the line, and then interpolate the x value for any y value (or vice-versa) along the line.

## Intercepts
When we use a linear equation to plot a line, we can easily see where the line intersects the X and Y axes of the plot. These points are known as *intercepts*. The *x-intercept* is where the line intersects the X (horizontal) axis, and the *y-intercept* is where the line intersects the Y (horizontal) axis.

Let's take a look at the line from our linear equation with the X and Y axis shown through the origin (0,0).

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.show()
```

The x-intercept is the point where the line crosses the X axis, and at this point, the **y** value is always 0. Similarly, the y-intercept is where the line crosses the Y axis, at which point the **x** value is 0. So to find the intercepts, we need to solve the equation for **x** when **y** is 0.

For the x-intercept, our equation looks like this:

\begin{equation}0 = \frac{3x - 4}{2} \end{equation}

Which can be reversed to make it look more familar with the x expression on the left:

\begin{equation}\frac{3x - 4}{2} = 0 \end{equation}

We can multiply both sides by 2 to get rid of the fraction:

\begin{equation}3x - 4 = 0 \end{equation}

Then we can add 4 to both sides to get rid of the constant on the left:

\begin{equation}3x = 4 \end{equation}

And finally we can divide both sides by 3 to get the value for x:

\begin{equation}x = \frac{4}{3} \end{equation}

Which simplifies to:

\begin{equation}x = 1\frac{1}{3} \end{equation}

So the x-intercept is 11/3 (approximately 1.333).

To get the y-intercept, we solve the equation for y when x is 0:

\begin{equation}y = \frac{3\cdot0 - 4}{2} \end{equation}

Since 3 x 0 is 0, this can be simplified to:

\begin{equation}y = \frac{-4}{2} \end{equation}

-4 divided by 2 is -2, so:

\begin{equation}y = -2 \end{equation}

This gives us our y-intercept, so we can plot both intercepts on the graph:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.annotate('x-intercept',(1.333, 0))
plt.annotate('y-intercept',(0,-2))
plt.show()
```

The ability to calculate the intercepts for a linear equation is useful, because you can calculate only these two points and then draw a straight line through them to create the entire line for the equation.

## Slope
It's clear from the graph that the line from our linear equation describes a slope in which values increase as we travel up and to the right along the line. It can be useful to quantify the slope in terms of how much **x** increases (or decreases) for a given change in **y**. In the notation for this, we use the greek letter Δ (*delta*) to represent change:

\begin{equation}slope = \frac{\Delta{y}}{\Delta{x}} \end{equation}

Sometimes slope is represented by the variable ***m***, and the equation is written as:

\begin{equation}m = \frac{y_{2} - y_{1}}{x_{2} - x_{1}} \end{equation}

Although this form of the equation is a little more verbose, it gives us a clue as to how we calculate slope. What we need is any two ordered pairs of x,y values for the line - for example, we know that our line passes through the following two points:
- (0,-2)
- (6,7)

We can take the x and y values from the first pair, and label them x1 and y1; and then take the x and y values from the second point and label them x2 and y2. Then we can plug those into our slope equation:

\begin{equation}m = \frac{7 - -2}{6 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{7 + 2}{6 - 0} \end{equation}

That gives us the result 9/6 which is 11/2 or 1.5 .

So what does that actually mean? Well, it tells us that for every change of **1** in x, **y** changes by 11/2 or 1.5. So if we start from any point on the line and move one unit to the right (along the X axis), we'll need to move 1.5 units up (along the Y axis) to get back to the line.

You can plot the slope onto the original line with the following Python code to verify it fits:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# set the slope
m = 1.5

# get the y-intercept
yInt = -2

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

### Slope-Intercept Form
One of the great things about algebraic expressions is that you can write the same equation in multiple ways, or *forms*. The *slope-intercept form* is a specific way of writing a 2-variable linear equation so that the equation definition includes the slope and y-intercept. The generalised slope-intercept form looks like this:

\begin{equation}y = mx + b \end{equation}

In this notation, ***m*** is the slope and ***b*** is the y-intercept.

For example, let's look at the solved linear equation we've been working with so far in this section:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Now that we know the slope and y-intercept for the line that this equation defines, we can rewrite the equation as:

\begin{equation}y = 1\frac{1}{2}x + -2 \end{equation}

You can see intuitively that this is true. In our original form of the equation, to find y we multiply x by three, subtract 4, and divide by two - in other words, x is half of 3x - 4; which is 1.5x - 2. So these equations are equivalent, but the slope-intercept form has the advantages of being simpler, and including two key pieces of information we need to plot the line represented by the equation. We know the y-intecept that the line passes through (0, -2), and we know the slope of the line (for every x, we add 1.5 to y.

Let's recreate our set of test x and y values using the slope-intercept form of the equation, and plot them to prove that this  describes the same line:

```python
%matplotlib inline

import pandas as pd
from matplotlib import pyplot as plt

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Define slope and y-intercept
m = 1.5
yInt = -2

# Add a y column by applying the slope-intercept equation to x
df['y'] = m*df['x'] + yInt

# Plot the line
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# label the y-intercept
plt.annotate('y-intercept',(0,yInt))

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

```python

```

AI Professional/2 - Essential Mathematics For Artificial Intelligence/DAT256x/Module01/01-02-Linear Equations.ipynb


```python
from sympy import *
x, y, z, t = symbols('x y z t')
```

## Linear algebra

A matrix $A \in \mathbb{R}^{m\times n}$ is a rectangular array of real numbers with $m$ rows and $n$ columns.
To specify a matrix $A$, we specify the values for its $mn$ components $a_{11}, a_{12}, \ldots, a_{mn}$
as a list of lists:

```python
A = Matrix( [[ 2,-3,-8, 7],
             [-2,-1, 2,-7],
             [ 1, 0,-3, 6]] )
```

Use the square brackets to access the matrix elements or to obtain a submatrix:

```python
A[0,1]         # row 0, col 1 of A
```

$\displaystyle -3$

```python
A[0:2,0:3]     # top-left 2x3 submatrix of A
```

$\displaystyle \left[\begin{matrix}2 & -3 & -8\\-2 & -1 & 2\end{matrix}\right]$

Some commonly used matrices can be created with shortcut methods:

```python
eye(2)         # 2x2 identity matrix
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

```python
zeros(2, 3)
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

Standard algebraic operations like
addition `+`, subtraction `-`, multiplication `*`,
and exponentiation `**` work as expected for `Matrix` objects.
The `transpose` operation flips the matrix through its diagonal:

```python
A.transpose()  # the same as A.T
```

$\displaystyle \left[\begin{matrix}2 & -2 & 1\\-3 & -1 & 0\\-8 & 2 & -3\\7 & -7 & 6\end{matrix}\right]$

Recall that the transpose is also used to convert row vectors into column vectors and vice versa.

### Row operations

```python
M = eye(3)
M[1,:] = M[1,:] + 3*M[0,:]
M
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\3 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

The notation `M[i,:]` refers to entire rows of the matrix.
The first argument specifies the $0$-based row index,
for example the first row of~`M` is `M[0,:]`.
The code example above implements the row operation $R_2 \gets R_2 + 3R_1$.

To scale a row `i` by constant `c`, use the command `M[i,:] = c*M[i,:]`.

To swap rows `i` and `j`, use can use the `Python` tuple-assignment syntax `M[i,:], M[j,:] = M[j,:], M[i,:]`.

### Reduced row echelon form

The Gauss—Jordan elimination procedure is a sequence of row operations you can perform
on any matrix to bring it to its *reduced row echelon form* (RREF).
In `SymPy`, matrices have a `rref` method that computes their RREF:

```python
A = Matrix( [[2,-3,-8, 7],
             [-2,-1,2,-7],
             [1, 0,-3, 6]])
A.rref()  # RREF of A, location of pivots
```

    (Matrix([
     [1, 0, 0,  0],
     [0, 1, 0,  3],
     [0, 0, 1, -2]]),
     (0, 1, 2))

Note the `rref` method returns a tuple of values:
the first value is the RREF of $A$,
while the second tells you the indices of the leading ones (also known as pivots) in the RREF of $A$.
To get just the RREF of $A$, select the $0^\mathrm{th}$ entry form the tuple: `A.rref()[0]`.

### Matrix fundamental spaces

Consider the matrix $A \in \mathbb{R}^{m\times n}$.
The fundamental spaces of a matrix are its column space $\mathcal{C}(A)$,
its null space $\mathcal{N}(A)$,
and its row space $\mathcal{R}(A)$.
These vector spaces are important when you consider the matrix product
$A\vec{x}=\vec{y}$ as “applying” the linear transformation $T_A:\mathbb{R}^n \to \mathbb{R}^m$
to an input vector $\vec{x} \in \mathbb{R}^n$ to produce the output vector $\vec{y} \in \mathbb{R}^m$.

**Linear transformations** $T_A:\mathbb{R}^n \to \mathbb{R}^m$ (vector functions)
**are equivalent to $m\times n$ matrices**.
This is one of the fundamental ideas in linear algebra.
You can think of $T_A$ as the abstract description of the transformation
and $A \in \mathbb{R}^{m\times n}$ as a concrete implementation of $T_A$.
By this equivalence,
the fundamental spaces of a matrix $A$
tell us facts about the domain and image of the linear transformation $T_A$.
The columns space $\mathcal{C}(A)$ is the same as the image space space $\textrm{Im}(T_A)$ (the set of all possible outputs).
The null space $\mathcal{N}(A)$ is the same as the kernel $\textrm{Ker}(T_A)$ (the set of inputs that $T_A$ maps to the zero vector).
The row space $\mathcal{R}(A)$ is the orthogonal complement of the null space.
Input vectors in the row space of $A$ are in one-to-one correspondence with the output vectors in the column space of $A$.

Okay, enough theory! Let's see how to compute the fundamental spaces of the matrix $A$ defined above.
The non-zero rows in the reduced row echelon form of $A$ are a basis for its row space:

```python
[ A.rref()[0][r,:] for r in A.rref()[1] ]  # R(A)
```

    [Matrix([[1, 0, 0, 0]]), Matrix([[0, 1, 0, 3]]), Matrix([[0, 0, 1, -2]])]

The column space of $A$ is the span of the columns of $A$ that contain the pivots
in the reduced row echelon form of $A$:

```python
[ A[:,c] for c in  A.rref()[1] ]           # C(A)
```

    [Matrix([
     [ 2],
     [-2],
     [ 1]]),
     Matrix([
     [-3],
     [-1],
     [ 0]]),
     Matrix([
     [-8],
     [ 2],
     [-3]])]

Note we took columns from the original matrix $A$ and not its RREF.

To find the null space of $A$, call its `nullspace` method:

```python
A.nullspace()                              # N(A)
```

    [Matrix([
     [ 0],
     [-3],
     [ 2],
     [ 1]])]

### Determinants

The determinant of a matrix,
denoted $\det(A)$ or $|A|$,
is a particular way to multiply the entries of the matrix to produce a single number.

```python
M = Matrix( [[1, 2, 3],
             [2,-2, 4],
             [2, 2, 5]] )
M.det()
```

$\displaystyle 2$

Determinants are used for all kinds of tasks:
to compute areas and volumes,
to solve systems of equations,
and to check whether a matrix is invertible or not.

### Matrix inverse

For every invertible matrix $A$,
there exists an inverse matrix $A^{-1}$ which *undoes* the effect of $A$.
The cumulative effect of the product of $A$ and $A^{-1}$ (in any order)
is the identity matrix: $AA^{-1}= A^{-1}A=\mathbb{1}$.

```python
A = Matrix( [[1,2],
             [3,9]] )
A.inv()
```

$\displaystyle \left[\begin{matrix}3 & - \frac{2}{3}\\-1 & \frac{1}{3}\end{matrix}\right]$

```python
A.inv()*A
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

```python
A*A.inv()
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

The matrix inverse $A^{-1}$ plays the role of division by $A$.

### Eigenvectors and eigenvalues

When a matrix is multiplied by one of its eigenvectors the output
is the same eigenvector multiplied by a constant $A\vec{e}_\lambda =\lambda\vec{e}_\lambda$.
The constant $\lambda$ (the Greek letter *lambda*) is called an *eigenvalue* of $A$.

To find the eigenvalues of a matrix, start from the definition $A\vec{e}_\lambda =\lambda\vec{e}_\lambda$,
insert the identity $\mathbb{1}$,
and rewrite it as a null-space problem:

$$
A\vec{e}_\lambda =\lambda\mathbb{1}\vec{e}_\lambda
\qquad
\Rightarrow
\qquad
\left(A - \lambda\mathbb{1}\right)\vec{e}_\lambda = \vec{0}.
$$

This equation will have a solution whenever $|A - \lambda\mathbb{1}|=0$.(The invertible matrix theorem states
that a matrix has a non-empty null space if and only if its determinant is zero.)
The eigenvalues of $A \in \mathbb{R}^{n \times n}$,
denoted $\{ \lambda_1, \lambda_2, \ldots, \lambda_n \}$,\
are the roots of the *characteristic polynomial* $p(\lambda)=|A - \lambda \mathbb{1}|$.

```python
A = Matrix( [[ 9, -2],
             [-2,  6]] )
A.eigenvals()  # same as solve(det(A-eye(2)*x), x)
               # return eigenvalues with their multiplicity
```

    {10: 1, 5: 1}

```python
A.eigenvects()
```

    [(5,
      1,
      [Matrix([
       [1/2],
       [  1]])]),
     (10,
      1,
      [Matrix([
       [-2],
       [ 1]])])]

Certain matrices can be written entirely in terms of their eigenvectors and their eigenvalues.
Consider the matrix $\Lambda$ (capital Greek *L*) that has the eigenvalues of the matrix $A$ on the diagonal,
and the matrix $Q$ constructed from the eigenvectors of $A$ as columns:

$$
\Lambda =
\begin{bmatrix}
\lambda_1	&  \cdots  &  0 \\
\vdots 	&  \ddots  &  0  \\
0  	&   0      &  \lambda_n
\end{bmatrix}\!,
\ \
Q \: =
\begin{bmatrix}
|  &  & | \\
\vec{e}_{\lambda_1}  & \!  \cdots \! &  \large\vec{e}_{\lambda_n} \\
|  &  & |
\end{bmatrix}\!,
\ \
\textrm{then}
\ \
A = Q \Lambda Q^{-1}.
$$

Matrices that can be written this way are called *diagonalizable*.
To *diagonalize* a matrix $A$ is to find its $Q$ and $\Lambda$ matrices:

```python
Q, L = A.diagonalize()
Q            # the matrix of eigenvectors as columns
```

$\displaystyle \left[\begin{matrix}1 & -2\\2 & 1\end{matrix}\right]$

```python
Q.inv()
```

$\displaystyle \left[\begin{matrix}\frac{1}{5} & \frac{2}{5}\\- \frac{2}{5} & \frac{1}{5}\end{matrix}\right]$

```python
L            # the matrix of eigenvalues
```

$\displaystyle \left[\begin{matrix}5 & 0\\0 & 10\end{matrix}\right]$

```python
Q*L*Q.inv()  # eigendecomposition of A
```

$\displaystyle \left[\begin{matrix}9 & -2\\-2 & 6\end{matrix}\right]$

```python
Q.inv()*A*Q  # obtain L from A and Q
```

$\displaystyle \left[\begin{matrix}5 & 0\\0 & 10\end{matrix}\right]$

Not all matrices are diagonalizable.
You can check if a matrix is diagonalizable by calling its `is_diagonalizable` method:

```python
A.is_diagonalizable()
```

    True

```python
B = Matrix( [[1, 3],
            [0, 1]] )
B.is_diagonalizable()
```

    False

```python
B.eigenvals()  # eigenvalue 1 with multiplicity 2
```

    {1: 2}

```python
B.eigenvects()
```

    [(1,
      2,
      [Matrix([
       [1],
       [0]])])]

The matrix $B$ is not diagonalizable because it doesn't have a full set of eigenvectors.
To diagonalize a $2\times 2$ matrix, we need two orthogonal eigenvectors but $B$ has only a single eigenvector.
Therefore, we can't construct the matrix of eigenvectors $Q$ (we're missing a column!)
and so $B$ is not diagonalizable.

Non-square matrices don't have eigenvectors and therefore don't have an eigendecomposition.
Instead, we can use the *singular value decomposition* to break up a non-square matrix $A$ into
left singular vectors,
right singular vectors,
and a diagonal matrix of singular values.
Use the `singular_values` method on any matrix to find its singular values.


```python
import numpy as np
p = np.poly1d([1,0,0,0,0,0])
print (p)
print (p.integ())
p.integ()(1.0) - p.integ()(-1.0)
```

       5
    1 x
            6
    0.1667 x

    0.0

```python
from sympy import integrate, symbols
x, y = symbols('x y', real=True)
integrate(x**5, x)
```

    x**6/6

```python
integrate(x**5, (x, -1, 1))
```

    0

```python
from sympy import N, exp as Exp, sin as Sin
integrate(Exp(-x) * Sin(x), x)
```

    -exp(-x)*sin(x)/2 - exp(-x)*cos(x)/2

```python
integrate(Exp(-x) * Sin(x), (x, 0, 1))
```

    -exp(-1)*sin(1)/2 - exp(-1)*cos(1)/2 + 1/2

```python
N(_)
```

    0.245837007000237

```python
integrate(Sin(x) / x, x)
```

    Si(x)

```python
integrate(Sin(x) / x, (x, 0, 1))
```

    Si(1)

```python
N(_)
```

    0.946083070367183

```python
integrate(x**1, (x, 0, 1))
```

    1/2

```python
from sympy import oo
integrate(Exp(-x**2), (x,0,+oo))
```

    sqrt(pi)/2

```python

```


# The Gaussian (Normal) Distribution

## Probability Distribution Function

The Normal (Gaussian) distribution probability distribution function is

\begin{equation}
f\left(x; \mu, \sigma\right) = \frac{1}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2},
\end{equation}

normalized to unity,
$$
\int\limits_{-\infty}^{\infty} f\left(x; \mu,\sigma\right)\,dx = 1,
$$
is symmetrically distributed about its mean, $\mu$, with width $\sigma$.

## Full Width at Half Maximum (FWHM)

The full width at half maximum (FWHM) is the distance between points on a curve at which the function reaches half its maximum value. The FWHM is often used to describe the "width" of a distribution.

For a 1-dimensional Gaussian, it is seen that as the maximum value occurs as $x = \mu$ (by definition), half of the maximum value is

\begin{align*}
\left.\frac{1}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\right|_{x = x \text{ of (max/2)}} &= \frac{1}{2} f\left(x_{\text{max}}\right)\\
    &= \frac{1}{2} f\left(\mu\right) = \frac{1}{2} \frac{1}{\sqrt{2\pi}\sigma},
\end{align*}

resulting in the equality

\begin{equation*}
e^{-\left(x-\mu\right)^2/2\sigma^2} = \frac{1}{2},
\end{equation*}

which is (taking the log)

\begin{equation*}
-\frac{\left(x-\mu\right)^2}{2\sigma^2} = -\ln2.
\end{equation*}

Thus, solving the equality,
$$
\left(x-\mu\right)^2 = 2\sigma^2 \ln2,
$$
yields
$$
x_{\pm} = \pm \sigma \sqrt{2 \ln 2} + \mu.
$$

Thus, the FWHM is

\begin{align*}
\text{FWHM} &= x_{+} - x_{-}\\
    &= \left(\sigma \sqrt{2 \ln 2} + \mu\right) - \left(-\sigma \sqrt{2 \ln 2} + \mu\right)\\
    &= \boxed{2\sqrt{2\ln2}\sigma}\,.
\end{align*}

**N.B.:** It is seen that the FWHM for a Gaussian is _independent_ of both the normalization constant and the mean, and is only dependent on the standard devaiation of the Gaussian.

## Probability and the Error Function

The probability that a Normally distributed random variable will lie in a range of values symmetrically integrated over is given by

\begin{align*}
\text{Pr}\left(\mu - y \leq x \leq \mu + y\right) &= \int\limits_{\mu - y}^{\mu + y}\frac{1}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\,dx\\
    &= \int\limits_{\mu}^{\mu + y}\frac{2}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\,dx.
\end{align*}

Making the substitution
$$
t = \frac{\left(x-\mu\right)}{\sqrt{2}\sigma},
$$
then

\begin{align*}
\text{Pr}\left(\mu - y \leq x \leq \mu + y\right) &= \int\limits_{\mu}^{\mu + y}\frac{2}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\,dx\\
    &= \boxed{\frac{2}{\sqrt{\pi}} \int\limits_{0}^{y/\sqrt{2}\sigma}e^{-t^2}\,dt \equiv \text{erf}\left(\frac{y}{\sqrt{2}\sigma}\right)}\,.
\end{align*}

## Cumulative Distribution Function (cdf)

For the cumulative distribution function (cdf),
$$
\Phi\left(x\right) = \int\limits_{-\infty}^{x}f\left(t;\mu,\sigma\right)\,dt,
$$
it is seen, noting from the form of the error function, that

\begin{equation*}
\frac{2}{\sqrt{\pi}} \int\limits_{0}^{y/\sqrt{2}\sigma}e^{-t^2}\,dt = \frac{2}{\sqrt{2 \pi}} \int\limits_{0}^{y/\sigma}e^{-t^2/2}\,dt = \text{erf}\left(\frac{y}{\sqrt{2}\sigma}\right),
\end{equation*}

```python
import sympy as sym
sym.init_printing(use_unicode=True, wrap_line=False, no_global=True)
from sympy.abc import sigma
```

```python
t, y, = sym.symbols('t y')
sym.integrate((2/sym.sqrt(sym.pi)) * sym.exp(-t**2), (t, 0, y/(sym.sqrt(2) * sigma)))
```

```python
sym.integrate((2/sym.sqrt(2 * sym.pi)) * sym.exp(-t**2/2), (t, 0, y/sigma))
```

then for the standard Gaussian ($\mu = 0$, $\sigma=1$)

\begin{align*}
\Phi\left(x\right) &= \frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{x} e^{-t^2/2}\,dt\\
    &= \frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{0} e^{-t^2/2}\,dt + \frac{1}{\sqrt{2\pi}}\int\limits_{0}^{x} e^{-t^2/2}\,dt\\
    &= \frac{1}{2} + \frac{1}{2} \text{erf}\left(\frac{x}{\sqrt{2}}\right)\\
    &= \frac{1}{2} \left(1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)
\end{align*}

so it is likewise seen that

\begin{align*}
\text{Pr}\left(\mu - n \sigma \leq x \leq \mu + n\sigma\right) &= \Phi(n) - \Phi(-n)\\
    &= \Phi(n) - \left(1-\Phi(n)\right)\\
    &= \frac{1}{2} \left(1 + \text{erf}\left(\frac{n}{\sqrt{2}}\right)\right) - \left[1-\frac{1}{2} \left(1 + \text{erf}\left(\frac{n}{\sqrt{2}}\right)\right)\right]\\
    &= \text{erf}\left(\frac{n}{\sqrt{2}}\right).
\end{align*}

**Show this for:** generic normal distribution $f$ with mean $\mu$ and std $\sigma$

It is noted that in the case that $\left|y\right|=n\sigma$,
$$
\text{Pr}\left(\mu - y \leq x \leq \mu + y\right) = \text{Pr}\left(\mu - n\sigma \leq x \leq \mu + n\sigma\right) = \text{erf}\left(\frac{n}{\sqrt{2}}\right).
$$

So, for $n=1$,
$$
\text{Pr}\left(\mu - \sigma \leq x \leq \mu + \sigma\right) = \text{erf}\left(\frac{1}{\sqrt{2}}\right)
$$

However, at this point we are at an impass analytically, as the integral of a Gaussian function over a finite range has no analytical solution, and must be evaluated numerically.

```python
import math
from scipy import special as special
```

```python
def prob_n_sigma(n):
    return special.erf(n/math.sqrt(2.))
```

```python
prob_n_sigma(1)
```

## $p$-values

### Two-Tailed  $p$-value

### One-Tailed  $p$-value


$$f(x) = x^2$$

### The probability of getting (k) heads when flipping (n) coins is
\begin{equation*}
P(E)   = {n \choose k} p^k (1-p)^{ n-k}
\end{equation*}

\begin{equation}
\int y\, \mathrm{d}x
\end{equation}

## Mean, Median, Mode

Mean, median, and mode are different measures of center in a numerical data set. They each try to summarize a dataset with a single number to represent a "typical" data point from the dataset.

__Mean:__ The "average" number; found by adding all data points and dividing by the number of data points.

\begin{equation}
\bar{y} = \frac{\sum y}{N}
\end{equation}

- $\sum y$ Take all the values and add them up
- divide by the total number of observations you have

__Median__ The middle number found by ordering all data points and picking out the one in the middle (or if there are two middle numbers, taking the mean of those two numbers).

To get the median

\begin{equation}
y = \tilde{x}
\end{equation}

$ y = [1,2,3,4,4,3,2,1] $

$ y = 4 $

The __mode__ is the most commonly occurring data point in a dataset. The mode is useful when there are a lot of repeated values in a dataset. There can be no mode, one mode, or multiple modes in a dataset.

## Standard Deviation

__Standard deviation__ is a number used to tell how measurements for a group are spread out from the average (mean), or expected value. A low standard deviation means that most of the numbers are close to the average. A high standard deviation means that the numbers are more spread out.

$$s = \sqrt{\frac{1}{N-1} \sum_{i=1}^N (x_i - \overline{x})^2}$$

## Variance

The __Variance__ is defined as: The average of the squared differences from the Mean. To calculate the variance follow these steps: Find the __Mean__, Then for each number: subtract the __Mean__ and square the result (the squared difference).

$$\sigma^2 = \frac{\displaystyle\sum_{i=1}^{n}(x_i - \mu)^2} {n}$$

## What is RMSE

 $RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}{\Big(\frac{d_i -f_i}{\sum_i}\Big)^2}}$

 ## Root mean squared error (RMSE)
RMSE is a quadratic scoring rule that also measures the average magnitude of the error. It’s the square root of the average squared differences between prediction and actual observation.

 $RMSE = \sqrt{\frac{1}{n}\Sigma_{i=1}^{n}{\Big(\frac{d_i -f_i}{\Sigma_i}\Big)^2}}$

 we can do this simply from sklearn:

```python
from sklearn.metrics import mean_squared_error
from math import sqrt
rmse = sqrt(mean_squared_error(y_actual, y_predicted))
print(rmse)

# another way
n = len(predictions)
rmse = np.linalg.norm(predictions - targets) / np.sqrt(n)

# and another way
from sklearn.metrics import mean_squared_error
from math import sqrt
expected = [0.0, 0.5, 0.0, 0.5, 0.0]
predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
mse = mean_squared_error(expected, predictions)
rmse = sqrt(mse)
print('RMSE: %f' % rmse)

```
The mean square root and square root of it will be useful.

```python
#pip install sklearn
```

```python
from sklearn.metrics import mean_squared_error
from math import sqrt
expected = [0.0, 0.5, 0.0, 0.5, 0.0]
predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
mse = mean_squared_error(expected, predictions)
rmse = sqrt(mse)
print('RMSE: %f' % rmse)
```

    RMSE: 0.148324

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/RECAP_DS/01_BASIC_STATISTICS/01_Math_Basics.ipynb


# Generating Functions

Generating functions are functions that encode sequences of numbers as the coefficients of power series.

Consider a set $S$ with $n$ elements.

Pretend there is a picture function' $P(s)\ \forall s\in S$.

The picture function enables one, for example, to write the multiset $\{1,1,2\}$ as an expression $P(1)^2P(2)$.

The enumerating function $E_p(s)\ :s\in S$ is used to list combinations subject to restraints. For example, the enumerating function for all multisets that include either one or two times some element $a$ and between zero and two times some element $b$ is written:

\begin{equation}
\begin{array}{rl}
E_P(s) &= P(a)+ P(a)^2 + P(a)P(b) + P(a)^2P(b) +  + P(a)P(b)^2 + P(a)^2P(b)^2\\
&\left(P(a) + P(a)^2\right)\left(1 + P(b) + P(b)^2\right)
\end{array}
\end{equation}

This hints at how generating functions can make it easier to keep track of combinations.

## Example: Binomial Coefficients

Consider the case of a collection of $n$ indistinguishable objects $s$, and write $P(s) = x$. Then the enumerator for selecting any subset of those $n$ objects is given by:

\begin{equation}
E_P(s) = \prod_i^n (x^0+x^1) = (1+x)^n = \sum_i^n {n \choose i}x^i
\end{equation}

Where $(x^0 + x^1)$ corresponds to including an element zero or one times. The exponent encodes how many objects were included into a particular subset. This is one way of "proving" the binomial theorem, and one can says $(1_x)^n$ is the generating function for the binomial coefficients ${n \choose i}$.

## Example: Basket of Goods

An apple costs $20c$, a pear costs $25c$ and a banana costs $30c$. How many different fruit baskets can be bought for $100c$?

By replacing the picture function $P(s)$ with $x$, it was possible to identify subsets of $n$ objects by looking at the exponent of $x^n$ in the enumerating function. In this case, the exponent is supposed to show the price. This can be done by writing $P(apple) = x^{20}, P(pear) = x^{25}$ and $P(banana) = x^{30}$.

\begin{equation}
E_P(s) = \left( \sum_{i=0}^5 x^{20} \right)\left( \sum_{i=0}^4 x^{25} \right)\left( \sum_{i=0}^3 x^{30} \right)
\end{equation}

Which results in some power series of the form:

\begin{equation}
E_P(s) = 1x^0 + 1x^{20} + 1x^{25} + 1x^{30} + 1x^{40} +... + 2x^{60} + ... + 1x^{290}
\end{equation}

To obtain the number of combinations that correspond to a cost of exactly $100c$, one can apply the operator $\frac{1}{n!}\frac{d^n}{dx^n}$ and set $x=0$ to obtain the desired term. This is what is done with moment generating functions in statistics.

But actually it is easier to think through what the coefficients will be so that:

\begin{equation}
\sum_{l=0}^{n+m+h}d_l x^l = \left(\sum_{i=0}^n a_i x^i\right)\left(\sum_{j=0}^m b_j x^j\right)\left(\sum_{k=0}^h c_k x^k\right)
\end{equation}

\begin{equation}
d_l = \sum_{\begin{array}{c}i,j,k\\i+j+k=l\end{array}} a_i b_j c_k
\end{equation}

If the coefficients are all '1', which is the case for these combinations, then:

\begin{equation}
d_l = \sum_{\begin{array}{c}i,j,k\\i+j+k=l\end{array}} 1 = {l + 3 - 1 \choose l}
\end{equation}

Which is the number of all multisets of size $l$ and 3 classes. In this case the coefficients are equal to 1 for i=20, j=25, and k=30, and zero otherwise, so that there are 4 ways of summing to 100. Hence, $d_{100} = 4$ for the basket of goods above.

The sum can also be rewritten:

\begin{equation}
d_l = \sum^l_{i=0}\sum^{l-i}_{j=0}\sum^{l-i-j}_{k= l-i-j} a_i b_j c_k
\end{equation}

Where the last sum is actually just over a single term.

Of course, this is the discrete version of a convolution. So, it is no surprise that the addition of random variables winds up being a convolution.

## Example: Dice

How many ways are there for $n$ dice with $k$ faces to show $s$ eyes?

\begin{equation}
E_P = \left(\sum_{i=0}^\infty a_i x^i\right)^n = \sum_{i=0}^{\infty}d_i x^i
\end{equation}

Where $a_i = 1\ \forall i\in(1,k)$ and $a_i = 0$ otherwise.

\begin{equation}
E_P = \left(\sum_{i=1}^k x^i\right)^n = \left(x(1-x^k)\sum_{i=0}^\infty x^i\right)^n = x^n\left(\frac{1-x^k}{1-x}\right)^n = x^n \left(\sum_{i=0}^n (-1)^i {n \choose i} x^{ik}\right)\left(\sum_{j=0}^{\infty} (-1)^j {-n \choose j} x^j\right)
\end{equation}

The coefficient for $x^s$ is given the sum:

\begin{equation}
d_s = \sum_{ki+j=s-n} (-1)^{i+j}{n \choose i}{-n \choose j}\\
i \in [0,n] \\
j \in [0,\infty]
\end{equation}

Where the indices $i$ and $j$ satisfy $ki+j = s-n$. For example, for $s=7$, $n=2$ and $k=6$:

\begin{equation}
6i+j = 7-2 = 5
\end{equation}

Holds for $i=0, j=5$:

\begin{equation}
d_s = (-1)^5 {2 \choose 0}{-2 \choose 5} = (-1)^{10} 1 {2+5-1 \choose 5} = {6 \choose 5} = \frac{6!}{5!1!} = 6
\end{equation}

Indeed, there are 6 ways for two d6 to add to 7:

\begin{equation}
[6,1],[5,2],[4,3],[3,4],[2,5],[1,6]
\end{equation}

```python
import warnings

def memoize(func):
    """
    memoizing wrapper to speed up recursion by keeping track of previously calculated values.
    """
    S = {}
    def wrappingfunction(*args):
        if args not in S:
            S[args] = func(*args)
        return S[args]
    return wrappingfunction

@memoize
def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
        return res

@memoize
def binomial(n,k):
    if n > 40 or k > 40:
        warnings.warn('careful with large n or k - unresolved numerical stability issues.')

    if n >= k and k>= 0:
        return int(factorial(n)/(factorial(n-k)*factorial(k)))
    elif n < 0 and k >= 0:
        return int((-1)**k * binomial(-n+k-1,k))
    elif n < 0 and k <= n:
        return int((-1)**(n-k) * binomial(-k-1,n-k))
    else:
        return 0

f = lambda x: str(x) if x != 0 else '-'
N = 5

print('\t'.join(['     ']+['k=%i' % k for k in range(-N,N+1)]))
for n in range(-N,N+1)[::-1]:
    print('\t'.join(['n=%i' % n]+[f(binomial(n,k)) for k in range(-N,N+1)]))
```

         	k=-5	k=-4	k=-3	k=-2	k=-1	k=0	k=1	k=2	k=3	k=4	k=5
    n=5	-	-	-	-	-	1	5	10	10	5	1
    n=4	-	-	-	-	-	1	4	6	4	1	-
    n=3	-	-	-	-	-	1	3	3	1	-	-
    n=2	-	-	-	-	-	1	2	1	-	-	-
    n=1	-	-	-	-	-	1	1	-	-	-	-
    n=0	-	-	-	-	-	1	-	-	-	-	-
    n=-1	1	-1	1	-1	1	1	-1	1	-1	1	-1
    n=-2	-4	3	-2	1	-	1	-2	3	-4	5	-6
    n=-3	6	-3	1	-	-	1	-3	6	-10	15	-21
    n=-4	-4	1	-	-	-	1	-4	10	-20	35	-56
    n=-5	1	-	-	-	-	1	-5	15	-35	70	-126

```python
def d_s(s,n=2,k=6):
    """
    Number of Dice Combinations

    n = number of dice
    k = number of dice faces (faces: 1,2,3,4,5,...,k)
    s = sum of dice throw

    Beware of numerical stability issues for N >~ 45
    """
    res = 0
    for i in range(0,int((s-n)/k)+1):
        j = s-n-k*i
        res += (-1)**(i+j) * binomial(n,i) * binomial(-n,j)
    return res

print('2 d6')
for s in range(1,14):
    print('s = %i\t' % s, 'd = %i' % d_s(s,n=2,k=6))
```

    2 d6
    s = 1	 d = 0
    s = 2	 d = 1
    s = 3	 d = 2
    s = 4	 d = 3
    s = 5	 d = 4
    s = 6	 d = 5
    s = 7	 d = 6
    s = 8	 d = 5
    s = 9	 d = 4
    s = 10	 d = 3
    s = 11	 d = 2
    s = 12	 d = 1
    s = 13	 d = 0

```python
import matplotlib.pyplot as plt
import numpy as np

k = 6

N = 31
ss = list(range(1,N*(k-2)))
nn = list(range(1,N))

plt.figure(figsize=(12,5))
for n in nn:
    d = [d_s(s,n=n,k=k) for s in ss]
    d = np.array(d)/np.sum(d)

    plt.plot(ss,d,'.-',linewidth=2,markersize=8)
    plt.text(x=3.5*n+0.2,y=max(d)+0.0025,s='%i' % n)

plt.xlabel('s')
plt.ylabel('p(s)')
plt.title('Probability of d6 Throw')
plt.savefig('./img/d6throw.png')

xlim = plt.xlim()

k = 20

N = 31
ss = list(range(1,N*(k-2)))
nn = list(range(1,N))

plt.figure(figsize=(12,5))
for n in nn:
    d = [d_s(s,n=n,k=k) for s in ss]
    d = np.array(d)/np.sum(d)

    plt.plot(ss,d,'.-',linewidth=2,markersize=8)
    if 10.5*n <= max(xlim):
        plt.text(x=10.5*n+0.2,y=max(d)+0.00075,s='%i' % n)

plt.xlim(*xlim)
plt.xlabel('s')
plt.ylabel('p(s)')
plt.title('Probability of d20 Throw')
plt.savefig('./img/d20throw.png')
```

```python

```

Combinatorics - Generating Functions.ipynb


# Basic Naive Bayes

## Theory

Naive Bayes classifiers attempt to divine the probability of each class given a data set.  If we represent the predicted class as a random variable $Y$ and the set of features as a random variable $X$, then naive Bayes predicts a class from data $x_i$ by selecting the class $y_i$ that have the highest value of $P(Y=y_i|X=x_i)$.  In statistical nomenclature, the class with the highest posterior probability.  So training consists of figuring out how we can estimate these posterior probabilities from a set of training data.

To begin, let us assume that we are trying to predict the chance of borrower default given some historical data:

| married | diet | defaulted |
| --- | ---| ---|
| yes | conventional | no |
| no | conventional | yes |
| no | vegan | yes |
| yes | vegetarian | no |
| no | gluten-free | no |
| yes | West Coast | yes |

Given this data, what is the probability that a married vegan will default?

We can attempt to solve this with the standard conditional probability definition, $$P(default | (married \cap vegan)) = \frac{P(default \cap married \cap vegan)}{P(married \cap vegan)} $$ but find that this leaves us trying to approximate the joint probability of our features.  We likely didn't gather data with this estimation in mind (we are attempting to correlate attributes with our predicted class, not other attributes).  If we assume indepedence of attributes, our denominator is always 0.  Instead, we would like to estimate $P(default | (married \cap vegan))$ based on the relationships between our attributes and our predicted class.  We can accomplish this using Bayes' formula, which is the consequence of the following equalities:
$$ P(Y|X) = \frac{P(Y \cap X)}{P(X)} $$
$$ P(X|Y) = \frac{P(X \cap Y)}{P(Y)} $$
$$P(X \cap Y) = P(Y \cap X)$$

Solving the first two for their joint probabilities, equating them, and then dividing by $P(X)$ yields Bayes formula:

$$ P(Y | X) = \frac{P(X|Y)P(Y)}{P(X)} $$

As noted above, we can make classify data $x_i$ by selecting the class $y_i$ that has the highest value of $P(Y=y_i|X=x_i)$.  Since $P(X)$ is constant across classes, we can ignore it when searching for the highest posterior.  Estimating $P(Y=y_i)$ from the training data is easy as it is just the proportion of labels that belong to a particular class.  Estimating $P(X | Y)$ is more difficult, since X is a vector of attributes and therefore $P(X=x_i)$ is shorthand for $P(x_0 = x_{i0} \cap x_1 = x_{i1} \cap ... \cap x_m = x_{im})$.  With __naive bayes__, this calculation is greatly simplified by assuming that the attributes are conditionally independent.  This assumption yields:

$$P((x_0 = x_{i0} \cap x_1 = x_{i1} \cap ... \cap x_m = x_{im}) | Y=y) = \prod_{j=0}^{m}{P(x_j=x_{ij} | Y=y)}$$

Consequently, we can estimate our posterior (dropping the class constant denominator) as:

$$P(Y=y | X=x_i) \sim = \prod_{j=0}^{m}{P(x_j=x_{ij} | Y=y)}*P(Y=y)$$

### Training

The learned 'model' is a set of prior probabilities (i.e. $P(Y=y)$ for all classes $y$) and a set of conditional probabilities (i.e. $P(x_j=x_{ij} | Y=y$).  We learn the priors based on class proportion in the training set and calculate the conditional probabilities based on the training data.

### Classifying

To classify, we calculate the posterior (i.e. $P(Y=y | X=x_i)$) for every class y using our learned priors and conditional probabilities.  Our label is the class with the highest posterior.

### Misc

Arguably the most useful characteristic of a naive bayes classifier is that it returns a probability for each class, which can be used to guage classification confidence.

### Exercise

Given our borrower default data, calculate the naive bayes 'model' and use it to predict if a married vegan will default on their loan.

---
Our model:

| model param | value |
|---|---|
| $P(defaulted=yes)$ | 0.5|
| $P(defaulted=no)$ | 0.5|
| $P(married=yes | defaulted=yes)$ | 0.333|
| $P(married=no | defaulted=yes)$ | 0.666|
| $P(married=yes | defaulted=no)$ | 0.666|
| $P(married=no | defaulted=no)$ | 0.333|
| $P(diet=conventional | defaulted=yes)$ | 0.333|
| $P(diet=conventional | defaulted=no)$ | 0.333|
| ... | ... |
| $P(diet=West Coast | defaulted=yes)$ | 0.333|
| $P(diet=West Coast | defaulted=no)$ | 0|

Our posteriors:

$\begin{align}
 P(default=yes|(married=yes \cap diet=vegan)) &= \\
 & =  P(married=yes | defaulted = yes) * P(diet=vegan | defaulted = yes) * P(defaulted=yes)\\
 & = 0.333 * 0.333 * 0.5 \\
 & = 0.055
\end{align}$

$\begin{align}
 P(default=no|(married=yes \cap diet=vegan)) &= \\
 & =  P(married=yes | defaulted = no) * P(diet=vegan | defaulted = no) * P(defaulted=no)\\
 & = 0.666 * \rho * 0.5 \\
 & = 0.166\rho
\end{align}$

We see how to actually deal with 0 conditional probabilities later but for now lets just say that $\rho=0.333$, so we have the posterior for default=yes  is 0.055 and for default=no is 0.0833.  Therefore, our model would predict that a married vegan would not default.

### Edge cases

#### Conditionals of zero
As we saw in our example above, a zero conditional can occur when there is too little data (esp. with regard to a single attribute).  Since a zero conditional will zero out the product of conditionals, it is important to come up with a heuristic for dealing with them. A common approach is to use the m-estimate:

$$P(x_i|y_i)=\frac{n_c+mp}{n+m}$$

where $m$ (equivalent sample size) and $p$ are parameters, $n_c$ is the number of records with attribute value $x_i$ and class $y_i$, and n is the number of records with class $y_i$.  We note that if there is no training set then $P(x_i|y_i)=p$, so $p$ can be conceived of as the prior probability of attribute value $x_i$ given class $y_i$.

#### Estimating conditionals for continuous attributes
The astute student might notice that directly calculating conditionals for continuous variables would be impossible since any single point is unlikely to occur multiple times in our training set.  The simplest way to deal with this issue is discretization; divide up the range of values into regions, map the values to regions, and treat the region 'labels' as categorical variables.  The problem with this approach is that the analyst must select the regions and this is prone to error.

Alternatively, we can assume that the data fits a specific distribution (e.g. Gaussian) and use the class conditional probability of that distribution as our conditional.  Assuming a Gaussian distribution, we have:

$$P(X_i=x|Y=y_j)=\frac{1}{\sqrt{2\pi}\sigma_{ij}}e^{-\frac{(x-\mu_{ij})^2}{2\sigma^2_{ij}}}$$

Consequently, during the training phase we calculate the sample mean and sample variance of the attribute $X_i$'s values for records with class $y_j$ and use those values to estimate $\mu_{ij}$ and $\sigma_{ij}$ respectively.  At that point, we have a function for determining the conditional for a particular attribute value and class.

## Implementation

```python
#Assumptions : Only string (nominal values) or numeric fields
import pandas as pd
import math

class naive_bayes:

    def __init__(self, equivalent_sample_size=1, default_attr_prior=0.1):
        self._prior = default_attr_prior
        self._eq_sample_size = equivalent_sample_size

    def _m_cond(self, class_cnt, class_attr_cnt):
        return (class_attr_cnt + (self._eq_sample_size*self._prior)) / (class_cnt + self._eq_sample_size)

    def _calc_priors(self, labels):
        return labels.value_counts() / len(labels)

    def _categorical_conditionals(self, df):
        "Assumes 2 columns, attribute and label, and the attribute values are categorical"
        attr_col = df.columns[0]
        class_col = df.columns[1]

        conds = []
        for y, df in data.groupby(class_col):
            class_count = len(df)
            conds += [[val,self._m_cond(class_count, cnt),y] for val,cnt in df[attr_col].value_counts().items()]

        return pd.DataFrame(conds,columns=['Value','Probability','Class'])

    def _continuous_conditionals(self, df):
        "Assumes 2 columns, attribute and label, and the attribute values are numeric"
        attr_col = df.columns[0]
        class_col = df.columns[1]

        stats = []
        for y, df in data.groupby(class_col):
            stats += [[df[attr_col].mean(),df[attr_col].std(),y]]

        return pd.DataFrame(stats,columns=['Mean','Std','Class'])

    def fit(self, data,class_column):
        result = {}

        for col,is_numeric in ((data.dtypes == 'float64') | (data.dtypes == 'int64')).items():
            if col != class_column:
                result[col] = {'type' : 'continuous', 'conditionals' : self._continuous_conditionals(data[[col,class_column]])} if is_numeric else {'type' : 'categorical', 'conditionals' : self._categorical_conditionals(data[[col,class_column]])}

        self._model = result
        self._priors = self._calc_priors(data[class_column])

    def _gaussian(self, val, params):
        std = params['Std']
        return (1.0/(math.sqrt(2*math.pi) * std)) * math.exp(-((val-params['Mean'])**2)/(2*(std**2)))

    def _calc_posterior(self, record, col):
        model_params = self._model[col]
        x = record[col]
        conds = model_params['conditionals']

        if model_params['type'] == 'categorical':
            return conds[conds['Value'] == x][['Probability','Class']]

        #else continuous
        gauss = lambda stats : pd.Series([self._gaussian(x,stats), stats['Class']])
        tmp = conds.apply(gauss,axis=1)
        return tmp.rename(columns={0 : 'Probability', 1 : 'Class'})

    def predict(self, record):
        if not hasattr(self, '_model'):
            raise Exception('The model has not been fitted, prediction is impossible')

        tmp_df = pd.DataFrame()
        for col in record.index:
            tmp_df = tmp_df.append(self._calc_posterior(record,col),ignore_index=True)

        posteriors = tmp_df.groupby('Class').prod()['Probability']*self._priors
        print(posteriors)

        return posteriors.idxmax()

```

```python
data = pd.DataFrame({'Diet' : ['conventional','conventional','vegan','vegetarian','gluten-free','west-coast'],
                    'Married' : ['yes','no','no','yes','no','yes'],
                    'Salary' : [120000.0,34000.0,54000.0,75000.0,90000.0,65000.0],
                    'Default' : ['no','yes','yes','no','no','yes']})

nb = naive_bayes()

nb.fit(data,'Default')

new_record = pd.Series(['east-coast','no',50000.00],index=['Diet','Married','Salary'])

nb.predict(data.iloc[0,:-1])
nb.predict(data.iloc[1,:-1])
nb.predict(new_record)

```

    no     6.930807e-07
    yes    6.260629e-11
    dtype: float64
    no     1.902869e-08
    yes    1.020828e-06
    dtype: float64
    no     3.479887e-07
    yes    6.649849e-06
    dtype: float64

    'yes'