```python
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
init_printing()
```

## Calculus

Calculus is the study of the properties of functions. The operations of
calculus are used to describe the limit behaviour of functions, calculate
their rates of change, and calculate the areas under their graphs. In
this section we'll learn about the `SymPy` functions for calculating
limits, derivatives, integrals, and summations.

### Infinity

The infinity symbol is denoted `oo` (two lowercase `o`s) in `SymPy`. Infinity
is not a number but a process: the process of counting forever. Thus,
$\infty + 1 = \infty$, $\infty$ is greater than any finite number, and $1/\infty$ is an
infinitely small number. `Sympy` knows how to correctly treat infinity
in expressions:

```python
oo+1
```

```python
5000 < oo
```

```python
1/oo
```

### Limits

We use limits to describe, with mathematical precision, infinitely large
quantities, infinitely small quantities, and procedures with infinitely
many steps.

The number $e$ is defined as the limit $e \equiv \lim_{n\to\infty}\left(1+\frac{1}{n}\right)^n$:

```python
limit( (1+1/n)**n, n, oo)
```

This limit expression describes the annual growth rate of a loan with
a nominal interest rate of 100% and infinitely frequent compounding.
Borrow \$1000 in such a scheme, and you'll owe $2718.28 after one year.

Limits are also useful to describe the behaviour of functions. Consider
the function $f(x) = \frac{1}{x}$. The `limit` command shows us what happens
to $f(x)$ near $x = 0$ and as $x$ goes to infinity:

```python
limit( 1/x, x, 0, dir="+")
```

```python
limit( 1/x, x, 0, dir="-")
```

```python
limit( 1/x, x, oo)
```

As $x$ becomes larger and larger, the fraction $\frac{1}{x}$ becomes smaller
and smaller. In the limit where $x$ goes to infinity, $\frac{1}{x}$ approaches
zero: $\lim_{x\to\infty}\frac{1}{x} = 0$. On the other hand, when $x$ takes on smaller
and smaller positive values, the expression $\frac{1}{x}$ becomes infinite:
$\lim_{x\to0^+}\frac{1}{x} = \infty$. When $x$ approaches 0 from the left, we have
$\lim_{x\to0^-}\frac{1}{x}=-\infty$. If these calculations are not clear to you, study
the graph of $f(x) = \frac{1}{x}$.

Here are some other examples of limits:

```python
limit(sin(x)/x, x, 0)
```

```python
limit(sin(x)**2/x, x, 0)
```

```python
limit(exp(x)/x**100,x,oo)  # which is bigger e^x or x^100 ?
                           # exp f >> all poly f for big x
```

Limits are used to define the derivative and the integral operations.

### Derivatives

The derivative function, denoted $f'(x)$, $\frac{d}{dx}f(x)$, $\frac{df}{dx}$, or $\frac{dy}{dx}$,
describes the *rate of change* of the function $f(x)$.
The `SymPy` function `diff` computes the derivative of any expression:

```python
diff(x**3, x)
```

The differentiation operation knows about the product rule $[f(x)g(x)]^\prime=f^\prime(x)g(x)+f(x)g^\prime(x)$,
the chain rule $f(g(x))' = f'(g(x))g'(x)$,
and the quotient rule $\left[\frac{f(x)}{g(x)}\right]^\prime = \frac{f'(x)g(x) - f(x)g'(x)}{g(x)^2}$:

```python
diff( x**2*sin(x), x )
```

```python
diff( sin(x**2), x )
```

```python
diff( x**2/sin(x), x )
```

The second derivative of a function `f` is `diff(f,x,2)`:

```python
diff(x**3, x, 2)   # same as diff(diff(x**3, x), x)
```

The exponential function $f(x)=e^x$ is special because it is equal to its derivative:

```python
diff( exp(x), x )  # same as diff( E**x, x  )
```

A differential equation is an equation that relates some unknown function $f(x)$ to its derivative.
An example of a differential equation is $f'(x)=f(x)$.
What is the function $f(x)$ which is equal to its derivative?
You can either try to guess what $f(x)$ is or use the `dsolve` function:

```python
x = symbols('x')
f = symbols('f', cls=Function)  # can now use f(x)
dsolve( f(x) - diff(f(x),x), f(x) )
```

We'll discuss `dsolve` again in the section on mechanics.

### Tangent lines

The *tangent line* to the function $f(x)$ at $x=x_0$ is
the line that passes through the point $(x_0, f(x_0))$ and has
the same slope as the function at that point.
The tangent line to the function $f(x)$ at the point $x=x_0$ is described by the equation

$$
   T_1(x) =  f(x_0) \ + \  f'(x_0)(x-x_0).
$$

What is the equation of the tangent line to $f(x)=\frac{1}{2}x^2$ at $x_0=1$?

```python
f = S('1/2')*x**2
f
```

```python
df = diff(f,x)
df
```

```python
T_1 = f.subs({x:1}) + df.subs({x:1})*(x - 1)
T_1
```

The tangent line $T_1(x)$ has the same value and slope as the function $f(x)$ at $x=1$:

```python
T_1.subs({x:1}) == f.subs({x:1})
```

    True

```python
diff(T_1,x).subs({x:1}) == diff(f,x).subs({x:1})
```

    True

### Optimization

Optimization is about choosing an input for a function $f(x)$ that results in the best value for $f(x)$.
The best value usually means the *maximum* value
(if the function represents something desirable like profits)
or the *minimum* value
(if the function represents something undesirable like costs).

The derivative $f'(x)$ encodes the information about the *slope* of $f(x)$.
Positive slope $f'(x)>0$ means $f(x)$ is increasing,
negative slope $f'(x)<0$ means $f(x)$ is decreasing,
and zero slope $f'(x)=0$ means the graph of the function is horizontal.
The *critical points* of a function $f(x)$ are the solutions to the equation $f'(x)=0$.
Each critical point is a candidate to be either a maximum or a minimum of the function.

The second derivative $f^{\prime\prime}(x)$ encodes the information about the *curvature* of $f(x)$.
Positive curvature means the function looks like $x^2$,
negative curvature means the function looks like $-x^2$.

Let's find the critical points of the function $f(x)=x^3-2x^2+x$
and use the information from its second derivative
to find the maximum of the function
on the interval $x \in [0,1]$.

```python
x = Symbol('x')
f = x**3-2*x**2+x
diff(f, x)
```

```python
sols = solve( diff(f,x),  x)
sols
```

```python
diff(diff(f,x), x).subs( {x:sols[0]} )
```

```python
diff(diff(f,x), x).subs( {x:sols[1]} )
```

[It will help to look at the graph of this function.](https://www.google.com/#safe=off&q=plot+x**3-2*x**2%2Bx)
The point $x=\frac{1}{3}$ is a local maximum because it is a critical point of $f(x)$
where the curvature is negative, meaning $f(x)$ looks like the peak of a mountain at $x=\frac{1}{3}$.
The maximum value of $f(x)$ on the interval $x\in [0,1]$ is $f\!\left(\frac{1}{3}\right)=\frac{4}{27}$.
The point $x=1$ is a local minimum because it is a critical point
with positive curvature, meaning $f(x)$ looks like the bottom of a valley at $x=1$.

### Integrals

The *integral* of $f(x)$ corresponds to the computation of the area under the graph of $f(x)$.
The area under $f(x)$ between the points $x=a$ and $x=b$ is denoted as follows:

$$
 A(a,b) = \int_a^b f(x) \: dx.
$$

The *integral function* $F$ corresponds to the area calculation as a function
of the upper limit of integration:

$$
  F(c) \equiv \int_0^c \! f(x)\:dx\,.
$$

The area under $f(x)$ between $x=a$ and $x=b$ is obtained by
calculating the *change* in the integral function:

$$
   A(a,b) = \int_a^b \! f(x)\:dx  =  F(b)-F(a).
$$

In `SymPy` we use `integrate(f, x)` to obtain the integral function $F(x)$ of any function $f(x)$:
$F(x) = \int_0^x f(u)\,du$.

```python
integrate(x**3, x)
```

```python
integrate(sin(x), x)
```

```python
integrate(ln(x), x)
```

This is known as an *indefinite integral* since the limits of integration are not defined.

In contrast,
a *definite integral* computes the area under $f(x)$ between $x=a$ and $x=b$.
Use `integrate(f, (x,a,b))` to compute the definite integrals of the form $A(a,b)=\int_a^b f(x) \, dx$:

```python
integrate(x**3, (x,0,1))  # the area under x^3 from x=0 to x=1
```

We can obtain the same area by first calculating the indefinite integral $F(c)=\int_0^c \!f(x)\,dx$,
then using $A(a,b) = F(x)\big\vert_a^b \equiv F(b) - F(a)$:

```python
F = integrate(x**3, x)
F.subs({x:1}) - F.subs({x:0})
```

Integrals correspond to *signed* area calculations:

```python
integrate(sin(x), (x,0,pi))
```

```python
integrate(sin(x), (x,pi,2*pi))
```

```python
integrate(sin(x), (x,0,2*pi))
```

During the first half of its $2\pi$-cycle,
the graph of $\sin(x)$ is above the $x$-axis, so it has a positive contribution to the area under the curve.
During the second half of its cycle (from $x=\pi$ to $x=2\pi$),
$\sin(x)$ is below the $x$-axis, so it contributes negative area.
Draw a graph of $\sin(x)$ to see what is going on.

### Fundamental theorem of calculus

The integral is the “inverse operation” of the derivative.
If you perform the integral operation followed by the derivative operation on some function,
you'll obtain the same function:

$$
  \left(\frac{d}{dx} \circ \int dx \right) f(x) \equiv \frac{d}{dx} \int_c^x f(u)\:du = f(x).
$$

```python
f = x**2
F = integrate(f, x)
F
```

```python
diff(F,x)
```

Alternately, if you compute the derivative of a function followed by the integral,
you will obtain the original function $f(x)$ (up to a constant):

$$
  \left( \int dx \circ \frac{d}{dx}\right) f(x) \equiv \int_c^x f'(u)\;du = f(x) + C.
$$

```python
f = x**2
df = diff(f,x)
df
```

```python
integrate(df, x)
```

The fundamental theorem of calculus is important because it tells us how to solve differential equations.
If we have to solve for $f(x)$ in the differential equation $\frac{d}{dx}f(x) = g(x)$,
we can take the integral on both sides of the equation to obtain the answer $f(x) = \int g(x)\,dx + C$.

### Sequences

Sequences are functions that take whole numbers as inputs.
Instead of continuous inputs $x\in \mathbb{R}$,
sequences take natural numbers $n\in\mathbb{N}$ as inputs.
We denote sequences as $a_n$ instead of the usual function notation $a(n)$.

We define a sequence by specifying an expression for its $n^\mathrm{th}$ term:

```python
a_n = 1/n
b_n = 1/factorial(n)
```

Substitute the desired value of $n$ to see the value of the $n^\mathrm{th}$ term:

```python
a_n.subs({n:5})
```

The `Python` list comprehension syntax `[item for item in list]`
can be used to print the sequence values for some range of indices:

```python
[ a_n.subs({n:i}) for i in range(0,8) ]
```

```python
[ b_n.subs({n:i}) for i in range(0,8) ]
```

Observe that $a_n$ is not properly defined for $n=0$ since $\frac{1}{0}$ is a division-by-zero error.
To be precise, we should say $a_n$'s domain is the positive naturals $a_n:\mathbb{N}^+ \to \mathbb{R}$.
Observe how quickly the `factorial` function $n!=1\cdot2\cdot3\cdots(n-1)\cdot n$ grows:
$7!= 5040$, $10!=3628800$, $20! > 10^{18}$.

We're often interested in calculating the limits of sequences as $n\to \infty$.
What happens to the terms in the sequence when $n$ becomes large?

```python
limit(a_n, n, oo)
```

```python
limit(b_n, n, oo)
```

Both $a_n=\frac{1}{n}$ and $b_n = \frac{1}{n!}$ *converge* to $0$ as $n\to\infty$.

Many important math quantities are defined as limit expressions.
An interesting example to consider is the number $\pi$,
which is defined as the area of a circle of radius $1$.
We can approximate the area of the unit circle by drawing a many-sided regular polygon around the circle.
Splitting the $n$-sided regular polygon into identical triangular splices,
we can obtain a formula for its area $A_n$.
In the limit as $n\to \infty$,
the $n$-sided-polygon approximation to the area of the unit-circle becomes exact:

```python
A_n = n*tan(2*pi/(2*n))
limit(A_n, n, oo)
```

### Series

Suppose we're given a sequence $a_n$ and we want to compute the sum of all the values in this sequence $\sum_{n}^\infty a_n$.
Series are sums of sequences.
Summing the values of a sequence $a_n:\mathbb{N}\to \mathbb{R}$
is analogous to taking the integral of a function $f:\mathbb{R}\to \mathbb{R}$.

To work with series in `SymPy`,
use the `summation` function whose syntax is analogous to the `integrate` function:

```python
a_n = 1/n
summation(a_n, [n, 1, oo])
```

```python
b_n = 1/factorial(n)
summation(b_n, [n, 0, oo])
```

We say the series $\sum a_n$ *diverges* to infinity (or *is divergent*) while the series $\sum b_n$ converges (or *is convergent*).
As we sum together more and more terms of the sequence $b_n$, the total becomes closer and closer to some finite number.
In this case, the infinite sum $\sum_{n=0}^\infty \frac{1}{n!}$ converges to the number $e=2.71828\ldots$.

The `summation` command is useful because it allows us to compute `infinite` sums,
but for most practical applications we don't need to take an infinite number of terms in a series to obtain a good approximation.
This is why series are so neat: they represent a great way to obtain approximations.

Using standard `Python` commands,
we can obtain an approximation to $e$ that is accurate to six decimals by summing 10 terms in the series:

```python
import math
def b_nf(n):
    return 1.0/math.factorial(n)
sum( [b_nf(n) for n in range(0,10)] )
```

```python
E.evalf()  # true value
```

### Taylor series

Wait, there's more!
Not only can we use series to approximate numbers,
we can also use them to approximate functions.

A *power series* is a series whose terms contain different powers of the variable $x$.
The $n^\mathrm{th}$ term in a power series is a function of both the sequence index $n$ and the input variable $x$.

For example, the power series of the function $\exp(x)=e^x$ is

$$
 \exp(x) \equiv  1 + x + \frac{x^2}{2} + \frac{x^3}{3!} + \frac{x^4}{4!} + \frac{x^5}{5!} + \cdots
  =       \sum_{n=0}^\infty \frac{x^n}{n!}.
$$

This is, IMHO, one of the most important ideas in calculus:
you can compute the value of $\exp(5)$ by taking the infinite sum of the terms in the power series with $x=5$:

```python
exp_xn = x**n/factorial(n)
summation( exp_xn.subs({x:5}), [n, 0, oo] ).evalf()
```

```python
exp(5).evalf()  # the true value
```

Note that `SymPy` is actually smart enough to recognize that the infinite series
you're computing corresponds to the closed-form expression $e^5$:

```python
summation( exp_xn.subs({x:5}), [n, 0, oo])
```

Taking as few as 35 terms in the series is sufficient to obtain an approximation to $e$
that is accurate to 16 decimals:

```python
import math  # redo using only python
def exp_xnf(x,n):
    return x**n/math.factorial(n)
sum( [exp_xnf(5.0,i) for i in range(0,35)] )
```

The coefficients in the power series of a function (also known as the *Taylor series*)
The formula for the $n^\mathrm{th}$ term in the Taylor series of $f(x)$ expanded at $x=c$ is $a_n(x) = \frac{f^{(n)}(c)}{n!}(x-c)^n$,
where $f^{(n)}(c)$ is the value of the $n^\mathrm{th}$ derivative of $f(x)$ evaluated at $x=c$.
The term *Maclaurin series* refers to Taylor series expansions at $x=0$.

The `SymPy` function `series` is a convenient way to obtain the series of any function.
Calling `series(expr,var,at,nmax)`
will show you the series expansion of `expr`
near `var`=`at`
up to power `nmax`:

```python
series( sin(x), x, 0, 8)
```

```python
series( cos(x), x, 0, 8)
```

```python
series( sinh(x), x, 0, 8)
```

```python
series( cosh(x), x, 0, 8)
```

Some functions are not defined at $x=0$, so we expand them at a different value of $x$.
For example, the power series of $\ln(x)$ expanded at $x=1$ is

```python
series(ln(x), x, 1, 6)  # Taylor series of ln(x) at x=1
```

Here, the result `SymPy` returns is misleading.
The Taylor series of $\ln(x)$ expanded at $x=1$ has terms of the form $(x-1)^n$:

$$
  \ln(x) = (x-1) - \frac{(x-1)^2}{2} + \frac{(x-1)^3}{3} - \frac{(x-1)^4}{4} + \frac{(x-1)^5}{5} + \cdots.
$$

Verify this is the correct formula by substituting $x=1$.
`SymPy` returns an answer in terms of coordinates `relative` to $x=1$.

Instead of expanding $\ln(x)$ around $x=1$,
we can obtain an equivalent expression if we expand $\ln(x+1)$ around $x=0$:

```python
series(ln(x+1), x, 0, 6)  # Maclaurin series of ln(x+1)
```


# Algebra and Symbolic Math With Sympy
The mathematical problems and solutions
in our programs so far have all involved the
manipulation of numbers. But there’s another
way math is taught, learned, and practiced, and that’s
in terms of symbols and the operations between them.
Just think of all the xs and ys in a typical algebra prob-
lem. We refer to this type of math as symbolic math.
---

## Defining Symbols and Symbolic Operations

```python
from sympy import Symbol
a = Symbol('x')
a + a + 1
```

$\displaystyle 2 x + 1$

```python
p = x*(x + x)
p
```

$\displaystyle 2 x^{2}$

```python
p = (x + 2)*(x + 3)
p
```

$\displaystyle \left(x + 2\right) \left(x + 3\right)$

## Working with Expressions

### Factorizing and Expanding Expressions

```python
from sympy import Symbol
from sympy import factor

x = Symbol('x')
y = Symbol('y')

expr = x**2 - y**2
factor(expr)
```

$\displaystyle \left(x - y\right) \left(x + y\right)$

## Pretty Printing

```python
from sympy import Symbol
from sympy import factor
from sympy import init_printing

init_printing(order='rev-lex')

x = Symbol('x')
y = Symbol('y')

expr = x*x + 2*x*y + y*y

pprint(expr)
```

### Printing a Series

```python
from sympy import Symbol, pprint, init_printing
def print_series(n):
    # Initialize printing system with reverse order
    init_printing(order='rev-lex')

    x = Symbol('x')
    series = x
    for i in range(2, n+1):
        series = series + (x**i)/i
    pprint(series)

if __name__ == '__main__':
    n = input('Enter the number of terms you want in the series: ')
    print_series(int(n))
```

## Substituting in Values
Let’s see how we can use SymPy to plug values into an algebraic expression.
This will let us calculate the value of the expression for certain values of the
variables. Consider the mathematical expression x 2 + 2xy + y 2 , which can be
defined as follows:

```python
from sympy import Symbol

x = Symbol('x')
y = Symbol('y')
expr = x*x + x*y + x*y + y*y

res = expr.subs({x:1, y:2})
print("the answer is: " + str(res))
```

### On Python Dictionaries

```python
sampledict = {"key1":5, "key2":20}
sampledict['key1']
expr_subs = expr.subs({x:1-y})
```

### Calculating the Value of a Series
Let’s revisit the series-printing program. In addition to printing the series,
we want our program to be able to find the value of the series for a par-
ticular value of x. That is, our program will now take two inputs from the
user—the number of terms in the series and the value of x for which the
value of the series will be calculated. Then, the program will output both
the series and the sum. The following program extends the series printing
program to include these enhancements:

```python
from sympy import Symbol, pprint, init_printing
def print_series(n, x_value):
    # Initialize printing system with reverse order
    init_printing(order='rev-lex')

    x = Symbol('x')
    series = x
    for i in range(2, n+1):
        series = series + (x**i)/i

    pprint(series)

    # Evaluate the series at x_value
    series_value = series.subs({x:x_value})
    print('Value of the series at {0}: {1}'.format(x_value, series_value))

if __name__ == '__main__':
    n = input('Enter the number of terms you want in the series: ')
    x_value = input('Enter the value of x at which you want to evaluate the series: ')
    print_series(int(n), float(x_value))
```


```python
import loader
from sympy import *
init_printing()
from root.solver import *
```

Find the general solution of
$$
    \vec{\mathbf{x'}} = \begin{bmatrix}
        1 & 1 & 1 \\
        2 & 1 & -1 \\
        0 & -1 & 1 \\
    \end{bmatrix} \vec{\mathbf{x}}
$$

```python
_, p = system([
    [1, 1, 1],
    [2, 1, -1],
    [0, -1, 1]
])
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle 3 \lambda - \left(\lambda - 1\right)^{3} - 5 = 0$

$\displaystyle \text{Eigenvalues and eigenvectors}$

$\displaystyle \lambda_1 = -1$

$\displaystyle \left[\begin{matrix}2 & 1 & 1 & 0\\2 & 2 & -1 & 0\\0 & -1 & 2 & 0\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & 0 & \frac{3}{2} & 0\\0 & 1 & -2 & 0\\0 & 0 & 0 & 0\end{matrix}\right]\Rightarrow v = \left[\begin{matrix}- \frac{3}{2}\\2\\1\end{matrix}\right]$

$\displaystyle \lambda_2 = 2$

$\displaystyle \left[\begin{matrix}-1 & 1 & 1 & 0\\2 & -1 & -1 & 0\\0 & -1 & -1 & 0\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 1 & 0\\0 & 0 & 0 & 0\end{matrix}\right]\Rightarrow v = \left[\begin{matrix}0\\-1\\1\end{matrix}\right]$

$\displaystyle \text{Find the generalized eigenvector}\left( M - \lambda I \right) w = v $

$\displaystyle \left[\begin{matrix}-1 & 1 & 1 & 0\\2 & -1 & -1 & -1\\0 & -1 & -1 & 1\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & 0 & 0 & -1\\0 & 1 & 1 & -1\\0 & 0 & 0 & 0\end{matrix}\right]\Rightarrow w = \left[\begin{matrix}-1\\-1\\0\end{matrix}\right]$

$\displaystyle \text{General solution: }$

$\displaystyle \vec{\mathbf{x}} = C_{1}e^{- t}\left[\begin{matrix}- \frac{3}{2}\\2\\1\end{matrix}\right]+C_{2}e^{2 t}\left[\begin{matrix}0\\-1\\1\end{matrix}\right]+C_{3}e^{2 t}\left(\left[\begin{matrix}0\\-1\\1\end{matrix}\right]t + \left[\begin{matrix}-1\\-1\\0\end{matrix}\right]\right)$

```python
_, p = system([
    [3, -2],
    [4, -1]
])
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle \left(\lambda - 3\right) \left(\lambda + 1\right) + 8 = 0$

$\displaystyle \text{Eigenvalues and eigenvectors}$

$\displaystyle \lambda_1 = 1 - 2*I$

$\displaystyle \left[\begin{matrix}2 + 2 i & -2 & 0\\4 & -2 + 2 i & 0\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & - \frac{2 - 2 i}{4} & 0\\0 & 0 & 0\end{matrix}\right]\Rightarrow v = \left[\begin{matrix}\frac{2 - 2 i}{4}\\1\end{matrix}\right]$

$\displaystyle \text{Use Euler's formula to expand the imaginary part}$

$\displaystyle \left[\begin{matrix}\frac{2 - 2 i}{4}\\1\end{matrix}\right] e^{t - 2 i t} = e^{t} \left[\begin{matrix}\frac{1}{4} \left(2 - 2 i\right) \left(- i \sin{\left(2 t \right)} + \cos{\left(2 t \right)}\right)\\- i \sin{\left(2 t \right)} + \cos{\left(2 t \right)}\end{matrix}\right] = e^{t}\left( \left[\begin{matrix}- \frac{\sin{\left(2 t \right)}}{2} + \frac{\cos{\left(2 t \right)}}{2}\\\cos{\left(2 t \right)}\end{matrix}\right] + \left[\begin{matrix}- \frac{\sin{\left(2 t \right)}}{2} - \frac{\cos{\left(2 t \right)}}{2}\\- \sin{\left(2 t \right)}\end{matrix}\right]\right)$

$\displaystyle \text{General solution: }$

$\displaystyle \vec{\mathbf{x}} = e^{t}\left(C_{1}\left[\begin{matrix}- \frac{\sin{\left(2 t \right)}}{2} + \frac{\cos{\left(2 t \right)}}{2}\\\cos{\left(2 t \right)}\end{matrix}\right] + C_{2}\left[\begin{matrix}- \frac{\sin{\left(2 t \right)}}{2} - \frac{\cos{\left(2 t \right)}}{2}\\- \sin{\left(2 t \right)}\end{matrix}\right]\right)$

Find the solution and draw the phase portrait for the following system of ode
$$
    \vec{\mathbf{x'}} = \begin{bmatrix}
        2 & -5 \\
        1 & -2 \\
    \end{bmatrix} \vec{\mathbf{x}}
$$

```python
_, p = system([
    [2, -5],
    [1, -2]
])
p.display()

%matplotlib inline
phase_portrait([
        [2, -5],
        [1, -2]
    ],
    -4, 4, -4, 4, 15, "stream"
)
```

```python

```


# Second Order ODEs in Python

Our last assignment consisted of a series of first order ODEs. This assignment will be similarly structured; however, you will be using the same techniques to solve second order ODEs. In particular, the examples I will give will be for a damped spring-mass system and coupled spring mass system then the problems you will solve will be for a single damped pendulum and a coupled pendulum.

## ODEs in Python

In this class, we will make use for the [scipy.integrate toolbox](https://docs.scipy.org/doc/scipy/reference/integrate.html). Primarily, we will be accessing the [solve_ivp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) (for *initial* value problems) and the [solve_bvp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_bvp.html#scipy.integrate.solve_bvp) (for *boundary* value problems).
Through these two functions, Scipy seeks the best type of solver for a particular system. We'll import these and other necessary elements below.

```python
# Python Imports

# 3rd Party Numerical Imports
import numpy as np
from scipy.integrate import solve_ivp as ivp

# 3rd Party Plotting Utilities
from matplotlib import pyplot as plt
```

## Examples

### The Damped Spring-Mass System

You all may remember that the equation of motion for a damped spring-mass system is

\begin{equation}
    m \ddot{x} + c \dot{x} + k x = 0
\end{equation}

where $m$ is the mass, $c$ is the damping constant and $k$ is the spring constant. Just like we did in the first ODE assignment, we will have to rewrite this equation in terms of the highest order term.

\begin{equation}
    \ddot{x} = -2\gamma \dot{x} - \omega_0^2 x
\end{equation}

where $2\gamma = c/m$ is the normalized damping coefficient and $\omega_0^2 = k/m$ is the natural frequency of the system. Now, the only problem with this equation is that we still have a term of $\dot{x}$ which we will have to write out as a second equation. This way, the differential equation solver will know how $\ddot{x}$, $\dot{x}$ and $x$ are related. We know that $\dot{x} = v$. Therefore, we will tell the solver this information by using a *system* of equations which we will write as

\begin{align}
    \dot{x}  &= v \\
    \dot{v} &= -2\gamma v - \omega_0^2 x.
\end{align}

Now, as it did on the last assignment, the ODE solver in Python expects as its first argument a function that packages all the necessary information about the problem we are trying to solve. And, the specified function is meant to take the independent variable as the first argument ($t$ in this case), the dependent variables as a tuple in the second argument ($X$ in this case) and any other variables after that. To see this visually, check the example and input below.

```python
def odefunc(indepVar, depVars, *otherArgs)
```

```python
def dshm(t, X, gamma, omega0Sq):
    '''A function for the ODE solver that specifies the conditions for damped, simple harmonic oscillations.'''

    # Unpack the variables
    pos, vel = X

    # Return the Output of the Equations
    return [
        vel,
        -2*gamma*vel - omega0Sq*pos
    ]
```

Note from the equations and functions above that, as we have it written the first output will be the *integration* of $\dot{x}$ with respect to time which is the position and the second output will be the integration of $\dot{v} = \ddot{x}$ with respect to time which is the velocity. Essentially, ODE solvers do not know how to handle anything other than a first order ODE. Therefore, we have to reframe our problems of higher orders into ones of a system of first order ODEs.

At this point, we are ready to begin testing this equation against different initial conditions. Before doing so, let's take a look at the necessary arguments for Scipy's IVP solver. Their documentation states that the function call looks like

```python
solve_ivp(odefunc, tSpan, y0, args=None, tEval=None)
```

The function call takes other arguments which we will not utilize here. However, those listed above are explained below.

* `odefunc`: This represents the name of the function to be evaluated. The IVP solver will call this function to create the solution.
* `tSpan`: This is meant to be a two-tuple meant to represent the start and stop times.
* `y0`: The initial condition(s) of the system as a tuple or list
* `args`: A tuple of arguments meant to be passed on to the `odefunc`. Note: **they must be in the same order as they appear in the `odefunc`**.
* `tEval`: Used to specify specific times to evaluate.

Now that we're aware of these, let's solve the charging problem under various initial conditions.

```python
# Set the values of the variables
m = 1
c = 0.5
k = 2
gamma = c/(2*m)
omega0Sq = k/m

# Set the initial value of the position and velocity
X0 = [1, 0]  # pos = 1, vel = 0

# Set Time Specs
tSpan = (0, 10)  # The simulation times for our system
tEval = np.linspace(tSpan[0], tSpan[1], 101)

# Solve the Problem
sol = ivp(dshm, tSpan, X0, args=(gamma, omega0Sq), t_eval=tEval)

# Plot the Solution
fig, ax = plt.subplots(figsize=(10, 7))
_ = ax.plot(sol.t, sol.y[0], linewidth=2, label='Position (m)')    # Plot the Position (which is the zeroth solution in the solution list)
_ = ax.plot(sol.t, sol.y[1], linewidth=2, label='Velocity (m/s)')  # Plot the Velocity (which is the first solution in the solution list)
_ = ax.set_xlim(tSpan)
_ = ax.set_xlabel('Time (s)',     fontsize=14)
_ = ax.set_ylabel('Amplitude', fontsize=14)
_ = ax.set_title('Damped Harmonic Oscillator', fontsize=16)
_ = ax.grid(True)
_ = ax.legend()
```

Now, let's instead assume that the mass starts at the origin and is given some initial kick in the positive direction.

```python
# Set the values of the variables
m = 1
c = 0.5
k = 2
gamma = c/(2*m)
omega0Sq = k/m

# Set the initial value of the position and velocity
X0 = [0, 2]  # pos = 0, vel = 2

# Set Time Specs
tSpan = (0, 10)  # The simulation times for our system
tEval = np.linspace(tSpan[0], tSpan[1], 101)

# Solve the Problem
sol = ivp(dshm, tSpan, X0, args=(gamma, omega0Sq), t_eval=tEval)

# Plot the Solution
fig, ax = plt.subplots(figsize=(10, 7))
_ = ax.plot(sol.t, sol.y[0], linewidth=2, label='Position (m)')    # Plot the Position (which is the zeroth solution in the solution list)
_ = ax.plot(sol.t, sol.y[1], linewidth=2, label='Velocity (m/s)')  # Plot the Velocity (which is the first solution in the solution list)
_ = ax.set_xlim(tSpan)
_ = ax.set_xlabel('Time (s)',     fontsize=14)
_ = ax.set_ylabel('Amplitude', fontsize=14)
_ = ax.set_title('Damped Harmonic Oscillator', fontsize=16)
_ = ax.grid(True)
_ = ax.legend()
```

Finally, let's take a look at the *critically damped* case when $\gamma = \omega_0$.

```python
# Set the values of the variables
gamma = 1
omega0 = 1
omega0Sq = omega0**2

# Set the initial value of the position and velocity
X0 = [1, -3]  # pos = 1, vel = -1

# Set Time Specs
tSpan = (0, 10)  # The simulation times for our system
tEval = np.linspace(tSpan[0], tSpan[1], 101)

# Solve the Problem
sol = ivp(dshm, tSpan, X0, args=(gamma, omega0Sq), t_eval=tEval)

# Plot the Solution
fig, ax = plt.subplots(figsize=(10, 7))
_ = ax.plot(sol.t, sol.y[0], linewidth=2, label='Position (m)')    # Plot the Position (which is the zeroth solution in the solution list)
_ = ax.plot(sol.t, sol.y[1], linewidth=2, label='Velocity (m/s)')  # Plot the Velocity (which is the first solution in the solution list)
_ = ax.set_xlim(tSpan)
_ = ax.set_xlabel('Time (s)',     fontsize=14)
_ = ax.set_ylabel('Amplitude', fontsize=14)
_ = ax.set_title('Critically Damped Harmonic Oscillator', fontsize=16)
_ = ax.grid(True)
_ = ax.legend()
```

You are welcome to play around with the three cells above to see how changing $k$, $m$, $c$ or the initial conditions affects the system.

### Coupled, Damped Harmonic Oscillators

Now, let's consider the case where we have a mass connected to a spring which is attached to the left wall and a spring attached between another mass and the right wall with a spring connecting the two masses in between. We can write the equations of motion for the system as being

\begin{align}
    m_1 \ddot{x}_1 &= -c \dot{x}_1 - k_1 x_1 - k_2 \left( x_1 - x_2 \right) \\
    m_2 \ddot{x}_2 &= -c \dot{x}_2 - k_2 \left( x_2 - x_1 \right) - k_3 x_2
\end{align}

we can then rewrite this as

\begin{align}
    \dot{x}_1 &= v_1 \\
    \dot{x}_2 &= v_2 \\
    \dot{v}_1 &= -2\gamma v_1 - \omega_0^2 x_1 - \omega_0^2 \left( x_1 - x_2 \right) \\
    \dot{v}_2 &= -2\gamma v_2 - \omega_0^2 \left( x_2 - x_1 \right) - \omega_0^2 x_2
\end{align}

assuming $k_1=k_2=k$ and $m_1=m_2=m$.

```python
# The Predator-Prey Equation
def cdhm(t, X, gamma, omega0Sq):
    '''A function for the ODE solver that specifies the conditions for damped, simple harmonic oscillations.'''

    # Unpack the variables
    pos1, pos2, vel1, vel2 = X

    # Return the Output of the Equations
    return [
        vel1,
        vel2,
        -2*gamma*vel1 - omega0Sq*pos1 - omega0Sq*(pos1 - pos2),
        -2*gamma*vel2 - omega0Sq*(pos2 - pos1) - omega0Sq*pos2
    ]
```

```python
# Set the values of the variables
m = 1
c = 0.5
k = 2
gamma = c/(2*m)
omega0Sq = k/m

# Set the initial value of the position and velocity
X0 = [0, -1, 0, 0]  # pos1 = 1, pos2 = 1.5, vel = 0

# Set Time Specs
tSpan = (0, 10)  # The simulation times for our system
tEval = np.linspace(tSpan[0], tSpan[1], 301)

# Solve the Problem
sol = ivp(cdhm, tSpan, X0, args=(gamma, omega0Sq), t_eval=tEval)

# Plot the Solution
fig, ax = plt.subplots(2, 1, figsize=(10, 12), sharex=True)
_ = ax[0].plot(sol.t, sol.y[0], linewidth=2, label='$x_1$')  # Plot the Position1 (which is the zeroth solution in the solution list)
_ = ax[0].plot(sol.t, sol.y[1], linewidth=2, label='$x_2$')  # Plot the Position2 (which is the first solution in the solution list)
_ = ax[1].plot(sol.t, sol.y[2], linewidth=2, label='$v_1$')  # Plot the Position1 (which is the zeroth solution in the solution list)
_ = ax[1].plot(sol.t, sol.y[3], linewidth=2, label='$v_2$')  # Plot the Position2 (which is the first solution in the solution list)
_ = ax[0].set_xlim(tSpan)
_ = ax[1].set_xlim(tSpan)
_ = ax[1].set_xlabel('Time (s)',  fontsize=14)
_ = ax[0].set_ylabel('Position (m)',   fontsize=14)
_ = ax[1].set_ylabel('Velocity (m/s)', fontsize=14)
_ = fig.suptitle('Coupled, Damped Harmonic Oscillator', fontsize=16)
_ = ax[0].grid(True)
_ = ax[1].grid(True)
_ = ax[0].legend()
_ = ax[1].legend()
```

```python
# Set the values of the variables
m = 1
c = 0.5
k = 2
gamma = c/(2*m)
omega0Sq = k/m

# Set the initial value of the position and velocity
X0 = [0, 0, 0, -2]  # pos1 = 1, pos2 = 1.5, vel = 0

# Set Time Specs
tSpan = (0, 10)  # The simulation times for our system
tEval = np.linspace(tSpan[0], tSpan[1], 301)

# Solve the Problem
sol = ivp(cdhm, tSpan, X0, args=(gamma, omega0Sq), t_eval=tEval)

# Plot the Solution
fig, ax = plt.subplots(2, 1, figsize=(10, 12), sharex=True)
_ = ax[0].plot(sol.t, sol.y[0], linewidth=2, label='$x_1$')  # Plot the Position1 (which is the zeroth solution in the solution list)
_ = ax[0].plot(sol.t, sol.y[1], linewidth=2, label='$x_2$')  # Plot the Position2 (which is the first solution in the solution list)
_ = ax[1].plot(sol.t, sol.y[2], linewidth=2, label='$v_1$')  # Plot the Position1 (which is the zeroth solution in the solution list)
_ = ax[1].plot(sol.t, sol.y[3], linewidth=2, label='$v_2$')  # Plot the Position2 (which is the first solution in the solution list)
_ = ax[0].set_xlim(tSpan)
_ = ax[1].set_xlim(tSpan)
_ = ax[1].set_xlabel('Time (s)',  fontsize=14)
_ = ax[0].set_ylabel('Position (m)',   fontsize=14)
_ = ax[1].set_ylabel('Velocity (m/s)', fontsize=14)
_ = fig.suptitle('Coupled, Damped Harmonic Oscillator', fontsize=16)
_ = ax[0].grid(True)
_ = ax[1].grid(True)
_ = ax[0].legend()
_ = ax[1].legend()
```

## Assignment

### The Pendulum

The equation of motion for a damped pendulum is

\begin{equation}
    l \ddot{\theta} + c \dot{\theta} + g \sin \theta = 0
\end{equation}

which can be rewritten as

\begin{align}
    \dot{\theta} &= \omega \\
    \dot{\omega} &= -\frac{c}{l} \omega - \frac{g}{l} \sin \theta
\end{align}

Write the function for Scipy below.

```python

```

Write the code to solve the equation here for the initial conditions

\begin{align}
    \theta(0) &= \frac{9\pi}{10} \\
    \dot{\theta}(0) &= 0
\end{align}

and when $c=0.5$, $g=9.8$ and $l=1$. Show 10 seconds of simulation time.

```python

```

### Coupled Pendulums

In the case when two pendula are connected to each other via a spring, the equations of motion are

\begin{align}
    \ddot{\theta}_1 + \frac{g}{l} \sin \theta_1 + \frac{k}{m} (\theta_1 - \theta_2) &= 0 \\
    \ddot{\theta}_2 + \frac{g}{l} \sin \theta_2 + \frac{k}{m} (\theta_2 - \theta_1) &= 0 \\
\end{align}

which can be written as

\begin{align}
    \dot{\theta}_1 &= \omega_1 \\
    \dot{\theta}_2 &= \omega_2 \\
    \dot{\omega}_1 &= - \frac{g}{l} \sin \theta_1 - \frac{k}{m} (\theta_1 - \theta_2) \\
    \dot{\omega}_2 &= - \frac{g}{l} \sin \theta_2 - \frac{k}{m} (\theta_2 - \theta_1) \\
\end{align}

Write the function for Scipy below.

```python

```

And write the code to solve the differential equation below for the initial conditions

\begin{align}
    \theta_1(0) &= 0 \\
    \theta_2(0) &= \frac{\pi}{6} \\
    \dot{\theta}_1(0) &= 0 \\
    \dot{\theta}_2(0) &= 0 \\
\end{align}

and when $k=1$, $m=1$, $g=9.8$ and $l=2$. Make the simulation for 30 seconds.

```python

```


# Tutorial

We will solve the following problem using a computer to assist with the technical aspects:

```{admonition} Problem

The matrix $A$ is given by $A=\begin{pmatrix}a & 1 & 1\\ 1 & a & 1\\ 1 & 1 & 2\end{pmatrix}$.

1. Find the determinant of $A$
2. Hence find the values of $a$ for which $A$ is singular.
3. For the following values of $a$, when possible obtain $A ^ {- 1}$ and confirm
   the result by computing $AA^{-1}$:
    1. $a = 0$;
    2. $a = 1$;
    3. $a = 2$;
    4. $a = 3$.

```

`sympy` is once again the library we will use for this.

We will start by our matrix $A$:

```python
import sympy as sym

a = sym.Symbol("a")
A = sym.Matrix([[a, 1, 1], [1, a, 1], [1, 1, 2]])
```

We can now create a variable `determinant` and assign it the value of the
determinant of $A$:

```python
determinant = A.det()
determinant
```

$\displaystyle 2 a^{2} - 2 a$

A matrix is singular if it has determinant 0. We can find the values of $a$ for
which this occurs:

```python
sym.solveset(determinant, a)
```

$\displaystyle \left\{0, 1\right\}$

Thus it is not possible to find the inverse of $A$ for $a\in\{0, 1\}$.

However for $a = 2$:

```python
A.subs({a: 2})
```

$\displaystyle \left[\begin{matrix}2 & 1 & 1\\1 & 2 & 1\\1 & 1 & 2\end{matrix}\right]$

```python
A.subs({a: 2}).inv()
```

$\displaystyle \left[\begin{matrix}\frac{3}{4} & - \frac{1}{4} & - \frac{1}{4}\\- \frac{1}{4} & \frac{3}{4} & - \frac{1}{4}\\- \frac{1}{4} & - \frac{1}{4} & \frac{3}{4}\end{matrix}\right]$

To carry out matrix multiplication we use the `@` symbol:

```python
A.subs({a: 2}).inv() @ A.subs({a: 2})
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

and for $a = 3$:

```python
A.subs({a: 3}).inv()
```

$\displaystyle \left[\begin{matrix}\frac{5}{12} & - \frac{1}{12} & - \frac{1}{6}\\- \frac{1}{12} & \frac{5}{12} & - \frac{1}{6}\\- \frac{1}{6} & - \frac{1}{6} & \frac{2}{3}\end{matrix}\right]$

```python
A.subs({a: 3}).inv() @ A.subs({a: 3})
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

```{important}
In this tutorial we have

- Created a matrix.
- Calculated the determinant of the matrix.
- Substituted values in the matrix.
- Inverted the matrix.
```


# Eigenvalue and Eigenvector
Given an $m \times m$ matrix $A$, we say a ***nonzero vector*** $\vec{x} \in \mathbb{C}^m$ is an **eigenvector** of $A$, and a number $\lambda \in \mathbb{C}$ is its **eigenvalue**, if

$$A\vec{x} = \lambda \vec{x}$$

How to compute them?

$$A\vec{x} = \lambda \vec{x} \Longleftrightarrow \left( \lambda I - A \right)\vec{x} = \vec{0}$$

which has a ***nonzero solution*** $\vec{x}$, which requires that the matrix $\lambda I - A$ is ***singular***, thus

$$\DeclareMathOperator*{\det}{det} A\vec{x} = \lambda \vec{x} \Longleftrightarrow \left( \lambda I - A \right)\vec{x} = \vec{0} \Longleftrightarrow \det\left( \lambda I - A \right) = 0$$

Here $\det\left( \lambda I - A \right)$ is actually a polynomial of degree $m$ in $\lambda$, called the **characteristic polynomial** of matrix $A$.
1. Solve the **characteristic polynomial** to find possible $\lambda$s, the engenvalues
2. For each **engenvalue**, solve the linear system $\left( \lambda I - A \right)\vec{x} = \vec{0}$ to find the corresponding eigenvectors of $A$.

>**e.g.** Compute the eigenvalue of the matrix $A = \left[\begin{array}{cc}
2 & 1 \\
0 & 1
\end{array}\right]$
>
>>$$\lambda I - A = \lambda \cdot \left[\begin{array}{cc}
1 & 0 \\
0 & 1
\end{array}\right] - \left[\begin{array}{cc}
2 & 1 \\
0 & 1
\end{array}\right] = \left[\begin{array}{cc}
\lambda - 2 & 1 \\
0 & \lambda - 1
\end{array}\right]$$

>> So that the chracteristic polynomial of $A$ is $\left( \lambda - 2 \right)\left( \lambda - 1 \right)$. Therefore the eigenvalues of $A$ are $\lambda_1 = 2$ and $\lambda_2 = 1$. Corresponding to the eigenvalue $2$, the solution of the linear system $(\lambda_1 I - A) \vec{x} = \vec{0}$ would be

>>$$\left[\begin{array}{cc}
0 & -1 \\
0 & 1
\end{array}\right] \vec{x} = \left[\begin{array}{c}
0 \\
0
\end{array}\right] \Longrightarrow \vec{x} = k_1 \cdot \left[\begin{array}{c}
1 \\
0
\end{array}\right]
$$

>> where $k_1$ is an arbitrary constant. And the other one is $k_2 \cdot \left[\begin{array}{c}
1 \\
-1
\end{array}\right]$

# Eigenspace and multiplicity of eigenvalues
$\odot$ If a vector $\vec{x}$  is an eigenvector of $A$ corresponding to an eigenvalue $\lambda$, then any scalar multiple of $\vec{x}$ is also an eigenvector of $A$ corresponding to $\lambda$.$\Join$

**eigenspace** of $A$ corresponding to the eigenvalue $\lambda$: the solution space of $\left( A - \lambda I\right)\vec{x} = \vec{0}$, denoted a $E_{\lambda}$. And the dimension of $E_{\lambda}$ is also the **geometric multiplicity** of $\lambda$, which represents the maximum number of linearly linearly independent eigenvectors corresponding to the eigenvalue $\lambda$.

**algebraic multiplicity**, the multiplicity of each root $\lambda$.

Now denote all the distinct eigenvalues of a matrix $A$ be $\lambda_1, \lambda_2, \dots, \lambda_k$, so that the characteristic polynomial $\det(\lambda I - A)$ would be then expressed as

$$\det(\lambda I - A) = (\lambda - \lambda_1)^{l_1}(\lambda - \lambda_2)^{l_2 }\cdots(\lambda - \lambda_k)^{l_k}$$

Here $l_1, l_2, \dots, l_k$ are the algebraic multiplicities of $\lambda_1, \lambda_2, \dots, \lambda_k$ respectively.

$Conclusion$

For any eigenvalue $\lambda^*$ of a matrix $A$, its geometric multiplicity is always less than or equal to its algebric multiplicity.

$Proof$

Suppose a $m \times m$ matrix $A$ with a eigenvalue $\lambda^*$ whose geomatrix multiplicity is $l^*$. Then let the orthonormal basis of its eigenspace $E_{\lambda^*}$ be $\vec{v}_1, \vec{v}_2, \dots, \vec{v}_{l^*}$. And let the complementary orthonormal vector be $\vec{v}_{l^*+1},\vec{v}_{l^*+2},\dots,\vec{v}_{m}$, so that they can form the orthogonormal basis of the $m\text{-dimensional}$ vector space.

Let $V$ be the orthogonal matrix with $\vec{v}_{1},\vec{v}_{2},\dots,\vec{v}_{m}$ as its columns, then we have

$$V^{\mathrm{T}}AV = \left[\begin{array}{cc}
\lambda I & C \\
\mathbf{0} & D
\end{array}\right]$$

where $I$ is the $l^* \times l^*$ identity matrix, $C$ is an $l^*\times (m-l^*)$ matrix, and $D$ is an $(m-l^*) \times (m-l^*)$ matrix. Note that $\det(V^{\mathrm{T}})\det(V) = 1$. So we have

$$\det(\lambda I - A) = \det(\lambda I - V^{\mathrm{T}}AV) = \det( (\lambda - \lambda^*)I ) \cdot \det(\lambda I - D) = (\lambda - \lambda^*)^{l^*} \det(\lambda I - D)$$

which implies that the algebraic multiplicity of $\lambda^*$ is at least $l^*$.

# Similarity and Diagonalizability
$Def$

A matrix $A$ is said to be **similar** to another $B$ $iff$ there exists a ***nonsigular matrix*** $X$ such that

$$A = XBX^{-1}$$

$Conclusion$

$$\det(A - \lambda I) = \det(XBX^{-1} - \lambda XIX^{-1}) = \det(B - \lambda I)$$

And therefore they have the same eigenvalues.

Here $m \times m$ matrix $A$ is called **diagonalizable** if it is similar to a diagonal matrix $\Lambda $.

$$A = X\Lambda X^{-1} \Longrightarrow AX = X\Lambda $$

We can also see that the columns of $X$ are eigenvectors of the matrix $A$ corresponding to eigenvalues on the diagonal of $\Lambda $: $A \vec{x}^c_i = \lambda_i \vec{x}^c_i$.

**defective**: An eigenvalue whose algebraic mutiplicity exceeds its geometric multiplicity is a **defective eigenvalue**; A matrix with defective eigenvalues is a **defective matrix**. This promises its eigenvalue decomposition. And of course any diagonal matrix is nondefective.

***
For simplicity, now we only consider real symmetric matrix $A$ ($A^{\mathrm{T}} = A$). So that the eigenvalues are all real numbers, and eigenvectors can form a orthonormal basis of the $m\text{-dimensional}$ vector space.

$Theorem$ 1

For any real symmetric $m \times m$ matrix $A$, there exist a real orthogonal matrix $Q$, and a real diagonal matrix $\Lambda$, such that

$$A = Q \Lambda Q^{\mathrm{T}} \Longrightarrow A\left[
\begin{array}{cccc}
\vec{q}_1^c & \vec{q}_2^c & \cdots & \vec{q}_m^c \\
\end{array}
\right] = \left[
\begin{array}{cccc}
\vec{q}_1^c & \vec{q}_2^c & \cdots & \vec{q}_m^c \\
\end{array}
\right] \left[
\begin{array}{cccc}
\lambda_1 & & & \\
& \lambda_2 & & \\
& & \ddots & \\
& & & \lambda_m \\
\end{array}
\right]$$

This is the **eigenvalue decomposition** of matrix $A$. We can also give them an order so that $\left| \lambda_1 \right| \geq \left| \lambda_2 \right| \geq \cdots \geq \left| \lambda_1 \right| \geq\left| \lambda_m \right| \geq 0$

# Conditioning of Eigenvalue Problems
Given an $m \times m$ matrix $A$ which is diagonalizable with eigenvalues $\lambda_1, \lambda_2, \dots \lambda_m$, and a set of eigenvectors $\vec{x}^c_1,\vec{x}^c_2, \dots, \vec{x}^c_m$ so that we can write $A = X\Lambda X^{-1}$ or equivalently $\Lambda = X^{-1}AX$.

Assume there is a small perturbation $\Delta A$ on the matrix $A$. Let us consider eigenvalues of the perturbed matrix $A + \Delta A$. We denote

$$X^{-1}\Delta A X = F$$

So that $X^{-1}\left( A + \Delta A \right)X = \Lambda + F$ and let $\mu$ be the eigenvalue of $\left( A + \Delta A \right)$ and $D+F$ corresponding to the eigenvector $\vec{v}$.

$$\left( D + F \right) \vec{v} = \mu \vec{v} \Longrightarrow \mu \vec{v} - D \vec{v} = F\vec{v}$$

Therefore we have

$$\begin{align}
\left\| \vec{v} \right\|_2 &= \left\| \left( \mu I - D \right)^{-1} F \vec{v} \right\|_2 \\
&\leq \left\| \left( \mu I - D \right)^{-1} \right\|_2 \cdot \left\| F \right\|_2 \cdot \left\| \vec{v} \right\|_2 \\
&= \frac{\left\| F \right\|_2 \cdot \left\| \vec{v} \right\|_2} {\left| \mu - \lambda_k \right|}
\end{align}$$

So that

$$\DeclareMathOperator*{\Cond}{Cond}
\begin{align}
\left| \mu - \lambda_k \right| &\leq \left\| F \right\|_2 \\
&\leq \left\| X \right\|_2 \cdot \left\| \Delta A \right\|_2 \cdot \left\| X^{-1} \right\|_2 \\
&= \Cond(X) \left\|\Delta A \right\|_2
\end{align}$$

where $\lambda_k$ is the closest to $\mu$ among the eigenvalues of the matrix $A$.

If the matrix $A$ is symmetric, then we know From $Theorem$ 1 that the matrix $Q$ is orthogonal. Therefore solving the eigenvalues of a symmetric matrix is always well-conditioned, since $\Cond(Q) = 1$ for orthogonal matrix Q.

# Algorithms for solving eigenvalue problems
To find the eigenvalue is to solve the corresponding characteristic polynomial. But that's an ill-conditioned problem. At the same time, there is no exact formula to express the roots of a general high order ($> 5$) polynomials. This means that in general finding the eigenvalues of a matrix cannot be achieved within a finite number of steps of operations. Therefore an eigensolver has to be *iterative*.

Similar to the idea from $QR$ and $LU$, we wish to multiply a sequence of orthogonal matrices and their transposes to the both sides of $A$, such that

$$Q_m \cdots Q_2 Q_1 A Q^{\mathrm{T}}_1 Q^{\mathrm{T}}_2 \cdots Q^{\mathrm{T}}_m = \Lambda$$

And then

$$A= Q^{\mathrm{T}}_1Q^{\mathrm{T}}_2\cdots Q^{\mathrm{T}}_m \Lambda Q_1Q_2 \cdots Q_m = Q^{\mathrm{T}}\Lambda Q$$

However such an approach is just an imagination. An eigenvalue problem cannot be solved in a finite number of steps in general. The approach to solve eigenvalue problem is in general to produce a sequence

$$Q_j \cdots Q_2 Q_1 A Q^{\mathrm{T}}_1 Q^{\mathrm{T}}_2 \cdots Q^{\mathrm{T}}_j$$

such that it converges to an diagonal matrix $\Lambda$ as $ j \to 1$. In this sense we say

**Any eigenvalue solver must be iterative.**

Computational/Intro to Numerical Computing/Note_Chap08_Fundamentals of Eigenvalue Problems.ipynb


```python
#format the book
from __future__ import division, print_function
%matplotlib inline
import sys
sys.path.insert(0, '..')
import book_format
book_format.set_style()
```

<style>
.output_wrapper, .output {
    height:auto !important;
    max-height:100000px;
}
.output_scroll {
    box-shadow:none !important;
    webkit-box-shadow:none !important;
}
</style>

# Computing and plotting PDFs of discrete data

So let's investigate how to compute and plot probability distributions.

First, let's make some data according to a normal distribution. We use `numpy.random.normal` for this. The parameters are not well named. `loc` is the mean of the distribution, and `scale` is the standard deviation. We can call this function to create an arbitrary number of data points that are distributed according to that mean and std.

```python
import numpy as np
import numpy.random as random

mean = 3
std = 2

data = random.normal(loc=mean, scale=std, size=50000)
print(len(data))
print(data.mean())
print(data.std())
```

    50000
    3.004197839408925
    1.995905183997337

```python
data = 1.8 + np.random.randn(50000)*0.414
data.shape
```

    (50000,)

```python
data.std(), data.mean()
```

    (0.413856930662519, 1.801027366859001)

As you can see from the print statements we got 5000 points that have a mean very close to 3, and a standard deviation close to 2.

We can plot this Gaussian by using `scipy.stats.norm` to create a frozen function that we will then use to compute the pdf (probability distribution function) of the Gaussian.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.stats as stats

def plot_normal(xs, mean, std, **kwargs):
    norm = stats.norm(mean, std)
    plt.plot(xs, norm.pdf(xs), **kwargs)

xs = np.linspace(-5, 15, num=200)
plot_normal(xs, mean, std, color='k')
```

```python
norm = stats.norm(mean, std)
norm.pdf([1,2,3,4])
```

    array([0.12098536, 0.17603266, 0.19947114, 0.17603266])

But we really want to plot the PDF of the discrete data, not the idealized function.

There are a couple of ways of doing that. First, we can take advantage of `matplotlib`'s `hist` method, which computes a histogram of a collection of data. Normally `hist` computes the number of points that fall in a bin, like so:

```python
plt.hist(data, bins=200)
plt.show()
```

that is not very useful to us - we want the PDF, not bin counts. Fortunately `hist` includes a `density` parameter which will plot the PDF for us.

```python
plt.hist(data, bins=200, density=True)
plt.show()
```

I may not want bars, so I can specify the `histtype` as 'step' to get a line.

```python
plt.hist(data, bins=200, density=True, histtype='step', lw=2)
plt.show()
```

To be sure it is working, let's also plot the idealized Gaussian in black.

```python
plt.hist(data, bins=200, density=True, histtype='step', lw=2)
norm = stats.norm(mean, std)
plt.plot(xs, norm.pdf(xs), color='k', lw=2)
plt.show()
```

There is another way to get the approximate distribution of a set of data. There is a technique called *kernel density estimate* that uses a kernel to estimate the probability distribution of a set of data. SciPy implements it with the function `gaussian_kde`. Do not be mislead by the name - Gaussian refers to the type of kernel used in the computation. This works for any distribution, not just Gaussians. In this section we have a Gaussian distribution, but soon we will not, and this same function will work.

```python
kde = stats.gaussian_kde(data)

xs = np.linspace(-5, 15, num=200)
plt.plot(xs, kde(xs))
plt.show()
```

## Monte Carlo Simulations

We (well I) want to do this sort of thing because I want to use monte carlo simulations to compute distributions. It is easy to compute Gaussians when they pass through linear functions, but difficult to impossible to compute them analytically when passed through nonlinear functions. Techniques like particle filtering handle this by taking a large sample of points, passing them through a nonlinear function, and then computing statistics on the transformed points. Let's do that.

We will start with the linear function $f(x) = 2x + 12$ just to prove to ourselves that the code is working. I will alter the mean and std of the data we are working with to help ensure the numbers that are output are unique It is easy to be fooled, for example, if the formula multipies x by 2, the mean is 2, and the std is 2. If the output of something is 4, is that due to the multication factor, the mean, the std, or a bug? It's hard to tell.

```python
def f(x):
    return 2*x + 12

mean = 1.
std = 1.4
data = random.normal(loc=mean, scale=std, size=50000)

d_t = f(data) # transform data through f(x)

plt.hist(data, bins=200, density=True, histtype='step', lw=2)
plt.hist(d_t, bins=200, density=True, histtype='step', lw=2)

plt.ylim(0, .35)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

This is what we expected. The input is the Gaussian $\mathcal{N}(\mu=1, \sigma=1.4)$, and the function is $f(x) = 2x+12$. Therefore we expect the mean to be shifted to $f(\mu) = 2*1+12=14$. We can see from the plot and the print statement that this is what happened.

Before I go on, can you explain what happened to the standard deviation? You may have thought that the new $\sigma$ should be passed through $f(x)$ like so $2(1.4) + 12=14.81$. But that is not correct - the standard deviation is only affected by the multiplicative factor, not the shift. If you think about that for a moment you will see it makes sense. We multiply our samples by 2, so they are twice as spread out as before. Standard deviation is a measure of how spread out things are, so it should also double. It doesn't matter if we then shift that distribution 12 places, or 12 million for that matter - the spread is still twice the input data.

## Nonlinear Functions

Now that we believe in our code, lets try it with nonlinear functions.

```python
def f2(x):
    return (np.cos((1.5*x + 2.1))) * np.sin(0.3*x) - 1.6*x

d_t = f2(data)
plt.subplot(121)
plt.hist(d_t, bins=200, density=False, histtype='step', lw=2)
# plt.hist(data, bins=200, density=True, histtype='step', lw=2)

plt.subplot(122)
kde = stats.gaussian_kde(d_t)
xs = np.linspace(-10, 10, 200)
plt.plot(xs, kde(xs), 'k')
plot_normal(xs, d_t.mean(), d_t.std(), color='g', lw=3)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

Here I passed the data through the nonlinear function $f(x) = \cos(1.5x+2.1)\sin(\frac{x}{3}) - 1.6x$. That function is quite close to linear, but we can see how much it alters the pdf of the sampled data.

There is a lot of computation going on behind the scenes to transform 50,000 points and then compute their PDF. The Extended Kalman Filter (EKF) gets around this by linearizing the function at the mean and then passing the Gaussian through the linear equation. We saw above how easy it is to pass a Gaussian through a linear function. So lets try that.

We can linearize this by taking the derivative of the function at x. We can use sympy to get the derivative.

```python
#its not necessary that the tangent
```

```python
import sympy
x = sympy.symbols('x')
f = sympy.cos(1.5*x+2.1) * sympy.sin(x/3) - 1.6*x
dfx = sympy.diff(f, x)
dfx
```

    -1.5*sin(x/3)*sin(1.5*x + 2.1) + cos(x/3)*cos(1.5*x + 2.1)/3 - 1.6

We can now compute the slope of the function by evaluating the derivative at the mean.

```python
m = dfx.subs(x, mean)
m
```

    -1.66528051815545

The equation of a line is $y=mx+b$, so the new standard deviation should be $~1.67$ times the input std. We can compute the new mean by passing it through the original function because the linearized function is just the slope of f(x) evaluated at the mean. The slope is a tangent that touches the function at $x$, so both will return the same result. So, let's plot this and compare it to the results from the monte carlo simulation.

```python
plt.hist(d_t, bins=200, density=True, histtype='step', lw=2)
plot_normal(xs, f2(mean), abs(float(m)*std), color='k', lw=3, label='EKF')
plot_normal(xs, d_t.mean(), d_t.std(), color='r', lw=3, label='MC')
plt.legend()
plt.show()
```

We can see from this that the estimate from the EKF (in red) is not exact, but it is not a bad approximation either.


# Poisson Distribution

> ***GitHub***: https://github.com/czs108

## Definition

\begin{equation}
P(X = r) = \frac{e^{-\lambda} \cdot {\lambda}^{r}}{r!}
\end{equation}

\begin{equation}
\lambda = \text{The mean number of occurrences in the interval or the rate of occurrence.}
\end{equation}

If a variable $X$ follows a *Poisson Distribution* where the *mean* number of occurrences in the interval or the rate of occurrence is $\lambda$. This can be written as

\begin{equation}
X \sim Po(\lambda)
\end{equation}

## Expectation

\begin{equation}
E(X) = \lambda
\end{equation}

## Variance

\begin{equation}
Var(X) = \lambda
\end{equation}

## Approximation

A *Binomial Distribution* $X \sim B(n,\, p)$ can be approximated by $X \sim Po(np)$ if $n$ is *large* and $p$ is *small*.

Because both the *expectation* and *variance* of $X \sim Po(np)$ are $np$. When $n$ is large and $p$ is small, $(1 - p) \approx 1$ and for $X \sim B(n,\, p)$:

\begin{equation}
E(X) = np
\end{equation}

\begin{equation}
Var(X) \approx np
\end{equation}

The approximation is typically very close if $n > 50$ and $p < 0.1$.


# Quadratic Equations

Consider the following equation:

\begin{equation}y = 2(x - 1)(x + 2)\end{equation}

If you multiply out the factored ***x*** expressions, this equates to:

\begin{equation}y = 2x^{2} + 2x - 4\end{equation}

Note that the highest ordered term includes a squared variable (x2).

Let's graph this equation for a range of ***x*** values:

```python
import pandas as pd

# Create a dataframe with an x column containing values to plot
df = pd.DataFrame ({'x': range(-9, 9)})

# Add a y column by applying the quadratic equation to x
df['y'] = 2*df['x']**2 + 2 *df['x'] - 4

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that the graph shows a *parabola*, which is an arc-shaped line that reflects the $x$ and $y$ values calculated for the equation.

Now let's look at another equation that includes an ***x2*** term:

\begin{equation}y = -2x^{2} + 6x + 7\end{equation}

What does that look like as a graph?:

```python
import pandas as pd

# Create a dataframe with an x column containing values to plot
df = pd.DataFrame ({'x': range(-8, 12)})

# Add a y column by applying the quadratic equation to x
df['y'] = -2*df['x']**2 + 6*df['x'] + 7

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Again, the graph shows a parabola, but this time instead of being open at the top, the parabola is open at the bottom.

Equations that assign a value to ***y*** based on an expression that includes a squared value for ***x*** create parabolas. If the relationship between ***y*** and ***x*** is such that ***y*** is a *positive* multiple of the ***x2*** term, the parabola will be open at the top; when ***y*** is a *negative* multiple of the ***x2*** term, then the parabola will be open at the bottom.

These kinds of equations are known as *quadratic* equations, and they have some interesting characteristics. There are several ways quadratic equations can be written, but the *standard form* for quadratic equation is:

\begin{equation}y = ax^{2} + bx + c\end{equation}

Where ***a***, ***b***, and ***c*** are numeric coefficients or constants.

Let's start by examining the parabolas generated by quadratic equations in more detail.

## Parabola Vertex and Line of Symmetry
Parabolas are symmetrical, with x and y values converging exponentially towards the highest point (in the case of a downward opening parabola) or lowest point (in the case of an upward opening parabola). The point where the parabola meets the line of symmetry is known as the *vertex*.

Run the following cell to see the line of symmetry and vertex for the two parabolas described previously (don't worry about the calculations used to find the line of symmetry and vertex - we'll explore that later):

```python
%matplotlib inline

def plot_parabola(a, b, c):
    import pandas as pd
    import numpy as np
    from matplotlib import pyplot as plt

    # get the x value for the line of symmetry
    vx = (-1*b)/(2*a)

    # get the y value when x is at the line of symmetry
    vy = a*vx**2 + b*vx + c

    # Create a dataframe with an x column containing values from x-10 to x+10
    minx = int(vx - 10)
    maxx = int(vx + 11)
    df = pd.DataFrame ({'x': range(minx, maxx)})

    # Add a y column by applying the quadratic equation to x
    df['y'] = a*df['x']**2 + b *df['x'] + c

    # get min and max y values
    miny = df.y.min()
    maxy = df.y.max()

    # Plot the line
    plt.plot(df.x, df.y, color="grey")
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid()
    plt.axhline()
    plt.axvline()

    # plot the line of symmetry
    sx = [vx, vx]
    sy = [miny, maxy]
    plt.plot(sx,sy, color='magenta')

    # Annotate the vertex
    plt.scatter(vx,vy, color="red")
    plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy + 5)* np.sign(a)))

    plt.show()

plot_parabola(2, 2, -4)

plot_parabola(-2, 3, 5)
```

## Parabola Intercepts
Recall that linear equations create lines that intersect the **x** and **y** axis of a graph, and we call the points where these intersections occur *intercepts*. Now look at the graphs of the parabolas we've worked with so far. Note that these parabolas both have a y-intercept; a point where the line intersects the y axis of the graph (in other words, when x is 0). However, note that the parabolas have *two* x-intercepts; in other words there are two points at which the line crosses the x axis (and y is 0). Additionally, imagine a downward opening parabola with its vertex at -1, -1. This is perfectly possible, and the line would never have an x value greater than -1, so it would have *no* x-intercepts.

Regardless of whether the parabola crosses the x axis or not, other than the vertex, for every ***y*** point in the parabola, there are *two* ***x*** points; one on the right (or positive) side of the axis of symmetry, and one of the left (or negative) side. The implications of this are what make quadratic equations so interesting. When we solve the equation for ***x***, there are *two* correct answers.

Let's take a look at an example to demonstrate this. Let's return to the first of our quadratic equations, and we'll look at it in its *factored* form:

\begin{equation}y = 2(x - 1)(x + 2)\end{equation}

Now, let's solve this equation for a ***y*** value of 0. We can restate the equation like this:

\begin{equation}2(x - 1)(x + 2) = 0\end{equation}

The equation is the product of two expressions **2(x - 1)** and **(x + 2)**. In this case, we know that the product of these expressions is 0, so logically *one or both of the expressions must return 0*.

Let's try the first one:

\begin{equation}2(x - 1) = 0\end{equation}

If we distrbute this, we get:

\begin{equation}2x - 2 = 0\end{equation}

This simplifies to:

\begin{equation}2x = 2\end{equation}

Which gives us a value for *x* of **1**.

Now let's try the other expression:

\begin{equation}x + 2 = 0\end{equation}

This gives us a value for *x* of **-2**.

So, when *y* is **0**, *x* is **-2** or **1**. Let's plot these points on our parabola:

```python
import pandas as pd

# Assign the calculated x values
x1 = -2
x2 = 1

# Create a dataframe with an x column containing some values to plot
df = pd.DataFrame ({'x': range(x1-5, x2+6)})

# Add a y column by applying the quadratic equation to x
df['y'] = 2*(df['x'] - 1) * (df['x'] + 2)

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = 2*(vx -1)*(vx + 2)

# get min and max y values
miny = df.y.min()
maxy = df.y.max()

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# Plot calculated x values for y = 0
plt.scatter([x1,x2],[0,0], color="green")
plt.annotate('x1',(x1, 0))
plt.annotate('x2',(x2, 0))

# plot the line of symmetry
sx = [vx, vx]
sy = [miny, maxy]
plt.plot(sx,sy, color='magenta')

# Annotate the vertex
plt.scatter(vx,vy, color="red")
plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy - 5)))

plt.show()
```

So from the plot, we can see that both of the values we calculated for ***x*** align with the parabola when ***y*** is 0. Additionally, because the parabola is symmetrical, we know that every pair of ***x*** values for each ***y*** value will be equidistant from the line of symmetry, so we can calculate the ***x*** value for the line of symmetry as the average of the ***x*** values for any value of ***y***. This in turn means that we know the ***x*** coordinate for the vertex (it's on the line of symmetry), and we can use the quadratic equation to calculate ***y*** for this point.

## Solving Quadratics Using the Square Root Method
The technique we just looked at makes it easy to calculate the two possible values for ***x*** when ***y*** is 0 if the equation is presented as the product two expressions. If the equation is in standard form, and it can be factored, you could do the necessary manipulation to restate it as the product of two expressions. Otherwise, you can calculate the possible values for x by applying a different method that takes advantage of the relationship between squared values and the square root.

Let's consider this equation:

\begin{equation}y = 3x^{2} - 12\end{equation}

Note that this is in the standard quadratic form, but there is no *b* term; in other words, there's no term that contains a coeffecient for  ***x*** to the first power. This type of equation can be easily solved using the square root method. Let's restate it so we're solving for ***x*** when ***y*** is 0:

\begin{equation}3x^{2} - 12 = 0\end{equation}

The first thing we need to do is to isolate the ***x2*** term, so we'll remove the constant on the left by adding 12 to both sides:

\begin{equation}3x^{2} = 12\end{equation}

Then we'll divide both sides by 3 to isolate x2:

\begin{equation}x^{2} = 4\end{equation}

No we can isolate ***x*** by taking the square root of both sides. However, there's an additional consideration because this is a quadratic equation. The ***x*** variable can have two possibe values, so we must calculate the *principle* and *negative* square roots of the expression on the right:

\begin{equation}x = \pm\sqrt{4}\end{equation}

The principle square root of 4 is 2 (because 22 is 4), and the corresponding negative root is -2 (because -22 is also 4); so *x* is **2** or **-2**.

Let's see this in Python, and use the results to calculate and plot the parabola with its line of symmetry and vertex:

```python
import pandas as pd
import math

y = 0
x1 = int(- math.sqrt(y + 12 / 3))
x2 = int(math.sqrt(y + 12 / 3))

# Create a dataframe with an x column containing some values to plot
df = pd.DataFrame ({'x': range(x1-10, x2+11)})

# Add a y column by applying the quadratic equation to x
df['y'] = 3*df['x']**2 - 12

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = 3*vx**2 - 12

# get min and max y values
miny = df.y.min()
maxy = df.y.max()

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# Plot calculated x values for y = 0
plt.scatter([x1,x2],[0,0], color="green")
plt.annotate('x1',(x1, 0))
plt.annotate('x2',(x2, 0))

# plot the line of symmetry
sx = [vx, vx]
sy = [miny, maxy]
plt.plot(sx,sy, color='magenta')

# Annotate the vertex
plt.scatter(vx,vy, color="red")
plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy - 20)))

plt.show()
```

## Solving Quadratics Using the Completing the Square Method
In quadratic equations where there is a *b* term; that is, a term containing **x** to the first power, it is impossible to directly calculate the square root. However, with some algebraic manipulation, you can take advantage of the ability to factor a polynomial expression in the form *a2 + 2ab + b2* as a binomial *perfect square* expression in the form *(a + b)2*.

At first this might seem like some sort of mathematical sleight of hand, but follow through the steps carefull and you'll see that there's nothing up my sleeve!

The underlying basis of this approach is that a trinomial expression like this:

\begin{equation}x^{2} + 24x + 12^{2}\end{equation}

Can be factored to this:

\begin{equation}(x + 12)^{2}\end{equation}

OK, so how does this help us solve a quadratic equation? Well, let's look at an example:

\begin{equation}y = x^{2} + 6x - 7\end{equation}

Let's start as we've always done so far by restating the equation to solve ***x*** for a ***y*** value of 0:

\begin{equation}x^{2} + 6x - 7 = 0\end{equation}

Now we can move the constant term to the right by adding 7 to both sides:

\begin{equation}x^{2} + 6x = 7\end{equation}

OK, now let's look at the expression on the left: *x2 + 6x*. We can't take the square root of this, but we can turn it into a trinomial that will factor into a perfect square by adding a squared constant. The question is, what should that constant be? Well, we know that we're looking for an expression like *x2 + 2**c**x + **c**2*, so our constant **c** is half of the coefficient we currently have for ***x***. This is **6**, making our constant **3**, which when squared is **9** So we can create a trinomial expression that will easily factor to a perfect square by adding 9; giving us the expression *x2 + 6x + 9*.

However, we can't just add something to one side without also adding it to the other, so our equation becomes:

\begin{equation}x^{2} + 6x + 9 = 16\end{equation}

So, how does that help? Well, we can now factor the trinomial expression as a perfect square binomial expression:

\begin{equation}(x + 3)^{2} = 16\end{equation}

And now, we can use the square root method to find x + 3:

\begin{equation}x + 3 =\pm\sqrt{16}\end{equation}

So, x + 3 is **-4** or **4**. We isolate ***x*** by subtracting 3 from both sides, so ***x*** is **-7** or **1**:

\begin{equation}x = -7, 1\end{equation}

Let's see what the parabola for this equation looks like in Python:

```python
import pandas as pd
import math

x1 = int(- math.sqrt(16) - 3)
x2 = int(math.sqrt(16) - 3)

# Create a dataframe with an x column containing some values to plot
df = pd.DataFrame ({'x': range(x1-10, x2+11)})

# Add a y column by applying the quadratic equation to x
df['y'] = ((df['x'] + 3)**2) - 16

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = ((vx + 3)**2) - 16

# get min and max y values
miny = df.y.min()
maxy = df.y.max()

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# Plot calculated x values for y = 0
plt.scatter([x1,x2],[0,0], color="green")
plt.annotate('x1',(x1, 0))
plt.annotate('x2',(x2, 0))

# plot the line of symmetry
sx = [vx, vx]
sy = [miny, maxy]
plt.plot(sx,sy, color='magenta')

# Annotate the vertex
plt.scatter(vx,vy, color="red")
plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy - 10)))

plt.show()
```

## Vertex Form
Let's look at another example of a quadratic equation in standard form:

\begin{equation}y = 2x^{2} - 16x + 2\end{equation}

We can start to solve this by subtracting 2 from both sides to move the constant term from the right to the left:

\begin{equation}y - 2 = 2x^{2} - 16x\end{equation}

Now we can factor out the coefficient for x2, which is **2**. 2x2 is 2 • x2, and -16x is 2 • 8x:

\begin{equation}y - 2 = 2(x^{2} - 8x)\end{equation}

Now we're ready to complete the square, so we add the square of half of the -8x coefficient on the right side to the parenthesis. Half of -8 is -4, and -42 is 16, so the right side of the equation becomes *2(x2 - 8x + 16)*. Of course, we can't add something to one side of the equation without also adding it to the other side, and we've just added 2 • 16 (which is 32) to the right, so we must also add that to the left.

\begin{equation}y - 2 + 32 = 2(x^{2} - 8x + 16)\end{equation}

Now we can simplify the left and factor out a perfect square binomial expression on the right:

\begin{equation}y + 30 = 2(x - 4)^{2}\end{equation}

We now have a squared term for ***x***, so we could use the square root method to solve the equation. However, we can also isolate ***y*** by subtracting 30 from both sides. So we end up restating the original equation as:

\begin{equation}y = 2(x - 4)^{2} - 30\end{equation}

Let's just quickly check our math with Python:

```python
from random import randint
x = randint(1,100)

2*x**2 - 16*x + 2 == 2*(x - 4)**2 - 30
```

    True

So we've managed to take the expression ***2x2 - 16x + 2*** and change it to ***2(x - 4)2 - 30***. How does that help?

Well, when a quadratic equation is stated this way, it's in *vertex form*, which is generically described as:

\begin{equation}y = a(x - h)^{2} + k\end{equation}

The neat thing about this form of the equation is that it tells us the coordinates of the vertex - it's at ***h,k***.

So in this case, we know that the vertex of our equation is 4, -30. Moreover, we know that the line of symmetry is at ***x = 4***.

We can then just use the equation to calculate two more points, and the three points will be enough for us to determine the shape of the parabola. We can simply choose any ***x*** value we like and substitute it into the equation to calculate the corresponding ***y*** value. For example, let's calculate ***y*** when x is **0**:

\begin{equation}y = 2(0 - 4)^{2} - 30\end{equation}

When we work through the equation, it gives us the answer **2**, so we know that the point (0, 2) is in our parabola.

So, we know that the line of symmetry is at ***x = h*** (which is 4), and we now know that the ***y*** value when ***x*** is 0 (***h*** - ***h***) is 2. The ***y*** value at the same distance from the line of symmetry in the negative direction will be the same as the value in the positive direction, so when ***x*** is ***h*** + ***h***, the ***y*** value will also be 2.

The following Python code encapulates all of this in a function that draws and annotates a parabola using only the ***a***, ***h***, and ***k*** values from a quadratic equation in vertex form:

```python
def plot_parabola_from_vertex_form(a, h, k):
    import pandas as pd
    import math

    # Create a dataframe with an x column a range of x values to plot
    df = pd.DataFrame ({'x': range(h-10, h+11)})

    # Add a y column by applying the quadratic equation to x
    df['y'] = (a*(df['x'] - h)**2) + k

    # get min and max y values
    miny = df.y.min()
    maxy = df.y.max()

    # calculate y when x is 0 (h+-h)
    y = a*(0 - h)**2 + k

    # Plot the line
    %matplotlib inline
    from matplotlib import pyplot as plt

    plt.plot(df.x, df.y, color="grey")
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid()
    plt.axhline()
    plt.axvline()

    # Plot calculated y values for x = 0 (h-h and h+h)
    plt.scatter([h-h, h+h],[y,y], color="green")
    plt.annotate(str(h-h) + ',' + str(y),(h-h, y))
    plt.annotate(str(h+h) + ',' + str(y),(h+h, y))

    # plot the line of symmetry (x = h)
    sx = [h, h]
    sy = [miny, maxy]
    plt.plot(sx,sy, color='magenta')

    # Annotate the vertex (h,k)
    plt.scatter(h,k, color="red")
    plt.annotate('v=' + str(h) + ',' + str(k),(h, k), xytext=(h - 1, (k - 10)))

    plt.show()

# Call the function for the example discussed above
plot_parabola_from_vertex_form(2, 4, -30)
```

It's important to note that the vertex form specifically requires a *subtraction* operation in the factored perfect square term. For example, consider the following equation in the standard form:

\begin{equation}y = 3x^{2} + 6x + 2\end{equation}

The steps to solve this are:
1. Move the constant to the left side:
\begin{equation}y - 2 = 3x^{2} + 6x\end{equation}
2. Factor the ***x*** expressions on the right:
\begin{equation}y - 2 = 3(x^{2} + 2x)\end{equation}
3. Add the square of half the x coefficient to the right, and the corresponding multiple on the left:
\begin{equation}y - 2 + 3 = 3(x^{2} + 2x + 1)\end{equation}
4. Factor out a perfect square binomial:
\begin{equation}y + 1 = 3(x + 1)^{2}\end{equation}
5. Move the constant back to the right side:
\begin{equation}y = 3(x + 1)^{2} - 1\end{equation}

To express this in vertex form, we need to convert the addition in the parenthesis to a subtraction:

\begin{equation}y = 3(x - -1)^{2} - 1\end{equation}

Now, we can use the a, h, and k values to define a parabola:

```python
plot_parabola_from_vertex_form(3, -1, -1)
```

## Shortcuts for Solving Quadratic Equations
We've spent some time in this notebook discussing how to solve quadratic equations to determine the vertex of a parabola and the ***x*** values in relation to ***y***. It's important to understand the techniques we've used, which incude:
- Factoring
- Calculating the Square Root
- Completing the Square
- Using the vertex form of the equation

The underlying algebra for all of these techniques is the same, and this consistent algebra results in some shortcuts that you can memorize to make it easier to solve quadratic equations without going through all of the steps:

### Calculating the Vertex from Standard Form
You've already seen that converting a quadratic equation to the vertex form makes it easy to identify the vertex coordinates, as they're encoded as ***h*** and ***k*** in the equation itself - like this:

\begin{equation}y = a(x - \textbf{h})^{2} + \textbf{k}\end{equation}

However, what if you have an equation in standard form?:

\begin{equation}y = ax^{2} + bx + c\end{equation}

There's a quick and easy technique you can apply to get the vertex coordinates.

1. To find ***h*** (which is the x-coordinate of the vertex), apply the following formula:
\begin{equation}h = \frac{-b}{2a}\end{equation}
2. After you've found ***h***, use it in the quadratic equation to solve for ***k***:
\begin{equation}\textbf{k} = a\textbf{h}^{2} + b\textbf{h} + c\end{equation}

For example, here's the quadratic equation in standard form that we previously converted to the vertex form:

\begin{equation}y = 2x^{2} - 16x + 2\end{equation}

To find ***h***, we perform the following calculation:

\begin{equation}h = \frac{-b}{2a}\;\;\;\;=\;\;\;\;\frac{-1 \cdot16}{2\cdot2}\;\;\;\;=\;\;\;\;\frac{16}{4}\;\;\;\;=\;\;\;\;4\end{equation}

Then we simply plug the value we've obtained for ***h*** into the quadratic equation in order to find ***k***:

\begin{equation}k = 2\cdot(4^{2}) - 16\cdot4 + 2\;\;\;\;=\;\;\;\;32 - 64 + 2\;\;\;\;=\;\;\;\;-30\end{equation}

Note that a vertex at 4,-30 is also what we previously calculated for the vertex form of the same equation:

\begin{equation}y = 2(x - 4)^{2} - 30\end{equation}

### The Quadratic Formula
Another useful formula to remember is the *quadratic formula*, which makes it easy to calculate values for ***x*** when ***y*** is **0**; or in other words:

\begin{equation}ax^{2} + bx + c = 0\end{equation}

Here's the formula:

\begin{equation}x = \frac{-b \pm \sqrt{b^{2} - 4ac}}{2a}\end{equation}

Let's apply that formula to our equation, which you may remember looks like this:

\begin{equation}y = 2x^{2} - 16x + 2\end{equation}

OK, let's plug the ***a***, ***b***, and ***c*** variables from our equation into the quadratic formula:

\begin{equation}x = \frac{--16 \pm \sqrt{-16^{2} - 4\cdot2\cdot2}}{2\cdot2}\end{equation}

This simplifes to:

\begin{equation}x = \frac{16 \pm \sqrt{256 - 16}}{4}\end{equation}

This in turn (with the help of a calculator) simplifies to:

\begin{equation}x = \frac{16 \pm 15.491933384829668}{4}\end{equation}

So our positive value for ***x*** is:

\begin{equation}x = \frac{16 + 15.491933384829668}{4}\;\;\;\;=7.872983346207417\end{equation}

And the negative value for ***x*** is:

\begin{equation}x = \frac{16 - 15.491933384829668}{4}\;\;\;\;=0.12701665379258298\end{equation}

The following Python code uses the vertex formula and the quadtratic formula to calculate the vertex and the -x and +x for y = 0, and then plots the resulting parabola:

```python
def plot_parabola_from_formula (a, b, c):
    import math

    # Get vertex
    print('CALCULATING THE VERTEX')
    print('vx = -b / 2a')

    nb = -b
    a2 = 2*a
    print('vx = ' + str(nb) + ' / ' + str(a2))

    vx = -b/(2*a)
    print('vx = ' + str(vx))

    print('\nvy = ax^2 + bx + c')
    print('vy =' + str(a) + '(' + str(vx) + '^2) + ' + str(b) + '(' + str(vx) + ') + ' + str(c))

    avx2 = a*vx**2
    bvx = b*vx
    print('vy =' + str(avx2) + ' + ' + str(bvx) + ' + ' + str(c))

    vy = avx2 + bvx + c
    print('vy = ' + str(vy))

    print ('\nv = ' + str(vx) + ',' + str(vy))

    # Get +x and -x (showing intermediate calculations)
    print('\nCALCULATING -x AND +x FOR y=0')
    print('x = -b +- sqrt(b^2 - 4ac) / 2a')

    b2 = b**2
    ac4 = 4*a*c
    print('x = ' + str(nb) + '+-sqrt(' + str(b2) + ' - ' + str(ac4) + ')/' + str(a2))

    sr = math.sqrt(b2 - ac4)
    print('x = ' + str(nb) + ' +- ' + str(sr) + ' / ' + str(a2))
    print('-x = ' + str(nb) + ' - ' + str(sr) + ' / ' + str(a2))
    print('+x = ' + str(nb) + ' + ' + str(sr) + ' / ' + str(a2))

    posx = (nb + sr) / a2
    negx = (nb - sr) / a2
    print('-x = ' + str(negx))
    print('+x = ' + str(posx))

    print('\nPLOTTING THE PARABOLA')
    import pandas as pd

    # Create a dataframe with an x column a range of x values to plot
    df = pd.DataFrame ({'x': range(round(vx)-10, round(vx)+11)})

    # Add a y column by applying the quadratic equation to x
    df['y'] = a*df['x']**2 + b*df['x'] + c

    # get min and max y values
    miny = df.y.min()
    maxy = df.y.max()

    # Plot the line
    %matplotlib inline
    from matplotlib import pyplot as plt

    plt.plot(df.x, df.y, color="grey")
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid()
    plt.axhline()
    plt.axvline()

    # Plot calculated x values for y = 0
    plt.scatter([negx, posx],[0,0], color="green")
    plt.annotate('-x=' + str(negx) + ',' + str(0),(negx, 0), xytext=(negx - 3, 5))
    plt.annotate('+x=' + str(posx) + ',' + str(0),(posx, 0), xytext=(posx - 3, -10))

    # plot the line of symmetry
    sx = [vx, vx]
    sy = [miny, maxy]
    plt.plot(sx,sy, color='magenta')

    # Annotate the vertex
    plt.scatter(vx,vy, color="red")
    plt.annotate('v=' + str(vx) + ',' + str(vy),(vx, vy), xytext=(vx - 1, vy - 10))

    plt.show()

plot_parabola_from_formula (2, -16, 2)
```

```python

```

MathsToML/Module01-Equations, Graphs, and Functions/01-07-Quadratic Equations.ipynb


## Learning Objectives

By the end of this session you should be able to...

1. Take the derivative of a function over one variable
1. Take the partial derivative of a function over all of its variables
1. Find the minimum of the function to obtain the best line that represents relationships between two variables in a dataset

## Why are derivatives important?

Derivatives are the foundation for Linear Regression (a topic we'll cover later in the course) that allows us to obtain the best line that represents relationships between two variables in a dataset.

## Introduction to Derivatives

The process of fidning a derivative is called **Differentiation**, which is a technique used to calculate the slope of a graph at different points.

### Activity - Derivative Tutorial:

1. Go through this [Derivative tutorial from Math Is Fun](https://www.mathsisfun.com/calculus/derivatives-introduction.html) (15 min)
1. When you're done, talk with a partner about topics you still have questions on. See if you can answer each other's questions. (5 min)
1. We'll then go over questions on the tutorial as a class (10 min)

### Review Diagram

Review the below diagram as a class, and compare with what you just learned in the above Derivative Tutorial. Note that a Gradient Function is just another name for the Derivative of a function:

## Derivative Formula

- Choose small $\Delta x$

- $f^\prime(x) = \frac{d}{dx}f(x) = \frac{\Delta y}{\Delta x}  = \frac{f(x + \Delta x) - f(x)}{\Delta x}$

Remember that $\Delta x$ approaches 0. So if plugging in a value in the above formula, choose a _very_ small number, or simplify the equation further such that all $\Delta x = 0$, like we saw in the tutorial

## Activity: Write a Python function that calculates the gradient of $x^2$ at $x = 3$ and $x = -2$ using the above definition

```python
def f(x):
    return x**2

eps = 1e-6
x = 3
print((f(x + eps) - f(x)) / eps)
x = -2
print((f(x + eps) - f(x)) / eps)
```

    6.000001000927568
    -3.999998999582033

Note that these values match $2x$, our derivative of $x^2$:

$2*3 = 6$

$2 * -2 = -4$

## Derivative Table

As a shortcut, use the second page of this PDF to find the derivative for common formulas. Utilize this as a resource going forward!

- https://www.qc.edu.hk/math/Resource/AL/Derivative%20Table.pdf

## Extend Gradient into Two-Dimensional Space

Now we know how to calculate a derivative of one variable. But what if we have two?

To do this, we need to utilize **Partial Derivatives**. Calculating a partial derivative is essentially calculating two derivatives for a function: one for each variable, where they other variable is set to a constant.

### Activity - Partial Derivative Video

Lets watch this video about Partial Derivative Intro from Khan Academy: https://youtu.be/AXqhWeUEtQU

**Note:** Here are some derivative shortcuts that will help in the video:

$\frac{d}{dx}x^2 = 2x$

$\frac{d}{x}sin(x) = cos(x)$

$\frac{d}{dx}x = 1$

### Activity - Now You Try!
Consider the function $f(x, y) = \frac{x^2}{y}$

- Calculate the first order partial derivatives ($\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$) and evaluate them at the point $P(2, 1)$.

## We can use the Symbolic Python package (library) to compute the derivatives and partial derivatives

```python
from sympy import symbols, diff
# initialize x and y to be symbols to use in a function
x, y = symbols('x y', real=True)
f = (x**2)/y
# Find the partial derivatives of x and y
fx = diff(f, x, evaluate=True) # partial derivative of f(x,y) with respect to x
fy = diff(f, y, evaluate=True) # partial derivative of f(x,y) with respect to y
print(fx)
print(fy)
# print(f.evalf(subs={x: 2, y: 1}))
print(fx.evalf(subs={x: 2, y: 1}))
print(fy.evalf(subs={x: 2, y: 1}))
```

    2*x/y
    -x**2/y**2
    4.00000000000000
    -4.00000000000000

## Optional Reading: Tensorflow is a powerful package from Google that calculates the derivatives and partial derivatives numerically

```python
import tensorflow as tf

x = tf.Variable(2.0)
y = tf.Variable(1.0)

with tf.GradientTape(persistent=True) as t:
    z = tf.divide(tf.multiply(x, x), y)

# Use the tape to compute the derivative of z with respect to the
# intermediate value x and y.
dz_dx = t.gradient(z, x)
dz_dy = t.gradient(z, y)

print(dz_dx)
print(dz_dy)

# All at once:
gradients = t.gradient(z, [x, y])
print(gradients)

del t
```

## Optional Reading: When x and y are declared as constant, we should add `t.watch(x)` and `t.watch(y)`

```python
import tensorflow as tf

x = tf.constant(2.0)
y = tf.constant(1.0)

with tf.GradientTape(persistent=True) as t:
    t.watch(x)
    t.watch(y)
    z = tf.divide(tf.multiply(x, x), y)

# Use the tape to compute the derivative of z with respect to the
# intermediate value y.
dz_dx = t.gradient(z, x)
dz_dy = t.gradient(z, y)
```

# Calculate Partial Derivative from Definition

```python
def f(x, y):
    return x**2/y

eps = 1e-6
x = 2
y = 1
print((f(x + eps, y) - f(x, y)) / eps)
print((f(x, y + eps) - f(x, y)) / eps)
```

    4.0000010006480125
    -3.9999959997594203

Looks about right! This works rather well, but it is just an approximation. Also, you need to call `f()` at least once per parameter (not twice, since we could compute `f(x, y)` just once). This makes this approach difficult to control for large systems (for example neural networks).

## Why Do we need Partial Gradients?

In many applications, more specifically DS applications, we want to find the Minimum of a cost function

- **Cost Function:** a function used in machine learning to help correct / change behaviour to minimize mistakes. Or in other words, a measure of how wrong the model is in terms of its ability to estimate the relationship between x and y. [Source](https://towardsdatascience.com/machine-learning-fundamentals-via-linear-regression-41a5d11f5220)

Why do we want to find the minimum for a cost function? Given that a cost function mearues how wrong a model is, we want to _minimize_ that error!

In Machine Learning, we frequently use models to run our data through, and cost functions help us figure out how badly our models are performing. We want to find parameters (also known as **weights**) to minimize our cost function, therefore minimizing error!

We find find these optimal weights by using a **Gradient Descent**, which is an algorithm that tries to find the minimum of a function (exactly what we needed!). The gradient descent tells the model which direction it should take in order to minimize errors, and it does this by selecting more and more optimal weights until we've minimized the function! We'll learn more about models when we talk about Linear Regression in a future lesson, but for now, let's review the Gradient Descent process with the below images, given weights $w_0$ and $w_1$:

Look at that bottom right image. Looks like we're using partial derivatives to find out optimal weights. And we know exactly how to do that!

## Finding minimum of a function

Assume we want to minimize the function $J$ which has two weights $w_0$ and $w_1$

We have two options to find the minimum of $J(w_0, w_1)$:

1. Take partial derivatives of $J(w_0, w_1)$ with relation to $w_0$ and $w_1$:

$\frac{\partial J(w_0, w_1)}{\partial w_0}$

$\frac{\partial J(w_0, w_1)}{\partial w_1}$

And find the appropriate weights such that the partial derivatives equal 0:

$\frac{\partial J(w_0, w_1)}{\partial w_0} = 0$

$\frac{\partial J(w_0, w_1)}{\partial w_1} = 0$

In this approach we should solve system of linear or non-linear equation

2. Use the Gradient Descent algorithm:

First we need to define two things:

- A step-size alpha ($\alpha$) -- also called the *learning rate* -- as a small number (like $1.e-5$)
- An arbitrary random initial value for $w_0$ and $w_1$: $w_0 = np.random.randn()$ and $w_1 = np.random.randn()$

Finally, we need to search for the most optimal $w_0$ and $w_1$ by using a loop to update the weights until we find the most optimal weights. We'll need to establish a threshold to compare weights to know when to stop the loop. For example, if the weight update -- the change in the weight parameter -- from one iteration is within 0.0001 of the weight from the previous iteration, we can stop the loop (0.0001 is our threshold here)

Let's review some pseudocode for how to implement this algorithm:

```
# initialization
initialize the following:
    a starting weight value -- an initial guess, could be random
    the learning rate (alpha), a small number (we'll choose 1.e-5)
    the threshold -- set this to 1.e-4
    the current weight update -- initialize to 1

# weight update loop
while the weight update is greater than the threshold:
    store the current values of the weights into a previous value variable
    set the weight values to new values based on the algorithm, by adding the weight updates
```

How do we `set the weight values to new values based on the algorithm`? by using the below equations:

$w_0 = w_0 - \alpha \frac{\partial J(w_0, w_1)}{\partial w_0}$

$w_1 = w_1 - \alpha \frac{\partial J(w_0, w_1)}{\partial w_1}$

Finish the "starter code" block below, creating real code from the pseudocode!

**Stretch Challenge:** We may also want to limit the number of loops we do, in addition to checking the threshold. Determine how we may go about doing that

## Resources

- [Derivative tutorial from Math Is Fun](https://www.mathsisfun.com/calculus/derivatives-introduction.html)
- [Derivative Table](https://www.qc.edu.hk/math/Resource/AL/Derivative%20Table.pdf)
- [Khan Academy - Partial Derivatives video](https://www.youtube.com/watch?v=AXqhWeUEtQU&feature=youtu.be)
- [Towards Data Science - Machine Learning Fundamentals: cost functions and gradient Descent](https://towardsdatascience.com/machine-learning-fundamentals-via-linear-regression-41a5d11f5220)

## Gradient descent in one dimension

```python
# pseudo-code
def minimize(f):
    # Initialize

    # run the weight update loop until it terminates

    # return the current weights
```

```python
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(42)
eps = 1.e-10

# define an interesting function to minimize
x = np.linspace(-10,10, 100)
fx =lambda x: (1/50)*x**4 - 2*x**2 + x + 1
df_dx = lambda x: (4/50)*x**3 - 4*x + 1
plt.plot(x,fx(x),label = 'function f(x)')
plt.plot(x,df_dx(x),'r--',label = 'derivative df/dx')
plt.legend()
plt.grid()
```

### gradient descent function

```python
def minimize(fx,x_init):
    # Initialize
    alpha = 1.e-6
    thresh = 1.e-4
    weight_update = 1.
    x_values = []
    max_iter = 1000
    n_iter = 0
    x = x_init
    eps = 1.e-10

    # run the weight update loop until it terminates
    while np.abs(weight_update) > thresh: #and n_iter < max_iter:
        n_iter+=1
        df_dx = (fx(x+eps) - fx(x))/eps
        weight_update = -alpha*df_dx
        x = x + weight_update
        x_values.append(x)

    # return the final value of the weight -- which should correspond to the minimum of the function
    return x, x_init, x_values, n_iter
```

### Choose an initial value for x, then run gradient descent

```python
np.random.seed(42)
x_init = np.random.uniform(-10,10,1) # choose a random starting point
print(x_init)
x_star, x_init, x_values, n_iter = minimize(f,x_init)
print(f'Started at {x_init}, found minimum {x_star} after {n_iter} iterations')
```

    [-2.50919762]
    Started at [-2.50919762], found minimum [-2.5092074] after 1 iterations

### Check that the derivative of the function is indeed zero at the minimum found by gradient descent

```python
# derivative, from calculus
print(f'derivative from calculus: {df_dx(x_star)}')

# derivative, from definition
print(f'derivative from calculus:  {(fx(x_star + eps) - fx(x_star) )/eps}')
```

    derivative from calculus: [10.01321418]
    derivative from calculus:  [10.01321692]

```python
x_values = np.array(x_values)
plt.plot(x,fx(x),label = 'function f(x)')
plt.plot(np.array(x_values),fx(x_values),'.',markersize = 1, label='gradient descent updates')
plt.plot(x_init,fx(x_init),'r.',markersize = 10, label = '$x_{init}$')
plt.plot(x_star,fx(x_star),'k*',markersize = 10, label = '$x_{star}$')

plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()
plt.legend(loc='best');
```

```python

```


```python
from sympy.integrals.quadrature import gauss_legendre
```

```python
gauss_legendre(4, 17)
```

    ([-0.86113631159405258,
      -0.33998104358485626,
      0.33998104358485626,
      0.86113631159405258],
     [0.34785484513745386,
      0.65214515486254614,
      0.65214515486254614,
      0.34785484513745386])

Compare against:

https://github.com/certik/hfsolver/blob/0693c60abc8da820b9c626f2092bf9d767644a8b/src/quadrature.f90#L30

Points:

```fortran
   case(4)
      xi(1)=-0.86113631159405258_dp
      xi(2)=-0.33998104358485626_dp
      xi(3)=0.33998104358485626_dp
      xi(4)=0.86113631159405258_dp
```

Weights:

```fortran
   case(4)
      w(1)=0.34785484513745386_dp
      w(2)=0.65214515486254614_dp
      w(3)=0.65214515486254614_dp
      w(4)=0.34785484513745386_dp
```

```python
gauss_legendre??
```

applications/Applications -- Numerical Quadrature.ipynb


```python
from sympy import *
```

```python
x,y,d,z = symbols('x,y,d,z')
```

ecuación

```python
 d = (x + y) ** 2
 d
```

$\displaystyle \left(x + y\right)^{2}$

Expandimos la ecuación

```python
d = expand((x+y)**2)
d
```

$\displaystyle x^{2} + 2 x y + y^{2}$

```python
d= expand((x+y)**3)
d
```

$\displaystyle x^{3} + 3 x^{2} y + 3 x y^{2} + y^{3}$

Ecuación con dos variables

```python
d =  5 * x + 3 * y
d
```

$\displaystyle 5 x + 3 y$

Ecuación con 3 variables

```python
d = 5 * x + 2 * y + 12 * z
d
```

$\displaystyle 5 x + 2 y + 12 z$