# Introduction and Motivation

Binomial distribution plays an important role in real world applications. It measures the probability of having $k$ successes out of $n$ i.i.d. trials with each trial having a success rate $p$. The PDF of binomial distribution with success rate $p$, total number of trials $n$ and the number of successes $k$ is
\begin{equation}
f(k,n,p)={n\choose k}p^{k}(1-p)^{n-k}
\label{bino_pdf}
\end{equation}

However, it may be difficult to directly use formula because it may contain large and small terms. Typically, ${n\choose k}$ is very large, whereas $p^k$ and $(1-p)^{n-k}$ are very small. These may cause overflow or underflow when computing them. It would be more convenient if we could reformulate the formula that avoids this issue. Is it possible to do? Maybe we can observe an example of this distribution to get some ideas.

Given the success rate $p$ of i.i.d. trials and the total number of trials $n$, the above equation gives the relationship between $k$ and $f(k,n,p)$. Below is an example of binomial distribution PDF with $n=3000$ and $p=0.3$.

The above figure looks very similar to a normal distribution, which makes intuitive sense. The peak of the distribution should coorespond to $k=np$. Furthermore, it is also very likely that $k$ can deviate around $np$ because it's very possible that we may get some more or fewer successes than expected. However, it is unlikely that the $k$ we get deviates too far from $np$. All of these intuitions can be visualized on the graph. Therefore, one may ask, is it possible to have a normal distribution to approximate the binomial distribution? It turns out the answer is yes. In this project, we will study the relationships between binomial distributions their corresponding normal approximation.

# Derivation

We begin by presenting the derivation from this paper[1].
Before we start, it is worth noting an approximation for factorials called Stirling's formula[2].
$$
\begin{aligned}
n!&=\sqrt{2\pi n}\left(\frac{n}{e}\right)^n\left(1+\frac{1}{12n}+\frac{1}{288n^2}-\frac{1}{51840n^3}+...\right)\\
&=\sqrt{2\pi n}\left(\frac{n}{e}\right)^n\left(1+O\left(\frac{1}{n}\right)\right)
\end{aligned}
$$

Equipped with this formula, we can proceed on making the approximation.
$$
\begin{aligned}
f(k,n,p)&={n\choose k}p^{k}(1-p)^{n-k}\\
&=\frac{n!}{k!(n-k)!}p^{k}(1-p)^{n-k}\\
&=\sqrt{\frac{n}{2\pi k(n-k)}}\left(\frac{np}{k}\right)^{k}\left(\frac{n(1-p)}{n-k}\right)^{n-k}\left[1+O\left(\frac1n\right)\right]
\end{aligned}
$$

Defining $\delta=k-np$, we have $k=\delta+np$ and $n-k=n(1-p)-\delta$. Expanding $\textrm{ln}(1+k)=k-\frac{1}{2}k^2+O(k^3)$, we have,
$$
\textrm{ln}\left[\left(\frac{np}{k}\right)^k\left(\frac{n(1-p)}{n-k}\right)^{n-k}\right]=-\frac{\delta^2}{2np(1-p)}+O\left(\frac{\delta^3}{n^2}\right)
$$

Exponentiating the result, we have
$$
\begin{aligned}
\left(\frac{np}{k}\right)^k\left(\frac{n(1-p)}{n-k}\right)^{n-k}&=\textrm{exp}\left(-\frac{\delta^2}{2np(1-p)}+O\left(\frac{\delta^3}{n^2}\right)\right)\\
&=\textrm{exp}\left(-\frac{\delta^2}{2np(1-p)}\right)\textrm{exp}\left(O\left(\frac{\delta^3}{n^2}\right)\right)\\
\end{aligned}
$$

Because $\delta$ is just the difference between observed $k$ and the expected value of $k$ which is $np$, $\delta$ should be on the scale of a few standard deviations, which is $\sqrt{np(1-p)}$. Therefore, we argue that $\delta\sim\sqrt{n}$.
Therefore,
$$
O\left(\frac{\delta^3}{n^2}\right)=O\left(\frac{n^{\frac32}}{n^2}\right)=O\left(\frac{1}{\sqrt{n}}\right).
$$
As $n$ gets large, $\frac{1}{\sqrt{n}}$ gets small.
Using $\textrm{exp}(x)=1+x+\frac{x^2}{2}+...=1+x+O(x^2)$, the above equation becomes
$$
\begin{aligned}
\left(\frac{np}{k}\right)^k\left(\frac{n(1-p)}{n-k}\right)^{n-k}=\textrm{exp}\left(-\frac{\delta^2}{2np(1-p)}\right)\left[1+O\left(\frac{1}{\sqrt{n}}\right)\right].
\end{aligned}
$$

For the square root part, we have
$$
\begin{aligned}
\sqrt{\frac{n}{2\pi k(n-k)}}&=\sqrt{\frac{n}{2\pi(np+\delta)(n(1-p)-\delta)}}\\
&=\sqrt{\frac{1}{2\pi np(1-p)}}\left[1+O\left(\frac{1}{\sqrt{n}}\right)\right]
\end{aligned}
$$

Multiplying them together, we obtain
$$
\begin{aligned}
f(k,n,p)=\sqrt{\frac{1}{2\pi np(1-p)}}\textrm{exp}\left(-\frac{(k-np)^2}{2np(1-p)}\right)\left[1+O\left(\frac{1}{\sqrt{n}}\right)\right]
\end{aligned}
$$

We know that for binomial distribution $f(k)$ with fixed $n$ and $p$, its mean of $k$ is $\mu=np$, and its variance is $\sigma^2=np(1-p)$. Therefore, we can rewrite the formula above as
$$
f(k)=\frac{1}{\sqrt{2\pi} \sigma}\textrm{exp}\left(-\frac{(k-\mu)^2}{2\sigma^2}\right)\left[1+O\left(\frac{1}{\sqrt{n}}\right)\right]
$$

This is exactly a normal distribution with mean $\mu$ and variance $\sigma^2$ with a correction term $O\left(\frac{1}{\sqrt{n}}\right)$. As $n\rightarrow\infty$, $O\left(\frac{1}{\sqrt{n}}\right)$ becomes a small value compared to $1$.

There's a caveat here. we arrived at this result by assuming that $\delta\sim\sqrt{n}$, which means that this formula is applicable only when $k$ is within a few standard deviations to $np$. This formula may fail for significantly deviated $k$. This makes intuitive sense. After all, the domain of $f(k)$ is $[0,n]$ whereas the domain of normal distribution is $(-\infty,\infty)$. For $k$ smaller than $0$ or larger than $n$, our approximation returns a positive value whereas the true binomial formula returns $0$, and the ratio between them will be infinity. This is not to be worried about because our approximation disregards those exceptional cases where $\delta\sim n$ rather than $\delta\sim\sqrt{n}$.

We can now define the approximation $g(k)$ of $f(k)$ by neglecting the $O\left(\frac{1}{\sqrt{n}}\right)$ term.
$$
g(k)=\frac{1}{\sqrt{2\pi} \sigma}\textrm{exp}\left(-\frac{(k-\mu)^2}{2\sigma^2}\right).
$$

# Approximation Error Visualizations

As previously discussed, our approximation works well when $n$ is large $k$ is within a few standard deviations to $\mu$. We can visualize both the approximation and the exact function with varying $n$. Here are some examples. The x-axis limits are determined by $\mu\pm 5\sigma$ for all of the plots. The blue and red curves represent the approximations $g(k)$ and the exact functions $f(k)$, respectively.

As $n$ gets larger, the differences between two plots becomes more similar. At $n=400$, they are essentially indistinguishable. However, we may still be interested in seeing how the differences decrease with increasing $n$. Defining $d(k)=g(k)-f(k)$, we can visualize $d(k)$.

We can see that as $n$ increases the magnitudes of $d(k)$ becomes smaller, but it gets wider. Moreover, its general structures are more or less preserved. Take $n = 400$ and $n=1000$ as a comparison, if we scale y-coordinates the $n=400$ plot down, but scale up its x-coordinates, we will get some results similar to the $n=1000$ plot. In other words, if we ignore the x and y scales, the $n=400$ plot and $n=1000$ plot look pretty much identical.

We should be aware that the correction is also a function of $k$, because otherwise $d(k)$ will be a constant function. Furthermore, the observations above suggest that on the x-axis, the pattern induced by the correction lengthens in accordance to $\sigma$. For example, if $\sigma$ doubles, we should see the distance between the two local minima around the center peak of $d(k)$ also doubles. That is, the plot becomes twice as wide. For a normal distribution, if $\sigma$ doubles, we need to go $\sqrt{2}$ as far from $\mu$ to achieve the same percentile as before. Because $\sigma$ scales with $\sqrt{n}$, it implies that when $n$ is large, the contribution from the term $\frac{k}{\sqrt{n}}$ dominates those from other terms as a function $y\left(\frac{x}{\sqrt{n}}\right)$ is $\sqrt{n}$ times as wide as $y(x)$, and the disproportionalities of the graphs caused by other terms, if they exist, are visually negligible.

We may also visualize the correction terms itself by defining a new function $r(k)$ such that
$$
\begin{aligned}
r(k)&=-\frac{d(k)}{g(k)}\\
&=-\frac{\frac{-1}{\sqrt{2\pi} \sigma}\textrm{exp}\left(-\frac{(k-\mu)^2}{2\sigma^2}\right)}{\frac{1}{\sqrt{2\pi} \sigma}\textrm{exp}\left(-\frac{(k-\mu)^2}{2\sigma^2}\right)}O\left(\frac{1}{\sqrt{n}}\right)\\
&=O\left(\frac{1}{\sqrt{n}}\right)
\end{aligned}
$$

The plots above confirms the claims that for large $n$ and $k$ not too far from $\mu$, the correction terms are small and therefore the approximations are very accurate.

An interesting observation of $r(k)$ is that its local maxima doesn't seem to be at $\mu$, but rather at some distances away from $\mu$. We can zoom in around $k=\mu$ to see more details.

It looks like at $k=\mu$, the correction $r(k)$ is negative, making $g(\mu)>f(\mu)$. As $k$ deviates from $\mu$, the difference becomes smaller and eventually $g(k)<f(k)$ for positive corrections. Then, the corrections starts decreasing which will make $g(k)>f(k)$ again.

Scale wise, while we already know that $r(k)$ roughly widens as $\sqrt{n}$, its amplitudes decrease roughly as $n^{-1}$. Take $n=100$ and $n=1000$ as a comparison, $r(\mu)$ for $n=1000$ is roughly $10$ times smaller than that for $n=100$.

# Conclusion

We have constructed a normal approximation to a binomial distribution. The error analysis indicates that the approximation errors tends to be insignificant if $n$ is large and $k$ is not too far from $\mu$. More specifically, the corrections stays "in sync" with the approximation in a sense that the final graphs corresponding to different $n$'s are roughly proportional. Moreover, the amplitudes of the corrections decreases  roughly as $n^{-1}$.

# Generalizations

We may generalize our discussions about binomial distributions to multinomial distributions, using the same technique. In fact, the generalization has already been discussed. For a multinomial $f(\mathbf{k},n,\mathbf{p})$ such that
$$
\begin{aligned}
\mathbf{k}&=[k_1,k_2,...k_m]\\
\mathbf{p}&=[p_1,p_2,...p_m]\\
n&=\sum_{i=1}^{m}k_i\\
0&<=p_1,p_2,...,p_m<=1
\end{aligned}
$$
for varibles $\mathbf{X}=[X_1,X_2,...X_M]$, where $X_1,X_2,...,X_m$ are i.i.d. only within themselves. We can approximate this multinomial distribution PDF as
$$
\mathcal{N}(n\mathbf{p},n\mathbf{M}),
$$
where $\mathbf{M}=\mathbf{P}-\mathbf{p}\mathbf{p}^\top$
where $\mathbf{P}$ is a diagonal matrix with diagonal entries being elements of $\mathbf{p}$.

# References

[1]The Normal Approximation to the Binomial Distribution. http://scipp.ucsc.edu/~haber/ph116C/NormalApprox.pdf

[2]Stirling's approximation. In Wikipedia. https://en.wikipedia.org/wiki/Stirling%27s_approximation

[3]Geyer, C. Stat 5151 Notes: Brand Name Distributions, http://www.stat.umn.edu/geyer/5102/notes/brand.pdf

```julia

```


Euler Problem 70
================

Euler's Totient function, φ(n) [sometimes called the phi function], is used to
determine the number of positive numbers less than or equal to n which are
relatively prime to n. For example, as 1, 2, 4, 5, 7, and 8, are all less than
nine and relatively prime to nine, φ(9)=6. The number 1 is considered to be
relatively prime to every positive number, so φ(1)=1.

Interestingly, φ(87109)=79180, and it can be seen that 87109 is a permutation
of 79180.

Find the value of n, 1 < n < 10^7, for which φ(n) is a permutation of n and the
ratio n/φ(n) produces a minimum.

```python
from sympy import sieve, primepi
N = 10 ** 7
n = int(N ** 0.5)
min_ratio = 1.005
best_n = None
primes = list(sieve.primerange(1, N))
pi = primepi(n)
num_primes = len(primes)
for i in range(pi, -1, -1):
    p = primes[i]
    ratio = p / (p - 1)
    if ratio > min_ratio:
        break
    for j in range(i+1, num_primes):
        q = primes[j]
        n = p * q
        if n > N:
            break
        if p / (p - 1) > min_ratio:
            break
        if sorted(str(n)) == sorted(str(n - p - q + 1)):
            ratio = 1.0 * p * q / (p - 1) / (q - 1)
            if ratio < min_ratio:
                min_ratio = ratio
                best_n = n
print(best_n)
```

    8319823

**Discussion:** The ratio n/φ(n) is equal to the product of p/(p-1) for all distinct prime factors p of n.
We may assume that n has no repeated factors.

If n is prime then φ(n) = n - 1, so the digits of φ(n) cannot be a permutation of the digits of n.

If n is the product of three or more prime factors, then its smallest prime factor is less than 200, so n/φ(n) > 1.005.

Suppose that n is the product of two distinct prime factors p and q (p < q). Then n/φ(n) = p/(p-1) * q/(q-1). If the minimum value realized in this case is less than 1.005, then we have found the optimal value of n.


# Part 1 - Scalars and Vectors

For the questions below it is not sufficient to simply provide answer to the questions, but you must solve the problems and show your work using python (the NumPy library will help a lot!) Translate the vectors and matrices into their appropriate python  representations and use numpy or functions that you write yourself to demonstrate the result or property.

## 1.1 Create a two-dimensional vector and plot it on a graph

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

two_d_vector = [.4, 1]

plt.arrow(0,0, two_d_vector[0], two_d_vector[1],head_width=.1, head_length=.1, color ='orange')
plt.xlim(-.2,1)
plt.ylim(-.2,1.2)
```

## 1.2 Create a three-dimensional vector and plot it on a graph

```
vectors = np.array([[0, 0, 0, 2, 3, 4],
                    [0, 0, 0, 7, 4, 6],
                    [0, 0, 0, 2, 8, 9]])

X, Y, Z, U, V, W = zip(*vectors)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.quiver(X, Y, Z, U, V, W, length=.5)
ax.set_xlim([0, 4])
ax.set_ylim([0, 6])
ax.set_zlim([0, 5])
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
plt.show()
```

```

```

## 1.3 Scale the vectors you created in 1.1 by $5$, $\pi$, and $-e$ and plot all four vectors (original + 3 scaled vectors) on a graph. What do you notice about these vectors?

```
from math import e, pi

two_d_vector = [.4, 1]
scaled_to_e = np.multiply(e, two_d_vector)
scaled_to_pi = np.multiply(pi, two_d_vector)
scaled_by_5 = np.multiply(5, two_d_vector)

plt.arrow(0,0, scaled_by_5[0], scaled_by_5[1],head_width=.1, head_length=.1, color ='green')
plt.arrow(0,0, scaled_to_pi[0], scaled_to_pi[1],head_width=.1, head_length=.1, color ='blue')
plt.arrow(0,0, scaled_to_e[0], scaled_to_e[1],head_width=.1, head_length=.1, color ='red')
plt.arrow(0,0, two_d_vector[0], two_d_vector[1],head_width=.1, head_length=.1, color ='orange')

plt.xlim(-.2,3)
plt.ylim(-.2,6)
```

```

```

## 1.4 Graph vectors $\vec{a}$ and $\vec{b}$ and plot them on a graph

\begin{align}
\vec{a} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}
\qquad
\vec{b} = \begin{bmatrix} 3 \\4 \end{bmatrix}
\end{align}

```
a = np.array([5,7])
b = np.array([3,4])

plt.arrow(0,0,a[0],a[1], head_width=.2, head_length=.2, color='b')
plt.arrow(0,0,b[0],b[1], head_width=.2, head_length=.2, color='r')

plt.xlim(0, 6)
plt.ylim(0,8)

```

## 1.5 find $\vec{a} - \vec{b}$ and plot the result on the same graph as $\vec{a}$ and $\vec{b}$. Is there a relationship between vectors $\vec{a} \thinspace, \vec{b} \thinspace \text{and} \thinspace \vec{a-b}$

```
a = np.array([5,7])
b = np.array([3,4])
c = a - b

plt.arrow(0,0,a[0],a[1], head_width=.2, head_length=.2, color='b')
plt.arrow(0,0,b[0],b[1], head_width=.2, head_length=.2, color='r')
plt.arrow(0,0,c[0],c[1], head_width=.2, head_length=.2, color='y')

plt.xlim(0, 6)
plt.ylim(0,8)
```

## 1.6 Find $c \cdot d$

\begin{align}
\vec{c} = \begin{bmatrix}7 & 22 & 4 & 16\end{bmatrix}
\qquad
\vec{d} = \begin{bmatrix}12 & 6 & 2 & 9\end{bmatrix}
\end{align}

```
c = np.array([7, 22, 4, 16])
d = np.array([12, 6, 2, 9])

c_dot_d = (c*d).sum()

c_dot_d
```

    368

```
np.vdot(c, d )
```

    368

##  1.7 Find $e \times f$

\begin{align}
\vec{e} = \begin{bmatrix} 5 \\ 7 \\ 2 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\4 \\ 6 \end{bmatrix}
\end{align}

```
e = np.array([5, 7, 2])
f = np.array([3, 4, 6])

e_cross_f = np.cross(e,f)

e_cross_f
```

    array([ 34, -24,  -1])

```
np.array
```

## 1.8 Find $||g||$ and then find $||h||$. Which is longer?

\begin{align}
\vec{g} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 8 \end{bmatrix}
\qquad
\vec{h} = \begin{bmatrix} 3 \\3 \\ 3 \\ 3 \end{bmatrix}
\end{align}

```
g = np.array([1, 1, 1, 8])
h = np.array([3, 3, 3, 3])

g_magnitude = np.sqrt((g**2).sum())
h_magnitude = np.sqrt((h**2).sum())

if g_magnitude > h_magnitude:
  print(f"The magnitude of g ({g_magnitude}) is greater than the magnitude of h ({h_magnitude})")
else:
    print(f"The magnitude of h ({h_magnitude}) is greater than the magnitude of g ({g_magnitude})")
```

    The magnitude of g (8.18535277187245) is greater than the magnitude of h (6.0)

# Part 2 - Matrices

## 2.1 What are the dimensions of the following matrices? Which of the following can be multiplied together? See if you can find all of the different legal combinations.
\begin{align}
A = \begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
\qquad
B = \begin{bmatrix}
2 & 4 & 6 \\
\end{bmatrix}
\qquad
C = \begin{bmatrix}
9 & 6 & 3 \\
4 & 7 & 11
\end{bmatrix}
\qquad
D = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\qquad
E = \begin{bmatrix}
1 & 3 \\
5 & 7
\end{bmatrix}
\end{align}

```
A = np.array([[1, 2],
              [3, 4],
              [5,6]])
B = np.array([2, 4, 6])
C = np.array([[9, 6, 3],
              [4, 7, 11]])
D = np.array([[1, 0, 0],
              [0, 1, 0],
              [0, 0, 1]])
E = np.array([[1, 3],
              [5, 7]])
```

A - 3x2
> A*B = No // A*C = Yes // A*D = No // A*E = Yes

B - 1x3

> BA = Yes // BC = No // BD = Yes // BE = No

C - 2x3

> CA = Yes // CB = No // CD = Yes // CE = No

D - 3x3

> DA = Yes // DB = No // DC = No // DE = No

E - 2x2

> EA = No // EB = No // EC = Yes // ED = No

## 2.2 Find the following products: CD, AE, and BA. What are the dimensions of the resulting matrices? How does that relate to the dimensions of their factor matrices?

```
print(np.matmul(C,D))
print(np.matmul(A,E))
print(np.matmul(B,A))

```

    [[ 9  6  3]
     [ 4  7 11]]
    [[11 17]
     [23 37]
     [35 57]]
    [44 56]

CD = 2x3
AE = 3x2
BA = 1x2

## 2.3  Find $F^{T}$. How are the numbers along the main diagonal (top left to bottom right) of the original matrix and its transpose related? What are the dimensions of $F$? What are the dimensions of $F^{T}$?

\begin{align}
F =
\begin{bmatrix}
20 & 19 & 18 & 17 \\
16 & 15 & 14 & 13 \\
12 & 11 & 10 & 9 \\
8 & 7 & 6 & 5 \\
4 & 3 & 2 & 1
\end{bmatrix}
\end{align}

The main diagonal in F, and it's transpose, F.T are the same.

F - 5x4
F.T - 4x5

```
F = np.array([[20,19,18,17],
              [16,15,14,13],
              [12,11,10,9],
              [8,7,6,5],
              [4,3,2,1]])

F.T
```

    array([[20, 16, 12,  8,  4],
           [19, 15, 11,  7,  3],
           [18, 14, 10,  6,  2],
           [17, 13,  9,  5,  1]])

# Part 3 - Square Matrices

## 3.1 Find $IG$ (be sure to show your work) 😃

You don't have to do anything crazy complicated here to show your work, just create the G matrix as specified below, and a corresponding 2x2 Identity matrix and then multiply them together to show the result. You don't need to write LaTeX or anything like that (unless you want to).

\begin{align}
G=
\begin{bmatrix}
13 & 14 \\
21 & 12
\end{bmatrix}
\end{align}

```
G = np.array([[13,14],
              [21,12]])

G_identity = np.array([[1,0],
                       [0,1]])

# If the product is equal to G, we have the proper IG
np.matmul(G, G_identity)
```

    array([[13, 14],
           [21, 12]])

## 3.2 Find $|H|$ and then find $|J|$.

\begin{align}
H=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\qquad
J=
\begin{bmatrix}
0 & 1 & 2 \\
7 & 10 & 4 \\
3 & 2 & 0
\end{bmatrix}
\end{align}

```
H = np.array([[12,11],
              [7,10]])

J = np.array([[0,1,2],
              [7,10,4],
              [3,2,0]])
```

```
np.linalg.det(H)
```

    43.000000000000014

```
np.linalg.det(J)
```

    -19.999999999999996

## 3.3 Find $H^{-1}$ and then find $J^{-1}$

$H^{-1}$ = \begin{bmatrix}
10 & -11 \\
-7 & 12
\end{bmatrix}

---

 $J^{-1}$ = No Determinant

## 3.4 Find $HH^{-1}$ and then find $J^{-1}J$. Is $HH^{-1} == J^{-1}J$? Why or Why not?

Please ignore Python rounding errors. If necessary, format your output so that it rounds to 5 significant digits (the fifth decimal place).

I would say no, as $J^{-1}$ doesn't exist

# Stretch Goals:

A reminder that these challenges are optional. If you finish your work quickly we welcome you to work on them. If there are other activities that you feel like will help your understanding of the above topics more, feel free to work on that. Topics from the Stretch Goals sections will never end up on Sprint Challenges. You don't have to do these in order, you don't have to do all of them.

- Write a function that can calculate the dot product of any two vectors of equal length that are passed to it.
- Write a function that can calculate the norm of any vector
- Prove to yourself again that the vectors in 1.9 are orthogonal by graphing them.
- Research how to plot a 3d graph with animations so that you can make the graph rotate (this will be easier in a local notebook than in google colab)
- Create and plot a matrix on a 2d graph.
- Create and plot a matrix on a 3d graph.
- Plot two vectors that are not collinear on a 2d graph. Calculate the determinant of the 2x2 matrix that these vectors form. How does this determinant relate to the graphical interpretation of the vectors?


```python
from IPython.display import Image
from IPython.core.display import HTML
Image(url= "https://i.imgur.com/NLtIf3P.png")
```

```python
# Start with a coordinate system
from sympy.vector import CoordSys3D, Del
delop = Del()
C = CoordSys3D('C')
```

```python
C
```

$\displaystyle CoordSys3D\left(C, \left( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \  \mathbf{\hat{0}}\right)\right)$

```python
delop
```

$\displaystyle Del\left(\right)$

```python
from sympy import symbols, Function
v1, v2, v3, f = symbols('v1 v2 v3 f', cls=Function)
```

```python
# Define the vector field as vfield and the scalar field as sfield.
vfield = v1(C.x, C.y, C.z)*C.i + v2(C.x, C.y, C.z)*C.j + v3(C.x, C.y, C.z)*C.k
ffield = f(C.x, C.y, C.z)
```

```python
vfield
```

$\displaystyle (\operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)})\mathbf{\hat{i}_{C}} + (\operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)})\mathbf{\hat{j}_{C}} + (\operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)})\mathbf{\hat{k}_{C}}$

```python
ffield
```

$\displaystyle f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)}$

```python
# Construct the expression for the LHS of the equation using Del()
lhs = (delop.dot(ffield * vfield)).doit()
```

```python
lhs
```

$\displaystyle f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)}$

```python
# Similarly, the RHS would be defined.
rhs = ((vfield.dot(delop(ffield))) + (ffield * (delop.dot(vfield)))).doit()
```

```python
rhs
```

$\displaystyle \left(\frac{\partial}{\partial \mathbf{{x}_{C}}} \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \frac{\partial}{\partial \mathbf{{y}_{C}}} \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \frac{\partial}{\partial \mathbf{{z}_{C}}} \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)}\right) f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)}$

```python
# Now, to prove the product rule, we would just need to equate the expanded and simplified versions of the lhs and the rhs,
# so that the SymPy expressions match.
lhs.expand().simplify() == rhs.expand().doit().simplify()
```

    True

```python
lhs.expand().simplify()
```

$\displaystyle f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)}$

```python
rhs.expand().doit().simplify()
```

$\displaystyle f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{1}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{x}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{2}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{y}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} + \operatorname{v_{3}}{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)} \frac{\partial}{\partial \mathbf{{z}_{C}}} f{\left(\mathbf{{x}_{C}},\mathbf{{y}_{C}},\mathbf{{z}_{C}} \right)}$

```python
# Thus, the general form of the third product rule mentioned above can be proven using sympy.vector.

# source: https://docs.sympy.org/latest/modules/vector/examples.html
```

Personal_Projects/vector math calculus 2.ipynb


# Bayesian estimation

A core concept in machine learning and related fields

Probablistic views and concepts

Bayes classifiers

## Conditional probability

$p(A \, | \, B)$: the probability of event $A$ given $B$

From basic probability:
$$
\begin{align}
p(A \, | \, B)
&=
\frac{p(A \bigcap B)}{p(B)}
\end{align}
$$
where $p(A \bigcap B)$ is the joint probability, of $A$ and $B$ both happen.

Alternative representation:
$$
\begin{align}
p(A \bigcap B)
&=
p(A, B)
\end{align}
$$

$$
\begin{align}
p(A \, | \, B)
&=
\frac{p(A, B)}{p(B)}
\end{align}
$$

Assume $p(B) > 0$ as otherwise the question is ill-defined.

### Bayes' rule

Apply conditional probability to the numerator again:
$$
\begin{align}
p(A \, | \, B)
&=
\frac{p(A, B)}{p(B)}
\\
&=
\frac{p(B \, | \, A) p(A)}{p(B)}
\end{align}
$$

To help remember, consider symmetry:
$$
\begin{align}
p(A, B)
&=
p(A \, | \, B) p(B)
\\
&=
p(B \, | \, A) p(A)
\end{align}
$$

## Classification

$p(C \, | \, \mathbf{x})$

From an observed data $\mathbf{x}$ we want to predict probability of $C$, the class label.

We take a probabilistic view because real world is often non-deterministic.

### Iris example

$C$: type of flower

$\mathbf{x}$: flower features

Setosa

Versicolor

Virginica

### Banking example

A bank decides whether to make a loan to a customer:
* $
\mathbf{x} =
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix}
$
: customer income $x_1$ and asset $x_2$

* $C$: 0/1 if the customer is unlike/likely to pay back the loan

Make a loan if $P(C = 1 \, | \, \mathbf{x}) > 0.5$ or other threshold value

### Bayes' rule

How to compute $P(C \, | \, \mathbf{x})$, which is unknown?

From conditional probability:
$$
\begin{align}
p\left(C \, | \, \mathbf{x}\right)
&=
\frac{p\left(\mathbf{x} \, | \, C\right) p\left(C\right)}{p\left(\mathbf{x}\right)}
\\
&\propto
p\left(\mathbf{x} \, | \, C\right) p\left(C\right)
\end{align}
$$

* $p(C \, | \, \mathbf{x})$: posterior

the likelihood of $C$ given $\mathbf{x}$

* $p(C)$: prior

how likely $C$ is before observing $\mathbf{x}$

* $p(\mathbf{x} \, | \, C)$: likelihood

how likely $\mathbf{x}$ is if it belongs to $C$

* $p(\mathbf{x})$: marginal/evidence

constant for given $\mathbf{x}$

We can compute $P(C \, | \, \mathbf{x})$ (posterior), if given $p(C)$ (prior) and $p(\mathbf{x} \, | \, C)$ (likelihood)

### Rational decision making

Humans tend to over focus on rare events

For example:

$\mathbf{x}$: get killed

$C$: cause of death
* $C_{car}$: car crash
* $C_{plane}$: airplane crash

Plane crash is much more deadly than car crash:
$
p(\mathbf{x} \, | \, C_{plane}) \gg p(\mathbf{x} \, | \, C_{car})
$

say
$
\begin{align}
p(\mathbf{x} \, | \, C_{plane}) &= 1.0
\\
p(\mathbf{x} \, | \, C_{car}) &= 0.1
\end{align}
$

But plane crash is much rarer than car crash:
$
p(C_{plane}) \ll p(C_{car})
$

say
$
\begin{align}
p(C_{plane}) &= 0.001 \\
p(C_{car}) &= 0.1
\end{align}
$

Multiply together:
$
\begin{align}
p(\mathbf{x} \, | \, C_{plane}) p(C_{plane}) &= 0.001
\\
p(\mathbf{x} \, | \, C_{car}) p(C_{car}) &= 0.01
\end{align}
$

$
\begin{align}
\frac{p(C_{plane} \, | \, \mathbf{x})}{p(C_{car} \, | \, \mathbf{x})}
&=
\frac{p(\mathbf{x} \, | \, C_{plane}) p(C_{plane})}{p(\mathbf{x} \, | \, C_{car}) p(C_{car})}
\end{align}
$

Thus plane travel is actually safter than car travel:
$
p(C_{plane} \, | \, \mathbf{x}) < p(C_{car} \, | \, \mathbf{x})
$

# Breast cancer example

$C$: has cancer

$\overline{C}$: no cancer

$1/0$: positive/negative cancer screening result

$p(C) = 0.01$

$C$:
* $p(1 | C) = 0.8$
* $p(0 | C) = 0.2$

$\overline{C}$:
* $p(1 | \overline{C}) = 0.1$
* $p(0 | \overline{C}) = 0.9$

What is $p(C | 1)$?

https://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/

## Other applications

### Value

$R(C_i\, | \mathbf{x})$: expected value (e.g. utility/loss/risk) for taking class $C_i$ given data $\mathbf{x}$

$R_{ik}$: value for taking class $C_i$ when the actual class is $C_k$

$$
\begin{align}
R(C_i \, | \mathbf{x})
=
\sum_{k} R_{ik} p(C_k \, | \, \mathbf{x})
\end{align}
$$

Goal: select $C_i$ to optimize $R(C_i \, | \, \mathbf{x})$ given $\mathbf{x}$

### Association

$
X \rightarrow Y
$
* $X$: antecedent
* $Y$: consequent

Example: basket analysis for shopping,
$X$ and $Y$ can be sets of item(s).

Support:
$$
p(X, Y)
$$
, the statistical significance of having $X$ and $Y$ together

Confidence:
$$
p(Y | X)
$$
, how likely $Y$ can be predicted from $X$

Lift:
$$
\begin{align}
\frac{p(X, Y)}{p(X)p(Y)}
&=
\frac{p(Y | X)}{p(Y)}
\end{align}
$$
, $> 1$, $< 1$, $=1$, $X$ makes $Y$ more, less, equally likely

## Learning

From training data $\mathbf{X}$ we want to estimate model parameters $\Theta$.

$$
\begin{align}
p\left(\Theta | \mathbf{X}\right)
&=
\frac{p\left(\mathbf{X} | \Theta\right) p\left(\Theta\right)}{p\left(\mathbf{X}\right)}
\\
&\propto
p\left(\mathbf{X} | \Theta\right) p\left(\Theta\right)
\end{align}
$$

* $p(\Theta | \mathbf{X})$: posterior

the likelihood of $\Theta$ given $\mathbf{X}$

* $p(\Theta)$: prior

how likely $\Theta$ is before observing $\mathbf{X}$

* $p(\mathbf{X} | \Theta)$: likelihood

how likely $\mathbf{X}$ is if the model parameters are $\Theta$

* $p(\mathbf{X})$: marginal/evidence

constant for given $\mathbf{X}$

### MAP (maximum a posteriori) estimation
$$
\Theta_{MAP} = argmax_{\Theta} p(\Theta | \mathbf{X})
$$

If we don't have $p(\Theta)$, it can be assumed to be flat
$
p(\Theta) = 1
$
and MAP is equivalent to ML:

### ML (maximum likelihood) estimation
$$
\Theta_{ML} = argmax_{\Theta} p(\mathbf{X} | \Theta)
$$

Under the often iid (identical and independently distributed) assumption:
$$
p(\mathbf{X} | \Theta) = \prod_{t=1}^{N} p(\mathbf{x}^{(t)} | \Theta)
$$
, where $\{\mathbf{x}^{t}\}$ are the individual samples within $\mathbf{X}$.
($\mathbf{X}$ is a matrix and $\{\mathbf{x}^{t}\}$ are its individual columns.)

### Bayes estimator
The expected value of the posterior density:
$$
\begin{align}
\Theta_{Bayes}
&=
E(\Theta | \mathbf{X})
\\
&=
\int \Theta p(\Theta | \mathbf{X}) d\Theta
\end{align}
$$

The best estimate of a random variable/vector is its mean.

## Gaussian example

$$
\begin{align}
p(\mathbf{X} | \Theta)
&=
\frac{1}{(2\pi)^{N/2}\sigma^N} \exp\left(-\frac{\sum_{t=1}^N (\mathbf{x}^{(t)} - \Theta)^2}{2\sigma^2}\right)
\\
p(\Theta)
&=
\frac{1}{\sqrt{2\pi}\sigma_0} \exp\left( -\frac{(\Theta - \mu_0)^2}{2\sigma_0^2} \right)
\end{align}
$$

### ML (maximum likelihood) estimation
$$
\Theta_{ML} = argmax_{\Theta} p(\mathbf{X} | \Theta)
$$

Compute the data $\mathbf{X}$ mean and variance:
$$
\begin{align}
\Theta_{ML} &= \mathbf{m} = \frac{\sum_{t=1}^N \mathbf{x}^{(t)}}{N}
\\
\sigma_{ML}^2 &= s^2 = \frac{\sum_{t=1}^N \left(\mathbf{x} - \mathbf{m} \right)^2}{N}
\end{align}
$$

The $\mathbf{m}$ part holds regardless of $\sigma$ (constant or a variable to be optimized).

### MAP estimation
$$
\begin{align}
\Theta_{MAP}
&=
argmax_{\Theta} p(\Theta | \mathbf{X})
\\
&=
argmax_{\Theta} p(\mathbf{X} | \Theta) p(\Theta)
\end{align}
$$

It can be shown that
$$
\Theta_{MAP} =
\frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2} \mathbf{m}
+
\frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2} \mu_0
$$
, i.e. weighted average of prior mean $\mu_0$ and sample $\mathbf{X}$ mean $\mathbf{m}$, with weights inversely proportional to variances.

Note that if we don't know $p(\Theta)$, we can assume it is a constant distribution $p(\Theta) = 1$, i.e. $\sigma_0 = \infty$.
This will give us $\Theta_{MAP} = \mathbf{m} = \Theta_{ML}$, as expected.

### Bayes estimation

$$
\Theta_{Bayes} = E(\Theta | \mathbf{X}) =
\frac{N/\sigma^2}{N/\sigma^2 + 1/\sigma_0^2} \mathbf{m}
+
\frac{1/\sigma_0^2}{N/\sigma^2 + 1/\sigma_0^2} \mu_0
$$
, i.e. same as $\Theta_{MAP}$.

The math derivations are left as exercise.

# Naive Bayes

Naive Bayes assume the features are independent for the likelihood, i.e. for an $n$-dimensional data vector $\mathbf{x}$
$$
\begin{align}
p(\mathbf{x} | \Theta)
&=
p(\mathbf{x}_1, \mathbf{x}_2, \cdots \mathbf{x}_n | \Theta)
\\
&=
\prod_{k=1}^n p( \mathbf{x}_k | \Theta)
\end{align}
$$

We can generalize from a single data item $\mathbf{x}$ (a vector) to an entire data set $\mathbf{X}$ (a matrix whose columns are data items) by considering columns of $\mathbf{X}$ as features, i.e. $\mathbf{X}_{(k)}$

Put the above into our Bayesian rule:
$$
\begin{align}
p(\Theta | \mathbf{X})
&=
\frac{p(\Theta) p\left(\mathbf{X} | \Theta\right)}{p(\mathbf{X})}
\\
&=
\frac{p(\Theta) \prod_{k=1}^n p\left(\mathbf{X}_{(k)} | \Theta\right)}{p(\mathbf{X})}
\end{align}
$$

The main merit of naive Bayes is that the estimation/computation of individual
$
p\left(\mathbf{X}_{(k)} | \Theta\right)
$
terms is easier/faster than the joint term
$
p\left(\mathbf{X} | \Theta \right)
$
.

This feature independence is just an assumption, but tends to work well in practice.

More details can be found under:
* http://scikit-learn.org/stable/modules/naive_bayes.html
* https://en.wikipedia.org/wiki/Naive_Bayes_classifier

# Code example

scikit learn supports naive Bayes with different math functions for the likelihood, such as Gaussian, Bernoulli, and multinomial.

```python
from sklearn import datasets
import numpy as np

iris = datasets.load_iris()

X = iris.data[:, [2, 3]]
y = iris.target
```

```python
from sklearn.cross_validation import train_test_split

# splitting data into 70% training and 30% test data:
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)
```

```python
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
sc.fit(X_train)
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test)
```

```python
from sklearn.metrics import accuracy_score
```

```python
from sklearn.naive_bayes import GaussianNB

gnb = GaussianNB()

_ = gnb.fit(X_train_std, y_train)
```

```python
y_pred = gnb.predict(X_test_std)

print('Accuracy: %.2f' % accuracy_score(y_test, y_pred))
```

    Accuracy: 0.98


# 15.1. Diving into symbolic computing with SymPy

https://ipython-books.github.io/151-diving-into-symbolic-computing-with-sympy/

### Ref

* Math constant - https://docs.sympy.org/0.7.1/modules/mpmath/functions/constants.html

```python
from sympy import *
init_printing()
```

```python
var('x y')   # use in interactive session
```

```python
x, y = symbols('x y')  # use in script
```

```python
expr1 = (x + 1) ** 2
expr2 = x**2 + 2 * x + 1
```

```python
expr1, expr2
```

```python
expr1 == expr2
```

    False

```python
simplify(expr1 - expr2)
```

```python
simplify(expr2)
```

```python
expr1.subs(x, expr1)
```

```python
expr1.subs(x, pi)
```

```python
v = expr1.subs(x, S(1) / 2)
```

```python
type(v)
```

    sympy.core.numbers.Rational

```python
v.evalf()
```

```python
pi.evalf()               # convert sympy expression to numerical value
```

```python
exp(1).evalf()
```

```python
f = lambdify(x, expr1)   # convert sympy expression to python function
```

```python
import numpy as np
y2 = f(np.linspace(-2., 2., 5))
```

```python
type(y2)
```

    numpy.ndarray

```python
len(y2)
```

```python
type(f)
```

    function

```python
f(-1)
```

### Plot function

```python
%matplotlib inline
var('x y')
y = x**2
plot(y, (x, -4, 4))
```

```python

```


# Analytical Solutions to 2nd-order ODEs

Next, we'll focus on analytical solutions for 2nd-order ODEs, and specifically initial-value problems (IVPs). As before, let's categorize problems based on their solution approach.

## 1. Solution by direct integration

If you have a 2nd-order ODE of this form:
\begin{equation}
\frac{d^2 y}{dx^2} = f(x)
\end{equation}
then you can solve by **direct integration**.

For example, let's say we are trying to solve for the deflection of a cantilever beam $y(x)$ with a force $P$ at the end, where $E$ is the modulus and $I$ is the moment of intertia, and the initial conditions are $y(0)=0$ and $y^{\prime}(0) = 0$:

\begin{align}
\frac{d^2 y}{dx^2} &= \frac{-P (L-x)}{EI} \\
\frac{d}{dx} \left(\frac{dy}{dx}\right) &= \frac{-P}{EI} (L-x) \\
\int d \left(\frac{dy}{dx}\right) &= \frac{-P}{EI} \int (L-x) dx \\
y^{\prime} = \frac{dy}{dx} &= \frac{-P}{EI} \left(Lx - \frac{x^2}{2}\right) + C_1 \\
\int dy &= \int \left( \frac{-P}{EI} \left(Lx - \frac{x^2}{2}\right) + C_1 \right) dx \\
y(x) &= \frac{-P}{EI} \left(\frac{L}{2} x^2 - \frac{1}{6} x^3\right) + C_1 x + C_2
\end{align}
That is our general solution; we can obtain the specific solution by applying our two initial conditions:
\begin{align}
y(0) &= 0 = C_2 \\
y^{\prime}(0) &= 0 = C_1 \\
\therefore y(x) &= \frac{P}{EI} \left( \frac{x^3}{6} - \frac{L x^2}{2} \right)
\end{align}

## 2. Solution by substitution

If we have a 2nd-order ODE of this form:
\begin{equation}
\frac{d^2 y}{dx^2} = f(x, y^{\prime})
\end{equation}
then we can solve by **substitution**, meaning by substituting a new variable for $y^{\prime}$. (Notice that $y$ itself does not show up in the ODE.)

Let's substitute $u$ for $y^{\prime}$ in the above ODE:
\begin{align}
u &= y^{\prime} \\
u^{\prime} &= y^{\prime\prime} \\
\rightarrow u^{\prime} &= f(f, u)
\end{align}
Now we have a 1st-order ODE! Then, we can apply the methods previously discussed to solve this; once we find $u(x)$, we can integrate that once more to get $y(x)$.

### Example: falling object

For example, consider a falling mass where we are solving for the downward distance as a function of time, $y(t)$, that is experiencing the force of gravity downward and a drag force upward. It starts at some reference point so $y(0) = 0$, and has a zero initial downward velocity: $y^{\prime}(0) = 0$. The governing equation is:
\begin{equation}
m \frac{d^2 y}{dt^2} = mg - c \left( \frac{dy}{dt} \right)^2
\end{equation}
where $m$ is the mass, $g$ is acceleration due to gravity, and $c$ is the drag proportionality constant.
We can substitute $V$ for $y^{\prime}$, which gives us a first-order ODE:
\begin{align}
\text{let} \quad \frac{dy}{dt} = V \\
\rightarrow m \frac{dV}{dt} &= mg - c V^2 \\
\end{align}
Then, we can solve this for $V(t)$ using our initial condition for velocity $V(0) = 0$. Once we have that, we can integrate once more:
\begin{equation}
y(t) = \int V(t) dt
\end{equation}
and apply our initial condition for position, $y(0) = 0$, to obtain $y(t)$.

Here is the full process:
\begin{align}
\frac{dV}{dt} &= g - \frac{c}{m} V^2 \\
\frac{dV}{g - \frac{c}{m} V^2} &= dt \\
\frac{m}{c} \int \frac{dV}{a^2 - V^2} &= \int dt = t + \bar{c}, \quad \text{where} \quad a = \sqrt{\frac{mg}{c}} \\
\frac{m}{c} \frac{1}{a} \tanh^{-1} \left(\frac{V}{a}\right) &= t + c_1 \\
V &= a \tanh \left( \frac{a c}{m} t + c_1 \right) \\
\therefore V(t) &= \sqrt{\frac{mg}{c}} \tanh \left(\sqrt{\frac{gc}{m}} t + c_1\right)
\end{align}
Applying the initial condition for velocity, $V(0) = 0$:
\begin{align}
V(0) &= 0 = \sqrt{\frac{mg}{c}} \tanh \left(0 + c_1\right) \\
\therefore c_1 &= 0 \\
V(t) &= \sqrt{\frac{mg}{c}} \tanh \left(\sqrt{\frac{gc}{m}} t\right)
\end{align}
Then, to get $y(t)$, we just need to integrate once more:
\begin{align}
\frac{dy}{dt} = V(t) &= \sqrt{\frac{mg}{c}} \tanh \left(\sqrt{\frac{gc}{m}} t\right) \\
\int dy &= \sqrt{\frac{mg}{c}} \int \tanh \left(\sqrt{\frac{gc}{m}} t\right) dt \\
y(t) &= \sqrt{\frac{mg}{c}} \sqrt{\frac{m}{gc}} \log\left(\cosh\left(\sqrt{\frac{gc}{m}} t\right)\right) + c_2 \\
\rightarrow y(t) &= \frac{m}{c} \log\left(\cosh\left(\sqrt{\frac{gc}{m}} t\right)\right) + c_2
\end{align}
Finally, we can apply the initial condition for position, $y(0) = 0$, to get our solution:
\begin{align}
y(0) &= 0 = \frac{m}{c} \log\left(\cosh\left(0\right)\right) + c_2 = c_2 \\
\rightarrow c_2 &= 0 \\
y(t) &= \frac{m}{c} \log\left(\cosh\left(\sqrt{\frac{gc}{m}} t\right)\right)
\end{align}

### Example: catenary problem

The catenary problem describes the shape of a hanging chain or rope fixed between two points. (It was also a favorite of one of my professors, [Joe Prahl](https://en.wikipedia.org/wiki/Joseph_M._Prahl), and I like to teach it as an example in his honor.) The downward displacement of the hanging string/chain/rope as a function of horizontal position, $y(x)$, is governed by the equation
\begin{equation}
y^{\prime\prime} = \sqrt{1 + (y^{\prime})^2}
\end{equation}

This is actually a **boundary value problem**, with the boundary conditions for the displacement at one side $y(0) = 0$ and that the slope is zero in the middle: $\frac{dy}{dx}\left(\frac{L}{2}\right) = 0$. (Please note that I have skipped the derivation of the governing equation, and left some details out.)

We can solve this via substitution, by letting a new variable $u = y^{\prime}$; then, $u^{\prime} = \frac{du}{dx} = y^{\prime\prime}$. This gives is a first-order ODE, which we can integrate:
\begin{align}
\frac{du}{dx} &= \sqrt{1 + u^2} \\
\int \frac{du}{\sqrt{1 + u^2}} &= \int dx \\
\sinh^{-1}(u) &= x + c_1, \quad \text{where } \sinh(x) = \frac{e^x - e^{-x}}{2} \\
u(x) &= \sinh(x + c_1)
\end{align}

Then, we can integrate once again to get $y(x)$:
\begin{align}
\frac{dy}{dx} &= u(x) = \sinh(x + c_1) \\
\int dy &= \int \sinh(x + c_1) dx = \int \left(\sinh(x)\cosh(c_1) + \cosh(x)\sinh(c_1)\right)dx \\
y(x) &= \cosh(x)\cosh(c_1) + \sinh(x)\sinh(c_1) + c_2 \\
\rightarrow y(x) &= \cosh(x + c_1) + c_2
\end{align}
This is the general solution to the catenary problem, and applies to any boundary conditions.

For our specific case, we can apply the boundary conditions and find the particular solution, though it involves some algebra...:
\begin{align}
y(0) &= 0 = \cosh(c_1) + c_2 \\
\frac{dy}{dx}\left(\frac{L}{2}\right) &= u(0) = \sinh \left(\frac{L}{2} + c_1\right) \\
\rightarrow c_1 &= -\frac{L}{2} \\
0 &= \cosh \left( -\frac{L}{2} \right) + c_2 \\
\rightarrow c_2 &= -\cosh\left( -\frac{L}{2} \right) = -\cosh\left(\frac{L}{2} \right)
\end{align}
So, the overall solution for the catenary problem with the given boundary conditions is
\begin{equation}
y(x) = \cosh \left(x - \frac{L}{2}\right) - \cosh\left( \frac{L}{2} \right)
\end{equation}

Let's see what this looks like:

```matlab
L = 1.0;
x = linspace(0, 1);
y = cosh(x - (L/2.)) - cosh(L/2.);
plot(x, y)
```

Please note that I've made some simplifications in the above work, and skipped the details of how the ODE is derived. In general, the solution for the shape is
\begin{equation}
y(x) = C \cosh \frac{x + c_1}{C} + c_2
\end{equation}
where you would solve for the constants $C$, $c_1$, and $c_2$ using the constraints:
\begin{align}
\int_{x_a}^{x_b} \sqrt{1 + (y^{\prime})^2} dx &= L \\
y(x_a) &= y_a \\
y(x_b) &= y_b \;,
\end{align}
where $L$ is the length of the rope/chain.

You can read more about the catenary problem here (for example): <http://euclid.trentu.ca/aejm/V4N1/Chatterjee.V4N1.pdf>

## 3. Homogeneous 2nd-order ODEs

An important category of 2nd-order ODEs are those that look like
\begin{equation}
y^{\prime\prime} + p(x) y^{\prime} + q(x) y = 0
\end{equation}
"Homogeneous" means that the ODE is unforced; that is, the right-hand side is zero.

Depending on what $p(x)$ and $q(x)$ look like, we have a few different solution approaches:

- constant coefficients: $y^{\prime\prime} + a y^{\prime} + by = 0$
- Euler-Cauchy equations: $x^2 y^{\prime\prime} + axy^{\prime} + by = 0$
- Series solutions

First, let's talk about the characteristics of linear, homogeneous 2nd-order ODEs:

#### Solutions have two parts:
Solutions have two parts: $y(x) = c_1 y_1 + c_2 y_2$, where $y_1$ and $y_2$ are each a basis of the solution.

#### Linearly independent:

The two parts of the solution $y_1$ and $y_2$ are linearly independent.

One way of defining this is that $a_1 y_1 + a_2 y_2 = 0$ only has the trivial solution $a_1=0$ and $a_2=0$.

Another way of thinking about this is that $y_1$ and $y_2$ are linearly *dependent* if one is a multiple of the other, like $y_1 = x$ and $y_2 = 5x$. This *cannot* be solutions to a linear, homogeneous ODE.

#### Both parts satisfy the ODE:

$y_1$ and $y_2$ each satisfy the ODE. Meaning, you can plug each of them into the ODE for $y$ and obtain 0.

However, we need both parts together to fully solve the ODE.

#### Reduction of order:

If $y_1$ is known, we can get $y_2$ by **reduction of order**. Let $y_2 = u y_1$, where $u$ is some unknown function of $x$. Then, put $y_2$ into the ODE $y^{\prime}{\prime} + p(x) y^{\prime} + q(x) y = 0$:
\begin{align}
y_2 &= u y_1 \\
y_2^{\prime} &= u y_1^{\prime} + u^{\prime} y_1 \\
y_2^{\prime\prime} &= 2 u^{\prime} y_1^{\prime} + u^{\prime\prime} y_1 + u y_1^{\prime\prime} \\
\rightarrow u^{\prime\prime} &= - \left[ p(x) + \left(\frac{2 y_1^{\prime}}{y_1}\right) \right] u^{\prime}
\text{or, } u^{\prime\prime} &= - \left( g(x) \right) u^{\prime}
\end{align}
Now, we have an ODE with only $u^{\prime\prime}$, $u^{\prime}$, and some function $g(x)$—so we can solve by substitution! Let $u^{\prime} = v$, and then we have $v^{\prime} = -g(x) v$:
\begin{align}
\frac{dv}{dx} &= - \left( p(x) + \frac{2 y_1^{\prime}}{y_1} \right) v \\
\int \frac{dv}{v} &= - \int \left(p(x) + \frac{2 y_1^{\prime}}{y_1} \right) dx \\
\text{Recall } 2 \frac{d}{dx} \left( \ln y_1 \right) &= 2 \frac{y_1^{\prime}}{y_1} \\
\therefore \int \frac{dv}{v} &= - \int \left(p(x) + 2 \frac{d}{dx} \left( \ln y_1 \right) \right) dx \\
\ln v &= -\int p(x) dx - 2 \ln y_1 \\
\rightarrow v &= \frac{\exp\left( -\int p(x)dx \right)}{y_1^2}
\end{align}

So, the actual solution procedure is then:

  1. Solve for $v$: $v = \frac{\exp\left( -\int p(x)dx \right)}{y_1^2}$
  2. Solve for $u$: $u = \int v dx$
  3. Get $y_2$: $y_2 = u y_1$

Here's an example, where we know one part of the solution $y_1 = e^{-x}$:
\begin{align}
y^{\prime\prime} + 2 y^{\prime} + y &= 0 \\
\text{Step 1:} \quad v = \frac{\exp \left( -\int 2dx \right)}{ \left(e^{-x}\right)^2} = \frac{e^{-2x}}{e^{-2x}} &= 1 \\
\text{Step 2:} \quad u = \int v dx = \int 1 dx &= x \\
\text{Step 3:} \quad y_2 &= x e^{-x}
\end{align}
Then, the general solution to the ODE is $y(x) = c_1 e^{-x} + c_2 x e^{-x}$.

### 3a. Equations with constant coefficents

A common category of 2nd-order homogeneous ODEs are equations with constant coefficients, of the form:
\begin{equation}
y^{\prime\prime} + a y^{\prime} + by = 0
\end{equation}
Note that these are unforced, and the right-hand side is zero.

Solutions to these equations take the form $y(x) = e^{\lambda x}$, and inserting this into the ODE gives us the characteristic equation
\begin{equation}
\lambda^2 + a \lambda + b = 0
\end{equation}
which we can solve to find the solution for given coefficients $a$ and $b$ and initial conditions. Depending on those coefficients and the solution to the characteristic equation, our solution can fall into one of three cases:

* Real roots: $\lambda_1$ and $\lambda_2$. This is an **overdamped** system and the full solution takes the form
\begin{equation}
y(x) = c_1 e^{\lambda_1 x} + c_2 e^{\lambda_2 x}
\end{equation}

* Repeated roots: $\lambda_1 = \lambda_2 = \lambda$. This is a **critically damped** system and the full solution is
\begin{equation}
y(x) = c_1 e^{\lambda x} + c_2 x e^{\lambda x}
\end{equation}
(Where does that second part come from, you might ask? Well, we know that $y_1$ is $e^{\lambda x}$, but the second part cannot also be $e^{\lambda x}$ because those are linearly dependent. So, we use *reduction of order* to find $y_2$, which is $x e^{\lambda x}$.

* Imaginary roots: $\lambda = \frac{-a}{2} \pm \beta i$, where $\beta = \frac{1}{2} \sqrt{4b - a^2}$. This is an **underdamped** system and the solution takes the form
\begin{equation}
y(x) = e^{-ax/2} \left( c_1 \sin \beta x + c_2 \cos \beta x \right)
\end{equation}

Some examples:

1. $y^{\prime\prime} + 3 y^{\prime} + 2y = 0$
\begin{align}
\rightarrow \lambda^2 + 3\lambda + 2 &= 0 \\
(\lambda + 2)(\lambda + 1) &= 0 \\
\lambda &= -2, -1 \\
y(x) &= c_1 e^{-x} + c_2 e^{-2x}
\end{align}
Then, we would use the initial conditions given for $y(0)$ and $y^{\prime}(0)$ to find $c_1$ and $c_2$.

2. $y^{\prime\prime} + 6 y^{\prime} + 9y = 0$
\begin{align}
\rightarrow \lambda^2 + 6\lambda + 9 &= 0 \\
(\lambda + 3)(\lambda + 3) &= 0 \\
\lambda &= -3 \\
y(x) &= c_1 e^{-3x} + c_2 x e^{-3x}
\end{align}

3. $y^{\prime\prime} + 6 y^{\prime} + 25 y = 0$
\begin{align}
\rightarrow \lambda^2 + 6\lambda + 25 &= 0 \\
\lambda &= -3 \pm 4i \\
y(x) &= e^{-3x} \left( c_1 \sin 4x + c_2 \cos 4x \right)
\end{align}

### 3b. Euler-Cauchy equations

Euler-Cauchy equations are of the form
\begin{equation}
x^2 y^{\prime\prime} + axy^{\prime} + by = 0
\end{equation}

Solutions take the form $y = x^m$, which when plugged into the ODE leads to a different characterisic equation to find $m$:
\begin{align}
y &= x^m \\
y^{\prime} &= m x^{m-1} \\
y^{\prime\prime} &= m (m-1) x^{m-2} \\
\rightarrow x^2 m (m-1) x^{m-2} + axmx^{m-1} + bx^m &= 0 \\
m^2 + (a-1)m + b &= 0
\end{align}
This is our new characteristic formula for these problems, and solving for the roots of this equation gives us $m$ and thus our general solution.

Like equations with constant coefficients, we have three solution forms depending on the roots of the characteristic equation:

* Real roots: $y(x) = c_1 x^{m_1} + c_2 x^{m_2}$
* Repeated roots: $y(x) = c_1 x^m + c_2 x^m \ln x$
* Imaginary roots: $m = \alpha \pm \beta i$, and $y(x) = x^{\alpha} \left[c_1 \cos (\beta \ln x) + c_2 \sin (\beta \ln x)\right]$

## 4. Inhomogeneous 2nd-order ODEs

Inhomogeneous, or forced, 2nd-order ODEs with constant coefficients take the form
\begin{equation}
y^{\prime\prime} + a y^{\prime} + by = F(t)
\end{equation}
with initial conditions $y(0) = y_0$ and $y^{\prime}(0) = y_0^{\prime}$. Depending on the form of the forcing function $F(t)$, we can solve with techniques such as

 - the method of undetermined coefficients
 - variation of parameters
 - LaPlace transforms

The solution in general to inhomogeneous ODEs includes two parts:
\begin{equation}
y(t) = y_{\text{H}} + y_{\text{IH}} = c_1 y_1 + c_2 y_2 + y_{\text{IH}} \;,
\end{equation}
where $y_{\text{H}}$ is the solution from the equivalent homogeneous ODE $y^{\prime\prime} + a y^{\prime} + b y = 0$.

The forcing function $F(t)$ may be

 - continuous
 - periodic
 - aperiodic/discontinuous

### 4a. Continuous $F(t)$: method of undetermined coefficients

For continuous forcing functions, we have two solution methods: the method of undetermined coefficients, and variation of parameters.

Generally you'll want to use the method of undetermined coefficients when possible, which depends on if $F(t)$ matches one of a set of functions. In that case, the form of the inhomogeneous solution $y_{\text{IH}}(t)$ follows that of the forcing function $F(t)$, with one or more unknown constants:

| $F(t)$        | $y_{\text{IH}}(t)$ |
| ------------- | :-------:|
| constant      | $K$ |
| $\cos \omega t$ | $K_1 \cos \omega t + K_2 \sin \omega t$ |
| $\sin \omega t$ | $K_1 \cos \omega t + K_2 \sin \omega t$ |
| $e^{-at}$ | $K e^{-at}$   |
| $(A) t$   | $K_0 + K_1 t$ |
| $t^n$     | $K_0 + K_1 t + K_2 t^2 + \ldots + K_n t^n$|

For combinations of these functions, we can combine functions; for example, given
\begin{align}
F(t) &= e^{-at} \cos \omega t \quad \text{or} e^{-at} \sin \omega t \\
y_{\text{IH}} &= K_1 e^{-at} \cos \omega t + K_2 e^{-at} \sin \omega t
\end{align}
(Note how in all the above cases how the inhomogeneous solution follows the functional form of the forcing function; for example, the exponential decay rate $a$ or the sinusoidal frequency $\omega$ match.

The method of undetermined coefficients works by plugging the candidate inhomogeneous solutionn $y_{\text{IH}}$ into the full ODE, and solving for the constants (e.g., $K$)—but **not** from the initial conditions.

For example, let's solve
\begin{equation}
y^{\prime\prime} + 2y^{\prime} + y = e^{-x}
\end{equation}
with initial conditions $y(0) = y^{\prime}(0) = 0$. First, we should find the solution to the homogeneous equation
\begin{equation}
y^{\prime\prime} + 2y^{\prime} + y = 0 \;.
\end{equation}
We can do this by using the associated characteristic formula
\begin{align}
\lambda^2 + 2 \lambda + 1 &= 0 \\
(\lambda + 1)(\lambda + 1) &= 0 \\
\rightarrow y_{\text{H}} &= c_1 e^{-x} + c_2 x e^{-x}
\end{align}

To find the inhomogeneous solution, we would look at the table above to find what matches the forcing function $e^{-x}$. Normally, we'd grab $K e^{-x}$, but that would not be linearly independent from the first part of the homogeneous solution $y_{\text{H}}$. The same is true for $K x e^{-x}$, which is linearly dependent with the second part of $y_{\text{H}}$, but $K x^2 e^{-x}$ works! Then, we just need to find $K$ by plugging this into the ODE:
\begin{align}
y_{\text{IH}} &= K x^2 e^{-x} \\
y^{\prime} &= K e^{-x} (2x - x^2) \\
y^{\prime\prime} &= K e^{-x} (x^2 - 4x + 2) \\
2 K &= 1 \\
\rightarrow K &= \frac{1}{2} \\
y_{\text{IH}} &= \frac{1}{2} x^2 e^{-x}
\end{align}
Thus, the overall general solution is
\begin{equation}
y(x) = c_1 e^{-x} + c_2 x e^{-x} + \frac{1}{2} x^2 e^{-x}
\end{equation}
and we would solve for the integration constants $c_1$ and $c_2$ using the initial conditions.

Important points to remember:

- The constants of the inhomogeneous solution $y_{\text{IH}}$ come from the ODE, **not** the initial conditions.
- Only solve for the integration constants $c_1$ and $c_2$ (part of the homogeneous solution) once you have the full general solution $y = c_1 y_1 + c_2 y_2 + y_{\text{IH}}$.

### 4b. Continuous $F(t)$: variation of parameters

We have the variation of parameters approach to solve for inhomogeneous 2nd-order ODEs that are more general:
\begin{equation}
y^{\prime\prime} + p(x) y^{\prime} + q(x) y = r(x)
\end{equation}
In this case, we can assume a solution $y(x) = y_1 u_1 + y_2 u_2$.

The solution procedure is:

1. Obtain $y_1$ and $y_2$ by solving the homogeneous equation: $y^{\prime\prime} + p(x) y^{\prime} + q(x) y = 0$

2. Solve for $u_1$ and $u_2$:
\begin{align}
u_1 &= - \int \frac{y_2 r(x)}{W} dx + c_1 \\
u_2 &= \int \frac{y_1 r(x)}{W} dx + c_2 \\
W &= \begin{vmatrix}
y_1 & y_2\\ y_1^{\prime} & y_2^{\prime}\\
\end{vmatrix} = y_1 y_2^{\prime} - y_2 y_1^{\prime} \;,
\end{align}
where $W$ is the Wronksian.

3. Then, we have the general solution:
\begin{align}
y &= u_1 y_1 + u_2 y_2 \\
&= \left( -\int \frac{y_2 r(x)}{W} dx + c_1 \right) y_1 + \left( \int \frac{y_1 r(x)}{W} dx + c_2 \right) y_2 \;,
\end{align}
where we solve for $c_1$ and $c_2$ using the two initial conditions.

#### Example 1: variation of parameters

First, let's try the same example we used for the method of undetermined coefficients above:
\begin{equation}
y^{\prime\prime} + 2 y^{\prime} + y = e^{-x}
\end{equation}
We already found the homogeneous solution, so we know that $y_1 = e^{-x}$ and $y_2 = x e^{-x}$.
Next, let's get the Wronksian, and then $u_1$ and $u_2$.
\begin{align}
W &= \begin{vmatrix} y_1 & y_2 \\ y_1^{\prime} & y_2^{\prime} \end{vmatrix} = e^{-x} e^{-x}(1-x) - x e^{-x} (-e^{-x}) = e^{-2x} \\
%
u_1 &= -\int \frac{x e^{-x} e^{-x}}{e^{-2x}} dx + c_1 = -\int x dx + c_1 = -\frac{1}{2} x^2 + c_1 \\
u_2 &= \int \frac{e^{-x} e^{-x}}{e^{-2x}} dx + c_2 = \int dx + c_2 = x + c_2 \\
y(x) &= \left(-\frac{1}{2} x^2 + c_1\right) e^{-x} + (x + c_2) x e^{-x} \\
\end{align}
After simplifying, we obtain the same solution as via the method of undetermined coefficients (but with a bit more work):
\begin{equation}
y(x) = x_1 e^{-x} + c_2 x e^{-x} + \frac{1}{2} x^2 e^{-x}
\end{equation}

#### Example 2: variation of parameters

Now let's try an example that we could *not* solve using the method of undetermined coefficients, with a forcing term that involves hyperbolic cosine (cosh); recall that $\cosh(x) = \frac{e^x + e^{-x}}{2}$.
\begin{equation}
y^{\prime\prime} + 4 y^{\prime} + 4y = \cosh(x)
\end{equation}
First, we need to find the homogeneous solution:
\begin{align}
y^{\prime\prime} + 4 y^{\prime} + 4y &= 0 \\
\lambda^2 + 4 \lambda + 4 &= 0 \\
\rightarrow \lambda &= -2
\end{align}
So our homogeneous solution involves repeated roots:
\begin{equation}
y_H = c_1 e^{-2x} + c_2 x e^{-2x}
\end{equation}
where $y_1 = e^{-2x}$ and $y_2 = x e^{-2x}$.

Then, we need to find $u_1$ and $u_2$, so let's get the Wronksian and then solve
\begin{align}
W &= \begin{vmatrix} y_1 & y_2 \\ y_1^{\prime} & y_2^{\prime} \end{vmatrix} = e^{-2x} (e^{-2x}) (1 - 2x) - x e^{-2x}(-2 e^{-2x}) = e^{-4x} \\
%
u_1 &= - \int \frac{x e^{-2x} \cosh x}{e^{-4x}} dx + c_1 = -\int \frac{x \frac{1}{2}(e^x + e^{-x})}{e^{-2x}} dx + c_1 \\
 &= -\frac{1}{2} \int x (e^{3x} + e^x) dx + c_1 = -\frac{1}{2} \left[ \frac{1}{9} e^{3x}(3x-1) + e^x(x-1) \right] + c_1 \\
u_1 &= -\frac{1}{18} e^{3x}(3x-1) - \frac{1}{2} e^x (x-1) + c_1 \\
%
u_2 &= \int \frac{e^{-2x} \cosh x}{e^{-4x}} dx + c_2 = \frac{1}{2} \int e^{2x}(e^x + e^{-x}) dx + c_2 = \frac{1}{2} \int (e^{3x} + e^x) dx + c_2 \\
u_2 &= \frac{1}{6} e^{3x} + \frac{1}{2} e^x + c_2
\end{align}

Then, when we put these all together, we get the full (complicated) solution:
\begin{equation}
y(x) = \left[ -\frac{1}{18} e^{3x} (3x-1) - \frac{1}{2} e^x (x-1) + c_1 \right] e^{-2x} + \left( \frac{1}{6} e^{3x} + \frac{1}{2} e^x + c_2 \right) x e^{-2x}
\end{equation}


## Classical Mechanics - Week 9

### Last Week:
- We saw how a potential can be used to analyze a system
- Gained experience with plotting and integrating in Python

### This Week:
- We will study harmonic oscillations using packages
- Further develope our analysis skills
- Gain more experience wtih sympy

```python
# Let's import packages, as usual
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
sym.init_printing(use_unicode=True)
```

Let's analyze a spring using sympy. It will have mass $m$, spring constant $k$, angular frequency $\omega_0$, initial position $x_0$, and initial velocity $v_0$.

The motion of this harmonic oscillator is described by the equation:

eq 1.) $m\ddot{x} = -kx$

This can be solved as

eq 2.) $x(t) = A\cos(\omega_0 t - \delta)$, $\qquad \omega_0 = \sqrt{\dfrac{k}{m}}$

Use SymPy below to plot this function. Set $A=2$, $\omega_0 = \pi/2$ and $\delta = \pi/4$.

(Refer back to ***Notebook 7*** if you need to review plotting with SymPy.)

```python
# Plot for equation 2 here
A, omega0, t, delta = sym.symbols('A, omega_0, t, delta')
x=A*sym.cos(omega0*t-delta)
x
```

```python
x1 = sym.simplify(x.subs({A:2, omega0:sym.pi/2, delta:sym.pi/4}))
x1
```

```python
sym.plot(x1,(t,0,10), title='position vs time',xlabel='t',ylabel='x')
```

## Q1.) Calculate analytically the initial conditions, $x_0$ and $v_0$, and the period of the motion for the given constants.  Is your plot consistent with these values?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible answers #######

Initial position $x_0=2\cos(-\pi/4)=\sqrt{2}$.

Initial velocity $v_0=-2(\pi/2)\sin(-\pi/4)=\pi/\sqrt{2}$.

The period is $2\pi/(\pi/2)=4$.

The initial position and the period are easily seen to agree with these values.  The initial velocity is
positive and the initial slope looks like it agrees.

####### Possible answers #######

#### Now let's make plots for underdamped, critically-damped, and overdamped harmonic oscillators.
Below are the general equations for these oscillators:

- Underdamped, $\beta < \omega_0$ :

eq 3.)    $x(t) = A e^{-\beta t}cos(\omega ' t) + B e^{-\beta t}sin(\omega ' t)$ , $\omega ' = \sqrt{\omega_0^2 - \beta^2}$

 ___________________________________

- Critically-damped, $\beta = \omega_0$:

eq 4.)    $x(t) = Ae^{-\beta t} + B t e^{-\beta t}$

 ___________________________________

- Overdamped, $\beta > \omega_0$:

eq 5.)    $x(t) = Ae^{-\left(\beta + \sqrt{\beta^2 - \omega_0^2}\right)t} + Be^{-\left(\beta - \sqrt{\beta^2 - \omega_0^2}\right)t}$

 _______________________

In the cells below use SymPy to create the Position vs Time plots for these three oscillators.

Use $\omega_0=\pi/2$ as before, and then choose an appropriate value of $\beta$ for the three different damped oscillator solutions. Play around with the variables, $A$, $B$, and $\beta$, to see how different values affect the motion and if this agrees with your intuition.

```python
# Put your code for graphing Underdamped here
A, B, omega0, beta, t = sym.symbols('A, B, omega_0, beta, t')
omegap=sym.sqrt(omega0**2-beta**2)
x=sym.exp(-beta*t)*(A*sym.cos(omegap*t)+B*sym.sin(omegap*t))
x
```

```python
x1 = sym.simplify(x.subs({A:0, B:2, omega0:sym.pi/2, beta:sym.pi/40}))
sym.plot(x1,(t,0,30), title='Underdamped Oscillator',xlabel='t',ylabel='x')
```

```python
# Put your code for graphing Critical here
A, B, omega0, beta, t = sym.symbols('A, B, omega_0, beta, t')
x=sym.exp(-beta*t)*(A+B*t)
x
```

```python
x1 = sym.simplify(x.subs({A:0, B:2, omega0:sym.pi/2, beta:sym.pi/2}))
sym.plot(x1,(t,0,30), title='Critically-damped Oscillator',xlabel='t',ylabel='x')
```

```python
# Put your code for graphing Overdamped here
A, B, omega0, beta, t = sym.symbols('A, B, omega_0, beta, t')
beta1=beta+sym.sqrt(beta**2-omega0**2)
beta2=beta-sym.sqrt(beta**2-omega0**2)
x=A*sym.exp(-beta1*t)+B*sym.exp(-beta2*t)
x
```

```python
x1 = sym.simplify(x.subs({A:-2, B:2, omega0:sym.pi/2, beta:sym.pi}))
sym.plot(x1,(t,0,30), title='Overdamped Oscillator',xlabel='t',ylabel='x')
```

## Q2.) How would you compare the 3 different oscillators?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible answers #######

Underdamped: Oscillations, with amplititude decreasing over time

Critical Damping: No oscillations. Curve dies the fastest of the three (for fixed $\omega_0$).

Overdamped: No oscillations. Curve dies slower than critically damped case.

####### Possible answers #######

# Here's another simple harmonic system, the pendulum.

The equation of motion for the pendulum is:

eq 6.) $ml\dfrac{d^2\theta}{dt^2} + mg \sin(\theta) = 0$, where $v=l\dfrac{d\theta}{dt}$ and $a=l\dfrac{d^2\theta}{dt^2}$

In the small angle approximation $\sin\theta\approx\theta$, so this can be written:

eq 7.) $\dfrac{d^2\theta}{dt^2} = -\dfrac{g}{l}\theta$

We then find the period of the pendulum to be $T = \dfrac{2\pi}{\sqrt{l/g}}$ and the angle at any given time
(if released from rest) is given by

$\theta = \theta_0\cos{\left(\sqrt{\dfrac{g}{l}} t\right)}$.

Let's use Euler's Forward method to solve equation (7) for the motion of the pendulum in the small angle approximation, and compare to the analytic solution.

First, let's graph the analytic solution for $\theta$. Go ahead and graph using either sympy, or the other method we have used, utilizing these variables:

- $t:(0s,50s)$
- $\theta_0 = 0.5$ radians
- $l = 40$ meters

```python
# Plot the analytic solution here
l=40
g=9.81

theta0, omega0, t = sym.symbols('theta_0, omega_0, t')
theta = theta0*sym.cos(omega0*t)
theta1 = sym.simplify(theta.subs({omega0:(g/l)**0.5,theta0:0.5}))
sym.plot(theta1,(t,0,50),title='Pendulum Oscillation, Small Angle Approximation',xlabel='time (s)',ylabel='Theta (radians)')
plt.show()

```

```python
# The same analytic plot, but now using matplotlib
# This is easier for comparing with the Euler's method calculation

ti=0
tf=50
dt=0.001
t=np.arange(ti,tf,dt)

theta0=0.5
l=40
g=9.81

omega0=np.sqrt(g/l)

theta=theta0*np.cos(omega0*t)
plt.grid()
plt.xlabel("Time (s)")
plt.ylabel("Theta (radians)")
plt.title("Theta (radians) vs Time (s), small angle approximation")
plt.plot(t,theta)
plt.show()
```

Now, use Euler's Forward method to obtain a plot of $\theta$ as a function of time $t$ (in the small angle approximation).  Use eq (7) to calculate $\ddot{\theta}$ at each time step.
Try varying the time step size to see how it affects the Euler's method solution.

```python
# Perform Euler's Method Here
theta1=np.zeros(len(t))
theta1[0]=theta0
dtheta=0
ddtheta=-omega0**2*theta1[0]

for i in range(len(t)-1):
    theta1[i+1] = theta1[i] + dtheta*dt
    dtheta += ddtheta*dt
    ddtheta = -omega0**2*theta1[i+1]

import matplotlib.patches as mpatches

plt.grid()
plt.xlabel("Time (s)")
plt.ylabel("Theta (radians)")
plt.title("Theta (radians) vs Time (s), small angle approximation")
blue_patch = mpatches.Patch(color = 'b', label = 'analytic')
red_patch = mpatches.Patch(color = 'r', label = 'Euler method')
plt.legend(handles=[blue_patch,red_patch],loc='lower left')
plt.plot(t,theta1,color='r')
plt.plot(t,theta,color='b')
plt.show()
```

You should have found that if you chose the time step size small enough, then the Euler's method solution was
indistinguishable from the analytic solution.

We can now trivially modify this, to solve for the pendulum **exactly**, without using the small angle approximation.
The exact equation for the acceleration is

eq 8.) $\dfrac{d^2\theta}{dt^2} = -\dfrac{g}{l}\sin\theta$.

Modify your Euler's Forward method calculation to use eq (8) to calculate $\ddot{\theta}$ at each time step in the cell below.

```python
theta2=np.zeros(len(t))
theta2[0]=theta0
dtheta=0
ddtheta=-omega0**2*np.sin(theta2[0])

for i in range(len(t)-1):
    theta2[i+1] = theta2[i] + dtheta*dt
    dtheta += ddtheta*dt
    ddtheta = -omega0**2*np.sin(theta2[i+1])

plt.grid()
plt.xlabel("Time (s)")
plt.ylabel("Theta (radians)")
plt.title("Theta (radians) vs Time (s)")
blue_patch = mpatches.Patch(color = 'b', label = 'small angle approximation')
red_patch = mpatches.Patch(color = 'r', label = 'EXACT')
plt.legend(handles=[blue_patch,red_patch],loc='lower left')
plt.plot(t,theta2,color='r')
plt.plot(t,theta1,color='b')
plt.show()
```

# Q3.) What time step size did you use to find agreement between Euler's method and the analytic solution (in the small angle approximation)? How did the exact solution differ from the small angle approximation?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible answers #######

A time step of 0.001 was sufficient so that Euler's method and the analytic formula were indistinguishable in the plots (for small angle approximation).
(Different time steps could be found, depending on how closely one compared the plots.)

The exact pendulum solution has a slightly longer period than the small angle approximation (in agreement with what we learned last week.)

####### Possible answers #######

### Now let's do something fun:

In class we found that the 2-dimensional anisotropic harmonic motion can be solved as

eq 8a.) $x(t) = A_x \cos(\omega_xt)$

eq 8b.) $y(t) = A_y \cos(\omega_yt - \delta)$

If $\dfrac{\omega_x}{\omega_y}$ is a rational number (*i.e,* a ratio of two integers), then the trajectory repeats itself after some amount of time. The plots of $x$ vs $y$ in this case are called Lissajous figures (after the French physicists Jules Lissajous).  If $\dfrac{\omega_x}{\omega_y}$ is not a rational number, then the trajectory does not repeat itself, but it still shows some very interesting behavior.

Let's make some x vs y plots below for the 2-d anisotropic oscillator.

First, recreate the plots in Figure 5.9 of Taylor.  (Hint: Let $A_x=A_y$. For the left plot of Figure 5.9, let $\delta=\pi/4$ and for the right plot, let $\delta=0$.)

Next, try other rational values of $\dfrac{\omega_x}{\omega_y}$ such as 5/6, 19/15, etc, and using different phase angles $\delta$.

Finally, for non-rational $\dfrac{\omega_x}{\omega_y}$, what does the trajectory plot look like if you let the length of time to be arbitrarily long?

\[For these parametric plots, it is preferable to use our original plotting method, *i.e.* using `plt.plot()`, as introduced in ***Notebook 1***.\]

```python
# Plot the Lissajous curves here
Ax=1
Ay=1
omegay=1

ti=0
tf=10
dt=0.1
t=np.arange(ti,tf,dt)

r=2
delta=np.pi/4

omegax=r*omegay

X=Ax*np.cos(omegax*t)
Y=Ay*np.cos(omegay*t-delta)

plt.plot(X,Y)

```

```python
ti=0
tf=24.1
dt=0.1
t=np.arange(ti,tf,dt)

r=np.sqrt(2)
delta=0

omegax=r*omegay

X=Ax*np.cos(omegax*t)
Y=Ay*np.cos(omegay*t-delta)

plt.plot(X,Y)
```

```python
ti=0
tf=50
dt=0.1
t=np.arange(ti,tf,dt)

r=5/6
delta=np.pi/2

omegax=r*omegay

X=Ax*np.cos(omegax*t)
Y=Ay*np.cos(omegay*t-delta)

plt.plot(X,Y)
```

```python
ti=0
tf=100
dt=0.1
t=np.arange(ti,tf,dt)

r=19/15
delta=np.pi/2

omegax=r*omegay

X=Ax*np.cos(omegax*t)
Y=Ay*np.cos(omegay*t-delta)

plt.plot(X,Y)
```

```python
ti=0
tf=1000
dt=0.1
t=np.arange(ti,tf,dt)

r=np.sqrt(2)
delta=0

omegax=r*omegay

X=Ax*np.cos(omegax*t)
Y=Ay*np.cos(omegay*t-delta)

plt.plot(X,Y)
```

# Q4.) What are some observations you make as you play with the variables? What happens for non-rational $\omega_x/\omega_y$ if you let the oscillator run for a long time?

✅ Double click this cell, erase its content, and put your answer to the above question here.

####### Possible answers #######

For rational $\omega_x/\omega_y = n/m$ the curve closes.  In general, the larger $n$ and $m$ (with no common factors) the longer it takes the curve to close.

For non-rational $\omega_x/\omega_y$, the curve essentially fills in the entire rectangle if you let it run to long enough time.

####### Possible answers #######

# Notebook Wrap-up.
Run the cell below and copy-paste your answers into their corresponding cells.

```python
from IPython.display import HTML
HTML(
"""

"""
)
```

# Well that's that, another Notebook! It's now been 10 weeks of class

You've been given lots of computational and programing tools these past few months. These past two weeks have been practicing these tools and hopefully you are understanding how some of these pieces add up. Play around with the code and see how it affects our systems of equations. Solve the Schrodinger Equation for the Helium atom. Figure out the unifying theory. The future is limitless!


```python
import sympy as sp
x,y,z = sp.symbols('x,y,z')
sp.init_printing(use_unicode=False, wrap_line=False, no_global=True)

%run ../display_helpers.py
```

# SymPy - Polynomials

http://docs.sympy.org/latest/modules/polys/index.html

----
## Basics
http://docs.sympy.org/latest/modules/polys/basics.html

----

### Polynomial Division

```python
subhead("Polynomial Division - Rational (Q)")

f = 5*x**2 + 10*x +3
g = 2*x + 2
q,r = sp.div(f, g, domain='QQ')

print_eq(('{} = {},\;r\;{}', f/g, sp.factor(q), r))
```

```python
subhead("Polynomial Division - Integer (Z)")

q,r = sp.div(f, g, domain='ZZ')

print_eq(('{} = {},\;r\;{}', f/g, q, r))
```

```python
subhead("Polynomial Division - Multiple Variables")

f = x*y + y*z
g = 3*x + 3*z
q,r = sp.div(f, g, domain='QQ')

print_eq(('{} = {},\;r\;{}', f/g, q, r))
```

----

### Greatest Common Divisor (GCD) / Lowest Common Multiple (LCM)

```python
subhead("Greatest Common Divisor (GCD)")

f = (12*x + 12) * x
g = 16*x**2
q = sp.gcd(f, g)

print_eq(
    ('gcd \Big(  {}\;,\;{}  \Big) = {}', f, g, q)
)
```

```python
subhead("Greatest Common Divisor (GCD)")
desc('If the coefficients are rational, the polynomial answer is monic')
f = 3 * x **2 /2
g = 9 * x/4
q = sp.gcd(f, g)

print_eq(
    ('gcd \Big(  {}\;,\;{}  \Big) = {}', f, g, q)
)
```

```python
subhead("LCM with GCD")
f = x * y**2 + x**2 * y
g = x**2 * y**2
q = sp.gcd(f, g)
r = sp.lcm(f, g)

print_eq(
    ('gcd \Big( {}\;,\;{} \Big) &= {}', f, g, q),
    ('lcm \Big( {}\;,\;{} \Big) &= {}', f, g, r)
)
```

```python
subhead("LCM with GCD")
f = x * y**2 + x**2 * y
g = x**2 * y**2
q = sp.gcd(f, g)
r = sp.lcm(f, g)

print_eq(
    ('f &= {}', f),
    ('g &= {}', g),
    ('gcd \Big( f\;,\;g \Big) &= {}', q),
    ('lcm \Big( f\;,\;g \Big) &= {}', r),
    ('f.g &= {}', (f*g).expand() )
)
```

----
### Factorization

```python
subhead("Square-Free Factorization (SQF)")
desc('For univariate polynomials')
f = 2 * x**2 + 5 * x**3 + 4 * x**4 + x**5
q = sp.sqf_list(f)
r = sp.sqf(f)

print_eq(
    ('f &= {}', f),
    ('&= {}', r),
)

print('\n\nsqf_list(f) = ', q)
```

```python
subhead("Factorization")
desc('For univariate & multivariate polynomials with rational coefficient')
f = x**4/2 + 5*x**3/12 - x**2/3
q = sp.factor(f)

print_eq(
    ('a)\; f &= {}', f),
    ('&= {}', q),
)

print('\n\n\n')

f = x**2 + 4*x*y + 4*y**2
q = sp.factor(f)
print_eq(
    ('b)\; f &= {}', f),
    ('&= {}', q),
)
```

```python
subhead("Groebner Bases")
desc('Buchberger’s algorithm is implemented, supporting various monomial orders')
f = [x**2 + 1, y**4*x + x**3]
q = sp.groebner( f, x, y, order='lex')

print_eq(
    ('f &= {}', f)
)
print(q)

print('\n\n')

f = [x**2 + 1, y**4*x + x**3, x*y*z**3]
q = sp.groebner( f, x, y, z, order='grevlex')
print_eq(
    ('f &= {}', f)
)
print(q)
```

```python
subhead("Solve Equations")
desc('solve')
f = x**3 + 2*x + 3
q = sp.solve( f, x )

print_eq(
    ('f &= {}', f),
    ('x &= {}', q)
)

print('\n\n\n')
f = x**2 + y*x + z
q = sp.solve( f, x )

print_eq(
    ('f &= {}', f),
    ('x &= {}', q)
)
```

```python
desc('solve poly system')
f = [y-x, x-5]
q = sp.solve_poly_system(f, x, y)

print_eq(
    ('{}', f),
    ('x = {}', q)
)

print('\n\n')

f = [y**2 - x**3 + 1, y*x]
q = sp.solve_poly_system(f, x, y)

print_eq(
    ('{}', f)
)
print_eq(
    ('x = {}', q)
)
```

----

## Examples

http://docs.sympy.org/latest/modules/polys/wester.html

Simple univariate poylnomial factorization
Univariate GCD, resultant and factorization
Multivarite GCD, and factorization
Support for symbols in exponents
Testing if polynomials have common zeros
Normalizing simple rational functions
Expanding expressions and factoring back
Factoring in terms of cuclotomic polynomials
Univarite factoring over Gaussian numbers
Computing with automatic field extensions
Univariate factoring over various domains
Factoring polynomials into linear factors
Advanced factoring over finite fields
Working with Expressions as polynomials
Computing reduced Gröbner bases
Multivariate factoring over algebraic numbers
Partial fraction decomposition

- factor
- primitive
- expand
- resultant
- apart
- cancel
- solve
- groebner
-

----

- Examples
- Polynomial Manipulation
- AGCA (Algebraic Geometry * Cummutative Algebra Module)
- Internals
- Series Maniuplation
- Literature

### Glossary

#### Associative
An expression is associative *if* $(a * b) * c = a * (b * c)$

#### Commutative
An expression is commutative *if* $a * b = b * a$

#### Distributive
An expression is distributive *if* $a \times (b + c) = a \times b + a \times c$

#### Group
Examples - clock/modular arithmetic, symmetries, integer arithmetic
Definition
- Set of elements (generally G)
- Operations (such as + or $\times$) (generall- y *)
- Closed under operation (produces another value in the set) $x, y \in G \implies x*y \in G$
- Inverses $x^{-1}$ exists for all x, and $x.x^{-1} = e$
- Identity x*e = e*x = x
- Associative $(a * b) * c = a * (b * c)$
- G may or may not be commutative $x*y \ne y*x$ (symmetries are not commutative)
    - If G is commutative, then it is a Commutative/Abelian Group
    - Otherwise it is a Noncommutative/Non-Albian Group

#### Identity
The identity of a value and operation will produce the same value. $n + 0 = n$, $n \times 1 = n$

#### Inverses
When an inverse of a value/operation are applied to a value, the result is the identity. $n + (-n) = 0$ and $n.n^{-1} = 1$

#### Monic
A Univratiate polynomial, where the leading coefficient (non-zero) is 1. i.e. $x^3 + 2x^2 - 8x +4 $
- Closed under multiplication (2 monic polys multiplied make another monic

#### Ring
Set of elements that allow + - and $\times$ operations. $\div$ and commutative $\times$ are not necessary. Examples - integers, vectors, matricies, polynomials with integer coefficients.
- Set of elements
- 2 operations + and $\times$ (subtraction can be performed with -ve numbers)
- Addition is commutative
- Multiplication is associative (but not necessarily commutative)
- They're distributive

#### Univariate Polynomial
Single-variable polynomial, i.e. $x^4 + 3x^3 - 7x + 1$


## Variational Inference: Ising Model

This notebook focuses on Variational Inference (VI) for the Ising model in application to binary image de-noising. The Ising model is an example of a Markov Random Field (MRF) and it originated from statistical physics. The Ising model assumes that we have a grid of nodes, where each node can be in one of two states. In the case of binary images, you can think of each node as being a pixel with a black or white color. The state of each node depends on the neighboring nodes through interaction potentials. In the case of images, this translates to a smoothness constraint, i.e. a pixel prefers to be of the same color as the neighboring pixels. In the image denoising problem, we assume that we have a 2-D grid of noisy pixel observations of an underlying true image and we would like to recover the true image. Thus, we can model the image as a grid:

In the figure above, the shaded nodes are the noisy observations $y_i$ of binary latent variables $x_i \in \{-1, +1\}$. We can write down the joint distribution as follows:

\begin{equation}
    p(x,y) = p(x)p(y|x) = \prod_{(s,t)\in E} \Psi_{st}(x_s, x_t) \prod_{i=1}^{n}p(y_i|x_i) = \prod_{(s,t)\in E} \exp \{x_s w_{st} x_t \} \prod_{i=1}^{N} N(y_i|x_i, \sigma^2)
\end{equation}

where the interaction potentials are represented by $\Psi_{st}$ for every pair of nodes $x_s$ and $x_t$ in a set of edges $E$ and the observations $y_i$ are Gaussian with mean $x_i$ and variance $\sigma^2$. Here, $w_{st}$ is the coupling strength and assumed to be constant and equal to $J>0$ indicating a preference for the same state as neighbors (i.e. potential $\Psi(x_s, x_t) = \exp\{x_s J x_t\}$ is higher when $x_s$ and $x_t$ are both either $+1$ or $-1$).

The basic idea behind variational inference is to choose an approximating disribution $q(x)$ which is close to the original distribution $p(x)$ where the distance is measured by KL divergence:

\begin{equation}
    KL(q||p) = \sum_x q(x) \log \frac{q(x)}{p(x)}
\end{equation}

This makes inference into an optimization problem in which the objective is to minimize KL divergence or maximize the Evidence Lower BOund (ELBO). We can derive the ELBO as follows:

\begin{equation}
    \log p(y) = \log \sum_{x} p(x,y) = \log \sum_x \frac{q(x)}{q(x)}p(x,y) = \log E_{q(x)}\big[\frac{p(x,y)}{q(x)} \big] \geq E_{q(x)}\big[\log \frac{p(x,y)}{q(x)} \big] = E_{q(x)}\big[\log p(x,y) \big] - E_{q(x)}\big[\log q(x) \big]
\end{equation}

In application to the Ising model, we have:

\begin{equation}
    \mathrm{ELBO} = E_{q(x)}\big[\log p(x,y) \big] - E_{q(x)}\big[\log q(x) \big] = E_{q(x)}\big[\sum_{(s,t)\in E}x_s w_{st}x_t + \sum_{i=1}^{n} \log N(x_i, \sigma^2) \big] - \sum_{i=1}^{n} E_{q_i(x)}\big[\log q_i(x) \big]
\end{equation}

In *mean-field* variational inference, we assume a *fully-factored* approximation q(x):

\begin{equation}
    q(x) = \prod_{i=1}^{n} q(x_i; \mu_i)
\end{equation}

It can be shown [1] that $q(x_i;\mu_i)$ that minimizes the KL divergence is given by:

\begin{equation}
    q_i(x_i) = \frac{1}{Z_i}\exp \big[E_{-q_i}\{\log p(x) \} \big]
\end{equation}

where $E_{-q_i}$ denotes an expectation over every $q_j$ except for $j=i$. To compute $q_i(x_i)$, we only care about the terms that involve $x_i$, i.e. we can isolate them as follows:

\begin{equation}
    E_{-q_i}\{\log p(x)\} = E_{-q_i}\{x_i \sum_{j\in N(i)} w_{ij}x_j + \log N(x_i,\sigma^2) + \mathrm{const} \} = x_i \sum_{j\in N(i)}J\times \mu_j + \log N(x_i, \sigma^2) + \mathrm{const}
\end{equation}

where $N(i)$ denotes the neighbors of node $i$ and $\mu_j$ is the mean of a binary random variable:

\begin{equation}
    \mu_j = E_{q_j}[x_j] = q_j(x_j=+1)\times (+1) + q_j(x_j=-1)\times (-1)
\end{equation}

In order to compute this mean, we need to know the values of $q_j(x_j=+1)$ and $q_j(x_j=-1)$. Let $m_i = \sum_{j\in N(i)} w_{ij}\mu_j$ be the mean value of neighbors and let $L_{i}^{+} = N(x_i=+1; \sigma^2)$ and $L_{i}^{-} = N(x_i=-1; \sigma^2)$, then we can compute the mean as follows:

\begin{equation}
    q_i(x_i=+1) = \frac{\exp\{m_i + L_{i}^{+}\}}{\exp\{m_i + L_{i}^{+}\} + \exp\{-m_i + L_{i}^{-}\}} = \frac{1}{1+\exp\{-2m_i+L_{i}^{-}-L_{i}^{+}\}} = \frac{1}{1+\exp\{-2 a_i\}} = \sigma(2a_i)
\end{equation}

\begin{equation}
    q_i(x_i=-1) = 1 - q_i(x_i=+1) = 1 - \sigma(2a_i) = \sigma(-2a_i)
\end{equation}

\begin{equation}
    \mu_i = E_{q_i}[x_i] = \sigma(2a_i) - \sigma(-2a_i) = \tanh(a_i)
\end{equation}

where $a_i = m_i + 1/2\big(L_{i}^{+} - L_{i}^{-}\big)$. In other words, our mean-field variational updates of the parameters $\mu_i$ at iteration $k$ are computed as follows:

\begin{equation}
    \mu_{i}^{(k)} = \tanh \bigg(\sum_{j\in N(i)}w_{ij}\mu_{j}^{(k-1)} + \frac{1}{2}\bigg[\log \frac{N(x_i=+1, \sigma^2)}{N(x_i=-1, \sigma^2)} \bigg] \bigg) \times \lambda + (1-\lambda)\times \mu_{i}^{(k-1)}
    \end{equation}

where we added a learning rate parameter $\lambda$. The figure below shows the parametric form of our mean-field approximation of the Ising model:

Now that we derived the variational updates and the ELBO, let's implement this in Python in application to binary image denoising!

```python
%matplotlib inline
import numpy as np
import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt

from PIL import Image
from tqdm import tqdm
from scipy.special import expit as sigmoid
from scipy.stats import multivariate_normal

np.random.seed(0)
sns.set_style('whitegrid')
```

Let's load a grayscale (single channel) image, add Gaussian noise and binarize it based on mean threshold. We can then define variational inference parameters such as the coupling strength, noise level, smoothing rate and max number of iterations:

```python
#load data
print "loading data..."
data = Image.open('./figures/bayes.bmp')
img = np.double(data)
img_mean = np.mean(img)
img_binary = +1*(img>img_mean) + -1*(img 0$, we were able to find the mean parameters for our approximating distribution $q_i(x_i)$ that maximized the ELBO objective and resulted in mostly denoised image. We can visualize the ELBO objective as a function of iterations as follows:

```python
plt.figure()
plt.plot(ELBO, color='b', lw=2.0, label='ELBO')
plt.title('Variational Inference for Ising Model')
plt.xlabel('iterations'); plt.ylabel('ELBO objective')
plt.legend(loc='upper left')
plt.savefig('./figures/ising_vi_elbo.png')
```

Notice that the ELBO is monotonically increasing and flattening out after about 10 iterations. To get further insight into de-noising, we can plot the average entropy $\frac{1}{n}\sum_{i=1}^{n}H_q(x_i)$. We expect early entropy to be high due to random initialization, however, as the number of iterations increases, mean-field updates converge on binary values of $x_i$ that are consistent with observations and the neighbors resulting in a decrease in average entropy:

```python
plt.figure()
plt.plot(Hx_mean, color='b', lw=2.0, label='Avg Entropy')
plt.title('Variational Inference for Ising Model')
plt.xlabel('iterations'); plt.ylabel('average entropy')
plt.legend(loc="upper right")
plt.savefig('./figures/ising_vi_avg_entropy.png')
```

The 2-D Ising model can be extended in multiple ways, for example: 3-D grids and K-states per node (aka Potts model).

### References

[1] K. Murphy, "Machine Learning: A Probabilistic Perspective", The MIT Press, 2012
[2] E. Sudderth, "CS242: Probabilistic Graphical Models", http://cs.brown.edu/courses/cs242/lectures/


```python
import numpy as np
import matplotlib.pyplot as plt
```

Table of content:
- Formal definition (of a perceptron)
- Geometric interpretation (of a perceptron)

Table of content:
- Formal definition
- Geometric interpretation

## Formal definition

For a binary classification problem, a perceptron is formally defined as follows:

For a input feature vector of $I$ entries, $\vec{\phi} = \{x_1, \cdots, x_i, \cdots, x_I\}$, its classification, $y\in\{-1, 1\}$, is given by

$$y=\text{sign}(w_0 + \sum_{i=1}^I w_i \phi_i)$$

where
- $\sum_{i=1}^I w_i \phi_i$ is a linear combination of the input variables.
- $w_0$ is called the bias; $-w_0$ is called the threshold, the minimum value of $\sum_{i=1}^I w_i \phi_i$ for the $\text{sign}$ function to output 1.
-
$
\text{sign}(a) =
\begin{cases}
    -1, & \text{when } a < 0 \\
    +1, & \text{when } a \geq 0
\end{cases}
$

By adding another entry, $x_0=1$, to $\vec{\phi}$, we can write the perceptron in a more convenient vector notation:

$$
\begin{align}
y&=\text{sign}(w_0 + \sum_{i=1}^I w_i \phi_i) \\
&=\text{sign}(\sum_{i=0}^I w_i \phi_i) \\
&=\text{sign}(\vec{w} \cdot \vec{\phi}) \text{, or } \text{sign}(\vec{w}^T \vec{\phi})
\end{align}
$$

By adding another entry, $x_0=1$, to $\vec{\phi}$, we can write the perceptron in a more convenient vector notation (that will help us interpret the perceptron):

$$
\begin{align}
y&=\text{sign}(w_0 + \sum_{i=1}^I w_i \phi_i) \\
&=\text{sign}(\sum_{i=0}^I w_i \phi_i) \\
&=\text{sign}(\vec{w} \cdot \vec{\phi}) \text{, or } \text{sign}(\vec{w}^T \vec{\phi})
\end{align}
$$

## Geometric Interpretation

$$y=\text{sign}(\vec{w} \cdot \vec{\phi})$$

$$y=\text{sign}(\color{red}{\vec{w} \cdot \vec{\phi}})$$

$$y=\color{red}{\text{sign}}(\vec{w} \cdot \vec{\phi})$$

Let's look at the vectorized form of the perceptron. What do you see here? (wait for 3 seconds.) For me, I see two procedures being applied to the input vector $\vec{\phi}$. First, I see $\vec{\phi}$ being projected onto $\vec{w}$ through the dot product. Second, I see that the sign function checks whether that project is positive or negative. So, what's the consequence of these two procedures?

Let's first focus on the interpretation of the first procedure, the dot product, in two dimensions:

## Geometric interpretation of the dot product

```python
w = np.array([[3], [4]]) / 5
```

```python
np.linalg.norm(w)
```

    1.0

Let's create an arbitrary weight vector. I made it length one. Why? Because

When $||\vec{w}||\neq1$,

$$\vec{w}\cdot\vec{\phi} = ||\vec{w}|| ||\vec{\phi}|| \cos(\theta)$$

When $||\vec{w}=1||$,

$$\vec{w}\cdot\vec{\phi} = ||\vec{\phi}|| \cos(\theta)$$

According to the dot product formula above
- When $||\vec{w}||\neq1$, the dot product is the project of $\vec{\phi}$ in the direction of $\vec{w}$ scaled by the norm of $\vec{w}$.
- When $||\vec{w}||=1$, the dot product is the projection of $\vec{\phi}$ in the direction of $\vec{w}$.

Obviously, the second case is easier for interpretation. Note that in practice $||\vec{w}||$ does not neccessarily have a norm of 1.

### Create $\vec{\phi}$s with different norms at different angles from $\vec{w}$, and compute dot products

```python
phi_initial = w
```

We start from the input vector that points in the same direction as $\vec{w}$. For simplicity, I used a copy of $\vec{w}$.

```python
def get_rotation_matrix(angle_in_rad):
    return np.array([
        [np.cos(angle_in_rad), -np.sin(angle_in_rad)],
        [np.sin(angle_in_rad), np.cos(angle_in_rad)]
    ])
```

To obtain $\vec{\phi}$s at different angles from $\vec{w}$, we define a rotation matrix function, which returns a matrix that can rotate any 2D vector by a specified angle.

```python
angles_in_rad = np.arange(0, np.pi * 2, 0.05)
projections = []
for angle_in_rad in angles_in_rad:

    rotation_matrix = get_rotation_matrix(angle_in_rad)
    phi = rotation_matrix @ (phi_initial * float(np.random.uniform(0.1, 1)))

    projection = float(w.T @ phi)
    projections.append(projection)

projections = np.array(projections)
```

```python
angles_in_rad = np.arange(0, np.pi * 2, 0.05)   # creates many angles
projections = []  # creates an accumulator for projections

for angle_in_rad in angles_in_rad:

    # rotate phi_initial by angle_in_rad, and change its norm
    rotation_matrix = get_rotation_matrix(angle_in_rad)
    phi = rotation_matrix @ phi_initial * float(np.random.uniform(0.1, 1))

    # compute dot product
    projection = float(w.T @ phi)

    projections.append(projection)

projections = np.array(projections)  # for plotting
```

### Plot projection of $\vec{\phi}$ onto $\vec{w}$ against the angle from $\vec{\phi}$ to $\vec{w}$

```python
plt.figure(figsize=(7, 7))
plt.polar(angles_in_rad, np.zeros(len(angles_in_rad)), label='The zero line')
plt.polar(angles_in_rad[projections > 0], projections[projections > 0], 'r.', label='Positive projection')
plt.polar(angles_in_rad[projections < 0], projections[projections < 0], 'b.', label='Negative projection')
plt.legend()
plt.show()
```

Now, let's plot the projection of $\vec{\phi}$ onto $\vec{w}$ against the angle from $\vec{\phi}$ to $\vec{w}$.

The circular angle axis around the circle represents the angle between $\vec{\phi}$ and $\vec{w}$. For example, $\vec{w}$ is represented by the zero degree mark since it is zero degree away from itself.

The axis here (move mouse over to the radar-like axis) represents the projection of $\vec{\phi}$ onto $\vec{w}$.

 At ninety and two-seventy degrees, the projection is zero. From immediately after two-seventy degrees to immediately before ninety degrees counterclockwise, the projection is positive. From immediately after ninety degrees to immediately before two-seventy degrees counterclockwise, the projection is negative.

## Geometric interpretation of the sign function

```python
plt.figure(figsize=(7, 7))
plt.polar(angles_in_rad, np.zeros(len(angles_in_rad)), label='The zero line')
plt.polar(angles_in_rad[projections >= 0], np.sign(projections[projections >= 0]), 'r.', label='Positive output')
plt.polar(angles_in_rad[projections < 0], np.sign(projections[projections < 0]), 'b.', label='Negative output')
plt.legend()
plt.show()
```

Now, let's apply the sign function to the projections.

## Summary

```python

```

## Final

The vector $w$ is perpendicular to the decision boundary. When the weight vector is appropriately chose, the +1 -1 assignment of the perceptron will match the label provided by the training data. But how is a perceptron trained? In Episode 2 of the Perceptron series, I will discuss the Perceptron Learning Algorithm (PLA) in detail.

If you liked this video, please hit the like botton down below and subscribe for future videos. Good luck with your study in machine learning.

```python

```

```python

```


```python
from sympy import symbols, Matrix
```

```python
a, b, c, d = symbols('a b c d')
Yt_1, Yt_2, Gt_1 = symbols('Yt_1 Yt_2 Gt_1')
C, I, G, Yt = symbols('C I G Yt')
Yt_1, Yt_2, Gt_1 = symbols('Yt_1 Yt_2 Gt_1')
e, u, v = symbols('e u v')

```

```python
A = Matrix([
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 0],
    [-1, -1, -1, 1]
])
```

```python
B = Matrix([
    [-b, -a, 0, 0],
    [0, -c, c, 0],
    [0, 0, 0, -d],
    [0, 0, 0, 0]
])
```

```python
Y = Matrix([
    [C],
    [I],
    [G],
    [Yt]
])
```

```python
X = Matrix([
    [1],
    [Yt_1],
    [Yt_2],
    [Gt_1]
])
```

```python
U = Matrix([
    [e],
    [u],
    [v],
    [0]
])
```

```python
# Y = M*X + A.inv() * U
```

```python
Y
```

$\displaystyle \left[\begin{matrix}C\\I\\G\\Yt\end{matrix}\right]$

```python
M = -A.inv() * B
M
```

$\displaystyle \left[\begin{matrix}b & a & 0 & 0\\0 & c & - c & 0\\0 & 0 & 0 & d\\b & a + c & - c & d\end{matrix}\right]$

```python
X
```

$\displaystyle \left[\begin{matrix}1\\Yt_{1}\\Yt_{2}\\Gt_{1}\end{matrix}\right]$

```python
A.inv()
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\1 & 1 & 1 & 1\end{matrix}\right]$

```python
U
```

$\displaystyle \left[\begin{matrix}e\\u\\v\\0\end{matrix}\right]$

```python

```

```python

```


# Integer Programming

* All decision variables are integers

\begin{align}
\text{maximize}\  & \mathbf{c}^T\mathbf{x} \\
\text{subject to } & \\
& A\mathbf{x} &&\leq \mathbf{b} \\
& \mathbf{x} &&\geq 0 \\
& \mathbf{x} \in \mathbb{Z}^n
\end{align}

* Binary integer programming: Variables are restricted to be either 0 or 1

# Binary Knapsack Problem
* Combinatorial optimization problem
* Problem of packing the most valuable or useful items without overloading the luggage.
    * A set of items ($N$ items), each with a weight($w$) and a value($v$)
    * Fixed capacity
    * Maximize the total value possible

## Problem Formulation
\begin{align}
\text{maximize}\  & \sum_{i=0}^{N-1}v_{i}x_{i} \\
\text{subject to } & \\
& \sum_{i=0}^{N-1}w_{i}x_{i} & \leq C \\
& x_i \in \{0,1\} & \forall i=0,\dots,N-1
\end{align}

## Coding in Python

## Creating the data (weights and values)

```python
w = [4,2,5,4,5,1,3,5]
v = [10,5,18,12,15,1,2,8]
C = 15
N = len(w)
```

## Step 2: Importing docplex package

```python
from docplex.mp.model import Model
```

## Step 3: Create an optimization model

```python
knapsack_model = Model('knapsack')
```

## Step 4: Add multiple binary decision variables

Adds a list of binary decision variables and stores them in the model.
```python
binary_var_list(keys,                 # sequence of objects / an integer
                lb=None,              # lower bound
                ub=None,              # upper bound
                name=<type 'str'>,    # name
                key_format=None)      # a format string or None for naming the variables
```

```python
x = knapsack_model.binary_var_list(N, name="x")
```

## Step 5: Add the constraints

$$\sum_{i=1}^{N} w_{i}x_{i} \leq C$$

```python
# \sum_{i=1}^{N} w_{i}*x_{i} <= C
knapsack_model.add_constraint(sum(w[i]*x[i] for i in range(N)) <= C)
```

    docplex.mp.LinearConstraint[](4x_0+2x_1+5x_2+4x_3+5x_4+x_5+3x_6+5x_7,LE,15)

## Step 6: Define the objective function

$$\sum_{i=1}^{N} v_{i}x_{i}$$

```python
# \sum_{i=1}^{N} v_{i}*x_{i}
obj_fn = sum(v[i]*x[i] for i in range(N))
knapsack_model.set_objective('max',obj_fn)

knapsack_model.print_information()
```

    Model: knapsack
     - number of variables: 8
       - binary=8, integer=0, continuous=0
     - number of constraints: 1
       - linear=1
     - parameters: defaults
     - objective: maximize
     - problem type is: MILP

## Step 7: Solve the model and output the solution

```python
knapsack_model.solve()
print('Optimization is done. Objective Function Value: %.2f' % knapsack_model.objective_value)
# Get values of the decision variables
knapsack_model.print_solution()
```

    Optimization is done. Objective Function Value: 46.00
    objective: 46
      x_2=1
      x_3=1
      x_4=1
      x_5=1

```python

```