# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
alpha = symbols('alpha')
beta = symbols('beta')
f = Function('f')
t = symbols('t')
```

```python
quad_growth_expression = alpha*f(t) + beta*f(t)**2
```

```python
eq_quad = Eq(dfdt, quad_growth_expression)
```

```python
solution_eq_quad = dsolve(eq_quad)
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


$$
\huge \text{Deriving the Quadratic Formula with SymPy}\\
\large \text{Andrew Ribeiro}\\
\text{December 2017}
$$

```python
import sympy as sp
from IPython.display import display
sp.init_printing(order="lex",use_latex='mathjax')
```

Quadratic equations are of the form:

$$ ax^2 + bx^1 + cx^0 = ax^2 + bx + c = 0$$

Where $a,b,c$ are coeficients. The coeficients can be integers, real numbers, or imaginary numbers. We know from the quadratic formula that such equations can be solved for x by completing the square. In this exercise we will use SymPy to help us derive the quadratic formula. Let's first define some symbols we will use in our symbolic calculations.

```python
a,b,c = sp.symbols("a b c")
z,k = sp.symbols("z k")
x = sp.symbols("x")
```

Now we can easily represent the quadratic equation.

```python
lhs = a*x**2 + b*x + c
rhs = 0
quadraticEqn = sp.Eq(lhs,rhs)
quadraticEqn
```

$$a x^{2} + b x + c = 0$$

## Completing the square

Before we start our derivation, let's talk about completing the square. Say we have binomial rased to a power.

```python
f =(a+b)**2
f
```

$$\left(a + b\right)^{2}$$

If we expand this we get the form:

```python
fExp = f.expand()
fExp
```

$$a^{2} + 2 a b + b^{2}$$

Going from this expanded form back to the binomial is called *factoring.*

```python
fExp.factor()
```

$$\left(a + b\right)^{2}$$

We use SymPy to define a function for completing the square of a symbolic polynomial. This will work for different types of polynomials, but in the case of a quadratic, this works by solving for $z$ and $k$:

$$
ax^2 + bx + c = (x+z)^2 + k \\
ax^2 + bx + c - (x+z)^2 + k = 0
$$

```python
def completeSquare(poly):
    z,k = sp.symbols("z k")
    completedSquareForm = (x+z)**2+k
    sol = sp.solve(poly-completedSquareForm,[z,k])
    squareRes = sp.Pow(x+sol[0][0],2,evaluate=False)
    constantRes = sol[0][1]
    return squareRes + constantRes

```

Consider the following polynomial, which is not a perfect square.

```python
poly1 = x**2 + 10*x + 28
poly1
```

$$x^{2} + 10 x + 28$$

If we try to factor this with SymPy it will throw up its hands and do nothing because it cannot be factored.

```python
poly1.factor()
```

$$x^{2} + 10 x + 28$$

We still can, however, complete the square.

```python
completedSquareForm = (x+b)**2+c
poly1Eqn = sp.Eq(poly1,completedSquareForm)
poly1Eqn
```

$$x^{2} + 10 x + 28 = c + \left(b + x\right)^{2}$$

```python
sol = sp.solve(poly1 - completedSquareForm,[b,c])
sp.Eq(poly1Eqn,completeSquare(poly1))
```

$$x^{2} + 10 x + 28 = c + \left(b + x\right)^{2} = \left(x + 5\right)^{2} + 3$$

Now consider a polynomial which is a perfect square.

```python
poly2 = ((x+5)**2).expand()
poly2
```

$$x^{2} + 10 x + 25$$

Factoring now works, but we can also complete the square with $c=0$

```python
poly2.factor()
```

$$\left(x + 5\right)^{2}$$

```python
completeSquare(poly2)
```

$$\left(x + 5\right)^{2}$$

We will use this function below to help us derive the quadratic formula.

## Using the completion of the square to derive the quadratic formula

```python
quadApart = (quadraticEqn/a).apart(a)
quadApart
```

$$x^{2} + \frac{1}{a} \left(b x + c\right)$$

```python
expanded = quadApart.expand()
lhs = expanded
lhs
```

$$x^{2} + \frac{b x}{a} + \frac{c}{a}$$

Subtract both sides by $\large \frac{c}{a}$

```python
lhs = lhs - c/a
rhs = rhs - c/a
sp.Eq(lhs,rhs)
```

$$x^{2} + \frac{b x}{a} = - \frac{c}{a}$$

We would now like to know what term we must add to both sides of the equation such that we can complete the square of the left hand side so we can isolate x. We know from previous results that it is:

$$
\begin{align}
\large \left( \frac{b}{2a} \right)^2 = \frac{b^2}{4a^2}
\end{align}
$$

But let's derive this using sympy. To do this we will need to solve the following equation for $z$. We can also solve for $k$ to get the completed square, but we will derive this later.

$$
\large x^2+\frac{b}{a}x+z = \left( x+k \right)^2
$$

Subtracting the right hand side from both sides will put this in a form SymPy favors for solving:

$$
\large x^2+\frac{b}{a}x+z - \left( x+k \right)^2 = 0
$$

```python
solvingForZK = sp.solve(lhs+z -(x+k)**2,z,k)
print("Z:")
# Sympy automatically applies the square.
display(solvingForZK[0][0])
print("K:")
display(solvingForZK[0][1])
```

    Z:

$$\frac{b^{2}}{4 a^{2}}$$

    K:

$$\frac{b}{2 a}$$

Thus we see if we'd like to write the left hand side of our equation as a square, we need to add $ \left( \frac{b}{2a} \right)^2$ to both sides of our equation.

```python
completingSquareTerm = sp.Pow((b/(2*a)),2,evaluate=False)
nLhs = lhs + completingSquareTerm
nRhs = rhs + completingSquareTerm
sp.Eq(nLhs,nRhs)
```

$$x^{2} + \left(\frac{b}{2 a}\right)^{2} + \frac{b x}{a} = \left(\frac{b}{2 a}\right)^{2} - \frac{c}{a}$$

We can use the function we defined above to complete the square of the left hand side.

```python
nLhs = completeSquare(nLhs)
nLhs
```

$$\left(x + \frac{b}{2 a}\right)^{2}$$

As we see, $\frac{b}{2a}$ is the $k$ we computed before. We now have:

```python
sp.Eq(nLhs,nRhs)
```

$$\left(x + \frac{b}{2 a}\right)^{2} = \left(\frac{b}{2 a}\right)^{2} - \frac{c}{a}$$

We have finally found a form where we can isolate $x$! The remainder of the derivation is just a simple matter of rearanging terms.

We need to get the right hand side into a form easier for sympy to work with later. This requires a little wizardry with polynomial manipulation module. We could have done this operation on one line, but I will show the steps here.

We first factor.

```python
nRhs = nRhs.factor()
nRhs
```

$$- \frac{1}{4 a^{2}} \left(4 a c - b^{2}\right)$$

As you see this gives us a strange form. We can resolve this by expanding, then bringing the terms together again.

```python
nRhs = nRhs.expand()
nRhs
```

$$- \frac{c}{a} + \frac{b^{2}}{4 a^{2}}$$

```python
nRhs = nRhs.together()
nRhs
```

$$\frac{1}{4 a^{2}} \left(- 4 a c + b^{2}\right)$$

We now square both sides.

Since we did not define our symbol type, sympy will not apply the square root because it does not always hold for all types of numbers that $\sqrt{x^2} = x^2$; however we can force it to make this assumption with a utility function called powdnest.

```python
nLhs = sp.powdenest(sp.sqrt(nLhs),force=True)
nRhs = sp.powdenest(sp.sqrt(nRhs),force=True)
sp.Eq(nLhs,nRhs)
```

$$x + \frac{b}{2 a} = \frac{1}{2 a} \sqrt{- 4 a c + b^{2}}$$

Now we subtract $\frac{b}{2a}$ from both sides.

```python
nLhs = nLhs - b/(2*a)
nRhs = nRhs - b/(2*a)
sp.Eq(nLhs,nRhs)
```

$$x = - \frac{b}{2 a} + \frac{1}{2 a} \sqrt{- 4 a c + b^{2}}$$

We simplify the right hand side and get our familiar quadratic equation.

```python
sp.Eq(x,nRhs.simplify())
```

$$x = \frac{1}{2 a} \left(- b + \sqrt{- 4 a c + b^{2}}\right)$$

As you can see, there is no $\pm$ we are accustom to seeing. There is obviously no plus or minus operator inherent in mathematics. We would have introduced this when we took the square root of both sides because the square of $x$ could either be positive or negative.

Notebooks/Exercise - Deriving the Quadratic Formula with SymPy.ipynb


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
beta = symbols('beta')
```

```python
eq3 = Eq(diff(f(t), t), alpha*f(t) + beta*f(t)**2)
```

```python
solution_eq = dsolve(eq3)
```

```python
general = solution_eq.rhs
```

```python
at_0 = general.subs(t, 0)
```

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

```python
particular = general.subs(C1, value_of_C1)
```

```python
particular = simplify(particular)
```

```python
particular.subs(t, 0)
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


# Part 1 - Scalars and Vectors

For the questions below it is not sufficient to simply provide answer to the questions, but you must solve the problems and show your work using python (the NumPy library will help a lot!) Translate the vectors and matrices into their appropriate python  representations and use numpy or functions that you write yourself to demonstrate the result or property.

```
import numpy as np
import pandas as pd
import math
import matplotlib.pyplot as plt
```

## 1.1 Create a two-dimensional vector and plot it on a graph

```
# vector samples
green = [-.6, .7]
purple = [.3, .2]
orange = [.4, -.4]
```

```
# examine component
orange[0]
```

    0.4

```
# loop
for color in [green, purple, orange]:
  print(color[1])
```

    0.7
    0.2
    0.4

```
# plotting using matplotlib
fig, ax = plt.subplots()
ax.grid()
plt.xlim(-.8, .8)
plt.ylim(-.8, .8)
plt.arrow(0,0,
          green[0],
          green[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='green')
plt.arrow(0,0,
          purple[0],
          purple[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='purple')
plt.arrow(0,0,
          orange[0],
          orange[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='orange');
```

```
print(green, purple, orange)
```

    [-0.6, 0.7] [0.3, 0.2] [0.4, -0.4]

## 1.2 Create a three-dimensional vecor and plot it on a graph

```
# create 3 dimensional vector
green = [.5, .5, .5]
purple = [.4, .3, .2]
orange = [.2, .1, .1]
myvector = np.array([[0,0,0, .5, .5, .5],
                     [0,0,0, .4, .3, .2],
                     [0,0,0, .2, .1, .1]])
myvector[0]
```

    array([0. , 0. , 0. , 0.5, 0.5, 0.5])

```
# unzip vector
X, Y, Z, U, V, W = zip(*myvector)
for letter in [X, Y, Z, U, V, W]:
  print(letter)
```

    (0.0, 0.0, 0.0)
    (0.0, 0.0, 0.0)
    (0.0, 0.0, 0.0)
    (0.5, 0.4, 0.2)
    (0.5, 0.3, 0.1)
    (0.5, 0.2, 0.1)

```
# displaying in 3d
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.quiver(X[0], Y[0], Z[0], U[0], V[0], W[0], length=1, color='green')
ax.quiver(X[1], Y[1], Z[1], U[1], V[1], W[1], length=1, color='purple')
ax.quiver(X[2], Y[2], Z[2], U[2], V[2], W[2], length=1, color='orange')

ax.set_xlim([0, 1])
ax.set_ylim([0, 1])
ax.set_zlim([0, 1])
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
plt.show()
```

## 1.3 Scale the vectors you created in 1.1 by $5$, $\pi$, and $-e$ and plot all four vectors (original + 3 scaled vectors) on a graph. What do you notice about these vectors?

```
from math import e, pi
print(e)
print(pi)
```

    2.718281828459045
    3.141592653589793

```
# scale by 5
pink = np.multiply(5, green)
yellow = np.multiply(5, purple)
red = np.multiply (5, orange)
pink, yellow, red
```

    (array([-3. ,  3.5]), array([1.5, 1. ]), array([ 2., -2.]))

```
# scale by pi
blue = np.multiply(pi, green)
cyan = np.multiply(pi, purple)
brown = np.multiply(pi, orange)
blue, cyan, brown
```

    (array([-1.88495559,  2.19911486]),
     array([0.9424778 , 0.62831853]),
     array([ 1.25663706, -1.25663706]))

```
# scale by e
magenta = np.multiply(e, green)
teal = np.multiply(e, purple)
sienna = np.multiply(e, orange)
magenta, teal, sienna
```

    (array([-1.6309691 ,  1.90279728]),
     array([0.81548455, 0.54365637]),
     array([ 1.08731273, -1.08731273]))

```
# plot on graph
ax.grid()
plt.xlim(-3, 3)
plt.ylim(-3, 3)
plt.arrow(0,0,
          green[0],
          green[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='green')
plt.arrow(0,0,
          purple[0],
          purple[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='purple')
plt.arrow(0,0,
          orange[0],
          orange[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='orange')
plt.arrow(0,0,
          pink[0],
          pink[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='pink')
plt.arrow(0,0,
          yellow[0],
          yellow[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='yellow')
plt.arrow(0,0,
          red[0],
          red[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='red')
plt.arrow(0,0,
          blue[0],
          blue[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='blue')
plt.arrow(0,0,
          cyan[0],
          cyan[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='cyan')
plt.arrow(0,0,
          brown[0],
          brown[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='brown')
plt.arrow(0,0,
          magenta[0],
          magenta[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='magenta')
plt.arrow(0,0,
          teal[0],
          teal[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='teal')
plt.arrow(0,0,
          sienna[0],
          sienna[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='sienna');
```

```
# the vectors follow the same direction of their multlipicand
```

```

```

## 1.4 Graph vectors $\vec{a}$ and $\vec{b}$ and plot them on a graph

\begin{align}
\vec{a} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}
\qquad
\vec{b} = \begin{bmatrix} 3 \\4 \end{bmatrix}
\end{align}

```
a = [5, 7]
b = [3, 4]
```

```
np_a = np.array(a)
np_b = np.array(b)
print(np_a, np_b)
```

    [5 7] [3 4]

```
# graph
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.grid()
plt.xlim(-0,8)
plt.ylim(0,8)

plt.arrow(0,0,
          a[0],
          a[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='gold')
plt.arrow(0,0,
          b[0],
          b[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='grey');
```

## 1.5 find $\vec{a} - \vec{b}$ and plot the result on the same graph as $\vec{a}$ and $\vec{b}$. Is there a relationship between vectors $\vec{a} \thinspace, \vec{b} \thinspace \text{and} \thinspace \vec{a-b}$

```
c = np_a - np_b
c
```

    array([2, 3])

```
# graph
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.grid()
plt.xlim(-0,8)
plt.ylim(0,8)

plt.arrow(0,0,
          a[0],
          a[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='gold')
plt.arrow(0,0,
          b[0],
          b[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='grey')
plt.arrow(0,0,
          c[0],
          c[1],
          head_width=.02,
          head_length=.02,
          linewidth=3,
          color='tan');
```

## 1.6 Find $c \cdot d$

\begin{align}
\vec{c} = \begin{bmatrix}7 & 22 & 4 & 16\end{bmatrix}
\qquad
\vec{d} = \begin{bmatrix}12 & 6 & 2 & 9\end{bmatrix}
\end{align}

```
# define c and d
c = np.array([7, 22, 4, 16])
d = np.array([12, 6, 2 , 9])
c, d
```

    (array([ 7, 22,  4, 16]), array([12,  6,  2,  9]))

```
# dot-product method
np.dot(c,d)
```

    368

##  1.7 Find $e \times f$

\begin{align}
\vec{e} = \begin{bmatrix} 5 \\ 7 \\ 2 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\4 \\ 6 \end{bmatrix}
\end{align}

```
e = np.array([5, 7, 2])
f = np.array([3, 4, 6])
e, f
```

    (array([5, 7, 2]), array([3, 4, 6]))

```
e * f
```

    array([15, 28, 12])

## 1.8 Find $||g||$ and then find $||h||$. Which is longer?

\begin{align}
\vec{g} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 8 \end{bmatrix}
\qquad
\vec{h} = \begin{bmatrix} 3 \\3 \\ 3 \\ 3 \end{bmatrix}
\end{align}

```
g = ([1, 1, 8, 8])
h = ([3, 3, 3, 3])
g, h
```

    ([1, 1, 8, 8], [3, 3, 3, 3])

```
# norm_g is longer
norm_g = np.linalg.norm(g)
norm_h = np.linalg.norm(h)
norm_g, norm_h
```

    (11.40175425099138, 6.0)

# Part 2 - Matrices

## 2.1 What are the dimensions of the following matrices? Which of the following can be multiplied together? See if you can find all of the different legal combinations.
\begin{align}
A = \begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
\qquad
B = \begin{bmatrix}
2 & 4 & 6 \\
\end{bmatrix}
\qquad
C = \begin{bmatrix}
9 & 6 & 3 \\
4 & 7 & 11
\end{bmatrix}
\qquad
D = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\qquad
E = \begin{bmatrix}
1 & 3 \\
5 & 7
\end{bmatrix}
\end{align}

```
# A is 3 x 2
# B is 1 x 3
# C is 2 x 3
# D is 3 x 3
# E is 2 x 2
```

```
# compatible matrices
# A & B
# A & C
# A & D
# E & A
# D & B
# D & C
# C & E
```

## 2.2 Find the following products: CD, AE, and BA. What are the dimensions of the resulting matrices? How does that relate to the dimensions of their factor matrices?

```
# Define Matrices
A = np.array([[1, 2],
     [3, 4],
     [5, 6]])
B = np.array([[2, 4, 6]])
C = np.array([[9, 6, 3],
              [4, 7, 11]])
D = np.array([[1, 0, 0],
              [0, 1, 0],
              [0, 0, 1]])
E = np.array([[1, 3],
              [5, 7]])
A, B, C, D, E
```

    (array([[1, 2],
            [3, 4],
            [5, 6]]), array([[2, 4, 6]]), array([[ 9,  6,  3],
            [ 4,  7, 11]]), array([[1, 0, 0],
            [0, 1, 0],
            [0, 0, 1]]), array([[1, 3],
            [5, 7]]))

```
# CD - 2x3
pd.DataFrame(np.matmul(C, D))
```

      0
      1
      2

      0
      9
      6
      3

      1
      4
      7
      11

```
# AE - 3x2
pd.DataFrame(np.matmul(A, E))
```

      0
      1

      0
      11
      17

      1
      23
      37

      2
      35
      57

```
# BA - 1x2
pd.DataFrame(np.matmul(B, A))
```

      0
      1

      0
      44
      56

```
# the resulting matrices usually takes on the shape of the smaller factor
```

## 2.3  Find $F^{T}$. How are the numbers along the main diagonal (top left to bottom right) of the original matrix and its transpose related? What are the dimensions of $F$? What are the dimensions of $F^{T}$?

\begin{align}
F =
\begin{bmatrix}
20 & 19 & 18 & 17 \\
16 & 15 & 14 & 13 \\
12 & 11 & 10 & 9 \\
8 & 7 & 6 & 5 \\
4 & 3 & 2 & 1
\end{bmatrix}
\end{align}

```
F = np.array([[20, 19, 19, 17],
              [16, 15, 14, 13],
              [12, 11, 10, 9],
              [8, 7, 6, 5],
              [4, 3, 2, 1]])
F
```

    array([[20, 19, 19, 17],
           [16, 15, 14, 13],
           [12, 11, 10,  9],
           [ 8,  7,  6,  5],
           [ 4,  3,  2,  1]])

```
pd.DataFrame(F.T)
# the dimension changed from 5x4 to 4x5, thus, the first 4 diagonal
# numbers remained
```

      0
      1
      2
      3
      4

      0
      20
      16
      12
      8
      4

      1
      19
      15
      11
      7
      3

      2
      19
      14
      10
      6
      2

      3
      17
      13
      9
      5
      1

# Part 3 - Square Matrices

## 3.1 Find $IG$ (be sure to show your work) 😃

You don't have to do anything crazy complicated here to show your work, just create the G matrix as specified below, and a corresponding 2x2 Identity matrix and then multiply them together to show the result. You don't need to write LaTeX or anything like that (unless you want to).

\begin{align}
G=
\begin{bmatrix}
13 & 14 \\
21 & 12
\end{bmatrix}
\end{align}

```
# G Matrix
G = np.array([[13, 14],
              [21, 12]])
pd.DataFrame(G)
```

      0
      1

      0
      13
      14

      1
      21
      12

```
# Identity matrix
I = np.array ([[1, 0],
               [0, 1]])
pd.DataFrame(I)
```

      0
      1

      0
      1
      0

      1
      0
      1

```
# multipy
pd.DataFrame(np.matmul(I, G))
```

      0
      1

      0
      13
      14

      1
      21
      12

## 3.2 Find $|H|$ and then find $|J|$.

\begin{align}
H=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\qquad
J=
\begin{bmatrix}
0 & 1 & 2 \\
7 & 10 & 4 \\
3 & 2 & 0
\end{bmatrix}
\end{align}

```
# H & J
H = np.array([[12, 11],
              [7,10]])
J = np.array([[0, 1, 2],
      [7, 10, 4],
      [3, 2, 0]])
```

```
pd.DataFrame(H)
```

      0
      1

      0
      12
      11

      1
      7
      10

```
pd.DataFrame(J)
```

      0
      1
      2

      0
      0
      1
      2

      1
      7
      10
      4

      2
      3
      2
      0

## 3.3 Find $H^{-1}$ and then find $J^{-1}$

```
# H inverse
pd.DataFrame(np.linalg.inv(H))
```

      0
      1

      0
      0.232558
      -0.255814

      1
      -0.162791
      0.279070

```
# J inverse
pd.DataFrame(np.linalg.inv(J))
```

      0
      1
      2

      0
      0.4
      -0.20
      0.80

      1
      -0.6
      0.30
      -0.70

      2
      0.8
      -0.15
      0.35

## 3.4 Find $HH^{-1}$ and then find $J^{-1}J$. Is $HH^{-1} == J^{-1}J$? Why or Why not?

Please ignore Python rounding errors. If necessary, format your output so that it rounds to 5 significant digits (the fifth decimal place).

```
HH1 = np.matmul(H, np.linalg.inv(H))
pd.DataFrame(HH1)
```

      0
      1

      0
      1.000000e+00
      5.551115e-16

      1
      2.220446e-16
      1.000000e+00

```
JJ1 = np.matmul(J, np.linalg.inv(J))
pd.DataFrame(JJ1)
```

      0
      1
      2

      0
      1.0
      -5.551115e-17
      0.0

      1
      0.0
      1.000000e+00
      0.0

      2
      0.0
      0.000000e+00
      1.0

```
np.round(HH1, decimals=5)
```

    array([[1., 0.],
           [0., 1.]])

```
np.round(JJ1, decimals=5)
```

    array([[ 1., -0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

# Stretch Goals:

A reminder that these challenges are optional. If you finish your work quickly we welcome you to work on them. If there are other activities that you feel like will help your understanding of the above topics more, feel free to work on that. Topics from the Stretch Goals sections will never end up on Sprint Challenges. You don't have to do these in order, you don't have to do all of them.

- Write a function that can calculate the dot product of any two vectors of equal length that are passed to it.
- Write a function that can calculate the norm of any vector
- Prove to yourself again that the vectors in 1.9 are orthogonal by graphing them.
- Research how to plot a 3d graph with animations so that you can make the graph rotate (this will be easier in a local notebook than in google colab)
- Create and plot a matrix on a 2d graph.
- Create and plot a matrix on a 3d graph.
- Plot two vectors that are not collinear on a 2d graph. Calculate the determinant of the 2x2 matrix that these vectors form. How does this determinant relate to the graphical interpretation of the vectors?


# Linear Regression Derived

#### Cost function

$ C = \sum_i (y_i - mx_i - c)^2 $

#### Calculate the partial derivative with respect to $m$

$
\begin{align}
\frac{\partial C}{\partial m} &= \sum_i 2(y_i - m x_i -c)(-x_i) \\
&= -2 \sum_i x_i (y_i - m x_i -c) \\
\end{align}
$

#### Set derivative to zero

$
\begin{align}
& \frac{\partial C}{\partial m} = 0 \\
\Rightarrow & -2 \sum_i x_i (y_i - m x_i -c) = 0 \\
\Rightarrow & \sum_i ( x_i y_i - m x_i x_i - x_i c ) = 0 \\
\Rightarrow & \sum_i x_i y_i - \sum_i m x_i x_i - \sum_i x_i c = 0 \\
\Rightarrow & \sum_i x_i y_i - c \sum_i x_i = m \sum_i x_i x_i \\
\Rightarrow & m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i}
\end{align}
$

#### Calculate the partial derivative with respect to $c$

$
\begin{align}
\frac{\partial C}{\partial c} &= \sum_i 2(y_i - m x_i -c)(-1) \\
&= -2 \sum_i (y_i - m x_i -c) \\
\end{align}
$

#### Set the derivative to zero

$
\begin{align}
& \frac{\partial C}{\partial c} = 0 \\
\Rightarrow & -2 \sum_i (y_i - m x_i - c) = 0 \\
\Rightarrow & \sum_i (y_i - m x_i - c) = 0 \\
\Rightarrow & \sum_i y_i - \sum_i m x_i - \sum_i c = 0 \\
\Rightarrow & \sum_i y_i - m \sum_i x_i - c \sum_i 1 = 0 \\
\Rightarrow & \sum_i y_i - m \sum_i x_i = c \sum_i 1 \\
\Rightarrow & c = \frac{\sum_i y_i - m \sum_i x_i}{\sum_i 1} \\
\Rightarrow & c = \frac{\sum_i y_i}{\sum_i 1} - m \frac{\sum_i x_i}{\sum_i 1} \\
\Rightarrow & c = \bar{y} - m \bar{x} \\
\end{align}
$

#### Combine the estimates

$
\begin{align}
& m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i} \\
& c = \bar{y} - m \bar{x} \\
& \Rightarrow m = \frac{\sum_i x_i y_i - (\bar{y} - m \bar{x}) \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i  + m \bar{x} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i}  + m \frac{\bar{x} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m - m \frac{\bar{x} \sum_i x_i}{\sum_i x_i x_i} = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m(1 - \frac{\bar{x} \sum_i x_i}{\sum_i x_i x_i}) = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m(\frac{\sum_i x_i x_i - \bar{x} \sum_i x_i}{\sum_i x_i x_i}) = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m(\sum_i x_i x_i - \bar{x} \sum_i x_i) = \sum_i x_i y_i - \bar{y} \sum_i x_i \\
& \Rightarrow m = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i - \bar{x} \sum_i x_i} \\
\end{align}
$

#### End


# Assignment: Traffic Flow

---

Consider the flow of traffic on a one-lane highway.
How can we describe the general behavior of this system?

We can define the speed of traffic, $V$, in kilometers per hour.
We can also define a traffic density, the number of cars per unit length of highway, $\rho$, in cars per kilometer.
Finally, we can define a traffic flux, or the flow rate of cars, $F$,in cars per hour.

Now make a few assumptions about how traffic moves.

If $\rho$ approaches $0$, i.e. there are very few cars on the road, then cars will drive as fast as they can at some $V_\text{max}$.

Conversely, if cars are bumper to bumper along the road, then $\rho$ approaches a $\rho_\text{max}$ and $V$ approaches $0$.

One possible equation to describe this behavior is

$$
\begin{equation}
    V = V_\text{max} \left( 1 - \frac{\rho}{\rho_\text{max}} \right)
\end{equation}
$$

If we describe the traffic flux, $F$ as $F = \rho V$, then

$$
\begin{equation}
    F = F \left( \rho \right) = V \left( \rho \right) \rho = V_\text{max} \rho \left( 1 - \frac{\rho}{\rho_{max}} \right)
\end{equation}
$$

Unsteady traffic flow can be modeled as the non-linear convection of car density, so we apply the non-linear convection equation with $V$, the wave speed, equal to $\frac{\partial F}{\partial \rho}$:

$$
\begin{equation}
    \frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial \rho} \frac{\partial \rho}{\partial x} = 0
\end{equation}
$$

Applying the chain rule of calculus, this becomes

$$
\begin{equation}
    \frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0
\end{equation}
$$

where $F$ is defined as above.
The PDE above is written in conservation form.
(See section at the end of Notebook for more details about the conservation form.)

You will integrate the traffic flow equation (in conservation form) using the modified Euler's method (also called midpoint method or Runge-Kutta second-order method).
You will discretize the first-order spatial derivative with a backward-difference quotient:

$$
\begin{equation}
    \left. \frac{\partial F}{\partial x} \right|_i = \frac{F_i - F_{i-1}}{\Delta x} = \frac{F(\rho_i) - F(\rho_{i-1})}{\Delta x}
\end{equation}
$$

Examine a stretch of road using the following conditions:

* $V_\text{max} = 90 \; \text{km/h}$
* $L = 25 \; \text{km}$
* $\rho_\text{max} = 100 \; \text{cars/km}$
* $nx = 101$ (number of points to discretize the road of length $L$)
* $\Delta t = 0.001 \; \text{hours}$

## Implement your solution (40 points)

---

Implement your solution in this section.
You can use as many code cells as you want.

```python
# YOUR CODE HERE
import numpy
import sympy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
sympy.init_printing()
```

```python
# Set parameters.
nx = 101
L = 25.0
dx = L / (nx - 1)
dt = 0.001
Vmax = 90.0
𝜌max = 100
```

```python
# Get the grid point coordinates.
x = numpy.linspace(0.0, L, num=nx)
```

## Assessment (60 points)

---

Answer questions in this section.

Do not try to delete or modify empty code cells that are already present.
For each question, provide your answer in the cell **just above** the empty cell.
(This empty cell contains hidden tests to assert the correctness of your answer and cannot be deleted.)
Pay attention to the name of the variables we ask you to create to store computed values; if the name of the variable is misspelled, the test will fail.

```python
#######################################
# IMPORTANT
#
# Import mooc38 if you use Python 3.8
# It will not work for other versions
#
#######################################

# Import module to check your answers.
import mooc38 as mooc
```

### Part A

Use the following initial condition

$$
\begin{equation}
    \rho_0 = \rho \left(x, t = 0 \right) =
    \begin{cases}
        50 \quad \text{if} \; 2 \leq x \leq 4.2 \\
        10 \quad \text{otherwise}
    \end{cases}
    ,\quad \forall x \in \left[ 0, L \right]
\end{equation}
$$

and the following boundary condition

$$
\begin{equation}
    \rho \left( x=0, t \right) = \rho \left( x=L, t \right) = 10, \quad \forall t
\end{equation}
$$

to answer questions Q1, Q2, Q3, and Q4.

```python
# YOUR CODE HERE
# Set the initial conditions.𝜌
u0 = numpy.ones(nx)
for i in range(len(u0)):
    if u0[i] == 1:
        u0[i] = 10
mask = numpy.where(numpy.logical_and(x >= 2.0, x <= 4.2))
u0[mask] = 50.0
```

```python
# Plot the initial conditions.
pyplot.figure(figsize=(4.0, 4.0))
pyplot.title('Initial conditions')
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, color='C0', linestyle='--', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 55);
```

```python
# get the non-linear solution
nt = int(1/15/dt)
u = u0.copy()
for n in range(1, nt):
    un = u.copy()
    for i in range(1, nx):
        u[i] = un[i] - dt/dx * (un[i]*Vmax*(1-un[i]/𝜌max) - un[i-1]*Vmax*(1-un[i-1]/𝜌max))
```

```python
# Plot the solution after nt time steps
    # along with the initial conditions.
pyplot.figure(figsize=(4.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, label='Initial',
                color='C0', linestyle='--', linewidth=2)
pyplot.plot(x, u, label='nt = {}'.format(nt),
                color='C1', linestyle='-', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 55);
```

```python

```

* **Q1 (5 points):** What's the minimum initial velocity in meters per second?

Store your result in the variable `v0_min`; you can check your answer by calling the function `mooc.check('hw2_answer1', v0_min)`.

```python
# YOUR CODE HERE
a = max(u0)
v0_min = Vmax * (1 - a / 𝜌max) * 1000 / 3600
print(v0_min)
mooc.check('hw2_answer1', v0_min)
```

    12.5
    [hw2_answer1] Good job!

```python

```

* **Q2 (10 points):** What's the average velocity, in meters per second, along the road after $4$ minutes?

Store your result in the variable `v4_mean`; you can check your answer with the function `mooc.check('hw2_answer2', v4_mean)`.

```python
# YOUR CODE HERE
b = 0
for i in u:
    b = b + i
𝜌4 = b / len(u)
v4_mean = Vmax*(1-𝜌4/𝜌max)*1000/3600
print(v4_mean)
mooc.check('hw2_answer2', v4_mean)
```

    21.60891089108911
    [hw2_answer2] Good job!

```python

```

* **Q3 (10 points):** What's the minimum velocity, in meters per second, after $8$ minutes?

Store your result in the variable `v8_min`; you can check your answer with the function `mooc.check('hw2_answer3', v8_min)`.

```python
# YOUR CODE HERE
nx = 101
L = 25.0
dx = L / (nx - 1)
dt = 0.001
Vmax = 90.0
𝜌max = 100
nt = int(8/60/dt)

x = numpy.linspace(0.0, L, num=nx)

u0 = numpy.ones(nx)
for i in range(len(u0)):
    if u0[i] == 1:
        u0[i] = 10
mask = numpy.where(numpy.logical_and(x >= 2.0, x <= 4.2))
u0[mask] = 50.0
u0[0] = 10
u0[-1] = 10

u = u0.copy()
for n in range(1, nt):
    un = u.copy()
    for i in range(1, nx):
        u[i] = un[i] - dt/dx * (un[i]*Vmax*(1-un[i]/𝜌max) - un[i-1]*Vmax*(1-un[i-1]/𝜌max))

c = max(u)
print(c)
v8_min = Vmax * (1 - c / 𝜌max) * 1000 / 3600
print(v8_min)
mooc.check('hw2_answer3', v8_min)
```

    31.807381516040245
    17.04815462098994
    [hw2_answer3] Try again!

```python

```

* **Q4 (5 points):** What's the maximum car density, in cars per kilometer, along the road after 8 minutes?

Store you result in the variable `rho8_max`; you can check your answer with `mooc.check('hw2_answer4', rho8_max)`.

```python
# YOUR CODE HERE

```

```python

```

### Part B

Now, set $V_\text{max} = 130 \; \text{km/hr}$ and redo the simulations using the following initial condition for the traffic density

$$
\begin{equation}
    \rho_0 = \rho \left(x, t = 0 \right) =
    \begin{cases}
        50 \quad \text{if} \; 2 \leq x \leq 4.2 \\
        20 \quad \text{otherwise}
    \end{cases}
    ,\quad \forall x \in \left[ 0, L \right]
\end{equation}
$$

and the following boundary condition

$$
\begin{equation}
    \rho \left( x=0, t \right) = \rho \left( x=L, t \right) = 20, \quad \forall t
\end{equation}
$$

Redo the simulations to answer questions Q5, Q6, Q7, and Q8.

```python
# YOUR CODE HERE
nx = 101
L = 25.0
dx = L / (nx - 1)
dt = 0.001
Vmax = 90.0
𝜌max = 100

x = numpy.linspace(0.0, L, num=nx)
```

* **Q5 (5 points):** What's the minimum initial velocity in meters per second?

Store your result in the variable `v0_min2`; you can check your answer by calling the function `mooc.check('hw2_answer5', v0_min2)`.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python

```

* **Q6 (10 points):** What's the average velocity, in meters per second, along the road after $4$ minutes?

Store your result in the variable `v4_mean2`; you can check your answer with the function `mooc.check('hw2_answer6', v4_mean2)`.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python

```

* **Q7 (10 points):** What's the minimum velocity, in meters per second, after $8$ minutes?

Store your result in the variable `v8_min2`; you can check your answer with the function `mooc.check('hw2_answer7', v8_min2)`.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python

```

* **Q8 (5 points):** What's the average car density, in cars per kilometer, along the road after 8 minutes?

Store your result in the variable `rho8_mean2`; you can check your answer with the function `mooc.check('hw2_answer8', rho8_mean2)`.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python

```

## Conservation and non-conservation forms

---

The traffic flow is modeled by the following equation

$$
\begin{equation}
    \frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0
\end{equation}
$$

where $\rho$ is the density and $F$ is the flux of density given by

$$
\begin{equation*}
    F = F \left( \rho \right) = \rho V \left( \rho \right) = \rho V_\text{max} \left( 1 - \frac{\rho}{\rho_\text{max}} \right)
\end{equation*}
$$

The equation above is called the **conservation form** of the traffic-flow equation.

Alternatively, we could also write another equation by deriving the flux with respect to $x$:

$$
\begin{eqnarray*}
    \frac{\partial \rho}{\partial t} &=& -\frac{\partial F}{\partial x} \\
    &=& -\frac{\partial F}{\partial \rho} \frac{\partial \rho}{\partial x} \\
    &=& - V_\text{max} \left( 1 - 2 \frac{\rho}{\rho_\text{max}} \right) \frac{\partial \rho}{\partial x}
\end{eqnarray*}
$$

i.e.,

$$
\begin{equation}
    \frac{\partial \rho}{\partial t} + V_\text{max} \left( 1 - 2 \frac{\rho}{\rho_\text{max}} \right) \frac{\partial \rho}{\partial x} = 0
\end{equation}
$$

The equation above is call the **non-conservation form** of the traffic-flow equation.

Although the two equations are mathematically identical, their respective discrete version is different and will not lead to the same numerical solution.

Suppose, we want to advance the solution in time using a first-order Euler's method while computing the space derivative using a backward-difference technique.

The discrete version of the equation in conservation form is

$$
\begin{equation}
    \frac{\rho_i^{n + 1} - \rho_i^n}{\Delta t} + \frac{F_i^n - F_{i-1}^n}{\Delta x} = 0
\end{equation}
$$

which leads to

$$
\begin{equation}
    \frac{\rho_i^{n + 1} - \rho_i^n}{\Delta t} + \frac{\rho_i^n V_\text{max} \left( 1 - \frac{\rho_i^n}{\rho_\text{max}} \right) - \rho_{i - 1}^n V_\text{max} \left( 1 - \frac{\rho_{i - 1}^n}{\rho_\text{max}} \right)}{\Delta x} = 0
\end{equation}
$$

The discrete version of the equation in non-conservation form is

$$
\begin{equation}
    \frac{\rho_i^{n + 1} - \rho_i^n}{\Delta t} + V_\text{max} \left( 1 - 2 \frac{\rho_i^n}{\rho_\text{max}} \right) \frac{\rho_i^n - \rho_{i - 1}^n}{\Delta x} = 0
\end{equation}
$$

You can see that the two discrete equations are different, and thus will not lead to the same numerical solution.

For the second assignment of this class, you should discretize the traffic-flow equation in its conservation form.


We are answering questions in <cite data-cite="bibtex_lane2019online">(Lane, 2019)</cite>

The first question we answer is on how to find the smallest absolute difference for the set of numbers $S=\left\{2,3,4,9,16\right\}$

```python
s=[2,3,4,9,16]
result=[]
for i in range(10,1,-1):
    sum=0
    for j in s:
        sum += abs(j-i)
    result.append((i,sum))
result.sort(key=lambda x: x[1])
print("The number that gives the smallest absolute difference is %d.  The sum of absolute differences is %d."
      % (result[0][0],result[0][1]))
```

    The number that gives the smallest absolute difference is 4.  The sum of absolute differences is 20.

We can generalize the logic in the cell above into a that operates on a list of numbers.  Let us make the convention that we enumerate the elements $s_j \in S$ starting with $0$, so if $S$ has three elements then $S=\left\{s_0,s_1,s_2\right\}.$  We know the number we should need to subtract from the elements of $s_j \in S$ should be
\begin{equation}
\underset{i} {\mathrm{argmin}} \sum_{j=0}^{\left|S\right|-1} \left|s_j-i\right|.
\label{equ:argmin-1}
\end{equation}

We do not claim this range to search is optimal, but we show that we can choose a range that is guaranteed to find the value of $i$ that minimizes the expression above.  Let $s_{\text{max}}=\text{max}\left(S\right)$ be the largest element in $S$, and let $s_{\text{min}}=\text{min}\left(S\right)$ be the smallest element in $S$.  Then the value of $i$ that satisfies equation \ref{equ:argmin-1} is in the closed interval  $\left[ s_{\text{min}}, s_{\text{max}} \right].$

To see why this is so, if we use any number $k$ less than the smallest element $s_{\text{min}}$ or greater than the largest element $s_{\text{max}}$ then we are adding at least $\left|s_{\text{min}}-k\right|$ or $\left|s_{\text{max}}-k\right|$ unnecessarily to every term in the sum.

```python
def find_min_abs_diff_val(s):
    """
    returns the integer that when subtracted from every element of s
    gives the smallest sum of the absolute values of all the differences.
    for clarification see the section titled, "Smallest Absolute Deviation,"
    in http://onlinestatbook.com/2/summarizing_distributions/what_is_ct.html
    """
    s.sort()
    s_min=s[0]
    s_max=s[len(s)-1]
    result=[]
    for i in range(s_min, s_max, 1):
        sum=0
        for j in s:
            sum += abs(j-i)
        result.append((i,sum))
    result.sort(key=lambda x: x[1])
    return result[0][0]

print(find_min_abs_diff(s))
```

    4


# Week 2: Day 2 AM // PDF & Sampling

Today we will learn about

1. PDF, PMF, and CDF
2. Sampling and Sampling Distribution
3. Central Limit Theorem

```python
import numpy as np
from matplotlib import pyplot as plt
import scipy
import scipy.stats

%matplotlib inline
```

# CDF, PDF, and PMF

## CDF

$$ F_{X}(x)=P[X \le x] $$

In probability theory and statistics, the cumulative distribution function (CDF) of a real-valued random variable ${X}$, or just distribution function of ${X}$, evaluated at ${x}$, is the probability that ${X}$ will take a value less than or equal to ${x}$

In the case of a scalar continuous distribution, it gives the area under the probability density function from minus infinity to ${x}$. Cumulative distribution functions are also used to specify the distribution of multivariate random variables.

Max value of CDF is always one and min values is always 0

```python
def my_dist(x,myu=0,sigma=1):
    pf=np.exp(-0.5*((x-myu)/sigma)**2)
    return pf/(sigma*((2*np.pi)**(1/2)))

import numpy as np
from pylab import *

x = np.arange(-3, 3,0.001)
p = my_dist(x)

# Normalize the data to a proper PDF
Y = p
dx = 0.001
Y /= (dx * Y).sum()

# Compute the CDF
CY = np.cumsum(Y * dx)

# Plot both
plot(x, Y)
plot(x, CY, 'r--')

show()
```

## PDF

In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can be interpreted as providing a relative likelihood that the value of the random variable would equal that sample.

Properties of PDF:
1. Area under PDF is always accumulated to one
2. Integrate the pdf to get the cdf.

```python
fig, ax = plt.subplots(1,2, figsize =(18,4))

def my_dist(x,myu=0,sigma=1):
    pf=np.exp(-0.5*((x-myu)/sigma)**2)
    return pf/(sigma*((2*np.pi)**(1/2)))

x = np.arange(-3, 3,0.001)
p = my_dist(x)
ax[0].set_title('Handmade norm function')
ax[0].plot(x, p)

rv_norm = scipy.stats.norm(0, 1)
p_scp = rv_norm.pdf(x)
ax[1].set_title('Norm function with SciPy')
ax[1].plot(x, p_scp)
plt.show()
```

## PMF

In probability and statistics, a probability mass function is a function that gives the probability that a discrete random variable is exactly equal to some value. Sometimes it is also known as the discrete density function. It can be regarded as PDF in term of discrete variable.

```python
from scipy.stats import poisson
fig, ax = plt.subplots(1, 1)
mu = 3
x = np.arange(poisson.ppf(0.00001, mu),poisson.ppf(0.99999, mu))
ax.plot(x, poisson.pmf(x, mu), 'bo', ms=8, label='poisson pmf')
ax.vlines(x, 0, poisson.pmf(x, mu), colors='b', lw=5, alpha=0.5)
```

## Application of PDF, CDF, PMF: Probability of Events

Suppose we have a distribution of cat weights with mean of 3 and standard deviation of 0.5. For now, do not worry about the meaning mean and std, just keep in mind it is the parameter of a normal distribution. What is the chances for us founding a cat with weight above 3.8 Kg?

```python
fig, ax = plt.subplots(1,1, figsize =(10,4), sharey=True)
x = np.arange(0, 6,0.001)
rv_norm = scipy.stats.norm(3, 0.5)
p= rv_norm.pdf(x)
ax.plot(x, p)
ax.set_xlabel('Weight of cat (Kg)')
thresh_r=3.8
ax.fill_betweenx(p,x1=x,x2=thresh_r, where = x > thresh_r,color='r')
plt.show()
```

The cumulative sum (area) highlighted by the read determine the probability of us finding cat with weight larger than 3.8. This logic of finding an event probability can be applied for many functions, such as anomaly detection.

We can use CDF to find the probability, as we know CDF is the integral of PDF. Combining it with the concept of complement, we can use the following calculation.

Remember that CDF is defined by integral of PDF starting from the left side to certain point.

```python
1-rv_norm.cdf(3.8)
```

    0.054799291699557995

# Common Distribution Functions

On every distribution functions, there are some key characteristics that we need to take note:
1. PDF
2. CDF
3. Mean (Expected Value)
4. Variance

## Gaussian Distribution

Commonly found on "natural" distribution

$$ f(x) = \frac{e^{-(x - \mu)^{2}/(2\sigma^{2}) }} {\sigma\sqrt{2\pi}} $$

```python
from scipy.stats import norm
fig, ax = plt.subplots(1, 1)
x = np.linspace(norm.ppf(0.01),norm.ppf(0.99), 100)
ax.plot(x, norm.pdf(x),'r-', lw=5, alpha=0.6, label='norm pdf')
```

```python

```

## Uniform Distribution

$$ f(x) = \frac{1} {B - A}  \;\;\;\;\;\;\;  \mbox{for}  \ A \le x \le B $$

```python
import random
random.randint(1,5)
```

    1

```python
x
```

    array([-2.32634787, -2.27935095, -2.23235402, -2.18535709, -2.13836017,
           -2.09136324, -2.04436631, -1.99736939, -1.95037246, -1.90337553,
           -1.85637861, -1.80938168, -1.76238475, -1.71538783, -1.6683909 ,
           -1.62139397, -1.57439705, -1.52740012, -1.48040319, -1.43340627,
           -1.38640934, -1.33941241, -1.29241549, -1.24541856, -1.19842163,
           -1.15142471, -1.10442778, -1.05743085, -1.01043393, -0.963437  ,
           -0.91644007, -0.86944314, -0.82244622, -0.77544929, -0.72845236,
           -0.68145544, -0.63445851, -0.58746158, -0.54046466, -0.49346773,
           -0.4464708 , -0.39947388, -0.35247695, -0.30548002, -0.2584831 ,
           -0.21148617, -0.16448924, -0.11749232, -0.07049539, -0.02349846,
            0.02349846,  0.07049539,  0.11749232,  0.16448924,  0.21148617,
            0.2584831 ,  0.30548002,  0.35247695,  0.39947388,  0.4464708 ,
            0.49346773,  0.54046466,  0.58746158,  0.63445851,  0.68145544,
            0.72845236,  0.77544929,  0.82244622,  0.86944314,  0.91644007,
            0.963437  ,  1.01043393,  1.05743085,  1.10442778,  1.15142471,
            1.19842163,  1.24541856,  1.29241549,  1.33941241,  1.38640934,
            1.43340627,  1.48040319,  1.52740012,  1.57439705,  1.62139397,
            1.6683909 ,  1.71538783,  1.76238475,  1.80938168,  1.85637861,
            1.90337553,  1.95037246,  1.99736939,  2.04436631,  2.09136324,
            2.13836017,  2.18535709,  2.23235402,  2.27935095,  2.32634787])

```python
from scipy.stats import uniform
fig, ax = plt.subplots(1, 1)
#x = np.linspace(uniform.ppf(0.01,loc=1,scale=6),uniform.ppf(0.99,loc=1,scale=6), 100)
x = np.linspace(0,7, 100)
ax.plot(x, uniform.pdf(x,loc=2,scale=3),'r-', lw=5, alpha=0.6, label='uniform pdf')
```

```python
from scipy.stats import uniform
fig, ax = plt.subplots(1, 1,figsize=(15,3))
colors = ['r','b','m']
for idx,scale in enumerate([0.5,1,2]):
    color = colors[idx]
    #x = np.arange(1,10,1)
    #x = np.linspace(uniform.ppf(0.01,scale=scale),uniform.ppf(0.99,scale=scale), 100)
    x = np.linspace(0,5, 100)
    ax.plot(x, uniform.pdf(x,loc=1,scale=scale),str(color)+'-', lw=5, alpha=0.6, label='Scale='+str(scale))
    ax.legend()
```

```python

```

## Exponential Distribution

Consider a packet router that passes arriving packets to next destination routers. Measure the time
between successive packets arrivals, referred to as the packet **inter-arrival time** (expected value).

$$ f(x) = \lambda e^{-\lambda x} $$

$ \lambda $ = rate of distribution

```python
from scipy.stats import expon
fig, ax = plt.subplots(1, 1)

x = np.linspace(expon.ppf(0.01),expon.ppf(0.99), 100)
ax.plot(x, expon.pdf(x),'r-', lw=5, alpha=0.6, label='expon pdf')
```

```python

```

```python
from scipy.stats import expon
fig, ax = plt.subplots(1, 1)

colors = ['r','b','m']
for idx,lambda_ in enumerate([0.5,1,2]):
    color = colors[idx]
    x = np.linspace(0,5, 100)
    scale = 1/lambda_
    ax.plot(x, expon.pdf(x,scale=scale),str(color)+'-', lw=5, alpha=0.6, label='Lambda='+str(lambda_))
    ax.legend()
```

https://en.wikipedia.org/wiki/Exponential_distribution

## Gamma Distribution

$$ f(x) = \frac{(\frac{x-\mu}{\beta})^{\gamma - 1}\exp{(-\frac{x-\mu}
{\beta}})} {\beta\Gamma(\gamma)}  \hspace{.2in}  x \ge \mu; \gamma,
\beta > 0 $$

```python
from scipy.stats import gamma
fig, ax = plt.subplots(1, 1)
beta = 5
x = np.linspace(gamma.ppf(0.000000001,beta),gamma.ppf(0.99,beta), 100)
ax.plot(x, gamma.pdf(x,beta),'r-', lw=5, alpha=0.6, label='norm pdf')
```

## Poisson Distribution

The number of arrivals has a Poisson distribution.

$$ p(x;\lambda) = \frac{e^{-\lambda}\lambda^{x}} {x!} \mbox{    for     }
x = 0, 1, 2, \cdots $$

```python
from scipy.stats import poisson
fig, ax = plt.subplots(1, 1)
mu = 1
x = np.arange(poisson.ppf(0.00001, mu),poisson.ppf(0.99999, mu))
ax.plot(x, poisson.pmf(x, mu), 'bo', ms=8, label='poisson pmf')
ax.vlines(x, 0, poisson.pmf(x, mu), colors='b', lw=5, alpha=0.5)
```

## Binomial Distribution

Toss a biased coin. The coin falls down heads with probability p.

$$ P(x;p,n) = \left( \begin{array}{c} n \\ x \end{array} \right)
 (p)^{x}(1 - p)^{(n-x)} \;\;\;\;\;\; \mbox{for $x = 0, 1, 2, \cdots , n$} $$

```python
from scipy.stats import binom
fig, ax = plt.subplots(1, 1)

n, p = 10, 0.5

x = np.arange(binom.ppf(0.01, n, p),
              binom.ppf(0.99, n, p))
ax.plot(x, binom.pmf(x, n, p), 'bo', ms=8, label='binom pmf')
ax.vlines(x, 0, binom.pmf(x, n, p), colors='b', lw=5, alpha=0.5)
```

## Geometric Distribution

Toss a biased coin. k is the number of times we toss the coin until we see the first head.

$$ f(k)=(1-p)^{k-1}p \;\;\;\;\;\; \mbox{for $k \ge 1, 0<p \le 1 $}  $$

```python
from scipy.stats import geom
fig, ax = plt.subplots(1, 1)

p = 0.5
x = np.arange(geom.ppf(0.01, p),geom.ppf(0.99, p))
ax.plot(x, geom.pmf(x,p), 'bo', ms=8, label='geom pmf')
ax.vlines(x, 0, geom.pmf(x, p), colors='b', lw=5, alpha=0.5)
```

```python
from scipy.stats import geom
fig, ax = plt.subplots(1, 3,figsize=(15,3))

for idx,p in enumerate([0.2,0.5,0.8]):
    x = np.arange(1,10,1)
    ax[idx].plot(x, geom.pmf(x,p), 'bo', ms=8, label='geom pmf')
    ax[idx].vlines(x, 0, geom.pmf(x, p), colors='b', lw=5, alpha=0.5)
    ax[idx].set_title('P='+str(p))
```

```python
from scipy.stats import geom
fig, ax = plt.subplots(1, 3,figsize=(15,3))

for idx,p in enumerate([0.2,0.5,0.8]):
    x = np.arange(1,10,1)
    ax[idx].plot(x, geom.cdf(x,p), 'ro', ms=8, label='geom pmf')
    ax[idx].vlines(x, 0, geom.cdf(x, p), colors='r', lw=5, alpha=0.5)
    ax[idx].set_title('P='+str(p))
```

## Probability Distribution Family Tree

Most statistical analysis involves working with distributions - usually of sample data.

## Sampling and Sampling Distributions
As we discussed earlier, when working with statistics, we usually base our calculations on a sample and not the full population of data. This means we need to allow for some variation between the sample statistics and the true parameters of the full population.

In the previous example, we knew the probability that a security search would be triggered was 25%, so it's pretty easy to calculate that the expected value for a random variable indicating the number of searches per 100 passengers is 25. What if we hadn't known the probability of a search? How could we estimate the expected mean number of searches for a given number of passengers based purely on sample data collected by observing passengers go through security?

### Creating a Proportion Distribution from a Sample
We know that the each passenger will either be searched or not searched, and we can assign the values ***0*** (for not searched) and ***1*** (for searched) to these outcomes. We can conduct a Bernoulli trial in which we sample 16 passengers and calculate the fraction (or *proportion*) of passengers that were searched (which we'll call ***p***), and the remaining proportion of passengers (which are the ones who weren't searched, and can be calculated as ***1-p***).

Let's say we record the following values for our 16-person sample:

    0,1,0,0,1,0,0,0,0,0,0,0,1,0,0,0

In this sample, there were 3 searches out of 16 passengers; which as a proportion is 3/16 or 0.1875. This is our proportion (or **p**); but because we know that this is based on a sample, we call it **p̂** (or p-hat). The remaining proportion of passengers is 1-p; in this case 1 - 0.1875, which is 0.8125.

The data itself is *qualitative* (categorical) - we're indicating "no search" or "search"; but because we're using numeric values (0 and 1), we can treat these values as numeric and create a binomial distribution from them - it's the simplest form of a binomial distribution - a Bernoulli distribution with two values.

Because we're treating the results as a numberic distribution, we can also calculate statistics like *mean* and *standard deviation*:

To calculate these, you can use the following formulae:

$$
\begin{equation}\mu_{\hat{p}} = \hat{p}\end{equation}
$$

$$
\begin{equation}\sigma_{\hat{p}} = \sqrt{\hat{p}(1-\hat{p})}\end{equation}
$$

The mean is just the value of **p̂**, so in the case of the passenger search sample it is 0.1875.

The standard deviation is calculated as:

$$
\begin{equation}\sigma_{\hat{p}} = \sqrt{0.1875 \times 0.8125} \approx 0.39\end{equation}
$$

We can use Python to plot the sample distribution and calculate the mean and standard deviation of our sample like this:

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np

searches = np.array([0,1,0,0,1,0,0,0,0,0,0,0,1,0,0,0])

# Set up the graph
plt.xlabel('Search Results')
plt.ylabel('Frequency')
plt.hist(searches)
plt.show()
print('Mean: ' + str(np.mean(searches)))
print('StDev: ' + str(np.std(searches)))
```

When talking about probability, the *mean* is also known as the *expected value*; so based on our single sample of 16 passengers, should we expect the proportion of searched passengers to be 0.1875 (18.75%)?

Well, using a single sample like this can be misleading because the number of searches can vary with each sample. Another person observing 100 passengers may get a (very) different result from you. One way to address this problem is to take multiple samples and combine the resulting means to form a *sampling* distribution. This will help us ensure that the distribution and statistics of our sample data is closer to the true values; even if we can't measure the full population.

### Creating a Sampling Distribution of a Sample Proportion
So, let's collect mulitple 16-passenger samples - here are the resulting sample proportions for 12 samples:

| Sample | Result |
|--------|--------|
| p̂1| 0.1875 |
| p̂2| 0.2500 |
| p̂3| 0.3125 |
| p̂4| 0.1875 |
| p̂5| 0.1250 |
| p̂6| 0.3750 |
| p̂7| 0.2500 |
| p̂8| 0.1875 |
| p̂9| 0.3125 |
| p̂10| 0.2500 |
| p̂11| 0.2500 |
| p̂12| 0.3125 |

We can plot these as a sampling distribution like this:

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np

searches = np.array([0.1875,0.25,0.3125,0.1875,0.125,0.375,0.25,0.1875,0.3125,0.25,0.25,0.3125])

# Set up the graph
plt.xlabel('Search Results')
plt.ylabel('Frequency')
plt.hist(searches)
plt.show()
```

#### The Central Limit Theorem
You saw previously with the binomial probability distribution, with a large enough sample size (the *n* value indicating the number of binomial experiments), the distribution of values for a random variable started to form an approximately *normal* curve. This is the effect of the *central limit theorem*, and it applies to any distribution of sample data if the size of the sample is large enough. For our airport passenger data, if we collect a large enough number of samples, each based on a large enough number of passenger observations, the sampling distribution will be approximately normal. The larger the sample size, the closer to a perfect *normal* distribution the data will be, and the less variance around the mean there will be.

Run the cell below to see a simulated distribution created by 10,000 random 100-passenger samples:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

n, p, s = 100, 0.25, 10000
df = pd.DataFrame(np.random.binomial(n,p,s)/n, columns=['p-hat'])

# Plot the distribution as a histogram
means = df['p-hat']
means.plot.hist(title='Simulated Sampling Distribution')
plt.show()
print ('Mean: ' + str(means.mean()))
print ('Std: ' + str(means.std()))
```

```python
df
```

      p-hat

      0
      0.18

      1
      0.22

      2
      0.17

      3
      0.19

      4
      0.23

      ...
      ...

      9995
      0.19

      9996
      0.24

      9997
      0.22

      9998
      0.27

      9999
      0.27

10000 rows × 1 columns

### Mean and Standard Error of a Sampling Distribution of Proportion
The sampling distribution is created from the means of multiple samples, and its mean is therefore the mean of all the sample means. For a distribution of proportion means, this is considered to be the same as **p** (the population mean). In the case of our passenger search samples, this is 0.25.

Because the sampling distribution is based on means, and not totals, its standard deviation is referred to as its *standard error*, and its formula is:

$$
\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{p(1-p)}{n}}\end{equation}
$$

In this formula, *n* is the size of each sample; and we divide by this to correct for the error introduced by the average values used in the sampling distribution. In this case, our samples were based on observing 16-passengers, so:

$$
\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{16}} \approx 0.11\end{equation}
$$

In our simulation of 100-passenger samples, the mean remains 0.25. The standard error is:

$$
\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{100}} \approx 0.043\end{equation}
$$

Note that the effect of the central limit theorem is that as you increase the number and/or size of samples, the mean remains constant but the amount of variance around it is reduced.

Being able to calculate the mean (or *expected value*) and standard error is useful, because we can apply these to what we know about an approximately normal distribution to estimate probabilities for particular values. For example, we know that in a normal distribution, around 95.4% of the values are within two standard deviations of the mean. If we apply that to our sampling distribution of ten thousand 100-passenger samples, we can determine that the proportion of searched passengers in 95.4% of the samples was between 0.164 (16.4%) and 0.336 (36.6%).

How do we know this?

We know that the mean is ***0.25*** and the standard error (which is the same thing as the standard deviation for our sampling distribution) is ***0.043***. We also know that because this is a *normal* distribution, ***95.4%*** of the data lies within two standard deviations (so 2 x 0.043) of the mean, so the value for 95.4% of our samples is 0.25 ± (*plus or minus*) 0.086.

The *plus or minus* value is known as the *margin of error*, and the range of values within it is known as a *confidence interval* - we'll look at these in more detail later. For now, run the following cell to see a visualization of this interval:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

n, p, s = 100, 0.25, 10000
df = pd.DataFrame(np.random.binomial(n,p,s)/n, columns=['p-hat'])

# Plot the distribution as a histogram
means = df['p-hat']
m = means.mean()
sd = means.std()
moe1 = m - (sd * 2)
moe2 = m + (sd * 2)

means.plot.hist(title='Simulated Sampling Distribution')

plt.axvline(m, color='red', linestyle='dashed', linewidth=2)
plt.axvline(moe1, color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(moe2, color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

### Creating a Sampling Distribution of Sample Means
In the previous example, we created a sampling distribution of proportions; which is a suitable way to handle discrete values, like the number of passengers searched or not searched. When you need to work with continuous data, you use slightly different formulae to work with the sampling distribution.

For example, suppose we want to examine the weight of the hand luggage carried by each passenger. It's impractical to weigh every bag that is carried through security, but we could weigh one or more samples, for say, 5 passengers at a time, on twelve occassions. We might end up with some data like this:

| Sample | Weights |
|--------|---------|
| 1      | [4.020992,2.143457,2.260409,2.339641,4.699211] |
| 2      | [3.38532,4.438345,3.170228,3.499913,4.489557] |
| 3      | [3.338228,1.825221,3.53633,3.507952,2.698669] |
| 4      | [2.992756,3.292431,3.38148,3.479455,3.051273] |
| 5      | [2.969977,3.869029,4.149342,2.785682,3.03557] |
| 6      | [3.138055,2.535442,3.530052,3.029846,2.881217] |
| 7      | [1.596558,1.486385,3.122378,3.684084,3.501813] |
| 8      | [2.997384,3.818661,3.118434,3.455269,3.026508] |
| 9      | [4.078268,2.283018,3.606384,4.555053,3.344701] |
| 10     | [2.532509,3.064274,3.32908,2.981303,3.915995] |
| 11     | [4.078268,2.283018,3.606384,4.555053,3.344701] |
| 12     | [2.532509,3.064274,3.32908,2.981303,3.915995] |

Just as we did before, we could take the mean of each of these samples and combine them to form a sampling distribution of the sample means (which we'll call **X**, and which will contain a mean for each sample, which we'll label x̄n):

| Sample | Mean Weight |
|--------|---------|
| x̄1 | 3.092742  |
| x̄2 | 3.7966726 |
| x̄3 | 2.98128   |
| x̄4 | 3.239479  |
| x̄5 | 3.36192   |
| x̄6 | 3.0229224 |
| x̄7 | 2.6782436 |
| x̄8 | 3.2832512 |
| x̄9 | 3.5734848 |
| x̄10 | 3.1646322 |
| x̄11 | 3.5734848 |
| x̄12 | 3.1646322 |

We can plot the distribution for the sampling distribution like this:

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np

meanweights = np.array([3.092742,
                        3.7966726,
                        2.98128,
                        3.239479,
                        3.36192,
                        3.0229224,
                        2.6782436,
                        3.2832512,
                        3.5734848,
                        3.1646322,
                        3.5734848,
                        3.1646322])

# Set up the graph
plt.xlabel('Mean Weights')
plt.ylabel('Frequency')
plt.hist(meanweights, bins=6)
plt.show()

print('Mean: ' + str(meanweights.mean()))
print('Std: ' + str(meanweights.std()))
```

Just as before, as we increase the sample size, the central limit theorem ensures that our sampling distribution starts to approximate a normal distribution. Our current distribution is based on the means generated from twelve samples, each containing 5 weight observations. Run the following code to see a distribution created from a simulation of 10,000 samples each containing weights for 500 passengers:

>This may take a few minutes to run. The code is not the most efficient way to generate a sample distribution, but it reflects the principle that our sampling distribution is made up of the means from multiple samples. In reality, you could simulate the sampling by just creating a single sample from the ***random.normal*** function with a larger ***n*** value.

```python

```

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

mu, sigma, n = 3.2, 1.2, 500
samples = list(range(0, 10000))

# data will hold all of the sample data
data = np.array([])

# sampling will hold the means of the samples
sampling = np.array([])

# Perform 10,000 samples
for s in samples:
    # In each sample, get 500 data points from a normal distribution
    sample = np.random.normal(mu, sigma, n)
    data = np.append(data,sample)
    sampling = np.append(sampling,sample.mean())

# Create a dataframe with the sampling of means
df = pd.DataFrame(sampling, columns=['mean'])

# Plot the distribution as a histogram
means = df['mean']
means.plot.hist(title='Simulated Sampling Distribution', bins=100)
plt.show()

# Print the Mean and StdDev for the full sample and for the sampling distribution
print('Sample Mean: ' + str(data.mean()))
print('Sample StdDev: ' + str(data.std()))
print ('Sampling Mean: ' + str(means.mean()))
print ('Sampling StdErr: ' + str(means.std()))
```

### Mean and Variance of the Sampling Distribution

The following variables are printed beneath the histogram:

- **Sample Mean**: This is the mean for the complete set of sample data - all 10,000 x 500 bag weights.
- **Sample StdDev**: This is the standard deviation for the complete set of sample data - all 10,000 x 500 bag weights.
- **Sampling Mean**: This is the mean for the sampling distribution - the means of the means!
- **Sampling StdErr**: This is the standard deviation (or *standard error*) for the sampling distribution

If we assume that **X** is a random variable representing every possible bag weight, then its mean (indicated as **μx**) is the population mean (**μ**). The mean of the **X** sampling distribution (which is indicated as **μx̄**) is considered to have the same value. Or, as an equation:

$$
\begin{equation}\mu_{x} = \mu_{\bar{x}}\end{equation}
$$

In this case, the full population mean is unknown (unless we weigh every bag in the world!), but we do have the mean of the full set of sample observations we collected (**x̄**), and if we check the values generated by Python for the sample mean and the sampling mean, they're more or less the same: around 3.2.

To find the standard deviation of the sample mean, which is technically the *standard error*, we can use this formula:

$$
\begin{equation}\sigma_{\bar{x}}  = \frac{\sigma}{\sqrt{n}}\end{equation}
$$

In this formula, ***σ*** is the population standard deviation and ***n*** is the size of each sample.

Since our the population standard deviation is unknown, we can use the full sample standard deviation instead:

$$
\begin{equation}SE_{\bar{x}} \approx \frac{s}{\sqrt{n}}\end{equation}
$$

In this case, the standard deviation of our set of sample data is around 1.2, and we have used 500 variables in each sample to calculate our sample means, so:

$$
\begin{equation}SE_{\bar{x}} \approx \frac{1.2}{\sqrt{500}} = \frac{1.2}{22.36} \approx 0.053\end{equation}
$$

## Confidence Intervals
A confidence interval is a range of values around a sample statistic within which we are confident that the true parameter lies. For example, our bag weight sampling distribution is based on samples of the weights of bags carried by passengers through our airport security line. We know that the mean weight (the *expected value* for the weight of a bag) in our sampling distribution is 3.2, and we assume this is also the population mean for all bags; but how confident can we be that the true mean weight of all carry-on bags is close to the value?

Let's start to put some precision onto these terms. We could state the question another way. What's the range of weights within which are confident that the mean weight of a carry-on bag will be 95% of the time? To calculate this, we need to determine the range of values within which the population mean weight is likely to be in 95% of samples. This is known as a *confidence interval*; and it's based on the Z-scores inherent in a normal distribution.

Confidence intervals are expressed as a sample statistic ± (*plus or minus*) a margin of error. To calculate the margin of error, you need to determine the confidence level you want to find (for example, 95%), and determine the Z score that marks the threshold above or below which the values that are *not* within the chosen interval reside. For example, to calculate a 95% confidence interval, you need the critical Z scores that exclude 5% of the values under the curve; with 2.5% of them being lower than the values in the confidence interval range, and 2.5% being higher. In a normal distribution, 95% of the area under the curve is between a Z score of ± 1.96. The following table shows the critical Z values for some other popular confidence interval ranges:

| Confidence  | Z Score |
|-------------|---------|
| 90%         | 1.645   |
| 95%         | 1.96    |
| 99%         | 2.576   |

To calculate a confidence interval around a sample statistic, we simply calculate the *standard error* for that statistic as described previously, and multiply this by the approriate Z score for the confidence interval we want.

To calculate the 95% confidence interval margin of error for our bag weights, we multiply our standard error of 0.053 by the Z score for a 95% confidence level, which is 1.96:

$$
\begin{equation}MoE = 0.053 \times 1.96 = 0.10388 \end{equation}
$$

So we can say that we're confident that the population mean weight is in the range of the sample mean ± 0.10388 with 95% confidence. Thanks to the central limit theorem, if we used an even bigger sample size, the confidence interval would become smaller as the amount of variance in the distribution is reduced. If the number of samples were infinite, the standard error would be 0 and the confidence interval would become a certain value that reflects the true mean weight for all carry-on bags:

$$
\begin{equation}\lim_{n \to \infty} \frac{\sigma}{\sqrt{n}} = 0\end{equation}
$$

In Python, you can use the *scipy.stats.**norm.interval*** function to calculate a confidence interval for a normal distribution. Run the following code to recreate the sampling distribution for bag searches with the same parameters, and display the 95% confidence interval for the mean (again, this may take some time to run):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from scipy import stats

mu, sigma, n = 3.2, 1.2, 500
samples = list(range(0, 10000))

# data will hold all of the sample data
data = np.array([])

# sampling will hold the means of the samples
sampling = np.array([])

# Perform 10,000 samples
for s in samples:
    # In each sample, get 500 data points from a normal distribution
    sample = np.random.normal(mu, sigma, n)
    data = np.append(data,sample)
    sampling = np.append(sampling,sample.mean())

# Create a dataframe with the sampling of means
df = pd.DataFrame(sampling, columns=['mean'])

# Get the Mean, StdDev, and 95% CI of the means
means = df['mean']
m = means.mean()
sd = means.std()
ci = stats.norm.interval(0.95, m, sd)

# Plot the distribution, mean, and CI
means.plot.hist(title='Simulated Sampling Distribution', bins=100)
plt.axvline(m, color='red', linestyle='dashed', linewidth=2)
plt.axvline(ci[0], color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(ci[1], color='magenta', linestyle='dashed', linewidth=2)
plt.show()

# Print the Mean, StdDev and 95% CI
print ('Sampling Mean: ' + str(m))
print ('Sampling StdErr: ' + str(sd))
print ('95% Confidence Interval: ' + str(ci))
```

In Normal Distribution 95% confidence interval are located at $(\mu - 1.645\sigma,\mu + 1.645\sigma)$


## Exercise 2.9 Conditional independence
(Source: Koller.) Are the following properties true? Prove or disprove. Note that we are not restricting
attention to distributions that can be represented by a graphical model.
* a. True or false? $(X\perp W|Z,Y)\land(X\perp Y|Z)\Rightarrow(X\perp Y,W|Z)$
* b. True or false?$(X\perp Y|Z)\land(X\perp Y|W)\Rightarrow(X\perp Y|Z,W)$

### Solution
#### (a)
We have two information about conditional independence: The first one is

$$
X\perp W |Z, Y.
$$

Based on this, we can state:
\begin{equation}
p(x, w |z, y) = p(x |z, y) p(w|z, y)
\end{equation}

The second one is:

$$
X\perp Y|Z
$$

Based on this, we can state that:

\begin{equation}
p(x, y|z) = p(x|z)p(y|z)
\end{equation}

Now we have to see if $X\perp Y, W|Z$ is true.

\begin{equation}
p(x, y, w|z) = p(w|y, x, z)p(y|x, z)p(x|z) = p(x|z)p(y|z)p(w|y, z) = p(x|z)p(y, w|z)
\end{equation}

- The first passage is the chain rule of probability
- The second makes use of the conditional independences given to us ($Y$ does not dependent on $X$ given $Z$ and $W$ does not depend on $X$ given $Z$ and $Y$)
- The third puts the joint distribution back together. So the proposition is **True**.

#### (b)
Prove this proposition is false with a counter example. Start by defining $X$, $Y$ and $Z$ to be iid random variables with the following distribution:

\begin{equation}\left\{
\begin{array}{l}
P(X=1) = 0.5 \\
P(X=-1) = 0.5
\end{array}\right.
\end{equation}

Also, define $W=XYZ$. First prove that the two conditional independences $X\perp Y|Z$ and $X\perp Y|W$ are true.