# Bite Size Bayes

Copyright 2020 Allen B. Downey

License: [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
def prob(A):
    """Computes the probability of a proposition, A.

    A: Boolean series

    returns: probability
    """
    return A.mean()
```

```python
def conditional(A, B):
    """Conditional probability of A given B.

    A: Boolean series
    B: Boolean series

    returns: probability
    """
    return prob(A[B])
```

## Two coins

> Suppose I flip two fair coins and tell you (honestly) that at least one of the coins is heads.  What is the probability that both coins are heads?

The answer is 1/3, and here's an argument that explains it.

1. If you toss two coins, there are 4 equally likely outcomes: HH, HT, TH, TT

2. If I tell you that at least one is heads, that eliminates TT.

3. The remaining 3 outcomes are still equally likely, so their probability is now 1/3 each.

4. Therefore, the probability of HH is now 1/3.

However, you might still have some doubts.  For me, Step 3 feels like an unsupported assertion: How do we know the 3 remaining outcomes are still equally likely?

The following simulation might help convince you.

First I'll generate two sets of coin flips.

```python
size = 10000
first = np.random.choice(['H', 'T'], size)
second = np.random.choice(['H', 'T'], size)
```

We can confirm that each coin has a 50% chance of landing heads:

```python
prob(first == 'H')
```

    0.5067

```python
prob(second == 'H')
```

    0.493

Now we can compute a Boolean Series that is `True` if either coin landed heads, or both.

```python
at_least_one = (first == 'H') | (second == 'H')
```

And we can confirm that happens 75% of the time:

```python
prob(at_least_one)
```

    0.7491

We can compute a Boolean Series that is `True` if both coins landed heads.

```python
both = (first == 'H') & (second == 'H')
```

And confirm that it happens 25% of the time.

```python
prob(both)
```

    0.2506

Finally, we can compute the conditional probability of `both` given `at_least_one`:

```python
conditional(both, at_least_one)
```

    0.33453477506340945

## The Monty Hall problem

From [Wikipedia](https://en.wikipedia.org/wiki/Monty_Hall_problem):

> Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

To avoid ambiguities, we have to make some assumptions about the behavior of the host:

1. The host never opens the door you picked.

2. The host never opens the door with the car.

3. If you choose the door with the car, the host chooses one of the other doors at random.

4. The host always offers you the option to switch.

Under these assumptions, are you better off sticking or switching?

The correct answer is that you are better off switching.  If you stick, you win 1/3 of the time.  If you switch, you win 2/3 of the time.

Here's one of many arguments that might persuade you.

> If you always stick, you win if you initially choose the door with the car, so the probability is 1/3.
>
> If you always switch, you win if you did _not_ choose the door with the car, so the probability is 2/3.

However, many people do not find any verbal arguments persuasive.  So, maybe a simulation will help.

**Exercise:** Write a simulation that confirms that you are better off switching if the host opens Door 3.

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

## Bayes's Theorem

In the previous two examples, you might have noticed a seeming contradiction:

* In the coin example, we start with four hypotheses with equal probability; one of them is eliminated by the data, and I argue that the other three still have equal probability.

* In the Monty Hall example, we start with three hypotheses with equal probability; one of them is eliminated by the data, but it turns out that the other two do _not_ have equal probability.

When one hypothesis is eliminated, its probability is redistributed to the remaining hypotheses, but it seems like there is no general rule for _how_ it is redistibuted.

Fortunately, Bayes's Theorem resolves this contradiction; if we apply it carefully, it tells us exactly how the probability should be redistributed.

First I'll solve the coin problem using a Bayes table.  Again, we start with four hypotheses with equal prior probability.  Just for fun, I'll use `Fraction` objects so the results are represented as rational numbers rather than floating-point.

```python
from fractions import Fraction

hypos = ['HH', 'HT', 'TH', 'HH']
table = pd.DataFrame(index=hypos)
table['prior'] = Fraction(1, 4)
table
```

      prior

      HH
      1/4

      HT
      1/4

      TH
      1/4

      HH
      1/4

The data, in this example, is my report that there is at least one heads.  Assuming that I report honestly, we can compute the likelihood of the data under each hypothesis.

```python
table['likelihood'] = [1, 1, 1, 0]
table
```

      prior
      likelihood

      HH
      1/4
      1

      HT
      1/4
      1

      TH
      1/4
      1

      HH
      1/4
      0

And we fill in the rest of the table in the usual way.

```python
table['unnorm'] = table['prior'] * table['likelihood']
prob_data = table['unnorm'].sum()
table['posterior'] = table['unnorm'] / prob_data
table
```

      prior
      likelihood
      unnorm
      posterior

      HH
      1/4
      1
      1/4
      1/3

      HT
      1/4
      1
      1/4
      1/3

      TH
      1/4
      1
      1/4
      1/3

      HH
      1/4
      0
      0
      0

In this example the remaining hypotheses have the same posterior probability because the likelihood of the data is the same under any of them.

For the Monty Hall problem, that is not the case.  We'll start with three hypotheses, one for each door, and equal priors.

```python
hypos = ['Door 1', 'Door 2', 'Door 3']
table = pd.DataFrame(index=hypos)
table['prior'] = Fraction(1, 3)
table
```

      prior

      Door 1
      1/3

      Door 2
      1/3

      Door 3
      1/3

Now, the data is that the host opens Door 3.  So the question is, what is the probability of the data under each hypothesis.  In reverse order:

* Door 3: If the car is behind Door 3 and you choose Door 1, the host has no choice but to open Door 2, so the probability that he opens Door 3 is 0.

* Door 2: If the car is behind Door 2 and you choose Door 1, the host has not choice but to open Door 3, so the probability that he opens Door 3 is 1.

* Door 1: If the car is behind Door 1 and you choose Door 1, the host has a choice, and the statement of the problem indicates that he chooses either Door 2 or Door 3 with equal probability, so the probability that he opens Door 3 is 1/2.

That's all we need to fill in the likelihoods:

```python
table['likelihood'] = [Fraction(1,2), 1, 0]
table
```

      prior
      likelihood

      Door 1
      1/3
      1/2

      Door 2
      1/3
      1

      Door 3
      1/3
      0

And we fill in the rest of the table in the usual way.

```python
table['unnorm'] = table['prior'] * table['likelihood']
prob_data = table['unnorm'].sum()
table['posterior'] = table['unnorm'] / prob_data
table
```

      prior
      likelihood
      unnorm
      posterior

      Door 1
      1/3
      1/2
      1/6
      1/3

      Door 2
      1/3
      1
      1/3
      2/3

      Door 3
      1/3
      0
      0
      0

In this example the likelihood of the data is not the same under the remaining hypotheses, so the posterior probabilities are not the same.

**Exercise:** Here's a variation on the Monty Hall problem.  Suppose that whenever the host has a choice, he opens Door 3.  In that case, what are the posterior probabilities for the three doors?

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

**Exercise:** Suppose that whenever the host has a choice he chooses Door 3 with probability `p` and Door 2 with probability `1-p`.  What are the posterior probabilities for the three doors?

Hint: If you use SymPy to create a symbol for `p`, it will carry through the computation.

```python
from sympy import symbols

p = symbols('p')
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python

```


# Semi-Monocoque Theory: corrective solutions

```python
from pint import UnitRegistry
import sympy
import networkx as nx
import numpy as np
import matplotlib.pyplot as plt
import sys
%matplotlib inline
from IPython.display import display
```

Import **Section** class, which contains all calculations

```python
from Section import Section
```

Initialization of **sympy** symbolic tool and **pint** for dimension analysis (not really implemented rn as not directly compatible with sympy)

```python
ureg = UnitRegistry()
sympy.init_printing()
```

Define **sympy** parameters used for geometric description of sections

```python
A, A0, t, t0, a, b, h, L, E, G = sympy.symbols('A A_0 t t_0 a b h L E G', positive=True)
```

We also define numerical values for each **symbol** in order to plot scaled section and perform calculations

```python
values = [(A, 150 * ureg.millimeter**2),(A0, 250  * ureg.millimeter**2),(a, 400 * ureg.millimeter), \
          (b, 20 * ureg.millimeter),(h, 150 * ureg.millimeter),(L, 2000 * ureg.millimeter), \
          (t, 0.8 *ureg.millimeter),(E, 72e3 * ureg.MPa), (G, 28e3 * ureg.MPa)]
datav = [(v[0],v[1].magnitude) for v in values]
```

# First example: Simple rectangular symmetric section

Define graph describing the section:

1) **stringers** are **nodes** with parameters:
- **x** coordinate
- **y** coordinate
- **Area**

2) **panels** are **oriented edges** with parameters:
- **thickness**
- **lenght** which is automatically calculated

```python
stringers = {1:[(a,h),A],
             2:[(sympy.Rational(1,2)*a,h),A],
             3:[(sympy.Integer(0),h),A],
             4:[(sympy.Integer(0),sympy.Integer(0)),A],
             5:[(sympy.Rational(1,2)*a,sympy.Integer(0)),A],
             6:[(a,sympy.Integer(0)),A]}

panels = {(1,2):t,
          (3,4):t,
          (4,5):t,
          (5,6):t,
          (6,1):t,
          (5,2):t}
```

Define section and perform first calculations

```python
S1 = Section(stringers, panels)
```

```python
S1.cycles
```

## Plot of **S1** section in original reference frame

Define a dictionary of coordinates used by **Networkx** to plot section as a Directed graph.
Note that arrows are actually just thicker stubs

```python
start_pos={ii: [float(S1.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }
```

```python
plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);
```

## Plot of **S1** section in inertial reference Frame

Section is plotted wrt **center of gravity** and rotated (if necessary) so that *x* and *y* are principal axes.
**Center of Gravity** and **Shear Center** are drawn

```python
positions={ii: [float(S1.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }
```

```python
x_ct, y_ct = S1.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);
```

## Standard Solution

```python
Tx, Ty, Nz, Mx, My, Mz, F, ry, ry, mz = sympy.symbols('T_x T_y N_z M_x M_y M_z F r_y r_x m_z')
```

```python
S1.set_loads(_Tx=0, _Ty=0, _Nz=0, _Mx=0, _My=0, _Mz=Mz)
```

## Loads

```python
S1.compute_stringer_actions()
S1.compute_panel_fluxes();
```

```python
S1.N
```

```python
S1.q
```

```python
S1.set_loads(_Tx=0, _Ty=Ty, _Nz=0, _Mx=Mx, _My=0, _Mz=0)
```

```python
S1.compute_stringer_actions()
S1.compute_panel_fluxes();
```

```python
S1.N
```

```python
S1.q
```

# Autosolutions

Compute **L** matrix: with 6 nodes we expect 3 **dofs**, one with _symmetric load_ and one with _antisymmetric load_

```python
S1.compute_L()
```

```python
S1.L
```

Compute **H** matrix

```python
S1.compute_H()
```

```python
S1.H
```

Compute $\tilde{K}$ and $\tilde{M}$ as:

$$\tilde{K} = L^T \cdot \left[ \frac{A}{A_0} \right] \cdot L$$
$$\tilde{M} = H^T \cdot \left[ \frac{l}{l_0}\frac{t_0}{t} \right] \cdot L$$

```python
S1.compute_KM(A,a/2,t,datav)
```

```python
S1.Ktilde
```

```python
S1.Mtilde
```

Compute **eigenvalues** and **eigenvectors** as:

$$\left| \mathbf{I} \cdot \beta^2 - \mathbf{\tilde{K}}^{-1} \cdot \mathbf{\tilde{M}} \right| = 0$$

We substitute some numerical values to simplify the expressions

```python
sol_data = (sympy.N(S1.Ktilde.subs(datav).inv())*(sympy.N(S1.Mtilde.subs(datav)))).eigenvects()
```

**Eigenvalues** correspond to $\beta^2$

```python
β2 = [sol[0] for sol in sol_data]
β2
```

**Eigenvectors** are orthogonal as expected

```python
X = [sol[2][0] for sol in sol_data]
X
```

From $\beta_i^2$ we compute:
$$\lambda_i = \sqrt{\frac{E A_0 l_0}{G t_0} \beta_i^2}$$

substuting numerical values

```python
λ = [sympy.N(sympy.sqrt(E*A*b/(G*t)*βi).subs(datav)) for βi in β2]
λ
```

```python

```


# Stochastic Processes:
Data Analysis and Computer Simulation

# Distribution function and random number

# 2. Generating random numbers with Gaussian/binomial/Poisson distributions

# 2.1. Preparations

## Python built-in functions for random numbers (numpy.random)

1. `seed(seed)`: Initialize the generator with an integer "seed".
1. `rand(d0,d1,...,dn)`: Return a multi-dimensional array of uniform random numbers of shape (d0, d1, ... , dn).
1. `randn(d0,d1,...,dn)`: The same as above but from the standard normal distribution.
1. `binomial(M,p,size)`: Draw samples from a binomial distribution with "M" and "p".
1. `poisson(a,size)`: Draw samples from a Poisson distribution with "a".
1. `choice([-1,1],size)`: Generates random samples from the two choices, -1 or 1 in this case.
1. `normal(ave,std,size)`: Draw random samples from a normal distribution.
1. `uniform([low,high,size])`: Draw samples from a uniform distribution.

- See the Scipy website for details

https://docs.scipy.org/doc/numpy-dev/reference/routines.random.html

## Import common libraries

```python
% matplotlib inline
import numpy as np # import numpy library as np
import math # use mathematical functions defined by the C standard
import matplotlib.pyplot as plt # import pyplot library as plt
plt.style.use('ggplot') # use "ggplot" style for graphs
```

# 2.2. Normal / Gaussian distribution

## Generate random numbers, $x_0, x_1, \cdots, x_N$

```python
ave = 0.0 # set average
std = 1.0 # set standard deviation
N = 100000 # number of generated random numbers
np.random.seed(0) # initialize the random number generator with seed=0
X = ave+std*np.random.randn(N) # generate random sequence and store it as X
plt.ylim(-10,10) # set y-range
plt.xlabel(r'$i$',fontsize=16) # set x-label
plt.ylabel(r'$x_i$',fontsize=16) # set y-label
plt.plot(X,',') # plot x_i vs. i (i=1,2,...,N) with dots
plt.show() # draw plots
```

## Compare the distribution with the normal distribution function

\begin{equation}
P(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\langle X \rangle)^2}{2\sigma^2}\right]\tag{D1}
\end{equation}

```python
plt.hist(X,bins=25,normed=True) # plot normalized histogram of R using 25 bins
x = np.arange(-10,10,0.01) # create array of x from 0 to 1 with increment 0.01
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # create array of y as y=1
plt.xlim(-10,10) # set x-range
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$x$',fontsize=16) # set x-label
plt.ylabel(r'$P(x)$',fontsize=16) # set y-label
plt.legend([r'Gaussian',r'histogram'], fontsize=16) # set legends
plt.show() # display plots
```

## Calculate the auto-correlation function $\varphi(i)$

### The definition

\begin{equation}
\varphi(i)=\frac{1}{N}\sum_{j=1}^{N} \left(x_j - \langle X\rangle\right)\left(x_{i+j}-\langle X\rangle\right) \tag{D2}
\end{equation}

\begin{equation}
\varphi(i=0)=\frac{1}{N}\sum_{j=1}^{N} \left(x_j - \langle X\rangle\right)^2=\langle x_j - \langle X\rangle\rangle^2=\sigma^2\tag{D3}
\end{equation}

\begin{equation}
\varphi(i\ne 0)= \langle x_j - \langle X\rangle\rangle\langle x_{i\ne j} - \langle X\rangle\rangle=0\hspace{5mm}
(\rightarrow{\rm White\ noise})\hspace{-12mm}
\tag{D4}
\end{equation}

### A code example to calculate auto-correlation

```python
def auto_correlate(x):
    cor = np.correlate(x,x,mode="full")
    return cor[N-1:]
c = np.zeros(N)
c = auto_correlate(X-ave)/N
plt.plot(c)
plt.xlim(-1000,10000)
plt.xlabel(r'$i$',fontsize=16)
plt.ylabel(r'$\varphi(i)$',fontsize=16)
print('\sigma^2  =',std**2)
plt.show()
```

# 2.3. Binomial distribution

## Generate random numbers, $n_0, n_1, \cdots, n_N$

```python
p = 0.5 # set p, propability to obtain "head" from a coin toss
M = 100 # set M, number of tosses in one experiment
N = 100000 # number of experiments
np.random.seed(0) # initialize the random number generator with seed=0
X = np.random.binomial(M,p,N) # generate the number of heads after M tosses, N times, and store it as X
plt.xlabel(r'$i$',fontsize=16) # set x-label
plt.ylabel(r'$n_i$',fontsize=16) # set y-label
plt.plot(X,',') # plot n_i vs. i (i=1,2,...,N) with dots
plt.show() # draw plots
```

## Compare the distribution with the Binomial distribution function

\begin{equation}
P(n)=\frac{M!}{n!(M-n)!}p^{n}(1-p)^{M-n}\tag{D5}
\end{equation}

\begin{equation}
\langle n \rangle=Mp\tag{D6}
\end{equation}

\begin{equation}
\sigma^2=Mp(1-p)\tag{D7}
\end{equation}

```python
def binomial(n,m,p):
    comb=math.factorial(m)/(math.factorial(n)*math.factorial(m-n))
    prob=comb*p**n*(1-p)**(m-n)
    return prob
plt.hist(X,bins=20,normed=True) # plot normalized histogram of R using 22 bins
x = np.arange(M) # generate array of x values from 0 to 100, in intervals of 1
y = np.zeros(M) # generate array of y values, initialized to 0
for i in range(M):
    y[i]=binomial(i,M,p) # compute binomial distribution P(n), Eq. (D5)
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Binomial',r'histogram'], fontsize=16) # set legends
plt.show() # display plots
```

## Calculate the auto-correlation function $\varphi(i)$

### The definition

\begin{equation}
\varphi(i)=\frac{1}{N}\sum_{j=1}^{N} \left(n_j - \langle n\rangle\right)\left(n_{i+j}-\langle n\rangle\right) \tag{D8}
\end{equation}

\begin{equation}
\varphi(i=0)=\frac{1}{N}\sum_{j=1}^{N} \left(n_j - \langle n\rangle\right)^2=\langle n_j - \langle n\rangle\rangle^2=\sigma^2=Mp(1-p)\tag{D9}
\end{equation}

\begin{equation}
\varphi(i\ne 0)= \langle n_j - \langle n\rangle\rangle\langle n_{i\ne j} - \langle n\rangle\rangle=0\hspace{8mm}
(\rightarrow{\rm White\ noise})\hspace{10mm}
\tag{D10}
\end{equation}

### A code example to calculate auto-correlation

```python
def auto_correlate(x):
    cor = np.correlate(x,x,mode="full")
    return cor[N-1:]
c = np.zeros(N)
c = auto_correlate(X-M*p)/N
plt.plot(c)
plt.xlim(-1000,10000)
plt.xlabel(r'$i$',fontsize=16)
plt.ylabel(r'$\varphi(i)$',fontsize=16)
print('\sigma^2  =',M*p*(1-p))
plt.show()
```

# 2.4. Poisson distribution

## Generate random numbers, $n_0, n_1, \cdots, n_N$

```python
a = 10.0 # set a, the expected value
N = 100000 #  number of generated random numbers
np.random.seed(0) # initialize the random number generator with seed=0
X = np.random.poisson(a,N) # generate randon numbers from poisson distribution
plt.xlabel(r'$i$',fontsize=16) # set x-label
plt.ylabel(r'$n_i$',fontsize=16) # set y-label
plt.plot(X,',') # plot n_i vs. i (i=1,2,...,N) with dots
plt.show() # draw plots
```

## Compare the distribution with the binomial distribution function

\begin{equation}
P(n)=\frac{a^n e^{-a}}{n!}\tag{D11}
\end{equation}

\begin{equation}
\langle n \rangle=a\tag{D12}
\end{equation}

\begin{equation}
\sigma^2=a\tag{D13}
\end{equation}

```python
def poisson(n,a):
    prob=a**n*np.exp(-a)/math.factorial(n)
    return prob
plt.hist(X,bins=25,normed=True) # plot normalized histogram of X using 25 bins
x = np.arange(M) # generate array of x values from 0 to 100, in intervals of 1
y = np.zeros(M) # generate array of y values, initialized to zero
for i in range(M):
    y[i]=poisson(i,a) # Compute Poisson distribution for n, Eq. (D11)
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Poisson',r'histogram'], fontsize=16) # set legends
plt.show() # display plots
```


# Number Theory on Primes

```python
from math import factorial, log
import numpy as np
import matplotlib.pyplot as plt
%run primes.py
%run ../../python-data-science/display_helpers.py
```

```python
%%timeit -r 1
A = Prime.nth(10000)
B = Prime.nth(20000)
P = A*B

# (104743, 110609, 11585518487)
display(A, B, P)
```

    104743

    224743

    23540256049

    104743

    224743

    23540256049

    13 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

----

## Wilson's Theorem

- p is prime

$ (p-1)! \equiv p-1 \bmod p$

```python
S = 3000
N = 4000
f = lambda p: (factorial(p-1) % p) / p
x = np.arange(S,N)
y = np.array( [f(p) for p in x] )

plt = configure_plot((S, N), (0, 1.5), figsize=(24, 8))
#plt.plot(x, y, color = 'tomato')
plt.scatter(x, y, s=1)
display(plt)
```

----

## Fermat's Little Theorem

- p is prime
- a, *s.t* p $\nmid$ a ( *is not divisble by*  )

$a^{p-1} \equiv -1 \pmod p$

```python
p = Prime.nth(3000)
a = 2 # not prime is divisble by 2, 2 is also faster for computer
a**(p-1) % p
```

    1

Usage:

Find the remainder when you divide $3^{100,000}$ by 53.

p = 53 is prime, and 53 $\nmid$ 3 so

$\begin{equation}
\begin{split}
3^{53-1} &\equiv 1 \pmod{53}  \\
3^{52} &\equiv 1 \pmod{53}  \\
{ \text{trick: raise both sides to a large power} } \\
\frac{100, 000}{52} : quotient &= 1923, rem =  4 \\
(3^{52})^{1923} &\equiv 1^{1923} \pmod{53} \\
3^{99,996} &\equiv 1 \pmod{53} \\
3^4(3^{99,996}) &\equiv 3^4 \pmod{53} \\
3^{100,000} &\equiv 81 \pmod{53} \\
3^{100,000} &\equiv 28 \pmod{53} \\
\end{split}
\end{equation}
$

Proof:
https://www.youtube.com/watch?v=w0ZQvZLx2KA&list=PLi01XoE8jYojnxiwwAPRqEH19rx_mtcV_&index=3

- p is prime
- a, *s.t* p $\nmid$ a ( *is not divisble by*  )

Every integer is congruent to one of `0, 1, 2, 3, ... p-1 (mod p)`

Use only non-zero congruence classes, because 0 (pod p) contains the multiples of p, and p $\nmid$ a

Multiply all other values `1, 2, 3 ... p-1` with a.

`a, 2a, 3a ... (p-1)a`

show this list is a rearrangement of all the elements in the group.

> case 1: None of these are congruent to 0.

> suppose $r.a \equiv 0 \pmod p$, then p $\mid$ a, which is a contradiction

> case 2: These are distinct; no two are congruent to each other.

  > Pick 2 values - r.a, s.a, where 0 < r < p, and 0 < s
 Show that $r.a \not\equiv s.a \pmod p$
  > So r.a - s.a = a.(r-s)
  > By assumption, p $\mid$ a
  > Can p divide r-s?
  >    0 < r < p ,   -p < -s < 0   =>  -p < r-s
    $r -s \ne 0$, therefore $p \nmid r-s$

$ (p-1)!a^{p-1} \equiv (p-1)! \pmod p$

$ a^{p-1} \equiv 1 \pmod p$

----
## The Prime Number Theorem

Definition: P(n) = number of primes $\leq$ n

P(6) = 3
P(10) = 4

```python
S = 2
N = 50000

h = (N / log(N)) * 1.2
r = range(S,N)
pi = Prime.pi()

x = np.arange(S, N)
y = np.array([next(pi) for a in r])
y_asym = x / np.log(x)

plt = configure_plot((0, N), (0, y_asym[-1]*1.2), figsize=(12, 8))
plt.plot( x, y, color = 'tomato')
plt.plot( x, y_asym, color = 'tomato', dashes = [4,4])
display(plt)
```

$P(n) \sim \frac{n}{log n} $

(i.e. ~ is asymptotically equivalent as n approaches infinity)

$\displaystyle  a_n \sim b_n if \lim_{n\to\inf} \frac{a_n}{b_n} = 1$

```python

```


# Visualizing stationary paths of a  functional

Last revised: 02-Feb-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Consider the functional

$\begin{align}
  S = \int_{x_1}^{x_2} f[y(x), y'(x), x] \, dx
\end{align}$

with $y_1 = y(x_1)$ and $y_2 = y(x_2)$ fixed.  We denote by $y^*(x)$ the path that minimizes $S$ (or, more generally, makes it stationary).  Then we consider the class of candidate paths $y(x)$ given by

$\begin{align}
  y(x) = y^*(x) + \alpha \eta(x)
\end{align}$

where $\eta(x)$ is some function that vanishes at the endpoints: $\eta(x_1) = \eta(x_2) = 0$.  We can derive the Euler-Lagrange equations by minimizing $S(\alpha)$ with respect to $\alpha$.

Here we visualize this problem by considering a particular $S$, choosing among some possible $\eta(x)$ definitions, and seeing how $S$ is minimized with respect to $\alpha$.  We will also allow for an incorrect determination of $y^*(x)$, in which case we expect that the minimum alpha will give us a reasonable reproduction of the true $y^*(x)$.  The variation of $\alpha$ and the choice of functions will be made using widgets from `ipywidgets`.

## Looking at a plot of the functional evaluation versus $\alpha$

\[We'll use `%matplotlib notebook` so that we can modify figures without redrawing them.\]

```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt

import ipywidgets as widgets
from IPython.display import display

```

### Functional from Taylor problem 6.9

This problem states: "Find the equation of the path from the origin $O$ to the point $P(1,1)$ in the $xy$ plane that makes the integral $\int_O^P (y'^2 + y y' + y^2)$ stationary.  The answer from solving the Euler-Lagrange equation is $y^*(x) = \sinh(x)/\sinh(1)$.

```python
def y_star(x):
    """Path that minimizes the functional in Taylor problem 6.9."""
    return np.sinh(x) / np.sinh(1.)
```

```python
delta_x = 0.001
x_pts = np.arange(0., 1., delta_x)

fig = plt.figure(figsize=(6,3),
                 num='Visualizing stationary paths of a functional')
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)

def setup_figure():
    ax1.set_title('Show paths')
    ax1.plot(x_pts, y_star(x_pts), color='black', lw=2)
    ax1.set_xlabel('x')
    ax1.set_ylabel('y(x)')

    ax2.set_title('Evaluate functional')
    ax2.set_xlabel(r'$\alpha$')
    ax2.set_ylabel('functional')
    ax2.set_xlim(-0.4, 0.4)
    ax2.set_ylim(1.5, 3.)
    #ax2.axvline(0., color='black', alpha=0.3)
    ax2.axhline(evaluate_functional(x_pts, y_star(x_pts)),
                                    color='black', alpha=0.3)

    fig.tight_layout()

def evaluate_functional(x_pts, y_pts):
    """Given arrays of x and y points, evaluate the functional from 6.9."""
    # The numpy gradient function takes the derivative of an array y_pts
    #  that is a function of points x in x_pts.
    y_deriv_pts = np.gradient(y_pts, x_pts)
    f = y_deriv_pts**2 + y_pts * y_deriv_pts + y_pts**2

    # Use the numpy trapezoid rule (trapz) to do the integral over f.
    return np.trapz(f, x_pts)

def make_path(alpha, ax1_passed, ax2_passed,
              base_function='exact', eta_function='sine'):
    """Given a base function, which may be the exact y^*(x) or a guess that
       is not correct, generate and plot the path corresponding to adding
       alpha*eta(x) to the base function, with eta(x) chosen among some
       functions that vanish at the endpoints in x.
    """
    # map x_pts to zero to 1 (it may already be there)
    x_mapped_pts = (x_pts - x_pts[0]) / (x_pts[-1] - x_pts[0])

    # Choices for the base function
    if (base_function == 'exact'):
        base = lambda x : y_star(x)
    elif (base_function == 'guess 1'):
        base = lambda x : np.sinh(2.*x) / np.sinh(2.)
    elif (base_function == 'guess 2'):
        base = lambda x : x**3

    if (eta_function == 'sine'):
        eta = lambda x : np.sin(np.pi * x)
    elif (eta_function == 'parabola'):
        eta = lambda x : 4. * x * (1. - x)

    y_new_pts = base(x_pts) + alpha * eta(x_mapped_pts)

    ax1_passed.plot(x_pts, y_new_pts, color='red', lw=1)
    ax2_passed.plot(alpha, evaluate_functional(x_pts, y_new_pts), '.',
                    color='red')

def reset_graph(event):
    ax1.clear()
    ax2.clear()
    setup_figure()

button = widgets.Button(
    description='reset graph'
)
button.on_click(reset_graph)

widgets.interact(make_path,
                 alpha=widgets.FloatSlider(min=-1., max=1., step=.05,
                                           value=0.0, description=r'$\alpha$',
                                           continuous_update=False),
                 ax1_passed=widgets.fixed(ax1),
                 ax2_passed=widgets.fixed(ax2),
                 base_function=widgets.Dropdown(options=['exact', 'guess 1',
                                                         'guess 2'],
                                                value='exact',
                                                description='base function'),
                 eta_function=widgets.Dropdown(options=['sine', 'parabola'],
                                               value='sine',
                                               description=r'$\eta(x)$')
                )

setup_figure()
button
```

```python

```


```python
%matplotlib inline
```

```python
from sympy import *
from sympy.plotting import plot_implicit

init_printing()
x = Symbol('x')
y = Symbol('y')
f =  x ** 2 / 25 + y ** 2 / 9  - 1

plot_implicit(f, (x, -10, 10), (y, -10, 10))
```

```python
solve(Eq(f, 0), y ** 2)
```

```python
dydx = diff((- 9 * x ** 2 / 25 + 9), x) / 2
dydx
```

```python
g = Function('g')
g = dydx * (x - 1) - y + f(x)
```

```python
f.subs([x, 1])
```

```python
g = x + y
g.subs([x, 1])
```

```python

```


# Vectorizing t-test and F-tests for unequal variances

Almost all modern data science tasks begin with exploratory data analysis ([EDA](https://en.wikipedia.org/wiki/Exploratory_data_analysis)) phase. Visualizing summary statistics and testing for associations forms the basis of hypothesis generation and subsequent exploration and modeling. Applied statisticians need to be careful not to over-interpret the results of EDA since the p-values generated during this phase do not correspond to their formal definition when used in a highly proscribed scenario. Speed is often an asset during EDA. I recently encountered the problem of needing to assess thousands of AUROC statistics (discussed in the past [here](http://www.erikdrysdale.com/auc_CI/) and [here](http://www.erikdrysdale.com/auc_max/)), and found the bootstrapping procedure to be too slow for high-throughput assessment. By relying on the asymptotic normality of the AUROC statistic (which is an instance of the [Mann-Whitney U-test](https://en.wikipedia.org/wiki/Mann–Whitney_U_test)), rapid inference could be performed because an analytic solution was available. However, I needed to develop code that could properly address the bottlenecks of my analysis:

1. Use only the moments of the data (mean, variance, and sample size)
2. (Possibly) accounting for unequal variances
3. Vectorizing all functions

In the rest of the post, I'll provide simple functions in `python` that will vectorize the [Student's t-test](https://en.wikipedia.org/wiki/Student%27s_t-test) and the [F-test](https://en.wikipedia.org/wiki/F-test) for the multiple comparisons problem. Each of these functions will rely on only the first two moments of the data distribution (plus the sample size). Using only the sufficient statistics of the data helps to reduce the memory overhead that other functions normally have. Means and variances can be computed quickly using methods that are already part of `pandas` and `numpy` classes. The functions in this post will also be able to account for unequal variances, which to best of my knowledge, is not available in existing `python` packages for the F-test.

## (1) Student's t-test for equal means

Suppose there are two normally distributed samples: $x = (x_1, \dots, x_n) \sim N(\mu_x, \sigma^2_x)$ and $y=(y_1,\dots,y_m)\sim N(\mu_y,\sigma^2_y)$, and we would like to test the null hypothesis that $H_0: \mu_x = \mu_y$. If the variances of the distributions were known in practice than the average difference of the two means would have a normal distribution,

$$
\begin{align*}
\frac{\bar{x} - \bar{y}}{\sqrt{\sigma^2_x/n + \sigma^2_y/m}} \sim N(0,1) \hspace{2mm} | \hspace{2mm} H_0 \text{ is true},
\end{align*}
$$

So that a test statistic with a known distribution could be easily constructed. However, since the variance of the two distributions needs to be estimated in practice, the statistic seen above would actually be the ratio of a normal to a chi-squared distribution, in other words a [Student's t-distribution](https://en.wikipedia.org/wiki/Student's_t-distribution):

$$
\begin{align}
d &= \frac{\bar{x} - \bar{y}}{\sqrt{\hat\sigma^2_x/n + \hat\sigma^2_y/m}} \label{eq:dstat} \\
d &\sim t(\nu).
\end{align}
$$

When the variances are not equivalent, some modifications need to be made to the degrees of freedom parameter ($\nu$), using now classic derivations from [Welch](https://www.jstor.org/stable/2332510?seq=1) in 1947:

$$
\begin{align*}
\nu &= \begin{cases}
n + m - 2 & \text{ if } \sigma^2_x = \sigma^2_y \\
\frac{(\hat\sigma^2_x/n + \hat\sigma^2_y/m)^2}{(\hat\sigma^2_x/n)^2/(x-1)+(\hat\sigma^2_y/m)^2/(m-1)} & \text{ if } \sigma^2_x \neq \sigma^2_y
\end{cases}.
\end{align*}
$$

Because the test statistic $d$ is only a function of the first two moments,

$$
\begin{align*}
d = f(\bar x, \bar y, \hat\sigma^2_x, \hat\sigma^2_y, n, m),
\end{align*}
$$

A function can written that takes uses only these sufficient statistics from the data. The code block below will provide the first function `tdist_2dist` to carry out the testing and return the test statistic and associated p-values from a two-sided hypothesis test.

```python
# Import modules needed to reproduce results
import os
import plotnine
from plotnine import *
import pandas as pd
from scipy import stats
import numpy as np
from statsmodels.stats.proportion import proportion_confint as prop_CI

def tdist_2dist(mu1, mu2, se1, se2, n1, n2, var_eq=False):
    var1, var2 = se1**2, se2**2
    num = mu1 - mu2
    if var_eq:
        nu = n1 + n2 - 2
        sp2 = ((n1-1)*var1 + (n2-1)*var2) / nu
        den = np.sqrt(sp2*(1/n1 + 1/n2))
    else:
        nu = (var1/n1 + var2/n2)**2 / ( (var1/n1)**2/(n1-1) + (var2/n2)**2/(n2-1) )
        den = np.sqrt(var1/n1 + var2/n2)
    dist_null = stats.t(df=nu)
    tstat = num / den
    pvals = 2*np.minimum(dist_null.sf(tstat), dist_null.cdf(tstat))
    return tstat, pvals

# Useful short wrappers for making row or columns vectors
def rvec(x):
    return np.atleast_2d(x)

def cvec(x):
    return rvec(x).T
```

As a rule, I always conduct statistical simulations to make sure the functions I have written actually perform the way I expect them to when the null is known. If you can't get your method to work on a data generating procedure of your choosing, it should not leave the statistical laboratory! In the simulations below, $\mu_y = 0$, and $\mu_x$ will vary from zero to 0.2. At the same time, both variance homoskedasticity ($\sigma_y = \sigma_x$) and heteroskedasticity ($\sigma_y \neq \sigma_x$) will be assessed. To further ensure the approach works, the respective sample sizes, $n$ and $m$, for each of the `nsim`=100K experiments will be a random integer between 25 and 75. In order to avoid an inner loop and rely of pure `numpy` vectorization, a data matrix of dimension 75 x 100000 will be generated. To account for the different sample sizes, if $n$ or $m$ is less than 75, the corresponding difference in rows will be set as a missing value `np.NaN`. The `np.nanmean` and `np.nanstd` functions will be used to handle missing values.

Note that in all of the subsequent simulations, the type-I error rate target will be fixed to 5% ($\alpha=0.05$), and 100K simulations will be run.

```python
# Parameters of simulations
nsim = 100000
alpha = 0.05
nlow, nhigh = 25, 75
n1, n2 = np.random.randint(nlow, nhigh+1, nsim), np.random.randint(nlow, nhigh+1, nsim)
se1, se2 = np.exp(np.random.randn(nsim)), np.exp(np.random.randn(nsim))
mu_seq = np.arange(0,0.21,0.01)
tt_seq, method_seq = np.repeat(['eq','neq'],2), np.tile(['neq','eq'],2)
holder = []
np.random.seed(1234)
for mu in mu_seq:
    # Generate random data
    x1 = mu + se1*np.random.randn(nhigh, nsim)
    x2a = se1 * np.random.randn(nhigh, nsim)
    x2b = se2 * np.random.randn(nhigh, nsim)
    idx = np.tile(np.arange(nhigh),[nsim,1]).T
    # Find which rows to set to missing
    idx1, idx2 = idx < rvec(n1), idx < rvec(n2)
    x1, x2a, x2b = np.where(idx1, x1, np.nan), np.where(idx2, x2a, np.nan), np.where(idx2, x2b, np.nan)
    mu_hat1, mu_hat2a, mu_hat2b  = np.nanmean(x1, 0), np.nanmean(x2a, 0), np.nanmean(x2b, 0)
    se_hat1, se_hat2a, se_hat2b = np.nanstd(x1, 0, ddof=1), np.nanstd(x2a, 0, ddof=1), np.nanstd(x2b, 0, ddof=1)
    # Calculate statistics and p-values
    tstat_neq_a, pval_neq_a = tdist_2dist(mu_hat1, mu_hat2a, se_hat1, se_hat2a, n1, n2, False)
    tstat_eq_a, pval_eq_a = tdist_2dist(mu_hat1, mu_hat2a,  se_hat1, se_hat2a, n1, n2, True)
    tstat_neq_b, pval_neq_b = tdist_2dist(mu_hat1, mu_hat2b, se_hat1, se_hat2b, n1, n2, False)
    tstat_eq_b, pval_eq_b = tdist_2dist(mu_hat1, mu_hat2b, se_hat1, se_hat2b, n1, n2, True)
    # Find hypothesis rejection probability
    power_neq_a, power_eq_a = np.mean(pval_neq_a < alpha), np.mean(pval_eq_a < alpha)
    power_neq_b, power_eq_b = np.mean(pval_neq_b < alpha), np.mean(pval_eq_b < alpha)
    power_seq = np.array([power_neq_a, power_eq_a, power_neq_b, power_eq_b])
    holder.append(pd.DataFrame({'mu':mu,'tt':tt_seq,'method':method_seq, 'power':power_seq}))
# Power comparison
di_method = {'eq':'Equal','neq':'Not Equal'}
res_power = pd.concat(holder).assign(nsim=nsim)
res_power[['tt','method']] = res_power[['tt','method']].apply(lambda x: x.map(di_method))
res_power = res_power.rename(columns={'tt':'Variance'}).assign(nreject=lambda x: (x.power*x.nsim).astype(int))
res_power = pd.concat([res_power.drop(columns=['nsim','nreject']),
            pd.concat(prop_CI(count=res_power.nreject,nobs=nsim,method='beta'),1)],1)
res_power.rename(columns={0:'lb',1:'ub'}, inplace=True)

plotnine.options.figure_size = (8, 3.5)
gg_power_ttest = (ggplot(res_power,aes(x='mu',y='power',color='method')) +
                  theme_bw()  + geom_line() +
                  geom_hline(yintercept=0.05,linetype='--') +
                  scale_color_discrete(name='Variance assumption') +
                  geom_linerange(aes(ymin='lb',ymax='ub')) +
                  ggtitle('Vertical lines show 95% CI') +
                  labs(y='Prob. of rejecting null',x='Mean difference') +
                  facet_wrap('~Variance',labeller=label_both) +
                  theme(legend_position=(0.5,-0.1),legend_direction='horizontal'))
gg_power_ttest
```

Figure 1 above shows that the `tdist_2dist` function is working as expected. When the variances of $x$ and $y$ are equivalent, there is no difference in performance between approaches. When the mean difference is zero, the probability of rejecting the null is exactly equivalent to the level of the test (5%). However, when the variances differ, using the degrees of freedom calculation assuming they are equal leads to an inflated type-I error rate. Whereas using the adjustment from [Welch's t-test](https://en.wikipedia.org/wiki/Welch%27s_t-test) gets to the right nominal level.

## (2) Checking power calculations

After checking that function's test-statistic has the right nominal coverage on simulated data, I find is useful to check whether the power of the test can be predicted for different values of the alternative hypothesis. For some test statistics, this is not possible to do analytically, since the distribution of the test statistic under the alternative may not be known. However, for the student-t distribution, a difference in true means amounts to a [noncentral t-distribution](https://en.wikipedia.org/wiki/Noncentral_t-distribution).

$$
\begin{align*}
T &= \frac{Z + c}{\sqrt{V/\nu}} \\
T &\sim \text{nct}(\nu,c) \\
Z&\sim N(0,1), \hspace{3mm} V\sim \chi^2(\nu), \hspace{3mm} \mu \neq 0
\end{align*}
$$

The statistic $d$ from \eqref{eq:dstat} can be modified to match the noncentral t-distribution:

$$
\begin{align*}
d + \underbrace{\frac{\mu_x - \mu_y}{\sqrt{\sigma^2_x/n + \sigma^2_y/m}}}_{c}.
\end{align*}
$$

The power simulations below will fix $n=25$, $m=75$, and unit variances when $\sigma_x=\sigma_y$ and $\sigma_x=1$ and $\sigma_y=2$ in the heteroskedastic case.

```python
n1, n2 = 25, 75
se1 = 1
se2a, se2b = se1, se1 + 1
var1, var2a, var2b = se1**2, se2a**2, se2b**2
# ddof under different assumptions
nu_a = n1 + n2 - 2
nu_b = (var1/n1 + var2b/n2)**2 / ( (var1/n1)**2/(n1-1) + (var2b/n2)**2/(n2-1) )
mu_seq = np.round(np.arange(0, 1.1, 0.1),2)

# Pre-calculate power
crit_ub_a, crit_lb_a = stats.t(df=nu_a).ppf(1-alpha/2), stats.t(df=nu_a).ppf(alpha/2)
crit_ub_b, crit_lb_b = stats.t(df=nu_b).ppf(1-alpha/2), stats.t(df=nu_b).ppf(alpha/2)
lam_a = np.array([mu/np.sqrt(var1*(1/n1 + 1/n2)) for mu in mu_seq])
lam_b = np.array([mu/np.sqrt((var1/n1 + var2b/n2)) for mu in mu_seq])
dist_alt_a, dist_alt_b = stats.nct(df=nu_a, nc=lam_a), stats.nct(df=nu_b, nc=lam_b)
power_a = (1-dist_alt_a.cdf(crit_ub_a)) + dist_alt_a.cdf(crit_lb_a)
power_b = (1-dist_alt_b.cdf(crit_ub_b)) + dist_alt_b.cdf(crit_lb_b)
dat_theory = pd.concat([pd.DataFrame({'mu':mu_seq,'theory':power_a,'method':'eq'}),
                        pd.DataFrame({'mu':mu_seq,'theory':power_b,'method':'neq'})])

# Run simulations to confirm
np.random.seed(1234)
holder = []
for mu in mu_seq:
    x1 = mu + se1 * np.random.randn(n1, nsim)
    x2a = se2a * np.random.randn(n2, nsim)
    x2b = se2b * np.random.randn(n2, nsim)
    mu_hat1, mu_hat2a, mu_hat2b  = x1.mean(0), x2a.mean(0), x2b.mean(0)
    se_hat1, se_hat2a, se_hat2b = x1.std(0,ddof=1), x2a.std(0, ddof=1), x2b.std(0, ddof=1)
    stat_a, pval_a = tdist_2dist(mu_hat1, mu_hat2a, se_hat1, se_hat2a, n1, n2, var_eq=True)
    stat_b, pval_b = tdist_2dist(mu_hat1, mu_hat2b, se_hat1, se_hat2b, n1, n2, var_eq=False)
    reject_a, reject_b = np.mean(pval_a < 0.05), np.mean(pval_b < 0.05)
    holder.append(pd.DataFrame({'mu': mu,'method':['eq','neq'], 'power': [reject_a, reject_b]}))
res_theory = pd.concat(holder).merge(dat_theory).sort_values(['method','mu']).reset_index(None, True)
res_theory = res_theory.assign(nreject=lambda x: (x.power*nsim).astype(int))
res_theory = pd.concat([res_theory.drop(columns='nreject'),
                        pd.concat(prop_CI(count=res_theory.nreject,nobs=nsim,method='beta'),1)],1)
res_theory.rename(columns={0:'lb',1:'ub','method':'Variance'}, inplace=True)
res_theory = res_theory.assign(Variance=lambda x: x.Variance.map(di_method))

plotnine.options.figure_size = (8, 3.5)
gg_power_theory = (ggplot(res_theory,aes(x='theory',y='power')) +
                   theme_bw() + geom_point() +
                   geom_linerange(aes(ymin='lb',ymax='ub')) +
                   facet_wrap('~Variance', labeller=label_both) +
                   theme(legend_position=(0.5, -0.1), legend_direction='horizontal') +
                   labs(x='Expected power',y='Actual power') +
                   scale_y_continuous(limits=[0,1]) + scale_x_continuous(limits=[0,1]) +
                   geom_abline(slope=1,intercept=0,color='blue',linetype='--'))
gg_power_theory
```

Figure 2 shows that the power calculations line up exactly with the analytical expectations for both equal and unequal variances. Having thoroughly validated the type-I and type-II errors of this function we can now move onto testing whether the means from multiple normal distributions are equal.

## (3) F-test for equality of means

Suppose there are $K$ normal data vectors: $x_1=(x_{1,1},\dots,x_{1,n_1})$ to $x_k=(x_{k,1},\dots,x_{1,n_k})$, and we want to test the null hypothesis of $\mu_1 = \mu_2 = \dots = \mu_K$ against an alternative hypothesis that there is at least 1 inequality in the means, where $x_{k,i} \sim N(\mu_k,\sigma^2_k)$. As before, the variances of each vector may or may not be equal. When the variances are equal, the sum of squared differences between the total mean and any one group mean will be chi-square. Similarly, the sum of the sample variances will also have a chi-square distribution. Hence, the F-test for equality of means is the ratio of the variation "between" versus "within" the groups,

$$
\begin{align*}
R &= \frac{\frac{1}{K-1}\sum_{k=1}^K n_k (\bar x_k - \bar x)^2 }{\frac{1}{N-K}\sum_{k=1}^K (n_k - 1)\hat\sigma^2_k}, \\
R &\sim F(K-1, N-K) \hspace{3mm} \text{ if } \sigma^2_k = \sigma^2 \hspace{3mm} \forall k \in \{1,\dots,K\}
\end{align*}
$$

Where $N = \sum_k n_k$. To account for heteroskedasticity in the data (i.e. non-equal variances), both the test and degrees of freedom need to be modified using an [approach](https://doi.org/10.2307/2332579) Welch proposed in 1951.

$$
\begin{align*}
R_W &= \frac{\frac{1}{K-1}\sum_{k=1}^K w_k (\bar x_k - \bar x_w)^2 }{1 + \frac{2}{3}((K-2)\nu)}, \\
w_k &= n_k / \hat\sigma^2_k \\
\bar x_w &= \frac{\sum_{k=1}^K w_k \bar x_k}{\sum_{k=1}^K w_k}\\
\nu &= \frac{3\cdot \sum_{k=1}^K \Bigg[ \frac{1}{n_k - 1} \Big( 1 - \frac{w_k}{\sum_{k=1}^K w_k} \Big)^2 \Bigg]^2}{K^2-1} \\
R_W &\sim F(K-1, 1/\nu) \hspace{3mm} \text{ if } \sigma^2_k \neq \sigma^2_{-k} \hspace{3mm} \text{for at least one }k
\end{align*}
$$

The `fdist_anova` function below carries out an F-test for the equality of means using only the empirical means, standard deviations, and sample sizes for either variance assumption. In `R` this would be equivalent to using `aov` for equal variances or `oneway.test` for unequal variances. In `python`, it will replicate the `scipy.stats.f_oneway` function (for equal variances). I am unaware of a `python` function that does a Welch-adjustment (if you know please message me and I will provide an update with this information). As before, because the function only relies on the moments of the data, it can be fully vectorized to handle matrices of means, variances, and sample sizes.

The simulation below assesses how well the two F-test approaches (homoskedasticity vs heteroskedasticity) do when the ground truth variances are either all equal or vary. To vary the signal in the data, I generate the $K$ different means from $(-\mu,\dots,0,\dots,\mu)$, where $\mu$ is referred to as "mean dispersion" in the subsequent figures.

```python
def fdist_anova(mus, ses, ns, var_eq=False):
    lshape = len(mus.shape)
    assert lshape <= 2
    assert mus.shape == ses.shape
    if len(ns.shape) == 1:
        ns = cvec(ns.copy())
    else:
        assert ns.shape == mus.shape
    if lshape == 1:
        mus = cvec(mus.copy())
        ses = cvec(ses.copy())
    vars = ses ** 2  # variance
    n, k = ns.sum(0), len(ns)  # Total samples and groups
    df1, df2 = (k - 1), (n - k)
    if var_eq:  # classical anova
        xbar = np.atleast_2d(np.sum(mus * ns, 0) / n)
        vb = np.sum(ns*(xbar - mus)**2,0) / df1  # numerator is variance between
        vw = np.sum((vars * (ns - 1)), 0) / df2  # den is variance within
        fstat = vb / vw
        pval = stats.f(dfn=df1,dfd=df2).sf(fstat)
    else:
        w = ns / vars
        xbar = np.sum(w * mus, 0) / np.sum(w,0)
        num = np.sum(w * (xbar - mus) ** 2,0) / df1
        v = 3*np.sum((1-w/w.sum(0))**2 / (ns-1),0) / (k**2 - 1)
        den = 1 + 2*((k-2)*v)/3
        fstat = num / den
        pval = stats.f(dfn=df1, dfd=1/v).sf(fstat)
    return fstat, pval

nlow, niter = 25, 5
k_seq = [5, 7, 9]
disp_seq = np.round(np.arange(0, 0.51, 0.1),2)
dgp_seq = np.repeat(['eq', 'neq'], 2)
method_seq = np.tile(['eq', 'neq'], 2)

holder = []
np.random.seed(1)
for k in k_seq:
    n_seq = np.arange(nlow, nlow+k * niter, niter)
    n_seq = np.tile(n_seq, [nsim, 1]).T
    nhigh = np.max(n_seq)
    dim_3d = [1, 1, k]
    for disp in disp_seq:
        mu_k = np.linspace(-disp, disp, num=k)
        se_k1 = np.repeat(1,k).reshape(dim_3d)
        se_k2 = np.exp(np.random.randn(k)).reshape(dim_3d)
        X1 = mu_k + se_k1 * np.random.randn(nhigh,nsim,k)
        X2 = mu_k + se_k2 * np.random.randn(nhigh, nsim, k)
        idx = np.tile(np.arange(nhigh),[k,nsim,1]).T <= np.atleast_3d(n_seq).T
        X1, X2 = np.where(idx, X1, np.nan), np.where(idx, X2, np.nan)
        # Calculate means and variance : (k x nsim)
        mu_X1, mu_X2 = np.nanmean(X1, 0).T, np.nanmean(X2, 0).T
        se_X1, se_X2 = np.nanstd(X1, 0, ddof=1).T, np.nanstd(X2, 0, ddof=1).T
        assert n_seq.shape == mu_X1.shape == se_X1.shape
        # Calculate significance
        fstat_eq1, pval_eq1 = fdist_anova(mus=mu_X1, ses=se_X1, ns=n_seq, var_eq=True)
        fstat_neq1, pval_neq1 = fdist_anova(mus=mu_X1, ses=se_X1, ns=n_seq, var_eq=False)
        fstat_eq2, pval_eq2 = fdist_anova(mus=mu_X2, ses=se_X2, ns=n_seq, var_eq=True)
        fstat_neq2, pval_neq2 = fdist_anova(mus=mu_X2, ses=se_X2, ns=n_seq, var_eq=False)
        reject_eq1, reject_neq1 = np.mean(pval_eq1 < alpha), np.mean(pval_neq1 < alpha)
        reject_eq2, reject_neq2 = np.mean(pval_eq2 < alpha), np.mean(pval_neq2 < alpha)
        reject_seq = [reject_eq1, reject_neq1, reject_eq2, reject_neq2]
        tmp = pd.DataFrame({'k':k,'disp':disp,'dgp':dgp_seq,'method':method_seq,'reject':reject_seq})
        # print(tmp)
        holder.append(tmp)
res_f = pd.concat(holder).reset_index(None,True)
res_f[['dgp','method']] = res_f[['dgp','method']].apply(lambda x: x.map(di_method),0)
res_f.rename(columns={'dgp':'Variance'}, inplace=True)

plotnine.options.figure_size = (8, 6)
gg_fdist = (ggplot(res_f, aes(x='disp',y='reject',color='method.astype(str)')) +
            theme_bw() + geom_line() + geom_point() +
            facet_grid('k~Variance',labeller=label_both) +
            labs(x='Mean dispersion',y='Prob. of rejecting null') +
            geom_hline(yintercept=0.05,linetype='--') +
            scale_y_continuous(limits=[0,1]) +
            scale_color_discrete(name='Variance assumption'))
gg_fdist
```

The simulations in Figure 3 show a similar finding to the that of t-test: when the ground truth variances are equal, there is almost no differences between the tests, and an expected 5% false positive rate occurs when the means are equal. However, for the unequal variance situation, the assumption of homoskedasticity leads to an inflated type-I error rate (as was the case for the t-test), but also lower power when the null is false (which was not the case for the t-test). Using the Welch adjustment is better in both cases. The one surprising finding is that the power of the test is not monotonically increasing in the heteroskedastic case. I am not completely sure why this is the case. One theory could be that since a higher mean dispersion leads to a higher variance of $\bar{x}_w$, the ratio of the degrees of freedom may be more stable for lower values of $\mu$, leading to a more consistent rejection rate.

## (4) Quick sanity checks

After confirming the frequentist properties of a test statistic, it is worthwhile checking the results of any custom function to similar functions from other libraries. The `tdist_2dist` function will be compared to it's `scipy` counterpart on the [Iris](https://en.wikipedia.org/wiki/Iris_flower_data_set) dataset.

```python
from sklearn import datasets
ix, iy = datasets.load_iris(return_X_y=True)
v1, v2 = ix[:,0], ix[:,1]
k = 1
all_stats = [stats.ttest_ind(v1, v2, equal_var=True)[k],
             tdist_2dist(v1.mean(), v2.mean(), v1.std(ddof=1), v2.std(ddof=1), len(v1), len(v2), var_eq=True)[k],
             stats.ttest_ind(v1, v2, equal_var=False)[k],
             tdist_2dist(v1.mean(), v2.mean(), v1.std(ddof=1), v2.std(ddof=1), len(v1), len(v2), var_eq=False)[k]]
pd.DataFrame({'test':'t-test',
              'method':np.tile(['scipy','custom'],2),
              'pval':all_stats})
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>test</th>
      <th>method</th>
      <th>pval</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>t-test</td>
      <td>scipy</td>
      <td>7.027919e-112</td>
    </tr>
    <tr>
      <th>1</th>
      <td>t-test</td>
      <td>custom</td>
      <td>7.027919e-112</td>
    </tr>
    <tr>
      <th>2</th>
      <td>t-test</td>
      <td>scipy</td>
      <td>1.459543e-96</td>
    </tr>
    <tr>
      <th>3</th>
      <td>t-test</td>
      <td>custom</td>
      <td>1.459543e-96</td>
    </tr>
  </tbody>
</table>
</div>

So far so good. Next, we'll use `rpy2` to get the results in `R` which supports equal and unequal variances with two different functions.

```python
import rpy2.robjects as robjects

moments_x = pd.DataFrame({'x':ix[:,0],'y':iy}).groupby('y').x.describe()[['mean','std','count']]

all_stats = [np.array(robjects.r('summary(aov(Sepal.Length~Species,iris))[[1]][1, 5]'))[0],
             fdist_anova(moments_x['mean'], moments_x['std'], moments_x['count'], var_eq=True)[1][0],
             np.array(robjects.r('oneway.test(Sepal.Length~Species,iris)$p.value'))[0],
             fdist_anova(moments_x['mean'], moments_x['std'], moments_x['count'], var_eq=False)[1][0]]
pd.DataFrame({'test':'F-test',
              'method':np.tile(['R','custom'],2),
              'pval':all_stats})
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>test</th>
      <th>method</th>
      <th>pval</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>F-test</td>
      <td>R</td>
      <td>1.669669e-31</td>
    </tr>
    <tr>
      <th>1</th>
      <td>F-test</td>
      <td>custom</td>
      <td>1.669669e-31</td>
    </tr>
    <tr>
      <th>2</th>
      <td>F-test</td>
      <td>R</td>
      <td>1.505059e-28</td>
    </tr>
    <tr>
      <th>3</th>
      <td>F-test</td>
      <td>custom</td>
      <td>1.505059e-28</td>
    </tr>
  </tbody>
</table>
</div>

Once again the results are identical to the benchmark functions.

## (5) Application to AUROC inference

The empirical AUROC has an asymptotically normal distribution. Consequently, the difference between two AUROCs will also have an asymptotically normal distribution. For small sample sizes, the [Hanley and McNeil](http://www.med.mcgill.ca/epidemiology/hanley/software/Hanley_McNeil_Radiology_82.pdf) adjustment to the AUROC standard error will obtain slightly better coverage. For a review of the notation and meaning of the AUROC, see a previous post [here](http://www.erikdrysdale.com/auc_CI).

$$
\begin{align*}
AUC &= \frac{1}{n_1 n_0} \sum_{i: y_i = 1} \sum_{j: y_j=0} I(s_i > s_j) \\
\sigma_{N} &= \sqrt{\frac{n_1 + n_0 + 1}{12\cdot n_1 n_0}} \\
\sigma_{HM} &= \sqrt{\frac{AUC\cdot (1-AUC) + q_1 + q_0}{n_1 n_0}} \\
q_1 &= (n_1 - 1)\cdot ( AUC / (2-AUC) - AUC^2) \\
q_0&= (n_0- 1)\cdot ( AUC^2 / (1+AUC) - AUC^2)
\end{align*}
$$

The standard error from the normal approximation ($\sigma_N$) is only a function of the positive ($n_1$) and negative ($n_0$) class sample sizes whereas the Hanley and McNeil adjustment ($\sigma_{HM}$) uses the empirical AUROC as well. The previous t- and F-tests relied on the fact that the sample mean had a variance that $O(1/n)$ so that $\bar x \sim N(\mu, \sigma^2/n)$. As can be seen from either formula, the sample variance for the AUROC can not be nearly re-written as a function of the sample size. We can still appeal to the t-test, the only difference being that the sample size is built into the variance estimate:

$$
\begin{align*}
\frac{AUC_A - AUC_B}{\sqrt{\sigma^2_{HM_A} + \sigma^2_{HM_B}}} &\sim N(0,1) \hspace{3mm} \text{ if $H_0$ is true}
\end{align*}
$$

In the simulation below, scores will come from one of two distributions. The negative class will have 200 samples drawn from a standard normal ($n_0$). The positive class scores will have 100 samples ($n_1$) drawn from either a standard normal (for the null distribution) and a normal with a mean at or above zero. The difference in AUROCs between these two distributions will be evaluated. Since the null distribution will have an (average) AUROC of 50%, the difference in these distribution will be above zero when the mean from the alternative is greater than zero.

```python
n1, n0 = 100, 200
n = n1 + n0
n1n0 = n1 * n0
mu_seq = np.round(np.arange(0, 1.01, 0.1),2)

def se_auroc_hanley(auroc, n1, n0):
    q1 = (n1 - 1) * ((auroc / (2 - auroc)) - auroc ** 2)
    q0 = (n0 - 1) * ((2 * auroc ** 2) / (1 + auroc) - auroc ** 2)
    se_auroc = np.sqrt((auroc * (1 - auroc) + q1 + q0) / (n1 * n0))
    return se_auroc

def se_auroc_normal(n1, n0):
    return np.sqrt( (n1 + n0 + 1) / (12 * n1 * n0) )

np.random.seed(1)
holder = []
for mu in mu_seq:
    x1_null, x0 = np.random.randn(n1, nsim), np.random.randn(n0, nsim)
    x1 = mu + np.random.randn(n1, nsim)
    x, x_null = np.concatenate((x1, x0)), np.concatenate((x1_null, x0))
    auc = (np.sum(stats.rankdata(x, axis=0)[:n1],0) - n1*(n1+1)/2) / n1n0
    auc_null = (np.sum(stats.rankdata(x_null, axis=0)[:n1], 0) - n1 * (n1 + 1) / 2) / n1n0
    se_HM, se_null_HM = se_auroc_hanley(auc, n1, n0), se_auroc_hanley(auc_null, n1, n0)
    se_N = se_auroc_normal(n1, n0)
    # Do pairwise t-test
    dauc = auc - auc_null
    t_score_HM = dauc / np.sqrt(se_HM**2 + se_null_HM**2)
    t_score_N = dauc / np.sqrt(2 * se_N**2)
    dist_null = stats.t(df=2*n - 2)
    pval_HM = 2 * np.minimum(dist_null.sf(t_score_HM), dist_null.cdf(t_score_HM))
    pval_N = 2 * np.minimum(dist_null.sf(t_score_N), dist_null.cdf(t_score_N))
    reject_HM, reject_N = np.mean(pval_HM < alpha), np.mean(pval_N < alpha)
    tmp = pd.DataFrame({'method':['HM','N'],'mu':mu, 'reject':[reject_HM, reject_N]})
    holder.append(tmp)
# Merge and analyse
res_auc = pd.concat(holder).reset_index(None, True)
res_auc = res_auc.assign(auc=lambda x: stats.norm.cdf(x.mu/np.sqrt(2)),
                         nreject=lambda x: (x.reject*nsim).astype(int))
res_auc = pd.concat([res_auc.drop(columns='nreject'),
            pd.concat(prop_CI(count=res_auc.nreject,nobs=nsim,method='beta'),1)],1)
res_auc.rename(columns={0:'lb',1:'ub'},inplace=True)

# plot
plotnine.options.figure_size = (5, 4)
gg_auc = (ggplot(res_auc,aes(x='auc',y='reject',color='method')) + theme_bw() +
          geom_line() +
          labs(x='Alternative hypothesis AUROC',y='Prob. of rejecting null') +
          geom_hline(yintercept=0.05,linetype='--') +
          geom_linerange(aes(ymin='lb',ymax='ub')) +
          scale_color_discrete(name='Method',labels=['Hanley-McNeil','Normal']))
gg_auc
```

Figure 4 shows that the standard errors from both methods yield almost identical results. Furthermore, the standard errors are conservative (too large), leading to an under-rejection of the null hypothesis when the null is true (i.e. the alternative hypothesis AUROC is 50%). The alternative hypothesis AUROC needs to reach around 53% before the rejection rate reaches the expected normal level. However, between 53%-70%, the power of the test approaches 100% for this sample size combination.


# Import Packages

```python
%load_ext autoreload
%autoreload 2
```

```python
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import sympy as sym
import statsmodels.api as sm
import typing as tp
```

    Bad key "text.kerning_factor" on line 4 in
    /Users/michaelnowotny/anaconda3/envs/continuous_time_mcmc/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    https://github.com/matplotlib/matplotlib/blob/v3.1.3/matplotlibrc.template
    or from the matplotlib source distribution

```python
from state_space.data_reader import (
    load_monthly_and_annual_ff_factors,
    compute_monthly_returns
)
```

```python
from state_space import (
    SymbolicStateSpaceModelViaMaximumLikelihood,
    LambdaParameterTransformation,
    RectangularParameterTransformation,
    linearize_vector_valued_state_function
)
```

# Load Data

```python
data_directory_name = 'data'
ford_data_filename = os.path.join(data_directory_name, 'F_monthly.csv')
gspc_data_filename = os.path.join(data_directory_name, '^GSPC_monthly.csv')
ff_data_filename = os.path.join(data_directory_name, 'F-F_Research_Data_Factors_monthly.CSV')
```

```python
ford_prices = pd.read_csv(ford_data_filename).set_index('Date')
gspc_prices = pd.read_csv(gspc_data_filename).set_index('Date')
```

Load Fama-French factors

```python
ff_monthly, ff_annual = load_monthly_and_annual_ff_factors(ff_data_filename)
```

# Prepare Data

Compute Ford returns

```python
ford_returns = compute_monthly_returns(ford_prices, name='ford')
```

Compute S&P 500 index returns

```python
gspc_returns = compute_monthly_returns(gspc_prices, name='gspc')
```

Compute excess returns on both

```python
gspc_excess_returns = gspc_returns - ff_monthly['RF']
ford_excess_returns = ford_returns - ff_monthly['RF']
```

Put Ford and S&P 500 returns in a single dataframe

```python
joint_returns = pd.DataFrame({'ford': ford_returns, 'gspc': gspc_returns, 'rf': ff_monthly['RF']}).dropna()
```

```python
joint_excess_returns = pd.DataFrame({'ford': ford_excess_returns, 'gspc': gspc_excess_returns}).dropna()
```

Restrict the time horizon from 1990 to 2004

```python
restricted_joint_excess_returns = joint_excess_returns.query('Date >= 1990 and Date < 2004')
```

# Conditional Factor Model and State-Space Representation

## Stochastic Process Model

Consider a univariate linear factor model with time-varying coefficients, where the factor is the return on an equity index. This model can be written as

$r_t = \alpha_t + \beta_t r_{M, t} + e_t$

$\alpha_{t+1} = \alpha_t + \eta_t$

$\beta_{t+1} = \beta_t + \epsilon_t$

where $r_t$ is the excess return of the stock over period $t$ and $r_{M, t}$ is the excess return of the market over period $t$. The innovations $e$, $\eta$, and $\epsilon$ are both serially and mutually independent and distributed as follows:

$e_t \sim N(0, \sigma^2_e)$

$\eta_t \sim N(0, \sigma^2_{\eta})$

$\epsilon_t \sim N(0, \sigma^2_{\epsilon})$

## State Space Representation

This model can be represented in statespace form as:

$s_t = \begin{pmatrix} \alpha_t \\ \beta_t \end{pmatrix}$ (state),

$y_t = r_t$ (observation)

$s_{t+1} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \cdot s_t + \begin{pmatrix} \eta_t \\ \epsilon_t \end{pmatrix}$

$y_t = \begin{pmatrix} 1 & r_{M, t} \end{pmatrix} \cdot s_t + e_t $

We can further combine the processes

$\begin{pmatrix} s_{t+1} \\ y_t \end{pmatrix} = \delta_t + \Phi_t s_t + u_t$,

where

$\delta_t = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}$, $\Phi_t = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & r_{M, t} \end{pmatrix}$, $u_t = \begin{pmatrix} \eta_t \\ \epsilon_t \\ e_t \end{pmatrix}$

## Representation in SymPy

Initialize symbols

```python
sigma_e, sigma_eta, sigma_epsilon, alpha, beta, r, r_M = \
    sym.symbols('sigma_e, sigma_eta, sigma_epsilon, alpha, beta, r, r_M')
```

Model parameters to be estimated

```python
parameter_symbols = (sigma_e, sigma_eta, sigma_epsilon)
```

Define the state vector

```python
state_vector_symbols = (alpha, beta)
```

Define the vector of observations

```python
observation_vector_symbols = (r, )
```

Define mapping between data symbols and Pandas series/NumPy arrays

```python
data_symbol_to_data_map = {r: restricted_joint_excess_returns['ford'],
                           r_M: restricted_joint_excess_returns['gspc']}
```

Set starting values for the maximum likelihood estimation via numerical optimization

```python
ford_return_std = np.std(restricted_joint_excess_returns['ford'])
parameter_symbols_to_start_parameters_map = {sigma_e: ford_return_std,
                                             sigma_eta: ford_return_std,
                                             sigma_epsilon: ford_return_std}
```

Define transformation functions to ensure that the variance parameters remain positive (the optimizer is unconstrained)

Option A: Using a vectorized transformation function (the same for all parameters)

```python
parameter_transformation = \
    LambdaParameterTransformation(transform_function=lambda x: x**2,
                                  untransform_function=lambda x: x**0.5)
```

Option B: Using individual lower and upper bounds

```python
# parameter_transformation = \
#     RectangularParameterTransformation(parameter_symbols=parameter_symbols,
#                                        parameter_symbol_to_bounds_map={sigma_e: (0, np.inf),
#                                                                        sigma_eta: (0, np.inf),
#                                                                        sigma_epsilon: (0, np.inf)})
```

Construct transition matrix in SymPy

```python
transition_matrix = sym.eye(2)
transition_matrix
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

Construct design matrix in SymPy

```python
design_matrix = sym.Matrix([[1, r_M]])
design_matrix
```

$\displaystyle \left[\begin{matrix}1 & r_{M}\end{matrix}\right]$

We can alternatively express the transition and observation functions as a vector valued functions of state variables and linearize the function. For non-linear transition and observation functions this leads in the extended Kalman filter (EKF).

```python
f = sym.Matrix([[alpha], [beta]])
g = sym.Matrix([[alpha + beta * r_M]])
```

```python
state_intercept_vector_ekf, transition_matrix_ekf = \
    linearize_vector_valued_state_function(f, state_vector_symbols=state_vector_symbols)
print(state_intercept_vector_ekf)
print(transition_matrix_ekf)
```

    Matrix([[0], [0]])
    Matrix([[1, 0], [0, 1]])

```python
observation_intercept_vector_ekf, design_matrix_ekf = \
    linearize_vector_valued_state_function(g, state_vector_symbols=state_vector_symbols)
print(observation_intercept_vector_ekf)
print(design_matrix_ekf)
```

    Matrix([[0]])
    Matrix([[1, r_M]])

Construct selection matrix in SymPy

```python
selection_matrix = sym.eye(2)
selection_matrix
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

Construct state covariance matrix matrix in SymPy

```python
state_covariance_matrix = sym.diagonalize_vector(sym.Matrix([sigma_eta**2, sigma_epsilon**2]))
state_covariance_matrix
```

$\displaystyle \left[\begin{matrix}\sigma_{\eta}^{2} & 0\\0 & \sigma_{\epsilon}^{2}\end{matrix}\right]$

Construct observation covariance matrix matrix in SymPy

```python
observation_covariance_matrix = sym.Matrix([[sigma_e**2]])
```

Compile numeric state space representation for MLE from the symbolic definition

```python
conditional_factor_model = \
    SymbolicStateSpaceModelViaMaximumLikelihood(parameter_symbols=parameter_symbols,
                                                state_vector_symbols=state_vector_symbols,
                                                observation_vector_symbols=observation_vector_symbols,
                                                data_symbol_to_data_map=data_symbol_to_data_map,
                                                parameter_symbols_to_start_parameters_map=parameter_symbols_to_start_parameters_map,
                                                parameter_transformation=parameter_transformation,
                                                design_matrix=design_matrix,
                                                observation_covariance_matrix=observation_covariance_matrix,
                                                selection_matrix=selection_matrix,
                                                state_covariance_matrix=state_covariance_matrix,
                                                transition_matrix=transition_matrix)
```

# Estimation

Fit the model using MLE (recall that we are fitting the three variance parameters)

```python
res = conditional_factor_model.fit(disp=False)
```

## Estimates

```python
print(res.summary())
```

                                            Statespace Model Results
    =======================================================================================================
    Dep. Variable:                                               y   No. Observations:                  168
    Model:             SymbolicStateSpaceModelViaMaximumLikelihood   Log Likelihood                 148.798
    Date:                                         Fri, 16 Apr 2021   AIC                           -291.596
    Time:                                                 21:09:05   BIC                           -282.260
    Sample:                                                      0   HQIC                          -287.806
                                                             - 168
    Covariance Type:                                           opg
    =================================================================================
                        coef    std err          z      P>|z|      [0.025      0.975]
    ---------------------------------------------------------------------------------
    sigma_e           0.0929      0.004     22.945      0.000       0.085       0.101
    sigma_eta         0.0024      0.002      1.342      0.179      -0.001       0.006
    sigma_epsilon     0.1192      0.085      1.398      0.162      -0.048       0.286
    ===================================================================================
    Ljung-Box (Q):                       42.51   Jarque-Bera (JB):                38.55
    Prob(Q):                              0.36   Prob(JB):                         0.00
    Heteroskedasticity (H):               1.09   Skew:                             0.73
    Prob(H) (two-sided):                  0.76   Kurtosis:                         4.86
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

## Filtered and Smoothed State

Filtered $\alpha$

```python
plt.plot(restricted_joint_excess_returns.index, res.filtered_state[0, :]);
```

Smoothed $\alpha$

```python
plt.plot(restricted_joint_excess_returns.index, res.smoothed_state[0, :]);
```

Filtered $\beta$

```python
plt.plot(restricted_joint_excess_returns.index, res.filtered_state[1, :]);
```

Smoothed $\beta$

```python
plt.plot(restricted_joint_excess_returns.index, res.smoothed_state[1, :]);
```

```python

```

notebooks/Conditional Linear Factor Model.ipynb


Estimating The Period of an Oscillator
======================================

This week we'll be investigating the period of a pendulum. This is a problem that has been studied for a *very* long time (e.g., one of earliest workers in the pendulum field was Galeleo himself!). However, most folks (e.g., physics students) just investigate the low amplitude behavior (when the angle between the pendulum and the vertical is pretty small, say 5 or 10 degrees). We are not going to satisfied with such a simple analysis! No, on the contrary, we'll be looking at how the period of the pendulum *varies* with amplitude. We could use the momentum principle as our startign point for this analsysis, as is usually done, but we're also going to be somewhat unconventional and work instead with the energy principle. Note that this same technique works with any non-linear oscilator (think resonance circuits in electronics, or lasers, or molecules). So, lots to learn! This project has two primary components:

1. Integrating a function, that is not integrable, numerically so that it's integral can be used as part of the analysis of a problem. As we work out the exact expression for the period of a pendulum we'll encounter such a function and we'll integrate it numerically.

2. Applying our techniques from last week of fitting functions to data to extract very precise times from a video that we'll collect of the pendulum in motion. This will allow us to compare our analytical approximation and the numerical integral with actual data gathered from a real pendulum.

Let's say we have a system with a potential energy given by some,
possibly complex expression $U(x)$. For any system with position
$x$ and a potential energy $U$ we have:

\begin{equation}
E = {1 \over 2}m\left(\frac{dx}{dt}\right)^2 + U(x).
\end{equation}

If we solve this for $dx/dt$ we get the result we want:
\begin{equation}
 \frac{dx}{dt} = \sqrt{2(E - U(x))/m}.
\end{equation}

or, solving for $dt$ we get:

\begin{equation}
dt = \sqrt{m \over 2 (E-U(x)) } dx.
\end{equation}

If the system is bound we can get the time for one oscillation by
integrating the time for one complete period! If the potential is symmetric
with respect to $x$ then we only have to do 1/4 of a period and multiply
by 4:

\begin{equation}
T = 4 \int_0^{x_{\rm max}} \sqrt{m \over 2 (E-U(x)) } dx.
\end{equation}

Let's apply this idea to the simple pendulum. Use this potential (and use polar coordinates!) to study the simple pendulum:

\begin{equation}
U(\theta) = -mgl {\rm cos}(\theta)
\end{equation}

Use this idea to show that for small
values of $\theta_m$ the result is the same as we got in
General Physics ($T = 2 \pi \sqrt{l/g}$). You can **watch the slides** for this lesson see how to carry out the expansion to the
next order and find the second order correction to this result.
To do the integral it helps to first use: ${\rm cos}\,\theta = 1 - 2 {\rm sin}^2\,(\theta/2)$ and a similar
substitution for ${\rm cos}\,\theta_m$. Then change variables to
${\rm sin}\,\phi = {\rm sin}\,(\theta/2)/
{\rm sin}\,(\theta_m/2)$.Then expand in powers of $\theta_m$ At the
last step before expansion you should have:

\begin{equation}
4 \sqrt {l/g} \int_0^{\pi/2} {d\phi \over {\sqrt{1 - \sin^2\phi \sin^2{\theta_m \over 2}}}}
\end{equation}

The integral here is called the "Complete Elliptic Integral of the First Kind". It's famous. It's impossible to do in closed form.

However we can approximate it in several ways. First, we can evalutate the integral numerically (see Simpon's Rule below) or we can estimate the value of the integral by using a Taylor Expansion WRT a small parameter (in our case the amplitude of the pedulum) to get a series that gives us a correction to the simple theory. We'll do both of these things and comapare the results.

Below is a simple python implementation of [Simpson's Rule.](https://en.wikipedia.org/wiki/Simpson's_rule).

```python
%pylab inline

def simpson(f, a, b, N, args=None):
    """
    Use Simpson's Rule to estimate an integral

    f: function to be integrated
    a, b: lower, upper limits
    N: "half the number of samples" to take.
    args: list of additional arguments (besides 'x') for the function.
    """
    if args is None:
        args=[]
    x=linspace(a,b,2*N)  # sample the function
    h=(b-a)/(2*N-1)
    y=f(*([x] +list(args)))
    evens =  y[2:-2:2]
    odds = y[1:-1:2]
    return (y[0] + y[-1] + 2*odds.sum() + 4*evens.sum())*h/3.0

def K_int(phi, m):
    """
    Integrand for K(m,N) below. This defines the
    "Complete Elliptic Integral of the First Kind".
    """
    return 1.0/sqrt(1-m*sin(phi)**2)

def K(m, N=1000):
    """
    K(m,N) computes the:
    "Complete Elliptic Integral of the First Kind"
    with 2*N samples using the simple Simpson's Rule integrator above.
    """
    return simpson(K_int, 0, pi/2, N=N, args=(m,))

```

    Populating the interactive namespace from numpy and matplotlib

The `scipy.integrate` package has a number of numerical [integrators](https://docs.scipy.org/doc/scipy-0.18.1/reference/integrate.html) that you can study. For the purposes of this example we'll use `quad` which is a simple general purpose integrator. Let's compare that with the Simpson's Rule integrator above.

```python
#
# scipy has an easy "built in" integration routine: integrate.quad
# We'll use this as well for comparison.
#

from scipy.integrate import quad  # import a numerical integrator

siList=[]  # scipy integrate.quad list
saList=[]  # small angle list
srList=[]  # Simpson's Rule, N=1000 list

thetArr = linspace(0.0, 70, 150) # range of theta values

for theta in thetArr:
    thetaRad=theta*pi/180.0
    m=sin(thetaRad/2.0)**2
    simpresult = K(m)
    quadResult = quad(K_int, 0, pi/2, args=(m,))
    smallAngle = (pi/2)*(1.0+m/4)
    srList.append(2*simpresult/pi)
    siList.append(2*quadResult[0]/pi)
    saList.append(2*smallAngle/pi)

title("Compare integration results (normalized)")
xlabel("$\\theta$")
ylabel("K(m)/K(0)")

plot(thetArr, srList, 'b-', label="Simpson's Rule (N=1000)")
plot(thetArr, siList, 'r-', label="scipy integrate.quad")
plot(thetArr, saList, 'g-', label="Small Angle")
legend(loc=2)
```

Project 8
==========

For project 8 we'll take data from a video of a simple pendulum. This year I plan to collect new data. We have enough students that we can divide and conquer to get a significant data set. From that data we'll get experimental values for period. I'd like you to compare those experimental results with a) The small angle analytical results and b) the exact theory using Simpson's Rule. You can find below example code that illustrates the idea. In your project you should show how you:

1. Analyzed the video and contributed some data to the full data set
2. Took at least one full cycle of low amplitude data (in the last few seconds of video) to get the right offset for the data you collected and to confirm the low amplitude graph.

```python
#
# This is an example of getting data out of tracker.
# Note that we can get the angle and the time with the same fit.
#

import os
import csv
from scipy.optimize import curve_fit  # automated curve fitting

sigTheta=0.18/58.0 # ~0.18 cm (1 pixel)  error, r~= 58cm

def fQuad(t, theta0, alpha, t0):
    """
    Use a quadratic to fit the angle vs. time
    """
    return theta0 + alpha*(t-t0)**2

fname = 'track270.csv'
dataFile = open(os.path.join('p8-data',fname))
reader = csv.reader(dataFile, delimiter=',')
lineCnt=0
tlist=[]
xlist=[]
ylist=[]
thetaList=[]

for line in reader:
    if lineCnt in [0,1]:
        pass
    else:
        try:
            tlist.append(float(line[0]))
            xlist.append(float(line[1]))  # tracker has x-y backwards for some reason.
            ylist.append(float(line[2]))
            thetaList.append(arctan2(ylist[-1],xlist[-1]))
        except:
            print("Ack!", line)

    lineCnt+=1
print("Number of data points:", len(tlist))

tarr=array(tlist)
thArr=array(thetaList)

thDeg=thArr*180/pi     # get theta array in degress for display

theta0e = thArr.sum()/len(thArr)
dt=tarr[1]-tarr[0]
om0 = (thArr[1]-thArr[0])/dt
omf = (thArr[-1]-thArr[-2])/dt
alpha0e = (omf-om0)/(tarr[-1]-tarr[0])
t0e = tarr.sum()/len(tarr)

popt, pcov = curve_fit(fQuad, tarr, thArr, p0=(theta0e, alpha0e, t0e),sigma=sigTheta)

theta0=popt[0]
th0Deg=theta0*180.0/pi         # theta0 in degrees for display
thErr=sqrt(pcov[0,0])*180.0/pi # theta0 error in degrees

alpha=popt[1]
t0=popt[2]
t0Err=sqrt(pcov[2,2])

print("fit parameters:"+fname)
print("-----------------")
print("theta0:", th0Deg,"+/-",thErr)
print("alpha:", alpha)
print("t0", t0,"+/-",t0Err)
print("K(sin^2(theta0/2)", K(sin(theta0/2.0)**2)) # estimate elliptic integral

tstar = linspace(tarr[0],tarr[-1],50)   # range of times for quadratic model
ystar = fQuad(tstar, theta0, alpha, t0) # value of quadratic model
trange = tarr[-1]-tarr[0]               # get range of times
thRange = max(thDeg)-min(thDeg)      # range of displayed angles

title("Data: "+fname)
xlabel("time(s)")
ylabel("angle(deg)")
errorbar(tarr, thArr*180/pi,fmt='b.',yerr=sigTheta*180.0/pi, label="theta")
plot(tstar, ystar*180/pi, 'r-', label="quad fit")
plot([tarr[0],tarr[-1]],[th0Deg,th0Deg],'k-')            # draw horiz. cross hair
plot([t0,t0],[min(thDeg)-0.3*thRange,max(thDeg)],'k-')   # draw vertical cross hair
legend()

```

```python
def fPeriodic(t, A, omega, phi, offset):
    """
    Use a quadratic to fit the angle vs. time
    """
    return A*cos(omega*t + phi) + offset

fname = 'track_full_lowamp.csv'
dataFile = open(os.path.join('p8-data',fname))
reader = csv.reader(dataFile, delimiter=',')
lineCnt=0
tlist=[]
xlist=[]
ylist=[]
thetaList=[]

for line in reader:
    if lineCnt in [0,1]:
        pass
    else:
        try:
            tlist.append(float(line[0]))
            xlist.append(float(line[1]))  # tracker has x-y backwards for some reason.
            ylist.append(float(line[2]))
            thetaList.append(arctan2(ylist[-1],xlist[-1]))
        except:
            print("Ack!", line)

    lineCnt+=1
print(len(tlist))

tarr=array(tlist)
thArr=array(thetaList)

omega0 = 2*pi/(120.1-118.6)  # estimate omega (from the data)
A0 = (6-(-8))*pi/360.0       # estimate Amplitude ((max-min)/2)*(pi/180)
phi0 = 0                     # estimate phi  (just a guess)
offset0 = 0.0                # estimate offset

print("initial estimates")
print("-----------------")
print("omega0", omega0)
print("A0", A0)

popt, pcov = curve_fit(fPeriodic, tarr, thArr, p0=(A0, omega0, phi0, offset0),sigma=sigTheta)

A=popt[0]
omega=popt[1]
phi=popt[2]
offset=popt[3]
T0 = (2*pi/omega)/(1+(A**2)/16)  # T0 corrected for amplitude effect

print("fit parameters")
print("-----------------")
print("A:", A*180.0/pi)
print("omega:", omega, "corrected period:", T0, "s")
print("phi:", phi*180.0/pi)
print("offset:", offset*180.0/pi)
print()

tstar = linspace(tarr[0],tarr[-1],50)
ystar = fPeriodic(tstar, A, omega, phi, offset)

title("Data: "+fname)
xlabel("time(s)")
ylabel("angle(deg)")
errorbar(tarr, thArr*180/pi, fmt='b.',yerr=sigTheta*180.0/pi, label="theta")
plot(tstar, ystar*180/pi, 'r-', label="quad fit")
legend()

```

```python
fname = 'raw-data.csv'
dataFile = open(os.path.join('p8-data',fname))
reader = csv.reader(dataFile, delimiter=',')
lineCnt=0
t1_list=[]
t2_list=[]
theta1_list=[]
theta2_list=[]
for line in reader:
    if lineCnt in [0]:
        pass
    else:
        try:
            t1_list.append(float(line[1]))
            theta1_list.append(float(line[2])*pi/180.0)  # convert to radians
            t2_list.append(float(line[3]))
            theta2_list.append(float(line[4])*pi/180.0)  # convert to radians
        except:
            print("Ack!", line)

    lineCnt+=1

N=len(t1_list)
t1arr=array(t1_list)
t2arr=array(t2_list)
th1arr=array(theta1_list)+offset # theta 1 corrected for offset
th2arr=array(theta2_list)+offset # theta 2 corrected for offset

ratio = (t2arr-t1arr)/T0     # array of experimental periods devided by zero amplitude period
thAvg = (th1arr+th2arr)/2.0

vsaa = 1.0*ones(N)
saa = 1.0+thAvg*thAvg/16
sraList=[]
for theta in thAvg:
    sraList.append(2*K(sin(theta/2.0)**2)/pi)

thDeg=thAvg*180/pi

  # array of average amplitudes (in radians)

title("Amplitude of a simple pendulum")
xlabel("$\\theta_{\\rm max}$ (degrees)")
ylabel("T/To")
axis([30,70,0.99,1.1])
plot(thDeg, ratio,'b.', label="experiment")
plot(thDeg, saa, 'r-',label="small angle approx.")
plot(thDeg, vsaa, 'g-', label="very small angle approx.")
plot(thDeg, sraList, 'c-', label="exact theory")
legend(loc=2)
```

```python

```


# Diferenciação Numérica

> Material didático baseado no livro: *Python Programming and Numerical Methods - A Guide for Engineers and Scientists by Qingkai Kong Timmy Siauw and Alexandre Bayen. Imprint: Academic Press*.

Muitos sistemas de engenharia e ciência mudam com o tempo e com o espaço.  Em matemática, as derivadas de funções são usadas para modelar essas mudanças.  No entanto, na prática, a função pode não ser explicitamente conhecida ou pode ser representada implicitamente por um conjunto de pontos de dados.  Nestes casos e em outros, pode ser interessante calcular as derivadas numericamente em vez de analiticamente.

 O foco deste tutorial é a diferenciação numérica.  Ao final desta aula, você será capaz de derivar alguns esquemas básicos de diferenciação numérica e estimar sua precisão.

## Definindo o problema

Uma grade numérica é um conjunto de pontos uniformemente espaçados sobre o domínio de uma função (ou seja, a variável independente), em algum intervalo.  O espaçamento ou tamanho do passo de uma grade numérica é a distância entre pontos adjacentes na grade. Para os fins deste tutorial, se  $x$ é uma grade numérica, então $x_j$ é o j-ésimo ponto na grade numérica e h
é o espaçamento entre $x_j$ e $x_{j-1}$

A figura a seguir mostra um exemplo de grade numérica.

Existem várias funções do Python que podem ser usadas para gerar grades numéricas. Para grades numéricas em uma dimensão, é suficiente usar a função *linspace*

 Em Python, uma função $f(x)$ pode ser representada em um intervalo calculando seu valor em uma grade.  Embora a função em si possa ser contínua, esta representação discreta ou discretizada é útil para cálculos numéricos e corresponde a conjuntos de dados que podem ser adquiridos em situações reais de engenharia ou ciências.

 Especificamente, o valor da função só pode ser conhecido em pontos discretos.  Por exemplo, um sensor de temperatura pode fornecer pares de temperatura versus tempo em intervalos de tempo regulares.  Embora a temperatura seja uma função de tempo suave e contínua, o sensor apenas fornece valores em intervalos de tempo discretos e, neste caso particular, a função subjacente nem mesmo seria conhecida.

 Seja $f$ uma função analítica ou uma representação discreta de uma, gostaríamos de obter métodos de aproximação da derivada de $f$ sobre uma grade numérica e determinar sua precisão.

A derivada $f^{\prime}(x)$ da função $f(x)$ no ponto $x=a$ é definida por:
\begin{equation}
f'(a) = \lim\limits_{x \to a}\frac{f(x) - f(a)}{x-a}
\end{equation}

A derivada em $x = a$ é a inclinação neste ponto. Em aproximações de diferenças finitas dessa inclinação, podemos usar valores da função na vizinhança do ponto $x = a$ para atingir o objetivo. Existem várias fórmulas de diferenças finitas usadas em diferentes aplicações, e três delas, onde a derivada é calculada usando os valores de dois pontos, serão apresentadas a seguir.

A diferença para a frente é a estimativa da inclinação da função em $x_j$ usando a linha que conecta $(x_j, f(x_j))$ e $(x_{j+1}, f(x_{j+1}))$
\begin{equation}
f'(x_j) = \frac{f(x_{j+1}) - f(x_j)}{x_{j+1}-x_j}
\end{equation}

A diferença para a trás é a estimativa da inclinação da função em $x_j$ usando a linha que conecta $(x_{j-1}, f(x_{j-1}))$ e $(x_j, f(x_j))$
\begin{equation}
f'(x_j) = \frac{f(x_j) - f(x_{j-1})}{x_j - x_{j-1}}
\end{equation}

A diferença central é a estimativa da inclinação da função em $x_j$  usando a linha que conecta $(x_{j-1}, f(x_{j-1}))$ e $(x_{j+1}, f(x_{j+1}))$
\begin{equation}
f'(x_j) = \frac{f(x_{j+1}) - f(x_{j-1})}{x_{j+1} - x_{j-1}}
\end{equation}

A figura a seguir ilustra os três tipos diferentes de fórmulas para estimar a inclinação.

## Aproximando Diferenças Finitas com Séries de Taylor

Para obter uma aproximação para a derivada de $f$, vamos usar a série de Taylor. Para uma função arbitrária $f(x)$ a série de Taylor de $f$ em torno de $a$ é dada por:

\begin{equation}
f(x) = \frac{f(x_j)(x - x_j)^0}{0!} + \frac{f^{\prime}(x_j)(x - x_j)^1}{1!} + \frac{f''(x_j)(x - x_j)^2}{2!} + \frac{f'''(x_j)(x - x_j)^3}{3!} + \cdots.
\end{equation}

Se $x$ está em uma grade de pontos com espaçamento $h$, podemos calcular a série de Taylor em $x = x_{j+1}$

\begin{equation}
f(x_{j+1}) = \frac{f(x_j)(x_{j+1} - x_j)^0}{0!} + \frac{f^{\prime}(x_j)(x_{j+1}- x_j)^1}{1!} + \frac{f''(x_j)(x_{j+1} - x_j)^2}{2!} + \frac{f'''(x_j)(x_{j+1} - x_j)^3}{3!} + \cdots.
\end{equation}

Substituindo $h = x_{j+1} - x_j$ e resolvendo para $f^{\prime}(x_j)$:

\begin{equation}
f^{\prime}(x_j) = \frac{f(x_{j+1}) - f(x_j)}{h} + \left(-\frac{f''(x_j)h}{2!} -\frac{f'''(x_j)h^2}{3!} - \cdots\right).
\end{equation}

Os termos entre parênteses $-\frac{f''(x_j)h}{2!} -\frac{f'''(x_j)h^2}{3!} - \cdots$ são chamados de termos de ordens maiores de $h$. Esses termos vão a zero quando $h\approx0$.

Usando a abreviatura $O(h)$ para esses termos e substituindo na equação obtida anteriormente
\begin{equation}
f^{\prime}(x_j) = \frac{f(x_{j+1}) - f(x_j)}{h} + O(h).
\end{equation}

Isto nos fornece a fórmula da diferenciação para a frente
\begin{equation}
f^{\prime}(x_j) \approx \frac{f(x_{j+1}) - f(x_j)}{h},
\end{equation}
e dizemos que essa fórmula é $O(h)$, ou seja, de ordem $h$.

Aqui, $O(h)$ descreve a **precisão** da fórmula de diferença para frente (*forward*) para aproximar derivadas. Para uma aproximação que é $O(h^p)$, dizemos que $p$ é a ordem da precisão da aproximação. Com poucas exceções, a precisão de ordem superior é melhor do que a de ordem inferior. Para ilustrar esse ponto, suponha que $q
h> 0$, vai para 0, $h^p$ vai para 0 mais rápido do que $h^q$. Portanto, conforme $h$ vai para 0, uma aproximação de um valor que é $O(h^p)$ se aproxima do valor verdadeiro mais rapidamente do que um que é $O(h^q)$.

Calculando a série de Taylor em torno de $a=x_j$ em $x=x_{j-1}$ e novamente resolvendo para $f′(x_j)$, obtemos a fórmula da diferença reversa

\begin{equation}
f^{\prime}(x_j) \approx \frac{f(x_j) - f(x_{j-1})}{h},
\end{equation}

que também é $O(h)$. Tente verificar esse resultado por conta própria.

Intuitivamente, as fórmulas de diferença para frente e para trás para a derivada em $x_j$ são apenas as inclinações entre o ponto em $x_j$ e os pontos $x_{j + 1}$ e $x_{j − 1}$, respectivamente.

Podemos construir uma aproximação melhorada da derivada pela manipulação inteligente dos termos da série de Taylor tomados em pontos diferentes. Para ilustrar, podemos calcular a série de Taylor em torno de $a = x_j$ em $x_{j + 1}$ e $x_{j − 1}$

\begin{equation}
f(x_{j+1}) = f(x_j) + f^{\prime}(x_j)h + \frac{1}{2}f''(x_j)h^2 + \frac{1}{6}f'''(x_j)h^3 + \cdots
\end{equation}

e

\begin{equation}
f(x_{j-1}) = f(x_j) - f^{\prime}(x_j)h + \frac{1}{2}f''(x_j)h^2 - \frac{1}{6}f'''(x_j)h^3 + \cdots.
\end{equation}

Subtraindo as duas equações:
\begin{equation}
f(x_{j+1}) - f(x_{j-1}) = 2f^{\prime}(x_j) + \frac{2}{3}f'''(x_j)h^3 + \cdots,
\end{equation}

que quando resolvido para $f^{\prime}(x_j)$ dá a fórmula da diferença central

\begin{equation}
f^{\prime}(x_j) \approx \frac{f(x_{j+1}) - f(x_{j-1})}{2h}.
\end{equation}

Por causa de como subtraímos as duas equações, os termos $h$ se cancelaram; portanto, a fórmula de diferença central é $O (h^2)$, embora requeira a mesma quantidade de esforço computacional que as fórmulas de diferença para frente e para trás! Assim, a fórmula de diferença central obtém uma ordem extra de precisão gratuitamente. Em geral, as fórmulas que utilizam pontos simétricos em torno de $x_j$, por exemplo $x_{j − 1}$ e $x_{j + 1}$, têm melhor precisão do que as assimétricas, como as fórmulas de diferença para frente e para trás.

A figura a seguir mostra a diferença progressiva (linha que conecta $(x_j, y_j)$ e $(x_{j+1}, y_{j+1})$, a diferença regressiva (linha que conecta $(x_j, y_j)$ e $(x_{j-1}, y_{j-1}$), e a diferença central (linha que conecta $(x_{j-1}, y_{j-1})$ e $(x_{j+1}, y_{j+1}$) na aproximação da derivada de uma função $f$. Como pode ser visto, a diferença no valor da inclinação pode ser mudar significativamente com base no tamanho de $h$ e a natureza da função.

# Exemplo 1

Considere a função $f (x) = \cos (x)$. Sabemos que a derivada de $\cos (x)$ é $−\sin (x)$. Embora, na prática, possamos não saber a função subjacente para a qual estamos encontrando a derivada, usamos esse exemplo simples para ilustrar os métodos de diferenciação numérica mencionados acima e sua precisão. O código a seguir calcula as derivadas numericamente.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-poster')

```

```python
# Tamanho do passo:
h = 0.1
# Definindo a grade:
x = np.arange(0, 2*np.pi, h)
# Calculando a função:
y = np.cos(x)

```

```python
# Calculando o vertor com diferenças para frente
forward_diff = np.diff(y)/h
# Calcular a grade correspondente
x_diff = x[:-1]
# Solução exata:
exact_solution = -np.sin(x_diff)

```

```python

# Plotando as soluções
plt.figure(figsize = (12, 8))
plt.plot(x_diff, forward_diff, '--', \
         label = 'Aproximação por diferença finita')
plt.plot(x_diff, exact_solution, \
         label = 'Solução Exata')
plt.legend()
plt.show()

```

```python
# Calcular o erro entre solução exata e aproximada

max_error = max(abs(exact_solution - forward_diff))
print(max_error)
```

    0.049984407218554114

# Exemplo 2
Calcule a derivada numérica de $f (x) = \cos (x)$ usando a fórmula de diferença para frente diminuindo o tamanho dos passos, $h$. Em seguida, plote o erro máximo entre a derivada aproximada e a derivada verdadeira versus $h

```python
# define step size
h = 1
# define number of iterations to perform
iterations = 20
# list to store our step sizes
step_size = []
# list to store max error for each step size
max_error = []
```

```python
for i in range(iterations):
    # halve the step size
    h /= 2
    # store this step size
    step_size.append(h)
    # compute new grid
    x = np.arange(0, 2 * np.pi, h)
    # compute function value at grid
    y = np.cos(x)
    # compute vector of forward differences
    forward_diff = np.diff(y)/h
    # compute corresponding grid
    x_diff = x[:-1]
    # compute exact solution
    exact_solution = -np.sin(x_diff)

    # Compute max error between
    # numerical derivative and exact solution
    max_error.append(\
            max(abs(exact_solution - forward_diff)))
```

```python
# produce log-log plot of max error versus step size
plt.figure(figsize = (12, 8))
plt.loglog(step_size, max_error, 'v')
plt.show()
```

A inclinação da linha no espaço log-log é 1; portanto, o erro é proporcional a $h^1$, o que significa que, como esperado, a fórmula da diferença para frente é $O (h)$.

# Exemplo 3
Use a diferença regressiva para calcular a derivada aproximada de $\sin(x)$

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
x = np.linspace(0,5*np.pi,100)
f = lambda x: np.sin(x)

```

```python
h = 0.1
diff_reg = lambda a: (f(a) - f(a - h))/h
```

```python
derivada = diff_reg(x)
```

```python
d_analitica = np.cos(x)
```

```python
plt.figure(figsize=(12,5))
plt.plot(x,derivada,'r.',label='Derivada Regressiva')
plt.plot(x,d_analitica,'b',label='Derivada Analítica')

plt.title('Derivada Regressiva de y = cos(x)')
plt.legend(loc='best')
plt.show()
```

# Exemplo 4
Construa uma função que calcule derivadas progressivas, regressivas e centrais para qualquer função matemática:

```python
def derivadas(f,a,method='central',h=0.01):
    if method == 'central':
        return (f(a + h) - f(a - h))/(2*h)
    elif method == 'progressiva':
        return (f(a + h) - f(a))/h
    elif method == 'regressiva':
        return (f(a) - f(a - h))/h
    else:
        raise ValueError("O método deve ser 'central', 'progressiva' ou 'regressiva'.")
```

```python
derivadas(np.cos,0,method='progressiva',h=1e-8)

```

    0.0

```python
x = np.linspace(0,5*np.pi,100)
dydx = derivadas(np.sin,x)
```

Vamos calcular e representar graficamente a derivada de uma função complicada:
\begin{equation}
y=\left(\frac{4x^2+2x+1}{x+2e^x}\right)^x
\end{equation}

```python
x = np.linspace(0,6,100)
f = lambda x: ((4*x**2 + 2*x + 1)/(x + 2*np.exp(x)))**x
y = f(x)
dydx = derivadas(f,x)

plt.figure(figsize=(12,5))
plt.plot(x,y,label='y=f(x)')
plt.plot(x,dydx,label="Diferença Central y=f'(x)")
plt.legend()
plt.grid(True)

plt.show()
```

```python

```