```python
%run proof_setup
```

## Real case

First, the real case is relatively simple.
The integral that we want to do is:

$$
k_\Delta(\tau) = \frac{1}{\Delta^2}\int_{t_i-\Delta/2}^{t_i+\Delta/2} \mathrm{d}t \,\int_{t_j-\Delta/2}^{t_j+\Delta/2}\mathrm{d}t^\prime\,k(t - t^\prime)
$$

For celerite kernels it helps to make the assumtion that $t_j + \Delta/2 < t_i - \Delta/2$ (in other words, the exposure times do not overlap).

```python
import sympy as sm

cr = sm.symbols("cr", positive=True)
ti, tj, dt, t, tp = sm.symbols("ti, tj, dt, t, tp", real=True)

k = sm.exp(-cr * (t - tp))
k0 = k.subs([(t, ti), (tp, tj)])
kint = sm.simplify(
    sm.integrate(
        sm.integrate(k, (t, ti - dt / 2, ti + dt / 2)), (tp, tj - dt / 2, tj + dt / 2)
    )
    / dt ** 2
)
res = sm.simplify(kint / k0)
print(res)
```

    (exp(2*cr*dt) - 2*exp(cr*dt) + 1)*exp(-cr*dt)/(cr**2*dt**2)

This is the factor that we want.
Let's make sure that it is identical to what we have in the note.

```python
kD = 2 * (sm.cosh(cr * dt) - 1) / (cr * dt) ** 2
sm.simplify(res.expand() - kD.expand())
```

    0

Excellent.

Let's double check that this reduces to the original kernel in the limit $\Delta \to 0$:

```python
sm.limit(kD, dt, 0)
```

    1

## Complex case

The complex cases proceeds similarly, but it's a bit more involved.
In this case,

$$
k(\tau) = (a + i\,b)\,\exp(-(c+i\,d)\,(t_i-t_j))
$$

```python
a, b, c, d = sm.symbols("a, b, c, d", real=True, positive=True)
k = sm.exp(-(c + sm.I * d) * (t - tp))
k0 = k.subs([(t, ti), (tp, tj)])
kint = sm.simplify(sm.integrate(k, (t, ti - dt / 2, ti + dt / 2)) / dt)
kint = sm.simplify(sm.integrate(kint.expand(), (tp, tj - dt / 2, tj + dt / 2)) / dt)
print(sm.simplify(kint / k0))
```

    (2*cos(I*dt*(c + I*d)) - 2)/(dt**2*(c**2 + 2*I*c*d - d**2))

That doesn't look so bad!

But, I'm going to re-write it by hand and make sure that it's correct:

```python
coeff = (c - sm.I * d) ** 2 / (dt * (c ** 2 + d ** 2)) ** 2
coeff *= sm.exp((c + sm.I * d) * dt) + sm.exp(-(c + sm.I * d) * dt) - 2
sm.simplify(coeff * k0 - kint)
```

    0

Good.

Now we need to work out nice expressions for the real and imaginary parts of this.
First, the real part.
I found that it was easiest to look at the prefactors for the trig functions directly and simplify those.
Here we go:

```python
res = (a + sm.I * b) * coeff
A = sm.simplify((res.expand(complex=True) + sm.conjugate(res).expand(complex=True)) / 2)
```

```python
sm.simplify(sm.poly(A, sm.cos(dt * d)).coeff_monomial(sm.cos(dt * d)))
```

    (a*c**2*exp(2*c*dt) + a*c**2 - a*d**2*exp(2*c*dt) - a*d**2 + 2*b*c*d*exp(2*c*dt) + 2*b*c*d)*exp(-c*dt)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
sm.simplify(
    sm.poly(
        sm.poly(A, sm.cos(dt * d)).coeff_monomial(1), sm.sin(dt * d)
    ).coeff_monomial(sm.sin(dt * d))
)
```

    (2*a*c*d*exp(2*c*dt) - 2*a*c*d - b*c**2*exp(2*c*dt) + b*c**2 + b*d**2*exp(2*c*dt) - b*d**2)*exp(-c*dt)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
sm.simplify(
    sm.poly(
        sm.poly(A, sm.cos(dt * d)).coeff_monomial(1), sm.sin(dt * d)
    ).coeff_monomial(1)
)
```

    2*(-a*c**2 + a*d**2 - 2*b*c*d)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

Then, same thing for the imaginary part:

```python
B = sm.simplify(
    -sm.I * (res.expand(complex=True) - sm.conjugate(res).expand(complex=True)) / 2
)
```

```python
sm.simplify(sm.poly(B, sm.cos(dt * d)).coeff_monomial(sm.cos(dt * d)))
```

    (-2*a*c*d*exp(2*c*dt) - 2*a*c*d + b*c**2*exp(2*c*dt) + b*c**2 - b*d**2*exp(2*c*dt) - b*d**2)*exp(-c*dt)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
sm.simplify(
    sm.poly(
        sm.poly(B, sm.cos(dt * d)).coeff_monomial(1), sm.sin(dt * d)
    ).coeff_monomial(sm.sin(dt * d))
)
```

    (a*c**2*exp(2*c*dt) - a*c**2 - a*d**2*exp(2*c*dt) + a*d**2 + 2*b*c*d*exp(2*c*dt) - 2*b*c*d)*exp(-c*dt)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
sm.simplify(
    sm.poly(
        sm.poly(B, sm.cos(dt * d)).coeff_monomial(1), sm.sin(dt * d)
    ).coeff_monomial(1)
)
```

    2*(2*a*c*d - b*c**2 + b*d**2)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

Ok.

Now let's make sure that the simplified expressions are right.

```python
C1 = a * c ** 2 - a * d ** 2 + 2 * b * c * d
C2 = b * c ** 2 - b * d ** 2 - 2 * a * c * d
cos_term = (sm.exp(c * dt) + sm.exp(-c * dt)) * sm.cos(d * dt) - 2
sin_term = (sm.exp(c * dt) - sm.exp(-c * dt)) * sm.sin(d * dt)
denom = dt ** 2 * (c ** 2 + d ** 2) ** 2

A0 = (C1 * cos_term - C2 * sin_term) / denom
B0 = (C2 * cos_term + C1 * sin_term) / denom
```

```python
sm.simplify(A.expand() - A0.expand())
```

    0

```python
sm.simplify(B.expand() - B0.expand())
```

    0

Finally let's rewrite things in terms of hyperbolic trig functions.

```python
sm.simplify(2 * (sm.cosh(c * dt) * sm.cos(d * dt) - 1).expand() - cos_term.expand())
```

    0

```python
sm.simplify(2 * (sm.sinh(c * dt) * sm.sin(d * dt)).expand() - sin_term.expand())
```

    0

Looks good!

Let's make sure that this actually reproduces the target integral:

```python
sm.simplify(((a + sm.I * b) * kint / k0 - (A + sm.I * B)).expand(complex=True))
```

    0

Finally, let's make sure that this reduces to the original kernel when $\Delta \to 0$:

```python
sm.limit(A, dt, 0), sm.limit(B, dt, 0)
```

    (a, b)

```python
sm.simplify(A.subs([(d, 0), (b, 0)]))
```

    a*((exp(c*dt) - 2)*exp(c*dt) + 1)*exp(-c*dt)/(c**2*dt**2)

```python
sm.simplify(B.subs([(d, 0), (b, 0)]))
```

    0

## Overlapping exposures & the power spectrum

If we directly evaluate the power spectrum of this kernel, we'll have some issues because there will be power from lags where our assumption of non-overlapping exposures will break down.
Instead, we can evaluate the correct power spectrum by realizing that the integrals that we're doing are convolutions.
Therefore, the power spectrum of the integrated kernel will be product of the original power spectrum with the square of the Fourier transform of the top hat exposure function.

```python
omega = sm.symbols("omega", real=True)
sm.simplify(sm.integrate(sm.exp(sm.I * t * omega) / dt, (t, -dt / 2, dt / 2)))
```

    Piecewise((2*sin(dt*omega/2)/(dt*omega), Ne(dt*omega, 0)), (1, True))

Therefore, the integrated power spectrum is

$$
S_\Delta(\omega) = \frac{\sin^2(\Delta\,\omega/2)}{(\Delta\,\omega/2)^2}\,S(\omega)
 = \mathrm{sinc}^2(\Delta\,\omega/2)\,S(\omega)
$$

For overlapping exposures, some care must be taken when computing the autocorrelation because of the absolute value.
This also means that celerite cannot be used (as far as I can tell) to evaluate exposure time integrated models with overlapping exposures.
In this case, the integral we want to do is:

$$
k_\Delta(\tau) = \frac{1}{\Delta^2}\int_{t_i-\Delta/2}^{t_i+\Delta/2} \mathrm{d}t \,\int_{t_j-\Delta/2}^{t_j+\Delta/2}\mathrm{d}t^\prime\,k(|t - t^\prime|)
$$

which can be broken into three integrals when $\tau = |t_i - t_j| \le \Delta$ (assuming still that $t_i \ge t_j$):

$$
\Delta^2\,k_\Delta(\tau)
= \int_{t_j+\Delta/2}^{t_i+\Delta/2} \mathrm{d}t \,\int_{t_j-\Delta/2}^{t_j+\Delta/2}\mathrm{d}t^\prime\,k(t - t^\prime)
+ \int_{t_i-\Delta/2}^{t_j+\Delta/2} \mathrm{d}t \,\int_{t_j-\Delta/2}^{t}\mathrm{d}t^\prime\,k(t - t^\prime)
+ \int_{t_i-\Delta/2}^{t_j+\Delta/2} \mathrm{d}t \,\int_{t}^{t_j+\Delta/2}\mathrm{d}t^\prime\,k(t^\prime - t)
$$

```python
tau = sm.symbols("tau", real=True, positive=True)
kp = sm.exp(-cr * (t - tp))
km = sm.exp(-cr * (tp - t))
k1 = sm.simplify(
    sm.integrate(
        sm.integrate(kp, (tp, tj - dt / 2, tj + dt / 2)), (t, tj + dt / 2, ti + dt / 2)
    )
    / dt ** 2
)
k2 = sm.simplify(
    sm.integrate(sm.integrate(kp, (tp, tj - dt / 2, t)), (t, ti - dt / 2, tj + dt / 2))
    / dt ** 2
)
k3 = sm.simplify(
    sm.integrate(sm.integrate(km, (tp, t, tj + dt / 2)), (t, ti - dt / 2, tj + dt / 2))
    / dt ** 2
)
kD = sm.simplify((k1 + k2 + k3).expand())
res = sm.simplify(kD.subs([(ti, tau + tj)]))
res
```

    (2*cr*dt*exp(cr*tau) - 2*cr*tau*exp(cr*tau) + exp(cr*(-dt + 2*tau)) - 2 + exp(-cr*dt))*exp(-cr*tau)/(cr**2*dt**2)

```python
kint = (
    2 * cr * (dt - tau)
    + sm.exp(-cr * (dt - tau))
    - 2 * sm.exp(-cr * tau)
    + sm.exp(-cr * (dt + tau))
) / (cr * dt) ** 2
sm.simplify(kint - res)
```

    0

Ok. That's the result for the real case. Now let's work through the result for the complex case.

```python
arg1 = ((a + sm.I * b) * kint.subs([(cr, c + sm.I * d)])).expand(complex=True)
arg2 = ((a - sm.I * b) * kint.subs([(cr, c - sm.I * d)])).expand(complex=True)
res = sm.simplify((arg1 + arg2) / 2)
res
```

    (2*dt*(a*c**3 + a*c*d**2 + b*c**2*d + b*d**3)*exp(c*(2*dt + 2*tau)) - 2*tau*(a*c**3 + a*c*d**2 + b*c**2*d + b*d**3)*exp(c*(2*dt + 2*tau)) + 2*(-a*c**2*cos(d*tau) + 2*a*c*d*sin(d*tau) + a*d**2*cos(d*tau) - b*c**2*sin(d*tau) - 2*b*c*d*cos(d*tau) + b*d**2*sin(d*tau))*exp(c*(2*dt + tau)) + (a*c**2*cos(d*(dt - tau)) - 2*a*c*d*sin(d*(dt - tau)) - a*d**2*cos(d*(dt - tau)) + b*c**2*sin(d*(dt - tau)) + 2*b*c*d*cos(d*(dt - tau)) - b*d**2*sin(d*(dt - tau)))*exp(c*(dt + 3*tau)) + (a*c**2*cos(d*(dt + tau)) - 2*a*c*d*sin(d*(dt + tau)) - a*d**2*cos(d*(dt + tau)) + b*c**2*sin(d*(dt + tau)) + 2*b*c*d*cos(d*(dt + tau)) - b*d**2*sin(d*(dt + tau)))*exp(c*(dt + tau)))*exp(-2*c*(dt + tau))/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
C1 = a * c ** 2 - a * d ** 2 + 2 * b * c * d
C2 = b * c ** 2 - b * d ** 2 - 2 * a * c * d
denom = dt ** 2 * (c ** 2 + d ** 2) ** 2

dpt = dt + tau
dmt = dt - tau
cos_term = (
    sm.exp(-c * dmt) * sm.cos(d * dmt)
    + sm.exp(-c * dpt) * sm.cos(d * dpt)
    - 2 * sm.exp(-c * tau) * sm.cos(d * tau)
)
sin_term = (
    sm.exp(-c * dmt) * sm.sin(d * dmt)
    + sm.exp(-c * dpt) * sm.sin(d * dpt)
    - 2 * sm.exp(-c * tau) * sm.sin(d * tau)
)

ktest = 2 * (a * c + b * d) * (c ** 2 + d ** 2) * dmt
ktest += C1 * cos_term + C2 * sin_term
ktest /= denom

sm.simplify(ktest - res)
```

    0

This corresponds to adding a function to the kernel:

```python
delta = sm.simplify(
    ktest.expand(trig=True)
    - (A * sm.exp(-c * tau) * sm.cos(d * tau) + B * sm.exp(-c * tau) * sm.sin(d * tau))
)

C1 = a * c ** 2 - a * d ** 2 + 2 * b * c * d
C2 = b * c ** 2 - b * d ** 2 - 2 * a * c * d
sinh = (sm.exp(c * (dt - tau)) - sm.exp(-c * (dt - tau))) / 2
cosh = (sm.exp(c * (dt - tau)) + sm.exp(-c * (dt - tau))) / 2
norm = dt ** 2 * (c ** 2 + d ** 2) ** 2

delta_test = (
    2
    * (
        C2 * cosh * sm.sin(d * (dt - tau))
        - C1 * sinh * sm.cos(d * (dt - tau))
        + (a * c + b * d) * (dt - tau) * (c ** 2 + d ** 2)
    )
    / norm
)
sm.simplify(delta.expand(trig=True) - delta_test.expand(trig=True))
```

    0

And now, finally, I think we're done.

## What about the diagonal?

By default, the diagonal terms in a celerite model are $\sum a_j$.
In this case, that would be $A$, but that would be wrong because $\tau = 0 < \Delta$.
Let's work out what extra term should be added to the diagonal:

```python
delta = sm.simplify(ktest.subs([(tau, 0)]) - A)
```

```python
sm.simplify(sm.poly(delta, sm.cos(dt * d)).coeff_monomial(sm.cos(dt * d)))
```

    (-a*c**2*exp(2*c*dt) + a*c**2 + a*d**2*exp(2*c*dt) - a*d**2 - 2*b*c*d*exp(2*c*dt) + 2*b*c*d)*exp(-c*dt)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
sm.simplify(
    sm.poly(
        sm.poly(delta, sm.cos(dt * d)).coeff_monomial(1), sm.sin(dt * d)
    ).coeff_monomial(sm.sin(dt * d))
)
```

    (-2*a*c*d*exp(2*c*dt) - 2*a*c*d + b*c**2*exp(2*c*dt) + b*c**2 - b*d**2*exp(2*c*dt) - b*d**2)*exp(-c*dt)/(dt**2*(c**4 + 2*c**2*d**2 + d**4))

```python
sm.simplify(
    sm.poly(
        sm.poly(delta, sm.cos(dt * d)).coeff_monomial(1), sm.sin(dt * d)
    ).coeff_monomial(1)
)
```

    2*(a*c + b*d)/(dt*(c**2 + d**2))

Let's check that our simplified expression is correct:

```python
C1 = a * c ** 2 - a * d ** 2 + 2 * b * c * d
C2 = b * c ** 2 - b * d ** 2 - 2 * a * c * d
norm = dt ** 2 * (c ** 2 + d ** 2) ** 2
sinh = (sm.exp(c * dt) - sm.exp(-c * dt)) / 2
cosh = (sm.exp(c * dt) + sm.exp(-c * dt)) / 2
delta_test = (
    2
    * (
        C2 * cosh * sm.sin(d * dt)
        - C1 * sinh * sm.cos(d * dt)
        + (a * c + b * d) * dt * (c ** 2 + d ** 2)
    )
    / norm
)

sm.simplify(
    delta.expand(trig=True, complex=True) - delta_test.expand(trig=True, complex=True)
)
```

    0

And check the result for the real term:

```python
sm.simplify(delta_test.subs([(d, 0), (b, 0)]) - 2 * a * (c * dt - sinh) / (dt * c) ** 2)
```

    0

Now let's check to make sure that the diagonal will always be positive:

```python
diag_test = (
    2
    * (
        (a * c + b * d) * (c ** 2 + d ** 2) * dt
        - C1
        + sm.exp(-c * dt) * (C1 * sm.cos(d * dt) + C2 * sm.sin(d * dt))
    )
    / (dt * (c ** 2 + d ** 2)) ** 2
)

sm.simplify(ktest.subs([(tau, 0)]) - diag_test)
```

    0

```python
sm.limit(diag_test, dt, 0)
```

    a

```python
sm.limit(diag_test, dt, sm.oo)
```

    0

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Discussion: Week 6

## Exercise: Two-Period Cake-Eating Problem

Recall the two-period cake-eating problem. A person has an initial quantity of cake $K_0>0$ and receives utility from consuming cake in period 0 and period 1:

\begin{align}
U(C_0,C_1) & = \log C_0 + \beta \log C_1.
\end{align}

The person chooses how much to consume in each period, $C_0$ and $C_1$, and a quantity of cake $K_1$ to carry forward from period 0 to maximize her utility subject to the following two budget constraints:

\begin{align}
C_0 + K_1 & = K_0\\
C_1 & = K_1,
\end{align}

where the boundary condition $K_2 = 0$ has been imposed. Use the budget constraints to substitute $C_0$ and $C_1$ out of the utility function. Then the maximization problem can be written concisely as a choice of just $K_1$:

\begin{align}
\max_{K_1} \; \log (K_0-K_1) + \beta \log K_1
\end{align}

The first-order condition with respect to $K_1$ is:

\begin{align}
-\frac{1}{K_0-K_1} + \frac{\beta}{K_1} & = 0,
\end{align}

Given $K_1$, values for $C_0$ and $C_1$ are implied by the budget constraints

The function in the next cell computes the solution for the two-period cake-eating problem and provides an example of how to use the function.

```python
# Define a function to compute the solution for the two-period cake-eating problem. CELL PROVIDED.
def two_period_cake(beta,cake0):

    '''Function for computing solution to two-period cake-eating problem

            Args:
            beta (float):   subjective discount factor
            cake0 (float):  initial quantity of cake

        Returns:
            numpy array, numpy array
    '''

    # Initialize arrays for cake and consumption
    cake = np.zeros(3)
    consumption = np.zeros(2)

    # Assign values to cake array elements
    cake[0] = cake0
    cake[1] = beta/(1+beta)*cake[0]

    # Assign values to consumption array elements as implied by the budget constraints
    consumption[0] = cake[0]-cake[1]
    consumption[1] = cake[1]

    return cake,consumption

# Example of function use: Plot cake path for beta=0.99 and cake0=100
beta = 0.99
cake0 = 100

cake,consumption = two_period_cake(beta=beta,cake0=cake0)

plt.plot(cake)
plt.title('Cake Quantity for $\\beta='+str(beta)+'$')
plt.grid()
```

Suppose that the initial quantity of cake is $K_0 = 100$. For $\beta = 0, 0.25, 0.5, 0.75, 0.99, 1$, use the function `two_period_cake()` to plot the optimal path for the quantity of cake for periods 0 through 2. Add labels to the plot command so you know which line corresponds to which value of $\beta$.

```python
# Create a variable called 'cake0' that stores the initial quantity of cake
cake0=100

# Plot optimal path of cake for the different values of beta.
for beta in [1, 0.99, 0.75, 0.5, 0.25, 0]:

    cake,consumption = two_period_cake(beta=beta,cake0=cake0)
    plt.plot(cake,'o-',lw=3,alpha=0.75,label='$\\beta=$'+str(beta))

plt.title('Cake Quantity for different $\\beta$ values')

# Put a legend to the right of the current axis. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

plt.grid()
```

**Questions**

1. How does increasing $\beta$ affect the otpimal path of $K$?
2. For $\beta=0$, $K_1 = 0$. Why does this make sense?

**Answers**

1. As $\beta$ increases, the person is placing more weight on the utility flow from consumption in period 1 and so the optimal path entails more cake saved for period 1 consummption. The path of cake is more smooth.<!-- answer -->
2. For $\beta=0$, the person places no weight on utility flow in period 1 and so the person consumes all of the cake in period 0. This person is very impatient!<!-- answer -->

## Exercise: Three-Period Cake-Eating Problem

Now consider a the three-period cake-eating problem. A person has an initial quantity of cake $K_0>0$ and receives utility from consuming cake in period 0, period 1, and period 2:

\begin{align}
U(C_0,C_1,C_2) & = \log C_0 + \beta \log C_1+ \beta^2 \log C_2.
\end{align}

The person chooses how much to consume in each period, $C_0$, $C_1$, and $C_2$, and quantities of cake $K_1$ and $K_2$ to carry foward from period 0 and from period 1 to maximize her utility subject to the following two budget constraints:

\begin{align}
C_0 + K_1 & = K_0\\
C_1 + K_2 & = K_1\\
C_2 & = K_2,
\end{align}

where the boundary condition $K_3 = 0$ has been imposed. Use the budget constraints to substitute $C_0$, $C_1$, and $C_2$ out of the utility function. Then the maximiation problem can be written concisely as a choice of $K_1$ and $K_2$:

\begin{align}
\max_{K_1,K_2} \; \log (K_0-K_1) + \beta\log (K_1-K_2) + \beta^2 \log K_2
\end{align}

The first-order conditions with respect to $K_1$ and $K_2$ are:

\begin{align}
-\frac{1}{K_0-K_1} + \frac{\beta}{K_1-K_2} & = 0\\
-\frac{1}{K_1-K_2} + \frac{\beta}{K_2} & = 0,
\end{align}

### Part (a)

Use the first-order conditions for the three-period cake-eating problem to solve by hand for:

1. $K_2$ as a function of $K_1$
2. $K_1$ as a function of $K_0$

Use what you derive in this part to do **Part (b)** and **Part (c)**.

### Part (b)

Copy and paste the full definition of the function `two_period_cake()` into the cell below and do the following:

1. Rename the function `three_period_cake`
2. Edit the function so that it computes the optimal path for the three-period problem

```python
# Define a function to compute the solution for the three-period cake-eating problem.
def three_period_cake(beta,cake0):

    '''Function for computing solution to three-period cake-eating problem

            Args:
            beta (float):   subjective discount factor
            cake0 (float):  initial quantity of cake

        Returns:
            numpy array, numpy array
    '''

    # Initialize arrays for cake and consumption. NOT PROVIDED
    cake = np.zeros(4)
    consumption = np.zeros(3)

    # Assign values to cake array elements. NOT PROVIDED
    cake[0] = cake0
    cake[1] = (beta+ beta**2)/(1+beta+beta**2)*cake[0]
    cake[2] = beta/(1+beta)*cake[1]

    # Assign values to consumption array elements as implied by the budget constraints. NOT PROVIDED
    consumption[0] = cake[0]-cake[1]
    consumption[1] = cake[1]-cake[2]
    consumption[2] = cake[2]

    return cake,consumption
```

### Part (c)

Suppose that the initial quantity of cake is $K_0 = 100$. For $\beta = 0, 0.25, 0.5, 0.75, 0.99, 1$, plot the optimal path for the quantity of cake for periods 0 through 3. Add labels to the plot command so you know which line corresponds to which value of $\beta$.

```python
# Create a variable called 'cake0' that stores the initial quantity of cake
cake0=100

# Plot optimal path of cake for the different values of beta.
for beta in [1, 0.99, 0.75, 0.5, 0.25, 0]:

    cake,consumption = three_period_cake(beta=beta,cake0=cake0)
    plt.plot(cake,'o-',lw=3,alpha=0.75,label='$\\beta=$'+str(beta))

plt.title('Cake Quantity for different $\\beta$ values')

# Put a legend to the right of the current axis. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

plt.grid()
```


# Internal stability of Runge-Kutta methods

Internal stability deals with the growth of errors (such as roundoff) introduced at the Runge-Kutta stages during a single Runge-Kutta step.  It is usually important only for methods with a large number of stages, since that is when the internal amplification factors can be large.  An excellent explanation of internal stability is given in [this paper](http://oai.cwi.nl/oai/asset/1652/1652A.pdf).  Here we demonstrate some tools for studying internal stability in NodePy.

First, let's load a couple of RK methods:

```
from nodepy import rk
reload(rk)
rk4 = rk.loadRKM('RK44')
ssprk4 = rk.loadRKM('SSP104')
print rk4
print ssprk4
```

    Classical RK4
    The original four-stage, fourth-order method of Kutta
     0   |
     1/2 | 1/2
     1/2 |      1/2
     1   |           1
    _____|____________________
         | 1/6  1/3  1/3  1/6
    SSPRK(10,4)
    The optimal ten-stage, fourth order SSP Runge-Kutta method
     0    |
     1/6  | 1/6
     1/3  | 1/6   1/6
     1/2  | 1/6   1/6   1/6
     2/3  | 1/6   1/6   1/6   1/6
     1/3  | 1/15  1/15  1/15  1/15  1/15
     1/2  | 1/15  1/15  1/15  1/15  1/15  1/6
     2/3  | 1/15  1/15  1/15  1/15  1/15  1/6   1/6
     5/6  | 1/15  1/15  1/15  1/15  1/15  1/6   1/6   1/6
     1    | 1/15  1/15  1/15  1/15  1/15  1/6   1/6   1/6   1/6
    ______|____________________________________________________________
          | 1/10  1/10  1/10  1/10  1/10  1/10  1/10  1/10  1/10  1/10

## Absolute stability regions

First we can use NodePy to plot the region of absolute stability for each method.  The absolute stability region is the set

\begin{align} \label{1}
\{ z \in C : |\phi (z)|\le 1 \}
\end{align}

That's \eqref{1}.

That's \ref{1}.

where $\phi(z)$ is the *stability function* of the method:

$1 + z b^T (I-zA)^{-1}$

If we solve $u'(t) = \lambda u$ with a given method, then $z=\lambda \Delta t$ must lie inside this region or the computation will be unstable.

```
p,q = rk4.stability_function()
print p
h1=rk4.plot_stability_region()
```

             4          3       2
    0.04167 x + 0.1667 x + 0.5 x + 1 x + 1

```
p,q = ssprk4.stability_function()
print p
h2=ssprk4.plot_stability_region()
```

# Internal stability

The stability function tells us by how much errors from one step are amplified in the next one.  This is important since we introduce truncation errors at every step.  However, we also introduce roundoff errors at the each stage within a step.  Internal stability tells us about the growth of those.  Internal stability is typically less important than (step-by-step) absolute stability for two reasons:

 - Roundoff errors are typically much smaller than truncation errors, so moderate amplification of them typically is not significant
 - Although the propagation of stage errors within a step is governed by internal stability functions, in later steps these errors are propagated according to the (principal) stability function

Nevertheless, in methods with many stages, internal stability can play a key role.

Questions: *In the solution of PDEs, large spatial truncation errors enter at each stage.  Does this mean internal stability becomes more significant?  How does this relate to stiff accuracy analysis and order reduction?*

## Internal stability functions

We can write the equations of a Runge-Kutta method compactly as

\begin{align}
y & = u^n e + h A F(y) \\
u^{n+1} & = u^n + h b^T F(y),
\end{align}

where $y$ is the vector of stage values, $u^n$ is the previous step solution, $e$ is a vector with all entries equal to 1, $h$ is the step size, $A$ and $b$ are the coefficients in the Butcher tableau, and $F(y)$ is the vector of stage derivatives.  In floating point arithmetic, roundoff errors will be made at each stage.  Representing these errors by a vector $r$, we have

$y = u^n e + h A F(y) + r.$

Considering the test problem $F(y)=\lambda y$ and solving for $y$ gives

$y = u^n (I-zA)^{-1}e + (I-zA)^{-1}r,$

where $z=h\lambda$.  Substituting this result in the equation for $u^{n+1}$ gives

$u^{n+1} = u^n (1 + zb^T(I-zA)^{-1}e) + zb^T(I-zA)^{-1}r = \psi(z) u^n + \theta(z)^T r.$

Here $\psi(z)$ is the *stability function* of the method, that we already encountered above.  Meanwhile, the vector $\theta(z)$ contains the *internal stability functions* that govern the amplification of roundoff errors $r$ within a step:

$\theta(z) = z b^T (I-zA)^{-1}.$

Let's compute $\theta$ for the classical RK4 method:

```
theta=rk4.internal_stability_polynomials()
theta
```

    [poly1d([1/24, 1/12, 1/6, 1/6, 0], dtype=object),
     poly1d([1/12, 1/6, 1/3, 0], dtype=object),
     poly1d([1/6, 1/3, 0], dtype=object),
     poly1d([1/6, 0], dtype=object)]

```
for theta_j in theta:
    print theta_j
```

             4           3          2
    0.04167 x + 0.08333 x + 0.1667 x + 0.1667 x
             3          2
    0.08333 x + 0.1667 x + 0.3333 x
            2
    0.1667 x + 0.3333 x

    0.1667 x

Thus the roundoff errors in the first stage are amplified by a factor $z^4/24 + z^3/12 + z^2/6 + z/6$, while the errors in the last stage are amplified by a factor $z/6$.

## Internal instability

Usually internal stability is unimportant since it relates to amplification of roundoff errors, which are very small.  Let's think about when things can go wrong in terms of internal instability.  If $|\theta(z)|$ is of the order $1/\epsilon_{machine}$, then roundoff errors could be amplified so much that they destroy the accuracy of the computation.  More specifically, we should be concerned if $|\theta(z)|$ is of the order $tol/\epsilon_{machine}$ where $tol$ is our desired error tolerance.  Of course, we only care about values of $z$ that lie inside the absolute stability region $S$, since internal stability won't matter if the computation is not absolutely stable.

We can get some idea about the amplification of stage errors by plotting the curves $|\theta(z)|=1$ along with the stability region.  Ideally these curves will all lie outside the stability region, so that all stage errors are damped.

```
rk4.internal_stability_plot()
```

```
ssprk4.internal_stability_plot()
```

For both methods, we see that some of the curves intersect the absolute stability region, so some stage errors are amplified.  But by how much?  We'd really like to know the maximum amplification of the stage errors under the condition of absolute stability.  We therefore define the *maximum internal amplification factor* $M$:

$M = \max_j \max_{z \in S} |\theta_j(z)|$

```
print rk4.maximum_internal_amplification()
print ssprk4.maximum_internal_amplification()
```

    2.15239281554
    4.04399941143

We see that both methods have small internal amplification factors, so internal stability is not a concern in either case.  This is not surprising for the method with only four stages; it is a surprisingly good property of the method with ten stages.

Questions: *Do SSP RK methods always (necessarily) have small amplification factors?  Can we prove it?*

Now let's look at some methods with many stages.

## Runge-Kutta Chebyshev methods

The paper of Verwer, Hundsdorfer, and Sommeijer deals with RKC methods, which can have very many stages.  The construction of these methods is implemented in NodePy, so let's take a look at them.  The functions `RKC1(s)` and `RKC2(s)` construct RKC methods of order 1 and 2, respectively, with $s$ stages.

```
s=4
rkc = rk.RKC1(s)
print rkc
```

    RKC41

     0    |
     1/16 |  1/16
     1/4  |  1/8   1/8
     9/16 |  3/16  1/4   1/8
    ______|________________________
          |   1/4   3/8   1/4   1/8

```
rkc.internal_stability_plot()
```

It looks like there could be some significant internal amplification here.  Let's see:

```
rkc.maximum_internal_amplification()
```

    11.760869405962685

Nothing catastrophic.  Let's try a larger value of $s$:

```
s=20
rkc = rk.RKC1(s)
rkc.maximum_internal_amplification()
```

    42.665327220219126

As promised, these methods seem to have good internal stability properties.  What about the second-order methods?

```
s=20
rkc = rk.RKC2(s)
rkc.maximum_internal_amplification()
```

    106.69110992619214

Again, nothing catastrophic.  We could take $s$ much larger than 20, but the calculations get to be rather slow (in Python) and since we're using floating point arithmetic, the accuracy deteriorates.

Remark: *we could do the calculations in exact arithmetic using Sympy, but things would get even slower.  Perhaps there are some optimizations that could be done to speed this up.  Or perhaps we should use Mathematica if we need to do this kind of thing.*

Remark 2: *of course, for the RKC methods the internal stability polynomials are shifted Chebyshev polynomials.  So we could evaluate them directly in a stable manner using the three-term recurrence (or perhaps scipy's special functions library).  This would also be a nice check on the calculations above.*

## Other methods with many stages

Three other classes of methods with many stages have been implemented in NodePy:

 - SSP families
 - Integral deferred correction (IDC) methods
 - Extrapolation methods

### SSP Families

```
s=20
ssprk = rk.SSPRK2(s)
ssprk.internal_stability_plot()
ssprk.maximum_internal_amplification()
```

```
s=25 # # of stages
ssprk = rk.SSPRK3(s)
ssprk.internal_stability_plot()
ssprk.maximum_internal_amplification()
```

The SSP methods seem to have excellent internal stability properties.

### IDC methods

```
p=6 #order
idc = rk.DC(p-1)
print len(idc)
idc.internal_stability_plot()
idc.maximum_internal_amplification()
```

IDC methods also seem to have excellent internal stability.

### Extrapolation methods

```
p=6 #order
ex = rk.extrap(p)
print len(ex)
ex.internal_stability_plot()
ex.maximum_internal_amplification()
```

Not so good.  Let's try a method with even more stages (this next computation will take a while; go stretch your legs).

```
p=10 #order
ex = rk.extrap(p)
print len(ex)
ex.maximum_internal_amplification()
```

    46

    28073.244376758907

Now we're starting to see something that might cause trouble, especially since such high order extrapolation methods are usually used when extremely tight error tolerances are required.  Internal amplification will cause a loss of about 5 digits of accuracy here, so the best we can hope for is about 10 digits of accuracy in double precision.  Higher order extrapolation methods will make things even worse.  How large are their amplification factors?  (Really long calculation here...)

```
pmax = 12
ampfac = np.zeros(pmax+1)
for p in range(1,pmax+1):
    ex = rk.extrap(p)
    ampfac[p] = ex.maximum_internal_amplification()
    print p, ampfac[p]
```

```
semilogy(ampfac,linewidth=3)
```

We see roughly geometric growth of the internal amplification factor as a function of the order $p$.  It seems clear that very high order extrapolation methods applied to problems with high accuracy requirements will fall victim to internal stability issues.


### University of Washington: Machine Learning and Statistics

# Lecture 4: Dimensionality Reduction

Andrew Connolly and Stephen Portillo

##### Resources for this notebook include:
- [Textbook](https://press.princeton.edu/books/hardcover/9780691198309/statistics-data-mining-and-machine-learning-in-astronomy) Chapter 8.
- [astroML website](https://www.astroml.org/index.html)

This notebook is developed based on material from A. Connolly, Z. Ivezic, M. Juric, S. Portillo, G. Richards, B. Sipocz, J. VanderPlas, D. Hogg, and many others.

The notebook and assoociated material are available from [github](https://github.com/uw-astro/astr-598a-win22).

xMake sure you are using the latest version of astroML
> pip install --pre -U astroml

## This notebook includes:

[The curse of dimensionality](#curse)

[Principal Component Analysis](#PCA)

[Gappy PCA](#gappy)

[Comparing PCA, NMF and ICA](#all3)

[Manifold Learning](#LLE)

## The curse of dimensionality
[Go to top](#toc)

Fitting and overfitting get worse with ''curse of dimensionality'' Bellman 1961

Think about a hypersphere. Its volume is  given by

$$  V_D(r) = \frac{2r^D\pi^{D/2}}{D\  \Gamma(D/2)}$$

where $\Gamma(z)$ is the complete gamma function, $D$ is the dimension, and $r$ the radius of the sphere.

If you populated a hypercube of size $2r$ how much data would be enclosed by the hypersphere
- as $D$ increases the fractional volume enclosed by the hypersphere goes to 0!

For example: the SDSS comprises a sample of 357 million sources.
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size..) and normalizing the data range $-1$ to $1$

probability of having one of the 357 million sources reside within a unit hypersphere 1 in 1.4$\times 10^5$.

### Solution: dimensionality reduction!

## Increasing interest in dimensionality reduction

## Principal Component Analysis
[Go to top](#toc)

Points are  correlated along a particular direction which doesn't align with the initial choice of axes.
* we should rotate our axes to align with this correlation.
* rotation preserves the relative ordering of data

Choose  rotation to maximize the ability to discriminate between the data points
*   first axis, or principal component, is direction of maximal variance
*   second principal component is orthogonal to the first component and maximizes the residual variance

#### Singular value decomposition (SVD)
Common approach is eigenvalue decomposition of the covariance or correlation matrix,
or singular value decomposition (SVD) of the data matrix

$$U \Sigma V^T = \frac{1}{\sqrt{N - 1}} X$$

columns of $U$ are  _left-singular vectors_

columns of $V$ are the _right-singular vectors_

The columns of $U$ and $V$ form orthonormal bases ($U^TU = V^TV = I$)

Covariance matrix is

$$
  C_X = \left[\frac{1}{\sqrt{N - 1}}X\right]^T \left[\frac{1}{\sqrt{N - 1}}X\right]\nonumber\\
      = V \Sigma U^T U \Sigma V^T\nonumber\\
      = V \Sigma^2 V^T.
$$

right singular vectors $V$ are the principal components so principal from the SVD of $X$ dont need $C_X$.

#### Application of PCA

To compute the PCA components we use `PCA` from `scikit-learn`.

- Center data by subtracting the mean of each dimension
-  For heterogeneous data (e.g., galaxy shape and flux) divide by variance (whitening). why?
- For spectra or images normalize each row so integrated flux of each object is one.

For our astronomy use case, we are using the SDSS spectroscopic dataset.
The SDSS spectra come from galaxies at a range of redshifts,
and have sections of unreliable or missing data due to sky absorption, cosmic rays, bad detector pixels,
or other effects. AstroML provides a set of spectra which have been moved to rest frame, corrected
for masking using an iterative PCA reconstruction technique, and resampled to 1000 common wavelength bins.

The spectra can be downloaded using `fetch_sdss_corrected_spectra()`. In the following example we plot
15 of these spectra:

```python
def warn(*args, **kwargs):
    pass
import warnings
warnings.warn = warn
```

```python
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra_raw = data['spectra']
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA: returns mean + PCA vectors
def compute_PCA(n_components=20):
    nrows = 5000
    ind = np.random.randint(spectra.shape[0], size=nrows)
    spec_mean = spectra[ind].mean(0)

    # PCA
    pca = PCA(n_components - 1)
    pca.fit(spectra[ind])
    pca_comp = np.vstack([spec_mean,
                          pca.components_])
    evals = pca.explained_variance_ratio_

    return pca_comp, evals, ind

n_components = 5
decompositions, evals, index = compute_PCA(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(11, 20))

fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(5):
    ax = fig.add_subplot(n_components, 2, 2*j+2)

    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, decompositions[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

## Exercise: Plot the eigenvalues and what they meam in terms of explained variance

#### Interpreting the PCA

Reconstruction of spectrum, ${x}(k)$, from the
eigenvectors, ${e}_i(k)$

>$ \begin{equation}
  {x}_i(k) = {\mu}(k) + \sum_j^R \theta_{ij} {e}_j(k),
\end{equation}
$

Truncating this expansion (i.e., $r<R$)

>$\begin{equation}
{x}_i(k) = {\mu}(k) + \sum_i^{r<R} \theta_i {e}_i(k),
\end{equation}
$

- eigenvectors ordered by their associated eigenvalues
- eigenvalues reflect variance  within each eigenvector (sum of the eigenvalues is total variance of the system).
- project each spectrum onto these first few eigenspectra is a compression of the data

This is the sense in which PCA gives for dimensionality reduction.

```python
#  Because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(10, 10))
fig.subplots_adjust(hspace=0, top=0.95, bottom=0.1, left=0.12, right=0.93)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.02, 0.93, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

## PCA with missing data
[Go to top](#toc)

Observed spectrum, $x^o$ is the true spectrum, ${x}$ plus a wavelength-dependent weight, ${w}$.

Weight is zero where data are missing and $1/{\sigma}^2$ for rest

Minimizing the quadratic deviation between  ${x}^o$ truncated reconstruction, $\sum_i \theta_i {e}_i$
and solving for $\theta_i$ gives

$$
	\sum_k \theta_i {w}(k) {e}_i(k) {e}_j(k) =
	\sum_k {w}(k) {x}^o(k) {e}_j(k),
$$

If $M_{ij} = \sum_k {w}(k) {e}_i(k) {e}_j(k)$ and $F_i = \sum_k {w}(k) {x}^o(k) {e}_i(k)$ then

$$
	\theta_i = \sum_j M_{ij}^{-1} F_{j},
$$

- $F_j$ are  coefficients derived from  gappy data
- $M_{ij}^{-1}$ shows how correlated  eigenvectors are over the missing regions.

An estimate of the uncertainty on the
reconstruction coefficients is given by

$$
%Cov(\theta_i,\theta_j) = \frac{1}{N}M_{ij}^{-1}
{\rm Cov}(\theta_i,\theta_j) = M_{ij}^{-1}.
$$

Accuracy of this reconstruction will depend on the distribution of
the gaps within the data vector.

The principal component vectors defined for the SDSS spectra can be used to interpolate across or reconstruct
missing data. Examples of three masked spectral regions are shown comparing the reconstruction of the input
spectrum (black line) using the mean and the first ten eigenspectra (blue line) The gray bands represent the
masked region of the spectrum.

```python
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((wavelengths > 5750) & (wavelengths < 6350))
wavelength = wavelengths[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(10, 12.5))
fig.subplots_adjust(left=0.09, bottom=0.08, hspace=0, right=0.92, top=0.95)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spectra_i_centered = spectra[i] / norms[i] - mu
    coeffs = np.dot(spectra_i_centered, evecs.T)

    # plot the raw  spectrum
    spectra_i = spectra[i]
    ax.plot(wavelength, spectra_i[i_plot], '--', color='k',
            label='Masked spectrum', lw=1.5)

    # blank out masked regions
    mask_i = mask[i]
    spectra_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(wavelength, spectra_i[i_plot], '-', color='k',
            label='True spectrum', lw=1.5)

    # plot two levels of reconstruction
    for nev in [0,2]:
        if nev == 0:
            label = 'mean'
            color='red'
        else:
            label = 'reconstruction\n(nev=%i)' % nev
            color='grey'
        spectra_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(wavelength, spectra_i_recons[i_plot], label=label, color=color)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([wavelength[:1], wavelength, wavelength[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(wavelength[0], wavelength[-1])
    ax.set_ylim(ylim)
    #ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1)

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()
```

## Comparing PCA, NMF and ICA
[Go to top](#toc)

### Nonnegative Matrix Factorization (NMF)

#### Non negative factorization

Eigenvectors are defined relative to a mean data vector. Principal components that can be positive or negative but for many physical systems we know data are a linear sum of positive components (e.g. galaxy spectrum is a linear sum of stellar components)

Nonnegative matrix factorization (NMF) applies positivity constraint

Assumes that the data matrix $X$ is a product of two positive matrices, $X=Y*W$.

Solved iteratively for $Y$ and $W$ by minimizing the reconstruction error $||(X-WY)^2||$

```python
from sklearn.decomposition import NMF

X = np.random.random((100, 3))  # 100 points in 3 dims, all positive
nmf = NMF(n_components=3)  # setting n_components is optional
nmf.fit(X)
proj = nmf.transform(X)  # project to 3 dimensions

comp = nmf.components_  # 3 x 10 array of components
err = nmf.reconstruction_err_  # how well 3 components captures data

```

### Independent component analysis

Independent component analysis (ICA) solves the ``cocktail party problem'', where there are multiple microphones situated through out a room containing
$N$ people. Each microphone picks up a linear combination of the $N$
voices. The goal of ICA is to use the concept of statistical
independence to isolate (or unmix) the individual signals.

Each spectrum, $x_i(k)$, can be described by

>$x_1(k) = a_{11}s_1(k) + a_{12}s_2(k) + a_{13}s_3(k) + \cdots\\
x_2(k) = a_{21}s_1(k) + a_{22}s_2(k) + a_{23}s_3(k) + \cdots \\
x_3(k) = a_{31}s_1(k) + a_{32}s_2(k) + a_{33}s_3(k) + \cdots \\
$

where $s_i(k)$ are the individual  spectra and $a_{ij}$
the appropriate mixing amplitudes.

In matrix form,
>$ X = A S$,

 Extracting these signal spectra is equivalent to estimating the appropriate weight matrix,
$W$, such that
>$ S = W X$.

Underlying ICA is the observation that the
input signals, $s_i(k)$, should be **statistically
independent** - the joint probability distribution, $f(x,y)$, can be
fully described by  $f(x^p,y^q)=f(x^p)f(y^q)$

 The rationale for ICA is that the
sum of any two independent random variables will always be more
Gaussian than either of the individual random variables (i.e., from the
central limit theorem). If we identify an unmixing matrix, $W$, that
maximizes the non-Gaussianity of the distributions, then we would be
identifying the input signals.

```python
from sklearn.decomposition import FastICA

X = np.random.normal(size=(100, 2))  # 100 pts in 2 dims
R = np.random.random((2, 5))  # mixing matrix
X = np.dot(X, R)  # X is now 2D data in 5D space
ica = FastICA(2)  # fit two components

sources = ica.fit_transform(X)
proj = ica.transform(X)  # 100 x 2 projection of data
comp = ica.components_  # the 2 x 5 matrix of indep. components
```

### Comparing PCA, NMF and ICA

A comparison of the decomposition of SDSS spectra using PCA (left panel), ICA (middle panel) and
NMF (right panel). The rank of the component increases from top to bottom. For the ICA and PCA
the first component is the mean spectrum (NMF does not require mean subtraction). All of these
techniques isolate a common set of spectral features (identifying features associated with the
continuum and line emission). The ordering of the spectral components is technique dependent.

```python
from sklearn.decomposition import NMF, FastICA, PCA

data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA, ICA, and NMF components
def compute_PCA_ICA_NMF(n_components=5):
    spec_mean = spectra.mean(0)

    # PCA:
    pca = PCA(n_components - 1, random_state=0, svd_solver='randomized')
    pca.fit(spectra)
    pca_comp = np.vstack([spec_mean,
                          pca.components_])

    # ICA treats sequential observations as related.  Because of this, we need
    # to fit with the transpose of the spectra
    ica = FastICA(n_components - 1, random_state=0)
    ica.fit(spectra.T)
    ica_comp = np.vstack([spec_mean,
                          ica.transform(spectra.T).T])

    # NMF requires all elements of the input to be greater than zero
    spectra[spectra < 0] = 0
    nmf = NMF(n_components, random_state=0)
    nmf.fit(spectra)
    nmf_comp = nmf.components_

    return pca_comp, ica_comp, nmf_comp

n_components = 5
decompositions = compute_PCA_ICA_NMF(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['PCA components', 'ICA components', 'NMF components']

for i, comp in enumerate(decompositions):
    for j in range(n_components):
        ax = fig.add_subplot(n_components, 3, 3 * j + 1 + i)

        ax.yaxis.set_major_formatter(plt.NullFormatter())
        ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
        if j < n_components - 1:
            ax.xaxis.set_major_formatter(plt.NullFormatter())
        else:
            ax.xaxis.set_major_locator(
                plt.FixedLocator(list(range(3000, 7999, 1000))))
            ax.set_xlabel(r'wavelength ${\rm (\AA)}$')

        ax.plot(wavelengths, comp[j], '-k', lw=1)

        # plot zero line
        xlim = [3000, 8000]
        ax.plot(xlim, [0, 0], '-', c='gray', lw=1)

        if j == 0:
            ax.set_title(titles[i])

        if titles[i].startswith('PCA') or titles[i].startswith('ICA'):
            if j == 0:
                label = 'mean'
            else:
                label = 'component %i' % j
        else:
            label = 'component %i' % (j + 1)

        ax.text(0.03, 0.94, label, transform=ax.transAxes,
                ha='left', va='top')

        for l in ax.get_xticklines() + ax.get_yticklines():
            l.set_markersize(2)

        # adjust y limits
        ylim = plt.ylim()
        dy = 0.05 * (ylim[1] - ylim[0])

        ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)
        ax.set_xlim(xlim)

plt.show()
```

## Manifold Learning

[Go to top](#toc)

Real-world data sets often have very nonlinear features (e.g. QSOs with broad lines) which are hard to describe compactly using linear eigenvectors. Manifold learning techniques search for a representation of these data within a lower dimensional space

#### Local Linear Embedding:

Step 1: define local geometry
- local neighborhoods determined from $k$ nearest neighbors.
- for each point calculate weights that reconstruct a point from its $k$ nearest
neighbors
>$
\begin{equation}
  \mathcal{E}_1(W) = \left|X - WX\right|^2
\end{equation}
$
- essentially this is finding the hyperplane that describes the local surface at each point within the data set.

Step 2: embed within a lower dimensional space
- set all $W_{ij}=0$ except when point $j$ is one of the $k$ nearest neighbors of point $i$.
- $W$ becomes very sparse for $k \ll N$.
- minimize
>$\begin{equation}
  \mathcal{E}_2(Y) = \left|Y - W Y\right|^2,
\end{equation}
$

with $W$ fixed to find an $N$ by $d$ matrix ($d$ is the new dimensionality)

Step 1 requires a nearest-neighbor search.

Step 2 requires an
eigenvalue decomposition of the matrix $C_W \equiv (I-W)^T(I-W)$,

LLE has been applied to data as diverse as galaxy spectra, stellar spectra, and photometric light curves.

[Survey of LLE variants](https://arxiv.org/pdf/2011.10925.pdf)

```python
from astroML.plotting.tools import discretize_cmap
from astroML.decorators import pickle_results
from sklearn import manifold, neighbors

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Set up color-map properties
clim = (1.5, 6.5)
cmap = discretize_cmap(plt.cm.jet, 5)
cdict = ['unknown', 'star', 'absorption galaxy',
         'galaxy', 'emission galaxy',
         'narrow-line QSO', 'broad-line QSO']
cticks = [2, 3, 4, 5, 6]
formatter = plt.FuncFormatter(lambda t, *args: cdict[int(np.round(t))])

#------------------------------------------------------------
# Fetch the data; PCA coefficients have been pre-computed
spec = sdss_corrected_spectra.reconstruct_spectra(data)
color = data['lineindex_cln']

#------------------------------------------------------------
# Compute the LLE projection; save the results
def compute_spec_LLE(n_neighbors=10, out_dim=3):
    # Compute the LLE projection
    LLE = manifold.LocallyLinearEmbedding(n_neighbors=n_neighbors,
                                          n_components=out_dim,
                                          method='modified',
                                          eigen_solver='dense')
    Y_LLE = LLE.fit_transform(spec)
    print (" - finished LLE projection")

    # remove outliers for the plot
    BT = neighbors.BallTree(Y_LLE)
    dist, ind = BT.query(Y_LLE, n_neighbors)
    dist_to_n = dist[:, -1]
    dist_to_n -= dist_to_n.mean()
    std = np.std(dist_to_n)
    flag = (dist_to_n > 0.25 * std)
    print (" - removing %i outliers for plot" % flag.sum())

    return Y_LLE[~flag], color[~flag]

coeffs_LLE, c_LLE = compute_spec_LLE(10, 3)

#----------------------------------------------------------------------
# Plot the results:
for (c, coeffs, xlim) in zip([c_LLE],
                             [coeffs_LLE],
                             [(-0.01, 0.014)]):
    fig = plt.figure(figsize=(5, 5))
    fig.subplots_adjust(hspace=0.05, wspace=0.05)

    # axes for colorbar
    cax = plt.axes([0.525, 0.525, 0.02, 0.35])

    # Create scatter-plots
    scatter_kwargs = dict(s=4, lw=0, edgecolors='none', c=c, cmap=cmap)

    ax1 = plt.subplot(221)
    im1 = ax1.scatter(coeffs[:, 0], coeffs[:, 1], **scatter_kwargs)
    im1.set_clim(clim)
    ax1.set_ylabel('$c_2$')

    ax2 = plt.subplot(223)
    im2 = ax2.scatter(coeffs[:, 0], coeffs[:, 2], **scatter_kwargs)
    im2.set_clim(clim)
    ax2.set_xlabel('$c_1$')
    ax2.set_ylabel('$c_3$')

    ax3 = plt.subplot(224)
    im3 = ax3.scatter(coeffs[:, 1], coeffs[:, 2], **scatter_kwargs)
    im3.set_clim(clim)
    ax3.set_xlabel('$c_2$')

    fig.colorbar(im3, ax=ax3, cax=cax,
                 ticks=cticks,
                 format=formatter)

    ax1.xaxis.set_major_formatter(plt.NullFormatter())
    ax3.yaxis.set_major_formatter(plt.NullFormatter())

    ax1.set_xlim(xlim)
    ax2.set_xlim(xlim)

    for ax in (ax1, ax2, ax3):
        ax.xaxis.set_major_locator(plt.MaxNLocator(5))
        ax.yaxis.set_major_locator(plt.MaxNLocator(5))

plt.show()

```

#### Summary of pracical properties of PCA, NMF, ICA, and LLE

The following table is a simple summary of the trade-offs along our axes of
accuracy, interpretability, simplicity, and speed in dimension
reduction methods, expressed in terms of high (H), medium
(M), and low (L) categories.

|Method | Accuracy | Interpretability | Simplicity | Speed |
|-------|----------|------------------|------------|-------|
|Principal component analysis | H | H | H |  H |
|Nonnegative matrix factorization | H | H | M | M|
|Independent component analysis | M | M | L | L |
|Locally linear embedding | H | M | H | H |

### Exercise: for a set of spectra compare the coefficients from LLE and PCA coefficients by plotting the coefficicents  as a function of galaxy type

```python
# print the entries in the SDSS spectroscpic data set
for k in data.iterkeys():
    print (k)
```

#### spec_cln

- SPEC_UNKNOWN = 0
- SPEC_STAR    = 1
- SPEC_GALAXY  = 2
- SPEC_QSO     = 3
- SPEC_HIZ_QSO = 4 /* high redshift QSO, z>2.3, Ly-alpha finding code is triggered */
- SPEC_SKY     = 5
- STAR_LATE    = 6 /* star dominated by molecular bands, Type M or later           */
- GAL_EM       = 7

 #### lineindex_cln

- 0 : unknown (SPEC_CLN = 0)
- 1 : star (SPEC_CLN = 1)
- 2 : absorption galaxy (H-alpha seen in absorption)
- 3 : normal galaxy (no significant H-alpha emission or absorption)
- 4 : emission line galaxies (below line-ratio curve)
- 5 : narrow-line QSO (above line-ratio curve)
- 6 : broad-line QSO (SPEC_CLN = 3)
- 7 : Sky (SPEC_CLN = 4)
- 8 : Hi-z QSO (SPEC_CLN = 5)
- 9 : Late-type star (SPEC_CLN = 6)
- 10 : Emission galaxy (SPEC_CLN = 7)