```python
import numpy as np
from scipy import sparse
from scipy import linalg as la
from scipy.integrate import solve_ivp
from matplotlib import pyplot as plt
```

```python
plt.rcParams["figure.dpi"] = 300
plt.rcParams["figure.figsize"] = (12, 4)
plt.rcParams["text.usetex"] = True
```

# ROM Operator Inference Tutorial

The `rom_operator_inference` package constructs reduced-order models (ROM) for large dynamical systems.
Such systems often arise from the numerical solution of partial differentials equations (PDE).
In this tutorial, we give an example of solving a \*very\* simple heat equation with a ROM learned from data via operator inference.

## Problem Statement and Numerical Setting

Let $\Omega = [0,L]\subset \mathbb{R}$ be the spatial domain indicated by the variable $\omega$, and let $[0,T]\subset\mathbb{R}$ be the time domain with variable $t$.
We consider the one-dimensional heat equation with homogeneous Dirichlet boundary conditions,
$$
\begin{align}
    \frac{\partial}{\partial t} x(\omega,t) - \frac{\partial^2}{\partial\omega^2}x(\omega,t) &= 0
    & \omega &\in\Omega,\quad t\in(0,T],
    \\
    x(0,t) = x(L,t) &= 0
    & t &\in[0,T],
    \\
    x(\omega,0) &= \omega(1 - \omega),
    & \omega &\in \Omega.
\end{align}
$$

This is a model for a one-dimensional rod that conducts heat: the temperature at the ends of the rod are fixed at $0$ and heat is allowed to flow out of the rod through the ends.
This is the canonical model of diffusion.

To solve the problem numerically, let $\{\omega_i\}_{i=0}^{n+1}$ be an equidistant grid of $n+2$ points on $\Omega$, i.e.,
$$
\begin{align*}
    0 &= \omega_0 < \omega_1 < \cdots < \omega_n < \omega_{n+1} = L
    &
    &\text{and}
    &
    \delta\omega &= \frac{L}{n+1} = \omega_{i+1} - \omega_{i},\quad i=1,\ldots,n-1.
\end{align*}
$$

Since the boundary conditions prescribe $x(\omega_0,t) = x(\omega_{n+1},t) = 0$, we wish to compute the state vector

$$
\mathbf{x}(t) = \left[\begin{array}{c} x(\omega_1,t) \\ \vdots \\ x(\omega_n,t)\end{array}\right]\in\mathbb{R}^n
$$

for various $t\in[0,T]$.

Introducing the finite difference approximation

$$
\begin{align*}
    \frac{\partial^2}{\partial\omega^2}x(\omega,t) &\approx \frac{x(\omega-\delta\omega,t) - 2x(\omega,t) + x(\omega+\delta\omega,t)}{(\delta\omega)^2}
    % &
    % \Longrightarrow&
    % &
    % \frac{\partial^2}{\partial\omega^2}x(\omega_i,t) &\approx \frac{x(\omega_{i-1},t) - 2x(\omega_{i},t) + x(\omega_{i+1},t)}{(\delta\omega)^2}
    % \\
    &
    &\Longrightarrow
    &
    \frac{\partial^2}{\partial\omega^2}x_{i} &\approx \frac{x_{i-1} - 2x_{i} + x_{i+1}}{(\delta\omega)^2},
\end{align*}
$$

we obtain the semi-discrete linear system

$$
\boxed{
\frac{d}{dt}\mathbf{x}(t) = A\mathbf{x}(t),
\qquad
\mathbf{x}(0) = \mathbf{x}_0
}\,,
$$

where

$$
\begin{align}
    A &= \frac{1}{(\delta\omega)^2}\left[\begin{array}{ccccc}
        -2 & 1 & & & \\
        1 & -2 & 1 & & \\
        & \ddots & \ddots & \ddots & \\
        & & 1 & -2 & 1 \\
        & & & 1 & -2 \\
    \end{array}\right] \in\mathbb{R}^{n\times n},
    &
    \mathbf{x}_0 &= \left[\begin{array}{c}
    \omega_1 (1 - \omega_1) \\ \omega_2 (1 - \omega_2) \\ \vdots \\ \omega_{n-1} (1 - \omega_{n-1}) \\ \omega_n (1 - \omega_n)
    \end{array}\right] \in\mathbb{R}^{n}.
\end{align}
$$

The boxed equation is the _full-order model_ (FOM).
Our goal is to construct a reduced-order model (ROM) that approximates the FOM.

### Snapshot Data Generation

For simplicity, let $L = T = 1$.
We begin by simulating the full-order system described above with a maximal time step size $\delta t = 10^{-3}$, resulting in $k = 10^3+1$ time steps (1000 steps past the initial condition).
The result is the snapshot matrix $X\in\mathbb{R}^{n\times k}$, where the $j$th column is the solution trajectory at time $t_j$.
We also compute the derivative at each snapshot, obtaining $\dot{X}\in\mathbb{R}^{n\times k}$.

```python
# Construct the spatial domain.
W = 1                           # Spatial domain length.
n = 127                         # Spatial grid size.
w_all = np.linspace(0, W, n+2)  # Full spatial grid.
w = w_all[1:-1]                 # Interior spatial grid (where x is unknown).
dw = w[1] - w[0]                # Spatial resolution.
dw2inv = 1 / dw**2

# Construct the temporal domain.
T = 1                           # Temporal domain length (final simulation time).
k = T*1000+1                    # Temporal grid size.
t = np.linspace(0, T, k)        # Temporal grid.
dt = t[1] - t[0]                # Temporal resolution.

print(f"Spatial step size dw = {dw}")
print(f"Temporal step size dt = {dt}")
```

    Spatial step size dw = 0.0078125
    Temporal step size dt = 0.001

```python
# Construct the state matrix A.
diags = np.array([1, -2, 1]) * dw2inv
A = sparse.diags(diags, [-1,0,1], (n,n))

# Define the full-order model dx/dt = f(t,x),  x(0) = x0.
f = lambda t,x: A @ x
x0 = w * (1 - w)

print(f"shape of A:\t{A.shape}")
print(f"shape of x0:\t{x0.shape}")
```

    shape of A:	(127, 127)
    shape of x0:	(127,)

```python
# Compute snapshots by solving the equation with SciPy.
X = solve_ivp(f, [0,T], x0, t_eval=t, method="BDF").y

# Also compute velocities (dx/dt) at each snapshot.
Xdot = A @ X

print(f"shape of X:\t{X.shape}")
print(f"shape of Xdot:\t{Xdot.shape}")
```

    shape of X:	(127, 1001)
    shape of Xdot:	(127, 1001)

Finally, we visualize the snapshots to get a sense of how the solution looks qualitatively.

```python
def plot_heat_data(Z, title):
    fig, ax = plt.subplots(1, 1, figsize=(12,4))

    # Plot a few snapshots over the spatial domain.
    sample_columns = [0, 20, 80, 160, 320, 640]
    color = iter(plt.cm.viridis_r(np.linspace(0, 1, len(sample_columns))))
    for j in sample_columns:
        x_all = np.concatenate([[0], Z[:,j], [0]])
        ax.plot(w_all, x_all, color=next(color), label=rf"$x(\omega,t_{{{j}}})$")
    ax.set_xlim(0, 1)
    ax.set_xlabel(r"$\omega$")
    ax.set_ylabel(r"$x(\omega,t)$")
    ax.legend(loc="upper left")
    fig.suptitle(title)
```

```python
plot_heat_data(X, "Snapshot Data")
```

Initially there is more heat toward the center of the rod, which then diffuses out of the ends of the rod.

### Summary

At this point, we have gathered some training data by simulating the FOM.
We also have an initial condition and space and time domains.

| Name | Symbol | Code Variable |
| :--- | :----: | :------------ |
| State snapshots | $X$ | `X` |
| Velocity snapshots | $\dot{X}$ | `Xdot` |
| Initial state | $\mathbf{x}_0$ | `x0` |
| | |
| Spatial variable | $\Omega$ | `w` |
| Time domain | $[0,T]$ | `t` |

## Using The Package to Solve the Problem

Since the FOM problem has the form
$$
    \frac{d}{dt}\mathbf{x}(t) = A\mathbf{x}(t),\qquad\mathbf{x}(0) = \mathbf{x}_0,
$$
we seek a reduced model with that same structure,
$$
    \frac{d}{dt}\hat{\mathbf{x}}(t) = \hat{A}\hat{\mathbf{x}}(t),\qquad\hat{\mathbf{x}}(0) = \hat{\mathbf{x}}_0,
$$
but with $\hat{\mathbf{x}}\in \mathbb{R}^{r}$ and $\hat{A}\in\mathbb{R}^{r\times r}$ for some $r\ll n$.

We have several tasks to consider:
1. Choosing the reduced-model dimension,
2. Constructing a low-dimensional subspace,
3. Constructing the ROM via operator inference,
4. Simulating the ROM, and
5. Evaluating the performance of the ROM.

We will do this quickly, then show each step in more detail.

```python
import rom_operator_inference as roi

Vr = roi.pre.pod_basis(X, r=2)                      # Construct the reduced basis.
model = roi.InferredContinuousROM(modelform="A")    # Define the model structure.
model.fit(X, Xdot, Vr)                              # Construct the ROM with operator inference.
X_ROM = model.predict(x0, t)                        # Simulate the ROM.
roi.post.frobenius_error(X, X_ROM)[1]               # Calculate the relative error of the ROM simulation.
```

    0.0009116078625495159

### Choosing the Reduced-model Dimension

There are several ways to choose $r$ in an informed way.
A simple choice is to look at the singular values $\{\sigma_j\}_{j=1}^{n}$ of the snapshot matrix $X$ and select the number of $\sigma_{j}$ that are greater than a given threshold.
This also gives us a sense of whether or not we expect model reduction to be successful: if the singular values do not decay quickly, then we will need many modes to capture the behavior of the system.

```python
roi.pre.significant_svdvals?
```

    Signature: roi.pre.significant_svdvals(X, eps, plot=False)
    Docstring:
    Count the number of singular values of X that are greater than eps.

    Parameters
    ----------
    X : (n,k) ndarray
        A matrix of k snapshots. Each column is a single snapshot.

    eps : float or list(floats)
        Cutoff value(s) for the singular values of X.

    plot : bool
        If True, plot the singular values and the cutoff value(s) against the
        singular value index.

    Returns
    -------
    ranks : int or list(int)
        The number of singular values greater than the cutoff value(s).
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/pre.py
    Type:      function

```python
roi.pre.significant_svdvals(X, 1e-4, plot=True)
```

We can also look at the relative contribution magnitudes of the singular values.

```python
roi.pre.energy_capture?
```

    Signature: roi.pre.energy_capture(X, thresh, plot=False)
    Docstring:
    Compute the number of singular values of X needed to surpass a given
    energy threshold. The energy of j singular values is defined by

        energy_j = sum(singular_values[:j]) / sum(singular_values).

    Parameters
    ----------
    X : (n,k) ndarray
        A matrix of k snapshots. Each column is a single snapshot.

    thresh : float or list(floats)
        Energy capture threshold(s).

    plot : bool
        If True, plot the singular values and the energy capture against
        the singular value index.

    Returns
    -------
    ranks : int or list(int)
        The number of singular values required to capture more than each
        energy capture threshold.
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/pre.py
    Type:      function

```python
r = roi.pre.energy_capture(X, .999999, plot=True)
print(f"r = {r}")
```

This indicates that we can capture 99.9999% of the behavior of the FOM with only 2 modes.
This is a very small choice of $r$, but it is also a very simple problem, so for now we select $r = 2$.

### Constructing a Low-dimensional Subspace

Next, we need a reduced basis $V_{r}$ to define the linear subspace to which the ROM states will be confined.
One of the most standard strategies, which aligns with our analysis of the singular values of $X$, is the _POD basis of rank $r$_ corresponding to $X$.
If $X$ has the singular value decomposition
$$
X = \Phi \Sigma \Psi^\mathsf{T},
$$
then the POD basis of rank $r$ consists of the first $r$ columns of $X$:
$$
V_{r} := \Phi_{:,:r}.
$$

```python
roi.pre.pod_basis?
```

    Signature: roi.pre.pod_basis(X, r, mode='simple', **options)
    Docstring:
    Compute the POD basis of rank r corresponding to the data in X.
    This function does NOT shift or scale the data before computing the basis.

    Parameters
    ----------
    X : (n,k) ndarray
        A matrix of k snapshots. Each column is a single snapshot.

    r : int
        The number of POD basis vectors to compute.

    mode : str
        The strategy to use for computing the truncated SVD of X. Options:
        * "simple" (default): Use scipy.linalg.svd() to compute the entire SVD
            of X, then truncate it to get the first r left singular vectors of
            X. May be inefficient for very large matrices.
        * "arpack": Use scipy.sparse.linalg.svds() to compute only the first r
            left singular vectors of X. This uses ARPACK for the eigensolver.
        * "randomized": Compute an approximate SVD with a randomized approach
            using sklearn.utils.extmath.randomized_svd(). This gives faster
            results at the cost of some accuracy.

    options
        Additional parameters for the SVD solver, which depends on `mode`:
        * "simple": scipy.linalg.svd()
        * "arpack": scipy.sparse.linalg.svds()
        * "randomized": sklearn.utils.extmath.randomized_svd()

    Returns
    -------
    Vr : (n,r) ndarray
        The first r POD basis vectors of X. Each column is one basis vector.
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/pre.py
    Type:      function

```python
r=2
Vr = roi.pre.pod_basis(X, r, mode="simple")
print(f"Shape of Vr: {Vr.shape}")
```

    Shape of Vr: (127, 2)

To get a sense of the kinds of solutions we may see, we plot the columns of $V_r$.
All solutions of the resulting ROM can only be linear combinations of these columns.

```python
for j in range(Vr.shape[1]):
    plt.plot(w_all, np.concatenate(([0], Vr[:,j], [0])), label=f"Column {j+1}")
plt.legend()
plt.show()
```

### Constructing the ROM via Operator Inference

We now have training data _and_ a linear basis for a low-dimensional subspace.

| Name | Symbol | Code Variable |
| :--- | :----: | :------------ |
| State snapshots | $X$ | `X` |
| Velocity snapshots | $\dot{X}$ | `Xdot` |
| POD basis | $V_{r}$ | `Vr` |
| Initial state | $\mathbf{x}_0$ | `x0` |
| | |
| Spatial variable | $\Omega$ | `w` |
| Time domain | $[0,T]$ | `t` |

Next, we initialize a `rom_operator_inference` "ROM" class and fit it to the data.
Since the problem is continuous (time-dependent) and there are no external parameters, we use the `InferredContinuousROM` class.
The constructor takes a single parameter, `modelform`, that specifies the structure of the desired model.

| Character | Name | Reduced-order Model Term |
| :-------- | :--- | :------- |
| `c` | Constant |  $\hat{\mathbf{c}}$ |
| `A` | Linear |  $\hat{A}\hat{\mathbf{x}}(t)$ |
| `H` | Quadratic |  $\hat{H}\left(\hat{\mathbf{x}}\otimes\hat{\mathbf{x}}\right)(t)$ |
| `B` | Input |  $\hat{B}\mathbf{u}(t)$ |

Since we seek a ROM of the form $\frac{d}{dt}\hat{\mathbf{x}}(t) = \hat{A}\hat{\mathbf{x}}(t)$, we want `modelform="A"`.
If there were a constant term, $\frac{d}{dt}\hat{\mathbf{x}}(t) = \mathbf{c} + \hat{A}\hat{\mathbf{x}}(t)$, we would use `modelform="cA"`, and so on.

```python
model = roi.InferredContinuousROM("A")
print(model)
```

    Reduced-order model structure: dx / dt = Ax(t)

We now fit the model to the data by solving the least squares problem
$$
    \min_{\hat{A}\in\mathbb{R}^{r\times r}}\left\|\hat{X}^\mathsf{T}\hat{A}^\mathsf{T} - \dot{\hat{X}}^\mathsf{T}\right\|_{F}^2,
$$
where
$$
\begin{align*}
    \hat{X} &= V_r^\mathsf{T}X,
    &
    \dot{\hat{X}} &= V_r^\mathsf{T}\dot{X}.
\end{align*}
$$
This is all done in the `fit()` method, given $X$, $\dot{X}$, and $V_r$.

```python
model.fit?
```

    Signature: model.fit(X, Xdot, Vr, U=None, P=0)
    Docstring:
    Solve for the reduced model operators via regularized least squares.

    Parameters
    ----------
    X : (n,k) ndarray
        Column-wise snapshot training data (each column is a snapshot).

    Xdot : (n,k) ndarray
        Column-wise velocity training data.

    Vr : (n,r) ndarray
        The basis for the linear reduced space (e.g., POD basis matrix).

    U : (m,k) or (k,) ndarray or None
        Column-wise inputs corresponding to the snapshots. If m=1 (scalar
        input), then U may be a one-dimensional array. Required if 'B' is
        in `modelform`; must be None if 'B' is not in `modelform`.

    P : (d,d) ndarray or float
        Tikhonov regularization matrix. If nonzero, the least-squares
        problem problem takes the form min_{x} ||Ax - b||^2 + ||Px||^2.
        If a nonzero number is provided, the regularization matrix is
        P * I (a scaled identity matrix). Here d is the dimension of the
        data matrix for the least-squares problem, e.g., d = r + m for a
        linear model with inputs.

    Returns
    -------
    self
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/_core.py
    Type:      method

```python
model.fit(X, Xdot, Vr)
```

    <rom_operator_inference._core.InferredContinuousROM at 0x15223e5e10>

After fitting the model, we can directly examine the inferred operators of the model.

```python
model.A_
```

    array([[ -9.87390272,  -0.62666937],
           [ -0.62666937, -92.61560522]])

In this case, because this is such a simple problem, operator inference essentially recovers the exact same operator $\hat{A}$ as intrusive projection, i.e., $\hat{A} = V_r^\mathsf{T} A V_r$:

```python
Vr.T @ A @ Vr
```

    array([[ -9.87390272,  -0.62666937],
           [ -0.62666937, -92.61560522]])

```python
np.allclose(model.A_, Vr.T @ A @ Vr)
```

    True

### Simulating the Reduced Model

Once the model is fit, we simulate it with the `predict()` method.
This method takes an initial condition from the original space $\mathbb{R}^n$, projects it to $\mathbb{R}^r$, simulates the ROM in $\mathbb{R}^r$, and maps the results to $\mathbb{R}^n$.

```python
model.predict?
```

    Signature: model.predict(x0, t, u=None, **options)
    Docstring:
    Simulate the learned ROM with scipy.integrate.solve_ivp().

    Parameters
    ----------
    x0 : (n,) ndarray
        The initial (high-dimensional) state vector to begin a simulation.

    t : (nt,) ndarray
        The time domain over which to integrate the reduced-order system.

    u : callable or (m,nt) ndarray
        The input as a function of time (preferred) or the input at the
        times `t`. If given as an array, u(t) is approximated by a cubic
        spline interpolating the known data points.

    options
        Arguments for solver.integrate.solve_ivp(), such as the following:
        method : str
            The ODE solver for the reduced-order system.
            * 'RK45' (default): Explicit Runge-Kutta method of order 5(4).
            * 'RK23': Explicit Runge-Kutta method of order 3(2).
            * 'Radau': Implicit Runge-Kutta method of the Radau IIA family
                of order 5.
            * 'BDF': Implicit multi-step variable-order (1 to 5) method
                based on a backward differentiation formula for the
                derivative.
            * 'LSODA': Adams/BDF method with automatic stiffness detection
                and switching. This wraps the Fortran solver from ODEPACK.
        max_step : float
            The maximimum allowed integration step size.
        See https://docs.scipy.org/doc/scipy/reference/integrate.html.

    Returns
    -------
    X_ROM: (n,nt) ndarray
        The reduced-order approximation to the full-order system over `t`.
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/_core.py
    Type:      method

```python
X_ROM = model.predict(x0, t, method="BDF")
X_ROM.shape
```

    (127, 1001)

### Evaluating the Performance of the ROM

To see how the ROM does, we begin by visualizing the simulation output.
It should look similar to the plot of the snapshot data.

```python
plot_heat_data(X_ROM, "ROM Output")
```

For more detail, we evaluate the $\ell^p$ error of the ROM output in time, comparing it to the snapshot set.

```python
roi.post.lp_error?
```

    Signature: roi.post.lp_error(X, Y, p=2, normalize=False)
    Docstring:
    Compute the absolute and relative lp-norm errors between two snapshot
    sets X and Y where Y is an approximation to X,

        Abs_Err_j = ||X_j - Y_j||_p,
        Rel_Err_j = ||X_j - Y_j||_p / ||X_j||_p = Abs_Err_j / ||X_j||_p.

    Parameters
    ----------
    X : (n,k) or (n,) ndarray
        The "true" data. Each column is one snapshot, i.e., X[:,j] is the data
        at some time t[j]. If one-dimensional, all of X is a single snapshot.

    Y : (n,k) or (n,) ndarray
        An approximation to X, i.e., Y[:,j] approximates X[:,j] and corresponds
        to some time t[j]. If one-dimensional, all of Y is a single snapshot
        approximation.

    p : float
        The order of the lp norm (default p=2 is the Euclidean norm). Used as
        the `ord` argument for scipy.linalg.norm(); see options at
        docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html.

    normalize : bool
        If true, compute the normalized absolute error instead of the relative
        error, defined by

            Normalized_Abs_Err_j = ||X_j - Y_j||_2 / max_j{||X_j||_2}.

    Returns
    -------
    abs_err : (k,) ndarray or float
        The absolute error of each pair of snapshots X[:,j] and Y[:,j]. If X
        and Y are one-dimensional, X and Y are treated as single snapshots, so
        the error is a float.

    rel_err : (k,) ndarray or float
        The relative or normed absolute error of each pair of snapshots X[:,j]
        and Y[:,j]. If X and Y are one-dimensional, X and Y are treated as
        single snapshots, so the error is a float.
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/post.py
    Type:      function

```python
plt.semilogy(t, roi.post.lp_error(X, X_ROM)[0])
plt.title(r"Absolute $\ell^2$ Error")
plt.show()
```

We can also get a scalar error measurement by calculating the relative Frobenius norm error.

```python
roi.post.frobenius_error?
```

    Signature: roi.post.frobenius_error(X, Y)
    Docstring:
    Compute the absolute and relative Frobenius-norm errors between two
    snapshot sets X and Y where Y is an approximation to X,

        Abs_Err = ||X - Y||_F,
        Rel_Err = ||X - Y||_F / ||X||_F = Abs_Err / ||X||_F.

    Parameters
    ----------
    X : (n,k)
        The "true" data. Each column is one snapshot, i.e., X[:,j] is the data
        at some time t[j].

    Y : (n,k)
        An approximation to X, i.e., Y[:,j] approximates X[:,j] and corresponds
        to some time t[j].

    Returns
    -------
    abs_err : float
        The absolute error ||X - Y||_F.

    rel_err : float
        The relative error ||X - Y||_F / ||X||_F.
    File:      ~/Desktop/Research/Operator-Inference-Python/rom_operator_inference/post.py
    Type:      function

```python
roi.post.frobenius_error(X, X_ROM)[1]  # RELATIVE Frobenius error.
```

    0.0009231209515532893

In other words, the ROM simulation is within 0.1% of the snapshot data.


# Demo - Some fast transforms

**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **May 27, 2021**

**Summary.** This demo will show how to compute fast forward transforms for the three
different Dirichlet bases that are implemented for Chebyshev
polynomials in Shenfun.

## Forward and backward transforms

A function $u(x)$ can be approximated in a finite global spectral
expansion $u_N(x)$ as

$$
\label{eq:expansion} \tag{1}
u_N(x) = \sum_{k=0}^{N-1} \hat{u}_k \phi_k(x), \quad \forall \, x \, \in [-1, 1],
$$

where $\phi_k(x)$ are the basis functions and $\boldsymbol{\hat{u}} = \{\hat{u}_k\}_{k=0}^{N-1}$
are the expansion coefficients. The function $u_N(x)$ is continuous
on the interval domain $[-1, 1]$. The span of the basis functions
$V_N = \text{span} \{\phi_k\}_{k=0}^{N-1}$ represents a functionspace.
Associated with this functionspace is a set of quadrature points
$\{x_k\}_{k=0}^{N-1}$ that, along with quadrature weights $\{\omega_k\}_{k=0}^{N-1}$, can be used
for efficient integration. We can also evaluate the function $u_N(x)$ at
these quadrature points to get the sequence
$\boldsymbol{u} = \{u_N(x_k)\}_{k=0}^{N-1}$. If $\boldsymbol{\hat{u}}=\{\hat{u}_k\}_{k=0}^{N-1}$ are *known*,
then $\boldsymbol{u}$ can be evaluated directly from
Eq. ([1](#eq:expansion))

$$
\label{eq:expansionQ} \tag{2}
u_N(x_j) = \sum_{k=0}^{N-1} \hat{u}_k \phi_k(x_j), \quad \forall \, j=0,1, \ldots, N-1.
$$

This would correspond to a *backward* transform according to
the Shenfun terminology. A direct evaluation of the backward
([2](#eq:expansionQ)) transform takes $\mathcal{O}(N^2)$
operations since it requires a double sum (over both $j$
and $k$). A *fast* transform is
a transform that can be computed in $\mathcal{O}(N \log N)$ operations.
This is what the Fast Fourier Transform (FFT) does. It computes a double
sum, like ([2](#eq:expansionQ)), in $\mathcal{O}(N \log N)$ operations.

The other way around, computing $\{\hat{u}_k\}_{k=0}^{N-1}$ from the
*known* $\{u_N(x_k)\}_{k=0}^{N-1}$ corresponds to a *forward* transform.
The forward transform is computed using a projection of $u$
into $V_N$, which is formulated as: find $u_N \in V_N$ such that

$$
\label{eq:projection} \tag{3}
(u_N-u, v)_{\omega^{\sigma}} = 0, \quad \forall \, v \in V_{N},
$$

where $(a, b)_{\omega^{\sigma}} = \int_{I} a b \omega^{\sigma} dx$ is the
inner product in $L^2_{\omega^{\sigma}}(I)$, and $\omega^{\sigma}(x)=(1-x^2)^{\sigma}$  is a weight function.
For Chebyshev polynomials the weight function is usually $\omega^{-1/2}=(1-x^2)^{-1/2}$.
Inserting for $u_N$ and $v=\phi_k$, we get

$$
\begin{equation}
\sum_{j=0}^{N-1}(\phi_j, \phi_k)_{\omega^{\sigma}} \hat{u}_{j} = (u, \phi_k)_{\omega^{\sigma}},
\label{_auto1} \tag{4}
\end{equation}
$$

$$
\begin{equation}
B \boldsymbol{\hat{u}} = \boldsymbol{\tilde{u}},
\label{_auto2} \tag{5}
\end{equation}
$$

$$
\begin{equation}
\boldsymbol{\hat{u}} = B^{-1} \boldsymbol{\tilde{u}},
\label{_auto3} \tag{6}
\end{equation}
$$

where
$\boldsymbol{\tilde{u}} = \{(u, \phi_k)_{\omega^{\sigma}}\}_{k=0}^{N-1}$ and the mass matrix
$B = (b_{kj})_{k,j=0}^{N-1}$, with $b_{kj}=(\phi_j, \phi_k)_{\omega^{\sigma}}$.

Note that the forward transform requires both an inner product
$\boldsymbol{\tilde{u}}$ and a matrix inversion. By a *fast* forward transform
we mean a transform that can be computed in $\mathcal{O}(N \log N)$
operations. If $B$ is a diagonal or banded matrix, the matrix inversion costs $\mathcal{O}(N)$,
and the limiting factor is then the inner product. Like for the backward transform,
the inner product, computed with quadrature, is a double sum

$$
(u, \phi_k)_{\omega^{\sigma}} = \sum_{j=0}^{N-1} u(x_j) \phi_k(x_j) \omega_j, \quad \forall \, k = 0, 1, \ldots, N-1,
$$

where $\{\omega_j\}_{j=0}^{N-1}$ are the quadrature weights.

A naive implementation of the inner product
takes $\mathcal{O}(N^2)$ operations. However,
for Chebyshev polynomials we can compute the double loop with
fast $\mathcal{O}(N \log N)$ discrete sine or cosine transforms,
that are versions of the FFT. To see this, assume that the basis functions are $\phi_k(x) =T_k(x)$, where
$T_k(x)$ is the $k$'th Chebyshev polynomial of the first kind,
and the weight function is $\omega^{-1/2}$.
We then choose Gauss-Chebyshev points $x_j = \cos(\theta_j)$,
where $\theta_j=\pi (2j+1)/(2N)$, and the associated quadrature weights
that are constant $\omega_j = \pi/N$. The Chebyshev polynomials evaluated
on the quadrature points can now
alternatively be written as $T_k(x_j) = \cos(k \theta_j)$,
and the inner product becomes

$$
(u, T_k)_{\omega^{-1/2}} = \sum_{j=0}^{N-1} u(x_j) \cos(k \theta_j) \pi/N, \quad \forall \, k = 0, 1, \ldots, N-1.
$$

From the [FFTW documentation](http://www.fftw.org/fftw3_doc/1d-Real_002deven-DFTs-_0028DCTs_0029.html#g_t1d-Real_002deven-DFTs-_0028DCTs_0029)
we recognise this sum as half a DCT-II (the FFTW DCT-II has a factor
2 in front of the sum) of $\boldsymbol{u}\pi/N$. Hence, we can compute the inner product as

$$
(u, T_k)_{\omega^{-1/2}} = \frac{\pi}{2N} \text{dct}^{II}(\boldsymbol{u})_k, \quad k = 0, 1, \ldots, N-1.
$$

## Dirichlet bases

The basis function $T_k$ satisfies $T_k(\pm 1) = (\pm 1)^k$ at the
boundaries of the domain, and the space $S_N=\text{span}\{T_k\}_{k=0}^{N-1}$,
of dimension $N$,
is thus not associated with any specific set of boundary conditions.
A functionspace for homogeneous Dirichlet boundary conditions is
given as $V_N=\{v\in S_N | v(\pm 1)=0 \}$. Because of the two restrictions
the space has dimension $N-2$.

There are several different choices of basis functions
for $V_N$.
The most interesting we name $\phi_k^n$, for integer $n$, and
define them as

$$
\begin{equation}
\phi^n_k = \omega T^{(n)}_{k+n} = (1-x^2) T^{(n)}_{k+n},
\label{_auto4} \tag{7}
\end{equation}
$$

where $T^{(n)}_{k+n}$ is the $n$'th derivative of $T_{k+n}$. We have
for any integer $n$ that $V_N=\text{span}\{\phi^n_k\}_{k=0}^{N-3}$, and an
expansion in any of these basis functions is

$$
\begin{equation}
\label{eq:uNgeneric} \tag{8}
    u_N = \sum_{k=0}^{N-3} \hat{u}^n_k \phi^n_k.
\end{equation}
$$

We can find the sequence $\{\hat{u}^n_{k}\}_{k=0}^{N-3}$ for any $n$
using a projection into the space $V_N$. The projection is computed
by using Eq. ([8](#eq:uNgeneric)) and $v=\phi^n_k$ in
Eq. ([3](#eq:projection))

$$
\begin{equation}
\label{eq:projortho} \tag{9}
   \sum_{j=0}^{N-3} ( T^{(n)}_{j+n},  T^{(n)}_{k+n})_{\omega^{\sigma+2}} \hat{u}^{n}_j = (u, T^{(n)}_{k+n})_{\omega^{\sigma+1}}.
\end{equation}
$$

Now how can this projection be computed as efficiently as possible?
The Chebyshev polynomials and their derivatives are known to satisfy
the following orthogonality relation

$$
\begin{equation}
\label{eq:orthon} \tag{10}
  (T^{(n)}_j, T^{(n)}_k)_{\omega^{n-1/2}} = \alpha^{n}_k \delta_{kj}, \quad \text{for}\, n \ge 0,
\end{equation}
$$

where $\delta_{kj}$ is the Kronecker delta function and

$$
\begin{equation}
    \alpha^n_k = \frac{c_{k+n}\pi k (k+n-1)!}{2(k-n)!},
\label{_auto5} \tag{11}
\end{equation}
$$

where $c_0=2$ and $c_k=1$ for $k>0$. This can be used in
computing ([9](#eq:projortho)), because we just
need to choose the $\sigma$ that leads to a diagonal mass matrix.
For $n=(0, 1, 2)$ this will be $\sigma=-5/2, -3/2$ and $-1/2$,
respectively. So, choosing $\sigma=-5/2, -3/2$ and $-1/2$
for $n=0, 1$ and 2, respectively, will lead to a diagonal
mass matrix $( T^{(n)}_{j+n},  T^{(n)}_{k+n})_{\omega^{\sigma+2}}$.
Using these $\sigma$'s we can invert the diagonal mass matrices
in Eq. ([9](#eq:projortho)) to get

$$
\begin{equation}
    \hat{u}^n_k = \frac{1}{\alpha^n_{k+n}}(u, T^{(n)}_{k+n})_{\omega^{\sigma+1}}, \quad k=0, 1, \ldots, N-3, \text{ for } n \in (0, 1, 2).
\label{_auto6} \tag{12}
\end{equation}
$$

Using now quadrature, $1-x^2_i=\sin^2 \theta_i$ and the
fast transforms $(u, T_k)_{\omega^{-1/2}} = \pi/2/N \text{dct}^{II}(\boldsymbol{u})_k$
and  $(u, U_k)_{\omega^{-1/2}} = \pi/2/N \text{dst}^{II}(\boldsymbol{u}/\sin \boldsymbol{\theta})_k$,
where $\boldsymbol{u}/\sin \boldsymbol{\theta}$ implies element-wise division,
we get

$$
\begin{equation}
    \hat{u}^0_k = \frac{1}{c_k N} \text{dct}^{II}(\boldsymbol{u}/\sin^2 \boldsymbol{\theta})_k, \quad k = 0, 1, \ldots, N-3, \label{eq:fast1} \tag{13}
\end{equation}
$$

$$
\begin{equation}
    \hat{u}^1_k = \frac{1}{(k+1)N}\text{dst}^{II}(\boldsymbol{u}/\sin \boldsymbol{\theta})_k, \quad k = 0, 1, \ldots, N-3, \label{eq:fast2} \tag{14}
\end{equation}
$$

$$
\begin{equation}
    \hat{u}^2_k = \frac{1}{2(k+2)}\left(\hat{u}^1_k - \hat{u}^1_{k+2} \right), \quad k=0, 1, \ldots, N-3. \label{eq:fast3} \tag{15}
\end{equation}
$$

The last one requires some work, using the identity
$\phi^2_k=(1-x^2)T''_{k+2}=0.5(k+2)(k+3)(U_k - (k+1)/(k+3)U_{k+2})$.

## Verification

To validate all the fast methods we compute the projection first regularly
using the Shenfun function [project](https://github.com/spectralDNS/shenfun/blob/master/shenfun/forms/project.py),
which is using $\sigma=-1/2$, and then the fast methods above. The two
projections should be the same, but they will not give identical results.
In general, the fast transforms above should be both faster and more
accurate, because they only take a discrete transform and merely a diagonal
mass matrix inversion.

We start the implementation by importing necessary modules from Shenfun
and mpi4py-fft

```
from shenfun import *
from mpi4py_fft import fftw
```

The three bases $\{\phi^n_k\}_{k=0}^{N-3}$ are implemented
with slightly different scaling in shenfun.
The first, with $n=0$, is obtained with no special scaling using

```
N = 20
D0 = FunctionSpace(N, 'C', bc=(0, 0), basis='Heinrichs')
```

The second basis is implemented in Shenfun as $\phi_k = \frac{2}{k+1}\phi^1_k$,
which can be simplified as

$$
\label{eq:ft:shen} \tag{16}
\phi_k(x) = T_k-T_{k+2}, \quad k=0,1, \ldots, N-3,
$$

and implemented as

```
D1 = FunctionSpace(N, 'C', bc=(0, 0)) # this is the default basis
```

Because of the scaling the expansion coefficients for $\phi_k$  are
$\hat{u}^{\phi}_k=\frac{k+1}{2}\hat{u}^1_k$. Using ([14](#eq:fast2)) we get

$$
\hat{u}^{\phi}_k = \frac{1}{2N}\text{dst}^{II}(\boldsymbol{u}/\sin \boldsymbol{\theta})_k, \quad k = 0, 1, \ldots, N-3.
$$

The third basis is also scaled and implemented in Shenfun as $\psi_k = \frac{2}{(k+3)(k+2)}\phi^2_k$,
which can be simplified using Chebyshev polynomials of the second
kind $U_k$

$$
\label{eq:ft:dirichletU} \tag{17}
\psi_k(x) = U_k-\frac{k+1}{k+3}U_{k+2}, \quad k=0,1, \ldots, N-3.
$$

We get the basis using

```
D2 = FunctionSpace(N, 'C', bc=(0, 0), basis='DirichletU')
```

and the expansion coefficients are found as
$\hat{u}^{\psi}_k = \frac{(k+3)(k+2)}{2} \hat{u}^2_k$.

For verification of all the fast transforms we first create a vector
consisting of random expansion coefficients, and then transform
it backwards to physical space

```
f = Function(D0, buffer=np.random.random(N))
f[-2:] = 0
fb = f.backward().copy()
```

Next, we perform the regular projections into the three spaces
`D0`, `D1` and `D2`, using the inner product
in $L^2_{\omega^{-1/2}}$. Now `u0`, `u1` and `u2` will be the
three solution vectors
$\boldsymbol{\hat{u}}^{\varphi}$, $\boldsymbol{\hat{u}}^{\phi}$
and $\boldsymbol{\hat{u}}^{\psi}$, respectively.

```
u0 = project(fb, D0)
u1 = project(fb, D1)
u2 = project(fb, D2)
```

Now compute the fast transforms and assert that they are equal to `u0`, `u1` and `u2`

```
theta = np.pi*(2*np.arange(N)+1)/(2*N)
# Test for n=0
dct = fftw.dctn(fb.copy(), type=2)
ck = np.ones(N); ck[0] = 2
d0 = dct(fb/np.sin(theta)**2)/(ck*N)
assert np.linalg.norm(d0-u0) < 1e-8, np.linalg.norm(d0-f0)
# Test for n=1
dst = fftw.dstn(fb.copy(), type=2)
d1 = dst(fb/np.sin(theta))/(2*N)
assert np.linalg.norm(d1-u1) < 1e-8
# Test for n=2
ut = d1
k = np.arange(N)
d2 = Function(D2)
d2[:-2] = (k[:-2]+3)/2/(k[:-2]+1)*ut[:-2]
d2[:-2] = d2[:-2] - 0.5*ut[2:]
assert np.linalg.norm(d2-u2) < 1e-8
```

That's it! If you make it to here with no errors, then the three tests pass, and the fast transforms are equal to the slow ones, at least within given precision.

Let's try some timings

```
%timeit project(fb, D1)
```

```
%timeit dst(fb/np.sin(theta))/(2*N)
```

We can precompute the sine term, because it does not change

```
dd = np.sin(theta)*2*N
%timeit dst(fb/dd)
```

The other two transforms are approximately the same speed.

```
%timeit dct(fb/np.sin(theta)**2)/(ck*N)
```


```python
import sys

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import colors
from utils import bimshow

sys.path.append('../')

%matplotlib inline
```

# Morphological Image Processing

## Preliminaries

1. The **reflection** of a set $B$ is defined as:

$$\begin{equation}
\hat{B} \triangleq \{ -b | b \in B \}
\end{equation}$$

2. The **translation** of a set $B$ by point $z$ is defined as:

$$\begin{equation}
(B)_z \triangleq \{ b + z | b \in B\}
\end{equation}$$

3. A **Structuring Element** is a small set or a subimage used to probe an image under study for properties of interest.
  * In addition to its elements, its origin must be specified. Otherwise, reflection and translation on it are
  ambiguous.
  * When the SE is symmetric and its origin is the symmetric center, the origin can be omitted.

## Erosion and Dilation

### Dilation

The **dilation** of $A$ by $B$ is defined as:

$$\begin{equation}
A \oplus B \triangleq \{ a + b | a \in A, b \in B\}
\end{equation}$$

That is, the dilation of $A$ by $B$ is just the union of translations of $A$ by every point in $B$, i.e.:

$$\begin{equation}
A \oplus B = \bigcup_{b \in B}(A)_b
\end{equation}$$

Or equivalently, the dilation of $A$ by $B$ is just the union of translations of $B$ by every point in $A$, i.e.:

$$ A \oplus B = \bigcup_{a \in A} (B)_a $$

which can be interpreted as the result that each point in $A$ 'grows' into $B$ (with itself becoming the origin of $B$).

Clearly, dilation is:

* associative: $(A \oplus B) \oplus C = A \oplus (B \oplus C)$
* symmetric: $A \oplus B = B \oplus A$

Moreover, if we denote the set containing the origin alone by $\mathcal{O}$, then we have:

* zero-element: $A \oplus \mathcal{O} = A$

But generally, there is no set $B$ s.t. $A \oplus B = \mathcal{O}$ unless $A = \mathcal{O}$.

```python
square = np.zeros((20, 20), dtype=np.uint8)
square[6:14, 6:14] = 1

bimshow(square, figsize=(8, 8))
plt.show()
```

```python
mask = np.ones((3, 3), dtype=np.uint8)
mask[0, 0], mask[2, 0], mask[0, 2], mask[2, 2] = 0, 0, 0, 0

bimshow(mask, ticklabels='off', figsize=(1, 1))
```

```python
from morphology import bimage2set
```

```python
mask_set = bimage2set(mask, origin=(1, 1))
mask_set
```

    array([[-1,  0],
           [ 0, -1],
           [ 0,  0],
           [ 0,  1],
           [ 1,  0]])

```python
square_set = bimage2set(square)
square_set.shape
```

    (64, 2)

```python
square_set_ext = np.expand_dims(square_set, axis=0)  # (1, 64, 2)
mask_set_ext = np.expand_dims(mask_set, axis=1)  # (9, 1, 2)

dilation = (square_set_ext + mask_set_ext).reshape((-1, 2))
dilation.shape
```

    (320, 2)

```python
dilated = np.zeros_like(square, dtype=np.uint8)

dilated[dilation[:, 0], dilation[:, 1]] = 1

bimshow(dilated, figsize=(8, 8))
```

### Erosion

The **erosion** of $A$ by $B$ is defined as:

$$\begin{equation}
A \ominus B \triangleq \{z | (B)_z \subseteq A\}
\end{equation}$$

Similarly, $z$ belongs to $A \ominus B$ iff for any $b \in B$, there is a $a \in A$, s.t. $z = a - b$, that is:

$$\begin{equation}
A \ominus B = \bigcap_{b \in B} \bigcup_{a \in A} \{a - b\} = \bigcap_{b \in B} (A)_{-b}
\end{equation}$$

### Duality

From the duality between cup and cap, it's obvious that:

$$\begin{equation}
\big(A \oplus B\big)^c = \big( \bigcup_{b \in B} (A)_b\big)^c
               = \bigcap_{b \in B} \big((A)_b\big)^c
               = \bigcap_{b \in B} (A^c)_b
               = \bigcap_{b \in \hat{B}} (A^c)_{-b}
               = A^c \ominus \hat{B}
\end{equation}$$

Replacing $A^c$ with $A$ and $\hat{B}$ with $B$, the above equation reads: $(A^c \oplus \hat{B})^c = A \ominus B$, that is:

$$\begin{equation}
\big(A \ominus B\big)^c = A^c \oplus \hat{B}
\end{equation}$$

## Opening and Closing

### Opening

The **opening** of set $A$ by a structuring element $B$ is defined as:

$$\begin{equation}
A \circ B = \big(A \ominus B\big) \oplus B
\end{equation}$$

Property of opening:

* $A \circ B \subseteq A$

* $A \subseteq C \Longrightarrow A \circ B \subseteq C \circ B$

* $\big(A \circ B\big) \circ B = A \circ B$

### Closing

The **closing** of set $A$ by a SE $B$ is defined as:

$$\begin{equation}
A \bullet B = \big(A \oplus B\big) \ominus B
\end{equation}$$

Property of closing:

* $A \subseteq A \bullet B$

* $A \subseteq C \Longrightarrow A \bullet B \subseteq C \bullet B$

* $\big(A \bullet B\big) \bullet B = A \bullet B$

### Duality

As in the case with dialtion and erosion, opening and closing are duals of each other w.r.t. set complementation and
reflection. That is,

$$\begin{equation}
\big(A \circ B\big)^c = \big(A \ominus B\big)^c \ominus \hat{B} = (A^c \oplus \hat{B}) \ominus \hat{B} = A^c \bullet \hat{B}
\end{equation}$$

and similarly,

$$\begin{equation}
\big(A \bullet B\big)^c = \big(A \oplus B\big)^c \oplus \hat{B} = (A^c \ominus \hat{B}) \oplus \hat{B} = A^c \circ \hat{B}
\end{equation}$$

## Hit-or-Miss Transformation

## Some Basic Morphological Algorithms

### Boundary Extraction

The boundary of $A$ can be obtained as following:

$$\begin{equation}
\beta(A) = A - \big(A \ominus B\big)
\end{equation}$$

where $B$ is a suitable SE.

### Hole Filling

A *hole* may be defined as a background region surrounded by a connnected border of foreground pixels. Holes can be filled via dilation if the correct points to start with are known.

Let $X_0$ be the binary image with one point in each hole marked as 1 (the rest are 0s). Then the following procedure
fills all the holes with 1s:

$$\begin{equation}
X_k = \big(X_{k - 1} \oplus B\big) \bigcap A^c \quad k = 1, 2, \ldots
\end{equation}$$

where $B$ is a suitable SE. The algorithm terminates iff $X_k = X_{k - 1}$ for some $k$. Then the union $A\cup X_k$ is the desired result.

### Extraction of Connected Components

Similar to hole filling, the extraction of connected components can also be done by dilation.

Let $X_0$ be the binary image with one point in each connected component marked as 1 (the rest are 0s). Then the
following procedure extracts all the connected components:

$$\begin{equation}
X_k = \big( X_{k - 1} \oplus B \big) \bigcap A \quad k = 1, 2, \ldots
\end{equation}$$

where $B$ is a suitable SE. The algorithm terminates iff $X_k = X_{k - 1}$ for some $k$. Then $X_k$ is the desired result.

### Convext Hull

### Thinning

### Thickening

### Skeletons

### Pruning

### Morphological Reconstruction

## Gray-scale Morphology


```python
try:
    from openmdao.utils.notebook_utils import notebook_mode
except ImportError:
    !python -m pip install openmdao[notebooks]
```

# Simple Optimization

## The Paraboloid

This example finds the minimum point in a paraboloid defined by:

\begin{align}
f(x,y) &= (x-3)^2 + x y + (y+4)^2 - 3
\end{align}

```python
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-50, 50, 100)
y = np.linspace(-50, 50, 100)
XX, YY = np.meshgrid(x, y)
ZZ = (XX - 3)**2 + XX * YY + (YY + 4)**2 - 3

plt.figure(figsize=(8, 6))
plt.contour(XX, YY, ZZ, levels=np.linspace(np.min(ZZ), np.max(ZZ), 50))
plt.xlabel('x')
plt.ylabel('y')
plt.colorbar(label='f')
plt.suptitle('Contour of the paraboloid', fontsize=18);

```

## Finding the minimum with OpenMDAO

The following script uses OpenMDAO to find the minimum of the paraboloid.

```python
import openmdao.api as om

# build the model
prob = om.Problem()

prob.model.add_subsystem('paraboloid', om.ExecComp('f = (x-3)**2 + x*y + (y+4)**2 - 3'))

# setup the optimization
prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'

prob.model.add_design_var('paraboloid.x', lower=-50, upper=50)
prob.model.add_design_var('paraboloid.y', lower=-50, upper=50)
prob.model.add_objective('paraboloid.f')

prob.setup()

# Set initial values.
prob.set_val('paraboloid.x', 3.0)
prob.set_val('paraboloid.y', -4.0)

# run the optimization
prob.run_driver();
```

```python
def test_1():
    from openmdao.utils.assert_utils import assert_near_equal
    assert_near_equal(prob.get_val('paraboloid.x'), 6.66666666, tolerance=1.0E-5)
    assert_near_equal(prob.get_val('paraboloid.y'), -7.33333333, tolerance=1.0E-5)
    assert_near_equal(prob.get_val('paraboloid.f'), -27.33333333, tolerance=1.0E-5)

```

## Step-by-step explanation

First, we import the OpenMDAO API into the namespace `om`

```python
import openmdao.api as om
```

### Instantiating a Problem

An OpenMDAO `Problem` couples a computational _model_ with some sort of _driver_ that executes that model.

The default model is an empty _Group_, to which we can add _Components_ or _Groups_ of _Components_.

The default driver simply executes the model a single time.  More commonly, you'll use drivers that perform optimization (such as ScipyOptimizeDriver or pyOptSparseDriver) or execute the model to perform a design-of-experiments (DOEDriver).

```python
prob = om.Problem()
```

### Building our Model

In this case, our model consists of a single component; an `ExecComp` that executes a user-defined function.  We've given it the mathematical equation for our paraboloid.

```python
prob.model.add_subsystem('paraboloid', om.ExecComp('f = (x-3)**2 + x*y + (y+4)**2 - 3'));
```

### Specifying the Driver

In this case we wish to find the minimum value on the contour of the paraboloid.  To do this, we use an optimizaation driver.  Optimization drivers vary the values of some _design variables_ to minimize the value of some _objective function_.  Many optimiation techniques can also respect some set of _constraints_ imposed upon the optimization, but this particular problem is unconstrained.  We're going to use OpenMDAO's built in `ScipyOptimizeDriver`, which uses the optimizers from [scipy.optimize.minimize](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) to perform the optimization.  In this case, the design variables are `x` and `y`, and the objective is `f`.

```python
# setup the optimization
prob.driver = om.ScipyOptimizeDriver()
prob.driver.options['optimizer'] = 'SLSQP'

prob.model.add_design_var('paraboloid.x', lower=-50, upper=50)
prob.model.add_design_var('paraboloid.y', lower=-50, upper=50)
prob.model.add_objective('paraboloid.f')
```

### Setup the Problem

The next step is setting up the problem.  This is a bit analogous to compiling source code when programming.  OpenMDAO allocates the necessary data structures that enable pass data between different parts of our model and to compute the derivatives of our model for the optimizer.

```python
prob.setup()
```

### Specifying Initial Values

During the course of optimization, OpenMDAO is going to vary the value of the design variables (`x` and `y`) in an attempt to find the minimum value of `f`.  The optimization algorithms involved generally need the user to specify a starting point.  This is accomplished using the `set_val` method of the `Problem` class.

```python
# Set initial values.
prob.set_val('paraboloid.x', 3.0)
prob.set_val('paraboloid.y', -4.0)
```

### Running the Optimization Driver

Finally, we run the optimization driver to actually find the minimum value.  The ScipyOptimizeDriver will output the results of the optimization to standard output.  In addition, `run_driver` will return a `fail` flag.  If `True`, this means the optimization driver believes it failed to successfully find an optimal solution.

```python
# run the optimization
prob.run_driver()
```

### Getting the results

We can access the values of variables in our model by using the `get_val` method of `Problem`.

```python
print('Optimal x value: ', prob.get_val('paraboloid.x'))
print('Optimal y value: ', prob.get_val('paraboloid.y'))
print('Objective value: ', prob.get_val('paraboloid.f'))
```

```python
from openmdao.utils.assert_utils import assert_near_equal
assert_near_equal(prob.get_val('paraboloid.x'), 6.66666666, tolerance=1.0E-5)
assert_near_equal(prob.get_val('paraboloid.y'), -7.33333333, tolerance=1.0E-5)
assert_near_equal(prob.get_val('paraboloid.f'), -27.33333333, tolerance=1.0E-5)
```


# Sistema de Ecuaciones Lineales
## Factorización PALU y Cholesky

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
from scipy import linalg as spla
from scipy import sparse as scsp
from time import time
```

## Número de Condición

### Matriz de Hilbert

Matriz cuadrada simétrica y positiva definida, cuyas entradas se definen como:
\begin{equation}
    H_{ij} = \frac{1}{i+j-1}
\end{equation}

Por ejemplo, la matriz $H$ de $5\times 5$ es
\begin{equation}
     H=
     \begin{bmatrix}
         1 & \frac {1}{2} & \frac {1}{3} & \frac {1}{4} & \frac {1}{5}\\
         \frac {1}{2} & \frac {1}{3} & \frac {1}{4} & \frac {1}{5} & \frac {1}{6}\\
         \frac {1}{3} & \frac {1}{4} & \frac {1}{5} & \frac {1}{6} & \frac {1}{7}\\
         \frac {1}{4} & \frac {1}{5} & \frac {1}{6} & \frac {1}{7} & \frac {1}{8}\\
         \frac {1}{5} & \frac {1}{6} & \frac {1}{7} & \frac {1}{8} & \frac {1}{9}
     \end{bmatrix}
\end{equation}

```python
n = 5
H = spla.hilbert(n)
```

¿Simétrica?

```python
np.all(H == H.T)
```

    True

¿Definida positiva?

```python
np.all(np.linalg.eigvals(H) > 0)
```

    True

Cuidado ver lo que pasa para valores de $n$ grandes.

Número de condición $\kappa(A)$

```python
np.linalg.cond(H), np.max(np.abs(np.linalg.eigvals(H)))/np.min(np.abs(np.linalg.eigvals(H)))
```

    (476607.25024100044, 476607.2502420452)

En teoría esta matriz es simétrica y definida positiva, en la práctica para $n$ grandes no se cumple la ultima propiedad. Además es una matriz mal condicionada. ¿Qué pasará si debemos resolver un sistema de ecuaciones lineales con esta matriz?

# Matriz definida positiva

Una manera de comprobar si una matriz es definida positiva es calcular los valores propios y comprobar que todos estos sean positivos. Veamos el caso para $n=2$.

```python
np.random.seed(2)
n = 2
A = np.random.rand(n, n)
A = np.dot(A, A.T) + np.eye(n)
eigenvalues = np.linalg.eig(A)[0]
print(eigenvalues)
print(eigenvalues > 0)
```

    [1.0486246  1.63377355]
    [ True  True]

La otra forma es comprobar $\mathbf{x}^{T} A \mathbf{x} > 0$, $\mathbf{x}\neq\mathbf{0}$, para eso podemos hacer uso de simpy:

```python
x, y = sym.symbols('x y')
x = np.array([x,y])
f = sym.factor(sym.expand(np.dot(np.dot(x,A),x)))
fl = sym.lambdify(x, f)
f
```

$\displaystyle 1.49163442510266 \left(0.798294611703302 x^{2} + 0.336458187477782 x y + 1.0 y^{2}\right)$

Si analizamos el polinomio anterior tenemos:

\begin{equation}
    \begin{split}
        0.798294611703302 x^{2} + 0.336458187477782 xy + y^2 & \\
        0.798294611703302 x^{2} + 0.336458187477782 xy + y^2 + 0.028301027980208573x^2 - 0.028301027980208573x^2 & \\
        0.7699935837230933 x^{2} + (0.168229093738891x + y)^{2} &
    \end{split}
\end{equation}

El polinomio es positivo para cualquier valor de $x,y$, entonces estamos en presencia de una matriz definida positiva

Otra idea, pero que podría ser más complicado para $\mathbb{R}^{n}$ con $n>3$, es utilizar el criterio de la segunda derivada (en este caso parcial).

```python
fx = sym.diff(f, x[0])
fy = sym.diff(f, x[1])
fxy = sym.diff(fx, x[1])
fyx = sym.diff(fy, x[0])
fxx = sym.diff(fx, x[0])
fyy = sym.diff(fy, x[1])
fxxl = sym.lambdify(x, fxx)
fxyl = sym.lambdify(x, fxy)
fyyl = sym.lambdify(x, fyy)
D = lambda a, b: fxxl(a, b) * fyyl(a, b) - (fxyl(a, b)) ** 2
```

```python
fx, fy
```

    (2.38152744838122*x + 0.501872615049506*y,
     0.501872615049506*x + 2.98326885020533*y)

```python
sym.solve([fx, fy])
```

    {y: 0.0, x: 0.0}

```python
D(0, 0), fxxl(0, 0)
```

    (6.852860530928045, 2.38152744838122)

```python
fl(0,0)
```

    0.0

La función tienen un mínimo en $(x,y)=(0,0)$ pero por definición no nos interesa ese punto, así que la función efectivamente es positiva. Por lo tanto $A$ es definida positiva.

# Complejidad temporal y espacial

Estimación del tamaño del arreglo en [MB]

```python
size = lambda a: a.nbytes * 1e-6
```

```python
Ne = 10 # Repetición de experimentos
Nf = 10
N = 2 ** np.arange(7, Nf + 1) # N = [2^7, 2^{10}]
Nn = N.shape[-1]
# Para tiempos
times_palu = np.zeros(Nn)
times_chol = np.zeros(Nn)
# Para espacio
storage_palu = np.zeros(Nn)
storage_chol = np.zeros(Nn)
```

## Experimentos

```python
for i in range(Nn):
    n = N[i]
    A = np.random.rand(n, n)
    A = np.dot(A, A.T) + np.eye(n)
    # Time PALU
    start_time= time()
    for j in range(Ne):
        P, L, U = spla.lu(A)
    end_time = time()
    storage_palu[i] = size(P) + size(L) + size(U)
    times_palu[i] = (end_time - start_time) / Ne
    # Time Cholesky
    start_time = time()
    for j in range(Ne):
        R = np.linalg.cholesky(A) # R corresponde a R^T
    end_time = time()
    times_chol[i] = (end_time - start_time) / Ne
    # Storage
    storage_chol[i] = size(R)
```

## Análisis de tiempo computacional

```python
plt.figure(figsize=(12, 6))
plt.plot(N, times_palu, 'bd', label="PALU")
plt.plot(N, times_chol, 'go', label="Cholesky")
# Deben adaptar el coeficiente que acompaña a N**k según los tiempos que obtengan en su computador
plt.plot(N, 1e-8 * N ** 2, 'g--', label=r"$O(n^2)$")
plt.plot(N, 1e-10 * N ** 3, 'r--', label=r"$O(n^3)$")
plt.grid(True)
plt.yscale('log')
plt.xscale('log')
plt.xlabel(r"$n$")
plt.ylabel("Time [s]")
plt.legend()
plt.show()
```

Los algoritmos tienen una complejidad que tiende a $O(n^3)$. Dado que hay optimizaciones en las bibliotecas, pueden ver exponentes menores a $3$. En teoría el análisis debería funcionar para $n\to\infty$.

## Análisis de memoria

```python
plt.figure(figsize=(12, 6))
plt.plot(N, storage_palu, 'bd', label="PALU")
plt.plot(N, storage_chol, 'go', label="Cholesky")
plt.plot(N, 1.5e-5 * N ** 2, 'g-', label=r"$O(n^2)$")
plt.grid(True)
plt.yscale('log')
plt.xscale('log')
plt.xlabel(r"$n$")
plt.ylabel("Size [GB]")
plt.legend()
plt.show()
```

Si guardamos todos los elementos de las matrices, el espacio utlizado debería ser del orden $\sim n^2$. ¿Existirá una manera de ahorrar espacio?

# Alternativas de almacenamiento

En el caso de $PALU$ debemos almacenar $P$, $L$ y $U$ $\sim 3n^2$. Con Cholesky solo debemos almacenar $\sim n^2$.
Dado que hay muchos elementos iguales a $0$, podríamos almacenar de manera eficiente solo los valores distintos de $0$. Las matrices que tienen muchos valores que son $0$ se conocen como **Sparse Matrix**, *Matriz dispersa o rala* en español. $L$, $U$ y $R$ son matrices triangulares, y $P$ solo tiene $n$ elementos distintos de $0$.

Una matriz triangular, tiene $\displaystyle n^2 - \sum_{i=1}^{n-1}i=n^2 - \frac{n(n-1)}{2}=\frac{n^2+ n}{2}$ elementos. Es decir, con una representación *dispersa*, $PALU$ requiere mantener $n^2 + 2n$ elementos mientras que Cholesky solo requiere $\displaystyle \frac{n^2+ n}{2}$.

## Almacenamiento *Sparse*

### Coordinate list (COO)

Se almacena una lista de tuplas con $(fila, columna, valor)$.

### Compressed sparse row (CSR)

La matriz se almacena por fila. Se mantienen 3 arreglos, $valores$, $indice\_columnas$, $puntero\_fila$. El primer arreglo guarda los valores no nulos, el segundo guarda el índice de la columna donde se encuentra el coeficiente y el último indica qué parte del primer y segundo arreglo corresponde a cada fila.

### Compressed sparse column (CSC)

Análogo al anterior pero se almacena por columnas. Además el segundo arreglo mantiene el $indice\_filas$ y el tercero $puntero\_columna$.

## Ejemplo

```python
M = np.array([[1, 0, 3, 0], [0, 1, 8, 0], [0, 0, 0, 0], [0, 1, 0, 1]])
```

```python
M
```

### COO

```python
M_coo = scsp.coo_matrix(M)
```

```python
print("Datos:", M_coo.data)
print("Indice filas:", M_coo.row)
print("Indice columnas:", M_coo.col)
```

### CSR

```python
M_csr = scsp.csr_matrix(M)
```

```python
print("Datos:", M_csr.data)
print("Indice columnas:", M_csr.indices)
print("Puntero filas:", M_csr.indptr)
```

```python
for i in range(M_csr.indptr.shape[-1]-1):
    print("Los elementos de la fila %d, se encuentran entre las posiciones [%d, %d) del arreglo de indices" % (i, M_csr.indptr[i], M_csr.indptr[i+1]))
```

### CSC

```python
M_csc = scsp.csc_matrix(M)
```

```python
print("Datos:", M_csc.data)
print("Indice filas:", M_csc.indices)
print("Puntero columnas:", M_csc.indptr)
```

```python
for i in range(M_csc.indptr.shape[-1]-1):
    print("Los elementos de la columna %d, se encuentran entre las posiciones [%d, %d) del arreglo de indices" % (i, M_csc.indptr[i], M_csc.indptr[i+1]))
```

Los formatos *CSR* y *CSC* necesitan almacenar $2nz+n+1$ elementos distintos de $0$ ($nz$).

## Matrices generadas por los métodos de factorización

```python
# Para la visualización de las matrices.
def plotMatrix(M):
    # Solo para ver coeficientes distintos de 0, se muestra la magnitud -> log(|M+eps|), eps para evitar el log(0)
    plt.imshow(np.log(np.abs(M+1e-16)))
    plt.show()
```

```python
plotMatrix(A)
```

## PALU

```python
plotMatrix(P); plotMatrix(L); plotMatrix(U)
```

### Matrices densas

```python
palu_dense = size(P) + size(L) + size(U)
palu_dense
```

### Matrices dispersas

#### Formato Coordenadas

```python
Pc = scsp.coo_matrix(P)
Lc = scsp.coo_matrix(L)
Uc = scsp.coo_matrix(U)
```

```python
palu_sparse_co = size(Pc.data) + size(Pc.row) + size(Pc.col) + size(Lc.data) + size(Lc.row) + size(Lc.col) + size(Uc.data) + size(Uc.row) + size(Uc.col)
palu_sparse_co
```

#### Formato Comprimido

```python
Psr = scsp.csr_matrix(P)
Lsr = scsp.csr_matrix(L)
Usr = scsp.csr_matrix(U)
Psc = scsp.csc_matrix(P)
Lsc = scsp.csc_matrix(L)
Usc = scsp.csc_matrix(U)
```

```python
palu_sparse_r = size(Psr.data) + size(Psr.indices) + size(Psr.indptr) + size(Lsr.data) + size(Lsr.indices) + size(Lsr.indptr) + size(Usr.data) + size(Usr.indices) + size(Usr.indptr)
palu_sparse_r
```

```python
len(Psr.data)
```

```python
palu_sparse_c = size(Psc.data) + size(Psc.indices) + size(Psc.indptr) + size(Lsc.data) + size(Lsc.indices) + size(Lsc.indptr) + size(Usc.data) + size(Usc.indices) + size(Usc.indptr)
palu_sparse_c
```

% memoria matrices *densas* vs *dispersas* utilizando coordenadas

```python
palu_sparse_co / palu_dense
```

% memoria matrices *densas* vs *dispersas* utilizando formato comprimido

```python
palu_sparse_r / palu_dense
```

## Cholesky

```python
plotMatrix(R)
```

### Matriz densa

```python
cholesky_dense = size(R)
cholesky_dense
```

### Matriz dispersa

#### Formato Coordenadas

```python
Rc = scsp.coo_matrix(R)
```

#### Formato Comprimido

```python
Rsr = scsp.csr_matrix(R)
```

```python
Rsc = scsp.csc_matrix(R)
```

Tamaño utilizando representación *sparse*

```python
cholesky_sparse_co = size(Rc.data) + size(Rc.row) + size(Rc.col)
cholesky_sparse_co
```

```python
cholesky_sparse_r = size(Rsr.data) + size(Rsr.indices) + size(Rsr.indptr)
cholesky_sparse_r
```

```python
cholesky_sparse_c = size(Rsc.data) + size(Rsc.indices) + size(Rsc.indptr)
cholesky_sparse_c
```

% de memoria utilizando representación *dispersa* formato coordenadas

```python
cholesky_sparse_co / cholesky_dense
```

% memoria utilizando representación *dispersa* comprimida.

```python
cholesky_sparse_r / cholesky_dense
```

### Comparación

Si analizamos teóricamente la memoria que podemos ahorrar para ambos tipos de factorizaciones utilizando la representación *sparse* tenemos:

* $PALU$:
\begin{equation}
    \frac{\text{Representación dispersa}}{\text{Representación densa}}=
    \frac{n^2 + 2n}{3n^2} = \frac{n+2}{3n} = \frac{1}{3} + \frac{2}{3n}
    \implies \lim_{n\to\infty} \left(\frac{1}{3} + \frac{2}{3n}\right) = \dfrac{1}{3}
\end{equation}

* Cholesky
\begin{equation}
    \frac{\text{Representación dispersa}}{\text{Representación densa}}=
    \frac{\frac{n^2 + n}{2}}{n^2} = \frac{n+1}{2n} = \frac{1}{2} + \frac{1}{2n}
    \implies \lim_{n\to\infty} \left(\frac{1}{2} + \frac{1}{2n}\right) = \dfrac{1}{2}
\end{equation}

Notar que esto es válido solo si almacenamos los coeficientes distintos de $0$. En la práctica estas representaciones deben guardar información adicional sobre la posición de los elementos, entre otros.

Caso $PALU$ (considerando valores no nulos)

```python
(Pc.data.shape[-1] + Lc.data.shape[-1] + Uc.data.shape[-1]) / (3 * N[-1] ** 2)
```

Caso Cholesky (considerando valores no nulos)

```python
Rc.data.shape[-1] / N[-1] ** 2
```

Se hace énfasis en que este análisis solo compara los valores de coeficientes, pero no se incluye el tamaño de las estructuras adicionales que requieren estas representaciones.

## Resolución sistema de ecuaciones

### Sistema de ecuaciones lineales

\begin{equation}
  A \mathbf{x} = \mathbf{b}
\end{equation}

### Resolución utilizando $PA=LU$

\begin{equation}
  \begin{split}
    A\, \mathbf{x} & = \mathbf{b} \\
    PA\, \mathbf{x} & = P\,\mathbf{b} \\
    LU\, \mathbf{x} & = P\, \mathbf{b} \\
    L\, \mathbf{c} & = P\, \mathbf{b}
  \end{split}
\end{equation}

Algoritmo:
1. Obtener descomposición PALU: $PA=LU$
2. Resolver para $\mathbf{c}$: $L\,\mathbf{c} = P\, \mathbf{b}$
3. Resolver para $\mathbf{x}$: $U\,\mathbf{x} = \mathbf{c}$
4. Retornar $\mathbf{x}$

### Resolución utilizando Cholesky

\begin{equation}
  \begin{split}
    A \, \mathbf{x} & = \mathbf{b} \\
    R^T\,R \,\mathbf{x} & = \mathbf{b}
  \end{split}
\end{equation}

Algoritmo:
1. Obtener descomposición de Cholesky: $A=R^T\,R$
2. Resolver para $\mathbf{c}$: $R^T\mathbf{c} = \mathbf{b}$
3. Resolver para $\mathbf{x}$: $R\,\mathbf{x} = \mathbf{c}$
4. Retornar $\mathbf{x}$

## Ejemplo

¿Cómo se resuelven sistemas utilizando las factorizaciones $PA=LU$ y Cholesky?

```python
n = 100 # Probar con distintos valores de n
A = np.random.rand(n, n)
A = np.dot(A, A.T) + np.eye(n)
# Luego de probar con la matriz A simétrica y definida positiva, probar con la matriz de Hilbert...
#A = spla.hilbert(n)
```

```python
np.linalg.cond(A)
```

```python
x = np.arange(1, n+1) # Solucion
b = np.dot(A, x)
```

```python
#x
```

```python
np.all(A == A.T)
```

```python
np.all(np.linalg.eigvals(A) > 0)
```

```python
x_n = np.linalg.solve(A, b)
```

### $PA=LU$

```python
P, L, U = spla.lu(A)
Pb = np.dot(P.T, b) # Permutacion
c = spla.solve_triangular(L, Pb, lower=True) # Lc = Pb
x_p = spla.solve_triangular(U, c) # Ux = c
```

```python
np.linalg.norm(np.dot(P, np.dot(L, U)) - A) # Verificar que PLU = A
```

### Cholesky

```python
R = np.linalg.cholesky(A) # Entrega R^T de acuerdo a nuestras diapositivas
c = spla.solve_triangular(R, b, lower=True) # R^Tc = b
x_c = spla.solve_triangular(R.T, c) # Rx = c
```

```python
np.linalg.norm(np.dot(R, R.T) - A) # Veridiar que R^TR = A
```

```python
# Descomentar para ver soluciones
#print(x_n)
#print(x_p)
#print(x_c)
```

## Backward y Forward Error

Analizar el error de las soluciones...

```python
backwardError = lambda A, x_c, b: np.linalg.norm(b - np.dot(A, x_c), np.inf)
forwardError  = lambda x, x_c: np.linalg.norm(x - x_c, np.inf)
```

```python
backwardError(A, x_n, b), forwardError(x, x_n) # Solver de NumPy
```

```python
backwardError(A, x_p, b), forwardError(x, x_p) # PALU
```

```python
backwardError(A, x_c, b), forwardError(x, x_c) # Cholesky
```


# 重要性采样/Importance Sampling

```python
import random
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## 继续前面的例子

- 我们使用被积函数作为 PDF
    - 但是需要它进行归一化，也就是说在样本空间，也就是$[a,b]$区间内积分要等于 1
    - 这样我们就需要对这个函数乘上一个系数 $C$
    - 通过下面这个简单的推导，就可以得出 $C$ 的值
$$
\begin{align}
& p(x) = C 3 x^2 \\
\Rightarrow & \quad \int_{a}^{b} C 3 x^2 dx = 1 \\
\Rightarrow & \quad C x^3 \big|_{a}^{b} = 1 \\
\Rightarrow & \quad C (b^3 - a^3) = 1 \\
\Rightarrow & \quad C = \frac{1}{b^3 - a^3}
\end{align}
$$

- 求 CDF

$$
P(x) = \int_a^x 3C t^2 \mathrm{d} t \\
P(x) = C t^3 \big|_{a}^{x} \\
P(x) = C (x^3 - a^3)
$$

- 求 CDF 的反函数

$$
P^{-1}(x) = \sqrt[3]{\frac{x}{C} + a^3}
$$

- 所以，我们的采样函数为
    - 其中 $\xi$ 即为 Uniform Sampling 的样本

$$
X = \sqrt[3]{\frac{\xi}{C} + a^3}
$$

```python
# 定义被积函数
def my_f(x):
    return 3*(x**2)

# 定义原函数
def my_F(x):
    return x**3

# 定义积分区间
a = 4
b = 8

# 使用原函数求解析解
def analytic_solution(F, a, b):
    return F(b)-F(a)
```

```python
def my_monte_carlo(func, a, b, n):
    result = 0
    C = (b**3)-(a**3)

    for i in range(n):
        xi = random.uniform(a,b)
        x = (xi/C+a**3)**(1.0/3)
        pdf = 3*(x**2)/C
        y = func(x)
        result += y/pdf

    return result/n
```

```python
analytic = analytic_solution(my_F, a, b)
print("analytic solution = ", analytic)

my_result = my_monte_carlo(my_f, a, b, 10000)
print("my_monte_carlo.result = {result}, error = {error}".format(result=my_result,error=analytic-my_result))
```

    analytic solution =  448
    my_monte_carlo.result = 448.0, error = 0.0

## 完美 PDF 并不存在

通过上述试验，我们发现，同样的样本数量，蒙特卡洛方法的误差已经惊人的降低到了$0$！

另外，你还可以尝试一下，把样本数量设置为1，同样可以得到误差为 0 的结果！这是因为我们使用了完美的 PDF 函数，并且成功求出了它的 CDF，以及 CDF 的反函数。这只是一个简单的试验的例子，这种理想的情况在实际工作中是不存在的。因为如果你可以像我前面那面求出完美的 PDF 的话，你也就可以得到解析解，也就用不到蒙特卡洛方法了。:)

在后面对于 BRDF 那种复杂函数采样的话，一般会选用一个尽量和它的分布接近的简单函数作为 PDF，然后推导出重要性采样的方法。

## 参考资料

- [Physically Based Rendering: Sampling Random Variables](https://www.pbr-book.org/3ed-2018/Monte_Carlo_Integration/Sampling_Random_Variables)
- [UCSD CSE 168 - Computer Graphics II: Rendering](https://cseweb.ucsd.edu/~viscomp/classes/cse168/sp20/info.html)

```python

```


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

## Marginal and Conditional Densities ##

Let random variables $X$ and $Y$ have the joint density defined by
$$
f(x, y) ~ = ~
\begin{cases}
30(y-x)^4, ~~~ 0 < x < y < 1 \\
0 ~~~~~~~~ \text{otherwise}
\end{cases}
$$

```python
def jt_dens(x,y):
    if y < x:
        return 0
    else:
        return 30 * (y-x)**4

Plot_3d(x_limits=(0,1), y_limits=(0,1), f=jt_dens, cstride=4, rstride=4)
```

Then the possible values of $(X, Y)$ are in the upper right hand triangle of the unit square.

```python
# HIDDEN
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('Possible Values of $(X, Y)$');
```

Here is a quick check by `SymPy` to see that the function $f$ is indeed a joint density.

```python
declare('x', interval=(0, 1))
declare('y', interval=(0, 1))
joint_density = 30*(y-x)**4
```

```python
Integral(joint_density, (y, x, 1), (x, 0, 1)).doit()
```

### Density of $X$ ###
We can use the joint density $f$ to find the density of $X$. Call this density $f_X$. We know that

\begin{align*}
f_X(x)dx &\sim P(X \in dx) \\
&= \int_y P(X \in dx, Y \in dy) \\
&= \int_y f(x, y)dxdy \\
&= \big{(} \int_y f(x, y)dy \big{)}dx
\end{align*}

You can see the reasoning behind this calculation in the graph below. The blue strip shows the event $\{ X \in dx \}$ for a value of $x$ very near 0.25. To find the volume $P(X \in dx)$, we hold $x$ fixed and add over all $y$.

```python
# HIDDEN
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
plt.plot([0.25, 0.25], [0.25, 1], color='blue', lw=3, alpha=0.3)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$X \in dx$');
```

So the density of $X$ is given by
$$
f_X(x) ~ = ~ \int_y f(x, y)dy ~~~~~ \text{for all } x
$$

By analogy with the discrete case, $f_X$ is sometimes called the *marginal density* of $X$.

In our example, the possible values of $(X, Y)$ are the upper left hand triangle as shown above. So for each fixed $x$, the possible values of $Y$ go from $x$ to 1.

Therefore for $0 < x < 1$, the density of $X$ is given by

\begin{align*}
f_X(x) &= \int_x^1 30(y-x)^4 dy \\
&= 30 \cdot \frac{1}{5} (y-x)^5 \Big{\rvert}_x^1 \\
&= 6(1-x)^5
\end{align*}

Here is the joint density surface again. You can see that $X$ is much more likely to be near 0 than near 1.

```python
Plot_3d(x_limits=(0,1), y_limits=(0,1), f=jt_dens, cstride=4, rstride=4)
```

That can be seen in the shape of the density of $X$.

```python
# HIDDEN
x_vals = np.arange(0, 1.01, 0.01)
f_X = 6*(1-x_vals)**5
plt.plot(x_vals, f_X, color='darkblue', lw=2)
plt.xlabel('$x$')
plt.ylabel('$f_X(x)$', rotation=0)
plt.title('$f_X$: Density of $X$');
```

### Density of $Y$ ###
Correspondingly, the density of $Y$ can be found by fixing $y$ and integrating over $x$ as follows:

$$
f_Y(y) = \int_x f(x, y)dx ~~~~ \text{for all } y
$$

In our example, the joint density surface indicates that $Y$ is more likely to be near 1 than near 0, which is confirmed by calculation. Remember that $y > x$ and therefore for each fixed $y$, the possible values of $x$ are 0 through $y$.

For $0 < y < 1$,
$$
f_Y(y) ~ = ~ \int_0^y 30(y-x)^4dx ~ = ~ 6y^5
$$

```python
# HIDDEN
y_vals = np.arange(0, 1.01, 0.01)
f_Y = 6*y_vals**5
plt.plot(y_vals, f_Y, color='darkblue', lw=2)
plt.xlabel('$y$')
plt.ylabel('$f_Y(y)$', rotation=0)
plt.title('$f_Y$: Density of $Y$');
```

### Conditional Densities ###
Consider the conditional probability $P(Y \in dy \mid X \in dx)$. By the division rule,

$$
P(Y \in dy \mid X \in dx) ~ = ~
\frac{P(X \in dx, Y \in dy)}{P(X \in dx)} ~ = ~
\frac{f(x, y)dxdy}{f_X(x)dx} ~ = ~
\frac{f(x, y)}{f_X(x)} dy
$$

This gives us a division rule for densities. For a fixed value $x$, the *conditional density of $Y$ given $X=x$* is defined by

$$
f_{Y\mid X=x} (y) ~ = ~ \frac{f(x, y)}{f_X(x)} ~~~~ \text{for all } y
$$

Since $X$ has a density, we know that $P(X = x) = 0$ for all $x$. But the ratio above is of densities, not probabilities. It might help your intuition to think of "given $X=x$" to mean "given that $X$ is just around $x$".

Visually, the shape of this conditional density is the vertical cross section at $x$ of the joint density graph above. The numerator determines the shape, and the denominator is part of the constant that makes the density integrate to 1.

Note that $x$ is constant in this formula; it is the given value of $X$. So the denominator $f_X(x)$ is the same for all the possible values of $y$.

To see that the conditional density does integrate to 1, let's do the integral.

$$
\int_y f_{Y\mid X=x} (y)dy ~ = ~ \int_y \frac{f(x, y)}{f_X(x)} dy ~ = ~
\frac{1}{f_X(x)} \int_y f(x, y)dy ~ = ~ \frac{1}{f_X(x)} f_X(x) ~ = ~ 1
$$

In our example, let $x = 0.2$ and consider finding the conditional density of $Y$ given $X = 0.4$. Under that condition, the possible values of $Y$ are in the range 0.4 to 1, and therefore

$$
f_{Y \mid X=0.4} (y) ~ = ~ \frac{30(y - 0.4)^4}{6(1 - 0.4)^5} ~ = ~
\frac{5}{0.6^5} (y - 0.4)^4 ~~~~ y \in (0.4, 1)
$$

This is a density on $(0.4, 1)$:

```python
declare('y', interval=(0.4, 1))
cond_density = (5/(0.6**5)) * (y - 0.4)**4
Integral(cond_density, (y, 0.4, 1)).doit()
```

The figure below shows the overlaid graphs of the density of $Y$ and the conditional density of $Y$ given $X = 0.4$. You can see that the conditional density is more concentrated on large values of $Y$, because under the condition $X = 0.4$ you know that $Y$ can't be small.

```python
# HIDDEN
plt.plot(y_vals, f_Y, color='darkblue', lw=2, label='Density of $Y$')
new_y = np.arange(0.4, 1.01, 0.01)
dens = (5/(0.6**5)) * (new_y - 0.4)**4
plt.plot(new_y, dens, color='gold', lw=2, label='Density of $Y$ given $X=0.4$')
plt.legend()
plt.xlim(0, 1)
plt.xlabel('$y$');
```

#### Using a Conditional Density ####
We can use conditional densities to find probabilities and expectations, just as we would use an ordinary density. Here are some examples of calculations. In each case we will set up the integrals and then use `SymPy`.

$$
P(Y > 0.9 \mid X = 0.4) = \int_{0.9}^1 \frac{5}{0.6^5} (y - 0.4)^4 dy
$$

The answer is about 60%.

```python
declare('y', interval=(0, 1))
cond_density = (5/(0.6**5))*(y - 0.4)**4
Integral(cond_density, (y, 0.9, 1)).doit()
```

Now we will use the conditional density to find a conditional expectation. Remember that in our example, given that $X = 0.4$ the possible values of $Y$ go from $0.4$ to 1.

$$
E(Y \mid X = 0.4) ~ = ~ \int_{0.4}^1 y \frac{5}{0.6^5} (y - 0.4)^4 dy ~ = ~ 0.9
$$

```python
Integral(y*cond_density, (y, 0.4, 1)).doit()
```

For any fixed value of $y$, the conditional density of $X$ given $Y = y$ is

$$
f_{X \mid Y=y} (x) ~ = ~ \frac{f(x, y)}{f_Y(y)} ~~~~~ \text{for all } x
$$

All the examples in this section and the previous one have started with a joint density function that apparently emerged out of nowhere. In the next section, we will study a context in which they arise.

```python

```


# From transfer function to difference equation
In approximately the middle of Peter Corke's lecture [Introduction to digital control](https://youtu.be/XuR3QKVtx-g?t=34m56s), he explaines how to go from a transfer function description of a controller (or compensator) to a difference equation that can be implemented on a microcontroller.

The idea is to recognize that the term $$ sX(s) $$ in a transfer function is the laplace transform of the derivative of $x(t)$,
\begin{equation}
sX(s) + x(0) \quad \overset{\mathcal{L}}{\longleftrightarrow} \quad \frac{d}{dt} x(t),
\end{equation}
where the inital value $x(0)$ is often taken to be zero. We then make use of a discrete approximation of the derivative
$$ \frac{d}{dt}x(t) \approx \frac{x(t-h) - x(t)}{h}, $$
where $h$ is the time between the samples in the sampled version of signal $x(t)$.

The steps to convert the system on transfer function form
$$ Y(s) = F(s)U(s) = \frac{s+b}{s+a}U(s) $$
are to write
$$ (s+a)Y(s) = (s+b)U(s) $$
$$ sY(s) + aY(s) = sU(s) + bU(s), $$
take the inverse Laplace transform
$$ \frac{d}{dt} y + ay = \frac{d}{dt} u + bu$$
and use the discrete approximation of the derivative
$$ \frac{y_k - y_{k-1}}{h} + ay_k = \frac{u_k - u_{k-1}}{h} + bu_k $$
which can be written
$$ (1+ah) y_k = y_{k-1} + u_k - u_{k-1} + bh u_k,$$
or
$$ y_k = \frac{1}{1+ah} y_{k-1} + \frac{1+bh}{1+ah}u_k - \frac{1}{1+ah}u_{k-1}. $$

## Example
With the system
$$ F(s) =  \frac{s+1}{s+2} $$
and the sampling time
$$ h=0.1 $$
we get the difference equation
$$ y_k = \frac{1}{1.2}y_{k-1} + \frac{1.1}{1.2}u_k - \frac{1}{1.2} u_{k-1}. $$

Let's implement the system and see how the discrete approximation compares to the continuous-time system for the case of a step-response.

```python
import numpy as np
import scipy.signal as signal
import matplotlib.pyplot as plt
%matplotlib inline

# Define the continuous-time linear time invariant system F
a = 2
b = 1
num = [1, b]
den = [1, a]
F = signal.lti(num, den)

# Plot a step response
(t, y) = signal.step(F)
plt.figure(figsize=(14,6))
plt.plot(t, y, linewidth=2)

# Solve the difference equation y_k = c y_{k-1} + d_0 u_k + d_1 u_{k-1}
h = 0.1 # The sampling time

c = 1.0/(1 + a*h)
d0 = (1 + b*h) / (1 + a*h)
d1 = -c

td = np.arange(35)* h #The sampling time instants
ud = np.ones(35) # The input signal is a step, limited in time to 3.5 seconds
yd = np.zeros(35) # A vector to hold the discrete output signal
yd[0] = c*0 + d0*ud[0] - d1*0 # The first sample of the output signal
for k in range(1,35): # And then the rest
    yd[k] = c*yd[k-1] + d0*ud[k] + d1*ud[k-1]

plt.plot(td, yd, 'o', markersize=8)
plt.xlim([0, 3.5])
plt.ylim([0, 1])
plt.legend(('Continuous-time system', 'Discrete approximation'))
```

## Exercise
1. Why is the error in the discrete approximation larger in the beginning than at the end of the step-response?
2. Make a discrete approximation of the transfer function $$ F(s) = \frac{3}{s+3} $$ using the sampling time $$ h=0.2 $$ Then simulate and plot a step-response for the continuous- and discrete system, following the example above. *Hint*: Copy the python code for the example above into the code cell below and modify for the exercise.

```python
## Your python code goes here
```

# Recursively computing values of a polynomial using difference equations
In the lecture by Peter Corke, he talks about the historical importance of difference equations for computing values of a polynomial. Let's look at this in some more detail.

## A first order polynomial
Consider the polynomial
$$ p(x) =  4x + 2. $$
The first difference is
$$ \Delta p(x) = p(x) - p(x-h) = 4x + 2 - \big( 4(x-h) + 2 \big) = 4h, $$
and the second order difference is zero (as are all higher order differences):
$$ \Delta^2 p(x) = \Delta p(x) - \Delta p(x-h) = 4h - 4h = 0. $$

Using the firs order difference, we can also write the second order difference $ \Delta p(x) - \Delta p(x-h) = \Delta^2 p(x) $
as
$$ p(x) - p(x-h) - \Delta p(x-h) = \Delta^2p(x) $$
or
$$ p(x) = p(x-h) + \Delta p(x-h) + \Delta^2 p(x)$$
which for the first order polynomial above becomes
$$ p(x) = p(x-h) + \Delta p(x-h) = p(x-h) + 4h. $$

```python
def p1(x): return 4*x + 2 # Our first-order polynomial
# Compute values for x=[0,0.2, 0.4, ... 2] recursively using the difference equation
h = 0.2
x = h*np.arange(11) # Gives the array [0,0.2, 0.4, ... 2]
pd = np.zeros(11)
d1 = 4*h

# Need to compute the first value as the initial value for the difference equation,
pd[0] = p1(x[0])

for k in range(1,10): # Solve difference equation
    pd[k] = pd[k-1] + d1

plt.figure(figsize=(14,6))
plt.plot(x, p1(x), linewidth=2)
plt.plot(x, pd, 'ro')
```

## Second order polynomial
For a second order polynomial
$$ p(x) = a_0x^2 + a_1x + a_2 $$
we have
$$ p''(x) = 2a_0, $$
and the differences
$$ \Delta p(x) = p(x) - p(x-h) = a_0x^2 + a_1x + a_2 - \big( a_0(x-h)^2 + a_1(x-h) + a_2 \big) = h(2a_0x + a_1) -a_0h^2, $$
$$ \Delta^2 p(x) = \Delta p(x) - \Delta p(x-h) = h(2a_0x+a_1) - a_0h^2 - \big( h(2a_0(x-h) + a_1) - a_0 h^2 \big) =  h^22a_0 $$

Recall the difference equation using the second order difference
$$ p(x) = p(x-h) + \Delta p(x-h) + \Delta^2 p(x)$$
We now get
$$ p(x) = p(x-h) + \Delta p(x-h) + \Delta^2 p(x) = p(x-h) + \Delta p(x-h) + h^2 2 a_0,$$
or, using the definition of the first-order difference $\Delta p(x-h)$
$$ p(x) = 2p(x-h) - p(x-2h) + h^2 2 a_0,$$

Consider the second order polynomial
$$ p(x) = 2x^2 - 3x + 2, $$
and compute values using the difference equation.

```python
a0 = 2
a1 = -3
a2 = 2
def p2(x): return a0*x**2 + a1*x + a2 # Our second-order polynomial

# Compute values for x=[0,0.2, 0.4, ... 8] recursively using the difference equation
h = 0.2
x = h*np.arange(41) # Gives the array [0,0.2, 0.4, ... 2]
d1 = np.zeros(41) # The first differences
pd = np.zeros(41)
d2 = h**2*2*a0 # The constant, second difference

# Need to compute the first two values to get the initial values for the difference equation,
pd[0] = p2(x[0])
pd[1] = p2(x[1])

for k in range(2,41): # Solve difference equation
    pd[k] = 2*pd[k-1] - pd[k-2] + d2

plt.figure(figsize=(14,6))
plt.plot(x, p2(x), linewidth=2) # Evaluating the polynomial
plt.plot(x, pd, 'ro') # The solution using the difference equation
```

## Exercise
What order would the difference equation be for computing valuse of a third-order polynomial? What is the difference equation?

```python

```


# Lecture 8: Random Variables and Their Distributions

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Two Basic Types of Random Variables

There are two basic types of random variables.

* _Discrete random variables_ are random variables whose values can be enumerated, such as $a_1, a_2, \dots , a_n$. Here, _discrete_ does not necessarily mean only integers.

* _Continuous random variables_, on the other hand, can take on values in $\mathbb{R}$.

Note that there are other types that might mix the two definitions.

### Cumulative distribution function

Given $X \le x$ is an event, and a function $F(x) = P(X \le x)$.

Then $F$ is known as the __cumulative distribution function__ (CDF) of $X$. In contrast with the probability mass function, the CDF is more general, and is applicable for both discrete and continuous random variables.

### Probability mass function

The __probability mass function__ (PMF) is $P(X=a_j)$, for all $\text{j}$. A pmf must satisfy the following conditions:

\begin{align}
  P_j &\ge 0 \\
  \sum_j P_j &= 1
\end{align}

In practice, using a PMF is easier than a CDF, but a PMF is only applicable in the case of __discrete random variables__ only.

----

## Revisiting the Binomial Distribution $\operatorname{Bin}(n,p)$

$X \sim \operatorname{Bin}(n, p)$, where

* $n$ is an integer greater than 0
* $p$ is between 0.0 and 1.0

Here are a few different ways to explain the Binomial distribution.

### Explanation with a story

$X$ is the number of successes in $n$ _independent_ $\operatorname{Bern}(p)$ trials.

### Explanation using sum of indicator random variables

\begin{align}
  &X = X_1 + X_2 + \dots + X_j
  & &\text{where } X_j =
  \begin{cases}
    1, &\text{if } j^{\text{th}} \text{ trial succeeds} \\
    0, &\text{ otherwise }
  \end{cases} \\
  & & &\text{and } X_1, X_2, \dots , X_j \text{ are i.i.d } \operatorname{Bern}(n,p)
\end{align}

### Explanation using a probability mass function

\begin{align}
  P(X=k) = \binom{n}{k} p^k q^{n-k} \text{, where } q = 1 - p \text{ and } k \in \{0,1,2, \dots ,n\}
\end{align}

A probability mass function sums to 1. Note that

\begin{align}
  \sum_{k=0}^n \binom{n}{k} p^k q^{n-k} &= (p + q)^n & &\text{by the Binomial Theorem} \\
  &= (p + 1 - p)^n \\
  &= 1^n \\
  &= 1
\end{align}
so the explanation by PMF holds. By the way, it is this relation to the [Binomial Theorem](https://en.wikipedia.org/wiki/Binomial_theorem) that this distribution gets its name.

----

## Sum of Two Binomial Random Variables

Recall that we left Lecture 7 with this parting thought:

\begin{align}
    & X \sim \operatorname{Bin}(n,p) \text{, } Y \sim \operatorname{Bin}(m,p) \rightarrow X+Y \sim \operatorname{Bin}(n+m, p)
\end{align}

Let's see how this is true by using the three explanation approaches we used earlier.

### Explanation with a story

$X$ and $Y$ are functions, and since they both have the same sample space $S$ and the same domain.

$X$ is the number of successes in $n \operatorname{Bern}(p)$ trials.

$Y$ is the number of successes in $m \operatorname{Bern}(p)$ trials.

$\therefore X + Y$ is simply the sum of successes in $n+m \operatorname{Bern}(p)$ trials.

### Explanation using a sum of indicator random variables

\begin{align}
    &X = X_1 + X_2 + \dots + X_n, ~~~~ Y = Y_1 + Y_2 + \dots + Y_m \\
    &\Rightarrow X+Y = \sum_{i=1}^n X_j + \sum_{j=1}^m Y_j \text{, which is } n + m \text{ i.i.d. } \operatorname{Bern}(p) \\
    &\Rightarrow \operatorname{Bin}(n + m, p)
\end{align}

### Explanation using a probability mass function

We start with a PMF of the _convolution_ $X+Y=k$

\begin{align}
    P(X+Y=k) &= \sum_{j=0}^k P(X+Y=k|X=j)P(X=j) & &\text{wishful thinking, assume we know } x \text{ and apply LOTP} \\
    &= \sum_{j=0}^k P(Y=k-j|X=j) \binom{n}{j} p^j q^{n-j} & &\text{... but since events }Y=k-j \text{ and } X=j \text{ are independent}\\
    &= \sum_{j=0}^k P(Y=k-j) \binom{n}{j} p^j q^{n-j} & &P(Y=k-j|X=j) = P(Y=k-j) \\
    &= \sum_{j=0}^k \binom{m}{k-j} p^{k-j} q^{m-k+j}~~~ \binom{n}{j} p^j q^{n-j} \\
    &= p^k q^{m+n-k} \underbrace{\sum_{j=0}^k \binom{m}{k-j}\binom{n}{j}}_{\text{Vandermonde's Identity}} & &\text{see Lecture 2, Story Proofs, Ex.3} \\
    &= \binom{m+n}{k} p^k q^{m+n-k} ~~~~ \blacksquare
\end{align}

----

## Hypergeometric Distribution

A common mistake in trying to use the Binomial distribution is forgetting that

* the trials are independent
* they have the same probability of success

#### Example: How Many Aces in a 5-card Hand?

A 5-card hand out of a standard 52-card deck of playing cards, what is the the distribution (PMF or CDF) of the number of aces.

Let $X = (\text{# of aces})$.

Find $P(X=k)$ where $x \in \text{\{0,1,2,3,4\}}$. At this point, we can conclude that the distribution is __not__ Binomial, _as the trials are not independent_.

The PMF is given by
\begin{align}
   P(X=k) &= \frac{\binom{4}{k}\binom{48}{5-k}}{\binom{52}{5}}
\end{align}

#### Example: Sampling White/Black Marbles

Suppose we have a total of $b$ black and $w$ white marbles. Pick a random sample of $n$ marbles. What is the distribution on the # white marbles?

\begin{align}
    P(X=k) &= \frac{\binom{w}{k}\binom{b}{n-k}}{\binom{w+b}{n}}
\end{align}

The distribution in the examples above is called the __Hypergeometric distribution__. In the hypergeometric distribution, we sample _without replacement_, and so the trials cannot be independent.

Note that when the total number of items in our sample is exceedingly large, it becomes highly unlikely that we would choose the same item _more than once_. Therefore, it doesn't matter if you are sampling with replacement or without, suggesting that in this case the hypergeometric behaves as the binomial.

#### Is the Hypergeometric distribution a valid PMF?

_Is the hypergeometric non-negative?_

Yes, obviously.

_Does the PMF sum to 1?_

\begin{align}
    P(X=k) &=  \sum_{k=0}^w \frac{\binom{w}{k}\binom{b}{n-k}}{\binom{w+b}{n}} \\
    &= \frac{1}{\binom{w+b}{n}} \sum_{k=0}^w \binom{w}{k}\binom{b}{n-k} & &\text{since }\binom{w+b}{n} \text{ is constant} \\
    &= \frac{1}{\binom{w+b}{n}} \binom{w+b}{n} & &\text{by Vandermonde's Identity } \\
    &= 1 ~~~~ \blacksquare
\end{align}

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.stats import hypergeom

%matplotlib inline

plt.xkcd()
_, ax = plt.subplots(figsize=(12,8))

# seme Hypergeometric parameters
# population and # white marbles are fixed
population_sizes = [200, 200, 200]
w_marbles = [30, 30, 30]
# let sample size n vary
sample_sizes = [30, 60, 90]
params = list(zip(population_sizes, w_marbles, sample_sizes))

# colorblind-safe, qualitative color scheme
colors = ['#1b9e77', '#d95f02', '#7570b3']

for i,(N,w,n) in enumerate(params):
    rv = hypergeom(N, w, n)
    x = np.arange(0, w+1)
    pmf_w = rv.pmf(x)
    ax.plot(x, pmf_w, 'o', ms=8, color=colors[i], label='n={}'.format(n))
    ax.vlines(x, 0, pmf_w, lw=2, color=colors[i], alpha=0.3)

# legend styling
legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

# y-axis
ax.set_ylim([0.0, 0.25])
ax.set_ylabel(r'$P(X=k)$')

# x-axis
ax.set_xlim([0, 25])
ax.set_xlabel('# of white marbles k out of {} in sample size n'.format(w))

# x-axis tick formatting
majorLocator = MultipleLocator(5)
majorFormatter = FormatStrFormatter('%d')
minorLocator = MultipleLocator(1)
ax.xaxis.set_major_locator(majorLocator)
ax.xaxis.set_major_formatter(majorFormatter)
ax.xaxis.set_minor_locator(minorLocator)

ax.grid(color='grey', linestyle='-', linewidth=0.3)

plt.suptitle(r'Hypergeometric PMF: $P(X=k) = \frac{\binom{w}{k}\binom{b}{n-k}}{\binom{w+b}{n}}$')
plt.title(r'({} white + {} black marbles)'.format(w, N-w))

plt.show()
```

----

## Appendix A: Independent, Identically Distributed

A sequence of random variables are independent and identically distributed (i.i.d) if each r.v. has the same probability distribution and all are mutually independent.

Think of the Binomial distribution where we consider a string of coin flips. The probability $p$ for head's is the *same across all coin flips*, and seeing a head (or tail) in a previous toss *does not affect any of the coin flips that come afterwards*.

----

View [Lecture 8: Random Variables and Their Distributions | Statistics 110](http://bit.ly/2MlKHmi) on YouTube.