## pomegranate: fast and flexible probabilistic modelling

Author: Jacob Schreiber

Contact: <jmschreiber91@gmail.com>

```python
%matplotlib inline
import time
import pandas
import random
import numpy
import matplotlib.pyplot as plt
import seaborn; seaborn.set_style('whitegrid')
import itertools

from pomegranate import *

random.seed(0)
numpy.random.seed(0)
numpy.set_printoptions(suppress=True)

%load_ext watermark
%watermark -m -n -p numpy,scipy,pomegranate
```

    Fri Jan 10 2020

    numpy 1.18.1
    scipy 1.4.1
    pomegranate 0.12.0

    compiler   : Clang 10.0.0 (clang-1000.11.45.5)
    system     : Darwin
    release    : 17.7.0
    machine    : x86_64
    processor  : i386
    CPU cores  : 4
    interpreter: 64bit

pomegranate is a probabilistic modeling library for Python that aims to be easy to use, flexible, and fast. It is easy to use because it maintains a consistent and minimal API that mirrors the scikit-learn API when possible. It is flexible because it allows complicated probabilistic models to be built from simpler ones by easily stacking components on top of each other. It is fast due to its Cython backend which allows for fast numerics and multi-threading without cluttering the interface. pomegranate was developed at the University of Washington with the generous support from the following organizations:

### The API

One of the core tenant of pomegranate is that everything is fundamentally a probability distribution and should be treated the same way. Most people are familiar with simple probability distributions such as the normal or the uniform distribution. However, a mixture of many simple distributions is still a probability distribution because the density under it still adds up to 1. A hidden Markov model is still a probability distribution over sequences because the probability of all potential sequences must add up to 1. A Bayesian network is literally a probability distribution that has been factorized along a graphical structure, with a discrete Bayesian network factorizing a very large joint probability table and a linear Gaussian Bayesian network factorizing a multivariate Gaussian distribution that has many 0's in the inverse covariance matrix.

A benefit of considering all methods as probability distributions is that it simplifies and unifies the API across all of the models. The common API is the following methods. Let's specify some distribution and take a look.

```python
model = NormalDistribution(5, 1)
```

***model.probability(X)*** and ***model.log_probability(X)***

These methods return the probability of a single sample or a vector of probabilities if provided a vector of samples. This is equal to P(D|M) where D is the data and M is the model.

```python
print(model.probability([4., 6., 7.]))
print(model.log_probability([4., 6., 7.]))
```

    [0.24197072 0.24197072 0.05399097]
    [-1.41893853 -1.41893853 -2.91893853]

***model.sample(n=1)***

This method will return a random sample from the probability distribution or a vector of random samples if n is set to be greater than 1.

```python
model.sample(n=10)
```

    array([6.76405235, 5.40015721, 5.97873798, 7.2408932 , 6.86755799,
           4.02272212, 5.95008842, 4.84864279, 4.89678115, 5.4105985 ])

***model.fit(X, weights=None, inertia=0.0, other arguments)***

This method will fit the distribution to the data. For simple distributions and Bayesian networks this corresponds to weighted maximum likelihood estimates. For other compositional distributions such as hidden Markov models this corresponds to an iterative expectation-maximization allgorithm. Optionally, weights can be provided for each of the samples. Different models will have different arguments that can be specified as well.

```python
X = numpy.random.normal(7, 2, size=(100,))

model.fit(X)
model
```

    {
        "class" :"Distribution",
        "name" :"NormalDistribution",
        "parameters" :[
            7.036955079959913,
            2.0668514243367655
        ],
        "frozen" :false
    }

***model.summary(X, weights=None)***

This method implements the first part of the fitting process, which is summarizing a batch of data down to its sufficient statistics and storing them. These sufficient statistics are additive and will be updated for each successive batch that's seen. The sufficient statistics can be reset either through a call to `from_summaries` or a call to `clear_summaries`.

```python
X = numpy.random.normal(8, 1.5, size=(100,))

model.summarize(X)
model
```

    {
        "class" :"Distribution",
        "name" :"NormalDistribution",
        "parameters" :[
            7.036955079959913,
            2.0668514243367655
        ],
        "frozen" :false
    }

***model.from_summaries(inertia=0.0)***

This method implements the second part of the fitting process, which is using the stored sufficient statistics in order to update the model parameters.

```python
model.from_summaries()
model
```

    {
        "class" :"Distribution",
        "name" :"NormalDistribution",
        "parameters" :[
            8.068146018374335,
            1.4490322240381275
        ],
        "frozen" :false
    }

***model.clear_summaries()***

This method resets the sufficient statistics stored to the model.

```python
X = numpy.random.normal(3, 0.2, size=(100,))

model.summarize(X)
model.clear_summaries()
model.from_summaries()
model
```

    {
        "class" :"Distribution",
        "name" :"NormalDistribution",
        "parameters" :[
            8.068146018374335,
            1.4490322240381275
        ],
        "frozen" :false
    }

***Model.from_samples(distributions, n_components, X, weights=None)*** or ***Model.from_samples(X, weights=None)***

This class method will initialize and then fit the parameters of a model to some data. This differs from the `fit` function in that the `fit` function will update the parameters of a pre-defined model whereas the `from_samples` method will initialize a model to data and return the best parameters given the data.

```python
X = numpy.random.normal(6, 1, size=(250, 1))

model = NormalDistribution.from_samples(X)
model
```

    {
        "class" :"Distribution",
        "name" :"NormalDistribution",
        "parameters" :[
            5.878631757664192,
            1.0120288481111088
        ],
        "frozen" :false
    }

```python
model = GeneralMixtureModel.from_samples(NormalDistribution, 3, X)
model
```

    {
        "class" : "GeneralMixtureModel",
        "distributions" : [
            {
                "class" : "Distribution",
                "name" : "NormalDistribution",
                "parameters" : [
                    5.901815709887363,
                    0.47865628749453537
                ],
                "frozen" : false
            },
            {
                "class" : "Distribution",
                "name" : "NormalDistribution",
                "parameters" : [
                    5.175959799664007,
                    0.8774489898709738
                ],
                "frozen" : false
            },
            {
                "class" : "Distribution",
                "name" : "NormalDistribution",
                "parameters" : [
                    6.9902780582522634,
                    0.597808757836137
                ],
                "frozen" : false
            }
        ],
        "weights" : [
            0.3148521575093671,
            0.42381846635660686,
            0.261329376134026
        ]
    }

***model.to_json(separators=(',', ' :'), indent=4)***

This method returns the JSON serialization of the distribution as a string.

```python
print(model.to_json())
```

    {
        "class" : "GeneralMixtureModel",
        "distributions" : [
            {
                "class" : "Distribution",
                "name" : "NormalDistribution",
                "parameters" : [
                    5.901815709887363,
                    0.47865628749453537
                ],
                "frozen" : false
            },
            {
                "class" : "Distribution",
                "name" : "NormalDistribution",
                "parameters" : [
                    5.175959799664007,
                    0.8774489898709738
                ],
                "frozen" : false
            },
            {
                "class" : "Distribution",
                "name" : "NormalDistribution",
                "parameters" : [
                    6.9902780582522634,
                    0.597808757836137
                ],
                "frozen" : false
            }
        ],
        "weights" : [
            0.3148521575093671,
            0.42381846635660686,
            0.261329376134026
        ]
    }

***Model.from_json(s)***

This class method returns the deserialization of the JSON string back to the model object. s can either be the JSON string or a filename ending in `.json` to read from.

```python
model = NormalDistribution(5, 2)

model2 = Distribution.from_json(model.to_json())
model2
```

    {
        "class" :"Distribution",
        "name" :"NormalDistribution",
        "parameters" :[
            5.0,
            2.0
        ],
        "frozen" :false
    }

Compositional models, i.e., those that are not simple distributions, have three additional methods whose named are inspired by scikit-learn. These methods relate to the posterior probabilities P(M|D) of each of the components of the model given some data.

```python
d1 = ExponentialDistribution(5.0)
d2 = ExponentialDistribution(0.3)

model = GeneralMixtureModel([d1, d2])
model
```

    {
        "class" : "GeneralMixtureModel",
        "distributions" : [
            {
                "class" : "Distribution",
                "name" : "ExponentialDistribution",
                "parameters" : [
                    5.0
                ],
                "frozen" : false
            },
            {
                "class" : "Distribution",
                "name" : "ExponentialDistribution",
                "parameters" : [
                    0.3
                ],
                "frozen" : false
            }
        ],
        "weights" : [
            0.5,
            0.5
        ]
    }

***model.predict(X)***

This method returns the most likely component for each sample. In the case of a mixture model it returns the component that is most likely, in the case of a hidden Markov model it returns the most likely component for each observation in the sequence.

```python
X = numpy.random.exponential(3, size=(10,1))

model.predict(X)
```

    array([1, 0, 0, 0, 1, 0, 0, 1, 0, 1], dtype=int32)

***model.predict_proba(X)***

This method returns the probability of each component for each sample. It is similar to the `predict` method except that it returns the probabilities instead of simply the most likely component.

```python
model.predict_proba(X)
```

    array([[0.00046883, 0.99953117],
           [0.74132615, 0.25867385],
           [0.58846532, 0.41153468],
           [0.89449455, 0.10550545],
           [0.        , 1.        ],
           [0.94041259, 0.05958741],
           [0.50991095, 0.49008905],
           [0.00002574, 0.99997426],
           [0.83434281, 0.16565719],
           [0.        , 1.        ]])

***model.predict_log_proba(X)***

Like predict_proba except that it returns the log probabilities instead of the probabilities.

```python
model.predict_log_proba(X)
```

    array([[ -7.66527111,  -0.00046894],
           [ -0.29931461,  -1.35218726],
           [ -0.53023729,  -0.88786198],
           [ -0.11149647,  -2.24899266],
           [-46.97709403,   0.        ],
           [ -0.06143657,  -2.820311  ],
           [ -0.67351918,  -0.71316817],
           [-10.56732239,  -0.00002574],
           [ -0.18111092,  -1.79783472],
           [-27.30692393,  -0.        ]])

### Flexibility

#### Modeling different features as different distributions

A second benefit of treating all models as probability distributions is that it greatly increases the flexibility that pomegranate provides. When people build naive Bayes classifiers, they typically will use a Gaussian distribution. However, there's no reason that one has to use a Gaussian distribution, you can drop in any type of distribution that you'd like. For example:

```python
X = numpy.random.normal(5, 1, size=(100, 2))
X[50:] += 1

y = numpy.zeros(100)
y[50:] = 1

model1 = NaiveBayes.from_samples(NormalDistribution, X, y)
model2 = NaiveBayes.from_samples(LogNormalDistribution, X, y)
```

It is easy to drop in whatever probability distribution you'd like because it should be easy. Mathematically, the naive Bayes model relies on Bayes' rule, which says:

\begin{equation}
P(M|D) = \frac{P(D|M)P(M)}{P(D)}
\end{align}

Because the "naive" part of a "naive" Bayes model means that the model treats each feature independently, we can rewrite the $P(D|M)$ aspect as the product of these probabilities over all $d$ features:

\begin{equation}
P(M|D) = \frac{P(M) \prod\limits_{i=1}^{d} P(D_{i}|M)}{P(D)}
\end{equation}

Now, because each feature is independent, they can be modeled by different probability distributions.

```python
mu = numpy.random.normal(7, 2, size=1000)
std = numpy.random.lognormal(-0.8, 0.8, size=1000)
dur = numpy.random.exponential(50, size=1000)

data = numpy.concatenate([numpy.random.normal(mu_, std_, int(t)) for mu_, std_, t in zip(mu, std, dur)])

plt.figure(figsize=(14, 4))
plt.title("Randomly Generated Signal", fontsize=16)
plt.plot(data)
plt.xlabel("Time", fontsize=14)
plt.ylabel("Signal", fontsize=14)
plt.xlim(0, 3000)
plt.show()
```

If someone was trying to model this signal, they could potentially try to segment it and then extract properties of those segments, such as the mean, the variance, and the duration. We can plot what thos would look like:

```python
plt.figure(figsize=(12, 3))
plt.subplot(131)
plt.title("mu", fontsize=14)
plt.hist(mu, bins=numpy.arange(0, 15))

plt.subplot(132)
plt.title("sigma", fontsize=14)
plt.hist(std, bins=numpy.arange(0.00, 1.75, 0.05))

plt.subplot(133)
plt.title("Duration", fontsize=14)
plt.hist(dur, bins=numpy.arange(0, 150, 10))
plt.show()
```

We can see that the mean of the segments does have a Gaussian distribution, but that neither the standard deviation or the duration do. It would be suboptimal to assume all features were Gaussian, as a simple Gaussian naive Bayes model would. pomegranate allows you to define a different distribution for each feature, like the following:

```python
X1 = numpy.array([numpy.random.normal(7, 2, size=400),
                  numpy.random.lognormal(-0.8, 0.8, size=400),
                  numpy.random.exponential(50, size=400)]).T

X2 = numpy.array([numpy.random.normal(8, 2, size=600),
                  numpy.random.lognormal(-1.2, 0.6, size=600),
                  numpy.random.exponential(100, size=600)]).T

X = numpy.concatenate([X1, X2])
y = numpy.zeros(1000)
y[400:] = 1

NaiveBayes.from_samples([NormalDistribution, LogNormalDistribution, ExponentialDistribution], X, y)
```

    {
        "class" : "NaiveBayes",
        "models" : [
            {
                "class" : "Distribution",
                "name" : "IndependentComponentsDistribution",
                "parameters" : [
                    [
                        {
                            "class" : "Distribution",
                            "name" : "NormalDistribution",
                            "parameters" : [
                                6.905835442537852,
                                2.09604452629164
                            ],
                            "frozen" : false
                        },
                        {
                            "class" : "Distribution",
                            "name" : "LogNormalDistribution",
                            "parameters" : [
                                -0.880499005246798,
                                0.7642112111076668
                            ],
                            "frozen" : false
                        },
                        {
                            "class" : "Distribution",
                            "name" : "ExponentialDistribution",
                            "parameters" : [
                                0.01979944881580867
                            ],
                            "frozen" : false
                        }
                    ],
                    [
                        1.0,
                        1.0,
                        1.0
                    ]
                ],
                "frozen" : false
            },
            {
                "class" : "Distribution",
                "name" : "IndependentComponentsDistribution",
                "parameters" : [
                    [
                        {
                            "class" : "Distribution",
                            "name" : "NormalDistribution",
                            "parameters" : [
                                8.03534029261696,
                                2.063613757125447
                            ],
                            "frozen" : false
                        },
                        {
                            "class" : "Distribution",
                            "name" : "LogNormalDistribution",
                            "parameters" : [
                                -1.2016720901139963,
                                0.5891919855702662
                            ],
                            "frozen" : false
                        },
                        {
                            "class" : "Distribution",
                            "name" : "ExponentialDistribution",
                            "parameters" : [
                                0.010167268183697146
                            ],
                            "frozen" : false
                        }
                    ],
                    [
                        1.0,
                        1.0,
                        1.0
                    ]
                ],
                "frozen" : false
            }
        ],
        "weights" : [
            0.4,
            0.6
        ]
    }

Modeling each feature independently is allowed for naive Bayes, mixtures, and hidden Markov models. This is very useful when trying to capture different dynamics in different features.

#### Stacking models on top of each other

Next, another feature that emerges when you treat all models as probability distributions is that they can be easily stacked within each other. For example, if a mixture is just a probability distribution, then naturally you should be able to make a mixture Bayes classifier by dropping a `GeneralMixtureModel` into a `BayesClassifier` just as easily as a Gaussian Bayes classifier by dropping a `MultivariateGaussianDistribution` into the `BayesClassifier`.

```python
X = numpy.concatenate([numpy.random.normal((5, 1), 1, size=(200, 2)),
                       numpy.random.normal((6, 4), 1, size=(200, 2)),
                       numpy.random.normal((3, 5), 1, size=(350, 2)),
                       numpy.random.normal((7, 6), 1, size=(250, 2))])

y = numpy.zeros(1000)
y[400:] = 1

model = BayesClassifier.from_samples(MultivariateGaussianDistribution, X, y)
print(model.log_probability(X).sum())

d1 = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 2, X[y == 0])
d2 = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 2, X[y == 1])
model2 = BayesClassifier([d1, d2], [0.4, 0.6])
print(model2.log_probability(X).sum())
```

    -4012.4121095570604
    -3904.723017794966

In this case we build data that intentionally has 4 clusters, of which each class is composed of two of the clusters. A simple normal distribution is unlikely to capture this well, but a mixture model is built specifically to model distributions that are composed of multiple parts. We can see that the correspond log probability is much higher when we incorporate the mixture model in. There is an implementation detail that doesn't allow you yet to pass in a compositional model into the `from_samples` method quite yet, but in this case it is fairly simple to break up the data such that we build the model ourselves.

### Speed

pomegranate uses a Cython backend for the computationally intensive aspects of the calculation, both dramatically speeding up calculations and allowing for multithreading to be utilized.

#### numpy

pomegranate used to be consistently faster than numpy at fitting probability distributions to data, when compared to performing the same operations in numpy. However, due to consistent improvements in numpy, it is now a more complicated picture. For example, let's look at fitting a normal distribution to 1,000 samples.

```python
X = numpy.random.normal(4, 1, size=1000)

%timeit -n 1 -r 1 numpy.mean(X), numpy.std(X)
%timeit -n 1 -r 1 NormalDistribution.from_samples(X)
```

    152 µs ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)
    336 µs ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

pomegranate appears to be a bit faster, but the total amount of time is negligble one way or another. If we increase the size of the data we're using to 10 million samples:

```python
X = numpy.random.normal(4, 1, size=10000000)

%timeit numpy.mean(X), numpy.std(X)
%timeit NormalDistribution.from_samples(X)
```

    52.9 ms ± 903 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
    67.8 ms ± 577 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

It looks like pomegranate is more comparable to numpy.

Let's now look at fitting a multivariate Gaussian distribution. The calculation of a covariance matrix requires a dot product, which numpy accelerates using BLAS. pomegranate uses a Cython wrapper of BLAS and so can take advantage of those speed improvements. Let's look at fitting to one million data points with 3 dimensions.

```python
X = numpy.random.normal(4, 1, size=(1000000, 3))

%timeit numpy.mean(X, axis=0), numpy.cov(X, rowvar=False, ddof=0)
%timeit MultivariateGaussianDistribution.from_samples(X)
```

    51.2 ms ± 838 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
    33.2 ms ± 247 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

It looks like pomegranate is around the same speed here. However, if we fit to a 1000 dimensional sample rather than a 3 dimensional one:

```python
X = numpy.random.normal(4, 1, size=(100000, 1000))

%timeit numpy.mean(X, axis=0), numpy.cov(X, rowvar=False, ddof=0)
%timeit MultivariateGaussianDistribution.from_samples(X)
```

    2.45 s ± 76.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    4.99 s ± 612 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Now it looks like pomegranate is slower than numpy.

The main take-away when compared to numpy is that pomegranate and numpy appear to perform similarly. As the samples become larger numpy becomes faster than pomegranate, but they're both within an order of magnitude of each other one way or another. If you're trying to do basic operations on a large amount of data, you may want to stick with using numpy.

#### scipy

scipy can be used to calculate probabilities of samples given a distribution. This corresponds to the `probability` and `log_probability` functions in pomegranate. Let's see how long it takes to calculate these probabilities in the two packages given a normal distrubution.

```python
from scipy.stats import norm

d = NormalDistribution(0, 1)
x = numpy.random.normal(0, 1, size=(10000000,))

%timeit norm.logpdf(x, 0, 1)
%timeit NormalDistribution(0, 1).log_probability(x)

print("\nlogp difference: {}".format((norm.logpdf(x, 0, 1) - NormalDistribution(0, 1).log_probability(x)).sum()))
```

    581 ms ± 12.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    47.2 ms ± 883 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

    logp difference: -3.990151116406437e-06

Looks like it can be significantly faster. Let's also look at a large multivariate normal distribution with 2500 dimensions.

```python
from scipy.stats import multivariate_normal

dim = 2500
n = 1000

mu = numpy.random.normal(6, 1, size=dim)
cov = numpy.eye(dim)

X = numpy.random.normal(8, 1, size=(n, dim))

d = MultivariateGaussianDistribution(mu, cov)

%timeit multivariate_normal.logpdf(X, mu, cov)
%timeit MultivariateGaussianDistribution(mu, cov).log_probability(X)
%timeit d.log_probability(X)

print("\nlogp difference: {}".format((multivariate_normal.logpdf(X, mu, cov) - d.log_probability(X)).sum()))
```

    3.39 s ± 25.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    796 ms ± 6.07 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    171 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

    logp difference: 8.183596946764737e-07

One of the reasons which pomegranate can be so fast at calculating log probabilities is that it is able to cache parts of the logpdf equation so that it doesn't need to do all of the calculations each time. For example, let's look at the Normal distribution pdf equation:

\begin{equation}
P(X|\mu, \sigma) = \frac{1}{\sqrt{2\pi}\sigma} exp \left( -\frac{(x - \mu)^{2}}{2\sigma^{2}} \right) \\
\end{equation}

We can take the log of this to simplify it.

\begin{equation}
logP(X|\mu, \sigma) = -\log \left(\sqrt{2\pi}\sigma \right) - \frac{(x-\mu)^{2}}{2\sigma^{2}}
\end{equation}

pomegranate speeds up this calculation by caching $-\log(\sqrt{2\pi}\sigma)$ and $2\sigma^{2}$ when the object is created. This means that the equation is simplified to the following:

\begin{equation}
logP(X|\mu, \sigma) = \alpha - \frac{(x - \mu)^{2}}{\beta}
\end{equation}

We don't need to calculate any logs or exponentials here, just a difference, a multiplication, a division, and a subtraction.

#### scikit-learn

scikit-learn and pomegranate overlap when it comes to naive Bayes classifiers and mixture models. In pomegranate, both of these model types can be used with any distribution, allowing mixtures of exponentials or log-normals to be made just as easily as mixtures of Gaussians, whereas in scikit-learn only Gaussians or multinomials are allowed. Let's compare speed of the overlap.

```python
from sklearn.mixture import GaussianMixture

X = numpy.random.normal(8, 1, size=(10000, 100))

%timeit model1 = GaussianMixture(5, max_iter=10).fit(X)
%timeit model2 = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 5, X, max_iterations=10)
```

    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)
    /Users/ksachdeva/Desktop/Dev/myoss/pomegranate/env/lib/python3.7/site-packages/sklearn/mixture/_base.py:267: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      % (init + 1), ConvergenceWarning)

    1.79 s ± 160 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    1.43 s ± 107 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

It looks like the two are approximately the same speed on a single thread. Let's look at naive Bayes now.

```python
from sklearn.naive_bayes import GaussianNB

X = numpy.random.normal(8, 1, size=(100000, 500))
X[:50000] += 1

y = numpy.zeros(100000)
y[:50000] = 1

%timeit GaussianNB().fit(X, y)
%timeit NaiveBayes.from_samples(NormalDistribution, X, y)
```

    1.14 s ± 104 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    1.37 s ± 10.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

It looks like scikit-learn is a bit faster when it comes to training a Gaussian naive Bayes model.

Much like with the comparisons to numpy, if scikit-learn has something easily implemented, it's likely better to try to use their implementation. The goal of pomegranate is not to step on the feet of these other packages, but rather to provide similarly-fast implementations that extend to other probability distributions. For example, you can train a log normal naive Bayes in the following manner:

```python
model = NaiveBayes.from_samples(LogNormalDistribution, X, y)
```

No such functionality yet exists in scikit-learn to train naive Bayes models for arbitrary distributions.


# **[HW8] Practice_6**
1. Install packages
2. PageRank

선형대수 실습은, exercise 혹은 coding 문제풀이에 집중하기 보다는, 수업시간에 배웠던 개념들을 visualize 하면서 이를 폭넓게 이해하는 데에 초점을 맞추고 있습니다. 실습에서 사용한 예시 외에도, 다양한 matrix들을 visualize 하면서 개념을 익혀보시기 바랍니다.

# 1. Install packages

> 필요한 package를 설치하고 import합니다

```python
# visualization을 위한 helper code입니다.
from urllib.request import urlretrieve
URL = 'https://go.gwu.edu/engcomp4plot'
urlretrieve(URL, 'plot_helper.py')

import sys
sys.path.append('../scripts/')

# 다음 세 custom function (1)plot_vector, (2)plot_linear_transformation, (3) plot_linear_transformations
# 을 사용할 것입니다.
from plot_helper import *
```

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy

sy.init_printing()
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

# 2. PageRank

> 이번 실습에선 Google의 PageRank 알고리즘을 구현하면서, eigenvector가 실제로 어떻게 사용되는지 알아보도록 하겠습니다.

> PageRank는 Google의 검색 알고리즘 중 가장 먼저 사용되었던 알고리즘입니다. PageRank는 web**Page**의 **rank**를 매기기 위해, 해당 webpage와 연결되어 있는 다른 webpage들의 개수가 몇 개인지,  (ii) 연결된 webpage들은 각각 얼마나 중요한 webpage인지, 두 가지를 고려하게 됩니다. PageRank에 깔려있는 가정은, **더 중요한 webpage 일수록, 다른 중요한 webpage들로부터 link가 걸려있을 가능성이 높다**는 것입니다 (https://en.wikipedia.org/wiki/PageRank).

> 어떤 webpage의 점수는, 해당 webpage를 link 건 다른 webpage들의 중요도의 weighted average로 계산합니다. 다음 예시를 통해서 알아봅시다.

Example network of 4 webpages, 출처: Ref. [1]

> 위 그림에선, 4개의 wegbpage가 서로 link를 걸고 있음을 알 수 있습니다 (화살표가 링크를 걸었다는 의미입니다). D의 rank(중요도)는 몇일까요? 위 그림에선, D를 link 건 webpage가 A, C입니다. 따라서 A와 C는 D에게 본인의 rank를 나눠주게 됩니다. 이때 A는 나머지 3개 webpage를 전부 link걸었기 때문에, B,C,D 세 webpage에 1/3씩 본인의 rank를 나눠주게 됩니다. B는 B,C 두 webpage를 link걸었기 때문에, B,C 두 webpage에 1/2씩 본인의 rank를 나눠주게 됩니다.

* Page A : B,C,D에게 본인 rank의 1/3씩 나눠주게 됨
* Page B : C,D에게 본인 rank의 1/2씩 나눠주게 됨
* Page C : Link가 없음
* Page D : A,C에게 본인 rank의 1/2씩 나눠주게 됨

> 위 내용을 행렬로 정리하면, 다음과 같이 표현할 수 있습니다.

$$M = \begin{bmatrix} 0   & 0   & 0 & 1/2 \\
                       1/3 & 0   & 0 & 0  \\
                       1/3 & 1/2 & 0 & 1/2 \\
                       1/3 & 1/2 & 0 & 0 \end{bmatrix}
$$

> 그럼 최종적으로 A,B,C,D의 rank는 어떻게 계산될까요? 최종 rank는 모든 webpage들의 rank가 변하지 않을 때까지 나눠주는 행위를 반복한 다음, 수렴된 값을 최종 rank로 생각하게 됩니다.

$$
\begin{bmatrix} 1/4 \\ 1/4 \\ 1/4 \\ 1/4 \end{bmatrix}
\xrightarrow{M} Mx=\begin{bmatrix} 1/8 \\ 1/12 \\ 1/3 \\ 5/24 \end{bmatrix}\xrightarrow{M} M^2x = \begin{bmatrix} 5/48 \\ 1/24 \\ 3/16 \\ 1/12 \end{bmatrix} \xrightarrow{M} \cdots \xrightarrow{M} M^{n}x \xrightarrow{M} M^{n+1}x
$$

> $M^nx = M^{n+1}x$가 된다면, 더이상 값이 바뀌지 않는다는 의미이고, 그 값이 A,B,C,D 각각의 최종 rank가 됩니다.

> 그런데 문제점이 있습니다. Webpage C는 다른 webpage로 link를 걸기 때문에, rank를 나눠주지 않고 webpage C의 rank는 소멸되어버리기 때문에, 최종적으로 모든 webpage의 rank가 0으로 되어버리게 됩니다 (위의 예시에서 모든 rank의 합이 1, 0.75, 0.4167, ... 으로 점점 주는 것을 알 수 있습니다).

```python
M = np.array([[0,0,0,1/2],[1/3,0,0,0],[1/3,1/2,0,1/2],[1/3,1/2,0,0]])
x = np.array([1/4,1/4,1/4,1/4])
```

```python
print(M@x)
print(M@M@M@M@M@M@M@M@M@M@M@M@M@M@x)
```

    [0.125 0.083 0.333 0.208]
    [0. 0. 0. 0.]

> 따라서, 이 문제를 해결하기 위해 $M$을 약간 수정해야 합니다. 가장 단순한 해결책은, M의 zero-column을 $\begin{bmatrix} 1/4 \\ 1/4 \\ 1/4 \\ 1/4 \end{bmatrix}$로 바꿔버리는 것입니다. 이를 modified matrix, $M'$이라고 합시다. 최종적으로는, 이 $M'$과 모든 entry가 1/4인 행렬과의 가중 평균을 사용하게 됩니다. 이 모든 entry가 1/4인 행렬을 더해주게 되면, 모든 entry가 0 이상이 되고, 이는 어떤 임의의 webpage에서 본인을 포함한 다른 webpage로 향하는 모든 link를 만들어줬다고 생각할 수 있습니다.

$$
  M' = \begin{bmatrix} 0   & 0   & 1/4 & 1/2 \\
                      1/3 & 0   & 1/4 & 0  \\
                      1/3 & 1/2 & 1/4 & 1/2 \\
                      1/3 & 1/2 & 1/4 & 0 \end{bmatrix}
\\
  B = n \times n \text{ matrix with 1/n in every entry} \\
  G = (1-p)\, M^{\prime} + p\, B
$$

> $p$는 *damping factor*인데, 둘 간의 가중치를 정하기 위해 우리가 지정하는 hyper-parameter입니다. 결론적으로는, $G$는 $M$과 다르게 $x$과 non-zero vector로 수렴하게 됩니다. 이를 좀 더 엄밀하게 알고 싶으신 분은, 다음 강의를 참조하시기 바랍니다 (https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/lecture-24-markov-matrices-fourier-series/.)

### Exercise

```python
# answer
p=0.15

modified_M = M
modified_M[:,2] = np.ones(4)*1/4
B = np.ones((4,4))*1/4
G = 0.85*modified_M + 0.15*B
```

> 위 내용을 참조하여, PageRank algorithm을 구현해 봅시다. Parameter로 $M'$, damping factor $p$, tolerance $\epsilon$를 받아서, 최종 수렴된 vector를 return하면 됩니다. 위 설명처럼 $G^nx=G^{n+1}x$가 됐다면 rank vector가 수렴되었음을 의미하는데, 둘의 차가 정확히는 같지 않더라도 $\epsilon$ 이하라면 수렴되었다고 간주합니다.

$$ ||G^{n+1}x - G^nx||^2_2 < \epsilon \rightarrow \text{ rank vector is converged to }G^nx $$

### Exercise

```python
# answer
def pagerank(modified_M, p=0.15, eps=1e-3):

    n = modified_M.shape[1]
    v = np.random.rand(n)
    v = v/np.linalg.norm(v)
    G = (1-p)*modified_M + p*(1/n)*np.ones((n,n))
    vold = np.zeros(n)

    while np.linalg.norm(v-vold, 2)>eps:
        vold = v.copy()
        v = G.dot(v)

    return G, v/np.sum(v)
```

```python
pagerank(modified_M)
```

    (array([[0.037, 0.037, 0.25 , 0.462],
            [0.321, 0.037, 0.25 , 0.037],
            [0.321, 0.462, 0.25 , 0.462],
            [0.321, 0.462, 0.25 , 0.037]]), array([0.219, 0.175, 0.356, 0.25 ]))

> 실제로 G를 여러번 곱했을 때, 해당 값으로 수렴함을 확인할 수 있습니다.

```python
G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@G@x
```

    array([0.219, 0.175, 0.356, 0.25 ])

> 즉 세 번쨰, C가 가장 높은 rank를 획득했음을 알 수 있습니다. B는 가장 낮은 rank를 획득하였습니다. 각 rank는 임의의 user가 각 page에 도달할 확률이라고 해석할 수 있습니다.

### Exercise

> 다음 network에 pagerank 알고리즘을 적용하여 최종 rank vector를 구하시오 (damping factor $p$=0.15).

Example network of 8 webpages, 출처: Ref. [2]

```python
# answer

K1 = np.array([[0,1/2,1/2,0,0,0,0,0],[0,0,0,1,0,0,0,0],[0,1/2,0,0,1/2,0,0,0],[0,1/2,0,0,0,1/2,0,0],[0,0,0,0,0,1/3,1/3,1/3],
              [0,0,0,0,0,0,0,1],[1/3,0,0,0,1/3,0,0,1/3],[0,0,0,0,0,1/2,1/2,0]])
K1 = K1.T

K2, v2 = pagerank(K1)
v2
```

    array([0.06 , 0.112, 0.044, 0.114, 0.079, 0.195, 0.147, 0.249])

> 위와 같이 수렴된 vector는, 사실 eigenvector를 이용하면 쉽게 풀 수 있습니다. 첫 번쨰 예시 G에서는 $G^nx = G^{n+1}x$을 만족하는 $G^nx$를 찾으면 됐습니다. 이때 $G^nx = v$라고 하면, 이는 $v = Gv$가 됩니다. 즉, $v$는 $G$의 eigenvalue가 1인 eigenvector가 됩니다.

```python
eigenvalues, eigenvectors = np.linalg.eig(K2)
eigenvalues
```

    array([ 1.   +0.j   , -0.763+0.j   ,  0.397+0.154j,  0.397-0.154j,
           -0.551+0.j   , -0.164+0.204j, -0.164-0.204j,  0.   +0.j   ])

> 이 중 eigenvalue가 1인, 첫번째 eigenvector는 위에서 구한 rank vector와 같음을 알 수 있습니다.

```python
eigenvectors
```

    array([[ 0.152+0.j   , -0.165+0.j   , -0.205+0.065j, -0.205-0.065j,
             0.27 +0.j   ,  0.038-0.225j,  0.038+0.225j, -0.   +0.j   ],
           [ 0.28 +0.j   ,  0.108+0.j   ,  0.276+0.107j,  0.276-0.107j,
             0.253+0.j   , -0.051+0.067j, -0.051-0.067j,  0.   +0.j   ],
           [ 0.111+0.j   ,  0.092+0.j   , -0.167+0.135j, -0.167-0.135j,
            -0.208+0.j   , -0.323+0.18j , -0.323-0.18j ,  0.   +0.j   ],
           [ 0.285+0.j   , -0.12 +0.j   ,  0.591+0.j   ,  0.591-0.j   ,
            -0.39 +0.j   ,  0.273-0.006j,  0.273+0.006j, -0.   +0.j   ],
           [ 0.199+0.j   , -0.217+0.j   , -0.312+0.251j, -0.312-0.251j,
             0.43 +0.j   ,  0.595+0.j   ,  0.595-0.j   ,  0.802+0.j   ],
           [ 0.49 +0.j   ,  0.512+0.j   ,  0.227-0.234j,  0.227+0.234j,
            -0.224+0.j   , -0.145-0.18j , -0.145+0.18j , -0.267+0.j   ],
           [ 0.369+0.j   ,  0.446+0.j   , -0.323-0.021j, -0.323+0.021j,
            -0.525+0.j   ,  0.14 +0.158j,  0.14 -0.158j, -0.   +0.j   ],
           [ 0.624+0.j   , -0.656+0.j   , -0.086-0.304j, -0.086+0.304j,
             0.395+0.j   , -0.527+0.006j, -0.527-0.006j, -0.535+0.j   ]])

```python
rank_vector = K2 @ eigenvectors[:,0]
rank_vector = rank_vector / rank_vector.sum()
rank_vector
```

    array([0.06 +0.j, 0.112+0.j, 0.044+0.j, 0.114+0.j, 0.079+0.j, 0.195+0.j,
           0.147+0.j, 0.249+0.j])

> j는 허수 부분을 나타내는데, 계수가 0이므로 무시하면 됩니다. 즉, $G$의 eigenvalue가 1인 eigenvector는 $G$를 여러번 곱해서 얻어진 rank_vector와 같음을 알 수 있습니다.

# References

1. Dan Margalit and Joseph Rabinoff (2018), Interactive Linear Algebra, an open book under the GNU Free Documentation License, Version 1.2.
2. David Austin (2019), Understanding Linear Algebra, an open book under CC-By license.
3. https://github.com/jclosure/EngComp4_landlinear

```python

```


```python
import numpy as np
import scipy.linalg as sla
```

## Decomposição LU

Nas aulas anteriores, nós trabalhamos o método de escalonamento gaussiano como forma de resolver sistemas de equações lineares não triviais.

Como lembramos, o método tinha dois passos: o primeiro, de eliminação ou escalonamento, tinha como intenção triangularizar a matriz de coeficientes $A$ a partir de uma série de operações que eram aplicados tanto à matriz quanto ao vetor de termos independentes $\vec{b}$, resultando, dessa forma, numa matriz equivalente, triangular superior, e num novo vetor de termos independentes; e o segundo era a solução direta do sistema triangular superior por substituição regressiva.

Embora esse método, principalmente e versão com pivotamento, seja seguro no sentido de garantir a solução correta, ele apresenta duas limitações:

1. a etapa de escalonamento da eliminação gaussiana, principalmente com pivotamento, tem elevado custo computacional, principalmente para grandes sistemas de equações, como ilustra o quadro abaixo, que ilustra o número de _flops_ (_floating-point operations per second_  - operações de ponto flutuante por segundo) para a eliminação gaussiana. Para comparar, basta considerar que a eliminação gaussiana requer $\mathcal{O}(n^3/3)$ operações matemáticas para determinar $\vec{x}$. No entanto, resolver um sistema triangular superior ou inferior, requer somente $\mathcal{O}(2n^2)$. O quadro abaixo dimensiona, a título de ilustração, a redução de custo computacional de se trabalhar com sistemas triangulares se comparados com a eliminação gaussiana.

|$n$ | $n^3/3$ | $2n^2$| redução de custo computacional|
| :-:|:-----:|:-------:|:-:|
|10| $3,3 \cdot 10^2$ | $2 \cdot 10^2$ | 40% |
|100|  $3,3 \cdot 10^5$ | $2 \cdot 10^4$ | 94% |
|1000|  $3,3 \cdot 10^8$ | $2 \cdot 10^6$ | 99,4% |

2. esse custo computacional torna essa abordagem particularmente ineficiente para encontrar a solução de diversos sistemas que tenham mesma matriz de coeficientes, mas diferentes vetores de termos independentes, uma vez que, em todos os sistemas, todo o processo terá de ser inteiramente aplicado.

Uma alternativa à eliminação gaussiana, que trabalharemos essa semana, é a utilização da fatoração ou decomposição LU, que basicamente se propõe a decompor a matriz $\mathbf{A}$ em duas matrizes $\mathbf{L}$, triangular inferior, e $U$, triangular superior, de forma que

$$
\mathbf{A} = \mathbf{LU}
$$

De forma bem simplificada, a solução de sistemas de equações lineares por decomposição LU tem dois passos bem definidos:

1. __decomposição LU:__ $\mathbf{A}$ é fatorada em duas matrizes triangulares superior e inferior;
2. __substituição:__ $\mathbf{L}$ e $\mathbf{U}$ são usadas para determinar o vetor solução $x$, considerando, basicamente, que

\begin{eqnarray*}
\mathbf{A}\vec{x} = \vec{b}~~~ \Rightarrow~~~ \mathbf{LU}\vec{x} = \vec{b}
\end{eqnarray*}

Fazendo o sistema intermediário $\mathbf{U}\vec{x} = \vec{y}$, obtemos outro sistema intermediário, $\mathbf{L}\vec{y} = \vec{b}$. E podemos resolver o sistema original em duas etapas:

* primeiro, se resolve o sistema intermediário $\mathbf{L}\vec{y} = \vec{b}$, para encontrar o vetor $\vec{y}$ usando substituição progressiva;

* em seguida, o vetor $\vec{y}$ é usado como vetor de termos independentes do outro sistema intermediário, $\mathbf{U}\vec{x} = \vec{y}$, e, usando substituição retroativa, se encontra o vetor solução desejado.

A figura abaixo ilustra o método.

Essa abordagem diminui sensivelmente o custo computacional se comparado a eliminação gaussiana, uma vez que todas as operações, após a decomposição, serão feitas em um sistema triangular inferior ou superior (e como vimos acima isso reduz drásticamente o custo computacional), além de permitir que, uma vez que a decomposição se realiza somente na matriz de coeficientes, diferentes sistemas com mesma matriz de coeficientes mas diferentes vetores de termos independentes possam ser resolvidos, com uma única execução da decomposição.

## Estratégias de Decomposição LU

Não existe uma única forma de obter as matrizes $\mathbf{L}$ e $\mathbf{U}$. Na verdade, existem inúmeras possíveis abordagens, que se diferenciam, basicamente, por certas restrições usadas para definir algumas características das matrizes $\mathbf{L}$ ou $\mathbf{U}$, e que acabam diferenciando as abordagens. No geral, podemos pensar nas seguintes abordagens:

1. a decomposição derivada da eliminação gaussiana, que pode ser trabalhada de duas formas:
    * a decomposição de Doolittle, que tem como restrição o fato de diagonal principal da matriz triangular inferior ter todos os elementos iguais a 1;
    * a decomposição de Crout, que tem como restrição a diagonal principal da matriz triangular superior com todos os elementos unitários e
2. a decomposição de Choleski, em que a matriz triangular inferior é a transposta da matriz triangular superior.

O quadro abaixo sumariza esses métodos.

| Método | Características |
|:-----:|:-:|
| Doolittle | eliminação gaussiana com diagonal principal da matriz triangular inferior igual a 1 ($L_{ii} = 1$)  |
| Crout | eliminação gaussiana com  diagonal principal da matriz triangular superior igual a 1 ($U_{ii} = 1$)
| Choleski | matrizes triangulares superior e inferior transpostas ($\mathbf{L} = \mathbf{U^T}$)

### Decomposição LU a Partir da Eliminação Gaussiana

A própria eliminação gaussiana pode ser usada de maneira relativamente simples para implementarmos a decomposição LU. Para entendermos o funcionamento, trabalharemos com a decomposição de Doolittle, que tem como característica a diagonal principal da matriz $\mathbf{L}$ com valores iguais a 1.

Nesse processo, temos dois passos:

* o escalonamento da matriz $\mathbf{A}$ (não a matriz estendida) nos retornará a matriz $\mathbf{U}$;

* os multiplicadores serão os elementos da matriz $\mathbf{L}$ (no caso, os elementos abaixo da diagonal principal).

Ainda no processo de escalonamento, como nós vimos para a eliminação gaussiana, pode ser necessário a realização de permutas de linhas para garantir que o maior valor de cada coluna esteja na diagonal principal, por meio de escalonamento parcial.

Nesse processo, utilizaremos uma abordagem de uso de uma matriz de permutação $\mathbf{P}$, que é uma matriz formada apenas de zeros e uns, sendo o valor de apenas um elemento por linha e por coluna que igual a um, marcando, justamente que linha foi permutada e cujo objetivo é gerar uma permutação dos elementos de um vetor ou entre linhas ou colunas de uma matriz.

No caso de usarmos a matriz de permutação, teremos uma pequena variação da expressão que representa o sistema, que é

$$
\mathbf{P A}\vec{x} = \mathbf{P} \vec{b},
$$

sendo no caso, a decomposição,

$$
\mathbf{P A} = \mathbf{LU}
$$

Vamos considerar o exemplo a seguir pra gente entender o funcionamento dessa abordagem. Para nosso entendimento a linha $i$ será referenciada como $E_i$.

$$
\mathbf{A} = \underbrace{\begin{bmatrix}\color{red}{1}&1&0\\\color{red}{2}&1&-1\\\color{red}{3}&-1&-1\end{bmatrix}}_{P = \begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}}
\xrightarrow{\text{troca linhas 1 e 3}}
\underbrace{\begin{bmatrix}\color{red}{3}&-1&-1\\\color{red}{2}&1&-1\\\color{red}{1}&1&0\end{bmatrix}}_{P = \begin{bmatrix}0&0&1\\0&1&0\\1&0&0\end{bmatrix}}\\[4cm]
\xrightarrow{\begin{align*}E^{(1)}_2 &= E_2-(2/3)E_1\\E^{(1)}_3 &= E_3-(1/3)E_1\end{align*}}
\underbrace{\begin{bmatrix}\color{red}{3}&-1&-1\\\color{red}{0}&\frac{5}{3}&\frac{1}{3}\\\color{red}{0}&\frac{4}{3}&\frac{1}{3}\end{bmatrix}}_{P = \begin{bmatrix}0&0&1\\0&1&0\\1&0&0\end{bmatrix}}\\[4cm]
$$

$$
\mathbf{A^{(1)}} =
\underbrace{\begin{bmatrix}3& \color{red}{-1}&-1\\0& \color{red}{\frac{5}{3}}&\frac{1}{3}\\0& \color{red}{\frac{4}{3}}&\frac{1}{3}\end{bmatrix}}_{P = \begin{bmatrix}0&0&1\\0&1&0\\1&0&0\end{bmatrix}}
\xrightarrow{E^{(2)}_3 = E^{(1)}_3-(4/5)E_2^{(1)}}
\underbrace{\begin{bmatrix}3& \color{red}{-1}&-1\\0& \color{red}{\frac{5}{3}}&\frac{1}{3}\\0& \color{red}{0}&\frac{3}{5}\end{bmatrix}}_{P = \begin{bmatrix}0&0&1\\0&1&0\\1&0&0\end{bmatrix}}
$$

Assim, obtemos a matriz de permutação

$$
P = \begin{bmatrix}0&0&1\\0&1&0\\1&0&0\end{bmatrix}
$$

a matriz $\mathbf{U}$ como a matriz escalonada

$$
\mathbf{U} = \begin{bmatrix}3& -1&-1\\0&\frac{5}{3}&\frac{1}{3}\\0&0&\frac{3}{5}\end{bmatrix}
$$

e a matriz $\mathbf{L}$, usando os multiplicadores de cada operação de cada linha

$$
\mathbf{L} = \begin{bmatrix}1& 0&0\\\frac{2}{3}&1&0\\\frac{1}{3}&\frac{4}{5}&1\end{bmatrix}
$$

#### Implementação do algoritmo de Doolittle

Considerando as matrizes generalizadas,

\begin{equation}
\mathbf{A} = \begin{bmatrix}a_{11}&a_{12}&\ldots&a_{1n}\\a_{21}&a_{22}& \cdots&a_{2n}\\\vdots& \vdots&\ddots& \vdots\\a_{n1}&a_{n2}&\ldots&a_{nn}\end{bmatrix},
\mathbf{L} = \begin{bmatrix}1&0&\ldots&0\\l_{21}&1& \cdots&0\\\vdots& \vdots&\ddots& \vdots\\l_{n1}&l_{n2}&\ldots&1\end{bmatrix}, \\[2cm]
\mathbf{U} = \begin{bmatrix}u_{11}&u_{12}&\ldots&u_{1n}\\0&u_{22}& &u_{2n}\\\vdots& \vdots&\ddots& \vdots\\0&0&\ldots&u_{nn}\end{bmatrix},
\mathbf{P} = \begin{bmatrix}p_{11}&p_{12}&\ldots&p_{1n}\\p_{21}&\ddots& &p_{2n}\\\vdots& &\ddots& \vdots\\p_{n1}&\ldots&\ldots&p_{nn}\end{bmatrix} \\[2cm]
\end{equation}

o algoritmo de Doolittle é

| Algoritmo de Doolittle | |
| --:   | :-- |
|___Entrada.___ | Matriz $\mathbf{A}$|
|___Passo  1.___  | Inicialize $\mathbf{L = P = I}$, de dimensão $n \times n$, e $\mathbf{U = A}$ |
| ___Passo  2.___  | Para $j = 1, \ldots, n-1$ |
|     | $\phantom{--}$ Se $|u_{pj}| = \max{(|u_{(j:n,j)}|)}> u_{jj}, \text{ com } j \leq p < n$ |
|     | $\phantom{----}$ Permutar linha $U_j$ com linha $U_{p}$ |
|   | $\phantom{----}$ Permutar linha $P_j$ com linha $P_{p}$ |
| ___Passo 3.___  | $\phantom{----}$ Para $i = j+1, \ldots, n$ |
|     | $\phantom{------}$ $l_{ij} = \dfrac{u_{ij}}{u_{jj}}$ |
|    | $\phantom{------}$ $U_{i}=U_{i} - l_{ij} U_{j}$ (em que $U_i, U_j$ representam as linhas $i$ e $j$ da matriz $\mathbf{U}$, respectivamente)

### Solução de Equações Após Decomposição LU

Para solucionar o sistema completo, uma vez que temos as matrizes $\mathbf{P}$, $\mathbf{L}$ e $\mathbf{U}$, é, como comentamos anteriormente, resolver os dois sistemas intermediários:

$$\mathbf{L}\vec{y} = \mathbf{P}\vec{b}~~~~~~\mathbf{U}\vec{x} = \vec{y}$$

Notemos que $\mathbf{P}$ multiplica o vetor de termos independentes para que possamos a versão modificada pelo pivotamento.

Como os dois sistemas são triangulares, e nós já implementamos, em aulas anteriores, funções para resolver esses sistemas, basta, então, construir um implementação integrando todas as "partes" do processo. Considerando que usaremos essas mesmas funções já implementadas, `sist_lin_tri_sup()` e `sist_lin_tri_inf()`, o algoritmo a seguir descreve isso

| Solução de sistemas com decomposição LU | |
| --:   | :-- |
|___Entrada.___  | Matrizes $\mathbf{P}$, $\mathbf{L}$ e $\mathbf{U}$ e vetor de termos indendentes $\vec{b}$|
| ___Passo  1.___  | Resolução do sistema triangular inferior $\mathbf{L}\vec{y} = \mathbf{P}\vec{b}$, usando a função `sist_lin_tri_inf(L,Pb)`, que retornará o vetor $\vec{y}$
| ___Passo  2.___  | Resolução do sistema triangular superior $\mathbf{U}\vec{x} = \vec{y}$, usando a função `sist_lin_tri_sup(U,y)`, que retornará o vetor $\vec{x}$

### Decomposição LU usando `Scipy`

A biblioteca `scipy` possui três funções, no submódulo `linalg`, sobre decomposição LU:

* [lu](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lu.html): calcula a decomposição LU com pivotamento de uma matriz $\mathbf{A}$, decompondo no formato $\mathbf{A = PLU}$. Esse método retorna, se o parâmetro `permute_l == False`, a matriz de permutação $\mathbf{P}$, a matriz $\mathbf{L}$, com elementos unitários na diagonal principal, e a matriz $\mathbf{U}$, e se `permute_l == True`, retorna a matriz $\mathbf{PL}$ e a matriz $\mathbf{U}$

* [lu_factor](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lu_factor.html): também calcula a decomposição LU com pivotamento de uma matriz $\mathbf{A}$, decompondo no formato $\mathbf{A = PLU}$, retornando, diferentemente da função anterior, uma matriz $\mathbf{[LU]}$, cujos elementos de $\mathbf{U}$ na parte triangular superior e de $\mathbf{L}$ na triangular inferior (os elementos unitários da diagonal de $\mathbf{L}$ não são armazenados) e um vetor `piv` contendo os índices do pivotamento, indicando que a linha `i` foi permutada com a linha `piv[i]`.

* [lu_solve](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lu_solve.html): soluciona um sistema $\mathbf{A}\vec{x} = \vec{b}$, a usando fatoração LU. Como entrada, recebe uma tupla formada pelas matrizes $\mathbf{[LU]}$ e `piv` (saída da função `lu_factor`) e vetor de termos independentes $\vec{b}$.

Em resumo, podemos pensar em dois casos sobre o uso dessas funções:

1. se a necessidade for realizar a decomposição LU, para encontrar as partes da decomposição, a função para se usar é `lu`;

2. se a necessidade for resolver o sistema, então é preciso usar a função `lu_factor`, e usar sua saída como entrada para a função `lu_solve`.

Para podermos visualizar melhor o uso dessas funções, vamos resolver o sistema

$$
  \begin{split}
    x_1 + x_2 +x_3  &= -2\\
    2x_1 + x_2 -x_3  &= 1\\
    2x_1 - x_2 +x_3  &= 3\\
  \end{split}
$$

cuja matriz de coeficientes e vetor de termos independentes pode ser escrito como

```python
A = np.array([[1,1,1],[2,1,-1],[2,-1,1]])
A
```

    array([[ 1,  1,  1],
           [ 2,  1, -1],
           [ 2, -1,  1]])

```python
b = np.array([-2,1,3])
b
```

    array([-2,  1,  3])

Se quisermos saber quem são as matrizes $\mathbf{P}$, $\mathbf{L}$ e $\mathbf{U}$, fazemos

```python
P,L,U = sla.lu(A)
```

```python
P
```

    array([[0., 0., 1.],
           [1., 0., 0.],
           [0., 1., 0.]])

```python
L
```

    array([[ 1.  ,  0.  ,  0.  ],
           [ 1.  ,  1.  ,  0.  ],
           [ 0.5 , -0.25,  1.  ]])

```python
U
```

    array([[ 2.,  1., -1.],
           [ 0., -2.,  2.],
           [ 0.,  0.,  2.]])

```python
P@L@U
```

    array([[ 1.,  1.,  1.],
           [ 2.,  1., -1.],
           [ 2., -1.,  1.]])

Já se desejamos encontrar o vetor solução do sistema, precisamos fazer a sequência

```python
LU, piv = sla.lu_factor(A)
```

```python
LU
```

    array([[ 2.  ,  1.  , -1.  ],
           [ 1.  , -2.  ,  2.  ],
           [ 0.5 , -0.25,  2.  ]])

```python
piv
```

    array([1, 2, 2], dtype=int32)

```python
x = sla.lu_solve((LU,piv),b)
x
```

    array([ 1., -2., -1.])

E, testando, de fato temos

```python
A@x
```

    array([-2.,  1.,  3.])


# Trajectory Optimization

## Notebook Setup

### Julia Setup

```julia
using LaTeXStrings
using Plots
using Polynomials
import Base: ctranspose

# The Poly class has an odd quirk in that is defines the conjugate transpose
# operator A' as differentiation of the polynomial. While this makes some
# sense for an isolated polynomial objects, for arrays it has unexpected
# behavior: [p q]' == [polyder(p), polyder(q)']. In this notebook, I will be
# manipulating vectors and matrices of polynomial objects; I do not want these
# objects to be differentiated when transposed. Therefore, I'm redefining the
# ctranspose operator to simply return the original polynomial object.
ctranspose{T}(p::Poly{T}) = p

# The line selects which plotting backend will be used to generate the plots
# shown in this notebook.
pyplot();

# The function below allows quick printing of arrays of rational numbers
function print_rational(title::String, data::Array{Rational{Int64},2})
    @printf("%s:\n", title)
    for i in 1:size(data,1)
        for j in 1:size(data,2)
            @printf("%4d/%-4d", data[i,j].num, data[i,j].den)
        end
        println()
    end
    println()
end
```

    WARNING: Method definition ctranspose

    print_rational (generic function with 1 method)

### Latex Macros

- Vector Typeface: $ \newcommand{\vec}[1]{\boldsymbol{#1}} $
- Matrix Typeface: $ \newcommand{\mat}[1]{\boldsymbol{#1}} $

## Overview

The objective of this project is to develop a trajectory optimization tool that can be used to assess the maximum range of a Conventional Prompt Global Strike (CPGS) weapon system. CPGS weapon systems, have been a major research focus of the US Department of Defense for over a decade, with several development programs currently underway ([Ref. 1](https://fas.org/sgp/crs/nuke/R41464.pdf)). This project is specifically interested in CPGS systems that employ the “boost-glide” technique whereby a maneuverable glide vehicle is accelerated to hypersonic velocities using a multi-stage solid rocket booster.

## Trajectory Computation via Implicit CG Method

One of the key elements of the trajectory optimization problem is discretizing the equations of motion that define how the vehicle responds to the control input. This discretization can be accomplished in many ways, with modern state-of-the-art codes predominantly utilizing high-order
collocation methods. In this project we discretize the governing equations using an implicit Continuous Galerkin (CG) method in conjunction with quintic Hermite basis functions.

### Governing Differential Equations

The trajectory of a hypersonic boost-glide systems can be modeled using a pair of coupled second-order differential equations, similar to those shown below.

\begin{align}
    \label{eq:governing_x} \ddot{x} &= f^x(t, x, y, \dot{x}, \dot{y}, \alpha) \\
    \label{eq:governing_y} \ddot{y} &= f^y(t, x, y, \dot{x}, \dot{y}, \alpha)
\end{align}

| Variables   | Description
|:-----------:|-------------
| $x$,$y$     | Vehicle position relative to inertial space
| $f^x$,$f^y$ | Force acting on the vehicle (per unit mass)
| $\alpha$    | Vehicle Angle of attack (control variable)

Note that for the purposes of this study, we consider the angle of attack to be the control variable that allows us to shape the trajectory. In reality, the angle of attack is a consequence of how the vehicle thrust is vectored and/or how the aerodynamic control surfaces are deflected. However, modeling the how the vehicle attitude changes in response to control inputs is beyond the scope of what is necessary for this project. Instead, we simply assume that a control system exists and that it can trim the vehicle to follow the desired angle of attack profile. We can always place limits on the angle of attack profile if necessary during the optimization process.

### CG System of Equations

The CG discretization proceeds by converting the differential equations above into an equivalent weak form by multiplying each equation by an arbitrary “weighting” or “test” function, $w(t)$, and integrating. (Note: from here on out, we will only show the $x$-equation; the $y$-equation is handled in an identical fashion.)

\begin{align}
    \int_{t_0}^{t_f} w\ddot{x}~dt & = \int_{t_0}^{t_f} wf^x~dt \\
    w\dot{x}\big|_{t_0}^{t_f} - \int_{t_0}^{t_f} \dot{w}\dot{x}~dt & = \int_{t_0}^{t_f} wf^x~dt
\end{align}

Introducing the notation $(u,v)=\int_{t_0}^{t_f}uv~dt$ yields a concise statement of the governing equation in weak form:

\begin{equation}
    (\dot{w},\dot{x}) + (w,f^x) + w\dot{x}\big|_{t_0} - w\dot{x}\big|_{t_f} = 0
\end{equation}

At this point, it is necessary to select a method of parameterizing the unknown functions $x(t)$ and $w(t)$. For this application, we will utilize high-order (quintic) Hermite basis functions. The reason for selecting this basis set is as follows:

1. Since we can safely assume the vehicles acceleration history will be at least $C^0$ continuous, the vehicle position history will be at minimum $C^2$ continuous. Using quintic Hermite shape functions ensures that that the computed trajectory belongs to the space of $C^2$ continuous functions. (Note: high-jerk events like stage separation will be addressed by stitching together two separate CG discretizations using optimizer equality constraints).

2. Since Hermite shape functions utilize the nodal values of position, velocity, etc. as the shape function coefficients, implementing boundary conditions and trajectory constraints (altitude limits, terminal velocity requirements, etc.) is trivial.

3. Quintic Hermite interpolation is 6-th order accurate, which enables highly accurate trajectory solutions with small degrees of freedom counts.

Let $t_i,~i = 1,...,N$ be a series of ordered points spanning the interval $[t_0, t_f]$; the points need not uniformly spaced. The vector-valued function $\vec{\phi}_i(t)$ represents the Hermite basis functions associated with the $i$-th node. The vector $\vec{x}_i$ represents the coefficients used to scale the basis
functions for that node. In the case of quintic Hermite basis functions, these coefficients are simply the value of the solution function and its first two derivatives at the $i$-th node:

\begin{equation}
    \vec{x}_i = \left[ x(t_i), \dot{x}(t_i), \ddot{x}(t_i) \right]^T
\end{equation}

The function $x(t)$ is then constructed via a linear super-position of the coefficient-weighted basis functions for all nodes, as shown below. The Galerkin test function, $w(t)$, is parameterized similarly.

\begin{align}
    x(t) &= \sum_{i=1}^{N} \vec{\phi}_i^T(t)\cdot \vec{x}_i \\
    w(t) &= \sum_{i=1}^{N} \vec{\phi}_i^T(t)\cdot \vec{w}_i
\end{align}

We then substitute these parameterizations into the weak form of the governing equation. For brevity, I will drop the explicit summation symbols; summation is implied by products with repeated indices.

\begin{equation}
    \left( \vec{\dot\phi}_i^T\vec{w}_i, \vec{\dot\phi}_j^T\vec{w}_j \right) +
    \left( \vec{\phi}_i^T\vec{w}_i, f^x \right) +
    \left( \vec{\phi}_i^T(t_0)\vec{w}_i \right) \cdot \left( \vec{\dot\phi}_j^T(t_0)\vec{x}_j \right) -
    \left( \vec{\phi}_i^T(t_f)\vec{w}_i \right) \cdot \left( \vec{\dot\phi}_j^T(t_f)\vec{x}_j \right) =
    0
\end{equation}

Note that since $\vec{\phi}_i^T\vec{w}_i = \vec{w}_i^T\vec{\phi}_i$, we can rewrite the above equation and factor out parameterization constants for the basis functions as shown below:

\begin{equation}
    \vec{w}_i^T \left[
        \left[
           \left( \vec{\dot\phi}_i, \vec{\dot\phi}^T_j \right) +
           \left. \vec{\phi}_i\vec{\dot\phi}_j^T \right|_{t_0} -
           \left. \vec{\phi}_i\vec{\dot\phi}_j^T \right|_{t_f}
        \right] \vec{x}_j +
        \left( \vec{\phi}_i, f^x \right)
    \right] = 0
\end{equation}

Since the weighting function is arbitrary, the values of the $\vec{w}_i$’s are free parameters. Thus, the only way for the above equation to be uniformly zero for any possible value of the $\vec{w}_i$’s is for the bracketed expression itself to be zero for every value of $i = 1,...,N$. The result is a system of simultaneous, non-linear equations that must be solved for the unknown values of the $\vec{x}_j$’s:

\begin{equation}
    \label{eq:cg_system_of_equations}
    \left[
        \left( \vec{\dot\phi}_i, \vec{\dot\phi}^T_j \right) +
        \left. \vec{\phi}_i\vec{\dot\phi}_j^T \right|_{t_0} -
        \left. \vec{\phi}_i\vec{\dot\phi}_j^T \right|_{t_f}
    \right] \vec{x}_j +
    \left( \vec{\phi}_i, f^x \right) =
    0, ~~~~
    \forall i = 1,...,N
\end{equation}

### Quintic Hermite Shape Functions

Hermite shape functions enable construction of a continuous interpolant on the interval $\tau \in [0,1]$ given the value of a function and its derivatives at the endpoints of the interval. The canonical form of the quintic (5th order) Hermite shape function are:

\begin{align}
    H_0^5(\tau) &= 1 - 10\tau^3 + 15\tau^4 - 6\tau^5 \\
    H_1^5(\tau) &= \tau - 6\tau^3 + 8\tau^4 - 3\tau^5 \\
    H_2^5(\tau) &= \frac{1}{2}\left( \tau^2 - 3\tau^3 + 3\tau^4 - \tau^5 \right) \\
    H_3^5(\tau) &= 10\tau^3 - 15\tau^4 + 6\tau^5 \\
    H_4^5(\tau) &= -4\tau^3 + 7\tau^4 -3\tau^5 \\
    H_5^5(\tau) &= \frac{1}{2}\left( \tau^3 - 2\tau^4 + \tau^5 \right)
\end{align}

```julia
# Define and plot quintic Hermite shape functions

H = [
    Poly([ 1,   0,   0, -10,  15,  -6 ]//1, :τ)
    Poly([ 0,   1,   0,  -6,   8,  -3 ]//1, :τ)
    Poly([ 0,   0,   1,  -3,   3,  -1 ]//2, :τ)
    Poly([ 0,   0,   0,  10, -15,   6 ]//1, :τ)
    Poly([ 0,   0,   0,  -4,   7,  -3 ]//1, :τ)
    Poly([ 0,   0,   0,   1,  -2,   1 ]//2, :τ)
]

τ  = linspace(0,1)
H1 = polyder.(H)
H2 = polyder.(H1)

plot(
    plot(τ, polyval.(H',τ),  ylabel="Function Value"),
    plot(τ, polyval.(H1',τ), ylabel="Function 1st Derivative"),
    plot(τ, polyval.(H2',τ), ylabel="Function 2nd Derivative"),
    label=["\$H^5_$i\$" for i in (0:5)'],
    xlabel=L"\tau",
    layout=(3,1),
    size=(600,1200),
)

```

    (Polynomials.Poly{#T<:Any}) in module Polynomials at C:\Users\Jeff\.julia\v0.5\Polynomials\src\Polynomials.jl:440 overwritten in module Main at In[2]:13.

The interpolant is constructed from the shape functions as follows:

\begin{equation}
\begin{split}
    x(\tau) &= H_0^5(\tau)\cdot x\bigg|_{\tau=0}
             + H_1^5(\tau)\cdot \frac{dx}{d\tau}\bigg|_{\tau=0}
             + H_2^5(\tau)\cdot \frac{d^2x}{d\tau^2}\bigg|_{\tau=0} \\
            &+ H_3^5(\tau)\cdot x\bigg|_{\tau=1}
             + H_4^5(\tau)\cdot \frac{dx}{d\tau}\bigg|_{\tau=1}
             + H_5^5(\tau)\cdot \frac{d^2x}{d\tau^2}\bigg|_{\tau=1}
\end{split}
\end{equation}

Three of the shape functions are associated with the degrees of freedom on the left end of the interval while the other three are assoicated with the right end of the interval. Therefore, let’s define

\begin{align}
    \vec{h}_L(\tau) &= [H_0^5(\tau), H_1^5(\tau), H_2^5(\tau)]^T \\
    \vec{h}_R(\tau) &= [H_3^5(\tau), H_4^5(\tau), H_5^5(\tau)]^T \\
\end{align}

This allows us to write the interpolant as

\begin{equation}
    x(\tau) =
        \vec{h}_L^T\cdot\left[
            \begin{array}{c}
                x \\
                \frac{dx}{d\tau} \\
                \frac{d^2x}{d\tau^2} \\
            \end{array}
        \right]_{\tau=0}
        +
        \vec{h}_R^T\cdot\left[
            \begin{array}{c}
                x \\
                \frac{dx}{d\tau} \\
                \frac{d^2x}{d\tau^2} \\
            \end{array}
        \right]_{\tau=1}
\end{equation}

Notice that the derivatives required for this interpolant are with respect to $\tau$, the interval-local coordinate. We previously defined the $\vec{x}_i$’s as specifying the true time derivatives, so a scaling must be applied. (Note: given that the node spacing need not be uniform, the relationships between $t$ and $\tau$ may differ on either side of a given node implying $\lim_{t \rightarrow t_i+}\frac{dx}{d\tau} \neq \lim_{t \rightarrow t_i-}\frac{dx}{d\tau}$). If we define $\Delta{}t_i = t_{i+1} - t_i$ as the length of the $i$-th interval, we can then construct $\mat{S}_i$, a matrix that scales global derivatives by the Jacobian of the coordinate transformation:

\begin{equation}
    \mat{S}_i = \left[
        \begin{array}{ccc}
            1  &      0      &      0        \\
            0  & \Delta{}t_i &      0        \\
            0  &      0      & \Delta{}t_i^2 \\
        \end{array}
    \right]
\end{equation}

Therefore, the interpolate for the $i$-th interval can be written as:

\begin{equation}
    \label{eq:interval_interpolation_equation}
    x(\tau) = \vec{h}_L^T\mat{S}_i\vec{x}_i
            + \vec{h}_R^T\mat{S}_i\vec{x}_{i+1}
\end{equation}

Finally, if we consider two adjacent intervals, we can write the functional form of the global shape functions associated with the $i$-th node as follows:

\begin{equation}
    \label{eq:global_basis_functions}
    \vec{\phi}_i = \left\{
        \begin{array}{cl}
            \mat{S}_{i-1}\vec{h}_R,  & t_{i-1} \lt t \leq t_{ i } \\
            \mat{S}_{i}\vec{h}_L,    & t_{ i }~~~ \lt t \leq t_{i+1} \\
            0,                       & \text{otherwise}
        \end{array}
    \right.
\end{equation}

### Evaluation of Basis Function Integrals

With the functional form of the global shape functions defined, we will now return to the fundamental non-linear equations shown in Equation $\ref{eq:cg_system_of_equations}$, and evaluate the various shape function integrals.

First, consider the term $\left( \vec{\dot\phi}_i,\vec{\dot\phi}_j^T \right)\vec{x}_j$. Due to the compact support of the nodal basis functions, the weighting matrix $\left( \vec{\dot\phi}_i,\vec{\dot\phi}_j^T \right)$ will be zero unless $j \in \{i-1, i, i+1\}$. Assuming $j=i-1$, we can write an explicit expression for the weighting matrix as follows:

\begin{split}
    \left(\vec{\dot\phi}_i,\vec{\dot\phi}_{i-1}^T \right)
        & = \int_{t_0}^{t_f} \vec{\dot\phi}_i\vec{\dot\phi}_{i-1}^T~dt \\
        & = \int_{t_{i-1}}^{t_i}
            \left(\mat{S}_{i-1}\dot{h}_R\right)
            \left(\mat{S}_{i-1}\dot{h}_L\right)^T
            ~dt \\
        & = \mat{S}_{i-1} \cdot \int_0^1
            \left( \frac{d\tau}{dt}\frac{d\vec{h}_R}{d\tau} \right)
            \left( \frac{d\tau}{dt}\frac{d\vec{h}_L}{d\tau} \right)^T
            \frac{dt}{d\tau}~d\tau \cdot \mat{S}_{i-1}\\
        & = \frac{1}{\Delta t_{i-1}} \mat{S}_{i-1} \cdot \int_0^1
            \vec{h}_R^\prime \vec{h}_L^{\prime T}~d\tau\ \cdot \mat{S}_{i-1} \\
        & = \frac{1}{\Delta t_{i-1}} \mat{S}_{i-1} \mat{K}^{RL} \mat{S}_{i-1}
\end{split}

A similar derivation can be performed for $j = i,i+1$. The results is a block tri-diagonal system of equations. The tri-diagonal structure means this matrix can be inverted efficiently, which suggests that we should consider using a gradient-based method to converge the discrete system of equations.

\begin{align}
    \label{eq:stiffness_matrix}
    \left(\vec{\dot\phi}_i,\vec{\dot\phi}_{j}^T \right)\vec{x}_j = \left\{
    \begin{array}{ll}
        \frac{1}{\Delta t_1} \mat{S}_1 \mat{K}^{LL} \mat{S}_1 \vec{x}_1 +
        \frac{1}{\Delta t_1} \mat{S}_1 \mat{K}^{LR} \mat{S}_1 \vec{x}_2,
        & i = 1 \\
        \frac{1}{\Delta t_{i-1}} \mat{S}_{i-1} \mat{K}^{RL} \mat{S}_{i-1} \vec{x}_{i-1} +
        \left(
            \frac{1}{\Delta t_{i-1}} \mat{S}_{i-1} \mat{K}^{RR} \mat{S}_{i-1} +
            \frac{1}{\Delta t_{i}} \mat{S}_{i} \mat{K}^{LL} \mat{S}_{i}
        \right)\vec{x}_i +
        \frac{1}{\Delta t_{i}} \mat{S}_{i} \mat{K}^{LR} \mat{S}_{i} \vec{x}_{i+1},
        & i = 2,...,N-1\\
        \frac{1}{\Delta t_{N-1}} \mat{S}_{N-1} \mat{K}^{RL} \mat{S}_{N-1} \vec{x}_{N-1} +
        \frac{1}{\Delta t_{N-1}} \mat{S}_{N-1} \mat{K}^{RR} \mat{S}_{N-1} \vec{x}_{N}
        & i = N \\
    \end{array}
    \right.
\end{align}

The code snippet below evaluates the various stiffness matrices $\mat{K}^{RL}, \mat{K}^{RR},$ etc. to obtain their exact numerical values. Note that the matrices are not symmetric as would be the case for a standard linear finite element method. This is because of the anti-symmetry of the basis function associated with the function derivative at a given node.

```julia
# Compute shape function integrals for K_RR, K_RL, etc.
hlp = H1[1:3]
hrp = H1[4:6]
print_rational("K_RL", polyval.(polyint.(hrp * hlp'), 1))
print_rational("K_RR", polyval.(polyint.(hrp * hrp'), 1))
print_rational("K_LL", polyval.(polyint.(hlp * hlp'), 1))
print_rational("K_LR", polyval.(polyint.(hlp * hrp'), 1))
```

    K_RL:
     -10/7     -3/14    -1/84
       3/14    -1/70    -1/210
      -1/84     1/210    1/1260

    K_RR:
      10/7     -3/14     1/84
      -3/14     8/35    -1/60
       1/84    -1/60     1/630

    K_LL:
      10/7      3/14     1/84
       3/14     8/35     1/60
       1/84     1/60     1/630

    K_LR:
     -10/7      3/14    -1/84
      -3/14    -1/70     1/210
      -1/84    -1/210    1/1260

Next, let's consider the boundary terms that result from the integration by parts. For these terms, we note that the only shape functions that are non-zero at the boundaries are those that are associated with the boundary degrees of freedom, $\vec\phi_1$ and $\vec\phi_N$. Thus, the boundary terms are non-zero only when $i = j = 1,N$.

\begin{array}{lll}
    \vec\phi_1\vec{\dot\phi}_j^T\bigg|_{t_0}\vec{x}_j
         &= \left(\mat{S}_1\vec{h}_L\right)
            \left(\mat{S}_1\vec{\dot{h}}_L\right)^T\bigg|_{t_0}
            \vec{x}_1
         &= \frac{1}{\Delta t_1} \left(
                \mat{S}_1
                \left[ \vec{h}_L \vec{h}_L^{\prime T} \right]_{\tau=0}
                \mat{S}_1
            \right) \vec{x}_1\\
    \vec\phi_N\vec{\dot\phi}_j^T\bigg|_{t_f}\vec{x}_j
         &= \left(\mat{S}_{N-1}\vec{h}_R\right)
            \left(\mat{S}_{N-1}\vec{\dot{h}}_R\right)^T\bigg|_{t_f}
            \vec{x}_N
         &= \frac{1}{\Delta t_{N-1}} \left(
                \mat{S}_{N-1}
                \left[ \vec{h}_R \vec{h}_R^{\prime T} \right]_{\tau=1}
                \mat{S}_{N-1}
            \right)
            \vec{x}_N
\end{array}

The code below evaluates the shape function products $\vec{h}_L\vec{h}_L^{\prime T}$ and $\vec{h}_R\vec{h}_R^{\prime T}$ at their respective endpoints. There is only one non-zero element in the array: it picks out the first derivative of the unknown function at the boundary which will then be added or subtracted from the function value at the boundary.

```julia
# Compute boundary terms
hl = H[1:3]
hr = H[4:6]
print_rational("hl*hlp' @ tau=0:", polyval.(hl*hlp', 0))
print_rational("hr*hrp' @ tau=1:", polyval.(hr*hrp', 1))
```

    hl*hlp' @ tau=0::
       0/1      1/1      0/1
       0/1      0/1      0/1
       0/1      0/1      0/1

    hr*hrp' @ tau=1::
       0/1      1/1      0/1
       0/1      0/1      0/1
       0/1      0/1      0/1

The last term we need to evaluate is the integral of the basis functions multiplied by the forcing function.

\begin{split}
    \left( \vec{\phi}_i, f^x \right)
        &= \int_{t_0}^{t_f} \vec{\phi}_i \cdot f_x ~dt \\
        &= \int_{t_{i-1}}^{t_{ i }} \left(\mat{S}_{i-1}\vec{h}_R\right) \cdot f^x ~dt
         + \int_{t_{ i }}^{t_{i+1}} \left(\mat{S}_{ i }\vec{h}_L\right) \cdot f^x ~dt \\
        &= \Delta t_{i-1} \mat{S}_{i-1} \int_0^1 \vec{h}_R \cdot f^x ~d\tau
         + \Delta t_{ i } \mat{S}_{ i } \int_0^1 \vec{h}_L \cdot f^x ~d\tau
\end{split}

For most practical problems we will not have have a closed form expression for the forcing function, so the integrals in the final line of the above equation must be evaluated approximately using quadrature:

\begin{equation}
    \int_0^1 \vec{h} \cdot f^x ~d\tau
    \approx
    \sum_k w_k \cdot \vec{h}(\tau_k) \cdot f^x(\tau_k)
\end{equation}

Defining

\begin{align}
    \mat{H}^R &= [ \vec{h}_R(\tau_1), \vec{h}_R(\tau_2), ... ] \\
    \mat{H}^L &= [ \vec{h}_R(\tau_1), \vec{h}_R(\tau_2), ... ] \\
    \vec{f}^x &= [ f^x(\tau_1), f^x(\tau_2), ... ]^T \\
    \mat{W}   &= \left[ \begin{array}{ccc}
        w_1, &  0,  &        \\
         0,  & w_2, &        \\
             &      & \ddots
    \end{array} \right]
\end{align}

We can write express the forcing function integral as follows:

\begin{equation}
    \left( \vec{\phi}_i, f^x \right)
       = \Delta t_{i-1} \mat{S}_{i-1} \mat{H}^R \mat{W} \vec{f}^x_{i-1}
       + \Delta t_{ i } \mat{S}_{ i } \mat{H}^L \mat{W} \vec{f}^x_{ i }
\end{equation}

Interestingly, all matrices that multiply $\vec{f}_{i-1}^x$ and $\vec{f}_{i}^x$ are constants, so they may be precomputed an stored as a single weighting matrix. Furthermore, if the node spacing in uniform, these matrices are the same for every node. This feature will enable the design of highly memory- and compute-efficient solution algorithms.

### Discrete System Summary

General Form:
\begin{equation}
    \label{eq:discrete_system_summary}
    \mat{\bar{K}}\vec{x} + \vec{\bar{f}}(\vec{x}) = 0
\end{equation}

Matrix Definitions:
\begin{array}{ll}
    i = 1:            & 0 = \left( \mat{\bar{K}}^{LL}_1 + \mat{\bar{B}}^L \right)\vec{x}_1
                          + \mat{\bar{K}}^{LR}_1\vec{x}_2
                          + \mat{\bar{H}}^L_1\mat{W}\vec{f}_1 \\
    i = 2,\dots,N-1:  & 0 = \mat{\bar{K}}^{RL}_{i-1}\vec{x}_{i-1}
                          + \left( \mat{\bar{K}}^{RR}_{i-1} + \mat{\bar{K}}^{LL}_{i} \right) \vec{x}_{i}
                          + \mat{\bar{K}}^{LR}_i\vec{x}_{i+1}
                          + \mat{\bar{H}}^{R}_{i-1}\mat{W}\vec{f}_{i-1} + \mat{\bar{H}}^{L}_{i}\mat{W}\vec{f}_{i} \\
    i = N:            & 0 = \mat{\bar{K}}^{RL}_{N-1}\vec{x}_{N-1}
                          + \left( \mat{\bar{K}}^{RR}_{N-1} - \mat{\bar{B}}^R \right)\vec{x}_N
                          + \mat{\bar{H}}^R_{N-1}\mat{W}\vec{f}_{N-1}
\end{array}

|      Coefficient       |   Definition$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$                                                                  |
|:----------------------:|------------------------------------------------------------------------------------------------------------------|
| $\mat{\bar{K}}^{RL}_i$ |  $\frac{1}{\Delta{t}_i} \mat{S}_i \left( \int_0^1 \vec{h}_R^\prime \vec{h}_L^{\prime T}~d\tau \right)\mat{S}_i$  |
| $\mat{\bar{K}}^{RR}_i$ |  $\frac{1}{\Delta{t}_i} \mat{S}_i \left( \int_0^1 \vec{h}_R^\prime \vec{h}_R^{\prime T}~d\tau \right)\mat{S}_i$  |
| $\mat{\bar{K}}^{LL}_i$ |  $\frac{1}{\Delta{t}_i} \mat{S}_i \left( \int_0^1 \vec{h}_L^\prime \vec{h}_L^{\prime T}~d\tau \right)\mat{S}_i$  |
| $\mat{\bar{K}}^{LR}_i$ |  $\frac{1}{\Delta{t}_i} \mat{S}_i \left( \int_0^1 \vec{h}_L^\prime \vec{h}_R^{\prime T}~d\tau \right)\mat{S}_i$  |
| $\mat{\bar{H}}^{L}_i$  |  $\Delta t_i \mat{S}_i  \left[ \vec{h}_L(\tau_1), \vec{h}_L(\tau_2) ... \right]$                                 |
| $\mat{\bar{H}}^{R}_i$  |  $\Delta t_i \mat{S}_i  \left[ \vec{h}_R(\tau_1), \vec{h}_R(\tau_2) ... \right]$                                 |
| $\mat{\bar{B}}^{L}$    |  $\frac{1}{\Delta t_{ 1 }} \mat{S}_{ 1 } \left[ \vec{h}_L \vec{h}_L^{\prime T} \right]_{\tau=0} \mat{S}_{ 1 }$   |
| $\mat{\bar{B}}^{R}$    |  $\frac{1}{\Delta t_{N-1}} \mat{S}_{N-1} \left[ \vec{h}_R \vec{h}_R^{\prime T} \right]_{\tau=1} \mat{S}_{N-1}$   |

### Boundary Conditions

In the derivation above, we have neglected discussion of boundary conditions. In general, a trajectory integration problems requires specification of the full initial state (position and velocity in both the $x$ and $y$ directions) or specification of an equivalent number of initial and final states. We will also require that the integration time be specified a-priori; this requirement can be relaxed in the future once we get the fixed-time integrator running.

Imposing Dirichlet or Neumann boundary conditions using the discrete equations above is relatively straight forward. Since we have chosen to use Hermite basis functions, the discrete system contains explicit equations for positions and velocity at each end of the interval. To impose a specified boundary conditions, e.g. $\dot{x}_N = v_f$,

1. We zero out the associated row in $\mat{\bar{K}}$ and set the diagonal entry to unity.

2. We set the associated row in $\vec{\bar{f}}$ equal to the specified value.

Note that this procedure can be trivially generalized to allow assigning specified values to internal degrees of freedom. For example, we may choose to ensure that the trajectory apogee or perigee occurs at the mid-point of the trajectory, so we set the vertical velocity to zero at the mid-point node. This capability maybe worth implementing further down the road.

Another interesting generalization that would be fairly easy to implement would be boundary conditions that are linear combinations of the state variables:

\begin{equation}
    \sum_k a_k x_{i,k} = c
\end{equation}

Perhaps most importantly, this could be used to ensure specific flight path constraints are met at impact. In cases like this, instead of setting the diagonal entries to unity in $\mat{\bar{K}}$, we would inject the coefficients for each state and set then assign the constant term to the forcing vector. However, in this case it's not clear which row in $\mat{\bar{K}}$ should be replaced: do we replace the equation for the first state involved in the boundary condition, or should we always prefer replacing the velocity variables? These types of boundary conditions are also amenable to Lagrange multiplier techniques, so perhaps that method would be preferred (that will destroy the diagonal structure of the LHS matrix, but an efficient solution may still be possible via partitioning).

General non-linear boundary conditions will require some more thought (although I can't think of a practical example of one, so maybe we don't need it). I suspect we would just add a hook that allows the user to call a custom function that computes an appropriate RHS for the constraint and then use some kind of automatic differentiation or finite differences to linearize that function if needed.

### Linearization

Efficiently solving the non-linear system shown in Equation $\ref{eq:discrete_system_summary}$ will require linearizing the residual with respect to the unknown state variables. Let's consider the discrete residual equations for the $i$-th interior node:

\begin{align}
    \label{eq:x_residual}
    \vec{R}_i^x &= \mat{\bar{K}}^{RL}_{i-1}\vec{x}_{i-1}
       + \left( \mat{\bar{K}}^{RR}_{i-1} + \mat{\bar{K}}^{LL}_i \right) \vec{x}_{i}
       + \mat{\bar{K}}^{LR}_i\vec{x}_{i+1}
       + \mat{\bar{H}}^{R}_{i-1}\mat{W}\vec{f}_{i-1}^x + \mat{\bar{H}}^{L}_i\mat{W}\vec{f}_i^x \\
    \vec{R}_i^y &= \mat{\bar{K}}^{RL}_{i-1}\vec{y}_{i-1}
       + \left( \mat{\bar{K}}^{RR}_{i-1} + \mat{\bar{K}}^{LL}_i \right) \vec{y}_{i}
       + \mat{\bar{K}}^{LR}_i\vec{y}_{i+1}
       + \mat{\bar{H}}^{R}_{i-1}\mat{W}\vec{f}_{i-1}^y + \mat{\bar{H}}^{L}_i\mat{W}\vec{f}_i^y \\
\end{align}

As shown in Equations $\ref{eq:governing_x}$ and $\ref{eq:governing_y}$, the force terms are in general a function of $t,x,y,\dot{x},\dot{y},\alpha$. Therefore, the differential of a forcing function evaluated at the $k$-th quadrature point of the $i$-th interval is given by:

\begin{equation}
    \label{eq:nodal_differential}
    \delta f_k = \left.\frac{\partial f}{\partial t}\right|_{\tau_k} \delta t
               + \left.\frac{\partial f}{\partial x}\right|_{\tau_k} \delta x
               + \left.\frac{\partial f}{\partial y}\right|_{\tau_k} \delta y
               + \left.\frac{\partial f}{\partial \dot{x}}\right|_{\tau_k} \delta\dot{x}
               + \left.\frac{\partial f}{\partial \dot{y}}\right|_{\tau_k} \delta\dot{y}
               + \left.\frac{\partial f}{\partial \alpha}\right|_{\tau_k} \delta\alpha
\end{equation}

For the time being, we will consider $t$ and $\alpha$ to be specified, and therefore do require linearization. Furthermore, since we have selected Hermite basis functions, $x,\dot{x}$ on the $i$-th interval are functions of only $x_i,x_{i+1}$. A similar relationship exists for the $y$ variable. Therefore, the total differential can be rewritten as:

\begin{equation}
    \begin{split}
    \delta f_k
        &=
        \left[
            \frac{\partial f}{\partial x} \frac{\partial x}{\partial \vec{x}_i} +
            \frac{\partial f}{\partial \dot{x}} \frac{\partial \dot{x}}{\partial \vec{x}_i}
        \right]_{\tau_k} \delta \vec{x}_i
        +
        \left[
            \frac{\partial f}{\partial x} \frac{\partial x}{\partial \vec{x}_{i+1}} +
            \frac{\partial f}{\partial \dot{x}} \frac{\partial \dot{x}}{\partial \vec{x}_{i+1}}
        \right]_{\tau_k} \delta \vec{x}_{i+1} \\
        &+
        \left[
            \frac{\partial f}{\partial y} \frac{\partial y}{\partial \vec{y}_i} +
            \frac{\partial f}{\partial \dot{y}} \frac{\partial \dot{y}}{\partial \vec{y}_i}
        \right]_{\tau_k} \delta \vec{y}_i
        +
        \left[
            \frac{\partial f}{\partial y} \frac{\partial y}{\partial \vec{y}_{i+1}} +
            \frac{\partial f}{\partial \dot{y}} \frac{\partial \dot{y}}{\partial \vec{y}_{i+1}}
        \right]_{\tau_k} \delta \vec{y}_{i+1}
    \end{split}
\end{equation}

From this expression, we see that in order to compute the partial derivatives of a given $\vec{f}_i$ with respect to the nodal degrees of freedom we need two things:

1. The sensitivity of the forcing function with respect to its input parameters ($x$,$\dot{x}$,etc.), evaluated at each quadrature point in the interval.

2. The sensitivity of the forcing function inputs ($x$,$\dot{x}$, etc.) with respect to the nodal degrees of freedom  ($\vec{x}_i, \vec{x}_{i+1}$, etc.) for each point quadrature point in the interval.

While the former will in general be a function of the solution and must be re-computed as the solution evolves, the later can be precomputed and stored. Differentiating Equation $\ref{eq:interval_interpolation_equation}$ with respect to $\vec{x}_i, \vec{x}_{i+1}$ yields:

\begin{align}
    \frac{\partial x}{\partial \vec{x}_i} &= \vec{h}_L^T\mat{S}_i \\
    \frac{\partial x}{\partial \vec{x}_{i+1}} &= \vec{h}_R^T\mat{S}_i
\end{align}

These expressions can then be differentiated with respect to time:

\begin{align}
    \frac{\partial \dot{x}}{\partial \vec{x}_i} &= \frac{1}{\Delta t_i} \vec{h}_L^{\prime T}\mat{S}_i \\
    \frac{\partial \dot{x}}{\partial \vec{x}_{i+1}} &= \frac{1}{\Delta t_i} \vec{h}_R^{\prime T}\mat{S}_{i} \\
\end{align}

Substituting these expressions into Equation $\ref{eq:nodal_differential}$ and stacking the results for each quadrature points, we can derive the following expression for the linearization of the force vector on the $i$-th interval with respect to the nodal degrees of freedom:

\begin{align}
    \frac{\partial}{\partial x_{ i }}\vec{f}_i =
        \frac{1}{\Delta t_i^2}\left(
            \mat{\bar{H}}^{L}_i \mat{F}^{xx} +
            \mat{\bar{H}}^{\prime L}_i \mat{F}^{x\dot{x}}
        \right)^T \\
    \frac{\partial}{\partial x_{i+1}}\vec{f}_i =
        \frac{1}{\Delta t_i^2}\left(
            \mat{\bar{H}}^{R}_i \mat{F}^{xx} +
            \mat{\bar{H}}^{\prime R}_i \mat{F}^{x\dot{x}}
        \right)^T \\
    \mat{F}^{pq} = \left[ \begin{array}{ccc}
        \left.\frac{\partial f^p}{\partial q}\right|_{\tau_1} & &  \\
        & \left.\frac{\partial f^p}{\partial q}\right|_{\tau_2} &  \\
        & & \ddots
    \end{array}\right]
\end{align}

With the identities above, we may now write the linearization of Equation $\ref{eq:x_residual}$ as follows:

... this gets ugly fast. Doable, but ugly.