# Workshop Tutorial 1: Practical Apricot Usage

Welcome to a tutorial on how to reduce redundancy in massive data sets using submodular optimization! In this tutorial, we will explore submodular optimization at a high level and see how it can be used to select representative subsets of data; these subsets can then be used on their own, such as to create a highlight reel for an album, or to create smaller training sets for machine learning models that achieve similar performance in a fraction of the time. Although submodular optimization is as general-purpose as convex optimization, this tutorial will focus on using basic optimization algorithms on two main functions: a feature-based function, and facility location functions. Finally, this tutorial will focus on practical usage of apricot. Please see the other tutorials for more of the theory behind how these functions work.

Let's get started!

```python
%pylab inline
import seaborn
seaborn.set_style('whitegrid')

from tqdm import tqdm
```

## Feature-based Selection

A simple class of submodular functions are the feature-based ones. At a high level, feature-based functions are those that maximize diversity in the observed feature values themselves. This property means that they work well in settings where each feature represents some quality of the data and higher values mean that the example has more of that value: for instance, when vectorizing text data, each feature might represent a word and the value would be the number of times that the word appears in the document.

More formally, feature-based functions take the form

\begin{equation}
f(X) = \sum\limits_{u \in U} w_{u} \phi_{u} \left( \sum\limits_{x \in X} m_{u}(x) \right)
\end{equation}
where $x$ is a single example, $X$ is the set of all examples, $u$ is a single feature, $U$ is the set of all features, $w$ is a weight foe each feature, and $\phi$ is a saturating concave function such as log or sqrt.

### 20 newsgroups

Let's start off with some text data. Below, some code is provided to download a shuffled version of the 20 newsgroups data set, which contains articles and labels for 20 topics. However, as we can see, the downloaded text is not in a convenient featurized form that can be used by machine learning models.

```python
from sklearn.datasets import fetch_20newsgroups

train_data = fetch_20newsgroups(subset='train', random_state=0, shuffle=True)
train_data.data[2]
```

Processing this to get rid of the weird characters like "\n" and converting it to a vectorized form is not really the point of this tutorial, so let's use sklearn's built-in vectorizer to get a clean feature matrix to operate on. Please fill in the below cells.

```python
from sklearn.feature_extraction.text import CountVectorizer

vectorizer = ...

X_train = ...
X_train.shape
```

Let's take a look at how dense that data is. We can do this by creating a heatmap where each red dot represents that the feature hasa non-zero value. If you implemented the above code correctly you should get a density of 0.08395.

```python
i = 1000

X_random_block = X_train[:i].toarray()
random_density = (X_random_block != 0).mean()

plt.figure(figsize=(12, 6))
plt.scatter(*numpy.where(X_random_block.T[:i] != 0), c='r', s=0.05)
plt.xlim(0, i)
plt.ylim(0, i)

plt.title("Words in Text Blobs: Density={:4.4}".format(random_density), fontsize=14)
plt.xlabel("Word Index", fontsize=12)
plt.ylabel("Text Blob Index", fontsize=12)
plt.show()
```

The above heat map is made up of the first 1000 entries in the data set after shuffling. It doesn't seem particularly dense; fewer than 10% of the values in the matrix are positive. This may not be particularly problematic when restricting to 1000 features, but since more features generally means more accuracy in this setting, is there a way to ensure that our subset sees a higher percentage of the features?

Well, choosing examples that exhibit values in a diverse set of features is exactly what submodular optimization and feature based functions are good at. We can define a feature-based function easily using apricot, choose an equal sized subset of examples using submodular optimization, and re-visualize the chosen examples.

Fill in the next code block, using a feature-based selector to choose 1000 samples with everything else set to the default parameters. You can also set `verbose=True` to see a progress bar during selection. Note that, while apricot can operate on sparse matrices, you might need to use the `toarray()` method to convert a sparse array to a dense array for the subsequent visualization step.

```python
from apricot import FeatureBasedSelection

selector = ...

X_submodular_block = ...
```

Now that you've selected the examples, we can visualize the block in the same way that we visualized the randomly selected examples. If you implemented the selector correctly, you should get a density of 0.2103. Visually, the heatmap should also look significantly more red. This is because we are intentionally choosing examples that have many non-zero values, i.e., rows that would have red in a lot of columns.

```python
submodular_density = (X_submodular_block != 0).mean()

plt.figure(figsize=(12, 6))
plt.scatter(*numpy.where(X_submodular_block.T[:i] != 0), c='r', s=0.05)
plt.xlim(0, i)
plt.ylim(0, i)

plt.title("Words in Text Blobs: Density={:4.4}".format(submodular_density), fontsize=14)
plt.xlabel("Word Index", fontsize=12)
plt.ylabel("Text Blob Index", fontsize=12)
plt.show()
```

Next, we can take a look at the number of words that are observed at least once as we select more and more examples, either randomly, or using submodular optimization. If your implementation of selecting a subset of examples using apricot is correct you should see that a larger number of words are observed earlier in the selection process when submodular optimization is used. You do not need to do anything here.

```python
random_seen_words = (numpy.cumsum(X_random_block, axis=0) > 0).sum(axis=1)
submodular_seen_words = (numpy.cumsum(X_submodular_block, axis=0) > 0).sum(axis=1)

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("# Total Uniques Words Seen", fontsize=14)
plt.plot(random_seen_words, color='0.5', linewidth=2, label="Random")
plt.plot(submodular_seen_words, color='#FF6600', linewidth=2, label="Submodular Optimization")
plt.xlabel("# Examples Chosen", fontsize=12)
plt.ylabel("# Words Seen At Least Once", fontsize=12)
plt.legend(fontsize=12)

plt.subplot(122)
plt.title("# New Words Seen per Example", fontsize=14)
plt.plot(numpy.diff(random_seen_words), color='0.5', linewidth=2, label="Random")
plt.plot(numpy.diff(submodular_seen_words), color='#FF6600', linewidth=2, label="Submodular Optimization")
plt.xlabel("# Examples Chosen", fontsize=12)
plt.ylabel("# New Words in Example", fontsize=12)
plt.legend(fontsize=12)

plt.tight_layout()
plt.show()
```

Next, we're going to move on to the primary goal of apricot: choosing subsets for training machine learning models. Unfortunately, this is not always straightforward. As an example, we are going to consider classifying a subset of classes from the 20 newsgroups data set. Here are the classes.

```python
fetch_20newsgroups(subset="train").target_names
```

As an initial example, we will focus on two classes that are somewhat related conceptually but will likely have distinct vocabularies. We will use the TF-IDF vectorizer instead of the count vectorizer because TF-IDF is a straightforward way to downweight words that appear in many articles and to upweight words that are somewhat rare and more likely to be topic-specific. Please fill in the below code involving processing the training and test data properly.

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import SGDClassifier

categories = ['sci.med', 'sci.space']

train_data = fetch_20newsgroups(subset='train', categories=categories, random_state=0, shuffle=True)
test_data = fetch_20newsgroups(subset='test', categories=categories, random_state=0)

vectorizer = ...

X_train = ...
X_test = ...

y_train = ...
y_test = ...

X_train.shape, X_test.shape, y_train.shape, y_test.shape
```

Next, use a feature-based function to select 1000 examples from the training data. 1000 examples is almost all of the data, but because the selection process is greedy we can use it to rank most of the data and then choose increasingly large subsets to train the model.

```python
selector = ...
```

Now, let's train a SGG classifier on subsets of increasing size and compare to ten draws of similarly sized random subsets. Please fill in the below code, keeping in mind that the `selector.ranking` attribute contains a ranking of indices from the original data set. For example, if the first element was `10`, that would mean that `X_train[10]` was the first element chosen by the greedy optimization process.

```python
model = SGDClassifier(random_state=0)
counts = numpy.arange(10, 1001, 10)

random_idxs = numpy.array([numpy.random.choice(X_train.shape[0], replace=False, size=1000) for i in range(10)])
random_accuracies, submodular_accuracies = [], []

for count in tqdm(counts):
    idxs = selector.ranking[:count]

    ...

    y_hat = model.predict(X_test)
    acc = (y_hat == y_test).mean()
    submodular_accuracies.append(acc)

    accs = []
    for i in range(10):
        r_idxs = random_idxs[i, :count]

        ...

        y_hat = model.predict(X_test)
        acc = (y_hat == y_test).mean()
        accs.append(acc)

    random_accuracies.append(accs)

plt.title("20 Newsgroups Classification", fontsize=14)
plt.plot(counts, numpy.mean(random_accuracies, axis=1), color='0.5', linewidth=2, label="Random")
plt.plot(counts, submodular_accuracies, color='#FF6600', linewidth=2, label="Submodular Optimization")

plt.xlabel("# Chosen Examples", fontsize=12)
plt.ylabel("Classification Accuracy")
plt.legend(loc=4, fontsize=12)

seaborn.despine()
plt.tight_layout()
plt.show()
```

Looks like we can get almost the same performance with just 100 examples (~93% with submodular optimization and ~85% with random selection) as we could with the full set of 1000 examples. It is worth noting that there is a lot of variance when the number of examples chosen is very small, but that performance picks up pretty quickly. If you're not seeing these trends, it's possible that you implemented something incorrectly.

If you'd like to explore apricot's abilities more broadly, try out the above cells using different sets of categories from the 20 newsgroups corpus and different types of classifiers.

### A Warning: Gaussian Blobs

Unfortunately, not all data is amenable to feature-based functions. Specifically, data sets where the features don't follow the semantics that are assumed, i.e., non-negative and a higher value conveys some notion of having "more" of some feature. If you have features like coordinates or embeddings from a pre-trained model or projections from a method like tSNE or UMAP, they may not work as you'd like.

Here, we will look at using data drawn from random Gaussian blobs.

```python
from sklearn.datasets import make_blobs

numpy.random.seed(0)
centers = numpy.random.normal(100, 5, (5, 2))
X, y = make_blobs(n_samples=2500, n_features=2, centers=centers, random_state=0)
```

This time, we will fill in how to use a selector for you.

```python
selector = FeatureBasedSelection(n_samples=100)
selector.fit(X)
```

Now, let's apply the selector and get our representative subset!

```python
plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Gaussian Blob Data", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.axis('off')

plt.subplot(122)
plt.title("Feature Based Selected Examples", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*X[selector.ranking].T, color='#FF6600', s=10, label="Selected Examples")
plt.axis('off')
plt.legend(loc=(1.01, 0.5), fontsize=12)

plt.tight_layout()
plt.show()
```

Oops. That doesn't look like a representative subset.

Does this mean that feature-based functions cannot work in settings where the data doesn't have the same semantics as our assumptions? No! We just need to engineer features that do follow those semantics.

### Feature Engineering for Feature-based Functions: Gaussian Mixture Models

Potentially, one of the most straightforward ways to transform this Gaussian data would be to, first, apply a Gaussian mixture model to it, and second, use the posterior probabilities from that model as the features. Basically, instead of applying submodular optimization to the original feature values themselves, we apply them to the predicted class probabilities from the mixture model. These probabilities have all the properties that we would like: (1) because they are between zero and one they must be non-negative, (2) a higher value means an enrichment for that feature, i.e., a higher probability means an enrichment for class membership from that class.

Using the `GaussianMixture` object below, transform the above data from in `X` from the original feature values into the posterior probabilities. Because the data was generated from five clusters, your mixture should have five components. If done correctly, the resulting shape should be `(2500, 5)`.

```python
from sklearn.mixture import GaussianMixture

model = ...
X_posteriors = ...
X_posteriors.shape
```

Now, apply a feature-based selector as you've done in the past.

```python
selector = FeatureBasedSelection(n_samples=100)
selector.fit(...)
```

Now, let's plot the mixture centroids as well as the selected examples.

```python
plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Gaussian Blob Data", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*model.means_.T, color='b', s=10)
plt.axis('off')

plt.subplot(122)
plt.title("Feature Based Selected Examples", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*model.means_.T, color='b', s=10, label="GMM Centroids")
plt.scatter(*X[selector.ranking].T, color='#FF6600', s=10, label="Selected Examples")
plt.axis('off')
plt.legend(loc=(1.01, 0.5), fontsize=12)

plt.tight_layout()
plt.show()
```

Does this look like what you might expect?

If not, think more closely about the feature-based function and the data here. The sum of each example should be equal to one, so there are no examples that have a higher coverage of the feature space than other examples. However, the feature-based function includes a saturation function that diminishes the benefit of high values in one feature versus spreading them out across several features. Combined, these facts mean that the method will always try to choose examples that are split between multiple classes. Put another way, `numpy.sqrt([0.5, 0.5]).sum() = 1.414` is larger than `numpy.sqrt([1.0, 0]).sum() = 1.0`.

Regardless of the explanation, this isn't exactly what we were expecting. What we'd like to do is have a way that our feature-based function can select examples near the middle of each cluster without needing cluster labels. The problem with using the posteriors, which are normalized to sum to 1, is that examples that are purest for a particular cluster are not the ones closest to the centroid but rather the ones that are on the other side of the centroid as all the other centroids.

What does that mean? Well, let's use a simple trick to try to pick out the purest examples from each cluster. First, we need to transform these values such that values near one become bigger, so that purity is valued higher, but values near zero remain the same. We can use an `arctanh` function for that, but you should try out any other function you'd like to see the effects. Below is an example `arctanh` function.

```python
x = numpy.arange(0, 1, 0.001)
plt.plot(x, numpy.arctanh(x))
plt.title("Example Non-Linear Function", fontsize=14)
plt.xlabel("x", fontsize=12)
plt.ylabel("arctanh(x)", fontsize=12)
plt.show()
```

```python
X_arctanh = numpy.arctanh(X_posteriors - 1e-12) + 1e-12

selector = FeatureBasedSelection(n_samples=100)
selector.fit(X_arctanh)

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Gaussian Blob Data", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*model.means_.T, color='b', s=10)
plt.axis('off')

plt.subplot(122)
plt.title("Feature Based Selected Examples", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*model.means_.T, color='b', s=10, label="GMM Centroids")
plt.scatter(*X[selector.ranking].T, color='#FF6600', s=10, label="Selected Examples")
plt.axis('off')
plt.legend(loc=(1.01, 0.5), fontsize=12)

plt.tight_layout()
plt.show()
```

We can see some interesting trends here. Unlike the previous plot where all the chosen examples were near boundaries, most of the chosen examples are on very edge of the convex hull. A notable exception, however, is the top cluster. This is likely because the top cluster is so far away from the others that any example in it is considered "pure."

Finally, let's get to the expected behavior. We would like to design a transformation such that our selection chooses elements that are neat representations of each cluster individually. We saw previously that using the normalized posterior probabilities can be an issue because the normalization process encourages the chosen examples to be far away from the other centroids, rather than close to any particular centroid. If we get rid of that normalization process and instead use the raw probabilities that each example belongs to a particular mixture component, we can get around this.

In the cell below, use the `multivariate_normal` method from scipy to calculate an array of probabilities for each example under each mixture component. Hint: you will need to do this separately for each component as part of a loop. Make sure that your final output is of shape `(n_samples, 5)`.

```python
from scipy.stats import multivariate_normal

X_probs = ...

selector = FeatureBasedSelection(n_samples=100)
selector.fit(X_probs)

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Gaussian Blob Data", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*model.means_.T, color='b', s=10)
plt.axis('off')

plt.subplot(122)
plt.title("Feature Based Selected Examples", fontsize=14)
plt.scatter(*X.T, color='0.5', s=10)
plt.scatter(*model.means_.T, color='b', s=10, label="GMM Centroids")
plt.scatter(*X[selector.ranking].T, color='#FF6600', s=10, label="Selected Examples")
plt.axis('off')
plt.legend(loc=(1.01, 0.5), fontsize=12)

plt.tight_layout()
plt.show()
```

If you've done this correctly, you'll notice that all of the chosen examples are near the centroids of the clusters.

At this point, you might be wondering "why do I need submodular optimization to do this?" because you can just take the examples closest to centroids. The answer is two-fold: first, submodular optimization can be applied to any type of transformation where it may not be obvious how to do it by hand. Second, submodular optimization automatically balances the number of examples chosen per centroid based on their distance. This isn't a particularly complicated task here where all of the clusters are distinct, but consider this example:

```python
numpy.random.seed(0)
centers = numpy.random.normal(100, 4, (5, 2))
X2, _ = make_blobs(n_samples=1000, n_features=2, centers=centers, cluster_std=3, random_state=0)

model2 = GaussianMixture(5, random_state=0).fit(X2)
X_probs = numpy.array([multivariate_normal.pdf(X2, model2.means_[i], model2.covariances_[i]) for i in range(5)]).T

selector = FeatureBasedSelection(n_samples=100)
selector.fit(X_probs)

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Gaussian Blob Data", fontsize=14)
plt.scatter(*X2.T, color='0.5', s=10)
plt.scatter(*model2.means_.T, color='b', s=10)
plt.axis('off')

plt.subplot(122)
plt.title("Feature Based Selected Examples", fontsize=14)
plt.scatter(*X2.T, color='0.5', s=10)
plt.scatter(*model2.means_.T, color='b', s=10, label="GMM Centroids")
plt.scatter(*X2[selector.ranking].T, color='#FF6600', s=10, label="Selected Examples")
plt.axis('off')
plt.legend(loc=(1.01, 0.5), fontsize=12)

plt.tight_layout()
plt.show()
```

Here, the selected examples are close to one of the centroids, which is arguably in low-data areas and probably have a smaller variance. Because the other four centroids are in data-richer areas their variances likely overlap significantly, and so the chosen examples are in the central region between the three of them. Simply choosing the points near the centroid would not give the same results. This isn't to say that this is always exactly the most representative set from this data, just that this is a case where submodular optimization will provide different results from a simpler approach.

## Facility Location Selection

An alternative to feature-based functions are graph-based functions. These functions operate on a similarity matrix (note: a similarity matrix is the inverse of a distance matrix, where the most similar elements in a distance matrix have a pairwise value of zero whereas the most distant elements in a similarity matrix have a pairwise value of zero) instead of the feature values directly. Graph-based functions are generally more versatile than feature-based ones because any featurization of data can be converted into a similarity graph by calculating the Euclidean distance or correlation between examples, but data types that are inherently graphs can also be operated on.

A specific graph-based function is the facility location function, which has been used in the past to literally locate new facilities. In this setting, one wants to identify the next location that would serve the most people that are currently underserved without having to move any of the previous locations. The facility location function takes the following form:

\begin{equation}
f(X, V) = \sum\limits_{v \in V} \max\limits_{x \in X} \phi(x, v)
\end{equation}
where $x$ is a selected example $X$ is the set of already selected examples, $v$ is an unselected example, $V$ is the set of unselected examples, and $\phi$ is a similarity function that either returns an entry in a pre-defined similarity matrix or calculates the similarity between two examples.

A challenge with using graph-based functions is that the similarity matrix has to be calculated and stored in memory for efficient computation, which can be challenging for massive data sets. However, it is more versatile because similarities can be calculated that are more informative than simple featurizations.

### Greedy Version of K-means Clustering

A simple way to think about using submodular optimization to optimize a facility location function is that it is a greedy version of k-medoids clustering. As a refresher, k-medoids clustering is similar to k-means except that the cluster centroids must be examples from the training set. It is similar to the difference between calculating the mean and calculating the median. Submodular optimization on a facility location function involves iteratively choosing the example that best explains the previously explained examples, i.e., that maximizes the increase in similarity between all of the examples and all of the chosen examples.

What does that look like in practice? Implement a facility location selection object to choose 50 examples. You'll notice that, despite being a graph-based function, you can still pass in a feature matrix and it will automatically calculate a similarity graph from that.

```python
from apricot import FacilityLocationSelection

selector = ...

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Gaussian Blob Data", fontsize=14)
plt.scatter(*X.T, color='0.7', s=10)
plt.scatter(*model.means_.T, color='b', s=10)
plt.axis('off')

plt.subplot(122)
plt.title("Facility Location Selected Examples", fontsize=14)
plt.scatter(*X.T, color='0.7', s=10)
plt.scatter(*model.means_.T, color='b', s=10, label="GMM Centroids")
plt.scatter(*X[selector.ranking].T, color='#FF6600', s=10, label="Selected Examples")
plt.axis('off')
plt.legend(loc=(1.01, 0.5), fontsize=12)

plt.tight_layout()
plt.show()
```

The selected examples should appear to be fairly uniformly distributed across the space. If you're noticing a concentration of points anywhere, you may have incorrectly implemented something.

To get a sense for the selection process, let's visualize the iterative process of selecting examples.

```python
plt.figure(figsize=(14, 8))
for i in range(10):
    plt.subplot(2, 5, i+1)
    plt.title("{} Selections".format(i+1), fontsize=14)
    plt.scatter(*X.T, color='0.7', s=10)

    if i > 0:
        plt.scatter(*X[selector.ranking[:i]].T, color='0.1', s=10, label="Selected Examples")

    plt.scatter(*X[selector.ranking[i]].T, color='#FF6600', s=10, label="Next Selection")
    plt.axis('off')

plt.legend(loc=(1.01, 0.5), fontsize=12)
plt.tight_layout()
plt.show()
```

You'll notice that the first example comes from around the center of the data set. As a greedy approach, the optimizer is trying to find the single best example without knowing if it will be able to choose future ones. Then, the second example comes from an underrepresented area, etc.

### Digits Data Set

Now, let's apply facility location functions to choosing data for machine learning. A constraint of the feature-based functions is that they only work when the features follow a particular set of semantics. Although there are powerful approaches for transforming features into new features that follow those semantics, it's also nice to not have to do anything fancy to get a good set of items. A good example of data where the assumptions of feature-based functions don't work out of the box are those that involve images.

Let's download a reduced version of the digits data set and try training a machine learning model using selected subsets or random subsets, as we did before.

```python
from sklearn.datasets import load_digits

random.seed(0)

X, y = load_digits(return_X_y=True)

idxs = numpy.arange(X.shape[0])
numpy.random.shuffle(idxs)

X = X[idxs]
y = y[idxs]

X_train, y_train = X[:1000], y[:1000]
X_test, y_test = X[1000:], y[1000:]
```

Now, impelment a facility location function to choose 1000 examples and a feature-based function to also choose 1000 examples, for comparison.

```python
fl_selector = ...

fb_selector = ...
```

Let's see how the subsets selected using facility location fare against those selected using random selection or feature-based selection.

```python
numpy.random.seed(0)

model = SGDClassifier(random_state=0)
counts = numpy.arange(10, 1001, 10)

random_idxs = numpy.array([numpy.random.choice(X_train.shape[0], replace=False, size=1000) for i in range(10)])
random_accuracies, fl_accuracies, fb_accuracies = [], [], []

for count in tqdm(counts):
    #
    idxs = ...

    y_hat = model.predict(X_test)
    acc = (y_hat == y_test).mean()
    fl_accuracies.append(acc)

    #
    idxs = ...

    y_hat = model.predict(X_test)
    acc = (y_hat == y_test).mean()
    fb_accuracies.append(acc)

    accs = []
    for i in range(10):
        r_idxs = random_idxs[i, :count]
        ...

        y_hat = model.predict(X_test)
        acc = (y_hat == y_test).mean()
        accs.append(acc)

    random_accuracies.append(accs)
```

```python
plt.title("Reduced MNIST Classification", fontsize=14)
plt.plot(counts, numpy.mean(random_accuracies, axis=1), color='0.5', linewidth=2, label="Random")
plt.plot(counts, fl_accuracies, color='#FF6600', linewidth=2, label="Facility Location Optimization")
plt.plot(counts, fb_accuracies, color='g', linewidth=2, label="Feature-Based Optimization")

plt.xlabel("# Chosen Examples", fontsize=12)
plt.ylabel("Classification Accuracy")
plt.legend(loc=4, fontsize=12)

seaborn.despine()
plt.tight_layout()
plt.show()
```

Looks like the facility location function achieves high accuracy with only a small number of examples! Using only 40 examples achieves almost 90% accuracy, whereas it takes almost 200 randomly selected examples to get hat performance on average.

tutorials/Workshop Tutorial 1. Practical Apricot Usage - Worksheet.ipynb


#  NMA Model Types Tutorial 2: "How" models

In this tutorial we will explore models that can potentially explain *how* the spiking data we have observed is produced. That is, the models will tell us something about the *mechanism* underlying the physiological phenomenon.

Our objectives:
- Write code to simulate a  simple "leaky integrate-and-fire" neuron model
- Make the model more complicated — but also more realistic — by adding more physiologically-inspired details

```python
#@title Video: "How" models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='yWPQsBud4Cc', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=yWPQsBud4Cc

## Setup

**Don't forget to execute the hidden cells!**

```python
#@title Imports
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

import ipywidgets as widgets
```

```python
#@title Helper Functions
def histogram(counts, bins, vlines=(), ax=None, ax_args=None, **kwargs):
  """Plot a step histogram given counts over bins."""
  if ax is None:
    _, ax = plt.subplots()

  # duplicate the first element of `counts` to match bin edges
  counts = np.insert(counts, 0, counts[0])

  ax.fill_between(bins, counts, step="pre", alpha=0.4, **kwargs)  # area shading
  ax.plot(bins, counts, drawstyle="steps", **kwargs)  # lines

  for x in vlines:
    ax.axvline(x, color='r', linestyle='dotted')  # vertical line

  if ax_args is None:
    ax_args = {}

  # heuristically set max y to leave a bit of room
  ymin, ymax = ax_args.get('ylim', [None, None])
  if ymax is None:
    ymax = np.max(counts)
    if ax_args.get('yscale', 'linear') == 'log':
      ymax *= 1.5
    else:
      ymax *= 1.1
      if ymin is None:
        ymin = 0

  if ymax == ymin:
    ymax = None

  ax_args['ylim'] = [ymin, ymax]

  ax.set(**ax_args)
  ax.autoscale(enable=False, axis='x', tight=True)

def plot_neuron_stats(v, spike_times):
  fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))

  # membrane voltage trace
  ax1.plot(v[0:100])
  ax1.set(xlabel='Time', ylabel='Voltage')
  # plot spike events
  for x in spike_times:
    if x >= 100:
      break
    ax1.axvline(x, color='limegreen')

  # ISI distribution
  isi = np.diff(spike_times)
  n_bins = bins = np.arange(isi.min(), isi.max() + 2) - .5
  counts, bins = np.histogram(isi, bins)
  vlines = []
  if len(isi) > 0:
    vlines = [np.mean(isi)]
  xmax = max(20, int(bins[-1])+5)
  histogram(counts, bins, vlines=vlines, ax=ax2, ax_args={
    'xlabel': 'Inter-spike interval',
    'ylabel': 'Number of intervals',
    'xlim': [0, xmax]
  })
```

## The Linear Integrate-and-Fire Neuron

One of the simplest models of spiking neuron behavior is the linear integrate-and-fire model neuron. In this model, the neuron increases its membrane potential $V_m$ over time in response to excitatory input currents $I$ scaled by some factor $\alpha$:

\begin{align}
  dV_m = {\alpha}I
\end{align}

Once $V_m$ reaches a threshold value of 1, a spike is emitted, the neuron resets $V_m$ back to 0, and the process continues.

#### Spiking Inputs

We now have a model for the neuron dynamics. Next we need to consider what form the input $I$ will take. How should we represent the presynaptic neuron firing behavior providing the input coming into our model nueuron? We learned previously that a good approximation of spike timing is a Poisson random variable, so we can do that here as well

\begin{align}
  I \sim Poisson(\lambda)
\end{align}

where $\lambda$ is the average rate of incoming spikes.

### Exercise: Compute $dV_m$

For your first exercise, you will write the code to compute the change in voltage $dV_m$ of the linear integrate-and-fire model neuron. The rest of the code to handle the neuron dynamics are provided for you, so you just need to fill in a definition for `dv` in the `lif_neuron` method below. The value for $\lambda$ needed for the Poisson random variable is named `rate`.

TIP: The [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html) package is a great resource for working with and sampling from various probability distributions.

```python
def lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  """ Simulate a linear integrate-and-fire neuron.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    rate (int): The mean rate of incoming spikes

  """
  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):

    #######################################################################
    ## TODO for students: compute dv, then remove the NotImplementedError #
    #######################################################################
    I = stats.poisson.rvs(rate)
    dv = alpha * I
    # raise NotImplementedError("Student excercise: compute the change in membrane potential")

    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Uncomment these lines after completing the lif_neuron function
v, spike_times = lif_neuron()
plot_neuron_stats(v, spike_times)
```

```python
I = stats.poisson.rvs(10)
print(I)
```

    5

```python
# to_remove solution
def lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  """ Simulate a linear integrate-and-fire neuron.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    rate (int): The mean rate of incoming spikes
  """
  # precompute Poisson samples for speed
  exc = stats.poisson(rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = alpha * exc[i]
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

v, spike_times = lif_neuron()
with plt.xkcd():
  plot_neuron_stats(v, spike_times)
```

### Parameter Exploration

Here's an interactive demo that shows how the model behavior changes for different parameter values.

**Remember to enable the demo by running the cell.**

```python
#@title Linear Integrate-and-Fire Model Neuron Explorer

def _lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  exc = stats.poisson(rate).rvs(n_steps)
  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = alpha * exc[i]
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0
  return v, spike_times

@widgets.interact(
  n_steps=widgets.FloatLogSlider(1000.0, min=2, max=4),
  alpha=widgets.FloatLogSlider(0.01, min=-4, max=-1),
  rate=widgets.IntSlider(10, min=1, max=20)
)
def plot_lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  v, spike_times = _lif_neuron(int(n_steps), alpha, rate)
  plot_neuron_stats(v, spike_times)
```

## Inhibitory signals

Our linear integrate-and-fire neuron from the previous section was indeed able to produce spikes, but the actual spiking behavior did not line up with our expectations of exponentially distributed ISIs. This means we need to refine our model!

In the previous model we only considered excitatory behavior -- the only way the membrane potential could decrease is upon a spike event. We know, however, that there are other factors that can drive $V_m$ down. First is the natural tendancy of the neuron to return to some steady state or resting potential. We can update our previous model as follows:

\begin{align}
  dV_m = -{\beta}V_m + {\alpha}I
\end{align}

where $V_m$ is the current membrane potential and $\beta$ is some leakage factor. This is a basic form of the popular Leaky Integrate-and-Fire model neuron (for a more detailed discussion of the LIF Neuron, see the Appendix).

We also know that in addition to excitatory presynaptic neurons, we can have inhibitory presynaptic neurons as well. We can model these inhibitory neurons with another Poisson random variable, giving us

\begin{align}
I = I_{exc} - I_{inh} \\
I_{exc} \sim Poisson(\lambda_{exc}) \\
I_{inh} \sim Poisson(\lambda_{inh})
\end{align}

where $\lambda_{exc}$ and $\lambda_{inh}$ are the rates of the excitatory and inhibitory presynaptic neurons, respectively.

### Exercise: Compute $dV_m$ with inhibitory signals

For your second exercise, you will again write the code to compute the change in voltage $dV_m$, though now of the LIF model neuron described above. Like last time, the rest of the code needed to handle the neuron dynamics are provided for you, so you just need to fill in a definition for `dv` below.

```python
def lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """
  v = np.zeros(n_steps)

  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)
  curr = exc - inh

  spike_times = []
  for i in range(1, n_steps):

    ############################################################
    ## Students: compute dv and remove the NotImplementedError #
    ############################################################
    dv = -beta * v[i-1] + alpha * (exc[i] - inh[i])
    # raise NotImplementedError("Student excercise: compute the change in membrane potential")

    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Uncomment these lines do make the plot once you've completed the function
v, spike_times = lif_neuron_inh()
plot_neuron_stats(v, spike_times)
```

```python
# to_remove solution
def lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """
  # precompute Poisson samples for speed
  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = -beta * v[i-1] + alpha * (exc[i] - inh[i])
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times
```

```python
v, spike_times = lif_neuron_inh()
plot_neuron_stats(v, spike_times)
```

### Parameter Exploration

Like last time, you can now explore how you LIF model behaves when the various parameters of the system are changed.

```python
#@title LIF Model Neuron with Inhibitory Inputs Explorer

def _lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """
  # precompute Poisson samples for speed
  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = -beta * v[i-1] + alpha * (exc[i] - inh[i])
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

@widgets.interact(n_steps=widgets.FloatLogSlider(1000.0, min=2, max=4),
                  alpha=widgets.FloatLogSlider(0.5, min=-2, max=1),
                  beta=widgets.FloatLogSlider(0.1, min=-2, max=0),
                  exc_rate=widgets.IntSlider(10, min=1, max=20),
                  inh_rate=widgets.IntSlider(10, min=1, max=20))
def plot_lif_neuron(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  v, spike_times = _lif_neuron_inh(int(n_steps), alpha, beta, exc_rate, inh_rate)
  plot_neuron_stats(v, spike_times)
```

```python
#@title Video: Balanced inputs
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='buXEQPp9LKI', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=buXEQPp9LKI

## Appendix

### Why do neurons spike?

A neuron stores energy in an electric field across its cell membrane, by controlling the distribution of charges (ions) on either side of the membrane. This energy is rapidly discharged to generate a spike when the field potential (or membrane potential) crosses a threshold. The membrane potential may be driven toward or away from this threshold, depending on inputs from other neurons: excitatory or inhibitory, respectively. The membrane potential tends to revert to a resting potential, for example due to the leakage of ions across the membrane, so that reaching the spiking threshold depends not only on the amount of input ever received following the last spike, but also the timing of the inputs.

The storage of energy by maintaining a field potential across an insulating membrane can be modeled by a capacitor. The leakage of charge across the membrane can be modeled by a resistor. This is the basis for the leaky integrate-and-fire neuron model.

### The LIF Model Neuron

The full equation for the LIF neuron is

\begin{align}
C_{m}\frac{dV_m}{dt} = -(V_m - V_{rest})/R_{m} + I
\end{align}

where $C_m$ is the membrane capacitance, $R_M$ is the membrane resistance, $𝑉_{𝑟𝑒𝑠𝑡}$ is the resting potential, and  𝐼  is some input current (from other neurons, an electrode, ...).

In our above examples we set many of these properties to convenient values ($C_m = R_m = dt = 1$, $V_{rest} = 0$) to focus more on the overall behavior, though these too can be manipulated to achieve different dynamics.


# Field, Goldsmith, & Habing Multi-Phase ISM

Figure 1.10-1.12 from Chapter 1 of *Interstellar and Intergalactic Medium* by Ryden & Pogge, 2021,
Cambridge University Press.

This notebook creates figures illustrating the Field, Goldsmith, and Habing (FGH) multi-phase interstellar
medium model [Field, Goldsmith, & Habing 1969, ApJ, 155, L149](https://ui.adsabs.harvard.edu/abs/1969ApJ...155L.149F/abstract)

There are 3 figures
 * Figure 1.10 - FGH Cooling function $\Lambda(T)$
 * Figure 1.11 - Equilibrium density $n_{eq}(T)$
 * Figure 1.12 - Pressure $P$ vs density $n_{eq}$

```python
%matplotlib inline

import math
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

# SciPy bits we use for analysis

from scipy.signal import argrelmin, argrelmax
from scipy import stats

import warnings
warnings.filterwarnings('ignore',category=UserWarning, append=True)
```

## Standard Plot Format

Setup the standard plotting format and make the plot. Fonts and resolution adopted follow CUP style.

```python
# graphic aspect ratio = width/height

aspect = 4.0/3.0

# Text width in inches - don't change, this is defined by the print layout

textWidth = 6.0 # inches

# output format and resolution

figFmt = 'png'
dpi = 600

# Graphic dimensions

plotWidth = dpi*textWidth
plotHeight = plotWidth/aspect
axisFontSize = 14
labelFontSize = 10
lwidth = 0.5
axisPad = 5
wInches = textWidth
hInches = wInches/aspect

# LaTeX is used throughout for markup of symbols, Times-Roman serif font

plt.rc('text', usetex=True)
plt.rc('font', **{'family':'serif','serif':['Times-Roman'],'weight':'bold','size':'16'})

# Font and line weight defaults for axes

matplotlib.rc('axes',linewidth=lwidth)
matplotlib.rcParams.update({'font.size':axisFontSize})

# axis and label padding

plt.rcParams['xtick.major.pad'] = f'{axisPad}'
plt.rcParams['ytick.major.pad'] = f'{axisPad}'
plt.rcParams['axes.labelpad'] = f'{axisPad}'
```

## FGH Model Calculation

The basic parameters for these models are set in the code section below:

 * Ionization fraction: $x_e$=0.001
 * Temperature Range: $T_e$=10 to 20000K (logratithmic)
 * gain factor: $G$=20, defined such that $n_{eq}$=G/$\Lambda$ ($\Lambda$ is the total cooling rate)

The model assumes three sources of collisional cooling using these scaling relations:

HI Lyman-$\alpha$ Cooling (Eqn 1.38):
\begin{equation}
  \frac{\Lambda^{e}_{Ly\alpha}}{10^{-27}{\rm erg\,cm^3\,s^{-1}}} \approx
  6\times10^{5} \left(\frac{x}{0.001}\right)
  \left(\frac{T}{10^{4}K}\right)^{-1/2}
  \exp\left(-\frac{1.18\times10^{5}K}{T}\right)
\end{equation}

Carbon (CII) Cooling (Eqn 1.35) electron collisional term:
\begin{equation}
  \frac{\Lambda^{e}_{CII}}{10^{-27}{\rm erg\,cm^3\,s^{-1}}} \approx
  3.1 \left(\frac{x}{0.001}\right)
  \left(\frac{T}{100K}\right)^{-0.5}
  \exp\left(-\frac{91.2K}{T}\right)
\end{equation}
and H collisional term:
\begin{equation}
  \frac{\Lambda^{H}_{CII}}{10^{-27}{\rm erg\,cm^3\,s^{-1}}} \approx
  5.2\left(\frac{T}{100K}\right)^{0.13}
  \exp\left(-\frac{91.2K}{T}\right)
\end{equation}

Oxygen (OI) Cooling:
\begin{equation}
   \frac{\Lambda^{H}_{OI}}{10^{-27}{\rm erg\,cm^3\,s^{-1}}} \approx
   4.1\left(\frac{T}{100K}\right)^{0.42}
   \exp\left(-\frac{228K}{T}\right)
\end{equation}

We compute total cooling ($\Lambda=\Lambda_{Ly\alpha}+\Lambda_{CII}+\Lambda_{OII}$), equilibrium density
($n_{eq}$), and pressure ($P=n_{eq}kT$) as a function of logarithmic steps in temperature.

We have adopted the Lodders (2010) abundances for C and O, as used in the ISM/IGM book
(see Chapter 1, Table 1.2).

```python
xe = 0.001
minT = 10.0
maxT = 20000.
gain = 20.0

# Boltzmann Constant (CODATA 2018)

k = 1.380649e-16 # erg K^-1

minLogT = math.log10(minT)
maxLogT = math.log10(maxT)

logT = np.linspace(minLogT,maxLogT,num=1001)
T = 10.0**logT

xfac = xe/0.001
TH = 118000.0 # hydrogen excitation temperature in K
TC = 91.2     # carbon excitation temperature in K
TO = 228.0    # oxygen excitation temperature in K

# Lyman-alpha cooling

coolLya = 6.0e5*(xfac/np.sqrt(T/1.0e4))*np.exp(-TH/T)

# Carbon cooling

coolC = 3.1*(xfac/np.sqrt(T/100.0))*np.exp(-TC/T) + 5.2*((T/100.0)**0.13)*np.exp(-TC/T)

# Oxygen cooling

coolO = 4.1*((T/100.0)**0.42)*np.exp(-TO/T)

# Total cooling

coolTot = (coolLya + coolC + coolO)

# equilibrium density

neq = gain/coolTot

# pressure

P = neq*k*T

```

## FGH Cooling Function - Figure 1.10

Plot the cooling function $\Lambda(T)$ vs $T$ including the curves for the individual contributions

```python
plotFile = f'Fig1_10.{figFmt}'

fig,ax = plt.subplots()
fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

ax.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
ax.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

# Limits

minCool = 1.0e-30 # erg cm^3 s^-1
maxCool = 1.0e-24

# Labels

xLabel = r'Temperature [K]'
yLabel = r'$\Lambda$ [erg cm$^3$ s$^{-1}$]'

plt.xlim(minT,maxT)
ax.set_xscale('log')
ax.set_xticks([10,100,1000,1.0e4])
ax.set_xticklabels(['10','100','1000','10$^{4}$'])
plt.xlabel(xLabel)

plt.ylim(minCool,maxCool)
ax.set_yscale('log')
ax.set_yticks([1.0E-30,1.0E-29,1.0E-28,1.0E-27,1.0e-26,1.0e-25,1.0e-24])
ax.set_yticklabels(['$10^{-30}$','10$^{-29}$','10$^{-28}$','10$^{-27}$','10$^{-26}$','10$^{-25}$','10$^{-24}$'])
plt.ylabel(yLabel)

# Plot the total and individual cooling functions

plt.plot(T,1.0e-27*coolTot,'-',color='black',lw=2,zorder=10)
plt.plot(T,1.0e-27*coolLya,'--',color='black',lw=1,zorder=10)
plt.plot(T,1.0e-27*coolC,':',color='black',lw=1,zorder=10)
plt.plot(T,1.0e-27*coolO,'-.',color='black',lw=1,zorder=10)

# label components

lfs = np.rint(1.2*axisFontSize)
plt.text(1000.0,1.7e-26,'Total',fontsize=lfs,rotation=10.0,ha='center',va='bottom')
plt.text(80.0,1.0e-28,r'$[\textsc{O\,i}]\,\lambda$63$\mu m$',fontsize=lfs)
plt.text(3000.0,3.5e-27,r'$[\textsc{C\,ii}]\,\lambda$158$\mu m$',fontsize=lfs,rotation=3.0,ha='center')
plt.text(5400.0,1.0e-28,r'Ly$\alpha$',fontsize=lfs,ha='center')

# make the figure

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```

## FGH equilibrium density - Figure 1.11

Plot the equlibrium density function $n_{eq}$ vs $T$ for the FGH model.

```python
plotFile = f'Fig1_11.{figFmt}'

fig,ax = plt.subplots()
fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

ax.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
ax.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

# Limits

minNe = 0.01     # cm^{-3}
maxNe = 20000.0

# Labels

xLabel = r'Temperature [K]'
yLabel = r'$n$ [cm$^{-3}$]'

plt.xlim(minT,maxT)
ax.set_xscale('log')
ax.set_xticks([10,100,1000,1.0e4])
ax.set_xticklabels(['10','100','1000','10$^{4}$'])
plt.xlabel(xLabel)

plt.ylim(minNe,maxNe)
ax.set_yscale('log')
ax.set_yticks([0.01,0.1,1.0,10.,100.,1e3,1e4])
ax.set_yticklabels(['0.01','0.1','1','10','100','1000','10$^{4}$'])
plt.ylabel(yLabel)

# Plot neq vs T

plt.plot(T,neq,'-',color='black',lw=2,zorder=10)
plt.fill_between(T,neq,maxNe,facecolor="#eaeaea")

# label regions above and below

lfs = np.rint(1.2*axisFontSize)
plt.text(200.0,0.1,'Net heating',fontsize=lfs,ha='center',zorder=10)
plt.text(1000.0,20.0,'Net cooling',fontsize=lfs,ha='center',zorder=10)

# make the figure

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```

## FGH pressure vs density - Figure 1.12

Plot the equlibrium pressure vs density for the FGH model.

We numerically search for the stability region pressure limits and the crossing points at a reference pressure of
P= 2×10−13 dyne/cm 2 . The methods used are a little dodgy, but are robust here as the pressure-density curve is
well-behaved.

```python
plotFile = f'Fig1_12.{figFmt}'

fig,ax = plt.subplots()
fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

plt.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
plt.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

# Limits

minNe = 0.02     # cm^{-3}
maxNe = 10000.0

minP = 4.0e-14   # dyne cm^-2
maxP = 1.0e-11

# Labels

xLabel = r'$n$ [cm$^{-3}$]'
yLabel = r'$P$ [dyne cm$^{-2}$]'

plt.xlim(minNe,maxNe)
plt.xscale('log')
ax.set_xticks([0.1,1.0,10.,1.0e2,1.0e3,1.0e4])
ax.set_xticklabels(['0.1','1.0','10','100','1000','10$^4$'])
plt.xlabel(xLabel)

plt.ylim(minP,maxP)
ax.set_yscale('log')
ax.set_yticks([1.0e-13,1.0e-12,1.0e-11])
ax.set_yticklabels(['10$^{-13}$','10$^{-12}$','10$^{-11}$'])
plt.ylabel(yLabel)

# plot the n-P curve

plt.plot(neq,P,'-',color='black',lw=2,zorder=10)
plt.fill_between(neq,P,maxP,facecolor="#eaeaea")

# FGH stability region - estimate from array using scipy.signal argrelmin() and argrelmax()
# peak-finding functions

iMin = argrelmin(P)[0]
iMax = argrelmax(P)[0]

plt.hlines(P[iMin],minNe,maxNe,color='black',ls='--',lw=0.5)
plt.hlines(P[iMax],minNe,maxNe,color='black',ls='--',lw=0.5)

# Reference pressure, 2e-13 dyne/cm^2

pFGH = 2.0e-13

# The FGH points are at zero crossings of P(n)-fghP.  Find the nearest zero-crossing, then
# fit a line to +/-3 points around it and find the crossing point.  This is dodgy generally
# but we get away with it because the P-n curve is well-behaved.

iFGH = np.where(np.diff(np.sign(P-pFGH)))[0]

nFGH = []
for i in iFGH:
    slope, inter, rVal, pVal, stdErr = stats.linregress(neq[i-3:i+3],P[i-3:i+3]-pFGH)
    xZero = -inter/slope
    nFGH.append(xZero)
    # print(f'n_eq = {xZero:.5e} cm^-3')

lfs = np.rint(1.2*axisFontSize)

plt.plot(nFGH[0],pFGH,color='black',marker='o',ms=8,mfc='black')
plt.text(1.4*nFGH[0],pFGH,'F',fontsize=lfs,va='center',zorder=10)
plt.plot(nFGH[1],pFGH,color='black',marker='o',ms=8,mfc='black')
plt.text(1.4*nFGH[1],pFGH,'G',fontsize=lfs,va='center',zorder=10)
plt.plot(nFGH[2],pFGH,color='black',marker='o',ms=8,mfc='black')
plt.text(1.4*nFGH[2],pFGH,'H',fontsize=lfs,va='center',zorder=10)

plt.text(10.0,1.1*P[iMax],'Net cooling',fontsize=lfs,ha='center',va='bottom',zorder=10)
plt.text(1300.0,pFGH,'Net heating',fontsize=lfs,ha='center',va='center',zorder=10)

# make the figure

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```


We now extend SEIR model to SEAI5R:

\begin{align}
\dot{S_{i}}	&=-\lambda_{i}(t)S_{i}+\sigma_{i}, \\
\dot{E}_{i}	&=\lambda_{i}(t)S_{i}-\gamma_{E}E_{i},\\
\dot{A}_{i}	&=\gamma_{E}E_{i}-\gamma_{A}A_{i},\\
\dot{I}_{i}^{a}	&=\alpha\gamma_{A}A_{i}-\gamma_{I^{a}}I_{i}^{a},\\
\dot{I}_{i}^{s}	&=\bar{\alpha}\gamma_{A}A_{i}-\gamma_{I^{s}}I_{i}^{s},\\
\dot{I}_{i}^{h}	&=h_{i}\gamma_{I^{s}}I_{i}^{s}-\gamma_{I^{h}}I_{i}^{h},\\
\dot{I}_{i}^{c}	&=c_{i}\gamma_{I^{h}}I_{i}^{h}-\gamma_{I^{c}}I_{i}^{c},\\
\dot{I}_{i}^{m}	&=m_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{R}_{i} &=\gamma_{I^{a}}I_{i}^{a}+\bar{h}_{i}\gamma_{I^{s}}I_{i}^{s}+\bar{c}_{i}\gamma_{I^{h}}I_{i}^{h}+\bar{m}_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{N}_{i}	&=\sigma_{i}-I_{i}^{m}.
\end{align}

$\lambda_{i}(t)=\beta\sum_{j=1}^{M}\left(C_{ij}^{a}\frac{A_{j}}{N_{j}}+C_{ij}^{a}\frac{I_{j}^{a}}{N_{j}}+C_{ij}^{s}\frac{I_{j}^{s}}{N_{j}}+C_{ij}^{h}\frac{I_{j}^{h}}{N_{j}}\right)$.

Here
* $I^{a}$ : asymptomatic infectives
* $I^{s}$ : symptomatic infectives
* $I^{h}$ : hospitalized infectives
* $I^{c}$ : ICU cases
* $I^{m}$ : mortality
* ${h}_{i}=1-\bar h_{i}$ is the fraction of syptomatics who are hospitalized
* $c_{i}=1-\bar{c}_{i}$, is the fraction of hospitalizeds, who are in ICU
* $m_{i}=1-\bar{m}_{i}$, is the fraction of ICU which leads to mortality
* $C_{ij}^{s}=f^{s}C_{ij}^{a}\equiv f^{s}C_{ij}$
* $C_{ij}^{h}=f^{h}C_{ij}^{a}\equiv f^{h}C_{ij}$.

S ---> E

E ---> A

A ---> Ia, Is

Ia ---> R

Is ---> Ih, R

Ih ---> Ic, R

Ic ---> Im, R

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
```

```python
M=16  # number of age groups

# load age structure data
my_data = np.genfromtxt('../data/age_structures/UK.csv', delimiter=',', skip_header=1)
aM, aF = my_data[:, 1], my_data[:, 2]

# set age groups
Ni=aM+aF;   Ni=Ni[0:M];  N=np.sum(Ni)
```

```python
# contact matrices
CH, CW, CS, CO = pyross.contactMatrix.UK()

## matrix of total contacts
C=CH+CW+CS+CO

fig,aCF =  plt.subplots(2,2);
aCF[0][0].pcolor(CH, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[0][1].pcolor(CW, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][0].pcolor(CS, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][1].pcolor(CO, cmap=plt.cm.get_cmap('GnBu', 10));
```

## Fraction of asymptomatic infectives is constant

```python
beta  = 0.036692            # infection rate

gE    = 1/5
gA    = 1/3
gIa   = 1./7                # recovery rate of asymptomatic infectives
gIs   = 1./7                # recovery rate of symptomatic infectives
alpha = 0.3                 # fraction of asymptomatic infectives
fsa   = 0.2                 # the self-isolation parameter
fh    = 0
gIh   = 1/14
gIc   = 1/14

sa    = 100*np.ones(M)      #  rate of additional/removal of population by birth etc
sa[0]     = 1500            # birth
sa[12:16] = -300            # mortality

hh   = 0.1*np.ones(M)       # fraction which goes from Is to hospital
cc   = 0.05*np.ones(M)      # fraction which goes from  hospital to ICU
mm   = 0.4*np.ones(M)       # mortality from IC

# initial conditions
Is_0 = np.zeros((M));  #Is_0[6:13]=8;  Is_0[2:6]=4; Is_0[13:16]=4

Ia_0 = 1000*np.ones((M));
R_0  = np.zeros((M))
E_0  = np.zeros((M))
A_0  = np.zeros((M))
Ih_0  = np.zeros((M))
Ic_0  = np.zeros((M))
Im_0  = np.zeros((M))

S_0  = Ni - (E_0 + A_0 + Ia_0 + Is_0 + Ih_0 + Ic_0  + R_0)

# matrix for linearised dynamics
L0 = np.zeros((M, M))
L  = np.zeros((2*M, 2*M))

for i in range(M):
    for j in range(M):
        L0[i,j]=C[i,j]*Ni[i]/Ni[j]

L[0:M, 0:M]     =    alpha*beta/gIs*L0
L[0:M, M:2*M]   = fsa*alpha*beta/gIs*L0
L[M:2*M, 0:M]   =    ((1-alpha)*beta/gIs)*L0
L[M:2*M, M:2*M] = fsa*((1-alpha)*beta/gIs)*L0

r0 = np.max(np.linalg.eigvals(L))
print("The basic reproductive ratio for these parameters is", r0)
```

    The basic reproductive ratio for these parameters is (1.3199080176950944+0j)

```python
# duration of simulation and data file
Tf=200;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE, 'gA':gA,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEAI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC  = np.zeros((Nf))
for i in range(1*M):
        IC += data['X'][:,3*M+i]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC, '-', lw=4, color='#A60628', label='forecast', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```

## Fraction of asymptomatic infectives is age-dependent

```python
alpha = 0.5*np.ones(M)      #  rate of additional/removal of population by birth etc
alpha[6:M] = 0.1              # everyone is symtomatic in large age-group

Ia_0 = 1000*np.ones((M));
```

```python

# duration of simulation and data file
Tf=200;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE, 'gA':gA,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEAI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC1  =  data['X'][:,3*M+1];
IC2  =  data['X'][:,3*M+8]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC1, '-', lw=4, color='#A60628', label='M=1', alpha=0.8)
plt.plot(t, IC2, '-', lw=4, color='green', label='M=16', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```


```python
# Imports and declarations
%matplotlib inline
import sys
import numpy as np
import matplotlib.pyplot as plt
plt.style.use("ggplot")
plt.rcParams["image.cmap"] = "binary_r"
sys.path.append("../../semi-supervised")

import torch
import torch.nn.functional as F
from torchvision import datasets, transforms
cuda = torch.cuda.is_available()
```

# Auxiliary Deep Generative Model

In this notebook we will run the Auxiliary Deep Generative Model as described in (Maaløe 2016). The model is similar to the Deep Generative Model described in (Kingma 2014), with the addition of an auxiliary variable $a$. The idea is that introducing another latent variable that is marginalised out will allow the autoencoder to learn more complicated distributions and will be able to seperate the latent factors with higher precision.

In the MNIST case, we can represent the variables as:

* $y$: the label of the digit
* $a$: the style of the digit

```python
from data.limitedmnist import LimitedMNIST
from utils import generate_label, onehot

batch_size = 64

labels = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
n = len(labels)

# Load in data
mnist_lab = LimitedMNIST('./', train=True, transform=torch.bernoulli, target_transform=onehot(n), digits=labels, fraction=0.005)
mnist_ulab = LimitedMNIST('./', train=True, transform=torch.bernoulli, target_transform=onehot(n), digits=labels, fraction=1.0)
mnist_val = LimitedMNIST('./', train=False, transform=torch.bernoulli, target_transform=onehot(n), digits=labels)

# Unlabelled data
unlabelled = torch.utils.data.DataLoader(mnist_ulab, batch_size=batch_size, shuffle=True, num_workers=2)

# Validation data
validation = torch.utils.data.DataLoader(mnist_val, batch_size=batch_size, shuffle=True, num_workers=2)

# Labelled data
labelled = torch.utils.data.DataLoader(mnist_lab, batch_size=batch_size, shuffle=True, num_workers=2)
```

```python
f, axarr = plt.subplots(1, 10, figsize=(10, 2))
images, labels = next(iter(labelled))
_, labels = torch.max(labels, 1)

for i in range(10):
    axarr[i].imshow(images[i].numpy().reshape(28, 28))
    title = labels[i]# if labels[i] < 5 else "Unknown"
    axarr[i].set_title(title)
    axarr[i].axis("off")

f.suptitle("Data samples after Bernoulli transform")
plt.show()
```

Again `VariationalInferenceWithLabels` is used to model the objective, shown below:

\begin{align}
\log p(x, y) &\geq \mathbb{E}_{q_{\phi}(a, z|x, y)} [ \log p_{\theta}(x, y, a, z) - \log q_{\phi}(a, y, z|x) ]\\
&= \mathbb{E}_{q_{\phi}(a, z|x, y)} [ \log p_{\theta}(x, y, a, z) - \log q_{\phi}(a | x) -  \log q_{\phi}(z | a, y, x) ] = - \mathcal{L}(x, y)
\end{align}

```python
from models import AuxiliaryDeepGenerativeModel
from inference.loss import VariationalInferenceWithLabels, kl_divergence_normal, discrete_uniform_prior

# Numerical stability
epsilon = 1e-7

# ADGM with a single hidden layer in both the encoder and decoder
model = AuxiliaryDeepGenerativeModel(ratio=len(mnist_ulab)/len(mnist_lab), dims=[28 * 28, n, 16, [128, 64]])

if cuda: model.cuda()

def binary_cross_entropy(r, x):
    return torch.sum((x * torch.log(r + epsilon) + (1 - x) * torch.log((1 - r) + epsilon)), dim=-1)

def cross_entropy(y, logits):
    return -torch.sum(y * torch.log(logits + epsilon), dim=1)

objective = VariationalInferenceWithLabels(binary_cross_entropy, kl_divergence_normal, discrete_uniform_prior)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)

model
```

    AuxiliaryDeepGenerativeModel (
      (encoder): Encoder (
        (hidden): ModuleList (
          (0): Linear (16 -> 128)
          (1): Linear (128 -> 64)
        )
        (sample): StochasticGaussian (
          (mu): Linear (64 -> 16)
          (log_var): Linear (64 -> 16)
        )
      )
      (decoder): Decoder (
        (hidden): ModuleList (
          (0): Linear (16 -> 64)
          (1): Linear (64 -> 128)
        )
        (reconstruction): Linear (128 -> 784)
        (output_activation): Sigmoid ()
      )
      (aux_encoder): Encoder (
        (hidden): ModuleList (
          (0): Linear (784 -> 128)
          (1): Linear (128 -> 64)
        )
        (sample): StochasticGaussian (
          (mu): Linear (64 -> 16)
          (log_var): Linear (64 -> 16)
        )
      )
      (aux_decoder): Decoder (
        (hidden): ModuleList (
          (0): Linear (16 -> 64)
          (1): Linear (64 -> 128)
        )
        (reconstruction): Linear (128 -> 16)
        (output_activation): Sigmoid ()
      )
      (classifier): Classifier (
        (dense): Linear (784 -> 128)
        (logits): Linear (128 -> 10)
      )
      (transform_x_to_z): Linear (784 -> 16)
      (transform_y_to_z): Linear (10 -> 16)
      (transform_z_to_x): Linear (16 -> 784)
    )

Something about unlabelled lower bound.

```python
class DGMTrainer():
    """
    Class for training Deep Generative Models.
    :param model: Object of class `DeepGenerativeModel`
    :param objective: Loss function for labelled data, e.g. `VariationalInferenceWithLabels`
    :param optimizer: A PyTorch-enabled optimizer
    :param cuda: Optional parameter whether to use CUDA acceleration
    """
    def __init__(self, model, objective, optimizer, cuda=False):
        self.model = model
        self.objective = objective
        self.optimizer = optimizer
        self.cuda = cuda

    def calculate_loss(self, x, y=None):
        """
        Given a semi-supervised problem (x, y) pair where y
        is only occasionally observed, calculates the
        associated loss.
        :param x: Features
        :param y: Labels (optional)
        :returns L_alpha if labelled, U if unlabelled.
        """
        is_unlabelled = True if y is None else False

        x = Variable(x)
        logits = self.model(x)

        # If the data is unlabelled, sum over all classes
        if is_unlabelled:
            [batch_size, *_] = x.size()
            x = x.repeat(n, 1)
            y = torch.cat([generate_label(batch_size, i, n) for i in range(n)])

        y = Variable(y.type(torch.FloatTensor))

        if self.cuda:
            x = x.cuda()
            y = y.cuda()

        # Compute lower bound (the same as -L)
        reconstruction, _, z, a = self.model(x, y)
        ELBO = self.objective(reconstruction, x, y, [z, a])

        # In the unlabelled case calculate the entropy H and return U
        if is_unlabelled:
            ELBO = ELBO.view(logits.size())
            loss = torch.sum(torch.mul(logits, ELBO - torch.log(logits)), -1)
            loss = -torch.mean(loss)
        # In the case of labels add cross entropy and return L_alpha
        else:
            loss = ELBO + self.model.beta * -cross_entropy(y, logits)
            loss = -torch.mean(loss)

        return loss

    def train(self, labelled, unlabelled):
        """
        Trains a DGM model based on some data.
        :param labelled: Labelled data loader
        :param inlabelled: Unlabelled data loader
        :return L, U: Final loss values.
        """
        for (x, y), (u, _) in zip(labelled, unlabelled):
            L = self.calculate_loss(x, y)
            U = self.calculate_loss(u, None)

            J = L + U

            J.backward()
            self.optimizer.step()
            self.optimizer.zero_grad()

        return L, U
```

## Visualisation

Here you can optionally enable the use of `visdom` to visualise the training process.

```python
use_visdom = True

if use_visdom:
    import visdom
    vis = visdom.Visdom()

class Visualiser():
    def __init__(self):
        self.loss = vis.line(X=np.array([0]), Y=np.array([0]), opts=dict(title="Training Loss", xlabel="Epoch"))
        self.acc  = vis.line(X=np.array([0]), Y=np.array([0]), opts=dict(title="Accuracy", xlabel="Epoch"))

    def update_loss(self, L, U):
        vis.updateTrace(X=np.array([epoch]), Y=L.data.numpy(), win=self.loss, name="Labelled")
        vis.updateTrace(X=np.array([epoch]), Y=U.data.numpy(), win=self.loss, name="Unlabelled")

    def update_accuracy(self, model):
        accuracy = []
        for x, y in validation:
            _, prediction = torch.max(model(Variable(x)), 1)
            _, y = torch.max(y, 1)

            accuracy += [torch.mean((prediction.data == y).float())]

        vis.updateTrace(X=np.array([epoch]), Y=np.array([np.mean(accuracy)]), win=self.acc)

    def update_images(self, model):
        x, y = next(iter(validation))
        input = Variable(x[:5])
        label = Variable(y[:5].type(torch.FloatTensor))
        x_hat, *_ = model(input, label)
        images = x_hat.data.numpy().reshape(-1, 1, 28, 28)

        vis.images(images, opts=dict(width=5*64, height=64, caption="Sample epoch {}".format(epoch)))
```

Controlling which dataloaders to use, we gather the losses into a single combined loss that we can backpropagate.

$$\mathcal{J}^{\alpha} = \mathcal{L}^{\alpha} + \mathcal{U}$$

```python
from torch.autograd import Variable

trainer = DGMTrainer(model, objective, optimizer)
visual = Visualiser()

for epoch in range(1001):
    L, U = trainer.train(labelled, unlabelled)

    if use_visdom:
        # Plot the last L and U of the epoch
        visual.update_loss(L, U)
        visual.update_accuracy(model)

        if epoch % 50 == 0:
            visual.update_images(model)
```

```python
z = Variable(torch.randn(16, 16))
a = Variable(torch.randn(16, 16))
y = Variable(generate_label(16, 0, 10))

x = model.sample(z, a, y)

images = x.view(-1, 28, 28).data.numpy()

f, axarr = plt.subplots(4, 4, figsize=(4, 4))

for i in range(16):
    axarr[i%4, i//4].imshow(images[i])
    axarr[i%4, i//4].axis("off")
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
```

```python
plt.style.use('fivethirtyeight')
```

# Problem 11.4 - Fogler - 5ed

The elementary, irreversible, organic liquid-phase reaction
\begin{equation}
    A + B \rightarrow C
\end{equation}

is carried out adiabatically in a flow reactor. An equal molar feed in A and B enters at 27C, and the volumetric flow rate is 2 dm3/s and CA0 = 0.1 kmol/m$^3$.

_Additional information_:

Parameter | Value | Parameter | Value
- | - | - | -
HA(273 K) | -20 kcal/mol | CpA | 15 cal/mol/K
HB(273 K) | -15 kcal/mol | CpB | 15 cal/mol/K
HC(273 K) | -41 kcal/mol | CpC | 30 cal/mol/K
k (300 K) | 0.01 dm$^3$/mol/s | E | 10,000 cal/mol

**PFR**
1. Plot and then analyze the conversion and temperature as a function of PFR volume up to where X = 0.85. Describe the trends.
2. What is the maximum inlet temperature one could have so that the boiling point of the liquid (550 K) would not be exceeded even for complete conversion?
3. Plot the heat that must be removed along the reactor ( Q ̇ vs. V) to maintain isothermal operation.
4. Plot and then analyze the conversion and temperature profiles up to a PFR reactor volume of 10 dm3 for the case when the reaction is reversible with KC = 10 m3/kmol at 450 K. Plot the equilibrium conversion profile. How are the trends different than part (a)?

**CSTR**
- What is the CSTR volume necessary to achieve 90% conversion?

# Data

```python
HA_273 = -20000    # Standard enthalpy for A, cal/mol
HB_273 = -15000    # Standard enthalpy for B, cal/mol
HC_273 = -41000    # Standard enthalpy for C, cal/mol

To = 300.15        # Feed temperature, K
vo = 2             # inlet volumetruc flow rate, dm3/s
Cao = 0.1          # Initial concetration for A, mol/dm3

CpA = 15           # heat capacity for A, cal/mol/K
CpB = 15           # heat capacity for B, cal/mol/K
CpC = 30           # heat capacity for C, cal/mol/K

k_300 = 0.01       # kinetic constant at 300 K, dm3/mol/s
Ea = 10000         # Activation energy, cal/mol

Rgas = 1.987       # Universal gas constant, cal/mol/K

Xamax = 0.85
```

```python
Fao = Cao * vo
```

```python
Dh = HC_273 - HB_273 - HA_273

print('The reaction heat at 300K is {:.2f} cal/mol'.format(Dh))
```

    The reaction heat at 300K is -6000.00 cal/mol

# PFR
## Part (a)

We need to solve the mole balance and energy balance, simultaneously

The mole balance:

\begin{align}
&\frac{dx}{dV} = -\frac{r_A}{F_{Ao}} \\
&r_A = - k(T) \, C_{Ao}^2 \, \left(1-x\right)^2\\
&k(T) = k(300) \exp\left(\frac{-Ea}{R}\left( \frac{1}{T}-\frac{1}{300}\right)\right)
\end{align}

The energy balance

\begin{equation}
T = To - \frac{\Delta H\, x}{\sum{\theta_i \, Cp_i} + \Delta Cp \, x}
\end{equation}

```python
def system(x, v):
    t = To - (Dh * x) / (CpA + CpB)
    k = k_300 * np.exp(-Ea/Rgas * (1/t - 1/To))

    ra = -k * Cao**2 * (1-x)**2

    dv_dx = -Fao / ra
    return dv_dx

```

```python
# initial condition
ICs = [0]            # the initial volume of the reactor is 0

xspan = [0, Xamax]  #  independent variable range

solODE = solve_ivp(system, xspan, ICs, t_eval=np.linspace(0, Xamax))
```

```python
print('The reactor volume is {:.2f} dm^3'.format(solODE.y[0][-1]))
```

    The reactor volume is 305.10 dm^3

The temperature profile is calculated from the energy balance based on the conversion profile

```python
Temp = To - (Dh * solODE.t) / (CpA + CpB)
```

```python
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 7))

ax1.plot(solODE.y[0], solODE.t)
ax1.set_xlabel(r'Reactor volume, dm$^3$')
ax1.set_ylabel('Conversion, adim')

ax2.plot(solODE.y[0], Temp)
ax2.set_xlabel(r'Reactor volume, dm$^3$')
ax2.set_ylabel('Reactor tempeture, K')

plt.tight_layout()
plt.show()
```

## Part (b)

This part is direct because can be calculated from the energy balance

\begin{equation}
Tf = To - \frac{\Delta H_r \, X}{CpA + CpB}
\end{equation}

Therefore

\begin{equation}
To = Tf + \frac{\Delta H_r \, X}{CpA + CpB}
\end{equation}

```python
Tf_max = 550
To_max = Tf_max + Dh * Xamax / (CpA + CpB)

print('The maximum inlet temperature is {:.2f} K'.format(To_max))
```

    The maximum inlet temperature is 380.00 K

## Part (c)

The heat that should be removed is the heat of reaction times fractional conversion

```python
heat_to_be_removed = -Dh * solODE.t
```

```python
fig, ax = plt.subplots(figsize=(6, 6))

ax.plot(solODE.y[0], heat_to_be_removed)
ax.set_xlabel(r'Reactor volume, dm$^3$')
ax.set_ylabel('Heat to be removed, cal/mol')

plt.tight_layout()
plt.show()
```

## Part (d)

This part requires to change the kinetic law and include the equilibrium constant into the system

The mole balance:

\begin{align}
&\frac{dx}{dV} = -\frac{r_A}{F_{Ao}} \\
&r_A = - k(T) \left(C_A \,C_B - \frac{C_C}{K_c(T)} \right) \\
&k(T) = k(300) \exp\left(\frac{-Ea}{R}\left( \frac{1}{T}-\frac{1}{300}\right)\right)\\
&Kc(T) = k(450) \exp\left(\frac{-\Delta H}{R}\left( \frac{1}{T}-\frac{1}{450}\right)\right)
\end{align}

with
\begin{align}
&C_A = C_{Ao} \, (1 - x) \\
&C_B = C_{Bo} - C_{Ao} \, x \\
&C_C = C_{Ao} \, x
\end{align}

```python
Vreactor = 10
Kc_450 = 10
```

```python
def system2(v, x):
    t = To - (Dh * x) / (CpA + CpB)
    k = k_300 * np.exp(-Ea/Rgas * (1/t - 1/To))
    Kc = Kc_450 * np.exp(-Dh/Rgas * (1/t - 1/450))

    Ca = Cao * (1-x)
    Cb = Ca
    Cc = Cao * x

    ra = -k * (Ca * Cb - Cc/ Kc)

    dx_dv = -ra / Fao
    return dx_dv

```

```python
vspan = [0, Vreactor]  #  independent variable range

solODE2 = solve_ivp(system2, vspan, ICs, t_eval=np.linspace(0, Vreactor))

Temp2 = To - (Dh * solODE2.y[0]) / (CpA + CpB)
```

Let plot both results for the same reactor size (10 dm^3)

```python
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(12, 7))

ax1.plot(solODE.y[0], solODE.t)
ax1.set_xlabel(r'Reactor volume, dm$^3$')
ax1.set_ylabel('Conversion, adim')
ax1.set_xlim([0,10])

ax2.plot(solODE2.t, solODE2.y[0], color='red')
ax2.set_xlabel(r'Reactor volume, dm$^3$')
ax2.set_ylabel('Conversion, adim')

ax3.plot(solODE.y[0], Temp)
ax3.set_xlabel(r'Reactor volume, dm$^3$')
ax3.set_ylabel('Reactor tempeture, K')
ax3.set_xlim([0,10])

ax4.plot(solODE2.t, Temp2, color='red')
ax4.set_xlabel(r'Reactor volume, dm$^3$')
ax4.set_ylabel('Reactor tempeture, K')

plt.tight_layout()
plt.show()
```

```python
print('The final conversion for the reversible scenario is {:.3f}'.format(solODE2.y[0][-1]))
```

    The final conversion for the reversible scenario is 0.005

```python
print('The final temperature for the reversible scenario is {:.2f} K'.format(Temp2[-1]))
```

    The final temperature for the reversible scenario is 301.17 K

At the equilibrium

\begin{equation}
C_A\,C_B = \frac{C_C}{K_C(T)}
\end{equation}

solving

\begin{equation}
C_{A0}\,Kc(T) - \left(2C_{A0}\,Kc(T)+1 \right)\,Xe + C_{A0}\,Kc(T) \,Xe^2= 0
\end{equation}

```python
def x_equilibrium(t):
    kc = Kc_450 * np.exp(-Dh/Rgas * (1/t - 1/450))
    a = Cao * kc
    b = - (2 * Cao * kc + 1)
    c = Cao * kc
    xe = (-b - np.sqrt(b**2 - 4 * a * c))/(2*a)
    return xe
```

```python
Xe = x_equilibrium(Temp2)
```

```python
fig, ax = plt.subplots(figsize=(6, 6))

ax.plot(solODE2.t, Xe)
ax.set_xlabel(r'Reactor volume, dm$^3$')
ax.set_ylabel('Equilibrium conversion, adim')

plt.tight_layout()
plt.show()
```

# CSTR

```python
def system_cstr(x):
    t = To - (Dh * x) / (CpA + CpB)
    k = k_300 * np.exp(-Ea/Rgas * (1/t - 1/To))

    ra = -k * Cao**2 * (1-x)**2

    v = (Fao * x ) / -ra
    return v
```

```python
Volume_CSTR = system_cstr(0.9)
```

```python
print('The required volume for CSTR is {:.2f} dm^3'.format(Volume_CSTR))
```

    The required volume for CSTR is 335.27 dm^3

```python

```


> Code to accompany **Chapter 10: Defending Against Adversarial Inputs**

# Fashion-MNIST - Dropout for Adversarial Detection

In this notebook, we'll create a classifier that uses drop-out during test as well as training time. This introduces
uncertainty to the network's predictions so each time an image is passed to the network, it will generated a slightly
different answer.

Adding uncertainty to test time may result in reduced accuracy. However, the point of a model of this type is to distinguish adversarial from non-adversarial. Therefore, it might run alongside another model during operations.

In theory, the uncertainty (variation) in the answers for a particular image will be greater when an image is not
within the training data set distribution. Refer to  __Chapter 10__ for a more detailed discussion.

## Import and get the data

```python
import tensorflow as tf
from tensorflow import keras
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images/255.0
test_images = test_images/255.0
```

## Create a Simple Network with drop-out for Image Classification

We need to use the Keras __functional API__ (rather than the sequential API) to access the
dropout capability with `training = True` at test time.

You can experiment by moving the drop-out layer or the level of dropout.

```python
from tensorflow.keras.layers import Input, Dense, Flatten, Dropout
from tensorflow.keras.models import Model

inputs = Input(shape=(28,28))
x = Flatten()(inputs)
x = Dense(56, activation='relu')(x)
x = Dropout(0.2)(x, training=True)
x = Dense(56, activation='relu')(x)
predictions = Dense(10, activation='softmax')(x)
model = Model(inputs=inputs, outputs=predictions)

model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.summary()
```

    WARNING:tensorflow:From C:\Users\katyw\Anaconda3\envs\strengthening-dnns\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    WARNING:tensorflow:From C:\Users\katyw\Anaconda3\envs\strengthening-dnns\lib\site-packages\tensorflow\python\keras\layers\core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_1 (InputLayer)         (None, 28, 28)            0
    _________________________________________________________________
    flatten (Flatten)            (None, 784)               0
    _________________________________________________________________
    dense (Dense)                (None, 56)                43960
    _________________________________________________________________
    dropout (Dropout)            (None, 56)                0
    _________________________________________________________________
    dense_1 (Dense)              (None, 56)                3192
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                570
    =================================================================
    Total params: 47,722
    Trainable params: 47,722
    Non-trainable params: 0
    _________________________________________________________________

That's the model architecture defined. Let's train it.

Note that more training epochs are used with drop-out; it takes longer to learn as the network needs to contend with the uncertainty.

```python
model.fit(train_images, train_labels, epochs=12)
```

    Epoch 1/12
    60000/60000 [==============================] - 4s 62us/sample - loss: 0.5811 - acc: 0.7922
    Epoch 2/12
    60000/60000 [==============================] - 4s 59us/sample - loss: 0.4296 - acc: 0.8439
    Epoch 3/12
    60000/60000 [==============================] - 4s 59us/sample - loss: 0.3992 - acc: 0.8540
    Epoch 4/12
    60000/60000 [==============================] - 4s 68us/sample - loss: 0.3797 - acc: 0.8607
    Epoch 5/12
    60000/60000 [==============================] - 4s 69us/sample - loss: 0.3610 - acc: 0.8678
    Epoch 6/12
    60000/60000 [==============================] - 4s 65us/sample - loss: 0.3531 - acc: 0.8694
    Epoch 7/12
    60000/60000 [==============================] - 4s 62us/sample - loss: 0.3440 - acc: 0.8724
    Epoch 8/12
    60000/60000 [==============================] - 4s 62us/sample - loss: 0.3361 - acc: 0.8747
    Epoch 9/12
    60000/60000 [==============================] - 4s 62us/sample - loss: 0.3268 - acc: 0.8784
    Epoch 10/12
    60000/60000 [==============================] - 4s 63us/sample - loss: 0.3224 - acc: 0.8794
    Epoch 11/12
    60000/60000 [==============================] - 4s 63us/sample - loss: 0.3182 - acc: 0.8803
    Epoch 12/12
    60000/60000 [==============================] - 4s 71us/sample - loss: 0.3094 - acc: 0.8856

    <tensorflow.python.keras.callbacks.History at 0x1e2b89cfb70>

Take a look at the model's accuracy.
This will change due to the uncertainty of the network each time you re-run this cell.

```python
test_loss, test_acc = model.evaluate(test_images, test_labels)
print('Model accuracy based on test data:', test_acc)
```

    10000/10000 [==============================] - 0s 37us/sample - loss: 0.4024 - acc: 0.8618
    Model accuracy based on test data: 0.8618

## Get some test data

We'll take two test datasets and check each for its level of uncertainty:

- x_images - original images (non-adversarial)
- x_images_adv - adversarial images

The hypothesis is that the network will create more uncertain results with the adversarial images than with the non-adversarial ones.

First, generate some adversarial test cases.

```python
num_images = 1000
```

```python
# Import helper function
import sys
sys.path.append('..')
from strengtheningdnns.adversarial_utils import generate_adversarial_data
```

```python
import foolbox

fmodel = foolbox.models.TensorFlowModel.from_keras(model, bounds=(0, 1))
attack_criterion = foolbox.criteria.Misclassification()
x_images = test_images[0:num_images, :]
attack_fn = foolbox.attacks.GradientSignAttack(fmodel,
criterion=attack_criterion,
distance=foolbox.distances.Linfinity)

(x_images_adv, _, _) = generate_adversarial_data(original_images = x_images,
                                                 predictions = model.predict(x_images),
                                                 attack_fn = attack_fn)

```

In case the total number of adversarial images couldn't be generated, recalculate the number of images.

```python
num_images = x_images_adv.shape[0]
```

Next, get the non-adversarial images (the original test data).

```python
import numpy as np
x_images = test_images[:num_images]
```

We've now got the two test datasets:

- x_images
- x_images_adv

Repeatedly call the drop-out model to get its predictions across the batches of images.
Each time it will return a different set of predictions.

We build up two prediction matrices:

- predictions matrix
- predictions_matrix_adv

```python
L = 100
num_classes = 10

predictions_matrix = np.zeros((L, num_images, num_classes))
predictions_matrix_adv = np.zeros((L, num_images, num_classes))

for i in range(L):
    predictions = model.predict(x_images)
    predictions_adv = model.predict(x_images_adv)
    predictions_matrix[i] = predictions
    predictions_matrix_adv[i] = predictions_adv

```

## Measuring the uncertainty of the responses

The following method calculates a value representing the level of uncertainty over a set of responses __for a single image__ returned from the network.

\begin{align}
U(x) = (\frac{1}{L}{\sum_{i=1}\|F_r(x)\|}) -
        \|\frac{1}{L}{\sum_{i=1}F_r(x)}\|
\end{align}

$$\text{where}\:\|y\| = \text{the squared}\: L_2\text{-norm}$$

Reference: https://arxiv.org/abs/1705.07263

```python
def uncertainty(predictions):
    return(np.sum(np.square(predictions))/predictions.shape[0]
           - np.sum(np.square(np.mean(predictions, axis=0))))
```

Calculate the uncertainty for the two image batches.

```python
uncertainty_results = np.zeros((num_images))
uncertainty_results_adv = np.zeros((num_images))

for i in range(num_images):
    uncertainty_results[i] = uncertainty(predictions_matrix[:,i])
    uncertainty_results_adv[i] = uncertainty(predictions_matrix_adv[:,i])
```

Plot the results

```python
import matplotlib.pyplot as plt

plt.hist((uncertainty_results, uncertainty_results_adv), bins=50, label=('Nonadversarial','Adversarial'))

plt.title("Model prediction uncertainty")
plt.xlabel("Uncertainty per image")
plt.ylabel("Number of images")
plt.legend(loc='right')

#plt.savefig("out.png", bbox_inches='tight', dpi=1000)

plt.show()
```

## Where next?
You can use this notebook to further experiment with randomized dropout, such
as changing the dropout parameters or altering the attack methods.

Assuming the attacker has complete knowledge of this defence, they might create their adversarial examples using the drop-out network. To see how the uncertainty of the network affects the ability to create adversarial examples, take a look at
[../chapter10/FashionMNIST_dropout_for_detection.ipynb]().

```python

```


# Estadística inferencial: Ajuste de Modelos

La inferencia busca extraer **conclusiones** a partir de **hechos u observaciones** a través de un **método o premisa**

En el caso particular de la **inferencia estadística** podemos realizar las siguientes asociaciones

- Hechos: Datos
- Premisa: Modelo probabilístico
- Conclusión: Una cantidad no observada que es interesante

Y lo que buscamos es

> Cuantificar la incerteza de la conclusión dado los datos y el modelo

**Niveles de la estadística inferencial**

La inferencia estadística puede dividirse en los siguientes tres niveles

1. Ajustar un modelo a nuestros datos
1. Verificar que el modelo sea confiable
1. Responder una pregunta usando el modelo

En lo que sigue estudiaremos las herramientas más utilizadas asociadas a cada uno de estos niveles

1. **Estimador de máxima verosimilitud**
1. **Bondad de ajuste** e **Intervalos de confianza**
1. **Test de hipótesis**

## Estimación de máxima verosimilitud

En este nivel de inferencia se busca **ajustar** un modelo teórico sobre nuestros datos. Nos enfocaremos en **modelos de tipo parámetrico**, es decir aquellos donde **se explicita una distribución de probabilidad**

Ajustar un modelo que se distribuye $\mathcal{N}(\mu, \sigma^2)$ a nuestros datos corresponde a

> encontrar los valores de $\mu$ y $\sigma$ tal que el modelo "se parezca" lo más posible a la distribución empírica de los datos

A continuación definiremos estas nociones

**¿Qué distribución ajustar?**

Antes de ajustar debemos suponer una distribución. En general podemos ajustar cualquier distribución pero un mal supuesto podría invalidar nuestra inferencia

:::{hint}

Podemos usar las herramientas de **estadística descriptiva** para estudiar nuestros datos y tomar esta decisión de manera informada

:::

**¿Cómo ajustar mi modelo?**

A continuación describiremos un procedimiento para ajustar modelos paramétricos llamado *maximum likelihood estimation* (MLE)

Sea un conjunto de datos $\{x_1, x_2, \ldots, x_N\}$. Antes de empezar haremos dos supuestos

- **Supuesto 1:** Los datos siguen el modelo $f(x;\theta)$ donde $f(\cdot)$ es una distribución y $\theta$ son sus parámetros
- **Supuesto 2:** Las datos son independientes e idénticamente distribuidos (iid)

Usando estos supuestos podemos desarrolar la distribución conjunta de los datos como como

$$
\begin{split}
f(x_1, x_2, \ldots, x_N |\theta) &= f(x_1|\theta) f(x_2|\theta) \ldots f(x_N|\theta)  \\
& = \prod_{i=1}^N f(x_i|\theta)  \\
& = \mathcal{L}(\theta)
\end{split}
$$

donde $\mathcal{L}(\theta)$ se conoce como la **verosimilitud** o probabilidad inversa de $\theta$

Si consideramos que los datos son fijos podemos buscar el valor de $\theta$ de máxima verosimilitud como

$$
\begin{align}
\hat \theta &= \text{arg} \max_\theta \mathcal{L}(\theta) \nonumber \\
&= \text{arg} \max_\theta \log \mathcal{L}(\theta) \nonumber \\
&= \text{arg} \max_\theta \sum_{i=1}^N \log f(x_i|\theta)
\end{align}
$$

:::{note}

El segundo paso es valido por que el argumento máximo de $g(x)$ y $\log(g(x))$ son equivalentes (el logaritmo es monoticamente creciente)

:::

Ahora sólo falta encontrar el máximo. Podemos hacerlo

- Analíticamente, derivando con respecto a $\theta$ e igualando a cero
- Usando técnicas de optimización iterativas como gradiente descedente

**Ejemplo analítico:**

Su profesor quiere medir su peso pero sospecha que su pesa está defectuosa. Para comprobarlo mide su peso $N$ veces obteniendo un conjunto de observaciones $\{z_i\}$. ¿Es posible obtener un estimador del peso real $\hat z$ a partir de estas observaciones?

Podemos modelar las observaciones como

$$
z_i = \hat z + \varepsilon_i
$$

donde $\varepsilon_i$ corresponde al ruido o error del instrumento y asumiremos que $\varepsilon_i \sim \mathcal{N}(0, \sigma_\varepsilon^2)$, es decir que el ruido es **independiente** y **Gaussiano** con media cero y **varianza** $\sigma_\varepsilon^2$ **conocida**

Entonces la distribución de $z_i$ es

$$
f(z_i|\hat z) = \mathcal{N}(\hat z, \sigma_\varepsilon^2)
$$

Para encontrar $\hat z$, primero escribimos el logaritmo de la **verosimilitud**

$$
\begin{align}
\log \mathcal{L}(\hat z) &=  \sum_{i=1}^N \log f(z_i|\hat z) \nonumber \\
&= \sum_{i=1}^N  \log \frac{1}{\sqrt{2\pi\sigma_\varepsilon^2}}  \exp \left ( - \frac{1}{2\sigma_\varepsilon^2} (z_i - \hat z)^2 \right)  \nonumber \\
&= -\frac{N}{2}\log(2\pi\sigma_\varepsilon^2)  - \frac{1}{2\sigma_\varepsilon^2}  \sum_{i=1}^N  (z_i - \hat z)^2  \nonumber
\end{align}
$$

Luego debemos resolver

$$
\begin{align}
\hat \theta &= \text{arg} \max_\theta \log \mathcal{L}(\theta) \nonumber \\
&= \text{arg} \max_\theta - \frac{1}{2\sigma_\varepsilon^2}  \sum_{i=1}^N  (z_i - \hat z)^2
\end{align}
$$

donde podemos ignorar el primer término de la verosimilitud ya que no depende de $\theta$

Para encontrar el máximo derivamos la expresión anterior e igualamos a cero

$$
-\frac{1}{2\sigma_\varepsilon^2} \sum_{i=1}^N 2(z_i - \hat z) = 0.
$$

Finalmente si despejamos llegamos a que

$$
\hat z = \frac{1}{N} \sum_{i=1}^N z_i,
$$

que se conoce como el estimador de máxima verosimilitud **para la media de una Gaussiana**

:::{note}

Podemos comprobar que es un máximo utilizando la segunda derivada

:::

A continuación veremos como hacer ajuste de parámetros con MLE para distintas distribuciones conocidas utilizando `scipy`

## Estimación MLE con `scipy`

El módulo [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html) provee de un gran número de distribuciones teóricas. Los objetos de tipo distribución comparten algunos métodos, entre ellos:

- `pdf`/`pmf(x)`: Retorna la función de densidad/masa de probabilidad evaluada en $x$
- `cdf(x)`: Retorna la función de densidad/masa acumulada evaluada en $x$
- `ppf(p)`: Retorna  el inverso de la distribución acumulada
- `rvs(size=N)`: Retorna $N$ muestras a partir de la distribución
- `fit(data)`: Retorna los parámetros de la distribución ajutados al arreglo `data`

:::{important}

El método `fit` realiza estimación de máxima verosimilitud para obtener los parámetros

:::

:::{note}

El método `fit` sólo está disponible para distribuciones univariadas (continuas o discretas)

:::

Para ejemplificar el uso de `fit` utilizaremos los siguientes datos

```python
import pickle
import numpy as np
import scipy.stats

with open("data/mistery_data.pkl", "rb") as f:
    mistery_data = pickle.load(f)
```

> Para ajustar un modelo parámetrico a estos datos primero creamos un objeto distribución y luego llamamos su método `fit`

Por ejemplo si ajustamos una distribución normal a `mistery_data` obtenemos:

```python
normal_params = scipy.stats.norm.fit(mistery_data)
normal_params
```

    (1.7598526388767328, 0.5437958926009934)

El resultado son los estimadores de máxima verosimilitud de la media y la desviación estándar de la distribución normal

Podemos visualizar el resultado del modelo ajustado utilizando `pdf`, sobre una objeto distribución creado a partir de los parámetros obtenidos

Observemos el resultado en conjunto a un histograma de `mistery_data`

```python
fitted_distribution = scipy.stats.norm(*normal_params)

x_eval = np.linspace(0, 4, num=100)
p_eval = fitted_distribution.pdf(x_eval)
```

```python
%matplotlib inline
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(7, 4), tight_layout=True)
ax.hist(mistery_data, density=True, bins=20, label='data', alpha=0.75)
ax.plot(x_eval, p_eval, lw=2, c='k', label='model')
ax.legend();
```

La visualización revela que los datos tienen una distribución asimétrica (cola derecha más larga)

> Ajustar con un modelo normal no fue una buena decisión

Intentemos nuevamente el ajuste pero con distribuciones de cola derecha larga:

```python
fig, ax = plt.subplots(figsize=(7, 4), tight_layout=True)
ax.hist(mistery_data, density=True, label='data', alpha=0.75, bins=20)

for dist in [scipy.stats.gamma,
             scipy.stats.lognorm,
             scipy.stats.gumbel_r]:
    mle_params = dist.fit(mistery_data)
    print(mle_params)
    ax.plot(x_eval, dist(*mle_params).pdf(x_eval), lw=2, label=dist.name)

ax.legend();
```

A simple vista, el resultado del ajuste es superior al caso normal

> ¿Cómo decidir cual modelo es superior?

Para eso profundizaremos en el segundo nivel de la estadística inferencial

## Verificación de modelos

Una vez que hemos ajustado un modelo es buena práctica verificar la calidad o bondad del ajuste. Esto se puede hacer gráficamente utilizando histogramas, gráficos probabilidad-probabilidad (pp plot) o [gráficos cuantil-cuantil](https://es.wikipedia.org/wiki/Gr%C3%A1fico_Q-Q) (qq plot)

:::{seealso}

Para crear qq-plots o pp-plots en Python sugiero la [librería statsmodels](https://www.statsmodels.org/dev/graphics.html)

:::

También es muy usual visualizar la CDF de nuestro modelo teórico contra la CDF empírica de los datos. Por ejemplo:

```python
def ECDF(data):
    data = np.sort(data)
    n = len(data)
    def result(x):
        return np.searchsorted(data, x, side='right')/n
    return result
```

```python
fig, ax = plt.subplots(figsize=(7, 4), tight_layout=True)

ax.plot(x_eval, ECDF(mistery_data)(x_eval), lw=2, ls='--', label='ECDF')

for dist in [scipy.stats.norm,
             scipy.stats.gamma]:
    mle_params = dist.fit(mistery_data)
    ax.plot(x_eval, dist(*mle_params).cdf(x_eval), lw=2,
            label=dist.name, alpha=1.)

ax.legend();
```

Visualmente, el modelo gamma sigue más de cerca a la ECDF que el modelo normal

**Test de Kolmogorov-Smirnov**

Existen pruebas como el test de Kolmogorov-Smirnov (KS) y el test de Anderson-Darling (AS) que miden la diferencia entre la CDF empírica y teórica

En particular el test de KS se basa en el siguiente estadístico de prueba

$$
D_n = \sup_x |F_n(x) - F(x)|,
$$

es decir la distancia absoluta más grande entre $F_n(x)$, la CDF empírica, y $F(x)$, una CDF teórica de referencia

:::{warning}

El test de KS sólo se puede usar para distribuciones univariadas y continuas

:::

:::{note}

El test de KS puede usarse para comparar dos muestras o para comparar una muestra con una distribución teórica

:::

El test de KS de una muestra está implementado en [`scipy.stats.kstest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html). Los principales argumentos de esta función son

- un ndarray con las observaciones de nuestra muestra
- un método `cdf` de una distribución continua de scipy.stats

La función retorna un objeto cuyo atributo `statistic` es equivalente a $D_n$

A continuación se calcula el valor de $D_n$ para cuatro distribuciones teóricas contra la CDF empírica de `mistery_data`:

```python
for dist in [scipy.stats.norm,
             scipy.stats.gamma,
             scipy.stats.lognorm,
             scipy.stats.gumbel_r]:

    mle_params = dist.fit(mistery_data)
    fitted_distribution = dist(*mle_params)
    result = scipy.stats.kstest(mistery_data, fitted_distribution.cdf)
    print(f"{dist.name:<10}: Dn = {result.statistic:0.4f}")

```

    norm      : Dn = 0.0981
    gamma     : Dn = 0.0538
    lognorm   : Dn = 0.0525
    gumbel_r  : Dn = 0.0493

La distribución de menor distancia es "gumbel_r", sin embargo

> ¿Es significativa la diferencia en distancia que estamos observando?

Para responder esta pregunta podemos realizar un **test de hipótesis**. Este es el tema principal de la lección siguiente

:::{warning}

`scipy.stats.kstest` también retorna un p-value asociado a la hipótesis nula de que las distribuciones que se están comparando son iguales. Este p-value sólo es válido si la CDF teórica no fue ajustada con los datos de la CDF empírica, por lo tanto lo hemos ignorado en este ejemplo.

:::

:::{seealso}

- Para calcular un p-value para el test de KS donde la distribución teórica fue ajustada utilizando MLE la opción es utilizar *bootstrap*. Lo veremos en una lección futura
- Para calcular el estadístico de Anderson-Darling (AD) sugiero la librería [statsmodels](https://www.statsmodels.org/stable/generated/statsmodels.stats.diagnostic.anderson_statistic.html). AD es más sensible que KS cuando la diferencia está concentrada en las colas de la distribución
- Para variables discretas se puede usar el [test $\chi^2$](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html) de bondad de ajuste

:::

```python

```