## Naive Bayes and Bayes Classifiers

author: Jacob Schreiber

contact: jmschreiber91@gmail.com

Bayes classifiers are some of the simplest machine learning models that exist, due to their intuitive probabilistic interpretation and simple fitting step. Each class is modeled as a probability distribution, and the data is interpreted as samples drawn from these underlying distributions. Fitting the model to data is as simple as calculating maximum likelihood parameters for the data that falls under each class, and making predictions is as simple as using Bayes' rule to determine which class is most likely given the distributions. Bayes' Rule is the following:

\begin{equation}
P(M|D) = \frac{P(D|M)P(M)}{P(D)}
\end{equation}

where M stands for the model and D stands for the data. $P(M)$ is known as the prior, because it is the probability that a sample is of a certain class before you even know what the sample is. This is generally just the frequency of each class. Intuitively, it makes sense that you would want to model this, because if one class occurs 10x more than another class, it is more likely that a given sample will belong to that distribution. $P(D|M)$ is the likelihood, or the probability, of the data under a given model. Lastly, $P(M|D)$ is the posterior, which is the probability of each component of the model, or class, being the component which generated the data. It is called the posterior because the prior corresponds to probabilities before seeing data, and the posterior corresponds to probabilities after observing the data. In cases where the prior is uniform, the posterior is just equal to the normalized likelihoods. This equation forms the basis of most probabilistic modeling, with interesting priors allowing the user to inject sophisticated expert knowledge into the problem directly.

pomegranate implements two distinct models of this format with the simpler being the naive Bayes classifier. The naive Bayes classifier assumes that each feature is independent from each other feature, and so breaks down $P(D|M)$ to be $\prod\limits_{i=1}^{d} P(D_{i}|M_{i})$ where $i$ is a specific feature in a data set that has $d$ features in it. This typically means faster calculations because covariance across features doesn't need to be considered, but it also is a natural way to model each feature as a different distribution because it ignores the complexities of modeling the covariance between, say, an exponential distribution and a normal distribution.

The more general model is the Bayes classifier which does not assume that each feature is independent of the others. This allows you to plug in anything for the likelihood function, whether it be a multivariate Gaussian distribution or a whole other compositional model. For instance, one could create a classifier whose components are each large mixture models. This enables much more complex models to be learned within the simple framework of Bayes' rule.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn; seaborn.set_style('whitegrid')
import numpy

from pomegranate import *

numpy.random.seed(0)
numpy.set_printoptions(suppress=True)

%load_ext watermark
%watermark -m -n -p numpy,scipy,pomegranate
```

    Fri Jan 10 2020

    numpy 1.18.1
    scipy 1.4.1
    pomegranate 0.12.0

    compiler   : Clang 10.0.0 (clang-1000.11.45.5)
    system     : Darwin
    release    : 17.7.0
    machine    : x86_64
    processor  : i386
    CPU cores  : 4
    interpreter: 64bit

### Simple Gaussian Example

```python
X = numpy.concatenate((numpy.random.normal(3, 1, 200), numpy.random.normal(10, 2, 1000)))
y = numpy.concatenate((numpy.zeros(200), numpy.ones(1000)))

x1 = X[:200]
x2 = X[200:]

plt.figure(figsize=(16, 5))
plt.hist(x1, bins=25, color='m', edgecolor='m', label="Class A")
plt.hist(x2, bins=25, color='c', edgecolor='c', label="Class B")
plt.xlabel("Value", fontsize=14)
plt.ylabel("Count", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

The data seems like it comes from two normal distributions, with the cyan class being more prevalent than the magenta class. A natural way to model this data would be to create a normal distribution for the cyan data, and another for the magenta distribution.

Let's take a look at doing that. All we need to do is use the `from_samples` class method of the `NormalDistribution` class.

```python
d1 = NormalDistribution.from_samples(x1)
d2 = NormalDistribution.from_samples(x2)
idxs = numpy.arange(0, 15, 0.1)

p1 = list(map(d1.probability, idxs))
p2 = list(map(d2.probability, idxs))

plt.figure(figsize=(16, 5))
plt.plot(idxs, p1, color='m'); plt.fill_between(idxs, 0, p1, facecolor='m', alpha=0.2)
plt.plot(idxs, p2, color='c'); plt.fill_between(idxs, 0, p2, facecolor='c', alpha=0.2)
plt.xlabel("Value", fontsize=14)
plt.ylabel("Probability", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

It looks like some aspects of the data are captured well by doing things this way-- specifically the mean and variance of the normal distributions. This allows us to easily calculate $P(D|M)$ as the probability of a sample under either the cyan or magenta distributions using the normal (or Gaussian) probability density equation:

\begin{align}
P(D|M) &= P(x|\mu, \sigma) \\
       &= \frac{1}{\sqrt{2\pi\sigma^{2}}} exp \left(-\frac{(x-u)^{2}}{2\sigma^{2}} \right)
\end{align}

However, if we look at the original data, we see that the cyan distributions is both much wider than the purple distribution and much taller, as there were more samples from that class in general. If we reduce that data down to these two distributions, we lose the class imbalance. We want our prior to model this class imbalance, with the reasoning being that if we randomly draw a sample from the samples observed thus far, it is far more likely to be a cyan than a magenta sample. Let's take a look at this class imbalance exactly.

```python
magenta_prior = 1. * len(x1) / len(X)
cyan_prior = 1. * len(x2) / len(X)

plt.figure(figsize=(4, 6))
plt.title("Prior Probabilities P(M)", fontsize=14)
plt.bar(0, magenta_prior, facecolor='m', edgecolor='m')
plt.bar(1, cyan_prior, facecolor='c', edgecolor='c')
plt.xticks([0, 1], ['P(Magenta)', 'P(Cyan)'], fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

The prior $P(M)$ is a vector of probabilities over the classes that the model can predict, also known as components. In this case, if we draw a sample randomly from the data that we have, there is a ~83% chance that it will come from the cyan class and a ~17% chance that it will come from the magenta class.

Let's multiply the probability densities we got before by this imbalance.

```python
d1 = NormalDistribution.from_samples(x1)
d2 = NormalDistribution.from_samples(x2)
idxs = numpy.arange(0, 15, 0.1)

p_magenta = numpy.array(list(map(d1.probability, idxs))) * magenta_prior
p_cyan = numpy.array(list(map(d2.probability, idxs))) * cyan_prior

plt.figure(figsize=(16, 5))
plt.plot(idxs, p_magenta, color='m'); plt.fill_between(idxs, 0, p_magenta, facecolor='m', alpha=0.2)
plt.plot(idxs, p_cyan, color='c'); plt.fill_between(idxs, 0, p_cyan, facecolor='c', alpha=0.2)
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M)P(D|M)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

This looks a lot more faithful to the original data, and actually corresponds to $P(M)P(D|M)$, the prior multiplied by the likelihood. However, these aren't actually probability distributions anymore, as they no longer integrate to 1. This is why the $P(M)P(D|M)$ term has to be normalized by the $P(D)$ term in Bayes' rule in order to get a probability distribution over the components. However, $P(D)$ is difficult to determine exactly-- what is the probability of the data? Well, we can sum over the classes to get that value, since $P(D) = \sum_{i=1}^{c} P(D|M)P(M)$ for a problem with c classes. This translates into $P(D) = P(M=Cyan)P(D|M=Cyan) + P(M=Magenta)P(D|M=Magenta)$ for this specific problem, and those values can just be pulled from the unnormalized plots above.

This gives us the full Bayes' rule, with the posterior $P(M|D)$ being the proportion of density of the above plot coming from each of the two distributions at any point on the line. Let's take a look at the posterior probabilities of the two classes on the same line.

```python
magenta_posterior = p_magenta / (p_magenta + p_cyan)
cyan_posterior = p_cyan / (p_magenta + p_cyan)

plt.figure(figsize=(16, 5))
plt.subplot(211)
plt.plot(idxs, p_magenta, color='m'); plt.fill_between(idxs, 0, p_magenta, facecolor='m', alpha=0.2)
plt.plot(idxs, p_cyan, color='c'); plt.fill_between(idxs, 0, p_cyan, facecolor='c', alpha=0.2)
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M)P(D|M)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)

plt.subplot(212)
plt.plot(idxs, magenta_posterior, color='m')
plt.plot(idxs, cyan_posterior, color='c')
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M|D)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

The top plot shows the same densities as before, while the bottom plot shows the proportion of the density belonging to either class at that point. This proportion is known as the posterior $P(M|D)$, and can be interpreted as the probability of that point belonging to each class. This is one of the native benefits of probabilistic models, that instead of providing a hard class label for each sample, they can provide a soft label in the form of the probability of belonging to each class.

We can implement all of this simply in pomegranate using the `NaiveBayes` class.

```python
idxs = idxs.reshape(idxs.shape[0], 1)
X = X.reshape(X.shape[0], 1)

model = NaiveBayes.from_samples(NormalDistribution, X, y)
posteriors = model.predict_proba(idxs)

plt.figure(figsize=(14, 4))
plt.plot(idxs, posteriors[:,0], color='m')
plt.plot(idxs, posteriors[:,1], color='c')
plt.xlabel("Value", fontsize=14)
plt.ylabel("P(M|D)", fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

Looks like we're getting the same plots for the posteriors just through fitting the naive Bayes model directly to data. The predictions made will come directly from the posteriors in this plot, with cyan predictions happening whenever the cyan posterior is greater than the magenta posterior, and vice-versa.

## Naive Bayes

In the univariate setting, naive Bayes is identical to a general Bayes classifier. The divergence occurs in the multivariate setting, the naive Bayes model assumes independence of all features, while a Bayes classifier is more general and can support more complicated interactions or covariances between features. Let's take a look at what this means in terms of Bayes' rule.

\begin{align}
P(M|D) &= \frac{P(M)P(D|M)}{P(D)} \\
       &= \frac{P(M)\prod_{i=1}^{d}P(D_{i}|M_{i})}{P(D)}
\end{align}

This looks fairly simple to compute, as we just need to pass each dimension into the appropriate distribution and then multiply the returned probabilities together. This simplicity is one of the reasons why naive Bayes is so widely used. Let's look closer at using this in pomegranate, starting off by generating two blobs of data that overlap a bit and inspecting them.

```python
X = numpy.concatenate([numpy.random.normal(3, 2, size=(150, 2)), numpy.random.normal(7, 1, size=(250, 2))])
y = numpy.concatenate([numpy.zeros(150), numpy.ones(250)])

plt.figure(figsize=(8, 8))
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c')
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m')
plt.xlim(-2, 10)
plt.ylim(-4, 12)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

Now, let's fit our naive Bayes model to this data using pomegranate. We can use the `from_samples` class method, pass in the distribution that we want to model each dimension, and then the data. We choose to use `NormalDistribution` in this particular case, but any supported distribution would work equally well, such as `BernoulliDistribution` or `ExponentialDistribution`. To ensure we get the correct decision boundary, let's also plot the boundary recovered by sklearn.

```python
from sklearn.naive_bayes import GaussianNB

model = NaiveBayes.from_samples(NormalDistribution, X, y)
clf = GaussianNB().fit(X, y)

xx, yy = numpy.meshgrid(numpy.arange(-2, 10, 0.02), numpy.arange(-4, 12, 0.02))
Z1 = model.predict(numpy.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
Z2 = clf.predict(numpy.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

plt.figure(figsize=(16, 8))
plt.subplot(121)
plt.title("pomegranate naive Bayes", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c')
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m')
plt.contour(xx, yy, Z1, cmap='Blues')
plt.xlim(-2, 10)
plt.ylim(-4, 12)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)

plt.subplot(122)
plt.title("sklearn naive Bayes", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c')
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m')
plt.contour(xx, yy, Z2, cmap='Blues')
plt.xlim(-2, 10)
plt.ylim(-4, 12)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.show()
```

Drawing the decision boundary helps to verify that we've produced a good result by cleanly splitting the two blobs from each other.

Bayes' rule provides a great deal of flexibility in terms of what the actually likelihood functions are. For example, when considering a multivariate distribution, there is no need for each dimension to be modeled by the same distribution. In fact, each dimension can be modeled by a different distribution, as long as we can multiply the $P(D|M)$ terms together.

### Classifying Signal with Different Distributions on Different Features

Let's consider the example of some noisy signals that have been segmented. We know that they come from two underlying phenomena, the cyan phenomena and the magenta phenomena, and want to classify future segments. To do this, we have three features-- the mean signal of the segment, the standard deviation, and the duration.

```python
def plot_signal(X, n):
    plt.figure(figsize=(16, 6))
    t_current = 0
    for i in range(n):
        mu, std, t = X[i]
        chunk = numpy.random.normal(mu, std, int(t))
        plt.plot(numpy.arange(t_current, t_current+t), chunk, c='cm'[i % 2])
        t_current += t

    plt.xticks(fontsize=14)
    plt.yticks(fontsize=14)
    plt.xlabel("Time (s)", fontsize=14)
    plt.ylabel("Signal", fontsize=14)
    plt.ylim(20, 40)
    plt.show()

def create_signal(n):
    X, y = [], []
    for i in range(n):
        mu = numpy.random.normal(30.0, 0.4)
        std = numpy.random.lognormal(-0.1, 0.4)
        t = int(numpy.random.exponential(50)) + 1
        X.append([mu, std, int(t)])
        y.append(0)

        mu = numpy.random.normal(30.5, 0.8)
        std = numpy.random.lognormal(-0.3, 0.6)
        t = int(numpy.random.exponential(200)) + 1
        X.append([mu, std, int(t)])
        y.append(1)

    return numpy.array(X), numpy.array(y)

X_train, y_train = create_signal(1000)
X_test, y_test = create_signal(250)
plot_signal(X_train, 20)
```

We can start by modeling each variable as Gaussians, like before, and see what accuracy we get.

```python
model = NaiveBayes.from_samples(NormalDistribution, X_train, y_train)
print("Gaussian Naive Bayes: ", (model.predict(X_test) == y_test).mean())

clf = GaussianNB().fit(X_train, y_train)
print("sklearn Gaussian Naive Bayes: ", (clf.predict(X_test) == y_test).mean())
```

    Gaussian Naive Bayes:  0.794
    sklearn Gaussian Naive Bayes:  0.794

We get identical values for sklearn and for pomegranate, which is good. However, let's take a look at the data itself to see whether a Gaussian distribution is the appropriate distribution for the data.

```python
plt.figure(figsize=(14, 4))
plt.subplot(131)
plt.title("Mean")
plt.hist(X_train[y_train == 0, 0], color='c', alpha=0.5, bins=25)
plt.hist(X_train[y_train == 1, 0], color='m', alpha=0.5, bins=25)

plt.subplot(132)
plt.title("Standard Deviation")
plt.hist(X_train[y_train == 0, 1], color='c', alpha=0.5, bins=25)
plt.hist(X_train[y_train == 1, 1], color='m', alpha=0.5, bins=25)

plt.subplot(133)
plt.title("Duration")
plt.hist(X_train[y_train == 0, 2], color='c', alpha=0.5, bins=25)
plt.hist(X_train[y_train == 1, 2], color='m', alpha=0.5, bins=25)
plt.show()
```

So, unsurprisingly (since you can see that I used non-Gaussian distributions to generate the data originally), it looks like only the mean follows a normal distribution, whereas the standard deviation seems to follow either a gamma or a log-normal distribution. We can take advantage of that by explicitly using these distributions instead of approximating them as normal distributions. pomegranate is flexible enough to allow for this, whereas sklearn currently is not.

```python
model = NaiveBayes.from_samples(NormalDistribution, X_train, y_train)
print("Gaussian Naive Bayes: ", (model.predict(X_test) == y_test).mean())

clf = GaussianNB().fit(X_train, y_train)
print("sklearn Gaussian Naive Bayes: ", (clf.predict(X_test) == y_test).mean())

model = NaiveBayes.from_samples([NormalDistribution, LogNormalDistribution, ExponentialDistribution], X_train, y_train)
print("Heterogeneous Naive Bayes: ", (model.predict(X_test) == y_test).mean())
```

    Gaussian Naive Bayes:  0.794
    sklearn Gaussian Naive Bayes:  0.794
    Heterogeneous Naive Bayes:  0.822

It looks like we're able to get a small improvement in accuracy just by using appropriate distributions for the features, without any type of data transformation or filtering. This certainly seems worthwhile if you can determine what the appropriate underlying distribution is.

Next, there's obviously the issue of speed. Let's compare the speed of the pomegranate implementation and the sklearn implementation.

```python
%timeit GaussianNB().fit(X_train, y_train)
%timeit NaiveBayes.from_samples(NormalDistribution, X_train, y_train)
%timeit NaiveBayes.from_samples([NormalDistribution, LogNormalDistribution, ExponentialDistribution], X_train, y_train)
```

    710 µs ± 15.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    547 µs ± 3.91 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    568 µs ± 5.77 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Looks as if on this small dataset they're all taking approximately the same time. This is pretty much expected, as the fitting step is fairly simple and both implementations use C-level numerics for the calculations. We can give a more thorough treatment of the speed comparison on larger datasets. Let's look at the average time it takes to fit a model to data of increasing dimensionality across 25 runs.

```python
pom_time, skl_time = [], []

n1, n2 = 15000, 60000,
for d in range(1, 101, 5):
    X = numpy.concatenate([numpy.random.normal(3, 2, size=(n1, d)), numpy.random.normal(7, 1, size=(n2, d))])
    y = numpy.concatenate([numpy.zeros(n1), numpy.ones(n2)])

    tic = time.time()
    for i in range(25):
        GaussianNB().fit(X, y)
    skl_time.append((time.time() - tic) / 25)

    tic = time.time()
    for i in range(25):
        NaiveBayes.from_samples(NormalDistribution, X, y)
    pom_time.append((time.time() - tic) / 25)
```

```python
plt.figure(figsize=(14, 6))
plt.plot(range(1, 101, 5), pom_time, color='c', label="pomegranate")
plt.plot(range(1, 101, 5), skl_time, color='m', label="sklearn")
plt.xticks(fontsize=14)
plt.xlabel("Number of Dimensions", fontsize=14)
plt.yticks(fontsize=14)
plt.ylabel("Time (s)")
plt.legend(fontsize=14)
plt.show()
```

It appears as if the two implementations are basically the same speed. This is unsurprising given the simplicity of the calculations, and as mentioned before, the low level implementation.

### Multivariate Gaussian Bayes Classifiers

The natural generalization of the naive Bayes classifier is to allow any multivariate function take the place of $P(D|M)$ instead of it being the product of several univariate probability distributions. One immediate difference is that now instead of creating a Gaussian model with effectively a diagonal covariance matrix, you can now create one with a full covariance matrix. Let's see an example of that at work.

```python
tilt_a = [[-2, 0.5], [5, 2]]
tilt_b = [[-1, 1.5], [3, 3]]

X = numpy.concatenate((numpy.random.normal(4, 1, size=(250, 2)).dot(tilt_a), numpy.random.normal(3, 1, size=(800, 2)).dot(tilt_b)))
y = numpy.concatenate((numpy.zeros(250), numpy.ones(800)))

model_a = NaiveBayes.from_samples(NormalDistribution, X, y)
model_b = BayesClassifier.from_samples(MultivariateGaussianDistribution, X, y)

xx, yy = numpy.meshgrid(numpy.arange(-5, 30, 0.02), numpy.arange(0, 25, 0.02))
Z1 = model_a.predict(numpy.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
Z2 = model_b.predict(numpy.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

plt.figure(figsize=(18, 8))
plt.subplot(121)
plt.contour(xx, yy, Z1, cmap='Blues')
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.xlim(-5, 30)
plt.ylim(0, 25)

plt.subplot(122)
plt.contour(xx, yy, Z2, cmap='Blues')
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.xlim(-5, 30)
plt.ylim(0, 25)
plt.show()
```

It looks like we are able to get a better boundary between the two blobs of data. The primary for this is because the data don't form spherical clusters, like you assume when you force a diagonal covariance matrix, but are tilted ellipsoids, that can be better modeled by a full covariance matrix. We can quantify this quickly by looking at performance on the training data.

```python
print("naive training accuracy: {:4.4}".format((model_a.predict(X) == y).mean()))
print("bayes classifier training accuracy: {:4.4}".format((model_b.predict(X) == y).mean()))
```

    naive training accuracy: 0.9219
    bayes classifier training accuracy: 0.9695

Looks like there is a significant boost. Naturally you'd want to evaluate the performance of the model on separate validation data, but for the purposes of demonstrating the effect of a full covariance matrix this should be sufficient.

### Gaussian Mixture Model Bayes Classifier

While using a full covariance matrix is certainly more complicated than using only the diagonal, there is no reason that the $P(D|M)$ has to even be a single simple distribution versus a full probabilistic model. After all, all probabilistic models, including general mixtures, hidden Markov models, and Bayesian networks, can calculate $P(D|M)$. Let's take a look at an example of using a mixture model instead of a single gaussian distribution.

```python
X = numpy.empty(shape=(0, 2))
X = numpy.concatenate((X, numpy.random.normal(4, 1, size=(200, 2)).dot([[-2, 0.5], [2, 0.5]])))
X = numpy.concatenate((X, numpy.random.normal(3, 1, size=(350, 2)).dot([[-1, 2], [1, 0.8]])))
X = numpy.concatenate((X, numpy.random.normal(7, 1, size=(700, 2)).dot([[-0.75, 0.8], [0.9, 1.5]])))
X = numpy.concatenate((X, numpy.random.normal(6, 1, size=(120, 2)).dot([[-1.5, 1.2], [0.6, 1.2]])))
y = numpy.concatenate((numpy.zeros(550), numpy.ones(820)))

model_a = BayesClassifier.from_samples(MultivariateGaussianDistribution, X, y)

gmm_a = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 2, X[y == 0])
gmm_b = GeneralMixtureModel.from_samples(MultivariateGaussianDistribution, 2, X[y == 1])
model_b = BayesClassifier([gmm_a, gmm_b], weights=numpy.array([1-y.mean(), y.mean()]))

xx, yy = numpy.meshgrid(numpy.arange(-10, 10, 0.02), numpy.arange(0, 25, 0.02))
Z1 = model_a.predict(numpy.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
Z2 = model_b.predict(numpy.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
centroids1 = numpy.array([distribution.mu for distribution in model_a.distributions])
centroids2 = numpy.concatenate([[distribution.mu for distribution in component.distributions] for component in model_b.distributions])

plt.figure(figsize=(18, 8))
plt.subplot(121)
plt.contour(xx, yy, Z1, cmap='Blues')
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.scatter(centroids1[:,0], centroids1[:,1], color='k', s=100)

plt.subplot(122)
plt.contour(xx, yy, Z2, cmap='Blues')
plt.scatter(X[y == 0, 0], X[y == 0, 1], color='c', alpha=0.3)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='m', alpha=0.3)
plt.scatter(centroids2[:,0], centroids2[:,1], color='k', s=100)
plt.show()
```

Frequently in the real world you will end up with data that looks like the ones in the above plots, not neatly falling into any single simple distribution. Using a mixture here allowed us to model the various components separately and get a more sophisticated decision boundary that doesn't seem to be extremely overfit.

If one wanted to use hidden Markov models to model sequences, all that needs to change is passing in `HiddenMarkovModel` objects instead of `GeneralMixtureModel` ones. Likewise, Bayesian networks are similarly supported and can be passed in just as easily. In fact, the most stacking one could reasonably do is to create a Bayes classifier that distinguishes mixtures of hidden Markov models with mixture emissions (GMM-HMM-GMMs) from each other!


# Lecture 8: Overview of different performance metrics

## 1. Review

### 1.1 Bipolar model

**Definition:** Let  $\tilde{\Phi} = \{ (x_i,e_i, y_i,h_i)\}$ represent the marked Poisson point process such that:

(1) The ground process is $\Phi = \{ x_i\}$ where $x_i$ denotes the location of the potential transmitters with density $\lambda>0$ in TXs/m$^2$.

(2) $\{e_i\}$ is the medium access indicator: node $i$ is allowed to transmit if $e_i=1$ and it is not allowed if $e_i=0$. For slotted ALOHA protocol. $e_i$ is a random variable independent and identically distributed such that $\mathbb{P}(e_i=1) = p$ and  $\mathbb{P}(e_i=0) = 1-p$.

(3) $\{y_i\}$ denotes the location of the receiver associated with the transmitter $x_i$ to form a communication pair. We assume that the random vectors ${x_i - y_i}$ are independent and identically distributed such that  $|x_i - y_i| = r$. Here we consider that $r$ is constant.

(4)  $\{h_i\}$ represents the channel gain due to fading in relation to the typical link receiver, located at the origin. We assume that $\{h_i\}$  is also an independent and identically distributed random variables.

```python
#################
#Illustration####
#################

from matplotlib import pyplot as plt
import numpy as np

#########################
###(1) Ground process###
########################
#Square side: L2
L2 = 10
#Average number of points (Poisson)
x_average_PPP = 20
#Number of points N that is a Poisson random variable
x_realization_PPP = np.random.poisson(x_average_PPP,1)
#Random points in axis x
position_x_realization_PPP_TX =  np.random.uniform(0,L2,x_realization_PPP)
#Random points in axis y
position_y_realization_PPP_TX = np.random.uniform(0,L2,x_realization_PPP)

fig, axes = plt.subplots(1, 3, figsize=(19,6), dpi=1200)

#Plot commands
axes[0].plot(position_x_realization_PPP_TX, position_y_realization_PPP_TX, marker='o', color='k',linestyle = '')
#plt.axis('off')

#####################
###(2) ALOHA #######
####################

#aloha probability
p=0.7

#declaring a list of unknown size
position_x_realization_PPP_TX_e_1 = []
position_y_realization_PPP_TX_e_1 = []
position_x_realization_PPP_TX_e_0 = []
position_y_realization_PPP_TX_e_0 = []

for xx in range(0,x_realization_PPP[0]):
    if np.random.random() < p:
        #add element to the list
        position_x_realization_PPP_TX_e_1.append(position_x_realization_PPP_TX[xx])
        position_y_realization_PPP_TX_e_1.append(position_y_realization_PPP_TX[xx])
    else:
        #add element to the list
        position_x_realization_PPP_TX_e_0.append(position_x_realization_PPP_TX[xx])
        position_y_realization_PPP_TX_e_0.append(position_y_realization_PPP_TX[xx])
    xx+=1

#converting the list to a np.array
position_x_realization_PPP_TX_e_1 = np.array(position_x_realization_PPP_TX_e_1)
position_y_realization_PPP_TX_e_1 = np.array(position_y_realization_PPP_TX_e_1 )
position_x_realization_PPP_TX_e_0 = np.array(position_x_realization_PPP_TX_e_0)
position_y_realization_PPP_TX_e_0 = np.array(position_y_realization_PPP_TX_e_0)

axes[1].plot(position_x_realization_PPP_TX_e_1, position_y_realization_PPP_TX_e_1, marker='o', color='b',linestyle = '')
axes[1].plot(position_x_realization_PPP_TX_e_0, position_y_realization_PPP_TX_e_0, marker='o', color='r',linestyle = '')

#####################
###(3) Receivers####
####################

#distance between TX-RX (fixed)
dist_TX_RX = 0.75

#angles of the receivers
angle_RX = np.random.uniform(0,2*np.pi,x_realization_PPP)
#setting arrays
position_x_realization_PPP_RX = np.zeros(x_realization_PPP)
position_y_realization_PPP_RX = np.zeros(x_realization_PPP)

#RXs position
for xx in range(0,x_realization_PPP[0]):
    position_x_realization_PPP_RX[xx] = position_x_realization_PPP_TX[xx] + dist_TX_RX*np.cos(angle_RX[xx])
    position_y_realization_PPP_RX[xx] = position_y_realization_PPP_TX[xx] + dist_TX_RX*np.sin(angle_RX[xx])
    xx+=1

axes[2].plot(position_x_realization_PPP_RX, position_y_realization_PPP_RX, marker='x', color='k',linestyle = '')
axes[2].plot(position_x_realization_PPP_TX_e_1, position_y_realization_PPP_TX_e_1, marker='o', color='b',linestyle = '')
axes[2].plot(position_x_realization_PPP_TX_e_0, position_y_realization_PPP_TX_e_0, marker='o', color='r',linestyle = '')
#plot lines between links - bipoles
axes[2].plot([position_x_realization_PPP_TX, position_x_realization_PPP_RX],
         [position_y_realization_PPP_TX, position_y_realization_PPP_RX],  color='k',linestyle = '-')
plt.show()
```

### 1.2 Performance analysis

**Scenario description**

- Bipolar wireless network $\tilde{\Phi} = \{ (x_i,e_i, y_i, h_i)\}$ such that the ground process is a two dimensional homogeneous Poisson point process with intensity $\lambda$, the transmitters access the network with probability $p$ (slotted ALOHA), the receivers are located at a distance $r$ far from its respective transmitter and $h_i$ is related to Rayleigh fading with mean 1.

- The distance-dependent path-loss function is given by $\ell(x) =  x^{-\alpha}$ where $\alpha>2$ is the path-loss exponent.

- The network is assumed *interference-limited* so that the noise effect is neglected.

- The receivers employ the decoding rule *treat-interference-as-noise* where the message can be successfully decoded if $\textrm{SIR} > \beta$.

### 1.3 Analytical results

[Step 1] The point process $\tilde{\Phi}$ is stationary and then it is worth to analyze a typical link, which will be referred by using the index $0$. The point process is then $\tilde{\Phi} \cup \{x_0,e_0,y_0,h_0\}$. The receiver of the typical link will be located at the origin - i.e. $y_0 = (0,0)$.

[Step 2] Find the process of active links as thinning of  $\tilde{\Phi}$  keeping only the points (transmitters) that $e_i = 1$. Let us denote the thinned marked process $\tilde{\Phi}_1$ whose density is $p \lambda$.

[Step 3] Compute the probability $p_\mathrm{s}$ that a message is successfully decoded a typical receiver (i.e $\textrm{SIR}_0 > \beta$). From the analysis done in the previous lecture:

 $\displaystyle p_\mathrm{s} = e^{-p \lambda \pi r^2 \beta^{2/\alpha} \kappa}$,

where $\displaystyle \kappa = \Gamma(1 + 2/\alpha)\Gamma(1-2/\alpha)$.

```python
#Testing
import numpy as np
from scipy.spatial import distance
from __future__ import division
import statsmodels.api as sm # recommended import according to the docs
import math
#################
##Inputs########
###############

#pathloss
alpha=4
#SIR required
beta= 2
#aloha prob
p=0.9
#Average number of points (Poisson)
x_average_PPP = p*15
#Square side: L
L = 20
#number of realizations to get the outage prob for 1 point
num_realizations = 2000
#number of simulation points
num_of_point_for_outage = 50
#
outage_prob = np.zeros(num_of_point_for_outage)
r = np.linspace(0,4,num_of_point_for_outage)

for count2 in range(1, num_of_point_for_outage):
    outage = 0
    for count in range(0, num_realizations):
        #Number of points N that is a Poisson random variable
        x_realization_PPP = np.random.poisson(x_average_PPP,1)
        #Random points in axis x
        position_x_realization_PPP =  np.random.uniform(0,L,x_realization_PPP)
        #Random points in axis y
        position_y_realization_PPP = np.random.uniform(0,L,x_realization_PPP)
        #Manipulating arrays
        position_x_realization_PPP_t=np.transpose(position_x_realization_PPP)
        position_y_realization_PPP_t=np.transpose(position_y_realization_PPP)
        position_final = []
        position_final = [[position_x_realization_PPP_t[ix], position_y_realization_PPP_t[ix]] for ix in range(0, x_realization_PPP)]
        #Distances to the origin (L/2,L/2)
        distances_to_origin = [distance.euclidean(position_final[ix],[L/2,L/2]) for ix in range(0, x_realization_PPP)]
        #getting the power of each interferer
        path_loss_fading = np.array([np.random.exponential(1) * distances_to_origin[xx]**(-alpha) for xx in range(0, x_realization_PPP)])
        #computing aggregate interference
        interference = sum(path_loss_fading)
        if (np.random.exponential(1) * r[count2]**(-alpha)/interference) <= beta:
                outage = outage + 1
        count = count + 1
    outage_prob[count2] = outage/count
    count2 = count2 + 1

###Plot

k = math.gamma(1 + 2/alpha) * math.gamma(1- 2/alpha)
plt.figure(figsize=(10,8), dpi=1200)
plt.plot(r,outage_prob,'o k',label = "Simulation")
x = np.linspace(0,4,100)
plt.xlabel('$r$', fontsize=16)
plt.ylabel('Outage probability', fontsize=16)
plt.plot(x, 1 -  np.exp(- (x**2) * k * beta**(2/alpha) * np.pi * p * x_average_PPP/(L**2)),'k-' ,label = "Analysis")
plt.axis([0,4,0,1])
plt.grid(color='k', alpha=0.5, linestyle='dashed', linewidth=0.5)
plt.legend(loc=2,fontsize=16)
plt.show()
```

## 2. Performance analysis

### 2.1 Expected progress in [m]

 $\displaystyle \mathrm{EP} = r \times (1- P_\mathrm{out}) = r \; e^{-p \lambda \pi r^2 \beta^{2/\alpha} \kappa}$

```python
#Expected progress
#
alpha = 3
beta = 3
density = 0.05
p = 0.7
k = math.gamma(1 + 2/alpha) * math.gamma(1- 2/alpha)
r_opt = (2/(np.pi * k * density * p))**(1/2)/(2 * beta**(1/alpha))

plt.figure(figsize=(10,8), dpi=1200)
r = np.linspace(0,4,1000)
plt.xlabel('$r$', fontsize=16)
plt.ylabel('Expected progress', fontsize=16)
plt.plot(r, r * np.exp(- (r**2) * beta**(2/alpha) * np.pi * p * k * density),'k-')
plt.plot(r_opt, r_opt * np.exp(- (r_opt**2) * beta**(2/alpha) * np.pi * p * k * density),'k o')
plt.grid(color='k', alpha=0.5, linestyle='dashed', linewidth=0.5)
plt.text(1.2, 0.57, 'Optimal point', fontsize=14)
plt.show()
```

*Exercise:* Find the optimal point

```python
# Using sympy to solve analytically the optimization problem
from sympy import Symbol, symbols, init_printing, oo
from sympy import simplify
from sympy import *

# From future
from __future__ import division

# Setting things up
init_printing(use_unicode = False, wrap_line = False, no_global = True)
```

```python
#Defining variables
r2, p2, lamb2, beta2, alpha2, kappa = symbols('r p lambda beta alpha kappa', real = True, positive = True)
```

```python
#Success probability
p_s= exp(- p2 * kappa* lamb2 * pi * r2**2 * beta2**(2/alpha2))

#EP
EP = r2 * p_s

EP
```

```python
#Best r
simplify(solve(diff(EP,r2),r2))
```

```python
#Verifying
alpha = 3
beta = 3
density = 0.05
p = 0.7
k = math.gamma(1 + 2/alpha) * math.gamma(1- 2/alpha)
(2/(np.pi * k * density * p))**(1/2)/(2 * beta**(1/alpha))
```

```python
r_opt * np.exp(- (r_opt**2) * beta**(2/alpha) * np.pi * p * k * density)
```

### 2.2 Information efficiency in [bit-m/s/Hz]

 $\displaystyle \mathrm{IE} = r \times \log(1+\beta) \times (1- P_\mathrm{out})  = r \; \log(1+\beta) \; e^{-p \lambda \pi r^2 \beta^{2/\alpha} \kappa}$

```python
#IE
#
alpha = 4
density = 0.05
r1 = 2.5
r2 = 3.5
r3 = 5
p = 1
k = math.gamma(1 + 2/alpha) * math.gamma(1- 2/alpha)

plt.figure(figsize=(10,8), dpi=1200)
beta = np.linspace(0,4,300)
plt.xlabel(r'$\beta$', fontsize=16)
plt.ylabel('Information efficiency', fontsize=16)
plt.plot(beta, r1 * (np.log(1+beta)/np.log(2)) * np.exp(- (r1**2) * k * beta**(2/alpha) * np.pi * p * density),'k-',label = "$r=2.5$")
plt.plot(beta, r2 * (np.log(1+beta)/np.log(2)) * np.exp(- (r2**2) * k * beta**(2/alpha) * np.pi * p * density),'g-',label = "$r=3.5$")
plt.plot(beta, r3 * (np.log(1+beta)/np.log(2)) * np.exp(- (r3**2) * k * beta**(2/alpha) * np.pi * p * density),'b-',label = "$r=5.0$")
#plt.axis([0,4,0,1])
plt.legend(loc=1,fontsize=16)
plt.grid(color='k', alpha=0.5, linestyle='dashed', linewidth=0.5)
plt.show()
```

### 2.3 Spatial throughput (or area spectral efficiency or density of throughput) in [bit/s/Hz/m$^2$]

 $\displaystyle \mathrm{ST} = p \times \lambda  \times (1- P_\mathrm{out}) \times \log(1+\beta)  = p \lambda \; \log(1+\beta) \; e^{-p \lambda \pi r^2 \beta^{2/\alpha} \kappa}$

```python
#Spatial throughput
import numpy as np
import statsmodels.api as sm # recommended import according to the docs
import matplotlib.pyplot as plt
#
alpha = 3
r = 2
density1 = 0.05
density2 = 0.1
density3 = 0.2
p = 1

fig, axes = plt.subplots(1, 2, figsize=(12,5), dpi=1200)

beta = np.linspace(0,9,300)
axes[0].set_xlabel(r'$\beta$', fontsize=16)
axes[0].set_ylabel('Spatial throughput', fontsize=16)
axes[0].plot(beta, p * density1 * (np.log(1+beta)/np.log(2)) * np.exp(- (r**2) * beta**(2/alpha) * np.pi * p * density1),'k-',
             label = "$\lambda=0.05$")
axes[0].plot(beta, p *density2 * (np.log(1+beta)/np.log(2)) * np.exp(- (r**2) * beta**(2/alpha) * np.pi * p * density2),'g-',
             label = "$\lambda=0.10$")
axes[0].plot(beta, p * density3 * (np.log(1+beta)/np.log(2)) * np.exp(- (r**2) * beta**(2/alpha) * np.pi * p * density3),'b-',
             label = "$\lambda=0.15$")
#plt.axis([0,4,0,1])
axes[0].legend(loc=1,fontsize=16)
axes[0].grid(color='k', alpha=0.5, linestyle='dashed', linewidth=0.5)

#####

alpha = 3
r = 2
beta1 = 0.5
beta2 = 2.5
beta3 = 5
p = 1
density = np.linspace(0,0.5,300)
axes[1].set_xlabel(r'$\lambda$', fontsize=16)
axes[1].plot(density, p * density * (np.log(1+beta1)/np.log(2)) * np.exp(- (r**2) * beta1**(2/alpha) * np.pi * p * density),'k-',
             label = r'$\beta=0.5$')
axes[1].plot(density, p *density * (np.log(1+beta2)/np.log(2)) * np.exp(- (r**2) * beta2**(2/alpha) * np.pi * p * density),'g-',
             label = r'$\beta=2.5$')
axes[1].plot(density, p *density * (np.log(1+beta3)/np.log(2)) * np.exp(- (r**2) * beta3**(2/alpha) * np.pi * p * density),'b-',
             label = r'$\beta=5.0$')

#plt.axis([0,4,0,1])
axes[1].legend(loc=1,fontsize=16)
axes[1].grid(color='k', alpha=0.5, linestyle='dashed', linewidth=0.5)

fig.show()
```

### 2.4 Transmission capacity in [bit/s/Hz/m$^2$]

*Definition:* Transmission capacity is the maximum spatial throughput (in terms of the network density) such that the outage probability $p_\mathrm {o}$ must be lower or equal to a given constraint $\epsilon$ (which is usually a small number). Mathematically:

 $\displaystyle \mathrm{TC} = \max\limits_ {\lambda}  \mathrm{ST} = p \lambda \; \log(1+\beta) \; e^{-p \lambda \pi r^2 \beta^{2/\alpha} \kappa}\\ \mathrm{s.t.}\; \; p_\mathrm {o} = 1-p_\mathrm{s} = e^{-p \lambda \pi r^2 \beta^{2/\alpha} \kappa} \leq \epsilon$

*Exercise:* Compute the transmission capacity.

## 4. References

[1] M. Haenngi, *Stochastic Geometry for Wireless Networks*, 105-108.

[2] F. Baccelli, B.BlaszczyszynStochastic, *Geometry and Wireless Networks, Volume II- Applications*, pages 3-27

[3] P. Nardelli, *Analysis of the spatial throughput in interference networks*, pages 22-26

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

```python

```


```python
# initial setup
try:
    # settings colab:
    import google.colab
except ModuleNotFoundError:
    # settings local:
    %run "../../../common/0_notebooks_base_setup.py"
```

---

#### Distribución Exponencial

La distribución exponencial es una **distribución de probabilidad continua** que describe el tiempo entre eventos en un proceso de Poisson.

Su función de densidad de probabilidad

\begin{equation}
    f_X(x) = P(x) = \begin{cases}
    \lambda e^{-\lambda x} & \text{si $x \ge 0$} \\
    0 & \text{en otro caso}
  \end{cases}
\end{equation}

Su función de distribución acumulada es
\begin{equation}
F_X(x) = P(X <= x) = \begin{cases}
    0 & \text{si $x \lt 0$} \\
    1 - e^{-\lambda x} & \text{si $x \ge 0$}
  \end{cases}
\end{equation}

Función de densidad de probabilidad:

Función de distribución de probabilidad:

#### Ejemplos

* X: Tiempo entre el ingreso de pacientes consecutivos en la guardia de un hospital en una hora

* X: Tiempo entre la llegada de autos consecutivos a una cabina de peaje

* X: Tiempo entre llamados consecutivos que llegan a un operador en un call center en la mañana.

---

Prueben ahora generando 100000 datos para esta distribución con distintos valores de lambda (0.5, 1, 1.5, 10), usando como modelo el código que vemos a continuación, y comparen los gráficos. ¿Qué pueden concluir?

Observemos en la documentación, que el parámetro que recibe es beta en lugar de lambda: $\lambda$ = 1/$ \beta$https://docs.scipy.org/doc/numpy/reference/random/generated/numpy.random.Generator.exponential.html#numpy.random.Generator.exponential

```python
import seaborn as sns
def distribution_plotter(data, label, bins=None):
    sns.set(rc={"figure.figsize": (10, 7)})
    sns.set_style("white")
    dist = sns.distplot(data, bins= bins, hist_kws={'alpha':0.2}, kde_kws={'linewidth':5})
    dist.set_title('Distribucion de ' + label + '\n', fontsize=16)
```

```python
import numpy as np
random_generator = np.random.default_rng()

lambda_value = .5
beta_value = 1.0 / lambda_value
sample_size = 100000
random_exponential_data_1 = random_generator.exponential(scale=beta_value, size = sample_size)

lambda_value = 1
beta_value = 1.0 / lambda_value
sample_size = 100000
random_exponential_data_2 = random_generator.exponential(scale=beta_value, size = sample_size)

lambda_value = 1.5
beta_value = 1.0 / lambda_value
sample_size = 100000
random_exponential_data_3 = random_generator.exponential(scale=beta_value, size = sample_size)

distribution_plotter(random_exponential_data_1, "Exponential")
distribution_plotter(random_exponential_data_2, "Exponential")
distribution_plotter(random_exponential_data_3, "Exponential")
```

---

#### Referencias

https://towardsdatascience.com/what-is-exponential-distribution-7bdd08590e2a

Gráficos: https://es.wikipedia.org/wiki/Distribuci%C3%B3n_exponencial


```python

from __future__ import division
from time import time

from sympy import *
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from scipy.integrate import dblquad, nquad, simps
from scipy import constants
from scipy import interpolate
from mpl_toolkits.mplot3d import axes3d

%matplotlib inline

fsize = 20
plt.rcParams['font.size'] = 14
plt.rcParams['font.family'] = 'serif'

init_printing()

rho = 7400  # (kg/m**3) density of neodymium the magnet is assumed to be made of
M = 1e6  # magnetization of the neodymium cylinder (in A/m)

a = 0.01  # radius of the cylinder (in m)
L = 0.002  # height of the cylinder (in m)

B0 = -constants.mu_0/(4*np.pi)*M  # this is the constant in the above equations

dwl = 0.001  # diameter (in m) of wire's cross section
rwl = a + 0.5*dwl  # radius (in m) of the wire loop through which the magnet is falling
rhom = 1.68e-8  # (Om*m) the resistivity of copper
z0 = 0.  # initial z coordinate (in m) of the magnet over the wire loop
zwl = 0.2  # z coordinate (in m) of wire loop's position on the pipe

# These are the integrands in the above equations

def funcBz(phi, R):
    return (R*(0.5*L - z)/(R**2 + (0.5*L - z)**2 + r**2 - 2*R*r*np.cos(phi))**1.5 +
            R*(0.5*L + z)/(R**2 + (0.5*L + z)**2 + r**2 - 2*R*r*np.cos(phi))**1.5)

def funcBr(phi, R):
    return (-R*(r - R*np.cos(phi))/(R**2 + (0.5*L - z)**2 + r**2 - 2*R*r*np.cos(phi))**1.5 +
            R*(r-R*np.cos(phi))/(R**2 + (0.5*L + z)**2 + r**2 - 2*R*r*np.cos(phi))**1.5)

# Cartesian coordinate grid for a projection of the magnetic field and pipe on the (X,Z) plane

la = 4 # a half of the size of the plotted domain (cube) in units of a
L2 = la*a # same in m
n = 11 # number of mesh points along each of the coordinate axes
nm = int(n/2)

xx = np.linspace(-L2, L2, n)
# yy = np.linspace(-L2, L2, n) # yy is needed only for a 3d plot
zz = np.linspace(-L2, L2, n)

# to plot a projection on the (X,Z) plane
yy = np.linspace(0, 0, n)

bx = np.zeros((n, n, n), dtype=float)
by = np.zeros((n, n, n), dtype=float)
bz = np.zeros((n, n, n), dtype=float)

# find z and r components of the magnetic field in the cylindrical coordiantes and transform them to
# x, y, and z components in the Cartesian coordinate system
# for this, we compute the above integrals using nquad or dblquad functions from scipy.integrate

t = time()

maxerz = 0.
maxfz = 0.
maxerr = 0.
maxfr = 0.

for i in range(n):
    x = xx[i]
    for j in range(n):
        y = yy[j]
        r = np.sqrt(x**2+y**2)
        alpha = np.arctan2(y, x)
        for k in range(n):
            z = zz[k]
            fz = nquad(funcBz, [[0, 2*np.pi], [0, a]])
            fr = nquad(funcBr, [[0, 2*np.pi], [0, a]])
            #fz=dblquad(funcBz, 0, a, lambda phi: 0, lambda phi: 2*np.pi)
            #fr=dblquad(funcBr, 0, a, lambda phi: 0, lambda phi: 2*np.pi)
            Br = B0*fr[0]
            bx[i, j, k] = Br * (np.cos(alpha))
            by[i, j, k] = Br * (np.sin(alpha))
            bz[i, j, k] = B0 * fz[0]

            # find maximum errors of the integration for the z and r components
            if abs(fz[0]) > 0.:
                erz = abs(fz[1]/fz[0])
            else:
                erz = 0.
            if erz > maxerz:
                maxerz = erz
                maxfz = fz[0]

            if abs(fr[0]) > 0.:
                err = abs(fr[1]/fr[0])
            else:
                err = 0.
            if err > maxerr:
                maxerr = err
                maxfr = fr[0]

xx, yy, zz = np.meshgrid(xx, yy, zz)

# 3d figure of the (X,Z) projection
fig = plt.figure()
ax = fig.gca(projection='3d')
fig.set_size_inches(10, 10)

# or its projection on the (X,Z) plane

# TO GET A GOOD PLOT TRY ONE OF THESE COMMANDS
# ax.quiver(xx,yy,zz,bx[:,nm,:],by[:,nm,:],bz[:,nm,:],color='b',length=0.5*a,pivot='middle')
ax.quiver(xx, yy, zz, bx[:, nm, :], by[:, nm, :], bz[:, nm, :],
          normalize=True, color='b', length=0.5*a, pivot='middle')

# Plot the magnetic cylinder surrounded by a pipe

def cylinder(r):
    beta = np.linspace(-2*np.pi, 2*np.pi, 100)
    x = r*np.cos(beta)
    y = r*np.sin(beta)
    return x, y

cx, cy = cylinder(a)
cx2, cy2 = cylinder(rwl)

for i in np.linspace(-L, L, 100):
    ax.plot(cx, cy, i, label='Cylinder', color='r', alpha=0.8)

for i in np.linspace(-L2, L2, 100):
    ax.plot(cx2, cy2, i, label='Pipe', color='g', alpha=0.4)

plt.title('Projection of the pipe (green) and magnetic field of the cylinder (red) on the (X,Z) plane')
ax.set_ylim([-L2, L2])
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')

plt.show()

```

```python

```


# 💃 Classifying differential equations 💃

Questions:
- What is a differential equation?
- What is the difference between an ordinary (ODE) and partial (PDE) differential equation?
- How do I classify the different types of differential equations?

Objectives:
- Identify the dependent and independent variables in a differential equation
- Distinguish between and ODE and PDE
- Identify the order of a differential equation
- Distinguish between linear and non-linear equations
- Distinguish between heterogeneous and homogeneous equations
- Identify a separable equation

### A differential equation is an equation that relates one or more functions and their derivatives

- The functions usually represent physical quantities (e.g. $\mathbf{F}$))
- The derivative represents a rate of change (e.g. acceleration)
- The differential equation represents the relationship between the two.
- For example, Newton's second law for $n$ particles of mass $m$:

\begin{equation}
\mathbf{F}(t,\mathbf{x},\mathbf{v}) = m\frac{d\mathbf{v}}{dt}
\end{equation}

### An independent variable is... a quantity that varies independently...

- An independent variable does not depend on other variables
- A  dependent variable depends on the independent variable

\begin{equation}
\mathbf{F}(t,\mathbf{x},\mathbf{v}) = m\frac{d\mathbf{v}}{dt}
\end{equation}

- $t$ is an independent variable
- $x$ and $v$ are dependent variables
- Writing $x = x(t)$ makes this relationship clear.

### Differential equations can be classified in a variety of ways

There are several ways to describe and classify differential equations. There are standard solution methods for each type, so it is useful to understand the classifications.

Once you can cook a single piece of spaghetti, you can cook all pieces of spaghetti!

### An ODE contains differentials with respect to only one variable

For example, the following equations are ODEs:

\begin{equation}
\frac{d x}{d t} = at
\end{equation}
\begin{equation}
\frac{d^3 x}{d t^3} + \frac{x}{t} = b
\end{equation}

As in each case the differentials are with respect to the single variable $t$.

### Partial differential equations (PDE) contain differentials with respect to several independent variables.

An example of a PDE is:

\begin{equation}
\frac{\partial x}{\partial t} = \frac{\partial x}{\partial y}
\end{equation}

As there is one differential with respect to $t$ and one differential with respect to $y$.

Note also the difference in notation - for ODEs we use $d$ whilst for PDEs we  use $\partial$.

### The order of a differential equation is the highest order of any differential contained in it.

For example:

$\frac{d x}{d t} = at$ is first order.

$\frac{d^3 x}{d t^3} + \frac{x}{t} = b$ is third order.

**Note:** $\frac{d^3 x}{d t^3}$ does not equal $\left(\frac{d x}{d t}\right)^3$!

###  Linear equations do not contain higher powers of either the dependent variable or its differentials

For example:

$\frac{d^3 x}{d t^3} = at$ and $\frac{\partial x}{\partial t} = \frac{\partial x}{\partial y} $ are linear.

$(\frac{d x}{d t})^3 = at$ and $\frac{d^3 x}{d t^3} = x^2$ are non-linear.

Non-linear equations can be particularly nasty to solve analytically, and so are often tackled numerically.

### Homogeneous equations do not contain any non-differential terms

For example:

$\frac{\partial x}{\partial t} = \frac{\partial x}{\partial y}$ is a homogeneous equation.

$\frac{\partial x}{\partial t} - \frac{\partial x}{\partial y}=a$ is a heterogeneous equation (unless $a=0$!).

### Separable equations can be written as a product of two functions of different variables

A separable differential equation takes the form

\begin{equation}
f(x)\frac{d x}{d t} = g(t)
\end{equation}

Separable equations are some of the easiest to solve as we can split the equation into two independent parts with fewer variables, and solve each in turn - we will see an example of this in the next tutorial.

Keypoints:

- An independent variable is a quantity that varies independently
- Differential equations can be classified in a variety of ways
- An ODE contains differentials with respect to only one variable
- The order is the highest order of any differential contained in it
- Linear equations do not contain higher powers of either the dependent variable or its differentials
- Homogeneous equations do not contain any non-differential terms

# Radioactive Decay

----
Questions:
- How can I describe radioactive decay using a first-order ODE?
- What are initial conditions and why are  they important?
----

----
Objectives:
- Map between physical notation for a particular problem and the more general notation for all differential equations
- Solve a linear, first order, separable ODE using integration
- Understand the physical importance of initial conditions
----

### Radioactive decay can be modelled a linear, first-order ODE

As our first example of an ODE we will model radioactive decay using a differential equation.

We know that the decay rate is proportional to the number of atoms present.  Mathematically, this relationship can be expressed as:

\begin{equation}
\frac{d N}{d t} = -\lambda N
\end{equation}

Note that we could choose to use different variables, for example:

\begin{equation}
\frac{d y}{d x} = cy
\end{equation}

However we try to use variables connected to the context of the problem. For example $N$ for the Number of atoms.

If we know that 10% of atoms will decay per second we could write:

\begin{equation}
\frac{d N}{d t} = -0.1 N
\end{equation}

where $N$ is the number of atoms and $t$ is time measured in seconds.

This equation is linear and first-order.

| physical notation | generic notation $\left(\frac{dy}{dx} = f(y)\right)$ |
|-----|-----|
|number of  atoms $N$ | dependent  variable $y$|
| time $t$ | independent variable  $x$|
| decay rate $\frac{dN}{dt}$ | differential $\frac{dy}{dx}$|
| constant of proportionality $\lambda=0.1 $ | parameter(s) |

### The equation for radioactive decay is separable and has an analytic solution

The radioactive decay equation is separable:

\begin{equation}
\frac{1}{N}\frac{d N}{d t} = -\lambda
\end{equation}

\begin{equation}
\frac{dN}{N} = -\lambda dt.
\end{equation}

We can then integrate each side:

\begin{equation}
\ln N = -\lambda t + const.
\end{equation}

and solve for N:

\begin{equation}
N = e^{-\lambda t}e^{\textrm{const.}}
\end{equation}

### To model a physical system an initial value has to be provided

\begin{equation}
N(t) = e^{-\lambda t}e^{\textrm{const.}}
\end{equation}

At the beginning (when $t=0$):

\begin{equation}
N (t_0) = e^{0}e^{\textrm{const.}} = e^{\textrm{const.}}
\end{equation}

So we can identify $e^{\textrm{const.}}$ as the amount of radioactive material that was present in the beginning. We denote this starting amount as $N_0$.

Substituting this back into Equation 4, the final solution can be more meaningfully written as:

\begin{equation}
N = N_0 e^{-\lambda t}
\end{equation}

We now have  not  just one solution, but a whole class  of solutions that are dependent on the initial amount of radioactive material $N_0$.

Remember that not all mathematical solutions make physical sense. To model a physical system, this initial value (also known as initial condition) has to be provided alongside the constant of proportionality $\lambda$.

### ODEs can have initial values or boundary values

ODEs have either initial values or boundary values. Take Newton's second law as an example:

\begin{equation}
\frac{\mathrm{d}^2x}{\mathrm{d}t^2} = -g
\end{equation}

An initial value problem would be where we know the starting position and velocity. A boundary value problem would be where we specify the position of the ball at times $t=t_0$ and $t=t_1$.

For second-order ODEs (such as acceleration under gravity) we need to provide two initial/boundary conditions, for third-order ODEs we would need to provide three, and so on. Our radioactive decay example was a first-order ODE and so we only had to provide a single initial condition.

-----
Keypoints:
- Radioactive decay can be modelled a linear, first-order ODE
- The equation for radioactive decay is separable and has an analytic solution
- To model a physical system an initial value has to be provided
- The number of  initial conditions depends on the order of  the differential equation
-----

# Euler's Method

-------
Questions:
- How do I use Euler's method to solve a first-order ODE?

--------

---------
Objectives:
- Use Euler's method, implemented in Python, to solve a first-order ODE
- Understand that this method is approximate and the significance of step size $h$
- Compare results at different levels of approximation using the `matplotlib` library.
------

### There are a variety of ways to solve an ODE

In the previous lesson we considered nuclear decay:

\begin{equation}
\frac{\mathrm{d} N}{\mathrm{d} t} = -\lambda N
\end{equation}

This is one of the simplest examples of am ODE - a first-order, linear, separable differential equation with one dependent variable. We saw that we could model the number of atoms $N$ by finding an analytic solution through integration:

\begin{equation}
N = N_0 e^{-\lambda t}
\end{equation}

However there is more than one way to crack an egg (or solve a differential equation). We could have, instead, used an approximate, numerical method. One such method - Euler's method - is this subject of this lesson.

### A function can be approximated using a Taylor expansion

The Taylor series is a polynomial expansion of a function about a point.
For example, the image below shows $\mathrm{sin}(x)$ and its Taylor approximation by polynomials of degree 1, 3, 5, 7, 9, 11, and 13 at $x = 0$.

Credit: Image By IkamusumeFan - CC BY-SA 3.0, Commons Wikimedia

The Taylor series of $f(x)$ evaluated at point $a$ can be expressed as:

\begin{equation}
f(x) = f(a) + \frac{\mathrm{d} f}{\mathrm{d} x}(x-a) + \frac{1}{2!} \frac{\mathrm{d} ^2f}{\mathrm{d} x^2}(x-a)^2 + \frac{1}{3!} \frac{\mathrm{d} ^3f}{\mathrm{d} x^3}(x-a)^3
\end{equation}

Returning to our example of nuclear decay, we can use a Taylor expansion to write the value of $N$ a short interval $h$ later:

\begin{equation}
N(t+h) = N(t) + h\frac{\mathrm{d}N}{\mathrm{d}t} + \frac{1}{2}h^2\frac{\mathrm{d}^2N}{\mathrm{d}t^2} + \ldots
\end{equation}

\begin{equation}
N(t+h) = N(t) + hf(N,t) + \mathcal{O}(h^2)
\end{equation}

If $h$ is small and $h^2$ is very small we can neglect the terms in $h^2$ and higher and we get:

\begin{equation}
N(t+h) = N(t) + hf(N,t).
\end{equation}

```python
# https://gist.github.com/dm-wyncode/55823165c104717ca49863fc526d1354
"""Embed a YouTube video via its embed url into a notebook."""
from functools import partial

from IPython.display import display, IFrame

width, height = (560, 315, )

def _iframe_attrs(embed_url):
    """Get IFrame args."""
    return (
        ('src', 'width', 'height'),
        (embed_url, width, height, ),
    )

def _get_args(embed_url):
    """Get args for type to create a class."""
    iframe = dict(zip(*_iframe_attrs(embed_url)))
    attrs = {
        'display': partial(display, IFrame(**iframe)),
    }
    return ('YouTubeVideo', (object, ), attrs, )

def youtube_video(embed_url):
    """Embed YouTube video into a notebook.

    Place this module into the same directory as the notebook.

    >>> from embed import youtube_video
    >>> youtube_video(url).display()
    """
    YouTubeVideo = type(*_get_args(embed_url)) # make a class
    return YouTubeVideo() # return an object

```

```python

youtube_video("https://www.youtube.com/embed/3d6DsjIBzJ4").display()
```

### Euler's method can be used to approximate the solution of differential equations

We can keep applying the equation above so that we calculate $N(t)$ at a succession of equally spaced points for as  long as we want. If $h$ is small enough we can get a good approximation to the solution of the equation. This method for solving differential equations is called Euler's method, after Leonhard Euler, its inventor.

**Note**: Although we are neglecting terms $h^2$ and higher, Euler's method typically has an error linear in $h$ as the error accumulates over repeated steps. This means that if we want to double the accuracy of our calculation we need to double the number of steps, and double the calcuation time.

### Euler's method can be applied using the Python skills we have developed

Let's use Euler's method to solve the differential equation for nuclear decay. We will model the decay process over a period of 10 seconds, with the decay constant $\lambda=0.1$ and the initial condition $N_0 = 1000$.

\begin{equation}
\frac{\mathrm{d}N}{\mathrm{d} t} = -0.1 N
\end{equation}

----

Keypoints:

- There are a variety of ways to solve an ODE
- A function can be approximated using a Taylor expansion
- If the step size $h$ is small then higher  order terms can be neglected
- Euler's method can be used to approximate the solution of differential equations
- Euler's method can be applied using the Python skills we have developed
- We can easily visualise our results, and compare against the analytical solution, using the matplotlib plotting library
-----

# The Strange Attractor

Questions:
- How do I solve simultaneous ODEs?
- How do I solve second-order ODEs (and higher)?

Objectives:
- Use Euler's method, implemented in Python, to solve a set of simultaneous ODEs
- Use Euler's method, implemented in Python, to solve a second-order ODE
- Understand how the same method could be applied to higher order ODEs

### Computers don't care so much about the type of differential equation

- In the previous lesson we used Euler's method to model radioactive decay. But we have seen that this equation can also be solved analytically

- However there are a large number of physical equations that cannot be solved analytically, and that rely on numerical methods for their modelling.

- For example, the Lotka-Volterra equations for studying predator-prey interactions have multiple dependent variables and the Cahn-Hilliard equation for modelling phase separation in fluids is non-linear. These equations can be solved analytically for particular, special cases only.

- Computers don't care whether an equation is linear or non-linear, multi-variable or single-variable, the numerical method for studying it is much the same

- The caveat is that numerical methods are approximate and so we need to think about the accuracy of the methods used.

### Simultaneous ODEs can be solved using numerical methods

One class of problems that are difficult to solve analytically are <bold>simultaenous ODEs</bold>. These are equations where the derivative of each dependent variable can depend on any of the variables (dependent or independent). For example,

\begin{eqnarray}
\frac{\mathrm{d}x}{\mathrm{d}t} &=& xt + y \\
\frac{\mathrm{d}y}{\mathrm{d}t} &=& \mathrm{sin}^2\omega t - xy
\end{eqnarray}

### Euler's method is easily extended to simultaneous ODEs

In the previous lesson we were introduced to Euler's method for the single variable case:

\begin{equation}
N(t+h) = N(t) + hf(N,t).
\end{equation}

This can be easily extended to the multi-variable case using vector notation:

\begin{equation}
\mathbf{r}(t+h) = \mathbf{r}(t) + h\mathbf{f}(\mathbf{r},t).
\end{equation}

We have seen that arrays can be easily represented in Python using the NumPy library. This allows us to do arithmetic with vectors directly (rather than using verbose workarounds such as `for` loops), so the code is not much more complicated than the one-variable case.

### The Lorenz equations are a famous set of simultaneous ODEs

The Lorenz equations demonstrate that systems can be deterministic and yet be inherently unpredictable (even in the absence of quantum effects).

\begin{eqnarray}
\frac{\mathrm{d}x}{\mathrm{d}t} &=& \sigma(y-x) \\
\frac{\mathrm{d}y}{\mathrm{d}t} &=& rx-y-xz \\
\frac{\mathrm{d}z}{\mathrm{d}t} &=& xy-bz
\end{eqnarray}

There three dependent variables - $x$, $y$ and $z$, and one independent variable $t$. There are also three constants -  $\sigma$, $r$ and $b$.

For particular values of $\sigma$, $r$ and $b$ the Lorenz systems has <bold>chaotic behaviour</bold> (a strong dependence on the initial conditions) and for a subset of these, there is also fractal structure called the <bold>strange attractor</bold>.

### Higher order ODEs can be re-cast as simultaneous ODEs and solved in the same way

Many physical equations are second-order or higher. The general form for a second-order differential equation with one dependent variable is:

\begin{equation}
\frac{\mathrm{d}^2x}{\mathrm{d}t^2} = f\left(x,\frac{\mathrm{d}x}{\mathrm{d}t},t\right)
\end{equation}

Luckily, we can re-cast a higher order equation as a set of simultaneous equations, and then solve in the same way as above.

Let's use the non-linear pendulum as an example. For a pendulum with an art of length $l$ and a bob of mass $m$, Newton's second law ($F=ma$) provides the following equation of motion:

\begin{equation}
ml\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2} = -mg\sin(\theta),
\end{equation}

where $\theta$ is the angle of displacement of the arm from the vertical and $l\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2}$ is the acceleration of the mass in the tangential direction. The exact solution to this equation is unknown, but we now have the knowledge needed to find a numerical approximation.

The equation can be re-written as:

\begin{equation}
\frac{\mathrm{d}^2\theta}{\mathrm{d}t^2} = -\frac{g}{l}\sin(\theta)
\end{equation}

We define a new variable $\omega$:

\begin{equation}
\frac{\mathrm{d}\theta}{\mathrm{d}t} = \omega
\end{equation}

and substitute this into the equation of motion:

\begin{equation}
\frac{\mathrm{d}\omega}{\mathrm{d}t} = -\frac{g}{l}\sin(\theta).
\end{equation}

These two expressions form a set of simultaneous equations that can be solved numerically using the method outlined above.

-----
Keypoints:
- Computers don't care so much about the type of differential equation
- Simulatenous ODEs can also be solved using numerical methods
- Euler's method is easily extended to the multi-variable case
- Higher order ODEs can be re-cast as simultaneous ODEs and solved the same way
-----

# The Runge-Kutta method

Questions:
- How do I use the Runge-Kutta method for more accurate solutions?

Objectives:
- Use the Runge-Kutta method, implemented in Python, to solve a first-order ODE
- Compare results at different levels of approximation using the `matplotlib` library.

### The Runge-Kutta method is more accurate than Euler's method and runs just as fast

So far we have used Euler's method for solving ODEs. We have learnt that, using this method, the final expression for the total error is linear in $h$.

However for roughly the same compute time we can reduce the total error so it is of order $h^2$ by implementing another method - the <bold>Runge-Kutta method</bold>.

> Note: It is common to use the Runge-Kutta method for solving ODEs given the improved accuracy over Euler's method. However Euler's method is still commonly used for PDEs (where there are other, larger, sources of error).

> Note: The Runge-Kutta method is actually a family of methods. In fact, Euler's method is the first-order Runge-Kutta method. There is then the second-order Runge-Kutta method, third-order Runge-Kutta method, and so on..

Euler's method does not take into account the curvature of the solution, whilst Runge-Kutta methods do, by calculating the gradient at intermediate points in the (time-)step.

### Runge-Kutta methods are derived from Taylor expansion(s) around intermediate point(s)

To derive the second-order Runge-Kutta method we:

1) estimate $x(t+h)$ using a Taylor expansion around $t+\frac{h}{2}$:

\begin{equation}
x(t+h) = x(t+\frac{h}{2}) + \frac{h}{2}\left(\frac{\mathrm{d}x}{\mathrm{d}t}\right)_{t+\frac{h}{2}} + \frac{h^2}{8}\left(\frac{\mathrm{d}^2x}{\mathrm{d}t^2}\right)_{t+\frac{h}{2}}+\mathcal{O}(h^3)
\end{equation}

2) estimate $x(t)$ using a Taylor expansion around $t-\frac{h}{2}$:

\begin{equation}
x(t) = x(t+\frac{h}{2}) - \frac{h}{2}\left(\frac{\mathrm{d}x}{\mathrm{d}t}\right)_{t+\frac{h}{2}} + \frac{h^2}{8}\left(\frac{\mathrm{d}^2x}{\mathrm{d}t^2}\right)_{t+\frac{h}{2}}+\mathcal{O}(h^3)
\end{equation}

3) Subtract Equation 2 from Equation 1 and re-arrange:

\begin{eqnarray}
x(t+h) &=& x(t) + h\left(\frac{\mathrm{d}x}{\mathrm{d}t}\right))_{t+\frac{h}{2}}+\mathcal{O}(h^3) \\
x(t+h) &=& x(t) + hf(x(t+\frac{h}{2}),t+\frac{h}{2})+\mathcal{O}(h^3)
\end{eqnarray}

> Note: the $h^2$ term has completely disappeared, and the error term is now order $h^3$. We can say that this approximation is now accurate to order $h^2$.

The problem is that this requires knowledge of $x(t+\frac{h}{2})$ which we don't currently have. We can however estimate this using the Euler method!

\begin{equation}
x(t+\frac{h}{2}) = x(t) + \frac{h}{2}f(x,t).
\end{equation}

Substituting this into Equation 3 above, we can write the method for a single step as follows:

\begin{eqnarray}
k_1 &=& hf(x,t) \\
k_2 &=& hf(x+\frac{k_1}{2},t+\frac{h}{2})\\
x(t+h) &=& x(t) + k_2
\end{eqnarray}

See how $k_1$ is used to give an estimate for $x(t+\frac{h}{2})$ ($k_2$), which is then substituted into the third equation to give an estimate for  $x(t+h)$.

> Note: Higher orde Runge-Kutta methods can be derived in a similar way - by calculating the Taylor series around various points and then taking a linear combination of the resulting expansions. As we increase the number of intermediate points, we increase the accuracy of the method. The downside is that the equations get increasingly complicated.

Keypoints:
- The Runge-Kutta method is more accurate than Euler's method and runs just as fast
- Runge-Kutta methods are derived from Taylor expansion(s) around intermediate point(s)
- Runge-Kutta methods can be applied using the Python skills we have developed
- We can easily compare our various models using the `matplotlib` plotting library


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Eigenfunctions

An [eigenfunction](https://en.wikipedia.org/wiki/Eigenfunction) of a system is defined as the input signal $x(t)$ which produces the output signal $y(t) = \mathcal{H}\{ x(t) \} = \lambda \cdot x(t)$ with $\lambda \in \mathbb{C}$. The weight $\lambda$ associated with $x(t)$ is known as scalar eigenvalue of the system. Hence, besides a weighting factor, an eigenfunction is not modified by passing through the system.

[Complex exponential signals](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $e^{s t}$ with $s \in \mathbb{C}$ are eigenfunctions of linear time-invariant (LTI) systems. This can be proven by applying the properties of LTI systems. Lets assume a generic LTI system with input signal $x(t) = e^{s t}$ and output signal $y(t) = \mathcal{H}\{ x(t) \}$. The response of the LTI system to the shifted input signal $x(t-\tau) = e^{s (t-\tau)}$ reads

\begin{equation}
y(t - \tau) = \mathcal{H}\{ x(t-\tau) \} = \mathcal{H}\{ e^{-s \tau} \cdot e^{s t}  \}
\end{equation}

due to the implied shift-invariance. Now considering the implied linearity this can be reformulated as

\begin{equation}
y(t - \tau) =  e^{-s \tau} \cdot \mathcal{H}\{ e^{s t}  \} = e^{-s \tau} \cdot y(t)
\end{equation}

It is straightforward to show that $y(t) = \lambda e^{st}$ fulfills above equation

\begin{equation}
\lambda e^{s t} e^{-s \tau} = e^{-s \tau} \lambda e^{st}
\end{equation}

**Example**

An LTI system whose input/output relation is given by the following inhomogeneous linear ordinary differential equation (ODE) with constant coefficients is investigated

\begin{equation}
a_0 y(t) + a_1 \frac{d y(t)}{dt} + a_2 \frac{d^2 y(t)}{dt^2} = x(t)
\end{equation}

with $a_i \in \mathbb{R} \quad \forall i$. In the remainder, the output signal $y(t)$ of the system is computed by explicit solution of the ODE for $x(t) = e^{s t}$ as input signal. Integration constants are discarded for ease of illustration.

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

t, s, a0, a1, a2 = sym.symbols('t s a:3')
x = sym.exp(s * t)
y = sym.Function('y')(t)

ode = sym.Eq(a0*y + a1*y.diff(t) + a2*y.diff(t, 2), x)
solution = sym.dsolve(ode)
solution.subs({'C1': 0, 'C2': 0})
```

**Exercises**

* Is the complex exponential signal an eigenfunction of the system?
* Introduce $x(t) = e^{s t}$ and $y(t) = \lambda \cdot e^{s t}$ into the ODE and solve manually for the eigenvalue $\lambda$. How is the result related to above result derived by solving the ODE?
* Can you generalize your findings to an ODE of arbitrary order?

**Example**

The following inhomogeneous linear ODE with time-dependent coefficient is considered as an example for a **time-variant** but linear system

\begin{equation}
t \cdot \frac{d y(t)}{dt} = x(t)
\end{equation}

The output signal $y(t)$ of the system for a complex exponential signal at the input $x(t) = e^{st}$ is computed by explicit solution of the ODE. Again integration constants are discarded.

```python
ode = sym.Eq(t*y.diff(t), x)
solution = sym.dsolve(ode)
solution.subs('C1', 0)
```

Note, $\text{Ei}(\cdot)$ denotes the [exponential integral](http://docs.sympy.org/latest/modules/functions/special.html#sympy.functions.special.error_functions.Ei). The response $y(t)$ of the time-variant system is not equal to a weighted complex exponential signal $\lambda \cdot e^{s t}$. It can be concluded that complex exponentials are no eigenfunctions of this particular time-variant system.

**Example**

A final example considers the following non-linear inhomogeneous ODE with constant coefficients

\begin{equation}
\left( \frac{d y(t)}{dt} \right)^2 = x(t)
\end{equation}

as example for a **non-linear** but time-invariant system. Again, the output signal $y(t)$ of the system for a complex exponential signal at the input $x(t) = e^{st}$ is computed by explicit solution of the ODE. As before, integration constants are discarded.

```python
ode = sym.Eq(y.diff(t)**2, x)
solution = sym.dsolve(ode)
[si.subs('C1', 0) for si in solution]
```

Obviously for this non-linear system complex exponential signals are no eigenfunctions.

## Transfer Function

The complex eigenvalue $\lambda$ constitutes the weight of a complex exponential signal $e^{st}$ (using complex frequency $s$) experiences when passing through an LTI system. It is commonly termed as [*transfer function*](https://en.wikipedia.org/wiki/Transfer_function) and is denoted by $H(s)=\lambda(s)$. Using this definition, the output signal $y(t)$ of an LTI system for a complex exponential signal at the input reads

\begin{equation}
y(t) = \mathcal{H} \{ e^{st} \} = H(s) \cdot e^{st}
\end{equation}

Note that the concept of the transfer function is directly linked to the linearity and time-invariance of a system. Only in this case, complex exponential signals are eigenfunctions of the system and $H(s)$ describes the properties of an LTI system with respect to these.

Above equation can be rewritten in terms of the magnitude $| H(s) |$ and phase $\varphi(s) = \arg \{ H(s) \}$ of the complex transfer function $H(s)$

\begin{equation}
y(t) = | H(s) | \cdot e^{s t + j \varphi(s)}
\end{equation}

The magnitude $| H(s) |$ provides the frequency dependent attenuation/amplification of the eigenfunction $e^{st}$ by the system, while $\varphi(s)$ provides the introduced phase-shift.

## Link between Transfer Function and Impulse Response

In order to establish a link between the transfer function $H(s)$ and the impulse response $h(t)$, the output signal $y(t) = \mathcal{H} \{ x(t) \}$ of an LTI system with input signal $x(t)$ is considered. It is given by convolving the input signal with the impulse response

\begin{equation}
y(t) = x(t) * h(t) = \int_{-\infty}^{\infty} x(t-\tau) \cdot h(\tau) \; d\tau
\end{equation}

For a complex exponential signal as input $x(t) = e^{st}$, the output of an LTI system is given as $y(t) = \mathcal{H} \{ e^{st} \} = H(s) \cdot e^{st}$. Introducing both signals into the convolution integral yields

\begin{equation}
H(s) \cdot e^{st} = \int_{-\infty}^{\infty} e^{st} e^{-s \tau} \cdot h(\tau) \; d\tau
\end{equation}

which after canceling $e^{s t}$ (the integral depends not on $t$) results in

\begin{equation}
H(s) = \int_{-\infty}^{\infty} h(\tau) \cdot e^{-s \tau}  \; d\tau
\end{equation}

under the assumption that the integral converges.
The transfer function $H(s)$ can be computed from the impulse response $h(t)$ by integrating over the impulse response multiplied with the complex exponential function $e^{- s \tau}$. This constitutes an integral transformation, which is later introduced in more detail as [Laplace transform](https://en.wikipedia.org/wiki/Laplace_transform).
Usually the temporal variable $t$ is then used

\begin{equation}
H(s) = \int_{-\infty}^{\infty} h(t) \cdot e^{-s t}  \; d t
\end{equation}
rather than $\tau$ which remained from the convolution integral calculus.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Demo 1: Flow

## Background/Motivation

Flow is a fundamental driver of processes in the surface and subsurface. In this set of demonstrations we will introduce

  * Richards Model for variably saturated subsurface flow
  * Diffusion Wave model for overland flow (surface flow)
  * A fully coupled (integrated hydrology) model

For each model we will provide the mathematical equations, highlight information about the discretization and provide tips on setting boundary conditions, initial conditions, source terms and key parameters.

## Subsurface Flow (Richards Equation)

Richards (1931) model for variably saturated flow:

$\newcommand{porosity}{\phi}$
$\newcommand{dens}{n_l}$
$\newcommand{sat}{s}$
$\newcommand{flux}{\mathbf{q}}$
$\newcommand{krel}{k_r}$
$\newcommand{kabs}{\kappa}$
$\newcommand{viscosity}{\mu}$

$$
\begin{align}
  \frac{\partial}{\partial t} (\porosity \dens \sat) &+ \nabla \cdot \flux = 0 \\
  \flux &= - \dens \frac{\krel \kabs}{\viscosity} (\nabla p + \rho \mathbf{g})
\end{align}
$$

**Note that this uses latex macros defined in this cell.**
Water retention models

Blah, Blah, lots more about the model and equations ...

These tests demonstrate solution of Richards equation on a single column. A single column of cells, oriented vertically, and are initialized as the hydrostatic solution with water table at 7m. Infilitration is turned on, and water fills the domain.

```python
%matplotlib inline
```

```python
import sys,os
sys.path.append(os.path.join(os.environ['ATS_SRC_DIR'],'tools', 'utils'))
import ats_xdmf
import plot_column_data
import numpy as np
from matplotlib import pyplot as plt
```

```python
def plot(dirname, title):
    """A generic plotting function that plots saturation as a function of depth."""
    fig = plt.figure()

    vis = ats_xdmf.VisFile(dirname, time_unit='d')
    vis.loadMesh(columnar=True)

    # plot the solution
    ax = fig.subplots(1,1)
    plot_column_data.plot_subsurface(vis, ['saturation_liquid',], ax, None, cmap='jet')
    ax.set_title(title)
    ax.set_xlabel('saturation [-]')
    ax.set_ylabel('z [m]')
```

```python
plot("./richards/run/", "infiltration example")
```

```python
import numpy as np
from matplotlib import pyplot as plt

flux_data = np.loadtxt("./richards/run/observation_flux.dat",skiprows=11)
flux_data = np.transpose(flux_data)
fig = plt.figure()
ax = fig.subplots(1,1)
plt.plot(flux_data[0]/86400, flux_data[1])
ax.set_xlabel("days")
ax.set_ylabel(r"$mol\cdot m^{-2}\cdot s^{-1}$")

```

## Overland Flow (Diffusion Wave Equation)

Diffusion wave model for overland flow

$\newcommand{ponded_depth}{\delta}$
$\newcommand{liquid_source}{q_l}$

## Integrated Hydrology

```python

```

```python

```


```python
import numpy as np
from skspatial.objects import plane
from sympy import Plane

def perpendicular(a):
    b = np.empty_like(a)
    b[0] = -a[1]
    b[1] = a[0]
    return b

def normalize(a):
    a = np.array(a)
    return a/np.linalg.norm(a)

def get2DProjection(origin, target_point):
    points = [origin,
              target_point,
              [target_point[0], target_point[1], target_point[2] + 1]]  # add 1 in order not to be collinear

    plane = Plane(points[0], points[1], points[2])
    plane_normal = np.array(plane.normal_vector)

    target_point, origin = np.array(target_point), np.array(origin)

    x_axis = normalize(np.array([target_point[0], target_point[1], 0]))
    y_axis = normalize(np.array([0, 0, 1]))

    s = np.dot(plane_normal, target_point-origin)
    x_coord = np.dot(x_axis, target_point-origin)
    y_coord = np.dot(y_axis, target_point-origin)

    return s, x_coord, y_coord

if __name__ == "__main__":
    target_point = [2, 2, 1]
    origin = [0, 0, 0]
    s, t_1, t_2 = get2DProjection(target_point, origin)

    print("s:", s)
    print("t_1:", t_1)
    print("t_2:", t_2)
```

    s: 0
    t_1: nan
    t_2: -1.0

    /var/folders/sd/1vc_q83x5rn9jjrd0x47_cc00000gn/T/ipykernel_92659/4172601782.py:14: RuntimeWarning: invalid value encountered in true_divide
      return a/np.linalg.norm(a)

```python

```

Mathematics/Linear Algebra/Python Visualization Notebooks/scikit-spatial/Projecting-3D-points-to-2D-plane.ipynb