# Bayesian linear regression

* [1. What is Bayesian linear regression?](#intro-blr)
* [2. Recap linear regression](#recap-lr)
* [3. Fundamental concepts](#fundamental-concepts)
* [4. Linear regression from a probabilistic perspective](#prob-lr)
* [5. Linear regression with basis functions](#lr-with-basis-functions)
    * [5.1 Example basis functions](#example-basis-function)
    * [5.2 The design matrix](#design-matrix)
* [6. Bayesian Linear Regression](#blr)
    * [6.1 Step 1: Probabilistic Model](#prob-model)
    * [6.2 Generating a dataset](#dataset)
    * [6.3 Step 2: Posterior over the parameters](#param-posterior)
    * [6.4 Visualizing the parameter posterior](#param-posterior-visualization)
    * [6.5 Step 3: Posterior predictive distribution](#predictive-posterior)
    * [6.6 Visualizing the predictive posterior](#predictive-posterior-visualization)

* [Sources and further reading](#sources)

## Link to interactive demo

[Click here](https://mybinder.org/v2/gh/zotroneneis/machine_learning_basics/HEAD?filepath=bayesian_linear_regression.ipynb) to run the notebook online (using Binder) without installing jupyter or downloading the code.

## 1. What is Bayesian linear regression (BLR)?
Bayesian linear regression is the *Bayesian* interpretation of linear regression. What does that mean? To answer this question we first have to understand the Bayesian approach. In most of the algorithms we have looked at so far we computed *point estimates* of our parameters. For example, in linear regression we chose values for the weights and bias that minimized our mean squared error cost function. In the Bayesian approach we don't work with exact values but with *probabilities*. This allows us to model the *uncertainty* in our parameter estimates. Why is this important?

In nearly all real-world situations, our data and knowledge about the world is incomplete, indirect and noisy. Hence, uncertainty must be a fundamental part of our decision-making process. This is exactly what the Bayesian approach is about. It provides a formal and consistent way to reason in the presence of uncertainty. Bayesian methods have been around for a long time and are widely-used in many areas of science (e.g. astronomy). Although Bayesian methods have been applied to machine learning problems too, they are usually less well known to beginners. The major reason is that they require a good understanding of probability theory.

In the following notebook we will work our way from linear regression to Bayesian linear regression, including the most important theoretical knowledge and code examples.

## 2. Recap linear regression

- In linear regression, we want to find a function $f$ that maps inputs $x \in \mathbb{R}^D$ to corresponding function values $f(x) \in \mathbb{R}$.
- We are given an input dataset $D = \big \{ \mathbf{x}_n, y_n \big \}_{n=1}^N$, where $y_n$ is a noisy observation value: $y_n = f(x_n) + \epsilon$, with $\epsilon$ being an i.i.d. random variable that describes measurement/observation noise
- Our goal is to infer the underlying function $f$ that generated the data such that we can predict function values at new input locations
- In linear regression, we model the underlying function $f$ using a linear combination of the input features:

$$
\begin{split}
y &= \theta_0 + \theta_1 x_1 + \theta_2 x_2 + ... + \theta_d x_d \\
&= \boldsymbol{x}^T \boldsymbol{\theta}
\end{split}
$$

- For more details take a look at the [notebook on linear regression](https://github.com/zotroneneis/machine_learning_basics/blob/master/linear_regression.ipynb)

## 3. Fundamental concepts
- One fundamental tool in Bayesian learning is [Bayes' theorem](https://en.wikipedia.org/wiki/Bayes%27_theorem)
- Bayes' theorem looks as follows:
$$
\begin{equation}
p(\boldsymbol{\theta} | \mathbf{x}, y) = \frac{p(y | \boldsymbol{x}, \boldsymbol{\theta})p(\boldsymbol{\theta})}{p(\boldsymbol{x}, y)}
\end{equation}
$$
- $p(y | \boldsymbol{x}, \boldsymbol{\theta})$ is the *likelihood*. It describes the probability of the target values given the data and parameters.
- $p(\boldsymbol{\theta})$ is the *prior*. It describes our initial knowledge about which parameter values are likely and unlikely.
- $p(\boldsymbol{x}, y)$ is the *evidence*. It describes the joint probability of the data and targets.
- $p(\boldsymbol{\theta} | \boldsymbol{x}, y)$ is the *posterior*. It describes the probability of the parameters given the observed data and targets.

- Another important tool you need to know about is the [Gaussian distribution](https://en.wikipedia.org/wiki/Normal_distribution). If you are not familiar with it I suggest you pause for a minute and understand its main properties before reading on.

In general, Bayesian inference works as follows:
1. We start with some prior belief about a hypothesis $p(h)$
2. We observe some data, representating new evidence $e$
3. We use Bayes' theorem to update our belief given the new evidence: $p(h|e) = \frac{p(e |h)p(h)}{p(e)}$

For more information take a look at the [Wikipedia article on Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference).

## 4. Linear regression from a probabilistic perspective

In order to pave the way for Bayesian linear regression we will take a probabilistic spin on linear regression. Let's start by explicitly modelling the observation noise $\epsilon$. For simplicity, we assume that $\epsilon$ is normally distributed with mean $0$ and some known variance $\sigma^2$: $\epsilon \sim \mathcal{N}(0, \sigma^2)$.

As mentioned in the beginning, a simple linear regression model assumes that the target function $f(x)$ is given by a linear combination of the input features:
$$
\begin{split}
y = f(\boldsymbol{x}) + \epsilon \\
  = \boldsymbol{x}^T \boldsymbol{\theta} + \epsilon
\end{split}
$$

This corresponds to the following likelihood function:
$$p(y | \boldsymbol{x}, \boldsymbol{\theta}) = \mathcal{N}(\boldsymbol{x}^T \boldsymbol{\theta}, \sigma^2)$$

Our goal is to find the parameters $\boldsymbol{\theta} = \{\theta_1, ..., \theta_D\}$ that model the given data best.
In standard linear regression we can find the best parameters using a least-squares, maximum likelihood (ML) or maximum a posteriori (MAP) approach. If you want to know more about these solutions take a look at the [notebook on linear regression](https://github.com/zotroneneis/machine_learning_basics/blob/master/linear_regression.ipynb) or at chapter 9.2 of the book [Mathematics for Machine Learning](https://mml-book.com).

## 5. Linear regression with basis functions
The simple linear regression model above is linear not only with respect to the parameters $\boldsymbol{\theta}$ but also with respect to the inputs $\boldsymbol{x}$. When $\boldsymbol{x}$ is not a vector but a single value (that is, the dataset is one-dimensional) the model $y_i = x_i \cdot \theta$ describes straight lines with $\theta$ being the slope of the line.

The plot below shows example lines produced with the model $y = x \cdot \theta$, using different values for the slope $\theta$ and intercept 0.

Having a model which is linear both with respect to the parameters and inputs limits the functions it can learn significantly. We can make our model more powerful by making it *nonlinear* with respect to the inputs. After all, *linear regression* refers to models which are linear in the *parameters*, not necessarily in the *inputs* (linear in the parameters means that the model describes a function by a linear combination of input features).

Making the model nonlinear with respect to the inputs is easy. We can adapt it by using a nonlinear transformation of the input features $\phi(\boldsymbol{x})$. With this adaptation our model looks as follows:
$$
\begin{split}
y &= \boldsymbol{\phi}^T(\boldsymbol{x}) \boldsymbol{\theta} + \epsilon \\
&= \sum_{k=0}^{K-1} \theta_k \phi_k(\boldsymbol{x}) + \epsilon
\end{split}
$$

Where $\boldsymbol{\phi}: \mathbf{R}^D \rightarrow \mathbf{R}^K$ is a (non)linear transformation of the inputs $\boldsymbol{x}$ and $\phi_k: \mathbf{R}^D \rightarrow \mathbf{R}$ is the $k-$th component of the *feature vector* $\boldsymbol{\phi}$:

$$
\boldsymbol{\phi}(\boldsymbol{x})=\left[\begin{array}{c}
\phi_{0}(\boldsymbol{x}) \\
\phi_{1}(\boldsymbol{x}) \\
\vdots \\
\phi_{K-1}(\boldsymbol{x})
\end{array}
\right]
\in \mathbb{R}^{K}
$$

With our new nonlinear transformation the likelihood function is given by

$$
p(y | \boldsymbol{x}, \boldsymbol{\theta}) = \mathcal{N}(\boldsymbol{\phi}^T(\boldsymbol{x}) \boldsymbol{\theta},\, \sigma^2)
$$

### 5.1 Example basis functions

#### Linear regression
The easiest example for a basis function (for one-dimensional data) would be simple linear regression, that is, no non-linear transformation at all. In this case we would choose $\phi_0(x) = 1$ and $\phi_i(x) = x$. This would result in the following vector $\boldsymbol{\phi}(x)$:

$$
\boldsymbol{\phi}(x)=
\left[
\begin{array}{c}
\phi_{0}(x) \\
\phi_{1}(x) \\
\vdots \\
\phi_{K-1}(x)
\end{array}
\right] =
\left[
\begin{array}{c}
1 \\
x \\
\vdots \\
x
\end{array}
\right]
\in \mathbb{R}^{K}
$$

#### Polynomial regression
Another common choice of basis function for the one-dimensional case is polynomial regression. For this we would set $\phi_i(x) = x^i$ for $i=0, ..., K-1$. The corresponding feature vector $\boldsymbol{\phi}(x)$ would look as follows:

$$
\boldsymbol{\phi}(x)=
\left[
\begin{array}{c}
\phi_{0}(x) \\
\phi_{1}(x) \\
\vdots \\
\phi_{K-1}(x)
\end{array}
\right] =
\left[
\begin{array}{c}
1 \\
x \\
x^2 \\
x^3 \\
\vdots \\
x^{K-1}
\end{array}
\right]
\in \mathbb{R}^{K}
$$

With this transformation we can lift our original one-dimensional input into a $K$-dimensional feature space. Our function $f$ can be any polynomial with degree $\le K-1$: $f(x) = \sum_{k=0}^{K-1} \theta_k x^k$

### 5.2 The design matrix
To make it easier to work with the transformations $\boldsymbol{\phi}(\boldsymbol{x})$ for the different input vectors $\boldsymbol{x}$ we typically create a so called *design matrix* (also called *feature matrix*). Given our dataset $D = \big \{ \mathbf{x}_n, y_n \big \}_{n=1}^N$ we define the design matrix as follows:

$$
\boldsymbol{\Phi}:=\left[\begin{array}{c}
\boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{1}\right) \\
\vdots \\
\boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{N}\right)
\end{array}\right]=\left[\begin{array}{ccc}
\phi_{0}\left(\boldsymbol{x}_{1}\right) & \cdots & \phi_{K-1}\left(\boldsymbol{x}_{1}\right) \\
\phi_{0}\left(\boldsymbol{x}_{2}\right) & \cdots & \phi_{K-1}\left(\boldsymbol{x}_{2}\right) \\
\vdots & & \vdots \\
\phi_{0}\left(\boldsymbol{x}_{N}\right) & \cdots & \phi_{K-1}\left(\boldsymbol{x}_{N}\right)
\end{array}\right] \in \mathbb{R}^{N \times K}
$$

Note that the design matrix is of shape $N \times K$. $N$ is the number of input examples and $K$ is the output dimension of the non-linear transformation $\boldsymbol{\phi}(\boldsymbol{x})$.

## 6. Bayesian linear regression
What changes when we consider a Bayesian interpretation of linear regression? Our data stays the same as before: $D = \big \{ \mathbf{x}_n, y_n \big \}_{n=1}^N$. Given the data $D$ we can define the set of all inputs as $\mathcal{X} := \{\boldsymbol{x}_1, ..., \boldsymbol{x}_n\}$ and the set of all targets as $\mathcal{Y} := \{y_1, ..., y_n \}$.

In simple linear regression we compute point estimates of our parameters (e.g. using a maximum likelihood approach) and use these estimates to make predictions. Different to this, Bayesian linear regression estimates *distributions* over the parameters and predictions. This allows us to model the uncertainty in our predictions.

To perform Bayesian linear regression we follow three steps:
1. We set up a probabilistic model that describes our assumptions how the data and parameters are generated
2. We perform inference for the parameters $\boldsymbol{\theta}$, that is, we compute the posterior probability distribution over the parameters
3. With this posterior we can perform inference for new, unseen inputs $y_*$. In this step we don't compute point estimates of the outputs. Instead, we compute the parameters of the posterior distribution over the outputs.

### 6.1 Step 1: Probabilistic model

We start by setting up a probabilistic model that describes our assumptions how the data and parameters are generated. For this, we place a prior $p(\boldsymbol{\theta})$ over our parameters which encodes what parameter values are plausible (before we have seen any data). Example: With a single parameter $\theta$, a Gaussian  prior $p(\theta) = \mathcal{N}(0, 1)$ says that parameter values are expected to lie in the interval $[−2,2]$ which is two standard deviations around the mean value 0.

To keep things simple we will assume a Gaussian prior over the parameters: $p(\boldsymbol{\theta}) = \mathcal{N}(\boldsymbol{m}_0, \boldsymbol{S}_0)$. Let's further assume that the likelihood function is Gaussian, too: $p(y \mid \boldsymbol{x}, \boldsymbol{\theta})=\mathcal{N}\left(y \mid \boldsymbol{\phi}^{\top}(\boldsymbol{x}) \boldsymbol{\theta}, \sigma^{2}\right)$.

Note: When considering the set of all targets $\mathcal{Y} := \{y_1, ..., y_n \}$, the likelihood function becomes a multivariate Gaussian distribution: $p(\mathcal{Y} \mid \mathcal{X}, \boldsymbol{\theta})=\mathcal{N}\left(\boldsymbol{y} \mid \boldsymbol{\Phi} \boldsymbol{\theta}, \sigma^{2} \boldsymbol{I}\right)$

The nice thing about choosing a Gaussian distribution for our prior is that the posterior distributions will be Gaussian, too (keyword [conjugate prior](https://en.wikipedia.org/wiki/Conjugate_prior))!

We will start our `BayesianLinearRegression` class with the knowledge we have so far - our probabilistic model. As mentioned in the beginning we assume that the variance $\sigma^2$ of the noise $\epsilon$ is known. Furthermore, to allow plotting the data later on we will assume that it's two dimensional (d=2).

```python
from scipy.stats import multivariate_normal
import numpy as np

class BayesianLinearRegression:
    """ Bayesian linear regression

    Args:
        prior_mean: Mean values of the prior distribution (m_0)
        prior_cov: Covariance matrix of the prior distribution (S_0)
        noise_var: Variance of the noise distribution
    """

    def __init__(self, prior_mean: np.ndarray, prior_cov: np.ndarray, noise_var: float):
        self.prior_mean = prior_mean[:, np.newaxis] # column vector of shape (1, d)
        self.prior_cov = prior_cov # matrix of shape (d, d)
        # We initalize the prior distribution over the parameters using the given mean and covariance matrix
        # In the formulas above this corresponds to m_0 (prior_mean) and S_0 (prior_cov)
        self.prior = multivariate_normal(prior_mean, prior_cov)

        # We also know the variance of the noise
        self.noise_var = noise_var # single float value
        self.noise_precision = 1 / noise_var

        # Before performing any inference the parameter posterior equals the parameter prior
        self.param_posterior = self.prior
        # Accordingly, the posterior mean and covariance equal the prior mean and variance
        self.post_mean = self.prior_mean # corresponds to m_N in formulas
        self.post_cov = self.prior_cov # corresponds to S_N in formulas

# Let's make sure that we can initialize our model
prior_mean = np.array([0, 0])
prior_cov = np.array([[0.5, 0], [0, 0.5]])
noise_var = 0.2
blr = BayesianLinearRegression(prior_mean, prior_cov, noise_var)
```

### 6.2 Generating a dataset

Before going any further we need a dataset to test our implementation. Remember that we assume that our targets were generated by a function of the form $y = \boldsymbol{\phi}^T(\boldsymbol{x}) \boldsymbol{\theta} + \epsilon$ where $\epsilon$ is normally distributed with mean $0$ and some known variance $\sigma^2$: $\epsilon \sim \mathcal{N}(0, \sigma^2)$.

To keep things simple we will work with one-dimensional data and simple linear regression (that is, no non-linear transformation of the inputs). Consequently, our data generating function will be of the form
$$ y = \theta_0 + \theta_1 \, x + \epsilon $$

Note that we added a parameter $\theta_0$ which corresponds to the intercept of the linear function. Until know we assumed $\theta_0 = 0$. As mentioned earlier, $\theta_1$ represents the slope of the linear function.

```python
import matplotlib.pyplot as plt
%matplotlib inline

def compute_function_labels(slope: float, intercept: float, noise_std_dev: float, data: np.ndarray) -> np.ndarray:
    """
    Compute target values given function parameters and data.

    Args:
        slope: slope of the function (theta_1)
        intercept: intercept of the function (theta_0)
        data: input feature values (x)
        noise_std_dev: standard deviation of noise distribution (sigma)

    Returns:
        target values, either true or corrupted with noise
    """
    n_samples = len(data)
    if noise_std_dev == 0: # Real function
        return slope * data + intercept
    else: # Noise corrupted
        return slope * data + intercept + np.random.normal(0, noise_std_dev, n_samples)
```

```python
# Set random seed to ensure reproducibility
seed = 42
np.random.seed(seed)

# Generate true values and noise corrupted targets
n_datapoints = 1000
intercept = -0.7
slope = 0.9
noise_std_dev = 0.5
noise_var = noise_std_dev**2
lower_bound = -1.5
upper_bound = 1.5

# Generate dataset
features = np.random.uniform(lower_bound, upper_bound, n_datapoints)
labels = compute_function_labels(slope, intercept, 0., features)
noise_corrupted_labels = compute_function_labels(slope, intercept, noise_std_dev, features)
```

```python
# Plot the dataset
plt.figure(figsize=(10,7))
plt.plot(features, labels, color='r', label="True values")
plt.scatter(features, noise_corrupted_labels, label="Noise corrupted values")
plt.xlabel("Features")
plt.ylabel("Labels")
plt.title("Real function along with noisy targets")
plt.legend();
```

### 6.3 Step 2: Posterior over the parameters

We finished setting up our probabilistic model. Next, we want to use this model and our dataset $\mathcal{X, Y}$ to estimate the parameter posterior $p(\boldsymbol{\theta} | \mathcal{X, Y})$. Keep in mind that we don't compute point estimates of the parameters. Instead, we determine the mean and variance of the (Gaussian) posterior distribution and use this entire distribution when making predictions.

We can estimate the parameter posterior using Bayes theorem:
$$
p(\boldsymbol{\theta} \mid \mathcal{X}, \mathcal{Y})=\frac{p(\mathcal{Y} \mid \mathcal{X}, \boldsymbol{\theta}) p(\boldsymbol{\theta})}{p(\mathcal{Y} \mid \mathcal{X})}
$$

- $p(\mathcal{Y} \mid \mathcal{X}, \boldsymbol{\theta})$ is the likelihood function, $p(\mathcal{Y} \mid \mathcal{X}, \boldsymbol{\theta})=\mathcal{N}\left(\boldsymbol{y} \mid \boldsymbol{\Phi} \boldsymbol{\theta}, \sigma^{2} \boldsymbol{I}\right)$
- $p(\boldsymbol{\theta})$ is the prior distribution, $p(\boldsymbol{\theta})=\mathcal{N}\left(\boldsymbol{\theta} \mid \boldsymbol{m}_{0}, \boldsymbol{S}_{0}\right)$
- $p(\mathcal{Y} \mid \mathcal{X})=\int p(\mathcal{Y} \mid \mathcal{X}, \boldsymbol{\theta}) p(\boldsymbol{\theta}) \mathrm{d} \boldsymbol{\theta}$ is the evidence which ensures that the posterior is normalized (that is, that it integrates to 1).

The parameter posterior can be estimated in closed form (for proof see theorem 9.1 in the book [Mathematics for Machine Learning](https://mml-book.com)):
$$
\begin{aligned}
p(\boldsymbol{\theta} \mid \mathcal{X}, \mathcal{Y}) &=\mathcal{N}\left(\boldsymbol{\theta} \mid \boldsymbol{m}_{N}, \boldsymbol{S}_{N}\right) \\
\boldsymbol{S}_{N} &=\left(\boldsymbol{S}_{0}^{-1}+\sigma^{-2} \boldsymbol{\Phi}^{\top} \boldsymbol{\Phi}\right)^{-1} \\
\boldsymbol{m}_{N} &=\boldsymbol{S}_{N}\left(\boldsymbol{S}_{0}^{-1} \boldsymbol{m}_{0}+\sigma^{-2} \boldsymbol{\Phi}^{\top} \boldsymbol{y}\right)
\end{aligned}
$$

Coming back to our `BayesLinearRegression` class we need to add a method which allows us to update the posterior distribution given a dataset.

```python
from scipy.stats import multivariate_normal
from scipy.stats import norm as univariate_normal
import numpy as np

class BayesianLinearRegression:
    """ Bayesian linear regression

    Args:
        prior_mean: Mean values of the prior distribution (m_0)
        prior_cov: Covariance matrix of the prior distribution (S_0)
        noise_var: Variance of the noise distribution
    """

    def __init__(self, prior_mean: np.ndarray, prior_cov: np.ndarray, noise_var: float):
        self.prior_mean = prior_mean[:, np.newaxis] # column vector of shape (1, d)
        self.prior_cov = prior_cov # matrix of shape (d, d)
        # We initalize the prior distribution over the parameters using the given mean and covariance matrix
        # In the formulas above this corresponds to m_0 (prior_mean) and S_0 (prior_cov)
        self.prior = multivariate_normal(prior_mean, prior_cov)

        # We also know the variance of the noise
        self.noise_var = noise_var # single float value
        self.noise_precision = 1 / noise_var

        # Before performing any inference the parameter posterior equals the parameter prior
        self.param_posterior = self.prior
        # Accordingly, the posterior mean and covariance equal the prior mean and variance
        self.post_mean = self.prior_mean # corresponds to m_N in formulas
        self.post_cov = self.prior_cov # corresponds to S_N in formulas

    def update_posterior(self, features: np.ndarray, targets: np.ndarray):
        """
        Update the posterior distribution given new features and targets

        Args:
            features: numpy array of features
            targets: numpy array of targets
        """
        # Reshape targets to allow correct matrix multiplication
        # Input shape is (N,) but we need (N, 1)
        targets = targets[:, np.newaxis]

        # Compute the design matrix, shape (N, 2)
        design_matrix = self.compute_design_matrix(features)

        # Update the covariance matrix, shape (2, 2)
        design_matrix_dot_product = design_matrix.T.dot(design_matrix)
        inv_prior_cov = np.linalg.inv(self.prior_cov)
        self.post_cov = np.linalg.inv(inv_prior_cov +  self.noise_precision * design_matrix_dot_product)

        # Update the mean, shape (2, 1)
        self.post_mean = self.post_cov.dot(
                         inv_prior_cov.dot(self.prior_mean) +
                         self.noise_precision * design_matrix.T.dot(targets))

        # Update the posterior distribution
        self.param_posterior = multivariate_normal(self.post_mean.flatten(), self.post_cov)

    def compute_design_matrix(self, features: np.ndarray) -> np.ndarray:
        """
        Compute the design matrix. To keep things simple we use simple linear
        regression and add the value phi_0 = 1 to our input data.

        Args:
            features: numpy array of features
        Returns:
            design_matrix: numpy array of transformed features

        >>> compute_design_matrix(np.array([2, 3]))
        np.array([[1., 2.], [1., 3.])
        """
        n_samples = len(features)
        phi_0 = np.ones(n_samples)
        design_matrix = np.stack((phi_0, features), axis=1)
        return design_matrix

    def predict(self, features: np.ndarray):
        """
        Compute predictive posterior given new datapoint

        Args:
            features: 1d numpy array of features
        Returns:
            pred_posterior: predictive posterior distribution
        """
        design_matrix = self.compute_design_matrix(features)

        pred_mean = design_matrix.dot(self.post_mean)
        pred_cov = design_matrix.dot(self.post_cov.dot(design_matrix.T)) + self.noise_var

        pred_posterior = univariate_normal(loc=pred_mean.flatten(), scale=pred_cov**0.5)
        return pred_posterior
```

### 6.4 Visualizing the parameter posterior

To ensure that our implementation is correct we can visualize how the posterior over the parameters changes as the model sees more data. We will visualize the distribution using a [contour plot](https://en.wikipedia.org/wiki/Contour_line) - a method for visualizing three-dimensional functions. In our case we want to visualize the density of our bi-variate Gaussian for each point (that is, each slope/intercept combination). The plot below shows an example which illustrates how the lines  and colours of a contour plot correspond to a Gaussian distribution:

As we can see, the density is highest in the yellow regions decreasing when moving further out into the green and blue parts. This should give you a better understanding of contour plots.

To analyze our Bayesian linear regression class we will start by initializing a new model. We can visualize its prior distribution over the parameters *before* the model has seen any real data.

```python
# Initialize BLR model
prior_mean = np.array([0, 0])
prior_cov = 1/2 * np.identity(2)
blr = BayesianLinearRegression(prior_mean, prior_cov, noise_var)

def plot_param_posterior(lower_bound, upper_bound, blr, title):
    fig = plt.figure()
    mesh_features, mesh_labels = np.mgrid[lower_bound:upper_bound:.01, lower_bound:upper_bound:.01]
    pos = np.dstack((mesh_features, mesh_labels))
    plt.contourf(mesh_features, mesh_labels, blr.param_posterior.pdf(pos), levels=15)
    plt.scatter(intercept, slope, color='red', label="True parameter values")
    plt.title(title)
    plt.xlabel("Intercept")
    plt.ylabel("Slope")
    plt.legend();

# Visualize parameter prior distribution
plot_param_posterior(lower_bound, upper_bound, blr, title="Prior parameter distribution")
```

The plot above illustrates both the prior parameter distribution and the true parameter values that we want to find. If our model works correctly, the posterior distribution should become more narrow and move closer to the true parameter values as the model sees more datapoints. This can be visualized with contour plots, too! Below we update the posterior distribution iteratively as the model sees more and more data. The contour plots for each step show how the parameter posterior develops and converges close to the true values in the end.

```python
n_points_lst = [1, 5, 10, 50, 100, 200, 500, 1000]
previous_n_points = 0
for n_points in n_points_lst:
    train_features = features[previous_n_points:n_points]
    train_labels = noise_corrupted_labels[previous_n_points:n_points]
    blr.update_posterior(train_features, train_labels)

    # Visualize updated parameter posterior distribution
    plot_param_posterior(lower_bound,
                         upper_bound,
                         blr,
                         title=f"Updated parameter distribution using {n_points} datapoints")

    previous_n_points = n_points
```

### 6.5 Step 3: Posterior predictive distribution

Given the posterior distribution over the parameters we can determine the predictive distribution (= posterior over the outputs) for a new input $(\boldsymbol{x}_*, y_*)$. This is the distribution we are really interested in. A trained model is not particularly useful when we can't use it to make predictions, right?

The posterior predictive distribution looks as follows:

$$
\begin{aligned}
p\left(y_{*} \mid \mathcal{X}, \mathcal{Y}, \boldsymbol{x}_{*}\right) &=\int p\left(y_{*} \mid \boldsymbol{x}_{*}, \boldsymbol{\theta}\right) p(\boldsymbol{\theta} \mid \mathcal{X}, \mathcal{Y}) \mathrm{d} \boldsymbol{\theta} \\
&=\int \mathcal{N}\left(y_{*} \mid \boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{*}\right) \boldsymbol{\theta}, \sigma^{2}\right) \mathcal{N}\left(\boldsymbol{\theta} \mid \boldsymbol{m}_{N}, \boldsymbol{S}_{N}\right) \mathrm{d} \boldsymbol{\theta} \\
&=\mathcal{N}\left(y_{*} \mid \boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{*}\right) \boldsymbol{m}_{N}, \boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{*}\right) \boldsymbol{S}_{N} \boldsymbol{\phi}\left(\boldsymbol{x}_{*}\right)+\sigma^{2}\right)
\end{aligned}
$$

First of all: note that the predictive posterior for a new input $\boldsymbol{x}_{*}$ is a *univariate* Gaussian distribution. We can see that the mean of the distribution is given by the product of the design matrix for the new example ($\boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{*}\right)$) and the mean of the parameter posterior ($\boldsymbol{m}_{N}$). The variance $(\boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{*}\right) \boldsymbol{S}_{N} \boldsymbol{\phi}\left(\boldsymbol{x}_{*}\right)+\sigma^{2}$) of the predictive posterior has two parts:
1. $\sigma^{2}$: The variance of the noise
2. $\boldsymbol{\phi}^{\top}\left(\boldsymbol{x}_{*}\right) \boldsymbol{S}_{N} \boldsymbol{\phi}\left(\boldsymbol{x}_{*}\right)$: The posterior uncertainty associated with the parameters $\boldsymbol{\theta}$

Let's add a `predict` method to our `BayesianLinearRegression` class which computes the predictive posterior for a new input (you will find the method in the class definition above):

```python
def predict(self, features: np.ndarray):
    """
    Compute predictive posterior given new datapoint

    Args:
        features: 1d numpy array of features
    Returns:
        pred_posterior: predictive posterior distribution
    """
    design_matrix = self.compute_design_matrix(features)

    pred_mean = design_matrix.dot(self.post_mean)
    pred_cov = design_matrix.dot(self.post_cov.dot(design_matrix.T)) + self.noise_var

    pred_posterior = univariate_normal(pred_mean.flatten(), pred_cov)
    return pred_posterior
```

### 6.6 Visualizing the predictive posterior

Our original dataset follows a simple linear function. After training the model it should be able to predict labels for new datapoints, even if they lie beyond the range from [-1.5, 1.5]. But how can we get from the predictive distribution that our model computes to actual labels? That's easy: we *sample* from the predictive posterior.

To make sure that we are all on the same page: given a new input example our Bayesian linear regression model predicts not a single label but a *distribution* over possible labels. This distribution is Gaussian. We can get actual labels by sampling from this distribution.

The code below implements and visualizes this:
- We create some test features for which we want predictions
- Each feature is given to the trained BLR model which returns a univariate Gaussian distribution over possible labels (`pred_posterior = blr.predict(np.array([feat]))`)
- We sample from this distribution (`sample_predicted_labels = pred_posterior.rvs(size=sample_size)`)
- The predicted labels are saved in a format that makes it easy to plot them
- Finally, we plot each input feature, its true label and the sampled predictions. Remember: the samples are generated from the predictive posterior returned by the `predict` method. Think of a Gaussian distribution plotted along the y-axis for each feature. We visualize this with a histogram: more likely values close to the mean will be sampled more often than less likely values.

```python
import pandas as pd
import seaborn as sns

all_rows = []
sample_size = 1000
test_features = [-2, -1, 0, 1, 2]
all_labels = []

for feat in test_features:
    true_label = compute_function_labels(slope, intercept, 0, np.array([feat]))
    all_labels.append(true_label)
    pred_posterior = blr.predict(np.array([feat]))
    sample_predicted_labels = pred_posterior.rvs(size=sample_size)
    for label in sample_predicted_labels:
        all_rows.append([feat, label])

all_data = pd.DataFrame(all_rows, columns=["feature", "label"])
sns.displot(data=all_data, x="feature", y="label")
plt.scatter(x=test_features, y=all_labels, color="red", label="True values")
plt.title("Predictive posterior distributions")
plt.legend()
plt.plot();
```

## Sources and further reading

The basis for this notebook is chapter 9.2 of the book [Mathematics for Machine Learning](https://mml-book.com). I can highly recommend to read through chapter 9 to get a deeper understanding of (Bayesian) linear regression.

You will find explanations and an implementation of simple linear regression in the [notebook on linear regression](https://github.com/zotroneneis/machine_learning_basics/blob/master/linear_regression.ipynb)

```python

```


# Regressão Linear Simples

```
import pandas as pd
```

```
pasta_raiz = '/content/drive/My Drive/datasets'
```

```
df = pd.read_csv(f'{pasta_raiz}/base_pulmao.csv')
df.head()
```

      taxa_fluxo_expiratorio
      qtde_anos_exposto

      0
      390
      0

      1
      410
      0

      2
      430
      0

      3
      460
      0

      4
      420
      1

```
df.describe()
```

      taxa_fluxo_expiratorio
      qtde_anos_exposto

      count
      122.000000
      122.000000

      mean
      365.655738
      14.081967

      std
      105.132641
      6.959850

      min
      110.000000
      0.000000

      25%
      300.000000
      7.000000

      50%
      365.000000
      17.000000

      75%
      430.000000
      20.000000

      max
      610.000000
      23.000000

```
%matplotlib inline
#se utiliza para versiones anteriores, en los mas nuevos no es necesario, era para decir a Jupiter renderizar su imagen
df.plot(kind='scatter', x='qtde_anos_exposto', y='taxa_fluxo_expiratorio')
```

```
X = df['qtde_anos_exposto'].copy()
y = df['taxa_fluxo_expiratorio'].copy()
```

```
import statsmodels.api as sm
```

    /usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```
X = sm.add_constant(X)
```

```
X.head()
```

      const
      qtde_anos_exposto

      0
      1.0
      0

      1
      1.0
      0

      2
      1.0
      0

      3
      1.0
      0

      4
      1.0
      1

```
modelo_rls = sm.regression.linear_model.OLS(y, X).fit()
# .fit para executar la instancia, hasta la parte de OLS(y, X) instancia para guardarlo en memoria
```

```
print(modelo_rls.summary());
```

                                  OLS Regression Results
    ==================================================================================
    Dep. Variable:     taxa_fluxo_expiratorio   R-squared:                       0.077
    Model:                                OLS   Adj. R-squared:                  0.069
    Method:                     Least Squares   F-statistic:                     9.974
    Date:                    Wed, 04 Nov 2020   Prob (F-statistic):            0.00201
    Time:                            02:27:25   Log-Likelihood:                -735.68
    No. Observations:                     122   AIC:                             1475.
    Df Residuals:                         120   BIC:                             1481.
    Df Model:                               1
    Covariance Type:                nonrobust
    =====================================================================================
                            coef    std err          t      P>|t|      [0.025      0.975]
    -------------------------------------------------------------------------------------
    const               424.5828     20.796     20.417      0.000     383.408     465.757
    qtde_anos_exposto    -4.1846      1.325     -3.158      0.002      -6.808      -1.561
    ==============================================================================
    Omnibus:                        0.767   Durbin-Watson:                   1.111
    Prob(Omnibus):                  0.681   Jarque-Bera (JB):                0.891
    Skew:                          -0.162   Prob(JB):                        0.641
    Kurtosis:                       2.734   Cond. No.                         35.7
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```
print(modelo_rls.summary2())
```

                        Results: Ordinary least squares
    ========================================================================
    Model:              OLS                    Adj. R-squared:     0.069
    Dependent Variable: taxa_fluxo_expiratorio AIC:                1475.3501
    Date:               2020-11-04 02:27       BIC:                1480.9581
    No. Observations:   122                    Log-Likelihood:     -735.68
    Df Model:           1                      F-statistic:        9.974
    Df Residuals:       120                    Prob (F-statistic): 0.00201
    R-squared:          0.077                  Scale:              10290.
    ------------------------------------------------------------------------
                           Coef.   Std.Err.    t    P>|t|   [0.025   0.975]
    ------------------------------------------------------------------------
    const                 424.5828  20.7960 20.4165 0.0000 383.4081 465.7575
    qtde_anos_exposto      -4.1846   1.3250 -3.1582 0.0020  -6.8079  -1.5612
    ------------------------------------------------------------------------
    Omnibus:                0.767           Durbin-Watson:             1.111
    Prob(Omnibus):          0.681           Jarque-Bera (JB):          0.891
    Skew:                   -0.162          Prob(JB):                  0.641
    Kurtosis:               2.734           Condition No.:             36
    ========================================================================

```
# outra forma de fazer a regressao linear
import statsmodels.formula.api as smf

modelo = smf.ols('taxa_fluxo_expiratorio ~ qtde_anos_exposto', data=df).fit()
print(modelo.summary())
```

                                  OLS Regression Results
    ==================================================================================
    Dep. Variable:     taxa_fluxo_expiratorio   R-squared:                       0.077
    Model:                                OLS   Adj. R-squared:                  0.069
    Method:                     Least Squares   F-statistic:                     9.974
    Date:                    Wed, 04 Nov 2020   Prob (F-statistic):            0.00201
    Time:                            02:27:29   Log-Likelihood:                -735.68
    No. Observations:                     122   AIC:                             1475.
    Df Residuals:                         120   BIC:                             1481.
    Df Model:                               1
    Covariance Type:                nonrobust
    =====================================================================================
                            coef    std err          t      P>|t|      [0.025      0.975]
    -------------------------------------------------------------------------------------
    Intercept           424.5828     20.796     20.417      0.000     383.408     465.757
    qtde_anos_exposto    -4.1846      1.325     -3.158      0.002      -6.808      -1.561
    ==============================================================================
    Omnibus:                        0.767   Durbin-Watson:                   1.111
    Prob(Omnibus):                  0.681   Jarque-Bera (JB):                0.891
    Skew:                          -0.162   Prob(JB):                        0.641
    Kurtosis:                       2.734   Cond. No.                         35.7
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

# Regressão linear multipla

```
dados = pd.read_excel(f'{pasta_raiz}/base_cheque.xls', sheet_name='cheque-especial')
dados.head()
```

      Código do Cliente
      Idade
      Rendimento\nTotal
      Salário
      Limite de Crédito Imediato
      Valor Total\ndo Patrimônio
      Limite do Cheque Especial

      0
      1
      81
      6800.0
      6800.0
      380
      299109
      2000

      1
      2
      35
      5000.0
      5000.0
      1000
      120000
      1000

      2
      3
      39
      6320.0
      6320.0
      1550
      100000
      1640

      3
      4
      70
      10736.0
      5214.0
      400
      100000
      500

      4
      5
      54
      6000.0
      6000.0
      1790
      171745
      3600

```
!pip install pyjanitor
```

    Requirement already satisfied: pyjanitor in /usr/local/lib/python3.6/dist-packages (0.20.10)
    Requirement already satisfied: pytest-azurepipelines in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.8.0)
    Requirement already satisfied: sphinxcontrib-fulltoc==1.2.0 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (1.2.0)
    Requirement already satisfied: lxml in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (4.2.6)
    Requirement already satisfied: isort>=4.3.18 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (5.6.4)
    Requirement already satisfied: pandas-flavor in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.2.0)
    Requirement already satisfied: pyspark in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (3.0.1)
    Requirement already satisfied: natsort in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (5.5.0)
    Requirement already satisfied: pre-commit in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (2.8.2)
    Requirement already satisfied: pytest-cov in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (2.10.1)
    Requirement already satisfied: xarray in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.15.1)
    Requirement already satisfied: jupyter-client in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (5.3.5)
    Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.11.0)
    Requirement already satisfied: interrogate in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (1.3.2)
    Requirement already satisfied: setuptools>=38.5.2 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (50.3.2)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.22.2.post1)
    Requirement already satisfied: unyt in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (2.8.0)
    Requirement already satisfied: hypothesis>=4.4.0 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (5.41.1)
    Requirement already satisfied: ipykernel in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (4.10.1)
    Requirement already satisfied: nbsphinx>=0.4.2 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.8.0)
    Requirement already satisfied: pytest>=3.4.2 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (3.6.4)
    Requirement already satisfied: black>=19.3b0 in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (20.8b1)
    Requirement already satisfied: pandas-vet in /usr/local/lib/python3.6/dist-packages (from pyjanitor) (0.2.2)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from pandas-flavor->pyjanitor) (1.1.3)
    Requirement already satisfied: py4j==0.10.9 in /usr/local/lib/python3.6/dist-packages (from pyspark->pyjanitor) (0.10.9)
    Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (5.3.1)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (2.0.0)
    Requirement already satisfied: importlib-resources; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (3.1.0)
    Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (1.5.0)
    Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (1.5.9)
    Requirement already satisfied: virtualenv>=20.0.8 in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (20.1.0)
    Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (3.2.0)
    Requirement already satisfied: toml in /usr/local/lib/python3.6/dist-packages (from pre-commit->pyjanitor) (0.10.1)
    Requirement already satisfied: coverage>=4.4 in /usr/local/lib/python3.6/dist-packages (from pytest-cov->pyjanitor) (5.3)
    Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from xarray->pyjanitor) (1.18.5)
    Requirement already satisfied: jupyter-core>=4.6.0 in /usr/local/lib/python3.6/dist-packages (from jupyter-client->pyjanitor) (4.6.3)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from jupyter-client->pyjanitor) (2.8.1)
    Requirement already satisfied: traitlets in /usr/local/lib/python3.6/dist-packages (from jupyter-client->pyjanitor) (4.3.3)
    Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.6/dist-packages (from jupyter-client->pyjanitor) (19.0.2)
    Requirement already satisfied: tornado>=4.1 in /usr/local/lib/python3.6/dist-packages (from jupyter-client->pyjanitor) (5.1.1)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->pyjanitor) (1.4.1)
    Requirement already satisfied: matplotlib>=2.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->pyjanitor) (3.2.2)
    Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from interrogate->pyjanitor) (20.2.0)
    Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from interrogate->pyjanitor) (7.1.2)
    Requirement already satisfied: colorama in /usr/local/lib/python3.6/dist-packages (from interrogate->pyjanitor) (0.4.4)
    Requirement already satisfied: py in /usr/local/lib/python3.6/dist-packages (from interrogate->pyjanitor) (1.9.0)
    Requirement already satisfied: tabulate in /usr/local/lib/python3.6/dist-packages (from interrogate->pyjanitor) (0.8.7)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->pyjanitor) (0.17.0)
    Requirement already satisfied: sympy>=1.2 in /usr/local/lib/python3.6/dist-packages (from unyt->pyjanitor) (1.6.2)
    Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /usr/local/lib/python3.6/dist-packages (from hypothesis>=4.4.0->pyjanitor) (2.2.2)
    Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from ipykernel->pyjanitor) (5.5.0)
    Requirement already satisfied: jinja2 in /usr/local/lib/python3.6/dist-packages (from nbsphinx>=0.4.2->pyjanitor) (2.11.2)
    Requirement already satisfied: nbformat in /usr/local/lib/python3.6/dist-packages (from nbsphinx>=0.4.2->pyjanitor) (5.0.8)
    Requirement already satisfied: docutils in /usr/local/lib/python3.6/dist-packages (from nbsphinx>=0.4.2->pyjanitor) (0.16)
    Requirement already satisfied: nbconvert!=5.4 in /usr/local/lib/python3.6/dist-packages (from nbsphinx>=0.4.2->pyjanitor) (5.6.1)
    Requirement already satisfied: sphinx>=1.8 in /usr/local/lib/python3.6/dist-packages (from nbsphinx>=0.4.2->pyjanitor) (1.8.5)
    Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=3.4.2->pyjanitor) (1.4.0)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=3.4.2->pyjanitor) (1.15.0)
    Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest>=3.4.2->pyjanitor) (0.7.1)
    Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest>=3.4.2->pyjanitor) (8.5.0)
    Requirement already satisfied: dataclasses>=0.6; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (0.7)
    Requirement already satisfied: appdirs in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (1.4.4)
    Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (3.7.4.3)
    Requirement already satisfied: mypy-extensions>=0.4.3 in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (0.4.3)
    Requirement already satisfied: regex>=2020.1.8 in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (2020.10.28)
    Requirement already satisfied: typed-ast>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (1.4.1)
    Requirement already satisfied: pathspec<1,>=0.6 in /usr/local/lib/python3.6/dist-packages (from black>=19.3b0->pyjanitor) (0.8.0)
    Requirement already satisfied: flake8>3.0.0 in /usr/local/lib/python3.6/dist-packages (from pandas-vet->pyjanitor) (3.8.4)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->pandas-flavor->pyjanitor) (2018.9)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->pre-commit->pyjanitor) (3.3.1)
    Requirement already satisfied: distlib<1,>=0.3.1 in /usr/local/lib/python3.6/dist-packages (from virtualenv>=20.0.8->pre-commit->pyjanitor) (0.3.1)
    Requirement already satisfied: filelock<4,>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from virtualenv>=20.0.8->pre-commit->pyjanitor) (3.0.12)
    Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from traitlets->jupyter-client->pyjanitor) (4.4.2)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets->jupyter-client->pyjanitor) (0.2.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2->seaborn->pyjanitor) (1.2.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2->seaborn->pyjanitor) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2->seaborn->pyjanitor) (2.4.7)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.2->unyt->pyjanitor) (1.1.0)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->pyjanitor) (0.7.5)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->pyjanitor) (1.0.18)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->pyjanitor) (4.8.0)
    Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->pyjanitor) (2.6.1)
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from ipython>=4.0.0->ipykernel->pyjanitor) (0.8.1)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from jinja2->nbsphinx>=0.4.2->pyjanitor) (1.1.1)
    Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.6/dist-packages (from nbformat->nbsphinx>=0.4.2->pyjanitor) (2.6.0)
    Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (1.4.2)
    Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (0.8.4)
    Requirement already satisfied: bleach in /usr/local/lib/python3.6/dist-packages (from nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (3.2.1)
    Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (0.3)
    Requirement already satisfied: testpath in /usr/local/lib/python3.6/dist-packages (from nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (0.4.4)
    Requirement already satisfied: defusedxml in /usr/local/lib/python3.6/dist-packages (from nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (0.6.0)
    Requirement already satisfied: requests>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (2.23.0)
    Requirement already satisfied: sphinxcontrib-websupport in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (1.2.4)
    Requirement already satisfied: babel!=2.0,>=1.3 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (2.8.0)
    Requirement already satisfied: imagesize in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (1.2.0)
    Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (20.4)
    Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (2.0.0)
    Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (0.7.12)
    Requirement already satisfied: pycodestyle<2.7.0,>=2.6.0a1 in /usr/local/lib/python3.6/dist-packages (from flake8>3.0.0->pandas-vet->pyjanitor) (2.6.0)
    Requirement already satisfied: pyflakes<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from flake8>3.0.0->pandas-vet->pyjanitor) (2.2.0)
    Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from flake8>3.0.0->pandas-vet->pyjanitor) (0.6.1)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel->pyjanitor) (0.2.5)
    Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.6/dist-packages (from pexpect; sys_platform != "win32"->ipython>=4.0.0->ipykernel->pyjanitor) (0.6.0)
    Requirement already satisfied: webencodings in /usr/local/lib/python3.6/dist-packages (from bleach->nbconvert!=5.4->nbsphinx>=0.4.2->pyjanitor) (0.5.1)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (2020.6.20)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.0.0->sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (3.0.4)
    Requirement already satisfied: sphinxcontrib-serializinghtml in /usr/local/lib/python3.6/dist-packages (from sphinxcontrib-websupport->sphinx>=1.8->nbsphinx>=0.4.2->pyjanitor) (1.1.4)

```
# limpando o nome das colunas
import janitor

dados = dados.clean_names().rename(columns={'rendimento\ntotal': 'rendimento_total', 'valor_total\ndo_patrimonio': 'valor_total_patrimonion'})
dados.head()
```

    /usr/local/lib/python3.6/dist-packages/distributed/config.py:20: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      defaults = yaml.load(f)

      codigo_do_cliente
      idade
      rendimento_total
      salario
      limite_de_credito_imediato
      valor_total_patrimonion
      limite_do_cheque_especial

      0
      1
      81
      6800.0
      6800.0
      380
      299109
      2000

      1
      2
      35
      5000.0
      5000.0
      1000
      120000
      1000

      2
      3
      39
      6320.0
      6320.0
      1550
      100000
      1640

      3
      4
      70
      10736.0
      5214.0
      400
      100000
      500

      4
      5
      54
      6000.0
      6000.0
      1790
      171745
      3600

```
y = dados['limite_do_cheque_especial'].copy()
X = dados.drop(['codigo_do_cliente', 'limite_do_cheque_especial'], axis=1)
X = sm.add_constant(X)

modelo_rlm = sm.regression.linear_model.OLS(y, X).fit()
print(modelo_rlm.summary())
```

                                    OLS Regression Results
    =====================================================================================
    Dep. Variable:     limite_do_cheque_especial   R-squared:                       0.845
    Model:                                   OLS   Adj. R-squared:                  0.827
    Method:                        Least Squares   F-statistic:                     45.92
    Date:                       Wed, 04 Nov 2020   Prob (F-statistic):           5.87e-16
    Time:                               02:27:49   Log-Likelihood:                -441.42
    No. Observations:                         48   AIC:                             894.8
    Df Residuals:                             42   BIC:                             906.1
    Df Model:                                  5
    Covariance Type:                   nonrobust
    ==============================================================================================
                                     coef    std err          t      P>|t|      [0.025      0.975]
    ----------------------------------------------------------------------------------------------
    const                      -2218.7812   1766.471     -1.256      0.216   -5783.663    1346.101
    idade                          6.9218     24.853      0.279      0.782     -43.233      57.077
    rendimento_total              -0.2697      0.163     -1.659      0.105      -0.598       0.058
    salario                        0.7154      0.169      4.245      0.000       0.375       1.056
    limite_de_credito_imediato     1.0696      0.405      2.642      0.012       0.253       1.886
    valor_total_patrimonion        0.0011      0.000      2.828      0.007       0.000       0.002
    ==============================================================================
    Omnibus:                        4.846   Durbin-Watson:                   2.282
    Prob(Omnibus):                  0.089   Jarque-Bera (JB):                3.865
    Skew:                           0.673   Prob(JB):                        0.145
    Kurtosis:                       3.344   Cond. No.                     1.14e+07
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 1.14e+07. This might indicate that there are
    strong multicollinearity or other numerical problems.

# Ajustando o modelo

```
y = dados['limite_do_cheque_especial'].copy()
X = dados.drop(['codigo_do_cliente', 'limite_do_cheque_especial', 'idade'], axis=1)
X = sm.add_constant(X)

modelo_rlm = sm.regression.linear_model.OLS(y, X).fit()
print(modelo_rlm.summary())
```

                                    OLS Regression Results
    =====================================================================================
    Dep. Variable:     limite_do_cheque_especial   R-squared:                       0.845
    Model:                                   OLS   Adj. R-squared:                  0.831
    Method:                        Least Squares   F-statistic:                     58.64
    Date:                       Wed, 04 Nov 2020   Prob (F-statistic):           7.41e-17
    Time:                               02:27:52   Log-Likelihood:                -441.47
    No. Observations:                         48   AIC:                             892.9
    Df Residuals:                             43   BIC:                             902.3
    Df Model:                                  4
    Covariance Type:                   nonrobust
    ==============================================================================================
                                     coef    std err          t      P>|t|      [0.025      0.975]
    ----------------------------------------------------------------------------------------------
    const                      -1821.4240   1030.359     -1.768      0.084   -3899.342     256.494
    rendimento_total              -0.2672      0.161     -1.664      0.103      -0.591       0.057
    salario                        0.7138      0.167      4.284      0.000       0.378       1.050
    limite_de_credito_imediato     1.0767      0.400      2.694      0.010       0.271       1.883
    valor_total_patrimonion        0.0011      0.000      2.849      0.007       0.000       0.002
    ==============================================================================
    Omnibus:                        5.285   Durbin-Watson:                   2.264
    Prob(Omnibus):                  0.071   Jarque-Bera (JB):                4.335
    Skew:                           0.717   Prob(JB):                        0.114
    Kurtosis:                       3.333   Cond. No.                     6.70e+06
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 6.7e+06. This might indicate that there are
    strong multicollinearity or other numerical problems.

```
y = dados['limite_do_cheque_especial'].copy()
X = dados.drop(['codigo_do_cliente', 'limite_do_cheque_especial', 'idade', 'rendimento_total'], axis=1)
X = sm.add_constant(X)

modelo_rlm = sm.regression.linear_model.OLS(y, X).fit()
print(modelo_rlm.summary())
```

                                    OLS Regression Results
    =====================================================================================
    Dep. Variable:     limite_do_cheque_especial   R-squared:                       0.835
    Model:                                   OLS   Adj. R-squared:                  0.824
    Method:                        Least Squares   F-statistic:                     74.28
    Date:                       Wed, 04 Nov 2020   Prob (F-statistic):           2.97e-17
    Time:                               02:27:55   Log-Likelihood:                -442.97
    No. Observations:                         48   AIC:                             893.9
    Df Residuals:                             44   BIC:                             901.4
    Df Model:                                  3
    Covariance Type:                   nonrobust
    ==============================================================================================
                                     coef    std err          t      P>|t|      [0.025      0.975]
    ----------------------------------------------------------------------------------------------
    const                      -2170.9877   1028.808     -2.110      0.041   -4244.414     -97.562
    salario                        0.4571      0.064      7.115      0.000       0.328       0.587
    limite_de_credito_imediato     1.1467      0.405      2.829      0.007       0.330       1.963
    valor_total_patrimonion        0.0006      0.000      2.373      0.022    9.48e-05       0.001
    ==============================================================================
    Omnibus:                        3.894   Durbin-Watson:                   2.122
    Prob(Omnibus):                  0.143   Jarque-Bera (JB):                3.047
    Skew:                           0.343   Prob(JB):                        0.218
    Kurtosis:                       4.026   Cond. No.                     6.56e+06
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 6.56e+06. This might indicate that there are
    strong multicollinearity or other numerical problems.


$$
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert#1\right\rvert}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\inner}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\bra}[1]{\left\langle#1\right|}
\newcommand{\ket}[1]{\left|#1\right\rangle}
\newcommand{\braket}[2]{\left\langle#1|#2\right\rangle}
\newcommand{\ketbra}[2]{\left|#1\right\rangle\left\langle#2\right|}
\newcommand{\angleset}[1]{\left\langle#1\right\rangle}
$$

# Logic Gates

_prepared by Israel Gelover_

### Definition 1.3 Bit
A bit is a variable that can take one of two possible values and is the minimum unit of classical information.
$$ b \in \set{0,1} $$

#### Example:
- In an electrical circuit, we can have a terminal at 0V (bit 0) or at 5V (bit 1).
- A LED can be off (bit 0) or on (bit 1).

## Logic Gates

Bit manipulatation happens through a logic gate paradigm.

### Definition 1.4 Logic Gate
A logic gate is a function
$$ f:\set{0,1}^n \to \set{0,1}^m \enspace n,m \in \mathbb{N} $$

There are several ways to get to the paradigm of classical computing, one of the best known is through Turing machines. The set of computable functions (operations that can be done) with the formalization of Turing machines turns out to be the same set of computable functions that is obtained through the logic gates paradigm.

#### Example:

- Identity
\begin{equation*}
    \begin{split}
        \mbox{I}:\set{0,1} &\to \set{0,1} \\
        a &\to a
    \end{split}
\end{equation*}
- Not
\begin{equation*}
    \begin{split}
        \mbox{Not}:\set{0,1} &\to \set{0,1} \\
        a &\to \overline{a} = 1-a
    \end{split}
\end{equation*}
- And
\begin{equation*}
    \begin{split}
        \land:\set{0,1}^2 &\to \set{0,1} \\
        (a,b) &\to a \land b = a \cdot b
    \end{split}
\end{equation*}
- Or
\begin{equation*}
    \begin{split}
        \lor:\set{0,1}^2 &\to \set{0,1} \\
        (a,b) &\to a \lor b = a + b - a \cdot b
    \end{split}
\end{equation*}
- XOr
\begin{equation*}
    \begin{split}
        \oplus:\set{0,1}^2 &\to \set{0,1} \\
        (a,b) &\to a \oplus b = a + b \mod{2} = a + b - 2ab
    \end{split}
\end{equation*}
- NAnd = Not $\circ$ And
- NOr = Not $\circ$ Or
- Copy
\begin{equation*}
    \begin{split}
        \mbox{Copy}:\set{0,1} &\to \set{0,1}^2 \\
        a &\to (a,a)
    \end{split}
\end{equation*}
- Swap
\begin{equation*}
    \begin{split}
        \mbox{Swap}:\set{0,1}^2 &\to \set{0,1}^2 \\
        (a,b) &\to (b,a)
    \end{split}
\end{equation*}
- Toffoli
\begin{equation*}
    \begin{split}
        \mbox{Toffoli}:\set{0,1}^3 \rightarrow \set{0,1}^3 \\\\
        \mbox{Toffoli}(0,0,0) = (0,0,0), \\
        \mbox{Toffoli}(0,0,1) = (0,0,1), \\
        \mbox{Toffoli}(0,1,0) = (0,1,0), \\
        \mbox{Toffoli}(0,1,1) = (0,1,1), \\
        \mbox{Toffoli}(1,0,0) = (1,0,0), \\
        \mbox{Toffoli}(1,0,1) = (1,0,1), \\
        \mbox{Toffoli}(1,1,0) = (1,1,1), \\
        \mbox{Toffoli}(1,1,1) = (1,1,0). \\
    \end{split}
\end{equation*}
where $\circ$ denotes the usual function composition.

### Theorem 1.5 De Morgan's Laws

1. $\overline{a \land b} = \overline{a} \lor \overline{b}$
    $Not(And(a,b)) = Or(Not(a), Not(b))$

2. $\overline{a \lor b} = \overline{a} \land \overline{b}$
    $Not(Or(a,b)) = And((Not(a), Not(b)))$

_Proof:_
From the definition of Not, And y Or, we have
1.
\begin{equation*}
    \begin{split}
        \overline{a} \lor \overline{b} &= \overline{a} + \overline{b} - (\overline{a} \cdot \overline{b}) \\
        &= (1 - a) + (1 - b) - (1 - a)(1 - b) \\
        &= 1 - a + 1 - b - (1 - b - a + (a \cdot b)) \\
        &= 1 - a + 1 - b - 1 + b + a - (a \cdot b) \\
        &= 1 - (a \cdot b) \\
        &= \overline{a \land b}
    \end{split}
\end{equation*}
2.
\begin{equation*}
    \begin{split}
        \overline{a \lor b} &= 1 - (a + b - (a \cdot b)) \\
        &= 1 - a - b + (a \cdot b) \\
        &= (1 - a) - b(1-a) \\
        &= (1 - a)(1 - b) \\
        &= \overline{a} \land \overline{b}
    \end{split}
\end{equation*}

### Proposition 1.6
$$
XOr(a,b) = a \oplus b = (\overline{\overline{a} \land \overline{b}}) \land (\overline{a \land b})
$$

### Task 1.

(On paper) Write a proof for Proposition 1.6

Click here for solution

### Theorem 1.7 Conjunctive normal form
Any logic gate $f:\set{0,1}^n \to \set{0,1}^m$ can be built using gates And and Not.

_Proof:_ Let $f:\set{0,1}^n \to \set{0,1}^m$ be a logic gate.
We can express $f = (f_1, f_2, \dots, f_m)$ where each $f_j:\set{0,1}^n \to \set{0,1}, j \in \set{1, \dots, m}$

Let $k \in \set{1, \dots, m}$, we can define $S_k = \set{\overline{a} \in \set{0,1}^n \mid f_k(\overline{a}) = 1}$
Given $\overline{a}^{(l_k)} \in S_k$ with $l_k \in \set{1, \dots, \abs{S_k}}$, let us define

\begin{equation*}
    \begin{split}
        f_k^{(l_k)}:\set{0,1}^n &\to \set{0,1}\\
        \overline{a} &\to b_1 \land b_2 \land \dots \land b_n
    \end{split}
\end{equation*}

Where
$b_j = \begin{cases}
    a_j & \text{if } a_j^{(l_k)} = 1 \\
    \overline{a_j} & \text{if } a_j^{(l_k)} = 0
\end{cases}$

Note that $f_k^{(l_k)}\left(\overline{a}^{(l_k)}\right) = 1$ and $f_k^{(l_k)}\left(\overline{a}\right) = 0 \enspace \forall \overline{a} \neq \overline{a}^{(l_k)}$

(1)

\begin{equation}
    \implies \overbrace{\underbrace{f_k(\overline{a})}_{0 \enspace \overline{a} \not\in S_k}}^{1 \enspace \overline{a} \in S_k} = \overbrace{\underbrace{f_k^{(1)}(\overline{a}) \lor f_k^{(2)}(\overline{a}) \lor \dots \lor f_k^{(\abs{S_k})}(\overline{a})}_{\text{all equal }0}}^{\text{some equal }1}
\end{equation}

Because

\begin{equation*}
    \begin{split}
        \text{if } \overline{a} \in S_k &\implies \exists l_k \text{ such that } f_k^{(l_k)}(\overline{a}) = 1\\
        \text{if } \overline{a} \not\in S_k &\implies \forall l_k, f_k^{(l_k)}(\overline{a}) = 0\\
    \end{split}
\end{equation*}

Equation (1) tells us that we have constructed $f_k$ using Or and $f_k^{(l_k)}$ gates. On the other hand, each gate $f_k^{(l_k)}$ was built using only And and Not gates. And by De Morgan's Laws, Or can be built using only And and Not gates, and vice versa, we can build the And gate using only gates Or and Not.

**Remark:** The set {And, Not} is called a universal set of logic gates. The set {Or, Not} is also universal and is known as the _Disjunctive normal form_.

### Proposition 1.8

And and Not gates can be implemented using the Toffoli gate; that is, this gate by itself forms a computationally
universal set.

### Task 2.

(On paper) Write a proof for Proposition 1.8

Click here for solution


```python
% matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import colors, cm
import numpy as np
from numpy import matmul
from scipy.spatial.distance import pdist, squareform
from sklearn.datasets import load_diabetes
import pandas as pd
from scipy.linalg import cholesky
from scipy.linalg import solve
from scipy.optimize import minimize
import time
```

```python
# Developer notes
# 1) Cholesky decomposition produces NaNs (probably because K+I*s**2 is not pos semidef) causing solve to complain
# 2) Including gradient for likelihood made optimization much faster

class mintGP():
    """
    The implementation is based on Algorithm 2.1 of Gaussian Processes
    for Machine Learning (GPML) by Rasmussen and Williams.

    Takes 2D np-arrays
    """

    def __init__(self):
        pass

    def fit(self, X, Y):
        self.yscale = np.std(Y)
        self.Y = Y/self.yscale
        self.X = X
        self.n = np.shape(X)[0]
        # initialize with heuristics
        self.lengthscale = np.mean(pdist(X, metric='euclidean'))
        self.likelihood_variance =  1

        ###############################################################
        # Gradient descent on marginal likelihood with scipy L-BFGS-B #
        ###############################################################
        theta0 = np.array([self.lengthscale, self.likelihood_variance])
        bnds = ((1e-20, None), (1e-10, None))
        sol = minimize(self.neg_log_marg_like, theta0, args=(),
                       method='L-BFGS-B', bounds=bnds, jac=True)
        self.lengthscale, self.likelihood_variance = sol.x
        self.marginal_likelihood = np.exp(-sol.fun)

        # for prediction:
        K,_ = self.K(X,X,self.lengthscale)
        self.L = cholesky( K + self.likelihood_variance*np.eye(self.n), lower=True)
        print(sol.x, theta0)

    ##########################
    # Likelihood computation #
    ##########################
    def neg_log_marg_like(self, theta):
        """
        Compute negative log marginal likelihood for hyperparameter optimization
        """
        jitter=0
        K, D = self.K(self.X ,self.X, theta[0])
        L = cholesky( K + (theta[1]+jitter)*np.eye(self.n), lower=True)
        self.L = L
        alpha = solve(L.T, solve(L,self.Y, lower=True) )
        logmarglike = \
        - 0.5*matmul(self.Y.T, alpha)[0,0]     \
        - np.sum( np.log( np.diag( L ) ) )   \
        - 0.5*self.n*np.log(2*np.pi)

        # compute gradients
        prefactor = matmul(alpha, alpha.T) - solve(L.T, solve(L, np.eye(self.n) ) )
        Kd_lengthscale = np.multiply( D/theta[0]**3, K)
        Kd_likelihood_variance = np.eye(self.n)
        logmarglike_grad = 0.5*np.array( [ np.trace( matmul(prefactor, Kd_lengthscale) ),
                                           np.trace( matmul(prefactor, Kd_likelihood_variance) )] )

        return -logmarglike, -logmarglike_grad

    def nlml_grad(self):
        """
        Return gradient of negative log marginal likelihood
        """
        return self.logmarglike_grad

    ######################
    # Kernel computation #
    ######################
    def K(self, X, Z, lengthscale):
        n1 = np.shape(X)[0]
        n2 = np.shape(Z)[0]
        n1sq = np.sum(np.square(X), 1)
        n2sq = np.sum(np.square(Z), 1)

        D = (np.ones([n2, 1])*n1sq).T + np.ones([n1, 1])*n2sq -2*matmul(X,Z.T)
        return np.exp(-D/(2*lengthscale**2)), D

    def scalarK(self, x, z, lengthscale):
        return( np.exp( np.linalg.norm(x - z)**2/(2*lengthscale**2) ) )

    ###########################
    # Predictive distribution #
    ###########################
    def predict(self, Xnew, predvar=False):
        alpha = solve(self.L.T, solve(self.L,self.Y*self.yscale ) )
        if predvar:
            m = np.shape(Xnew)[0]
            Knew_N,_ = self.K(Xnew, self.X, self.lengthscale)
            Knew_new = np.array( [self.scalarK(Xnew[i], Xnew[i], self.lengthscale) for i in range(m)] ).reshape([m,1])
            v = solve(self.L, Knew_N.T)
            return matmul(Knew_N, alpha), np.diag( Knew_new + self.likelihood_variance - matmul(v.T, v) ).reshape(m,1)
        else:
            Knew_N,_ = self.K(Xnew, self.X, self.lengthscale)
            return matmul(Knew_N, alpha)

    ###############################
    # Gradient of predictive mean #
    ###############################
    def predictive_grad(self, Xnew):
        alpha = solve(self.L.T, solve(self.L, self.Y*self.yscale ) )
        Knew_N,_ = self.K(Xnew, self.X, self.lengthscale)

        return (-1/self.lengthscale**2)*matmul( np.tile(Xnew.T, self.n) - self.X.T, np.multiply(Knew_N.T, alpha) )
```

## 1D Toy example with missing data, gradient computation, likelihood surface plot

```python
N = 30

Nt = 400
X = np.linspace(-4,5,N).reshape(N,1);
# We can pick out some values to illustrate how the uncertainty estimate behaves
# it's interesting to see what happens to likelihood below

ind = np.bool8([1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1]); X = X[ ind ]; N=N-sum(~ind)
Xt = np.linspace(-4,5,Nt).reshape(Nt,1)

Y = np.sin(X)*np.exp(0.2*X) + np.random.randn(N,1)*0.3

t0 = time.time()

m = mintGP()
m.fit(X,Y)
print( m.lengthscale, m.likelihood_variance )

pred, var = m.predict(Xt,predvar=True)
t1 = time.time()
print('time to compute ',t1-t0)

fig, ax = plt.subplots()
ax.plot(Xt, pred, label="GP mean")
twostd = 2*np.sqrt(var)

ax.fill_between(Xt.ravel(), (pred-twostd).ravel(), (pred+twostd).ravel(), alpha=0.5)
ax.scatter(X,Y,label='data')
ax.legend(loc='best')

```

### Gradient

```python
grad = [ m.predictive_grad(x.reshape(1,1)) for x in Xt ]
grad = np.array(grad)
fig, ax = plt.subplots()
ax.plot(Xt, grad.ravel(), label="GP deriv")
ax.plot([-4,5], [0,0], label="GP deriv")

```

### Likelihood Surface

```python
#### Plot LML landscape
plt.figure(1)
M = 30
theta0 = np.logspace(-0.3, 0.4,M)#np.logspace(-1, 1, M)
theta1 = np.logspace(-1.5, 0, M)#np.logspace(-2.5, 0, M)
Theta0, Theta1 = np.meshgrid(theta0, theta1)
LML = [[m.neg_log_marg_like([Theta0[i, j], Theta1[i, j]])[0]
        for i in range(M)] for j in range(M)]
LML = np.array(LML).T

vmin, vmax = (LML).min(), (LML).max()
vmax = 50
level = np.around(np.logspace(np.log10(vmin), np.log10(vmax), 50), decimals=1)
plt.contour(Theta0, Theta1, LML,
            levels=level, norm=colors.LogNorm(vmin=vmin, vmax=vmax))
plt.colorbar()
plt.xscale("log")
plt.yscale("log")
plt.xlabel("Length-scale")
plt.ylabel("Noise-level")
plt.title("neg-log-marginal-likelihood")
plt.tight_layout()
```

### likehood surface gradient

```python
import sympy

# Plot LML landscape
plt.figure(1)
LML_grad = [[ m.neg_log_marg_like([Theta0[i, j], Theta1[i, j]])[1]
        for i in range(M)] for j in range(M)]
LML_grad = -np.array(LML_grad).T

plt.figure()
plt.quiver(Theta0,Theta1,LML_grad[0],LML_grad[1])
plt.xscale("log")
plt.yscale("log")
plt.show()
```

## 2D toy example

```python
N = 100
# training data
Nt = 400
X1 = np.random.uniform(-5,5,size = (N,1)) #np.linspace(-4,5,N).reshape(N,1)
X2 = np.random.uniform(-5,5,size = (N,1))
X = np.concatenate([X1,X2],1)
# test data
Xt = np.concatenate([np.linspace(-4,5,Nt).reshape(Nt,1), np.linspace(-4,5,Nt).reshape(Nt,1)], 1)
Y = X1**2 + X2**2 + np.random.randn(N,1)*0.3

t0 = time.time()
m = mintGP()
m.fit(X,Y)
print( m.lengthscale, m.likelihood_variance )
#pred, var = m.predict(Xt,predvar=True)
t1 = time.time()
print('time to compute ',t1-t0)

```

    [  3.46386299e+00   1.01896404e-03] [ 5.21543278  1.        ]
    3.46386298728 0.00101896404276
    time to compute  0.07127761840820312

```python
M = 50
grid = np.linspace(-5,5,M).reshape(M,1)
XX1,XX2 = np.meshgrid(grid,grid)
Z = [[m.predict( np.array([XX1[i,j], XX2[i,j] ]).reshape(1,2) )[0,0] for i in range(M)] for j in range(M)]
Z = np.array(Z)
# plot points and fitted surface
fig = plt.figure()
ax = fig.gca(projection='3d')

ax.plot_surface(XX1, XX2, Z, rstride=1, cstride=1, alpha=0.2)
ax.scatter(X1, X2, Y, c='r', s=30)
plt.xlabel('X')
plt.ylabel('Y')
ax.set_zlabel('Z')
ax.axis('equal')
#ax.axis('tight')
```

### Gradient as colorcode like in Emans sw33t pløts

```python
# Color function by norm of gradient

fig = plt.figure(figsize=[8,5])
ax = fig.gca(projection='3d')

my_col = cm.jet( gradnorm / np.amax(gradnorm) )
cbar = cm.ScalarMappable(cmap=cm.jet)
cbar.set_array(my_col)
surf = ax.plot_surface(XX1, XX2, Z, rstride=2, cstride=2, alpha=0.5, facecolors = my_col, linewidth = 0 )
fig.colorbar(cbar, shrink=0.8, aspect=8)
```

# Tensordot experiments for gradient function to be able to take NxD arrays

```python
a = np.arange(60.).reshape(3,4,5)
b = np.arange(24.).reshape(4,3,2)
c = np.tensordot(a,b, axes=([1,0],[0,1]))
c.shape
```

    (5, 2)

```python
c
```

    array([[ 4400.,  4730.],
           [ 4532.,  4874.],
           [ 4664.,  5018.],
           [ 4796.,  5162.],
           [ 4928.,  5306.]])

```python
# A slower but equivalent way of computing the same
d = np.zeros((5,2))
for i in range(5):
    for j in range(2):
        for k in range(3):
            for n in range(4):
                d[i,j] += a[k,n,i] * b[n,k,j]
c == d
```

    array([[ True,  True],
           [ True,  True],
           [ True,  True],
           [ True,  True],
           [ True,  True]], dtype=bool)

```python
xnew = np.array([[4,9],[1,8]], ndmin=2)
```


# Introduction

The galactic centre holds many mysteries, one of which is the flares. These flares occur on a daily basis, and last for a few hours at a time. Their luminosity is around $10^{34.5}$ erg/s, which translates to a total energy of around $10^{38}$ erg. One idea proposed to explain these flares is the [ablation of asteroids](http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1110.6872). It has been shown that this mechanism can deposit enough energy within the relevant time frame. In this notebook we explore how fast can this energy be radiated.

# Review of Wet Disruption

Let us consider an asteroid with size $r_a$ and mass density $\rho_a$. Suppose that it is moving radially toward a supermassive black hole with mass $M_h$. Suppose further that the number density of the diffuse matter varies with distance to the black hole as $n_d = k r^{-\omega}$, where $\omega = 3/2$, and the coefficient $k$ can be calibrated from the [Chandra measurements](http://iopscience.iop.org/article/10.1086/375145/fulltext/53430.text.html). As a result of the collision between the asteroid and the ambient medium, the asteroid heats up. We consider a monolithic, solid asteroid, and so to vaporise it, the energy per unit mass necessary is approximately given by the speed of elastic waves squared $c_e^2$. We proceed to calculate the radius at which enough energy is released due to the interaction between the asteroid and the ambient medium to vaporise the asteroid

```python
import sympy
sympy.init_printing()
k = sympy.Symbol('k') # Ambient density profile prefactor
omega = sympy.Symbol('omega') # Ambient density profile exponent
r = sympy.Symbol('r') # Distance from the black hole
n = sympy.Symbol('n') # Density of diffuse gas
c_e = sympy.Symbol('c_e') # Elastic wave speed
r_a = sympy.Symbol('r_a') # Asteroid size
rho_a = sympy.Symbol('rho_a') # Asteroid density
G = sympy.Symbol('G') # Graviation constant
M_h = sympy.Symbol('M_h') # Black hole mass
m_p = sympy.Symbol('m_p') # Proton mass
v_k = sympy.Symbol('v_k') # Keplerian velocity
temp = sympy.Eq(c_e**2*r_a**3*rho_a, m_p*n*v_k**3*r_a**2*r/v_k)
temp = temp.subs(n, k*r**(-omega))
temp = temp.subs(v_k, sympy.sqrt(G*M_h/r))
destruction_radius = sympy.solve(temp, r)[0]
destruction_radius
```

Substituting typical numbers

```python
import scipy.constants
from sympy.physics.units import meter, kg, second, km, cm, kelvin, joule, gram
M_sun = 2e30*kg
pc = 3e18*cm
au = 1.5e13*cm
r_ch = sympy.Symbol('r_ch') # Radius of chandra measurements
n_ch = sympy.Symbol('n_ch') # Number density in the chandra measurement
temp = destruction_radius
temp = temp.subs(sympy.solve(sympy.Eq(n_ch, k*r_ch**(-omega)),k,dict=True)[0])
temp = temp.subs({G:scipy.constants.G*meter**3/kg/second**2,
                  omega:sympy.Rational(3,2),
                  c_e: 1e3*km/second,
                  M_h: 4e6*M_sun,
                  m_p:scipy.constants.m_p*kg,
                  n_ch: 130/cm**3,
                  r_ch: 0.04*pc,
                  r_a: 1.0*km,
                  rho_a: kg/meter**3
                 })
temp = temp.subs(km, 1000*meter)
temp = temp.subs(meter, 100*cm)
temp/au
```

This is somewhat smaller than the value used in Zubovas et al 2011 (1 au). In the remainder of the notebook I'll still use the canonical value of 1 au, though.

Also, I underestimate the speed of elastic waves. According to [Kraus et al 2012](http://onlinelibrary.wiley.com/doi/10.1029/2012JE004082/full), the speed should be closer to 5 km/s.

```python
temp = (5130*kelvin)*(5150*joule/kg/kelvin)
temp = temp.subs(joule, kg*meter**2/second**2)
sympy.sqrt(temp).n()
```

Available kinetic energy at the destruction radius

```python
erg = gram*cm**2/second**2
temp = G*M_h*rho_a*r_a**3/destruction_radius
temp = temp.subs(sympy.solve(sympy.Eq(n_ch, k*r_ch**(-omega)),k,dict=True)[0])
temp = temp.subs({G:scipy.constants.G*meter**3/kg/second**2,
                  omega:sympy.Rational(3,2),
                  c_e: 1e3*km/second,
                  M_h: 4e6*M_sun,
                  m_p:scipy.constants.m_p*kg,
                  n_ch: 130/cm**3,
                  r_ch: 0.04*pc,
                  r_a: 10.0*km,
                  rho_a: kg/meter**3
                 })
temp = temp.subs(km, 1000*meter)
temp = temp.subs(meter, 100*cm)
temp = temp.subs(kg, 1000*gram)
temp/erg
```

This energy is also much smaller than the typical energy in a flare ($10^{38}$ erg)

# Radiation Efficiency

Let's ignore the problems in the previous section and suppose that enough energy ($10^{38}$ erg) is injected into a medium with a density $10^8 \, \rm cm^{-3}$. This is achieved by converting the kinetic energy of the asteroid to thermal energy, and so the thermal velocity of the particles is equal to the keplerian velocity $0.1 c$.

In this sections we explore four emission channels: Bremsstrahlung, inverse Compton, pair produciton and synchrotron emission.

## Bremsstrahlung

The cooling time of plasma heated up to thermal velocities of $0.1 c$ is

```python
year = 1e7*second
k_b = sympy.Symbol('k_b') # Boltzmann constant
T = sympy.Symbol('T') # Temperature
alpha = sympy.Symbol('alpha') # Fine structure constant
m_e = sympy.Symbol('m_e') # Electron mass
c = sympy.Symbol('c') # Speed of light
r_e = sympy.Symbol('r_e') # Classical electron radius
temp = n*k_b*T/(alpha*m_e*c**3*(n*r_e**3)**2*sympy.sqrt(k_b*T/m_e/c**2)/r_e**4)
temp = temp.subs(T, m_p*(0.1*c)**2/k_b)
temp = temp.subs({alpha:scipy.constants.alpha,
                 n:1e8/cm**3,
                 m_p:scipy.constants.m_p*kg,
                 m_e:scipy.constants.m_e*kg,
                 c:scipy.constants.c*meter/second,
                 r_e:scipy.constants.physical_constants['classical electron radius'][0]*meter})
temp = temp.subs(meter, 100*cm)
temp/year
```

## Inverse Compton

The [bolometric luminosity](https://arxiv.org/ftp/arxiv/papers/1003/1003.1519.pdf) from SGR A\* is $10^{35}$ erg/s. These photons can interact with the hot plasma and carry away energy. We calculate the cooling time from inverse compton

```python
L_h = sympy.Symbol('L_h')
temp = n*k_b*T/(r_e**2*L_h/r**2)/n/(k_b*T/m_e/c**2)**2
temp = temp.subs(T, m_p*(0.1*c)**2/k_b)
temp = temp.subs({alpha:scipy.constants.alpha,
                 n:1e8/cm**3,
                 r: au,
                 L_h: 1e35*erg/second,
                 m_p:scipy.constants.m_p*kg,
                 m_e:scipy.constants.m_e*kg,
                 c:scipy.constants.c*meter/second,
                 r_e:scipy.constants.physical_constants['classical electron radius'][0]*meter})
temp = temp.subs(kg, 1000*gram)
temp/year
```

## Pair production

Pair production can increase the number of charges in the plasma, and thus shorten the cooling time. However, with the energy available, pair production can only increase the density by a factor of

```python
L_h = sympy.Symbol('L_h')
temp = (k_b*T/m_e/c**2)
temp = temp.subs(T, m_p*(0.1*c)**2/k_b)
temp = temp.subs({alpha:scipy.constants.alpha,
                 n:1e8/cm**3,
                 r: au,
                 L_h: 1e35*erg/second,
                 m_p:scipy.constants.m_p*kg,
                 m_e:scipy.constants.m_e*kg,
                 c:scipy.constants.c*meter/second,
                 r_e:scipy.constants.physical_constants['classical electron radius'][0]*meter})
temp = temp.subs(kg, 1000*gram)
temp
```

Even if the Bremsstrahlung cooling time would be shorter by this factor, it would still be about 10 years, and hence much larger than the few hours time scale.

Pair production affect inverse Compton in a complex way. On the one hand, it increases the number of scatterers, which shortens the cooling time. On the other hand, it reduces the Lorentz factor, and hence increases the cooling time. The second effect is stronger, and hence the net effect is to increase the cooling time.

Pair production becomes important once the positron density becomes comparable to the baryon density. However, for this to happen, the hot gas has to first radiate 5% ($m_e c^2/ k T$) of its energy. From the discussion in the previous sections, this cannot happen on the flare time scale.

## Synchrotron radiation

The [magnetic field within a few au from SGR A\*](https://arxiv.org/pdf/1512.01220.pdf) is estimated to be around a few tens of gauss. The cooling time from synchrotron emission would be

```python
hour = 3600*second
B = sympy.Symbol('B') # Magnetic field
gauss = sympy.sqrt(erg/cm**3)
temp = k_b*T/(r_e**2*c*B**2)/(k_b*T/m_e/c**2)**2
temp = temp.subs(T, m_p*(0.1*c)**2/k_b)
temp = temp.subs({alpha:scipy.constants.alpha,
                 n:1e8/cm**3,
                 r: au,
                 B:30*gauss,
                 m_p:scipy.constants.m_p*kg,
                 m_e:scipy.constants.m_e*kg,
                 c:scipy.constants.c*meter/second,
                 r_e:scipy.constants.physical_constants['classical electron radius'][0]*meter})
temp = temp.subs(kg, 1000*gram)
temp = temp.subs(meter, 100*cm)
temp/hour
```

This magnetic field is close to the equipartition value of plasma

```python
temp = sympy.sqrt(n*k_b*T)
temp = temp.subs(T, m_p*(0.1*c)**2/k_b)
temp = temp.subs({alpha:scipy.constants.alpha,
                 n:1e8/cm**3,
                 r: au,
                 B:30*gauss,
                 m_p:scipy.constants.m_p*kg,
                 m_e:scipy.constants.m_e*kg,
                 c:scipy.constants.c*meter/second,
                 r_e:scipy.constants.physical_constants['classical electron radius'][0]*meter})
temp = temp.subs(kg, 1000*gram)
temp = temp.subs(meter, 100*cm)
(temp/gauss).n()
```

A synchrotron emission is consistent with [observations](http://adsabs.harvard.edu/abs/2009ApJ...698..676D). The effect of supra - thermal, Fermi accelerate particles has been considered in [this work](http://iopscience.iop.org/article/10.1088/0004-637X/726/1/54/pdf).

```python

```


Computational Seismology

 Lamb's problem

---

This notebook is part of the supplementary material
to [Computational Seismology: A Practical Introduction](https://global.oup.com/academic/product/computational-seismology-9780198717416?cc=de&lang=en&#),
Oxford University Press, 2016.

##### Authors:
* David Vargas ([@dvargas](https://github.com/davofis))
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))

(Based on the original code by Lane Johnson)

## Basic Equations

The fundamental analytical solution to the three-dimensional Lamb’s problem, the problem of determining the elastic disturbance resulting from a point force in a homogeneous half space, is implemented in this Ipython notebook. This solution provides fundamental information used as benchmark for comparison with entirely numerical solutions. A setup of the fundamental problem is illustrated below. The figure on the right hand side is published in [1] (Figure 1. System of coordinates)

Simulations of 3D elastic wave propagation need to be validated by the use of analytical solutions. In order to evaluate how healthy a numerical solution is, one may recreate conditions for which analytical solutions exist with the aim of reproducing and compare the different results.

We which to find the displacement wavefield $\mathbf{u}(\mathbf{x},t)$ at some distance $\mathbf{x}$ from a seismic  source with $ \mathbf{F} = f_1\mathbf{\hat{x}_1} + f_2\mathbf{\hat{x}_2} + f_3\mathbf{\hat{x}_3}$.

For a uniform elastic material and a Cartesian co-ordinate system the equation for the conservation of linear momentum can be written

\begin{align*}
\rho(x) \frac{\partial^2}{\partial t^2} \mathbf{u(\mathbf{x},t)} = (\lambda + \mu)\nabla(\nabla\mathbf{u(\mathbf{x},t)}) + \mu\nabla^2 \mathbf{u(\mathbf{x},t)} + \mathbf{f(\mathbf{x},t)}
\end{align*}

We will consider the case where the source function is localized in both time and space

\begin{align*}
\mathbf{f(\mathbf{x},t)} = (f_1\mathbf{\hat{x}_1} + f_2\mathbf{\hat{x}_2} + f_3\mathbf{\hat{x}_3})\delta(x_1 - x^{'}_{1})\delta(x_2 - x^{'}_{2})\delta(x_3 - x^{'}_{3})\delta(t - t^{'})
\end{align*}

For such a source we will refer to the displacement solution as a Green’s function, and use the standard notation

\begin{align*}
\mathbf{u(\mathbf{x},t)} = g_1(\mathbf{x},t;\mathbf{x^{'}},t^{'})\mathbf{\hat{x}_1} + g_2(\mathbf{x},t;\mathbf{x^{'}},t^{'})\mathbf{\hat{x}_2} + g_3(\mathbf{x},t;\mathbf{x^{'}},t^{'})\mathbf{\hat{x}_3}
\end{align*}

The complete solution is found after applying the Laplace transform to the elastic wave equation, implementing the stress-free boundary condition, defining some transformations, and performing some algebraic manoeuvres. Then, the Green's function at the free surface is given:

\begin{align*}
\begin{split}
\mathbf{G}(x_1,x_2,0,t;0,0,x^{'}_{3},0) & =  \dfrac{1}{\pi^2\mu r} \dfrac{\partial}{\partial t}\int_{0}^{((t/r)^2 - \alpha^{-2})^{1/2}}\mathbf{H}(t-r/\alpha)\mathbb{R}[\eta_\alpha\sigma^{-1}((t/r)^2 - \alpha^{-2} - p^2)^{-1/2}\mathbf{M}(q,p,0,t,x^{'}_{3})\mathbf{F}] dp \\
 & + \dfrac{1}{\pi^2\mu r} \dfrac{\partial}{\partial t}\int_{0}^{p_2}\mathbf{H}(t-t_2)\mathbb{R}[\eta_\beta\sigma^{-1}((t/r)^2 - \beta^{-2} - p^2)^{-1/2}\mathbf{N}(q,p,0,t,x^{'}_{3})\mathbf{F}] dp
\end{split}
\end{align*}

Details on the involved terms are found in the original paper [2]. The Green's $\mathbf{G}$ function consist of three components of displacement evolving from the application of three components of force $\mathbf{F}$. If we assume that each component of $\mathbf{F}$ provokes three components of displacement, then $\mathbf{G}$ is composed by nine independent components that correspond one to one to the matrices $\mathbf{M}$ and $\mathbf{N}$. Without losing generality it is shown that among them four are equal zero, and we end up only with five possible components.

 [1] Eduardo Kausel - Lamb's problem at its simplest, 2012

 [2] Lane R. Johnson - Green’s Function for Lamb’s Problem, 1974

```
# Import all necessary libraries, this is a configuration step for the exercise.
# Please run it before the simulation code!
import numpy as np
import matplotlib.pyplot as plt
import os
from ricker import ricker

# Show the plots in the Notebook.
plt.switch_backend("nbagg")
```

```
# Compile the source code (needs gfortran!)
!rm -rf lamb.exe output.txt
!gfortran canhfs.for -fcheck=all -o lamb.exe
```

### Calling the original FORTRAN code

```
# Initialization of setup:
# Figure 4 in Lane R. Johnson - Green’s Function for Lamb’s Problem, 1974
# is reproduced when the following parameters are given
# -----------------------------------------------------------------------------
r   = 10.0    # km
vp  = 8.0     # P-wave velocity km/s
vs  = 4.62    # s-wave velocity km/s
rho = 3.3     # Density kg/m^3
nt  = 512     # Number of time steps
dt  = 0.01    # Time step s
h   = 0.2     # Source position km (0.01 to reproduce Fig 2.16 of the book)
ti  = 0.0     # Initial time s

var = [vp, vs, rho, nt, dt, h, r, ti]

# -----------------------------------------------------------------------------
# Execute fortran code
# -----------------------------------------------------------------------------
with open('input.txt', 'w') as f:
    for i in var:
        print(i, file=f, end='  ') # Write input for fortran code

f.close()

os.system("./lamb.exe")   # Code execution

# -----------------------------------------------------------------------------
# Load the solution
# -----------------------------------------------------------------------------
G = np.genfromtxt('output.txt')

u_rx = G[:,0]    # Radial displacement owing to horizontal load
u_tx = G[:,1]    # Tangential displacement due to horizontal load
u_zx = G[:,2]    # Vertical displacement owing to horizontal load

u_rz = G[:,3]    # Radial displacement owing to a vertical load
u_zz = G[:,4]    # Vertical displacement owing to vertical load

t = np.linspace(dt, nt*dt, nt)    # Time axis

```

### Visualization of the Green's function

```
# Plotting
# -----------------------------------------------------------------------------
seis = [u_rx, u_tx, u_zx, u_rz, u_zz]  # Collection of seismograms
labels = ['$u_{rx}(t) [cm]$','$u_{tx}(t)[cm]$','$u_{zx}(t)[cm]$','$u_{rz}(t)[cm]$','$u_{zz}(t)[cm]$']
cols = ['b','r','k','g','c']

# Initialize animated plot
fig = plt.figure(figsize=(12,8), dpi=80)

fig.suptitle("Green's Function for Lamb's problem", fontsize=16)

plt.ion() # set interective mode
plt.show()

for i in range(5):
    st = seis[i]
    ax = fig.add_subplot(2, 3, i+1)
    ax.plot(t, st, lw = 1.5, color=cols[i])
    ax.set_xlabel('Time(s)')
    ax.text(0.8*nt*dt, 0.8*max(st), labels[i], fontsize=16)
    plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))

    ax.spines['left'].set_position('zero')
    ax.spines['right'].set_color('none')
    ax.spines['bottom'].set_position('zero')
    ax.spines['top'].set_color('none')
    ax.spines['left'].set_smart_bounds(True)
    ax.spines['bottom'].set_smart_bounds(True)
    ax.xaxis.set_ticks_position('bottom')
    ax.yaxis.set_ticks_position('left')

plt.show()
```

### Convolution

Let $S(t)$ be a general source time function, then the displacent seismogram is given in terms of the Green's function $G$ via

\begin{equation}
u(\mathbf{x},t) = G(\mathbf{x},t; \mathbf{x}',t') \ast S(t)
\end{equation}

#### Exercise
Compute the convolution of the source time function 'ricker' with the Green's function of a Vertical displacement due to vertical loads. Plot the resulting displacement.

```
# call the source time function
T = 1/5   # Period
src = ricker(dt,T)

# Normalize source time function
src = src/max(src)

# Initialize source time function
f = np.zeros(nt)
f[0:int(2 * T/dt)] = src
```

```

```

```

# Compute convolution
u = np.convolve(u_zz, f)
u = u[0:nt]

# ---------------------------------------------------------------
# Plot Seismogram
# ---------------------------------------------------------------
fig = plt.figure(figsize=(12,4), dpi=80)

plt.subplot(1,3,1)
plt.plot(t, u_zz, color='r', lw=2)
plt.title('Green\'s function')
plt.xlabel('time [s]', size=16)
plt.ylabel('Displacement [cm]', size=14)
plt.xlim([0,nt*dt])
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))

plt.subplot(1,3,2)
plt.plot(t, f, color='k', lw=2)
plt.title('Source time function')
plt.xlabel('time [s]', size=16)
plt.xlim([0,nt*dt])
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))

plt.subplot(1,3,3)
plt.plot(t, u, color='b', lw=2)
plt.title('Displacement [cm]')
plt.xlabel('time [s]', size=16)
plt.xlim([0,nt*dt])
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))

plt.grid(True)
plt.show()
```

notebooks/Computational Seismology/Wave Propagation & Analytical Solutions/Lambs_problem_3D/lambs_problem_exercise.ipynb