Osnabrück University - Machine Learning (Summer Term 2018) - Prof. Dr.-Ing. G. Heidemann, Ulf Krumnack

# Exercise Sheet 11

## Introduction

This week's sheet should be solved and handed in before the end of **Sunday, June 24, 2018**. If you need help (and Google and other resources were not enough), feel free to contact your groups' designated tutor or whomever of us you run into first. Please upload your results to your group's Stud.IP folder.

## Assignment 1: Uncertainty and probability [6 Points]

This exercise will focus on concepts introduced in the first part of lecture (ML-11).

### a) Modeling uncertainty

In the lecture it is claimed that probabilities can summarize several factors:

1. missing knowledge
1. incapability to devise complete models of complex domains
1. chance

Think of an example for each of these points and explain how probabilities can be applied in modeling your example.

### b) Inference by enumeration

Given the full joint distribution below, calculate the following:

|         |  toothache
 catch|
  ¬catch |   ¬toothache
 catch |
 ¬catch |
|:--------|-------------------------:|--------------------------:|--------------------------:|---------------------------:|
| cavity  |                    0.108 |                     0.012 |                     0.072 |                      0.008 |
| ¬cavity |                    0.016 |                     0.064 |                     0.144 |                      0.576 |

1. $P(\neg toothache)$
1. $P(cavity)$
1. $P(toothache \mid cavity)$
1. $P(cavity \mid toothache \vee catch)$

If you are familiar with `pandas` you can use the dataframe below to find the solutions. You can of course also write code without using pandas or calculate the answers manually.

1. This asks for the probability that Toothache is true.
$$P (toothache ) = 0.108 + 0.012 + 0.016 + 0.064 = 0.2$$
1. This asks for the vector of probability values for the random variable Cavity. It has two
values, which we list in the order htrue, falsei. First add up $0.108 + 0.012 + 0.072 +
0.008 = 0.2$. Then we have
$$P(Cavity) = \langle 0.2, 0.8\rangle.$$
1. This asks for the vector of probability values for Toothache, given that Cavity is true.
$$P(Toothache |cavity) = \langle(.108 + .012)/0.2, (0.072 + 0.008)/0.2\rangle = \langle 0.6, 0.4\rangle$$
1. This asks for the vector of probability values for Cavity, given that either Toothache or
Catch is true. First compute $P(toothache\vee catch) = 0.108 + 0.012 + 0.016 + 0.064 +
0.072 + 0.144 = 0.416$. Then
$$P(Cavity|toothache \vee catch) =
\langle(0.108 + 0.012 + 0.072)/0.416, (0.016 + 0.064 + 0.144)/0.416\rangle =
\langle 0.4615, 0.5384 \rangle$$

```python
import pandas as pd
columns = pd.MultiIndex.from_product((('toothache', '¬toothache'), ('catch', '¬catch')))
index = ('cavity', '¬cavity')
data = [[0.108, 0.012, 0.072, 0.008],
        [0.016, 0.064, 0.144, 0.576]]
joint_distribution = pd.DataFrame(data, index, columns)
joint_distribution
```

      toothache
      ¬toothache

      catch
      ¬catch
      catch
      ¬catch

      cavity
      0.108
      0.012
      0.072
      0.008

      ¬cavity
      0.016
      0.064
      0.144
      0.576

```python
# 1.
joint_distribution.sum().sum(level=0)
```

    toothache     0.2
    ¬toothache    0.8
    dtype: float64

```python
# 2.
joint_distribution.sum(axis='columns')
```

    cavity     0.2
    ¬cavity    0.8
    dtype: float64

```python
# 3.
joint_distribution.loc['cavity', :].sum(level=0) / joint_distribution.loc['cavity', :].sum()
```

    toothache     0.6
    ¬toothache    0.4
    Name: cavity, dtype: float64

```python
# 4.
toothache_or_catch = joint_distribution.sum(axis='columns') - joint_distribution.loc[:, ('¬toothache', '¬catch')]
toothache_or_catch / toothache_or_catch.sum()
```

    cavity     0.461538
    ¬cavity    0.538462
    dtype: float64

### c) Conditional probability

For each of the following statements, either prove it is true or give a counterexample.
1. If P (a | b, c) = P (b | a, c), then P (a | c) = P (b | c)
1. If P (a | b, c) = P (a), then P (b | c) = P (b)
1. If P (a | b) = P (a), then P (a | b, c) = P (a | c)

1. True. By the product rule we know P (b, c)P (a|b, c) = P (a, c)P (b|a, c), which by
assumption reduces to P (b, c) = P (a, c). Dividing through by P (c) gives the result.
1. False. The statement P (a|b, c) = P (a) merely states that a is independent of b and c,
it makes no claim regarding the dependence of b and c. A counter-example: a and b
record the results of two independent coin flips, and c = b.
1. False. While the statement P (a|b) = P (a) implies that a is independent of b, it does
not imply that a is conditionally independent of b given c. A counter-example: a and b
record the results of two independent coin flips, and c equals the xor of a and b.

### d) Independence and conditional independence

It is quite often useful to consider the effect of some specific propositions in the
context of some general background evidence that remains fixed, rather than in the complete
absence of information. The following questions ask you to prove more general versions of
the product rule and Bayes’ rule, with respect to some background evidence e:

1. Prove the conditionalized version of the general product rule:
$$P(X, Y \mid e) = P(X \mid Y, e)\cdot P(Y \mid e) .$$
1. Prove the conditionalized version of Bayes’ rule:
$$P(Y \mid X, e) = \frac{P(X \mid Y, e)\cdot P(Y \mid e)}{P(X \mid e)} $$

The basic axiom to use here is the definition of conditional probability:

1. We have
$$P(A, B|E) = \frac{P(A, B, E)}{P(E)}$$
and
$$P(A|B, E)\cdot P(B|E) = \frac{P(A, B, E)}{P(B, E)}\cdot \frac{P(B, E)}{P(E)}
=\frac{P(A, B, E)}{P(E)}$$
hence
$$P(A, B|E) = P(A|B, E)\cdot P(B|E)$$
1. The derivation here is the same as the derivation of the simple version of Bayes’ Rule. First write down the dual form of the conditionalized product rule, simply by switching A and B in the above derivation:
$$P(A, B|E) = P(B|A, E)\cdot P(A|E)$$
Therefore the two right-hand sides are equal:
$$P(B|A, E)\cdot P(A|E) = P(A|B, E)\cdot P(B|E)$$
Dividing through by $P(B|E)$ you get
$$P(A|B, E) = \frac{P(B|A, E)\cdot P(A|E)}{P(B|E)}$$

### e) Naive Bayes models

Text categorization is the task of assigning a given document to one of a fixed set of
categories on the basis of the text it contains. Naive Bayes models are often used for this
task. In these models, the query variable is the document category, and the “effect” variables
are the presence or absence of each word in the language; the assumption is that words occur
independently in documents, with frequencies determined by the document category.
1. Explain precisely how such a model can be constructed, given as “training data” a set of documents that have been assigned to categories.
1. Explain precisely how to categorize a new document.
1. Is the conditional independence assumption reasonable? Discuss.

1. The model consists of the prior probability $P(Category)$ and the conditional probabilities $P(Word_i|Category)$. For each category $c$, $P(Category = c)$ is estimated as the fraction of all documents that are of category $c$. Similarly,
$P(Word_i = true|Category = c)$ is estimated as the fraction of documents of category $c$ that contain $Word_i$.
1. Here, every evidence variable $Word_i$ is observed, since we can tell if any given word appears in a given document or not. Hence by Bayes rule we can estimate $P(Category|Word_i)$.
1. The independence assumption is clearly violated in practice. For example, the word pair "machine learning" occurs more frequently in any given document category than would be suggested by multiplying the probabilities of "machine" and "learning".

## Assignment 2: Bayes networks [4 Points]

### a) Bayes networks

Explain in your own words the idea of a Bayes network. How is conditional independence represented in such a network? How can the full joint distribution be regained from such a network?

x

### b) Independence in Bayes networks

Consider the Bayes network in (ML-11 slide 32):
1. If no evidence is observed, are Burglary and Earthquake independent? Prove this from the numerical semantics and from the topological semantics.
1. If we observe Alarm = true, are Burglary and Earthquake independent? Justify your answer by calculating whether the probabilities involved satisfy the definition of conditional independence.

1. Yes. According to (ML-11 slide 33)
\begin{align*}
  P(X_1,\ldots,X_n) = \prod_{i=1}^n P(X_i\mid\textit{Parents}(X))
\end{align*}
So numerically one can compute that in our case
\begin{align*}
  P(B,E,A,J,M)
  & = P(B|\textit{Parents}(B))\cdot
  P(E|\textit{Parents}(E))\cdot
  P(A|\textit{Parents}(A))\cdot
  P(J|\textit{Parents}(J))\cdot
  P(M|\textit{Parents}(M)) \\
  & = P(B)\cdot
  P(E)\cdot
  P(A|B,E)\cdot
  P(J|A)\cdot
  P(M|A)
\end{align*}
To get $P(B,E)$ we can apply marginalization:
\begin{align*}
  P(B,E,A,J,M)
  &= \sum_{a}\sum_{j}\sum_{m} P(B,E,a,j,m) \\
  &= \sum_{a}\sum_{j}\sum_{m} P(B)\cdot P(E)\cdot
  P(a|B,E)\cdot
  P(j|a)\cdot
  P(m|a) \\
  &= \sum_{a} P(B)\cdot P(E)\cdot
  P(a|B,E)\cdot 1 \cdot 1 \\
  &= P(B)\cdot P(E)\cdot 1
\end{align*}
This shows the independence of $B$ and $E$. Topologically $B$ and $E$ are d-separated by $A$ (i.e. any path connecting $B$ and $E$ goes through $A$).
1.  We check whether $P(B,E|a) = P(B|a)P(E|a)$. First computing $P(B,E|a)$:
\begin{align} P(B,E|a) = \alpha P(a|B,E)P(B, E)
&= \alpha
\begin{cases}
 0.29 \cdot 0.001 \cdot 0.002 & \text{if $B=b$ and $E=e$} \\
 0.94 \cdot 0.001 \cdot 0.998 & \text{if $B=b$ and $E=\neg e$} \\
 0.29 \cdot 0.999 \cdot 0.002 & \text{if $B=\neg b$ and $E=e$} \\
 0.001 \cdot 0.999 \cdot 0.998 & \text{if $B=\neg b$ and $E=\neg e$}
 \end{cases}
\\
&= \alpha
\begin{cases}
 0.0008 & \text{if $B=b$ and $E=e$} \\
 0.3728 & \text{if $B=b$ and $E=\neg e$} \\
 0.2303 & \text{if $B=\neg b$ and $E=e$} \\
 0.3962 & \text{if $B=\neg b$ and $E=\neg e$}
 \end{cases}
\end{align}
where $\alpha$ is a normalization constant. Checking $B = \neg b$ whether $P = (b, e|a) = P (b|a)\cdot P(e|a)$ we
find
$$P(b,e|a) = 0.0008\neq 0.0863 = 0.3736 \cdot 0.2311 = P(b|a)\cdot P(e|a)$$
showing that $B$ and $E$ are not conditionally independent given $A$.

[RN, ex 14.4]

# Recap (part I)

This part of the sheet is intended to revise some topics from the lecture, a second part is following on the next sheet. These exercises do not need to be solved in order to qualify for the final exam but it is highly recommended for preparation. Also if you hit any question that should be discussed in more detail, please let us know.

## Recap 1: Concept Learning [2 Points]

### a) Concept Learning

What is Concept Learning? Is it supervised? Is it local?

Concept learning aims at acquiring knowledge that allows to distinguish exemplars from non exemplars of a given category (concept). It can be formalized as learning a unary predicate $p_c$ on the domain $X$ or equivalently an indicator function $c:X\to\{0,1\}$.

Concept learning is usually supervised: the teacher tells the learner if an example falls under the concept or not.

As soon as there are is some metric given on the data, there may be local and global concept learners. One may for example use a nearest neighbor learner (local) or a multilayer-perceptron (global) to learn concepts.

### b) Find-S
Describe the Find-S Algorithm in pseudo code. What is its inductive bias? What are its advantages and drawbacks?

    1. Initialize $h$ to the most specific hypothesis in H.
    2. For each positive training instance x do
           For each attribute constraint $a_i$ in h do
               If ($a_$i is not satisfied by x) then
                   Replace $a_i$ in h by the next more general constraint
                     that is satisfied by x.
               End if
           End for
       End for
    3. Output h.

Inductive Bias: The target concept can be described in its hypothesis space (in our case: it is a conjunction of features). All instances are negative instances unless demonstrated otherwise.

Drawback: it does not take negative instances into account.

### c) Hypotheses space

What is the hypotheses space for Candidate-Elimination used in the lecture?

The hypothesis space for Candidate-Elimination spreads between the most general and most specific hypotheses. The other hypotheses are made up by conjunction of features which biases the learner and makes it impossible to find a disjunctive solution.

The version space on the other hand is a subset of the hypotheses space. It is the set of all hypotheses between and including the general and the specific boundary.

## Recap 2: Decision Trees [2 Points]

### a) Overfitting
What is overfitting? How can it be avoided?

Overfitting means an overly specific adaptation of the learner to the training data. Not only the general structure of the training data has been learned, but also its specific noise, i.e. artifacts, are learned and hence the learner looses the capability to generalize and work on other data.

Overfitting can be detected by using a separate test data set. If the error on the test data increases during training, this indicates overfitting.

### b) Pruning

Name one method for pruning a decision tree and describe it!

Pruning can be applied to reduce overfitting of a decision tree. Two types of pruning have been introduced in the lecture:

*Reduced error pruning:* removes nodes from the decision tree to achieve better generalization on the test set.

*Rule based pruning:* translate the decision tree into a set of rules and then prune an individual rule by removing any preconditions that result in improving its accuracy on the
validation set.

### c) Information gain
What are entropy and information gain? Provide explanation and formulae. How are they used in ID3?

Entropy measures the inhomogeneity of a data set (the minimal number of bits needed to encode elements from the set)
$$E(S) = -p_{+}\log_2 p_{+} - p_{-}\log_2p_{-}$$
where $p_{+}$ denotes the fraction of positive and $p_{-}$ that of negative examples in the data set. A set $S$ with only positive (or only negative) examples would have no entropy (i.e. $E(S)=0$), while a set with the same number of positive and negative examples has maximal entropy ($E(S)=1$).

Information gain is the expected reduction in entropy due to splitting the data set $S$ based on one attribute $A$: denote for every value $v\in\operatorname{Values}(A)$ the subset of elements from $S$ where $A=v$ by $S_v$. Then the information gain is given by
$$\operatorname{Gain}(S,A) = E(S) - \sum_{v\in\operatorname{Values}(A)}E(S_v)\cdot\frac{|S_v|}{|S|}$$
that is, from the entropy of $S$ the entropy values for $S_v$ are subtracted and weighted by their respective sizes. If the subsets $S_v$ are all homogeneous ($E(S_v)=0$), then the information gain is maximal, namely $E(S)$, i.e. the data set can be fully explained by the single attribute $A$. On the other hand, if all $S_v$ have maximal entropy,  no information is gained by splitting based on $A$. In practice, something between these extremes will be the case.

ID3 places the node with highest information gain at the root of the decision tree.

## Recap 3: Data Mining [2 Points]

### a) Missing values

How can you deal with missing values? Name an important algorithm and explain how to use it.

Data records with missing values may be simply ignored, or one may try to "fix" the record by inserting artificial values into empty slots. The most simple way is to insert just zeros (or some other value) but this will lead to poor data quality. Better approaches try to use statistical properties of the data set to introduce "natural" fillers. One approach is to use the mean of the missing attribute, however this ignores possible dependencies between the different attributes. A more sophisticated approach is expectation maximization (EM) to estimate the joint probability distribution of all attributes in an iterative process. Once it is computed, one can use it to determine the most likely value for the missing datapoint.

### b) Outliers

What are outliers? Can we detect them? If so, how?

An outlier is a value that seemingly does not belong to the rest of the data. It is probably caused by some measurement error (but it may also reflect some real phenomenon).

A simple method to detect outliers is to consider their distances from the mean (or median) of the full data set. If this is too large (e.g. greater than 3 standard deviations), the data point is considered to be an outlier (z-test). The Rosner test iteratively removes those outliers until the dataset does not contain anymore of them.

### c)
What does the Q-function express in the EM algorithm?

The EM algorithm aims at finding model parameters $\theta$ that best explain observed data $x$ (which may have missing values $h$). It does so by alternating steps of calculating the expected value of the (log) likelihood function $L(\theta,x,h)$, using the current estimated parameters $\theta_t$ (E step), and then finding parameter values $\theta'$ that maximize this quantity (M step). The $Q$-function expresses the expected likelihood function:
$$Q(\theta\mid\theta_{t}) = E_{h\mid x,\theta_t}[\log L(\theta,x,h)] = \int P(h\mid x,\theta_t)\cdot \log P(h\mid x,\theta)\operatorname{d} h+\log P(x\mid\theta)$$

## Recap 4: Clustering [2 Points]

### a) Clustering

Explain the difference between single-linkage and complete-linkage clustering.

Single-linkage clustering is based on the *minimum distance* that defines the distance between two clusters from the distance of their closest points. Single-linkage clustering tends to chaining.

Complete-linkage clustering is based on the *maximum distance* that defines the distance of two clusters to be the maximal distance of two of their points. Complete linkage clustering prefers compact clusters.

### b) Metrics

Name three different distance measures and briefly explain them. Check the metric axioms for one of them.

* Hamming distance: the number of positions where two strings of equal length differ
* Chebyshev distance (also: maximum distance): maximal absolute difference in a single coordinate.
* p-norm: family of norms, defined by the formula $\sqrt[p](\sum_{i=1}^{L}|x_i-y_i|^p)$. Important special cases: city block (aka Manhattan, p=1), euclidean distance (p=2)
* Jaccard distance: for binary attributes

Metric axioms for Chebyshev distance $d(\mathbf{x},\mathbf{y}) := \max_{i=1,\ldots,L}|x_i-y_i|$:
1. Symmetry: Here we use that the absolute value of the difference is symmetric: $|a-b| = |b-a|$, hence $d(\mathbf{x},\mathbf{y}) = \max_{i=1,\ldots,L}|x_i-y_i| = \max_{i=1,\ldots,L}|y_i-x_i| = d(\mathbf{y},\mathbf{x})$
2. Coincidence (identity of indiscernibles): $d(\mathbf{x},\mathbf{x}) = \max_{i=1,\ldots,L}|x_i-x_i| = 0$
3. Triangle equation: Here we apply that the triangle inequality holsd for the absolute value of the difference: $|a-c|+|c-b|\geq|a-b|$, and hence
\begin{align}
  d(\mathbf{x},\mathbf{z}) + d(\mathbf{z},\mathbf{y}) =
\max_{i=1,\ldots,L}|x_i-z_i| + \max_{i=1,\ldots,L}|z_i-y_i| &
 \geq \max_{i=1,\ldots,L}(|x_i-z_i| + |z_i-y_i|) \\
& \geq \max_{i=1,\ldots,L}(|x_i-y_i|) = d(\mathbf{x},\mathbf{y})
\end{align}

### c) Mixture models

What is a mixture model? Explain. Can you provide a formula?

A mixture model describes a two-step process to mix different (simple) data distributions. Such an approach can be used to model a large population with different subpopulations, each which individual characteristics.

Formally, one provides a specific distribution $P(X\mid Z=z)$ for every subpopulation $z$. These are mixed according to the probability $P(Z=z)$ to select an individual from that subpopulation, i.e.
$$P(X=x) = \sum_{z}P(Z=z)\cdot P(X=x\mid Z=z)$$

## Recap 5: Dimension Reduction [2 Points]

### a) Visualization

Name three different data visualization techniques to visualize high dimensional data. Explain one in detail.

* a *scatterplot matrix* shows 2D projections of the data for all combination of axes
* *Chernoff faces:* map parameters to facial features
* *parallel coordinates:* map the different data dimensions to different x-coordinates and plot the corresponding values at the y-axis.

### b) PCA

Draw a few data points (ASCII arts or on a sheet of paper) and mark the principal components. What are the principal components?

Principal components form a set of linear independent vectors, pointing into the direction of the largest variance. Their length corresponds to the variance in that direction.

### c) Covariance matrix
What does a covariance matrix express? How is it computed from data? How is it used in PCA?

The covariance matrix contains the covariance values for all pairs of coordinates. A positive covariance value means that high values for the first coordinate correspond to high values for the second coordinate. A negative covariance value expresses a correspondence of high values in the first coordinate with low values in the second coordinate. A value of $0$ means, that the values of the two coordinates do not correspond to each other.

Given a set of $n$ data points in a $d$-dimensional data space as an $n\times d$-matrix $D$, the covariance matrix is computed as $$C=(D-\mu)^T\cdot(D-\mu)$$ where $\mu$ denotes the mean vector of the data set.

In PCA, the principal components are computed as eigenvectors of the covariance matrix.


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 6

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Feb. 27 at **12:30pm.**

## Exercise: Four-Period Cake-Eating Problem

Now consider a the four-period cake-eating problem. A person has an initial quantity of cake $K_0>0$ and receives utility from consuming cake in period 0, period 1, period 2, and period 3:

\begin{align}
U(C_0,C_1,C_2,C_3) & = \log C_0 + \beta \log C_1+ \beta^2 \log C_2 + \beta^3 \log C_3.
\end{align}

The person chooses how much to consume in each period, $C_0$, $C_1$, $C_2$, $C_3$, and quantities of cake $K_1$, $K_2$, and $K_3$ to carry foward from period 0, period 1, and period 2 to maximize her utility subject to the following two budget constraints:

\begin{align}
C_0 + K_1 & = K_0\\
C_1 + K_2 & = K_1\\
C_2 + K_3 & = K_2\\
C_3 & = K_3,
\end{align}

where the boundary condition $K_4 = 0$ has been imposed.

### Part (a)

Find the first-order conditions with respect to $K_1$, $K_2$, and $K_3$ for the four-period cake-eating problem. Then solve by hand for:

1. $K_3$ as a function of $K_2$
1. $K_2$ as a function of $K_1$
2. $K_1$ as a function of $K_0$

Use what you derive in this part to do **Part (b)**, **Part (c)**, and **Part (d)**.

### Part (b)

Suppose that the initial quantity of cake is $K_0 = 100$. For $\beta = 0, 0.25, 0.5, 0.75, 0.99, 1, 2$, plot the optimal path for the quantity of cake for periods 0 through 4. Add labels to the plot command so you know which line corresponds to which value of $\beta$.

```python
# Define a function to compute the solution for the four-period cake-eating problem. CELL NOT PROVIDED
def four_period_cake(beta,cake0):

    '''Function for computing solution to four-period cake-eating problem

            Args:
            beta (float):   subjective discount factor
            cake0 (float):  initial quantity of cake

        Returns:
            numpy array, numpy array
    '''

    # Initialize arrays for cake and consumption. NOT PROVIDED
    cake = np.zeros(5)
    consumption = np.zeros(4)

    # Assign values to cake array elements. NOT PROVIDED
    cake[0] = cake0
    cake[1] = (beta+beta**2+beta**3)/(1+beta+beta**2+beta**3)*cake[0]
    cake[2] = (beta+beta**2)/(1+beta+beta**2)*cake[1]
    cake[3] = beta/(1+beta)*cake[2]

    # Assign values to consumption array elements as implied by the budget constraints. NOT PROVIDED
    consumption[0] = cake[0]-cake[1]
    consumption[1] = cake[1]-cake[2]
    consumption[2] = cake[2]-cake[3]
    consumption[3] = cake[3]

    return cake,consumption
```

```python
# Create a variable called 'cake0' that stores the initial quantity of cake. NOT PROVIDED
cake0=100

# Plot optimal path of cake for the different values of beta. NOT PROVIDED
for beta in [2,1, 0.99, 0.75, 0.5, 0.25, 0]:

    cake,consumption = four_period_cake(beta=beta,cake0=cake0)
    plt.plot(cake,'o-',lw=3,alpha=0.75,label='$\\beta=$'+str(beta))

plt.title('Cake Quantity for different $\\beta$ values')

# Put a legend to the right of the current axis. NOT PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

plt.grid()
```

### Part (c)

Suppose that the initial quantity of cake is $K_0 = 100$. For $\beta = 0, 0.25, 0.5, 0.75, 0.99, 1, 2$, plot the optimal path for cake consumption for periods 0 through 3. Add labels to the plot command so you know which line corresponds to which value of $\beta$.

```python
# Create a variable called 'cake0' that stores the initial quantity of cake. NOT PROVIDED
cake0=100

# Plot optimal path of cake for the different values of beta. NOT PROVIDED
for beta in [2,1, 0.99, 0.75, 0.5, 0.25, 0]:

    cake,consumption = four_period_cake(beta=beta,cake0=cake0)
    plt.plot(consumption,'o-',lw=3,alpha=0.75,label='$\\beta=$'+str(beta))

plt.title('Cake Consumption for different $\\beta$ values')

# Put a legend to the right of the current axis. NOT PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

plt.grid()
```

### Part (d)

Suppose that $\beta=0.95$. For $K_0 = 1, 10, 20, 50$, plot the optimal path for the quantity of cake for periods 0 through 4. Add labels to the plot command so you know which line corresponds to which value of $K_0$.

```python
# Create a variable called 'cake0' that stores the initial quantity of cake. NOT PROVIDED
beta=0.5

# Plot optimal path of cake for the different values of beta. NOT PROVIDED
for cake0 in [1,10,20,50]:

    cake,consumption = four_period_cake(beta=beta,cake0=cake0)
    plt.plot(cake,'o-',lw=3,alpha=0.75,label='$\\beta=$'+str(beta))

plt.title('Cake Quantity for different $K_0$ values')

# Put a legend to the right of the current axis. NOT PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

plt.grid()
```

## Exercise: The Stochastic Solow Growth Model

In class we simulated the stochastic Solow growth model with three endogenous variables: $K_t$, $Y_t$, and $A_t$. Now, reconsider the model with two additional variables, consumption $C_t$ and investment $I_t$, and two additional equations:

\begin{align}
Y_t & = A_t K_t^{\alpha}\\
C_{t} & = (1-s)Y_t\\
Y_{t} & = C_t + I_t\\
K_{t+1} & = sY_t + (1-\delta) K_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$. In this exercise, you will construct a 401 period (10-year) simulation of the model.

For the other parameter, use the following values for the simulation:

| $\rho$ | $\sigma$ | $s$  | $\alpha$ | $\delta $ | $T$  |
|--------|----------|------|----------|-----------|------|
| 0.8    | 0.005    | 0.12 | 0.33     |  0.025    | 401  |

```python
# Create a variable called 'parameters' that is a Pandas Series containing the parameter values for the simulation
parameters = pd.Series()
parameters['rhoa'] = .8
parameters['sigma_squared'] = 0.005**2
parameters['alpha'] = 0.33
parameters['delta'] = 0.025
parameters['s'] = 0.12

# Print the model's parameters
print(parameters)
```

    rhoa             0.800000
    sigma_squared    0.000025
    alpha            0.330000
    delta            0.025000
    s                0.120000
    dtype: float64

```python
# Define a function that evaluates the equilibrium conditions. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Production function
    prod_fn = cur.a*cur.k**p.alpha - cur.y

    # Capital evolution
    capital_evolution = p.s*cur.a*cur.k**p.alpha + (1 - p.delta)*cur.k - fwd.k

    # Consumption function
    consumption_function = (1-p.s)*cur.y - cur.c

    # Market clearing
    market_clearing = cur.c + cur.i - cur.y

    # Exogenous tfp
    tfp_proc = p.rhoa*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        prod_fn,
        capital_evolution,
        consumption_function,
        market_clearing,
        tfp_proc
        ])
```

```python
# Initialize the model. Note that state variables must be named first in the var_names argument and shock for each
# state variable must be named.
model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=['a','k','y','c','i'],
                 shock_names=['e_a','e_k'],
                 parameters = parameters)
```

```python
# Compute the steady state numerically using the .compute_ss() method.
guess = [1,4,1,1,1]
model.compute_ss(guess)
```

```python
# Use the .approximate_and_solve() method to compute the log-linear approximation of the model around the
# non-stochastic steady state and solve endogenous variables in terms of state variables.
model.approximate_and_solve()
```

```python
# Use .stoch_sim() method to compute stochastic simulation with the following arguments:
#    1. seed = 126
#    2. T = 401
#    3. cov_mat= [[parameters['sigma_squared'],0],[0,0]]
model.stoch_sim(seed=126,T=401,cov_mat= [[parameters['sigma_squared'],0],[0,0]])

# Print first 10 rows of model.simulated
print(model.simulated.head(10))
```

            e_a  e_k         a         k         y         c         i
    0  0.001237  0.0  0.004041  0.000614  0.004243  0.004243  0.004243
    1  0.001189  0.0  0.004421  0.000705  0.004654  0.004654  0.004654
    2  0.002629  0.0  0.006166  0.000803  0.006431  0.006431  0.006431
    3  0.009421  0.0  0.014354  0.000944  0.014665  0.014665  0.014665
    4 -0.001482  0.0  0.010001  0.001287  0.010426  0.010426  0.010426
    5 -0.011009  0.0 -0.003008  0.001516 -0.002508 -0.002508 -0.002508
    6  0.003379  0.0  0.000972  0.001415  0.001439  0.001439  0.001439
    7  0.005128  0.0  0.005905  0.001416  0.006373  0.006373  0.006373
    8 -0.001113  0.0  0.003611  0.001540  0.004119  0.004119  0.004119
    9 -0.005668  0.0 -0.002779  0.001604 -0.002249 -0.002249 -0.002249

```python
# Plot the computed stochastic simulation results. Creae two axes side-by-side.
#     1. Left axes: plot: a, k, i, y, c
#     1. Right axes: plot: shock to TFP
fig = plt.figure(figsize=(12,4))

ax1 = fig.add_subplot(1,2,1)
model.simulated[['a','y','k','c','i']].plot(lw=3,alpha=0.5,grid=True,ax = ax1).legend(loc='upper right',ncol=3)
ax1.set_title('Endogenous Variables')

ax2 = fig.add_subplot(1,2,2)
model.simulated[['e_a']].plot(lw=3,alpha=0.5,grid=True,ax = ax2).legend(loc='upper right',ncol=2)
ax2.set_title('TFP Shock Process')

fig.tight_layout()
```

```python
# Compute standard deviations of simulated TFP, output, consumption, investment, and capital
model.simulated[['a','y','c','i','k']].std()
```

    a    0.008863
    y    0.009033
    c    0.009033
    i    0.009033
    k    0.002546
    dtype: float64

```python
# Compute correlation coefficients of simulated TFP, output, consumption, investment, and capital
model.simulated[['a','y','c','i','k']].corr()
```

      a
      y
      c
      i
      k

      a
      1.000000
      0.995770
      0.995770
      0.995770
      0.156264

      y
      0.995770
      1.000000
      1.000000
      1.000000
      0.246354

      c
      0.995770
      1.000000
      1.000000
      1.000000
      0.246354

      i
      0.995770
      1.000000
      1.000000
      1.000000
      0.246354

      k
      0.156264
      0.246354
      0.246354
      0.246354
      1.000000

**Questions**

1. In the data simulated from the stochastic Solow growth model, which variables appear to be perfectly correlated?
2. Refer to the Notebook from Class 9. Compared with the actual correlations, does the stochastic Solow growth model under- or overpredict the correlation between output and consumption over the cycle? What about the correlation between output and investment?

**Answers**

1. Output, investment, and consumption appear to be perfectly correlated.
2. The stochastic Solow model *overpredicts* the correlation between output and consumption. The stochastic Solow model also *overpredicts* the correlation between output and investment.


# Multi-output Gaussian processes in GPflow

This notebook shows how to construct a multi-output GP model using GPflow, together with different interdomain inducing variables which lead to different approximation properties. GPflow provides a framework for specifying multioutput GP priors, and interdomain approximations which is
- modular, by providing a consistent interface for the user of the resulting `SVGP` model,
- extensible, by allowing new interdomain variables and kernels to be specified while reusing exising code where possible,
- efficient, by allowing the most efficient custom code path to be specified where desired.

Getting to grips with the maths and code can be a bit daunting, so to accompany the documentation there is an [in-depth review on arXiv](https://arxiv.org/abs/2003.01115), which provides a unified mathematical framework, together with a high-level description of software design choices in GPflow.

This notebook shows the various design choices that can be made, to show the reader the flexibility of the framework. This is done in the hope that an example is provided that can be easily adapted to the special case that the reader wants to implement.

A reader who just wants to use a multioutput kernel should simply choose the most efficient set of inducing variables.

To cite this framework, please reference our [arXiv paper](https://arxiv.org/abs/2003.01115).
```
@article{GPflow2020multioutput,
  author = {{van der Wilk}, Mark and Dutordoir, Vincent and John, ST and
            Artemev, Artem and Adam, Vincent and Hensman, James},
  title = {A Framework for Interdomain and Multioutput {G}aussian Processes},
  year = {2020},
  journal = {arXiv:2003.01115},
  url = {https://arxiv.org/abs/2003.01115}
}
```

\begin{equation}
\newcommand{\GP}{\mathcal{GP}}
\newcommand{\NN}{\mathcal{N}}
\newcommand{\LL}{\mathcal{L}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\valpha}{\boldsymbol\alpha}
\newcommand{\vf}{\mathbf{f}}
\newcommand{\vF}{\mathbf{F}}
\newcommand{\vg}{\mathbf{g}}
\newcommand{\vW}{\mathbf{W}}
\newcommand{\vI}{\mathbf{I}}
\newcommand{\vZ}{\mathbf{Z}}
\newcommand{\vu}{\mathbf{u}}
\newcommand{\vU}{\mathbf{U}}
\newcommand{\vX}{\mathbf{X}}
\newcommand{\vY}{\mathbf{Y}}
\newcommand{\identity}{\mathbb{I}}
\end{equation}

## Task
We will consider a regression problem for functions $f: \mathbb{R}^D \rightarrow \mathbb{R}^P$. We assume that the dataset is of the form $(X, f_1), \dots, (X, f_P)$, that is, we observe all the outputs for a particular input location (for cases where there are **not** fully observed outputs for each input, see [A simple demonstration of coregionalization](./coregionalisation.ipynb)).

Here we assume a model of the form:
\begin{equation}
f(x) = W g(x),
\end{equation}
where $g(x) \in \mathbb{R}^L$, $f(x) \in \mathbb{R}^P$ and $W \in \mathbb{R}^{P \times L}$. We assume that the outputs of $g$ are uncorrelated, and that by *mixing* them with $W$ they become correlated. In this notebook, we show how to build this model using Sparse Variational Gaussian Process (SVGP) for $g$, which scales well with the numbers of data points and outputs.

Here we have two options for $g$:
1. The output dimensions of $g$ share the same kernel.
1. Each output of $g$ has a separate kernel.

In addition, we have two further suboptions for the inducing inputs of $g$:
1. The instances of $g$ share the same inducing inputs.
1. Each output of $g$ has its own set of inducing inputs.

The notation is as follows:
- $X \in \mathbb{R}^{N \times D}$ denotes the input
- $Y \in \RR^{N \times P}$ denotes the output
- $k_{1..L}$, $L$ are kernels on $\RR^{N \times D}$
- $g_{1..L}$, $L$ are independent $\GP$s  with $g_l \sim \GP(0,k_l)$
- $f_{1..P}$, $P$ are correlated  $\GP$s  with $\vf = \vW \vg$

```python
import numpy as np
import matplotlib.pyplot as plt
import gpflow as gpf
import tensorflow as tf

from gpflow.utilities import print_summary
from gpflow.ci_utils import ci_niter

gpf.config.set_default_float(np.float64)
gpf.config.set_default_summary_fmt("notebook")
np.random.seed(0)
%matplotlib inline

MAXITER = ci_niter(2000)
```

## Generate synthetic data
We create a utility function to generate synthetic data. We assume that:

```python
N = 100  # number of points
D = 1  # number of input dimensions
M = 15  # number of inducing points
L = 2  # number of latent GPs
P = 3  # number of observations = output dimensions
```

```python
def generate_data(N=100):
    X = np.random.rand(N)[:, None] * 10 - 5  # Inputs = N x D
    G = np.hstack((0.5 * np.sin(3 * X) + X, 3.0 * np.cos(X) - X))  # G = N x L
    W = np.array([[0.5, -0.3, 1.5], [-0.4, 0.43, 0.0]])  # L x P
    F = np.matmul(G, W)  # N x P
    Y = F + np.random.randn(*F.shape) * [0.2, 0.2, 0.2]

    return X, Y
```

```python
X, Y = data = generate_data(N)
Zinit = np.linspace(-5, 5, M)[:, None]
```

We create a utility function for plotting:

```python
def plot_model(m, lower=-8.0, upper=8.0):
    pX = np.linspace(lower, upper, 100)[:, None]
    pY, pYv = m.predict_y(pX)
    if pY.ndim == 3:
        pY = pY[:, 0, :]
    plt.plot(X, Y, "x")
    plt.gca().set_prop_cycle(None)
    plt.plot(pX, pY)
    for i in range(pY.shape[1]):
        top = pY[:, i] + 2.0 * pYv[:, i] ** 0.5
        bot = pY[:, i] - 2.0 * pYv[:, i] ** 0.5
        plt.fill_between(pX[:, 0], top, bot, alpha=0.3)
    plt.xlabel("X")
    plt.ylabel("f")
    plt.title(f"ELBO: {m.elbo(data):.3}")
    plt.plot(Z, Z * 0.0, "o")
```

## Model the outputs of $f(x)$ directly
The three following examples show how to model the outputs of the model $f(x)$ directly. Mathematically, this case is equivalent to having:
\begin{equation}
f(x) = I g(x),
\end{equation}
i.e. $W = I$ and $P = L$.

### 1. Shared independent multi-output kernel (MOK) and shared independent inducing variables
Here the priors on all outputs are constrained to have the same kernel hyperparameters. We also share the inducing inputs between all outputs. The different GPs are independent both in the prior and the approximate posterior.

```python
# create multi-output kernel
kernel = gpf.kernels.SharedIndependent(
    gpf.kernels.SquaredExponential() + gpf.kernels.Linear(), output_dim=P
)
# initialization of inducing input locations (M random points from the training inputs)
Z = Zinit.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)
```

```python
# create SVGP model as usual and optimize
m = gpf.models.SVGP(kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, num_latent_gps=P)
print_summary(m)
```

name                                      	class    	transform       	prior  	trainable  	shape      	dtype  	value

SVGP.kernel.kernel.kernels[0].variance    	Parameter	Softplus        	       	True       	()         	float64	1.0

SVGP.kernel.kernel.kernels[0].lengthscales	Parameter	Softplus        	       	True       	()         	float64	1.0

SVGP.kernel.kernel.kernels[1].variance    	Parameter	Softplus        	       	True       	()         	float64	1.0

SVGP.likelihood.variance                  	Parameter	Softplus + Shift	       	True       	()         	float64	1.0

SVGP.inducing_variable.inducing_variable.Z	Parameter	Identity        	       	True       	(15, 1)    	float64	[[-5....

SVGP.q_mu                                 	Parameter	Identity        	       	True       	(15, 3)    	float64	[[0., 0., 0....

SVGP.q_sqrt                               	Parameter	FillTriangular  	       	True       	(3, 15, 15)	float64	[[[1., 0., 0....

```python
def optimize_model_with_scipy(model):
    optimizer = gpf.optimizers.Scipy()
    optimizer.minimize(
        model.training_loss_closure(data),
        variables=model.trainable_variables,
        method="l-bfgs-b",
        options={"disp": True, "maxiter": MAXITER},
    )

optimize_model_with_scipy(m)
```

```python
print_summary(m)
```

name                                      	class    	transform       	prior  	trainable  	shape      	dtype  	value

SVGP.kernel.kernel.kernels[0].variance    	Parameter	Softplus        	       	True       	()         	float64	0.9280219997458425

SVGP.kernel.kernel.kernels[0].lengthscales	Parameter	Softplus        	       	True       	()         	float64	0.7983779052885694

SVGP.kernel.kernel.kernels[1].variance    	Parameter	Softplus        	       	True       	()         	float64	1.2117302545578978

SVGP.likelihood.variance                  	Parameter	Softplus + Shift	       	True       	()         	float64	0.039660888636948194

SVGP.inducing_variable.inducing_variable.Z	Parameter	Identity        	       	True       	(15, 1)    	float64	[[-4.83950767...

SVGP.q_mu                                 	Parameter	Identity        	       	True       	(15, 3)    	float64	[[-0.9017192, 0.69909342, -1.47516425...

SVGP.q_sqrt                               	Parameter	FillTriangular  	       	True       	(3, 15, 15)	float64	[[[1.71258646e-02, 0.00000000e+00, 0.00000000e+00...

```python
# Plot predictions and observations
plot_model(m)
```

```python
print_summary(m.kernel)
m.kernel.kernel.kernels[0].lengthscales
```

name                                            	class    	transform  	prior  	trainable  	shape  	dtype  	   value

SharedIndependent.kernel.kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	0.928022

SharedIndependent.kernel.kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.798378

SharedIndependent.kernel.kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	1.21173

    <Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.7983779052885694>

### 2. Separate independent MOK and shared independent inducing variables
Here we allow different hyperparameters for the priors of each output. We still share the inducing inputs between all outputs.

```python
# Create list of kernels for each output
kern_list = [gpf.kernels.SquaredExponential() + gpf.kernels.Linear() for _ in range(P)]
# Create multi-output kernel from kernel list
kernel = gpf.kernels.SeparateIndependent(kern_list)
# initialization of inducing input locations (M random points from the training inputs)
Z = Zinit.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)
```

```python
# create SVGP model as usual and optimize
m = gpf.models.SVGP(kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, num_latent_gps=P)
```

```python
optimize_model_with_scipy(m)
```

    WARNING:tensorflow:From /home/circleci/project/.venv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:617: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Use fn_output_signature instead

```python
print_summary(m.kernel)
```

name                                                  	class    	transform  	prior  	trainable  	shape  	dtype  	   value

SeparateIndependent.kernels[0].kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	0.863966

SeparateIndependent.kernels[0].kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.934779

SeparateIndependent.kernels[0].kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	0.863697

SeparateIndependent.kernels[1].kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	0.750181

SeparateIndependent.kernels[1].kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.969772

SeparateIndependent.kernels[1].kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	0.531191

SeparateIndependent.kernels[2].kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	1.10845

SeparateIndependent.kernels[2].kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.748775

SeparateIndependent.kernels[2].kernels[1].variance    	Parameter	Softplus   	       	True       	()     	float64	2.23608

```python
plot_model(m)
```

```python
[k.kernels[0].lengthscales for k in m.kernel.kernels]
```

    [<Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.9347793068109054>,
     <Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.9697724701289211>,
     <Parameter: dtype=float64, shape=[], fn="softplus", numpy=0.7487746262609993>]

### 3. Separate independent kernel and separate independent inducing variables
Here we allow different hyperparameters for the priors of each output. We now allow different inducing inputs for each output.

```python
# Create list of kernels for each output
kern_list = [gpf.kernels.SquaredExponential() + gpf.kernels.Linear() for _ in range(P)]
# Create multi-output kernel from kernel list
kernel = gpf.kernels.SeparateIndependent(kern_list)
# initialization of inducing input locations, one set of locations per output
Zs = [Zinit.copy() for _ in range(P)]
# initialize as list inducing inducing variables
iv_list = [gpf.inducing_variables.InducingPoints(Z) for Z in Zs]
# create multi-output inducing variables from iv_list
iv = gpf.inducing_variables.SeparateIndependentInducingVariables(iv_list)
```

**NOTE:** While the inducing points are independent, there needs to be the same number of inducing points per dimension.

```python
# create SVGP model as usual and optimize
m = gpf.models.SVGP(kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, num_latent_gps=P)
```

```python
optimize_model_with_scipy(m)
```

```python
plot_model(m)
```

The following plot shows that we use different inducing *inputs* in each output dimension.

```python
for i in range(len(m.inducing_variable.inducing_variable_list)):
    q_mu_unwhitened, q_var_unwhitened = m.predict_f(m.inducing_variable.inducing_variable_list[i].Z)
    plt.plot(
        m.inducing_variable.inducing_variable_list[i].Z.numpy(),
        q_mu_unwhitened[:, i, None].numpy(),
        "o",
    )
plt.gca().set_xticks(np.linspace(-6, 6, 20), minor=True)
plt.gca().set_yticks(np.linspace(-9, 9, 20), minor=True)
plt.grid(which="minor")
```

```python
m.inducing_variable.inducing_variable_list
```

    ListWrapper([<gpflow.inducing_variables.inducing_variables.InducingPoints object at 0x7f21005c73c8>, <gpflow.inducing_variables.inducing_variables.InducingPoints object at 0x7f21005bdf98>, <gpflow.inducing_variables.inducing_variables.InducingPoints object at 0x7f21005bdb70>])

## Model $f(x)$ by doing inference in the $g$ space
### Mixed kernel and uncorrelated inducing variables

Remember the general case: $f(x) = W g(x)$, where $g(x) \in \mathbb{R}^L$, $f(x) \in \mathbb{R}^P$ and $W \in \mathbb{R}^{P \times L}$, where $L \leq P$.
We assume that the outputs of $g$ are uncorrelated, and by *mixing* them with $W$ they become correlated.
With this setup we perform the optimal routine to calculate the conditional. Namely, calculate the conditional of the uncorrelated latent $g$ and afterwards project the mean and variance using the mixing matrix: $\mu_f = W \mu_g$ and $\Sigma_f = W\Sigma_g W^\top$

- $K_{uu} = L \times M \times M$
- $K_{uf} = L \times M \times N$

```python
# Create list of kernels for each output
kern_list = [gpf.kernels.SquaredExponential() + gpf.kernels.Linear() for _ in range(L)]
# Create multi-output kernel from kernel list
kernel = gpf.kernels.LinearCoregionalization(
    kern_list, W=np.random.randn(P, L)
)  # Notice that we initialise the mixing matrix W
# initialisation of inducing input locations (M random points from the training inputs)
Z = Zinit.copy()
# create multi-output inducing variables from Z
iv = gpf.inducing_variables.SharedIndependentInducingVariables(
    gpf.inducing_variables.InducingPoints(Z)
)
```

```python
# initialize mean of variational posterior to be of shape MxL
q_mu = np.zeros((M, L))
# initialize \sqrt(Σ) of variational posterior to be of shape LxMxM
q_sqrt = np.repeat(np.eye(M)[None, ...], L, axis=0) * 1.0

# create SVGP model as usual and optimize
m = gpf.models.SVGP(
    kernel, gpf.likelihoods.Gaussian(), inducing_variable=iv, q_mu=q_mu, q_sqrt=q_sqrt
)
```

```python
optimize_model_with_scipy(m)
```

```python
plot_model(m)
```

## Illustration of GPflow's multi-output capabilities
This section shows the inheritance structure in GPflow's multi-output framework.

### Multi-output kernels (MOK) class diagram
We include three multi-output kernels:
- `SharedIndependent`: This kernel is included mainly as an illustration of specifying a conditional using the multiple dispatch framework. The same functionality is provided by using a normal kernel and passing in multiple approximate posteriors by stacking `q_mu` and `q_sqrt`.
- `SeparateIndependent`: This kernel allows you to use different priors for each output GP.
- `LinearCoregionalization`: This kernel describes the prior of the linear model of coregionalization. As shown previously, this implementation supports various inducing point approximations.

We include several base classes. Two are noteworthy:
- `MultioutputKernel` is included to be the base class for all multi-output kernels.
- `IndepedentLatent` is the base class for all multi-output kernels which are constructed from independent latent processes. Including this kernel allows the specification of a default approximation method which, while not the most efficient, does take advantage of _some_ structure. It can be applied to _any_ kernel constructed from independent latent processes.

There is a similarity in the meaning of `SeparateIndependent` and `IndependentLatent`. Both kernels indicate that independent processes are used, and that $\mathbf{K}_{\bf uu}$ can therefore be represented as a `[L, M, M]` tensor. It could therefore be suggested that `SeparateIndependent` be the parent class of all "independent latent" kernels, instead of having a separate `IndependentLatent` class. We decided against this because:
- this would increase the complexity in specifying `conditionals()` for the otherwise simple multi-output kernels `SeparateIndependent` and `SharedIndependent`.
- we did not want to specify too much of an implementation in `IndependentLatent`, leaving implementation details to child classes. Using `SeparateIndependent` as the base class would force all child classes to be a `Combination` kernel.

### Multi-output inducing variables class diagram

#### Inducing points
The goal of this class is to provide inducing variables that can be used with _any_ kernel, even if the method ends up being slow.

The multiouput framework extends `InducingPoints` to work with multi-output kernels. Just like for single-output kernels, we want `InducingPoints` to work for all `MultioutputKernel`s. We do this by defining `InducingPoints` to take _all_ outputs for specific inducing inputs as inducing variables.

#### Fallback shared/separate independent inducing variables
The goal of these classes is to provide a reasonably efficient implementation for kernels that give exploitable independence structure in the prior of inducing variables (that is, subclasses of `IndependentLatent`), while only needing to implement `Kuu()` and `Kuf()` methods.

#### Shared/separate independent inducing variables
The goal of these classes is to provide the most efficient code path for kernels that allow exploiting independence structure in the prior of inducing variables.

For more specialized multi-output kernels (i.e. `{Shared|Separate}Independent` or `LinearCoregionalization`) we define `{Shared|Separate}IndependentInducingVariables`. These wrap (a list of) single-output inducing variables to define groups of a-priori independent inducing variables, which leads to a $\mathbf{K}_{\bf uu}$ that can be represented as a `[L, M, M]` tensor. We saw the use of these previously.

`{Shared|Separate}IndependentInducingVariables` inherit from `Fallback{Shared|Separate}IndependentInducingVariables`, so the multiple dispatch will fall back on the slower but general implementation.

### Implemented combinations
Multiple dispatch is applied to both `Kuu()`, `Kuf()`, and `conditional()`. The return values of the covariances can therefore be tailored to a specific implementation of `conditional()`. The following table lists combinations which are currently available in GPflow. Thanks to the multiple dispatch code, implementing your own outside of GPflow should require only a small amount of code!

| Inducing variable class                                      | Kernel                  | Kuu           | Kuf           | conditional                         | note                                                                                |
|----------------------------------------------|-------------------------|---------------|---------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `InducingPoints`                               | `MultioutputKernel`       | `[M, P, M, P]` | `[M, P, N, P]` | `inducing_point_conditional()`, which calls `fully_correlated_conditional()`       | Works for all kernels, but might be very inefficient. In this case `q_mu` and `q_sqrt` should have shapes of `[1, MP]` and `[1, MP, MP]`  |
| `SharedIndependentInducingVariables`           | `SharedIndependent`       | `[M, M]`         | `[M, N]`         | `shared_independent_conditional()`, which calls `base_conditional()`                    | The combination of these two classes is in a sense redundant, because we can achieve the same behavior by using the single output Kernel and InducingVariable classes. They are added for illustrative purposes. Thanks to the conditional dispatch, the most efficient code path is used. |
| `SeparateIndependentInducingVariables`         | `SharedIndependent`       | `[P, M, M]`     | `[P, M, N]`     | `separate_independent_conditional()`, which calls `base_conditional()` P times               | We loop P times over the `base_conditional()`         |
| `SeparateIndependentInducingVariable`         | `SeparateIndependent`     | `[P, M, M]`     | `[P, M, N]`     |  `separate_independent_conditional()`, which calls `base_conditional()` P times                | We loop P times over the `base_conditional()`           |
| `SharedIndependentInducingVariables`           | `SeparateIndependent`     | `[P, M, M]`     | `[P, M, N]`     | `separate_independent_conditional()`, which calls `base_conditional()` P times                | We loop P times over the `base_conditional()`        |
| `FallbackSharedIndependentInducingVariables`   | `IndependentLatent`       | `[L, M, M]`     | `[M, L, N, P]` | `fallback_independent_latent_conditional()`, which calls `independent_interdomain_conditional()` | Implementation which only requires custom `Kuu()` and `Kuf()` |
| `FallbackSeparateIndependentInducingVariable` | `IndependentLatent`       | `[L, M, M]`     | `[M, L, N, P]` | `fallback_independent_latent_conditional()`, which calls `independent_interdomain_conditional()` | Implementation which only requires custom `Kuu()` and `Kuf()`  |
| `SharedIndependentInducingVariables`           | `LinearCoregionalization` | `[L, M, M]`     | `[L, M, N]`     | `coregionalization_conditional()`, which calls `base_conditional()`                    | This is the most efficient implementation for linear coregionalization. The inducing outputs live in g-space. Here we use the output of the base conditional and project the mean and covariance with the mixing matrix W.                                                                    |
| `SeparateIndependentInducingVariables`         | `LinearCoregionalization` | `[L, M, M]`     | `[L, M, N]`     | base_conditional                    | This is the most efficient implementation for linear coregionalization. The inducing outputs live in g-space. Here we use the output of the base conditional and project the mean and covariance with the mixing matrix W.                                                                    |

## Debugging: introspect

Given all these possibilities it can be hard to determine which conditional will be called for which set of kernel and inducing variable. The following method lets you proactively introspect which implementation will be executed. This can be useful when debugging new code.

```python
def inspect_conditional(inducing_variable_type, kernel_type):
    """
    Helper function returning the exact implementation called
    by the multiple dispatch `conditional` given the type of
    kernel and inducing variable.

    :param inducing_variable_type:
        Type of the inducing variable
    :param kernel_type:
        Type of the kernel

    :return: String
        Contains the name, the file and the linenumber of the
        implementation.
    """
    import inspect
    from gpflow.conditionals import conditional

    implementation = conditional.dispatch(object, inducing_variable_type, kernel_type, object)
    info = dict(inspect.getmembers(implementation))
    return info["__code__"]

# Example:
inspect_conditional(
    gpf.inducing_variables.SharedIndependentInducingVariables, gpf.kernels.SharedIndependent
)
```

## Further Reading:
- [A simple demonstration of coregionalization](./coregionalisation.ipynb), which details other GPflow features for multi-output prediction without fully observed outputs.


## Primeiro Trabalho Computacional: Álgebra, Números e Criptografia 2018

### Alifer Sales e Fernanda Scovino

**Data: 04/12/18**

Implementação do teste de Miller-Rabin para geração de números primos grandes.

---

### Teste de primalidade Miller-Rabin

Teste proposto por G.L. Miller em 1976. O teste avalia se um número $n$ é primo tomando por base o teorema de Fermat:

Seja $n$ um número primo e $a$ um número inteiro escolhido aleatoriamente, tal que $1 < b <n-1$. Seja $k=\max\{r\in \mathbb {N} : 2^{r}\mid (n-1)\}$ e $q = \frac{(n-1)}{2^{k}}$. Por definição de $s$, $d$ é necessariamente ímpar.

Sabemos, pelo pequeno teorema de Fermat, que se $n$ é um número primo e $mdc(b, n) = 1$, então $a^{n-1} \equiv b^{2^{k}q}\equiv 1\mod n$.

Seja $j \geq 1$ o menor expoente tal que $b^{2^{j}q}\equiv 1\mod n$, logo $b^{2^{j}q}\equiv (b^{2^{j-1}q}-1)(b{2^{j-1}q}+1) \equiv 1\mod n$ e n divide b{2^{j-1}q}+1, pois $j$ é o menor expoente tal que $b^{2^{j}q}\equiv 1\mod n$, pu seja,   $b^{2^{j-1}q}\equiv -1\mod n$.

Então, se n é primo, existe um $r\in \{0,1,\cdots ,k-1\}$ tal que $a^{2^{r}d}\equiv -1\mod n$. No caso de $j=0$, pode ser também que $b^{q}\equiv 1 \mod n$.

**Resumo**

* Das amostras geradas, o número de primos classificados errados em cada não ultrapassa 2,6%, somando menos de 2,2% % do total

**Fontes**

[1] *Números Inteiros e Criptografia RSA* — S. C. Coutinho

[2] [*How to generate big prime numbers — Miller-Rabin*. Antoine Prudhomme](https://medium.com/@prudywsh/how-to-generate-big-prime-numbers-miller-rabin-49e6e6af32fb)

[3] [*Teste de primalidade de Miller-Rabin*. Wikipédia](https://pt.wikipedia.org/wiki/Teste_de_primalidade_de_Miller-Rabin)

#### Pacotes auxiliares

```python
import matplotlib.pyplot as plt
import numpy as np
from pandas import DataFrame
from itertools import combinations
import random
```

#### Implementação

```python
def checa_primo(n, verbose):

    """
    Verifica o teste de primalidade de Miller.

    :param n: número candidato a primo
    :param verbose: booleano para exibição dos prints
    """

    # Fazemos os testes triviais
    if (n % 2 == 0) or (n % 3 == 0):

        if verbose:
            print('O último número testado é composto...')
        return False

    # Escolhemos k (número máximo de testes)
    q = n-1
    k = 0

    while q % 2 == 0:

        q = int(q/2)
        k +=1

    # Verificamos se n tem alta probabilidade de ser primo
    for i in range(k):

        b = np.random.randint(2, n - 1)
        r = pow(b, q, n)

        if (r != 1) and (r != n - 1):

            j = 1
            while (j < k) and (r != n - 1):

                # elevamos ao quadrado a potência
                r = pow(r, 2, n)

                if (r == 1):

                    if verbose:
                        print('O último número testado é composto...')
                    return False

                j += 1

            if (r != n - 1):

                if verbose:
                    print('O último número testado é composto...')
                return False

    if verbose:
        print('O número tem altas chances de ser primo!')
    return True

def teste_Miller(t, verbose=True):

    """
    Gera um número primo.

    :param t: número de testes a ser realizado
    :param verbose: booleano para exibição dos prints
    """

    if verbose:
        print('>>> Iniciando Teste 1...')

    # Geramos um candidato a primo grande
    n = np.random.randint(10**10, 10**12)

    i = 1
    while (not checa_primo(n, verbose)) and (i < t):

        if verbose:
            print('>>> Teste {} \n Candidato a primo: {}'.format(i, n))
        n = np.random.randint(10**10, 10**12)
        i += 1

    if i <= t:

        return n
```

```python
p = teste_Miller(100)
p
```

    >>> Iniciando Teste 1...
    O último número testado é composto...
    >>> Teste 1
     Candidato a primo: 399638026734
    O último número testado é composto...
    >>> Teste 2
     Candidato a primo: 195716752519
    O último número testado é composto...
    >>> Teste 3
     Candidato a primo: 981271346145
    O último número testado é composto...
    >>> Teste 4
     Candidato a primo: 706621774250
    O último número testado é composto...
    >>> Teste 5
     Candidato a primo: 139429199437
    O último número testado é composto...
    >>> Teste 6
     Candidato a primo: 113836711913
    O último número testado é composto...
    >>> Teste 7
     Candidato a primo: 393402269908
    O último número testado é composto...
    >>> Teste 8
     Candidato a primo: 330749653760
    O último número testado é composto...
    >>> Teste 9
     Candidato a primo: 854654764263
    O último número testado é composto...
    >>> Teste 10
     Candidato a primo: 815089848663
    O último número testado é composto...
    >>> Teste 11
     Candidato a primo: 987923881098
    O último número testado é composto...
    >>> Teste 12
     Candidato a primo: 879053720020
    O último número testado é composto...
    >>> Teste 13
     Candidato a primo: 601363971282
    O último número testado é composto...
    >>> Teste 14
     Candidato a primo: 992382060392
    O último número testado é composto...
    >>> Teste 15
     Candidato a primo: 626316648869
    O último número testado é composto...
    >>> Teste 16
     Candidato a primo: 184713852789
    O último número testado é composto...
    >>> Teste 17
     Candidato a primo: 818435488535
    O último número testado é composto...
    >>> Teste 18
     Candidato a primo: 79373733472
    O último número testado é composto...
    >>> Teste 19
     Candidato a primo: 977836960546
    O último número testado é composto...
    >>> Teste 20
     Candidato a primo: 408242354836
    O último número testado é composto...
    >>> Teste 21
     Candidato a primo: 479752220924
    O último número testado é composto...
    >>> Teste 22
     Candidato a primo: 339783847864
    O último número testado é composto...
    >>> Teste 23
     Candidato a primo: 308174145579
    O último número testado é composto...
    >>> Teste 24
     Candidato a primo: 517876224292
    O último número testado é composto...
    >>> Teste 25
     Candidato a primo: 377684999616
    O último número testado é composto...
    >>> Teste 26
     Candidato a primo: 127316224965
    O último número testado é composto...
    >>> Teste 27
     Candidato a primo: 210728548764
    O último número testado é composto...
    >>> Teste 28
     Candidato a primo: 46671591222
    O último número testado é composto...
    >>> Teste 29
     Candidato a primo: 10720449467
    O último número testado é composto...
    >>> Teste 30
     Candidato a primo: 332556784906
    O último número testado é composto...
    >>> Teste 31
     Candidato a primo: 510838150375
    O último número testado é composto...
    >>> Teste 32
     Candidato a primo: 987680444036
    O último número testado é composto...
    >>> Teste 33
     Candidato a primo: 573373328809
    O último número testado é composto...
    >>> Teste 34
     Candidato a primo: 304171799418
    O último número testado é composto...
    >>> Teste 35
     Candidato a primo: 294191493054
    O último número testado é composto...
    >>> Teste 36
     Candidato a primo: 806299599063
    O último número testado é composto...
    >>> Teste 37
     Candidato a primo: 153052161323
    O último número testado é composto...
    >>> Teste 38
     Candidato a primo: 440965163226
    O último número testado é composto...
    >>> Teste 39
     Candidato a primo: 982013272883
    O último número testado é composto...
    >>> Teste 40
     Candidato a primo: 156729842852
    O último número testado é composto...
    >>> Teste 41
     Candidato a primo: 515959138289
    O último número testado é composto...
    >>> Teste 42
     Candidato a primo: 450233483085
    O último número testado é composto...
    >>> Teste 43
     Candidato a primo: 537685282782
    O último número testado é composto...
    >>> Teste 44
     Candidato a primo: 481526844323
    O último número testado é composto...
    >>> Teste 45
     Candidato a primo: 344568211582
    O último número testado é composto...
    >>> Teste 46
     Candidato a primo: 298276197245
    O número tem altas chances de ser primo!

    739156430063

#### Resultados

Para verificar se o número é efetivamente primo, podemos importar o módulo `simpy.is_prime`.

```python
import sympy as sp

sp.isprime(p)
```

    True

E vamos gerar $100$ amostras com $5000$ números classificados como primos para verificarmos a eficácia do algoritmo.

```python
%%time

amostras = 50
tamanho = 5000

erros = []
for _ in range(amostras):

    primos = np.array([teste_Miller(100, verbose=False) for i in range(tamanho)])
    result = np.array(list(map(lambda x: sp.isprime(x), primos)))

    erro = (len(result) - sum(result))/len(result)
    erros.append(erro)
```

    CPU times: user 1min 38s, sys: 288 ms, total: 1min 38s
    Wall time: 1min 38s

```python
plt.plot(erros)
```

```python
e_max = max(erros)
e_max
```

    0.0266

Verificamos que, das amostras geradas, o número de primos classificados errados em cada não ultrapassa 2,6%.

```python
e_geral = sum(erros)*tamanho/(amostras*tamanho)
e_geral
```

    0.021748

E o erro geral (razão do número de classificados errados pelo total de números, somando todas as amostras) menor que 2,2%


Exercise material of the MSc-level course **Numerical Methods in Geotechnical Engineering**.
Held at Technische Universität Bergakademie Freiberg.

Comments to:

*Prof. Dr. Thomas Nagel
Chair of Soil Mechanics and Foundation Engineering
Geotechnical Institute
Technische Universität Bergakademie Freiberg.*

https://tu-freiberg.de/en/soilmechanics

```python
import numpy as np
import matplotlib.pyplot as plt

#Some plot settings
plt.style.use('seaborn-deep')
plt.rcParams['lines.linewidth']= 2.0
plt.rcParams['lines.color']= 'black'
plt.rcParams['legend.frameon']=True
plt.rcParams['font.family'] = 'serif'
plt.rcParams['legend.fontsize']=14
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.left'] = True
plt.rcParams['axes.spines.bottom'] = True
plt.rcParams['axes.axisbelow'] = True
plt.rcParams['figure.figsize'] = (12, 6)
```

```python
kf = 1.e-9
mu = 1.e-3
g = 9.81
rho = 1000.
k = kf*mu/(rho*g)
print(k)
```

    1.0193679918450563e-16

# Exercise 7 - 1D Consolidation, Terzaghi. Influence of mass lumping and FIC stabilization

## Governing differential equation

Neglecting lateral displacements and fluid flow

$$
	\epsilon_{xx} = \epsilon_{yy} = 0 \quad \text{and} \quad q_x = q_y = 0
$$

the volume balance simplifies to

$$
	\dot{\epsilon}_{zz} = \frac{\partial q_z}{\partial z}
$$

Now we substitute the constitutive law (linear elasticity) of the solid and Darcy's law for the fluid motion into the volume balance and get

$$
    \dot{\sigma}_{zz}^\text{eff} = E_\text{S} \dot{\epsilon}_{zz} \quad \text{and} \quad \frac{\partial q_z}{\partial z} = - \frac{k}{\mu_\text{F}} \frac{\partial^2 p}{\partial z^2} \quad \text{ergibt} \quad
		 \frac{\dot{\sigma}_{zz}^\text{eff}}{E_\text{S}} = - \frac{k}{\mu_\text{F}} \frac{\partial^2 p}{\partial z^2}
$$

Assuming constant total stress (constant external loads) we find with the use of the effective stress principle

$$
	\sigma_{zz} = \sigma_{zz}^\text{eff} + p = \text{const} \quad \rightarrow \quad \dot{\sigma}_{zz} = 0 = \dot{\sigma}_{zz}^\text{eff} + \dot{p}
$$

Substitution yields the partial, linear and homogeneous differential equation

$$
	\dot{p} = \frac{E_\text{S} k}{\mu_\text{F}} p_{,zz} = \frac{E_\text{S} K_\text{D}}{\gamma_\text{F}} p_{,zz} = c_\text{v} p_{,zz} \quad \text{with } K_\text{D} = \frac{k\rho_\text{F} g}{\mu_\text{F}}
$$

$[k] = \text{m}^2$: intrinsic permeability

$[K_\text{D}] = \text{m s}^{-1}$: hydraulic conductivity

$[c_\text{v}] = \text{m}^2\text{s}^{-1}$: coefficient of consolidation

## Weak form

The pore pressure can have (the essential/Dirichlet) boundary conditions in the form:

$$
    p = \bar{p}\ \forall z \in \partial \Omega_\mathrm{D}
$$

We now introduce a test function $\eta$ which vanishes where the pore pressure is given

$$
    \eta = 0\ \forall z \in \partial \Omega_\mathrm{D}
$$

and construct the weak form (using integration by parts):

\begin{align}
   0 &= \int \limits_0^H \eta \left[\dot{p} - c_\text{v} p_{,zz} \right] \text{d}z
   \\
   &= \int \limits_0^H \left[\eta \dot{p} - \left( \eta c_\text{v} p_{,z} \right)_{,z} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z
   \\
   &= \int \limits_0^H \left[\eta \dot{p} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z + \left[ \eta E_\text{S} q_z \right]^H_0
\end{align}

where the natural/Neumann boundary conditions have appeared.

## Finite elements in 1D

We have a soil column of height $H$ on top of the bed rock at $z=0$.

We first create an element class. An element knows the number of nodes it has, their IDs in the global node vector, and the coordinates of its nodes. Linear elements have 2 nodes and 2 quadrature points, quadratic elements 3 nodes and 3 quadrature points. The natural coordinates of the element run from -1 to 1, and the quadrature points and weights are directly taken from Numpy.

```python
#element class
class line_element():#local coordinates go from -1 to 1
    #takes number of nodes, global nodal coordinates, global node ids
    def __init__(self, nnodes=2, ncoords=[0.,1.], nids=[0,1]):
        self.__nnodes = nnodes
        if (len(ncoords) != self.__nnodes):
            raise Exception("Number of coordinates does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(ncoords)))
        else:
            self.__coords = np.array(ncoords)

        self.__natural_coords = (self.__coords-self.__coords[0])/(self.__coords[-1]-self.__coords[0])*2. - 1.

        if (len(nids) != self.__nnodes):
            raise Exception("Number of node IDs does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(nids)))
        else:
            self.__global_ids = np.array(nids)
        self.__quad_degree = self.__nnodes
        self.__quad_points, self.__quad_weights = np.polynomial.legendre.leggauss(self.__quad_degree)

```

Next, we wish to generate a one-dimensional mesh by specifying the length of a line, the number of elements into which the mesh is to be split, and the number of nodes per element.

```python
def number_of_nodes(nelems,nodes_per_elem):
    return nelems*nodes_per_elem - (nelems - 1)

def generate_mesh(domain_length,nelems,nodes_per_elem):
    nn = number_of_nodes(nelems,nodes_per_elem)
    #coordinate vector of global nodes
    global_nodal_coordinates = np.linspace(0.,domain_length,nn)
    global_solution = np.array([0.]*nn)

    #generate elements
    element_vector = []
    for i in range(nelems):
        node_start = (nodes_per_elem-1)*i
        element_vector.append(
            line_element(nodes_per_elem,
                         global_nodal_coordinates[node_start:node_start+nodes_per_elem],
                         list(range(node_start,node_start+nodes_per_elem))))

    return global_nodal_coordinates, element_vector, global_solution
```

## Shape functions in 1D

As in exercise 06, we allow linear and higher-order shape functions.

```python
#N
def shape_function(element_order,xi):
    if (element_order == 2): #-1,1
            return np.array([(1.-xi)/2., (1.+xi)/2.])
    elif (element_order == 3): #-1, 0, 1
            return np.array([(xi - 1.)*xi/2., (1-xi)*(1+xi), (1+xi)*xi/2.])
#dN_dxi
def dshape_function_dxi(element_order,xi):
    if (element_order == 2): #-1,1
        return np.array([-0.5*xi/xi, 0.5*xi/xi])  #xi only later for plotting dimensions
    elif (element_order == 3):#-1,0,1
        return np.array([xi - 0.5,-2.*xi,xi + 0.5])

#dz_dxi
def element_jacobian(element,xi):
    element_order = element._line_element__nnodes
    Jacobian = 0.
    Jacobian += dshape_function_dxi(element_order,xi).dot(element._line_element__coords)
    return Jacobian

#dN_dz
def grad_shape_function(element,xi):
    element_order = element._line_element__nnodes
    Jac = element_jacobian(element,xi)
    return dshape_function_dxi(element_order,xi)/Jac

```

## Time and space discretization, Picard iterations

Using a backward Euler approach for simplicity we find the time-discrete weak form as ($p_{n+1} \equiv p$)

$$
    0 = \int \limits_0^H \left[\eta \frac{p - p_n}{\Delta t} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z + \left[ \eta E_\text{S} q_z \right]^H_0
$$

Introducing standard FE approximations

$$
    p \approx N_i \hat{p}_i, \quad \eta \approx N_i \hat{\eta}_i, \quad \frac{\partial p}{\partial z} \approx \nabla  N_i \hat{p}_i, \quad \frac{\partial \eta}{\partial z} \approx \nabla N_i \hat{\eta}_i
$$

This yields

$$
\begin{align}
 0 &= \int \limits_0^H \left[ N_i \hat{\eta}_i \frac{N_k \hat{p}_k - N_k \hat{p}_{n,k}}{\Delta t} + \nabla N_i \hat{\eta}_i c_\text{v} \nabla N_k \hat{p}_k \right] \, \text{d}z + \left[ N_i \hat{\eta}_i E_\text{S} q_z \right]^H_0
 \end{align}
$$

Now we bring all quantities associated with the unknown pressure to the left-hand side (LHS) and all known quantities to the RHS:

$$
    \hat{\eta}_i \int \limits_0^H \left[ N_i \frac{1}{\Delta t} N_k + \nabla N_i c_\text{v} \nabla  N_k \right] \, \text{d}z\ \hat{p}_k =  \hat{\eta}_i \left[ N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=H} \delta_{n_\text{n}i} - N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=0} \delta_{i0} \right] +  \hat{\eta}_i \int \limits_0^H N_i \frac{1}{\Delta t} N_k \, \text{d}z\ \hat{p}_{n,k}
$$

can be simplified by realizing that the nodal test function values are arbitrary and thus

$$
    \int \limits_0^H \left[ N_i \frac{1}{\Delta t} N_k + \nabla N_i c_\text{v} \nabla N_k \right] \, \text{d}z\ \hat{p}_k = N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=H} \delta_{n_\text{n}i} - N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=0} \delta_{i0} +  \int \limits_0^H N_i \frac{1}{\Delta t} N_k \, \text{d}z\ \hat{p}_{n,k}
$$

which leaves us with $n_\text{n}$ equations for the $n_\text{n}$ unknown nodal pressures $\hat{p}_k$. If any coefficients in the above are taken as pressure-dependent, the system could be solved repeatedly usind Picard iterations. For strong non-linearities, a Newton linearization would typically be used.

### Question:

How would the equation differ for an explicit time integration scheme (forward Euler)?

What we require now is the local assembler to calculate the stiffness matrix and the local right-hand side. Local integration is performed by Gauss quadrature:

$$
    \int \limits_{-1}^1 f(\xi)\,\text{d}\xi \approx \sum \limits_{i=1}^{n_\text{gp}} f(\xi_i) w_i
$$

## Local assember

```python

```

```python
def Stiffness(z):
    E0 = 5.e6 #Pa
    return E0

def Conductivity(z):
    kf = 1.e-9#m/s
    return kf

def SpecificWeight(z):
    g = 9.81 #m/s²
    rhow = 1000. #kg/m³
    return g*rhow

def ConsolidationCoeff(z):#m²/s
    return Stiffness(z)*Conductivity(z)/SpecificWeight(z)
```

```python
def local_assembler(elem,dt,prev_sol,mass_lumping=False,stabilization=False):
    element_order = elem._line_element__nnodes
    K_loc = np.zeros((element_order,element_order))
    M_loc = np.zeros((element_order,element_order))
    b_loc = np.zeros(element_order)
    z_nodes = elem._line_element__coords
    for i in range(elem._line_element__quad_degree):
        #local integration point coordinate
        xi = elem._line_element__quad_points[i]
        #shape function
        N = shape_function(element_order,xi)
        #gradient of shape function
        dN_dX = grad_shape_function(elem,xi)
        #determinant of Jacobian
        detJ = np.abs(element_jacobian(elem,xi))
        #integration weight
        w = elem._line_element__quad_weights[i]

        #global integration point coordinate (for spatially varying properties)
        z_glob = np.dot(N,z_nodes)
        #evaluation of local material/structural properties
        E = Stiffness(z_glob)
        #evaluation of local body force
        CV = ConsolidationCoeff(z_glob)

        #assembly of local stiffness matrix
        M_loc = np.outer(N,N) / dt
        if (stabilization):
            #Stabilization term 1
            M_loc -= np.outer(N,dN_dX) * 0.5 * (z_nodes[-1] - z_nodes[0]) / dt
        if (mass_lumping):
            M_loc = np.diag(M_loc.sum(0)) #diagonal of column sum

        K_loc += (np.outer(dN_dX,dN_dX) * CV + M_loc)* w * detJ

        #assembly of local RHS
        p_prev = np.dot(N,prev_sol)#pressure in integration point
        b_loc += N * p_prev/dt * w * detJ
        if (stabilization):
            grad_p_prev = np.dot(dN_dX,prev_sol)#pressure gradient in integration point
            #Stabilization term 2
            b_loc -= N * grad_p_prev/dt * w * detJ * 0.5 * (z_nodes[-1] - z_nodes[0])
    return K_loc,b_loc
```

## Global assembly

Now we can construct the global matrix system $\mathbf{K}\mathbf{u} = \mathbf{f}$ or $\mathbf{A}\mathbf{x}=\mathbf{b}$ (see lecture script).

```python
def global_assembler(nodes,elements,solution,dt,mass_lumping=False,stabilization=False):
    K_glob = np.zeros((len(nodes),len(nodes)))
    b_glob = np.zeros(len(nodes))
    for i,elem in enumerate(elements):
        start_id = elem._line_element__global_ids[0]
        end_id = elem._line_element__global_ids[-1]
        K_i, b_i = local_assembler(elem,dt,solution[start_id:end_id+1],mass_lumping,stabilization)
        K_glob[start_id:end_id+1,start_id:end_id+1] += K_i
        b_glob[start_id:end_id+1] += b_i
    return K_glob, b_glob
```

## Application of boundary conditions

First we apply flux boundary conditions

```python
def apply_Neumann_bc(b_glob,node_id,value):
    b_glob[node_id] += value
    return b_glob
```

Then we apply Dirichlet bc

```python
def apply_Dirichlet_bc(K_glob,b_glob,node_id,value):
    K_glob[node_id,:] = 0.# = K_glob[:,node_id] = 0.
    K_glob[node_id,node_id] = 1.
    b_glob[node_id] = value
    return K_glob, b_glob
```

## Application of initial conditions

Since we're dealing with a time-dependent problem (rate problem) we require initial conditions for the pore pressure, i.e. $p_0 = p(t=0)\ \forall\ z$. Due to the very specific assumptions in deriving this consolidation equation, the initial pressure is given by the (suddenly) applied load:

$$
    p_0 = \sigma_{zz} = \text{const.}
$$

```python
def apply_initial_conditions(solution,sig_v):
    solution *= 0.
    solution += sig_v
    return
```

## Time loop and problem solution

We now establish the time loop and in each time step perform the global assembly, apply a vanishing traction on the top and constrain the displacement at the bottom to zero.

```python
def time_loop(dt,nodes,elements,solution,mass_lumping=False,stabilization=False):
    #Startwerte
    t_end = 366*24*60*60 #s
    absolute_tolerance = 1.e-6
    max_iter = 100
    iteration_counter = np.array([0])
    apply_initial_conditions(solution,200.e3)
    y = [solution] #create a list that will hold the solution vectors at all time points
    times = np.array([0.])
    #
    while times[-1]+dt < t_end: #repeat the loop as long as the final time step is below the end point
        times = np.append(times,times[-1]+dt) #here define the next time point as the previous time point plus the time increment dt
        y_old = y[-1] #Starting value for recursive update
        i = 0
        #
        while True:
            K, f = global_assembler(nodes,elements,y[-1],dt,mass_lumping,stabilization)
            #f = apply_Neumann_bc(f,len(nodes)-1,0)
            K, f = apply_Dirichlet_bc(K, f, len(nodes)-1, 0.)#free draining top
            solution = np.linalg.solve(K,f)
            i += 1
            if (np.abs(np.linalg.norm(solution) - np.linalg.norm(y_old)) < absolute_tolerance or i > max_iter): #if change is below tolerance, stop iterations
                break
            y_old = solution #preparation of next recursion

        y.append(solution) #append the new found solution to the solution vector
        iteration_counter = np.append(iteration_counter,i) #store how much iterations this time step took to converge
    return times, y,iteration_counter
```

```python
#spatial discretization
H = 10.
nel = 20
n_per_el = 3
nodes,elements,solution=generate_mesh(H,nel,n_per_el)

```

```python
times, sols, iters = time_loop(24*60*60,nodes,elements,solution,False,False)
```

```python
plt.xlabel('$z$ / m')
plt.ylabel('$p$ / kPa')
plt.title('Finite element solution')
plt.plot(nodes, sols[0]/1.e3, marker='o', label='$t = %i$ d' %(times[0]/60/60))
plt.plot(nodes, sols[1]/1.e3, marker='o', label='$t = %i$ d' %(times[1]/60/60/24))
plt.plot(nodes, sols[10]/1.e3, marker='o', label='$t = %i$ d' %(times[10]/60/60/24))
plt.plot(nodes, sols[30]/1.e3, marker='o', label='$t = %i$ d' %(times[30]/60/60/24))
plt.plot(nodes, sols[200]/1.e3, marker='o', label='$t = %i$ d' %(times[200]/60/60/24))
plt.plot(nodes, sols[-1]/1.e3, marker='o', label='$t = %i$ d' %(times[-1]/60/60/24))
plt.legend();
```

```python
#from matplotlib import animation
#from IPython.display import HTML

# First set up the figure, the axis, and the plot element we want to animate
#fig, ax = plt.subplots();
#ax.set_xlim(( 0, 11));
#ax.set_ylim((0, 210));

#line, = ax.plot([], [], lw=2);

# initialization function: plot the background of each frame
#def init():
#    line.set_data(nodes, sols[0]/1e3)
#    return (line,)

# animation function. This is called sequentially
#def animate(i):
    #x = nodes
    #y = sols[i]/1e3
    #line.set_data(x, y)
    #return (line,)

# call the animator. blit=True means only re-draw the parts that have changed.
#anim = animation.FuncAnimation(fig, animate, init_func=init,
#                               frames=len(sols)-1, interval=20, blit=True);
#HTML(anim.to_html5_video())
```

### Convergence study with mass lumping

```python
dts = [12.*3600,24.*3600,48.*3600]
nels = [10,20,40]
H = 10.

fig, ax = plt.subplots(nrows=3,ncols=3,figsize=(18,18))

for i,dt in enumerate(dts):
    for j,nel in enumerate(nels):
        #print("Running (n,dt) combination ", dt, nel)
        number_of_elements = nel
        nodes_per_element = 2
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        times, sols, iters = time_loop(dt,nodes,elements,solution)
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        #element size and random fore-factor
        times_lumped, sols_lumped, iters_lumped = time_loop(dt,nodes,elements,solution,True,False)
        #
        ax[i][j].plot(nodes, sols[0]/1.e3, marker='o', color='green', label='$t = %i$ h' %(times[0]/60/60))
        ax[i][j].plot(nodes, sols[1]/1.e3, marker='o', color='red', label='$t = %i$ h' %(times[1]/60/60))
        ax[i][j].plot(nodes, sols[10]/1.e3, marker='o', color='blue', label='$t = %i$ h' %(times[10]/60/60))
        #
        ax[i][j].plot(nodes, sols_lumped[0]/1.e3, marker='d', ls=':', color='green')
        ax[i][j].plot(nodes, sols_lumped[1]/1.e3, marker='d', ls=':', color='red')
        ax[i][j].plot(nodes, sols_lumped[10]/1.e3, marker='d', ls=':', color='blue')
        #
        ax[i][j].set_xlabel('$z$ / m')
        ax[i][j].set_ylabel('$p$ / kPa')
        ax[i][j].set_title('dt = %i h, nel = %i' %(dt/3600,nel))
        ax[i][j].legend(fontsize=10)
fig.tight_layout()
```

Mass lumping (dashed lines) reduces the oscillations but leads to an overly diffusive response particularly on coarse grids.

### Convergence study with FIC stabilization

```python
dts = [12.*3600,24.*3600,48.*3600]
nels = [10,20,40]
H = 10.

fig, ax = plt.subplots(nrows=3,ncols=3,figsize=(18,18))

for i,dt in enumerate(dts):
    for j,nel in enumerate(nels):
        #print("Running (n,dt) combination ", dt, nel)
        number_of_elements = nel
        nodes_per_element = 2
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        times, sols, iters = time_loop(dt,nodes,elements,solution)
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        #element size and random fore-factor
        times_lumped, sols_lumped, iters_lumped = time_loop(dt,nodes,elements,solution,False,True)
        #
        ax[i][j].plot(nodes, sols[0]/1.e3, marker='o', color='green', label='$t = %i$ h' %(times[0]/60/60))
        ax[i][j].plot(nodes, sols[1]/1.e3, marker='o', color='red', label='$t = %i$ h' %(times[1]/60/60))
        ax[i][j].plot(nodes, sols[10]/1.e3, marker='o', color='blue', label='$t = %i$ h' %(times[10]/60/60))
        #
        ax[i][j].plot(nodes, sols_lumped[0]/1.e3, marker='d', ls=':', color='green')
        ax[i][j].plot(nodes, sols_lumped[1]/1.e3, marker='d', ls=':', color='red')
        ax[i][j].plot(nodes, sols_lumped[10]/1.e3, marker='d', ls=':', color='blue')
        #
        ax[i][j].set_xlabel('$z$ / m')
        ax[i][j].set_ylabel('$p$ / kPa')
        ax[i][j].set_title('dt = %i h, nel = %i' %(dt/3600,nel))
        ax[i][j].legend(fontsize=10)
fig.tight_layout()
```

The FIC stabilization (dashed lines) reduces the oscillations and causes less artificial diffusion compared to  mass lumping.

```python

```