#
$\quad$Uncertainty in Mathematical Models

 ENGSCI263: Engineering Science Design I

$\,$

 Department of Engineering Science

$\,$

```python
# imports and environment: this cell must be executed before any other in the notebook
%matplotlib inline
from uncertainty263 import*
```

When using mathematical models to describe the real-world, we must contend with a staggering amount of **ignorance**1. Indeed, mathematical modelling as an activity is motivated by our own ignorance. Perhaps we are interested in the unknown value of some system parameter? Or maybe we harbour a general concern about the unpredictable future? Or, turning it around, if we already **know** the answer, what is the point of modelling?

**Minimizing** the effects of error and uncertainty, and articulating **quantitatively** that which remains, are nontrivial undertakings. Too often, these steps are neglected: a model prediction is made that, in the absence of any disclosure of uncertainty, is **implied to be perfect**. Vague caveats might sometimes be offered about the prediction's quality in the face of some model simplification or assumed parameter value. However, the greatest confidence is attained when model predictions are attached **probabilistic bounds**, e.g., "*by 2024, the largest earthquake triggered by gas extraction operations has a 90% likelihood of lying in the interval [3.8, 4.6], with the most likely outcome being a M 4.2*".

Error and uncertainty enter into the modelling process **at all stages**, for instance:
- When a model is first developed and we are required to make **simplifying assumptions** (structural error).
- When **imperfect or incomplete measurements** (observation error) are used to calibrate the model.
- When the model is used to **make a prediction** of the future (prediction uncertainty), even though its parameters are **not completely constrained** (parameter uncertainty).

Because uncertainty can never be eliminated entirely, it is important that rigorous methods are developed to handle it. For this module, you will be assessed on your ability to, for a specific scenario, **identify** where significant uncertainty could arise and, where possible, **quantify** its impact on modelling outcomes.

1  Note, this is not the type of ignorance that carries negative connotations, i.e., obstinate closed-mindedness, an indifference to (or even a conscious intent to ignore) fact or reason. Rather, we are referring to the gaps in our knowledge: always present, but recognised, and with a desire to minimise.

## 1	Observational error

When a quantity, $y_i$, is measured twice in succession, $\tilde{y}_i^{(1)}$ and $\tilde{y}_i^{(2)}$, it is unlikely that the same value will be obtained. Assuming that the true2 underlying value, $y_{i,0}$, is not changing, differences between repeated measurements, $\tilde{y}_i^{(j)}$ are generally attributed to **random error**, $\epsilon$, that reflects instrument limitations or uncontrollable variables. In the absence of systematic error, the measurement process is expressed algebraically

\begin{equation}
\tilde{y}_i^{(j)}=y_{i,0}+\epsilon,
\end{equation}

where $\epsilon$ is a normally distributed random variable with zero mean and variance, $\sigma_i^2$, i.e., $\epsilon\sim\mathcal{N}(0,\sigma_i^2)$. While any particular measurement might be contaminated with significant random error, as we take the mean of **repeated measurements**, the effects of measurement error will be averaged out.

We could replace the true value of $y_i$ above with the model, $f(x_i;\boldsymbol{\theta})$, to obtain a model for the observation process

\begin{equation}
\tilde{y}_i (\boldsymbol{\theta})=f(x_i;\boldsymbol{\theta})+\mathcal{N}(0,\sigma_i^2 ),
\end{equation}

which is a normal distribution centred on the model prediction, $f(x_i;\boldsymbol{\theta})$, i.e., $P(\tilde{y}_i│\boldsymbol{\theta})\sim\mathcal{N}(f(x_i;\boldsymbol{\theta}),\sigma_i^2)$.

The notation $P(\tilde{y}_i│\boldsymbol{\theta})$ should be read (slowly, several times over in your head) "*the probability of obtaining an observation, $\tilde{y}_i$, given a model of the process, $f(\cdot)$, which has the parameters, $\boldsymbol{\theta}$*" and it expresses the idea that "*the (underlying, true world) parameters **determine** the observations*".

**Observations** are the critical ingredient in model calibration. They serve as a sort of target that the model should be able to, at least approximately, replicate. What should we do then when the target is fuzzy or partially obscured by observational error? Observations provide **information** about parameters, and the purpose of calibration is to extract that information. Thus a consequence of observational error is to place **limits** on what can be learned about the parameters.

2  We shall use the subscript 0 to indicate a quantity is the "*true*", i.e., real world, value.

### 1.1 A Linear Example

***We shall return to the example below throughout this notebook.***

Consider some physical process that happens to be well-described by a linear model. Linear models have two parameters: a gradient, $m$, and $y$-intercept, $c$, i.e., $y_i=f(x_i;m,c)=mx_i+c$. For the purposes of demonstration, we shall cast ourselves in the role of **Omniscient Supreme Being**, in which case we are able to “know” the true parameter values: $m=2$ and $c=3$, or $\boldsymbol{\theta}_0=[2.0,3.0]$. However, the less-enlightened humans on the ground3 don’t have access to this information, and must instead satisfy themselves by making measurements and developing a model.

Being government employees, their instruments are not wildly sensitive ($\sigma_i^2=0.1$) and there is only budget to make ten observations, $\tilde{y}_i$. Remember: as almighty overlords, we can see both the true model (blue line) and the observations (open circles), but the poor humans can only see the latter.

3  Simple-minded plebs.

***Execute the cell below and answer the questions.***

```python
observation_error()
```

```python
# Move the slider N_obs to show more or less observations.

# What do the plotted normal distributions represent?

# Click the "True Process" checkbox. What does the centre of each normal distribution correspond to?

# Click the "Best Model" checkbox. Do the Best and True models agree? Why or why not?

# Click "ROLL THE DICE" and consider (i) the observations, (ii) the Best Model, and (iii) the True Process.
# Which ones have changed and which ones have stayed the same?

# Move the variance slider.
# How does agreement between Best and True models change as the variance decreases?

# Move the N_obs slider.
# How does agreement between Best and True models change as the more observations are taken?

```

## 2	Parameter uncertainty

To make a prediction with a model, a conscious choice must first be made about the values of **parameters** used as model inputs. However, parameters values are never known with **absolute certainty**. Instead, they are better characterised by a distribution4, which expresses our **belief about the true value** for that parameter. The two most common distributions are the uniform and normal distributions.

Distributions express our ignorance. The **uniform** distribution bounds the value of a parameter, $\boldsymbol{\theta}_i$, between minimum and maximum values, $[\boldsymbol{\theta}_{min},\boldsymbol{\theta}_{max}]$ and expresses the notion "*the true parameter value is somewhere in this range, but I have no idea where*". The **normal** distribution goes further, expressing what we think is the most likely value of the parameter (the mean) but also attaching a degree of uncertainty (the variance). Rarely, though, will the true parameter value turn out to be exactly the mean5.

4  For some parameters, the distribution might be quite narrow, e.g., the value of gravity. But even gravitational constants are only known to a finite level of precision.

5  A pregnant woman’s "due date" is the **most-likely** date of birth given the estimated date of conception and a distribution for the gestation time of a human foetus. That being said, only 4% of babies are actually born on their due date. So "most-likely" is certainly not the same as "likely".

***Execute the cell below and answer the questions.***

```python
priors()
```

```python
# What is the area under the uniform distribution (left)?

# How does the height of the uniform distribution change as you slide the limits left and right?

# What is the area under the normal distribution (right)?

# Does a uniform distribution categorically rule out any parameter values? What about the normal distribution?

```

When a parameter distribution is presented by the modeller based solely on consultation of the relevant literature or by drawing on their own expert knowledge, then it is called a **prior**. In Bayesian terms, the prior is a quantitative statement of belief (or ignorance) about a parameter’s value. When a set of observations, $\tilde{y}_i$, is used to provide further constraints on the parameter’s value (say, through model calibration), then we obtain a new distribution, called the **posterior**. Again, this is a quantitative statement of belief that, now, incorporates the relevant observations alongside our prior notions.

For the case of a uniform prior ("*I have no idea what the value of this parameter is...*"), a reformulation of the weighted sum-of-squares objective function, $S(\boldsymbol{\theta})$, provides a simple way to determine the posterior parameter distribution. Defining $S(\boldsymbol{\theta})$ as6:

\begin{equation}
	S(\boldsymbol{\theta})=\sum\limits_{i=1}^N\frac{1}{\sigma_i^2}\left(\tilde{y}_i-f(x_i;\boldsymbol{\theta})\right)^2,
\end{equation}

then the posterior distribution over multi-dimensional parameter space7 is

\begin{equation}
	P(\boldsymbol{\theta}|\tilde{y}_i)=A e^{-S(\boldsymbol{\theta})/2},
\end{equation}

where $A$ is a normalising constant for the probability distribution. A proper understanding of what $P(\boldsymbol{\theta}|\tilde{y}_i)$ actually represents requires a little quiet introspection on the following points:
1. **Information** about the true parameters, $\boldsymbol{\theta}_0$, is obtained by making **measurements**, $\tilde{y}_i$.
2. All measurements, even the most meticulous ones, contain some amount of **error**, $\epsilon$.
3. Therefore, the true parameters, $\boldsymbol{\theta}_0$, can **never be determined** exactly.
4. However, we can find parameters, $\hat{\boldsymbol{\theta}}_0$, that give a model **most consistent** with the data (found by minimising $S(\boldsymbol{\theta})$, i.e., calibration).
5. We concede that $\hat{\boldsymbol{\theta}}_0$ is not equal to $\boldsymbol{\theta}_0$, although it is **hopefully close**.
6. Therefore, it is incumbent upon us to consider the multitude of other parameter sets, $\boldsymbol{\theta}$, that are near to $\hat{\boldsymbol{\theta}}_0$, and thus are also **possible candidates** for $\boldsymbol{\theta}_0$.

The equation for $\tilde{y}_i (\boldsymbol{\theta})$ in Section 1 expresses a forward model for the measurement process: parameters ($\boldsymbol{\theta}$) determine observations ($\tilde{y}_i$). The posterior distribution $P(\boldsymbol{\theta}|\tilde{y}_i)$ expresses exactly the **opposite**, that observations determine8 the parameters, or "*what is the probability that the true parameter values are $\boldsymbol{\theta}$, given that I have made measurements, $\tilde{y}_i$*"9. Again, because $P(\boldsymbol{\theta}│\tilde{y}_i)$ is a distribution, it expresses both our knowledge ("*These values are likely...*") and our ignorance ("*...but we can't choose just one*").

6  Note, compared to the definition in the Calibration notebook, the weights on each term are the reciprocal of the variance for the associated measurement error. Fun fact to annoy your friends: when a set of measurements have different error, $\sigma_i^2$, they are said to be **heteroskedastic**.

7  For a derivation, see Appendix A. This is not examinable.

8  We don’t mean "determine" in the **deterministic** sense, i.e., to cause the parameters to take on particular values. Rather, we refer to the **inference** sense, i.e., to provide the information necessary to constrain the parameter values.

9  You have probably heard the old trope "*we cannot prove a hypothesis, only falsify it*". The posterior distribution has a nice interpretation in this context: each parameter set represents the hypothesis "*this is the true parameter set, $\boldsymbol{\theta}_0$*". By computing the posterior, we falsify (probabilistically) some of those hypotheses.

### 2.1	Computing posterior parameter distributions

According to the definition above, to compute the posterior we first need to calculate the **sum-of-squares objective function**, $S(\boldsymbol{\theta})$, a surface over multi-dimensional parameter space. For our purposes, it is sufficient to approximate this surface by computing $S(\boldsymbol{\theta}_k)$, where $\boldsymbol{\theta}_k$ are the points of a **discretised grid**. This is a grid search10, a brute force method that nevertheless parallelises well11. Once $S(\boldsymbol{\theta}_k)$ has been computed, it can be substituted into equation for the posterior to obtain the discretised probability density function. The coefficient $A$ is determined by numerical integration of $e^{-S(\boldsymbol{\theta})/2}$ using a suitable quadrature scheme.

Computing $P(\boldsymbol{\theta}_k |\tilde{y}_i)$  on a grid of parameters is **equivalent** to the assumption of a uniform prior: we are implicitly saying "*the likelihood of $\boldsymbol{\theta}_k$ being the true parameter, $\boldsymbol{\theta}_0$, is related only to its goodness-of-fit with the data and not to any prior notions I have about what $\boldsymbol{\theta}_0$ should be.*" Once $P(\boldsymbol{\theta}_k |\tilde{y}_i)$ has been computed, it should be **inspected** to assess whether the parameter grid is sufficiently large: $P(\boldsymbol{\theta}_k |\tilde{y}_i)$  should be **small** at the edges of the grid.

10  A grid search was used to generate contours of $S(\boldsymbol{\theta})$ for Figs. 1 and 2 in the Calibration notes, and some of the figures in the first two labs.

11  Computing $S(\boldsymbol{\theta}_1)$, which involves a forward run of the model for parameters $\boldsymbol{\theta}_1$, does not depend on knowing $S(\boldsymbol{\theta})$ at any other point. Contrast this to the gradient descent algorithm: we cannot compute $\boldsymbol{\theta}_4$ without first knowing $\boldsymbol{\theta}_3$.

### 2.2	Marginal parameter distributions

Suppose there are two parameter – let’s consider $c$ and $k$ again, for the case of damped car suspension – in which case $P(c,k|\tilde{y}_i)$ is a surface over 2D space. If we were able to determine the value of one of the parameters, say $c=c_0$, by other methods, the uncertain distribution of $k$ that remains is found by taking a slice through $P(c,k|\tilde{y}_i)$ at $c=c_0$ (see Fig. 2).

Another question we might ask runs along the lines: "*what is the uncertain distribution of $k$ given the uncertainty of $c$?*" or, alternatively "*yes, $c$ is uncertain, but I don’t care; I just need to know the uncertainty of $k$.*" This is called the **marginal distribution** of $k$ and is obtained by integrating $P(c,k|\tilde{y}_i)$ over $c$

\begin{equation}
P(k|\tilde{y}_i)=\int P(c,k|\tilde{y}_i)dc.
\end{equation}

An illustration of marginal distributions and their relation to the posterior is shown in Fig. 1.

**Figure 1: A sample objective function (left) over $c$-$k$ parameter space, and the corresponding posterior and marginal distributions (right). Note how $P(\boldsymbol{\theta}|\tilde{y}_i)$ goes to zero at the edges of the grid, indicating the distribution is well resolved.**

### 2.3 A Linear Example

***Recall the example from Section 1.1.***

Our group of human scientists have satisfied themselves that the physical process described previously is probably linear in nature and, therefore, a model of the form $y_i=mx_i+c$ will provide a suitable fit to the data. Using **gradient descent** calibration, they quickly determine that the best-fitting model has $m=2.2$ and $c=3.1$, i.e., $\hat{\boldsymbol{\theta}}_0=[2.2,3.1]$. Clearly, this is different to the true value, $\boldsymbol{\theta}_0=[2.0,3.0]$ (Section 1.1), but that was expected.

To gain a more in-depth understanding of the possible candidate models different from $\hat{\boldsymbol{\theta}}_0$, a grid search is undertaken to first determine $S(\boldsymbol{\theta})$ and then subsequently compute a posterior, $P(\boldsymbol{\theta}|\tilde{y}_i)$. The resulting distribution can be seen by running the cell below. Remember, the posterior distribution is making two important points:

1. We don’t currently know, in fact we will never know, the true parameter, $\boldsymbol{\theta}_0$. Even our best guess, the least-squares fit, $\hat{\boldsymbol{\theta}}_0$, will be wrong to some degree.
2. Nevertheless, we can determine which parameter combinations are consistent with the data, and rank their likelihood relative to each other.

Given these caveats, it is comforting to see that the true value, $\boldsymbol{\theta}_0$, is near enough to the centre of mass of $P(\boldsymbol{\theta}|\tilde{y}_i)$ that it is among the likely candidates.

***Execute the cell below and answer the questions.***

```python
Nm = 41
Nc = 41
posterior(Nm,Nc)
```

```python
# The righthand figure is computed by a grid search over c-m parameter space.

# Use the sliders to fit different models (left) and explore parameter space (right).

# The best-fit model is not the same as the true process. Why?

# Does the best-fit model or the true process correspond to the higher value on the righthand plot?

# The righthand inset shows the posterior as a surface. The shape is 2D Gaussian.

# If we were to fit a set of major and minor axes to the posterior, would they align with the c- and m- axes?

# Change the values of Nm and Nc in the cell above and rerun. How many grid samples are required to
# 'properly' estimate the posterior?

# What tradeoffs are there in estimating the posterior?

```

Note how the posterior - a 2D Gaussian distribution - is **rotated** with major and minor axes mis-aligned to the $c$- and $m$-axes. This indicates that the parameters $c$ and $m$ are **correlated** or, in this case, anti-correlated. In simpler terms, if we attempt to fit a steeper line (higher $m$) then we need a smaller intercept (smaller $c$) to obtain a reasonable match with the data.

In the context of **inverse modelling**, where the entire exercise is motivated by a desire to determine $\boldsymbol{\theta}$, then the posterior parameter distribution (and the corresponding marginal distributions) communicate appropriately both our **best estimate** of $\boldsymbol{\theta}$ (the vector of values that maximise $P(\boldsymbol{\theta}|\tilde{y}_i))$ as well as our **uncertainty**. However, when the goal is to make a prediction of the future, then model calibration is only an intermediate step. We must now consider how uncertainty in the parameters results in uncertainty in the predictions.

## 3	Prediction uncertainty

If we were **absolutely certain** that we had discovered the true values of all parameters, $\boldsymbol{\theta}_0$, then we could simply plug these into the forward model, $f(x_f;\boldsymbol{\theta}_0)$, to obtain a [rock solid prediction of the future](https://media.socastsrm.com/wordpress/wp-content/blogs.dir/697/files/2017/12/THE-ROCK-PRESIDENT-1.jpg), $y_f$. Unfortunately, when it comes to modelling, absolute certainty is in short supply. In fact, we saw in the previous section that, when observational error is allowed for, our best estimate, $\hat{\boldsymbol{\theta}}_0$, is always going to be wrong to some degree, which means the prediction, $f(x_f;\hat{\boldsymbol{\theta}}_0)$ **is always going to be wrong** as well. Oh dear. Not a great start.

If we were absolutely **averse to risk**, then we might take a no-surprises approach and say "*what are all the possible parameter sets, $\boldsymbol{\theta}$? Let’s predict the future for each of these, and then at least we’ll have an idea of the possible outcomes.*" This is equivalent to sampling parameter sets from a uniform prior and yields, not one, but an [**ensemble of models**](https://xkcd.com/1885/) with an ensemble of outcomes. In the context of, say, climate modelling, this ensemble of outcomes might tell us that the 21st century could see anywhere between 5$^{\circ}$C of warming and 2$^{\circ}$C of cooling, although neither outcome is ranked more or less likely than the other.

In fact though, we do know that some predictions are **more likely** to come true than others, because some of the parameter sets are more likely to be correct than others. For instance, suppose we have used some observations to compute a posterior distribution for the parameters, $P(\boldsymbol{\theta}|\tilde{y}_i)$, from which we learn that $\boldsymbol{\theta}_1$ is twice as likely to be the true parameter set than $\boldsymbol{\theta}_2$. Then, we would imagine the outcome $y_{f,1}=f(x_f;\boldsymbol{\theta}_1)$ to, similarly, be twice as likely as the outcome $y_{f,2}=f(x_f;\boldsymbol{\theta}_2)$. We shall use this reasoning as the basis for constructing a **future forecast**12, which is simply a set of predictions with a probability attached to each.

To construct the forecast, we take samples of possible parameter sets from the posterior distribution, $P(\boldsymbol{\theta}|\tilde{y}_i)$ and then compute a prediction for each. The posterior is not uniform but rather is weighted toward the most-likely parameter combinations and, therefore, the predictions too will be **weighted** towards the most-likely outcomes. As more and more samples are computed, a histogram of the predictions – the forecast – will begin to resemble a probability distribution, $P(y_f|\tilde{y}_i)$, which can be subsequently interrogated for its statistical properties. Of particular interest are the median outcome and an **interval** centred on it that encloses some substantial proportion of the outcomes13.

**How many** samples from the posterior should be used to generate the forecast? The answer depends on how long the forward model takes to run, the number of computers available for running models14, and the diminishing returns in resolving $P(y_f |\tilde{y}_i)$.

12  Here, we define the term forecast to mean a probabilistic prediction. However, take care, the terminology may mean different things to different people.

13  90% is good number, although a high stakes outcome may call for a wider interval.

14  Like the grid search, this step parallelises well

### 3.1 A Linear Example

***Recall the example from Section 2.3.***

The stupid humans wish to use their calibrated model and understanding of the posterior to make a prediction. For this rather contrived scenario, they will attempt to predict the physical process at $x_f=3.0$, which might be regarded as "*about twice as far into the future than the period we have data for, [0-1]*". The true future outcome is $y_{f,0}=f(x_f;\boldsymbol{\theta}_0 )=9$, but this is only learned long after the prediction is made. The prediction made using just the best-fit model is $y_f=f(x_f;\hat{\boldsymbol{\theta}}_0)=9.75$. Not bad for mere humans! But also wrong.

To be more rigorous, it is desirable to develop some sort of confidence in the prediction, $y_f$. As per Section 3, sample parameter sets are chosen15 from the posterior distribution developed in Section 2.3, and a prediction is computed for each. Compiling a histogram of 2000 such samples – a forecast of the future – we can see that the 90% confidence interval is rather wide, [6, 13], but it **does contain** the true future outcome.

15  For this step, I fitted a multivariate normal distribution and sampled from it using the `scipy.stats.multivariate_normal` class in Python. In practical situations, the posterior may not have a nice shape and alternative sampling methods might be required (e.g., bootstrapping).

***Execute the cell below and answer the questions.***

```python
var = 0.1
prediction(var)
```

```python
# The figures above show:
# (left) the modelled process
# (middle) the sampled posterior
# (right) the model forecast

# Add some more samples using the slider.
# How does the forecast change (width and shape) as you add more samples?

# Extend the forecast out using the slider for xf.
# How does the forecast change (width and shape) as you consider more distant futures?

# Increase the variance on the observation by changing 'var' and rerunning the cell. Try 0.01 and 1.0.
# How do the posterior and forecast change as you increase the observation error?

```

## 4	Structural error

While definitions of **structural (or model) error** can be a little grey, here we are referring to any item falling under the rather broad umbrella of "*improper model formulation*" or "*deviations between model and reality*". This includes all model assumptions related to:
- The governing physics, e.g., the differential equations being solved neglect gravity, which in practice has some influence on the outcome, but in this case is **assumed to be minor**.
- Time and space discretization, e.g., to solve the problem in a reasonable amount of time, we often must solve on a grid comprising a finite number of points and advancing the solution with finite sized time steps. Anything in between might be inferred by an interpolation scheme, but is **not explicitly captured** by the model.
- Represented features and homogenization. For instance, in a continuum model we might choose to **exclude** a finite-sized feature that has different properties to the rest of the continuum (an example from geothermal reservoir modelling is to not include a fault in the model). In practice, no continuum is perfectly homogeneous in its properties – all materials contain defects, imperfections or innate variability. However, characterizing and modelling heterogeneous materials is a non-trivial undertaking.

Assessment of the size or impact of structural error often amounts to assessing the consequences of model assumptions (see Design notes, Section 4.3). In some cases, this requires running the model with the assumption relaxed and satisfying oneself that the behaviour or predictions are not substantially altered.

The above assumptions – in fact any assumption made consciously, articulated or not – fall into a category of ignorance termed **known unknowns**. These are the things we know we don’t know or, as James Clerk Maxwell16 put it, "*thoroughly conscious ignorance*". Observation error, parameter and prediction uncertainty also fall under this heading.

More difficult to deal with are the **unknown unknowns**, i.e., the things we don’t know that we don’t know17. This includes the physical processes we’re not aware are operating or that we have interpreted incorrectly, important features or structures that cannot be directly observed (again, unknown faults and fractures are a big challenge in modelling reservoirs), or bad information we have been provided that we actually think is good (data, expert knowledge). There is little that can be done to guard against unknown unknowns beyond a rigorous, professional approach to problem solving, i.e., gathering data, considering different modelling options, providing impartial self-critique, and maintaining an open-minded, but sceptical, outlook.

16  19th century titan of physics, populariser of dimensional analysis, originator of the hipster beard.

17  When you are first born, everything is an unknown unknown, which perhaps explains why infants find mathematical modelling quite challenging, hyuk hyuk hyuk.

### 4.1 A Linear Example

***Recall the example from Section 3.1.***

Another group of scientists have developed a different model. They know without a doubt18 that the true process is logarithmic and is described by a relationship of the form: $y_i=a\,\ln(x_i-x_0)+b$. Having three parameters instead of two, their model is clearly more sophisticated than the linear one19. Undertaking a similar analysis of the data, $\tilde{y}_i$, they compute a best-fit model, compute the posterior parameter distribution and sample from it to obtain a forecast of the future (Sections 2-3). This is implemented in the cell below, alongside the forecast from the first group, who used a linear model. The 90% forecast interval for the logarithmic model excludes the true outcome by a large margin.

How did the second group get it so wrong? They followed the same procedure as the first group: fitting a model to the data, exploring parameter space to construct a posterior, and then sampling from it to construct a forecast. Indeed, with respect to the observations, $\tilde{y}_i$, the best-fit logarithmic model has a smaller objective function ($S(\hat{\boldsymbol{\theta}}_0 )=4.2$) than the best-fit linear model ($6.5$), and on that basis we might prefer the former.

The error they made occurred early on during model selection when the underlying physics were assumed to be logarithmic in nature. The particular choice may have been motivated by the data ("*the logarithmic model fits $\tilde{y}_i$ better*") or by an incorrect analysis of the physics. In either case, structural error was introduced during model development that, in the end, had a deleterious effect of future predictions.

18  "*It is very difficult to find a black cat in a dark room. Especially when there is no cat.*" - [Proverb](https://i.kym-cdn.com/photos/images/original/001/153/173/152.jpg).

19  Albeit, less correct.

***Execute the cell below and answer the questions (be patient, this one takes a few seconds to setup).***

```python
structural()
```

```python
# The cell above offers three alternative models (including the log model described above).
# - Switch between the models using the dropdown menu.
# - Zoom in to see how each fits the data.
# - Use the slider to change the prediction window.

# Rank the FOUR models (incl. linear) in terms of fit to the data.
# (the legend on the lefthand figures gives the obj. func. value S)

# If 'ability to fit the data' was the ONLY criteria to choose a model,
# which would you choose?

# Of the three competing models, which ones successfully 'predict' the future at xf = 3?

# Describe the magnitude of the prediction error for alternative models as xf increases.

# What (real-world) actions could you take to help choose between competing models?

```

## Further reading for the interested

Firestein, S (2012). *Ignorance: How it drives science*. Oxford University Press.

Marzocchi, W., and J. D. Zechar (2011). *Earthquake Forecasting and Earthquake Prediction: Different Approaches for Obtaining the Best Model*. Seismological Research Letters 82, 442-448.

Tarantola, A. (2006). *Popper, Bayes and the inverse problem*. Nature Physics 2, 492-494.

## Appendix A

Maximum Likelihood Estimation (MLE) is a method for estimating the parameters of a statistical model. Central to MLE is the concept of observations, $\tilde{y}_i$, which are drawn from a probability density function, $P(⋅;\boldsymbol{\theta})$, that has parameters, $\boldsymbol{\theta}$. The true function is denoted $P_0(⋅;\boldsymbol{\theta}_0)$ and the goal of MLE is to find an estimate of $\boldsymbol{\theta}_0$.
In our case, the act of making an observation is equivalent to sampling a normally distributed random error (variance, $\sigma_i^2$) and adding it to the true model value (mean, $f(x_i;\boldsymbol{\theta})$), therefore the density function for the observation $\tilde{y}_i$ is

\begin{equation}
	P(\tilde{y}_i|\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma_i^2}} \exp \left[ -\frac{\left(\tilde{y}_i-f(x_i;\boldsymbol{\theta})\right)^2}{2\sigma_i^2}\right],
\end{equation}

which clearly depends on $\boldsymbol{\theta}$.

For $n$ observations, one forms the joint density function by multiplying together the equation above for each observation, i.e.,

\begin{equation}
	P(\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n│\boldsymbol{\theta})=\prod\limits_{i=1}^n P(\tilde{y}_i |\boldsymbol{\theta}) .
\end{equation}

The goal is to find $\hat{\boldsymbol{\theta}}_0$ that maximises this quantity, or rather its logarithm, which we term the log-likelihood

\begin{equation}
	\ln L=\sum\limits_{i=1}^n -\frac{(\tilde{y}_i-f(x_i;\boldsymbol{\theta}))^2}{2\sigma_i^2}+\text{const}.
\end{equation}

We identify the equation above as the objective function defined in Section 2 (to within a multiplicative factor and a constant), and thus we can write $P(\tilde{y}_1,\tilde{y}_2,\dots \tilde{y}_n│\boldsymbol{\theta}) = A \exp(-S(\boldsymbol{\theta})/2)$.

Bayes theorem for multiple parameters and observations is written

\begin{equation}
	P(\boldsymbol{\theta}│\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n)=\frac{P(\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n│\boldsymbol{\theta})  P(\boldsymbol{\theta})}{P(\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n )},
\end{equation}

which can be read literally "*the probability of the parameters ($\boldsymbol{\theta}$) given the data $(\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n)$ is equal to the probability of the data given the parameters (the joint density) multiplied by the probability of the parameters (the prior) divided by the probability of the data averaged over all parameters.*" For a uniform prior, $P(\boldsymbol{\theta})$ and $P(\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n)$ are constant. Hence, the posterior $P(\boldsymbol{\theta}│\tilde{y}_1,\tilde{y}_2,\dots\tilde{y}_n)$ is the likelihood function computed above.


# Variational multi-scale finite element solution for compressible
# Navier-Stokes equations - Symbolic Differentiation Formulation

## Elisa Magliozzi

###  *November 2017*

The Variational Multiscale formulation for the compressible Navier-Stokes equations presented in this document was developed by the Dr.-Ing. Camilo A. Bayona Roa as part of his PhD thesis on $\textit{"Adaptive Mesh Simulations of Compressible Flows using Stabilized Formulations"}$, under the supervision of the Professor Ramon Codina and the Dr.-Ing. Joan Baiges.

## List of Symbols

```python
from KratosMultiphysics import *

from sympy import *
from sympy_fe_utilities import *
import pprint

params = {
            "dim": 3,			                                    # Dimension
            "mu": Symbol('mu', positive = True),			        # Dynamic viscosity
            "h" : Symbol('h', positive = True),	        	        # Element size
            "lambda" : Symbol('lambda', positive = True),           # Thermal Conductivity of the fluid
            "c_v" : Symbol('c_v', positive = True),			        # Specific Heat at Constant volume
            "gamma": Symbol('gamma',positive = True),			    # Gamma (Cp/Cv)
            "stab_c1" : Symbol('stab_c1', positive = True),			# Algorithm constant
            "stab_c2" : Symbol('stab_c2', positive = True),			# Algorithm constant

        }

dim = params["dim"]                         # Define Dimension in params.py

BlockSize = dim+2 					        # Dimension of the vector of Unknowns
do_simplifications = False
mode = "c"                                  # Output mode to a c++ file

if(dim == 2):
    nnodes = 3
elif(dim == 3):
    nnodes = 4

impose_partion_of_unity = False
N,DN = DefineShapeFunctions(nnodes, dim, impose_partion_of_unity)

# Unknown fields definition (Used later for the gauss point interpolation)
U = DefineMatrix('U',nnodes,BlockSize)	     # Vector of Unknowns ( Density,Velocity[dim],Total Energy )
Un = DefineMatrix('Un',nnodes,BlockSize)     # Vector of Unknowns one step back
Unn = DefineMatrix('Unn',nnodes,BlockSize)   # Vector of Unknowns two steps back
r = DefineVector('r',nnodes)                 # Sink term    #COMMENT for manufactured solution

# Test functions defintiion
w = DefineMatrix('w',nnodes,BlockSize)	    # Variables field test

# External terms definition
f_ext = DefineMatrix('f_ext',nnodes,dim)    # Forcing term #COMMENT for manufactured solution

# Definition of other symbols
bdf0 = Symbol('bdf0')                       # Backward differantiation coefficients
bdf1 = Symbol('bdf1')
bdf2 = Symbol('bdf2')
v_sc = Symbol('v_sc')                       # Shock capturing Viscosity
k_sc = Symbol('k_sc')                       # Shock capturing Conductivity

```

## 1. Compressible Navier-Stokes formulation

The physics related to a compressible fluid flow can fully described by the compressible Navier-Stokes.
It is necessary to consider the comprimibility effects of a fluid whenever the Mach number (ratio between the velocity and the local speed of sound) is greater than 0.3.
The solution implemented is based on the finite element method for fluids, using a variational multiscale framework to approximate the unresolved subscales.
A newtonian viscous fluid is considered.

1. Conservation of mass equation

    \begin{equation}
    \label{MassConserv}
    \frac{\partial \rho}{\partial t} + \nabla \cdot (\rho \textbf{u})  = 0.
    \end{equation}

2. Conservation of momentum equation

    \begin{equation}
    \label{MomentumCons}
    \frac{\partial (\rho \mathbf{u})}{\partial t} + \nabla \big(\rho \mathbf{u}\big) \mathbf{u} +\nabla \big(p\mathbf{I}-\mathbf{\tau}\big) = \rho \mathbf{f}
    \end{equation}

3. Conservation of energy equation

    \begin{equation}
    \label{EnergyCons}
    \frac{\partial}{\partial t} \bigg(\rho \big( e + \frac{1}{2}  \mathbf{u}\cdot\mathbf{u}\big)\bigg) + \nabla \bigg(\rho \mathbf{u} \big(\mathbf{h}+\frac{1}{2}\mathbf{u}\cdot\mathbf{u}\big) - \mathbf{u}\cdot \mathbf{\tau} + \mathbf{q}\bigg)= \rho \mathbf{f}\cdot\mathbf{u}+\rho r
    \end{equation}

Where *$\rho$* is the density, *p* is the pressure, * **u** * is the velocity, *$\tau$* is the viscous stress tensor, * **f** * is the body force vector, *e* is the internal energy, *h* is the enthalpy, * **q** * is the heat flux vector and *r* is a heat soure/sink term.

Now the equation is written in terms of the conservative variable *$\rho$*, *$\mathbf{m} = \rho \mathbf{u}$*  , and *$e_{tot}$*   *$ = \rho \big( e + \frac{1}{2}  \mathbf{u}\cdot\mathbf{u}\big)$*

1. Conservation of mass equation

    \begin{equation}
    \label{MassConserv}
    \frac{\partial \rho}{\partial t} + \nabla \cdot \textbf{m}  = 0.
    \end{equation}

2. Conservation of momentum equation

    \begin{equation}
    \label{MomentumCons}
    \frac{\partial \mathbf{m}}{\partial t} + \nabla \cdot \mathbf{m} \frac{\mathbf{m}}{\rho} +\nabla (p\mathbf{I}-\mathbf{\tau}) = \rho \mathbf{f}
    \end{equation}

3. Conservation of energy equation

    \begin{equation}
    \label{EnergyCons}
    \frac{\partial e_{tot}}{\partial t} + \nabla \bigg((\rho+e_{tot})\frac{\mathbf{m}}{\rho} -\frac{\mathbf{m}}{\rho}\cdot\mathbf{\tau}+ \mathbf{q}\bigg)= \mathbf{f}\mathbf{m}+\rho r
    \end{equation}

It is possible to group the Navier-Stokes equations in a system with the help of the Einstein summation, considering $\mathbf{U} = (\rho, \mathbf{m}, e_{tot})^T$ as the vector of the conservative variables.

\begin{equation}
    \label{System}
    \frac{\partial (\mathbf{U})}{\partial t} + \frac{\partial \mathbf{F}_j (\mathbf{U})}{\partial x_j}+\frac{\partial \mathbf{G}_j (\mathbf{U})}{\partial x_j}-\mathbf{S}(\mathbf{U}) = \mathbf{0}, \quad in \quad \Omega \subset \mathbb{R}^d, t>0,
\end{equation}

\begin{equation}
\label{Dirichlet}
 \textit{U}(\mathbf{U}_g)=\mathbf{U}_g,\quad \quad \quad \quad \quad \quad \quad \quad on\quad \Gamma_g,t>0,
 \end{equation}

 \begin{equation}
     \label{Neumann}
     \mathbf{F}_jn_j=\mathbf{h}, \quad \quad \quad \quad \quad \quad \quad \quad\quad \quad on\quad \Gamma_n, t>0,
 \end{equation}

 \begin{equation}
     \label{InitialCond}
     \mathbf{U}=\mathbf{U}_0(\mathbf{x}), \quad \quad \quad \quad \quad \quad \quad \quad\quad \quad in\quad \Omega, t=0
 \end{equation}

 The boundary conditions for the Dirichlet and Neumann boundaries are here introduced, together with the initial condition.

```python
 ### Construction of the variational equation
Ug = DefineVector('Ug',BlockSize)			# Dofs vector
H = DefineMatrix('H',BlockSize,dim)			# Gradient of U
f = DefineVector('f',dim)			        # Body force vector
rg = Symbol('rg', positive = True)		    # Source/Sink term
V = DefineVector('V',BlockSize)			    # Test function
Q = DefineMatrix('Q',BlockSize,dim)			# Gradient of V
acc = DefineVector('acc',BlockSize)         # Derivative of Dofs/Time
#G = DefineMatrix('G',BlockSize,dim)		# Diffusive Flux matrix
Gsc = DefineMatrix('G',BlockSize,dim)       # Diffusive Flux matrix with Shock Capturing
```

### 1.1 Convective Flux Matrix Implementation

The $\textit{(d+2)*d}$ $\textit{convective}$ flux matrix $\mathbf{F}$ is:

\begin{equation}
    \label{Fmat}
    \mathbf{F}_j(\mathbf{U}) = \bigg(m_j,\frac{m_j}{\rho}m_i+p\delta_{ij},(e_{tot}+p)\frac{m_j}{\rho}\bigg)^T \quad \quad 1\leq{i},j\leq{d}
\end{equation}

Expanded:

\begin{equation}
    \mathbf{F} =
  \begin{bmatrix}
    m_1 & m_2 & m_3 \\
    \frac{m_1}{\rho}m_1+p & \frac{m_1}{\rho}m_2 & \frac{m_1}{\rho}m_3\\
    \frac{m_2}{\rho}m_1+p & \frac{m_2}{\rho}m_2+p & \frac{m_2}{\rho}m_3\\
    \frac{m_2}{\rho}m_1+p & \frac{m_2}{\rho}m_2 & \frac{m_3}{\rho}m_3+p\\
    (e_{tot}+p)\frac{m_1}{\rho} & (e_{tot}+p)\frac{m_2}{\rho} & (e_{tot}+p)\frac{m_3}{\rho}
  \end{bmatrix}
\end{equation}

Where the pressure $\textit{p}$ is:

\begin{equation}
    \label{pressure}
    \textit{p} (\mathbf{U}) = (\gamma-1) \bigg(\textit{e}_{tot}-\frac{|\mathbf{m}|^2}{2 \rho} \bigg)
\end{equation}

From this

\begin{equation}
    \label{DiffAmat}
    \frac{\partial \mathbf{F}_j(\mathbf{U})}{\partial x_j}=\mathbf{A}_j(\mathbf{U})\frac{\partial \mathbf{U}}{\partial x_j}
\end{equation}

 it is possible to derive the $\textit{(d+2)*(d+2)*d}$ Euler Jacobian matrix $\mathbf{A}_j$ as:

\begin{equation}
    \label{Amatr}
    \mathbf{A}_j(\mathbf{U}) = \frac{\partial\mathbf{F}_j(\mathbf{U})}{\partial \mathbf{U}}
\end{equation}

\begin{equation}
    \mathbf{A} =
  \begin{bmatrix}
    \frac{\partial m_j}{\partial \rho} & \frac{\partial m_j}{\partial \mathbf{m}} & \frac{\partial m_j}{\partial e_{tot}} \\
    \frac{\partial \big(\frac{m_j}{\rho}m_i+p\delta_{ij}\big)}{\partial\rho} &
    \frac{\partial \big(\frac{m_j}{\rho}m_i+p\delta_{ij}\big)}{\partial\mathbf{m}}&
    \frac{\partial \big(\frac{m_j}{\rho}m_i+p\delta_{ij}\big)}{\partial e_{tot}}\\
    \frac{\partial \big((e_{tot}+p)\frac{m_j}{\rho}\big)}{\partial\rho} &
      \frac{\partial \big((e_{tot}+p)\frac{m_j}{\rho}\big)}{\partial\mathbf{m}} &
      \frac{\partial \big((e_{tot}+p)\frac{m_j}{\rho}\big)}{\partial e_{tot}}\\
  \end{bmatrix} \quad\quad\quad\quad\quad\quad 1\leq i, j\leq d
\end{equation}

For each direction d the matrix $\mathbf{a_d}$ is defined as below.

\begin{equation}
    \mathbf{a_1} =
  \begin{bmatrix}
   0 & 1 & 0 & 0 & 0 \\
    \frac{(\gamma -3)m_1^2}{2\rho^2}+\frac{(\gamma -1) }{2 \rho^2} (m_2^2+m_3^2)&
    \frac{(3-\gamma)m_1}{\rho}& \frac{(1-\gamma)m_2}{\rho} & \frac{(1-\gamma)m_3}{\rho} & \gamma-1 \\
    \frac{-m_1 m_2}{\rho^2} & \frac{m_2}{\rho} & \frac{m_1}{\rho} & 0 & 0 \\
    \frac{-m_1 m_3}{\rho^2} & \frac{m_3}{\rho} & 0 & \frac{m_1}{\rho} & 0 \\
    -\gamma\frac{e_{tot}}{\rho^2}m_1+ (\gamma-1)\frac{m_1}{\rho}\big( \frac{m_1^2}{\rho^2}+ \frac{m_2^2}{\rho^2}+
    \frac{m_3^2}{\rho^2}\big) &
    \gamma \frac{e_{tot}}{\rho}+\frac{1-\gamma}{2\rho^2}\big(3m_1^2+m_2^2+m_3^2\big) & \frac{(1-\gamma)}{\rho^2}m_1m_2 &\frac{(1-\gamma)}{\rho^2}m_1m_3 & \frac{\gamma m_1}{\rho} \\
  \end{bmatrix}
\end{equation}

\begin{equation}
    \mathbf{a_2} =
  \begin{bmatrix}
   0 & 0 & 1 & 0 & 0 \\
    -\frac{m_1m_2}{\rho^2}& \frac{m_2}{\rho}& \frac{m_1}{\rho} & 0 & 0 \\
    -\frac{m_2m_3}{\rho^2}& 0 & \frac{m_3}{\rho}& \frac{m_2}{\rho} & 0\\
    \frac{(\gamma -3)m_3^2}{2\rho^2}+\frac{(\gamma -1) }{2 \rho^2}(m_1^2+m_2^2) & \frac{1-\gamma}{\rho}m_1 & \frac{(1-\gamma)}{\rho}m_2 & \frac{(3-\gamma)}{\rho}m_3 & \gamma-1 \\
    -\gamma\frac{e_{tot}}{\rho^2}m_2+ (\gamma-1)\frac{m_2}{\rho}\big( \frac{m_1^2}{\rho^2}+ \frac{m_2^2}{\rho^2}+
    \frac{m_3^2}{\rho^2}\big) &
    \frac{(1-\gamma)}{\rho^2}m_1m_2 &  \gamma \frac{e_{tot}}{\rho}+\frac{1-\gamma}{2\rho^2}\big(m_1^2+3m_2^2+m_3^2\big)  &\frac{(1-\gamma)}{\rho^2}m_2m_3 & \frac{\gamma m_2}{\rho}\\
  \end{bmatrix}
\end{equation}

\begin{equation}
    \mathbf{a_3} =
  \begin{bmatrix}
   0 & 0 & 0 & 1 & 0 \\
    -\frac{m_1m_3}{\rho^2}& \frac{m_3}{\rho}& 0 & \frac{m_1}{\rho} & 0 \\
    \frac{-m_2 m_3}{\rho^2} & 0 & \frac{m_3}{\rho} & \frac{m_2}{\rho} & 0 \\
    \frac{(\gamma -3)m_2^2}{2\rho^2}+\frac{(\gamma -1) }{2 \rho^2}(m_1^2+m_3^2) & \frac{1-\gamma}{\rho}m_1 & \frac{(3-\gamma)}{\rho}m_2 & \frac{(1-\gamma)}{\rho}m_3 & \gamma-1 \\
       -\gamma\frac{e_{tot}}{\rho^2}m_3+ (\gamma-1)\frac{m_3}{\rho}\big( \frac{m_1^2}{\rho^2}+ \frac{m_2^2}{\rho^2}+
    \frac{m_3^2}{\rho^2}\big) &
    \frac{(1-\gamma)}{\rho^2}m_1m_3 & \frac{(1-\gamma)}{\rho^2}m_2m_3 & \gamma \frac{e_{tot}}{\rho}+\frac{1-\gamma}{2\rho^2}\big(m_1^2+m_2^2+3m_3^2\big)  & \frac{\gamma m_3}{\rho}\\
  \end{bmatrix}
\end{equation}

```python
## Computation of the Convective Matrix
def computeA(dofs, params):
    print("\nCompute Convective Matrix \n")
    dim = params["dim"]				        # Spatial dimensions

    ## Unknown field definition
    F = DefineMatrix('F',dim+2,dim)		    # Convective Flux matrix
    Ug = dofs                               # Data interpolation to the Gauss points

    ## Other symbols definitions
    y = params["gamma"]				        # Gamma (Cp/Cv)

    ## Pgauss - Pressure definition
    pg = (y-1)*Ug[dim+1]
    for i in range(0,dim):
        pg += (y-1)*(-Ug[i+1]*Ug[i+1]/(2*Ug[0]))

    ## F - Convective Flux Matrix definition
    for j in range(0,dim):
        F[0,j] = Ug[j+1]

    for i in range (1,dim+1):
        for j in range(0,dim):
            F[i,j] = Ug[i]*Ug[j+1]/Ug[0]
            if i==j+1:
               F[i,j]+=pg

    for j in range(0,dim):
        F[dim+1,j] = (Ug[dim+1]+pg)*Ug[j+1]/Ug[0]

    ## A - Jacobian Convective Matrix definition
    A = []

    for j in range(0,dim):
        tmp = DefineMatrix('tmp',dim+2,dim+2)
        for i in range(0,dim+2):
            for k in range(0,dim+2):
      	        tmp[i,k] = diff(F[i,j], Ug[k])
      	        #print(j,'	',i,k,'=',tmp[i,k])
        A.append(tmp)
    return A

## Printing the Convective Matrix
def printA(A,params):
    dim = params["dim"]
    tmp = []
    print("The convective matrix is:\n")
    for j in range(0,dim):
    	tmp = A[j]
    	for i in range(0,dim+2):
            for k in range(0,dim+2):
                print("A[",j,",",i,",",k,"]=",tmp[i,k],"\n")

    return 0
```

```python
## Matrix Computation

A = computeA(Ug,params)
#printA(A,params)
```

    Compute Convective Matrix

### 1.3 Diffusive Flux Matrix Implementation

The diffusive matrix is related to the viscous and thermal properties of the fluid. The former are presented both in the conservation of momentum and energy, while the latter are necessary to define the conservation of energy only.

The shear stress tensor is defined as follow:

\begin{equation}
    \tau_{ij} (\mathbf{u}) = \mu \bigg(\frac{\partial u_i}{\partial x_j} + \frac{\partial u_j}{\partial x_i} \bigg) -\frac{2 \mu}{3} \bigg(\frac{\partial u_l}{\partial x_l} \bigg) \delta_{ij}
\end{equation}

with the dynamic viscosity $\mu$. The relation can be rewritten in terms of conservative variables as:

\begin{equation}
    \tau_{ij}(\mathbf{U}) = \frac{\mu}{\rho}\bigg(\frac{\partial m_i}{\partial x_j}+\frac{\partial m_j}{\partial x_i}\bigg)-\frac{2\mu}{3\rho}\bigg(\frac{\partial m_k}{\partial x_k}\bigg)\delta_{ij}-\frac{\mu}{\rho^2}\bigg(m_i \frac{\partial\rho}{\partial x_j}+m_j \frac{\partial \rho}{\partial x_i}\bigg)+\frac{2\mu}{3\rho^2}\bigg(m_k \frac{\partial \rho}{\partial x_k}\bigg)\delta_{ij} \quad i,j,k<{d}
\end{equation}

Substituting $\mathbf{H}(\mathbf{U})= \frac{\partial \mathbf{U}}{\partial x_j}$, which is a $\textit{(dim+2)*(dim)}$ matrix, and $\mathbf{U}$ in the equation:

\begin{equation}
    \tau_{ij}(\mathbf{U}) = \frac{\mu}{U_0}\bigg(H_{i+1,j}+H_{j+1,i}\bigg)-\frac{2\mu}{3U_0}\bigg(H_{k+1,k}\bigg)\delta_{ij}-\frac{\mu}{U_0^2}\bigg(U_{i+1} H_{0,j}+U_{j+1}H_{0,i}\bigg)+\frac{2\mu}{3U_0^2}\bigg(U_{k+1} H_{0,k}\bigg)\delta_{ij} \quad i,j,k<{d}
\end{equation}

The heat flux vector $q_i(\mathbf{U})$ is defined by the Fourier's law as:

\begin{equation}
    q_i(\theta) = -\lambda \frac{\partial \theta}{\partial x_i}
\end{equation}

where $\lambda$ is the thermal conductivity and $\theta$ is the temperature of the fluid.

Two additional equations are necessary in order to close the problem. Here the caloric equation and the perfect gas state equation are used in order to compute the pressure $\textit{p}$ and the speed of sound $\textit{c}$.

\begin{equation}
    e = c_v(\theta)\theta \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad p = \rho R \theta
\end{equation}

Where $\textit{R} = c_p - c_v$ and $\gamma = \frac{c_p}{c_v}$ is the ratio between the specific heat at constant pressure $c_p$ and the specific heat at constant volume $c_v$. The internal energy can be expressed in terms of the conservative variables leading to the following equation for the heat flux vector.

\begin{equation}
    e = \frac{e_{tot}}{\rho}- \frac{|\textbf{m}|^2}{2}
\end{equation}

\begin{equation}
    q_i(\mathbf{U}) = \frac{\lambda \textit{e}_{tot}}{\rho^2 \textit{c}_v}\frac{\partial \rho}{\partial x_i}-\frac{\lambda m_jm_j}{\rho^3 \textit{c}_v}\frac{\partial \rho}{\partial x_i}+\frac{\lambda m_j}{\rho^2 \textit{c}_v}\frac{\partial m_j}{\partial x_i}-\frac{\lambda}{\rho \textit{c}_v}\frac{\partial \textit{e}_{tot}}{\partial x_i}
\end{equation}

Which substituting $\mathbf{H}$ and $\mathbf{U}$ becomes:
\begin{equation}
    q_i(\mathbf{U}) = \frac{\lambda U_{d+1}}{U_0^2 \textit{c}_v}H_{0,i}-\frac{\lambda U_{j+1}U_{j+1}}{U_0^3 \textit{c}_v}H_{0,i}+\frac{\lambda U_{j+1}}{U_0^2 \textit{c}_v}H_{j+1,i}-\frac{\lambda}{U_0 \textit{c}_v}H_{d+1}\quad i,j<d
\end{equation}

NB: $\rho$ is the first term of the vector of unknwons $\mathbf{U}$ and $\textit{e}_{tot}$ is the last one (Ug\[dim+1\]).
Remember to consider them as Unknowns and not fluid parameters.

The $\textit{(d+2)*d diffusive}$ flux matrix $\mathbf{G}$ is defined as follow.

\begin{equation}
    \label{Gmat}
    \mathbf{G}_j(\mathbf{U}) = \big(0,-\tau_{ji}, -\frac{m_i}{\rho}\tau_{ij}+q_j\big)^T \quad \quad i,j<{d}
\end{equation}

\begin{equation}
    \mathbf{G} =
  \begin{bmatrix}
    0 & 0 & 0 \\
    -\tau_{11} & -\tau_{21} & -\tau_{31}\\
    -\tau_{12} & -\tau_{22} & -\tau_{32}\\
    -\tau_{13} & -\tau_{23} & -\tau_{33}\\
    -\frac{m_1}{\rho}\tau_{11}-\frac{m_2}{\rho}\tau_{21}-\frac{m_3}{\rho}\tau_{31}+q_1 &
   -\frac{m_1}{\rho}\tau_{12}-\frac{m_2}{\rho}\tau_{22}-\frac{m_3}{\rho}\tau_{32}+q_2 &
   -\frac{m_1}{\rho}\tau_{13}-\frac{m_2}{\rho}\tau_{23}-\frac{m_3}{\rho}\tau_{33}+q_3
  \end{bmatrix}
\end{equation}

The fourth order tensor $\mathbf{K}_{kj}(\mathbf{U})$ $\textit{(d+2)*(d+2)*d*d}$ can be manually derived in order to satisfy this expression:

\begin{equation}
    \label{K}
    \frac{\partial\mathbf{G}_j(\mathbf{U})}{\partial x_j} =- \frac{\partial}{\partial x_k}\bigg(\mathbf{K}_{kj}(\mathbf{U})\frac{\partial \mathbf{U}}{\partial x_j}\bigg)
\end{equation}

In this project the $\mathbf{K}$ tensor was not computed in order to simplify the implementation.
This does not affect the solution as long as using linear shape of functions, due to the fact that the tensor is a second order term.

```python
## Computation of the Diffusive Matrix with Shock Capturing
def computeGsc(dofs,params,Hg,Gg,v_sc,k_sc):
    print("\nCompute Diffusive Matrix with Shock Capturing\n")
    dim = params["dim"]				                    # spatial dimensions

    ## Unknown fields definition
    H = Hg.copy()                               		# Gradient of U
    Gsc = Gg.copy()                               		# Diffusive Flux matrix
    tau_stress = DefineMatrix('tau_stress',dim,dim)		# Shear stress tensor for Newtonian fluid
    q = DefineVector('q',dim)			                # Heat flux vector

    ## Other simbols definition
    c_v = params["c_v"]				                    # Specific Heat at Constant volume
    gamma = params["gamma"]				                # Gamma (Cp/Cv)
    mu  = params["mu"]         			                # Dynamic viscosity
    l = params["lambda"]			                    # Thermal Conductivity

    ## Data interpolation to the Gauss points
    Ug = dofs

    ## Pgauss - Pressure definition
    pg = Ug[dim+1]
    for i in range(0,dim):
        pg += (-Ug[i+1]*Ug[i+1]/(2*Ug[0]))
    pg *= (gamma-1)

    ## tau - Shear stress tensor definition
    for i in range(0,dim):
        for j in range(i,dim):
            if i!=j:
               tau_stress[i,j] = (mu/Ug[0])*(H[i+1,j]+H[j+1,i])-(mu/Ug[0]**2)*(Ug[i+1]*H[0,j]+Ug[j+1]*H[0,i])
            if i==j:
               tau_stress[i,j]= (2*mu/Ug[0])*H[i+1,i]-(2*mu/Ug[0]**2)*Ug[i+1]*H[0,i]
               for k in range(0,dim):
                   tau_stress[i,j]+= -(2*mu/(3*Ug[0]))*H[k+1,k]+(2*mu/(3*Ug[0]**2))*Ug[k+1]*H[0,k]

    for i in range(1,dim):
        for j in range(0,dim-1):
            if j!=i:
               tau_stress[i,j] = tau_stress[j,i]

    ## q - Heat flux vector definition
    for i in range(0,dim):
        q[i] = l*Ug[dim+1]/(Ug[0]**2*c_v)*H[0,i]-(l*H[dim+1,i])/(Ug[0]*c_v)
        for j in range(0,dim):
            q[i] += -l*Ug[j+1]**2/(c_v*Ug[0]**3)*H[0,i]+l/(Ug[0]**2*c_v)*Ug[j+1]*H[j+1,i]
    #NB!!!There is an error in the definition of q[i] in the research proposal.
    #The second term of the equation has an opposite sign!!!NB#
    '''
    G [(dim+2)*(dim)]

    0                                   0
    -tau00                              -tau01
    -tau01                              -tau11
    -mu/rho*tau00-mv/rho*tau01+q0       -mu/rho*tau01-mv/rho*tau11+q1
    '''
    tau_sc = (1+(Ug[0]*v_sc)/mu)*tau_stress     # Stress tensor with shock capturing viscosity
    q_sc = (1+(Ug[0]*c_v*k_sc)/l)*q             # Heat flux with shock capturing conductivity

    ## Gsc - Diffusive Matrix definition
    for j in range(0,dim):
        Gsc[0,j]= 0 			                #Mass equation related

    for i in range(1,dim+1):
        for j in range(0,dim):
            Gsc[i,j]=-tau_sc[i-1,j]		        #Moment equation related

    for j in range(0,dim):                      #Energy equation related
        Gsc[dim+1,j] = q_sc[j]
        for k in range(0,dim):
            Gsc[dim+1,j] += -Ug[k+1]*tau_sc[k,j]/Ug[0]

    return Gsc

## Printing the Diffusive Matrix G
def printG(G,params):
    dim = params["dim"]
    print("The diffusive matrix is:\n")
    for ll in range(dim+2):
        for mm in range(dim):
            print("G[",ll,",",mm,"]=",G[ll,mm],"\n")

    return 0
```

### 1.4 Shock Capturing technique

The diffusive term is modified in order to include additional artificial viscosity and conductivity. These terms are necessary to guarantee stability even when, due to shock waves, there are sharp gradients in the solution.
There are different methods to include diffusion to capture the nonlinearities generated by the physical phenomenum of shock.
Below the implementation of an isotropic residual based shock capturing is presented.

In the isotropic case the artificial diffusion is added in all components, adding an extra term in the stress tensor and one in the heat flux vector.

\begin{equation}
    \breve{\tau_{ij}} = \bigg( 1+ \frac{\rho \nu_{sc}}{\mu}\bigg) \tau_{ij}
\end{equation}

\begin{equation}
    \breve{q_i} = \bigg( 1+ \frac{\rho c_v k_{sc}}{\lambda}\bigg) q_i
\end{equation}

where $\nu_{sc}$ and $\textit{k}_{sc}$ are respectively the kinematic viscosity and the thermal diffusivity coefficient related to the shock capturing.

The coefficients are computed using a residual based the technique. Below the definition of the residual related to the momenum equation and the one related to the total energy are shown. The nonlinear operator $\mathbf{L} (\mathbf{U} ; \mathbf{U})$ will be defined in details in section 2.1.

\begin{equation}
    \mathbf{R} (\mathbf{m_h}) = -\frac{\mathbf{U_m}}{\partial t}-\mathbf{L_m} (\mathbf{U};\mathbf{U_h} )
\end{equation}

\begin{equation}
    R(e_{tot}) =  -\frac{Ue_{tot}}{\partial t}-\mathbf{L}e_{tot} (\mathbf{U};\mathbf{U_h} )
\end{equation}

The coefficients are defined as follow:

\begin{equation}
    \nu_{sc} = \frac{1}{2} h \alpha \frac{\big| \mathbf{R}(\mathbf{m_h})\big|}{\big| \nabla \mathbf{m_h} \big |} \quad if \quad \big|\nabla \mathbf{m_h} \big|\neq 0,\quad \quad\quad \quad \quad \quad\quad\quad\quad\nu_{sc} = 0 \quad otherwise
\end{equation}

The Froebenius norm of the momentum gradient is computed as:

\begin{equation}
    \big| \nabla \mathbf{m_h} \big| = \Bigg(\sum_{i=1}^{d}\sum_{j=1}^{d} \bigg| \frac{\partial m_{i,h}}{\partial x_j}\bigg |^2 \Bigg) ^{\frac{1}{2}}
\end{equation}

\begin{equation}
    k_{sc} = \frac{1}{2} h \alpha \frac{\big| R(e_{tot},h) \big|}{\big | \nabla e_{tot},h \big|} \quad if \quad \big| \nabla e_{tot},h \big| \neq 0, \quad \quad\quad \quad \quad \quad\quad\quad k_{sc} = 0 \quad otherwise
\end{equation}

Nota bene: the shock capturing diffusion is activated in case the norm of the gradients is greater than a certain tolerance. The implementationi of the coefficients and the computation of the gradient norms is computed directly at the element level (in the .cpp template).

```python
Gsc = computeGsc(Ug,params,H,Gsc,v_sc,k_sc)
#printG(Gsc,params)
```

    Compute Diffusive Matrix with Shock Capturing

### 1.3 Source Term Matrix Implementation

The $\textit{source}$ term vector $\mathbf{S(\mathbf{U})}$ is written as a product of a $\textit{(d+2)*(d+2)}$ reactive matrix $\mathbf{S}$ and the vector of unknowns.

\begin{equation}
    \label{S(U)}
    \mathbf{S}(\mathbf{U}) = (0,\rho\mathbf{f},\mathbf{f}\cdot\mathbf{m}+\rho r)^T
\end{equation}

\begin{equation}
    \label{S}
    \mathbf{S}=
    \begin{pmatrix} 0 & 0 & 0\\ \mathbf{f} & 0 & 0 \\ r &\mathbf{f}^T &0 \end{pmatrix}
\end{equation}

```python
## Computation of the Source Matrix
def computeS(force,source,params):
    print("\nCompute Source Matrix \n")
    dim = params["dim"]				        # Spatial dimensions

    ## Unknown field definition
    f = force.copy()					    # Body force vector
    r = source			 		            # Heat Source/Sink Term  #Symbol('r', positive = True)
    '''
    S - Reactive Matrix definition
     0  0  0   0
     fx 0  0   0
     fy 0  0   0
     r  fx fy  0
    '''
    S = zeros(dim+2,dim+2)                  # Reactive matrix (Source terms)

    for i in range(1,dim+1):
        S[i,0]   = f[i-1]
        S[dim+1,i] = f[i-1]
    S[dim+1,0] = r

    return S

## Printing of the Source Matrix
def printS(S,params):
    dim = params["dim"]				        # Spatial dimensions
    print("The source term matrix is:\n")
    for i in range (0,dim+2):
        for j in range (0,dim+2):
            print("S[",i,",",j,"]=",S[i,j],"\n")

    return 0
```

```python
S = computeS(f,rg,params)
#printS(S,params)
```

    Compute Source Matrix

## 2. Approximation of the subscales

Nota bene: the diffusive term is neglected in the approximation of the subscales (in the definition of the non-linear operator and its adjoint) to simplify the problem.
As explained previously this will not affect the solution as the second order terms related with the diffusion cannot nevertheless be detected by the linear elements used here.

The finite element space is subdivided into a coarse and a fine subgrid space.

 $\mathbf{V} = \mathbf{V}_h+\widetilde{\mathbf{V}}, \quad \mathbf{V}_h \subset \textit{W}_h$
In the same way, the $\mathbf{U}$ vector is decomposed in:

$ \mathbf{U} = \mathbf{U}_h + \widetilde{\mathbf{U}}, \quad \mathbf{U} \in \textit{W}$

In order to understand the final formulation of the variational problem we need to define the nonlinear operator $\mathbf{\textit{L}}(\mathbf{U};\mathbf{U})$ , its adjoint $\mathbf{\textit{L}^{*}}(\mathbf{U};\mathbf{V}_h)$ and the finite element residual $\mathbf{R}(\mathbf{U};\mathbf{U}_h)$, together with the stabilization matrix $\mathbf{\tau}$.

### 2.1 Nonlinear Operator Definition

The problem can be rewritten using  the nonlinear operator $\mathbf{\textit{L}}(\mathbf{U};\mathbf{U})$ as below.

\begin{equation}
    \frac{\partial \mathbf{U}}{\partial t} + \mathbf{\textit{L}}(\mathbf{U} ; \mathbf{U}) = \mathbf{0}, \quad in \quad \Omega \subset \mathbb{R}^d, t>0,
\end{equation}

\begin{equation}
 \textit{U}(\mathbf{U}_g)=\mathbf{U}_g,\quad \quad \quad \quad \quad \quad \quad \quad on\quad \Gamma_g,t>0,
 \end{equation}

 \begin{equation}
     \mathbf{F}_jn_j=\mathbf{h}, \quad \quad \quad \quad \quad \quad \quad \quad\quad \quad on\quad \Gamma_n, t>0,
 \end{equation}

 \begin{equation}
     \mathbf{U}=\mathbf{U}_0(\mathbf{x}), \quad \quad \quad \quad \quad \quad \quad \quad\quad \quad in\quad \Omega, t=0
 \end{equation}
Where $\mathbf{\textit{L}}$ is define as here.

\begin{equation}
    \mathbf{\textit{L}}(\mathbf{U};\mathbf{U}) = \mathbf{A}_j\frac{\partial \mathbf{U}}{\partial x_j}-\mathbf{S} \mathbf{U}
\end{equation}

```python
## Nonlinear operator definition
l1 = Matrix(zeros(dim+2,1))		            # Convective Matrix*Gradient of U
A_small = []
for j in range(0,dim):
    A_small = A[j]
    for ll in range(BlockSize):
        for mm in range(BlockSize):
            l1[ll] += A_small[ll,mm]*H[mm,j]

l3 = S*Ug				                    # Source term
print("\nCompute Non-linear operator\n")
L = l1-l3                                   # Nonlinear operator

```

    Compute Non-linear operator

### 2.1 Nonlinear Adjoint Operator Definition
The nonlinear adjoint operator $\mathbf{\textit{L}^{*}}$ is here applied to the test functions vector $\mathbf{V}_h$.

\begin{equation}
    \label{Nonlinoper}
    \mathbf{\textit{L}^{*}}(\mathbf{U};\mathbf{V}_h) =-\frac{\partial}{\partial x_j}\bigg(\mathbf{A}_j^{T}(\mathbf{U})\mathbf{V}_h\bigg) - \mathbf{S}^T \mathbf{V}_h
\end{equation}

```python
## Nonlinear adjoint operator definition
m1 = Matrix(zeros(dim+2,1))		            # Convective term
psi = Matrix(zeros(dim+2,dim))

for j in range(0,dim):
    A_T = A[j].transpose()
    for l in range(0,dim+2):
        for m in range(0,dim+2):
            psi[l,j] += A_T[l,m]*Q[m,j]
            for n in range(0,dim+2):
                psi[l,j] +=diff(A_T[l,m],Ug[n])*H[n,j]*V[m]

for s in range(0,dim+2):
    for j in range(0,dim):
        m1[s] += psi[s,j]

m3 = S.transpose()*V			            # Source term

print("\nCompute Non-linear adjoint operator\n")
L_adj = -m1-m3                              # Nonlinear adjoint operator
```

    Compute Non-linear adjoint operator

### 2.3 Residual of the Finite Element Scale

The finite element residual is:
\begin{equation}
    \label{R}
    \mathbf{R}(\mathbf{U};\mathbf{U_h}) = -\frac{\partial \mathbf{U}_h}{\partial t}-\mathbf{\textit{L}}(\mathbf{U};\mathbf{U}_h)
\end{equation}

```python
## Residual definition
res = -acc - L
```

Below the definition of the momentum and energy residual, as presented in paragraph 1.4.

```python
## Istotropic Residual Based Shock Capturing
res_m = Matrix(zeros(dim,1))                # Momentum residual
for i in range(0,dim):
    res_m[i,0] = res[i+1,0]

res_e = Matrix(zeros(1,1))                  # Total Energy residual
res_e[0,0] = res[dim+1]
```

### 2.4 The Stabilization matrix

The stabilization matrix $\mathbf{\tau}$ is a $\textit{(d+2)*(d+2)}$ diagonal matrix defined like this:

\begin{equation}
    \label{tau}
    \boldsymbol{\tau}^{-1} = diag \big(\tau^{-1}_1,\tau^{-1}_2\mathbf{I}_d,\tau^{-1}_3 \big)  \\
    \tau^{-1}_1 = \textit{c}_2 \frac{\lvert {\frac{\mathbf{m}} {\rho}}\rvert +c}{\textit{h}} \\
    \tau^{-1}_2 = \textit{c}_1 \frac{\nu}{\textit{h}^2} +\textit{c}_2 \frac{\lvert {\frac{\mathbf{m}} {\rho}}\rvert +c}{\textit{h}} \\
    \tau^{-1}_3 = \textit{c}_1 \frac{\lambda}{\rho \textit{c}_p \textit{h}^2} +\textit{c}_2 \frac{\lvert {\frac{\mathbf{m}} {\rho}}\rvert +c}{\textit{h}} \\
\end{equation}

Where $\textit{c}$ is the speed of sound, while $\textit{c}_1$ and $\textit{c}_2$ are algorithm constants (here $\textit{c}_1$ = 4 and $\textit{c}_2$ = 2)

Nota bene: the definition of the tau terms is done directly at the element level (in the *.cpp file)

```python
def computeTau(params):
    print("\nCompute Stabilization Matrix\n")
    dim = params["dim"]				# Spatial dimensions
    Tau = zeros(dim+2,dim+2)        # Stabilization Matrix

    tau1 = Symbol('tau1')
    tau2 = Symbol('tau2')
    tau3 = Symbol('tau3')

    Tau[0,0] = tau1
    for i in range (0,dim):
        Tau[i+1,i+1] = tau2
    Tau[dim+1,dim+1] = tau3
    return(Tau)

def printTau(Tau, params):
    dim = params["dim"]				# Spatial dimensions
    print("The Stabilization term matrix is:\n")
    for i in range (0,dim+2):
        for j in range (0,dim+2):
            print("Tau[",i,",",j,"]=",Tau[i,j],"\n")

    return 0
```

```python
Tau = computeTau(params)
printTau(Tau,params)
```

    Compute Stabilization Matrix

    The Stabilization term matrix is:

    Tau[ 0 , 0 ]= tau1

    Tau[ 0 , 1 ]= 0

    Tau[ 0 , 2 ]= 0

    Tau[ 0 , 3 ]= 0

    Tau[ 0 , 4 ]= 0

    Tau[ 1 , 0 ]= 0

    Tau[ 1 , 1 ]= tau2

    Tau[ 1 , 2 ]= 0

    Tau[ 1 , 3 ]= 0

    Tau[ 1 , 4 ]= 0

    Tau[ 2 , 0 ]= 0

    Tau[ 2 , 1 ]= 0

    Tau[ 2 , 2 ]= tau2

    Tau[ 2 , 3 ]= 0

    Tau[ 2 , 4 ]= 0

    Tau[ 3 , 0 ]= 0

    Tau[ 3 , 1 ]= 0

    Tau[ 3 , 2 ]= 0

    Tau[ 3 , 3 ]= tau2

    Tau[ 3 , 4 ]= 0

    Tau[ 4 , 0 ]= 0

    Tau[ 4 , 1 ]= 0

    Tau[ 4 , 2 ]= 0

    Tau[ 4 , 3 ]= 0

    Tau[ 4 , 4 ]= tau3

    0

## 3. Variational Formulation

Finally the problem can be described using the equation below.

\begin{equation}
    \label{final}
    \bigg(\mathbf{V}_h,\frac{\partial \mathbf{U}_h}{\partial t}\bigg) + \bigg(\mathbf{V}_h,\mathbf{A}_j(\mathbf{U}_h)\frac{\partial \mathbf{U}_h}{\partial x_j}\bigg) - \bigg(\frac{\partial \mathbf{V}_h}{\partial x_j},\mathbf{G}_{j}(\mathbf{U}_h)\bigg) - \bigg(\mathbf{V}_h,\mathbf{S} \mathbf{U}_h\bigg) + \sum_{K} \bigg\langle \mathbf{\textit{L}^{*}}(\mathbf{U}_h;\mathbf{V}_h),
    \boldsymbol{\tau} (\mathbf{U}_h) \mathbf{R}(\mathbf{U}_h) \bigg\rangle_K = 0 \quad \forall \mathbf{V}_h \in  \textit{W}
\end{equation}

```python
## Variational Formulation - Final equation
n1 = V.transpose()*acc		                # Mass term - FE scale

temp = zeros(dim+2,1)
A_smalll = []
for i in range(0,dim):
    A_smalll = A[i]
    for ll in range(BlockSize):
        for mm in range(BlockSize):
            temp[ll] += A_smalll[ll,mm]*H[mm,i]

n2 = V.transpose()*temp			            # Convective term - FE scale

n3 = Matrix(zeros(1,1))                     # Diffusive term - FE scale

for j in range(0,dim):
    for k in range(BlockSize):
        n3[0,0] += Q[k,j]*(-Gsc[k,j])       # G with shock capturing - FE scale

n4 = -V.transpose()*(S*Ug)		            # Source term - FE scale

n5 = L_adj.transpose()*(Tau*res)	        # VMS_adjoint - Subscales

print("\nCompute Variational Formulation\n")
rv = n1+n2+n3+n4+n5 			            # VARIATIONAL FORMULATION - FINAL EQUATION

```

    Compute Variational Formulation

### 3.1 Numerical discretization of the Variational Formulation

```python
### Substitution of the discretized values at the gauss points
print("\nSubstitution of the discretized values at the gauss points\n")

## Data interpolation at the gauss points
U_gauss = U.transpose()*N
w_gauss = w.transpose()*N
f_gauss = f_ext.transpose()*N                     #COMMENT for manufactured solution
acc_gauss = (bdf0*U+bdf1*Un+bdf2*Unn).transpose()*N
r_gauss = (r.transpose()*N)[0]                    #COMMENT for manufactured solution
#r_gauss = Symbol('r_gauss', positive = True)     #USED for manufactured solution

## Gradients computation
grad_U = DfjDxi(DN,U).transpose()
grad_w = DfjDxi(DN,w).transpose()

print("\nSubstitution in the variational formulation\n")
SubstituteMatrixValue(rv, Ug, U_gauss)
SubstituteMatrixValue(rv, acc, acc_gauss)
SubstituteMatrixValue(rv, H, grad_U)
SubstituteMatrixValue(rv, V, w_gauss)
SubstituteMatrixValue(rv, Q, grad_w)
SubstituteMatrixValue(rv, f, f_gauss)           #COMMENT for manufactured solution
SubstituteScalarValue(rv, rg, r_gauss)          #COMMENT for manufactured solution

print("\nSubstitution in the residual of momentum\n")
SubstituteMatrixValue(res_m, Ug, U_gauss)
SubstituteMatrixValue(res_m, acc, acc_gauss)
SubstituteMatrixValue(res_m, H, grad_U)
SubstituteMatrixValue(res_m, f, f_gauss)       #COMMENT for manufactured solution
SubstituteScalarValue(res_m, rg, r_gauss)      #COMMENT for manufactured solution

print("\nSubstitution in the residual of total energy\n")
SubstituteMatrixValue(res_e, Ug, U_gauss)
SubstituteMatrixValue(res_e, acc, acc_gauss)
SubstituteMatrixValue(res_e, H, grad_U)
SubstituteMatrixValue(res_e, f, f_gauss)       #COMMENT for manufactured solution
SubstituteScalarValue(res_e, rg, r_gauss)      #COMMENT for manufactured solution

```

    Substitution of the discretized values at the gauss points

    Substitution in the variational formulation

    Substitution in the residual of momentum

    Substitution in the residual of total energy

    Matrix([[-N_0*(U_0_4*bdf0 + Un_0_4*bdf1 + Unn_0_4*bdf2) - N_1*(U_1_4*bdf0 + Un_1_4*bdf1 + Unn_1_4*bdf2) - N_2*(U_2_4*bdf0 + Un_2_4*bdf1 + Unn_2_4*bdf2) - N_3*(U_3_4*bdf0 + Un_3_4*bdf1 + Unn_3_4*bdf2) - gamma*(DN_0_0*U_0_4 + DN_1_0*U_1_4 + DN_2_0*U_2_4 + DN_3_0*U_3_4)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0) - gamma*(DN_0_1*U_0_4 + DN_1_1*U_1_4 + DN_2_1*U_2_4 + DN_3_1*U_3_4)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0) - gamma*(DN_0_2*U_0_4 + DN_1_2*U_1_4 + DN_2_2*U_2_4 + DN_3_2*U_3_4)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0) + (gamma - 1)*(DN_0_0*U_0_2 + DN_1_0*U_1_2 + DN_2_0*U_2_2 + DN_3_0*U_3_2)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (gamma - 1)*(DN_0_0*U_0_3 + DN_1_0*U_1_3 + DN_2_0*U_2_3 + DN_3_0*U_3_3)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (gamma - 1)*(DN_0_1*U_0_1 + DN_1_1*U_1_1 + DN_2_1*U_2_1 + DN_3_1*U_3_1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (gamma - 1)*(DN_0_1*U_0_3 + DN_1_1*U_1_3 + DN_2_1*U_2_3 + DN_3_1*U_3_3)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (gamma - 1)*(DN_0_2*U_0_1 + DN_1_2*U_1_1 + DN_2_2*U_2_1 + DN_3_2*U_3_1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (gamma - 1)*(DN_0_2*U_0_2 + DN_1_2*U_1_2 + DN_2_2*U_2_2 + DN_3_2*U_3_2)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 - (-(gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4 + (gamma - 1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4) - (gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)))/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0))*(DN_0_0*U_0_1 + DN_1_0*U_1_1 + DN_2_0*U_2_1 + DN_3_0*U_3_1) - (-(gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4 + (gamma - 1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4) - (gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)))/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0))*(DN_0_1*U_0_2 + DN_1_1*U_1_2 + DN_2_1*U_2_2 + DN_3_1*U_3_2) - (-(gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2 + (N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4 + (gamma - 1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4) - (gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)))/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0))*(DN_0_2*U_0_3 + DN_1_2*U_1_3 + DN_2_2*U_2_3 + DN_3_2*U_3_3) - (((gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2) + (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2) + (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2))*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0) - (N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4 + (gamma - 1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4) - (gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)))/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2)*(DN_0_0*U_0_0 + DN_1_0*U_1_0 + DN_2_0*U_2_0 + DN_3_0*U_3_0) - (((gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2) + (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2) + (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2))*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0) - (N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4 + (gamma - 1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4) - (gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)))/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2)*(DN_0_1*U_0_0 + DN_1_1*U_1_0 + DN_2_1*U_2_0 + DN_3_1*U_3_0) - (((gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2) + (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2) + (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2))*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0) - (N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4 + (gamma - 1)*(N_0*U_0_4 + N_1*U_1_4 + N_2*U_2_4 + N_3*U_3_4) - (gamma - 1)*(N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)) - (gamma - 1)*(N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)**2/(2*(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)))/(N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)**2)*(DN_0_2*U_0_0 + DN_1_2*U_1_0 + DN_2_2*U_2_0 + DN_3_2*U_3_0) + (N_0*U_0_0 + N_1*U_1_0 + N_2*U_2_0 + N_3*U_3_0)*(N_0*r_0 + N_1*r_1 + N_2*r_2 + N_3*r_3) + (N_0*U_0_1 + N_1*U_1_1 + N_2*U_2_1 + N_3*U_3_1)*(N_0*f_ext_0_0 + N_1*f_ext_1_0 + N_2*f_ext_2_0 + N_3*f_ext_3_0) + (N_0*U_0_2 + N_1*U_1_2 + N_2*U_2_2 + N_3*U_3_2)*(N_0*f_ext_0_1 + N_1*f_ext_1_1 + N_2*f_ext_2_1 + N_3*f_ext_3_1) + (N_0*U_0_3 + N_1*U_1_3 + N_2*U_2_3 + N_3*U_3_3)*(N_0*f_ext_0_2 + N_1*f_ext_1_2 + N_2*f_ext_2_2 + N_3*f_ext_3_2)]])

### 3.2 Computation of the RHS vector and LHS matrix

```python
dofs = Matrix(zeros(nnodes*(dim+2),1))
testfunc = Matrix(zeros(nnodes*(dim+2),1))
for i in range(0,nnodes):
     for j in range(0,dim+2):
        dofs[i*(dim+2)+j] = U[i,j]
        testfunc[i*(dim+2)+j] = w[i,j]

## Compute LHS and RHS
print("\nCompute RHS\n")
rhs = Compute_RHS(rv.copy(), testfunc, do_simplifications)
rhs_out = OutputVector_CollectingFactors(rhs, "rhs", mode)

print("\nCompute LHS\n")
lhs = Compute_LHS(rhs, testfunc, dofs, do_simplifications) # Compute the LHS
lhs_out = OutputMatrix_CollectingFactors(lhs, "lhs", mode)
```

    Compute RHS

    Compute LHS

```python
## Residual for shock capturing
res_m_out = OutputMatrix_CollectingFactors(res_m, "res_m", mode)
res_e_out = OutputMatrix_CollectingFactors(res_e, "res_e", mode)
```

### 3.3 Writing output to *.cpp file

```python
## Reading Template File
print("\nReading compressible_navier_stokes_cpp_template.cpp\n")
if(dim==2):
        templatefile = open("compressible_navier_stokes_cpp_template2D.cpp")
        outstring=templatefile.read()
        outstring = outstring.replace("//substitute_lhs_2D", lhs_out)
        outstring = outstring.replace("//substitute_rhs_2D", rhs_out)
        outstring = outstring.replace("//substitute_res_m_2D", res_m_out)
        outstring = outstring.replace("//substitute_res_e_2D", res_e_out)
        ## Write the modified template
        print("\nWriting compressible_navier_stokes2D.cpp\n")
        out = open("../../custom_elements/compressible_navier_stokes2D.cpp",'w')
        out.write(outstring)
        out.close()
elif(dim == 3):
        templatefile = open("compressible_navier_stokes_cpp_template3D.cpp")
        outstring=templatefile.read()
        outstring = outstring.replace("//substitute_lhs_3D", lhs_out)
        outstring = outstring.replace("//substitute_rhs_3D", rhs_out)
        outstring = outstring.replace("//substitute_res_m_3D", res_m_out)
        outstring = outstring.replace("//substitute_res_e_3D", res_e_out)

        ## Write the modified template
        print("\nWriting compressible_navier_stokes3D.cpp\n")
        out = open("../../custom_elements/compressible_navier_stokes3D.cpp",'w')
        out.write(outstring)
        out.close()
print("\nCompressible Navier Stokes Element Generated\n")
```

    Reading compressible_navier_stokes_cpp_template.cpp

    Writing compressible_navier_stokes3D.cpp

    Compressible Navier Stokes Element Generated

```python

```

```python

```