# Tutorial 00: P1 Finite Elements for the Poisson Equation

## Problem Formulation

In this exercise, we will solve our first PDE, the *Poisson equation*:

\begin{align}\label{eq:1a}
\begin{array}{rcll}
-\Delta u & = f && \text{in}\ \Omega, \\
u & = g && \text{on}\ \partial\Omega,
\end{array}
\end{align}

where $\Omega \in \mathbb{R}^d$ is a given, polyhedral domain
(elements with curved boundaries are possible in DUNE but will not be considered
here). This problem is one of the basic equations of mathematical physics
which describes gravitational and electric potential as well as stationary heat or
groundwater flow. Poisson's equation is an instance of an *elliptic partial differential equation*.

A function $u \in C^2(\Omega) \cap C^0(\bar{\Omega})$ (the space of twice continuously differentiable functions in $\Omega$ which are continuous up to the boundary) is called a *strong solution* if it satisfies the above equations pointwise.

Formally, one often reduces the equations to a problem with homogeneous Dirichlet boundary conditions g ≡ 0 as the starting point for theoretical considerations. We will deliberately not do this here, as it is also not done in the computer implementation of the method.

### Weak Formulation

Existence, uniqueness and stability of solutions, i.e. well-posedness in the sense of
Hadamard, is easier to prove for so-called weak solutions. As the weak formulation
is also the basis of the finite element method we explain it here.

As a start, suppose $u$ is a strong solution of the equations above and take
*any* function
$v\in C^1(\Omega)\cap C^0(\bar\Omega)$ with $v=0$ on $\partial\Omega$, then we
have by integration by parts:
\begin{equation}
\int_\Omega (-\Delta u) v \,dx =
\int_\Omega \nabla u \cdot \nabla v \,dx= \int_\Omega fv \,dx.
\end{equation}
Observe, that the boundary integral $\int_{\partial\Omega} \nabla u\cdot n v\,dx$ vanishes
due to the fact that $v=0$ on $\partial\Omega$. Loosely speaking $v=0$ is a consequence
of the Dirichlet boundary condition $u=g$ on $\partial\Omega$.

Introducing the abbreviations
\begin{equation}
a(u,v) = \int_\Omega \nabla u \cdot \nabla v \,dx, \qquad l(v) = \int_\Omega fv \,dx
\end{equation}
one can on the other hand ask the question: Is there a class,
more specific a vector space, of functions $V$ with $V_g=\{v\in V :
\text{$v=g$ on $\partial\Omega$}\}$ and $V_0=\{v\in V :
\text{$v=0$ on $\partial\Omega$}\}$ such that
the problem
\begin{equation}
u \in V_g :\qquad a(u,v) = l(v) \qquad \forall v\in V_0 \label{eq:weakform}
\end{equation}
has a unique solution.

The answer is yes and in particular one can prove the following:

1. With $V=H^1(\Omega)$, the Sobolev space of functions with square integrable
weak derivatives, the problem \eqref{eq:weakform} has a unique solution provided
the bilinear form $a : V\times V \to \mathbb{R}$ is continuous and coercive on the
subspace $V_0\subset V$ and the linear form $l: V \to \mathbb{R}$ is also continuous.
Coercivity on $V_0$ follows from Friedrich's inequality and for the continuity of the right hand
side functional $f\in L^2(\Omega)$ is a sufficient condition.
2. If in addition $u\in C^2(\Omega)\cap C^0(\bar\Omega)$, then the solutions
of \eqref{eq:weakform} and \eqref{eq:1a} coincide.

We call \eqref{eq:weakform} the *weak formulation* of \eqref{eq:1a}.
As it has a unique solution under more general conditions than \eqref{eq:1a},
e.g. for discontinuous right hand side functions $f$, it can be considered a generalization
of the problem.

## The Finite Element Method

The (conforming) finite element method, in a nutshell, is based on the weak formulation where
the function space $V$ is replaced by a subspace $V_h\subset V$
which is *finite dimensional*. Here, the subscript $h$ relates to the dimension of the
function space. One major part of the finite element method is to construct such so-called *finite element spaces*.
Typically, finite element spaces consist of piecewise polynomial functions.
We consider one particular choice, the space of linear functions on simplicial elements.
For a more detailed description of the underlying finite element method, please refer to `tutorial00.pdf`. Additionally, there are many text books about the finite element method, which are listed in the [further reading section](#FurtherReading) at the end of this tutorial.

## Implementation

The code presented in this tutorial solves the elliptic equation \eqref{eq:1a} with Dirichlet boundary conditions, where
\begin{align}
    f(x) = -2d \quad\text{and}\quad  g(x) = \sum_{i=1}^d (x)_i^2.
\end{align}

*Note that throughout this section there are 'show solution buttons' included.
They only become relevant for the exercises in [section 2](#Exercises), as these exercises require some modification within the code.*

First, we include all the Dune sources through one convenience header:

```c++
#include<dune/jupyter.hh>
```

The simulation code written below will depend on a number of runtime parameters that are provided in a configuration file `tutorial00.ini`. You find that file in the `notebooks` directory. Dune uses a nested data structured called `ParameterTree` for these configurations. In the following we parse the configuration, output it to the notebook and alter it from within the notebook:

```c++
Dune::ParameterTree ptree;
Dune::ParameterTreeParser::readINITree("tutorial00.ini", ptree);
```

```c++
ptree
```

```c++
ptree["grid.refinement"] = "0";
```

```c++
ptree
```

We are using a two-dimensional, unstructured mesh for this simulation. The mesh resolves the unit square $\Omega = [0,1]^2$. The file is specified in the `unitsquare.msh` file that was generated using Gmsh.

```c++
const int dim = 2;
using Grid = Dune::UGGrid<dim>;
std::string filename = ptree.get("grid.twod.filename", "unitsquare.msh");
Dune::GridFactory<Grid> factory;
Dune::GmshReader<Grid>::read(factory,filename,true,true);
std::unique_ptr<Grid> grid(factory.createGrid());
grid->globalRefine(ptree.get<int>("grid.refinement"));
auto gv = grid->leafGridView();
using GV = decltype(gv);
```

```c++
grid
```

With Dune making heavy use of C++ templates, the underlying floating point type of the simulation can be chosen quite freely. Looking for a solution $u: \Omega\rightarrow\mathbb{R}$, the *domain field* type is the floating point type used to realize $\Omega$ where as the *range field* type is used to realize $\mathbb{R}$. `DF` is defined by the grid implementation (usually to `double`), but we can freely choose `RF` - typically also to `double`:

```c++
using DF = Grid::ctype;
using RF = double;
```

In the following, first a *local finite element map* of type `PkLocalFiniteElementMap` is set up. The finite element map associates finite element basis functions, defined on the corresponding reference element, with each element of the mesh. Additionally, information is provided on how the global finite element space $V_h$ is constructed from the local, element-wise pieces.

The type `CON` provides a so-called constraints class, which provides a way to assemble constraints on a function space.

The type `VBE` provides a vector backend, this allows for the direct filling of different types of data into these vectors, without a copy step. This is usefull, as PDELab can use different iterative solver libraries, which provide their own data types for vectors and (sparse) matrices. In this code, DUNE's own iterative solver library ISTL provides the vector backend.

From the above described parameters, the type `GFS` from the class template `GridFunctionSpace`is defined. This combines all the given information to construct the global finite element space $V_h$ on a given grid view. In the last two lines an object of this class is instantiated and the space is given a name.

```c++
using FEM = Dune::PDELab::PkLocalFiniteElementMap<GV,DF,RF,1>;
FEM fem(gv);
using CON = Dune::PDELab::ConformingDirichletConstraints;
using VBE = Dune::PDELab::ISTL::VectorBackend<>;
using GFS = Dune::PDELab::GridFunctionSpace<GV,FEM,CON,VBE>;
GFS gfs(gv,fem);
gfs.name("P1");
```

In the next step, a variable that will later on contain the solution vector *z* is declared:

```c++
using Z = Dune::PDELab::Backend::Vector<GFS,RF>;
Z z(gfs);
```

The function g extends the Dirichlet boundary values into the inerior. This function can be used to provide e.g. the exact solution of the problem for testing purposes or an initial guess for the iterative solvers. *Note that as stated above, one exercise at the end of this notebook requires a change of this function. The solution for this part can be accessed through the "show solution" button below.*

```c++
auto g = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [](const auto& x){
        RF s=0.0;
        for (std::size_t i=0; i<x.size(); i++)
            s+=x[i]*x[i];
        return s;
    }
);
Dune::PDELab::interpolate(g,gfs,z);
```

*This is the solution for exercise 1.2. Note that you need to comment out the above lines before executing the notebook with the new function g.*

```c++
//solution to exercise 1.2
/*
auto g = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [](const auto& x){
        RF s=0.0;
        for (std::size_t i=0; i<x.size(); i++)
            s+=x[i]*x[i]*x[i];
        return s;
    }
);
Dune::PDELab::interpolate(g,gfs,z);
*/
```

In the following, the constraints for a specific grid function space are determined from the boundary condition type function `b`. They are saved into the constraints container `cc` of type `CC`. The separation of the function space $V_h$ and the constraints set $\mathcal{I}_h^{\partial \Omega}$ allows one to reuse the function space together with different constraints, e.g. for a system of partial differential equations.

```c++
using CC = typename GFS::template ConstraintsContainer<RF>::Type;
CC cc;
```

```c++
auto b = Dune::PDELab::makeBoundaryConditionFromCallable(
    gv,
    [](const auto& x){ return true; }
);
Dune::PDELab::constraints(b, gfs, cc);
std::cout << "constrained dofs=" << cc.size() << " of "
          << gfs.globalSize() << std::endl;
```

The right hand side $f$ of the partial differential equation is denoted by the function `f`. *Note that the exercise also requires a change of the function f, thus there is another solution.*

```c++
auto f = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [](const auto& x){
        return Dune::FieldVector<RF,1>(-2.0*x.size());
    }
);;
```

*This is the solution to exercise 1.2. Again make sure only one of the cells containg the function f is executed.*

```c++
//solution to exercise 1.2
/*
auto f = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [](const auto& x){
        RF s=0.0;
        RF t=0.0;
        for (std::size_t i=0; i<x.size(); i++){
          s+=x[i];
          t+=x[i]*x[i]*x[i];
        }
        return -6*s;
    }
);;
*/
```

*Exercise 2 requires additional changes, the solution can be found below.*

```c++
/*
auto k = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [](const auto& x){
      return x[0];
    }
);;
*/
```

```c++
/*
auto a = Dune::PDELab::makeGridFunctionFromCallable(
   gv,
   [](const auto& x) {
       RF s=0.0;
       for (std::size_t i=0; i<x.size(); i++)
         s+=x[i];
       return s;
   }
);;
*/
```

Next, we will construct the local operator. We include the local operator implementation for P1-conforming finite elements from the file `poissonp1.hh`. You find this file in the `notebooks` folder. After making changes to the file, you need to restart and rerun this notebook.

```c++
#include "poissonp1.hh"

using LOP = PoissonP1<decltype(f),FEM>;
LOP lop(f, fem.find(*gv.template begin<0>()));
```

*The following solutions are again for exercise 2*

*The cell beneath can be either executed with your own modification of the poisson class `mypoisson1.hh` or with the solution `solutionpoissonp1.hh`.*

```c++
/*#include "mypoissonp1.hh"
//#include "solutionpoissonp1.hh"

using LOP = PoissonP1<decltype(f),decltype(k),decltype(a),FEM>;
LOP lop(f,k,a,fem.find(*gv.template begin<0>()));*/
```

This local operator is used as one of the template arguments in the global assembler or grid operator, which additionally requires the types for ansatz and test space, a matrix backend, the types for the components of the coefficient vectors of ansatz and test space and the entries of the matrix *A* and finally the types of the constraints container for test and ansatz space.

```c++
using MBE = Dune::PDELab::ISTL::BCRSMatrixBackend<>;
MBE mbe(1<<(dim+1)); // guess nonzeros per row
using GO = Dune::PDELab::GridOperator<
  GFS,GFS,  /* ansatz and test space */
  LOP,      /* local operator */
  MBE,      /* matrix backend */
  RF,RF,RF, /* domain, range, jacobian field type*/
  CC,CC     /* constraints for ansatz and test space */
  >;
GO go(gfs,cc,gfs,cc,lop,mbe);
```

Next, a linear solver for the linear system $Az = b$ is selected. Since ISTL backends for vectors and matrices are used, also a linear solver from the ISTL library is chosen.
Complete solvers are packaged by PDELab for sequential and
parallel computations. Here we select the conjugate gradient method with algebraic
multigrid as preconditioner and symmetric successive overrelaxation as smoother in
multigrid in its sequential implementation. The linear solver object `ls` is initialized
with the maximum number of iterations and a verbose parameter:

```c++
using LS = Dune::PDELab::ISTLBackend_SEQ_CG_AMG_SSOR<GO>;
LS ls(100,2);
```

*The following cell contains a different linear iterative solver (see exercise 1.3). Again, only one of the cells should be executed.*

```c++
 //using LS = Dune::PDELab::ISTLBackend_SEQ_BCGS_SSOR;
 //LS ls(5000,true);
```

So far, no finite element computations have actually been performed, only the
necessary components have been configured to now do the real work together. We
can now set up the matrix *A* as well as the right hand side *b* and then solve the
system. Since this is required often, there is a class in PDELab for this:

```c++
using SLP = Dune::PDELab::StationaryLinearProblemSolver<GO,LS,Z>;
SLP slp(go,ls,z,1e-10);
slp.apply(); // here all the work is done!
```

In the given example a problem with a known exact solution, which is given by
the function *g* is solved. In order to compare the computed solution with the exact
solution we initialize another coeffcient vector with the Lagrange interpolant of the
exact solution and provide a grid function:

```c++
using ZDGF = Dune::PDELab::DiscreteGridFunction<GFS,Z>;
ZDGF zdgf(gfs,z);
Z w(gfs); // Lagrange interpolation of exact solution
Dune::PDELab::interpolate(g,gfs,w);
ZDGF wdgf(gfs,w);
Dune::VTKWriter<GV> vtkwriter(gv,Dune::VTK::conforming);
using VTKF = Dune::PDELab::VTKGridFunctionAdapter<ZDGF>;
vtkwriter.addVertexData(std::shared_ptr<VTKF>(new VTKF(zdgf,"fesol")));
vtkwriter.addVertexData(std::shared_ptr<VTKF>(new VTKF(wdgf,"exact")));
```

The visualization data can be generated in Jupyter by printing the VTKWriter instance:

```c++
vtkwriter
```

# Local Operator

The finite element method itself is implemented in the so-called *local operator* realized by the class template
`PoissonP1`. It provides all the necessary element-local computations as described **in Subsection 3.4** and is declared as follows:

```c++
template<typename F, typename FiniteElementMap>
class PoissonP1 :

  public Dune::PDELab::FullVolumePattern,
  public Dune::PDELab::LocalOperatorDefaultFlags
```
The first template parameter provides the right hand side function of the PDE and
the second parameter provides a finite element map giving access to finite element
basis functions on the reference element for all elements of the grid. The class derives
from the PDELab classes `FullVolumePattern` and `LocalOperatorDefaultFlags` which provide some default constants and methods.

The basic assumption of this implementation of the finite element method is that all elements of the mesh are simplices of dimension d which use the same polynomial degree 1. In order to make the code faster it is a good idea to do the evaluation of the basis functions and their gradients on the reference element once before the
computations start. This will be done in the constructor, but before we can do so we need to do some preparations.

## Type Definitions and Data Members

The class begins by extracting important types. The finite element map provides a finite element for each element of the map. Its type is
```c++
private:
  // define useful types
  typedef typename FiniteElementMap::Traits::FiniteElementType
     FiniteElementType;
```
Among other things the finite element contains the basis functions on the reference element which can be accessed via the following type:
```c++
  typedef typename FiniteElementType::Traits::LocalBasisType
     LocalBasisType;
```
DUNE thinks of basis functions on the reference element to be of
the most general form
$$\hat\phi : \mathbb{A}^d \to \mathbb{B}^k, \quad
\nabla\hat\phi : \mathbb{A}^d \to \mathbb{B}^{k\times d},$$
i.e. they may be vector-valued. The following type definitions
```c++
  typedef typename LocalBasisType::Traits::DomainType
     DomainType;
  typedef typename LocalBasisType::Traits::RangeFieldType
     RF;
  typedef typename LocalBasisType::Traits::RangeType
     RangeType;
  typedef typename LocalBasisType::Traits::JacobianType
     JacobianType;
```
provide types to represent arguments and results of basis function
evaluations. `DomainType` represents $\mathbb{A}^d$,
`RF` represents $\mathbb{B}$, `RangeType`
represents $\mathbb{B}^k$ and finally `JacobianType`
represents $\mathbb{B}^{k\times d}$.

Next, we extract some important constants, the dimension
of the grid and the number of basis functions per element:
```c++
  // data members
  enum {dim=LocalBasisType::Traits::dimDomain};
  enum {n=dim+1};
```
As private data members the class stores an instance of the right hand side function `f` provided by the `driver`:
```c++
  const F f;              // right hand side function
```
the midpoint quadrature rule
```c++
  DomainType qp;          // center of mass of refelem
  double weight;          // quadrature weight on refelem
```
where `qp` is $\hat S_d$ and `weight` is $w_d$, and the values of the basis functions at the quadrature
point and their gradients:
```c++
  double phihat[n];       // basis functions at qp
  double gradhat[dim][n]; // coordinate x #basisfct
```
Then, already in the public part, we need to define some constants that control the operation of the grid operator doing the global assembly:
```c++
public:
  // define flags controlling global assembler
  enum { doPatternVolume = true };
  enum { doAlphaVolume = true };
  enum { doLambdaVolume = true };
```
These constants are evaluated at *compile time* and tell he grid operator class which methods have been implemented in the local operator by the user. Actually, the base class `LocalOperatorDefaultFlags` provides all possible flags ith the value`false` and we just need to overwrite the nes that are needed.
The constant`doPatternVolume` ells the global assembler to determine the sparsity pattern of the
matrix $A$ from a method `pattern_volume` which i inherited from the base class `FullVolumePattern`.
This default implementation inserts nonzeros between all degrees f freedom of an element. The constants `doAlphaVolume` and `doLambdaVolume` determine that our finite element method contains a volume integral involving the finite element solution $u_h$ and a right hand side integral which does not involve the finite element solution.

Setting `doAlphaVolume` to true implies that the local operator class implements the methods `alpha_volume`,
`jacobian_apply_volume` and `jacobian_volume`.
Setting `doLambdaVolume` to true implies that the method `lambda_volume` must be implemented.

## Constructor

The constructor of the class has the following signature:
```c++
 // Constructor precomputes element independent data
  PoissonP1 (const F& f_, const FiniteElementType& fel)
    : f(f_)
```

It takes the right hand side function `f` and a finite element `fel` as argument. The finite element is obtained from the finite element map and the first element of the grid in the function `driver`.
First thing to do is to get the lowest order quadrature rule for simplices from DUNE and we check that this is actually the midpoint rule:
```c++
  {
    // select quadrature rule
    Dune::GeometryType gt = fel.type();
    const Dune::QuadratureRule<RF,dim>&
      rule = Dune::QuadratureRules<RF,dim>::rule(gt,1);
    if (rule.size()>1) {
      std::cout << "Wrong quadrature rule!" << std::endl;
      exit(1);
    }
```
Then we store the first quadrature point in the local data members:
```c++
    // position and weight of the quadrature point
    weight = rule[0].weight();
    qp = rule[0].position();
```
It is also a good idea to check that the basis given by the user has at least the correct size:
```c++
    // check size of the basis
    if (fel.localBasis().size()!=n) {
      std::cout << "Wrong basis!" << std::endl;
      exit(1);
    }
```
Now the basis functions can be evaluated at the quadrature point in the reference element and the results are stored in the data members of the class:
```c++
    // evaluate basis functions on refelem
    std::vector<RangeType> phi(n);
    fel.localBasis().evaluateFunction(qp,phi);
    for (int i=0; i<n; i++) phihat[i] = phi[i];
```
And the same now for the gradients:
```c++
    // evaluate gradients of basis functions on refelem
    std::vector<JacobianType> js(n);
    fel.localBasis().evaluateJacobian(qp,js);
    for (int i=0; i<n; i++)
      for (int j=0; j<dim; j++)
        gradhat[j][i] = js[i][0][j];
  }
```
Note that the last index loops over the number of basis functions.

## Method `lambda_volume`

This method computes the contributions $b_T$ to the right hand side vector for a given element. It has the following signature:
```c++
  // volume integral depending only on test functions
  template<typename EG, typename LFSV, typename R>
  void lambda_volume (const EG& eg, const LFSV& lfsv,
                      R& r) const
```
Argument`eg` provides the element $T$ in a wrapped form such that PDELab need not operate directly on a DUNE grid.
With `eg.geometry()` the geometry of the element can be accessed in the form of a `Dune::Geometry`. With `eg.entity()` one can access the underlying codim 0 entity of the DUNE grid.
The second argument `lfsv` provides the test functions on the reference element and `r` provides a container where the result should be stored.
First thing to do is to evaluate the right hand side function at the quadrature point:
```c++
  {
    typename F::Traits::RangeType fval;
    f.evaluate(eg.entity(),qp,fval);
```
Next, we compute the factor that is common to all entries of $b_T$:
```c++
    RF factor=fval*weight*eg.geometry().integrationElement(qp);
```

Note that the method `integrationElement` on the geometry provides the value of $|\det B_T|$.
Finally, we can compute the entries and store them in the results container:

```c++
    for (int i=0; i<n; i++)
      r.accumulate(lfsv,i,-factor*phihat[i]);
  }
```
Here it is important to note the minus sign because PDELab actually solves the weak formulation as
$$r(u_h,v) = a(u_h,v)-l(v) = 0 \quad \forall v\in V$$ since this is more appropriate in the case of nonlinear partial differential equations.

## Method `jacobian_volume`

Next we need to compute the element contributions to the stiffness matrix.
This is done by the method `jacobian_volume` with
```c++
  // jacobian of volume term
  template<typename EG, typename LFSU, typename X,
           typename LFSV, typename M>
  void jacobian_volume (const EG& eg, const LFSU& lfsu,
                        const X& x, const LFSV& lfsv,
                        M& mat) const
  {
```
Its arguments are: `eg` providing the wrapped codim 0 entity $T$,`lfsu` providing the basis functions of the ansatz space, `x` providing the coefficients of the current iterate of the finite element solution, `lfsv` providing the test functions and `mat` a container to store the result.

The `jacobian_volume` method works in the same way also for nonlinear problems. Nonlinear problems are solved iteratively, e.g. using Newton's method or a fixed-point iteration, where the method should provide a linearization at the current iterate given by the combination of `lfsu` and `x`.
In our case of a linear problem the result *does not depend* on the current iterate.
Moreover, the basis functions for the test space are precomputed so we need
not access them via `lfsv`. Note also that in general the ansatz and test space might be different.

First thing we need is to get $B_T^{-T}$ and store it into `S`:
```c++
    // get Jacobian and determinant
    // assume the transformation is linear
    const auto geo = eg.geometry();
    const auto S = geo.jacobianInverseTransposed(qp);
```
Next, $|\det B_T|$ is retrieved from the geometry and the factor that is common to all entries of the local stiffness matrix is computed:
```c++
RF factor = weight*geo.integrationElement(qp);
```
Now form the matrix of transformed gradients $G=B_T^{-T} \hat G$ and store it in `grad`:
```c++
  // compute gradients of basis functions in transformed element
    double grad[dim][n] = {{0.0}}; // coordinate x #basisfct
    for (int i=0; i<dim; i++) // rows of S
      for (int k=0; k<dim; k++) // columns of S
        for (int j=0; j<n; j++) // columns of gradhat
          grad[i][j] += S[i][k] * gradhat[k][j];
```
The computations are arranged in such a way that the innermost loop has the dimension number of basis functions. In 3d there are four basis functions and the loop has a chance to get vectorized.

Now the local stiffness matrix $A_T = G^T G$ (up to the factor $|\det B_T| w_d$) is formed
```c++
    // compute grad^T * grad
    double A[n][n] = {{0.0}};
    for (int i=0; i<n; i++)
      for (int k=0; k<dim; k++)
        for (int j=0; j<n; j++)
          A[i][j] += grad[k][i]*grad[k][j];
```
and stored in the results container (now multiplying with the common factor):
```c++
    // store in result
    for (int i=0; i<n; i++)
      for (int j=0; j<n; j++)
        mat.accumulate(lfsu,i,lfsu,j,A[i][j]*factor);
  }
```

## Method `alpha_volume`

The method `alpha_volume` provides the element-local computations for the matrix-free evaluation of $a(u_h,\phi_i)$ for all test functions $\phi_i$. It has the interface:
```c++
 // volume integral depending on test and ansatz functions
  template<typename EG, typename LFSU, typename X,
           typename LFSV, typename R>
  void alpha_volume (const EG& eg, const LFSU& lfsu,
                     const X& x, const LFSV& lfsv,
                     R& r) const
```
Its arguments are: `eg` providing the wrapped codim 0 entity $T$,`lfsu` providing the basis functions of the ansatz space,`x` providing the coefficients of the current iterate of the finite element solution, `lfsv` providing the test functions and `r` a container to store the result.

The computations are actually quite similar to those in `jacobian_volume`. In particular, the computation of $B_T^{-T}$, $|\det B_T|$ and $G = B_T^{-T} \hat G$ are the same.
```c++
  {
    // get Jacobian and determinant
    // assume the transformation is linear
    const auto geo = eg.geometry();
    const auto S = geo.jacobianInverseTransposed(qp);
    RF factor = weight*geo.integrationElement(qp);

    // compute gradients of basis functions in transformed element
    double grad[dim][n] = {{0.0}};  // coordinate x #basisfct
    for (int i=0; i<dim; i++) // rows of S
      for (int k=0; k<dim; k++) // columns of S
        for (int j=0; j<n; j++) // columns of gradhat
          grad[i][j] += S[i][k] * gradhat[k][j];
```
Extracting the element local coefficients $z_T = R_T z$ is done by:
```c++
    double z_T[n];
    for (int j=0; j<n; j++) z_T[j] = x(lfsu,j); // read coeffs
```
Now we may compute $\nabla u_h$ via $G z_T$:
```c++
    double graduh[dim] = {0.0};
    for (int k=0; k<dim; k++) // rows of grad
      for (int j=0; j<n; j++) // columns of grad
        graduh[k] += grad[k][j]*z_T[j];
```
Finally, the result $a_T = G^T \nabla u_h$ is formed:
```c++
    // scalar products
    double a_T[n] = {0.0};
    for (int k=0; k<dim; k++) // rows of grad
      for (int j=0; j<n; j++)
        a_T[j] += grad[k][j]*graduh[k];
```
and stored in the results container (while being multiplied with the common factor):
```c++
    // store in result
    for (int i=0; i<n; i++)
      r.accumulate(lfsv,i,a_T[i]*factor);
  }
```

## Method `jacobian_apply_volume`

In the case of a *nonlinear* partial differential equation the finite element method results in a weak form
$$u_h \in V_h \ : \quad r(u_h,v) = \alpha(u_h,v) - \lambda(v) = 0 \quad \forall v\in V_h$$
which is *nonlinear* in its *first* argument. Inserting the finite element basis results in a nonlinear algebraic problem
$$R(z) = 0$$
with $(R(z))_i = r(\text{FE}_h(z),\phi_i)$ which is typically solved by Newton's iteration
or some other iterative method. In case of Newton's method, each step
involves the solution of a {\em linear} system of the form
$$J(z)\, w = R(z)$$
where $(J(z))_{i,j} = \frac{(\partial R(z))_i}{\partial z_j} =
\frac{\partial \alpha(\text{FE}_h(z),\phi_i)}{\partial z_j}$ is the Jacobian of
the nonlinear map $R$.

Naturally, the nonlinear case also includes the linear case described in this tutorial
by setting $r(u,v) = a(u,v)-l(v)$. Then, due to the linearity of $a$ in its first
argument, one can show that $J(z) = A$ and
$$(J(z)\, w)_i = (Aw)_i = a(\text{FE}_h(w),\phi_i).$$
This is *not* true in the nonlinear case. There, the evaluation of the form $\alpha(\text{FE}_h(w),\phi_i)$ and the application of the Jacobian $J w$ are different operations. Therefore, PDELab provides two functions with
the application of the Jacobian implemented in `jacobian_apply_volume`
with the interface:
```c++
  //! apply local jacobian of the volume term
  template<typename EG, typename LFSU, typename X,
           typename LFSV, typename R>
  void jacobian_apply_volume (const EG& eg, const LFSU& lfsu,
                              const X& z, const LFSV& lfsv,
                              R& r) const
```
Note, this is the same interface as for `alpha_volume`. Since our problem is linear, Jacobian application is identical to bilinear form evaluation and therefore we may just forward the call to the
function `alpha_volume`:

```c++
  {
    alpha_volume(eg,lfsu,z,lfsv,r);
  }
```

# Exercises

## Exercise 1

1. **Warming up**

   Run the program with different refinement levels. Visualize the solution in ParaView, use the Calculator
   filter to visualize the difference $|u-u_h|$ and determine the
   maximum error. Note that $u$ and $u_h$ are called `exact` and `fesol` in the paraview output.

- *In the following $|u-u_h|$ for different refinement levels is depicted.*

2. **Solving a new problem**

   Now consider
   \begin{equation}
   f(x) = -\sum_{i = 1}^{d} 6(x)_i \quad \text{and} \quad g(x) = \sum_{i = 1}^d(x)_i^3
   \end{equation}
   and check that $u(x)=\sum_{i=1}^d (x)_i^3$ solves the PDE. Implement the new $f$ and $g$ and rerun your program.

- *The solution for the new implementation of f and g is embedded in the notebook, directly at the place of the defintions ([go to solution](#newfandg)). When running all cells, ensure that only one of the definitions is uncommented (a cell that is hidden through the 'Show/Hide Solution' button is still executed).*

3. **Analysis of finite element error**:

  Produce a sequence of output files for different levels of mesh
  refinements $0, 1, 2, \ldots$ with suitable output
  filenames. Visualize the error $|u-u_h|$ in Para\-View and determine
  the maximum error on each level.  If you want you can also try to
  calculate the L2-norm of the error in paraview and determine the
  convergence rate by hand. In a later exercise you will see how to
  calculate the L2-norm directly in the c++ source.

 - *The following table shows the maximum norm,which is given as $||f||_{\max} := \max|f(x)|$ and the L2-norm as $||f||_{0,\Omega} = \left( \int_{\Omega} f(x)^2 dx \right)^{1/2} $ for different mesh refinements. The L2 error is depicted in the graph on the right side.*

    refinement
    ${||u-u_h||}_\max$
    ${||u-u_h||}_{0,\Omega}$

    0
    1.18978e-05
    1.161176e-03

    1
    1.76823e-06
    2.59592e-04

    2
    2.02272e-07
    6.0531e-05

    3
    1.9762e-08
    1.4689e-05

    4
    1.7708e-09
    3.635e-06

    5
    1.50032e-10
    9.06e-07

 - *In general the convergence behaviour of the finite element method is
$$ {||u-u_h||}_{j,\Omega} \rightarrow 0 \quad \text{for } h \rightarrow 0,$$
with $j = 0$ for the L2-norm. It can be profen that
$${||u-u_h||}_{j,\Omega} \leq Ch^{\beta},$$
where the exponent $\beta$ depends on the polynomial degree, the norm in which the error is measured and the regularity of the solution (more details can be found [here](#FurtherReading)). The exponent $\beta$ is called the* convergence rate *of the method (w.r.t a given norm). With $\beta = k + 1 $, where k denotes the polynomial degree, the finite element method with polynomials of degree 1 has a convergence rate of $\beta = 2$ w.r.t th L2-norm. The plot on the right depicts the error in the L2-norm with respect to the mesh size, clearly indicating a quadratic convergence rate.*

 - *These are the basic steps in order to determine the L2-norm of the error in paraview/by hand:*
     1. *use the calculator for $(u(x)-u_h(x))^2$*
     2. *use 'integrate variables' filter for $\int_{\Omega}(u(x)-u_h(x))^2 \,dx$*
     3. *determine $\left( \int_{\Omega}(u(x)-u_h(x))^2 \right)^{1/2}$ by hand*

4. **Use a different solver**

   You may also try to exchange the iterative linear solver, the lines for the new solver are already given as solution next to the old linear solver. Compare the number of iterations which is given by the following lines
  in the output (here we have 12 iterations):

```bash
=== CGSolver

12      1.9016e-09

=== rate=0.144679, T=0.02362, TIT=0.00196833, IT=12
```

*The previous solver uses 7 iterations and the new solver uses 10 iterations.*

## Exercise 2

Now consider the extended equation of the form
\begin{align}
    \begin{array}{rcll}
      -\nabla\cdot (k(x) \nabla u) +a(x) u  & = & f & \text{ in } \Omega, \\
      u & = & g & \text{ on } \partial\Omega,
    \end{array}
\end{align}
with scalar functions $k(x)$, $a(x)$.

This is the linear convection-diffusion equation, with $k(x)$ being the diffusion coefficient and $a(x)$ the reaction term.

1. In a first step show that the weak formulation of the problem
  involves the new bilinear form
  \begin{align}
    a(u,v) = \int_\Omega k(x) \nabla u(x) \cdot \nabla v(x) + a(x) u(x) v(x) \,dx .
  \end{align}

*This closely follows the derivation of the bilinear form as shown in section [Weak Formulation](#weakFormulation)  for the unextended equation.*

2. The main work is to extend the local operator given by the class
  `PoissonP1`. Your modifications should be done in the file `mypoissonp1.hh`, which is simply a copy of `poissonp1.hh`. The extension can be done in several steps:

 - Provide analytic functions for $k(x)$ and $a(x)$ and pass
    grid functions to the local operator like it is done for $f(x)$.
 - Extend the local operator to first handle $k(x)$ by assuming the
    function $k(x)$ to be *constant* on mesh elements.
 - Now extend the local operator (in `mypoissonp1.hh`) to handle $a(x) u(x)$. Also assume the function $a(x)$ to be *constant* on mesh elements.

- *The solution for providing the analytic functions and passing grid functions to the local operator are embedded in the notebook [(go to solution)](#solex2)*.
- *The place to include your modified local operator or the solution can be found [here](solex2LOP)*.
- *The solution for changes within the local operator can be found in `solutionpoissonp1.hh`. Additionally, in the following an overview of the main changes within the local operator is provided*.

Overview Changes in `poissonp1.hh`
 - *The local operator is implemented as the class PoissonP1. In order to extend the local operator to handle the functions $k(x)$ and $a(x)$, it is provided with two additional data members, k and a. Note, that these are **scalars**, as $k(x)$ and $a(x)$ are constant on mesh elements. As these are provided from outside the local operator, the constructor needs to be adapted as well.*

```cpp
// data members
  ...
  const K k;              // diffusion coefficient
  const T a;              // reaction term

// Constructor precomputes element independent data
  PoissonP1 (const F& f_, const K& k_, const T& a_, const FiniteElementType& fel)
    : f(f_), k(k_), a(a_)
```

- *As described before, the method `jacobian_volume` computes the element contributions to the stiffness matrix for a given element. As the [new bilinear form $a(u,v)$](#newBilinearForm) includes $a(x)$ and $k(x)$, these need to be included in `jacobian_volume` as well.*

```cpp
void jacobian_volume (...) const
{
    // evaluate diffusion coefficient
    typename K::Traits::RangeType kval;
    k.evaluate(eg.entity(),qp,kval);

    // evaluate reaction coefficient
    typename T::Traits::RangeType aval;
    a.evaluate(eg.entity(),qp,aval);
               ...
    mat.accumulate(lfsu,i,lfsu,j,(kval*A[i][j]+aval*phihat[j]*phihat[i] )*factor);
}
```

- *Hence also the method `alpha_volume` needs to be adadpted. As a reminder, it provides the element-local computations for the matrix-free evaluation of $a(u_h,\phi_i)$ for all test functions $\phi_i$.*

```cpp
void alpha_volume (...) const{
    // evaluate diffusion coefficient
    typename K::Traits::RangeType kval;
    k.evaluate(eg.entity(),qp,kval);

    // evaluate reaction coefficient
    typename T::Traits::RangeType aval;
    a.evaluate(eg.entity(),qp,aval);
    ...
    r.accumulate(lfsv,i,(kval*a_T[i]+aval*u*phihat[i])*factor);
}
```

3. Use your new local operator and test it. In the solution the code is tested for $k(x) = (x)_1$,
  $a(x)=\sum_{i=1}^{d}(x)_i$ and $g=\sum_{i=1}^d (x)_i^3$ by choosing  the right hand side $f(x)$ in such a way that $u(x)=\sum_{i=1}^d (x)_i^3$ solves the PDE.

 - *Reversed engineering yields $f(x) = - \left[6 \left(\sum_{i=1}^{d}(x)_i - (x)_1\right)+ 9(x)_1 \right] \cdot x_1 + \sum_{i=1}^{d}(x)_i \cdot \sum_{j=1}^{d}(x)^3_i$.*

*Below the solution is depicted.*

# Further Reading

Finite element method:
1. K. Eriksson, D. Estep, P. Hansbo and C. Johnson. *Computational Differential Equations*. Cambridge University Press.1996.
2. A. Ern and J.-L. Guermond. *Theory and practice of finite element methods*. Springer. 2004.
3. P. G. Ciarlet. *The finite element method for elliptic problems*. SIAM. Classics in Applied Mathematics. 2002.
4. D. Braess. *Finite Elemente*. Springer. 2003.
5. S. C. Brenner and L. R. Scott. *The mathematical theory of finite element methods*. Springer. 1994.
6. H. Elman, D. Silvester and A. Wathen.*Finite Elements and Fast Iterative Solvers*. Oxford University Press. 2005.
7. C. Großmann and H.-G. Roos. *Numerische Behandlung partieller Differentialgleichungen*. Teubner. 2006.
8. W. Hackbusch. *Theorie und Numerik elliptischer Differentialgleichungen*. Teubner. 1986.
9. R. Rannacher. *Einführung in die Numerische Mathematik II (Numerik partieller Differentialgleichungen)*. 2006. http://numerik.iwr.uni-heidelberg.de/~lehre/notes.
10. P. Bastian. *Lecture Notes on Scientific Computing with Partial Differential Equations*. 2014. https://conan.iwr.uni-heidelberg.de/data/teaching/finiteelements_ws2017/num2.pdf.


```python
%config InlineBackend.figure_format = 'retina'

from matplotlib import rcParams
rcParams["savefig.dpi"] = 96
rcParams["figure.dpi"] = 96
```

# The polynomial decomposition

## Introduction

The polynomial decomposition approach was originally proposed by [Keery et al. (2012)](
https://doi.org/10.1190/geo2011-0244.1). It consists of performing Debye decomposition of the SIP spectra with the assumption that the relaxation time distribution (RTD) of that spectra can be approximated by a polynomial function. In this approach, the complex resistivity $\rho^*$ is given by

\begin{equation}
\rho^* = \rho_0 \left[ 1 - \sum_{l=1}^{L} m_l\left(1-\frac{1}{1+(i\omega\tau_l)} \right) \right],
\end{equation}

where $\omega$ is the measurement angular frequencies
($\omega=2\pi f$) and $i$ is the imaginary unit. BISIP uses a generalized Cole-Cole polynomial decomposition scheme by including a user-defined Cole-Cole exponent ($c$). If $c = 1$, the equation is equivalent to Debye decomposition. If $c=0.5$, the equation is known as Warburg decomposition. For any other values we refer to this scheme as the generalized Cole-Cole decomposition. The complex resistivity is thus given by

\begin{equation}
\rho^* = \rho_0 \left[ 1 - \sum_{l=1}^{L} m_l\left(1-\frac{1}{1+(i\omega\tau_l)^c} \right) \right],
\end{equation}

where $L$ is an arbitrary real number that defines the discretization level of the RTD. In practice BISIP uses $L = 2\times n_{freq}$, where $n_{freq}$ is the number of measurement points in the frequency range. The RTD is defined by the distribution of $m_l$ values as a function of $\tau_l$ values. According to [Keery et al. (2012)](https://doi.org/10.1190/geo2011-0244.1), the RTD can be written as

\begin{equation}
m_l = \sum_{p=0}^P a_p (\log_{10} \tau_l)^p,
\end{equation}

where $a_p$ are polynomial coefficients and $P$ is the polynomial degree of the approximation. In practice, $P$ is often set to 3, 4 or 5. The user should aim to use the lowest degree possible while maintaining a satisfying fit with the data. As a result, $\rho^*$ depends on $P+1$ parameters:

- $\rho_0 \in [0, \infty)$, the direct current resistivity $\rho_0 = \rho^*(\omega\to 0)$.
- $a_p$, the $P$ polynomial coefficients.

From the recovered $a_p$ and $\rho_0$ values, the user can then compute the RTD and extract several integrating parameters, such as total chargeability and mean relaxation time. The user is referred to [Keery et al. (2012)](https://doi.org/10.1190/geo2011-0244.1) for the definition of these parameters. In this tutorial, we will compare the total chargeability values obtained with Debye and Warburg decomposition for 5 rock samples.

## Debye decomposition

First import the required packages.

```python
import numpy as np
from bisip import PolynomialDecomposition
from bisip import DataFiles

np.random.seed(42)
```

Let's start by inspecting the data sets so we can decide which polynomial approximation would be appropriate to fit the observations.

```python
data_files = DataFiles()  # get the dictionary of BISIP data files

for n, p in data_files.items():  # n is the filename, p the filepath
    model = PolynomialDecomposition(filepath=p)
    fig = model.plot_data('phase')
    ax = fig.gca()
    ax.set_title(n)
```

Samples K389173 and K389176 can be explained with a 3rd degree polynomial approxation, whereas the rest of the samples would likely require a 4th order polynomial approximation. For this example we will use a 4th order approaximation for all data sets.

```python
# Make a dict to hold results
results_debye = {}
# Define MCMC and Debye parameters
params = {'nwalkers': 32,
          'nsteps': 1000,
          'c_exp': 1,
          'poly_deg': 4,
         }

for n, p in data_files.items():  # n is the filename, p the filepath
    # Initialize with 1 file at a time, and pass the params
    model = PolynomialDecomposition(filepath=p, **params)
    model.fit()
    # Store the fitted model in the results dict
    results_debye[n] = model
```

    100%|██████████| 1000/1000 [00:01<00:00, 505.89it/s]
    100%|██████████| 1000/1000 [00:01<00:00, 512.17it/s]
    100%|██████████| 1000/1000 [00:01<00:00, 508.69it/s]
    100%|██████████| 1000/1000 [00:01<00:00, 509.93it/s]
    100%|██████████| 1000/1000 [00:01<00:00, 532.47it/s]
    100%|██████████| 1000/1000 [00:01<00:00, 522.29it/s]

Now we have a dictionary of fitted models in `results_debye`. Let's quickly inspect the fit results while discarding half the steps as burn-in. Check the chains with the `plot_traces()` method to see why the value is appropriate.

```python
for name, model in results_debye.items():
    fig = model.plot_fit_pa(discard=500)
    ax = fig.gca()
    ax.set_title(n)
```

We need to do a bit of work to extract the total chargeability. We will use the pandas library to organize the results in a DataFrame.

```python
import pandas as pd

names, models = zip(*results_debye.items())

mean_values = [m.get_param_mean(discard=500) for m in models]

df_debye = pd.DataFrame(mean_values)
df_debye.columns= models[0].param_names
df_debye.index = names
df_debye.head(6)
```

      r0
      a0
      a1
      a2
      a3
      a4

      SIP-K389170
      0.989771
      0.014033
      -0.002350
      -0.004634
      -0.000338
      0.000219

      SIP-K389172
      1.018388
      0.017680
      -0.011578
      -0.003509
      0.002061
      0.000526

      SIP-K389173
      1.011923
      0.003390
      -0.000995
      -0.000499
      0.000396
      0.000172

      SIP-K389174
      1.004851
      0.007547
      -0.003206
      -0.001871
      0.000548
      0.000242

      SIP-K389175
      0.997613
      0.006870
      -0.003937
      -0.001338
      0.000741
      0.000219

      SIP-K389176
      1.006941
      0.002426
      -0.001084
      -0.000048
      0.000564
      0.000159

We need to sum the coefficients and log_tau values:

\begin{equation}
m_l = \sum_{p=0}^P a_p (\log_{10} \tau_l)^p.
\end{equation}

```python
# Take log_tau from any model (considering the
# frequency range is the same for all samples)
log_tau = models[0].log_tau
print(log_tau)
```

    [-6.         -5.79487179 -5.58974359 -5.38461538 -5.17948718 -4.97435897
     -4.76923077 -4.56410256 -4.35897436 -4.15384615 -3.94871795 -3.74358974
     -3.53846154 -3.33333333 -3.12820513 -2.92307692 -2.71794872 -2.51282051
     -2.30769231 -2.1025641  -1.8974359  -1.69230769 -1.48717949 -1.28205128
     -1.07692308 -0.87179487 -0.66666667 -0.46153846 -0.25641026 -0.05128205
      0.15384615  0.35897436  0.56410256  0.76923077  0.97435897  1.17948718
      1.38461538  1.58974359  1.79487179  2.        ]

Define this utility function to get the chargeability value ($m$) that corresponds to each $\tau$ value.

```python
def get_m(df, P):
    m = {}
    for idx in df.index:
        m[idx] = 0
        for p in range(P+1):
            coeff = df_debye.loc[idx, f'a{p}']
            m[idx] += coeff*log_tau**p
        # Add the total chargeability in the df
        df.loc[idx, 'total_m'] = np.sum(m[idx])
    return m

m_debye = get_m(df_debye, params['poly_deg'])
df_debye.head(6)
```

      r0
      a0
      a1
      a2
      a3
      a4
      total_m

      SIP-K389170
      0.989771
      0.014033
      -0.002350
      -0.004634
      -0.000338
      0.000219
      1.342871

      SIP-K389172
      1.018388
      0.017680
      -0.011578
      -0.003509
      0.002061
      0.000526
      1.210898

      SIP-K389173
      1.011923
      0.003390
      -0.000995
      -0.000499
      0.000396
      0.000172
      0.786556

      SIP-K389174
      1.004851
      0.007547
      -0.003206
      -0.001871
      0.000548
      0.000242
      0.930805

      SIP-K389175
      0.997613
      0.006870
      -0.003937
      -0.001338
      0.000741
      0.000219
      0.655028

      SIP-K389176
      1.006941
      0.002426
      -0.001084
      -0.000048
      0.000564
      0.000159
      0.545386

Let's inspect the recovered RTDs:

```python
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
for n in names:
    ax.plot(10**log_tau, m_debye[n], label=n)
    ax.set_ylabel(r'$m$')
    ax.set_xlabel(r'$\tau$ (s)')
ax.set_title('Debye decomposition')
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlim([1e-5, 1e1])
ax.set_ylim([1e-3, 1e-1])
ax.legend(ncol=2)
```

## Warburg decomposition

Now we will repeat these steps for a Warburg decomposition ($c = 0.5$).

```python
# Make a dict to hold results
results_warburg = {}
# Define MCMC and Debye parameters
params = {'nwalkers': 32,
          'nsteps': 1000,
          'c_exp': 0.5,
          'poly_deg': 4,
         }

for n, p in data_files.items():  # n is the filename, p the filepath
    # Initialize with 1 file at a time, and pass the params
    model = PolynomialDecomposition(filepath=p, **params)
    model.fit()
    # Store the fitted model in the results dict
    results_warburg[n] = model
```

    100%|██████████| 1000/1000 [00:03<00:00, 262.07it/s]
    100%|██████████| 1000/1000 [00:03<00:00, 265.81it/s]
    100%|██████████| 1000/1000 [00:03<00:00, 263.03it/s]
    100%|██████████| 1000/1000 [00:03<00:00, 264.32it/s]
    100%|██████████| 1000/1000 [00:03<00:00, 269.38it/s]
    100%|██████████| 1000/1000 [00:03<00:00, 268.75it/s]

```python
names, models = zip(*results_debye.items())

mean_values = [m.get_param_mean(discard=500) for m in models]

df_warburg = pd.DataFrame(mean_values)
df_warburg.columns= models[0].param_names
df_warburg.index = names

m_warburg = get_m(df_warburg, params['poly_deg'])
df_warburg.head(6)
```

      r0
      a0
      a1
      a2
      a3
      a4
      total_m

      SIP-K389170
      0.989771
      0.014033
      -0.002350
      -0.004634
      -0.000338
      0.000219
      1.342871

      SIP-K389172
      1.018388
      0.017680
      -0.011578
      -0.003509
      0.002061
      0.000526
      1.210898

      SIP-K389173
      1.011923
      0.003390
      -0.000995
      -0.000499
      0.000396
      0.000172
      0.786556

      SIP-K389174
      1.004851
      0.007547
      -0.003206
      -0.001871
      0.000548
      0.000242
      0.930805

      SIP-K389175
      0.997613
      0.006870
      -0.003937
      -0.001338
      0.000741
      0.000219
      0.655028

      SIP-K389176
      1.006941
      0.002426
      -0.001084
      -0.000048
      0.000564
      0.000159
      0.545386

Add the Debye and Warburg values to a single DataFrame and print it.

```python
df = pd.merge(df_debye, df_warburg,
              left_index=True, right_index=True,
              suffixes=('_Debye', '_Warburg'))
df[['total_m_Debye', 'total_m_Warburg']].head(6)
```

      total_m_Debye
      total_m_Warburg

      SIP-K389170
      1.342871
      1.342871

      SIP-K389172
      1.210898
      1.210898

      SIP-K389173
      0.786556
      0.786556

      SIP-K389174
      0.930805
      0.930805

      SIP-K389175
      0.655028
      0.655028

      SIP-K389176
      0.545386
      0.545386

The values are identical. Finally let's look at the RTDs to confirm that they are the same.

```python
fig, ax = plt.subplots()
for n in names:
    ax.plot(10**log_tau, m_debye[n], label=n)
    ax.set_ylabel(r'$m$')
    ax.set_xlabel(r'$\tau$ (s)')
ax.set_title('Warburg decomposition')
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlim([1e-5, 1e1])
ax.set_ylim([1e-3, 1e-1])
ax.legend(ncol=2)
```

## Conclusions

In this tutorial we have seen how to do batch inversions, organize the results in pandas DataFrames and how to extract integrating parameters from the polynomial decomposition approach. The total chargeability parameter was identical for Debye and Warburg decomposition approaches.


### Preamble Stuff (ignore)

```python
from sympyhelpers import *
init_printing()
```

### Define all the symbols you'll need

```python
th,psi,thd,psid,R,g,t,l,thdd,phdd,psidd,N1,N2,N3,mr,tau,J  = \
symbols('theta,psi,thetadot,psidot,R,g,t,l,thetaddot,phiddot,psiddot,N_1,N_2,N_3,m_r,tau,J')
diffdict = {th:thd,thd:thdd,psi:psid,psid:psidd}
```

### Define rotation matrices and angular velocities for $\psi$ rotation about $\mathbf{e}_3$ and $-\theta$ rotation about $\mathbf{b}_2$

```python
bCi = rotMat(3,psi) # I->B
cCb = rotMat(2,-th) # B->C
bCi,cCb
```

```python
iWb = Matrix([0,0,psid]) #B frame
bWc = Matrix([0,-thd,0]) #C frame
```

### Apply Euler's second law to the disk, assuming planar moment of inertia J about $\mathbf{b}_3$ - all in $\mathcal{B}$ frame

```python
h_O = J*iWb;
M_O = Matrix([0,R,0]).cross(Matrix([-N1,-N2,N3]));
h_O,M_O
```

```python
eom1 = solve(transportEq(h_O,t,diffdict,iWb)-M_O,psidd); eom1
```

### Calculate rod kinematics (of center of mass) - all in the $\mathcal{B}$ frame

```python
r_GO = Matrix([0,R,0]) + cCb.T*Matrix([0,0,l/2]); r_GO
```

```python
v_GO = transportEq(r_GO,t,diffdict,iWb)
v_GO
```

```python
a_GO = transportEq(v_GO,t,diffdict,iWb);
simplify(a_GO)
```

### Solve for normal forces by applying Euler's first law to rod COM

```python
Ns = solve(mr*a_GO - Matrix([N1,N2,mr*g-N3]),(N1,N2,N3));Ns
```

### Parallel axis theorem to get MOI matrix for the rod about $Q$

```python
r_QG = Matrix([0,0,-l/2])
I_Q = mr*l**2/12*(eye(3) - diag(0,0,1)) + mr*((r_QG.transpose()*r_QG)[0]*eye(3) - r_QG*r_QG.transpose())
I_Q
```

### Calculate angular velocity and momentum about $Q$ in $\mathcal C$ frame components

```python
iWc = bWc + cCb*iWb
h_Q = I_Q*iWc
iWc,h_Q
```

### Now apply Euler's second law for rod about $Q$ (all in $\mathcal C$ frame)

```python
dh_Q = transportEq(h_Q,t,diffdict,iWc)
M_Q = Matrix([tau,mr*g*l/2*sin(th),0])
r_QO = cCb*Matrix([0,R,0])
a_QO = simplify(transportEq(transportEq(r_QO,t,diffdict,iWc),t,diffdict,iWc))
dh_Q,M_Q,a_QO
```

```python
r_QG = Matrix([0,0,-l/2])
eom2 = solve(dh_Q - (M_Q + mr*r_QG.cross(a_QO)),thdd)
eom2
```

### Finally, combine the solutions to the disk and rod motions for the full equations of motion

```python
simplify(solve([thdd - eom2[thdd],psidd - eom1[psidd].subs(N1,Ns[N1])],(psidd,thdd)))
```


# Physics 256
## Linear Regression and Least Squares Fitting

## Last Time

### [Notebook Link: 18_Logistic Map](./18_Logistic_Map.ipynb)

- The logistic map
- a simple route to chaotic behavior
- period doubling, fixed points and attractors

## Today

- An introduction to the theory of curve fitting

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('../include/notebook.mplstyle');
%config InlineBackend.figure_format = 'svg'
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
```

## Least Squares Analysis
Science is exceedingly becoming a *data rich* pursuit and modern experiments or simulations can easily produce terabytes of data.  Curve fitting or regression analysis is the simplest type of *data analysis*.

We begin by considering $N$ two dimensional data points:

$$\{(x_0,y_0),\ldots,(x_{N-1},y_{N-1})\} $$

and we want to fit this data to a continuous function:

$$Y(x;\vec{a})$$

where $\vec{a}$ is a vector of $M$ adjustable parameters that we want to determine such that our data **fits** the function.

Intuitively, this means that our data points should fall *near* the curve $Y$, we can quantify by defining the residual $\Delta_i$ for each point:

\begin{equation}
\Delta_i \equiv Y(x_i;\vec{a}) - y_i .
\end{equation}

Our approach will be to find $\vec{a}$ such that it minimizes the **squares** of these errors:
\begin{equation}
\sum_{i=0}^{N-1} \Delta_i^2 = \sum_{i=0}^{N-1} \left[Y(x_i;\vec{a}) - y_i\right]^2 .
\end{equation}
This is known as **least-squares fitting** and it was first used by Gauss to determine the orbits of the comets from observational data.

If there are uncertainties in the data such that $y_i \to y_i \pm \sigma_i$ then we need to modify this equation to place less emphasis on those data points with large error.  This defines the $\chi^2$ residual:

\begin{equation}
\chi^2 = \sum_{i=0}^{N-1} \left(\frac{\Delta_i}{\sigma_i}\right)^2 = \sum_{i=0}^{N-1} \frac{1}{\sigma_i^2} \left[Y(x_i;\vec{a}) - y_i\right]^2 .
\end{equation}

If we assume that the errors $\sigma_i$ are normally distributed, we can make concrete statistical statements about the goodness of fit.

## Anscombe's quartet

Before we learn how to do this, it is important to note that fitting a curve to a given dataset does not in of itself produce anything meaningful.  One must carefully ask the question:

*What is the probability that the data, given their uncertainties, are described by the curve?*

This is the statistical science of hypothesis testing.  We will investigate a linear fit to a [special data set](https://en.wikipedia.org/wiki/Anscombe%27s_quartet) that highlights the need to actually graph our data and consider the fit.

```python
def linear(x,a):
    '''A linear fitting function.'''
    return a[0] + a[1]*x

def χ2(Y,y):
    '''Unscaled χ^2'''
    return np.sum((Y-y)**2)

anscombe = np.array([[10.0,8.04,10.0,9.14,10.0,7.46,8.0,6.58],[8.0,6.95,8.0,8.14,8.0,6.77,8.0,5.76],
[13.0,7.58,13.0,8.74,13.0,12.74,8.0,7.71],[9.0,8.81,9.0,8.77,9.0,7.11,8.0,8.84],
[11.0,8.33,11.0,9.26,11.0,7.81,8.0,8.47],[14.0,9.96,14.0,8.10,14.0,8.84,8.0,7.04],
[6.0,7.24,6.0,6.13,6.0,6.08,8.0,5.25],[4.0,4.26,4.0,3.10,4.0,5.39,19.0,12.50],
[12.0,10.84,12.0,9.13,12.0,8.15,8.0,5.56],[7.0,4.82,7.0,7.26,7.0,6.42,8.0,7.91],
[5.0,5.68,5.0,4.74,5.0,5.73,8.0,6.8]])
```

```python
fig, axes = plt.subplots(2,2,sharex=True, sharey=True, squeeze=False, figsize=(10,7))
plt.subplots_adjust(hspace=0.05,wspace=0.05)
x = np.linspace(4,20,1000)
for i, ax in enumerate(axes.flat):
    Y = linear(x,(3,0.5))
    ax.plot(x,Y,'-',color=colors[0], linewidth=2, label='y = 0.5x + 3')
    r2 = χ2(linear(anscombe[:,2*i],(3,0.5)),anscombe[:,2*i+1])
    ax.plot(anscombe[:,2*i],anscombe[:,2*i+1],'o', markeredgecolor='None', alpha=0.7, markersize=7,
           label=f'$\chi^2 = {r2:4.2f}$', mfc=colors[2])
    ax.legend(loc='lower right')

[ax.set_ylabel('y') for ax in axes[:,0]]
[ax.set_xlabel('x') for ax in axes[-1,:]]
```

## Linear Regression

How were the coefficients in the Ancombe Quartet linear fit determined?

Let us investigate $\chi^2$ for a linear fitting function defined by:

\begin{equation}
Y(x;a_0,a_1) = a_0 + a_1 x
\end{equation}

such that

\begin{equation}
\chi^2 = \sum_{i=0}^{N-1} \frac{1}{\sigma_i^2} \left(a_0 + a_1 x_i - y_i\right)^2 .
\end{equation}

We want to minimize $\chi^2$ over the parameters $\vec{a}$:
\begin{align}
\frac{\partial \chi^2}{\partial a_0} &= 2 \sum_i \frac{1}{\sigma_i^2}(a_0 + a_1 x_i - y_i) = 0\newline
\frac{\partial \chi^2}{\partial a_1} &= 2 \sum_i \frac{1}{\sigma_i^2}(a_0 + a_1 x_i - y_i)x_i = 0 .
\end{align}

Defining:
\begin{equation}
\Sigma_q = \sum_i \frac{q_i}{\sigma_i^2}
\end{equation}
and solving these two equations yields:

\begin{align}
a_0 &= \frac{\Sigma_y\Sigma_{x^2} - \Sigma_x \Sigma_{xy}}{\Sigma_1\Sigma_{x^2} - (\Sigma_x)^2 } \newline
a_1 &= \frac{\Sigma_1\Sigma_{xy} - \Sigma_y \Sigma_{x}}{\Sigma_1\Sigma_{x^2} - (\Sigma_x)^2 } .
\end{align}

**Note:** If the errors are constant, i.e. $\sigma_i = \sigma\ \forall\ i=0,\ldots,N-1$ then it drops out of these expressions.  Thus we can use this formulation for fitting even if we don't know the uncertainties by setting $\sigma_i = 1$.

### Computing the uncertainty in $\vec{a}$

We can determine the uncertainty or error in our fit parameters using conventional error propagation resulting in:

\begin{equation}
\sigma^2_{a_\alpha} = \sum_i \left( \frac{\partial a_\alpha}{\partial y_i}\right)^2 \sigma_i^2 .
\end{equation}

We find:

\begin{align}
\sigma^2_{a_0} &= \frac{\Sigma_{x^2}}{\Sigma_1\Sigma_{x^2} - (\Sigma_x)^2} \newline
\sigma^2_{a_1} &= \frac{\Sigma_{1}}{\Sigma_1\Sigma_{x^2} - (\Sigma_x)^2} .
\end{align}

### Let's write some functions to perform a linear least-squares fit

```python
def Σ(σ,q):
    '''Compute the Σ function needed for linear fits.'''
    return np.sum(q/σ**2)

def get_a(x,y,σ):
    '''Get the χ^2 best fit value of a0 and a1.'''

    # Get the individual Σ values
    Σy,Σx,Σx2,Σ1,Σxy = Σ(σ,y),Σ(σ,x),Σ(σ,x**2),Σ(σ,np.ones(x.size)),Σ(σ,x*y)

    # the denominator
    D = Σ1*Σx2 - Σx**2

    # compute the best fit coefficients
    a = np.array([Σy*Σx2 - Σx*Σxy,Σ1*Σxy - Σx*Σy])/D

    # Compute the error in a
    aErr = np.array([np.sqrt(Σx2/D),np.sqrt(Σ1/D)])

    return a,aErr
```

 Group Programming challenge

Load the experimental data for the temperature T along a metal rod of length 10 cm
    suspended between two constant temperature baths in file  rod_temperature.dat. Perform a linear fit to the data and display the resulting fit function and parameters in the legend.

```python
plt.xlabel('L (cm)')
plt.ylabel('Temperature (K)')
plt.xlim(0,10)
plt.legend(loc=2)
```

```python

```


```python
# ES control minimizing a quadratic function
# Dan Arnold, daniel.brian.arnold@gmail.com

import numpy as np
import math
import matplotlib.pyplot as plt
#%matplotlib inline
```

```python
#ES function
#this variation includes low pass filtering of the gradient estimate

def es_function_lpf(t, T, hpf, lpf, f, c, a_param, Jk, Jkm1, sigmakm1, psikm1, gammakm1, uhatkm1):

    #outputs:
    #uk: the control to be applied at the next timestep
    #uhatk: the output of the integrator
    #gammak: the output of the lowpass filter
    #psik: signal after demodulation (mutliplication by cosine)
    #sigmak: the output of the washout (highpass) filter

    #inputs:
    #Jk, Jkm1: values of the objective function (metric we're optimizing) at the present and last timestep
    #sigmakm1: output of highpass filter at last timestep
    #psikm1: signal after demodulation at last timestep
    #gammakm1: output of lowpass filter
    #uhatkm1: output of integrator at last timestep
    #t: present time at this timestep
    #T: distance between timesteps, delta_T
    #hpf: high pass filter gain, usually make this an order of magnitude lower than w
    #lpf: low pass filter gain, usually make this equal to hpf, but I usually end up adjusting this quite a bit
    #c: gain on the integrator - how much to value new measurements compared to old measurements
    #ak: the amplitude of the probing signal

    #calculate angular frequency:
    w = 2*np.pi*f

    #extract the effect of the probing signal in the objective function
    #do this by passing the signal through a highpass filter
    sigmak = (Jk - Jkm1 - (hpf*T/2-1)*sigmakm1)/(1+hpf*T/2)
    #print("sigma: ", sigmak)

    #the resulting signal is a sinusoid, multiply by a sinusoid of the same frequency
    #this results in a cos**2 term, that has a DC component (we call this demodulation)
    psik = sigmak*np.cos(w*t)

    #pass the demodulated signal through a lowpass filter, to eliminate noise and "jumpiness"
    gammak = (T*lpf*(psik + psikm1) - (T*lpf - 2)*gammakm1)/(2 + T*lpf)

    #pass the resulting signal through an integrator - this approximates a gradient descent
    uhatk = uhatkm1 + c*T/2*(gammak + gammakm1)

    #modulation - add the perturbation to the next control setpoint
    uk = (uhatk + a_param*np.cos(w*t))

    return (uk, sigmak, psik, gammak, uhatk)
```

We can try to build some intuition as to why the method works.  Consider a quadratic objective function of the form:

\begin{align}
    J(u) = u^{2}
\end{align}

We seek to find the minimum of $J$ which is 0.  ES operates by approximating a gradient descent.  The gradient of this function is $2u$.  ES extracts this gradient by introducing a sinusoid into $u$, or

\begin{align}
    u = \hat{u} + a\cos \omega t
\end{align}

Where $\hat{u}$ is the quantity we want to drive to 0.  Note that $\hat{u}$ and $u$ have the same value on average (as the integral of a sinusoid over a period of the sinusoid is 0).  Here's what happens when we plug $u$ into $J$:

\begin{align}
    J(\hat{u} + a\cos \omega t) = \hat{u}^{2} + 2a\hat{u} \cos \omega t + a^{2}\cos^{2} \omega t
\end{align}

Now $J$ has a DC term and a sinusoidal term.  To extract the gradient, we can multiply by $\cos \omega t$:

\begin{align}
    J(\hat{u} + a\cos \omega t)\cos \omega t = \hat{u}^{2}\cos \omega t + 2a\hat{u} \cos^{2} \omega t + a^{2}\cos^{3} \omega t
\end{align}

Recall now that $2\cos^{2} \omega t = 1 - \cos 2\omega t$.  So, even powers of sinusoids have DC components.  Let's expand:

\begin{align}
    J(\hat{u} + a\cos \omega t)\cos \omega t = \hat{u}^{2}\cos \omega t + a\hat{u} \left( 1 + \cos 2 \omega t\right) + \frac{a^{2}}{2}\left(1 - \cos 2\omega t\right)\cos \omega t
\end{align}

We can "average" the signal by integrating over a period of the sinusoid.  When we do this, the only term remaining is $\hat{u}_{av}$.  We're off by a factor of 2, but we can account for this by choice of gains.  The subscript "av" indicates that this gradient is being averaged over a period of the sinusoid.  Now that we've got the gradient, the only thing we need to do is pass this quantity through an integrator to get a gradient descent.  The high and lowpass filters are put in the loop to reduce the effect of noise.

```python
#Simulation

#Line search - the goal of this experiment is to minimize a quadratic function
#the initial value of the controller, u, is the starting position
#we want to drive this value to the minimum of the quadratic, or u=0

#setup time
T = 0.01 #delta T: the length of each timestep
time = np.arange(0.,2000.)
time = time*T

#setup vectors to store past values of the ES control
J = np.zeros(len(time)) #objective function
sigma = np.zeros(len(time)) #value after highpass (washout) filter
psi = np.zeros(len(time)) #value after demodulation
gamma = np.zeros(len(time)) #value after lowpass filter
uhat = np.zeros(len(time)) #value after integration
u = np.zeros(len(time))  #control outputs of ES control (position)

#initial conditions
u[0] = 10 #choosing an initial condition - in this case its a position at these many units away from the origin
u_target = 0
J[0] = (u[0] - u_target)**2
sigma[0] = u[0]
psi[0] = u[0]
gamma[0] = u[0]
uhat[0] = u[0]

#setup parameters for the ES controller
f = 2 #frequency of probe, in Hz
c = -1 #gain on the integrator
lpf = 1 #lowpass filter critical frequency
hpf = 1 #highpass filter critical frequency
a = 0.25 #amplitude of the probe

#J = np.append(J,1)
print("program running")

for i in range(0,len(time)):

    #don't do anything the first timestep

    if(i>=1):

        #evaluate objective function value
        #minimize a quadratic, minimum value is 0
        J[i] = (u[i-1] - u_target)**2

        #call the ES controller - low pass filter
        uk, sigmak, psik, gammak, uhatk = es_function_lpf(time[i], T, hpf, lpf, f, c, a, J[i], J[i-1], \
            sigma[i-1], psi[i-1], gamma[i-1], uhat[i-1])

        #store values for next timestep
        u[i] = uk
        sigma[i] = sigmak
        psi[i] = psik
        gamma[i] = gammak
        uhat[i] = uhatk
        #print("----------------------------")

print("program complete")

target = np.ones(len(u))*u_target;

#this is the output of the ES controller
fig1 = plt.figure(1)
plt.plot(time,u,label="controller")
plt.plot(time,target,'--',label="target")
plt.title('position')
plt.xlabel('time')
plt.legend()
fig1.savefig('ES_basic.png')

plt.figure(2)
plt.plot(time,J,'r')
plt.title('objective function value')
plt.xlabel('time')

plt.show()

```

```python

```


# Demo: Generalised Filtering with eXogenous input

Wouter Kouw, 03-08-2021

---

Consider a discrete-time dynamic system with states $x_k \in \mathbb{R}^{d_x}$ and observations $y_k \in \mathbb{R}^{d_y}$.

We have the following state-space model:

$$\begin{align}
z_k =&\ A(\theta) z_{k-1} + B(\eta) u_k + q_{k-1} \\
y_k =&\ x_k + r_{k} \, ,
\end{align}$$

where $A$ is a state transition matrix, $C$ is an observation emission matrix, $q_{k-1} \sim \mathcal{N}(0, Q)$ is Gaussian distributed process noise and $r_{k} \sim \mathcal{N}(0,R)$ is Gaussian distributed measurement noise.

### System

This project considers a [harmonic oscillator](https://en.wikipedia.org/wiki/Harmonic_oscillator#Driven_harmonic_oscillators). Consider $y(t)$ as observed displacement, $x(t)$ as the unobserved state, $u(t)$ as the driving force and $v(t), w(t)$ as measurement and process noise, respectively. The continuous-time dynamics of the system are:

$$\begin{align}
m \frac{d^2 x(t)}{dt^2} =&\ - c \frac{d x(t)}{dt} - k x(t) + u(t) \\
y(t) =&\ x(t) + e(t)
\end{align}$$
where
$$
m     = \text{mass} \, , \quad
c     = \text{damping} \, , \quad
k     = \text{spring stiffness} \, ,
$$
constitute the physical parameters.

The process noise is a Wiener process, where the increment is Gaussian distributed $w(t) \sim \mathcal{N}(0, \zeta^{-1}dt)$ with $\zeta$ representing the precision of the process. The measurement noise is also a Wiener process, with $v(t) \sim \mathcal{N}(0, \xi^{-1}dt)$ and $\xi$ as precision parameter.

We cast this to the following discrete-time system. Using a forward finite difference method, i.e.,

$$\begin{align}
\frac{d^2 x(t)}{dt^2} = \frac{x_{t+1} - 2x_t + x_{t-1}}{(\Delta t)^2} \, , \quad \text{and} \quad
\frac{d x(t)}{dt} = \frac{x_{t+1} - x_t}{\Delta t} \, ,
\end{align}$$

we get:

$$\begin{align}
m \frac{x_{t+1} - 2x_t + x_{t-1}}{(\Delta t)^2}  =& - c \frac{x_{t+1} - x_t}{\Delta t} -  k x_t + u_t \\
x_{t+1} =&\ -\frac{-2m + k (\Delta t)^2}{m + c \Delta t} x_t - \frac{m - c \Delta t}{m + c \Delta t} = \frac{(\Delta t)^2}{m + c \Delta t} u_t \, .
\end{align}$$

If we substitute variables we get:

$$\begin{align}
x_{t+1} =&\ \theta_1 x_t + \theta_2 x_{t-1} + \eta u_t \\
y_t =&\ x_t + v_t \, .
\end{align}$$

#### Data-generating process

```julia
using LinearAlgebra
using ProgressMeter
using Plots
pyplot();
```

```julia
# Length of time-series
T = 100

# Sampling step size
Δt = 0.1

# Coefficients
θ_true = [0.1, 0.2]
η_true = .8

# Utility matrices
S = [1 Δt; 0 1];
s = [0; -Δt];

# Emission matrix
C = [1., 0.]

# Process noise covariance
τ = 4.
Q = inv(τ) .*[Δt^3/3 Δt^2/2;
              Δt^2/2     Δt];

# Measurement noise variance
R = 1e-3

# Initial states
x0 = [0., 1.]

# Preallocate data array
states = zeros(2,T)
observations = zeros(1,T)

# Initialize previous state
prev_state = x0

for k = 1:T

    # State transition
    states[:,k] = (S + s*θ_true')*prev_state + -s*η_true + cholesky(Q).L*randn(2,)

    # Observation with added measurement noise
    observations[1,k] = C'*states[:,k] + sqrt(R)*randn(1,)[1]

    # Update "previous state"
    prev_state = states[:,k]

end
```

```julia
# Inspect data
p1 = plot((1:T).*Δt, states[1,:], linewidth=3, color="blue", xlabel="", ylabel="x", label="states")
scatter!((1:T).*Δt, observations[1,:], color="black", markersize=2, label="observed", legend=:topleft)
p2 = plot((1:T).*Δt, states[2,:], linewidth=3, color="blue", xlabel="time [Δt="*string(Δt)*" seconds]", ylabel="x'", label="states")
plot(p1,p2, layout=(2,1), size=(900,300))
```

### Model specification

It is often not trivial to formulate a probabilistic model for a given problem. In the case of a linear Gaussian dynamical system, it is fortunately relatively straightforward.

We can convert the state transition and observation likelihood from the system dynamics to probabilistic form by integrating out the noise variables:

$$\begin{align}
x_k \sim&\ \mathcal{N}(Ax_{k-1}, Q) \\
y_k \sim&\ \mathcal{N}(Cx_k, R) \, .
\end{align}$$

The noise covariance matrices are now the covariance matrices of the stochastic state transition and likelihood. In ForneyLab, we often work with precision matrices instead of covariance matrices as that tends to require less matrix inversions.

We will need priors for the unknown variables. We will be doing recursive estimation (filtering) in this notebook, we need only define a prior for the "previous state" variable $x_{k-1}$:

$$\begin{align}
x_{k-1} \sim \mathcal{N}(m_k, W_k^{-1}) \, ,
\end{align}$$

where $W_k$ is a precision matrix. We will model the unknown noise precision matrices with Wishart distributions since these are the conditionally [conjugate priors](https://en.wikipedia.org/wiki/Conjugate_prior) for Gaussian likelihoods:

$$\begin{align}
Q^{-1} \sim&\ \mathcal{W}(V_Q, n_Q) \\
R^{-1} \sim&\ \mathcal{W}(V_R, n_R) \, .
\end{align}$$

We now have a likelihood and priors. That is all we need for ForneyLab.

```julia
using ForneyLab
```

```julia
fg = FactorGraph()

# Measurement noise precision (R inverse)
@RV Ri ~ Wishart(placeholder(:V_R, dims=(2,2)), placeholder(:n_R))

# Process noise precision (Q inverse)
@RV Qi ~ Wishart(placeholder(:V_Q, dims=(4,4)), placeholder(:n_Q))

# Previous state vector
@RV x_kmin1 ~ GaussianMeanPrecision(placeholder(:m_k, dims=(4,)), placeholder(:W_k, dims=(4,4)))

# State transition
@RV x_k ~ GaussianMeanPrecision(A*x_kmin1, Qi)

# Observation likelihood
@RV y_k ~ GaussianMeanPrecision(C*x_k, Ri)

# Indicate y_k will be observed
placeholder(y_k, :y_k, dims=(2,));

ForneyLab.draw(fg)
```

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.47.1 (20210417.1919)
 -->
<!-- Title: G Pages: 1 -->
<svg width="795pt" height="746pt"
 viewBox="0.00 0.00 794.50 746.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 742)">
<title>G</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-742 790.5,-742 790.5,4 -4,4"/>
<!-- 7836607220857614326 -->
<g id="node1" class="node">
<title>7836607220857614326</title>
<polygon fill="lightgrey" stroke="black" points="488.5,-738 411.5,-738 411.5,-684 488.5,-684 488.5,-738"/>
<text text-anchor="middle" x="450" y="-708.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_y_k</text>
</g>
<!-- 1646826350774227711 -->
<g id="node11" class="node">
<title>1646826350774227711</title>
<polygon fill="none" stroke="black" points="503.5,-639 396.5,-639 396.5,-567 503.5,-567 503.5,-639"/>
<text text-anchor="middle" x="450" y="-605.8" font-family="Times New Roman,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="450" y="-595.8" font-family="Times New Roman,serif" font-size="9.00">gaussianmeanprecision_3</text>
</g>
<!-- 7836607220857614326--1646826350774227711 -->
<g id="edge10" class="edge">
<title>7836607220857614326--1646826350774227711</title>
<path fill="none" stroke="black" d="M450,-683.8C450,-670.31 450,-653.68 450,-639.1"/>
<text text-anchor="start" x="450" y="-659.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">y_k</text>
<text text-anchor="start" x="429" y="-641.7" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="429" y="-677.4" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
</g>
<!-- 5443584827191874034 -->
<g id="node2" class="node">
<title>5443584827191874034</title>
<polygon fill="lightgrey" stroke="black" points="385,-54 305,-54 305,0 385,0 385,-54"/>
<text text-anchor="middle" x="345" y="-24.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_W_k</text>
</g>
<!-- 14074949235902030186 -->
<g id="node3" class="node">
<title>14074949235902030186</title>
<polygon fill="lightgrey" stroke="black" points="592.5,-396 515.5,-396 515.5,-342 592.5,-342 592.5,-396"/>
<text text-anchor="middle" x="554" y="-366.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_n_R</text>
</g>
<!-- 1827920715093087870 -->
<g id="node4" class="node">
<title>1827920715093087870</title>
<polygon fill="lightgrey" stroke="black" points="178,-396 124,-396 124,-342 178,-342 178,-396"/>
<text text-anchor="middle" x="151" y="-366.8" font-family="Times New Roman,serif" font-size="9.00">clamp_2</text>
</g>
<!-- 14907013347372185915 -->
<g id="node5" class="node">
<title>14907013347372185915</title>
<polygon fill="lightgrey" stroke="black" points="578,-54 500,-54 500,0 578,0 578,-54"/>
<text text-anchor="middle" x="539" y="-24.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_m_k</text>
</g>
<!-- 13282723035730671914 -->
<g id="node6" class="node">
<title>13282723035730671914</title>
<polygon fill="lightgrey" stroke="black" points="80,-162 0,-162 0,-108 80,-108 80,-162"/>
<text text-anchor="middle" x="40" y="-132.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_V_Q</text>
</g>
<!-- 4433327480983599190 -->
<g id="node7" class="node">
<title>4433327480983599190</title>
<polygon fill="lightgrey" stroke="black" points="273,-162 195,-162 195,-108 273,-108 273,-162"/>
<text text-anchor="middle" x="234" y="-132.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_n_Q</text>
</g>
<!-- 5888902112194352978 -->
<g id="node8" class="node">
<title>5888902112194352978</title>
<polygon fill="none" stroke="black" points="385,-522 309,-522 309,-450 385,-450 385,-522"/>
<text text-anchor="middle" x="347" y="-488.8" font-family="Times New Roman,serif" font-size="9.00">×</text>
<text text-anchor="middle" x="347" y="-478.8" font-family="Times New Roman,serif" font-size="9.00">multiplication_2</text>
</g>
<!-- 5888902112194352978--1827920715093087870 -->
<g id="edge14" class="edge">
<title>5888902112194352978--1827920715093087870</title>
<path fill="none" stroke="black" d="M308.74,-462.55C270.31,-440 212.05,-405.82 178.13,-385.92"/>
<text text-anchor="start" x="255" y="-425.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">clamp_2</text>
<text text-anchor="start" x="178.13" y="-388.52" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="295.74" y="-465.15" font-family="Times New Roman,serif" font-size="8.00">3 a </text>
</g>
<!-- 15831686215905286561 -->
<g id="node16" class="node">
<title>15831686215905286561</title>
<polygon fill="none" stroke="black" points="400.5,-405 293.5,-405 293.5,-333 400.5,-333 400.5,-405"/>
<text text-anchor="middle" x="347" y="-371.8" font-family="Times New Roman,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="347" y="-361.8" font-family="Times New Roman,serif" font-size="9.00">gaussianmeanprecision_2</text>
</g>
<!-- 5888902112194352978--15831686215905286561 -->
<g id="edge6" class="edge">
<title>5888902112194352978--15831686215905286561</title>
<path fill="none" stroke="black" d="M347,-449.87C347,-435.74 347,-419.46 347,-405.31"/>
<text text-anchor="start" x="347" y="-425.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">x_k</text>
<text text-anchor="start" x="326" y="-407.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="326" y="-443.47" font-family="Times New Roman,serif" font-size="8.00">2 in1 </text>
</g>
<!-- 3202067625536609345 -->
<g id="node9" class="node">
<title>3202067625536609345</title>
<polygon fill="none" stroke="black" points="590,-522 518,-522 518,-450 590,-450 590,-522"/>
<text text-anchor="middle" x="554" y="-488.8" font-family="Times New Roman,serif" font-size="9.00">W</text>
<text text-anchor="middle" x="554" y="-478.8" font-family="Times New Roman,serif" font-size="9.00">wishart_1</text>
</g>
<!-- 3202067625536609345--14074949235902030186 -->
<g id="edge7" class="edge">
<title>3202067625536609345--14074949235902030186</title>
<path fill="none" stroke="black" d="M554,-449.87C554,-432.57 554,-412.06 554,-396.15"/>
<text text-anchor="start" x="554" y="-425.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">n_R</text>
<text text-anchor="start" x="533" y="-398.75" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="536" y="-443.47" font-family="Times New Roman,serif" font-size="8.00">3 nu </text>
</g>
<!-- 6714325340166659249 -->
<g id="node12" class="node">
<title>6714325340166659249</title>
<polygon fill="lightgrey" stroke="black" points="786.5,-396 707.5,-396 707.5,-342 786.5,-342 786.5,-396"/>
<text text-anchor="middle" x="747" y="-366.8" font-family="Times New Roman,serif" font-size="9.00">placeholder_V_R</text>
</g>
<!-- 3202067625536609345--6714325340166659249 -->
<g id="edge13" class="edge">
<title>3202067625536609345--6714325340166659249</title>
<path fill="none" stroke="black" d="M590.3,-463.37C623.61,-443.52 672.84,-414.19 707.2,-393.71"/>
<text text-anchor="start" x="656" y="-425.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">V_R</text>
<text text-anchor="start" x="686.2" y="-396.31" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="590.3" y="-465.97" font-family="Times New Roman,serif" font-size="8.00">2 v </text>
</g>
<!-- 2042325482365814062 -->
<g id="node10" class="node">
<title>2042325482365814062</title>
<polygon fill="none" stroke="black" points="495.5,-171 388.5,-171 388.5,-99 495.5,-99 495.5,-171"/>
<text text-anchor="middle" x="442" y="-137.8" font-family="Times New Roman,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="442" y="-127.8" font-family="Times New Roman,serif" font-size="9.00">gaussianmeanprecision_1</text>
</g>
<!-- 2042325482365814062--5443584827191874034 -->
<g id="edge12" class="edge">
<title>2042325482365814062--5443584827191874034</title>
<path fill="none" stroke="black" d="M409.97,-99C396.62,-84.4 381.37,-67.75 369.01,-54.23"/>
<text text-anchor="start" x="394" y="-74.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">W_k</text>
<text text-anchor="start" x="348.01" y="-56.83" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="394.97" y="-92.6" font-family="Times New Roman,serif" font-size="8.00">3 w </text>
</g>
<!-- 2042325482365814062--14907013347372185915 -->
<g id="edge8" class="edge">
<title>2042325482365814062--14907013347372185915</title>
<path fill="none" stroke="black" d="M474.03,-99C487.38,-84.4 502.63,-67.75 514.99,-54.23"/>
<text text-anchor="start" x="497" y="-74.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">m_k</text>
<text text-anchor="start" x="493.99" y="-56.83" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="458.03" y="-92.6" font-family="Times New Roman,serif" font-size="8.00">2 m </text>
</g>
<!-- 1646826350774227711--5888902112194352978 -->
<g id="edge9" class="edge">
<title>1646826350774227711--5888902112194352978</title>
<path fill="none" stroke="black" d="M396.4,-579.48C383.11,-571.6 370.09,-561.52 361,-549 355.36,-541.23 351.94,-531.58 349.87,-522.23"/>
<text text-anchor="start" x="361" y="-542.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">variable_2</text>
<text text-anchor="start" x="328.87" y="-524.83" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="380.4" y="-582.08" font-family="Times New Roman,serif" font-size="8.00">2 m </text>
</g>
<!-- 1646826350774227711--3202067625536609345 -->
<g id="edge5" class="edge">
<title>1646826350774227711--3202067625536609345</title>
<path fill="none" stroke="black" d="M481.77,-566.87C494.55,-552.74 509.27,-536.46 522.07,-522.31"/>
<text text-anchor="start" x="506" y="-542.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">Ri</text>
<text text-anchor="start" x="501.07" y="-524.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="466.77" y="-560.47" font-family="Times New Roman,serif" font-size="8.00">3 w </text>
</g>
<!-- 2265991541729979615 -->
<g id="node13" class="node">
<title>2265991541729979615</title>
<polygon fill="none" stroke="black" points="270,-288 198,-288 198,-216 270,-216 270,-288"/>
<text text-anchor="middle" x="234" y="-254.8" font-family="Times New Roman,serif" font-size="9.00">W</text>
<text text-anchor="middle" x="234" y="-244.8" font-family="Times New Roman,serif" font-size="9.00">wishart_2</text>
</g>
<!-- 2265991541729979615--13282723035730671914 -->
<g id="edge11" class="edge">
<title>2265991541729979615--13282723035730671914</title>
<path fill="none" stroke="black" d="M197.96,-229.64C164.54,-209.83 114.89,-180.39 80.19,-159.83"/>
<text text-anchor="start" x="143" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">V_Q</text>
<text text-anchor="start" x="80.19" y="-162.43" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="183.96" y="-232.24" font-family="Times New Roman,serif" font-size="8.00">2 v </text>
</g>
<!-- 2265991541729979615--4433327480983599190 -->
<g id="edge3" class="edge">
<title>2265991541729979615--4433327480983599190</title>
<path fill="none" stroke="black" d="M234,-215.87C234,-198.57 234,-178.06 234,-162.15"/>
<text text-anchor="start" x="234" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">n_Q</text>
<text text-anchor="start" x="213" y="-164.75" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="216" y="-209.47" font-family="Times New Roman,serif" font-size="8.00">3 nu </text>
</g>
<!-- 6062987519380755961 -->
<g id="node14" class="node">
<title>6062987519380755961</title>
<polygon fill="lightgrey" stroke="black" points="665,-162 611,-162 611,-108 665,-108 665,-162"/>
<text text-anchor="middle" x="638" y="-132.8" font-family="Times New Roman,serif" font-size="9.00">clamp_1</text>
</g>
<!-- 520186944529735059 -->
<g id="node15" class="node">
<title>520186944529735059</title>
<polygon fill="none" stroke="black" points="480,-288 404,-288 404,-216 480,-216 480,-288"/>
<text text-anchor="middle" x="442" y="-254.8" font-family="Times New Roman,serif" font-size="9.00">×</text>
<text text-anchor="middle" x="442" y="-244.8" font-family="Times New Roman,serif" font-size="9.00">multiplication_1</text>
</g>
<!-- 520186944529735059--2042325482365814062 -->
<g id="edge1" class="edge">
<title>520186944529735059--2042325482365814062</title>
<path fill="none" stroke="black" d="M442,-215.87C442,-201.74 442,-185.46 442,-171.31"/>
<text text-anchor="start" x="442" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">x_kmin1</text>
<text text-anchor="start" x="421" y="-173.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="421" y="-209.47" font-family="Times New Roman,serif" font-size="8.00">2 in1 </text>
</g>
<!-- 520186944529735059--6062987519380755961 -->
<g id="edge2" class="edge">
<title>520186944529735059--6062987519380755961</title>
<path fill="none" stroke="black" d="M480.26,-228.55C518.69,-206 576.95,-171.82 610.87,-151.92"/>
<text text-anchor="start" x="546" y="-191.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">clamp_1</text>
<text text-anchor="start" x="589.87" y="-154.52" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="480.26" y="-231.15" font-family="Times New Roman,serif" font-size="8.00">3 a </text>
</g>
<!-- 15831686215905286561--2265991541729979615 -->
<g id="edge15" class="edge">
<title>15831686215905286561--2265991541729979615</title>
<path fill="none" stroke="black" d="M312.48,-332.87C298.59,-318.74 282.6,-302.46 268.7,-288.31"/>
<text text-anchor="start" x="294" y="-308.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">Qi</text>
<text text-anchor="start" x="247.7" y="-290.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="297.48" y="-326.47" font-family="Times New Roman,serif" font-size="8.00">3 w </text>
</g>
<!-- 15831686215905286561--520186944529735059 -->
<g id="edge4" class="edge">
<title>15831686215905286561--520186944529735059</title>
<path fill="none" stroke="black" d="M376.02,-332.87C387.7,-318.74 401.14,-302.46 412.83,-288.31"/>
<text text-anchor="start" x="398" y="-308.6" font-family="Times New Roman,serif" font-size="8.00" fill="red">variable_1</text>
<text text-anchor="start" x="391.83" y="-290.91" font-family="Times New Roman,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="360.02" y="-326.47" font-family="Times New Roman,serif" font-size="8.00">2 m </text>
</g>
</g>
</svg>

### Compile inference algorithm

We cannot use exact Bayesian inference here and will therefore switch to an approximate inference scheme: variational message passing.

```julia
# Define a factorization for the posterior
q = PosteriorFactorization(x_k, x_kmin1, Qi, Ri, ids=[:x_k, :x_kmin1, :Q, :R])

# Define a variational message passing procedure
algorithm = messagePassingAlgorithm([x_k, x_kmin1, Qi, Ri], q)

# Compile message passing procedure to an inference algorithm
code = algorithmSourceCode(algorithm);

# Import compiled functions to workspace
eval(Meta.parse(code));

# Print compiled functions (uncomment when desired)
# println(code)
```

### Execute inference algorithm

We execute the algorithm in an online fashion: after each timestep the posteriors for the state and noise precisions are used as priors for the next timestep.

```julia
# Parameters for prior distributions
m_x_0 = [0., 0., 1., -1.]
W_x_0 = 10. .* Matrix{Float64}(I, 4,4)
V_Q_0 = 10 .*Matrix{Float64}(I, 4,4)
n_Q_0 = 1.0
V_R_0 = Matrix{Float64}(I, 2,2)
n_R_0 = 100.0;
```

```julia
# Number of iterations
num_iterations = 5

# Initialize arrays for storing parameter estimates
m_x_k = zeros(4,T)
W_x_k = zeros(4,4,T)
V_Q_k = zeros(4,4,T)
n_Q_k = zeros(T,)
V_R_k = zeros(2,2,T)
n_R_k = zeros(T,)

# Initialize previous parameter estimates
m_x_kmin1 = m_x_0
W_x_kmin1 = W_x_0
V_Q_kmin1 = V_Q_0
n_Q_kmin1 = n_Q_0
V_R_kmin1 = V_R_0
n_R_kmin1 = n_R_0

# Initialize data dictionary
data = Dict()

# Initialize marginal distributions
marginals = Dict()
marginals[:x_kmin1] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=m_x_0, w=W_x_0)
marginals[:x_k] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=m_x_0, w=W_x_0)
marginals[:Qi] = ProbabilityDistribution(Wishart, v=V_Q_0, nu=n_Q_0)
marginals[:Ri] = ProbabilityDistribution(Wishart, v=V_R_0, nu=n_R_0)

# Recursive estimation procedure (posteriors at k => priors at k+1)
@showprogress for k = 1:T

    # Store data for current time-step
    data[:y_k] = observations[:,k]
    data[:m_k] = m_x_kmin1
    data[:W_k] = W_x_kmin1
    data[:V_Q] = V_Q_kmin1
    data[:n_Q] = n_Q_kmin1
    data[:V_R] = V_R_kmin1
    data[:n_R] = n_R_kmin1

    # Coordinate descent
    for kk = 1:num_iterations

        # Update recognition factors
        stepx_kmin1!(data, marginals)
        stepx_k!(data, marginals)
        stepQ!(data, marginals)
        stepR!(data, marginals)

    end

    # Update book-keeping
    m_x_kmin1 = m_x_k[:,k] = ForneyLab.unsafeMean(marginals[:x_k])
    W_x_kmin1 = W_x_k[:,:,k] = ForneyLab.unsafePrecision(marginals[:x_k])
    V_Q_kmin1 = V_Q_k[:,:,k] = marginals[:Qi].params[:v]
    n_Q_kmin1 = n_Q_k[k] = marginals[:Qi].params[:nu]
    V_R_kmin1 = V_R_k[:,:,k] = marginals[:Ri].params[:v]
    n_R_kmin1 = n_R_k[k] = marginals[:Ri].params[:nu]

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:06

### Inspect results
We can plot the estimates of the position and velocity over time, including the uncertainty of the estimates.

```julia
# Inspect data
p1 = plot((1:T).*Δt, states[1,:], linewidth=2, color="blue", xlabel="", ylabel="position 1", label="states")
scatter!((1:T).*Δt, observations[1,:], color="black", markersize=2, label="observed", legend=:topleft, title="2D position of car over time")
plot!((1:T).*Δt, m_x_k[1,:], linewidth=3, color="red", ribbon=[sqrt.(inv.(W_x_k[1,1,:])) sqrt.(inv.(W_x_k[1,1,:]))], alpha=0.6, label="estimated")
p2 = plot((1:T).*Δt, states[2,:], linewidth=2, color="blue", xlabel="time [Δt="*string(Δt)*" seconds]", ylabel="position 2", label="states")
scatter!((1:T).*Δt, observations[2,:], color="black", markersize=2, label="observed", legend=:bottomleft)
plot!((1:T).*Δt, m_x_k[2,:], linewidth=3, color="red", ribbon=[sqrt.(inv.(W_x_k[2,2,:])) sqrt.(inv.(W_x_k[2,2,:]))], alpha=0.6, label="estimated")
plot(p1,p2, layout=(2,1), size=(700,400))
```

```julia
# Inspect data
p1 = plot((1:T).*Δt, states[3,:], linewidth=2, color="blue", xlabel="", ylabel="velocity 1", label="states", title="2D velocity of car over time")
plot!((1:T).*Δt, m_x_k[3,:], linewidth=3, color="red", ribbon=[sqrt.(inv.(W_x_k[3,3,:])) sqrt.(inv.(W_x_k[3,3,:]))], alpha=0.6, label="estimated")
p2 = plot((1:T).*Δt, states[4,:], linewidth=2, color="blue", xlabel="time [Δt="*string(Δt)*" seconds]", ylabel="velocity 2", label="states")
plot!((1:T).*Δt, m_x_k[4,:], linewidth=3, color="red", ribbon=[sqrt.(inv.(W_x_k[4,4,:])) sqrt.(inv.(W_x_k[4,4,:]))], alpha=0.6, label="estimated")
plot(p1,p2, layout=(2,1), size=(700,400))
```

```julia

```


# Introduction

(comp_numbers)=
## Complex numbers

In complex analysis we are dealing with the set of complex numbers $\mathbb{C}$ which are ordered pairs of the form $(x, y)$, where $x$ and $y$ are real numbers. If we introduce the imaginary unit $i = \sqrt{-1}$ we can write it in its *rectangular form*:

$$ z = x + iy $$ (rectform)

Every complex number $z = x + iy$ has a **real part** Re$z = x$ and an **imaginary part** Im$z = y$. Sometimes $\mathfrak{R}(z)$ and $\mathfrak{I}(z)$ are used instead of Re and Im. Two complex numbers $z_1$ and $z_2$ are equal iff their real and imaginary parts are equal,

$$ z_1 = z_2 \iff \text{Re}(z_1) = \text{Re}(z_2) \wedge \text{Im}(z_1) = \text{Im}(z_2). $$

If Re$(z) = 0$ and Im$(z) \neq 0$ we say that $z$ is *pure imaginary*. On the other hand, if Im$(z) = 0$ and Re$(z) \neq 0$ we recover the *pure real* number $x$. Therefore, we identify real numbers as complex numbers whose imaginary part is equal to 0, so $\mathbb{R}$ is a subset of $\mathbb{C}$, i.e. $\mathbb{R} \subset \mathbb{C}$.

(comp_algebra)=
## Complex algebra

### Addition and multiplication

Adding complex numbers is performed by adding real and imaginary parts separately:

$$ z_1 \pm z_2 = (x_1 + i y_1) \pm (x_2 + iy_2) = (x_1 \pm x_2) + i(y_1 \pm y_2) $$

Multiplying two complex numbers is done by expanding the brackets and grouping real and imaginary parts:

$$ \begin{align}
z_1 \cdot z_2 & = (x_1 + i y_1) \cdot (x_2 + iy_2) = x_1 x_2 + i^2 y_1 y_2 + i x_1 y_2 + i x_2 y_1 \\
& = (x_1 x_2 - y_1 y_2) + i(x_1 y_2 + x_2 y_1).
\end{align} $$

### Complex conjugate
```{index} Complex conjugate
```

For every complex number $z = x + iy$ we define its **complex conjugate** $z^*$ or $\overline{z}$ by changing the sign of its imaginary part:

$$ z^* \equiv x - iy $$

This is an [involutory operation](https://en.wikipedia.org/wiki/Involution_(mathematics)) since \\( (z^*)^* = z \\) with the following properties:

$$ (z_1 \pm z_2)^* = z_1^* \pm z_2^* \\
(z_1 \cdot z_2)^* = z_1^* \cdot z_2^* \\
(z^{-1})^* = (z^*)^{-1} $$

We can write the real and imaginary parts of $z$ using the complex conjugate:

$$ \text{Re}(z) = \frac{1}{2} (z + z^*), \quad \text{Im}(z) = \frac{1}{2i} (z - z^*) $$

### Modulus

```{index} Modulus
```

To every complex number $z$ we assign a modulus (absolute value) $| z |$, a non-negative real number defined as:

$$ | z | \equiv \sqrt{x^2 + y^2}
$$ (mdls)

Zero is the only complex number whose modulus is 0.

```{admonition} Properties

The modulus of a complex number satisfies the following properties:

$$ |z_1| \cdot |z_2| = |z_1 z_2|, \quad | z^{-1} | = |z|^{-1}, $$

$$|z^*| = |z|, \quad z^* \cdot z = |z|^2 $$

```

### Division

We divide a complex number $z_1$ by another complex number $z_2 \neq 0$ by expanding the fraction such that the denominator becomes a real number. We do this by multiplying both the numerator and the denominator by $z_2^*$ (see the last property of a modulus above).

$$ \begin{align}
\frac{z_1}{z_2} & = \frac{x_1 + iy_1}{x_2 + iy_2} = \frac{x_1 + iy_1}{x_2 + iy_2} \cdot \frac{x_2 - iy_2}{x_2 - iy_2} \\
& = \left ( \frac{x_1x_2 + y_1y_2}{x_2^2 + y_2^2} \right ) + i \left ( \frac{x_2y_1- x_1y_2}{x_2^2 + y_2^2} \right )
\end{align} $$

### Multiplicative inverse

Let us find the reciprocal of a complex number $z = x + iy$:

$$ \begin{align}
z^{-1} & = \frac{1}{z} = \frac{1}{x + iy} \cdot \frac{x - iy}{x -iy} = \frac{x - iy}{x^2 + y^2} \\ & = \frac{x}{x^2 + y^2} -i \frac{y}{x^2 + y^2}
\end{align} $$

### Powers and roots

Some operations are quite tedious and much more difficult to perform on complex numbers in their rectangular form $z = x + iy$. Raising a complex number to some power and taking a root are one of such operations. We will therefore show the formula here for these operations for a complex number in its {ref}`comp_trig_form` (we introduce this in the next notebook).

$$ z = r( cos \varphi + i \sin \varphi) \\
z^n = r^n ( cos n \varphi + i \sin n \varphi) $$

Let \\(w = z^{1/n} \\), then

$$ w = r^{1/n} \left ( \cos \left ( \frac{ \varphi}{n} + \frac{2k \pi}{n} \right ) + i \sin \left ( \frac{ \varphi}{n} + \frac{2k \pi}{n} \right ) \right ), $$

where $k \in \mathbb{Z}$.

```python

```