# Adaptive Finite Element Method for a Nonlinear Poisson Equation

In this tutorial we solve the nonlinear Poisson equation from tutorial 01 using
adaptive grid refinement. The finite element solution on a given mesh is used to
compute local error indicators that can be used to iteratively reduce the
discretization error. It is assumed that the nonlinearity of the PDE is not too
strong, so that it may be approximated by linearizing around the current finite
element solution. This allows the calculation of local error estimates
that quantify the discretization error on each element of the mesh.

This tutorial depends on tutorial 01 which discusses
the solution of the considered partial differential equation. It is assumed that
you have worked through tutorial 01 before. Additionally, the error estimator has
to assemble skeleton terms, which are explained in greater detail in tutorial 02.
It may therefore be useful to have worked through that tutorial as well, but the
actual method discussed there is not relevant here.

# Problem Formulation

We consider the following nonlinear Poisson equation with
Dirichlet and Neumann boundary conditions as introduced in tutorial 01:

\begin{align}  \label{eq:ProblemStrong}
-\Delta u + q(u) &= f &&\text{in $\Omega$},\\
u &= g &&\text{on $\Gamma_D\subseteq\partial\Omega$},\\
-\nabla u\cdot \nu &= j &&\text{on $\Gamma_N=\partial\Omega\setminus\Gamma_D$}.
\end{align}

$\Omega\subset\mathbb{R}^d$ is a domain, $q:\mathbb{R}\to\mathbb{R}$ is a given, possibly nonlinear function and $f: \Omega\to\mathbb{R}$ is the source term and $\nu$ denotes the unit outer normal to the domain.

The weak formulation of this problem is derived by multiplication with an appropriate test function and integrating by parts. This results in the abstract problem:

\begin{equation}
\text{Find $u\in U$ s.t.:} \quad r^{\text{NLP}}(u,v)=0 \quad \forall v\in V,
\label{Eq:BasicBuildingBlock}
\end{equation}

with the continuous residual form

\begin{equation*}
r^{\text{NLP}}(u,v) = \int_\Omega \nabla u \cdot \nabla v + (q(u)-f)v\,dx + \int_{\Gamma_N} jv\,ds
\label{eq:ResidualForm}
\end{equation*}

and the function spaces $U= \{v\in H^1(\Omega) \,:\, \text{''$v=g$'' on $\Gamma_D$}\}$ and $V= \{v\in H^1(\Omega) \,:\, \text{''$v=0$'' on $\Gamma_D$}\}$. We assume that $q$ is such that this problem has a unique solution.

The example application uses a domain $\Omega$ that is L-shaped and given by
\begin{equation*}
  \Omega = \left\{ (x,y) \in \mathbb{R}^2 \colon |x| < 1, |y| < 1, (x < 0 \lor y > 0) \right\}.
\end{equation*}
We set $f = 0$ and $q(u) = 0$, and only consider Dirichlet boundary conditions.
The resulting PDE in residual form is
\begin{equation*}
r^{\text{NLP}}(u,v) = \int_\Omega \nabla u \cdot \nabla v \,dx,
\end{equation*}
and  the function
\begin{equation*}
  u(r,\theta) = r^{2/3} \cdot \text{sin}\left(\frac{2}{3} \theta\right)
\end{equation*}
in polar coordinates is one of its solutions on the given domain. We choose the restriction of this function to the boundary of $\Omega$ for the Dirichlet boundary condition.

# Adaptive Grid Refinement

In the previous tutorials a fixed finite element space $V_h$ was used, based on a fixed finite element mesh with an ordered set

\begin{equation}
\mathcal{X}_h = \{x_1,\ldots,x_N\}
\end{equation}
of vertices and an ordered set
\begin{equation}
\mathcal{T}_h = \{T_1, \ldots, T_M\}
\end{equation}
of elements. The task of choosing an appropriate mesh was not discussed in detail.

Any choice of finite element space will lead to discretization errors due to the finite-dimensional approximation of the true solution of the PDE, but the size of this error can be controlled by choosing an adequate mesh. Let $u_h \in V_h$ be the finite
element solution and $\|\cdot\|$ an appropriate norm. The goal is then
\begin{equation*}
  \|u - u_h\| \leq \text{TOL}
\end{equation*}
with a given tolerance $\text{TOL}$, while keeping the complexity and size of $V_h$ as low as possible.

The *a-priori* error estimates of formal proofs are not uitable for this task, since they contain a constant $C$ that is typically unknown and don't provide information about the spatial distribution of the error. For this reason one is interested in **a-posteriori** error
estimates of the form

\begin{equation}
  \|u - u_h\| \leq \gamma(u_h) = \left( \sum_{T \in {\mathcal T}_h} \gamma_T^2(u_h) \right)^{1/2},
\end{equation}

where $\gamma_T(u_h)$ is a **local error estimator** for the element $T$. Such an error estimate should:
1. have comparatively low cost of calculation to make an adaptive approach feasible
2. accurately describe the spatial distribution and size of the discretization error

The local estimates $\gamma_T(u_h)$ can then be used in a strategy that tries to minimize $\gamma(u_h)$
for fixed $\text{dim}(V_h)$ and thereby find a mesh with low discretization error $\|u - u_h\|$.

If the considered PDE is linear, i.e. $q(u)$ is an affine linear function, then it can be shown [[1]](#ref)
that such an estimate of the discretization error is given by

\begin{equation*}
  \gamma_T^2(u_h) = h_T^2 \| R_T(u_h) \|_{0,T}^2 + \sum_{F \in \mathcal{F}_h \cap \partial T} h_T \| R_F(u_h) \|_{0,F}^2
\end{equation*}

with the **element residuals**

\begin{equation*}
  R_T(u_h) = f + \Delta u_h - q(u_h)
\end{equation*}

and the **face residuals**
\begin{equation*}
  R_F(u_h) = \begin{cases} 2^{-1/2} [-(\nabla u_h) \cdot \nu] & F \in \mathcal{F}_h^i \\ - (\nabla u_h) \cdot \nu - j & F \in \mathcal{F}_h^N \end{cases}
\end{equation*}

Here $h_T$ is the local mesh width of the element $T$, $\mathcal{F}_h$ the set of faces of the elements
in $\mathcal{T}_h$, $\partial T$ the boundary of element $T$, $\mathcal{F}_h^i \subset \mathcal{F}_h$
the set of interior faces, $\mathcal{F}_h^N \subset \mathcal{F}_h$ the set of Neumann boundary faces and $\nu$ the unit normal vector of the face $F$ pointing from one of the elements $T_F^-$ that belong to $F$ to the other $T_F^+$. $\|\cdot\|_{0,T}$ is the $L^2$ norm on $T$, $\|\cdot\|_{0,F}$ that on $F$, and $[\cdot]$ the jump of the given expression across the face $F$ in the direction of $\nu$. Broadly speaking, the element residual $R_T$ measures to which degree $u_h$ solves the PDE on the element $T$, and the face residual $R_F$ measures the consistency of the flux $-(\nabla u_h)\cdot \nu$ between the elements.

With these definitions one can show that
\begin{equation*}
  \|u - u_h\|_{1,\Omega} \leq C \gamma(u_h) = C \left( \sum_{T \in {\mathcal T}_h} \gamma_T^2(u_h) \right)^{1/2}
\end{equation*}
with $\|\cdot\|_{1,\Omega}$ the $H^1$ norm on $\Omega$. The right hand side of this inequality does not
depend on $u$, which means it can be evaluated and used to control the discretization error.

Several aspects have to be considered:
 - The derivation of the error estimator given above does not require additional regularity beyond $u \in H^1(\Omega)$. This is critical, since error control is especially important in problems with low regularity.
 - The constant $C$ in the estimate is usually not known exactly, and this should be taken into account when defining the tolerance $\text{TOL}$.
 - The error estimate may converge with a lower order than the actual discretization error which yields a very pessimistic stopping criterion. If the error estimate converges with the same order, then the estimator is called *efficient*. The lowest applicable constant $C$ is then the *efficiency index*.

All of these considerations only hold if the PDE is linear. If the function $q(u)$ is nonlinear, then the estimate $\gamma(u_h)$ is no longer guaranteed to be an upper bound for the discretization error $\|u-u_h\|_{1,\Omega}$. However, it may still be used as an error indicator that is used to refine a given mesh. This can produce misleading results, and therefore the conclusions drawn from the estimate should be rather conservative. The error estimate will become more reliable the closer the finite element solution $u_h$ is to the exact solution $u$, as long as the function $q(u)$ is regular enough to be linearized around the exact solution.

# Realization in PDELab

The `fem` section in the ini-file provides the parameters for the finite element method. The new parameters control the adaptive refinement: `steps` specifies the maximum number of iterations for the refinement loop, `uniformlevel` the number of iterations that should use global refinement in the beginning, `tol` denotes the error tolerance
used as a stopping criterion, and `fraction` allows setting the number of elements that should be marked in each step.

```ini
[fem]
steps=200
uniformlevel=0
tol=0.01
fraction=0.7
```

The following includes again the necessary headers.

```c++
#include <dune/jupyter.hh>
#include "nonlinearpoissonfemestimator.hh"
#include "problem.hh"
#include "../tutorial01/nonlinearpoissonfem.hh"
```

As described in the previous tutorials, a grid is instantiated in the following cell:

```c++
// open ini file
Dune::ParameterTree ptree;
Dune::ParameterTreeParser ptreeparser;
ptreeparser.readINITree("tutorial05.ini",ptree);

// read ini file
const int dim = 2;
const int degree = 1;

using Grid = Dune::UGGrid<2>;
std::string filename = ptree.get("grid.twod.filename",
                                 "ldomain.msh");
Dune::GridFactory<Grid> factory;
Dune::GmshReader<Grid>::read(factory,filename,true,true);
std::shared_ptr<Grid> gridp(factory.createGrid());
```

```c++
// get leaf gridview
auto gv = gridp->leafGridView();
typedef decltype(gv) GV;

// dimension and important types
const int dim = GV::dimension;
typedef typename GV::Grid::ctype DF; // type for coordinates
typedef double RF;                   // type for computations
```

The code alternates between solving the problem on the current mesh and using the current solution for adaptive refinement. These two steps are repeated until the error estimate is below the prescribed tolerance or the maximum number of iterations is reached. The parts of the problem that are not affected by the changing mesh are instantiated outside of the loop.

```c++
// make user functions
RF eta = ptree.get("problem.eta",(RF)1.0);
Problem<RF> problem(eta);
```

```c++
auto g = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [&](const auto& e, const auto& x){return problem.g(e,x);}
);;
```

```c++
auto b = Dune::PDELab::makeBoundaryConditionFromCallable(
  gv,
  [&](const auto& i, const auto& x){return problem.b(i,x);}
);;
```

The parameter class `Problem` defines functions for the right hand side $f$, the source term $q$ and the value of the Dirichlet boundary condition $g$. These functions are independent of the chosen discretization. This also holds for the lambda functions that encapsulate the Dirichlet boundary condition and its value.

The grid function space `GFS` is a representation of the finite element space $V_h$. Most of its internals are independent of the actual mesh. The basis functions defined by the finite element map `FEM`, the constraints container `CON` and the vector backend `VBE` do not need to be modified when the mesh is refined. The only
parts of the grid function space that have to be updated are the local to global map and the constrained indices, which will be considered later.

```c++
// Make grid function space
using FEM = Dune::PDELab::PkLocalFiniteElementMap<GV,DF,RF,degree>;
FEM fem(gv);
using CON = Dune::PDELab::ConformingDirichletConstraints;
using VBE = Dune::PDELab::ISTL::VectorBackend<>;
using GFS = Dune::PDELab::GridFunctionSpace<GV,FEM,CON,VBE> ;
GFS gfs(gv,fem);
gfs.name("Vh");
```

Then the constraints of the initial mesh and an initial guess for the finite element solution are assembled.

```c++
// Assemble constraints
typedef typename GFS::template ConstraintsContainer<RF>::Type CC;
CC cc;
Dune::PDELab::constraints(b,gfs,cc); // assemble constraints

// A coefficient vector
using Z = Dune::PDELab::Backend::Vector<GFS,RF>;
Z z(gfs); // initial value

// Fill the coefficient vector
Dune::PDELab::interpolate(g,gfs,z);
```

The central loop of the function starts by reading in the maximum number of refinement iterations and the number of steps that should use global refinement. This can be used to guarantee a maximum diameter of the elements of the resulting meshes. From this point on all statements are part of the loop that is repeated until the error estimate is small enough or the maximum number of iterations is reached.

```c++
// adaptation loop
// only grid function space and coefficient vector
// live outside this loop
int steps = ptree.get("fem.steps",(int)3);
int uniformlevel = ptree.get("fem.uniformlevel",(int)2);
for (int i=0; i<steps; i++)
{
  std::stringstream s;
  s << i;
  std::string iter;
  s >> iter;
  std::cout << "Iteration: " << iter
  //---   << "\thighest level in grid: " << grid.maxLevel()
  << "\thighest level in grid: " << gridp->maxLevel()
    << std::endl;
  std::cout << "constrained dofs=" << cc.size()
    << " of " << gfs.globalSize() << std::endl;

  // Make local operator
  typedef NonlinearPoissonFEM<Problem<RF>,FEM> LOP;
  LOP lop(problem);

  // Make a global operator
  typedef Dune::PDELab::ISTL::BCRSMatrixBackend<> MBE;
  MBE mbe((int)pow(1+2*degree,dim));
  typedef Dune::PDELab::GridOperator<
    GFS,GFS,  /* ansatz and test space */
    LOP,      /* local operator */
    MBE,      /* matrix backend */
    RF,RF,RF, /* domain, range, jacobian field type */
    CC,CC     /* constraints for ansatz and test space */
    > GO;
  GO go(gfs,cc,gfs,cc,lop,mbe);

  // Select a linear solver backend
  typedef Dune::PDELab::ISTLBackend_SEQ_CG_AMG_SSOR<GO> LS;
  LS ls(100,0);

  // solve nonlinear problem
  Dune::PDELab::Newton<GO,LS,Z> newton(go,z,ls);
  newton.setReassembleThreshold(0.0);
  newton.setVerbosityLevel(2);
  newton.setReduction(1e-6);
  newton.setMinLinearReduction(1e-6);
  newton.setMaxIterations(25);
  newton.setLineSearchMaxIterations(10);
  newton.apply();

  // set up error estimator
  typedef Dune::PDELab::P0LocalFiniteElementMap<DF,RF,dim> P0FEM;
  P0FEM p0fem(Dune::GeometryTypes::simplex(dim));
  typedef Dune::PDELab::NoConstraints NCON;
  typedef Dune::PDELab::GridFunctionSpace<GV,P0FEM,NCON,VBE> P0GFS;
  P0GFS p0gfs(gv,p0fem);
  typedef NonlinearPoissonFEMEstimator<Problem<RF>,FEM> ESTLOP;
  ESTLOP estlop(problem);
  typedef Dune::PDELab::EmptyTransformation NCC;
  typedef Dune::PDELab::GridOperator<
    GFS,P0GFS,    /* one value per element */
    ESTLOP,       /* operator for error estimate */
    MBE,RF,RF,RF, /* same as before */
    NCC,NCC       /* no constraints */
    > ESTGO;
  ESTGO estgo(gfs,p0gfs,estlop,mbe);

  // compute local error contribution and global error
  using Z0 = Dune::PDELab::Backend::Vector<P0GFS,RF>;
  Z0 z0(p0gfs,0.0);
  estgo.residual(z,z0);
  auto estimated_error = sqrt(z0.one_norm());
  std::cout << "Estimated error in step " << i
    << " is " << estimated_error << std::endl;

  // =========================
  // compute true L2 error
  // =========================
  typedef Dune::PDELab::DiscreteGridFunction<GFS,Z> ZDGF;
  ZDGF zdgf(gfs,z); // the FE function as a grid function; moved from below
  // put your code here !!!

  // vtk output
  std::cout << "VTK output" << std::endl;
  Dune::SubsamplingVTKWriter<GV>
    vtkwriter(gv,Dune::refinementIntervals(ptree.get("output.subsampling",(int)1)));
  typedef Dune::PDELab::VTKGridFunctionAdapter<ZDGF> VTKF;
  vtkwriter.addVertexData( // the FE solution
      std::shared_ptr<VTKF>(new VTKF(zdgf,"fesol")));
  // =========================
  // output error function from above
  // your code here !!!

  // =========================
  // output squares of the local estimator
  typedef Dune::PDELab::DiscreteGridFunction<P0GFS,Z0> Z0DGF;
  Z0DGF z0dgf(p0gfs,z0);
  typedef Dune::PDELab::VTKGridFunctionAdapter<Z0DGF> VTKF0;
  vtkwriter.addCellData(
      std::shared_ptr<VTKF0>(new VTKF0(z0dgf,"gammaT^2")));
  // now write the file
  vtkwriter.write(ptree.get("output.filename",
        (std::string)"output")+iter, Dune::VTK::appendedraw);

  // error control
  auto tol = ptree.get("fem.tol",(double)0.0);
  if (estimated_error<=tol) break;
  if (i==steps-1) break;

  // mark elements for refinement
  std::cout << "mark elements" << std::endl;
  auto fraction = ptree.get("fem.fraction",(double)0.5);
  RF eta_refine,eta_coarsen;
  Dune::PDELab::error_fraction(
      z0,fraction,0.0,eta_refine,eta_coarsen);
  if (fraction>=1.0 || i<uniformlevel) eta_refine=0.0;
  //--- Dune::PDELab::mark_grid(grid,z0,eta_refine,0.0,2);
  Dune::PDELab::mark_grid(*gridp,z0,eta_refine,0.0,2);

  // do refinement
  std::cout << "adapt grid and solution" << std::endl;
  //--- Dune::PDELab::adapt_grid(grid,gfs,z,2*(degree+1));
  Dune::PDELab::adapt_grid(*gridp,gfs,z,2*(degree+1));

  // recompute constraints
  std::cout << "constraints and stuff" << std::endl;
  Dune::PDELab::constraints(b,gfs,cc);

  // write correct boundary conditions in new vector
  Z znew(gfs);
  Dune::PDELab::interpolate(g,gfs,znew);

  // copy Dirichlet boundary to interpolated solution
  Dune::PDELab::copy_constrained_dofs(cc,znew,z);
}
```

Figure 1 shows the final mesh and the corresponding solution $u_h$. The discretization error is almost completely restricted to the elements that touch the reentrant corner in the point $(0,0)$ and therefore hard to visualize.

 Figure 1: Solution on the finest mesh and underlying elements. The colormap for the solution uses discrete intervals to make the large gradients in the reentrant corner visible. Refinement has mainly occurred in this area, as can be seen on the right.

### Loop Explanation

Information about the refinement level and dimension of the current finite element space is printed at the beginning of each iteration.

Then the local operator and global operator of the nonlinear Poisson equation are created. The local operator is identical to the one of tutorial 01. It does not store information about the mesh and could therefore also be created outside of the loop. The global operator,
however, has to take the discretization into account and therefore has to be updated after each iteration.

The next few lines construct a linear solver backend and an instance of the Newton solver and solve the nonlinear problem on the current mesh.

These lines construct the error estimator and local error indicators. Each of these indicators is a single value $\gamma_T$ that is associated with one of the elements $T$, and we use a finite element map `P0FEM` for piecewise constant functions to
store these values. The function space does not need additional constraints, similar to the finite volume space of tutorial 02, and therefore the `NoConstraints` class is passed to the grid function space. Then a local operator for the error estimator is
constructed. The parameter class of the PDE is passed as a template argument, since the estimator needs access to the problem definition. These components are then used to construct a grid operator of type `ESTGO` which is a representation of the
element-wise computation of the error estimate.
```c++
// set up error estimator
typedef Dune::PDELab::P0LocalFiniteElementMap<DF,RF,dim> P0FEM;
P0FEM p0fem(Dune::GeometryTypes::simplex(dim));
typedef Dune::PDELab::NoConstraints NCON;
typedef Dune::PDELab::GridFunctionSpace<GV,P0FEM,NCON,VBE> P0GFS;
P0GFS p0gfs(gv,p0fem);
typedef NonlinearPoissonFEMEstimator<Problem<RF>,FEM> ESTLOP;
ESTLOP estlop(problem);
typedef Dune::PDELab::EmptyTransformation NCC;
typedef Dune::PDELab::GridOperator<
  GFS,P0GFS,    /* one value per element */
  ESTLOP,       /* operator for error estimate */
  MBE,RF,RF,RF, /* same as before */
  NCC,NCC       /* no constraints */
  > ESTGO;
ESTGO estgo(gfs,p0gfs,estlop,mbe);
```

A vector `z0` for the local estimates is created, and the error estimate is computed as the residual of the estimation grid operator. This way, the infrastructure of the assembler for the residual form of the PDE can be reused for this task. The result is
a vector containing the squared values of the local error estimates. The norm of the estimated error is then printed to provide feedback about the refinement process.
```c++
    // compute local error contribution and global error
    using Z0 = Dune::PDELab::Backend::Vector<P0GFS,RF>;
    Z0 z0(p0gfs,0.0);
    estgo.residual(z,z0);
    auto estimated_error = sqrt(z0.one_norm());
    std::cout << "Estimated error in step " << i
      << " is " << estimated_error << std::endl;
```

A VTK output of the finite element solution and the squared error is created. This allows
careful inspection of the estimated error and how it changes in the course of the
adaptive refinement.
```c++
   // vtk output
    std::cout << "VTK output" << std::endl;
    Dune::SubsamplingVTKWriter<GV>
      vtkwriter(gv,Dune::refinementIntervals(ptree.get("output.subsampling",(int)1)));
    typedef Dune::PDELab::VTKGridFunctionAdapter<ZDGF> VTKF;
    vtkwriter.addVertexData( // the FE solution
        std::shared_ptr<VTKF>(new VTKF(zdgf,"fesol")));
    // output squares of the local estimator
    typedef Dune::PDELab::DiscreteGridFunction<P0GFS,Z0> Z0DGF;
    Z0DGF z0dgf(p0gfs,z0);
    typedef Dune::PDELab::VTKGridFunctionAdapter<Z0DGF> VTKF0;
    vtkwriter.addCellData(
        std::shared_ptr<VTKF0>(new VTKF0(z0dgf,"gammaT^2")));
    // now write the file
    vtkwriter.write(ptree.get("output.filename",
          (std::string)"output")+iter, Dune::VTK::appendedraw);
```

If the norm of the estimated error is below the desired tolerance, then the finite element
solution can be accepted. Additional refinement is unnecessary and the loop is exited. This
also happens if the last iteration is reached, since the modified mesh would not be
used for further computations.
```c++
    // error control
    auto tol = ptree.get("fem.tol",(double)0.0);
    if (estimated_error<=tol) break;
    if (i==steps-1) break;
```

If the norm of the error estimate is still too large, then it has to be decided which of the elements should be refined and which should be kept. The fraction of elements that should be refined is read from the ini-file. The function `error_fraction` is called to
translate this fraction of elements into a threshold `eta_refine` of the value of the local indicator that separates the elements that should be refined from those that should remain the same. It is also possible to request a fraction of elements that should be coarsened, but this value is zero here for simplicity. If the chosen fraction contains all elements or the loop is still in an iteration that should refine globally, then this threshold is set to zero. Afterwards the function `mark_grid` is called and marks
all elements which have a value in `z0` that is larger than `eta_refine` to request their refinement.
```c++
    // mark elements for refinement
    std::cout << "mark elements" << std::endl;
    auto fraction = ptree.get("fem.fraction",(double)0.5);
    RF eta_refine,eta_coarsen;
    Dune::PDELab::error_fraction(
        z0,fraction,0.0,eta_refine,eta_coarsen);
    if (fraction>=1.0 || i<uniformlevel) eta_refine=0.0;
    Dune::PDELab::mark_grid(grid,z0,eta_refine,0.0,2);
```

The next line is the central line that calls `adapt_grid` and modifies the mesh. In contrast to most other functions in PDELab, this function requires the grid as an argument, and not a `GridView`. Grid views are a read only concept, and `adapt_grid` has to modify its argument. The marks on the elements are passed to the grid manager, which decides how it can best achieve the requested refinement. Depending on the capabilities and restrictions of the grid manager, the resulting mesh may not be exactly as requested.
Additional refinement may be necessary to keep the mesh conforming, for example. However, the level that is requested is a lower bound for the resulting level at each point in the domain, i.e. elements are only coarsened if requested and any marked element is refined. The function `adapt_grid` also receives the grid function space `gfs` and the solution `z` as arguments. The grid function space is updated to reflect the changes of the underlying mesh, and the solution is transferred to the new grid function space, by default
through local $L^2$ projection onto the elements of the new mesh. The last argument specifies the order of the quadrature rule that is used in this process.
```c++
    // do refinement
    std::cout << "adapt grid and solution" << std::endl;
    Dune::PDELab::adapt_grid(grid,gfs,z,2*(degree+1));

```

The function `adapt_grid` has updated the grid function space and transferred the solution to the new mesh, but the Dirichlet boundary condition requires manual intervention. The function `constraints` is called again, which updates the list of constrained degrees of freedom on the Dirichlet boundary. Then the corresponding values of the Dirichlet boundary condition are computed and copied onto the constrained degrees of freedom of `z`. This guarantees that the function `g` of Dirichlet boundary values is always represented in the most accurate way possible on the mesh. With these last steps
the iteration is finished, and the next one begins.
```c++
    // recompute constraints
    std::cout << "constraints and stuff" << std::endl;
    Dune::PDELab::constraints(b,gfs,cc);

    // write correct boundary conditions in new vector
    Z znew(gfs);
    Dune::PDELab::interpolate(g,gfs,znew);

    // copy Dirichlet boundary to interpolated solution
    Dune::PDELab::copy_constrained_dofs(cc,znew,z);
```

### Output

Then the mesh file is read and some statistics about it are reported:

```bash
Reading 2d Gmsh grid...
version 2.2 Gmsh file detected
file contains 375 nodes
file contains 754 elements
number of real vertices = 375
number of boundary elements = 74
number of elements = 674
```
Now an instance of a DUNE grid is created, the initial setup is performed, and the refinement loop is started. The problem is solved on the initial mesh:

```bash
Iteration: 0	highest level in grid: 0
constrained dofs=74 of 375
  Initial defect:   2.9656e-02
  Newton iteration  1.  New defect:   2.9206e-11.  Reduction
    (this): 9.8481e-10.  Reduction (total):   9.8481e-10
```

Due to the missing reaction term $q$ the problem is actually linear, and therefore it is solved after just one Newton iteration. The program runs the error estimator and reports an estimate for the error $\|u - u_h\|$ using the $L^2$ norm:

```bash
Estimated error in step 0 is 0.285625
```

Then the program states that it performs the remaining tasks of the first iteration:

```bash
VTK output
mark elements
adapt grid and solution
Updating entity set
constraints and stuff
```

The current solution $u_h$ and the squared local error estimates are written out to a VTK file. Then the elements are marked for refinement based on the chosen fraction of elements and error estimate, the grid
manager modifies the grid, and the solution $u_h$ and the grid function space are updated to reflect these changes. This includes a reevaluation of the constraints and the values of the Dirichlet boundary condition.
Then the solution vector can be used as an initial guess for the next iteration.

The loop runs for several iterations and produces similar output in each step. The reported error estimates are:

```bash
Estimated error in step 0 is 0.285625
Estimated error in step 1 is 0.209272
Estimated error in step 2 is 0.145672
Estimated error in step 3 is 0.09893
Estimated error in step 4 is 0.0672581
Estimated error in step 5 is 0.0446733
```

The program also reports the number of elements after each refinement:

```bash
constrained dofs=74 of 375
constrained dofs=78 of 419
constrained dofs=90 of 647
constrained dofs=117 of 1409
constrained dofs=171 of 2957
constrained dofs=244 of 6804
```

These numbers can be collected in a table containing the refinement level, the estimated error, the experimental order of convergence (EOC) for the error, the number of elements, and the number of elements a globally refined mesh would have:

    <col width="50px" />
    <col width="50px" />
    <col width="50px" />
    <col width="50px" />

     level
     $\gamma(u_h)$
     $\#T$
     $\#T_\text{global}$

     0
     0.286
     375
     375

     1
     0.209
     419
     1.5e3

     2
     0.146
      647
     6.0e3

     3
     0.099
     1409
     2.4e4

     4
     0.067
     2957
     9.6e4

     5
     0.045
     6804
     3.8e5

## Classes `Problem` and `NonlinearPoissonFEM`

These two classes are nearly identical to those of tutorial 01 and are therefore skipped. The only differences are the definitions of the right hand side and Dirichlet boundary condition. The example application of the tutorial solves the Laplace equation for a known reference solution, and therefore we set $f = 0$.
```c++
  template<typename E, typename X>
  Number f (const E& e, const X& x) const
  {
    return 0.0;
  }
```

The value of the Dirichlet boundary condition is the restriction of the reference solution to the boundary, which is given by
\begin{equation*}
  u (r, \theta) = r^{2/3} \cdot \sin\left(\frac{2}{3} \theta\right)
\end{equation*}
in polar coordinates.

```c++
 //! Dirichlet extension
  template<typename E, typename X>
  Number g (const E& e, const X& xlocal) const
  {
    auto x = e.geometry().global(xlocal);
    double theta = std::atan2(x[1],x[0]);
    if(theta < 0.0) theta += 2*M_PI;
    auto r = x.two_norm();
    return pow(r,2.0/3.0)*std::sin(theta*2.0/3.0);
  }
```

## Local Operator `NonlinearPoissonFEMEstimator`

The class `NonlinearPoissonFEMEstimator` implements the element-wise computations of the error estimate introduced in [Section 3](#AdaptiveGridRef). The computation of the error estimate is then implemented as a grid operator that returns the square of the estimated error as its residual. The contributions $R_T$ for the elements are provided by an `alpha_volume` term, while the contributions $R_F$ for the faces are provided by `alpha_skeleton` and `alpha_boundary` terms.

The definition of class `NonlinearPoissonFEMEstimator` starts as follows:
```c++
template<typename Param, typename FEM>
class NonlinearPoissonFEMEstimator
  : public Dune::PDELab::LocalOperatorDefaultFlags
```

The class is parametrized by a parameter class and a finite element map, just as the `NonlinearPoissonFEM` class. The parameter class is the same in both cases and provides access to the problem definition. The finite element map is expected to provide element-wise constant functions that match the geometry of the mesh elements, since the calculated error estimate contains one value per
element. The class does not contain methods for the Jacobian and matrix-free Jacobian evaluation, as only the residual has to be assembled for the error estimator. Therefore, only the `LocalOperatorDefaultFlags` are inherited.

Just as the class `NonlinearPoissonFEM`, the local operator contains
three private data members, a cache for evaluation of the basis functions on the reference element:

```c++
  typedef typename FEM::Traits::FiniteElementType
    ::Traits::LocalBasisType LocalBasis;
  Dune::PDELab::LocalBasisCache<LocalBasis> cache;
```
a reference to the parameter object:
```c++
  Param& param;
```
and an integer value controlling the order of the formulas used for numerical quadrature:
```c++
  int incrementorder;
```

The class has an additional private method `diameter`:

```c++
template<class GEO>
  typename GEO::ctype diameter (const GEO& geo) const
  {
    typedef typename GEO::ctype DF;
    DF hmax = -1.0E00;
    for (int i=0; i<geo.corners(); i++)
      {
        auto xi = geo.corner(i);
        for (int j=i+1; j<geo.corners(); j++)
          {
            auto xj = geo.corner(j);
            xj -= xi;
            hmax = std::max(hmax,xj.two_norm());
          }
      }
    return hmax;
  }
```

This function iterates over all pairs of corners of a given entity and defines the maximum distance among these pairs as its diameter. This information is required to scale the element contributions and face contributions of the error estimate with the local mesh width.

The public part of the class again starts with the definition of the flags controlling the generic assembly process. The `doPatternVolume` and `doPatternSkeleton` flags are both `false` to indicate that the sparsity pattern of the Jacobian can be skipped for this local operator:

```c++
  enum { doPatternVolume = false };
  enum { doPatternSkeleton = false };
```

The residual assembly flags indicate that in this local operator we will provide the methods `lambda_volume`, `lambda_boundary`
and `alpha_volume`:

```c++
  enum { doAlphaVolume  = true };
  enum { doAlphaSkeleton  = true };
  enum { doAlphaBoundary  = true };
```

Next comes the constructor taking as an argument a reference to a
parameter object and the optional increment of the quadrature order:

```c++
NonlinearPoissonFEMEstimator (Param& param_, int incrementorder_=0)
    : param(param_), incrementorder(incrementorder_)
```

### Method `alpha_volume`

This method assembles the error estimate contribution $R_T$ for a single element $T$.
Its interface is
```c++
  template<typename EG, typename LFSU, typename X,
    typename LFSV, typename R>
  void alpha_volume (const EG& eg, const LFSU& lfsu,
      const X& x, const LFSV& lfsv, R& r) const
```
This method is similar to the `alpha_volume` method of
`NonlinearPoissonFEM`, but assembles a different function.
The method starts by extracting the floating point type to be used for computations:
```c++
typedef decltype(makeZeroBasisFieldValue(lfsu)) RF;
```
Then a quadrature rule is selected
```c++
auto geo = eg.geometry();
const int order =
  incrementorder+2*lfsu.finiteElement().localBasis().order();
auto rule = Dune::PDELab::quadratureRule(geo,order);
```
a variable `sum` to collect the contributions is created, and the quadrature loop is started
```c++
RF sum(0.0);
for (const auto& ip : rule)
  {
```
Within the quadrature loop the basis functions are evaluated
```c++
    // evaluate basis functions
    auto& phihat = cache.evaluateFunction(
      ip.position(),lfsu.finiteElement().localBasis());
```
and the value of $u_h$ at the quadrature point is computed.
```c++
  // evaluate u
    RF u=0.0;
    for (size_t i=0; i<lfsu.size(); i++) u += x(lfsu,i)*phihat[i];
```
We neglect contributions containing the gradient of $u_h$, since they would require second-order derivatives of the basis functions. This means we do not need to evaluate the gradients of the basis functions.

The right hand side $f$ and the nonlinear term $q(u_h)$ are evaluated:
```c++
    // evaluate reaction term
    auto q = param.q(u);

    // evaluate right hand side parameter function
    auto f = param.f(eg.entity(),ip.position());
```
and the local error estimate for this quadrature point is calculated:
```c++
  // integrate f^2
    RF factor =
      ip.weight() * geo.integrationElement(ip.position());
    sum += (f-q)*(f-q)*factor;
  }
```
After summing up all local contributions, we are in the position to finally compute the estimate on the element by multiplying with $h_T^2$:
```c++
// accumulate cell indicator
  auto h_T = diameter(eg.geometry());
  r.accumulate(lfsv,0,h_T*h_T*sum);
}
```

### Metthod `alpha_skeleton`

This method has a structure that is very similar to the skeleton terms of the finite volume method from tutorial 02. It implements the error estimate contributions $R_F$ for the interior faces of the mesh and has the following interface:

```c++
template<typename IG, typename LFSU, typename X,
  typename LFSV, typename R>
void alpha_skeleton (const IG& ig,
      const LFSU& lfsu_i, const X& x_i, const LFSV& lfsv_i,
      const LFSU& lfsu_o, const X& x_o, const LFSV& lfsv_o,
      R& r_i, R& r_o) const
```

s in tutorial 02, the arguments comprise an intersection, local trial function and local test space for both elements adjacent to the intersection and containers for the local residual contributions in both elements.
The subscripts `_i` and `_o` correspond to
``inside'' and ``outside''. W.r.t. our notation above ``inside'' corresponds to``-'' and ``outside'' corresponds to ``+''.

As already noted in the context of the finite volume method of tutorial 02, the `alpha_skeleton` method needs to assemble contributions for `both` elements next to the intersection.

It starts by extracting the geometries of the intersection relative to the two elements adjacent to the intersection
```c++
// geometries in local coordinates of the elements
auto insidegeo = ig.geometryInInside();
auto outsidegeo = ig.geometryInOutside();
```
the two elements themselves
```c++
// inside and outside cells
auto cell_inside = ig.inside();
auto cell_outside = ig.outside();
```
and their geometries
```c++
// geometries from local to global in elements
auto geo_i = cell_inside.geometry();
auto geo_o = cell_outside.geometry();
```
Then the dimension of the intersection is extracted and a quadrature rule is created:
```c++
// dimensions
const int dim = IG::Entity::dimension;

// select quadrature rule
auto globalgeo = ig.geometry();
const int order =
  incrementorder+2*lfsu_i.finiteElement().localBasis().order();
auto rule = Dune::PDELab::quadratureRule(globalgeo,order);
```

The quadrature rule integrates over the intersection and collects contributions as the rule in `alpha_skeleton` did:
```c++
// loop over quadrature points and integrate normal flux
typedef decltype(makeZeroBasisFieldValue(lfsu_i)) RF;
RF sum(0.0);
for (const auto& ip : rule)
  {
```
The coordinates on the intersection are translated to coordinates in the two elements to make the evaluation of $u_h$ and its gradient possible, and the unit vector on the intersection pointing from $T^-$ to $T^+$ is obtained.
```c++
    // position of quadrature point in local coordinates of elements
    auto iplocal_i = insidegeo.global(ip.position());
    auto iplocal_o = outsidegeo.global(ip.position());

    // unit outer normal direction
    auto n_F = ig.unitOuterNormal(ip.position());
```
The gradients of the basis functions are evaluated on the reference element, mapped onto the actual element, and used to calculate $(\nabla u_h) \cdot \nu$ for the element $T_F^-$. Afterwards the same steps are performed for $T_F^+$, which is not repeated here.

```c++
    // gradient in normal direction in self
    auto& gradphihat_i = cache.evaluateJacobian(
        iplocal_i,lfsu_i.finiteElement().localBasis());
    const auto S_i = geo_i.jacobianInverseTransposed(iplocal_i);
    RF gradun_i = 0.0;
    for (size_t i=0; i<lfsu_i.size(); i++)
      {
        Dune::FieldVector<RF,dim> v;
        S_i.mv(gradphihat_i[i][0],v);
        gradun_i += x_i(lfsu_i,i)*(v*n_F);
      }
```
The face residual $R_F$ is calculated as the difference between the two directional derivatives of $u_h$ across the face, and its square is added to the sum.
```c++
    // integrate
    RF factor =
      ip.weight()*globalgeo.integrationElement(ip.position());
    RF jump = gradun_i-gradun_o;
    sum += jump*jump*factor;
  }
```
After summing up all local contributions, the result is multiplied with $\frac{1}{2} h_T$ and added to the estimate on the element:
```c++
    // accumulate indicator
  auto h_T = diameter(globalgeo);
  r_i.accumulate(lfsv_i,0,0.5*h_T*sum);
  r_o.accumulate(lfsv_o,0,0.5*h_T*sum);
}
```

### Method `alpha_boundary`

The method `alpha_boundary` is largely identical with `alpha_skeleton` and therefore isn't repeated here. All calculations
concerning $T_F^+$ are dropped, and instead the Neumann boundary condition value $j$ is used in the calculation of the jump:

```c++
// Neumann boundary condition value
auto j = param.j(ig.intersection(),ip.position());

// integrate
RF factor =
  ip.weight()*globalgeo.integrationElement(ip.position());
RF jump = gradun_i+j;
sum += jump*jump*factor;
```
Additionally, the final accumulation scales with $h_T$, and not with
$\frac{1}{2} h_T$ as the `alpha_skeleton` method.

```c++
// accumulate indicator
auto h_T = diameter(globalgeo);
r_i.accumulate(lfsv_i,0,h_T*sum);
```

# Exercises

## Playing with the Parameters

The adaptive finite element algorithm is controlled by several   parameters which can be set in the ini-file.  Besides the polynomial degree these are:

 - `steps`: how many times the solve-- adapt cycle is executed.
 - `uniformlevel` : how many times uniform mesh refinement is used before adaptive refinement starts.
 - `tol` : tolerance where the adaptive algorithm is stopped.
 - `fraction` : Remember that the global error is estimated from element-wise contributions:
    $$\gamma^2(u_h) = \sum_{T\in \mathcal{T}_h} \gamma_T^2(u_h).$$ The fraction parameter controls which elements are marked for   refinement. More precisely, the largest threshold $\gamma^\ast$ is   determined such that
    $$\sum_{\{T\in\mathcal{T}_h \,:\, \gamma_T(u_h)\geq\gamma^\ast\}}
    \gamma_T^2(u_h) \geq \text{fraction} \cdot
    \gamma^2(u_h). $$

If `fraction` is set to zero no elements need to be refined, i.e. $\gamma^\ast=\infty$.  When`fraction` is set to one then all elements need to be refined, i.e. $\gamma^\ast=0$.  A smaller `fraction` parameter results in a more optimized mesh but needs more steps and thus might be more expensive. If `fraction` is chosen too  large then almost all elements are refined which might also be expensive, so there exists an (unknown) optimal value for `fraction`.

Carry out the following experiments:

1. Choose polynomial degree 1 and fix a tolerance, e.g. `tol`=0.01.  Set `steps` to a large number so that the algorithm is not stopped before the tolerance is reached. Set `uniformlevel` to zero. Now vary the `fraction` parameter and measure the execution time with

```bash
$ time ./exercise05
```

2. Repeat the previous experiment with polynomial degrees two and
three. Compare the optimal meshes generated for the different polynomial degrees.
3. The idea of the heuristic refinement algorithm refining only   the elements with the largest contribution to the error is to    *equilibrate* the error. To check how well this is achieved use the calculator filter in ParaView (use cell data) to plot  $\log(\gamma_T(u_h))$. Note that the cell data exported by the   (unchanged) code is $\gamma_T^2(u_h)$! Now check the minimum and   maximum error contributions. What happens directly at the singularity, i.e. the point $(0,0)$?

- *times for `degree` = 1, `tol` = 0.01, `steps` = 200 and `uniformlevel` = 0*

| fraction|execution time|
|--------:|-------------:|
|      0.2|   38s |
|      0.4|   24s |
|      0.6|   19s |
|      0.8|   17s |
|      0.9|   21s |

- *times for same setup as before, but `degree` = 2*
| fraction|execution time|
|--------:|-------------:|
|      0.2|   1.25 s |
|      0.4|   0.53 s |
|      0.6|   0.47 s |
|      0.8|   0.38 s |
|      0.9|   0.45 s |

- *times for same setup as before, but `degree` = 3*
| fraction|execution time|
|--------:|-------------:|
|      0.2|    1.81 s |
|      0.4|    1.12 s |
|      0.6|    0.71 s |
|      0.8|    0.62 s |
|      0.9|    0.60 s |

- *examples for optimal meshes for polynomial degrees 1,2,3 (fraction = 0.7)*

- *example for $log(\gamma(u_h))$ for degree = 3*

## Compute the True $L_2$-Error

*Disclaimer: In this exercise we compute and evalute the $L_2$-norm of the error. The residual based error estimator implemented in the code, however, estimates the $H^1$-seminorm (which is equivalent to the $H^1$-norm here) and thus also optimizes the mesh with respect to that norm.  This is done to make the exercise as simple as possible. It would be more appropriate to carry out the experiments either with the true $H^1$-norm or to change the estimator to the $L_2$-norm.*

To do this exercise we need some more background on PDELab grid functions. The following code known from several tutorials by now constructs a PDELab grid function object `g`:

```c++
Problem<RF> problem(eta);
auto glambda =
[&](const auto& e, const auto& x){return problem.g(e,x);};
auto g = Dune::PDELab::makeGridFunctionFromCallable(gv,glambda);
```
Also the following code segment used in many tutorials constructs a PDELab grid function object `zdgf` from a grid function space and a coefficient vector:

```c++
typedef Dune::PDELab::DiscreteGridFunction<GFS,Z> ZDGF;
ZDGF zdgf(gfs,z);
```

PDELab grid functions can be evaluated in local coordinates given  by an element and a local coordinate within the corresponding   reference element. The result is stored in an additional argument   passed by reference.  Here is a code segment doing the job:
```c++
Dune::FieldVector<RF,1> truesolution(0.0);
g.evaluate(e,x,truesolution);
```
Here we assume that `e` is an element and `x` is a local coordinate.  The result is stored in am`Dune::FieldVector` with one component of type `RF`. This allows also to return vector-valued results.

Now here are your tasks:
1. Extend the code in the notebook to provide a grid function that provides the error $u-u_h$. Note that the method `g` in the parameter class already returns the true solution (for $\eta=0$).  You can solve this task by writing a lambda function subtracting the values returned by `g.evaluate` and `zdgf.evaluate`.

- *solution*

```c++
// =========================
// compute true L2 error
// =========================
typedef Dune::PDELab::DiscreteGridFunction<GFS,Z> ZDGF;
ZDGF zdgf(gfs,z); // the FE function as a grid function; moved from below
auto errorlambda = // subtract exact solution from FE solution
  [&](const auto& e, const auto& x){
  Dune::FieldVector<RF,1> fesolution(0.0);
  zdgf.evaluate(e,x,fesolution);
  Dune::FieldVector<RF,1> truesolution(0.0);
  g.evaluate(e,x,truesolution);
  return truesolution[0]-fesolution[0];
};
auto error = Dune::PDELab::makeGridFunctionFromCallable(gv,errorlambda); // make grid function
```

2. Compute the $L_2$-norm of the error, i.e. $\|u-u_h\|_0 = \sqrt{\int_\Omega (u(x)-u_h(x))^2\,dx}$. This can be done by creating a grid function with the squared error using    `Dune::PDELab::SqrGridFunctionAdapter` from    `<dune/pdelab/function/sqr.hh>` and the function `Dune::PDELab::integrateGridFunction` contained in the header file   `<dune/pdelab/common/functionutilities.hh>`. Output the result to the console by writing in a single line a tag like `L2ERROR`, the number of degrees of freedom using `gfs.globalSize()` and the $L_2$-norm. This lets you grep the results later for post-processing.

- *solution*
```c++
typedef Dune::PDELab::SqrGridFunctionAdapter<decltype(error)> SQRERROR;  // square of the error
SQRERROR sqrerror(error);
Dune::FieldVector<RF,1> integral(0.0);
Dune::PDELab::integrateGridFunction(sqrerror,integral,2*degree); // integrate grid function
std::cout << "L2ERROR " << gfs.globalSize() << " " << sqrt(integral[0]) << std::endl;
```

3. Investigate the $L_2$-error for different polynomial degrees. Run the code using different polynomial degrees and also uniform and adaptive (use your optimal `fraction`) refinement. Use `gnuplot` to produce graphs such as those in Figure 2. An example gnuplot file `plot.gp` is given.

  Figure 2: Comparison of $L_2$-error for different polynomial degrees and adaptive and   uniform meshes.

 The graph shows that for uniform refinement polynomial degree 2 is better than polynomial degree 1 only by a   constant factor. For adaptive refinement a better convergence rate can be achieved. The plot also shows that for adaptive refinement the optimal convergence rate can be recovered. For $P_1$ finite elements and fully regular solution in $H^2(\Omega)$ the a-priori estimate for uniform refinement yields $\|u-u_h\|_0\leq C h^2$. Expressing $h$ in terms of the number of degrees of freedom $N$ yields $h\sim N^{-1/d}$, so
$\|u-u_h\|_0\leq C h^2 \leq C' N^{-1}$ for $d=2$. The curve $N^{-1}$ is shown for comparison in the plot. Clearly, the result for $P_1$ is parallel to that line.

4. Optional: Also write the true error to the VTK output file.

# Bibliography

[1] Ainsworth M. and J.T. Oden. *A posteriori error estimation in finite element analysis*. Wiley. 2000.

[2] P. Bastian. *Lecture Notes on Scientific Computing with Partial Differential Equations*. 2014. http://conan.iwr.uni-heidelberg.de/teaching/numerik2_ss2014/num2.pdf.

[3] Braess, D. *Finite Elemente*. Springer. 3rd edition. 2003.

[4] Brenner, S. C. and Scott, L. R. *The mathematical theory of finite element methods*. Springer. 1994.

[5] Ciarlet, P.G. *The finite element method for elliptic problems*. SIAM. Classics in Applied Mathematics. 2002.

[6] Elman, H., Silvester, D. and Wathen, A. *Finite Elements and Fast Iterative Solvers*. Oxford University Press. 2005.

[7] Eriksson, D., Estep, Hansbo,P. and Johnson, C. *Computational Differential Equations*. Cambridge University Press. 1996. http://www.csc.kth.se/~jjan/private/cde.pdf.

[8] Ern, A. and Guermond, J.-L. *Theory and practice of finite element methods*. SIAM. Classics in Applied Mathematics. 2002.

[9] Geuzaine, C. and Remacle, J.-F. *Gmsh: A 3-D finite element mesh generator with built-in pre- and post-processing facilities*. International Journal for Numerical Methods in Engineering. 2009.

[10] Großmann, C. and Roos, H.-G. *Numerische Behandlung partieller Differentialgleichungen*. Teubner. 2006.

[11] Hackbusch, W. *Theorie und Numerik elliptischer Differentialgleichungen*. Teubner. 1986. http://www.mis.mpg.de/preprints/ln/lecturenote-2805.pdf.

[12] Rannacher, R.  *Einführung in die Numerische Mathematik II (Numerik partieller Differentialgleichungen)*. 2006. http://numerik.iwr.uni-heidelberg.de/~lehre/notes


# Neuromatch Academy: Week 1, Day 3, Tutorial 2
# Model Fitting: Linear regression with MLE

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil  with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Madineh Sarvestani, Patrick Mineault, Ella Batty, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 2 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will use a different approach to fit linear models that incorporates the random 'noise' in our data.
- Learn about probability distributions and probabilistic models
- Learn how to calculate the likelihood of our model parameters
- Learn how to implement the maximum likelihood estimator, to find the model parameter with the maximum likelihood

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions
def plot_density_image(x, y, theta, sigma=1, ax=None):
  """ Plots probability distribution of y given x, theta, and sigma

  Args:

    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta (float): Slope parameter
    sigma (float): standard deviation of Gaussian noise

  """

  # plot the probability density of p(y|x,theta)
  if ax is None:
    fig, ax = plt.subplots()

  xmin, xmax = np.floor(np.min(x)), np.ceil(np.max(x))
  ymin, ymax = np.floor(np.min(y)), np.ceil(np.max(y))
  xx = np.linspace(xmin, xmax, 50)
  yy = np.linspace(ymin, ymax, 50)

  surface = np.zeros((len(yy), len(xx)))
  for i, x_i in enumerate(xx):
    surface[:, i] = stats.norm(theta * x_i, sigma).pdf(yy)

  ax.set(xlabel='x', ylabel='y')

  return ax.imshow(surface, origin='lower', aspect='auto', vmin=0, vmax=None,
            cmap=plt.get_cmap('Wistia'),
            extent=[xmin, xmax, ymin, ymax])

def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter.

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """
  theta_hat = (x.T @ y) / (x.T @ x)
  return theta_hat
```

---
# Section 1: Maximum Likelihood Estimation (MLE)

```python
#@title Video 1: Maximum Likelihood Estimation
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="8mpNmzLKNfU", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=8mpNmzLKNfU

In the previous tutorial we made the assumption that the data was drawn from a linear relationship with noise added, and found an effective approach for estimating model parameters based on minimizing the mean squared error.

In that case we treated the noise as simply a nuisance, but what if we factored it directly into our model?

Recall our linear model:

\begin{align}
y = \theta x + \epsilon.
\end{align}

The noise component $\epsilon$ is often modeled as a random variable drawn from a Gaussian distribution (also called the normal distribution).

The Gaussian distribution is described by its [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (pdf)
\begin{align}
\mathcal{N}(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(x-\mu)^2}
\end{align}

and is dependent on two parameters: the mean $\mu$ and the variance $\sigma^2$. We often consider the noise signal to be Gaussian "white noise", with zero mean and unit variance:

\begin{align}
\epsilon \sim \mathcal{N}(0, 1).
\end{align}

## Interactive Demo: Gaussian Distribution Explorer

Use the explorer widget below to see how varying the $\mu$ and $\sigma$ parameters change the location and shape of the samples.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(mu=widgets.FloatSlider(0.0, min=-2.0, max=2.0),
                  sigma=widgets.FloatSlider(1.0, min=0.5, max=2.0))
def plot_normal_dist(mu=0, sigma=1):

  # Generate pdf & samples from normal distribution with mu/sigma
  rv = stats.norm(mu, sigma)
  x = np.linspace(-5, 5, 100)
  y = rv.pdf(x)
  samples = rv.rvs(1000)

  # Plot
  fig, ax = plt.subplots()
  ax.hist(samples, 20, density=True, color='g', histtype='stepfilled', alpha=0.8,
          label='histogram')
  ax.plot(x, y, color='orange', linewidth=3, label='pdf')
  ax.vlines(mu, 0, rv.pdf(mu), color='y', linewidth=3, label='$\mu$')
  ax.vlines([mu-sigma, mu+sigma], 0, rv.pdf([mu-sigma, mu+sigma]), colors='red',
            color='b', linewidth=3, label='$\sigma$')
  ax.set(xlabel='x', ylabel='probability density', xlim=[-5, 5], ylim=[0, 1.0])
  ax.legend()
```

## Section 1.1: Probabilistic Models

Now that we have a model of our noise component $\epsilon$ as random variable, how do we incorporate this back into our original linear model from before? Consider again our simplified model $y = \theta x + \epsilon$ where the noise has zero mean and unit variance $\epsilon \sim \mathcal{N}(0, 1)$. We can now also treat $y$ as a random variable drawn from a Gaussian distribution where $\mu = \theta x$ and $\sigma^2 = 1$:

\begin{align}
y \sim \mathcal{N}(\theta x, 1)
\end{align}

which is to say that the probability of observing $y$ given $x$ and parameter $\theta$ is
\begin{align}
p(y|x,\theta) = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(y-\theta x)^2}
\end{align}

---

Let's revisit our original sample dataset where the true underlying model has $\theta = 1.2$.

```python
# @title

# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence

np.random.seed(121)
theta = 1.2
n_samples = 30
x = 10 * np.random.rand(n_samples) # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples) # sample from a standard normal distribution
y = theta * x + noise
```

This time we can plot the density of $p(y|x,\theta=1.2)$ and see how $p(y)$ changes for different values of $x$.

```python
#@title
#@markdown Execute this cell to visualize p(y|x, theta=1.2)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))

# Invokes helper function to generate density image plots from data and parameters
im = plot_density_image(x, y, 1.2, ax=ax1)
plt.colorbar(im, ax=ax1)
ax1.axvline(8, color='k')
ax1.set(title=r'p(y | x, $\theta$=1.2)')

# Plot pdf for given x
ylim = ax1.get_ylim()
yy = np.linspace(ylim[0], ylim[1], 50)
ax2.plot(yy, stats.norm(theta * 8, 1).pdf(yy), color='orange', linewidth=2)
ax2.set(
    title=r'p(y|x=8, $\theta$=1.2)',
    xlabel='y',
    ylabel='probability density');
```

## Section 1.2: Likelihood Estimation

Now that we have our probabilistic model, we turn back to our original challenge of finding a good estimate for $\theta$ that fits our data. Given the inherent uncertainty when dealing in probabilities, we talk about the [likelihood](https://en.wikipedia.org/wiki/Likelihood_function) that some estimate $\hat \theta$ fits our data. The likelihood function $\mathcal{L(\theta)}$ is equal to the probability density function parameterized by that $\theta$:

\begin{align}
\mathcal{L}(\theta|x,y) = p(y|x,\theta) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(y-\theta x)^2}
\end{align}

### Exercise 1: Likelihood Function

In this exercise you will implement the likelihood function $\mathcal{L}(\theta|x,y)$ for our linear model where $\sigma = 1$.

After implementing this function, we can produce probabilities that our estimate $\hat{\theta}$ generated the provided observations. We will try with one of the samples from our dataset.

TIP: Use `np.exp` and `np.sqrt` for the exponential and square root functions, respectively.

```python
def likelihood(theta_hat, x, y):
  """The likelihood function for a linear model with noise sampled from a
    Gaussian distribution with zero mean and unit variance.

  Args:
    theta_hat (float): An estimate of the slope parameter.
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    ndarray: the likelihood values for the theta_hat estimate
  """
  sigma = 1

  # Compute Gaussian likelihood
  pdf = stats.norm.pdf(y, theta_hat * x, sigma)

  return pdf

# Uncomment below to test your function
print(likelihood(1.0, x[1], y[1]))
```

    0.11344443599846923

We should see that $\mathcal{L}(\theta=1.0|x=2.1,y=3.7) \approx 0.11$. So far so good, but how does this tell us how this estimate is better than any others?

When dealing with a set of data points, as we are with our dataset, we are concerned with their joint probability -- the likelihood that all data points are explained by our parameterization. Since we have assumed that the noise affects each output independently, we can factorize the likelihood, and write:

\begin{align}
\mathcal{L}(\theta|X,Y) = \prod_{i=1}^N \mathcal{L}(\theta|x_i,y_i),
\end{align}

where we have $N$ data points $X = \{x_1,...,x_N\}$ and $Y = \{y_1,...,y_N\}$.

In practice, such a product can be numerically unstable. Indeed multiplying small values together can lead to [underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow), the situation in which the digital representation of floating point number reaches its limit. This problem can be circumvented by taking the logarithm of the likelihood because the logarithm transforms products into sums:

\begin{align}
\operatorname{log}\mathcal{L}(\theta|X,Y) = \sum_{i=1}^N \operatorname{log}\mathcal{L}(\theta|x_i,y_i)
\end{align}

We can take the sum of the log of the output of our `likelihood` method applied to the full dataset to get a better idea of how different $\hat\theta$ compare. We can also plot the different distribution densities over our dataset and see how they line up qualitatively.

```python
#@title
#@markdown Execute this cell to visualize different distribution densities
theta_hats = [0.5, 1.0, 2.2]
fig, axes = plt.subplots(ncols=3, figsize=(16, 4))
for theta_hat, ax in zip(theta_hats, axes):
  ll = np.sum(np.log(likelihood(theta_hat, x, y)))  # log likelihood
  im = plot_density_image(x, y, theta_hat, ax=ax)
  ax.scatter(x, y)
  ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat}, log likelihood: {ll:.2f}')
plt.colorbar(im, ax=ax);
```

Using the log likelihood calculation, we see that $\mathcal{L}(\theta=1.0) > \mathcal{L}(\theta=0.5) > \mathcal{L}(\theta=2.2)$.

This is great: now we have a way to compare estimators based on likelihood. But like with the MSE approach, we want an analytic solution to find the best estimator. In this case, we want to find the estimator that maximizes the likelihood.

## Section 1.3: Finding the Maximum Likelihood Estimator

We want to find the parameter value $\hat\theta$ that makes our data set most likely:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \mathcal{L}(\theta|X,Y)
\end{align}

We discussed how taking the logarithm of the likelihood helps with numerical stability, the good thing is that it does so without changing the parameter value that maximizes the likelihood. Indeed, the $\textrm{log}()$ function is *monotonically increasing*, which means that it preserves the order of its inputs. So we have:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \sum_{i=1}^m \textrm{log} \mathcal{L}(\theta|x_i,y_i)
\end{align}

Now substituting our specific likelihood function and taking its logarithm, we get:
\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} [-\frac{N}{2} \operatorname{log} 2\pi\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^N (y_i-\theta x_i)^2].
\end{align}

Note that maximizing the log likelihood is the same as minimizing the negative log likelihood (in practice optimization routines are developed to solve minimization not maximization problems). Because of the convexity of this objective function, we can take the derivative of our negative log likelihhood, set it to 0, and solve - just like our solution to minimizing MSE.

\begin{align}
\frac{\partial\operatorname{log}\mathcal{L}(\theta|x,y)}{\partial\theta}=\frac{1}{\sigma^2}\sum_{i=1}^N(y_i-\theta x_i)x_i = 0
\end{align}

This looks remarkably like the equation we had to solve for the optimal MSE estimator, and, in fact, we arrive to the exact same solution!

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \hat{\theta}_{\textrm{MSE}} = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

```python
# Compute theta_hat_MLE
theta_hat_mle = (x @ y) / (x @ x)

```

```python
#@title
#@markdown Execute this cell to visualize density with theta_hat_mle

# Plot the resulting distribution density
fig, ax = plt.subplots()
ll = np.sum(np.log(likelihood(theta_hat_mle, x, y))) # log likelihood
im = plot_density_image(x, y, theta_hat_mle, ax=ax)
plt.colorbar(im, ax=ax);
ax.scatter(x, y)
ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat_mle:.2f}, log likelihood: {ll:.2f}');
```

---
# Summary

- Likelihood vs probability
    - $\mathcal{L}(\theta|x, y) = p(y|\theta, x)$
    - $p(y|\theta, x)$ -> "probability of observing the response $y$ given parameter $\theta$ and input $x$"
    - $\mathcal{L}(\theta|x, y)$ -> "likelihood model that parameters $\theta$ produced response $y$ from input $x$"
- Log-likelihood maximization
    - We take the $\textrm{log}$ of the likelihood function for computational convenience
    - The parameters $\theta$ that maximize $\textrm{log}\mathcal{L}(\theta|x, y)$ are the model parameters that maximize the probability of observing the data.
- **Key point**:
  - the log-likelihood is a flexible cost function, and is often used to find model parameters that best fit the data.

---
# Appendix

We can also see $\mathrm{p}(\mathrm{y} | \mathrm{x}, \theta)$ as a function of $x$. This is the stimulus likelihood function, and it is useful in case we want to decode the input $x$ from observed responses $y$. This is what is relevant from the point of view of a neuron that does not have access to the outside world and tries to infer what's out there from the responses of other neurons!


# Stochastic Processes:
Data Analysis and Computer Simulation

# Distribution function and random number

# 3. The central limit theorem

# 3.1. Binomial distribution $\rightarrow$ Gauss distribution

## From the previous lesson

- The binomial distribution becomes equivalent to the Gaussian distribution in the limit $n, M\gg 1$, as shown in the 1st plot of this week.

\begin{equation}
P(n)=\frac{M!}{n!(M-n)!}{{p}^{n}}{{(1-p)}^{M-n}}\tag{C6}
\end{equation}

\begin{equation}
\hspace{20mm}
\xrightarrow[n\to cont.]{n,M\gg 1}\
% P(n)=
\frac{1}{\sqrt{2\pi {{\sigma }^{2}}}}\exp \left[ -\frac{{{(n-{{\mu }_{1}})}^{2}}}{2{{\sigma }^{2}}} \right]\tag{C1}
\end{equation}

\begin{equation}
\hspace{20mm}
\mu_1=Mp,\ \ \ \
\sigma^2=Mp(1-p) \tag{C7, C8}
\end{equation}

## Numerical experiment 1

- While the proof for the equivalence has been given in the supplemental note, let us examine this by performing numerical experiments for various values of $M=1,2,4,10,100$ and $1000$.

### Include libraries

```python
% matplotlib inline
import numpy as np # import numpy library as np
import math # use mathematical functions defined by the C standard
import matplotlib.pyplot as plt # import pyplot library as plt
plt.style.use('ggplot') # use "ggplot" style for graphs
```

```python
p = 0.5    # set p, propability to obtain "head" from a coin toss
M = 50      # set M, number of tosses in one experiment
N = 100000 # number of experiments
ave = M*p
std = np.sqrt(M*p*(1-p))
print('p =',p,'M =',M)
np.random.seed(0) # initialize the random number generator with seed=0
X = np.random.binomial(M,p,N) # generate the number of head come up N times store it as X
nmin=np.int(ave-std*5)
nmax=np.int(ave+std*5)
nbin=nmax-nmin+1
plt.hist(X,range=[nmin,nmax],bins=nbin,normed=True) # plot normalized histgram of R
x = np.arange(nmin,nmax,0.01/std) # create array of x from nmin to nmax with increment 0.01/std
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # calculate the theoretical distribution
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Gauss',r'histgram'], fontsize=16) # set legends
plt.show() # display plots
```

## What we can learn from the experiment

- Stochastic variable "$s$" is a result of single binary choice,

\begin{equation}
s=0\ {\rm or}\ 1
%\tag{D1}
\end{equation}

and Stochastic variable "$n^{M}$" is a sum of $M$ independent binary choices $s$, with the index $j$ representing the $j$-th choice.

\begin{equation}
%\color{red}
{
n^{M}=\sum^{M}_{j=1}s_j}
%\tag{D4}
\end{equation}

### For $M=1$

\begin{equation}
n^{M=1}=s_1=s=0\ {\rm or}\ 1
\tag{D1}
\end{equation}
- Distribution function $\rightarrow$ Binary choice,
$P^{M=1}(0)=1-p, \ P^{M=1}(1)=p, $ with

\begin{equation}
\hspace{15mm}\mu^{M=1}_1=p, \hspace{23mm} \sigma_{M=1}^2=p(1-p) \tag{D2, D3}
\end{equation}

### For $M\gg 1$

\begin{equation}
\color{red}{
n^{M}=\sum^{M}_{j=1}s_j=\sum^{M}_{j=1}n^{M=1}_j}  \tag{D4}
\end{equation}
- Distribution function $\rightarrow$ Gaussian with

\begin{equation}
%\hspace{20mm}
\color{red}{
\mu^{M\gg1}_1=M\mu^{M=1}_1,\hspace{20mm} \sigma_{M\gg1}^2=M\sigma_{M=1}^2} \tag{D5, D6}
\end{equation}

# 3.2. The central limiting theorem (CLT)

## Generalization of Eqs. (D4-D6) for $M\gg 1$

### CLT for sum of stochastic variables

- Stochastic variable "$n^M$" as a SUM of any $M$ independent stochastic variables $n^{M=1}$ with $\mu_1^{M=1}$ and $\sigma_{M=1}^2$,

\begin{equation}
n^{M}=\sum^{M}_{j=1}n^{M=1}_j \tag{D7}
\end{equation}
- Distribution function $\rightarrow$ Gauss with

\begin{equation}
%\hspace{20mm}
{
\mu^{M\gg1}_1=M\mu^{M=1}_1,\hspace{20mm} \sigma_{M\gg1}^2=M\sigma_{M=1}^2} \tag{D8, D9}
\end{equation}

### CLT for average of stochastic variables

- Stochastic variable "$n^M$" as an AVERAGE of any $M$ independent stochastic variables with $\mu^{M=1}$ and $\sigma_{M=1}^2$,

\begin{equation}
n^{M}=\frac{1}{M}\sum^{M}_{j=1}n^{M=1}_j \tag{D10}
\end{equation}
- Distribution function $\rightarrow$ Gauss with

\begin{equation}
%\hspace{20mm}
{
\mu^{M\gg1}_1=\mu^{M=1}_1,\hspace{20mm} \sigma_{M\gg1}^2=\frac{\sigma_{M=1}^2}{M}} \tag{D11, D12}
\end{equation}

- Eqs. (D7-D12) is called "the central limiting theorem".

# 3.3. Uniform distribution $\rightarrow$ Gauss distribution

## From CLT

### For $M=1$

- Stochastic variable "$x$" is uniformly distributed between 0 and 1,

\begin{equation}
x^{M=1}\in[0:1]
\tag{D13}
\end{equation}
- Distribution function:
$P^{M=1}(x)=1 \ ({\rm for\ } 0\le x < 1),\ P^{M=1}(x)=0\ {\rm (otherwise)}$

\begin{equation}
\hspace{15mm}\mu^{M=1}_1=\frac{1}{2}, \hspace{23mm} \sigma_{M=1}^2=\frac{1}{12} \tag{D14, D15}
\end{equation}

### For $M\gg 1$

- Stochastic variable "$x$" is a sum of $M$ independent uniform random numbers

\begin{equation}
{
x^{M}=\sum^{M}_{j=1}x^{M=1}_j} \tag{D16}
\end{equation}
- Distribution function $\rightarrow$ Gauss with

\begin{equation}
%\hspace{20mm}
{
\mu^{M\gg1}_1=M\mu^{M=1}_1=\frac{M}{2},\hspace{20mm} \sigma_{M\gg1}^2=M\sigma_{M=1}^2}=\frac{M}{12} \tag{D17, D18}
\end{equation}

## Numerical experiment 2

```python
M = 10    # set M, the number of random variables to add
N = 100000 # number samples to draw, for each of the random variables
ave = M/2
std = np.sqrt(M/12)
print('M =',M)
np.random.seed(0)        # initialize the random number generator with seed=0
X = np.zeros(N)
for i in range(N):
    X[i] += np.sum(np.random.rand(M)) # draw a random numbers for each of the x_i and add to get new value for x = \sum x_i
nmin=np.int(ave-std*5)
nmax=np.int(ave+std*5)
plt.hist(X,range=[nmin,nmax],bins=50,normed=True) # plot normalized histgram of R
x = np.arange(nmin,nmax,0.01/std) # create array of x from nmin to nmax with increment 0.01/std
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # calculate the theoretical distribution
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$x$',fontsize=16) # set x-label
plt.ylabel(r'$P(x)$',fontsize=16) # set y-label
plt.legend([r'Gauss',r'histgram'], fontsize=16) # set legends
plt.show() # display plots
```