# Tutorial 01: Conforming Finite Element Method for a Nonlinear Poisson Equation

In this tutorial we extend tutorial 00 in the following ways:

1. Solve a nonlinear stationary partial differential equation (PDE).
2. Use conforming finite element spaces of arbitrary order.
3. Use different types of (conforming) meshes (simplicial, cubed and mixed).
4. Use multiple types of boundary conditions.

Combined with the fact that the implementation works in any dimension (note: it is not claimed to be efficient in high dimension $d>3$) this comprises already a relatively large space of different methods, so the example illustrates the flexibility of PDELab. Moreover, the finite element method developed in this tutorial will serve as a building block for instationary problems, adaptive mesh refinement and
parallel solution in subsequent tutorials. This tutorial depends on tutorial 00 which discusses piecewise linear elements on simplicial elements. It is assumed that you have worked through tutorial 00 before.

## Problem Formulation

Here we consider the following nonlinear Poisson equation with
Dirichlet and Neumann boundary conditions:

\begin{align}\label{eq:ProblemStrong}
-\Delta u + q(u) &= f &&\text{in $\Omega$}, \\
u &= g &&\text{on $\Gamma_D\subseteq\partial\Omega$},\\
-\nabla u\cdot \nu &= j &&\text{on $\Gamma_N=\partial\Omega\setminus\Gamma_D$}.
\end{align}

$\Omega\subset\mathbb{R}^d$ is a domain, $q:\mathbb{R}\to\mathbb{R}$ is a given, possibly
nonlinear function and $f: \Omega\to\mathbb{R}$ is the source term and
$\nu$ denotes the unit outer normal to the domain.

The weak formulation of this problem is derived by multiplication with an appropriate
test function and integrating by parts. This results in the abstract problem:
\begin{equation}
\text{Find $u\in U$ s.t.:} \quad r^{\text{NLP}}(u,v)=0 \quad \forall v\in V,
\label{Eq:BasicBuildingBlock}
\end{equation}

with the continuous residual form
\begin{equation}
r^{\text{NLP}}(u,v) = \int_\Omega \nabla u \cdot \nabla v + (q(u)-f)v\,dx + \int_{\Gamma_N} jv\,ds
\label{eq:ResidualForm}
\end{equation}
and the function spaces
$U= \{v\in H^1(\Omega) \,:\, \text{''$v=g$'' on $\Gamma_D$}\}$
and $V= \{v\in H^1(\Omega) \,:\, \text{''$v=0$'' on $\Gamma_D$}\}$.
We assume that $q$ is such that this problem has a unique solution.

## Realization in PDELab

The structure of the code is very similar to that of tutorial 00. Again, all the Dune sources are included through one convenience header:

```c++
#include <dune/jupyter.hh>
#include "nonlinearpoissonfem.hh"
//#include "nitschenonlinearpoissonfem.hh"
#include "problem.hh"
```

As for the previous tutorial, a number of runtime paramters is provided in a configuration file `tutorial01.ini`, which can be found in the `notebooks/tutorial01` directory. Again the paramters are parsed into a `ParamterTree`.

```c++
Dune::ParameterTree ptree;
Dune::ParameterTreeParser ptreeparser;
ptreeparser.readINITree("tutorial01.ini",ptree);
```

Ini File not that important for jupyter notebooks

**First, a DUNE grid object is instantiated**

The dimension is set and the refinement is read from the ini file.

```c++
// read ini file
const int dim = 2;
const int refinement = ptree.get<int>("grid.refinement");
```

The first step is again, to instantiate a grid. We use again a two-dimensional, unstructured mesh for this simulation, as known from the previous tutorial. Again, the mesh resolves the unit square $\Omega = [0,1]^2$. The file is specified in the `unitsquare.msh` file that was generated using Gmsh.

```c++
using Grid = Dune::UGGrid<dim>;
std::string filename = ptree.get("grid.twod.filename",
                                 "unitsquare.msh");
Dune::GridFactory<Grid> factory;
Dune::GmshReader<Grid>::read(factory,filename,true,true);
std::unique_ptr<Grid> gridp(factory.createGrid());
Dune::Timer timer;
gridp->globalRefine(refinement);
std::cout << "Time for mesh refinement " << timer.elapsed()
          << " seconds" << std::endl;
using GV = Grid::LeafGridView;
using DF = Grid::ctype;
GV gv = gridp->leafGridView();
```

*Using yasp grid instead of uggrid. Note, that additionally another finite element map needs to be selected.*

```c++
//solution to task 1.2 using the structured grid factory
/*using Grid = Dune::YaspGrid<dim>;
using DF = Grid::ctype;

Dune::FieldVector<DF,dim> lowerleft(0.0);
Dune::FieldVector<DF,dim> upperright(1.0);

auto N  = Dune::filledArray<dim, unsigned int>(2);

auto grid = Dune::StructuredGridFactory<Grid>::createCubeGrid(lowerleft, upperright, N);
using GV = Grid::LeafGridView;
GV gv = grid->leafGridView();
*/
```

```c++
gridp
```

Here, a polynomial degree can bee chosen:

```c++
const int degree = 1;
```

```c++
//ug grid
using FEM = Dune::PDELab::PkLocalFiniteElementMap<GV,DF,double,degree>;
```

*Finite element map for yasp grid*

```c++
//yasp
//using FEM = Dune::PDELab::QkLocalFiniteElementMap<GV,DF,double,degree>;
```

```c++
FEM fem(gv);
```

### Driver

**The driver instantiates the necessary PDELab classes for solving a nonlinear stationary problem and finally solves the problem**

```c++
// type for calculations
using RF = double;
```

The interface of the parameter class is defined by the implementor of the local operator and is not part of PDELab. As shown in tutorial 00 it is perfectly possible to have a local operator without a parameter class.
The following code segment instantiates the problem class which is called `Problem` here (it is explained in detail below)

```c++
// make PDE parameter class
RF eta = ptree.get("problem.eta",(RF)1.0);
Problem<RF> problem(eta);
```

```c++
eta
```

Now there are two places where information from the PDE is used in PDELab. First of all we need to have an object that can be used as an argument to `Dune::PDELab::interpolate` to initialize a vector which represents the initial guess and the Dirichlet boundary conditions.
The class `Problem` defines a method which we need to use to define a class with the interface of `Dune::PDELab::GridFunction`.
This is accomplished by the following code:

```c++
auto g = Dune::PDELab::makeGridFunctionFromCallable(
    gv,
    [&](const auto& e, const auto& x) {
        return problem.g(e,x);
    }
);;
```

Similarly, we need an object that can be passed to `Dune::PDELab::constraints` to fill a constraints container which is used to build a subspace of a function space. Again, the class `Problem` defines such a method which is extracted with a lambda function:

```c++
auto b = Dune::PDELab::makeBoundaryConditionFromCallable(
    gv,
    [&](const auto& i, const auto& x){
        return problem.b(i,x);
    }
);;
```

The next step is to define the grid function space. This is
exactly the same code as in tutorial 00.

```c++
// Make grid function space
//== Exercise 2 {
using CON = Dune::PDELab::ConformingDirichletConstraints;
// using CON = Dune::PDELab::NoConstraints;
//== }
using VBE = Dune::PDELab::ISTL::VectorBackend<>;
using GFS = Dune::PDELab::GridFunctionSpace<GV,FEM,CON,VBE>;
GFS gfs(gv,fem);
gfs.name("Vh");
```

Now comes unchanged code to assemble the constraints, instantiate a coefficient vector, making a discrete grid function that can be used for visualization and interpolating the initial guess and Dirichlet boundary conditions:

```c++
// Assemble constraints
//== Exercise 2 {
using CC = typename GFS::template
ConstraintsContainer<RF>::Type;
CC cc;
Dune::PDELab::constraints(b,gfs,cc); // assemble constraints
std::cout << "constrained dofs=" << cc.size() << " of "
        << gfs.globalSize() << std::endl;
// using CC = Dune::PDELab::EmptyTransformation;
//== }

// A coefficient vector
using Z = Dune::PDELab::Backend::Vector<GFS,RF>;
Z z(gfs); // initial value

// Make a grid function out of it
using ZDGF = Dune::PDELab::DiscreteGridFunction<GFS,Z>;
ZDGF zdgf(gfs,z);

// Fill the coefficient vector
Dune::PDELab::interpolate(g,gfs,z);
```

The next step is to instantiate a local operator, called `NonlinearPoissonFEM`, containing the implementation of the element-wise computations of the finite element method. As explained above the local operator is parametrized by the class
`Problem`. In addition, also the finite element map is passed as a template parameter for reasons that will become clear below:

```c++
// Make a local operator
//== Exercise 2 {
using LOP = NonlinearPoissonFEM<Problem<RF>,FEM> ;
LOP lop(problem);
// RF stab = ptree.get("fem.stab",(RF)1);
// using LOP = NitscheNonlinearPoissonFEM<Problem<RF>,FEM> ;
// LOP lop(problem,stab);
//== }
```

Now the grid function space, local operator, matrix backend and
constraints container are used to set up a grid operator facilitating the global residual assembly, Jacobian assembly and matrix-free Jacobian application. The matrix backend is initialized with a guess of the approximate number
of nonzero matrix entries per row.

```c++
// Make a global operator
using MBE = Dune::PDELab::ISTL::BCRSMatrixBackend<>;
MBE mbe((int)pow(1+2*degree,dim));
typedef Dune::PDELab::GridOperator<
GFS,GFS,  /* ansatz and test space */
LOP,      /* local operator */
MBE,      /* matrix backend */
RF,RF,RF, /* domain, range, jacobian field type*/
CC,CC     /* constraints for ansatz and test space */
> GO;
//== Exercise 2 {
GO go(gfs,cc,gfs,cc,lop,mbe);
// GO go(gfs,gfs,lop,mbe);
//== }
```

In order to prepare for the solution process an appropriate linear solver needs to be selected:

```c++
using LS = Dune::PDELab::ISTLBackend_SEQ_CG_AMG_SSOR<GO> ;
LS ls(100,2);
```

Since the problem is nonlinear we use the implementation of
Newton's method in PDELab. It provides the inexact Newton method
in the sense that the iterative solution of the linear subproblems is stopped early and uses line search as globalization strategy:

```c++
Dune::PDELab::Newton<GO,LS,Z> newton(go,z,ls);
newton.setReassembleThreshold(0.0); // always reassemble J
newton.setVerbosityLevel(3);        // be verbose
newton.setReduction(1e-10);         // total reduction
newton.setMinLinearReduction(1e-4); // min. red. in lin. solve
newton.setMaxIterations(25);        // limit number of its
newton.setLineSearchMaxIterations(10); // limit line search
```

Now, finally do all the work and solve the problem:

```c++
newton.apply();
```

At the end we can write the VTK file with subsampling:

```c++
//write filename to .txt file
std::string filename=ptree.get("output.filename","output");
std::ofstream out("name.txt");
out << filename;
out.close();

// Write VTK output file
int subsampling = ptree.get("output.subsampling",degree);
Dune::SubsamplingVTKWriter<GV> vtkwriter(gv,Dune::refinementIntervals(subsampling));
using VTKF = Dune::PDELab::VTKGridFunctionAdapter<ZDGF> ;
vtkwriter.addVertexData(std::shared_ptr<VTKF>(new VTKF(zdgf,"fesol")));
```

*Visualize exact solution*

```c++
//solution to exercise 2.1.3
/*Z w(gfs);
Dune::PDELab::interpolate(g,gfs,w); // Lagrange interpolation of exact solution
ZDGF wdgf(gfs,w);*/
```

```c++
//solution to exercise 2.1.3
/*vtkwriter.addVertexData(std::shared_ptr<VTKF>(new VTKF(wdgf,"exact")));*/
```

The visualization data can be generated in Jupyter by printing the VTKWriter instance:

```c++
vtkwriter
```

### Problem.hh

Note that the following cells are note executable. Changes can be made in the file `problem.hh` which is provided in the same folder as this notebook.

The class `Problem` contained in the file `problem.hh` provides all parameter functions for the PDE problem. It is parameterized with the floating point type to be used:

```c++
template<typename Number>
class Problem
```

Its constructor takes a parameter $\eta$ as argument:

```c++
Problem (const Number& eta_) : eta(eta_) {}
```

Now come the parameter functions defining the PDE problem.
First is the nonlinearity $q(u)$:

```c++
Number q (Number u) const
{
  return eta*u*u;
}
```

We also provide the derivative of the function $q$ as a seperate method:

```c++
Number qprime (Number u) const
{
  return 2*eta*u;
}
```

This allows the implementation of an exact Jacobian later (illustrated in tutorial 02) and is actually not needed here as we will use a numerical Jacobian.

Next is the right hand side function $f$ which gets an element `e` and a local coordinate `x` within
the corresponding reference element as a parameter:

```c++
template<typename E, typename X>
Number f (const E& e, const X& x) const
{
  return -2.0*x.size();
}
```

The argument `x` can be expected to be an instance of`Dune::FieldVector` which has a method `size`
giving the number of components of the vector, i.e. the space dimension.

The next method simply called `b` is the boundary condition type function. It should return true if the position given by intersection `i` and a local coordinate `x` within the reference element of the intersection is on the Dirichlet boundary.
In the particular instance here we set $\Gamma_D=\partial\Omega$:

```c++
template<typename I, typename X>
bool b (const I& i, const X& x) const
{
  return true;
}
```

The value of the Dirichlet boundary condition is now defined by the method `g`. As explained above it
is more appropriate to provide a function $u_g$ that can be evaluated
on $\overline{\Omega}$ and gives the value of $g$ on the Dirichlet boundary
and the initial guess for the nonlinear solver on all other points:

```c++
template<typename E, typename X>
Number g (const E& e, const X& x) const
{
  auto global = e.geometry().global(x);
  Number s=0.0;
  for (std::size_t i=0; i<global.size(); i++) s+=global[i]*global[i];
  return s;
}
```

As with the function `f` above, the arguments are an element and a local coordinate
in its reference element. Here we evaluate it as $u_g(e,x) = \|\mu_e(x)\|^2$.

Finally, there is a method defining the value of the Neumann boundary condition.
Although there is no Neumann boundary here, the method has to be provided but is
never called. The arguments of the method are the same as for the boundary
condition type function `b`:

```c++
template<typename I, typename X>
Number j (const I& i, const X& x) const
{
  return 0.0;
}
```

### Local Operator

We now turn to how the residual can be evaluated in practice. The residual form \eqref{eq:ResidualForm} can be readily decomposed into elementwise contributions:
\begin{equation}
r^{\text{NLP}}\left(u,v\right) =
\sum_{T\in\mathcal{T}_h} \alpha_T^V(u,v)
  + \sum_{T\in\mathcal{T}_h} \lambda_T^V(v)
 + \sum_{F\in\mathcal{F}_h^{\partial\Omega}}\lambda_F^B(v)
\end{equation}
with
\begin{align}
\alpha_T^V(u,v) &= \int_T \nabla u \cdot \nabla v + q(u) v \,dx, &
\lambda_T^V(v) &= - \int_T f v \,dx, &
\lambda_F^B(v) &= \int_{F\cap\Gamma_N} j v\,ds.
\end{align}

Here $\mathcal{F}_h^{\partial\Omega}$ is the set of intersections of elements with the domain boundary $\partial\Omega$.
The element-wise computations can be classified on the one hand as volume integrals (superscript $V$), boundary integrals (superscript $B$) and skeleton integrals (superscript $S$, to be shown later) and on the
other hand as integrals depending on trial and test functions ($\alpha$-terms) and integrals depending only on test functions ($\lambda$-terms). Here we need three of these six possible combinations.

The three terms can now be evaluated using the techniques introduced in tutorial 00 with the small extension that for general maps $\mu_T$ we have
$$\nabla w(\mu_T(\hat x)) = J_{\mu_T}^{-T}(\hat x) \hat\nabla \hat w (\hat x)$$
with $J_{\mu_T}(\hat x)$ the Jacobian of $\mu_T$ at point $\hat x$.

For a linear map  $\mu_T(\hat{x}) = B_T\hat{x} + a_t $ the Jacobian is simply given by $J_T = B_T$.

The class `NonlinearPoissonFEM` implements the element-wise computations of the finite element method. Evaluation of the residual $R(z)$ is accomplished
by the three types of contributions shown in equation \eqref{eq:FinalResidualEvaluation}.
In order to make things as simple as possible we chose to implement the evaluation of the Jacobian and the matrix-free Jacobian application with finite differences.

The definition of class `NonlinearPoissonFEM` starts as follows:

```c++
template<typename Param, typename FEM>
class NonlinearPoissonFEM :
  public Dune::PDELab::
    NumericalJacobianVolume<NonlinearPoissonFEM<Param,FEM> >,
  public Dune::PDELab::
    NumericalJacobianApplyVolume<NonlinearPoissonFEM<Param,FEM> >,
  public Dune::PDELab::FullVolumePattern,
  public Dune::PDELab::LocalOperatorDefaultFlags
```

The class is parametrized by a parameter class and a finite element map.
Implementation of element-wise contributions to the Jacobian and matrix-free
Jacobian evaluation is achieved through inheriting from the
classes `NumericalJacobianVolume` and `NumericalJacobianApplyVolume`.
Using the *curiously recurring template pattern* these classes provide
the corresponding methods without any additional coding effort
based on the `alpha_volume` method explained below.
The other two base classes are the same as in tutorial 00.

The private data members are a cache for evaluation of the basis functions on the reference element:

```c++
typedef typename FEM::Traits::FiniteElementType::
     Traits::LocalBasisType LocalBasis;
  Dune::PDELab::LocalBasisCache<LocalBasis> cache;
```

a reference to the parameter object:

```c++
Param& param;
```

and an integer value controlling the order of the formulas used for numerical quadrature:

```c++
int incrementorder;
```

The public part of the class starts with the definition of the flags controlling
the generic assembly process. The `doPatternVolume` flag specifies that the sparsity pattern of the Jacobian is determined by couplings between degrees of freedom associated with single elements. The corresponding
default pattern assembly method is inherited from the class `FullVolumePattern`:

```c++
enum { doPatternVolume = true };
```

The residual assembly flags indicate that in this local operator we will provide the methods `lambda_volume`, `lambda_boundary` and `alpha_volume`:

```c++
enum { doLambdaVolume = true };
enum { doLambdaBoundary = true };
enum { doAlphaVolume = true };
```

Next comes the constructor taking as an argument a reference to a parameter object and the optional increment of the quadrature order:

```c++
NonlinearPoissonFEM (Param& param_, int incrementorder_=0)
    : param(param_), incrementorder(incrementorder_)
  {}
```

#### Method `lambda_volume`

For any $(T,m)\in C(i)$ we obtain
\begin{equation*}
\begin{split}
\lambda_T^V(\phi_i) &= - \int_T f \phi_i \,dx =
- \int_{\hat T} f(\mu_T(\hat x)) p_m^{\hat T}(\hat x) |\text{det} J_{\mu_T}(\hat x)|\, d\hat x .
\end{split}
\end{equation*}
This integral on the reference element is then computed by employing numerical integration of appropriate order.  $\hat{p}^{\hat{T}}$ are the local functions on the reference element,the global Lagrange basis functions spanning $V_h^{k,d}(\mathcal{T}_h)$ are then defined by
\begin{equation*}
\phi_i(x) = \left\{\begin{array}{ll}
p^{\hat T}_m(\mu_T^{-1}(x)) & x\in T \wedge (T,m)\in C(i) \\
0 & \text{else}
\end{array}\right. , \quad i\in\mathcal{I}\left(V_h^{k,d}(\mathcal{T}_h)\right) .
\end{equation*}

The evaluation for all test functions with support on element $T$ may be collected in a vector
\begin{equation*}
(\mathcal{L}_T^V)_m = - \int_{\hat T} f(\mu_T(\hat x)) p_m^{\hat T}(\hat x)
|\text{det} J_{\mu_T}(\hat x)|\, d\hat x.
\end{equation*}

This method was also present in the local operator `PoissonP1` in tutorial00. It implements the term $\mathcal{L}_T^V$ and has the interface:

```c++
//! right hand side integral
template<typename EG, typename LFSV, typename R>
void lambda_volume (const EG& eg, const LFSV& lfsv, R& r) const
```

The implementation here uses numerical quadrature of sufficiently high order
which is selected at the beginning of the method:

```c++
auto geo = eg.geometry();
const int order = incrementorder + 2*lfsv.finiteElement().localBasis().order();
auto rule = Dune::PDELab::quadratureRule(geo,order);
```

The DUNE quadrature rules provide a container of quadrature points that can be iterated over:

```c++
for (const auto& ip : rule)
      {
        // evaluate basis functions
        auto& phihat = cache.evaluateFunction(ip.position(), lfsv.finiteElement().localBasis());

        // integrate -f*phi_i
        decltype(ip.weight()) factor = ip.weight()*
          geo.integrationElement(ip.position());
        auto f=param.f(eg.entity(),ip.position());
        for (size_t i=0; i<lfsv.size(); i++)
          r.accumulate(lfsv,i,-f*phihat[i]*factor);
      }
```

At each quadrature point all basis functions are evaluated. The local function space argument `lfsv` provides all the basis functions on the reference element. Evaluations are cached for each point as the evaluation
may be quite costly, especially for high order. In addition, copying of data is avoided as the cache returns only a reference to the data stored in the cache. The integration factor is the product of the weight of the quadrature point and the value of $|\text{det} J_{\mu_T}(\hat x)|$. The implementation works also
for non-affine element transformation. The quadrature order should be increased by providing a value for `incrementorder` in the constructor.
Then the parameter function can be evaluated and finally the residual contributions for each test function are stored in the result object `r`.

#### Method `lambda_boundary`

For $F\in\mathcal{F}_h^{\partial\Omega}$ with $F\cap\Gamma_N\neq\emptyset$ and $(T_F^-,m)\in C(i)$ we obtain
\begin{equation*}
\begin{split}
\lambda_T^B(\phi_i) &= \int_{F} j v\,ds =
\int_{\hat F} j(\mu_F(s)) p_m^{\hat T}(\eta_F(s))
\sqrt{|\text{det} (J^T_{\mu_F}(s)J_{\mu_F}(s))|} \,ds
\end{split}
\end{equation*}
Because integration is over a face of codimension 1 now, two mappings are involved. The map $\mu_F$ maps the reference element $\hat F$ of $F$ into global coordinates while the map $\eta_F$ maps $\hat F$ into the reference element $\hat T$ of $T$. Also the integration element has to be redefined accordingly.
Again, all contributions of the face $F$ can be collected in a vector:
\begin{equation*}
(\mathcal{L}_T^B)_m =
\int_{\hat F} j(\mu_F(s)) p_m^{\hat T}(\eta_F(s))
\sqrt{|\text{det} J^T_{\mu_T}(s)J_{\mu_T}(s)|} \,ds .
\end{equation*}

The `lambda_boundary` implements the residual contributions due to Neumann boundary conditions.
It implements the term $\mathcal{L}_T^B$ and has the following interface:

```c++
template<typename IG, typename LFSV, typename R>
void lambda_boundary (const IG& ig, const LFSV& lfsv, R& r) const
 ```

The difference to `lambda_volume` is that now an intersection is provided as first argument.

The method begins by evaluating the type of the boundary condition at the midpoint of the edge:

```c++
auto localgeo = ig.geometryInInside();
auto facecenterlocal = referenceElement(localgeo).position(0,0);
bool isdirichlet = param.b(ig.intersection(),facecenterlocal);
```

To that end the center of the reference element of the intersection is computed in the variable`facecenterlocal` before the parameter function can be called.

If the boundary condition type evaluated at the face center is Dirichlet then the complete face is assumed to be part of the Dirichlet boundary:

```c++
// skip rest if we are on Dirichlet boundary
if (isdirichlet) return;
```

It is thus assumed that the mesh resolves all positions where the boundary type changes.

Now that we are on a Neumann boundary an appropriate quadrature rule is selected for integration:

```c++
auto globalgeo = ig.geometry();
const int order = incrementorder + 2*lfsv.finiteElement().localBasis().order();
auto rule = Dune::PDELab::quadratureRule(globalgeo,order);
```

And here is the integral over the face:

```c++
// loop over quadrature points and integrate normal flux
for (const auto& ip : rule)
{
  // quadrature point in local coordinates of element
  auto local = localgeo.global(ip.position());

  // evaluate shape functions (assume Galerkin method)
  auto& phihat = cache.evaluateFunction(local,lfsv.finiteElement().localBasis());

  // integrate j
  decltype(ip.weight()) factor = ip.weight()*globalgeo.integrationElement(ip.position());
  auto j = param.j(ig.intersection(),ip.position());
  for (size_t i=0; i<lfsv.size(); i++)
    r.accumulate(lfsv,i,j*phihat[i]*factor);
}
```

Every quadrature point on the face needs to be mapped to the reference of the volume element for evaluation of the basis functions. The evaluation uses the basis function cache. Then the integration factor is computed and the contributions for all the test functions are accumulated.

#### Method `alpha_volume`

For any $(T,m)\in C(i)$ we get
\begin{equation*}
\begin{split}
\alpha_T^V(u_h,\phi_i) &= \int_T \nabla u \cdot \nabla \phi_i + q(u) \phi_i \,dx,
= \int_T \sum_j (z)_j \left(\nabla \phi_j \cdot \nabla \phi_i \right)
+ q\left( \sum_j (z)_j \phi_j \right) \phi_i \,dx,\\
&= \int_{\hat T} \sum_{n} (z)_{g_T(n)} (J_{\mu_T}^{-T}(\hat x) \hat\nabla p_n^{\hat T}(\hat x) )
\cdot (J_{\mu_T}^{-T}(\hat x) \hat\nabla p_m^{\hat T}(\hat x) ) \\
&\hspace{40mm}+ q\left( \sum_n (z)_{g_T(n)} p_n^{\hat T}(\hat x) \right) p_m^{\hat T}(\hat x)
|\text{det} J_{\mu_T}(\hat x)| \,d\hat x
\end{split}
\end{equation*}
Again contributions for all test functions can be collected in a vector
\begin{equation*}
\begin{split}
(\mathcal{R}_T^V(R_T z))_m &=
\sum_{n} (z)_{g_T(n)} \int_{\hat T} (J_{\mu_T}^{-T}(\hat x) \hat\nabla p_n^{\hat T}(\hat x) )
\cdot (J_{\mu_T}^{-T}(\hat x) \hat\nabla p_m^{\hat T}(\hat x) ) |\text{det} J_{\mu_T}(\hat x)| \,d\hat x\\
&\hspace{30mm}+ \int_{\hat T} q\left( \sum_n (z)_{g_T(n)} p_n^{\hat T}(\hat x) \right) p_m^{\hat T}(\hat x)
|\text{det} J_{\mu_T}(\hat x)| \,d\hat x
\end{split}
\end{equation*}

This method was already present in tutorial00. It implements the term $\mathcal{R}_T^V(R_T z)$ and its interface is

```c++
template<typename EG, typename LFSU, typename X,
         typename LFSV, typename R>
void alpha_volume (const EG& eg, const LFSU& lfsu, const X& x,
                   const LFSV& lfsv, R& r) const
```

The method starts by extracting the space dimension and the floating point type to be used for computations:

```c++
const int dim = EG::Entity::dimension;
typedef decltype(Dune::PDELab::makeZeroBasisFieldValue(lfsu)) RF;
```

Then a quadrature rule is selected

```c++
auto geo = eg.geometry();
const int order = incrementorder + 2*lfsu.finiteElement().localBasis().order();
auto rule = Dune::PDELab::quadratureRule(geo,order);
```

and the quadrature loop is started

```c++
for (const auto& ip : rule)
  {
```

Within the quadrature loop the basis functions are evaluated

```c++
auto& phihat = cache.evaluateFunction(ip.position(),lfsu.finiteElement().localBasis());
```

and the value of $u_h$ at the quadrature point is computed.

```c++
RF u=0.0;
for (size_t i=0; i<lfsu.size(); i++)
  u += x(lfsu,i)*phihat[i];
```

Then the gradients of the basis functions on the reference element are evaluated via the evaluation cache:

```c++
auto& gradphihat = cache.evaluateJacobian(ip.position(), lfsu.finiteElement().localBasis());
```

Now the gradients need to be transformed from the reference element to the transformed element by multiplication with $J_{\mu_T}^{-1}(\hat x)$:

```c++
// transform gradients of shape functions to real element
const auto S = geo.jacobianInverseTransposed(ip.position());
auto gradphi = makeJacobianContainer(lfsu);
for (size_t i=0; i<lfsu.size(); i++)
  S.mv(gradphihat[i][0],gradphi[i][0]);
```

Note that, as explained in tutorial00, DUNE allows basis functions in general to be vector valued. Therefore `gradphi[i][0]` contains the gradient (with $d$ components) of the component 0 of basis function number $i$.

Now $\nabla u_h$ can be computed

```c++
Dune::FieldVector<RF,dim> gradu(0.0);
for (size_t i=0; i<lfsu.size(); i++)
  gradu.axpy(x(lfsu,i),gradphi[i][0]);
```

and we are in the position to finally compute the residual contributions:

```c++
// integrate (grad u)*grad phi_i + q(u)*phi_i
auto factor = ip.weight()*
  geo.integrationElement(ip.position());
auto q = param.q(u);
for (size_t i=0; i<lfsu.size(); i++)
  r.accumulate(lfsu,i,(gradu*gradphi[i][0]+
                       q*phihat[i])*factor);
}
```

Now with these definitions in place the evaluation of the algebraic residual is
\begin{equation}
R(z) =
\sum_{T\in\mathcal{T}_h} R_T^T \mathcal{R}_T^V(R_T z)
  + \sum_{T\in\mathcal{T}_h} R_T^T \mathcal{L}_T^V
 + \sum_{F\in\mathcal{F}_h^{\partial\Omega}\cap\Gamma_N} R_T^T \mathcal{L}_F^B
\label{eq:FinalResidualEvaluation}
\end{equation}

The Jacobian of the residual is
\begin{equation*}
(J(z))_{i,j} = \frac{\partial R_i}{\partial z_j} (z) =
\sum_{(T,m,n) : (T,m)\in C(i) \wedge (T,n)\in C(j)} \frac{\partial (\mathcal{R}_T^V)_m}{\partial z_n}
(R_T z)
\end{equation*}
Note that:
1. Entries of the Jacobian can be computed element by element.
2. The derivative is independent of the $\lambda$-terms as they only depend on the test functions.
3. In the implementation below the Jacobian is computed numerically by finite differences. This can be achieved automatically by deriving from an additional base class.

# Exercises

## Warming Up

1. Try out different polynomial degrees and values for $\eta$. The parameter $\eta$ can be changed in the ini file 'tutorial01.ini'. The polynomial degree can be changed from within the notebook.
Here are some suggestions that could be interesting:
 - use higher values for $\eta$
 - Try all combinations of `degree=1|2` and `subsampling=1|2`.  Look at your solutions using paraview and the `warp by scalar` filter.  You can see the underlying grid by choosing `surface with edges` instead of `surface` in the paraview drop down menu. How does subsampling change the output?

- *example for influence of nonlinearity $\eta$ on solution.
  Chosen parameters: `degree = 1` `uggrid`*

- *example for different choices of degree and subsampling for $\eta = 10$*

*The subsampling divides each cell into four additional cells, however just virtually for the visualiztation, the actual grid stays the same. As paraview visualizes either the value of a cell or the value of the vertices, a further subdivision for the visualization prevents the loss of accuracy if polynomials of higher order are employed. This is especially important, when paraview is used for the post processing of data. A higher subsampling than polynomial degree is thus not sensible and only increases the required memory.*

2. Try to use `yasp` grid instead of `ug` grid and visualize the solution with paraview. You can use the structured grid factory as presented in the grid-exercise.

- *The solution can be found within the notebook [(go to solution)](#yasp)*.

3. Try use a 3D grid implementation. For ug grid, another gmsh file needs to be provided through the ini file.

 - *change the dimension to 3 at the beginning of the notebook [(go to cell)](#dim)*. This already suffices for using a 3D yasp grid.
 - *for ug grid: use unitcube.msh instead of unitsquare.msh, i.e. change section of ini file that is read from `grid.twod` to `grid.threed`.*

```c++
using Grid = Dune::UGGrid<dim>;
std::string filename = ptree.get("grid.threed.filename",  //3D
                                 "unitcube.msh");

```

- *example 3D output*

4. It is easy to implement different nonlinearities. Use $q(u)=\exp(\eta u)$ by adjusting the file `problem.hh`.

 - *changes in 'problem.hh'*:
 ```c++
 Number q (Number u) const
  {
    // return eta*u*u;
    return std::exp(eta*u);
  }

  //! derivative of nonlinearity
  Number qprime (Number u) const
  {
    // return 2*eta*u;
    return eta*std::exp(eta*u);
  }
```

5. Go back to $q(u)=\eta u^2$.  Now we want to see how good our approximation is.  Change the function $f(x)$ in the file  problem.hh to $f(x)=-2d+\eta(\sum_{i=1}^d(x)_i^2)^2$ where $d$ is the dimension (and therefore size of $x$). Then $u(x)=\sum_{i=1}^d(x)_i^2=g(x)$ is the exact solution.  Visualize the exact solution like it is done in tutorial 00. Start with `degree = 1`, `eta = 10` and `subsampling = 1`. Use paraview to see how the maximal error $\max|u-u_h|$ behaves for different refienement levels `refinement=1|...|5`. Additionally you can calculate the L2-norm of the error, as before in tutorial 00. Then try again for `degree=2`.  What happens here?

 - *change of function f in `problem.hh`*:
 ```c++
  //! right hand side
  template<typename E, typename X>
  Number f (const E& e, const X& x) const
  {
    // return -2.0*x.size();

    // right hand side where g(x) is exact solution
    auto global = e.geometry().global(x);
    Number s=0.0;
    for (std::size_t i=0; i<global.size(); i++) s+=global[i]*global[i];

    return -2.0*x.size()+eta*s*s;
  }
```

- *The visualization for the exact solution is embedded in the notebook [(go to solution)](#exact).*

- *The following table shows the maximum norm,which is given as $||f||_{\max} := \max|f(x)|$ and the L2-norm as $||f||_{0,\Omega} = \left( \int_{\Omega} f(x)^2 dx \right)^{1/2} $ for different mesh refinements. The L2 error is depicted in the graph on the right side.*

         refinement
         ${||u-u_h||}_\max$
         ${||u-u_h||}_{0,\Omega}$

         1
         0.000784874
         0.000232161

         2
         0.000241339
         5.51530598e-05

         3
         7.14064e-05
         1.35100333e-05

         4
         2.18749e-05
         3.35459387e-06

         5
         6.61612e-06
         8.37383425e-07

- *For `degree = 2` the finite element solution $u_h$ matches the exact solution $u$*.

## Nitsche's Method for weak Dirichlet Boundary Conditions

In this exercise we want to implement Dirichlet boundary conditions in a weak sense by using Nitsche's method.  Instead of incorporating the Dirichlet boundary condition into the Ansatzspace we modify the residual:
  \begin{align*}
    r^{\text{Nitsche}}(u,v) &= \int_\Omega \nabla u \cdot \nabla v + (q(u)-f)v\,dx + \int_{\Gamma_N} jv\,ds \\
    &\quad - \int_{\Gamma_D} \nabla u \cdot\nu v\,ds - \int_{\Gamma_D} (u-g)\nabla v \cdot\nu\,ds
    + \eta_{stab} \int_{\Gamma_D} (u-g)v\,ds.
  \end{align*}
Here $\eta_{stab}$ denotes a stabilization parameter that should be equal to $\eta_{stab}=c/h$ for a constant $c>0$ large enough. This stabilization term is needed to ensure coercivity of the bilinear form.

In order to implement this method you have to do the following:

 - include the new file `nitschenonlinearpoissonfem` instead of `nonlinearpoissonfem`
 - you have to turn of the constraints. The code is already there you just have to comment/uncomment the parts marked with
```c++
//== Exercise 2 {
...
// ...
//== }
```

- By changing these lines you use no constraints, an empty constraints container and construct the grid operator without constraints.  Besides that you use the new `NitscheNonlinearPoissonFEM` local operator that expects the stabilization parameter $\eta_{stab}$.

- the key part is adding the `alpha_boundary` method to the new local operator in the file `nitschenonlinearpoissonfem.hh`. Take a close look at the `lambda_volume`, `lambda_boundary` and `alpha_volume` methods and you should be on your way.

 *Hint:* The code for generating the transformation is already there:

```c++
// transform gradients of shape functions to real element
const auto S = geo_inside.jacobianInverseTransposed(local);
```

*Note*: Remember that the quadrature points need to be mapped to the reference element of the inside cell for the evaluation of the basis functions.

- *solution for method `alpha_boundary`
The interface for the method `alpha_boundary` is basically the same as for the previously described method `alpha_volume`, only this time an intersection is provided instead of an entitiy.*
 ```c++
   //! boundary integral depending on test and ansatz functions
template<typename IG, typename LFSU, typename X,
         typename LFSV, typename R>
void alpha_boundary (const IG& ig, const LFSU& lfsu, const X& x,
                   const LFSV& lfsv, R& r) const
```

*As in `lambda_boundary` the method begins by evaluating the type of the boundary condition at the midpoint of the edge:*

```c++
{
  // evaluate boundary condition type
  auto localgeo = ig.geometryInInside();
  auto facecenterlocal =
    referenceElement(localgeo).position(0,0);
  bool isdirichlet=param.b(ig.intersection(),facecenterlocal);

  // skip rest if we are _not_ on Dirichlet boundary
  if (!isdirichlet) return;
```

*Reminder: Here, `localgeo` provides the mapping from the reference element of the intersection to the reference element of the inside element.*
*Note: In order to get the center of the reference element of the intersection both `localgeo` and `geo` can be used, as the method `referenceElement()` returns for both the reference element of the intersection.*

*Followed by the definition of usefull types and the choice of quadrature rule, as described before:*
```c++
  // types & dimension
  const int dim = IG::Entity::dimension;
  using RF = decltype(Dune::PDELab::makeZeroBasisFieldValue(lfsu));

  // select quadrature rule
  auto geo = ig.geometry();
  const int order = incrementorder+
    2*lfsu.finiteElement().localBasis().order();
  auto rule = Dune::PDELab::quadratureRule(geo,order);

  // geometry of inside cell
  auto geo_inside = ig.inside().geometry();
```

*As described for the method `lambda_volume`, every quadrature point on the face needs to be mapped to the reference element of the volume element for the evaluation of the basis functions. This is accomplished by the method `global()`. The evaluation uses the basis function cache.*

```c++
// loop over quadrature points
for (const auto& ip : rule)
{
  // quadrature point in local coordinates of element
  auto local = localgeo.global(ip.position());

  // evaluate basis functions
  auto& phihat = cache.evaluateFunction(local,
                   lfsu.finiteElement().localBasis());
```
*The evaluation of u, the gradient of shape functions and the the gradient of u is the same as described in `alpha_volume`.The function `jacobianInverseTransposed` is provided with the integration point `ip` in local coordinates `local` of the reference element of the inside cell.*
```c++
  // evaluate u
  RF u=0.0;
    for (size_t i=0; i<lfsu.size(); i++)
      u += x(lfsu,i)*phihat[i];

    // evaluate gradient of shape functions
    auto& gradphihat = cache.evaluateJacobian(local,
                         lfsu.finiteElement().localBasis());

    // transform gradients of shape functions to real element
    const auto S = geo_inside.jacobianInverseTransposed(local);
    auto gradphi = makeJacobianContainer(lfsu);
    for (size_t i=0; i<lfsu.size(); i++)
      S.mv(gradphihat[i][0],gradphi[i][0]);

    // compute gradient of u
    Dune::FieldVector<RF,dim> gradu(0.0);
    for (size_t i=0; i<lfsu.size(); i++)
      gradu.axpy(x(lfsu,i),gradphi[i][0]);
```
*Additionally, the value  of the Dirichlet BC and the unit outer normal vector are required for the computation of the integral*
```c++
    // get unit outer normal vector and g
    auto n = ig.unitOuterNormal(ip.position());
    auto g = param.g(ig.intersection(),ip.position());
```
*Finally, the integration factor is computed and the contributions for all the test functions are accumulated.
Note: `geo` maps the coordinates from the reference element of the intersection to global coordinates, so the provided argument is `ip.position` (and not local).*
```c++

   // integrate -(grad u)*n * phi_i + q(u)*phi_i
   auto factor = ip.weight()*
      geo.integrationElement(ip.position());
   for (size_t i=0; i<lfsu.size(); i++)
     r.accumulate(lfsu,i,(-(gradu*n)*phihat[i]-
                          (u-g)*(gradphi[i][0]*n)+
                          stab*(u-g)*phihat[i])*factor);
   }
}
```

When you have done all that test your implementation.  Use the test case from exercise 1 with $f(x)=-2d+\eta(\sum_{i=1}^d(x)_i^2)^2$ and exact solution $u(x)=\sum_{i=1}^d(x)_i^2=g(x)$ and compare it to your approximation.

Introduce the parameter
```ini
[fem]
stab = 100
```
in the ini file and look at the maximal error $\max|u-u_h|$ for `stab=10|100|1000.`

 - *$\eta_{stab}$ needs to be large enough to ensure coercivity. For too small $\eta_{stab}$ the error is dominated by the error on the boundaries, for sufficiently large $\eta_{stab}$ the error is distributed as before.*

- *Error for Dirchlet boundary conditions in ansatzspace and in residual.*


__Note__: This is best viewed on [NBViewer](http://nbviewer.ipython.org/github/tdhopper/notes-on-dirichlet-processes/blob/master/2015-10-14-collapsed-gibbs-sampling-for-mixture-models.ipynb). It is part of a series on [Dirichlet Processes and Nonparametric Bayes](https://github.com/tdhopper/notes-on-dirichlet-processes).

# Collapsed Gibbs Sampling for Bayesian Mixture Models (with a Nonparametric Extension)

[In an earlier notebook](http://nbviewer.ipython.org/github/tdhopper/notes-on-dirichlet-processes/blob/master/2015-09-02-fitting-a-mixture-model.ipynb), I showed how we can fit the parameters of a Bayesian mixture model using a Gibbs sampler. The sampler defines a Markov chain that, in steady state, samples from the posterior distribution of the mixture model. To move the chain forward by one step we:

* Sample the cluster assignment $z_i$.
* Sample the mixture weights $\pi$
* Sample the cluster means $\mu_n$.

It turns out that we can derive a Gibbs sampler that _just_ samples the assignments instead of the mixture weights and cluster means. This is known as a _collapsed_ Gibbs sampler. If we integrate out the cluster means $\theta_k$ and mixture weights from the margial distribution of cluster assignment
$$p(z_i=k \,|\,
        z_{\neg i}, \pi,
        \theta_1, \theta_2, \theta_3, \sigma, \mathbf{x}, \alpha
        )$$ we are left with
$$p(z_i\,|\, z_{\neg i}, \sigma, \mathbf{x}, \alpha).$$

By the conditional independence, we can factorize this marginal distribution
$$
\begin{align}
p(z_i=k\,|\, z_{\neg i}, \sigma, \mathbf{x}, \alpha)
    &\propto
    p(z_i=k\,|\, x_i, z_{\neg i}, \sigma, \mathbf{x}_{\neg i}, \alpha)\\
    &=
    p(z_i=k\,|\, z_{\neg i}, \sigma, \mathbf{x}_{\neg i}, \alpha)
    p(x_i \,|\, z, \sigma, \mathbf{x}_{\neg i}, \alpha)\\
    &=
    p(z_i=k \,|\, z_{\neg i}, \alpha) p(x_i \,|\, z, \mathbf{x}_{\neg i}, \sigma)\\
    &=
    p(z_i=k \,|\, z_{\neg i}, \alpha)p(x_i \,|\, z_i=k, z_{\neg i}, x_{\neg_i}, \sigma)\\
    &=
    p(z_i=k \,|\, z_{\neg i}, \alpha)p(x_i \,|\, \left\{x_j \,|\, z_j=k, j\neq i\right\}, \sigma).
\end{align}
$$

The two terms have intuitive explanations. $p(z_i = k \,|\, z_{\neg i}, \alpha)$ is the probability point $x_i$ will be assigned to component $k$ given the current assignments. Because we are using a symmetric Dirichlet prior, this is the predictive likelihood of a Dirichlet-categorical distribution. This is given by:
$$p(z_i=k \,|\, z_{\not i}, \alpha)=
    \frac{N_k^{-i}+\alpha / K}{N-1+\alpha}$$
 where $N_k^{-i}=\sum_{j\neq i} \delta(z_j, k)$ is the number of observation assigned to $k$ (except $x_i$). We also need to define $\bar{x}_k^{-i}$ to be the mean of all observations assigned to component $k$ (except $x_i$).

The second term is the predictive likelihood that point $x_i$ is distributed according to cluster $k$ (given the data currently in cluster $k$). For our example, we are assuming unknown cluster means are distributed according to a normal distribution with hyperparameter mean $\lambda_1$ and variance $\lambda_2^2$ and known cluster variance $\sigma^2$.

Thus,
$$
\begin{align}
p(x_i \,|\, \left\{x_j \,|\, z_j=k, j\neq i\right\}, \sigma)
    &= \mathcal{N}(x_i \,|\, \mu_k, \sigma_k^2+\sigma^2)
\end{align}
    $$
where
$$\sigma_k^2 = \left( \frac{N_k^{-i}}{\sigma^2} + \frac{1}{\lambda_2^2} \right)^{-1}$$
and
$$\mu_k = \sigma_k^2 \left(
    \frac{\lambda_1}{\lambda_2^2}+\frac{N_k^{-i}\cdot \bar{x}_k^{-i}}{\sigma^2}
\right).$$
This is derived in Kevin Murphey's fantastic article [Conjugate Bayesian analysis of the Gaussian distribution](http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf).

At each step of the collapsed sampler, we sample each $z_i$ as follows:

* For each cluster $k$, compute $$f_k(x_i) =p(x_i \,|\, \left\{x_j \,|\, x_j=k, j\neq i\right\}, \lambda).$$ This is the predictive probability that $x_i$ is in cluster $k$ given the data currently assigned to that cluster.
* Sample $$z_i\sim \frac{1}{Z_i}\sum_{k=1}^K(N_k^{-i}+\alpha/K)f_k(x_i)\delta(z_i,k)$$

where the normalizing constant is $Z_i=\sum_{k=1}^K(N_k^{-i}+\alpha/K)f_k(x_i)$.

Let's write code for this Gibbs sampler!

```python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from collections import namedtuple, Counter
from scipy import stats
from numpy import random
```

```python
np.random.seed(12345)
```

First, load the same dataset we used [previously](http://nbviewer.ipython.org/github/tdhopper/notes-on-dirichlet-processes/blob/master/2015-09-02-fitting-a-mixture-model.ipynb):

```python
data = pd.Series.from_csv("clusters.csv")
_=data.hist(bins=20)
```

Again, we want to define a state object and a function for updating the sufficient statistics of the state.

```python
SuffStat = namedtuple('SuffStat', 'theta N')

def initial_state(num_clusters=3, alpha=1.0):
    cluster_ids = range(num_clusters)

    state = {
        'cluster_ids_': cluster_ids,
        'data_': data,
        'num_clusters_': num_clusters,
        'cluster_variance_': .01,
        'alpha_': alpha,
        'hyperparameters_': {
            "mean": 0,
            "variance": 1,
        },
        'suffstats': {cid: None for cid in cluster_ids},
        'assignment': [random.choice(cluster_ids) for _ in data],
        'pi': {cid: alpha / num_clusters for cid in cluster_ids},
    }
    update_suffstats(state)
    return state

def update_suffstats(state):
    for cluster_id, N in Counter(state['assignment']).iteritems():
        points_in_cluster = [x
            for x, cid in zip(state['data_'], state['assignment'])
            if cid == cluster_id
        ]
        mean = np.array(points_in_cluster).mean()

        state['suffstats'][cluster_id] = SuffStat(mean, N)
```

Next we define functions to compute the two terms of our marginal distribution over cluster assignments (as we derived above).

```python
def log_predictive_likelihood(data_id, cluster_id, state):
    """Predictive likelihood of the data at data_id is generated
    by cluster_id given the currenbt state.

    From Section 2.4 of
    http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf
    """
    ss = state['suffstats'][cluster_id]
    hp_mean = state['hyperparameters_']['mean']
    hp_var = state['hyperparameters_']['variance']
    param_var = state['cluster_variance_']
    x = state['data_'][data_id]
    return _log_predictive_likelihood(ss, hp_mean, hp_var, param_var, x)

def _log_predictive_likelihood(ss, hp_mean, hp_var, param_var, x):
    posterior_sigma2 = 1 / (ss.N * 1. / param_var + 1. / hp_var)
    predictive_mu = posterior_sigma2 * (hp_mean * 1. / hp_var + ss.N * ss.theta * 1. / param_var)
    predictive_sigma2 = param_var + posterior_sigma2
    predictive_sd = np.sqrt(predictive_sigma2)
    return stats.norm(predictive_mu, predictive_sd).logpdf(x)

def log_cluster_assign_score(cluster_id, state):
    """Log-likelihood that a new point generated will
    be assigned to cluster_id given the current state.
    """
    current_cluster_size = state['suffstats'][cluster_id].N
    num_clusters = state['num_clusters_']
    alpha = state['alpha_']
    return np.log(current_cluster_size + alpha * 1. / num_clusters)
```

Given these two functions, we can compute the posterior probability distribution for assignment of a given datapoint. This is the core of our collapsed Gibbs sampler.

To simplify the computation of things like $N_k^{-i}$ (where we remove point $i$ from the summary statistics), we create two simple functions to add and remove a point from the summary statistics for a given cluster.

```python
def cluster_assignment_distribution(data_id, state):
    """Compute the marginal distribution of cluster assignment
    for each cluster.
    """
    scores = {}
    for cid in state['suffstats'].keys():
        scores[cid] = log_predictive_likelihood(data_id, cid, state)
        scores[cid] += log_cluster_assign_score(cid, state)
    scores = {cid: np.exp(score) for cid, score in scores.iteritems()}
    normalization = 1.0/sum(scores.values())
    scores = {cid: score*normalization for cid, score in scores.iteritems()}
    return scores

def add_datapoint_to_suffstats(x, ss):
    """Add datapoint to sufficient stats for normal component
    """
    return SuffStat((ss.theta*(ss.N)+x)/(ss.N+1), ss.N+1)

def remove_datapoint_from_suffstats(x, ss):
    """Remove datapoint from sufficient stats for normal component
    """
    return SuffStat((ss.theta*(ss.N)-x*1.0)/(ss.N-1), ss.N-1)
```

Finally, we're ready to create a function that takes a Gibbs step on the state. For each datapoint, it
1. Removes the datapoint from its current cluster.
2. Computes the posterior probability of the point being assigned to each cluster (given the other current assignments).
3. Assigns the datapoint to a cluster sampled from this probability distribution.

```python
def gibbs_step(state):
    pairs = zip(state['data_'], state['assignment'])
    for data_id, (datapoint, cid) in enumerate(pairs):

        state['suffstats'][cid] = remove_datapoint_from_suffstats(datapoint,
                                                                  state['suffstats'][cid])
        scores = cluster_assignment_distribution(data_id, state).items()
        labels, scores = zip(*scores)
        cid = random.choice(labels, p=scores)
        state['assignment'][data_id] = cid
        state['suffstats'][cid] = add_datapoint_to_suffstats(state['data_'][data_id], state['suffstats'][cid])
```

Here's our old function to plot the assignments.

```python
def plot_clusters(state):
    gby = pd.DataFrame({
            'data': state['data_'],
            'assignment': state['assignment']}
        ).groupby(by='assignment')['data']
    hist_data = [gby.get_group(cid).tolist()
                 for cid in gby.groups.keys()]
    plt.hist(hist_data,
             bins=20,
             histtype='stepfilled', alpha=.5 )
```

Randomly assign the datapoints to a cluster to start.

```python
state = initial_state()
plot_clusters(state)
```

Look what happens to the assignments after just one Gibbs step!

```python
gibbs_step(state)
plot_clusters(state)
```

Two:

```python
gibbs_step(state)
plot_clusters(state)
```

After just two steps, our assignments look _really_ good. We can run it a few more times and see the assignments again.

```python
for _ in range(20): gibbs_step(state)
plot_clusters(state)
```

### Nonparametric Mixture Models!

It turns out, the collapsed Gibbs sampler for mixture models is almost identical in the context of a _nonparametric_ model. This model uses a _Dirichlet process prior_ instead of a _Dirichlet distribution prior_. It doesn't require us to specify how many clusters we are looking for in our data.

The cluster assignment score changes slightly. It is proportional to $N_k^{-i}$ for each known cluster. We assign a datapoint to a _new_ cluster with probability proportional to $\alpha$ (which is now the DP dispersion parameter).

```python
def log_cluster_assign_score_dp(cluster_id, state):
    """Log-likelihood that a new point generated will
    be assigned to cluster_id given the current state.
    """
    if cluster_id == "new":
        return np.log(state["alpha_"])
    else:
        return np.log(state['suffstats'][cluster_id].N)
```

The predictive likelihood remains the same for known clusters. However, we need to know the likelihood of assigning a datapoint to a new cluster. In this case, we fall back on the hyperparameters to get:

$$
\begin{align}
p(x_i \,|\, z, x_{\neg_i}, \sigma)
    &= \mathcal{N}(x_i \,|\, \lambda_1, \lambda_2^2+\sigma^2)
\end{align}
    $$

```python
def log_predictive_likelihood_dp(data_id, cluster_id, state):
    """Predictive likelihood of the data at data_id is generated
    by cluster_id given the currenbt state.

    From Section 2.4 of
    http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf
    """
    if cluster_id == "new":
        ss = SuffStat(0, 0)
    else:
        ss = state['suffstats'][cluster_id]

    hp_mean = state['hyperparameters_']['mean']
    hp_var = state['hyperparameters_']['variance']
    param_var = state['cluster_variance_']
    x = state['data_'][data_id]
    return _log_predictive_likelihood(ss, hp_mean, hp_var, param_var, x)
```

Given this, we can define the marginal distribution over cluster assignment. The only change is that the "`new`" state enters in the distribution.

```python
def cluster_assignment_distribution_dp(data_id, state):
    """Compute the marginal distribution of cluster assignment
    for each cluster.
    """
    scores = {}
    cluster_ids = state['suffstats'].keys() + ['new']
    for cid in cluster_ids:
        scores[cid] = log_predictive_likelihood_dp(data_id, cid, state)
        scores[cid] += log_cluster_assign_score_dp(cid, state)
    scores = {cid: np.exp(score) for cid, score in scores.iteritems()}
    normalization = 1.0/sum(scores.values())
    scores = {cid: score*normalization for cid, score in scores.iteritems()}
    return scores
```

We also need to be able to create a new cluster when "`new`" is drawn, and destroy a cluster if its emptied.

```python
def create_cluster(state):
    state["num_clusters_"] += 1
    cluster_id = max(state['suffstats'].keys()) + 1
    state['suffstats'][cluster_id] = SuffStat(0, 0)
    state['cluster_ids_'].append(cluster_id)
    return cluster_id

def destroy_cluster(state, cluster_id):
    state["num_clusters_"] = 1
    del state['suffstats'][cluster_id]
    state['cluster_ids_'].remove(cluster_id)

def prune_clusters(state):
    for cid in state['cluster_ids_']:
        if state['suffstats'][cid].N == 0:
            destroy_cluster(state, cid)
```

Finally, we can define the `gibbs_step_dp` function. It's nearly identical to the earlier `gibbs_step` function except
* It uses `cluster_assignment_distribution_dp`
* It creates a new cluster when the sampled assignment is "`new`".
* It destroys a cluster any time it is emptied.

For clarity, I split out the code for sampling assignment to its own function.

```python
def sample_assignment(data_id, state):
    """Sample new assignment from marginal distribution.
    If cluster is "`new`", create a new cluster.
    """
    scores = cluster_assignment_distribution_dp(data_id, state).items()
    labels, scores = zip(*scores)
    cid = random.choice(labels, p=scores)
    if cid == "new":
        return create_cluster(state)
    else:
        return int(cid)

def gibbs_step_dp(state):
    """Collapsed Gibbs sampler for Dirichlet Process Mixture Model
    """
    pairs = zip(state['data_'], state['assignment'])
    for data_id, (datapoint, cid) in enumerate(pairs):
        state['suffstats'][cid] = remove_datapoint_from_suffstats(datapoint, state['suffstats'][cid])
        prune_clusters(state)
        cid = sample_assignment(data_id, state)
        state['assignment'][data_id] = cid
        state['suffstats'][cid] = add_datapoint_to_suffstats(state['data_'][data_id], state['suffstats'][cid])

```

This time, we will start by randomly assigning our data to two clusters.

```python
state = initial_state(num_clusters=2, alpha=0.1)
plot_clusters(state)
```

Here's what happens when we run our Gibbs sampler once.

```python
gibbs_step_dp(state)
plot_clusters(state)
```

We went from 2 to 4 clusters!

After 100 iterations:

```python
for _ in range(99): gibbs_step_dp(state)
plot_clusters(state)
```

After 100 iterations, our assignment looks correct! We went back to 3 clusters.

We can sample the mixture weights, if we need them, using the "Conditional Distribution of Mixture Weights" derived [here](http://nbviewer.ipython.org/github/tdhopper/notes-on-dirichlet-processes/blob/master/2015-09-02-fitting-a-mixture-model.ipynb#Conditional-Distribution-of-Mixture-Weights).

```python
ss = state['suffstats']
alpha = [ss[cid].N + state['alpha_'] / state['num_clusters_']
         for cid in state['cluster_ids_']]
stats.dirichlet(alpha).rvs(size=1).flatten()
```

    array([ 0.21330625,  0.29838101,  0.48831275])

We can also sample the cluster means using [the method we derived earlier](http://nbviewer.ipython.org/github/tdhopper/notes-on-dirichlet-processes/blob/master/2015-09-02-fitting-a-mixture-model.ipynb#Conditional-Distribution-of-Cluster-Means):

```python
for cluster_id in state['cluster_ids_']:
    cluster_var = state['cluster_variance_']
    hp_mean = state['hyperparameters_']['mean']
    hp_var = state['hyperparameters_']['variance']
    ss = state['suffstats'][cluster_id]

    numerator = hp_mean / hp_var + ss.theta * ss.N / cluster_var
    denominator = (1.0 / hp_var + ss.N / cluster_var)
    posterior_mu = numerator / denominator
    posterior_var = 1.0 / denominator

    mean = stats.norm(posterior_mu, np.sqrt(posterior_var)).rvs()
    print "cluster_id:", cluster_id, "mean", mean

```

    cluster_id: 1 mean -0.0176257860235
    cluster_id: 2 mean -0.400581819532
    cluster_id: 3 mean 0.600302879661

Much thanks to Erik Sudderth's excellent introduction to nonparametric Bayes in [Chapter 2 of his dissertation](http://cs.brown.edu/~sudderth/papers/sudderthPhD.pdf). Algorithms 2.2 and 2.3 in that piece are the clearest formulation of collapsed Gibbs sampling for mixture models that I have come across.


# Hierarchical Model for Abalone Length

Abalone were collected from various sites on the coast of California north of San Francisco. Here I'm going to develop a model to predict abalone lengths based on sites and harvest method - diving or rock-picking. I'm interested in how abalone lengths vary between sites and harvesting methods. This should be a hierarchical model as the abalone at the different sites are from the same population and should exhibit similar effects based on harvesting method. The hierarchical model will be beneficial since some of the sites are missing a harvesting method.

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt

import sampyl as smp
from sampyl import np

import pandas as pd
```

```python
plt.style.use('seaborn')
plt.rcParams['font.size'] = 14.
plt.rcParams['legend.fontsize'] = 14.0
plt.rcParams['axes.titlesize'] = 16.0
plt.rcParams['axes.labelsize'] = 14.0
plt.rcParams['xtick.labelsize'] = 13.0
plt.rcParams['ytick.labelsize'] = 13.0
```

Load our data here. This is just data collected in 2017.

```python
data = pd.read_csv('Clean2017length.csv')
data.head()
```

      data year
      full lengths
      group_id
      site_code
      Full_ID
      Mode

      0
      2017
      181
      1.0
      5
      2017_06_24_005_30_01_01
      R

      1
      2017
      182
      1.0
      5
      2017_06_24_005_30_01_01
      R

      2
      2017
      183
      1.0
      5
      2017_06_24_005_30_01_01
      R

      3
      2017
      191
      1.0
      5
      2017_06_24_005_30_01_01
      R

      4
      2017
      191
      1.0
      5
      2017_06_24_005_30_01_01
      R

Important columns here are:

* **full lengths:** length of abalone
* **mode:** Harvesting method, R: rock-picking, D: diving
* **site_code:** codes for 15 different sites

First some data preprocessing to get it into the correct format for our model.

```python
# Convert sites from codes into sequential integers starting at 0
unique_sites = data['site_code'].unique()
site_map = dict(zip(unique_sites, np.arange(len(unique_sites))))
data = data.assign(site=data['site_code'].map(site_map))

# Convert modes into integers as well
# Filter out 'R/D' modes, bad data collection
data = data[(data['Mode'] != 'R/D')]
mode_map = {'R':0, 'D':1}
data = data.assign(mode=data['Mode'].map(mode_map))
```

## A Hierarchical Linear Model

Here we'll define our model. We want to make a linear model for each site in the data where we predict the abalone length given the mode of catching and the site.

$$ y_s = \alpha_s + \beta_s * x_s + \epsilon $$

where $y_s$ is the predicted abalone length, $x$ denotes the mode of harvesting, $\alpha_s$ and $\beta_s$ are coefficients for each site $s$, and $\epsilon$ is the model error. We'll use this prediction for our likelihood with data $D_s$, using a normal distribution with mean $y_s$ and variance $ \epsilon^2$ :

$$  \prod_s P(D_s \mid \alpha_s, \beta_s, \epsilon) = \prod_s \mathcal{N}\left(D_s \mid y_s, \epsilon^2\right) $$

The abalone come from the same population just in different locations. We can take these similarities between sites into account by creating a hierarchical model where the coefficients are drawn from a higher-level distribution common to all sites.

$$
\begin{align}
\alpha_s & \sim \mathcal{N}\left(\mu_{\alpha}, \sigma_{\alpha}^2\right) \\
\beta_s & \sim \mathcal{N}\left(\mu_{\beta}, \sigma_{\beta}^2\right) \\
\end{align}
$$

```python
class HLM(smp.Model):

    def __init__(self, data=None):
        super().__init__()
        self.data = data

    # Now define the model (log-probability proportional to the posterior)
    def logp_(self, μ_α, μ_β, σ_α, σ_β, site_α, site_β, ϵ):

        # Population priors - normals for population means and half-Cauchy for population stds
        self.add(smp.normal(μ_α, sig=500),
                 smp.normal(μ_β, sig=500),
                 smp.half_cauchy(σ_α, beta=5),
                 smp.half_cauchy(σ_β, beta=0.5))

        # Priors for site coefficients, sampled from population distributions
        self.add(smp.normal(site_α, mu=μ_α, sig=σ_α),
                 smp.normal(site_β, mu=μ_β, sig=σ_β))

        # Prior for likelihood uncertainty
        self.add(smp.half_normal(ϵ))

        # Our estimate for abalone length, α + βx
        length_est = site_α[self.data['site'].values] + site_β[self.data['site'].values]*self.data['mode']

        # Add the log-likelihood
        self.add(smp.normal(self.data['full lengths'], mu=length_est, sig=ϵ))

        return self()
```

```python
sites = data['site'].values
modes = data['mode'].values
lengths = data['full lengths'].values

# Now define the model (log-probability proportional to the posterior)
def logp(μ_α, μ_β, σ_α, σ_β, site_α, site_β, ϵ):

    model = smp.Model()

    # Population priors - normals for population means and half-Cauchy for population stds
    model.add(smp.normal(μ_α, sig=500),
              smp.normal(μ_β, sig=500),
              smp.half_cauchy(σ_α, beta=5),
              smp.half_cauchy(σ_β, beta=0.5))

    # Priors for site coefficients, sampled from population distributions
    model.add(smp.normal(site_α, mu=μ_α, sig=σ_α),
              smp.normal(site_β, mu=μ_β, sig=σ_β))

    # Prior for likelihood uncertainty
    model.add(smp.half_normal(ϵ))

    # Our estimate for abalone length, α + βx
    length_est = site_α[sites] + site_β[sites]*modes

    # Add the log-likelihood
    model.add(smp.normal(lengths, mu=length_est, sig=ϵ))

    return model()
```

```python
model = HLM(data=data)
```

```python
start = {'μ_α': 201., 'μ_β': 5., 'σ_α': 1., 'σ_β': 1.,
         'site_α': np.ones(len(site_map))*201,
         'site_β': np.zeros(len(site_map)),
         'ϵ': 1.}
model.logp_(*start.values())
```

    -509218.07501755428

```python
start = {'μ_α': 201., 'μ_β': 5., 'σ_α': 1., 'σ_β': 1.,
         'site_α': np.ones(len(site_map))*201,
         'site_β': np.zeros(len(site_map)),
         'ϵ': 1.}

# Using NUTS is slower per sample, but more likely to give good samples (and converge)
sampler = smp.NUTS(logp, start)
chain = sampler(1100, burn=100, thin=2)
```

    /Users/mat/miniconda3/lib/python3.6/site-packages/autograd/tracer.py:14: UserWarning: Output seems independent of input.
      warnings.warn("Output seems independent of input.")
    /Users/mat/miniconda3/lib/python3.6/site-packages/autograd/tracer.py:48: RuntimeWarning: overflow encountered in exp
      return f_raw(*args, **kwargs)

    Progress: [##############################] 1100 of 1100 samples

There are some checks for convergence you can do, but they aren't implemented yet. Instead, we can visually inspect the chain. In general, the samples should be stable, the first half should vary around the same point as the second half.

```python
fig, ax = plt.subplots()
ax.plot(chain.site_α);
```

```python
fig.savefig('/Users/mat/Desktop/chains.png', dpi=150)
```

```python
chain.site_α.T.shape
```

    (14, 500)

```python
fig, ax = plt.subplots(figsize=(16,9))
for each in chain.site_α.T:
    ax.hist(each, range=(185, 210), bins=60, alpha=0.5)
ax.set_xticklabels('')
ax.set_yticklabels('');
```

```python
fig.savefig('/Users/mat/Desktop/posteriors.png', dpi=300)
```

With the posterior distribution, we can look at many different results. Here I'll make a function that plots the means and 95% credible regions (range that contains central 95% of the probability) for the coefficients $\alpha_s$ and $\beta_s$.

```python
def coeff_plot(coeff, ax=None):
    if ax is None:
        fig, ax = plt.subplots(figsize=(3,5))
    CRs = np.percentile(coeff, [2.5, 97.5], axis=0)
    means = coeff.mean(axis=0)
    ax.errorbar(means, np.arange(len(means)), xerr=np.abs(means - CRs), fmt='o')

    ax.set_yticks(np.arange(len(site_map)))
    ax.set_yticklabels(site_map.keys())
    ax.set_ylabel('Site')

    ax.grid(True, axis='x', color="#CCCCCC")
    ax.tick_params(axis='both', length=0)
    for each in ['top', 'right', 'left', 'bottom']:
        ax.spines[each].set_visible(False)

    return ax
```

Now we can look at how abalone lengths vary between sites for the rock-picking method ($\alpha_s$).

```python
ax = coeff_plot(chain.site_α)
ax.set_xlim(175, 225)
ax.set_xlabel('Abalone Length (mm)');
```

Here I'm plotting the mean and 95% credible regions (CR) of $\alpha$ for each site. This coefficient measures the average length of rock-picked abalones. We can see that the average abalone length varies quite a bit between sites. The CRs give a measure of the uncertainty in $\alpha$, wider CRs tend to result from less data at those sites.

Now, let's see how the abalone lengths vary between harvesting methods (the difference for diving is given by $\beta_s$).

```python
ax = coeff_plot(chain.site_β)
#ax.set_xticks([-5, 0, 5, 10, 15])
ax.set_xlabel('Mode effect (mm)');
```

Here I'm plotting the mean and 95% credible regions (CR) of $\beta$ for each site. This coefficient measures the difference in length of dive picked abalones compared to rock picked abalones. Most of the $\beta$ coefficients are above zero which indicates that abalones harvested via diving are larger than ones picked from the shore. For most of the sites, diving results in 5 mm longer abalone, while at site 72, the difference is around 12 mm. Again, wider CRs mean there is less data leading to greater uncertainty.

Next, I'll overlay the model on top of the data and make sure it looks right. We'll also see that some sites don't have data for both harvesting modes but our model still works because it's hierarchical. That is, we can get a posterior distribution for the coefficient from the population distribution even though the actual data is missing.

```python
def model_plot(data, chain, site, ax=None, n_samples=20):
    if ax is None:
        fig, ax = plt.subplots(figsize=(4,6))

    site = site_map[site]
    xs = np.linspace(-1, 3)

    for ii, (mode, m_data) in enumerate(data[data['site'] == site].groupby('mode')):

        a = chain.site_α[:, site]
        b = chain.site_β[:, site]

        # now sample from the posterior...
        idxs = np.random.choice(np.arange(len(a)), size=n_samples, replace=False)

        # Draw light lines sampled from the posterior
        for idx in idxs:
            ax.plot(xs, a[idx] + b[idx]*xs, color='#E74C3C', alpha=0.05)

        # Draw the line from the posterior means
        ax.plot(xs, a.mean() + b.mean()*xs, color='#E74C3C')

        # Plot actual data points with a bit of noise for visibility
        mode_label = {0: 'Rock-picking', 1: 'Diving'}
        ax.scatter(ii + np.random.randn(len(m_data))*0.04,
               m_data['full lengths'], edgecolors='none',
               alpha=0.8, marker='.', label=mode_label[mode])

    ax.set_xlim(-0.5, 1.5)
    ax.set_xticks([0, 1])
    ax.set_xticklabels('')
    ax.set_ylim(150, 250)

    ax.grid(True, axis='y', color="#CCCCCC")
    ax.tick_params(axis='both', length=0)

    for each in ['top', 'right', 'left', 'bottom']:
        ax.spines[each].set_visible(False)

    return ax
```

```python
fig, axes = plt.subplots(figsize=(10, 5), ncols=4, sharey=True)

for ax, site in zip(axes, [5, 52, 72, 162]):
    ax = model_plot(data, chain, site, ax=ax, n_samples=30)
    ax.set_title(site)

first_ax = axes[0]
first_ax.legend(framealpha=1, edgecolor='none')
first_ax.set_ylabel('Abalone length (mm)');
```

For site 5, there are few data points for the diving method so there is a lot of uncertainty in the prediction. The prediction is also pulled lower than the data by the population distribution. Similarly, for site 52 there is no diving data, but we still get a (very uncertain) prediction because it's using the population information.

Finally, we can look at the harvesting mode effect for the population. Here I'm going to print out a few statistics for $\mu_{\beta}$.

```python
fig, ax = plt.subplots()
ax.hist(chain.μ_β, bins=30);
b_mean = chain.μ_β.mean()
b_CRs = np.percentile(chain.μ_β, [2.5, 97.5])
p_gt_0 = (chain.μ_β > 0).mean()
print(
"""Mean: {:.3f}
95% CR: [{:.3f}, {:.3f}]
P(mu_b) > 0: {:.3f}
""".format(b_mean, b_CRs[0], b_CRs[1], p_gt_0))
```

We can also look at the population distribution for $\beta_s$ by sampling from a normal distribution with mean and variance sampled from $\mu_\beta$ and $\sigma_\beta$.

$$
\beta_s \sim \mathcal{N}\left(\mu_{\beta}, \sigma_{\beta}^2\right)
$$

```python
import scipy.stats as stats
```

```python
samples = stats.norm.rvs(loc=chain.μ_β, scale=chain.σ_β)
plt.hist(samples, bins=30);
plt.xlabel('Dive harvesting effect (mm)')
```

It's apparent that dive harvested abalone are roughly 5 mm longer than rock-picked abalone. Maybe this is a bias of the divers to pick larger abalone. Or, it's possible that abalone that stay in the water grow larger.


# Import

```python
#region
import matplotlib.pyplot as plt
import math

from sympy import *
import matplotlib.pyplot as plt
from numpy import linspace
import numpy as np
#endregion
t = symbols('t')
f = symbols('f', cls=Function)
```

# Input

```python
#read input
#region
def ReadArray(f):
    line = f.readline()
    result = list(map(lambda x: float(N(x)), line.split()))
    return result

def ReadMatrix(f):
    listCoef = []
    line = f.readline()
    while(line.strip() != ''):
        coef = list(map(lambda x: float(N(x)), line.split()))
        listCoef.append(coef)
        line = f.readline()
    #print('listCoef: ')
    #print(listCoef)
    return listCoef

def RandN(listCoef):
        # R & N
    R = listCoef[0][0]
    N = math.inf
    for coef in listCoef:
        if(R > coef[0]): R = coef[0]
        coef.pop(0)
        if(N > len(coef)): N = len(coef)
    if R <= 0:
        raise ValueError("invalid input: bán kính <= 0")

    return (R,N)

#endregion
```

# Hàm chính

```python
def calculate(initial, listCoef, N):
    result = initial              # mảng kết quả c_i
    k=len(listCoef)-1             # mảng mảng hệ số a_i và f
    for n in range(0,N-k):
        c=0
        offset = 1;
        for i in range(n+1,n+k+1): offset *= i
        #start calculating c_{n+k}
        for m in range(0,n+1):
            mult = 1
            for i in range(0,k):
                c += listCoef[i][n-m] * result[m+i] * mult
                mult *= m+i+1
        c= (listCoef[k][n]-c)/offset           # -1*n! / (n+k)!
        result.append(c)
    return result

#Program
def Polynomial(inputPath):
    f = open(inputPath,"r")
    initial = ReadArray(f)
    listCoef = ReadMatrix(f)
    f.close()
    R,N = RandN(listCoef)
    result = calculate(initial, listCoef, N)
    return (R, result)
def Restore(array):
    3
```

# Plot and save

```python
#region
def Save(result, outputPath, mode):
    f = open(outputPath, mode)
    f.write("Radius of convergence = " + str(result[0]) + ", Result: \n");
    f.write(str(result[1]))
    f.close()

def Plotf(f, interval):
    t_vals = linspace(interval[0], interval[1], 1000)
    lam_x = lambdify(t, f, modules = ['numpy'])
    x_vals = lam_x(t_vals)
    plt.plot(t_vals, x_vals)

def Plot(result, start, end, g = None):
    f = 0
    power = 0
    for i in result:
        f += i * (t ** power)
        power += 1
    Plotf(f, (start, end))
    if g is not None:
        Plotf(g, (start, end))
    return f
#endregion

#Frobenius
```

# Test

```python
test1 = 'example1.txt'
test2 = 'example2.txt'

output = 'outputPath_1.txt'
```

```python
R, array = Polynomial(test1)
print("Radius of convergence = " ,str(R), ", Result:")
np.set_printoptions(precision=1)
print(np.array(array))

f = Plot(array, -2 , 2, g = sin(3*t))
print(f.evalf(2))
Save((R,array),output,"w")
```

```python
R, array = Polynomial(test2)

print("Radius of convergence = " + str(R) + ", Result: \n")
print(array)

Plot(array, -1 , 1, g = sin(t))
Save((R,array),output,"w")

```

```python
Plot([1,2], -2 , 2, g = sin(3*t))
```

```python

```

```python

```

Topic 5 - Solving Differential Equations/26.2.PowerSeries/PowerSeries/.ipynb_checkpoints/PowerSeries-checkpoint.ipynb