# What's new?

- implementation of a saddle point problem into *BoSSS* (e.g. the Poisson equation as a system)
- central-difference-form and strong form
- comparison of both formulations
- advanced: algebraic reduction of the poisson problem

# Prerequisites

- implementation of numerical fluxes -> chapter *NumFlux*
- spatial operator -> chapter *SpatialOperator*

# 1 Problem statement

Within this exercise, we are going to investigate
the discretization of a Poisson equation as a system.
Obviously, it is possible to discretize the Poisson equation as a system of
first-order-PDE's, introducing a vector field $\vec{\sigma}$:
$$
\begin{align}
 \vec{\sigma}  + \nabla u & = 0, & & \text{ in } \Omega
 \\
 \operatorname{div}(\vec{\sigma}) &  = g_{\Omega}, & & \text{ in } \Omega
 \\
  u                                               & = g_D, & & \text{ on } \Gamma_D \\
  - \vec{\sigma} \cdot \vec{n}_{\partial \Omega} & = g_N, & & \text{ on } \Gamma_N
\end{align}
$$
resp. in matrix-notation:
$$
\begin{align*}
  \begin{bmatrix}
    1 & \nabla \\
    \operatorname{div} & 0 \\
  \end{bmatrix}\cdot
  \begin{bmatrix}
    \vec{\sigma}\\
    u
  \end{bmatrix}=
  \begin{bmatrix}
    0 \\
    g_{\Omega}
  \end{bmatrix}
\end{align*}
$$
This exercise, together with the previous one,
will form the foundation for an incompressible Stokes- resp. Navier-Stokes solver.

# 2 Solution within the BoSSS framework

```C#
#r "BoSSSpad.dll"
using System;
using System.Collections.Generic;
using System.Linq;
using ilPSP;
using ilPSP.Utils;
using BoSSS.Platform;
using BoSSS.Foundation;
using BoSSS.Foundation.Grid;
using BoSSS.Foundation.Grid.Classic;
using BoSSS.Foundation.IO;
using BoSSS.Solution;
using BoSSS.Solution.Control;
using BoSSS.Solution.GridImport;
using BoSSS.Solution.Statistic;
using BoSSS.Solution.Utils;
using BoSSS.Solution.Gnuplot;
using BoSSS.Application.BoSSSpad;
using BoSSS.Application.XNSE_Solver;
using static BoSSS.Application.BoSSSpad.BoSSSshell;
Init();

```

```C#
using ilPSP.LinSolvers;
using BoSSS.Solution.Tecplot;
using ilPSP.Connectors.Matlab;
```

# 2.1 Tests on the divergence

## Common base-class for $\text{div}$-implementations
We are going to implement two different formulations of the
divergence-operator for which going to show equivalence.
We implement a common base-class for both formulations:

```C#

abstract public class BaseDivergence :
        BoSSS.Foundation.IEdgeForm, // edge integrals
        BoSSS.Foundation.IVolumeForm     // volume integrals
{
    /// We don't use parameters (e.g. variable viscosity, ...)
    /// at this point: so the parameter list can be null, resp. empty:
    public IList<string> ParameterOrdering {
        get { return null; }
    }

    /// But we have a vector argument variable,
    /// $ [ \sigma_1, \sigma_2 ] = \vec{\sigma} $
    /// (our trial function):
    public IList<String> ArgumentOrdering {
        get { return new string[] { "sigma1", "sigma2" }; }
    }

    public TermActivationFlags VolTerms {
        get {
            return TermActivationFlags.AllOn;
        }
    }

    public TermActivationFlags InnerEdgeTerms {
        get {
            return (TermActivationFlags.AllOn);
        }
    }

    public TermActivationFlags BoundaryEdgeTerms {
       get {
           return TermActivationFlags.AllOn;
        }
    }

    /// The following functions cover the actual math.
    /// For any discretization of the divergence-operator, we have to specify:
    /// \begin{itemize}
    ///    \item a volume integrand,
    ///    \item an edge integrand for inner edges, i.e. on $ \Gamma_i$,
    ///    \item an edge integrand for boundary edges,
    ///          i.e. on $\partial \Omega$.
    /// \end{itemize}
    /// These functions are declared as \code{abstract}, meaning that one has
    /// to specify them in classes derived from \code{BaseLaplace}.

    abstract public double VolumeForm(ref CommonParamsVol cpv,
           double[] U, double[,] GradU,
           double V, double[] GradV);

    abstract public double InnerEdgeForm(ref CommonParams inp,
        double[] U_IN, double[] U_OT, double[,] GradU_IN, double[,] GradU_OT,
        double V_IN, double V_OT, double[] GradV_IN, double[] GradV_OT);

    abstract public double BoundaryEdgeForm(ref CommonParamsBnd inp,
        double[] U_IN, double[,] GradU_IN, double V_IN, double[] GradV_OT);
}
```

```C#
/// We are going to use both, Dirichlet- and Neumann-boundary conditions
/// in this exercise; the function \code{IsDirichletBndy} is used to
/// specify the type of boundary condition at point \code{X}:
static class BndyMap {
static public Func<double[],bool> IsDirichletBndy = delegate(double[] X) {
    double x = X[0];
    double y = X[1];
    if(Math.Abs(x - (-1.0)) < 1.0e-8)
        return true;
    if(Math.Abs(y - (-1.0)) < 1.0e-8)
        return true;
    return false;
};
}
```

## Formulation (i): Central-difference-form of $\text{div}$
The implementation of the central-difference form is as follows:

```C#
class Divergence_cendiff : BaseDivergence {

    /// The volume form is equal to
    /// $ -\vec{\sigma} \cdot \nabla v$:
    override public double VolumeForm(ref CommonParamsVol cpv,
        double[] Sigma, double[,] GradSigma,
        double V, double[] GradV) {
        double Acc = 0;
        for(int d = 0; d < cpv.D; d++) {
            Acc -= Sigma[d]*GradV[d];
        }
        return Acc;
    }

    /// At the cell boundaries, we use a central-difference-flux,
    /// i.e. $\mean{\vec{\sigma}} \cdot \vec{n}_{\Gamma} \jump{v}$:
    override public double InnerEdgeForm(ref CommonParams inp,
        double[] Sigma_IN, double[] Sigma_OT, double[,] GradSigma_IN, double[,] GradSigma_OT,
        double V_IN, double V_OT, double[] GradV_IN, double[] GradV_OT) {

        double Acc = 0;
        for(int d = 0; d < inp.D; d++) {
            Acc += 0.5*(Sigma_IN[d] + Sigma_OT[d])*inp.Normal[d]*(V_IN - V_OT);
        }
        return Acc;
    }

    override public double BoundaryEdgeForm(ref CommonParamsBnd inp,
        double[] Sigma_IN, double[,] GradSigma_IN, double V_IN, double[] GradV_OT) {

        double Acc = 0;

        if(BndyMap.IsDirichletBndy(inp.X)) {
            /// Dirichlet-boundary: by taking the inner value of $\vec{\sigma}$,
            /// this is a free boundary with respect to $\vec{\sigma}$.
            for(int d = 0; d < inp.D; d++) {
                Acc += Sigma_IN[d]*inp.Normal[d]*V_IN;
            }
        } else {
            /// Neumann-boundary
            double gNeu = 0.0;
            Acc += gNeu*V_IN;
        }
        return Acc;
    }
}
```

## Formulation (ii): 'Strong' form of $\text{div}$:
Here, we use the form
$$
   b(\vec{\sigma},v) =
   \oint_{\Gamma \backslash \Gamma_D}
          M(v) J(\vec{\sigma}) \cdot \vec{n}_\Gamma
   dA
   -
   \int_{\Omega} \text{div}(\vec{\sigma}) \cdot v dV
$$
where M,J denote the mean and jump operator, respectively. This is actually the negative divergence, which will be more useful
later on.

```C#

class Divergence_strong : BaseDivergence {

    /// We have to implement \code{VolumeForm},
    /// \emph{InnerEdgeForm} and \code{BoundaryEdgeForm}:
    override public double VolumeForm(ref CommonParamsVol cpv,
        double[] Sigma, double[,] GradSigma,
        double V, double[] GradV) {
        double Acc = 0;
        for(int d = 0; d < cpv.D; d++) {
            Acc -= GradSigma[d,d]*V;
        }
        return Acc;
    }

    override public double InnerEdgeForm(ref CommonParams inp,
        double[] Sigma_IN, double[] Sigma_OT, double[,] GradSigma_IN, double[,] GradSigma_OT,
        double V_IN, double V_OT, double[] GradV_IN, double[] GradV_OT) {

        double Acc = 0;
        for(int d = 0; d < inp.D; d++) {
            Acc += 0.5*(V_IN + V_OT)*(Sigma_IN[d] - Sigma_OT[d])*inp.Normal[d];
        }
        return Acc;
    }

    override public double BoundaryEdgeForm(ref CommonParamsBnd inp,
        double[] Sigma_IN, double[,] GradSigma_IN, double V_IN, double[] GradV_OT) {

        double Acc = 0;

        if(BndyMap.IsDirichletBndy(inp.X)) {
            Acc = 0;
        } else {
            double gNeu = 0.0;
            for(int d = 0; d < inp.D; d++) {
                Acc += Sigma_IN[d]*inp.Normal[d]*V_IN;
            }
            Acc -= gNeu*V_IN;
        }
        return Acc;
    }
}
```

# 3 Equality test
We are going to test the equivalence of both formulationt
on a 2D grid, using a DG basis of degree 1:

```C#

 var grd2D               = Grid2D.Cartesian2DGrid(GenericBlas.Linspace(-1,1,6),                                                    GenericBlas.Linspace(-1,1,7));
 var b                   = new Basis(grd2D, 1);
 SinglePhaseField sigma1 = new SinglePhaseField(b,"sigma1");
 SinglePhaseField sigma2 = new SinglePhaseField(b,"sigma2");
 CoordinateVector sigma  = new CoordinateVector(sigma1,sigma2);
 var TrialMapping        = sigma.Mapping;
 var TestMapping         = new UnsetteledCoordinateMapping(b);
```

```C#
    /// We create the matrix of the central-difference formulation:
 var OpDiv_cendiff = (new Divergence_cendiff()).Operator();
 var MtxDiv_cendiff = OpDiv_cendiff.ComputeMatrix(TrialMapping,
                                                  null,
                                                  TestMapping);
```

We create the matrix of the strong formulation and show that the matrices of both formulations are equal.

We use the \code{InfNorm(...)}-method to identify whether a matrix is (approximately) zero or not.

```C#

 var OpDiv_strong  = (new Divergence_strong()).Operator();
 var MtxDiv_strong = OpDiv_strong.ComputeMatrix(TrialMapping, null, TestMapping);
 var TestP         = MtxDiv_cendiff + MtxDiv_strong;
 TestP.InfNorm();
```

# 4 The gradient-operator

For the variational formulation of the gradient operator, a vector-valued
test-function is required. Unfourtunately, this is not supported by
*BoSSS*. Therefore we have to discretize the gradent component-wise,
i.e. as $\partial_{x}$ and $\partial_y$.

A single derivative
can obviously be expressed as a divergence by the
identity $ \partial_{x_d} = \text{div}( \vec{e}_d u ) $.

```C#

class Gradient_d :
        BoSSS.Foundation.IEdgeForm, // edge integrals
        BoSSS.Foundation.IVolumeForm     // volume integrals
{
    public Gradient_d(int _d) {
        this.d = _d;
    }

    /// The component index of the gradient:
    int d;

    /// As ususal, we do not use parameters:
    public IList<string> ParameterOrdering {
        get { return null; }
    }

    /// We have one argument $u$:
    public IList<String> ArgumentOrdering {
        get { return new string[] { "u" }; }
    }

    public TermActivationFlags VolTerms {
        get { return TermActivationFlags.AllOn; }
    }

    public TermActivationFlags InnerEdgeTerms {
        get { return (TermActivationFlags.AllOn); }
    }

    public TermActivationFlags BoundaryEdgeTerms {
       get { return TermActivationFlags.AllOn; }
    }

    /// Now, we implement
    /// \begin{itemize}
    ///    \item the volume form $u \vec{e}_d \cdot \nabla v$
    ///    \item the boundary form
    ///       $\mean{u \ \vec{e}_d} \cdot \vec{n}_\Gamma \jump{v}$
    /// \end{itemize}
    public double VolumeForm(ref CommonParamsVol cpv,
           double[] U, double[,] GradU,
           double V, double[] GradV) {

        double Acc = 0;
        Acc -= U[0]*GradV[this.d];
        return Acc;
    }

    public double InnerEdgeForm(ref CommonParams inp,
        double[] U_IN, double[] U_OT, double[,] GradU_IN, double[,] GradU_OT,
        double V_IN, double V_OT, double[] GradV_IN, double[] GradV_OT) {

        double Acc = 0;
        Acc += 0.5*(U_IN[0] + U_OT[0])*inp.Normal[this.d]*(V_IN - V_OT);
        return Acc;
     }

    public double BoundaryEdgeForm(ref CommonParamsBnd inp,
        double[] U_IN, double[,] GradU_IN, double V_IN, double[] GradV_OT) {

        double Acc = 0;
        if(BndyMap.IsDirichletBndy(inp.X)) {
            double u_Diri = 0.0;
            Acc += u_Diri*inp.Normal[this.d]*V_IN;
        } else {
            Acc += U_IN[0]*inp.Normal[this.d]*V_IN;
        }
        return Acc;
    }
}
```

Now, we are ready to assemble the full $\nabla$ operator
as $\left[ \begin{array}{c} \partial_x \\ \partial_y \end{array} \right]$.

```C#

var OpGrad = new SpatialOperator(1,2,QuadOrderFunc.Linear(),"u","c1","c2");
OpGrad.EquationComponents["c1"].Add(new Gradient_d(0));
OpGrad.EquationComponents["c2"].Add(new Gradient_d(1));
OpGrad.Commit();
```

As an additional test, we create the gradient-matrix and verify that
its transpose
is equal to the negative **MtxDiv**-matrix:

```C#

var MtxGrad = OpGrad.ComputeMatrix(TestMapping, null, TrialMapping);
var Test2   = MtxGrad.Transpose() - MtxDiv_strong;
Test2.InfNorm();
```

# 5 The complete Poisson-system
## Assembly of the system

We also need the identity-matrix in the top-left corner
of the Poisson-system:

```C#

public class Identity :
        BoSSS.Foundation.IVolumeForm
{
    public IList<string> ParameterOrdering {
        get { return new string[0]; }
    }

    public string component;

    public IList<String> ArgumentOrdering {
        get { return new string[] { component }; }
    }

    public TermActivationFlags VolTerms {
        get {
            return TermActivationFlags.AllOn;
        }
    }

    public double VolumeForm(ref CommonParamsVol cpv,
           double[] U, double[,] GradU,
           double V, double[] GradV) {
        return U[0]*V;
    }
}
```

    (10,19): warning CS0649: Field 'Identity.component' is never assigned to, and will always have its default value null

We are going to implement the linear Poisson-operator
$$
    \left[ \begin{array}{ccc}
     1           &  0         & \partial_x \\
     0           &  1         & \partial_y \\
     -\partial_x & -\partial_y & 0
\end{array} \right]
\cdot
\left[ \begin{array}{c} \sigma_0 \\ \sigma_1 \\ u \end{array} \right]
=
\left[ \begin{array}{c} c_0 \\ c_1 \\ c_2 \end{array} \right]
$$
The variables $c_0$, $c_1$ and $c_2$, which correspond to the
test functions are also called co-domain variables of the operator.
We are using the negative divergence, since this will lead to a
symmetric matrix, instead of a anti-symmetric one.
By doing so, we can e.g. use a Cholesky-factorization to determine
whether the system is definite or not.

```C#

var OpPoisson = new SpatialOperator(3, 3,
                      QuadOrderFunc.Linear(),
                      "sigma1", "sigma2", "u", // the domain-variables
                      "c1", "c2", "c3"); //       the co-domain variables
/// Now we add all required components to \code{OpPoisson}:
OpPoisson.EquationComponents["c1"].Add(new Gradient_d(0));
OpPoisson.EquationComponents["c1"].Add(new Identity() { component = "sigma1" });
OpPoisson.EquationComponents["c2"].Add(new Gradient_d(1));
OpPoisson.EquationComponents["c2"].Add(new Identity() { component = "sigma2" });
OpPoisson.EquationComponents["c3"].Add(new Divergence_strong());
OpPoisson.Commit();
```

We create mappings $[\sigma_1, \sigma_2, u ]$:
three different combinations of DG orders will be investigated:

- equal order: the same polynomial degree for $u$ and $\vec{\sigma}$
- mixed order: the degree of $u$ is lower than the degree
      of $\vec{\sigma}$.
- `strange' order: the degree of $u$ is higher than the degree of
      $\vec{\sigma}$.

```C#

var b3         = new Basis(grd2D, 3);
var b2         = new Basis(grd2D, 2);
var b4         = new Basis(grd2D, 4);
var EqualOrder = new UnsetteledCoordinateMapping(b3,b3,b3);
var MixedOrder = new UnsetteledCoordinateMapping(b4,b4,b3);
var StrngOrder = new UnsetteledCoordinateMapping(b2,b2,b3);
```

```C#
var MtxPoisson_Equal = OpPoisson.ComputeMatrix(EqualOrder, null, EqualOrder);
var MtxPoisson_Mixed = OpPoisson.ComputeMatrix(MixedOrder, null, MixedOrder);
var MtxPoisson_Strng = OpPoisson.ComputeMatrix(StrngOrder, null, StrngOrder);
```

We show that the matrices are symmetric
(use e.g. **SymmetryDeviation(...)**), but indefinite
(use e.g. **IsDefinite(...)**).

```C#
double symDev_Equal = MtxPoisson_Equal.SymmetryDeviation();
symDev_Equal
```

2.0920765120280294E-14

```C#
double symDev_Mixed = MtxPoisson_Mixed.SymmetryDeviation();
symDev_Mixed
```

3.755329380794592E-14

```C#
double symDev_Strng = MtxPoisson_Strng.SymmetryDeviation();
symDev_Strng
```

2.090688733247248E-14

```C#
MtxPoisson_Equal.IsDefinite()
```

False

```C#
MtxPoisson_Mixed.IsDefinite()
```

False

```C#
MtxPoisson_Strng.IsDefinite()
```

False

```C#
/// BoSSScmdSilent BoSSSexeSilent
NUnit.Framework.Assert.LessOrEqual(symDev_Equal, 1.0e-8);
NUnit.Framework.Assert.LessOrEqual(symDev_Mixed, 1.0e-8);
NUnit.Framework.Assert.LessOrEqual(symDev_Strng, 1.0e-8);
```

# 6 Advanced topics

```C#

```

## Algebraic reduction

Since the top-left corner of our matrix
$$
\left[ \begin{array}{cc}
1   & B \\
B^T & 0
\end{array} \right]
$$
is actually very easy to eliminate the variable $\vec{\sigma}$
from our system algebraically.
The matrix of the reduces system is obviously $B^T \cdot B$.

# Extraction of sub-matrices and elimination
From the mapping, we can actually obtain index-lists for each variable,
which can then be used to extract sub-matrices from
**MtxPoisson\_Equal**, **MtxPoisson\_Mixed**, resp.
**MtxPoisson\_Strng**.

```C#

long[] SigmaIdx_Equal = EqualOrder.GetSubvectorIndices(true, 0,1);
long[] uIdx_Equal     = EqualOrder.GetSubvectorIndices(true, 2);
long[] SigmaIdx_Mixed = MixedOrder.GetSubvectorIndices(true, 0,1);
long[] uIdx_Mixed     = MixedOrder.GetSubvectorIndices(true, 2);
long[] SigmaIdx_Strng = StrngOrder.GetSubvectorIndices(true, 0,1);
long[] uIdx_Strng     = StrngOrder.GetSubvectorIndices(true, 2);
```

The extraction of the sub-matrix and the elimination, for the equal order:

```C#

var MtxPoissonRed_Equal =
      MtxPoisson_Equal.GetSubMatrix(uIdx_Equal, SigmaIdx_Equal)  // -Divergence
    * MtxPoisson_Equal.GetSubMatrix(SigmaIdx_Equal, uIdx_Equal); // Gradient
```

Finally, we also
create the reduced system for the mixed and the strange
order, test for the definiteness of the reduced system.

Equal and mixed order are positive definite, while the strange order
is indefinite - a clear indication that something ist wrong:

```C#
var MtxPoissonRed_Mixed =
      MtxPoisson_Mixed.GetSubMatrix(uIdx_Mixed, SigmaIdx_Mixed)  // -Divergence
    * MtxPoisson_Mixed.GetSubMatrix(SigmaIdx_Mixed, uIdx_Mixed); // Gradient
var MtxPoissonRed_Strng =
      MtxPoisson_Strng.GetSubMatrix(uIdx_Strng, SigmaIdx_Strng)  // -Divergence
    * MtxPoisson_Strng.GetSubMatrix(SigmaIdx_Strng, uIdx_Strng); // Gradient
```

```C#
bool isdef_red_Equal = MtxPoissonRed_Equal.IsDefinite();
isdef_red_Equal
```

True

```C#
bool isdef_red_Mixed = MtxPoissonRed_Mixed.IsDefinite();
isdef_red_Mixed
```

True

```C#
bool isdef_red_Strng = MtxPoissonRed_Strng.IsDefinite();
isdef_red_Strng
```

False

```C#
/// BoSSScmdSilent BoSSSexeSilent
NUnit.Framework.Assert.IsTrue(isdef_red_Equal);
NUnit.Framework.Assert.IsTrue(isdef_red_Mixed);
NUnit.Framework.Assert.IsFalse(isdef_red_Strng);
```

We compute the condition number of all three matrices; we observe that
the mixed as well as the equal-order discretization result give rather
moderate condition numbers.

For the strange orders, the condition number
of the system is far to high:

```C#
double condest_Mixed = MtxPoissonRed_Mixed.condest();
condest_Mixed
```

7016.828053419778

```C#
double condest_Equal = MtxPoissonRed_Equal.condest();
condest_Equal
```

3577.101349907232

```C#
double condest_Strng = MtxPoissonRed_Strng.condest();
condest_Strng
```

6.159871811473017E+19

```C#
/// BoSSScmdSilent BoSSSexeSilent
NUnit.Framework.Assert.LessOrEqual(condest_Mixed, 1e5);
NUnit.Framework.Assert.LessOrEqual(condest_Equal, 1e5);
NUnit.Framework.Assert.Greater(condest_Strng, 1e10);
```

```C#

```


# Stochastic Differential Equations - Lecture 2

## Kostas Zygalakis

### Reminders

A stochastic integral is

$$
\begin{equation}
  \int_{t_0}^t G(t') \, dW(t') = \text{mean-square-}\lim_{n\to\infty} \left\{ \sum_{i=1}^n G(t_{i-1}) \left( W_{t_i} - W_{t_{i-1}} \right) \right\}
\end{equation}
$$

where $W$ (sometimes denoted $B$) is a *Wiener process* or *Brownian motion* or *Brownian path*.

The numerical solution of SDEs is given by

$$
\begin{equation}
  dX_t = f(X_t) \, dt + g(X_t) \, dW_t.
\end{equation}
$$

This is *not* the same as

$$
\begin{equation}
  \frac{d X_t}{dt} = f(X_t) + g(X_t) \frac{d W_t}{dt}
\end{equation}
$$

as the path itself is not differentiable. Instead it should be interpreted as the integral equation

$$
\begin{equation}
  X_t = X_0 + \int_0^t f(X(s)) \, ds + \int_0^t g(X(s)) \, dW_s.
\end{equation}
$$

Doing this discretely we have

$$
\begin{align}
  X((t+1)h) - X(th) & = X_0 + \int_0^{(t+1)h)} f(X(s)) \, ds + \int_0^{(t+1)h} g(X(s)) \, dW_s - X_0 - \int_0^{th} f(X(s)) \, ds - \int_0^{th} g(X(s)) \, dW_s \\
  & = \int_{th}^{(t+1)h)} f(X(s)) \, ds + \int_{th}^{(t+1)h} g(X(s)) \, dW_s \\
  & \simeq f(X(th)) \left[ (t+1)h - th \right] + g(X(th)) \left[ W \left( (t+1) h \right) - W(th) \right] \\
  & \simeq f(X(th)) h + g(X(th)) \xi_n.
\end{align}
$$

So the algorithm becomes

$$
\begin{equation}
  X_{n+1} - X_n = f(X_n) h + g(X_n) \xi_n
\end{equation}
$$

where $\xi_n$ is a random variable with

1. zero mean,
2. normally distributed,
3. varaiance $(t+1)h - th = h$.

This is often rewritten as (the **Euler-Maruyama** method)

$$
\begin{equation}
  X_{n+1} - X_n = f(X_n) h + \sqrt{h} g(X_n) \hat{\xi}_n, \qquad \hat{\xi}_n \sim N(0, 1).
\end{equation}
$$

The random variable $\hat{\xi}_n$ is therefore the standard normally distributed variable with mean zero and variance (or standard deviation) one.

## Accuracy and convergence

Consider the simple ODE

$$
\begin{equation}
  \frac{dX}{dt} = f(X(t)).
\end{equation}
$$

Given the simple Euler method

$$
\begin{equation}
  X_{n+1} = X_n + h f(X_n)
\end{equation}
$$

we have that the error (given the true solution $X^{(e)}(T)$ and the numerical solution $X^{(h)}(T)$ with step size $h$ at identical times $T$) is

$$
\begin{equation}
  | X^{(e)}(T) - X^{(h)}(T) | \le C(T) h^{\alpha}
\end{equation}
$$

where $\alpha = 1$.

**NOTE**: picture here looking at the difference between the exact integral and the "Euler" integral over the range of $h$.

Argument for the convergence result:

$$
\begin{align}
  && \frac{d f(X(t))}{dt} & = \frac{df}{dX} \frac{dX}{dt} \\
  \implies && \frac{df}{dt} & = f'(X(t))f(X(t)) \\
  \implies && f(X(t)) = f(X(0)) + \int_0^t f'(X(s)) f(X(s)) \, ds.
\end{align}
$$

Hence we have

$$
\begin{align}
  \int_0^h f(X(s)) \, ds & = \int_0^h \left[ f(X(0)) + \left( \int_0^s f'(X(l))f(X(l)) \right) \, dl \right] \, ds \\
  & = h f(X(0)) + \int_0^h \left( \int_0^s f'(X(l)) f(X(l)) \, dl \right) \, ds
\end{align}
$$

### Extending the analysis to SDEs

#### Version 1

$$
\begin{equation}
  X_{n+1} = X_n + h f(X_n) + g(X_n) \Delta W_n
\end{equation}
$$

#### Version 2

$$
\begin{equation}
  X_{n+1} = X_n + h f(X_n) + \sqrt{h} g(X_n) \xi_n, \qquad \xi_n \sim N(0, 1).
\end{equation}
$$

Measuring the errors is more difficult as we have a random variable, so each run will give different results. Instead we must *average*.

### Strong convergence

In this case we measure

$$
\begin{equation}
  \mathbb{E} | X^{(e)}(T) - X^{(h)}(T) | \le C h^{\gamma}.
\end{equation}
$$

This is the **mean error**. This *means* that we are assuming an *underlying* random process, or a *fixed* Brownian process, which is *approximated* by $\xi_n$ or by $\Delta W_n$. As we vary the step size $h$ we keep the same underlying Brownian path, just evaluate it differently (effectively integrating over segments of different lengths).

The expectation here is that the convergence rate for Euler-Maruyama here is $\gamma = 1/2$, thanks to the $\sqrt{h}$ factor in the random process. This "intuitively" follows from thinking about generalizing the above analysis to the stochastic case, looking at the extra terms in the process, and can easily be checked numerically.

### Weak convergence

In this case we measure

$$
\begin{equation}
  \left| \mathbb{E} (p(X^{(e)}(T)) - \mathbb{E} (p(X^{(h)}(T)) \right| \le C h^{\gamma}.
\end{equation}
$$

Here $p$ is some function of the solution. This is the **error of the means**. For Euler-Maruyama the result is that $\gamma = 1$.

If you consider $p(X) = X$ we can compare the weak convergence test

$$
\begin{equation}
  \left| \mathbb{E} (X^{(e)}(T)) - \mathbb{E} (X^{(h)}(T)) \right|
\end{equation}
$$

with the strong

$$
\begin{equation}
  \mathbb{E} \left| X^{(e)}(T) - X^{(h)}(T) \right| .
\end{equation}
$$

By considering two Gaussian random processes we can see that strong convergence implies weak convergence, but not the other way around (two processes with identical mean need not have the same realization).

## Stochastic chain rule

Without proof:

$$
\begin{equation}
  dW^2 = dt, \qquad dW^{2+N} = 0, \quad \text{for } N > 0.
\end{equation}
$$

#### Deterministic chain rule

$$
\begin{equation}
  \frac{dX}{dt} = f(X).
\end{equation}
$$

Hence

$$
\begin{align}
  \frac{d}{dt} V(X_t) & \simeq V(X(t) + dX(t)) - V(X(t)) \\
  & \simeq V(X(t)) + V'(X(t)) \, dX(t) + \frac{1}{2} V''(X(t)) \, (dX(t))^2 - V(x(t)) \\
  & \simeq V'(X(t)) \, dX(t) + \frac{1}{2} V''(X(t)) \, (dX(t))^2 \\
  & \simeq V'(X) f(X)\, dt + \underbrace{\frac{1}{2} V''(X) f^2(X) \, dt^2}_{\text{small}} \\
  \frac{dV}{dt} & = V'(X) f(X).
\end{align}
$$

#### Stochastic case

$$
\begin{align}
  dV(x) & = V(X(t) + dX(t)) - V(X(t)) \\
  & \simeq V'(X) \, dX + \frac{1}{2} V''(X) \, (dX)^2 \\
  & \simeq V'(X) \left[ f(X)\,dt + g(X)\,dW_t \right] + \frac{1}{2} V''(X) \left[ f(X)\,dt + g(X)\,dW_t \right]^2 \\
  & \simeq V'(X) \left[ f(X)\,dt + g(X)\,dW_t \right] + \frac{1}{2} V''(X) \left[ f(X)^2\,dt^2 + f(X) g(X)\,dt dW_t + g^2\,dW^2_t \right] \\
  & \simeq \left[ V' f + \frac{1}{2} V'' g^2 \right] \, dt + V' g \, dW_t + \text{higher order terms}.
\end{align}
$$

We see that we expect additional terms resulting from the Brownian path.

#### Application

Remember the stochastic integral

$$
\begin{equation}
  \int_{t_0}^t W_s \, dW_s = \frac{1}{2} W_t^2 - \frac{1}{2} W_{t_{0}}^2 - \frac{1}{2} (t - t_0).
\end{equation}
$$

If we choose $V = X^2$ for the equation $dX_t = dW_t$ with $f = 0$ and $g = 1$ then we get

$$
\begin{align}
  && dX_t^2 & = dt + 2 X_t \, dW_t. \\
  \implies && X_t^2 & = X_{t_0}^2 + \int_{t_0}^t ds + \int_{t_0}^t 2 X_s \, dW_s \\
  \implies && W_t^2 & = W_{t_0}^2 + (t - t_0) + 2 \int_{t_0}^t W_s \, dW_s.
\end{align}
$$

## Exercise

Use the stochastic chain rule to prove that

$$
\begin{equation}
  dX_t = \lambda X_t \, dt + \mu X_t \, dW_t
\end{equation}
$$

has a solution of the form

$$
\begin{equation}
  X_t = X_0 \exp \left[ (\lambda - \tfrac{1}{2} \mu) t + \mu W_t \right].
\end{equation}
$$

**Hint**: apply the chain rule to $V(x) = \log(x)$.

FEEG6016 Simulation and Modelling/2014/SDEs Lecture 2.ipynb


# Understanding the positional encoding

_This notebook is brought to you by [Vladislav Goncharenko](https://www.linkedin.com/in/vladislav-goncharenko/)_ ([girafe-ai](https://github.com/girafe-ai)).

```python
# If using Colab, uncomment this cell
!pip install -Uq plotly
```

         |████████████████████████████████| 13.2MB 300kB/s

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import plotly.express as px
```

```python
plt.rcParams.update({"font.size": 14})
```

Positional Encoding matrix components proposed in the article

$$
PE_{(pos,2i)} =
\sin \left( \frac{pos}{10000^{2i/d_{\text{model}}}} \right) \sim
$$

$$
\sim \sin \left( \exp \left( -\frac{2i}{d_{\text{model}}} \right) \cdot \text{pos} \right) =
\sin(\omega \cdot t)
$$

$$ \\ $$

$$
PE_{(pos,2i+1)} =
\cos(\dots) \sim \cos (\omega \cdot t)
$$

Let's treat $\text{pos}$ as time and number of embedding component as carrier frequency of our signal.

Note that carrier frequencies decrease exponentionally.

```python
def make_carriers(d_mod, denom):
    return 1 / np.power(denom, np.arange(d_mod) / d_mod)
```

```python
carriers = {i: make_carriers(10, 10 ** i) for i in (8, 4, 2)}

plt.figure(figsize=(15, 7))
for i, carrier in carriers.items():
    plt.plot(carrier, label=f"denominator: 10^{i}")
plt.legend()
plt.xlabel("Number of embedding components")
plt.ylabel("Carrier frequency")
plt.show()
```

In fact, different frequencies correspond to different "clocks" that we have to measure $\text{pos}$ as time.

Model doesn't know $\text{pos}$ value directly but it sees all the "times" (in fact phases) of different "clocks" ($\sin$s and $\cos$s of different frequencies).

By having representative suits of "clocks" we can definetly say _what time is it now ($\text{pos}$ value)_ for every given "moment".

```python
def make_pa_matrix(n_pos, d_mod, denom):
    res = np.empty((n_pos, d_mod))
    carriers = make_carriers(d_mod, denom)

    for pos in range(n_pos):
        if pos % 2:
            funct = np.sin
        else:
            funct = np.cos

        for i in range(d_mod):
            res[pos, i] = funct((pos // 2) * carriers[i])
    return res
```

## 2D case

```python
pa2 = make_pa_matrix(1000, 2, 10 ** 4)

plt.figure(figsize=(15, 12))
plt.scatter(pa2[:, 0], pa2[:, 1], c=np.arange(len(pa2)))
plt.colorbar()
plt.show()
```

**Nice and harmonic picture, isn't it?**

That's because the curve plotted is [Lissajou's curve](https://en.wikipedia.org/wiki/Lissajous_curve):

Curve implicitly specified by harmonic coordinates

$$\left\{ \begin{align}
  & x(t)=A\sin (at+\delta ) \\
 & y(t)=B\sin (bt) \\
\end{align} \right.
$$

In our case $\delta = \pi / 2$.

## 3D case

```python
pa3 = make_pa_matrix(250, 3, 2 ** 3)
```

```python
pa3_df = pd.DataFrame(
    np.concatenate((pa3, np.arange(len(pa3))[:, None]), axis=1),
    columns=["x", "y", "z", "c"],
)
pa3_df.head()
```

      x
      y
      z
      c

      0
      1.000000
      1.000000
      1.000000
      0.0

      1
      0.000000
      0.000000
      0.000000
      1.0

      2
      0.540302
      0.877583
      0.968912
      2.0

      3
      0.841471
      0.479426
      0.247404
      3.0

      4
      -0.416147
      0.540302
      0.877583
      4.0

```python
fig = px.scatter_3d(pa3_df, x="x", y="y", z="z", color="c")
fig.show()
```

<meta charset="utf-8" />

```python

```


## Calculate local available potential energy
**22 March 2022 by MiniUFO**

---
[TOC]

---
### 1. Introduction
Available potential energy (APE) is a useful quantity in diagnosing energetics of geofluid.  It is defined as the domain integral of the potential energy (PE) difference between actual state and a minimum PE state (or reference state):
$$
\begin{align}
E_a = \int e_a dV = g\int \left(\rho' z\right) dV = g\int \left(\rho - \rho_R\right) z dV \tag{1}
\end{align}
$$
where $\rho$ is the density of a instantaneous state， $\rho_R$ the density of stably-stratified (vertically-sorted) motionless state (or reference state), and $e_a$ the APE density.  It is easy to define APE density $e_p$ as:
$$
\begin{align}
e_a = \left(\rho - \rho_R\right)g z \tag{2}
\end{align}
$$
However, this is different from the eddy kinetic energy that is always positive.  Large cancellation between positive and negative $e_p$ will be observed in the calculation, which is not physically useful, as we expect energy should be positive definite.

---

### 2. Theoretical guide
Commonly used definition of APE density is:
$$
\begin{align}
e_a = -\frac{g}{2}\frac{\rho'^2}{\partial\rho/\partial z} \tag{3}
\end{align}
$$
This is a small-amplitude approximation or the QG version.  In the finite-amplitude case, one need the most accurate definition by [Holliday and McIntyre (1981)](https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/on-potential-energy-density-in-an-incompressible-stratified-fluid/75CBC2DE788C9D22EA211BE91C4F2167):
$$
\begin{align}
e_a &= \int_{z_r(\rho)}^z g\left[\rho-\rho_r\left(z'\right)\right] dz' \tag{4}\\
&= \int_0^\eta g\left[\rho-\rho_r\left(z+z'\right)\right] dz' \tag{4'}
\end{align}
$$
If we compare Eq. (4') with local wave activity proposed by [Huang and Nakamura](https://journals.ametsoc.org/view/journals/atsc/73/1/jas-d-15-0194.1.xml), we could find that they are mathematically similar [see also [Roullet and Klein (2009)](https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/available-potential-energy-diagnosis-in-a-direct-numerical-simulation-of-rotating-stratified-turbulence/906E71577A230B90B48D8CEB442EE873)].  The difference is their units and that local wave activity density is defined in the hoziontal plane while local APE density is in the vertical plane.

Based on this fact, the local APE density calculation is exactly the same as local wave activity density.  There could be a sign difference if one call `cal_local_wave_activity()` to calculate local APE density.  Since both of them are positive definite, just multiplying `-1` would give the right answer.

---

### 3. Calculation procedure
#### 3.1 Load data
Here for demonstrating purpose, we use a sample data of [internal wave breaking over topography simulated using MITgcm](https://github.com/miniufo/InternalWaveBreaking).

First, we read in the data and calculate the conservatively-sorted reference state:

```python
import xarray as xr
import numpy as np
import sys
sys.path.append('../../')
from xgcm import Grid
from xcontour.xcontour import Contour2D

##################################  load data  #######################################
ds = xr.open_dataset('../Data/internalwave.nc')

# add metrics for xgcm.Grid
grid = Grid(ds, metrics = {
        ('X',)    : ['dxG', 'dxF', 'dxC', 'dxV'], # X distances
        ('Z',)    : ['drW', 'drS', 'drC', 'drF', 'drG'], # Z distances
        ('X', 'Z'): ['yA']}) # Areas in X-Z plane

# get potential temperature and maskout due to topography
T = ds.THETA.where(ds.maskC)

# calculate buoyancy using linear EOS
alpha = 2E-4
T0 = 20
g = 9.81
b = (alpha * (T - T0) * g).rename('b')

###########################  calculate sorted state  #################################
# Initialize equally-spaced contours from minimum value to maximum
# value (within lat/lon dims).  Here will implicitly loop over each isentropic level
%matplotlib inline
N  = 121           # increase the contour number may get non-monotonic A(q) relation
increase = False   # Y-index increases with latitude (sometimes not)
lt = True          # northward of PV contours (larger than) is inside the contour
                   # change this should not change the result of Keff, but may alter
                   # the values at boundaries
dtype = np.float32 # use float32 to save memory
undef = -9.99e8    # for maskout topography if present

# initialize a Contour2D analysis class using grid and tracer
analysis = Contour2D(grid, b,
                     dims={'X':'XC','Z':'Z'},
                     dimEq={'Z':'Z'},
                     increase=increase,
                     lt=lt)
# evenly-spaced contours
ctr = analysis.cal_contours(N)

# Mask for A(q) relation table.
# This can be done analytically in simple case, but we choose to do it
# numerically in case there are undefined values inside the domain.
mask = ds.maskC

# calculate related quantities for
table = analysis.cal_area_eqCoord_table(mask) # A(Yeq) table
area  = analysis.cal_integral_within_contours(ctr).rename('intArea')
ZEq   = table.lookup_coordinates(area).rename('ZEq')

#####################  interpolat to equivalent latitudes  ###########################
# combined the results
ds_contour = xr.merge([ctr, area, ZEq])

# interpolate from contour space to original latitudes
preZs = ds.Z.astype(dtype)
# results in latEq space
ds_ZEq = analysis.interp_to_dataset(preZs, ZEq, ds_contour)
```

    case 1: decrease & lt

When we get the sorted profile of buoyancy, we can calculate the local APE density.

```python
#%% calculate local finite-amplitude wave activity
lape, contours, masks = analysis.cal_local_APE(b,          # buoyancy
                                            ds_ZEq.b,      # the sorted buoyancy
                                            mask_idx=[8,28,51,81])   # select these mask to see the area of LAPE
```

Now we can take a look at the results.

```python
import proplot as pplt

fig, axes = pplt.subplots(nrows=3, ncols=2, figsize=(10, 8.5), sharex=3, sharey=3)

fontsize = 12

def plot_time(tidx, add_colorbar=False):
    ax = axes[tidx, 0]
    m1=ax.contourf(-lape.where(mask)[tidx]*1e4, levels=np.linspace(0,50,26), cmap='reds') # minus sign to ensure positive definite
    ax.contour(b.where(mask)[tidx], levels=11, cmap='viridis', lw=0.8)
    if add_colorbar:
        ax.colorbar(m1, loc='b', ticks=5, label='')
    ax.set_title('buoyancy and local APE density (t={})'.format(tidx), fontsize=fontsize)
    ax.set_xlabel('x-coordinate (m)', fontsize=fontsize-1)
    ax.set_ylabel('z-coordinate (m)', fontsize=fontsize-1)
    ax.set_xticks([0, 2000, 4000, 6000, 8000])
    ax.set_yticks([-200, -150, -100, -50, 0])
    ax.set_ylim([-200, 0])

    ax = axes[tidx, 1]
    msk = sum(masks)
    lev = xr.concat(contours, 'Z').isel(time=tidx).values
    m1=ax.pcolormesh(msk.where(msk!=0)[tidx], cmap='bwr')
    ax.contour(b.where(mask)[tidx], levels=lev, lw=0.8, color='k')
    if add_colorbar:
        ax.colorbar(m1, loc='b', ticks=1, label='')
    ax.set_title('masks for local APE calculation', fontsize=fontsize)
    ax.set_xlabel('x-coordinate (m)', fontsize=fontsize-1)
    ax.set_ylabel('z-coordinate (m)', fontsize=fontsize-1)
    ax.set_xticks([0, 2000, 4000, 6000, 8000])
    ax.set_yticks([-200, -150, -100, -50, 0])
    ax.set_ylim([-200, 0])

plot_time(0)
plot_time(1)
plot_time(2, True)

axes.format(abc='(a)')
```

It is clear that local APE density is all positive, and large local APE centers are located at regions where contours greatly deviate from its equilibrant places.  However, it is shown that these large centers are not corresponding to internal-wave breaking regions where contours are greatly stretched and folded (zigzag).  Therefore, large local APE density is not a region of significant irreversible mixing.

#### References
1. Holliday, D., and M. E. McIntyre, 1981: [On potential energy density in an incompressible, stratified fluid](https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/on-potential-energy-density-in-an-incompressible-stratified-fluid/75CBC2DE788C9D22EA211BE91C4F2167). *J. Fluid Mech.*, **107**, 221-225.
2. Huang, C. S. Y., and N. Nakamura, 2016: [Local finite-amplitude wave activity as a diagnostic of anomalous weather events](https://journals.ametsoc.org/view/journals/atsc/73/1/jas-d-15-0194.1.xml). *J. Atmos. Sci.*, **73**, 211-229.
3. Roullet, G., and P. Klein, 2009: [Available potential energy diagnosis in a direct numerical simulation of rotating stratified turbulence](https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/available-potential-energy-diagnosis-in-a-direct-numerical-simulation-of-rotating-stratified-turbulence/906E71577A230B90B48D8CEB442EE873). *J. Fluid Mech.*, **624**, 45-55.


```python
%matplotlib inline
from IPython.display import display, Math
import sympy as sp
import numpy as np
from sympy import init_printing
init_printing() # doctest: +SKIP
from mueller_matrices import M_Retarder
```

## Sample: linear retarder with specified transmission and degree of polarization

#### Sample retardance: $\delta$, fast axis:$\theta$, slow axis: $\phi = \theta + \pi/2$

```python
delta, phi = sp.symbols('delta, phi')
M_retarder = M_Retarder(phi - sp.pi/2, delta)

M_retarder
```

#### Sample may also have transmission $t$ and degree of polarization $p$

```python
t, p = sp.symbols('t, p')
M_tp = sp.Matrix([[t, 0, 0, 0], [0, t*p, 0, 0], [0, 0, t*p, 0], [0, 0, 0, t*p]])
M_sample = M_tp * M_retarder

M_sample
```

## Illumination: Stokes vectors

```python
S_unp = sp.Matrix([[1, 0, 0, 0]]).T
S_rcp = sp.Matrix([[1, 0, 0, 1]]).T
S_lcp = sp.Matrix([[1, 0, 0, -1]]).T
S = sp.Matrix([sp.symbols('s0:4')]).T
```

## model 1: illumination by right circularly polarized light

The microscope measures the following Stokes vector of ligth $S_{sample}= \left< s_0, s_1, s_2, s_3 \right>$ after interacting with the sample.

```python
S_sample = M_sample * S_rcp

S_sample
```

From the above Stokes vector, physical properties can be computed as follows.

```python
transmission = S[0]
slow_axis = 0.5 * sp.atan(-S[1]/S[2])
retardance = sp.atan(sp.sqrt(S[1]**2+S[2]**2)/S[3])
polarization = sp.sqrt(S[1]**2+S[2]**2+S[3]**2)/S[0]

display(Math('transmission = '+sp.latex(transmission)))
display(Math('slow\, axis = '+sp.latex(slow_axis)))
display(Math('retardance = '+sp.latex(retardance)))
display(Math('polarization = '+sp.latex(polarization)))
```

$\displaystyle transmission = s_{0}$

$\displaystyle slow\, axis = - 0.5 \operatorname{atan}{\left(\frac{s_{1}}{s_{2}} \right)}$

$\displaystyle retardance = \operatorname{atan}{\left(\frac{\sqrt{s_{1}^{2} + s_{2}^{2}}}{s_{3}} \right)}$

$\displaystyle polarization = \frac{\sqrt{s_{1}^{2} + s_{2}^{2} + s_{3}^{2}}}{s_{0}}$

## model 2: background as ellitpical illumination.

We model the background as elliptial illumination. Experimentally, this is the polarization state in the background where no specimen is present.

```python
S_bg = sp.Matrix([sp.symbols('s_0:4^{bg}')]).T
S_out = sp.Matrix([sp.symbols('s_0:4^{out}')]).T
```

```python
S_measured = M_sample * S_bg

display(Math(sp.latex(S_out)+'='+sp.latex(M_sample * S_bg)))
```

$\displaystyle \left[\begin{matrix}s_0^{out}\\s_1^{out}\\s_2^{out}\\s_3^{out}\end{matrix}\right]=\left[\begin{matrix}s_0^{bg} t\\p s_1^{bg} t \left(\sin^{2}{\left(2 \phi \right)} \cos{\left(\delta \right)} + \cos^{2}{\left(2 \phi \right)}\right) + p s_2^{bg} t \left(1 - \cos{\left(\delta \right)}\right) \sin{\left(2 \phi \right)} \cos{\left(2 \phi \right)} + p s_3^{bg} t \sin{\left(\delta \right)} \sin{\left(2 \phi \right)}\\p s_1^{bg} t \left(1 - \cos{\left(\delta \right)}\right) \sin{\left(2 \phi \right)} \cos{\left(2 \phi \right)} + p s_2^{bg} t \left(\sin^{2}{\left(2 \phi \right)} + \cos{\left(\delta \right)} \cos^{2}{\left(2 \phi \right)}\right) - p s_3^{bg} t \sin{\left(\delta \right)} \cos{\left(2 \phi \right)}\\- p s_1^{bg} t \sin{\left(\delta \right)} \sin{\left(2 \phi \right)} + p s_2^{bg} t \sin{\left(\delta \right)} \cos{\left(2 \phi \right)} + p s_3^{bg} t \cos{\left(\delta \right)}\end{matrix}\right]$

### 2a. If S_in = S_rcp

```python
display(Math(sp.latex(S_out)+'='+sp.latex(S_measured.subs({'s_0^{bg}':1,'s_1^{bg}':0,'s_2^{bg}':0,'s_3^{bg}':1}))))
```

$\displaystyle \left[\begin{matrix}s_0^{out}\\s_1^{out}\\s_2^{out}\\s_3^{out}\end{matrix}\right]=\left[\begin{matrix}t\\p t \sin{\left(\delta \right)} \sin{\left(2 \phi \right)}\\- p t \sin{\left(\delta \right)} \cos{\left(2 \phi \right)}\\p t \cos{\left(\delta \right)}\end{matrix}\right]$

### 2b. If sample retardance is small, it is possible to correct for large background

use Taylor series expansion of $cos(\delta)$

$cos(\delta)=1-\frac{x^2}{2!}+\frac{x^4}{4!}+...$

```python
S2 = S_measured.subs({sp.cos(delta):1,p:1})
display(Math(sp.latex(S_out)+'='+sp.latex(sp.trigsimp(S2))))
```

$\displaystyle \left[\begin{matrix}s_0^{out}\\s_1^{out}\\s_2^{out}\\s_3^{out}\end{matrix}\right]=\left[\begin{matrix}s_0^{bg} t\\s_1^{bg} t + s_3^{bg} t \sin{\left(\delta \right)} \sin{\left(2 \phi \right)}\\s_2^{bg} t - s_3^{bg} t \sin{\left(\delta \right)} \cos{\left(2 \phi \right)}\\- s_1^{bg} t \sin{\left(\delta \right)} \sin{\left(2 \phi \right)} + s_2^{bg} t \sin{\left(\delta \right)} \cos{\left(2 \phi \right)} + s_3^{bg} t\end{matrix}\right]$

### $t=\frac{s_0^{out}}{s_0^{bg}}$

Now, normalize the LHS by $s_0^{out}$ and the RHS by $s_0^{bg}t$ and define  $s_{*n}=s_{*}/s_0$

```python
S2n = S2.subs({'s_0^{bg}':1,'t':1,'s_1^{bg}':'s_{1n}^{bg}','s_2^{bg}':'s_{2n}^{bg}','s_3^{bg}':'s_{3n}^{bg}'})
S_outn = S_out.subs({'s_0^{out}':1,'s_1^{out}':'s_{1n}^{out}','s_2^{out}':'s_{2n}^{out}','s_3^{out}':'s_{3n}^{out}'})
display(Math(sp.latex(S_outn)+'='+sp.latex(sp.trigsimp(S2n))))
```

$\displaystyle \left[\begin{matrix}1\\s_1^{out}\\s_2^{out}\\s_3^{out}\end{matrix}\right]=\left[\begin{matrix}1\\s_1^{bg} + s_3^{bg} \sin{\left(\delta \right)} \sin{\left(2 \phi \right)}\\s_2^{bg} - s_3^{bg} \sin{\left(\delta \right)} \cos{\left(2 \phi \right)}\\- s_1^{bg} \sin{\left(\delta \right)} \sin{\left(2 \phi \right)} + s_2^{bg} \sin{\left(\delta \right)} \cos{\left(2 \phi \right)} + s_3^{bg}\end{matrix}\right]$

$\phi=\frac{1}{2}\arctan(-\frac{s_{1n}^{out}-s_{1n}^{bg}}{s_{2n}^{out}-s_{2n}^{bg}})$

$\delta=\arcsin(\frac{\sqrt{(s_{1n}^{out}-s_{1n}^{bg})^2+(s_{2n}^{out}-s_{2n}^{bg})^2}}{s_{3n}^{bg}})$

## model 3: background as non-ideal linear retarder

```python
delta_bg, phi_bg = sp.symbols('delta_bg, phi_bg')

M_bg = M_Retarder(phi_bg - sp.pi/2, delta_bg)
M_bg
```

#### How does changing the order of background retarder affect the output Stokes vector?

```python
S_bgBefore = M_sample * M_bg * S_rcp
display(Math(sp.latex(S_bgBefore)))
```

$\displaystyle \left[\begin{matrix}t\\- p t \left(1 - \cos{\left(\delta \right)}\right) \sin{\left(\delta_{bg} \right)} \sin{\left(2 \phi \right)} \cos{\left(2 \phi \right)} \cos{\left(2 \phi_{bg} \right)} + p t \left(\sin^{2}{\left(2 \phi \right)} \cos{\left(\delta \right)} + \cos^{2}{\left(2 \phi \right)}\right) \sin{\left(\delta_{bg} \right)} \sin{\left(2 \phi_{bg} \right)} + p t \sin{\left(\delta \right)} \sin{\left(2 \phi \right)} \cos{\left(\delta_{bg} \right)}\\p t \left(1 - \cos{\left(\delta \right)}\right) \sin{\left(\delta_{bg} \right)} \sin{\left(2 \phi \right)} \sin{\left(2 \phi_{bg} \right)} \cos{\left(2 \phi \right)} - p t \left(\sin^{2}{\left(2 \phi \right)} + \cos{\left(\delta \right)} \cos^{2}{\left(2 \phi \right)}\right) \sin{\left(\delta_{bg} \right)} \cos{\left(2 \phi_{bg} \right)} - p t \sin{\left(\delta \right)} \cos{\left(\delta_{bg} \right)} \cos{\left(2 \phi \right)}\\- p t \sin{\left(\delta \right)} \sin{\left(\delta_{bg} \right)} \sin{\left(2 \phi \right)} \sin{\left(2 \phi_{bg} \right)} - p t \sin{\left(\delta \right)} \sin{\left(\delta_{bg} \right)} \cos{\left(2 \phi \right)} \cos{\left(2 \phi_{bg} \right)} + p t \cos{\left(\delta \right)} \cos{\left(\delta_{bg} \right)}\end{matrix}\right]$

```python
sp.trigsimp(S_bgBefore.subs({sp.cos(delta_bg):1,sp.sin(delta_bg):delta_bg}))
```

```python
S_bgAfter =  M_bg * M_sample * S_rcp
S_bgAfter
```

```python
S_bgAfter.subs({sp.cos(delta_bg):1,sp.sin(delta_bg):0})
sp.trigsimp(S_bgAfter.subs({sp.cos(delta_bg):1,sp.sin(delta_bg):delta_bg}))
```

#### TODO: How does depolarization affect the accuracy of background correction

#### TODO: Inaccuracy in recovery of retardance and slow axis as background retardance increases


```
import numpy as np
import scipy as sp
import sympy
import matplotlib.pyplot as plt
import matplotlib as mpl

```

```
# activate pop up plots
#%matplotlib qt
# or change to inline plots
%matplotlib inline
```

```
# some sample data
x = np.arange(0,10,0.1)
```

```
page_width_cm = 13
dpi = 200
inch = 2.54 # inch in cm
# setting global plot configuration using the RC configuration style
plt.rc('font', family='serif')
plt.rc('xtick', labelsize=12) # tick labels
plt.rc('ytick', labelsize=20) # tick labels
plt.rc('axes', labelsize=20)  # axes labels
# If you don’t need LaTeX, don’t use it. It is slower to plot, and text
# looks just fine without. If you need it, e.g. for symbols, then use it.
#plt.rc('text', usetex=True) #<- P-E: Doesn't work on my Mac
```

```
# create a figure instance, note that figure size is given in inches!
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(8,6))
# set the big title (note aligment relative to figure)
fig.suptitle("suptitle 16, figure alignment", fontsize=16)

# actual plotting
ax.plot(x, x**2, label="label 12")

# set axes title (note aligment relative to axes)
ax.set_title("title 14, axes alignment", fontsize=14)

# axes labels
ax.set_xlabel('xlabel 12')
ax.set_ylabel(r'$y_{\alpha}$ 12', fontsize=8)

# legend
ax.legend(fontsize=12, loc="best")

##############################################################

##Ploting Grid
plt.grid()
#Changing Legend
ax.legend(loc=4)

ax.plot(x, x**2,color="purple", lw=1, ls='-', marker='s',markevery=5, markersize=8,markerfacecolor="yellow", markeredgewidth=2, markeredgecolor="blue", label="label 12")

#Subplot
#SubPlot
fig, ax1 = plt.subplots(nrows=1, ncols=1, figsize=(8,6))
ax1.plot(x, x**3, label="label 12")
ax1.set_title("Sub-plotting", fontsize=14)

# saving the figure in different formats
fig.savefig('figure-%03i.png' % dpi, dpi=dpi)
fig.savefig('figure.svg')
fig.savefig('figure.eps')
```

```
### Countour Plotting

import math
from pylab import *

alpha = 0.7
phi_ext = 2 * math.pi * 0.5

def flux_qubit_potential(phi_m, phi_p):
    return 2 + alpha - 2 * np.cos(phi_p)*np.cos(phi_m) - alpha * np.cos(phi_ext - 2*phi_p)

phi_m = np.linspace(0, 2*math.pi, 100)
phi_p = np.linspace(0, 2*math.pi, 100)
X,Y = np.meshgrid(phi_p, phi_m)
Z = flux_qubit_potential(X, Y).T

fig, ax = plt.subplots()

cnt = contour(Z, cmap=cm.RdBu, vmin=abs(Z).min(), vmax=abs(Z).max(), extent=[0, 1, 0, 1])

# Second Axis
ax2 = ax.twinx()

ax2.set_ylabel(r"Temperature ($^\circ$C)")
ax2.set_ylim(0, 35)

#Add Line
axvline(x=0.5, ymin=0, ymax=1)
axhline(y=25,xmin=0,xmax=1)

#autoformat dates for nice printing on the x-axis using fig.autofmt_xdate()
fig.autofmt_xdate(bottom=0.2, rotation=30, ha='right')
```

# Advanced exercises
We are going to play a bit with regression
-Create a vector x of equally spaced number between x∈[0,5π] of 1000 points (keyword: linspace)

-create a vector y, so that y=sin(x) with some random noise

-plot it

-Try to do a polynomial fit on y(x) using numpy.polyfit, with different polynomial degree

```

x= np.linspace(0, 5*math.pi, 1000)

Y= np.sin(x)
noise = np.random.normal(-0.1,0.1,1000)

y=Y+noise

fig, ax1 = plt.subplots(nrows=1, ncols=1, figsize=(8,6))
fig.suptitle("Sin + Noise", fontsize=16)
ax1.plot(x, y,color="black",lw=2, ls='*', marker='.', label="Sin(x) + Noise")

plt.plot(x,Y,color="red", linewidth=2.00,label="Sin(x)")

for i in [2,4,6,8,10]:
    p = np.poly1d(np.polyfit(x,y,i))
    plt.plot(x,p(x), linewidth=2.00,label="Fit order: "+str(i))

ax1.legend(fontsize=12, loc="best")

```

```

```


# Quantum Harmonic Osscilator
### Christina Lee
### Prerequisites: Quantum Mechanics
Slinkies. They started out as toys.  I still have one to play with on my desk.
Rubber bands What was once something useful, is now a wonderful projectile weapon.
 Swings I still love them, but people seem to not make them in adult sizes for some reason.

A person's perception of these objects start to change as they enter their first physics class.  Even in that beginning classical mechanics, the problems are filled with harmonic osscilators, like slinkies, rubber bands, or swings, which exert a force proportional to their displacement
\begin{equation}
F=-kx
\end{equation}
and therefore a quadratic potential
\begin{equation}
V(x)=\frac{1}{2} k x^2
\end{equation}

This is all extremely fun and useful in the classical regime, but we add Quantum Mechanics to the mix, and LOW AND BEHOLD! we have one of the few exactly solvable models in Quantum Mechanics. Moreso, this solution demonstrates some extremely important properties of quantum mechanical systems.

##### The Hamiltonian
\begin{equation}
    {\cal H}= \frac{p^2}{2 m} + \frac{1}{2} m \omega ^2 x^2
\end{equation}

##### The Solution
\begin{equation}
    \Psi (x) = \frac{1}{\sqrt{2^n n!}} \left(\frac{m \omega}{\hbar \pi}\right)^{1/4} \mathrm{e}^{-m \omega x^2/2 \hbar}  H_n \left( \sqrt{\frac{m \omega}{\hbar}} x \right)
\end{equation}
Today, I just intend to present the form of the solution, calculate this equation numerically, and visualize the results.  If you wish to know how the equation is derived, you can find a standard quantum mechanics textbook, or stay tuned till I manage to write it up.

### Physicists' Hermite Polynomials
Note: These are not the same as the "probabilists' Hermite Polynomial".  The two functions differ by scaling factors.

Physicists' Hermite polynomials are defined as eigenfunctions for the differential equation
\begin{equation}
u^{\prime \prime}-2xu^{\prime} = -2 \lambda u
\end{equation}

\begin{equation}
H_n(x) = (-1)^n \mathrm{e}^{x^2} \frac{\mathrm{d}^n}{\mathrm{d}x^n}
\left( e^{-x^2} \right)
\end{equation}

I leave it as an exercise to the reader (muahahahaha) to
* demonstrate othogonality with respect to the measure $e^{-x^2}$, ie
\begin{equation}
\int_{-\infty}^{\infty} H_m(x) H_n(x) e^{-x^2} \mathrm{d}x = \sqrt{\pi} 2^n n! \delta_{mn}
\end{equation}
* demonstrate completeness.  This means we can describe every function by a linear combination of Hermite polynomials, provided it is suitably well behaved.

Though a formula exists or calculating a function at n directly, the most efficient method at low n for calculating polynomials relies on recurrence relationships.  These recurrence relationships will also be quite handy if you ever need to show orthogonality, or expectation values.
##### Recurrence Relations
\begin{equation}
H_{n+1}(x) = 2xH_n(x) - H^{\prime}_n(x)
\end{equation}
\begin{equation}
H^{\prime}_n(x) = 2n H_{n-1}(x)
\end{equation}
\begin{equation}
H_{n+1}(x) = 2x H_n(x) - 2n H_{n-1}(x)
\end{equation}

```julia
using Pkg
Pkg.add("Roots")
using Roots;
using Plots
gr()
```

      Updating registry at `~/.julia/registries/General`
      Updating git-repo `https://github.com/JuliaRegistries/General.git`
    [1mFetching: [========================================>]  100.0 %.0 % Resolving package versions...
      Updating `~/.julia/environments/v1.0/Project.toml`
      [f2b01f46] + Roots v0.8.1
      Updating `~/.julia/environments/v1.0/Manifest.toml`
     [no changes]

    ┌ Info: Recompiling stale cache file /home/shaula/.julia/compiled/v1.0/Roots/o0Xsi.ji for Roots [f2b01f46-fcfa-551c-844a-d8ac1e96c665]
    └ @ Base loading.jl:1187

    Plots.GRBackend()

#### Programming Tip!
Since Hermite polynomials are generated recursively, I wanted to generate and save all the functions up to a designated value at once.  In order to do so, I created an array, whose values are anonymous functions.

```julia
function GenerateHermite(n)
    Hermite=Function[]

    push!(Hermite,x->1);
    push!(Hermite,x->2*x);

    for ni in 3:n
        push!(Hermite,x->2*x.*Hermite[ni-1](x)-2*n.*Hermite[ni-2](x))
    end
    return Hermite
end
```

    GenerateHermite (generic function with 1 method)

So lets generate some Hermite polynomials and look at them.
 Make sure you don't call a Hermite you haven't generated yet!

```julia
Hermite=GenerateHermite(5)
```

    5-element Array{Function,1}:
     getfield(Main, Symbol("##3#6"))()
     getfield(Main, Symbol("##4#7"))()
     getfield(Main, Symbol("##5#8")){Int64,Array{Function,1},Int64}(5, Function[##3#6(), ##4#7(), ##5#8{Int64,Array{Function,1},Int64}(#= circular reference @-2 =#), ##5#8{Int64,Array{Function,1},Int64}(5, Function[#= circular reference @-2 =#], 4), ##5#8{Int64,Array{Function,1},Int64}(5, Function[#= circular reference @-2 =#], 5)], 3)
     getfield(Main, Symbol("##5#8")){Int64,Array{Function,1},Int64}(5, Function[##3#6(), ##4#7(), ##5#8{Int64,Array{Function,1},Int64}(5, Function[#= circular reference @-2 =#], 3), ##5#8{Int64,Array{Function,1},Int64}(#= circular reference @-2 =#), ##5#8{Int64,Array{Function,1},Int64}(5, Function[#= circular reference @-2 =#], 5)], 4)
     getfield(Main, Symbol("##5#8")){Int64,Array{Function,1},Int64}(5, Function[##3#6(), ##4#7(), ##5#8{Int64,Array{Function,1},Int64}(5, Function[#= circular reference @-2 =#], 3), ##5#8{Int64,Array{Function,1},Int64}(5, Function[#= circular reference @-2 =#], 4), ##5#8{Int64,Array{Function,1},Int64}(#= circular reference @-2 =#)], 5)

#### Programming Tip!
Since the Hermite polynomials, and the wavefunctions after them, are composed on anonymous functions, we need to use `map(f,x)` in order to map the function `f` onto the array `x`.  Otherwise our polynomials only work on single values.

```julia
x=collect(-2:.01:2);
plot(ylims=(-50,50))
for j in 1:5
    plot!(x,map(Hermite[j],x),label="H_$j (x)")
end
plot!()
```

```julia
# Lets make our life easy and set all units to 1
m=1
ω=1
ħ=1

#Finally, we define Ψ
Ψ(n,x)=1/sqrt(factorial(n)*2^n)*(m*ω/(ħ*π))^(1/4)*exp(-m*ω*x^2/(2*ħ))*Hermite[n](sqrt(m*ω/ħ)*x)
```

    Ψ (generic function with 1 method)

### Finding Zeros
The eigenvalue maps to the number of zeros in the wavefunction.  Below, I use Julia's roots package to indentify roots on the interval from -3 to 3.

```julia
zeds=Array{Array{Float64}}(undef,1)
zeds[1]=[] #ground state has no zeros
for j in 2:4
    push!(zeds,fzeros(y->Ψ(j,y),-3,3))
end
```

```julia
zeds[3]
```

    2-element Array{Float64,1}:
     -1.5811388300841895
      1.5811388300841895

```julia
# AHHHHH! So Much code!
# Don't worry; it's all just plotting
x=collect(-3:.01:3)  #Set some good axes

plot(xlim=(-3,3),ylim=(-.5,4.5))
for j in 1:4    #how many do you want to view?
    plot!(x,map.(y->Ψ(j,y),x).+j.-1,label="| $j >")
    plot!(x,(j-1)*ones(length(x)),
        color="black",label="")
    scatter!(zeds[j],(j-1)*ones(length(zeds[j])),
        label="")
end

plot!(x,.5*m*ω^2*x.^2,label="Potential")

scatter!([],[],label="Zeros")

plot!(xlabel="x",ylabel="Ψ+n",
    title="Eigenstates of a Harmonic Osscilator")
```

## Example Result

## More to come
This barely scratched the surface into the richness that can be seen in the quantum harmonic osscilator.  Here, just we developed a way for calculating the functions, and visualized the results.  Stay tuned to hear here about ground state energy, ladder operators, and atomic trapping.

```julia

```


```julia
using DifferentialEquations
using Plots
```

Consider the simple reaction:

\begin{align}
    A &\longrightarrow B\\
\end{align}

The reaction is elementary, and B acts as a solvent for A and the transition state. We will consider it in a few different solvent environments.

```julia
function batch(du, u, p, t)
    MAR = p["MAR"]
    PAR = p["PAR"]

    k10, K10, V = PAR

    NA   = u[:,1]
    NB   = u[:,2]

    NT     = NA + NB

    XA   = NA./NT
    XB   = NB./NT

    #For A in B
    A12A = MAR[1]

    #For B in A
    A12B = MAR[2]

    #For Transition State 1 in B
    A12TS1 = MAR[3]

    gammaA  = exp.(XB.^2*A12A)
    gammaB  = exp.(XA.^2*A12B)
    gammaTS1 = exp.(XB.^2*A12TS1)

    z1 = 1/K10*gammaB./gammaA.*XB./XA

    r1 = k10*gammaA./gammaTS1.*XA.*(1 .- z1).*NT/V

    RA  = -r1[1]
    RB  =  r1[1]

    du[1] = RA*V
    du[2] = RB*V
    return du, r1
end
```

    batch (generic function with 1 method)

```julia
k10  = 1
K10  = 1
V    = 1

NA0  = 100
NB0  = 0.0

var0     = [NA0 NB0]
span     = (0.0, 25.0);
```

```julia
#Solvate transition state relative to reactants
MARSET1 = zeros(3,3)
MARSET1[:,1] = [0.0, 0.0, 0.0] #no solvation
MARSET1[:,2] = [0.0, 0.0, 2.0] #destabilize TS1
MARSET1[:,3] = [0.0, 0.0, -2.0] #stabilize TS1

tcoar  = range(0.0, stop = maximum(span), length = 20)
tfine  = range(0.0, stop = maximum(span), length = 1000)
Xout   = zeros(length(tfine), size(MARSET1, 2))
r1out  = zeros(length(tfine), size(MARSET1, 2))
NAout  = zeros(length(tfine), size(MARSET1, 2))
NBout  = zeros(length(tfine), size(MARSET1, 2))

for i = 1:size(MARSET1, 2)
    p0   = Dict("MAR" => MARSET1[:,i], "PAR" => [k10, K10, V])
    prob = ODEProblem(batch, var0, span, p0)
    sol  = solve(prob, Rodas5(), abstol = 1e-18, reltol = 1e-18)
    solf = sol(tfine)
    NA   = solf[1,:]
    NB   = solf[2,:]
    NT   = NA + NB
    ext  = NB/NA0
    dut, rt1 = batch([0., 0.], [NA NB], p0, tfine)

    Xout[:,i]  = ext
    r1out[:,i] = rt1
    NAout[:,i] = NA
    NBout[:,i] = NB
end

Xout[Xout .>= 0.9999] .= 0.9999

plt1 = plot(tfine/maximum(tfine), Xout, xlabel  = "time", ylabel = "Conversion", labels = ["ideal" "destabilize TS" "stabilize TS"], legend = :bottomright)
plt1 =scatter!(plt1, tcoar/maximum(tcoar), 1 .- exp.(-k10*tcoar))
plt2 = plot(tfine/maximum(tfine), r1out, xlabel  = "time", ylabel = "rate", labels = ["ideal" "destabilize TS" "stabilize TS"], legend = :topright)
plt3 = plot(tfine/maximum(tfine), log.(1 .- Xout))
plt4 = plot(Xout, r1out)
r1out[r1out .<= 0.000001] .= 0.000001
plt5 = plot(log.(1 .- Xout), log.(r1out), legend = nothing)
# plt2 = plot!(tfine[2:end]/maximum(tfine), r1out[2:end,:], labels = ["r1" nothing nothing])
# plt3 = plot(Xout, r2out, xlabel  = "extent", ylabel = "rate", labels = ["r1" nothing nothing], legend = :topright)
# # plt3 = plot!(e2out, r2out, labels = ["r2" nothing nothing])
# plt4 = plot(Xout, z1out, xlabel  = "extent", ylabel = "z", labels = ["z1" nothing nothing], legend = :topright)
# plt4 = plot!(Xout, z2out, labels = ["z2" nothing nothing])
# plt5 = plot(tfine/maximum(tfine), z1out, xlabel  = "time", ylabel = "z", labels = ["z1" nothing nothing], legend = :topright)
# plt5 = plot!(tfine/maximum(tfine), z2out, labels = ["z2" nothing nothing])

display(plt1)
display(plt2)
display(plt3)
display(plt4)
display(plt5)
# plot(tfine/maximum(tfine), cov)
```


# Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

### Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = 2*(x) + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    'wget' n'est pas reconnu en tant que commande interne
    ou externe, un programme ex‚cutable ou un fichier de commandes.

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it deaccelerates slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(X, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(X, Y*15000000000000.)
plt.plot(X, Y, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =X/max(X)
ydata =Y/max(Y)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 4.978100, beta_2 = -0.004434

Now we plot our resulting regresssion model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here

# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

Double-click __here__ for the solution.

## Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: [SPSS Modeler](http://cocl.us/ML0101EN-SPSSModeler).

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at [Watson Studio](https://cocl.us/ML0101EN_DSX)

### Thanks for completing this lesson!

Notebook created by: Saeed Aghabozorgi

Copyright © 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/).