# Module 1: Setting up the problem

### Introduction

Geophysical surveys consist of a similar basic framework. An energy source is delivered into the earth, which can be natural (for example, the Earth's magnetic field) or human-made (current in the ground, acoustic wave energy, etc.), and this stimulates a response according to the variation in physical properties in the subsurface. At the surface, receivers pick up the a signal and record this as data.

The goal of inversion is to find a model of the physical property distribution in the earth that produced the data. This is a dificult process because (1) information about a physical property for each datum is encoded in a complex way, and (2) we have a finite amount of data and cannot represent the physical property distribution everywhere.

Inversion is a multistep process, often represented as a workflow *insert workflow image here*. The goal of this module is to cover the first section of the workflow, which discretizes the data and places the values of the functions onto a mesh. This will be done using the following steps:

(1) Start with an expression that relates a kernel function with the continuous distribution of a physical property.

(2) Discretize this expression, and introduce a simple example problem to illustrate the mathematics in detail.

(3) Define a mesh that organizes the data.

(4) Build up the matrix equation $d = Gm$.

(5) Generalize the form of the problem from the example.

(6) Implement the example problem in Python as a forward problem.

But first, here are some fundamental definitions:

The general mathematical description of the inverse problem can be written as follows:

\begin{equation} F_j[m]= d_j +n_j \quad \text{for} \quad j=1,..,N \; \text{where}\end{equation}

$\bf{F}$ is a foward modeling operator. $\bf{F}$ incorporates the survey design and the physics of the problem.

$\bf{m}$ is a generic symbol for a physical property distibution.

$\bf{d}$ represents the observed data, (sometimes also represented as $\bf{d^{obs}}$).

$\bf{n}$ is a term that represents additive noise.

This is, of course, the most general formuation of the problem. In this module we will consider the simplest case, which is (a) one dimensional (this can be likened to a survey that varies as a function of depth only) and (b) linear, in which our forward modeling operator becomes a matrix **G** in the matix equation:

\begin{equation}d = Gm\end{equation}

### Step 1: Physical property distribution and the kernel function.

Each datum ($d_i$) collected is a volumetric response, that is to say, every datum measures the response of the whole volume (within the range of the system); it records the superposition of effects caused by all the material in the ground, and is therefore naturally represented as an integral. A "kernel function" or "sensitivity function", $g(x,y,z)$, shows how a datum is affected by all the subsurface. It describes the physics of the problem. The model, $m(x,y,z)$, represents the distribution of a physical property in the volume. Since each datum measures the response of the kernel function with the physical property distribution in the volume, for a continuous medium we can express this relationship as the inner product of the kernel function and the model. In the one dimensional case the expression for the *ith* datum is written as:

\begin{equation}d_i = \int_a^b g_i(x) m(x) dx \end{equation}

where again:
\begin{equation}
    d :=\text{measured data} \\
    g :=\text{kernel function} \\
    m :=\text{physical property}\\
\end{equation}

### Step 2: Discretize the expression

While the integral expression describes the inner product in a continuum, it is not possible to maintain this representation in our data space, so the above equation must be discretized. Let us consider the case where we have $N$ data, and as we build up the matrix equation, let $i$ be the rows and $j$ be the columns of our matrix:

\begin{equation}d_i = \int_a^b g_i(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_i (x_j) m_j \Delta x\end{equation}

**A "Toy Problem"**

Consider a simple case where we have a one dimensional, linear problem that will generate two data points, $d_1$ and $d_2$,from five physical property values
 ($m_1, m_2, m_3, m_4, m_5$), and these two data points are generated using two kernel functions $g_1$ and $g_2$. Further, let us assume that our domain of interest lies on the interval [0,1]. The equation above is then expressed as the following two equations, one for each datum:

\begin{equation}d_1 = \int_0^1 g_1(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_1 (x_j) m_j \Delta x\end{equation}

\begin{equation}d_2 = \int_0^1 g_2(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_2 (x_j) m_j \Delta x\end{equation}

Given that our problem is small by design, it is instructive to visualize the summation notation as matrix-vector products. Doing so yields the following expressions:

(a) Data. We can collect our two data points into a column vector. Our data in vector notation is given by:

\begin{equation}
d = \left[
\begin{array}{c}
d_1\\
d_2
\end{array}
\right]
\end{equation}

(b) The x-spacing. The x-spacings, $\Delta x$, are represented by a diagonal matrix. Here we are in a one dimensional case, but in general our data will reside in a volume, so let this matrix be represented as $V$. In the most general situation, the x-spacings need not be equal, and there can be significant variation in the distances within a grid, depending on the amount of resolution one desires at a particular location. For the moment, let's ignore this complexity and assume equal spacing in our grid. Then let $V=diag(\Delta x)$, and given the dimensions of our problem, $V$ appears as follows:

\begin{equation}
V=\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\end{equation}

(c) The model. The
model, $m$, is a column vector with five rows:
\begin{equation}
m=
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

(d) The kernel functions. Recall that a kernel function shows how a datum is affected by all the subsurface. The full expression for the kernel function will be developed further in the next sections, but for the moment, let us put each kernel function on a row of a matrix, such that $g_1$ will be on row 1, and $g_2$ on row 2, and define this matrix as $\widetilde{G}$. This will yield the matrix:
\begin{equation}
\widetilde{G}=
\begin{bmatrix}
    g_{11} &g_{12} &g_{13} &g_{14} &g_{15}\\
    g_{21} &g_{22} &g_{23} &g_{24} &g_{25}
\end{bmatrix}
\end{equation}

Using the above, the assembled matrix vector forms for these equations becomes:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\begin{bmatrix}
    g_{11} &g_{12} &g_{13} &g_{14} &g_{15}\\
    g_{21} &g_{22} &g_{23} &g_{24} &g_{25}
\end{bmatrix}
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

Or to put it succinctly, $d = \widetilde{G} \, diag(\Delta x)\, m$.

### Step 3: Set up the mesh

Now that the data has ben discretized, lets look at where the data is to be placed in our data space. First, subdivide the one dimensional domain into cell centers and nodes, with spacings of $\Delta x$:

For convenience (and for applications that will be discussed in later modules), the model values reside in the cell centers, while the values for the kernel functions reside on the nodes.

Note that the kernel function in our example (represented above) has six values and the model values are five; moreover, the x-coordinates where $g(x) $  and $m(x)$ are evaluated are not coincident, and this leads to a complication when we want to perform the inner product of **m** and **g**. What is needed is a way to evaluate the kernel functions at the cell centers. To do this, we employ the trapeziodal rule for approximating integration. Simply put, to obtain the values of the kernel function on the cell centers, take the average of the kernel function values on the adjacent nodes. Let us then define two kernel function matrices, one with values evaluated on the cell centers, $G_c$ (represented by the black circles below), and another with the values evaluated on the nodes, $G_n$ (in white).

The relationship between $G_c $ and $G_n$ is an "averaging matix", $A_v$, such that $G_c = A_v G_n$. Putting each kernel function in the columns for both $G_c$ and $G_n$, and using again the dimensions of our toy problem, this relation appears as follows:

\begin{equation}
\begin{bmatrix}
    g_{c1}(x_1)    & g_{c2}(x_1)  \\
    g_{c1}(x_2)    & g_{c2}(x_2)  \\
    g_{c1}(x_3)    & g_{c2}(x_3)  \\
    g_{c1}(x_4)    & g_{c2}(x_4)  \\
    g_{c1}(x_5)    & g_{c2}(x_5)  \\
\end{bmatrix}
=
\frac{1}{2}
\begin{bmatrix}
    1 & 1 & 0 & 0 & 0 & 0\\
    0 & 1 & 1 & 0 & 0 & 0\\
    0 & 0 & 1 & 1 & 0 & 0\\
    0 & 0 & 0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
    g_{n1}(x_1)    & g_{n2}(x_1)  \\
    g_{n1}(x_2)    & g_{n2}(x_2)  \\
    g_{n1}(x_3)    & g_{n2}(x_3)  \\
    g_{n1}(x_4)    & g_{n2}(x_4)  \\
    g_{n1}(x_5)    & g_{n2}(x_5)  \\
    g_{n1}(x_6)    & g_{n2}(x_6)  \\
\end{bmatrix}
\end{equation}

Meanwhile, the relationship between $G_c$ and $\widetilde{G}$ in step 2 is such that $G_c = \widetilde{G}^T$.

### Step 4: Build up the matrix equation $d=Gm$¶

We now have all the required building blocks to assemble the matrix equation. At the end of step 2 we arrived at $d = \widetilde{G} \, diag(\Delta x)\, m$, and from the previous step we have $G_c = \widetilde{G}^T = A_v G_n$. Put more orderly, $\widetilde{G} = (A_v G_n)^T$, so substuting this into our expression gives:

\begin{equation}
d = (A_v G_n)^T \, diag(\Delta x)\, m
\end{equation}

If we group matrices together and let $G = (A_v G_n)^T diag(\Delta x)$ then we arrive at $d=Gm$, our desired form. Referring again to our example, this would appear as follows:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\frac{1}{2}
\left(
\begin{bmatrix}
    1 & 1 & 0 & 0 & 0 & 0\\
    0 & 1 & 1 & 0 & 0 & 0\\
    0 & 0 & 1 & 1 & 0 & 0\\
    0 & 0 & 0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
    g_{n1}(x_1)    & g_{n2}(x_1)  \\
    g_{n1}(x_2)    & g_{n2}(x_2)  \\
    g_{n1}(x_3)    & g_{n2}(x_3)  \\
    g_{n1}(x_4)    & g_{n2}(x_4)  \\
    g_{n1}(x_5)    & g_{n2}(x_5)  \\
    g_{n1}(x_6)    & g_{n2}(x_6)  \\
\end{bmatrix}
\right)^T
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

### Step 5: Generalize the form of the problem

In the above case we had two data points and five model values. We can generalize this to larger data sets easily. In the case where we have $M$ measured data and $N$ model values, we obtain a matrix equation of the following dimensions:

\begin{equation}
(M \times 1) = [(N \times (N+1)) \; ((N+1) \times M)]^T (N  \times N) (N \times 1)\\
d \qquad = \qquad \qquad  \qquad \quad   [A_v G_n]^T  \qquad \qquad diag(\Delta x) \quad m
\end{equation}

### Step 6: Implement the example problem in Python.

Next, we will build up our matrices one by one in Python. First import the SimPEG and numpy packages:

```
from SimPEG import *
from IPython.html.widgets import interactive
import numpy as np
```

Let start by formulating the forward problem, that is, let us assume that we have our model values $m$ and seek to generate synthetic data $d$. Recall that we are building up the matrix equation: $d = (A_v G_n)^T \, diag(\Delta x)\, m$, so let's start with the right hand side and go over each matrix, one by one.

(1) The $A_v$ matrix. We can build the "averaging matrix" as follows:

```
N=5              # RecalL is the number of data we have in m.
n=N+1            # Define n as the N+1 dimension of the matrix
w=n-1            # Define w and the n-1  dimentsion of the matix
s = (N,n)        # Store matrix values
Av = np.zeros(s) # Create a matrix of zeros of the correct dimensions
                 # and fill in with elements usin the loop below (note the 1/2 is included in here).
for i in range(N):
    j=i
    k=i+1
    Av[i,j] = 0.5
    Av[i,k] = 0.5
print Av
```

    [[ 0.5  0.5  0.   0.   0.   0. ]
     [ 0.   0.5  0.5  0.   0.   0. ]
     [ 0.   0.   0.5  0.5  0.   0. ]
     [ 0.   0.   0.   0.5  0.5  0. ]
     [ 0.   0.   0.   0.   0.5  0.5]]

(2) The kernel matrix on the nodes, $G_n$. To build this matrix, we must first define a sensitivity function that describes some physical phenomenon. For the sake of our example, even though it has no direct, physical meaning, let us define our kernel function for the toy problem as follows:

\begin{equation}
g_j(x) = e^{jpx} \cos(2 \pi j q x)
\end{equation}

```
g = lambda x, k, p, q: np.exp(-p*k*x)*np.cos(2*np.pi*q*k*x)   # create an anonymous function as immediately above
x=np.linspace(0,1,6)                                          # define the nodes of our x-array
#x = np.array([0., 0.2, 0.4, 0.6, 0.8, 1.])
p = 0.01                                                      # Set values for p, q, j
q = 0.1
j = np.array([1, 2])
Gn = np.zeros((len(x), len(j)))                               # preallocate a matrix Gn, and evaluate functions in loop below.

for i in range(len(j)):
    f = g(x,k[i],p,q)
    Gn[:,i] = f
#print f
print Gn
```

    [[ 1.          1.        ]
     [ 0.99013245  0.96471657]
     [ 0.96471657  0.86932419]
     [ 0.92421453  0.72027328]
     [ 0.86932419  0.52732179]
     [ 0.80096714  0.30289805]]

(3) The volume matrix $V$. This simply consists of making an $N \times N$ array of x-spacings, $\Delta x.$

```
# make the delta x array

Deltax = 0.2*np.ones(N) # set x-spacings
V = np.diag(Deltax)     # create diagonal matrix
print V

```

    [[ 0.2  0.   0.   0.   0. ]
     [ 0.   0.2  0.   0.   0. ]
     [ 0.   0.   0.2  0.   0. ]
     [ 0.   0.   0.   0.2  0. ]
     [ 0.   0.   0.   0.   0.2]]

(4) Input the model values, $m$. Given that we are making a forward problem, we assume these values as given, so we input ficticious values:

```
m = np.array([0.02, 0.05, 0.09, 0.07, 0.04])
print m
```

    [ 0.02  0.05  0.09  0.07  0.04]

(5) Generate our two data values. The remaining step is simply to perform the matix-vector multiplication :

```
d = np.dot(np.dot(np.transpose(np.dot(Av, Gn)), V),m)
print d

```

    [ 0.04999083  0.03946006]

```
# Put x array on cell centers as follow

x = np.array([0., 0.2, 0.4, 0.6, 0.8, 1.])
xc = 0.5*(x[1:] + x[0:-1])
print x
print xc
```

    [ 0.   0.2  0.4  0.6  0.8  1. ]
    [ 0.1  0.3  0.5  0.7  0.9]

This is just an extra cell with more equations if needed:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\frac{1}{2}
\left(
\begin{bmatrix}
    g_{n1}(x_1) + g_{n1}(x_2)    & g_{n2}(x_1) + g_{n2}(x_2)  \\
    g_{n1}(x_2) + g_{n1}(x_3)    & g_{n2}(x_2) + g_{n2}(x_3)   \\
    g_{n1}(x_3) + g_{n1}(x_4)    & g_{n2}(x_3) + g_{n2}(x_4)  \\
    g_{n1}(x_4) + g_{n1}(x_5)    & g_{n2}(x_4) + g_{n2}(x_5)  \\
    g_{n1}(x_5) + g_{n1}(x_6)    & g_{n2}(x_5) + g_{n2}(x_6)   \\
\end{bmatrix}
\right)^T
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}


## Convection / flow modelling (with underworld)

  - Intro to underworld - quick overview of the basics
  - Solvers: Global flow from local source of buoyancy or bc. (Why MG)
  - Analytic solutions in Underworld and how to use them to estimate / test
  - Scaling & Rayleigh number
  - Rayleigh number and Nusselt number
  - Meshes / interpolation ... checkpointing
  - Tracers / strain markers, analysis
  - Thermal / chemical convection and more

### The basics

Underworld is configured to solve a couple Stokes / advection-diffusion problem on a finite element mesh.

The basic tools we need are:

   - The mesh itself
   - Some unknowns living on the mesh
   - Some boundary conditions
   - A driving term for the flow
   - Constitutive terms (viscosity, elasticity, plasticity)
   - A way to solve the equations
   - Some means of analysing the result
   - A means of timestepping

### Equations

The dimensionless (indicated by primes) Navier-Stokes equation can be written

$$
	\frac{1}{\textrm{Pr}} \frac{D \mathbf{v}'  }{Dt' }  =
	\acute{\nabla}^2  \mathbf{v}'  -  \acute{\nabla} p' + \textrm{ Ra} T' \hat{\mathbf{z}}
$$

We will work in the infinite Prandtl number limit in which intertial terms vanish, and we will also assume incompressibility

$$
\nabla^2  \mathbf{v}  -  {\nabla} p = -\textrm{ Ra} T \hat{\mathbf{z}}
$$
$$
\nabla \cdot \mathbf{v} = 0
$$

The buoyancy force in the Stokes equation includes (but is not limited to) thermal variations which are themselves coupled to the solution of the Stokes equation through the thermal advection term.

$$
			\frac{\partial T}{\partial t} + \mathbf{v}\cdot \nabla T -
					\kappa \nabla^2 T = \frac{H}{C_p}
$$

### Hands on  - quasi-static problems

The simplest problem we can set up is a boundary driven flow

  - [Lid Driven Flow Notebook](020-LidDrivenFlow.ipynb)
     - Exercise: [Point Driving Velocity](021-Exercise-PointSource.ipynb)

The next thing to try is a buoyancy driven flow

  - [Thermally-driven flow](030-ThermalBuoyancy.ipynb)

Which works pretty much as expected - by adding an appropriate function to the buoyancy term in the Stokes equation template.

### Hands on - time-stepping

Let's look at how to make a time-dependent problem. We start by looking at
some passive tracers swirling around in a lid-driven flow.

   - [Lid Driven Flow with Tracers](040-LidDrivenFlowPlusTracers.ipynb)
      - Exercise (take away): Kellogg & Turcotte, 1990
      - Exercise: Streak plots

Nice, but now we should let the flow be buoyancy driven and allow the buoyancy to drive the flow.

   - [Rayleigh Bénard Convection](050-RayleighBenardConvection.ipynb)
   - [Rayleigh Bénard Convection - pt2 ](051-RayleighBenardConvection-pt2.ipynb)
       - Exercise (take away): Nu - Ra and Vrms - Ra plots (again)
   - [Rayleigh Bénard Convection - pt3](052-RayleighBenardConvection-pt3.ipynb)
       - Exercise (take away): Arrhenius rheology and non-linear viscosity

## What have we not covered ?

There are many aspects of the `underworld` code and of modelling in general that we have not dealt with here, or have only touched upon in a general way. Some of these issues apply to all the different numerical methods and tools you might come across.

   - **Where can I find out more ?**  _Python codes include 'docstrings' which can be quite helpful and you should always take a look. uw.help() grabs the docstrings for you and leaves out some of the developer-specific complexities._

   - **Treat the code like a black box, how do you know when to trust the output ?** _In `underworld`, we treat this question very seriously and we bundle a whole collection of analytic Stokes flow problems in with the code ( `uw.help(uw.fn.analytic)` )_

   - **What are the trade-offs between resolution, accuracy, choice of solvers, speed, parallel performance etc etc ?** _You probably want to find the answer(s) to that question out for yourself, I just want to make you aware that it is one of the key questions in modeling. Having analytic or approximate solutions to hand is very helpful in this case. Also seeing if the results **converge** to a specific answer as you put more resources into the solution._

   - **Particles and material history.** _One of the reasons `underworld` was written was to look at problems of complex fluids - ones where the material history is relevant to the constitutive behaviour (e.g. anisotropy, viscoelasticity, strain-softening ...). There are examples of these in the `underworld` user guide and tutorials._

   - **Parallel ?** _Most of the time, the python scripts will run fine in parallel, some of the more convoluted things we saw in the examples are the result of guiding you to parallel safe methods (e.g. the `uw.fn.view.maxmin` which works in parallel when numpy.max() does not). Also, the whole `gLucifer` visualisation framework is designed for use in parallel. Have a look at the examples, they all work in parallel._

   - **Saving data.** _You can save and read back all field and swarm variables and this all works in parallel. The file formats are hdf5 and these can be read later (in serial if you can't process your results easily in parallel). This is important as you may / will want to checkpoint and restart jobs that take a long time to run. **Warning:** you will have to make sure your scripts handle whatever branches arise from restarting as we cannot snapshot your scripts and save them._

   - **Workflow.** _There are many, many python packages available which will just work with underworld. For example, to allow you to set up complicated pattens of particles using projections, mapping data to material properties or comparing observations and model outputs. Many are already built-in to `scipy`. You might end up with some issues in parallel if you are not careful (e.g. `scipy`'s fft routines), but perhaps you can create a workflow in which this processing is handled in serial, saves hdf5 information, and is then used to initialise a parallel script._

```python
import underworld as uw
from underworld import function as fn
import glucifer
import numpy as np
```

```python
print help(uw)
```

    Help on package underworld:

    NAME
        underworld

    FILE
        /Users/rfarrington/Underworld/Underworld2-development/underworld/__init__.py

    DESCRIPTION
        Underworld2 is a python-friendly version of the Underworld geodynamics
        code which provides a programmable and flexible front end to all the
        functionality of the code running in a parallel HPC environment. This
        gives signficant advantages to the user, with access to the power of
        python libraries for setup of complex problems, analysis at runtime,
        problem steering, and coupling of multiple problems.

        Underworld2 is integrated with the literate programming environment of
        the jupyter notebook system for tutorials and as a teaching tool for
        solid Earth geoscience.

        Underworld is an open-source, particle-in-cell finite element code
        tuned for large-scale geodynamics simulations. The numerical algorithms
        allow the tracking of history information through the high-strain
        deformation associated with fluid flow (for example, transport of the
        stress tensor in a viscoelastic, convecting medium, or the advection of
        fine-scale damage parameters by the large-scale flow). The finite
        element mesh can be static or dynamic, but it is not contrained to move
        in lock-step with the evolving geometry of the fluid. This hybrid approach
        is very well suited to complex fluids which is how the solid Earth behaves
        on a geological timescale.

    PACKAGE CONTENTS
        _net (package)
        _stgermain
        _uwid
        conditions (package)
        container (package)
        function (package)
        mesh (package)
        swarm (package)
        systems (package)
        utils (package)

    FUNCTIONS
        barrier()
            Creates an MPI barrier. All processes wait here for others to catch up.

        matplotlib_inline()
            This function simply enables Jupyter Notebook inlined matplotlib results.
            This function should be called at the start of your notebooks as a
            replacement for the Jupyter Notebook *%matplotlib inline* magic. It provides
            the same functionality, however it allows notebooks to be converted to
            python without having to explicitly remove these calls.

        nProcs()
            Returns the number of processes being utilised by the simulation.

            Returns
            -------
            unsigned
                Number of processors.

        rank()
            Returns the rank of the current process.

            Returns
            -------
            unsigned
                Rank of current process.

    DATA
        __version__ = '2.4.0-dev'

    VERSION
        2.4.0-dev

    None

```python
## Tab completion:  fn.
## gives you a helpful list with all the available functions
## but it leaves out the _hidden_ ones which the following does not

print dir(fn.math)
```

    ['_Function', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_cfn', '_uw', 'abs', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atanh', 'cos', 'cosh', 'dot', 'erf', 'erfc', 'exp', 'log', 'log10', 'log2', 'pow', 'sin', 'sinh', 'sqrt', 'tan', 'tanh']

```python
print help(fn.analytic.SolCx)
```

    Help on class SolCx in module underworld.function.analytic:

    class SolCx(_SolBase)
     |  The boundary conditions are free-slip everywhere on a unit domain.
     |  There is a viscosity jump in the x direction at :math:`x=xc`.
     |  The flow is driven by the following density perturbation:
     |
     |  .. math::
     |      \rho = \sin (nz \pi z) \cos (nx \pi x)
     |
     |  Parameters
     |  ----------
     |  viscosityA : float
     |      Viscosty of region A.
     |  viscosityB : float
     |      Viscosty of region B.
     |  xc : float
     |      Location for viscosity jump.
     |  nx : unsigned
     |      Wavenumber in x axis.
     |
     |  Method resolution order:
     |      SolCx
     |      _SolBase
     |      __builtin__.object
     |
     |  Methods defined here:
     |
     |  __init__(self, viscosityA=1.0, viscosityB=2.0, xc=0.25, nx=1, *args, **kwargs)
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors inherited from _SolBase:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)
     |
     |  fn_bodyforce
     |
     |  fn_pressure
     |
     |  fn_strainRate
     |
     |  fn_stress
     |
     |  fn_velocity
     |
     |  fn_viscosity

    None

```python
print help(uw.systems.Stokes)
```

    Help on class Stokes in module underworld.systems._stokes:

    class Stokes(underworld._stgermain.StgCompoundComponent)
     |  This class provides functionality for a discrete representation
     |  of the Stokes flow equations.
     |
     |  Specifically, the class uses a mixed finite element method to
     |  construct a system of linear equations which may then be solved
     |  using an object of the underworld.system.Solver class.
     |
     |  The underlying element types are determined by the supporting
     |  mesh used for the 'velocityField' and 'pressureField' parameters.
     |
     |  The strong form of the given boundary value problem, for :math:`f`,
     |  :math:`g` and :math:`h` given, is
     |
     |  .. math::
     |      \begin{align}
     |      \sigma_{ij,j} + f_i =& \: 0  & \text{ in }  \Omega \\
     |      u_{k,k} + \frac{p}{\lambda} =& \: H  & \text{ in }  \Omega \\
     |      u_i =& \: g_i & \text{ on }  \Gamma_{g_i} \\
     |      \sigma_{ij}n_j =& \: h_i & \text{ on }  \Gamma_{h_i} \\
     |      \end{align}
     |
     |  where,
     |
     |  * :math:`\sigma_{i,j}` is the stress tensor
     |  * :math:`u_i` is the velocity,
     |  * :math:`p`   is the pressure,
     |  * :math:`f_i` is a body force,
     |  * :math:`\lambda` is a bulk viscosity,
     |  * :math:`H` is the compressible equation source term,
     |  * :math:`g_i` are the velocity boundary conditions (DirichletCondition)
     |  * :math:`h_i` are the traction boundary conditions (NeumannCondition).
     |
     |  The problem boundary, :math:`\Gamma`,
     |  admits the decompositions :math:`\Gamma=\Gamma_{g_i}\cup\Gamma_{h_i}` where
     |  :math:`\emptyset=\Gamma_{g_i}\cap\Gamma_{h_i}`. The equivalent weak form is:
     |
     |  .. math::
     |      \int_{\Omega} w_{(i,j)} \sigma_{ij} \, d \Omega = \int_{\Omega} w_i \, f_i \, d\Omega + \sum_{j=1}^{n_{sd}} \int_{\Gamma_{h_j}} w_i \, h_i \,  d \Gamma
     |
     |  where we must find :math:`u` which satisfies the above for all :math:`w`
     |  in some variational space.
     |
     |  Parameters
     |  ----------
     |  velocityField : underworld.mesh.MeshVariable
     |      Variable used to record system velocity.
     |  pressureField : underworld.mesh.MeshVariable
     |      Variable used to record system pressure.
     |  fn_viscosity : underworld.function.Function
     |      Function which reports a viscosity value.
     |      Function must return scalar float values.
     |  fn_bodyforce : underworld.function.Function, Default = None
     |      Function which reports a body force for the system.
     |      Function must return float values of identical dimensionality
     |      to the provided velocity variable.
     |  fn_lambda : Removed use, fn_one_on_lambda instead
     |  fn_minus_one_on_lambda: underworld.function.Function, Default = None
     |      Function which defines a non solenoidal velocity field via the relationship
     |      div(velocityField) = -fn_minus_one_on_lambda * pressurefield + fn_source
     |      When this is left as None a incompressible formulation of the stokes equation is formed, ie, div(velocityField) = 0.
     |      fn_minus_one_on_lambda is incompatible with the 'penalty' stokes solver, ensure a
     |      'penalty' equal to 0 is used when fn_minus_one_on_lambda is used. By default this is the case.
     |  fn_source : underworld.function.Function, Default = None
     |      Function which defines a non solenoidal velocity field via the relationship
     |      div(velocityField) = -fn_minus_one_on_lambda * pressurefield + fn_source.
     |      fn_minus_one_on_lambda is incompatible with the 'penalty' stokes solver, ensure
     |      the 'penalty' of 0, is used when fn_lambda is used. By default this is the case.
     |  voronoi_swarm : underworld.swarm.Swarm
     |      If a voronoi_swarm is provided, voronoi type numerical integration is
     |      utilised. The provided swarm is used as the basis for the voronoi
     |      integration. If no voronoi_swarm is provided, Gauss integration
     |      is used.
     |  conditions : underworld.conditions.SystemCondition
     |      Numerical conditions to impose on the system. This should be supplied as
     |      the condition itself, or a list object containing the conditions.
     |
     |  Notes
     |  -----
     |  Constructor must be called by collectively all processes.
     |
     |  Method resolution order:
     |      Stokes
     |      underworld._stgermain.StgCompoundComponent
     |      underworld._stgermain.StgClass
     |      underworld._stgermain.LeftOverParamsChecker
     |      __builtin__.object
     |
     |  Methods defined here:
     |
     |  __init__(self, velocityField, pressureField, fn_viscosity, fn_bodyforce=None, fn_one_on_lambda=None, fn_lambda=None, fn_source=None, voronoi_swarm=None, conditions=[], _removeBCs=True, _fn_viscosity2=None, _fn_director=None, _fn_stresshistory=None, **kwargs)
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  eqResiduals
     |      Returns the stokes flow equations' residuals from the latest solve. Residual calculations
     |      use the matrices and vectors of the discretised problem.
     |      The residuals correspond to the momentum equation and the continuity equation.
     |
     |      Return
     |      ------
     |      (r1, r2) - 2 tuple of doubles
     |          r1 is the momentum equation residual
     |          r2 is the continuity equation residual
     |
     |      Notes
     |      -----
     |      This method must be called collectively by all processes.
     |
     |  fn_bodyforce
     |      The body force function. You may change this function directly via this
     |      property.
     |
     |  fn_one_on_lambda
     |      A bulk viscosity parameter
     |
     |  fn_source
     |      The volumetric source term function. You may change this function directly via this
     |      property.
     |
     |  fn_viscosity
     |      The viscosity function. You may change this function directly via this
     |      property.
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes defined here:
     |
     |  __abstractmethods__ = frozenset([])
     |
     |  ----------------------------------------------------------------------
     |  Static methods inherited from underworld._stgermain.StgCompoundComponent:
     |
     |  __new__(cls, *args, **kwargs)
     |      Creates stgermain instances of underlying objects.
     |
     |      The list of required instances is provided as a dictionary ('_objectsDict') on the class,
     |      with the entry key being the object name, and entry value being the object type.
     |
     |      A second class data member ('_selfObjectName') provides the object name
     |      which should be considered the object 'self'.
     |
     |      For example:
     |      class Drawing(_stgermain.StgCompoundComponent):
     |          _selfObjectName = "_dr"  # child should set this
     |          _objectsDict = { "_cm": "lucColourMap",
     |                           "_dr": None            }
     |
     |      Note that this example is for an abstract class ('Drawing'), so it does not define
     |      a type for '_dr', but instead defers to child classes to define the type of '_dr'.
     |
     |      Args:
     |          objectsDictOverrule (dict): If provided, this will overwrite any objects
     |                                      in the class level _objectDicts. Useful for
     |                                      programatically modifying behaviour.
     |
     |
     |      Returns:
     |          New created instance of child class.
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes inherited from underworld._stgermain.StgCompoundComponent:
     |
     |  __metaclass__ = <class 'underworld._stgermain._SetupClass'>
     |      This metaclass allows us to invoke a _setup method after the the __init__ method.
     |      Borrowed from:
     |      http://stackoverflow.com/questions/22261763/in-python-is-it-possible-to-write-a-method-that-will-be-automatically-called-aft
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from underworld._stgermain.StgClass:
     |
     |  __del__(self)
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors inherited from underworld._stgermain.LeftOverParamsChecker:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)

    None

```python

```


```python
# Bunch of imports
import numpy as np
import numpy.polynomial.polynomial as n_poly
import numpy.polynomial.legendre as leg
from scipy import integrate
from scipy import optimize
from scipy import interpolate
import sympy as sym
%pylab inline
```

```python
# Set the interval for L2 and the number of basis functions
A=-1.
B=1.
N=25
```

```python
# Define the scalar product in L2([A,B])
t = sym.symbols('t')

def scalar_product(p0, p1):
#     tt = p0.free_symbols.pop()
    foo1 = sym.lambdify(t, p0, 'numpy')
    foo2 = sym.lambdify(t, p1, 'numpy')
    foo3 = lambda x : foo1(x)*foo2(x)
    foo = integrate.quad(foo3,A,B)
    return foo[0]

```

```python
# Derive and plot the Legendre polynomials
PLk = [1.+0.*t]
PMk = [1.+0.*t]

for k in range(1,N):
    s = 0.
    for j in range(k):
        s+= scalar_product(t**k,PLk[j])/scalar_product(PLk[j],PLk[j])*PLk[j]
    pk = t**k - s
    pk = pk/pk.subs(t,B)
    PLk.append(pk)
    PMk.append(t**k)

xx = np.linspace(A,B,100)
figure()
fs = sym.lambdify(t, PMk[0], 'numpy')
plot(xx, np.ones(xx.shape)*fs(xx))
for k in range(1,N):
    fs = sym.lambdify(t, PMk[k], 'numpy')
    plot(xx, fs(xx))
figure()
plot(xx, np.ones(xx.shape))
for k in range(1,N):
    fs = sym.lambdify(t, PLk[k], 'numpy')
    plot(xx, fs(xx))

figure()
for k in range(N):
    c = np.zeros(k)
    if(k==0):
        c=1
    else:
        c[-1]=1
    plot(xx, leg.legval(xx, c))
```

```python
#Best Approximation in L2([A,B]) using different basis
f = sym.cos(8.*np.pi/(B-A)*t)

M_LP = np.zeros([N,N])

M_MP = np.zeros([N,N])

f_M = np.zeros([N,1])

f_L = np.zeros([N,1])

for i in range(N):
    f_M[i] = scalar_product(PMk[i],f)
    f_L[i] = scalar_product(PLk[i],f)
    for j in range(N):
        M_LP[i,j] = scalar_product(PLk[i],PLk[j])
        M_MP[i,j] = scalar_product(PMk[i],PMk[j])

p_L = np.linalg.solve(M_LP, f_L)
p_M = np.linalg.solve(M_MP, f_M)

f_p_L = 0.
f_p_M = 0.

for i in range(N):
    f_p_L += p_L[i] * PLk[i]
    f_p_M += p_M[i] * PMk[i]

fpL = sym.lambdify(t, f_p_L[0], 'numpy')
fpM = sym.lambdify(t, f_p_M[0], 'numpy')

ff = sym.lambdify(t, f, 'numpy')

plot(xx, ff(xx), 'r*')
plot(xx, fpL(xx), 'b')
plot(xx, fpM(xx), 'g')
# See the condition numbers of the Mass Matrix
print(np.linalg.cond(M_LP))
print(np.linalg.cond(M_MP))
```

```python
# Verify Optimality
N_lagrange = N
x_interp = np.linspace(A,B,N_lagrange)
x_cheby = np.array([np.cos((2.*j-1)/(2.*N_lagrange)*np.pi) for j in range(1,N_lagrange + 1)])
lagrange_interp = interpolate.lagrange(x_interp, ff(x_interp))
cheb_interp = interpolate.lagrange(x_cheby, ff(x_cheby))

figure()
plot(xx, lagrange_interp(xx))
figure()
plot(xx, cheb_interp(xx))

lagrange_error = integrate.quad(lambda x : (ff(x)-lagrange_interp(x))**2, A,B)

cheb_error = integrate.quad(lambda x : (ff(x)-cheb_interp(x))**2, A,B)

fpM_error = integrate.quad(lambda x : (ff(x)-fpM(x))**2, A,B)

fpL_error = integrate.quad(lambda x : (ff(x)-fpL(x))**2, A,B)
```

```python
print lagrange_error
print cheb_error
print fpL_error
print fpM_error
```

```python
# Roots of Legendre polynomial

fs = sym.lambdify(t, PLk[8], 'numpy')
x0 = np.linspace(A,B,8)
roots = optimize.fsolve(fs, x0)
print(roots)
plot(xx, fs(xx))

x0[1] = -0.85
x0[-2] = 0.85
roots = optimize.fsolve(fs, x0)
print(roots)

c = np.zeros(8+1)
c[-1] = 1
print(leg.legroots(c))

xg, wg = leg.leggauss(8)

print(xg)
```

```python
# Gauss Integration
f_gauss = lambda x : 8.*np.sin(x)+3.*np.cos(15.*x)

I = integrate.quad(f_gauss,A,B)

n_gauss = 10

xg, wg = leg.leggauss(n_gauss)

In = np.dot(wg, f_gauss(xg))

print(I)
print(In)

# Integration Error

error = list()

Ngmax = 40
order = np.arange(1,Ngmax)
for i in order:
    xg, wg = leg.leggauss(i)
#     In = np.dot(wg, f_gauss(xg))
    In = np.einsum('j,j',wg,f_gauss(xg))
    error.append(np.abs(I[0]-In))

plot(order, error)
figure()
loglog(order,error)
loglog(order, np.ones(Ngmax-1)*I[1])
figure()
semilogy(order,error)
semilogy(order, np.ones(Ngmax-1)*I[1])
## REMARK QUADRATURE ORDER ON (f, vi) FOR BEST APPROXIMATION
```

```python

```


# Constraint Handling

Constraint handling is often neglected in frameworks but is indeed an essential aspect of optimization.
Depending on the characteristics of the constraints, one method might be more suitable and lead to better results than another.
In principle, two ways exist how constraints can be dealt with: a) use an algorithm that can handle constraints internally; b) redefine the problem to have no constraints but still guarantee the optimum to be a feasible solution.
Both strategies will be elaborated in the following.

By default, the returned optimum is **always** required to be feasible. It might be required to obtain the **least infeasible** solution if no feasible could have been found in some situations. To enable this behavior, you can pass `return_least_infeasible=True` to the `minimize` function.

```python
from pymoo.algorithms.soo.nonconvex.ga import GA
from pymoo.factory import get_problem
from pymoo.optimize import minimize

res = minimize(get_problem("g05"),
               GA(),
               return_least_infeasible=True,
               seed=1,
               verbose=False)

print("CV:", res.CV[0])
print("F:", res.F[0])
```

    Compiled modules for significant speedup can not be used!
    https://pymoo.org/installation.html#installation

    To disable this warning:
    from pymoo.config import Config
    Config.show_compile_hint = False

    CV: 0.18861877009391703
    F: 5819.467467631964

## Inequality Constraints

In pymoo, inequality constraints are always defined as "$\leq 0$" constraints. Thus, constraint violation is defined as follows: A solution is considered as **feasible** of all constraint violations are **less than** zero. A solution is considered as **infeasible** if at least one constraint violation is **larger than** zero.

$$
- x^4 \geq - 2
$$

Has to be converted to a "$\leq 0$" constraints to satisfy the implementation of constraints in pymoo:

$$
x^4  \leq 2\\
x^4 - 2 \leq 0
$$

Let us assume the whole optimization problem is then given by:

\begin{align}
\begin{split}
\min \;\; & f(x) = 1 + x^3 \\
\text{s.t.} \;\; & x^4 - 2\leq 0\\
& -2 \leq x \leq 2
\end{split}
\end{align}

The figure below shows the corresponding function and the optimum to be found. The constraint function is shown with a dashed line greater than zero (violated) and with a solid line where less or equal to zero (satisfied). The optimum is shown by a red cross, which is minimum in the feasible region.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.rc('font', family='serif')

X = np.linspace(-2, 2, 10000)

F = 1 + X ** 3
G = X**4 - 2

plt.figure(figsize=(7, 5))

plt.plot(X, F, linewidth=2.0, color="blue", label="$f(x) = 1 + x^3$")

I = G<=0
plt.plot(X[I], G[I], linewidth=4.0, color="green")
plt.plot(X, G, linewidth=2.0, color="green", label="$g(x) = x^4 - 2$", linestyle='dotted')

plt.scatter([-1.18920654], [-0.6817904], color="red", s=100, marker="X", label="Optimum")
plt.xlim(-2, 2)
plt.xlabel("$x$")
plt.ylabel("$f$")

plt.legend(loc='upper center', bbox_to_anchor=(0.5, 1.12),
          ncol=4, fancybox=True, shadow=False)

plt.tight_layout()
plt.show()
```

The problem can be implemented straightforwardly:

```python
import numpy as np
from pymoo.util.misc import stack
from pymoo.core.problem import Problem

class MyProblem(Problem):

    def __init__(self):
        super().__init__(n_var=1,
                         n_obj=1,
                         n_constr=1,
                         xl=-2,
                         xu=2)

    def _evaluate(self, X, out, *args, **kwargs):
        out["F"] = 1 + X ** 3
        out["G"] = X**4 - 2
```

And then being solved as usually by initiating the problem, algorithm and running the minimize method:

```python
from pymoo.algorithms.soo.nonconvex.pso import PSO
from pymoo.optimize import minimize

problem = MyProblem()

algorithm = PSO()

res = minimize(problem,
               algorithm,
               seed=1,
               verbose=False)

print("Best solution found: \nX = %s\nF = %s\nCV = %s" % (res.X, res.F, res.CV))
```

    Best solution found:
    X = [-1.18920683]
    F = [-0.68179163]
    CV = [0.]

For a bi-objective example with constraints, we like to refer to our [Getting Started Guide](../getting_started/index.ipynb) where a similar example is shown.

## Equality to Inequality Constraints

We got a couple of questions about how equality constraints should be handled in a genetic algorithm. In general, functions without any smoothness are challenging to handle for genetic algorithms. An equality constraint is an extreme case, where the constraint violation is 0 at precisely one point and otherwise 1.

Let us consider the following constraint $g(x)$ where $x$ represents a variable:

$g(x): x = 5$

An equality constraint can be expressed by an inequality constraint:

$g(x): |x - 5| \leq 0$

or

$g(x): (x-5)^2   \leq 0$

However, all of the constraints above are very strict and make **most search space infeasible**. Without providing more information to the algorithm, those constraints are challenging to satisfy.

Thus, the constraint can be smoothed by adding an epsilon to it and, therefore, having two inequality constraints:

$g'(x): 5 - \epsilon \leq x \leq 5 + \epsilon$

Also, it can be simply expressed in one inequality constraint by:

$g'(x): (x-5)^2  - \hat{\epsilon} \leq 0$

Depending on the $\epsilon$, the solutions will be more or less close to the desired value. However, the genetic algorithm does not know anything about the problem itself, making it challenging to handle and focus the search in the infeasible space.

## Parameter-less Approach / Feasibility First

Parameter-less constraint handling is the default constraint handling approach in **pymoo**. It strictly prefers feasible over infeasible solutions.

$$
F(x)=
\begin{cases}
f(x), \quad \text{if feasible} \\
f_{\max} + \sum_{j=1}^{|J|} \, \langle \, g_j (x) \, \rangle, \quad \rm{otherwise}
\end{cases}
$$

where $\langle \cdot \rangle$ is defined by

$$
\langle a \rangle =
\begin{cases}
a, \quad a \geq 0 \\
0, \quad \rm{otherwise}
\end{cases}
$$

This approach works well if an algorithm is based on comparisons. For instance, a genetic algorithm has to determine what individuals survive. Thus, this mapping can be utilized to create a rank for each individual.
If the $f_\max$ of the function is known beforehand, it can also be used to redefine the problem beforehand.
Redefining the problem is very similar to the approach discussed next.

## Penalty-based Approach / Lagrange Multipliers

The penalty-based constraint handling approach internally converts equality constraints to inequality constraints. Then, the violation of inequality and equality constraints penalize the objective value. This makes all non-constrained algorithms applicable; however, it also might create a more difficult fitness landscape to solve.

**NOTE**: Genetic algorithms are not made to handle equality constraints out of the box, but often need to be customized to satisfy the equality constraints directly. The approach below only works for a very simple and a very limited amount of equality constraints.

### Single-objective

\begin{align}
\begin{split}
\min \;\; & f(x) = x_1^2 + x_2^2 + x_3^2 \\
\text{s.t.} \;\; & x_1 \cdot x_2 \geq 1\\
& x_1 \cdot x_2 \leq 5\\
& x_2 + x_3 = 1\\
& 0 \leq x \leq 5
\end{split}
\end{align}

By converting the constraints to $\leq 0$ constraints the implementation of the problem looks as follows:

```python
def func_obj(x):
    return x[0] ** 2 + x[1] ** 2 + x[2] ** 2

constr_ieq = [
    lambda x: 1 - x[0] * x[1],
    lambda x: x[0] * x[1] - 5
]

constr_eq = [
    lambda x: 1 - x[1] - x[2]
]
```

For this example, we use the definition of a problem using the factory method `create_problem_from_funcs`:

```python
from pymoo.problems.functional import FunctionalProblem
from pymoo.problems.constr_as_penalty import ConstraintsAsPenalty

problem = FunctionalProblem(3,
                            func_obj,
                            xl=0,
                            xu=5,
                            constr_ieq=constr_ieq,
                            constr_eq=constr_eq
                            )

problem = ConstraintsAsPenalty(problem, penalty=1e6)
```

Then, an algorithm can be used to solve the problem as usual:

```python
from pymoo.algorithms.soo.nonconvex.pso import PSO
from pymoo.optimize import minimize

algorithm = PSO(pop_size=50)

res = minimize(problem, algorithm, seed=1)

opt = res.opt[0]
X, F, CV = opt.get("X", "__F__", "__CV__")

print("Best solution found: \nX = %s\nF = %s\nCV = %s\n" % (X, F, CV))
```

    Best solution found:
    X = [1.0076019  0.99245594 0.00754335]
    F = [2.00028729]
    CV = [0.]

### Multi-objective

The almost equivalent could be applied for multi-objective optimization problems. The example below shows the [getting started guide](../getting_started/index.ipynb) example with a penalty being used to deal with the constraints:

#### Functions

```python
objs = [
    lambda x: x[0]**2 + x[1]**2,
    lambda x: (x[0]-1)**2 + x[1]**2

]

constr_ieq = [
    lambda x: 2*(x[0]-0.1) * (x[0]-0.9) / 0.18,
    lambda x: - 20*(x[0]-0.4) * (x[0]-0.6) / 4.8
]

constr_eq = []
```

#### Problem

```python
from pymoo.problems.functional import FunctionalProblem
from pymoo.problems.constr_as_penalty import ConstraintsAsPenalty

problem = FunctionalProblem(2,
                            objs,
                            xl=-2,
                            xu=2,
                            constr_ieq=constr_ieq,
                            constr_eq=constr_eq
                            )

problem = ConstraintsAsPenalty(problem, penalty=1e6)
```

#### Optimize

```python
from pymoo.algorithms.moo.nsga2 import NSGA2
from pymoo.optimize import minimize

algorithm = NSGA2(pop_size=40)

res = minimize(problem,
               algorithm,
               seed=1)
```

#### Visualize

```python
from pymoo.visualization.scatter import Scatter

plot = Scatter(title = "Objective Space")
plot.add(res.F)
plot.show()
```

## Repair Operator

A simple approach is to handle constraints through a repair function. This is only possible if the equation of the constraint is known. The repair makes sure every solution that is evaluated is, in fact, feasible. Let us consider the following example where
the equality constraints need to consider more than one variable:

\begin{align}
\begin{split}
\min \;\; & f_1(x) = (x_1^2 + x_2^2) \\
\max \;\; & f_2(x) = -(x_1-1)^2 - x_2^2 \\[1mm]
\text{s.t.} \;\; & g_1(x_1, x_3) : x_1 + x_3 = 2\\[2mm]
& -2 \leq x_1 \leq 2 \\
& -2 \leq x_2 \leq 2 \\
& -2 \leq x_3 \leq 2
\end{split}
\end{align}

We implement the problem using by squaring the term and using an $\epsilon$ as we have explained above. The source code for the problem looks as follows:

```python
import numpy as np

from pymoo.core.problem import Problem

class MyProblem(Problem):

    def __init__(self):
        super().__init__(n_var=3,
                         n_obj=2,
                         n_constr=1,
                         xl=np.array([-2, -2, -2]),
                         xu=np.array([2, 2, 2]))

    def _evaluate(self, x, out, *args, **kwargs):
        f1 = x[:, 0] ** 2 + x[:, 1] ** 2
        f2 = (x[:, 0] - 1) ** 2 + x[:, 1] ** 2
        g1 = (x[:, 0] + x[:, 2] - 2) ** 2 - 1e-5

        out["F"] = np.column_stack([f1, f2])
        out["G"] = g1

```

As you might have noticed, the problem has similar characteristics to the problem in our getting started.
Before a solution is evaluated, a repair function is called. To make sure a solution is feasible, an approach would be to either set $x_3 = 2 - x_1$ or $x_1 = 2 - x_3$. Additionally, we need to consider that this repair might produce a variable to be out of bounds.

```python
from pymoo.core.repair import Repair

class MyRepair(Repair):

    def _do(self, problem, pop, **kwargs):
        for k in range(len(pop)):
            x = pop[k].X
            if np.random.random() < 0.5:

                x[2] = 2 - x[0]
                if x[2] > 2:
                    val = x[2] - 2
                    x[0] += val
                    x[2] -= val

            else:
                x[0] = 2 - x[2]
                if x[0] > 2:
                    val = x[0] - 2
                    x[2] += val
                    x[0] -= val

        return pop
```

Now the algorithm object needs to be initialized with the repair operator and then can be run to solve the problem:

```python
from pymoo.algorithms.moo.nsga2 import NSGA2
from pymoo.optimize import minimize
from pymoo.visualization.scatter import Scatter

algorithm = NSGA2(pop_size=100, repair=MyRepair(), eliminate_duplicates=True)

res = minimize(MyProblem(),
               algorithm,
               ('n_gen', 20),
               seed=1,
               verbose=True)

plot = Scatter()
plot.add(res.F, color="red")
plot.show()
```

In our case it is easy to verify if the constraint is violated or not:

```python
print(res.X[:, 0] + res.X[:, 2])
```

    [2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2.]

If you would like to compare the solution without a repair you will see how searching only in the feasible space helps:

```python
algorithm = NSGA2(pop_size=100, eliminate_duplicates=True)

res = minimize(MyProblem(),
               algorithm,
               ('n_gen', 20),
               seed=1,
               verbose=True)

plot = Scatter()
plot.add(res.F, color="red")
plot.show()

```

```python
print(res.X[:, 0] + res.X[:, 2])
```

    [1.99825371]

Here, in fact, the $\epsilon$ term is necessary to find any feasible solution at all.


## Exercise 1 (4 points): 1D CNN [Pen and Paper]

In this assignment, we want to understand convolutional neural networks (CNNs) and we start by applying them to a one-dimensional input so that we get used to the basic concepts. An implementation applied to images follows in Exercise 2.

In the context of this exercise, the output 𝑦𝑘 of the 𝑘-th neuron in the 𝑗-th feature map of a convolutional layer is calculated as <br/>

\begin{equation}y_{k,j} = f \left(\sum_{i=1}^M \sum_{c=1}^C w_{i, c; j}⋅ x_{i+k-1, c} + b_j\right)\end{equation}

with the weights 𝑤𝑖,𝑐;𝑗, the input 𝑥𝑖,𝑐, the bias 𝑏𝑗 and the transfer function 𝑓 (𝑥). 𝑀 denotes the filter length and 𝐶 the number of channels this filter pools from. Note: Equation 1 does not include the concepts of padding and stride sizes used below (cf. TensorFlow references for further details about these parameters).

Suppose that our CNN is defined as follows:

    I An input of length 10 consisting of 𝐶 = 2 channels.

    II One convolutional layer with 3 feature maps, a filter length of 𝑀 = 5, a stride of 1 and the padding set to valid. As activation function, rectified linear units, i.e. 𝑓 (𝑥) = max(𝑥, 0), are used.

    III A max-pooling layer with a pool size of 3, a stride of 2 and the padding set to same.

    IV Two output neurons with a linear activation function (the output of the pooling layer is flattened before it is passed on to the output layer).

Let’s use this network to perform some basic computations.
1. The upper part of the input matrix 𝑋 is given as follows
\begin{pmatrix}
x_{1, 1} & 0\\
1 & -1\\
2 & 0\\
0 & 0\\
1 & 1\\
\vdots & \vdots
\end{pmatrix}
We now consider the first filter (𝑗 = 1) of the convolutional layer.

    a) What is the dimension of the weight matrix 𝑊 (filter size)?

    b) Suppose that all weights which pool from the first channel are set to $w_{i,1;1} = 1$, the weights for the second channel are set to $w_{i,2;1} = 2$ and the bias is set to $b_1 = 0$. The output of the first neuron is $y_{1;1} = 5$. Calculate the missing input value $x_{1,1}$.

2. What is the difference between a one-dimensional convolutional layer (temporal convolution) which pools from multiple channels (as implemented by Equation 1) compared to a two-dimensional CNN layer (e.g. with a filter size of 5 × 2)?

3. For each layer in our network, fill out Table 1 by deciding about

    • the dimensions of the layer’s output tensor (e.g. output matrix of the convolutional layer).

    • the number of weights per layer (if any).

    • the number of biases per layer (if any)

4. Suppose we used a standard hidden layer with 10 neurons instead of the convolutional and pooling layers (the output neurons are still used), how does the total number of weights and biases in the network change (the input is flattened before it is passed on to the hidden layer)?

## Exercise 2 (6 points): 2D CNN

The convolution operation is often used in computer vision domain and it is hence not surprising that CNNs are especially helpful in image processing tasks. We now want to apply a CNN to a small image classification task based on the STL-10 dataset [1]. More concretely, we are using two classes with 500 images each and train a network to distinguish between them.

Our main goal is to visualize how the network processes the images, i.e. highlight the activations of the layers. Therefore, we disregard other aspects like the validation against a test set. Further, we are also going to cover some new TensorFlow aspects, namely datasets and iterators2 as a novel way to feed data to the network.

Please use the file CNN2D.ipynb attached to this exercise as the basis for your implementation. For the test script, please pay also attention to Table 2.

1. Load the images and the corresponding labels from the attached stl10.pickle file

2. It would not be very practical to visualize the network activations for all images so we only use a few examples. Create a subset of the images (and labels) which contains these example images. Make sure that you use at least one instance per class.

3. We need to feed the images and the corresponding labels to the network and we want to use datasets and iterators for this purpose.

    a) Define a training dataset which iterates over all images. Shuffle the dataset (set a buffer size which equals the total number of training instances) and then apply the specified batch size.

    b) Define another dataset for the example images selected previously. Batch over all instances at once.

    c) Create a reinitializable iterator and define an initialization operation for each dataset.

    d) To access the data, extract tensors for the images and labels from the iterator (iter ator.get_next())

4. We now have all ingredients together to build our network. Use the defined parameters whenever it is suitable.

    a) Create a hierarchy of subsequent convolutional and pooling layer pairs (n_cnn_la yers in total). That is, first process the input via a (two-dimensional) convolutional layer and then apply an average (two-dimensional) polling layer. Store the CNN layers so that you can later easily calculate the network activations for the example images.

    b) Reshape the output of the last pooling layer to a one-dimensional vector. Note that you still need a two-dimensional tensor with the first dimension denoting the batches.

    c) Use a final (linear) layer which calculates the logits for each class.

    d) Define a training operation based on the cross-entropy loss function (including regularization) and the Adam optimizer.

    e) Create a tensor which calculates the accuracy over the current batch

5. The next step is to train and evaluate the network. We also want to calculate (and store) the activation images whenever the state of the network changes. Note that the training can take a few minutes.

    a) Calculate the activation images for the initial state of the network (after initializing the variables).

    b) Train the network over multiple epochs. With each training step (one batch), calculate the loss and the accuracy and average all values together so that one loss and one accuracy value per epoch remains3. Keep track of these values.

    c) After the training phase of each epoch, calculate the activation images again.

6. Plot the loss and the accuracy values over the epochs.

7. Plot your selected images.

8. Create an animation (interact) for the activation images which lets you select the image, the epoch and the layer. For each configuration, the output of the four filters should be shown. Set explicit values for the data range when using the imshow() function (vmin and vmax) which correspond to the boundaries of the tanh(𝑥) function. An example can be seen in Figure 1.


```python
import sympy as sym
%matplotlib inline
sym.init_printing()

import matplotlib.pyplot as plt
from sympy.plotting.plot import MatplotlibBackend

def logx_plot(Hw, wrange, **kwargs):
    'create plot with logarithmic x-axis'
    p = sym.plot(Hw, wrange, **kwargs)
    backend = MatplotlibBackend(p)
    backend.process_series()
    backend.ax[0].spines['left'].set_position(('axes', 0))
    backend.ax[0].spines['bottom'].set_position(('axes', 0))
    plt.xscale('log')
    plt.grid(which='both')
    return p

def db(x):
    'compute dB value'
    return 20 * sym.log(sym.Abs(x), 10)
```

# Combinación de Modelos Frecuenciales

Es conveniente analizar sistemas elaborados como una representación de sistemas más sencillos. A continuación los principales tipos de combinación.

## Concatenación / Serie

Cuando dos sistemas LTI $H_1$ y $H_2$ están interconectados de manera que la salida de $H_1$ es la entrada de $H_2$. Esta es la conexión en **Serie** o **Concatenación de sistemas**.

Suponga que:
- La respuesta impulsional de $H_1$ es $h_1(t)$.
- La respuesta impulsional de $H_2$ es $h_2(t)$.
- La concatenación se excita con una señal $x(t)$.

La salida de $H_1$ es entonces $x(t)*h_1(t)$. La salida de la concatenación de sistemas es:

\begin{equation}
y(t) =  x(t) * h_1(t) * h_2(t)
\end{equation}

En el dominio de Laplace, la convolución anterior se transforma en:

\begin{equation}
Y(s) = \underbrace{H_1(s) \cdot H_2(s)}_{H(s)} \cdot X(s)
\end{equation}

Así, los dos sistemas concatenados tienen una equivalencia $H(s) = H_1(s) \cdot H_2(s)$.

Si se tienen $N$ sistemas conectados en serie, el modelo equivalente está dado por la multiplicación de las respectivas funciones de transferencia.

\begin{equation}
H(s) = \prod_{n=1}^{N} H_n(s)
\end{equation}

**Ejemplo**

Al concatenar filtros de bajo orden se obtiene un filtro de orden más elevado.

Suponga las siguientes funciones de transferencia para filtros de orden 2.

\begin{equation}
G(s) = \frac{1}{LC s^2 + RC s + 1}
\end{equation}

Si se concatenan $N$ filtros con acoples de impedancia, se obtiene:

\begin{equation}
H_N(s) = \left( \frac{1}{LC s^2 + RC s + 1} \right)^N
\end{equation}

A continuación los diagramas de Bode. Para valores $R$, $L$ y $C$.

```python
s = sym.symbols('s', complex=True)
w = sym.symbols('omega', real=True)
R = 1
L = 0.5
C = 0.4

N = sym.symbols('N', integer=True)

H0 = 1/(L*C*s**2 + R*C*s + 1)
HN = H0**N
HN

```

Ahora se definen valores para $s = \sigma + j \omega$.

```python
H0w = H0.subs(s, sym.I * w)
HNw = HN.subs(s, sym.I * w)

plt.rcParams['figure.figsize'] = 6, 2

m1 = logx_plot(db(H0w), (w, 0.1, 100),
               xlabel='$\omega$', ylabel='$20 \log_{10} | H(j \omega) |$ in dB',
               line_color='b', show=False);
m2 = logx_plot(db(HNw.subs(N, 2)), (w, 0.1, 100),
               xlabel='$\omega$', ylabel='$20 \log_{10} | H(j \omega) |$ in dB',
               line_color='r', show=False);
m5 = logx_plot(db(HNw.subs(N, 5)), (w, 0.1, 100),
               xlabel='$\omega$', ylabel='$20 \log_{10} | H(j \omega) |$ in dB',
               line_color='g', show=False);
f1 = logx_plot(sym.arg(H0w), (w, 0.1, 100),
               xlabel='$\omega$', ylabel=r'$\varphi(j \omega)$',
               line_color='b', show=False);
f2 = logx_plot(sym.arg(HNw.subs(N, 2)), (w, 0.1, 100),
               xlabel='$\omega$', ylabel=r'$\varphi(j \omega)$',
               line_color='r', show=False);
f5 = logx_plot(sym.arg(HNw.subs(N, 5)), (w, 0.1, 100),
               xlabel='$\omega$', ylabel=r'$\varphi(j \omega)$',
               line_color='g', show=False);

m1.extend(m2)
m1.extend(m5)

f1.extend(f2)
f1.extend(f5)
```

```python
backend = MatplotlibBackend(m1)
backend.process_series()
backend.ax[0].spines['left'].set_position(('axes', 0))
backend.ax[0].spines['bottom'].set_position(('axes', 0))
plt.xscale('log')
plt.grid(which='both')

backend = MatplotlibBackend(f1)
backend.process_series()
backend.ax[0].spines['left'].set_position(('axes', 0))
backend.ax[0].spines['bottom'].set_position(('axes', 0))
plt.xscale('log')
plt.grid(which='both')
```

## Paralelo

Cuando dos sistemas LTI comparten la misma entrada y sus salidas se combinan, están conectados en paralelo.

Cuando dos sistemas LTI $H_1$ y $H_2$ están interconectados de manera que tienen la misma entrada y sus salidas se combinan, esta es la conexión en **Serie** o **Concatenación de sistemas**.

Suponga que:
- La respuesta impulsional de $H_1$ es $h_1(t)$.
- La respuesta impulsional de $H_2$ es $h_2(t)$.
- La conexión paralela se excita con una señal $x(t)$.

La salida de $H_1$ es entonces $x(t)*h_1(t)$. La salida de $H_2$ es entonces $x(t)*h_2(t)$. La salida de la conexión paralela es:

\begin{equation}
y(t) = h_1(t) * x(t) + h_2(t) * x(t)
\end{equation}

En el dominio de Laplace, la combinación anterior se transforma en:

\begin{equation}
Y(s) = \underbrace{\left( H_1(s) + H_2(s)  \right)}_{H(s)} \cdot X(s)
\end{equation}

Así, los dos sistemas en paralelo tienen una equivalencia $H(s) = H_1(s) + H_2(s)$.

Si se tienen $N$ sistemas conectados en paralelo, el modelo equivalente está dado por la suma de las respectivas funciones de transferencia.

\begin{equation}
H(s) = \sum_{n=1}^{N} H_n(s)
\end{equation}

## Realimentación

En algunos casos, parte de la señal de salida de $H_1$ es devuelta a su entrada a través de otro sistema $H_2$. Esta conexión es llamada **realimentación**.

La salida $y(t)$ está dada por:

\begin{equation}
y(t) = (x(t) + y(t) * h_2(t)) * h_1(t)
\end{equation}

En el dominio de Laplace, la expresión se transforma en:

\begin{equation}
y(t) = (x(t) + y(t) * h_2(t)) * h_1(t)
\end{equation}

\begin{equation}
Y(s) = (X(s) + Y(s) \cdot H_2(s)) \cdot H_1(s)
\end{equation}

\begin{equation}
Y(s) = X(s) \cdot H_1(s) + Y(s) \cdot H_2(s) \cdot H_1(s)
\end{equation}

\begin{equation}
Y(s) -  Y(s) \cdot H_2(s) \cdot H_1(s) = X(s) \cdot H_1(s)
\end{equation}

\begin{equation}
Y(s) \cdot (1 -  H_2(s) \cdot H_1(s)) = X(s) \cdot H_1(s)
\end{equation}

\begin{equation}
Y(s) = \frac{H_1(s)}{1 - H_1(s) \cdot H_2(s)} \cdot X(s)
\end{equation}

La equivalencia es

\begin{equation}
H(s) = \frac{H_1(s)}{1 - H_1(s) \cdot H_2(s)}
\end{equation}

Si la realimentación es negativa, la equivalencia es:

\begin{equation}
H(s) = \frac{H_1(s)}{1 + H_1(s) \cdot H_2(s)}
\end{equation}

## Álgebra de bloques

Cuando hay varios subsistemas interactuando entre sí, se modelan por separado y se identifican las conexiones entre ellos. Para encontrar equivalencias deben reducirse el sistema a una sola expresión con las siguientes normas.

**Ejemplo**

## Ejercicio para entregar el 27 de noviembre

Reducir el siguiente diagrama.  Realice el ejercicio en papel y suba fotografías como evidencia.

## Próxima sesión

Transformada Z


Gate Decomposition

## Imports

```python
%matplotlib inline
```

```python
from IPython.display import display
```

```python
from sympy import init_printing
init_printing(use_latex=True)
```

```python
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

Example 1

Create a symbolic controlled-Y gate

```python
CY10 = CGate(1, Y(0));
CY10
```

Decompose it into elementary gates and plot it

```python
CY10.decompose()
```

```python
circuit_plot(CY10.decompose(), nqubits=2);
```

Example 2

Create a controlled-Z gate

```python
CZ01 = CGate(0, Z(1));
CZ01
```

Decompose and plot it

```python
CZ01.decompose()
```

```python
circuit_plot(CZ01.decompose(), nqubits=2);
```

Example 3

Create a SWAP gate

```python
SWAP10 = SWAP(1, 0);
SWAP10
```

Decompose and plot it

```python
SWAP10.decompose()
```

```python
circuit_plot(SWAP10.decompose(), nqubits=2);
```

All together now

```python
gates = [CGate(1,Y(0)), CGate(0,Z(1)), SWAP(1, 0)]
```

```python
for g in gates:
    dg = g.decompose()
    display(Eq(g, dg))
    circuit_plot(g, nqubits=2)
    circuit_plot(dg, nqubits=2)
```


# Seasonality in time series data

Consider the problem of modeling time series data with multiple seasonal components with different periodicities.  Let us take the time series $y_t$ and decompose it explicitly to have a level component and two seasonal components.

$$
y_t = \mu_t + \gamma^{(1)}_t + \gamma^{(2)}_t
$$

where $\mu_t$ represents the trend or level, $\gamma^{(1)}_t$ represents a seasonal component with a relatively short period, and $\gamma^{(2)}_t$ represents another seasonal component of longer period. We will have a fixed intercept term for our level and consider both $\gamma^{(2)}_t$ and $\gamma^{(2)}_t$ to be stochastic so that the seasonal patterns can vary over time.

In this notebook, we will generate synthetic data conforming to this model and showcase modeling of the seasonal terms in a few different ways under the unobserved components modeling framework.

```python
%matplotlib inline
```

```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
```

### Synthetic data creation

We will create data with multiple seasonal patterns by following equations (3.7) and (3.8) in Durbin and Koopman (2012).  We will simulate 300 periods and two seasonal terms parametrized in the frequency domain having periods 10 and 100, respectively, and 3 and 2 number of harmonics, respectively.  Further, the variances of their stochastic parts are 4 and 9, respectively.

```python
# First we'll simulate the synthetic data
def simulate_seasonal_term(periodicity, total_cycles, noise_std=1.,
                           harmonics=None):
    duration = periodicity * total_cycles
    assert duration == int(duration)
    duration = int(duration)
    harmonics = harmonics if harmonics else int(np.floor(periodicity / 2))

    lambda_p = 2 * np.pi / float(periodicity)

    gamma_jt = noise_std * np.random.randn((harmonics))
    gamma_star_jt = noise_std * np.random.randn((harmonics))

    total_timesteps = 100 * duration # Pad for burn in
    series = np.zeros(total_timesteps)
    for t in range(total_timesteps):
        gamma_jtp1 = np.zeros_like(gamma_jt)
        gamma_star_jtp1 = np.zeros_like(gamma_star_jt)
        for j in range(1, harmonics + 1):
            cos_j = np.cos(lambda_p * j)
            sin_j = np.sin(lambda_p * j)
            gamma_jtp1[j - 1] = (gamma_jt[j - 1] * cos_j
                                 + gamma_star_jt[j - 1] * sin_j
                                 + noise_std * np.random.randn())
            gamma_star_jtp1[j - 1] = (- gamma_jt[j - 1] * sin_j
                                      + gamma_star_jt[j - 1] * cos_j
                                      + noise_std * np.random.randn())
        series[t] = np.sum(gamma_jtp1)
        gamma_jt = gamma_jtp1
        gamma_star_jt = gamma_star_jtp1
    wanted_series = series[-duration:] # Discard burn in

    return wanted_series
```

```python
duration = 100 * 3
periodicities = [10, 100]
num_harmonics = [3, 2]
std = np.array([2, 3])
np.random.seed(8678309)

terms = []
for ix, _ in enumerate(periodicities):
    s = simulate_seasonal_term(
        periodicities[ix],
        duration / periodicities[ix],
        harmonics=num_harmonics[ix],
        noise_std=std[ix])
    terms.append(s)
terms.append(np.ones_like(terms[0]) * 10.)
series = pd.Series(np.sum(terms, axis=0))
df = pd.DataFrame(data={'total': series,
                        '10(3)': terms[0],
                        '100(2)': terms[1],
                        'level':terms[2]})
h1, = plt.plot(df['total'])
h2, = plt.plot(df['10(3)'])
h3, = plt.plot(df['100(2)'])
h4, = plt.plot(df['level'])
plt.legend(['total','10(3)','100(2)', 'level'])
plt.show()
```

### Unobserved components (frequency domain modeling)

The next method is an unobserved components model, where the trend is modeled as a fixed intercept and the seasonal components are modeled using trigonometric functions with primary periodicities of 10 and 100, respectively, and number of harmonics 3 and 2, respectively.  Note that this is the correct, generating model. The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \gamma^{(1)}_t + \gamma^{(2)}_t + \epsilon_t\\
\mu_{t+1} & = \mu_t \\
\gamma^{(1)}_{t} &= \sum_{j=1}^2 \gamma^{(1)}_{j, t} \\
\gamma^{(2)}_{t} &= \sum_{j=1}^3 \gamma^{(2)}_{j, t}\\
\gamma^{(1)}_{j, t+1} &= \gamma^{(1)}_{j, t}\cos(\lambda_j) + \gamma^{*, (1)}_{j, t}\sin(\lambda_j) + \omega^{(1)}_{j,t}, ~j = 1, 2, 3\\
\gamma^{*, (1)}_{j, t+1} &= -\gamma^{(1)}_{j, t}\sin(\lambda_j) + \gamma^{*, (1)}_{j, t}\cos(\lambda_j) + \omega^{*, (1)}_{j, t}, ~j = 1, 2, 3\\
\gamma^{(2)}_{j, t+1} &= \gamma^{(2)}_{j, t}\cos(\lambda_j) + \gamma^{*, (2)}_{j, t}\sin(\lambda_j) + \omega^{(2)}_{j,t}, ~j = 1, 2\\
\gamma^{*, (2)}_{j, t+1} &= -\gamma^{(2)}_{j, t}\sin(\lambda_j) + \gamma^{*, (2)}_{j, t}\cos(\lambda_j) + \omega^{*, (2)}_{j, t}, ~j = 1, 2\\
\end{align}
$$
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{j,t}$ are i.i.d. $N(0, \sigma^2_1)$, and  $\omega^{(2)}_{j,t}$ are i.i.d. $N(0, \sigma^2_2)$, where $\sigma_1 = 2.$

```python
model = sm.tsa.UnobservedComponents(series.values,
                                    level='fixed intercept',
                                    freq_seasonal=[{'period': 10,
                                                    'harmonics': 3},
                                                   {'period': 100,
                                                    'harmonics': 2}])
res_f = model.fit(disp=False)
print(res_f.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_f.smoother_results.smoothed_state[0,-1:][0]))

res_f.plot_components()
plt.show()

```

```python
model.ssm.transition[:, :, 0]
```

    array([[ 1.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        ,  0.80901699,  0.58778525,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        , -0.58778525,  0.80901699,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.30901699,  0.95105652,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        , -0.95105652,  0.30901699,
             0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            -0.30901699,  0.95105652,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            -0.95105652, -0.30901699,  0.        ,  0.        ,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.99802673,  0.06279052,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        , -0.06279052,  0.99802673,  0.        ,
             0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ,  0.9921147 ,
             0.12533323],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        , -0.12533323,
             0.9921147 ]])

Observe that the fitted variances are pretty close to the true variances of 4 and 9.  Further, the individual seasonal components look pretty close to the true seasonal components.  The smoothed level term is kind of close to the true level of 10.  Finally, our diagnostics look solid; the test statistics are small enough to fail to reject our three tests.

### Unobserved components (mixed time and frequency domain modeling)

The second method is an unobserved components model, where the trend is modeled as a fixed intercept and the seasonal components are modeled using 10 constants summing to 0 and trigonometric functions with a primary periodicities of 100 with 2 harmonics total.  Note that this is not the generating model, as it presupposes that there are more state errors for the shorter seasonal component than in reality. The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \gamma^{(1)}_t + \gamma^{(2)}_t + \epsilon_t\\
\mu_{t+1} & = \mu_t \\
\gamma^{(1)}_{t + 1} &= - \sum_{j=1}^9 \gamma^{(1)}_{t + 1 - j} + \omega^{(1)}_t\\
\gamma^{(2)}_{j, t+1} &= \gamma^{(2)}_{j, t}\cos(\lambda_j) + \gamma^{*, (2)}_{j, t}\sin(\lambda_j) + \omega^{(2)}_{j,t}, ~j = 1, 2\\
\gamma^{*, (2)}_{j, t+1} &= -\gamma^{(2)}_{j, t}\sin(\lambda_j) + \gamma^{*, (2)}_{j, t}\cos(\lambda_j) + \omega^{*, (2)}_{j, t}, ~j = 1, 2\\
\end{align}
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{t}$ are i.i.d. $N(0, \sigma^2_1)$, and  $\omega^{(2)}_{j,t}$ are i.i.d. $N(0, \sigma^2_2)$.

```python
model = sm.tsa.UnobservedComponents(series,
                                    level='fixed intercept',
                                    seasonal=10,
                                    freq_seasonal=[{'period': 100,
                                                    'harmonics': 2}])
res_tf = model.fit(disp=False)
print(res_tf.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_tf.smoother_results.smoothed_state[0,-1:][0]))

res_tf.plot_components()
plt.show()
```

The plotted components look good.  However, the estimated variance of the second seasonal term is inflated from reality.  Additionally, we reject the Ljung-Box statistic, indicating we may have remaining autocorrelation after accounting for our components.

### Unobserved components (lazy frequency domain modeling)

The third method is an unobserved components model with a fixed intercept and one seasonal component, which is modeled using trigonometric functions with primary periodicity 100 and 50 harmonics. Note that this is not the generating model, as it presupposes that there are more harmonics then in reality.  Because the variances are tied together, we are not able to drive the estimated covariance of the non-existent harmonics to 0.  What is lazy about this model specification is that we have not bothered to specify the two different seasonal components and instead chosen to model them using a single component with enough harmonics to cover both.  We will not be able to capture any differences in variances between the two true components.  The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \gamma^{(1)}_t + \epsilon_t\\
\mu_{t+1} &= \mu_t\\
\gamma^{(1)}_{t} &= \sum_{j=1}^{50}\gamma^{(1)}_{j, t}\\
\gamma^{(1)}_{j, t+1} &= \gamma^{(1)}_{j, t}\cos(\lambda_j) + \gamma^{*, (1)}_{j, t}\sin(\lambda_j) + \omega^{(1}_{j,t}, ~j = 1, 2, \dots, 50\\
\gamma^{*, (1)}_{j, t+1} &= -\gamma^{(1)}_{j, t}\sin(\lambda_j) + \gamma^{*, (1)}_{j, t}\cos(\lambda_j) + \omega^{*, (1)}_{j, t}, ~j = 1, 2, \dots, 50\\
\end{align}
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{t}$ are i.i.d. $N(0, \sigma^2_1)$.

```python
model = sm.tsa.UnobservedComponents(series,
                                    level='fixed intercept',
                                    freq_seasonal=[{'period': 100}])
res_lf = model.fit(disp=False)
print(res_lf.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_lf.smoother_results.smoothed_state[0,-1:][0]))

res_lf.plot_components()
plt.show()
```

Note that one of our diagnostic tests would be rejected at the .05 level.

### Unobserved components (lazy time domain seasonal modeling)

The fourth method is an unobserved components model with a fixed intercept and a single seasonal component modeled using a time-domain seasonal model of 100 constants. The process for the time series can be written as:

$$
\begin{align}
y_t & =\mu_t + \gamma^{(1)}_t + \epsilon_t\\
\mu_{t+1} &= \mu_{t} \\
\gamma^{(1)}_{t + 1} &= - \sum_{j=1}^{99} \gamma^{(1)}_{t + 1 - j} + \omega^{(1)}_t\\
\end{align}
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{t}$ are i.i.d. $N(0, \sigma^2_1)$.

```python
model = sm.tsa.UnobservedComponents(series,
                                    level='fixed intercept',
                                    seasonal=100)
res_lt = model.fit(disp=False)
print(res_lt.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_lt.smoother_results.smoothed_state[0,-1:][0]))

res_lt.plot_components()
plt.show()
```

The seasonal component itself looks good--it is the primary signal.  The estimated variance of the seasonal term is very high ($>10^5$), leading to a lot of uncertainty in our one-step-ahead predictions and slow responsiveness to new data, as evidenced by large errors in one-step ahead predictions and observations. Finally, all three of our diagnostic tests were rejected.

### Comparison of filtered estimates

The plots below show that explicitly modeling the individual components results in the filtered state being close to the true state within roughly half a period.  The lazy models took longer (almost a full period) to do the same on the combined true state.

```python
# Assign better names for our seasonal terms
true_seasonal_10_3 = terms[0]
true_seasonal_100_2 = terms[1]
true_sum = true_seasonal_10_3 + true_seasonal_100_2

```

```python
time_s = np.s_[:50]  # After this they basically agree
fig1 = plt.figure()
ax1 = fig1.add_subplot(111)
idx = np.asarray(series.index)
h1, = ax1.plot(idx[time_s], res_f.freq_seasonal[0].filtered[time_s], label='Double Freq. Seas')
h2, = ax1.plot(idx[time_s], res_tf.seasonal.filtered[time_s], label='Mixed Domain Seas')
h3, = ax1.plot(idx[time_s], true_seasonal_10_3[time_s], label='True Seasonal 10(3)')
plt.legend([h1, h2, h3], ['Double Freq. Seasonal','Mixed Domain Seasonal','Truth'], loc=2)
plt.title('Seasonal 10(3) component')
plt.show()
```

```python
time_s = np.s_[:50]  # After this they basically agree
fig2 = plt.figure()
ax2 = fig2.add_subplot(111)
h21, = ax2.plot(idx[time_s], res_f.freq_seasonal[1].filtered[time_s], label='Double Freq. Seas')
h22, = ax2.plot(idx[time_s], res_tf.freq_seasonal[0].filtered[time_s], label='Mixed Domain Seas')
h23, = ax2.plot(idx[time_s], true_seasonal_100_2[time_s], label='True Seasonal 100(2)')
plt.legend([h21, h22, h23], ['Double Freq. Seasonal','Mixed Domain Seasonal','Truth'], loc=2)
plt.title('Seasonal 100(2) component')
plt.show()
```

```python
time_s = np.s_[:100]

fig3 = plt.figure()
ax3 = fig3.add_subplot(111)
h31, = ax3.plot(idx[time_s], res_f.freq_seasonal[1].filtered[time_s] + res_f.freq_seasonal[0].filtered[time_s], label='Double Freq. Seas')
h32, = ax3.plot(idx[time_s], res_tf.freq_seasonal[0].filtered[time_s] + res_tf.seasonal.filtered[time_s], label='Mixed Domain Seas')
h33, = ax3.plot(idx[time_s], true_sum[time_s], label='True Seasonal 100(2)')
h34, = ax3.plot(idx[time_s], res_lf.freq_seasonal[0].filtered[time_s], label='Lazy Freq. Seas')
h35, = ax3.plot(idx[time_s], res_lt.seasonal.filtered[time_s], label='Lazy Time Seas')

plt.legend([h31, h32, h33, h34, h35], ['Double Freq. Seasonal','Mixed Domain Seasonal','Truth', 'Lazy Freq. Seas', 'Lazy Time Seas'], loc=1)
plt.title('Seasonal components combined')
plt.show()
```

##### Conclusions

In this notebook, we simulated a time series with two seasonal components of different periods.  We modeled them using structural time series models with (a) two frequency domain components of correct periods and numbers of harmonics, (b) time domain seasonal component for the shorter term and a frequency domain term with correct period and number of harmonics, (c) a single frequency domain term with the longer period and full number of harmonics, and (d) a single time domain term with the longer period.  We saw a variety of diagnostic results, with only the correct generating model, (a), failing to reject any of the tests.  Thus, more flexible seasonal modeling allowing for multiple components with specifiable harmonics can be a useful tool for time series modeling.  Finally, we can represent seasonal components with fewer total states in this way, allowing for the user to attempt to make the bias-variance trade-off themselves instead of being forced to choose "lazy" models, which use a large number of states and incur additional variance as a result.


# Stiff ODEs and implicit methods
In this notebook we look at examples where Runge-Kutta methods require very small steps to be accurate. For these **stiff** ODEs implicit methods are better.

Let's define our explicit methods first.

```python
import numpy as np
import matplotlib.pyplot as plt

# The below commands make the font and image size bigger
plt.rcParams.update({'font.size': 22})
plt.rcParams["figure.figsize"] = (15,10)
```

```python
def EulerStep(f, dx, xi, yi):
    return yi + dx*f(xi, yi)
```

```python
def RK2Step(f, dx, xi, yi):
    k1 = dx*f(xi, yi)
    k2 = dx*f(xi + dx, yi + k1)

    return yi + 0.5*(k1 + k2)
```

```python
def RK4Step(f, dx, xi, yi):
        k1 = dx*f(xi,yi)
        k2 = dx*f(xi + 0.5*dx, yi + 0.5*k1)
        k3 = dx*f(xi + 0.5*dx, yi + 0.5*k2)
        k4 = dx*f(xi + dx, yi + k3)

        return yi + 1/6*(k1 + 2*k2 + 2*k3 + k4)
```

```python
def ODESolve(f, dx, x0, y0, imax, method='RK4', plotSteps=False):

    xi = x0
    yi = y0

    # Create arrays to store the steps in
    steps = np.zeros((imax+1,2))
    steps[0,0] = x0
    steps[0,1] = y0

    i = 0
    while i < imax:
        if(method == 'RK4'):
            yi = RK4Step(f, dx, xi, yi)
        elif(method == 'RK2'):
            yi = RK2Step(f, dx, xi, yi)
        elif(method == 'Euler'):
            yi = EulerStep(f, dx, xi, yi)

        xi += dx
        i  += 1

        # Store the steps for plotting
        steps[i, 0] = xi
        steps[i, 1] = yi

    if(plotSteps):
        plt.scatter(steps[:,0], steps[:,1], color='red', linewidth='10')

    return [xi, yi]
```

# Stiff ODEs

Let's look at an example. Consider the *first-order* ODE:

$$ y'(x) = \lambda[-y(x) + \sin(x)] $$

with *initial conditions* $y(0) = 0$. This has the general solution of

$$ y(x) = C e^{-\lambda x} + \frac{\lambda^2 \sin(x) - \lambda \cos(x)}{1 + \lambda^2}$$

The initial condition gives

$$ C = \frac{\lambda}{1 + \lambda^2}$$

Let's define these equations and see how our RK methods do at solving the ODE. Try using the 'Euler', 'RK2' and 'RK4' methods below.

```python
lam = 30

def dydxStiff(x,y):
    global lam
    return lam*(-y + np.sin(x))

def yStiff(x):
    global lam
    C = lam/(1+lam**2)
    return C*np.exp(-lam*x) + (lam**2*np.sin(x) -lam*np.cos(x))/(1+lam**2)
```

```python
x = np.linspace(0, 2.0, 100)
y = yStiff(x)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.plot(x,y)

ODESolve(dydxStiff, 0.1, 0, 0, 10, 'RK4', True)
plt.legend(['Exact solution', 'Explicit RK4']);
```

We see that even the more accurate RK4 method does not do well. It can be made to perform better by decreasing the step size but we want to avoid this. Furthermore, if we increase $\lambda$ the step size required to accurately obtain the solution drops rapidly.

To overcome this, let's look at implicit methods. One of the simplest is the Backwards Euler method.

## Implicit method: backwards Euler

The backwards Euler method is given by:

$$ \begin{align}
    x_{i+1} &= x_i + \Delta x \\
    y_{i+1} &= y_i + \Delta x f(x_{i+1}, y_{i+1}) \\
    \end{align}
    $$

Notice how, unlike the *explicit* methods we've looked at until now this does not immediately give a recipe for the next step in terms of the previous steps. We must solve the second equation for $y_{i+1}$. In general we won't be able to find an analytic solution, so we might use *numerical root finding*.

Let's look at the stiff ODE example given above, where it turns out we can find an analytic solution. In this case we have:

$$ \begin{align}
    x_{i+1} &= x_i + \Delta x \\
    y_{i+1} &= y_i + \Delta x \lambda[-y_{i+1} + \sin(x_{i+1})]
\end{align}$$

In this case we can solve the second equation explicitly for $y_{i+1}$:

$$ y_{i+1} = \frac{y_i + \Delta x \lambda \sin(x_{i+1})}{1 + \Delta x \lambda}$$

Let's now implement this *specific* example in code and test it out.

```python
def BackwardsEulerSpecific(dx, x0, y0, imax, plotSteps=False):
    global lam

    xi = x0
    yi = y0

    # Create arrays to store the steps in
    steps = np.zeros((imax+1,2))
    steps[0,0] = x0
    steps[0,1] = y0

    i = 0
    while i < imax:
        yi = (yi + dx * lam*np.sin(xi+dx))/(1+ dx*lam)
        xi += dx
        i += 1

        # Store the steps for plotting
        steps[i, 0] = xi
        steps[i, 1] = yi

    if(plotSteps):
        plt.scatter(steps[:,0], steps[:,1], color='green', linewidth='10')

    return [xi, yi]
```

```python
x = np.linspace(0, 2.0, 100)
y = yStiff(x)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(True)
plt.plot(x,y)

ODESolve(dydxStiff, 0.1, 0, 0, 10, 'RK4', True)

BackwardsEulerSpecific(0.2, 0, 0, 10, True)

plt.legend(['Exact solution', 'Explicit RK4', 'Implicit backwards Euler']);
```

We can thus see that the (implicit) backwards Euler method is much more successful than the (explicit) RK4 method, even when it takes larger steps. This is great because each step of the RK4 method is more expensive than a step of the Euler method. The downside to implicit methods can be seen from the code above: our implicit ODE solver code is not general, it only works for this specific example. In general, implicit methods are more complicated to implement as they often require numerical methods, e.g., root finding to find $y_{i+1}$.

Note, unlike the forward Euler method, the backwards Euler method is stable across a much wider class of ODEs.

```python

```