# Moment dynamics generation

(c) 2020 Manuel Razo. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT).

---

```python
import pickle

# Library that we will use to export lambdify functions
import cloudpickle

# Library we'll use to generate possible pairs of numbers
import itertools

# Numerical workhorse
import numpy as np
import pandas as pd
import scipy as sp

# To compute symbolic expressions
import sympy
sympy.init_printing(use_unicode=True, use_latex=True) # print outputs in LaTeX

# Import matplotlib stuff for plotting
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib as mpl

# Seaborn, useful for graphics
import seaborn as sns

# Import the project utils
import ccutils

# Magic function to make matplotlib inline
%matplotlib inline

# This enables high resolution graphics inline.
%config InlineBackend.figure_format = 'retina'
```

```python
# Set PBoC plotting format
ccutils.viz.set_plotting_style()
# Increase dpi
mpl.rcParams['figure.dpi'] = 110
```

### $\LaTeX$ macros
$\newcommand{kpon}{k^{(p)}_{\text{on}}}$
$\newcommand{kpoff}{k^{(p)}_{\text{off}}}$
$\newcommand{kron}{k^{(r)}_{\text{on}}}$
$\newcommand{kroff}{k^{(r)}_{\text{off}}}$
$\newcommand{rm}{r _m}$
$\newcommand{gm}{\gamma _m}$
$\newcommand{ee}[1]{\left\langle #1 \right\rangle}$
$\newcommand{bb}[1]{\mathbf{#1}}$
$\newcommand{th}[1]{{#1}^{\text{th}}}$
$\newcommand{dt}[1]{{\partial{#1} \over \partial t}}$
$\newcommand{Km}{\bb{K}}$
$\newcommand{Rm}{\bb{R}_m}$
$\newcommand{Re}{\bb{R}_m'}$
$\newcommand{Gm}{\bb{\Gamma}_m}$
$\newcommand{Rp}{\bb{R}_p}$
$\newcommand{Gp}{\bb{\Gamma}_p}$

## General moment equation for simple repression architecture

For our system the master equation that describes the time evolution of the distribution is defined by either two or three differential equations, one for each state of the promoter such that

$$
P(m, p) = \sum_{s\in \text{states}} P_s(m, p),
\tag{1}
$$
where $s \in \{A, I, R\}$ defines the state $A =$ transcriptionally active, $I =$ transcriptionally inactive state, and $R =$ repressor bound. The third state is only include in the case where there is transcription factor present. Without loss of generality let's focus here on the three-state promoter. The results for the two-state promoter written in matrix notation look the same, only the definition of the matrices change. Let $\bb{P}(m, p) = (P_A(m, p), P_I(m, p), P_R(m, p))^T$ be the vector containing all distributions. Using this notation the system of PDEs that define the distribution is given by

$$
\dt{\bb{P}(m, p)} = \overbrace{
\left(\Km - \Rm - m\Gm - m\Rp -p\Gp \right) \bb{P}(m, p)
}^{\text{exit state }m,p}\\
\overbrace{
+ \Rm \bb{P}(m-1, p) + \Gm (m + 1) \bb{P}(m+1, p)\\
+ \Rp (m) \bb{P}(m, p-1) + \Gm (p + 1) \bb{P}(m, p+1)
}^{\text{enter state }m,p},
\label{master_matrix}
\tag{2}
$$
where $\Km$ is the matrix defining transition rates between states, $\Rm$ and $\Gm$ are the matrices defining the production and degradation rates of mRNA respectively, and $\Rp$ and $\Gp$ are the equivalent matrices for the production and degradation of protein.

Given this birth-death process with three different states of the promoter if we compute a moment $\ee{m^x p^y}$ we need to compute

$$
\ee{m^x p^y} = \ee{m^x p^y}_A + \ee{m^x p^y}_I + \ee{m^x p^y}_R,
\tag{3}
$$
i.e. the moment at each of the states of the promoter. Let
$\bb{\ee{m^x p^y}} = \left(\ee{m^x p^y}_A, \ee{m^x p^y}_I, \ee{m^x p^y}_R\right)^T$ be a vector containing all three moments. The moment PDE is then given by

$$
\dt{\bb{\ee{m^x p^y}}} = \dt{} \left[ \sum_m \sum_p m^x p^y \bb{P}(m,p)\right].
\tag{4}
$$

Applying this sum over all mRNA and protein counts we obtain

$$
\dt{\bb{\ee{m^x p^y}}} =
\sum_m \sum_p \left(\Km - \Rm - m\Gm - m\Rp -p\Gp \right)m^x p^y \bb{P}(m, p)\\
\overbrace{
+ \Rm \sum_m \sum_p m^x p^y \bb{P}(m-1, p)
}^{1}
\overbrace{
+ \Gm \sum_m \sum_p (m + 1) m^x p^y \bb{P}(m+1, p)
}^{2}\\
\overbrace{
+ \Rp \sum_m \sum_p (m) m^x p^y \bb{P}(m, p-1)
}^{3}
\overbrace{
+ \Gm \sum_m \sum_p (p + 1) m^x p^y \bb{P}(m, p+1)
}^{4}.
\tag{5}
$$
Each of the numbered terms have stereotypical "tricks" to simplify them. Let's list them (derivation left elsewhere):

$$
1: m' \equiv m - 1\\
\Rightarrow
\sum_m \sum_p m^x p^y \bb{P}(m-1, p) =
\sum_{m'} \sum_p (m' + 1)^x p^y \bb{P}(m', p) = \\
\bb{\ee{(m+1)^x p^y}},
\tag{6}
$$

$$
2: m' \equiv m + 1\\
\Rightarrow
\sum_m \sum_p (m + 1) m^x p^y \bb{P}(m + 1, p) =
\sum_{m'} \sum_p m' (m' - 1)^x p^y \bb{P}(m', p) = \\
\bb{\ee{m (m - 1)^x p^y}},
\tag{7}
$$

$$
3: p' \equiv p - 1\\
\Rightarrow
\sum_m \sum_p (m) m^x p^y \bb{P}(m, p-1) =
\sum_m \sum_{p'} m^{x + 1} (p' + 1)^y \bb{P}(m, p') = \\
\bb{\ee{m^{x + 1} (p +  1)^{y}}},
\tag{8}
$$

$$
4: p' \equiv p + 1\\
\Rightarrow
\sum_m \sum_p (p + 1) m^x p^y \bb{P}(m, p+1) =
\sum_m \sum_{p'} p' m^x (p' - 1)^y \bb{P}(m, p') = \\
\bb{\ee{m^x p (p - 1)^y}}.
\tag{9}
$$

Given these tricks we can write a **general** form for the moment PDE given by
$$
\dt{\bb{\ee{m^x p^y}}} =
\Km \bb{\ee{m^x p^y}} +\\
\Rm \left[ \bb{\ee{(m+1)^x p^y}} - \bb{\ee{m^x p^y}} \right] +\\
\Gm \left[ \bb{\ee{m (m - 1)^x p^y}} - \bb{\ee{m^{x + 1} p^y}} \right] +\\
\Rp \left[ \bb{\ee{m^{x + 1} (p +  1)^{y}}} - \bb{\ee{m^{x+1} p^y}} \right] +\\
\Gp \left[ \bb{\ee{m^x p (p - 1)^y}} - \bb{\ee{m^x p^{y+1}}} \right]
\tag{10}
$$

It can be shown that all moments $\bb{\ee{m^x p^y}}$ depend only on moments $\bb{\ee{m'^x p'^y}}$ such that two conditions are satisfied:

\begin{equation}
  \begin{aligned}
    &1) y' \leq y,\\
  &2) x' + y' \leq x + y.
  \end{aligned}
  \tag{11}
\end{equation}

What this implies is that this system has no moment closure problem since all moments depend only on lower degree moments. We can use this to our advantage to generate a general moment dynamics equation.

### General moment dynamics

Let $\bb{\mu^{(x, y)}}$ be a vector containing all moments up to $\bb{\ee{m^x p^y}}$ for all promoter states. This is

$$
\bb{\mu^{(x, y)}} = \left[ \bb{\ee{m^0 p^0}}, \bb{\ee{m^1 p^0}}, \ldots \bb{\ee{m^x p^y}} \right]^T.
\tag{12}
$$
Explicitly for the three-state promoter this vector looks like

$$
\bb{\mu^{(\bb{x, y})}} = \left[ \ee{m^0 p^0}_A, \ee{m^0 p^0}_I, \ee{m^0 p^0}_R, \ldots,
                 \ee{m^x p^y}_A, \ee{m^x p^y}_I, \ee{m^x p^y}_R \right]^T.
\tag{13}
$$
Given this definition we can define the general moment dynamics in matrix notation as

$$
\dt{\mu^{(\bb{x, y})}} = \bb{A \mu^{(\bb{x, y})}} + \bb{\nu},
\tag{14}
$$
where $\bb{A}$ is a square matrix that contains all the connections in the network, i.e. the numeric coefficients that relate each of the moments, and $\bb{\nu}$ is a vector of constant terms.

## Using `sympy` to compute moments

The objective of this notebook is to generate the matrix $\bb{A}$ and the vector $\bb{\nu}$ from the general moment equation derived earlier.

### Define general moment equation.

Let's define the `sympy` variables that we will need for the moment equation.

First we define the variables for the matrices.

```python
# Define the matrices involved in the general moment equation
Km = sympy.Symbol('{\mathbf{K}}')  # State transition matrix
Rm, Gm = sympy.symbols('{\mathbf{R}_m} {\mathbf{\Gamma}_m}')  # mRNA matrices
Rp, Gp = sympy.symbols('{\mathbf{R}_p} {\mathbf{\Gamma}_p}')  # protein matrices

Km, Rm, Gm, Rp, Gp
```

Let's define the variables that go into the matrices.

```python
# Define rate constant variables
kp_off, kp_on = sympy.symbols('{k_{off}^{(p)}} {k_{on}^{(p)}}')
kr_off, kr_on = sympy.symbols('{k_{off}^{(r)}} {k_{on}^{(r)}}')

# Define degradation rate and production rate
rm, gm = sympy.symbols('r_m gamma_m')
rp, gp = sympy.symbols('r_p gamma_p')

kp_off, kp_on, kr_off, kr_on, rm, gm, rp, gp
```

Now we define the mRNA and exponent variables

```python
# Define the mRNA and protein variables
m, p, = sympy.symbols('{\mathbf{m}} {\mathbf{p}}')
x, y = sympy.symbols('{\mathbf{x}} {\mathbf{y}}')

# As an extra variable let's define the time t
t = sympy.symbols('t')

# Let's also define the non-vector varirables for
# mRNA and protein
mm, pp = sympy.symbols('m p')

m**x, p**y, t, mm, pp
```

Let's now define the right hand side of the general moment equation.

```python
# Defining the general master moment equation
master_moment = Km * (m**x * p**y) +\
Rm * (p**y) * ((m + 1)**x - m**x) +\
Gm * (m * p**y) * ((m - 1)**x - m**x) +\
Rp * m**(x + 1) * ((p + 1)**y - p**y) +\
Gp * (m**x * p) * ((p - 1)**y - p**y)

master_moment
```

Having defined this equation now all we need to do to obtain any moment equation is to substitute $\bb{x}$ and $\bb{y}$. As a sanity check let's look at some examples that we already know the answer. Let's look at the first mRNA moment.

```python
# Define the first mRNA moment <m> equation
master_moment.subs([[x, 1], [y, 0]]).factor([m, p])
```

The term with $\Rm$ only is actually $\Rm \bb{m}^0$ which is exactly what one obtains when solving for this particular moment.

Let's look now at the second protein moment.

```python
master_moment.subs([[x, 0], [y, 2]]).factor([m, p])
```

This is again the answer one gets performing the calculation specifically for this moment.

### Extract coefficients of moment polynomial equation.

Let's now define a function that given an expression for a moment it returns a dictionary with all the coefficients of each of the elements in the equation. For example for the previous example of the second protein moment it should return something of the form

$$
\{
\bb{p}^2 : (\Km - 2\Gm), \;\;
\bb{p} : \Gp, \;\;
\bb{mp} : 2\Rp, \;\;
\bb{m} : \Rp
\}
\tag{15}
$$

This will be useful for when we substitute the matrices and vectors to solve the linear system.

```python
def coeff_dictionary(eq):
    '''
    Returns a dictionary with each of the coefficients for a given eqent
    equation.

    Parameter
    ---------
    eq : sympy expression.
        Sympy expression for the eqent equation

    Returns
    -------
    coeff_dict : dictionary.
        Dictionary containing all the coefficients of each of the elements
        in the polynomial eqent equation
    '''
    # Find the degree of the eqent for each of the variables
    if eq.has(m):
        m_degree = sympy.Poly(eq).degree(m)
    else:
        m_degree = 0
    if eq.has(p):
        p_degree = sympy.Poly(eq).degree(p)
    else:
        p_degree = 0

    return {m**x * p**y: eq.coeff(m**x * p**y).\
            subs([[m, 0], [p, 0]])
            for x in range(m_degree + 1)
            for y in range(p_degree + 1)}
```

Let's test the function.

```python
p2_dict = coeff_dictionary(master_moment.subs([[x, 0], [y, 2]]).factor([m, p]))
p2_dict
```

### Substituting definition of matrices

We now have the functions to use the general moment equation to obtain the coefficients for a specific moment. In order to assemble the matrix $\bb{A}$ we need to substitute the definition of the matrices $\Km$, $\Rm$, $\Gm$, $\Rp$, and $\Gp$ to convert from the matrix notation to the actual matrix.

**NOTE:** on `sympy` getting to substitute a term like $(\Km - \Gm)$ with the corresponding matrices and then actually doing the subtraction is unfortunately very convoluted. If you want to reproduce this make sure you follow the instructions.

```python
def coeff_subs(coeff_dict, Km_mat, Rm_mat, Gm_mat, Rp_mat, Gp_mat):
    '''

    Parameters
    ----------
    coeff_dict : dictionary.
        Dictionary containing all the coefficients associated with each of the
        moments in the moment equation.
    Km_mat, Rm_mat, Gm_mat, Rp_mat, Gp_mat: 2D sympy matrices.
        Sympy matrices that define the master equation.
        Km_mat : transition between states
        Rm_mat : mRNA produciton
        Gm_mat : mRNA degradation
        Rp_mat : protein production
        Gp_mat : protein degradation

    Returns
    -------
    mom_mat_dict : dictionary.
        Dictionary containing each of the substitutted coefficients into matrices
    '''
    # Initialize dictionary to save the matrices
    mom_mat_dict = dict()

    # Loop through each of the coefficients and compute the operation
    # NOTE: It is quite tricky to get it to work on sympy
    for key, value in coeff_dict.items():
        # Extract arguments for the item
        args = value.args

        # Check each of the possible cases

        # 1. args is empty and value is zero :
        # That is a term of the form {key : 0}
        # Generate a matrix of zeros
        if (len(args) == 0) & (value == 0):
            mom_mat_dict[key] = sympy.zeros(*Km_mat.shape)

        # 2. args is empty and value is not zero :
        # That is the case where the term is a single matrix
        # Substitute that value with the actual definition of the matrix
        elif (len(args) == 0) & (value != 0):
            mom_mat_dict[key] = value.subs([[Km, Km_mat],
                                            [Rm, Rm_mat],
                                            [Gm, Gm_mat],
                                            [Rp, Rp_mat],
                                            [Gp, Gp_mat]])

        # 3. args is not empty but one of the terms is an integer :
        # That is the case where we have Number * Matrix.
        # substitute the matrix and multiply it by the number
        elif (len(args) != 0) & (any([x.is_Number for x in args])):
            # Substitute value
            term_list = [x.subs([[Km, Km_mat],
                                 [Rm, Rm_mat],
                                 [Gm, Gm_mat],
                                 [Rp, Rp_mat],
                                 [Gp, Gp_mat]]) for x in value.args]
            # Multiply matrix by constant and register case
            mom_mat_dict[key] = np.prod(term_list)

        # 4. args is not empty and non of the elements is an integer :
        # Substitute matrices and reduce to single matrix.
        else:
            term_list = [x.subs([[Km, Km_mat],
                                 [Rm, Rm_mat],
                                 [Gm, Gm_mat],
                                 [Rp, Rp_mat],
                                 [Gp, Gp_mat]]) for x in value.args]

            # Perform a second round of checking. Elements that have for example
            # Number * Matrix are not explicitly multiplied. For this we will use
            # np.prod by splitting the terms again into its arguments and
            # multiplying the the arguments
            for i, term in enumerate(term_list):
                if len(term.args) == 2:
                    term_list[i] = np.prod(term.args)

            # Add the matrices. In order to do so:
            # the sum function has an optional "start" argument so you can
            # initialize it with a "zero object" of the kind you are adding.
            # In this case, with a zero matrix.
            mom_mat_dict[key] = sum(term_list, sympy.zeros(*Km_mat.shape))

    return mom_mat_dict
```

Let's test the matrix by defining the corresponding matrices for the three-state system.

```python
# Define the rate constant matrix
Km_reg = sympy.Matrix([[-kp_off, kp_on, 0],
                         [kp_off, -(kp_on + kr_on), kr_off],
                         [0, kr_on, -kr_off]])
# Define the production matrix
Rm_reg = sympy.Matrix([[rm, 0, 0],
                       [0, 0, 0],
                       [0, 0, 0]])
# Define the degradation matrix
Gm_reg = sympy.Matrix([[gm, 0, 0],
                       [0, gm, 0],
                       [0, 0, gm]])

# Define the production matrix
Rp_reg = sympy.Matrix([[rp, 0, 0],
                       [0, rp, 0],
                       [0, 0, rp]])

# Define the production matrix
Gp_reg = sympy.Matrix([[gp, 0, 0],
                       [0, gp, 0],
                       [0, 0, gp]])

Km_reg, Rm_reg, Gm_reg, Rp_reg, Gp_reg
```

Having defined these matrices, let's substitute it into the second-moment example we have been working with so far.

```python
p2_coeff_dict = coeff_subs(p2_dict, Km_reg, Rm_reg, Gm_reg,
                           Rp_reg, Gp_reg)

p2_coeff_dict
```

### Systematically finding the moments necessary to solve for $\bb{\ee{m^x p^y}}$

We established that a moment of the form $\bb{\ee{m^x p^y}}$ depends on lower moments that satisfy two conditions. We need to define a function that given the largest moment $\bb{\ee{m^x p^y}}$ to be included in the matrix it finds all of the moments $\bb{\ee{m^{x'} p^{y'}}}$ that satisfy such conditions.

```python
def expo_pairs(m_expo, p_expo):
    '''
    Finds all of the pairs of exponents x', y' that are necessary to compute the
    moment <m**m_expo * p**p_expo> that satisfy the necessary conditions:
    1) y' <= p_expo
    2) x' + y' <= m_expo + p_expo
    Parameters
    ----------
    m_expo, p_expo: int.
        Exponents of the highest moment to be included in the system.
        m_expo corresponds to the mRNA exponent
        p_expo corresponds to the protein exponent

    Returns
    -------
    pairs : list.
        List of sorted exponent pairs necessary for the computation
    NOTE: The sorting (which is not a necessary feature) given the general
    form of the equation only works up to the moment <m**0 * p**p_expo>.
    Any moment with x'>0 and y'=p_expo will be out of order appended at
    the end of the list.
    '''
    # Find all possible pair of exponents that satisfy
    # x' + y' <= m_expo + p_expo
    expo_pairs = list(itertools.permutations(range(m_expo + p_expo + 1), 2))

    # Add the (num, num) pair that are not being included
    expo_pairs = expo_pairs + [tuple([s, s]) for s in
                               range(max([m_expo, p_expo]) + 1)]

    # Remove pairs that do not satisfy the condition
    # y' <= p_expo
    expo_pairs = [s for s in expo_pairs if s[1] <= p_expo]

    # Remove pairs that do not satisfy the condition
    # x' <= m_expo + 1
#     expo_pairs = [x for x in expo_pairs if x[0] <= m_expo + 1]

    # # Remove pairs that do not satisfy the condition
    # x' + y' <= m_expo + p_expo
    expo_pairs = [s for s in expo_pairs if sum(s) <= m_expo + p_expo]

    ##  Moment sorting ##
    # Initialize list to append sorted moments
    expo_sorted = list()

    # Append mRNA moments
    mRNA_mom = sorted([s for s in expo_pairs if s[1] == 0])
    expo_sorted.append(mRNA_mom)

    # Find each protein moment
    protein_mom = sorted([s for s in expo_pairs if (s[0] == 0) & (s[1] != 0)])

    # Loop through each protein moment and find the cross correlations
    # associated with it
    for pr in protein_mom:
        cross_corr = sorted([s for s in expo_pairs
                             if (s[0] > 0) & (s[1] > 0) & (sum(s) == pr[1])],
                            reverse=True)
        # append it to the list
        expo_sorted.append(cross_corr)
        expo_sorted.append([pr])

    expo_sorted = list(itertools.chain.from_iterable(expo_sorted))

    # Append the other terms that are missing
    missing = [s for s in expo_pairs if s not in expo_sorted]

    return expo_sorted + missing
```

Let's test the function by finding the moments that would be needed to obtain the second moment of the protein distribution.

```python
# Define exponents for highest moment to be inferred
m_expo, p_expo = 0, 2

# Find the list of moments that need to be computed
expo_pairs(m_expo, p_expo)
```

These are indeed the necessary moments to compute the protein second moment. Now that we can generate this list we are able to build our matrix $\bb{A}$ by iteratively substitute these exponents on the general moment equation and fill the corresponding entries on the matrix.

```python
def A_matrix_gen(m_expo, p_expo, Km, Rm, Gm, Rp, Gp):
    '''
    Generates the matrix A to compute the moment dynamics of the form
    dµ/dt = A * µ
    It is basically a collection of the coefficients that go along
    each moment.

    Parameters
    ----------
    m_expo, p_expo: int.
        Exponents of the highest moment to be included in the system.
        m_expo corresponds to the mRNA exponent
        p_expo corresponds to the protein exponent
    Kmat : array-like.
        Matrix containing the transition rates between the promoter states.
    Rm : array-like.
        Matrix containing the mRNA production rate at each of the states.
    Gm : array-like.
        Matrix containing the mRNA degradation rate at each of the states.
    Rp : array-like.
        Matrix containing the protein production rate at each of the states.
    Gp : array-like.
        Matrix containing the protein degradation rate at each of the states.

    Returns
    -------
    A_matrix : 2D-array.
        Sympy matrix containing all of the coefficients to compute the
        moment dynamics.
    exponents : list.
        List of exponents as they are computed in the dynamics
    '''
    # Define number of dimensions
    n_dim = Km.shape[0]

    # Find exponents of moments needed to be computed
    exponents = expo_pairs(m_expo, p_expo)

    # Initalize matrix A
    A_matrix = sympy.Matrix(np.zeros([len(exponents) * n_dim,
                                      len(exponents) * n_dim]))

    # Generate dictionary that saves position of each moment
    # on the matrix
    idx_dict = {e: [i * n_dim, (i * n_dim) + (n_dim)] for
                i, e in enumerate(exponents)}

    # Loop through moments to subsittute exopnents into
    # general moment equation
    for i, (mexp, pexp) in enumerate(exponents):
        # Substitute exponents into general moment equation
        mom_eq = master_moment.subs([[x, mexp], [y, pexp]]).\
                               factor([m, p])
        # Obtain coefficients of equation
        mom_coeff = coeff_dictionary(mom_eq)
        # Substitute coefficients with matrices
        mom_subs = coeff_subs(mom_coeff, Km,
                              Rm, Gm, Rp, Gp)
        # Find row index on matrix A for substitutions
        row_idx = idx_dict[(mexp, pexp)]
        # Loop through coefficients to make substitutions
        for key, value in mom_subs.items():
            # Find exponents of the moment
            if key.has(m):
                m_degree = sympy.Poly(key).degree(m)
            else:
                m_degree = 0
            if key.has(p):
                p_degree = sympy.Poly(key).degree(p)
            else:
                p_degree = 0

            # Check if moment included in mom_sub.
            # Sometimes extra moments with no values are added
            if (m_degree, p_degree) in exponents:
                # Find index for columns
                col_idx = idx_dict[(m_degree, p_degree)]
                # Substitute values in corresponding entries
                A_matrix[row_idx[0]:row_idx[1],
                         col_idx[0]:col_idx[1]] = value

    return A_matrix, exponents
```

Let's test the function for a small case that we can visualize. Let's build the matrix to compute up to the first protein moment

```python
# Generate matrix for the first protein moment
A_mat, expo = A_matrix_gen(0, 1, Km_reg, Rm_reg, Gm_reg, Rp_reg, Gp_reg)

A_mat
```

Having built this matrix let's now generate $\bb{A}$ for the following cases:
1. unregulated promoter:

    A. mRNA up to $\bb{\ee{m^6}}$

    B. protein up to $\bb{\ee{p^6}}$

2. regulated promoter:

    A. mRNA up to $\bb{\ee{m^6}}$

    B. protein up to $\bb{\ee{p^6}}$

We will save the results as a `lambdify` function that we can import into other notebooks.

Let's first define the matrices for the unregulated case

```python
# Define the rate constant matrix
Km_unreg = sympy.Matrix([[-kp_off, kp_on],
                           [kp_off, -kp_on]])
# Define the mRNA production matrix
Rm_unreg = sympy.Matrix([[rm, 0],
                         [0, 0]])
# Define the mRNA degradation matrix
Gm_unreg = sympy.Matrix([[gm, 0],
                         [0, gm]])

# Define the protein production matrix
Rp_unreg = sympy.Matrix([[rp, 0],
                         [0, rp]])
# Define the protein degradation matrix
Gp_unreg = sympy.Matrix([[gp, 0],
                         [0, gp]])

Km_unreg, Rm_unreg, Gm_unreg, Rp_unreg, Gp_unreg
```

Now let's generate the dynamics for the unregulated case.

```python
# Generate mRNA dynamics
# Generate matrix for the first protein moment
A_m_unreg, expo_m = A_matrix_gen(6, 0, Km_unreg,
                                 Rm_unreg, Gm_unreg,
                                 Rp_unreg, Gp_unreg)

# Define array containing variables
var = [kp_on, kp_off, rm, gm]

# Generate labdify function
A_m_unreg_lamdify = sympy.lambdify(var, A_m_unreg)

# Export function into file
with open('./pkl_files/two_state_mRNA_dynamics_matrix.pkl', 'wb') as file:
    cloudpickle.dump(A_m_unreg_lamdify, file)
    cloudpickle.dump(expo_m, file)

# Generate protein dynamics
# Generate matrix for the first protein moment
A_p_unreg, expo_p = A_matrix_gen(0, 6, Km_unreg,
                               Rm_unreg, Gm_unreg,
                               Rp_unreg, Gp_unreg)

# Define array containing variables
var = [kp_on, kp_off, rm, gm, rp, gp]

# Generate labdify function
A_p_unreg_lamdify = sympy.lambdify(var, A_p_unreg)

# Export function into file
with open('./pkl_files/two_state_protein_dynamics_matrix.pkl', 'wb') as file:
    cloudpickle.dump(A_p_unreg_lamdify, file)
    cloudpickle.dump(expo_p, file)
```

Let's repeat these calculations for the regulated case

```python
# Generate mRNA dynamics
# Generate matrix for the first protein moment
A_m_reg, expo_m = A_matrix_gen(6, 0, Km_reg,
                               Rm_reg, Gm_reg,
                               Rp_reg, Gp_reg)

# Define array containing variables
var = [kr_on, kr_off, kp_on, kp_off, rm, gm]

# Generate labdify function
A_m_reg_lamdify = sympy.lambdify(var, A_m_reg)

# Export function into file
with open('./pkl_files/three_state_mRNA_dynamics_matrix.pkl', 'wb') as file:
    cloudpickle.dump(A_m_reg_lamdify, file)
    cloudpickle.dump(expo_m, file)

# Generate protein dynamics
# Generate matrix for the first protein moment
A_p_reg, expo_p = A_matrix_gen(0, 6, Km_reg,
                               Rm_reg, Gm_reg,
                               Rp_reg, Gp_reg)

# Define array containing variables
var = [kr_on, kr_off, kp_on, kp_off, rm, gm, rp, gp]

# Generate labdify function
A_p_reg_lamdify = sympy.lambdify(var, A_p_reg)

# Export function into file
with open('./pkl_files/three_state_protein_dynamics_matrix.pkl', 'wb') as file:
    cloudpickle.dump(A_p_reg_lamdify, file)
    cloudpickle.dump(expo_p, file)
```

# Numerical integration of moments

Having generated the matrix $\bb{A}$ we can now numerically integrate the moment dynamics
$$
\dt{\bb{\mu^{(\bb{x,y})}}} = \bb{A \mu^{(\bb{x, y})}}.
\tag{16}
$$

First we read the parameter values inferred for the unregulated promoter assuming a single promoter at steady state.

```python
# Define mRNA rate
# http://bionumbers.hms.harvard.edu/bionumber.aspx?id=105717&ver=3&trm=lacZ%20mRNA%20lifetime&org=
gm = 1 / (3 * 60)

# Load the flat-chain
with open('../../data/mcmc/lacUV5_constitutive_mRNA_prior.pkl',
          'rb') as file:
    unpickler = pickle.Unpickler(file)
    gauss_flatchain = unpickler.load()
    gauss_flatlnprobability = unpickler.load()

# Generate a Pandas Data Frame with the mcmc chain
index = ['kp_on', 'kp_off', 'rm']

# Generate a data frame out of the MCMC chains
df_mcmc = pd.DataFrame(gauss_flatchain, columns=index)

# rerbsine the index with the new entries
index = df_mcmc.columns

# map value of the parameters
max_idx = np.argmax(gauss_flatlnprobability, axis=0)
kp_on, kp_off, rm = df_mcmc.iloc[max_idx, :] * gm

# Define protein production and degradatino rates
gp = 1 / (60 * 100) # sec^-1
rp = 500 * gp # sec^-1
```

Let's now define the function to feed to the `scipy odeint` function that computes the right-hand side of the moment dynamics.

```python
# Read mRNA matrix
with open('./pkl_files/two_state_mRNA_dynamics_matrix.pkl', 'rb') as file:
    A_mat_unreg_lam = cloudpickle.load(file)
    expo = cloudpickle.load(file)

# Substitute value of parameters on matrix
A_mat_unreg = A_mat_unreg_lam(kp_on, kp_off, rm, gm)

def dmdt_unreg(mom, t):
    '''
    Function to integrate
    dµ/dt = Aµ
    for the unregulated promoter using the scipy.integrate.odeint
    routine

    Parameters
    ----------
    mom : array-like
        Array containing all of the moments included in the matrix
        dynamics A.
    t : array-like
        Time array

    Returns
    -------
    Right hand-side of the moment dynamics
    '''
    return np.dot(A_mat_unreg, mom)
```

Now we can define the time array to perform the integration along with the initial conditions to integrate the moment dynamics.

```python
# Define time on which to perform integration
t = np.linspace(0, 20 * 60, 301)

# Define initial conditions
mom_init = np.array([0] * len(expo) * 2)
# Set initial condition for zero moment
# Since this needs to add up to 1
mom_init[0] = 1

# Integrate dynamics
mom_dynamics = sp.integrate.odeint(dmdt_unreg, mom_init, t)

## Save results in tidy dataframe  ##
# Define promoter states names
states = ['P', 'E']
# Define names of columns
names = ['m{0:d}p{1:d}'.format(*x) + s for x in expo
         for s in states]

# Save as data frame
df_m_unreg = pd.DataFrame(mom_dynamics, columns=names)
# Add time column
df_m_unreg = df_m_unreg.assign(t_sec = t, t_min = t / 60)

df_m_unreg.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>m0p0P</th>
      <th>m0p0E</th>
      <th>m1p0P</th>
      <th>m1p0E</th>
      <th>m2p0P</th>
      <th>m2p0E</th>
      <th>m3p0P</th>
      <th>m3p0E</th>
      <th>m4p0P</th>
      <th>m4p0E</th>
      <th>m5p0P</th>
      <th>m5p0E</th>
      <th>m6p0P</th>
      <th>m6p0E</th>
      <th>t_sec</th>
      <th>t_min</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.0</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.659941</td>
      <td>0.340059</td>
      <td>1.493616</td>
      <td>0.364540</td>
      <td>4.888498</td>
      <td>0.901562</td>
      <td>19.407399</td>
      <td>2.876893</td>
      <td>89.243886</td>
      <td>11.155311</td>
      <td>461.904592</td>
      <td>50.250121</td>
      <td>2640.832786</td>
      <td>255.509555</td>
      <td>4.0</td>
      <td>0.066667</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.460954</td>
      <td>0.539046</td>
      <td>2.003875</td>
      <td>1.071424</td>
      <td>10.887711</td>
      <td>4.083923</td>
      <td>68.340001</td>
      <td>19.887471</td>
      <td>480.223792</td>
      <td>115.148767</td>
      <td>3705.985927</td>
      <td>760.370649</td>
      <td>31000.202542</td>
      <td>5583.922193</td>
      <td>8.0</td>
      <td>0.133333</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.344516</td>
      <td>0.655484</td>
      <td>2.103787</td>
      <td>1.810708</td>
      <td>15.580350</td>
      <td>9.076318</td>
      <td>130.500223</td>
      <td>57.722605</td>
      <td>1203.686080</td>
      <td>431.883456</td>
      <td>12034.052740</td>
      <td>3650.750231</td>
      <td>129018.530237</td>
      <td>34039.854937</td>
      <td>12.0</td>
      <td>0.200000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.276382</td>
      <td>0.723618</td>
      <td>2.057844</td>
      <td>2.471903</td>
      <td>18.765197</td>
      <td>15.023681</td>
      <td>192.706414</td>
      <td>115.706961</td>
      <td>2164.859611</td>
      <td>1043.653429</td>
      <td>26185.883182</td>
      <td>10583.813397</td>
      <td>337581.439606</td>
      <td>117837.242540</td>
      <td>16.0</td>
      <td>0.266667</td>
    </tr>
  </tbody>
</table>
</div>

Let's look at the dynamics of the first moment. We expect this to reach the steady state value given by
$$
\ee{m} = {r_m \over \gm} \left( {\kpon \over \kpon + \kpoff} \right).
\tag{17}
$$

```python
# Compute steady state value
m_ss = rm / gm * (kp_on / (kp_on + kp_off))

# Extract index for first and second moment
first_mom_names = [x for x in df_m_unreg.columns if 'm1p0' in x]

# Compute the mean mRNA copy number
m_mean = df_m_unreg.loc[:, first_mom_names].sum(axis=1)

# Initialize figure
fig = plt.figure(figsize=(4, 3))

# Plot mean mRNA
plt.plot(df_m_unreg.t_min, m_mean,
         label=r'$\left\langle m(t) \right\rangle$')

# Plot steady state
plt.hlines(m_ss, df_m_unreg.t_min.min(),
           df_m_unreg.t_min.max(),
           linestyle='--',
           label=r'$\left\langle m \right\rangle_{ss}$')

# Add legend
plt.legend(loc='lower right')

# Change axis extension
plt.xlim([-.5, 20])
# label axis
plt.xlabel('time (min)')
plt.ylabel(r'$\left\langle \right.$mRNA$\left. \right\rangle$ / cell')

plt.tight_layout()
```

The dynamics converge to the expected steady state value, showing that at least for $\ee{m}$ we obtain the expected result.

Let's now define a function that takes in general any matrix $\bb{A}$, a time array, initial conditions, and returns the dynamics in a tidy dataframe.

```python
def dmomdt(A_mat, expo, t, mom_init, states=['E', 'P', 'R']):
    '''
    Function to integrate
    dµ/dt = Aµ
    for any matrix A using the scipy.integrate.odeint
    function

    Parameters
    ----------
    A_mat : 2D-array
        Square matrix defining the moment dynamics
    expo : array-like
        List containing the moments involved in the
        dynamics defined by A
    t : array-like
        Time array in seconds
    mom_init : array-like. lenth = A_mat.shape[1]
    states : list with strings. Default = ['E', 'P', 'R']
        List containing the name of the promoter states
    Returns
    -------
    Tidy dataframe containing the moment dynamics
    '''
    # Define a lambda function to feed to odeint that returns
    # the right-hand side of the moment dynamics
    def dt(mom, time):
        return np.dot(A_mat, mom)

    # Integrate dynamics
    mom_dynamics = sp.integrate.odeint(dt, mom_init, t)

    ## Save results in tidy dataframe  ##
    # Define names of columns
    names = ['m{0:d}p{1:d}'.format(*x) + s for x in expo
             for s in states]

    # Save as data frame
    df = pd.DataFrame(mom_dynamics, columns=names)
    # Add time column
    df = df.assign(t_sec = t, t_min = t / 60)

    return df
```

Let's test this function by computing the unregulated protein dynamics.

```python
# Read protein ununregulated matrix
with open('./pkl_files/two_state_protein_dynamics_matrix.pkl', 'rb') as file:
    A_mat_unreg_lam = cloudpickle.load(file)
    expo = cloudpickle.load(file)

# Substitute value of parameters on matrix
A_mat_unreg = A_mat_unreg_lam(kp_on, kp_off, rm, gm, rp, gp)

# Define initial conditions
mom_init = np.array([0] * len(expo) * 2)
# Set initial condition for zero moment
# Since this needs to add up to 1
mom_init[0] = 1

# Define time on which to perform integration
t = np.linspace(0, 500 * 60, 2000)

# Define promoter states
states = ['P', 'E']

# Integrate dynamics
df = dmomdt(A_mat_unreg, expo, t, mom_init, states)

# Take a look at the resulting dataframe
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>m0p0P</th>
      <th>m0p0E</th>
      <th>m1p0P</th>
      <th>m1p0E</th>
      <th>m2p0P</th>
      <th>m2p0E</th>
      <th>m3p0P</th>
      <th>m3p0E</th>
      <th>m4p0P</th>
      <th>m4p0E</th>
      <th>...</th>
      <th>m3p3P</th>
      <th>m3p3E</th>
      <th>m2p4P</th>
      <th>m2p4E</th>
      <th>m1p5P</th>
      <th>m1p5E</th>
      <th>m0p6P</th>
      <th>m0p6E</th>
      <th>t_sec</th>
      <th>t_min</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>...</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000e+00</td>
      <td>0.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.290050</td>
      <td>0.709950</td>
      <td>2.075003</td>
      <td>2.317273</td>
      <td>18.096207</td>
      <td>13.500064</td>
      <td>177.707734</td>
      <td>99.653272</td>
      <td>1910.823426</td>
      <td>862.137474</td>
      <td>...</td>
      <td>1.019576e+05</td>
      <td>5.470264e+04</td>
      <td>9.188870e+04</td>
      <td>5.913926e+04</td>
      <td>9.056480e+04</td>
      <td>7.092658e+04</td>
      <td>9.701996e+04</td>
      <td>9.412560e+04</td>
      <td>15.007504</td>
      <td>0.250125</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.194978</td>
      <td>0.805022</td>
      <td>1.862248</td>
      <td>4.063813</td>
      <td>23.921085</td>
      <td>36.057456</td>
      <td>359.797968</td>
      <td>410.479192</td>
      <td>6011.375271</td>
      <td>5492.258641</td>
      <td>...</td>
      <td>5.081123e+06</td>
      <td>5.533142e+06</td>
      <td>7.661525e+06</td>
      <td>1.006979e+07</td>
      <td>1.216950e+07</td>
      <td>1.950664e+07</td>
      <td>2.034663e+07</td>
      <td>4.022157e+07</td>
      <td>30.015008</td>
      <td>0.500250</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.182246</td>
      <td>0.817754</td>
      <td>1.909144</td>
      <td>5.092269</td>
      <td>27.802650</td>
      <td>54.797810</td>
      <td>490.382834</td>
      <td>764.432627</td>
      <td>9863.377203</td>
      <td>12658.041675</td>
      <td>...</td>
      <td>4.175629e+07</td>
      <td>6.644891e+07</td>
      <td>9.164106e+07</td>
      <td>1.757869e+08</td>
      <td>2.102168e+08</td>
      <td>4.890169e+08</td>
      <td>5.043670e+08</td>
      <td>1.430709e+09</td>
      <td>45.022511</td>
      <td>0.750375</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.180542</td>
      <td>0.819458</td>
      <td>2.049678</td>
      <td>5.896106</td>
      <td>32.229689</td>
      <td>71.262453</td>
      <td>618.572091</td>
      <td>1116.776396</td>
      <td>13689.090548</td>
      <td>20872.732661</td>
      <td>...</td>
      <td>1.721680e+08</td>
      <td>3.382274e+08</td>
      <td>5.047451e+08</td>
      <td>1.188832e+09</td>
      <td>1.547368e+09</td>
      <td>4.379814e+09</td>
      <td>4.963345e+09</td>
      <td>1.690687e+10</td>
      <td>60.030015</td>
      <td>1.000500</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 58 columns</p>
</div>

Let's now take a look at the first moment of both mRNA and protein. Here we will plot the first moment dynamics normalized by the expected steady-state value. This will allow us to 1) compare both dynamics on the same plot, and 2) assess if the integration is converging to the expected value.

```python
# Compute steady state value
m_ss = rm / gm * (kp_on / (kp_on + kp_off))
p_ss = rp / gp * m_ss

# Extract index for first and second moment
m1p0_names = [x for x in df.columns if 'm1p0' in x]
m0p1_names = [x for x in df.columns if 'm0p1' in x]

# Compute the mean mRNA copy number
m_mean = df.loc[:, m1p0_names].sum(axis=1)
p_mean = df.loc[:, m0p1_names].sum(axis=1)

# Initialize figure
fig = plt.figure(figsize=(4, 3))

# Plot mean mRNA
plt.plot(df.t_min, m_mean / m_ss,
         label='mRNA')

# Plot mean protein
plt.plot(df.t_min, p_mean / p_ss,
         label=r'protein')

# Add legend
plt.legend(loc='lower right')

# Change axis extension
# label axis
plt.xlabel('time (min)')
plt.ylabel('normalized copy number')

plt.tight_layout()
```

We can see that given the difference in degradation rates between the mRNA and the protein, the protein dynamics take much longer to reach the steady state. This right away raises the question of how does the cell cycle affect the dynamics since the content of the cell is effectively halved every time the cell divides. In the next section we will explore this more explicitly.


Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2019-10-24 15:18:16 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.447527, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here
# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )
```

    Mean absolute error: 0.04
    Residual sum of squares (MSE): 0.00
    R2-score: 0.96

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.

8-9 Machine Learning with Python/ML0104EN-Reg-NoneLinearRegression-py-v1.ipynb


# Spectral Analysis of Deterministic Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Introduction

The analysis of the spectral properties of a signal plays an important role in digital signal processing. Some application examples are the

* [Spectrum analyzer](https://en.wikipedia.org/wiki/Spectrum_analyzer)
* Detection of (harmonic) signals
* [Estimation of fundamental frequency and harmonics](https://en.wikipedia.org/wiki/Modal_analysis)
* Spectral suppression: acoustic echo suppression, noise reduction, ...

In the practical realization of spectral analysis techniques, the [discrete Fourier transform](https://en.wikipedia.org/wiki/Discrete_Fourier_transform) (DFT) is applied to discrete finite-length signals in order to gain insights into their spectral composition. A basic task in spectral analysis is to determine the amplitude (and phase) of dominant harmonic contributions in a signal mixture. The properties of the DFT with respect to the analysis of an harmonic exponential signal are discussed in the following.

## The Leakage Effect

[Spectral leakage](https://en.wikipedia.org/wiki/Spectral_leakage) is a fundamental effect of the DFT. It limits the ability to detect harmonic signals in signal mixtures and hence the performance of spectral analysis. In order to discuss this effect, the DFT of a discrete exponential signal is revisited starting from the Fourier transform of the continuous exponential signal. The connections between the Fourier transform, the [discrete-time Fourier transform](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform) (DTFT) and the DFT for a uniformly sampled signal are illustrated below.

Consequently, the leakage effect is discussed in the remainder of this section by considering the following four steps:

1. Fourier transform of an harmonic exponential signal,
2. discrete-time Fourier transform (DTFT) of a discrete harmonic exponential signal, and
3. DTFT of a finite-length discrete harmonic exponential signal
4. sampling of the DTFT.

### Fourier Transform of an Exponential Signal

The harmonic exponential signal is defined as

\begin{equation}
x(t) = \mathrm{e}^{\,\mathrm{j}\, \omega_0 \, t}
\end{equation}

where $\omega_0 = 2 \pi f$ denotes the angular frequency of the signal. The Fourier transform of the exponential signal is

\begin{equation}
X(\mathrm{j}\, \omega) = \int\limits_{-\infty}^{\infty} x(t) \,\mathrm{e}^{\,- \mathrm{j}\, \omega \,t} \mathrm{d}t = 2\pi \; \delta(\omega - \omega_0)
\end{equation}

The spectrum consists of a single shifted Dirac impulse located at the angular frequency $\omega_0$ of the exponential signal. Hence the spectrum $X(\mathrm{j}\, \omega)$ consists of a clearly isolated and distinguishable event. In practice, it is not possible to compute the Fourier transform of a continuous signal by means of digital signal processing.

### Discrete-Time Fourier Transform of a Discrete Exponential Signal

Now lets consider sampled signals. The discrete exponential signal $x[k]$ is derived from its continuous counterpart $x(t)$ above by equidistant sampling $x[k] := x(k T)$ with the sampling interval $T$

\begin{equation}
x[k] = \mathrm{e}^{\,\mathrm{j}\, \Omega_0 \,k}
\end{equation}

where $\Omega_0 = \omega_0 T$ denotes the normalized angular frequency. The DTFT is the Fourier transform of a sampled signal. For the exponential signal it is given as (see e.g. [reference card discrete signals and systems](../reference_cards/RC_discrete_signals_and_systems.pdf))

\begin{equation}
X(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \sum_{k = -\infty}^{\infty} x[k]\, \mathrm{e}^{\,-\mathrm{j}\, \Omega \,k} = 2\pi \sum_{n = -\infty}^{\infty} \delta((\Omega-\Omega_0) - 2\,\pi\,n)
\end{equation}

The spectrum of the DTFT is $2\pi$-periodic due to sampling. As a consequence, the transform of the discrete exponential signal consists of a series Dirac impulses. For the region of interest $-\pi < \Omega \leq \pi$ the spectrum consists of a clearly isolated and distinguishable event, as for the continuous case.

The DTFT cannot be realized in practice, since is requires the knowledge of the signal $x[k]$ for all time instants $k$. In general, a measured signal is only known within a finite time-interval. The DFT of a signal of finite length can be derived from the DTFT in two steps:

1. truncation (windowing) of the signal and
2. sampling of the DTFT spectrum of the windowed signal.

The consequences of these two steps are investigated in the following.

### Discrete-Time Fourier Transform of a Truncated Discrete Exponential Signal

In general, truncation of a signal $x[k]$ to a length of $N$ samples is modeled by multiplying the signal with a window function $w[k]$ of length $N$

\begin{equation}
x_N[k] = x[k] \cdot w[k]
\end{equation}

where $x_N[k]$ denotes the truncated signal and $w[k] = 0$ for $\{k: k < 0 \wedge k \geq N \}$. The spectrum $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ can be derived from the multiplication theorem of the DTFT as

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \frac{1}{2 \pi} X(\mathrm{e}^{\,\mathrm{j}\, \Omega}) \circledast_N W(\mathrm{e}^{\,\mathrm{j}\, \Omega})
\end{equation}

where $\circledast_N$ denotes the cyclic/[circular convolution](https://en.wikipedia.org/wiki/Circular_convolution) of length $N$. A hard truncation of the signal to $N$ samples is modeled by the rectangular signal

\begin{equation}
w[k] = \text{rect}_N[k] = \begin{cases}
1 & \mathrm{for} \; 0\leq k<N \\
0 & \mathrm{otherwise}
\end{cases}
\end{equation}

Its spectrum is given as

\begin{equation}
W(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \mathrm{e}^{\,-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})}
\end{equation}

The DTFT $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ of the truncated exponential signal is derived by introducing the DTFT of the exponential signal and the window function into above cyclic convolution. Since, both the DTFT of the exponential signal and the window function are periodic with a period of $2 \pi$, the cyclic convolution with period $2 \pi$ is given by linear convolution of both spectra within $-\pi < \Omega \leq \pi$

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \delta(\Omega-\Omega_0) * \mathrm{e}^{\,-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})} =
\mathrm{e}^{\,-\mathrm{j}\, (\Omega-\Omega_0) \, \frac{N-1}{2}} \cdot \frac{\sin(\frac{N\, (\Omega-\Omega_0)}{2})}{\sin(\frac{(\Omega-\Omega_0)}{2})}
\end{equation}

Note that $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ is periodic with a period of $2 \pi$. Clearly the DTFT of the truncated harmonic exponential signal $x_N[k]$ is not given by a series of Dirac impulses. Above equation is evaluated numerically in order to illustrate the properties of $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

Om0 = 1  # frequency of exponential signal
N = 32  # length of signal

# DTFT of finite length exponential signal (analytic)
Om = np.linspace(-np.pi, np.pi, num=1024)
XN = np.exp(-1j * (Om-Om0) * (N-1) / 2) * (np.sin(N * (Om-Om0) / 2)) / (np.sin((Om-Om0) / 2))

# plot spectrum
plt.figure(figsize = (10, 8))
plt.plot(Om, abs(XN), 'r')
plt.title(r'Absolute value of the DTFT of a truncated exponential signal $e^{j \Omega_0 k}$ with $\Omega_0=$%2.2f' %Om0)
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|X_N(e^{j \Omega})|$')
plt.axis([-np.pi, np.pi, -0.5, N+5])
plt.grid()
```

**Exercise**

* Change the frequency `Om0` of the signal and rerun the example. How does the magnitude spectrum change?
* Change the length `N` of the signal and rerun the example. How does the magnitude spectrum change?

Solution: The maximum of the absolute value of the spectrum is located at the frequency $\Omega_0$. It should become clear that truncation of the exponential signal leads to a broadening of the spectrum. The shorter the signal, the wider the mainlobe becomes.

### The Leakage Effect of the Discrete Fourier Transform

The DFT is derived from the DTFT $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ of the truncated signal $x_N[k]$ by sampling the DTFT equidistantly at $\Omega = \mu \frac{2 \pi}{N}$

\begin{equation}
X[\mu] = X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})\big\vert_{\Omega = \mu \frac{2 \pi}{N}}
\end{equation}

For the DFT of the exponential signal we finally get

\begin{equation}
X[\mu] = \mathrm{e}^{\,\mathrm{j}\, (\Omega_0 - \mu \frac{2 \pi}{N}) \frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,(\Omega_0 - \mu \frac{2 \pi}{N})}{2})}{\sin(\frac{\Omega_0 - \mu \frac{2 \pi}{N}}{2})}
\end{equation}

The sampling of the DTFT is illustrated in the following example. Note that the normalized angular frequency $\Omega_0$ has been expressed in terms of the periodicity $P$ of the exponential signal $\Omega_0 = P \; \frac{2\pi}{N}$.

```python
N = 32  # length of the signal
P = 10.33  # periodicity of the exponential signal
Om0 = P * (2*np.pi/N)  # frequency of exponential signal

# truncated exponential signal
k = np.arange(N)
x = np.exp(1j*Om0*k)

# DTFT of finite length exponential signal (analytic)
Om = np.linspace(0, 2*np.pi, num=1024)
Xw = np.exp(-1j*(Om-Om0)*(N-1)/2)*(np.sin(N*(Om-Om0)/2))/(np.sin((Om-Om0)/2))

# DFT of the exponential signal by FFT
X = np.fft.fft(x)
mu = np.arange(N) * 2*np.pi/N

# plot spectra
plt.figure(figsize = (10, 8))
ax1 = plt.gca()

plt.plot(Om, abs(Xw), 'r', label=r'$|X_N(e^{j \Omega})|$')
plt.stem(mu, abs(X), label=r'$|X_N[\mu]|$', basefmt=' ')
plt.ylim([-0.5, N+5]);
plt.title(r'Absolute value of the DTFT/DFT of a truncated exponential signal $e^{j \Omega_0 k}$ with $\Omega_0=$%2.2f' %Om0, y=1.08)
plt.legend()

ax1.set_xlabel(r'$\Omega$')
ax1.set_xlim([Om[0], Om[-1]])
ax1.grid()

ax2 = ax1.twiny()
ax2.set_xlim([0, N])
ax2.set_xlabel(r'$\mu$', color='C0')
ax2.tick_params('x', colors='C0')
```

**Exercise**

* Change the periodicity `P` of the exponential signal and rerun the example. What happens if the periodicity is an integer? Why?
* Change the length `N` of the DFT? How does the spectrum change?
* What conclusions can be drawn for the analysis of a single exponential signal by the DFT?

Solution: You should have noticed that for an exponential signal whose periodicity is an integer $P \in \mathbb{N}$, the DFT consists of a discrete Dirac pulse $X[\mu] = N \cdot \delta[\mu - P]$. In this case, the sampling points coincide with the maximum of the main lobe or the zeros of the DTFT. For non-integer $P$, hence non-periodic exponential signals with respect to the signal length $N$, the DFT has additional contributions. The shorter the length $N$, the wider these contributions are spread in the spectrum. This smearing effect is known as *leakage effect* of the DFT. It limits the achievable frequency resolution of the DFT when analyzing signal mixtures consisting of more than one exponential signal or exponential signals under additive noise. This is illustrated by the following numerical examples.

### Analysis of Signal Mixtures by the Discrete Fourier Transform

In order to discuss the implications of the leakage effect when analyzing signal mixtures, the superposition of two exponential signals with different amplitudes and frequencies is considered

\begin{equation}
x_N[k] = A_1 \cdot e^{\mathrm{j} \Omega_1 k} + A_2 \cdot e^{\mathrm{j} \Omega_2 k}
\end{equation}

where $A_1, A_2 \in \mathbb{R}$. For convenience, a function is defined that calculates and plots the magnitude spectrum of $x_N[k]$.

```python
def dft_signal_mixture(N, A1, P1, A2, P2):
    # N: length of signal/DFT
    # A1, P1, A2, P2: amplitude and periodicity of 1st/2nd complex exponential

    # generate the signal mixture
    Om0_1 = P1 * (2*np.pi/N)  # frequency of 1st exponential signal
    Om0_2 = P2 * (2*np.pi/N)  # frequency of 2nd exponential signal
    k = np.arange(N)
    x = A1 * np.exp(1j*Om0_1*k) + A2 * np.exp(1j*Om0_2*k)

    # DFT of the signal mixture
    mu = np.arange(N)
    X = np.fft.fft(x)

    # plot spectrum
    plt.figure(figsize = (10, 8))
    plt.stem(mu, abs(X), basefmt=' ')
    plt.title(r'Absolute value of the DFT of a signal mixture')
    plt.xlabel(r'$\mu$')
    plt.ylabel(r'$|X[\mu]|$')
    plt.axis([0, N, -0.5, N+5]);
    plt.grid()
```

Lets first consider the case that the frequencies of the two exponentials are rather far apart in terms of normalized angular frequency

```python
dft_signal_mixture(32, 1, 10.3, 1, 15.2)
```

Investigating the magnitude spectrum one could conclude that the signal consists of two major contributions at the frequencies $\mu_1 = 10$ and $\mu_2 = 15$. Now lets take a look at a situation where the frequencies are closer together

```python
dft_signal_mixture(32, 1, 10.3, 1, 10.9)
```

From visual inspection of the spectrum it is rather unclear if the mixture consists of one or two exponential signals. So far the levels of both signals where chosen equal.

Lets consider the case where the second signal has a much lower level that the first one. The frequencies have been chosen equal to the first example

```python
dft_signal_mixture(32, 1, 10.3, 0.1, 15.2)
```

Now the contribution of the second exponential is almost hidden in the spread spectrum of the first exponential. From these examples it should have become clear that the leakage effect limits the spectral resolution of the DFT.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```python
%matplotlib inline
```

Neural Transfer with PyTorch
============================
Script modified from : http://pytorch.org/tutorials/advanced/neural_style_tutorial.html

**Author**: `Alexis Jacq <https://alexis-jacq.github.io>`_

Introduction
------------

Welcome! This tutorial explains how to impletment the
`Neural-Style <https://arxiv.org/abs/1508.06576>`__ algorithm developed
by Leon A. Gatys, Alexander S. Ecker and Matthias Bethge.

Neural what?
~~~~~~~~~~~~

The Neural-Style, or Neural-Transfer, is an algorithm that takes as
input a content-image (e.g. a tortle), a style-image (e.g. artistic
waves) and return the content of the content-image as if it was
'painted' using the artistic style of the style-image:

.. figure:: /_static/img/neural-style/neuralstyle.png
   :alt: content1

How does it work?
~~~~~~~~~~~~~~~~~

The principle is simple: we define two distances, one for the content
($D_C$) and one for the style ($D_S$). $D_C$ measures
how different the content is between two images, while $D_S$
measures how different the style is between two images. Then, we take a
third image, the input, (e.g. a with noise), and we transform it in
order to both minimize its content-distance with the content-image and
its style-distance with the style-image.

OK. How does it work?
^^^^^^^^^^^^^^^^^^^^^

Well, going further requires some mathematics. Let $C_{nn}$ be a
pre-trained deep convolutional neural network and $X$ be any
image. $C_{nn}(X)$ is the network fed by $X$ (containing
feature maps at all layers). Let $F_{XL} \in C_{nn}(X)$ be the
feature maps at depth layer $L$, all vectorized and concatenated
in one single vector. We simply define the content of $X$ at layer
$L$ by $F_{XL}$. Then, if $Y$ is another image of same
the size than $X$, we define the distance of content at layer
$L$ as follow:

\begin{align}D_C^L(X,Y) = \|F_{XL} - F_{YL}\|^2 = \sum_i (F_{XL}(i) - F_{YL}(i))^2\end{align}

Where $F_{XL}(i)$ is the $i^{th}$ element of $F_{XL}$.
The style is a bit less trivial to define. Let $F_{XL}^k$ with
$k \leq K$ be the vectorized $k^{th}$ of the $K$
feature maps at layer $L$. The style $G_{XL}$ of $X$
at layer $L$ is defined by the Gram produce of all vectorized
feature maps $F_{XL}^k$ with $k \leq K$. In other words,
$G_{XL}$ is a $K$\ x\ $K$ matrix and the element
$G_{XL}(k,l)$ at the $k^{th}$ line and $l^{th}$ column
of $G_{XL}$ is the vectorial produce between $F_{XL}^k$ and
$F_{XL}^l$ :

\begin{align}G_{XL}(k,l) = \langle F_{XL}^k, F_{XL}^l\rangle = \sum_i F_{XL}^k(i) . F_{XL}^l(i)\end{align}

Where $F_{XL}^k(i)$ is the $i^{th}$ element of
$F_{XL}^k$. We can see $G_{XL}(k,l)$ as a measure of the
correlation between feature maps $k$ and $l$. In that way,
$G_{XL}$ represents the correlation matrix of feature maps of
$X$ at layer $L$. Note that the size of $G_{XL}$ only
depends on the number of feature maps, not on the size of $X$.
Then, if $Y$ is another image *of any size*, we define the
distance of style at layer $L$ as follow:

\begin{align}D_S^L(X,Y) = \|G_{XL} - G_{YL}\|^2 = \sum_{k,l} (G_{XL}(k,l) - G_{YL}(k,l))^2\end{align}

In order to minimize in one shot $D_C(X,C)$ between a variable
image $X$ and target content-image $C$ and $D_S(X,S)$
between $X$ and target style-image $S$, both computed at
several layers , we compute and sum the gradients (derivative with
respect to $X$) of each distance at each wanted layer:

\begin{align}\nabla_{	extit{total}}(X,S,C) = \sum_{L_C} w_{CL_C}.\nabla_{	extit{content}}^{L_C}(X,C) + \sum_{L_S} w_{SL_S}.\nabla_{	extit{style}}^{L_S}(X,S)\end{align}

Where $L_C$ and $L_S$ are respectivement the wanted layers
(arbitrary stated) of content and style and $w_{CL_C}$ and
$w_{SL_S}$ the weights (arbitrary stated) associated with the
style or the content at each wanted layer. Then, we run a gradient
descent over $X$:

\begin{align}X \leftarrow X - \alpha \nabla_{	extit{total}}(X,S,C)\end{align}

Ok. That's enough with maths. If you want to go deeper (how to compute
the gradients) **we encourage you to read the original paper** by Leon
A. Gatys and AL, where everything is much better and much clearer
explained.

For our implementation in PyTorch, we already have everything
we need: indeed, with PyTorch, all the gradients are automatically and
dynamically computed for you (while you use functions from the library).
This is why the implementation of this algorithm becomes very
comfortable with PyTorch.

PyTorch implementation
----------------------

If you are not sure to understand all the mathematics above, you will
probably get it by implementing it. If you are discovering PyTorch, we
recommend you to first read this :doc:`Introduction to
PyTorch </beginner/deep_learning_60min_blitz>`.

Packages
~~~~~~~~

We will have recourse to the following packages:

-  ``torch``, ``torch.nn``, ``numpy`` (indispensables packages for
   neural networks with PyTorch)
-  ``torch.autograd.Variable`` (dynamic computation of the gradient wrt
   a variable)
-  ``torch.optim`` (efficient gradient descents)
-  ``PIL``, ``PIL.Image``, ``matplotlib.pyplot`` (load and display
   images)
-  ``torchvision.transforms`` (treat PIL images and transform into torch
   tensors)
-  ``torchvision.models`` (train or load pre-trained models)
-  ``copy`` (to deep copy the models; system package)

```python
from __future__ import print_function

import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.optim as optim

from PIL import Image
import matplotlib.pyplot as plt

import torchvision.transforms as transforms
import torchvision.models as models

import numpy as np

import copy
```

Cuda
~~~~

If you have a GPU on your computer, it is preferable to run the
algorithm on it, especially if you want to try larger networks (like
VGG). For this, we have ``torch.cuda.is_available()`` that returns
``True`` if you computer has an available GPU. Then, we can use method
``.cuda()`` that moves allocated proccesses associated with a module
from the CPU to the GPU. When we want to move back this module to the
CPU (e.g. to use numpy), we use the ``.cpu()`` method. Finally,
``.type(dtype)`` will be use to convert a ``torch.FloatTensor`` into
``torch.cuda.FloatTensor`` to feed GPU processes.

```python
use_cuda = torch.cuda.is_available()
dtype = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor
```

Load images
~~~~~~~~~~~

In order to simplify the implementation, let's start by importing a
style and a content image of the same dimentions. We then scale them to
the desired output image size (128 or 512 in the example, depending on gpu
availablity) and transform them into torch tensors, ready to feed
a neural network:

.. Note::
    Here are links to download the images required to run the tutorial:
    `picasso.jpg <http://pytorch.org/tutorials/_static/img/neural-style/picasso.jpg>`__ and
    `dancing.jpg <http://pytorch.org/tutorials/_static/img/neural-style/dancing.jpg>`__.
    Download these two images and add them to a directory
    with name ``images``

```python
# desired size of the output image
imsize = 512 if use_cuda else 128  # use small size if no gpu

loader = transforms.Compose([
    transforms.Scale(imsize),  # scale imported image
    transforms.ToTensor()])  # transform it into a torch tensor

def image_loader(image_name):
    image = Image.open(image_name)
    image = Variable(loader(image))
    # fake batch dimension required to fit network's input dimensions
    image = image.unsqueeze(0)
    return image

style_imgs = [
    image_loader("images/background.jpg").type(dtype)
]
content_img = image_loader("images/person.jpg").type(dtype)

# assert style_img.size() == content_img.size(), \
#    "we need to import style and content images of the same size"
```

    /home/samuel_cahyawijaya/anaconda2/envs/py36/lib/python3.6/site-packages/torchvision-0.2.0-py3.6.egg/torchvision/transforms/transforms.py:156: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.

Imported PIL images has values between 0 and 255. Transformed into torch
tensors, their values are between 0 and 1. This is an important detail:
neural networks from torch library are trained with 0-1 tensor image. If
you try to feed the networks with 0-255 tensor images the activated
feature maps will have no sense. This is not the case with pre-trained
networks from the Caffe library: they are trained with 0-255 tensor
images.

Display images
~~~~~~~~~~~~~~

We will use ``plt.imshow`` to display images. So we need to first
reconvert them into PIL images:

```python
unloader = transforms.ToPILImage()  # reconvert into PIL image

plt.ion()

def imshow(tensor, title=None):
    image = tensor.clone().cpu()  # we clone the tensor to not do changes on it
    image = image.view(3, imsize, imsize)  # remove the fake batch dimension
    image = unloader(image)
    plt.imshow(image)
    if title is not None:
        plt.title(title)
    plt.pause(0.001) # pause a bit so that plots are updated

for i in range(len(style_imgs)):
    plt.figure()
    imshow(style_imgs[i].data, title='Style Image')

plt.figure()
imshow(content_img.data, title='Content Image')
```

Content loss
~~~~~~~~~~~~

The content loss is a function that takes as input the feature maps
$F_{XL}$ at a layer $L$ in a network fed by $X$ and
return the weigthed content distance $w_{CL}.D_C^L(X,C)$ between
this image and the content image. Hence, the weight $w_{CL}$ and
the target content $F_{CL}$ are parameters of the function. We
implement this function as a torch module with a constructor that takes
these parameters as input. The distance $\|F_{XL} - F_{YL}\|^2$ is
the Mean Square Error between the two sets of feature maps, that can be
computed using a criterion ``nn.MSELoss`` stated as a third parameter.

We will add our content losses at each desired layer as additive modules
of the neural network. That way, each time we will feed the network with
an input image $X$, all the content losses will be computed at the
desired layers and, thanks to autograd, all the gradients will be
computed. For that, we just need to make the ``forward`` method of our
module returning the input: the module becomes a ''transparent layer''
of the neural network. The computed loss is saved as a parameter of the
module.

Finally, we define a fake ``backward`` method, that just call the
backward method of ``nn.MSELoss`` in order to reconstruct the gradient.
This method returns the computed loss: this will be useful when running
the gradient descent in order to display the evolution of style and
content losses.

```python
class ContentLoss(nn.Module):

    def __init__(self, target, weight):
        super(ContentLoss, self).__init__()
        # we 'detach' the target content from the tree used
        self.target = target.detach() * weight
        # to dynamically compute the gradient: this is a stated value,
        # not a variable. Otherwise the forward method of the criterion
        # will throw an error.
        self.weight = weight
        self.criterion = nn.MSELoss()

    def forward(self, input):
        self.loss = self.criterion(input * self.weight, self.target)
        self.output = input
        return self.output

    def backward(self, retain_graph=True):
        self.loss.backward(retain_graph=retain_graph)
        return self.loss
```

.. Note::
   **Important detail**: this module, although it is named ``ContentLoss``,
   is not a true PyTorch Loss function. If you want to define your content
   loss as a PyTorch Loss, you have to create a PyTorch autograd Function
   and to recompute/implement the gradient by the hand in the ``backward``
   method.

Style loss
~~~~~~~~~~

For the style loss, we need first to define a module that compute the
gram produce $G_{XL}$ given the feature maps $F_{XL}$ of the
neural network fed by $X$, at layer $L$. Let
$\hat{F}_{XL}$ be the re-shaped version of $F_{XL}$ into a
$K$\ x\ $N$ matrix, where $K$ is the number of feature
maps at layer $L$ and $N$ the lenght of any vectorized
feature map $F_{XL}^k$. The $k^{th}$ line of
$\hat{F}_{XL}$ is $F_{XL}^k$. We let you check that
$\hat{F}_{XL} \cdot \hat{F}_{XL}^T = G_{XL}$. Given that, it
becomes easy to implement our module:

```python
class GramMatrix(nn.Module):

    def forward(self, input):
        a, b, c, d = input.size()  # a=batch size(=1)
        # b=number of feature maps
        # (c,d)=dimensions of a f. map (N=c*d)

        features = input.view(a * b, c * d)  # resise F_XL into \hat F_XL

        G = torch.mm(features, features.t())  # compute the gram product

        # we 'normalize' the values of the gram matrix
        # by dividing by the number of element in each feature maps.
        return G.div(a * b * c * d)
```

The longer is the feature maps dimension $N$, the bigger are the
values of the gram matrix. Therefore, if we don't normalize by $N$,
the loss computed at the first layers (before pooling layers) will have
much more importance during the gradient descent. We dont want that,
since the most interesting style features are in the deepest layers!

Then, the style loss module is implemented exactly the same way than the
content loss module, but we have to add the ``gramMatrix`` as a
parameter:

```python
class StyleLoss(nn.Module):

    def __init__(self, target, weight):
        super(StyleLoss, self).__init__()
        self.target = target.detach() * weight
        self.weight = weight
        self.gram = GramMatrix()
        self.criterion = nn.MSELoss()

    def forward(self, input):
        self.output = input.clone()
        self.G = self.gram(input)
        self.G.mul_(self.weight)
        self.loss = self.criterion(self.G, self.target)
        return self.output

    def backward(self, retain_graph=True):
        self.loss.backward(retain_graph=retain_graph)
        return self.loss
```

Load the neural network
~~~~~~~~~~~~~~~~~~~~~~~

Now, we have to import a pre-trained neural network. As in the paper, we
are going to use a pretrained VGG network with 19 layers (VGG19).

PyTorch's implementation of VGG is a module divided in two child
``Sequential`` modules: ``features`` (containing convolution and pooling
layers) and ``classifier`` (containing fully connected layers). We are
just interested by ``features``:

```python
cnn = models.vgg19(pretrained=True).features

# move it to the GPU if possible:
if use_cuda:
    cnn = cnn.cuda()
```

A ``Sequential`` module contains an ordered list of child modules. For
instance, ``vgg19.features`` contains a sequence (Conv2d, ReLU,
Maxpool2d, Conv2d, ReLU...) aligned in the right order of depth. As we
said in *Content loss* section, we wand to add our style and content
loss modules as additive 'transparent' layers in our network, at desired
depths. For that, we construct a new ``Sequential`` module, in wich we
are going to add modules from ``vgg19`` and our loss modules in the
right order:

```python
# desired depth layers to compute style/content losses :
content_layers_default = ['conv_4']
style_layers_default = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

def get_style_model_and_losses(cnn, style_img, content_img,
                               style_weight=1000, content_weight=1,
                               content_layers=content_layers_default,
                               style_layers=style_layers_default):
    cnn = copy.deepcopy(cnn)

    # just in order to have an iterable access to or list of content/syle
    # losses
    content_losses = []
    style_losses = []

    model = nn.Sequential()  # the new Sequential module network
    gram = GramMatrix()  # we need a gram module in order to compute style targets

    # move these modules to the GPU if possible:
    if use_cuda:
        model = model.cuda()
        gram = gram.cuda()

    i = 1
    for layer in list(cnn):
        if isinstance(layer, nn.Conv2d):
            name = "conv_" + str(i)
            model.add_module(name, layer)

            if name in content_layers:
                # add content loss:
                target = model(content_img).clone()
                content_loss = ContentLoss(target, content_weight)
                model.add_module("content_loss_" + str(i), content_loss)
                content_losses.append(content_loss)

            if name in style_layers:
                # add style loss:
                target_feature = model(style_img).clone()
                target_feature_gram = gram(target_feature)
                style_loss = StyleLoss(target_feature_gram, style_weight)
                model.add_module("style_loss_" + str(i), style_loss)
                style_losses.append(style_loss)

        if isinstance(layer, nn.ReLU):
            name = "relu_" + str(i)
            model.add_module(name, layer)

            if name in content_layers:
                # add content loss:
                target = model(content_img).clone()
                content_loss = ContentLoss(target, content_weight)
                model.add_module("content_loss_" + str(i), content_loss)
                content_losses.append(content_loss)

            if name in style_layers:
                # add style loss:
                target_feature = model(style_img).clone()
                target_feature_gram = gram(target_feature)
                style_loss = StyleLoss(target_feature_gram, style_weight)
                model.add_module("style_loss_" + str(i), style_loss)
                style_losses.append(style_loss)

            i += 1

        if isinstance(layer, nn.MaxPool2d):
            name = "pool_" + str(i)
            model.add_module(name, layer)  # ***

    return model, style_losses, content_losses
```

.. Note::
   In the paper they recommend to change max pooling layers into
   average pooling. With AlexNet, that is a small network compared to VGG19
   used in the paper, we are not going to see any difference of quality in
   the result. However, you can use these lines instead if you want to do
   this substitution:

   ::

       # avgpool = nn.AvgPool2d(kernel_size=layer.kernel_size,
       #                         stride=layer.stride, padding = layer.padding)
       # model.add_module(name,avgpool)

Input image
~~~~~~~~~~~

Again, in order to simplify the code, we take an image of the same
dimensions than content and style images. This image can be a white
noise, or it can also be a copy of the content-image.

```python
input_img = content_img.clone()
# input_img = np.random.rand(content_img.shape[0],content_img.shape[1],content_img.shape[2])
# if you want to use a white noise instead uncomment the below line:
# input_img = Variable(torch.randn(content_img.data.size())).type(dtype)

# add the original input image to the figure:
plt.figure()
imshow(input_img.data, title='Input Image')
```

Gradient descent
~~~~~~~~~~~~~~~~

As Leon Gatys, the author of the algorithm, suggested
`here <https://discuss.pytorch.org/t/pytorch-tutorial-for-neural-transfert-of-artistic-style/336/20?u=alexis-jacq>`__,
we will use L-BFGS algorithm to run our gradient descent. Unlike
training a network, we want to train the input image in order to
minimise the content/style losses. We would like to simply create a
PyTorch  L-BFGS optimizer, passing our image as the variable to optimize.
But ``optim.LBFGS`` takes as first argument a list of PyTorch
``Variable`` that require gradient. Our input image is a ``Variable``
but is not a leaf of the tree that requires computation of gradients. In
order to show that this variable requires a gradient, a possibility is
to construct a ``Parameter`` object from the input image. Then, we just
give a list containing this ``Parameter`` to the optimizer's
constructor:

```python
def get_input_param_optimizer(input_img):
    # this line to show that input is a parameter that requires a gradient
    input_param = nn.Parameter(input_img.data)
    optimizer = optim.Adam([input_param])
    return input_param, optimizer
```

**Last step**: the loop of gradient descent. At each step, we must feed
the network with the updated input in order to compute the new losses,
we must run the ``backward`` methods of each loss to dynamically compute
their gradients and perform the step of gradient descent. The optimizer
requires as argument a "closure": a function that reevaluates the model
and returns the loss.

However, there's a small catch. The optimized image may take its values
between $-\infty$ and $+\infty$ instead of staying between 0
and 1. In other words, the image might be well optimized and have absurd
values. In fact, we must perform an optimization under constraints in
order to keep having right vaues into our input image. There is a simple
solution: at each step, to correct the image to maintain its values into
the 0-1 interval.

```python
def run_style_transfer(cnn, content_img, style_img, input_img, num_steps=2000,
                       style_weight=10000, content_weight=1):
    """Run the style transfer."""
    print('Building the style transfer model..')
    model, style_losses, content_losses = get_style_model_and_losses(cnn,
        style_img, content_img, style_weight, content_weight)
    input_param, optimizer = get_input_param_optimizer(input_img)

    print('Optimizing..')
    run = [0]
    while run[0] <= num_steps:

        def closure():
            # correct the values of updated input image
            input_param.data.clamp_(-5, 5)

            optimizer.zero_grad()
            model(input_param)
            style_score = 0
            content_score = 0

            for sl in style_losses:
                style_score += sl.backward()
            for cl in content_losses:
                content_score += cl.backward()

            run[0] += 1
            if run[0] % 50 == 0:
                print("run {}:".format(run))
                print('Style Loss : {:4f} Content Loss: {:4f}'.format(
                    style_score.data[0], content_score.data[0]))
                print()

            return style_score + content_score

        optimizer.step(closure)

    # a last correction...
    input_param.data.clamp_(0, 1)

    return input_param.data
```

Finally, run the algorithm

```python
for i in range(len(style_imgs)):
    output = run_style_transfer(cnn, content_img, style_imgs[i], input_img)

    img_data = output.cpu().numpy()
    img_data = (img_data * 255).astype('uint8').T.reshape((512, 512, 3))

    im = Image.fromarray(img_data)
    im.save('output/outfile_{}.jpg'.format(i))

    plt.figure()
    imshow(output, title='Output Image')

# sphinx_gallery_thumbnail_number = 4
plt.ioff()
plt.show()
```

```python

```


# Project 5, deadline  December 16, 2020

**[Computational Physics I FYS3150/FYS4150](http://www.uio.no/studier/emner/matnat/fys/FYS3150/index-eng.html)**, Department of Physics, University of Oslo, Norway

Date: **Fall semester 2020**

## Studies of social interactions using the Ising model

The Ising model is one of the most
frequently used models of statistical mechanics. Recently, this
model has also become widely studied in many other disciplines, including mathematics, statistics, biology, [economy](https://www.springer.com/gp/book/9783319477046) and [sociology](https://www.springer.com/gp/book/9781461420316).

**For this project you can hand in collaborative reports and programs.**
This project (together with projects 3 and 4) counts 1/3 of the final mark.

### Introduction

The aim of this project is to use the Ising model from project 4 and
apply it to the modeling of electoral patterns and social interactions.
This project aims at studying the work of [Katarzyna Sznajd-Weron and
Jozef Sznajd](https://arxiv.org/abs/cond-mat/0101130), see also their
published work in [Int. J. Mod. Phys. C11,(2000) 1157](https://www.worldscientific.com/doi/abs/10.1142/S0129183100000936).

We repeat
In its simplest form
the energy of the Ising model from project 4 without an externally applied magnetic field,

$$
E=-J\sum_{< kl >}^{N}s_ks_l
$$

with
$s_k=\pm 1$. The quantity $N$ represents the total number of spins and $J$ is a coupling
constant expressing the strength of the interaction between
neighboring spins.  The symbol $<kl>$ indicates that we sum over
nearest neighbors only.
ordering, viz $J> 0$.

We will  consider as our model a community which time and again should
take a stand in some matter, for example vote on a president in
a two-party system. If each member of the
community can take only two attitudes ($A$ or $B$) then in several
votes one expects some difference $m$ of voters for $A$ and against.
We assume three limiting cases:

1. all members of the community vote for $A$ (an **all** $A$ state),

2. all members of the community vote for $B$ (an **all** $B$ state),

3. $50\%$ vote for $A$ and $50\%$ vote for $B$,

where the latter is meant to  be the stable solutions of our model.

The aim here is to analyze the time evolution of $m$. To model the
above mentioned system we consider an Ising spins chain
($S_i;i=1,2,\ldots N$) with the following dynamic rules:

* if $S_iS_{i+1}=1$ then $S_{i-1}$ and $S_{i+2}$ take the direction of the pair (i,i+1),

* if $S_iS_{i+1}=-1$ then $S_{i-1}$ takes the direction of $S_{i+1}$ and $S_{i+2}$ the direction of $S_i$.

These rules describe the influence of a given pair on the decision of
its nearest neighbours. When members of a pair have the same opinion
then their nearest neighbours agree with them. On the contrary, when
members of a pair have opinions different then the nearest neighbour
of each member disagrees with her/him.  These dynamic rules lead to
the three steady states above.  However, the third steady state ($50\%$
for $A$ and $50\%$ for $B$) is realized in a very special way.  Every
member of the community disagrees with her/his nearest neighbour (it
is easy to see that the Ising model with only next nearest neighbour
interaction has such fixed points: ferro- and antiferromagnetic
state).  According to the authors of the above mentioned article,
this rule is in accordance with the well known sentence
*united we stand divided we fall* (USDF-model).

### Project 5a):

To investigate our model we perform a standard Monte Carlo simulation
with random updating.  Consider a chain of $N$ Ising spins with free
boundary conditions. In our simulation you could use values from
$N=1000$ up to $N=10000$. We start from a totally random initial state
i.e. to each site of the chain we assign an arrow with a randomly
chosen direction: up or down (Ising spin).  Show that you obtain as a
state, one of the three fixed points (1-3, i.e. AAAA, BBBB, ABAB) with
probability 0.25,0.25 and 0.5, respectively. The typical relaxation
time for $N=1000$ is $\sim 10^4$ Monte Carlo steps (MCS). Plot the
spatial distribution of spins from the initial to a steady state and
compare your results with figure 1 of [Katarzyna Sznajd-Weron and
Jozef Sznajd](https://arxiv.org/abs/cond-mat/0101130). Can you see
whether there is a formation of clusters?  Comment your results.

### Project 5b): Magnetization and Social Mood

Let us define the decision as a magnetization, i.e.:

$$
\begin{equation}
m=\frac{1}{N}\sum_{i=1}^N S_i.
\label{em} \tag{1}
\end{equation}
$$

Compute the magnetization and compare your results with figure 2 of [Katarzyna Sznajd-Weron and
Jozef Sznajd](https://arxiv.org/abs/cond-mat/0101130).

Without any external stimulation decision can change
dramatically in a relatively short time.
Such strongly non-monotonic behaviour of the change of $m$
is typically observed in the USDF-model when the system evolves
towards the third steady state (total disagreement or in magnetic
language the antiferromagnetic state). Comment your results.

### Project 5c): Autocorrelation Function

To measure the time correlation of $m$ one can employ the
classical autocorrelation function:

$$
\begin{equation}
G(\Delta t) = \frac
{\sum \left( m(t)-<m>\right) \left( m(t+ \Delta t)- <m>\right)}
{\sum (m(t)-<m>)^2}.
\label{_auto1} \tag{2}
\end{equation}
$$

In the work of Sznajd-Weron and Snajzd, there is a comparison of
simulation results with empirical data as shown in their figure 3. Make a plot similar to their figure 3 and comment your results.
Following the changes of one particular individual, the dynamics seem to lead to some interesting effects. If an individual changes her/his opinion at time $t$ she/he will probably
change it also at time $t+1$.

 On the other hand
an individual can stay for a long time without changing her/his decision.
Let us denote by $\tau$ the time needed by an individual to change  her/his
opinion. From the autocorrelation function  it can be seen that $\tau$ is usually very short,
but sometimes can be very long. The distribution of $(\tau)$ ($P(\tau)$)
follows seeminingly a power law behavior
with an exponent $-3/2$. Plot this distribution and compare your results with
figure 4 Sznajd-Weron and Snajzd. Comment your results.

### Project 5d): Initial Conditions

We will now study the influence of the initial conditions on the
evolution of the system by considering  two different ways -
randomly and in clusters.  In both cases you could start from an initial
concentration $c_B$ of opinion $B$.  In the random setup $c_B*N$
individuals are randomly (uniformly) chosen out of all $N$
individuals. In the cluster setup simply the first $c_B*N$ individuals
are chosen.

Study whether the distribution of decision time $\tau$ still follows
the power law with the same exponent as you found in the previous
part.  A non-monotonic behaviour of decision change is still typical
and sometimes even much stronger.  However, it is obvious that if
initially there is more $A$'s then $B$'s the final state should be
more often "all $A$" then "all $B$".  Compare your results to figure 5
of Sznajd-Weron and Snajzd and comment your results.

### Project 5e): Information Noise

It is well known that the changes of opinion are determined by the
social impact. Till now we have considered a community in which a
change of an individuals opinion is caused only by a contact with its
neighbours. It was the simplest social impact one can imagine.  Now,
we introduce to our model noise $p$, which is the probability that an
individual, instead of following the dynamic rules, will make a random
decision. We start from an "all $A$" state to investigate if there is
a $p \in (0,1)$ which does not throw the system out of this state.

See if you can reproduce figures 6 and 7 of the above authors and
comment your final results. Their discussion section contains several interesting observations you may consider discussing. The text by [Serge Galam](https://www.springer.com/gp/book/9781461420316) contains several other interesting obeservations which can be useful in the present analysi.

## Introduction to numerical projects

Here follows a brief recipe and recommendation on how to write a report for each
project.

  * Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  * Include the source code of your program. Comment your program properly.

  * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  * Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  * Try to give an interpretation of you results in your answers to  the problems.

  * Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  * Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

## Format for electronic delivery of report and programs

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

  * Use **Canvas** to hand in your projects, log in  at  <https://www.uio.no/english/services/it/education/canvas/> with your normal UiO username and password.

  * Upload **only** the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  Do not include library files which are available at the course homepage, unless you have made specific changes to them. Alternatively, you can just upload the address to your GitHub or GitLab repository.

  * In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

  * In this and all later projects, you should include tests (for example unit tests) of your code(s).

  * Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your **Canvas** domain and are only visible to you and the teachers of the course.

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.