# "Percolation"
> "In this blog post we will look at Percolation. Starting with some motivating examples we will solve the percolation problem on both a 1d and infinite dimensional lattice. We will use percolation to help understand how power-law behaviour arises from criticality near phase transitions."
- toc: true
- author: Lewis Cole (2020)
- branch: master
- badges: false
- comments: false
- categories: [Percolation, Criticality, Bethe-Lattice, Dimensionality, Theory, Probability]
- hide: false
- search_exclude: false
- image: https://github.com/lewiscoleblog/blog/raw/master/images/Percolation/coffee.jpg

In this blog post we shall take a first look at percolation and how we can study it mathematically.

## Introduction
Percolation theory and related studies is concerned with a simple problem: how does a substance pass through a random medium? Speaking in generalities such as this can obscure a concept so lets consider a real world example: we have a pool of oil sitting below a porous rock. We would like to know if we can "push" the oil through this rock or whether the rock is not "porous enough" for this to happen. (We could also ask a follow up question: if oil can pass can it do so in a timely matter?) Another real world example is a coffee percolator:

Here we have a mass of ground coffee that we wish to pass (near) boiling water through in order to extract all its caffeine related goodness. From experience we know that if we grind very finely and pack very tightly the water has a hard time passing and we may need a lot of heat and pressure to get any coffee out (in some cases water may never pass).

We can also consider percolation in a slightly different way. Imagine we have a sheet of paper and a hole punch, we randomly start punching holes in the paper - at some point the paper will break into 2 (or more) smaller seperate pieces of paper. What can we say about the number of holes required for this to happen?

In all the examples above we are observing **criticality** or a **phase transition**. At some point of porosity the rock can allow oil to pass, if the rock is any less porous no oil can pass. Similarly grind the coffee too fine and the water will not pass through, but coarsen ever so slightly we can make (perhaps a small) cup of coffee. In the paper example the phase transition is even more obvious as the paper will physically break into multiple parts when we punch the last hole. Perhaps the most common example of a phase transition however are states of matter - in particular water changing to ice at 0 celcius or water vapour at 100 celcius (this is unrelated to percolation however). The point at which the phase of these systems change is called the **critical point** and it is a very common behaviour in both physical and social systems. Studying these can be quite illuminating as behaviour often "gets weird" when a system nears a critical point.

## Mathematical Formulation
Now we have some real world intuition of what percolation theory is all about let's start to phrase it in a more mathematical setting so that we are better able to study it.

In percolation theory we are usually concerned with lattices. Lattices are a form of connected graph that can extend to infinity. A classic example would be a 2d square lattice where each site in the lattice has an "up", "down", "left" and "right" neighbour, we can imagine this as a sheet of squared paper. Other types of lattice can exist however; higher dimensions, different geometries and so on.

With our lattice we can then consider "populating" it, for each site in the lattice we randomly select whether it is populated or not. Typically we would consider this independently uniformly at random. We denote this probability $p$. One question we might ask is at what population probability is it possible to move from one side of the lattice to the other where each step we can move to a nearest neighbour only? Or we could ask the question what is the distribution of cluster size? What is the average cluster size? And so on.

In general we do this in relation to an "infinite" sized lattice since these are often easier to work with mathematically. When dealing with finite sizes we typicaly want to assume the lattices are sufficiently large that any "edge effects" (weirdness relating to edge sites having fewer neighbours than interior sites) can be ignored. Although investigations on smaller scale percolations are also an active field of research due to their practical application.

So far we have phrased the percolation problem in terms of **site percolation** - whereby the sites themselves are occupied or vacant. We could equally consider **bond percolation** whereby in the lattice we consider the "edges" between sites. With probability $p$ we select whether an edge exists. Now we can can consider "paths" travelling along the edges - can we move from the top of the lattice to the bottom? and so on. In our examples above the porous oil and coffee problems are examples of bond percolation, the paper problem is a site percolation.

One might expect these to be equivalent to one another however that is not always the case depending on the geometry of a lattice.

Given the formulation here we can see that percolation problems lend themselves to being studied computationally. We can randomly generate finite lattices and analyse them to get intiution about how these things behave. However using this approach will not allow us to ever have a definitive "result" without a corresponding mathematical proof.

Some extensions we may wish to consider include:
- What if the sites/bonds are not independent (or uniformly) distributed?
- What about the edge effects?
- What about directed networks?
- What if connectivity is not static over time?
- How do we find the "path of least resistance" to pass through a realisation of a lattice?

## Why Should We Care?
We have seen one potential application of percolation theory: predicting whether we can extract oil through a porous rock. In fact when extracting oil they do take a small sample of the rock to analyse to see whether percolation is possible. The coffee and paper examples, while providing intuition, are not applications in the sense we would not use percolation theory to solve them!

What are some other examples we might care about?
1. **Forest Fires** - we could imagine a forest being made up of sites on a 2d lattice, some occupied with trees and some empty. We could investigate how forest density leads to fire spreading, we would not want to run real-world experiments on this for obvious reasons! We can build in more sophistication through modelling damp/dry patches, wind, air temperature and so on. The results from models like these can influence policy around back-burning, controlled cut-downs etc. to reduce the likelihood of larger more devastating wild fires. (This is a site percolation problem)
2. **Social Networks** - We may be interested in how information spreads through a social network. We can study how network connectivity influences this. If we were running a viral marketing campaign we might wish to understand the penetration required of the social network for an idea to take off. (This is a bond percolation problem)
3. **Communication Networks** - Suppose we are a telecom provider interested in keeping a service running 24/7. We have a network that allows this to happen with a number of connections between the individual nodes. We may wish to know how many connections (or nodes) need to "go down" for the entire system to fall over. We can then grow the network in such a way as to ensure robustness. (This is a bond percolation problem)
4. **Electrical Engineering** - If we imagine impregnating an insulating material with (for example) carbon nano-tubes, we could ask how much carbon we would need to add to turn the insulator into a conductor? Essentially we are asking what is the proportion of carbon required to make a "bridge" across the material to allow for electron flow. (This is a site percolation problem).
5. **Fault Lines** - Another application is the study of fault lines for use in predicting earthquakes. We can study the degree of "cracking" required for a "split" using tools based on percolation.
6. **Biology/Medicine** - Percolation theory has been used to understand virus fragmentation as well as modelling disease spread and fragmentation of ecological systems.

## Percolation in 1 dimension
As with many physical and mathematical systems it makes sense to start with 1 dimension since this is often easy to visualize, the mathematics tends to be simple and we can generally use intuition to "solve" the system.

Our lattice in 1 dimension is essentially a line segment made up of a number of sites joined together. (We can equivalently consider a bond percolation in the exact same way, in this 1d example the 2 formulations are equivalent - not necessarily true in more complex scenarios!)

We assume that each site is occupied (independently) with fixed probability $p$. If we imagine taking this line segment and adding extra sites to the end one-by-one (to the limit of infinity), it does not take much to realise that if $p<1$ then eventually we will come across an un-occupied site and so there will not exist a path from one end of the lattice to the other (i.e. there is no percolation!) Clearly if $p=1$ then every site is occupied and it is possible to create a path. We denote this probability $p_c$ (critical probability), it is one of the quantities we wish to study. In more complicated lattices we will not be able to use intuition quite so easily!

We will now step back and solve the system mathematically. In this case it is not strictly required, however it will provide a blueprint for how to approach more complicated lattices. It also allows us to introduce some of the notation.

We will fix a lattice size $L$, we will assume this is suitably large so that we can ignore any edge effect complications. The probability that there exists a cluster of size $s<L$ (that is a group of adjacent sites that are all occupied) is:
$$n_s = p^s(1-p)^2 $$
This is since we need $s$ occupied sites (probability $p^s$) surrounded by 2 empty sites (probability $(1-p)^2$) - since we are assuming independence this works as desired. We can look at the expected number of such clusters via: $Ln_s$ since we assume any edge effects are minimal (we could work this out exactly but it is not necessary here). Typically however we want to scale $L \to \infty$ so we will work with probabilities instead. Quite clearly if $p<1$ then $n_s \to 0$ as $s \to \infty$ - which is just a way of expressing our intuition above through mathematics.

We can further note that the probability that a site belongs to a cluster of size $s$ is $s n_s$ (it can be positioned in any 1 of the $s$ sites in the cluster). We then have (for an infinite size lattice) the relation:
$$ \sum_s s n_s = p $$
To see this we can note:
\begin{align}
\sum_s s n_s &= \sum_s s p^s (1-p)^2 \\
             &= (1-p)^2 p \sum_s \frac{d}{dp} p^s \\
             &= (1-p)^2 p \frac{d}{dp} \sum_s p^s \\
             &= (1-p)^2 p  \frac{d}{dp} \left( \frac{1-p^{L'}}{1-p}   \right) \\
             &= (1-p)^2 p \left((1-P^{L'})(1-p)^{-2} - L'p^{L'-1}(1-p^{L'-1})(1-p)^{-1}   \right) \\
             &\xrightarrow{L' \to \infty} (1-p)^2 p (1-p)^{-2} \\
             &= p
\end{align}
Note: here we imagine a smaller "window" within our lattice such that $L' \leq L-2$ to avoid edge effects. This is a common "trick" to deal with asymptotic behaviours. There are other ways to deal with this such as using periodic boundary conditions too. In this example we could actually compute all the options if we wanted to, the extra terms would "disappear" when taking the limit.

This formula containing $p$ is actually fairly intuitive despite looking "complicated", it essentially states: "every occupied site has to belong to some cluster" - so the sum over all cluster sizes must equal the probability that the site is occupied! In some cases the seemingly intuitive explanations can turn out to be wrong, so it is a good habit to prove these results rather take them for granted and only later realise they are incorrect. This seems especially true in probability theory.

We can now define a new variable:
$$ w_s = \frac{s n_s}{\sum_{s'} s' n_{s'}}$$
Which is in some sense a "weighting" of relative cluster sizes. We can then calculate the average cluster size using:
$$ S_L = \sum_s s w_s $$
By taking the limit we can observe the asymptotic behaviour:
$$ S_{\infty} = \lim_{L \to \infty} \sum_s s w_s $$

Can we find a formula for this value? In the case of the 1d lattice we can, in other more complicated geometries we may not be able to. In the derivation below I exchange limits/sums/derivatives quite freely since this is valid in this case.
\begin{align}
S_{\infty} &= \lim_{L \to \infty} \sum_s  \frac{s^2 n_s}{\sum_{s'} s' n_{s'}} \\
           &= \lim_{L \to \infty} \sum_s  \frac{s^2 n_s}{p} \\
           &= \frac{(1-p)^2}{p} \left( p \frac{d}{dp} \right)^2 \lim_{L \to \infty} \sum_s p^s \\
           &= \frac{(1-p)^2}{p} \left( p \frac{d}{dp} \right)^2 \frac{1}{1-p} \\
           &= \frac{(1-p)^2}{p} \left( p \frac{d}{dp} \right) \frac{p}{(1-p)^2} \\
           &= \frac{(1-p)^2}{p} \left( \frac{p}{(1-p)^2} + \frac{2p^2}{(1-p)^3} \right) \\
           &= 1 + \frac{2p}{1-p} \\
           &= \frac{1+p}{1-p} \\
           &= \frac{p_c+p}{p_c-p}
\end{align}
For values of $p$ close to $p_c = 1$ we have:
$$ S_{\infty} \approx \frac{2p_c}{p_c - p} \propto |p_c - p|^{-1}$$
In general percolation problems the behaviour of cluster size around the critical probability is defined by the "critical exponent" which we denote $\gamma$:
$$ S_{\infty} \propto |p_c - p|^{-\gamma} $$
In the case of the 1d lattice percolation this is $\gamma = 1$.

Another concept we have is the "correlation function" $g(r)$ which represents the probability that 2 sites distance $r$ apart belong to the same cluster. In the 1d case we can represent this as:
$$g(r) = p^r$$
Which we can re-express in the form:
$$ g(r) = exp\left(\frac{-r}{\xi}\right)$$
Where $\xi = \frac{-1}{ln(p)}$ is called the correlation length. For $p$ close to $p_c$ we have $\xi \approx (p_c - p)^{-1} \approx S_{\infty}$. Which is another critical exponent: $\xi \approx (p_c - p)^{-\nu}$. In this case $\nu = \gamma$ but this is not necessarily true in general.

We can also express the cluster size via correlation functions:
$$ S_L = \sum_r g(r) $$

This completely solves the percolation problem in the 1d lattice case. Unfortunately things are not always this simple.

## Bethe Lattice
We now move onto another lattice that can be solved exactly: the Bethe lattice. A Bethe lattice is an infinite connected cycle-free graph where each node has exactly $z$ neighbours. It is also sometimes called a Caley tree. We can think of the 1d lattice case above as being a Bethe lattice with $z=2$.

If the sites are arranged in "shells" as in the diagram above the number of sites in the kth shell is:
$$N_k = z(z-1)^{k-1}$$

We can think of the Bethe lattice as an infinite dimensional lattice. Why is this? If we consider a 3d space, we note that: $$Area \propto L^2$$
For some length $L$ for a generic shape. similarly for volume we have:
$$Vol \propto L^3 $$
For example the volume of a sphere is $\frac{4}{3}\pi r^3$ and the surface area of the sphere is $4 \pi r^2$, and similar formulae exist for cubes, etc. We notice that:
$$ \frac{Area}{Vol}_{3d} \propto \frac{L^2}{L^3} = L^{1 - \frac{1}{3}}$$
If we repeat the same procedure with a 4d space we get:
$$ \frac{Area}{Vol}_{4d} \propto L^{1 - \frac{1}{4}}$$
And in general for a D dimensional space:
$$ \frac{Area}{Vol}_{Dd} \propto L^{1 - \frac{1}{D}}$$

What about the Bethe lattice? From above we know the number of sites in the kth shell is: $z(z-1)^{k-1}$ we can then image these shells "encompassing" all preceeding shells to get a "volume". So in this case:
$$Vol = 1 +\sum_{i=1}^k z(z-1)^{i-1} = 1 + \frac{z((z-1)^k - 1)}{z-2} = \frac{z(z-1)^k - 2}{z-2}$$
And so:
$$ \frac{Area}{Vol}_{Bethe} =  \frac{z(z-1)^{k-1}}{\frac{z(z-1)^k - 2}{z-2}} \xrightarrow{k \to \infty} \frac{z-2}{z-1} $$
And so the ratio tends to a constant, in other words:
$$ \frac{Area}{Vol}_{Bethe} \propto L^{0}$$
Which is in some sense equivalent to an "infinite dimensional" space (by taking $D=\infty$ into the formula for $\frac{Area}{Vol}_{Dd}$).

We can find the percolation threshold probability through a relatively simple argument. We start at the origin (centre site); if a site exists in the next shell it has $(z-1)$ possible "new" neighbours. Given this site is occupied with probability $p$, the number of (expected) occupied neigbours is: $p(z-1)$. For an infinite cluster to exist we require this quantity to be greater than $1$ and so we have:
$$p_c = \frac{1}{z-1}$$
This ties in with our finding for the 1d lattice ($z=2$) as we would expect. Also notice that again in this case we could exchange "bond" for "site" and the argument still holds and so we have the critical probability is the same for both site and bond percolation problems. This does not hold true in general lattices.

We now introduce a new concept: In the 1d case we could only approach $p_c$ from below (at least I know of no way to consider probabilities in excess of 1!) Whereas now we can consider what happens above the percolation threshold. From common sense intuition we know that if $p=1$ the infinite cluster will in some sense be "stronger" than if $p$ is only slightly above $p_c$. We would like a metric to use to illustrate this concept. The infinite cluster strength represents the probability that a given site is part of the infinite cluster (that is we can follow a path from this site to infinity). Cleary the strength is zero below the percolation threshold and $1$ for $p=1$. We denote the cluster strength by $P(p)$ - this is an example of an order parameter.

To derive the cluster strength we introduce a quantity $Q(p)$ which is the probabilty that a site is not connected to the infinite cluster. We can then write:
$$P(p) = p(1-Q^z(p))$$
Which is the probability that a site is occupied and at least one path to infinity exists for that site. But we can further write:
$$Q(p) = (1-p) + pQ^{z-1}(p)$$
The probability the node is not occupied or it is occupied but the neighbours do not have a path to infinity.

We find that there is a trivial solution $Q(p)=1$ with non-trivial solution:
$$Q(p) = 1 - \frac{2p(z-1)-2}{p(z-1)(z-2)} $$
And so:
$$P(p) = p\left(1 - \left( 1 - \frac{2p(z-1)-2}{p(z-1)(z-2)} \right)^z \right) $$
Via a rather messy Taylor expansion we can show:
$$ P(p) \propto |p_c - p|^{-1} $$
This is another critical exponent for the Bethe lattice. In general lattices we shall find:
$$ P(p) \propto |p_c - p|^{-\beta} $$
For some $\beta$.

We now move onto looking at the average cluster size $S(p)$. To do this we will introduce an intermediate varible $T$ which represents contribution from an individual bond in the lattice. The average cluster size containing the origin is then:
$$S(p) = 1 + zT$$
Since there are $z$ bonds. For each branch we note that either the neighbouring site is un-occupied (probability $(1-p)$ but contribution 0 to the average) or it is occupied (probability $p$ with $(z-1)$ neighbours) so we can express $T$ as:
$$ T = p(1 + (z-1)T) \iff T = \frac{p}{1-(z-1)p}$$
And so:
$$S(p) = 1 + \frac{zp}{1-(z-1)p} = \frac{p_c(1+p)}{p_c - p}$$
Where $p_c = \frac{1}{z-1}$. Thus the critical exponent is $\gamma = 1$ as in the 1d case. We can also expand on our previous identity from the 1d case to give:
$$P(p) + \sum_{s=1}^{\infty} s n_s(p) = p $$
This equation holds for all lattices not just the Bethe lattice. We may naturally ask what form does $n_s(p)$ take in the Bethe lattice? Clearly the formula from the 1d lattice is not correct. We can show that $n_s$ is of the form:
$$ n_s(p) = g_{s,z} p^s (1-p)^{2+s(z-2)} $$
Where $g_{s,z}$ is some constant fixed for each cluster size ($s$) and Bethe lattice number ($z$). This constant can be hard to calculate for large $s$ since there are many possible configurations of sites. We can however look at ratios so remove the pesky constant:
\begin{align}
\frac{n_s(p)}{n_s(p_c)} &= \frac{p^s (1-p)^{2+s(z-2)}}{p_c^s (1-p_c)^{2+s(z-2)}} \\
                        &= \left(\frac{1-p}{1-p_c}\right)^2 exp\left(s ln\left[ \frac{p(1-p)^{z-2}}{p_c(1-p_c)^{z-2}} \right] \right)\\
                        &= \left(\frac{1-p}{1-p_c}\right)^2 exp\left( \frac{-s}{s_\xi} \right)
\end{align}
With:
$$ s_\xi = \frac{-1}{ln \left( \frac{p(1-p)^{z-2}}{p_c(1-p_c)^{z-2}} \right)} $$
Through a Taylor expansion we can show:
$$ s_\xi \propto |p_c - p|^{\frac{-1}{\sigma}} $$
With another critical exponent parameter $\sigma$. In the case of the Bethe lattice  $\sigma = \frac{1}{2}$.

We can now use $s_\xi$ to investigate $n_s(p)$. Through some more work we can show:
$$n_s(p) \propto s^{-\tau} exp \left( \frac{-s}{s_{\xi}} \right) $$
For another critical exponent $\tau$. In the case of the Bethe lattice we have $\tau = \frac{5}{2}$. The quantity $s_{\xi}$ Is often called the "cutoff cluster size" since: for for clusters very much below this we have $n_s(p) \propto s^{-\tau}$ while above this clusters decay very quickly.

## Percolation in d-dimenison with 1 < d < $\infty$
We have seen percolation in 1 dimension and in infinite dimensions (Bethe lattice) - what about other dimensions? Unfortunately things get (much) more complicated. To see why this is we first saw that we could "count" the clusters of a given size in the 1d lattice fairly easily, in the Bethe lattice we could also rely on the "cycle free" structure to help us count the arrangement of sites. This is not so easy in other lattices. For example consider the 2d square lattice, if we just look at the different ways a site (in red) can contribute to a cluster (in black) we have:

And so (by counting the empty neigbouring sites):
$$n_3(p) = 6 p^3 (1-p)^8 + 12 p^3 (1-p)^7 $$
These formulae only get more complicated as the cluster size increases and as the dimension increases. We cannot easily find a general solution to this problem. Even if we could it is unlikely we'll be able to make much traction using the techniques we have discussed so far.

So far we have seen that site percolation is equivalent to bond percolation. In general this is not the case, it is not hard to see this by considering bond and site percolations on the 2d space. We note however that any bond percolation problem can be reformulated as a site percolation problem (in most cases in a different lattice geometry). As such most study is focussed on site percolation.

In many problems an exact anaytic solution to the percolation problem is not possible/very difficult so many results are uncovered using computer simulations.

The table below summarises some of the critical probabilites for various lattice dimension and geometries:

```python
#hide_input
import pandas as pd

idx =['1d', '2d Hex', '2d Square', '2d Triangle', '3d Diamond', '3d Cubic', '4d Cubic', '5d Cubic', '6d Cubic', 'Bethe (z)']

data = {'pc Site Percolation': pd.Series([1, 0.696, 0.593, '1/2', '0.430', 0.312, 0.197, 0.141, 0.107, '1/(z-1)'], index=idx),
       'pc Bond Percolation': pd.Series([1, '1-2sin(𝜋/18)', '1/2', '2sin(𝜋/18)', 0.388, 0.249, '0.160', 0.118, 0.094, '1/(z-1)'], index=idx)}

pd.DataFrame(data)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>pc Site Percolation</th>
      <th>pc Bond Percolation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1d</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <td>2d Hex</td>
      <td>0.696</td>
      <td>1-2sin(𝜋/18)</td>
    </tr>
    <tr>
      <td>2d Square</td>
      <td>0.593</td>
      <td>1/2</td>
    </tr>
    <tr>
      <td>2d Triangle</td>
      <td>1/2</td>
      <td>2sin(𝜋/18)</td>
    </tr>
    <tr>
      <td>3d Diamond</td>
      <td>0.430</td>
      <td>0.388</td>
    </tr>
    <tr>
      <td>3d Cubic</td>
      <td>0.312</td>
      <td>0.249</td>
    </tr>
    <tr>
      <td>4d Cubic</td>
      <td>0.197</td>
      <td>0.160</td>
    </tr>
    <tr>
      <td>5d Cubic</td>
      <td>0.141</td>
      <td>0.118</td>
    </tr>
    <tr>
      <td>6d Cubic</td>
      <td>0.107</td>
      <td>0.094</td>
    </tr>
    <tr>
      <td>Bethe (z)</td>
      <td>1/(z-1)</td>
      <td>1/(z-1)</td>
    </tr>
  </tbody>
</table>
</div>

When not expressed as a fraction or trigonometric function probabilities are expressed to 3 significant figures.

## Summary of Critical Exponents
Due to its simplicity for the 1d case we only looked at 1 critical exponent. In more complicated lattices we can describe 5 different critical behaviours $(\gamma, \beta, \nu,  \sigma, \tau)$:
\begin{align}
S(p) &\propto |p_c - p|^{-\gamma} \\
P(p) &\propto |p_c - p|^{-\beta}  \\
\xi(p) &\propto |p_c - p|^{-\nu}  \\
s_\xi(p) &\propto |p_c - p|^{\frac{-1}{\sigma}}  \\
n_s(p) &\propto s^{-\tau} exp \left( \frac{-s}{s_{\xi}} \right)
\end{align}
Where these proportionalities only hold when we are close to the critical percolation probability. We can see that these will vary depending on the geometry of the lattices involved.

We can see that the behaviour of these quantities around the critical point behave as a power-law. This is a universal behaviour of "criticality" and what makes studying behaviour around phase transitions so interesting. The behaviours can become highly unusual, the critical exponents provide a way of classifying these behaviours.

Another exponent of interest is $\alpha$. We define the probability that a site (e.g. the origin) is connected to another site distance $r$ away as:
$$q(r) \propto r^{\alpha} $$
Where the probability is evaluated at the critical percolation probability.

We can summarise these exponents int he table below:

```python
#hide_input
import pandas as pd

idx =['𝛼', '𝛽', '𝛾', '𝜈', '𝜎', '𝜏']

data = {'1d': pd.Series([0, 0, 1, 1, 1, 2], index=idx),
        '2d': pd.Series(['-2/3', '5/36', '43/18', '4/3', '36/91', '187/91'], index=idx),
        '3d': pd.Series([-.62, 0.41, '1.80', 0.88, .45, 2.18], index=idx),
        '4d': pd.Series([-.72, 0.64, '1.44', 0.68, .48, 2.31], index=idx),
        '5d': pd.Series([-.86, 0.84, '1.18', 0.57, .49, 2.41], index=idx),
        '6d': pd.Series([-1, 1, 1, '1/2', '1/2', '5/2'], index=idx),
        'Bethe': pd.Series([-1, 1, 1, '1/2', '1/2', '5/2'], index=idx)}

pd.DataFrame(data)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>1d</th>
      <th>2d</th>
      <th>3d</th>
      <th>4d</th>
      <th>5d</th>
      <th>6d</th>
      <th>Bethe</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>𝛼</td>
      <td>0</td>
      <td>-2/3</td>
      <td>-0.62</td>
      <td>-0.72</td>
      <td>-0.86</td>
      <td>-1</td>
      <td>-1</td>
    </tr>
    <tr>
      <td>𝛽</td>
      <td>0</td>
      <td>5/36</td>
      <td>0.41</td>
      <td>0.64</td>
      <td>0.84</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <td>𝛾</td>
      <td>1</td>
      <td>43/18</td>
      <td>1.80</td>
      <td>1.44</td>
      <td>1.18</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <td>𝜈</td>
      <td>1</td>
      <td>4/3</td>
      <td>0.88</td>
      <td>0.68</td>
      <td>0.57</td>
      <td>1/2</td>
      <td>1/2</td>
    </tr>
    <tr>
      <td>𝜎</td>
      <td>1</td>
      <td>36/91</td>
      <td>0.45</td>
      <td>0.48</td>
      <td>0.49</td>
      <td>1/2</td>
      <td>1/2</td>
    </tr>
    <tr>
      <td>𝜏</td>
      <td>2</td>
      <td>187/91</td>
      <td>2.18</td>
      <td>2.31</td>
      <td>2.41</td>
      <td>5/2</td>
      <td>5/2</td>
    </tr>
  </tbody>
</table>
</div>

We notice that the critical exponent behaviour depends on the dimension only and not any specific geometry, unlike the critical probability.

## Conclusion
In this blog post we have looked at the very basics of Percolation theory. We should have an idea of why we should study this phenomena through the use of some motivating examples. We gained some intuition by analysing 2 "simple" examples in 1d and infinite dimensions. We have also seen how the percolation problem gives rise to criticality and how behaviour around a critical point gives rise to power-law behaviours.

This is just a brief introduction and there are many areas we did not touch upon here. Since percolation theory can provide some truly interesting mathematics and insight I'm sure I will visit some of these topics in a later blog post.


```
!pip install -q pyomo
```

         |████████████████████████████████| 8.9 MB 3.8 MB/s
         |████████████████████████████████| 49 kB 5.1 MB/s

$Exercise\;1$
\begin{align}
\min \ c_1 x_1 + c_2 x_2 + \ldots + c_N x_N, \\
{\rm{s.t.}} \; x_1 + x_2 + \ldots + x_N & \leq b_1, \\
w_1 x_1 + w_2 x_2 + \ldots + w_N x_N  & = b_2, \\
x_i & \leq u_i, \quad i=1,\ldots,N, \\
x_i & \geq l_i, \quad i=1,\ldots,N. \\
\end{align}
---

$b_1 = 161.2,$ \\
$b_2 = 54.6,$ \\
$c = [−8.1, 10.15, 30.5, 50.05, 0.05, 80.5, −0.25, −31.02, 50.65, 0.725, −0.8, 100.6];$ \\
$w = [0, 1.6, −2.02, 0, 1.01, −2.005, 0, 1.39, 1, −1.214, 0, −5.32];$ \\
$l = [−∞, 1, 0, 0, 7, 0, −∞, −∞, 1, 1, −5, 2],$ \\
$u = [4, 3, +∞, 2, 10, +∞, 13, 20, +∞, 21, 5, 60]$

$1.\;Solving\;the\;given\;problem:$

```
from pyomo.environ import *
```

```
import numpy as np
```

```
model = ConcreteModel()
```

```
N = 12
```

```
obj_coef_c = np.array([-8.1, 10.15, 30.5, 50.05, 0.05, 80.5, -0.25, -31.02, 50.65, 0.725, -0.8, 100.6])
```

```
constr1_coef_p = np.array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
constr2_coef_w = np.array([0, 1.6, -2.02, 0, 1.01, -2.005, 0, 1.39, 1, -1.214, 0, -5.32])

constr1_rhs_b1 = 161.2
constr2_rhs_b2 = 54.6
```

```
lower_bound = np.array([-np.inf, 1, 0, 0, 7, 0, -np.inf, -np.inf, 1, 1, -5, 2])
upper_bound = np.array([4, 3, np.inf, 2, 10, np.inf, 13, 20, np.inf, 21, 5, 60])
```

```
col_indices = np.arange(N)
print(col_indices)
```

    [ 0  1  2  3  4  5  6  7  8  9 10 11]

```
model.x = Var(col_indices)
```

```
model.constraints = ConstraintList()
```

```
model.constraints.add(sum(constr1_coef_p[j]*model.x[j] for j in col_indices) <= constr1_rhs_b1)
```

    <pyomo.core.base.constraint._GeneralConstraintData at 0x7fb406844d00>

```
#next add the constraint w1 x1 + w2 x2 + ...+ wN xN >= b2
model.constraints.add(sum(constr2_coef_w[j]*model.x[j] for j in col_indices) == constr2_rhs_b2)
```

    <pyomo.core.base.constraint._GeneralConstraintData at 0x7fb406844d70>

```
for j in col_indices:
  model.x[j].setlb(lower_bound[j])
  model.x[j].setub(upper_bound[j])
```

```
model.objective = Objective(expr = sum(obj_coef_c[j]*model.x[j] for j in col_indices), sense=minimize)
```

```
model.pprint()
```

    2 Set Declarations
        constraints_index : Size=1, Index=None, Ordered=Insertion
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :    2 : {1, 2}
        x_index : Size=1, Index=None, Ordered=False
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :   12 : {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}

    1 Var Declarations
        x : Size=12, Index=x_index
            Key : Lower : Value : Upper : Fixed : Stale : Domain
              0 :  -inf :  None :   4.0 : False :  True :  Reals
              1 :   1.0 :  None :   3.0 : False :  True :  Reals
              2 :   0.0 :  None :   inf : False :  True :  Reals
              3 :   0.0 :  None :   2.0 : False :  True :  Reals
              4 :   7.0 :  None :  10.0 : False :  True :  Reals
              5 :   0.0 :  None :   inf : False :  True :  Reals
              6 :  -inf :  None :  13.0 : False :  True :  Reals
              7 :  -inf :  None :  20.0 : False :  True :  Reals
              8 :   1.0 :  None :   inf : False :  True :  Reals
              9 :   1.0 :  None :  21.0 : False :  True :  Reals
             10 :  -5.0 :  None :   5.0 : False :  True :  Reals
             11 :   2.0 :  None :  60.0 : False :  True :  Reals

    1 Objective Declarations
        objective : Size=1, Index=None, Active=True
            Key  : Active : Sense    : Expression
            None :   True : minimize : -8.1*x[0] + 10.15*x[1] + 30.5*x[2] + 50.05*x[3] + 0.05*x[4] + 80.5*x[5] - 0.25*x[6] - 31.02*x[7] + 50.65*x[8] + 0.725*x[9] - 0.8*x[10] + 100.6*x[11]

    1 Constraint Declarations
        constraints : Size=2, Index=constraints_index, Active=True
            Key : Lower : Body                                                                                : Upper : Active
              1 :  -Inf :        x[0] + x[1] + x[2] + x[3] + x[4] + x[5] + x[6] + x[7] + x[8] + x[9] + x[10] + x[11] : 161.2 :   True
              2 :  54.6 : 1.6*x[1] - 2.02*x[2] + 1.01*x[4] - 2.005*x[5] + 1.39*x[7] + x[8] - 1.214*x[9] - 5.32*x[11] :  54.6 :   True

    5 Declarations: x_index x constraints_index constraints objective

```
!apt-get install -y -qq glpk-utils
```

    Selecting previously unselected package libsuitesparseconfig5:amd64.
    (Reading database ... 160837 files and directories currently installed.)
    Preparing to unpack .../libsuitesparseconfig5_1%3a5.1.2-2_amd64.deb ...
    Unpacking libsuitesparseconfig5:amd64 (1:5.1.2-2) ...
    Selecting previously unselected package libamd2:amd64.
    Preparing to unpack .../libamd2_1%3a5.1.2-2_amd64.deb ...
    Unpacking libamd2:amd64 (1:5.1.2-2) ...
    Selecting previously unselected package libcolamd2:amd64.
    Preparing to unpack .../libcolamd2_1%3a5.1.2-2_amd64.deb ...
    Unpacking libcolamd2:amd64 (1:5.1.2-2) ...
    Selecting previously unselected package libglpk40:amd64.
    Preparing to unpack .../libglpk40_4.65-1_amd64.deb ...
    Unpacking libglpk40:amd64 (4.65-1) ...
    Selecting previously unselected package glpk-utils.
    Preparing to unpack .../glpk-utils_4.65-1_amd64.deb ...
    Unpacking glpk-utils (4.65-1) ...
    Setting up libsuitesparseconfig5:amd64 (1:5.1.2-2) ...
    Setting up libcolamd2:amd64 (1:5.1.2-2) ...
    Setting up libamd2:amd64 (1:5.1.2-2) ...
    Setting up libglpk40:amd64 (4.65-1) ...
    Setting up glpk-utils (4.65-1) ...
    Processing triggers for libc-bin (2.27-3ubuntu1.2) ...
    /sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

```
SolverFactory('glpk', executable='/usr/bin/glpsol').solve(model).write()
```

    # ==========================================================
    # = Solver Results                                         =
    # ==========================================================
    # ----------------------------------------------------------
    #   Problem Information
    # ----------------------------------------------------------
    Problem:
    - Name: unknown
      Lower bound: 775.965100000001
      Upper bound: 775.965100000001
      Number of objectives: 1
      Number of constraints: 3
      Number of variables: 13
      Number of nonzeros: 21
      Sense: minimize
    # ----------------------------------------------------------
    #   Solver Information
    # ----------------------------------------------------------
    Solver:
    - Status: ok
      Termination condition: optimal
      Statistics:
        Branch and bound:
          Number of bounded subproblems: 0
          Number of created subproblems: 0
      Error rc: 0
      Time: 0.011092901229858398
    # ----------------------------------------------------------
    #   Solution Information
    # ----------------------------------------------------------
    Solution:
    - number of solutions: 0
      number of solutions displayed: 0

```
# display solution
print('\nObjective = ', model.objective())

print('\nDecision Variables')
for i in col_indices:
  print('x[',i,'] = ', model.x[i].value)

print('\nConstraints')
model.constraints.display()
```

    Objective =  775.9651000000001

    Decision Variables
    x[ 0 ] =  4.0
    x[ 1 ] =  3.0
    x[ 2 ] =  0.0
    x[ 3 ] =  0.0
    x[ 4 ] =  10.0
    x[ 5 ] =  0.0
    x[ 6 ] =  13.0
    x[ 7 ] =  20.0
    x[ 8 ] =  23.754
    x[ 9 ] =  1.0
    x[ 10 ] =  5.0
    x[ 11 ] =  2.0

    Constraints
    constraints : Size=2
        Key : Lower : Body               : Upper
          1 :  None :             81.754 : 161.2
          2 :  54.6 : 54.599999999999994 :  54.6

$2.\; The\;optimal\;solution\;is: $ \\
$$x_1 =  4.0,\;x_2 =  3.0,\;x_3 =  0.0,\;x_4 =  0.0,\;x_5 =  10.0,\;x_6 =  0.0,\;x_7 =  13.0,\;x_8 =  20.0,\;x_9 =  23.754,\;x_10 =  1.0,\;x_{11} =  5.0,\; x_{12} =  2.0$$
$The\;value\;of\;the\;objective\;function\;at\;optimal\;solution\;is:\;775.9651000000001$

Use of numpy arrays  to store the coefficients of objective and constraints.ipynb


# Chapter 7: Moving Beyond Linearity
- **Chapter 7 from the book [An Introduction to Statistical Learning](https://www.statlearning.com/)**
- **By Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani**
- **Pages from $297$ to $299$**
- **By [Mosta Ashour](https://www.linkedin.com/in/mosta-ashour/)**

**Exercises:**
- **[1.](#1) [(a)](#1a) [(b)](#1b) [(c)](#1c) [(d)](#1d) [(e)](#1e)**
- **[2.](#2) [(a)](#2a) [(b)](#2b) [(c)](#2c) [(d)](#2d) [(e)](#2e)**
- **[3.](#3)**
- **[4.](#4)**
- **[5.](#5) [(a)](#5a) [(b)](#5b) [(c)](#5c)**

#  7.9 Exercises
##  Conceptual

### $1.$ It was mentioned in the chapter that a cubic regression spline with one knot at $\xi$ can be obtained using a basis of the form $x, x^2, x^3, (x - \xi)^3_+$, where $(x - \xi)^3_+ = (x - \xi)^3$ if $x > \xi$ and equals 0 otherwise.
 We will now show that a function of the form $$f(x) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4(x − \xi)^3_+$$
 is indeed a cubic regression spline, regardless of the values of $\beta_0, \beta_1, \beta_2, \beta_3, \beta_4.$

**$(a)$ Find a cubic polynomial $$f_1(x) = a_1 + b_1x + c_1 x^2 + d_1 x^3$$ such that $f(x) = f_1(x)$ for all $x \leq \xi$. Express $a_1, b_1, c_1, d_1$ in terms of $\beta_0, \beta_1, \beta_2, \beta_3, \beta_4.$**

**Answer:**

>- With $(x \leq \xi)$, the term $(x − \xi)^3_+ = 0$
>- The formula would be: $$f_1(x) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3$$
>- So, the coefficients are:
$
a_1 = \beta_0,\\
b_1 = \beta_1,\\
c_1 = \beta_2,\\
d_1 = \beta_3.
$

**$(b)$ Find a cubic polynomial $$f_2(x) = a_2 + b_2x + c_2 x^2 + d_2 x^3$$ such that $f(x) = f_2(x)$ for all $x > \xi$. Express $a_2, b_2, c_2, d_2$ in terms of $\beta_0, \beta_1, \beta_2, \beta_3, \beta_4.$. We have now established that f(x) is a piecewise polynomial.**

**Answer:**

>- For $x \gt \xi$, $f(x)$ has the form of:
$$
\begin{align}
f_2(x) &= \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4 (x - \xi)^3_+
\\
&= \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4 (x - \xi)^3
\\
&= \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4 (x^3 - 3 x^2 \xi + 3 x \xi^2 - \xi^3)
\\
&= (\beta_0 - \beta_4 \xi^3) + (\beta_1 + 3 \beta_4 \xi^2) x + (\beta_2 - 3 \beta_4 \xi) x^2 + (\beta_3 + \beta_4) x^3
\end{align}
$$
>- Thus,
$
a_2 = \beta_0 - \beta_4 \xi^3,\\
b_2 = \beta_1 + 3 \beta_4 \xi^2,\\
c_2 = \beta_2 - 3 \beta_4 \xi,\\
d_2 = \beta_3 + \beta_4.
$

**$(c)$ Show that $f_1(\xi) = f_2(\xi).$ That is, $f(x)$ is continuous at $\xi$.**

**Answer:**

>- **For $f_1(\xi)$:**
$$
f_1(\xi) = \beta_0 + \beta_1 \xi + \beta_2 \xi^2 + \beta_3 \xi^3 \tag{1}
$$
>- **For $f_2(\xi)$:**
$$
\begin{align}
f_2(\xi) &= \beta_0 + \beta_1 \xi + \beta_2 \xi^2 + \beta_3 \xi^3 + \color{red}{\beta_4 (\xi - \xi)^3}
\\
f_2(\xi) &= \beta_0 + \beta_1 \xi + \beta_2 \xi^2 + \beta_3 \xi^3 \tag{2}
\\
\end{align}
$$

>- From $(1)$ and $(2)$, $f_1(\xi) = f_2(\xi) = \beta_0 + \beta_1 \xi + \beta_2 \xi^2 + \beta_3 \xi^3.$

**$(d)$ Show that $f'_1(\xi) = f'_2(\xi).$ That is, $f'(x)$ is continuous at $\xi$.**

**Answer:**
>- **For $f'_1(\xi)$:**
$$
\begin{align}
f_1(x) &= \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3
\\
f_1'(x) &= \beta_1 + 2 \beta_2 x + 3 \beta_3 x^2
\\
f_1'(\xi) &= \beta_1 + 2 \beta_2 \xi + 3 \beta_3 \xi^2 \tag{1}
\end{align}
$$
>- **For $f'_2(\xi)$:**
$$
\begin{align}
f_2(x) &= \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4 (x - \xi)^3
\\
f'_2(x) &= \beta_1 + 2 \beta_2 x + 3 \beta_3 x^2 + 3 \beta_4 (x - \xi)^2
\\
f'_2(\xi) &= \beta_1 + 2 \beta_2 \xi + 3 \beta_3 \xi^2 + \color{red}{3 \beta_4 (\xi - \xi)^2}
\\
f'_2(\xi) &= \beta_1 + 2 \beta_2 \xi + 3 \beta_3 \xi^2 \tag{2}
\end{align}
$$

>- From $(1)$ and $(2)$, $f'_1(\xi) = f'_2(\xi) = \beta_1 + 2 \beta_2 \xi + 3 \beta_3 \xi^2.$

**$(e)$ Show that $f''_1 (ξ) = f''_2 (\xi)$. That is, $f''(x)$ is continuous at $\xi$.

Therefore, $f(x)$ is indeed a cubic spline.

Hint: Parts $(d)$ and $(e)$ of this problem require knowledge of single-variable
calculus. As a reminder, given a cubic polynomial
$$f_1(x) = a_1  + b_1 x + c_1 x^2 + d_1 x^3,$$
the first derivative takes the form
$$f'_1(x) = b_1 + 2 c_1 x + 3 d_1 x^2$$
and the second derivative takes the form
$$f''_1 (x) = 2 c_1 + 6 d_1 x.$$**

**Answer:**
>- **For $f''_1(\xi)$:**
$$
\begin{align}
f_1''(x) &= 2 \beta_2 + 6 \beta_3 x
\\
f_1''(\xi) &= 2 \beta_2 + 6 \beta_3 \xi \tag{1}
\end{align}
$$
>- **For $f''_2(\xi)$:**
$$
\begin{align}
f''_2(x) &= 2 \beta_2 + 6 \beta_3 x + 6 \beta_4 (x - \xi)
\\
f''_2(\xi) &= 2 \beta_2 + 6 \beta_3 \xi + \color{red}{6 \beta_4 (\xi - \xi)}
\\
&= 2 \beta_2 + 6 \beta_3 \xi \tag{2}
\end{align}
$$

>- From $(1)$ and $(2)$, $f''_1(\xi) = f''_2(\xi) = 2 \beta_2 + 6 \beta_3 \xi$.

### 2. Suppose that a curve $\hat{g}$ is computed to smoothly fit a set of n points using the following formula: $$\hat{g} = arg \min_{g}\
\bigg(\sum_{i=1}^n (y_i - g(x_i))^2\
+ \lambda \int [g^{(m)} (x)]^2 dx \bigg),$$
 where $g^{(m)}$ represents the $m$th derivative of $g$ (and $g^{(0)} = g$). Provide example sketches of $\hat{g}$ in each of the following scenarios.

**$(a)$ $\lambda = \infty,m = 0.$**

**Answer:**
>- In this case, it forces $g(x) = 0$. Thus, $\hat{g}(x) = 0$

**$(b)$ $\lambda = \infty,m = 1.$**

**Answer:**
>- In this case, it forces $g'(x) = 0$. Thus, $\hat{g}(x) = a$

**$(c)$ $\lambda = \infty,m = 2.$**

**Answer:**
>- In this case, it forces $g''(x) = 0$. Thus, $\hat{g}(x) = ax + b$

**$(d)$ $\lambda = \infty,m = 3.$**

**Answer:**
>- In this case, it forces $g'''(x) = 0$. Thus, $\hat{g}(x) = ax^2 + bx + c.$

**$(e)$ $\lambda = 0,m = 3.$**

**Answer:**
>- In this case, the penalty term doesn't matter anymore. This is the formula for linear regression, to choose $g$ based on minimizing $\text{RSS}$.

### $3.$ Suppose we fit a curve with basis functions $b_1(X) = X, b_2(X) = (X - 1)^2 I(X \geq 1)$. (Note that $I(X \geq 1)$ equals $1$ for $X \geq 1$ and $0$ otherwise.) We fit the linear regression model $$Y = \beta_0 + \beta_1 b_1(X) + \beta_2 b_2(X) + \epsilon,$$ and obtain coefficient estimates $\hat{\beta}_0 = 1,  \hat{\beta}_1 = 1, \hat{\beta}_2 = -2.$ Sketch the estimated curve between $X = -2$ and $X = 2$. Note the intercepts, slopes, and other relevant information.

```python
import numpy as np
import matplotlib.pyplot  as plt
np.random.seed(42)

# generate x and y
x = np.arange(-2, 2, 0.04)
e = np.random.rand(100)
b2x = (x-1)**2 * (x >= 1)
y = 1 + 1*x - 2 * b2x

# plot x vs y
plt.figure(figsize=(8,5))
plt.plot(x, y)
plt.axvline(x=1, color="black", linestyle="--")
plt.xlabel("X")
plt.ylabel("Y");
```

**Comment:**
>- For $X < 1$, the curve tend to be linear following the model: $Y = 1 + X$, with an intercept of $1$ and slop of $1$.
>- For $X \geq 1$, the quadratic term $(X-1)^2$ play a rule now in the model shown by the quadratic curve after $X \geq 1$ following the model: $Y = 1 + X -2 (X-1)^2$, the slope for $X \geq 1$ will vary as it is a function of $X$.

### $4.$ Suppose we fit a curve with basis functions $b_1(X) = I(0 \leq X \leq 2) - (X - 1)I(1 \leq X \leq 2), b_2(X) = (X - 3)I(3 \leq X \leq 4)+I(4 < X \leq 5).$ We fit the linear regression model $$Y = \beta_0 + \beta_1 b_1(X) + \beta_2 b_2(X) + \epsilon,$$ and obtain coefficient estimates $\hat{\beta_0} = 1, \hat{\beta_1} = 1, \hat{\beta_2} = 3$. Sketch the estimated curve between $X = -2$ and $X = 2$. Note the intercepts, slopes, and other relevant information.

```python
np.random.seed(42)

# generate x and y
x = np.arange(-2, 2, 0.04)
e = np.random.rand(100)
b1x = ((0 <= x) * (x <= 2)) - (x - 1) * ((1 <= x) * (x <= 2))
b2x = ((x-3) * (3 <= x) * (x <= 4)) + ((4 < x) * (x <= 5))
y = 1 + 1*b1x + 3*b2x

# plot x vs y
plt.figure(figsize=(8,5))
plt.plot(x, y)
plt.axvline(x=1, color="black", linestyle="--")
plt.xlabel("X")
plt.ylabel("Y");
```

**Comment:**
>- For $(-2 \leq X < 0)$, the curve is a constant horizontal line, following the model: $Y = 1$ as none of the indicator variables are true.
>  - With intercept of $1$, and slop of $0$.
>- For $(0 \leq X < 1)$, the curve is a constant horizontal line, following the model: $Y = 1 + 1 = 2$ as the first indicator variable is true.
>  - With intercept of $2$, and slop of $0$.
>- For $(1 \leq X \leq 2)$, the curve is linear following the model: $Y = 1 + 1 - (X -1) = 3 - X$ as both of the indicator variables are true.
>  - With intercept of $3$, and slop of $-1$.

### $5.$ Consider two curves, $\hat{g}_1$ and $\hat{g}_2$, defined by $$\hat{g}_1 = arg \min_{g}\
\bigg(\sum_{i=1}^n (y_i - g(x_i))^2\
+ \lambda \int [g^{(3)} (x)]^2 dx \bigg),$$ $$\hat{g}_2 = arg \min_{g}\
\bigg(\sum_{i=1}^n (y_i - g(x_i))^2\
+ \lambda \int [g^{(4)} (x)]^2 dx \bigg),$$ where $g^{(m)}$ represents the $m$th derivative of $g$.

**$(a)$ As $\lambda \to \infty$, will $\hat{g}_1$ or $\hat{g}_2$ have the smaller training $RSS$?**

**Answer:**
>- As $\lambda$ increases, the penalty term becomes more and more important in the equation above.
>- For $\hat{g}_1, \lambda \to \infty$ forces $g^{(3)}(x)\to 0$. The highest order (most flexible) polynomial to satisfy this would be of the form $\hat{g}(x)=ax^2+bx+c$ (as in Q: $2. d$). Thus, $\hat{g}_1$ will be the quadratic that minimizes the training $RSS$.
>- For $\hat{g}_2, \lambda \to \infty$ forces $g^{(4)}(x)\to 0$. The highest order (most flexible) polynomial to satisfy this would be of the form $\hat{g}(x)=ax^3+bx^2+cx+d$. Thus, $\hat{g}_2$ will be the cubic that minimizes the training $RSS$.

>- Since $\hat{g}_2$ would be more flexible as the higher order polynomial, $\hat{g}_2$ would have the smaller training $RSS$.

**$(b)$ As $\lambda \to \infty$, will $\hat{g}_1$ or $\hat{g}_2$ have the smaller test $RSS$?**
**Answer:**
>- As in $5.a$, $\hat{g}_2$ would be more flexible as the higher order polynomial, therefore $\hat{g}_1$ may have the smaller test $RSS$.
>- Still, it depends on the true relationship between $x$ and $y$. Is it better approximated with a cubic or quadratic relationship!

**$(c)$ For $\lambda = 0$, will $\hat{g}_1$ or $\hat{g}_2$ have the smaller training and test $RSS$?**

**Answer:**
>- As $\lambda = 0$, the penalty term doesn't matter anymore, means $\hat{g}_1 = \hat{g}_2$.
>- Therefore, both $\hat{g}_1$ and $\hat{g}_2$ would have the same training and test $RSS$.

# Done!


```python
!pip install nangs
```

    Collecting nangs
      Downloading https://files.pythonhosted.org/packages/4c/cd/7eeca93e6088a85d18ff99897743aaf429c94c12049e56f2a1e8b2091656/nangs-0.1.2-py3-none-any.whl
    Installing collected packages: nangs
    Successfully installed nangs-0.1.2

```python
pip install --upgrade git+https://github.com/adantra/nangs.git
```

    Collecting git+https://github.com/adantra/nangs.git
      Cloning https://github.com/adantra/nangs.git to /tmp/pip-req-build-2qdqzz11
      Running command git clone -q https://github.com/adantra/nangs.git /tmp/pip-req-build-2qdqzz11
    Building wheels for collected packages: nangs
      Building wheel for nangs (setup.py) ... done
      Created wheel for nangs: filename=nangs-0.1.2-cp37-none-any.whl size=13860 sha256=a09b04df717d9ca8fa7da6f4ff47dbd9ab21bbee1d99b05e3882c1e767d09903
      Stored in directory: /tmp/pip-ephem-wheel-cache-fbld0hdp/wheels/2c/7f/7f/677931a26064faa3c2f5f1f88c96cae03fadc9cffb901d8b1e
    Successfully built nangs
    Installing collected packages: nangs
      Found existing installation: nangs 0.1.2
        Uninstalling nangs-0.1.2:
          Successfully uninstalled nangs-0.1.2
    Successfully installed nangs-0.1.2

```python
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```python

```

```python
#imports

import numpy as np
import matplotlib.pyplot as plt
import torch
import pandas as pd
import nangs
from nangs import *

device = "cuda" if torch.cuda.is_available() else "cpu"

nangs.__version__, torch.__version__
```

    ('0.1.2', '1.9.0+cu102')

```python
device

```

    'cuda'

```python
alpha=10
class Difussion2d(PDE):
    def computePDELoss(self, inputs, outputs):
        p = outputs[:, 0]
        grads = self.computeGrads(p, inputs)
        dpdx, dpdt = grads[:, 0], grads[:, 1]
        # second order derivatives
        dp2dx2 = self.computeGrads(dpdx, inputs)[:, 0]
        #dp2dy2 = self.computeGrads(dpdy, inputs)[:, 1]
        return {'pde':dpdt - dp2dx2}
```

```python
class NeumannX(Neumann):
    def computeBocoLoss(self, inputs, outputs):
        p = outputs[:, 0]
        grads = self.computeGrads(p, inputs)
        dpdx = grads[:, 0]
        return {'gradX': 100*dpdx}
```

```python
class PBC0(Neumann):
    def computeBocoLoss(self, inputs, outputs):
        pbc = 0
        p = outputs[:, 0]
        return {'PWF': 100*(p-pbc)}
```

```python
pde = Difussion2d(inputs=('x', 't'), outputs='p')
```

```python
def append(arr, values, axis=None):
    arr = np.asanyarray(arr)
    if axis is None:
        if arr.ndim != 1:
            arr = arr.ravel()
        values = np.ravel(values)
        axis = arr.ndim-1
    return np.concatenate((arr, values), axis=axis)
```

```python
# mesh

x = np.linspace(0,1,100)

t = np.linspace(0,1,100)
#t = append(t,np.linspace(0.2,1,10))
#t=np.array([0])
#tt=np.logspace(-2,0,20)
#t = append(t,tt)
#mesh = Mesh({'x': x, 't':t}, device=device)
#pde.set_mesh(mesh)
```

```python
x = np.append(np.linspace(0,.01,10),x)

t = np.append(np.linspace(0,.01,10),t)
#t = append(t,np.linspace(0.2,1,10))
#t=np.array([0])
#tt=np.logspace(-2,0,20)
#t = append(t,tt)
mesh2 = Mesh({'x': x, 't':t}, device=device)
pde.set_mesh(mesh2)
```

```python
from nangs import Dirichlet
```

```python
t0 = np.array([0])
```

```python
_x = np.meshgrid(x)
#p0 = np.sin(2*np.pi*_x)*np.sin(2*np.pi*_y)
```

```python

_x[0].shape
```

    (110,)

```python
p0=np.zeros((len(x)))
p0[:]=1
p0[0]=0
#p0[47:53,47:53]=1
```

```python
p0.shape
```

    (110,)

```python
initial_condition = Dirichlet({'x': x,'t':t0}, {'p': p0.reshape(-1)}, device=device, name="initial")
```

```python
p0.shape
```

    (110,)

```python
pde.add_boco(initial_condition)
#p0=np.ones(len(y))
#p1=np.zeros(len(y))
```

```python
pwf = PBC0({'x': np.array([0]),'t':t}, name='pwf', device=device)
#left = Dirichlet({'x': x, 'y': y}, {'p': p0.reshape(-1)}, device=device, name="left")
```

```python

pde.add_boco(pwf)
```

```python
#right = PBC0({'x': np.array([1]), 'y': y, 't':t}, name='right', device=device)
#right = Dirichlet({'x': x[-1], 'y': y}, {'p': p1.reshape(-1)}, device=device, name="right")
right =  NeumannX({'x': x[-1],'t':t}, name='right', device=device)
```

```python
pde.add_boco(right)
```

```python
#from nangs import MLP

BATCH_SIZE = 128
LR = 1e-2
EPOCHS = 2
NUM_LAYERS = 4
NUM_HIDDEN = 50

mlp = MLP(len(pde.inputs), len(pde.outputs), NUM_LAYERS, NUM_HIDDEN).to(device)
optimizer = torch.optim.Adam(mlp.parameters())
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=LR, pct_start=0.1, total_steps=EPOCHS)
```

```python
mlp
```

    MLP(
      (mlp): Sequential(
        (0): Linear(in_features=2, out_features=50, bias=True)
        (1): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=50, bias=True)
        )
        (2): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=50, bias=True)
        )
        (3): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=50, bias=True)
        )
        (4): Sequential(
          (0): GELU()
          (1): Linear(in_features=50, out_features=1, bias=True)
        )
      )
    )

```python

EPOCHS = 300
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=LR, pct_start=0.1, total_steps=EPOCHS)
```

```python
pde.compile(mlp, optimizer, scheduler)
```

```python

%time hist = pde.solve(EPOCHS, BATCH_SIZE)
```

```python
hist['pde'][-1]
```

    0.004050046222676572

```python
t = 0.0
x = np.linspace(0,1,100)
#y = np.linspace(0,1,100)

_x, _t = np.meshgrid(x, t)
eval_mesh = Mesh({'x': x,  't':t}, device=device)
p = pde.eval(eval_mesh)
p = p.cpu().numpy()
#p5 = p.reshape((_y.shape[0],_x.shape[1]))
```

```python
p,grad,loss=pde.eval_with_grad(eval_mesh)
```

```python
ppp=[]
loss = []
x = np.linspace(0.0,1.,10)
qq=[]
for t in np.linspace(0.01,1,120):
  #_x,_t = np.meshgrid(x,t)
  eval_mesh = Mesh({'x': x,  't':t}, device=device)
  pp,g,l = pde.eval_with_grad(eval_mesh)
  qq.append(g)
  loss.append(l)
  #pp = pp.cpu().numpy()
  ppp.append(pp)
```

```python
for l in loss:
  plt.plot(np.power(l,2)/hist['pde'][-1])
```

```python
q0=[]
qL=[]
for q in qq:
  plt.plot(q.detach().cpu().numpy()[0])
  q0.append(q.detach().cpu().numpy()[0][0])
  qL.append(q.detach().cpu().numpy()[0][-1])
```

```python
plt.plot(q0)
```

```python
plt.plot(qL[10:])
```

```python
from sympy import Sum, factorial, oo, IndexedBase, Function, sin,exp
from sympy.abc import n
```

```python
def diff_sol(x,t,alpha=0.1):
    sol= Sum(4/((2*n + 1)*np.pi)*sin((2*n+1)*np.pi/2*x)*exp(-((2*n + 1)*np.pi/2)**2*t*alpha),(n,0,50))

    return sol #,sol_n
```

```python
xxx=np.linspace(0,1,100)
for t in np.linspace(0.1,1,11):
    d=[]
    for xx in xxx:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d)
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(p)
```

```python
xxx=np.linspace(0,1,10)
x = np.linspace(0,1,10)
for t in np.linspace(0.02,1,11):
    d=[]
    for xx in xxx:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d)
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(pp,marker='o',linestyle='None')
```

```python

out=pde.eval_with_grad(eval_mesh)
```

```python
out_np=out[1].detach().cpu().numpy()
```

```python
out_np.shape
```

    (100, 2)

```python
#xxx=np.linspace(0,1,10)
x = np.linspace(0,1,10)
for t in np.linspace(0.05,1,10):
    d=[]
    for xx in x:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d,marker='o',color='y')
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(pp,marker='+',linestyle='None',color='black')
```

```python
x = np.linspace(0,1,10)
for t in np.linspace(0.05,1,10):
    d=[]
    for xx in x:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d,marker='o',color='y')
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(pp,marker='+',linestyle='None',color='black')
```

```python
x = np.linspace(0,1,10)
for t in np.linspace(0.01,1,10):
    d=[]
    for xx in x:
        d.append(diff_sol(xx,t,alpha=1).doit())
    plt.plot(d,marker='o',color='y')
    eval_mesh = Mesh({'x': x,  't':t}, device=device)
    pp = pde.eval(eval_mesh)
    pp = pp.cpu().numpy()
    plt.plot(pp,marker='+',linestyle='None',color='black')
```

```python

```


```python
adj = [[]] # adj list representation 0th element is dummy and nodes indexed from 1
edgelist = []
fileprefix = "test"
with open(fileprefix+".graph","r") as infile:
    first = infile.readline()
    pars = first.split(" ")
    n = int(pars[0])
    edges = int(pars[1])
    K = int(pars[2])
    for i in range(edges):
        adj.append([])
    for i in range(edges):
        line = infile.readline()
        spl = line.split(" ")
        n1 = inr(spl[0])
        n2 = inr(spl[2])
        edgelist.append([n1,n2])
#         edgelist.append([n2,n1])
        adj[n1].append(n2)
        adj[n2].append(n1)
```

```python
sympy.to_cnf()
```

```python
a,b,c,d,e,f,g,h,i,j = symbols('a,b,c,d,e,f,g,h,i,j')
```

```python
exp = (a&b)|(c&d)|(e&f)|(g&h)|(i&j)
```

```python
exp
```

```python
exp2 = to_cnf(exp,simplify=True)
```

```python
simplify_logic(exp, form='dnf')
```

```python

```

```python
# from sympy import *
# import argparse
import sys

# Variable numbering in minsat starts with 1
# i, j is node, sub-graph number
def get1Dindex(i, j, k):
    return (i*k + j)

# Variable numbering in minsat starts with 1
def get2Dindex(idx, k):
    # can treat it as var starting from 0
    # idx = idx - 1
    i = idx // k
    j = idx - i*k
    return i,j

def getInput(fileprefix):

    # Construct graph from INPUT
    edgelist = []
    # fileprefix = "test"
    with open(fileprefix+".graph","r") as infile:
        first = infile.readline()
        pars = first.split(" ")
        n = int(pars[0])
        edgeMatrix = [[False for x in range(n)] for y in range(n)]
        edges = int(pars[1])
        K = int(pars[2])

        for i in range(edges):
            line = infile.readline()
            spl = line.split(" ")
            n1 = int(spl[0])
            n2 = int(spl[1])
            n1 -= 1
            n2 -= 1
            edgelist.append([n1,n2])

            # print(n1, n2)
            edgeMatrix[n1][n2] = True
            edgeMatrix[n2][n1] = True

    return edgelist, edgeMatrix, n, K, edges

def generate(fileprefix):

    edgeList, edgeMatrix, n, K, edges = getInput(fileprefix)
    # A clause is a list of (list of tuples)
    # Tuples of (Bool, number) where
    clause1 = []

    # variableMatrix[n][K] represents whether node n is present in k'th graph
    # variableMatrix = [[False for i in range(n)] for j in range(K)]

    for i in range(n):
        l = []
        for j in range(K):
            l.append((True, get1Dindex(i,j,K)))
        # element wise append
        clause1.append(l)

    # print("Printing clause1 ------- " , clause1)

    # DNF TYPE , or of 2 and's (x1^x2) V (x3^x4) V ...
    clause2 = []

    # print("EDGE MATRIX", edgeMatrix)
    for i in range(n):
        for j in range(i+1, n):
            if (edgeMatrix[i][j]):
                l = []
                for t in range(K):
                    l.append( [(True, get1Dindex(i, t, K)), (True, get1Dindex(j, t, K))] )
                clause2 = clause2 + l

    # numVar is the index of next free variable
    numVar = n*K + 1
    # print('Printing edge matrix', edgeMatrix)
    # print("Printing clause 2  ------- " , clause2)
    clause2, numVar = to_cnf(clause2, numVar)
    # print("Printing clause 2 ---- " , clause2)

    clause3 = []
    # i and j are nodes and t is subgraph
    for i in range(n):
        for j in range(i+1,n):
            l = []
            if(not edgeMatrix[i][j]):
                for t in range(K):
                    l.append([(False, get1Dindex(i, t, K)), (False, get1Dindex(j, t, K) )])
            # concat l with clause3
            clause3.extend(l)

    # print("Printing clause3 ------- " , clause3)
    # DNF TYPE , or of 2 and's (x1^x2) V (x3^x4) V ...
    clause4 = []

    # i and j are subgraphs
    for i in range(K):
        for j in range(K):
            if i == j:
                continue
            l = []
            for node in range(n):
                l.append([(True, get1Dindex(node, i, K)), (False, get1Dindex(node, j, K))])
            # concat l with clause4
            clause4 = clause4 + l

    # print("Printing clause4 ------ " , clause4)

    # numVar is the index of next free variable
    clause4, numVar = to_cnf(clause4, numVar)
    # print("Printing clause4 ------ " , clause4)

    # Clause 5, we thought it was included in previous constraints ?
    clause5 = []

    for i in range(n):
        for j in range(i+1,n):
            l = []
            if(edgeMatrix[i][j]):
                # print("PRINTING i, j in clause 5 --- ", i, j)
                for t in range(K):
                    # OR of pairwise and
                    l.append( [(True, get1Dindex(i, t, K)), (True, get1Dindex(j, t, K))] )
                l, numVar = to_cnf(l,numVar)
            clause5 = clause5 + l

    # print("Printing clause 5 ----- ", clause5)

    clause = clause1 + clause2 + clause3 + clause4 + clause5

    # print("Num Var ---- ",numVar)

    writeSatInput(fileprefix, clause, numVar)

def writeSatInput(fileprefix, clause, numVar):
    # 1 to numVar all are varialbes
    with open(fileprefix+".satinput","w") as outfile:

        string1 = "p cnf %d %d\n" %(numVar, len(clause))
        outfile.write(string1)

        for exp in clause:
            st = ""
            for term in exp:
                if (not term[0]):
                    st += "-"
                # To make the variables 1 indexed
                st += str(term[1]+1) + " "

            st += "0\n"
            outfile.write(st)

        outfile.close()

def to_cnf(dnf_clause, free_var_num):

    if (dnf_clause == []):
        return [], free_var_num

    outl = []
    num_ands = len(dnf_clause)
    tmpl = []
    for i in range(num_ands):
        tmpl.append((True, free_var_num+i))
    outl.append(tmpl)

    for i in range(num_ands):
        for var in dnf_clause[i]:
            outl.append([(False, free_var_num + i),var])

    return outl, (free_var_num + num_ands)

def get_out(fileprefix, num_vars):
    with open(fileprefix+".satoutput","r") as sat_out:
        issat = sat_out.readline()
        if issat != "SAT\n":
            print("0")
            exit(-1)
        sol  = sat_out.readline()
        assign = sol.split(" ")
        assign = assign[:-1] # check that this removes 0 only
        assignments = []
        for i in range(num_vars):
            if assign[i][0]=="-":
                assignments.append(False)
            else:
                assignments.append(True)

    return assignments

# UDIT: I have tested this function (for sample case),
# this is running as expected [assignments, i&j order etc]
def out(fileprefix):

    edgeList, edgeMatrix, n, K, edges = getInput(fileprefix)

    # print(edgeList, n, K)

    assignments = get_out(fileprefix, n*K)
    # print("Assignments are ----",assignments)
    variableMatrix = [[False for i in range(K)] for j in range(n)]
    for t in range(len(assignments)):
        i, j = get2Dindex(t, K)
        variableMatrix[i][j] = assignments[t]

    # print("Variable matrix is ---",variableMatrix)

    for t in range(K):
        count = 0
        for i in range(n):
            if(variableMatrix[i][t]):
                count+=1
        print("#%d %d\n"%(t+1, count), end="")
        for i in range(n):
            if(variableMatrix[i][t]):
                count -= 1
                print(i+1, end="")
                if(count > 0):
                    print(" ", end="")
        print()

# fileprefix = "jumbotest"
# mode = 0
# if mode == 0:
#     generate(fileprefix)
# else:
#     out(fileprefix)
```

```python
fileprefix = "test4"
mode = 0

```

```python
%load_ext line_profiler
```

```python
%lprun -f generate generate(fileprefix)
```

```python

```


```python
%load_ext autoreload
%autoreload 2
```

# First steps using TVB

**EITN Fall School, September 15th, 2021**

# Brain network modeling with TVB

***

Components to be covered
* node dynamics
* connectivity
* coupling functions
* monitors
* integrator
* stimulus

# Objectives

Here we:

* Build a brain network model using subject-specific structural connectivity,
* Simulate resting-state activity,
* Characterize the resting-state activity by calculating the functional connectivity (FC).

# How to do it with TVB?

***

In the first part of this tutorial, we presents the basic anatomy of a region simulation using The Virtual Brain's scripting interface.

The first thing we want to do is to import the modules we will need for a simulation.

```python
%%capture
import os
import time as tm

import numpy as np
import matplotlib.pyplot as plt

from tvb.simulator.lab import *
from utils import plot_connectivity
from phase_plane import phase_plane_interactive
```

A basic simulation of TVB consists of **5 main components**. Each of these components is an object within TVB:

### Connectivity

We start by loading and visualizing the structural connectivity matrix that represents the set of all existing connections between brain areas. Having loaded the default dataset, we can then alter the speed of signal propagation through the network:

```python
# Import the anatomical structural connectivity.
conn = connectivity.Connectivity().from_file(
    os.path.abspath('dataset/connectivity_76.zip')
)
```

```python
nregions = len(conn.region_labels)               # Number of regions
conn.speed = np.array(np.inf)                    # Set the conduction speed
conn.configure()
```

*Take a look at some of the properties of the `conn` object: `weights`, `delays`, `region_labels`, etc.*

```python
conn
```

Connectivity

	value

Number of connections
1560

Number of regions
76

Undirected
False

areas [min, median, max]
[0, 2580.89, 10338.2]

areas dtype
float64

areas shape
(76,)

tract_lengths (connections) [min, median, max]
[0, 55.8574, 138.454]

tract_lengths [min, median, max]
[0, 71.6635, 153.486]

tract_lengths-non-zero [min, median, max]
[4.93328, 74.0646, 153.486]

weights [min, median, max]
[0, 0, 3]

weights dtype
float64

weights shape
(76, 76)

weights-non-zero [min, median, max]
[0.00462632, 2, 3]

```python
plot_connectivity(conn)
```

```python
plt.imshow(conn.weights)
```

```python
print(conn.weights)
```

    [[2. 2. 0. ... 0. 0. 0.]
     [3. 2. 0. ... 0. 0. 0.]
     [0. 0. 2. ... 0. 0. 0.]
     ...
     [0. 0. 0. ... 2. 2. 0.]
     [0. 0. 0. ... 3. 2. 0.]
     [0. 0. 0. ... 0. 0. 0.]]

### Model

A set of differential equations describing the local neural dynamics. There are a number of predefined models available in TVB, as an example here we will use the **Generic 2-dimensional Oscillator** model:

\begin{align}
\dot{V} &= d \, \tau (-f V^3 + e V^2 + g V + \alpha W + \gamma I), \\
                \dot{W} &= \dfrac{d}{\tau}\,\,(c V^2 + b V - \beta W + a).
\end{align}

We can explore it using the **phase-plane** tool in TVB.

```python
# Create and launch the phase-plane tool
g2d = models.Generic2dOscillator()                 # The model to investigate
g2d.a = np.array([3])
heunint = integrators.HeunDeterministic(dt=1.0)    # Integrator to use.
                                                   # dt is the integration step

phase_plane_interactive(model=g2d, integrator=heunint)

# Try changing the default parameters to
#    1) a = 3   (limit cycle)
#    2) a = 1   (two stable and unstable fixed points)
```

For our simulation we set it to the limit-cycle regime.

*Note that the parameters of the `Model` class has to be set as numpy arrays.*

```python
g2d
```

Generic2dOscillator

	value

I [min, median, max]
[0, 0, 0]

I dtype
float64

I shape
(1,)

Type
Generic2dOscillator

a [min, median, max]
[3, 3, 3]

a dtype
float64

a shape
(1,)

alpha [min, median, max]
[1, 1, 1]

alpha dtype
float64

alpha shape
(1,)

b [min, median, max]
[-10, -10, -10]

b dtype
float64

b shape
(1,)

beta [min, median, max]
[1, 1, 1]

beta dtype
float64

beta shape
(1,)

c [min, median, max]
[0, 0, 0]

c dtype
float64

c shape
(1,)

d [min, median, max]
[0.02, 0.02, 0.02]

d dtype
float64

d shape
(1,)

e [min, median, max]
[3, 3, 3]

e dtype
float64

e shape
(1,)

f [min, median, max]
[1, 1, 1]

f dtype
float64

f shape
(1,)

g [min, median, max]
[0, 0, 0]

g dtype
float64

g shape
(1,)

gamma [min, median, max]
[1, 1, 1]

gamma dtype
float64

gamma shape
(1,)

gid
UUID('ce662352-d936-44ac-8546-33f9e862c2b2')

state_variable_range
{'V': array([-2.,  4.]), 'W': array([-6.,  6.])}

tau [min, median, max]
[1, 1, 1]

tau dtype
float64

tau shape
(1,)

title
Generic2dOscillator gid: ce662352-d936-44ac-8546-33f9e862c2b2

variables_of_interest
('V',)

```python
# Initialize the model
g2d = models.Generic2dOscillator(a=np.array(1.7402))
g2d
```

Generic2dOscillator

	value

I [min, median, max]
[0, 0, 0]

I dtype
float64

I shape
(1,)

Type
Generic2dOscillator

a [min, median, max]
[1.7402, 1.7402, 1.7402]

a dtype
float64

a shape
()

alpha [min, median, max]
[1, 1, 1]

alpha dtype
float64

alpha shape
(1,)

b [min, median, max]
[-10, -10, -10]

b dtype
float64

b shape
(1,)

beta [min, median, max]
[1, 1, 1]

beta dtype
float64

beta shape
(1,)

c [min, median, max]
[0, 0, 0]

c dtype
float64

c shape
(1,)

d [min, median, max]
[0.02, 0.02, 0.02]

d dtype
float64

d shape
(1,)

e [min, median, max]
[3, 3, 3]

e dtype
float64

e shape
(1,)

f [min, median, max]
[1, 1, 1]

f dtype
float64

f shape
(1,)

g [min, median, max]
[0, 0, 0]

g dtype
float64

g shape
(1,)

gamma [min, median, max]
[1, 1, 1]

gamma dtype
float64

gamma shape
(1,)

gid
UUID('967065c8-1882-43f2-a9a0-c28b1c09c645')

state_variable_range
{'V': array([-2.,  4.]), 'W': array([-6.,  6.])}

tau [min, median, max]
[1, 1, 1]

tau dtype
float64

tau shape
(1,)

title
Generic2dOscillator gid: 967065c8-1882-43f2-a9a0-c28b1c09c645

variables_of_interest
('V',)

### Coupling function

It is a function that is used to join the local `Model` dynamics at distinct spatial locations over the connections described in `Connectivity`. Proper setting of the parameters for this function requires some knowledge of the properties of both the model being used and the structure through which it is connected. For our present purposes, we happen to know that for this configuration of parameters of TVB's `Generic2dOscillator` connected through TVB's default connectivity matrix, a linear function with a slope of 0.0075 is a reasonable thing to use.

```python
# Initialise a Coupling function.
G = np.array(0.0075)
con_coupling = coupling.Scaling(a=G)
```

### Integrator

Now that we have defined our structure and dynamics, we need to select an integration scheme. While TVB supports a number of schemes, for most purposes you should use either `HeunDeterministic` or `HeunStochastic`.

Note that the most important thing here is to use a step size that is small enough for the integration to be numerically stable.

```python
# Initialise an Integrator scheme.
dt = 0.1                          # Integration step [ms]

# We can use the deteministic integrator:
# heunint = integrators.HeunDeterministic(dt=dt)

# Or stochastic integrator:
nsigma = 0.001                    # Standard deviation of the noise
hiss = noise.Additive(nsig=np.array([nsigma, 0]))
heunint = integrators.HeunStochastic(dt=dt, noise=hiss)
```

### Monitors

The last component is `Monitor`: observer models. Although there are Monitors which apply a biophysical measurement process to the simulated neural activity, such as EEG, MEG, etc, here we will select two simple monitors to get the time series of the model state variables:

* the `Raw` monitor takes no arguments and simply returns all the simulated data at the time resolution of the integration step,
* the `TemporalAverage` monitor averages over a time window of length period returning one time point every period (given in ms).

*Note: as a general rule, the `Raw` monitor shouldn't be used for anything but very short simulations as the amount of data returned can become prohibitively large*

```python
# Initialise some Monitors with period in physical time.
mon_raw = monitors.Raw()
mon_tavg = monitors.TemporalAverage(period=1)    # 1000 Hz

#Bundle them
what_to_watch = (mon_raw, mon_tavg)
```

***

### Go! Simulate

The last step is to bring all these components together into a `Simulator` object. We then need to run the configure method, which basically just acts to calculate information necessary for the simulation that draws on specific combinations of the components.

```python
# Initialise the Simulator.
sim = simulator.Simulator(model=g2d,
                          connectivity=conn,
                          conduction_speed=np.float(conn.speed),
                          coupling=con_coupling,
                          integrator=heunint,
                          monitors=what_to_watch)
sim.configure()
```

    /Users/mariolavanga/Documents/GitHub/eitn_school_2021_tvb_project/env/lib/python3.7/site-packages/ipykernel_launcher.py:4: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
    Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
      after removing the cwd from sys.path.

Simulator

	value

Type
Simulator

conduction_speed
inf

connectivity
Connectivity gid: 7550dcb2-080c-4758-a1a8-2ecd88cf0573

coupling
Scaling gid: 08cda14e-45a3-422f-b5be-51087c8bd684

gid
UUID('edbc5a45-0fd7-4904-9586-a136082bec47')

initial_conditions
None

integrator
HeunStochastic gid: e47b60cf-04e0-4285-94d0-146399e2ca19

model
Generic2dOscillator gid: 967065c8-1882-43f2-a9a0-c28b1c09c645

monitors
(<tvb.simulator.monitors.Raw object at 0x13456ce90>, <tvb.simulator.monitors.TemporalAverage object at 0x13456c6d0>)

simulation_length
1000.0

stimulus
None

surface
None

title
Simulator gid: edbc5a45-0fd7-4904-9586-a136082bec47

Now, we can run the simulation. All we need to do is iterate for some length, which we provide in *ms*, and collect the output.

The data returned by the simulator is in the form of a list of arrays. For most subsequent purposes it is much easier to deal with the data if it exists as a single contiguous array, hence the enumeration of the return values for the `sim.run`.

```python
# Perform the simulation.
tic = tm.time()

# this should take ~25s
(raw_time, raw_data), (tavg_time, tavg_data) = sim.run(simulation_length=5000.) # ~30s

'simulation required %0.3f seconds.' % (tm.time()-tic)
```

    'simulation required 11.425 seconds.'

For each monitor, the simulation returns the time points and the associated data array.

The data arrays have shapes $n_\text{timepoints} \times n_\text{variables of interest} \times n_\text{regions} \times n_\text{modes}$.

```python
f"raw_time: {raw_time.shape} | raw_data: {raw_data.shape}"
```

    'raw_time: (50000,) | raw_data: (50000, 1, 76, 1)'

***

### Visualize our simulation

And finally, we can look at the results of our simulation in terms of time series.

```python
# Remove the dimensions with one element for easier indexing
raw = np.squeeze(np.array(raw_data))
tavg = np.squeeze(np.array(tavg_data))
raw.shape, tavg.shape
```

    ((50000, 76), (5000, 76))

```python
# Normalize the time series for easier visualization
nraw = raw / (np.max(raw, 0) - np.min(raw, 0))
ntavg = tavg / (np.max(tavg, 0) - np.min(tavg, 0))
```

```python
# Plot the raw time series
fig1 = plt.figure(figsize=(10,8))
plt.plot(raw_time[:], nraw[:, :10] + np.r_[:10])
plt.title('Raw Neuronal Activity', fontsize=20)
plt.xlabel('Time [ms]', fontsize=20)
plt.yticks(range(10), conn.region_labels[:10], fontsize=10)

# Plot the temporally averaged time series
fig2 = plt.figure(figsize=(10,8))
plt.plot(tavg_time[:], ntavg[:, :10] + np.r_[:10])
plt.title('Temporally Averaged Neuronal Activity', fontsize=20)
plt.xlabel('Time [ms]', fontsize=20)
plt.yticks(range(10), conn.region_labels[:10], fontsize=10)

plt.show()
```

***

# Analysis example: Functional Connectivity (FC)

**Functional Connectivity (FC)** describes the connectedness of two brain regions by means of the covariance between their time series. The classic and most widely used method to infer the strength of network interactions or functional connections consists in estimating the linear (Pearson) correlation coefficient between temporal signals. If two regions activate and deactivate at the same time, there is likely a functional connection.

To calculate the FC, we ignore the first second of the simulation which might be affected by the initial conditions.

```python
fc = np.corrcoef(tavg[1000:].T)
```

And now we display the resulting FC matrix:

```python
# Visualize FC matrix
plt.figure(figsize=(8,8))
plt.imshow(fc, interpolation='nearest', cmap='jet')
plt.title('SIM FC', fontsize=20)
plt.xlabel('ROIs', fontsize=20); plt.ylabel('ROIs', fontsize=20)
plt.xticks([], fontsize=20); plt.yticks([], fontsize=20)
cb = plt.colorbar(shrink=0.7)
cb.set_label('PCC', fontsize=20)

plt.show()
```


```python
from sympy import *
init_printing(use_unicode=True)
from sympy.codegen.ast import Assignment
```

```python
C = Matrix( symarray('C', (2,2)) )
R = Matrix( symarray('R', (2,2)) )
n = Matrix( symarray('n', (2)) )
t = Matrix( symarray('t', (2)) )
```

```python
R[0,0]=t[0]
R[1,0]=t[1]
R[0,1]=n[0]
R[1,1]=n[1]
```

```python
C
```

```python
simplify(transpose(R)*C*R)
```

```python

```


# Tutorial - ``params``, ``options``, and simulation

In this first tutorial we will learn how to specify and simulate a simple model with **respy**. As economists love Robinsonades [1](#fn1), we will implement a simple Robinson Crusoe economy as a discrete choice dynamic programming model.

```python
%matplotlib inline

import pandas as pd
import respy as rp
import matplotlib.pyplot as plt
import seaborn as sns
```

## The model: a simple Robinson Crusoe economy

In this basic Robinson Crusoe economy, Robinson is alone on a desert island and must choose every period $t = \{0, 1, ... T-1\}$ between two alternatives: fishing or relaxing on his hammock. Each of the two alternatives is associated with a certain level of instantaneous utility, determined in each period $t$ by a deterministic component and a stochastic component. As Robinson cannot foresee these stochastic components, we speak of "choice under uncertainty".

In **respy**, the instantaneous utility of an alternative, $u_t$, consists of the sum of a pecuniary reward or wage $W_t$ and a non-pecuniary reward $N_t$.

Speaking of a wage in this context seems inappropriate since Robinson, being alone on an island, won't receive any wage for the time he spends fishing. However, this distinction is useful to keep in mind when tackling more complex models of labor supply.

The **wage** is zero if Robinson chooses to relax on his hammock. If Robinson goes fishing his wage at time $t$ is:

$$
W^f = exp\{x_t \beta^f + \epsilon_{ft}\}
$$

where:

- $f$ indicates the "fishing" alternative;
- $x_t$ denotes Robinson's experience in fishing in period $t$;
- $\beta^f$ denotes the return to experience in fishing;
- $\epsilon_{ft}$ is the random shock in period $t$.

In words, the deterministic component of Robinson's wage is generated each day by only one covariate, his current experience in fishing (which is 0 if Robinson never fished before, 1 if Robinson fished once before, and so on), times the return to experience in fishing. The stochastic component, $\epsilon_{ft}$, is drawn from a normal distribution, and (may) depend on the current time $t$ and on Robinson's current choice.

The **non-pecuniary reward** for fishing and relaxing on the hammock are simply set as constant terms, with an additional shock, also drawn from a Normal distribution, entering the latter additively:

$$\begin{align}
N^f = \alpha^f \\
N^h = \alpha^h + \epsilon_{ht}\
\end{align}$$

Therefore, the parameters of this simple model are $\alpha^h$ and $\alpha^f$ (the constants in the non-pecuniary rewards) and $\beta^f$ (return to experience in fishing), along with the time-preference parameter (see below) and the lower triangular elements of the variance-covariance matrix which governs the distribution of the shocks.

More generally, **respy** allows the wage and non-pecuniary rewards to be flexibly generated specifying a vector of choice- and time-dependent covariates and the associated vector of choice-dependent parameters.

    To how-to guide

    Find out more about setting up a model in  respy  in How to
    specify a model.

## Solution

Discrete choice models are usually derived under an assumption of utility-maximizing behavior by the decision-maker. Indeed, below we assume that Robinson is forward-looking and thus, in each period $t$, wants to maximize the expected present value of utility over the remaining periods. However, Robinson can discount the utility stream derived from future periods in different ways. **respy** allows for:

- Exponential discounting
- Hyperbolic discounting (with a completely naïve agent)

In the following, we assume Robinson to be an exponential discounter. This means that Robinson's time preferences can be described by:

$$
U_t(u_t, u_{t+1}, ...) \equiv \sum_{t=0}^T{\delta^t u_t}
$$

where $\delta \in (0, 1]$  is the long-run discount factor which capture Robinson's preferences for immediate
over future gratification.

If $\delta=0$, Robinson is completely myopic and in each period $t$ simply chooses the alternative associated with the highest instantaneous utility. If $\delta>0$, in each period $t$, Robinson doesn't compare levels of instantaneous utility
but *value functions* which incorporate the utility derived from future choices.

Robinson's problem (choosing the optimal sequence of alternatives) can be solved with dynamic programming: The resulting decision rule can be used to simulate the choices Robinson would make under a different set of parameters — that is, to simulate counterfactuals.

    To how-to guide

    See "Impatient Robinson"
    for more on hyperbolic discounting in  respy.

## Specification: ``params`` and ``options``

The parameters of this simple model can be easily loaded with **respy**. The resulting DataFrame contains a two-level MultiIndex to group parameters (identified by a unique name and value) in categories. In particular:

- ``"delta"`` refers to Robinson's long-run discount factor;
- ``"wage_fishing"``groups the covariates and associated parameters appearing in the wage. Indeed, ``"exp_fishing"`` represents Robinson's experience in fishing (which is determined endogenously and recomputed from period to period) while the associated ``value`` is the return to experience;
- ``"nonpec_fishing"`` groups the covariates and associated parameters appearing in the non-pecuniary rewards, and therefore contains two constant terms. Note that in the example below we set ``"nonpec_fishing"`` to be negative, to reflect the effort cost of fishing, while ``"nonpec_hammock"`` is set as positive to reflect the good feeling of relaxing on the hammock.
- ``"shocks_sdcorr"`` groups the lower triangular elements of the shocks variance-covariance matrix. In **respy**, shocks are allowed to be correlated across choices.

```python
params, options = rp.get_example_model("robinson_crusoe_basic", with_data=False)
```

```python
params
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.30

      nonpec_fishing
      constant
      -0.20

      nonpec_hammock
      constant
      2.00

      shocks_sdcorr
      sd_fishing
      0.50

      sd_hammock
      0.50

      corr_hammock_fishing
      0.00

``params`` is complemented with ``options``, which contain additional information needed for the solution, simulation and estimation of the model. For example, `"n_periods"` indicates the number of periods for which the model is solved and simulated, and ``"simulation_agents"`` indicates how many identical individuals are simulated.

```python
options
```

    {'solution_draws': 100,
     'solution_seed': 456,
     'n_periods': 5,
     'simulation_agents': 1000,
     'simulation_seed': 132,
     'estimation_draws': 100,
     'estimation_seed': 100,
     'estimation_tau': 0.001,
     'interpolation_points': -1,
     'covariates': {'constant': '1'}}

Note that the behavior across identical simulated individuals can differ, as each of them faces a different sequence of random shocks: Seeds (denoted by ``"_seed"``in the ``options``) are used in every model component to ensure reproducibility. You can use any seed you like or even repeat the same seed number. Internally, we ensure that randomness is completely uncorrelated.

    To reference guide

    Find out more in "Randomness an Reproducibility".

## Simulation

We are now ready to simulate the model: We will simulate 1000 identical Robinsons and let the decision rule from the solution of the model guide them for 5 periods. Although the 1000 Robinsons start with the same characteristics, their actions from period to period can differ because of the different shocks they experience.

We will then inspect the predicted shares of Robinsons who choose the "fishing" or "hammock" alternative respectively. In other words, we will inspect the predicted probability with which Robinson chooses between fishing and relaxing on his hammock in each period.

Performing a simulation returns a MultiIndex containing all the information needed to inspect Robinson's decisions period by period, including his choice, instantaneous utility and level of experience in each period. Each of the 1000 Robinsons chooses one alternative in each ``"Period"`` and has a personal ``"Identifier"``.

    To how-to guide

    See "Simulation"
    for a guide on how to simulate with  respy.

```python
simulate = rp.get_simulate_func(params, options)
df = simulate(params)
```

```python
df.head(10)
```

      Experience_Fishing
      Shock_Reward_Fishing
      Meas_Error_Wage_Fishing
      Shock_Reward_Hammock
      Meas_Error_Wage_Hammock
      Dense_Key
      Core_Index
      Choice
      Wage
      Discount_Rate
      ...
      Nonpecuniary_Reward_Fishing
      Wage_Fishing
      Flow_Utility_Fishing
      Value_Function_Fishing
      Continuation_Value_Fishing
      Nonpecuniary_Reward_Hammock
      Wage_Hammock
      Flow_Utility_Hammock
      Value_Function_Hammock
      Continuation_Value_Hammock

      Identifier
      Period

      0
      0
      0
      -0.035035
      1
      0.040965
      1
      0
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.982635
      0.782635
      9.172605
      8.831548
      2
      NaN
      2.020483
      9.339589
      7.704322

      1
      0
      0.074254
      1
      1.506491
      1
      1
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      1.037825
      0.837825
      6.911083
      6.392903
      2
      NaN
      2.753245
      8.355429
      5.897036

      2
      0
      -0.354560
      1
      1.185316
      1
      2
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.837545
      0.637545
      4.677616
      4.252706
      2
      NaN
      2.592658
      6.398066
      4.005693

      3
      0
      -0.109397
      1
      -0.785877
      1
      3
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.946770
      0.746770
      2.793888
      2.154861
      2
      NaN
      1.607061
      3.554221
      2.049642

      4
      0
      -1.063705
      1
      1.245234
      1
      4
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.587516
      0.387516
      0.387516
      0.000000
      2
      NaN
      2.622617
      2.622617
      0.000000

      1
      0
      0
      -0.713137
      1
      1.418950
      1
      0
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.700075
      0.500075
      8.890045
      8.831548
      2
      NaN
      2.709475
      10.028581
      7.704322

      1
      0
      -0.464134
      1
      -0.384774
      1
      1
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.792893
      0.592893
      6.666151
      6.392903
      2
      NaN
      1.807613
      7.409797
      5.897036

      2
      0
      0.066755
      1
      -0.844060
      1
      2
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      1.033941
      0.833941
      4.874012
      4.252706
      2
      NaN
      1.577970
      5.383379
      4.005693

      3
      0
      -0.748565
      1
      -1.348821
      1
      3
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.687783
      0.487783
      2.534901
      2.154861
      2
      NaN
      1.325589
      3.272749
      2.049642

      4
      0
      -0.130574
      1
      -0.539693
      1
      4
      0
      hammock
      NaN
      0.95
      ...
      -0.2
      0.936799
      0.736799
      0.736799
      0.000000
      2
      NaN
      1.730154
      1.730154
      0.000000

10 rows × 21 columns

Stacked bar charts help visualize Robinson's choices. All the plots below can be easily generated with the ``Matplotlib`` library.

```python
sns.set_style("white")
sns.set_context("notebook", font_scale=1.5)

fig, ax = plt.subplots(figsize=(7, 5))

df.groupby("Period").Choice.value_counts(normalize=True).unstack().plot.bar(
    width=0.4, stacked=True, ax=ax
)
plt.xticks(rotation="horizontal")
plt.legend(loc="lower center", bbox_to_anchor=(0.5, -0.35), ncol=2)
plt.title("Robinson's choices by period", pad=15)

plt.show()
```

A payoff of structural models is the possibility to simulate the same model in a counterfactual environment, to help answer "what if" questions that can be highly relevant for policy purposes. As an example, in models of educational choice, the analysis of comparative statics is often applied to evaluate the effect of alternative tuition policies on average educational attainment.

We will now show how, in this simple Robinson Crusoe economy, changing one parameter affects Robinson's behavior: How do Robinson's choices change if we increase the return to experience in fishing? We can simulate the model again under a different set of parameters and inspect Robinson's new choices.

```python
return_exp = [0.2, 0.25, 0.3, 0.35]
```

```python
list_df = []
for r in return_exp:
    params.loc[("wage_fishing", "exp_fishing"), "value"] = r
    simulate = rp.get_simulate_func(params, options)
    df = simulate(params)
    list_df.append(df)
```

```python
sns.set_context("notebook", font_scale=1.25)

fig, axs = plt.subplots(2, 2, figsize=(10, 8))

width = 0.5
axs = axs.flatten()

for ax, df, r in zip(axs, list_df, return_exp):
    hammock = (
        df.groupby("Period")
        .Choice.value_counts(normalize=True)
        .unstack()
        .loc[:, "hammock"]
    )
    fishing = (
        df.groupby("Period")
        .Choice.value_counts(normalize=True)
        .unstack()
        .loc[:, "fishing"]
    )

    ax.bar(fishing.index, fishing.values, width=width, label="fishing")
    ax.bar(hammock.index, hammock.values, width=width, bottom=fishing, label="hammock")

    ax.set_title("Return to experience = " + str(r))
    plt.xticks(rotation="horizontal")

plt.legend(loc="right", bbox_to_anchor=(0.3, -0.3), ncol=2)
plt.suptitle("Robinson's choices by period")

plt.show()
```

As ``"exp_fishing"`` increases, Robinson is more and more likely to fish in each period. For higher returns to experience, Robinson spends all of his time fishing, while for low values he spends all of his time on his hammock. Varying just one parameter results in very different predicted outcomes.

***

**Footnotes**

1
    One of the earliest references of Robinsonades in Economics can be found in Marx (1867). In the 37th footnote, he mentions that even Ricardo used the theme before him.

## References

Marx, K. (1867). Das Kapital, Bd. 1. *MEW*, Bd, 23, 405