# 2021-11-19 Mixed methods

## Last time

* Community presentations
* Petrov-Galerkin stabilization
* Mass lumping for time dependent problems
* Examples of finite element interfaces

## Today

* Community presentations
* More finite element interfaces: Deal.II and MOOSE
* Vector problems and mixed finite elements

# Community presentations

## [Marcel Corchado: PFSSPY](https://github.com/dstansby/pfsspy)

## [Morgan Byers: Krylov.jl](https://github.com/cu-numpde/2021-community/files/7571148/byers_pde_community.pdf)

# Finite element interfaces: [Deal.II](https://dealii.org)

\begin{gather*}
    v^T F(u) \sim \int_\Omega v \cdot \color{olive}{f_0(u, \nabla u)} + \nabla v \!:\! \color{olive}{f_1(u, \nabla u)} \quad
    v^T J w \sim \int_\Omega \begin{bmatrix} v \\ \nabla v \end{bmatrix}^T \color{teal}{\begin{bmatrix} f_{0,0} & f_{0,1} \\ f_{1,0} & f_{1,1} \end{bmatrix}}
    \begin{bmatrix} w \\ \nabla w \end{bmatrix} \\
    J w = \sum_e \mathcal E_e^T \underbrace{\begin{bmatrix} B_I \\ B_{\nabla} \end{bmatrix}^T
    \begin{bmatrix} I & \\ & \left( \frac{\partial X}{\partial x}\right)^T \end{bmatrix}}_{\texttt{fe\_values}} W_q \color{teal}{\begin{bmatrix} f_{0,0} & f_{0,1} \\ f_{1,0} & f_{1,1} \end{bmatrix}} \underbrace{\begin{bmatrix} I & \\ & \left( \frac{\partial X}{\partial x}\right) \end{bmatrix} \begin{bmatrix} B_I \\ B_{\nabla} \end{bmatrix}}_{\texttt{fe\_values}} \mathcal E_e w_L
\end{gather*}

* Deal.II [step-7](https://dealii.org/developer/doxygen/deal.II/step_7.html)

```python
for e in elems:
    fe_values.reinit()
    for q in q_points:
        for i in test_functions:
            for j in trial_functions
                K_e[i,j] += ...
            f_e[i] += ...
    for f in e.faces:
        if f.at_boundary():
            fe_face_values.reinit()
            for q in q_points:
                ...
```

# Finite element interfaces: [MOOSE](https://mooseframework.org/)

## [Materials](https://mooseframework.inl.gov/modules/index.html)
* Can be written without knowledge of finite elements
* Registration allows libraries of materials (some in MOOSE, others packaged separatle)
* Example: [crystal plasticity](https://mooseframework.inl.gov/source/materials/crystal_plasticity/ComputeMultipleCrystalPlasticityStress.html)
* Code is C++, so can do dirty things
  * table lookups, proprietary code
  * implicit materials (Newton solve at each quadrature point)

## Composition in configuration files

* Add fields and coupling
* Select materials from libraries
* Multiphysics composition
* Multiscale coupling

## Graphical interface: [Peacock](https://mooseframework.inl.gov/application_usage/peacock.html)

# [Periodic table of finite elements](https://www-users.cse.umn.edu/~arnold/femtable/index.html)

## Exactly satisfies discrete identities like

$$ \nabla\cdot\nabla\times u = 0$$
$$ \nabla\times\nabla u = 0$$

* Generalized concept: Finite Element Exterior Calculus (FEEC)
* Improved stability or numerical properties
* Higher order of accuracy for quantity of interest despite non-smooth problem

## Problems with constraints

Stokes equation: (slow) incompressible flow

\begin{align}
-\nabla\cdot(\eta \nabla^s \mathbf u - pI) &= 0 \\
\nabla\cdot \mathbf u &= 0
\end{align}
where $\nabla^s \mathbf u$ is the symmetric part of the $3\times 3$ gradient $\nabla \mathbf u$.

Weak form: find $(\mathbf u, p)$ such that
$$ \int \nabla^s \mathbf v \!:\! \eta \!:\! \nabla^s \mathbf u - p \nabla\cdot \mathbf v - q \nabla\cdot \mathbf u = 0, \quad \forall (\mathbf v, q)$$


```python
import sys
import numpy as np
import sys
print(sys.version)

from phasor.utilities.ipynb.displays import *
from phasor.utilities.ipynb.filters import *
from phasor.utilities.ipynb.hdf import *
#from YALL.utilities.tabulate import tabulate

from declarative import (
  OverridableObject, mproperty
)

import sympy
from phasor.utilities.ipynb.displays import *
from phasor.utilities.ipynb.ipy_sympy import *
import scipy.linalg
```

    3.5.2 (default, Sep 14 2016, 11:28:32)
    [GCC 6.2.1 20160901 (Red Hat 6.2.1-1)]

```python
r,t,rbs = sympy.var('r, t, r_bs', real = True)
a,b,c,d = sympy.var('a,b,c,d')
#rbs = 0
a,b,c,d = 1,-1,-1,1
#r = sympy.sqrt(1 - t**2)
rp = sympy.var('r_p', real = True)# r * sympy.sqrt(1 - rbs**2)
tp = sympy.var('t_p', real = True)# t * sympy.sqrt(1 - rbs**2)
theta = sympy.var('theta', real = True)# t * sympy.sqrt(1 - rbs**2)
X = sympy.exp(sympy.ps_In * theta)
Xc = X.conjugate()

theta = sympy.var('phi', real = True)# t * sympy.sqrt(1 - rbs**2)
Y = sympy.exp(sympy.ps_In * phi)
Yc = Y.conjugate()

R_matInner = sympy.Matrix([
    [a*rbs,  rp,    tp,     0],
    [rp,     b*rbs, 0,        tp],
    [tp,     0,        c*rbs, -rp],
    [0,      tp,    -rp,    d*rbs],
])
Rl_matY = sympy.Matrix([
    [Yc,0,0,0],
    [0,1,0,0],
    [0,0,Yc,0],
    [0,0,0,1],
])
Rr_matY = sympy.Matrix([
    [Y,0,0,0],
    [0,1,0,0],
    [0,0,Y,0],
    [0,0,0,1],
])
Rl_matX = sympy.Matrix([
    [X,0,0,0],
    [0,1,0,0],
    [0,0,1,0],
    [0,0,0,X],
])
Rr_matX = sympy.Matrix([
    [1,0,0,0],
    [0,Xc,0,0],
    [0,0,Xc,0],
    [0,0,0,1],
])
R_mat = Rl_matX *Rl_matY * R_matInner * Rr_matY * Rr_matX
R_mat
```

```python
M = (R_mat.transpose().conjugate() * R_mat)
M = M.subs(rp, r * sympy.sqrt(1 - rbs**2))
M = M.subs(tp, t * sympy.sqrt(1 - rbs**2))
M = M.subs(r, sympy.sqrt(1 - t**2))
M.simplify()
M
```

```python
R_matInner = sympy.Matrix([
    [a*rbs,  rp,     tp,     0],
    [rp,     b*rbs, 0,         tp],
    [ tp,     0,        c*rbs, -rp],
    [0,       tp,    -rp,    d*rbs],
])
I4 = sympy.eye(4)
O4 = sympy.zeros(4)
Lt  = sympy.var('Lt', real = True)
Lr  = sympy.var('Lr', real = True)
Lx  = sympy.var('Lx', real = True)
rL = sympy.var('r_L', real = True)
tL = sympy.var('t_L', real = True)
def enlarge(mat):
    return sympy.Matrix(numpy.bmat([[mat, O4],[O4, I4]]))
R_matBig = sympy.Matrix(numpy.bmat([[R_matInner, Lx*I4],[-Lx*I4, R_matInner]]))

R_matBig
R_mat = enlarge(Rl_matX) * enlarge(Rl_matY) * R_matBig * enlarge(Rr_matY) * enlarge(Rr_matX)
M = R_mat
#R_mat = R_matBig
M = M.subs(Lx, sympy.sqrt(Lt**2 + Lr**2))
M = M.subs(tp, sympy.sqrt(t**2 - Lt**2))
M = M.subs(rp, sympy.sqrt(r**2 - Lr**2 - rbs**2))
M = M.subs(r, sympy.sqrt(1 - t**2))
M
```

```python
M = (R_mat.transpose().conjugate() * R_mat)
M = M.subs(Lx, sympy.sqrt(Lt**2 + Lr**2))
M = M.subs(tp, sympy.sqrt(t**2 - Lt**2))
M = M.subs(rp, sympy.sqrt(r**2 - Lr**2 - rbs**2))
M = M.subs(r, sympy.sqrt(1 - t**2))
#M = M.expand()
M.simplify()
M
```

```python

```


# Generative models - auto-encoders

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A brief introduction to [generative models](#generative)
2. A formal presentation of [Auto-Encoders](#ae) (AEs)
3. An explanation of how to [implement AEs](#implement)
4. An [application](#application) of AEs for modeling images
4. An practical exemple of [convolutional denoising AEs](#denoising) for image data **(exercise)**

## Generative models

### Supervised refresher

Until now, we have mostly discussed models that are developed for _supervised_ and _discriminative_ tasks. To formalize this problem, we have a set of data $\{\mathbf{x}_{i}, \mathbf{y}_{i}\}_{i\in[1,n]}$, where the $\mathbf{x}_{i}$ and $\mathbf{y}_{i}$ are linked. Therefore, we want to approximate this relation through
\begin{equation}
    \mathbf{\hat{y}} = \mathcal{F}_\mathbf{\theta}(\mathbf{x})
\end{equation}
where we train the parameters $\mathbf{\theta}$ so that $\mathbf{\hat{y}}\approx\mathbf{y}$. The existence of a label $\mathbf{y}$ ("correct answer") defines a _supervised_ problem

### Going unsupervised

In some cases, we might only have a set of data $\{\mathbf{x}_{i}\}_{i\in[1,n]}$, and still be interested in learning some underlying properties or structure of this set. In that case, the problem is _unsupervised_ as we have to learn without a given answer.

Here, we can turn to _generative_ models [[1](#reference1)], which allows to create new data instances based on the observation of existing examples. Although these models are more naturally defined in a _probabilistic way_, we will assume that we have some simple _code_ $\mathbf{z}$, which allows to control the properties of the generation, and need to learn
\begin{equation}
    \mathbf{\hat{x}} = \mathcal{F}_\mathbf{\theta}(\mathbf{z})
\end{equation}
where we still need to learn $\mathbf{\theta}$, so that $\mathbf{\hat{x}}$ have similar properties to that of the examples in $\{\mathbf{x}_{i}\}_{i\in[1,n]}$.

Now the problem to solve is how we can learn directly from a set of data.

## Auto-encoders

One way to understand a set of data is to try to _compress_, or _simplify_ the corresponding dataset. So the idea is to learn simultaneously how to _encode_ our unlabeled input $\{\mathbf{x}_{i}\}_{i\in[1,n]}$ and to _decode_ the corresponding representation. This idea give rise to the notion of **auto-encoder**.

### Architecture

The auto-encoder is an unsupervised architecture originally proposed to perform _dimensionality reduction_ [[3](#reference3)]. As its name indicates, we will try to train this model to learn an efficient _encoding_ $\mathbf{z}$ of unlabeled input data $\mathbf{x}$. The only way to learn efficient parameters is to also learn a _decoding_ function to _reconstruct_ $\mathbf{x}$ from $\mathbf{z}$.

As shown here, a first model $\mathcal{E}_\phi$ _encodes_ the input into a _latent code_ $\mathbf{z}$ in order to provide a low-dimensional representation of the data. A second model $\mathcal{D}_\theta$ designated as the _decoder_ aims to generate outputs from $\mathbf{z}$ that are as close to the original inputs as possible.

### Formal definition

The latent code $\mathbf{z}$ can be seen as a compressed abstract representation, and may be used as an intermediate space for analysis or generation. This helps to govern the distribution of the data through a simpler and higher-level representation, while enhancing the _expressiveness_ of the generative model.
The behaviour of an auto-encoder can be formalized as:

$$
\begin{align}
\mathbf{z} &= \mathcal{E}_\phi(\mathbf{x}) \\
\mathbf{\hat{x}} &= \mathcal{D}_\theta(\mathbf{z})
\end{align}
$$

with the _encoder_ $\mathcal{E}_\phi$ and _decoder_ $\mathcal{D}_\theta$ functions parameterized respectively by $\phi$ and $\theta$. As we can see this defines the reconstruction relationship
$$
    \mathbf{\hat{x}} = \mathcal{D}_\theta(\mathcal{E}_\phi(\mathbf{x}))
$$

### Training

The training of an auto-encoder consists in finding the optimal functions of encoding $\mathcal{E}^*$ and decoding $\mathcal{D}^*$ by evaluating the \textit{reconstruction error} $\mathcal{L}$ between $\mathbf{x}$ and $\mathbf{\hat{x}}$, such that
\begin{equation}
    \mathcal{E}^*, \mathcal{D}^* = arg\,min_{ \phi, \theta}{\mathcal{L}}(\mathbf{x}, \mathcal{D}_\theta(\mathcal{E}_\phi(\mathbf{x})))
\end{equation}

As the latent space usually has a smaller dimensionality than the input, it acts as an incentive for the network to find the main attributes of variations in the dataset (and also explains its use for _dimensionality reduction_).

### Variants and discussion

There are several variants of auto-encoders, such as denoising auto-encoders or variational auto-encoders. Each address some downside of the basic AE model. For instance, the deterministic nature of the basic auto-encoder implies a point-wise mapping of the latent space, meaning that not all the latent positions can be leveraged to produce relevant reconstructions. Because of this reason, there is no way to ensure that the latent space could allow a robust generalization and that any random $\mathbf{z}$ would generate a meaningful output.

## Implementation

Here, we discuss how we can implement and train a simple auto-encoder network in Pytorch. As discussed earlier, an AE is composed of two parts, an **encoder** and a **decoder**. The goal of the encoder is to "compress" the dataset, representing its principal features with a very small code, while the goal of the decoder is to learn how to reproduce the initial input from this code. Hence, we will first need to use some basic imports and definition to setup our problem

### Import Pytorch and other libraries

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torchvision
from helper_plot import hdr_plot_style
hdr_plot_style();
```

### Load the dataset

To start with a pragmatic and simple to understand example, we  will try to train the basic AE using the Fashion MNIST dataset. This dataset contains images of size 28x28 pixels, with different pieces of clothing. The following code allows to load (and eventually download) the dataset, by using the `torchvision.datasets` module. We also plot some randomly selected test examples

```python
import matplotlib
dataset_dir = './data'
# Going to use 80%/20% split for train/valid
valid_ratio = 0.2
# Load the dataset for the training/validation sets
train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.ToTensor(), download=True)
# Split it into training and validation sets
nb_train = int((1.0 - valid_ratio) * len(train_valid_dataset))
nb_valid =  int(valid_ratio * len(train_valid_dataset))
train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])
# Load the test set
test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.ToTensor(),train=False)
# Prepare
num_threads = 4     # Loading the dataset is using 4 CPU threads
batch_size  = 128   # Using minibatches of 128 samples
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)
valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)
```

Here, we are going to check the properties of our different sets and also plot some random examples, in order to better understand what type of data we will be working with.

```python
print("The train set contains {} images, in {} batches".format(len(train_loader.dataset), len(train_loader)))
print("The validation set contains {} images, in {} batches".format(len(valid_loader.dataset), len(valid_loader)))
print("The test set contains {} images, in {} batches".format(len(test_loader.dataset), len(test_loader)))
nsamples = 10
classes_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal','Shirt', 'Sneaker', 'Bag', 'Ankle boot']
imgs_test, labels = next(iter(train_loader))
fig = plt.figure(figsize=(20,5))
for i in range(nsamples):
    ax = plt.subplot(1,nsamples, i+1)
    plt.imshow(imgs_test[i, 0, :, :], vmin=0, vmax=1.0, cmap=matplotlib.cm.gray)
    ax.set_title("{}".format(classes_names[labels[i]]), fontsize=15)
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
```

## Basic auto-encoder

We recall here that in order to define an autoencoder, we will need an `encoder`, which compresses the images into a small latent vector, and a `decoder`, that reconstructs the original image from this code. Here, we will start very basic and define the encoder and decoder as simple `Dense` layers. To define the model simply, we will use the [Keras API](https://www.tensorflow.org/guide/keras/custom_layers_and_models) defined in the `tensorflow.keras` module (Note that we pre-loaded the `layers` submodule

```python
class AE(nn.Module):
    def __init__(self, encoding_dim):
        super(AE, self).__init__()
        self.latent_dim = encoding_dim
        self.hidden_dim = 256
        self.encoder = nn.Sequential(
          nn.Linear(28 * 28, self.hidden_dim), nn.ReLU(),
          nn.Linear(self.hidden_dim, self.hidden_dim), nn.ReLU(),
          nn.Linear(self.hidden_dim, self.latent_dim), nn.Sigmoid()
        )
        self.decoder = nn.Sequential(
          nn.Linear(self.latent_dim, self.hidden_dim), nn.ReLU(),
          nn.Linear(self.hidden_dim, self.hidden_dim), nn.ReLU(),
          nn.Linear(self.hidden_dim, 28 * 28), nn.Sigmoid()
        )

    def forward(self, x):
        encoded = self.encoder(x.reshape(-1, 28 * 28))
        decoded = self.decoder(encoded)
        return decoded.reshape(-1, 1, 28, 28)
```

Here we can see that the model depends on a given `encoding_dim` variable, which defines the size of the latent code. Therefore, we can instantiate our model arbitrarliy with `64` dimensions

```python
latent_dim = 64
model = AE(latent_dim)
```

The only remaining part that we did not discuss yet is what type of _loss_ (defined as $\mathcal{L}$) we can use to train our model. First, we will simply rely on the _Mean Squared Error_ (MSE) loss, which is defined as
\begin{equation}
    \mathcal{L}_{MSE}(\hat{\mathbf{x}}, \mathbf{x}) = \mid \hat{\mathbf{x}}, \mathbf{x} \mid^{2}
\end{equation}

```python
# Loss function that we will use
loss_fn = torch.nn.MSELoss(reduction='sum')
```

### Training the model

Train the model using `x_train` as both the input and the target. The `encoder` will learn to compress the dataset from 784 dimensions to the latent space, and the `decoder` will learn to reconstruct the original images.
.

```python
# Learning rate
learning_rate = 1e-4
# Optimizer to fit the weights of the network
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
for t in range(20):
    full_loss = torch.Tensor([0])
    # Forward pass: compute predicted y by passing x to the model.
    for i, (x, _) in enumerate(train_loader):
        y_pred = model(x)
        # Compute the loss.
        loss = loss_fn(y_pred, x)
        # Before the backward pass, zero all of the network gradients
        optimizer.zero_grad()
        # Backward pass: compute gradient of the loss with respect to parameters
        loss.backward()
        # Calling the step function to update the parameters
        optimizer.step()
    print(loss)
```

    tensor(5817.4189, grad_fn=<MseLossBackward>)
    tensor(4374.0566, grad_fn=<MseLossBackward>)
    tensor(4219.4839, grad_fn=<MseLossBackward>)
    tensor(4174.4062, grad_fn=<MseLossBackward>)
    tensor(3791.4062, grad_fn=<MseLossBackward>)
    tensor(3565.2781, grad_fn=<MseLossBackward>)
    tensor(3286.8918, grad_fn=<MseLossBackward>)
    tensor(3004.1448, grad_fn=<MseLossBackward>)
    tensor(2834.7437, grad_fn=<MseLossBackward>)
    tensor(2691.8218, grad_fn=<MseLossBackward>)
    tensor(2746.5903, grad_fn=<MseLossBackward>)
    tensor(2284.3418, grad_fn=<MseLossBackward>)
    tensor(2400.9062, grad_fn=<MseLossBackward>)
    tensor(2347.1150, grad_fn=<MseLossBackward>)
    tensor(2389.3455, grad_fn=<MseLossBackward>)
    tensor(2055.0049, grad_fn=<MseLossBackward>)
    tensor(2046.2859, grad_fn=<MseLossBackward>)
    tensor(2127.6699, grad_fn=<MseLossBackward>)
    tensor(2059.9185, grad_fn=<MseLossBackward>)
    tensor(1946.4250, grad_fn=<MseLossBackward>)

Now that the model is trained, we can test it by encoding and decoding images from the test set.

```python
encoded_imgs = model.encoder(imgs_test.reshape(-1,28*28))
decoded_imgs = model.decoder(encoded_imgs).reshape(-1,28,28).detach().numpy()
```

By plotting the images, we can see that the model is able to perform an adequate (yet somewhat blurry) reconstruction of the input images. The interesting point is that this reconstruction comes from a code of only `64` dimensions, whereas the original images have `784` dimensions.

```python
n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(imgs_test[i, 0])
    plt.title("original"); plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    # display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i])
    plt.title("reconstructed"); plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()
```

### Improvements

Even though this very basic example seems to work, several improvements can be made over the original model. First, we can see that the overall framework does not depend on the exact nature of the `encoder` and `decoder`.

***
1. We rewrite the original class to accept any type of architecture for these (see code below).
2. Fill the missing code to have a new AE model
***

```python
class AE(nn.Module):
    def __init__(self, encoder, decoder, encoding_dim):
        super(AE, self).__init__()
        self.latent_dim = encoding_dim
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

######################
# YOUR CODE GOES HERE
######################

encoder = ...
decoder = ...
model = ...
```

Second, we only used the train dataset, whereas the use of a validation and test sets allows to ensure that we do not overfit our model.

***
1. Write a separate train function, that can be called on different sets
2. Re-write the training loop to test overfitting
***

```python
######################
# YOUR CODE GOES HERE
######################
```

## Exercise: Denoising AE

Imagine (for the sake of argument), that we choose an encoding dimension which is of same dimensionality as the input one. Then, one huge problem is that nothing prevents the AE from simply learning the _identity_ function (try to imagine why). An autoencoder can also be trained to remove noise from images. This type of _regularization_ prevents the model from learning this degenerate situation.

In this exercise, you will need to create your own denoising AE, by relying on a noisy version of the Fashion MNIST dataset (adding random Gaussian noise to each image). You will then train an autoencoder using the noisy image as input, and the original image as the target.

Let's reimport the dataset to omit the modifications made earlier.

```python
class GaussianNoise(object):
    def __init__(self, mean=0., std=1.):
        self.std = std
        self.mean = mean

    def __call__(self, tensor):
        return tensor + torch.randn(tensor.size()) * self.std + self.mean

    def __repr__(self):
        return self.__class__.__name__ + '(mean={0}, std={1})'.format(self.mean, self.std)
```

Here, we create two new train and test sets by adding random noise to the images

```python
# Load the datasets and use our Gaussian noise transform
train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(), GaussianNoise(0., 0.1)]), download=True)
train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])
test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(), GaussianNoise(0., 0.1)]),train=False)
# Create loaders
train_loader_noisy = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)
valid_loader_noisy = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)
test_loader_noisy = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)
```

Plot the noisy images.

```python
x_test, labels = next(iter(test_loader))
x_test_noisy, labels = next(iter(test_loader_noisy))
n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    ax = plt.subplot(2, n, i + 1)
    plt.title("original")
    plt.imshow(x_test[i].squeeze())
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    plt.gray()
    ax = plt.subplot(2, n, i + 1 + n)
    plt.title("original + noise")
    plt.imshow(x_test_noisy[i].squeeze())
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    plt.gray()
plt.show()
```

### Define a convolutional autoencoder

In this example, you will train a convolutional autoencoder using  [Conv2D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) layers in the `encoder`, and [Conv2DTranspose](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2DTranspose) layers in the `decoder`.

```python
######################
# YOUR CODE GOES HERE
######################

class DenoisingAE(AE):
    def __init__(self):
        super(DenoisingAE, self).__init__()
        self.encoder = ...
        self.decoder = ...

    def forward(self, x):
        encoded = ...
        decoded = ...
        return decoded

autoencoder = ...
```

Write down the full optimization loop to optimize your denoising auto-encoder

```python
######################
# YOUR CODE GOES HERE
######################
```

Plot both the noisy images and the denoised images produced by the autoencoder to check that your implementation is correct

```python
######################
# YOUR CODE GOES HERE
######################
encoded_imgs = ...
decoded_imgs = ...
plt.figure(figsize=(20, 4))
for i in range(n):
    ...
plt.show()
```

### References

[1] Rezende, Danilo Jimenez, and Shakir Mohamed. "Variational inference with normalizing flows." _arXiv preprint arXiv:1505.05770_ (2015). [link](http://arxiv.org/pdf/1505.05770)

[2] Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving Variational Inference with Inverse Autoregressive Flow." _arXiv preprint arXiv:1606.04934_ (2016). [link](https://arxiv.org/abs/1606.04934)

[3] Kingma, D. P., & Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. (2013). [link](https://arxiv.org/pdf/1312.6114)

[4] Rezende, D. J., Mohamed, S., & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082. (2014). [link](https://arxiv.org/pdf/1401.4082)

[5] Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., & Courville, A. (2016). Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05013. [link](https://arxiv.org/pdf/1611.05013)

[6] Van den Oord, A., & Vinyals, O. (2017). Neural discrete representation learning. In NIPS 2017 (pp. 6306-6315). [link](http://papers.nips.cc/paper/7210-neural-discrete-representation-learning.pdf)


#  Neuromatch Academy: Week 1, Day 1, Tutorial 2
# Model Types: "How" models
__Content creators:__ Matt Laporte, Byron Galbraith, Konrad Kording

__Content reviewers:__ Dalin Guo, Aishwarya Balwani, Madineh Sarvestani, Maryam Vaziri-Pashkam, Michael Waskom

___
# Tutorial Objectives
This is tutorial 2 of a 3-part series on different flavors of models used to understand neural data. In this tutorial we will explore models that can potentially explain *how* the spiking data we have observed is produced

To understand the mechanisms that give rise to the neural data we save in Tutorial 1,  we will build simple neuronal models and compare their spiking response to real data. We will:
- Write code to simulate a  simple "leaky integrate-and-fire" neuron model
- Make the model more complicated — but also more realistic — by adding more physiologically-inspired details

```python
#@title Video 1: "How" models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='PpnagITsb3E', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=PpnagITsb3E

# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets #interactive display

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")

```

```python
#@title Helper Functions
def histogram(counts, bins, vlines=(), ax=None, ax_args=None, **kwargs):
  """Plot a step histogram given counts over bins."""
  if ax is None:
    _, ax = plt.subplots()

  # duplicate the first element of `counts` to match bin edges
  counts = np.insert(counts, 0, counts[0])

  ax.fill_between(bins, counts, step="pre", alpha=0.4, **kwargs)  # area shading
  ax.plot(bins, counts, drawstyle="steps", **kwargs)  # lines

  for x in vlines:
    ax.axvline(x, color='r', linestyle='dotted')  # vertical line

  if ax_args is None:
    ax_args = {}

  # heuristically set max y to leave a bit of room
  ymin, ymax = ax_args.get('ylim', [None, None])
  if ymax is None:
    ymax = np.max(counts)
    if ax_args.get('yscale', 'linear') == 'log':
      ymax *= 1.5
    else:
      ymax *= 1.1
      if ymin is None:
        ymin = 0

  if ymax == ymin:
    ymax = None

  ax_args['ylim'] = [ymin, ymax]

  ax.set(**ax_args)
  ax.autoscale(enable=False, axis='x', tight=True)

def plot_neuron_stats(v, spike_times):
  fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))

  # membrane voltage trace
  ax1.plot(v[0:100])
  ax1.set(xlabel='Time', ylabel='Voltage')
  # plot spike events
  for x in spike_times:
    if x >= 100:
      break
    ax1.axvline(x, color='red')

  # ISI distribution
  isi = np.diff(spike_times)
  n_bins = np.arange(isi.min(), isi.max() + 2) - .5
  counts, bins = np.histogram(isi, n_bins)
  vlines = []
  if len(isi) > 0:
    vlines = [np.mean(isi)]
  xmax = max(20, int(bins[-1])+5)
  histogram(counts, bins, vlines=vlines, ax=ax2, ax_args={
    'xlabel': 'Inter-spike interval',
    'ylabel': 'Number of intervals',
    'xlim': [0, xmax]
  })
  plt.show()
```

# Section 1: The Linear Integrate-and-Fire Neuron

How does a neuron spike?

A neuron charges and discharges an electric field across its cell membrane. The state of this electric field can be described by the _membrane potential_. The membrane potential rises due to excitation of the neuron, and when it reaches a threshold a spike occurs. The potential resets, and must rise to a threshold again before the next spike occurs.

One of the simplest models of spiking neuron behavior is the linear integrate-and-fire model neuron. In this model, the neuron increases its membrane potential $V_m$ over time in response to excitatory input currents $I$ scaled by some factor $\alpha$:

\begin{align}
  dV_m = {\alpha}I
\end{align}

Once $V_m$ reaches a threshold value a spike is produced, $V_m$ is reset to a starting value, and the process continues.

Here, we will take the starting and threshold potentials as $0$ and $1$, respectively. So, for example, if $\alpha I=0.1$ is constant---that is, the input current is constant---then $dV_m=0.1$, and at each timestep the membrane potential $V_m$ increases by $0.1$ until after $(1-0)/0.1 = 10$ timesteps it reaches the threshold and resets to $V_m=0$, and so on.

Note that we define the membrane potential $V_m$ as a scalar: a single real (or floating point) number. However, a biological neuron's membrane potential will not be exactly constant at all points on its cell membrane at a given time. We could capture this variation with a more complex model (e.g. with more numbers). Do we need to?

The proposed model is a 1D simplification. There are many details we could add to it, to preserve different parts of the complex structure and dynamics of a real neuron. If we were interested in small or local changes in the membrane potential, our 1D simplification could be a problem. However, we'll assume an idealized "point" neuron model for our current purpose.

#### Spiking Inputs

Given our simplified model for the neuron dynamics, we still need to consider what form the input $I$ will take. How should we specify the firing behavior of the presynaptic neuron(s) providing the inputs to our model neuron?

Unlike in the simple example above, where $\alpha I=0.1$, the input current is generally not constant. Physical inputs tend to vary with time. We can describe this variation with a distribution.

We'll assume the input current $I$ over a timestep is due to equal contributions from a non-negative ($\ge 0$) integer number of input spikes arriving in that timestep. Our model neuron might integrate currents from 3 input spikes in one timestep, and 7 spikes in the next timestep. We should see similar behavior when sampling from our distribution.

Given no other information about the input neurons, we will also assume that the distribution has a mean (i.e. mean rate, or number of spikes received per timestep), and that the spiking events of the input neuron(s) are independent in time. Are these reasonable assumptions in the context of real neurons?

A suitable distribution given these assumptions is the Poisson distribution, which we'll use to model $I$:

\begin{align}
  I \sim \mathrm{Poisson}(\lambda)
\end{align}

where $\lambda$ is the mean of the distribution: the average rate of spikes received per timestep.

### Exercise 1: Compute $dV_m$

For your first exercise, you will write the code to compute the change in voltage $dV_m$ (per timestep) of the linear integrate-and-fire model neuron. The rest of the code to handle numerical integration is provided for you, so you just need to fill in a definition for `dv` in the `lif_neuron` function below. The value of $\lambda$ for the Poisson random variable is given by the function argument `rate`.

The [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html) package is a great resource for working with and sampling from various probability distributions. We will use the `scipy.stats.poisson` class and its method `rvs` to produce Poisson-distributed random samples. In this tutorial, we have imported this package with the alias `stats`, so you should refer to it in your code as `stats.poisson`.

```python
def lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  """ Simulate a linear integrate-and-fire neuron.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    rate (int): The mean rate of incoming spikes

  """
  # precompute Poisson samples for speed
  exc = stats.poisson(rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []

  for i in range(1, n_steps):

    dv = alpha * exc[i]

    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Uncomment these lines after completing the lif_neuron function
v, spike_times = lif_neuron()
plot_neuron_stats(v, spike_times)
```

```python
def llif_neuron(n_steps=1000, alpha=0.01, rate=10):
  """ Simulate a *leaky* linear integrate-and-fire neuron.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    rate (int): The mean rate of incoming spikes

  """
  # precompute Poisson samples for speed
  exc = stats.poisson(rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []

  for i in range(1, n_steps):

    # what if we add a leak term?
    dv = alpha * exc[i] - 0.1 * v[i-1]

    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Uncomment these lines after completing the lif_neuron function
v, spike_times = llif_neuron()
plot_neuron_stats(v, spike_times)
```

## Interactive Demo: Linear-IF neuron
Like last time, you can now explore how various parametes of the LIF model influence the ISI distribution.

```python
#@title

#@markdown You don't need to worry about how the code works – but you do need to **run the cell** to enable the sliders.

def _lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  exc = stats.poisson(rate).rvs(n_steps)
  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = alpha * exc[i]
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0
  return v, spike_times

@widgets.interact(
  n_steps=widgets.FloatLogSlider(1000.0, min=2, max=4),
  alpha=widgets.FloatLogSlider(0.01, min=-2, max=-1),
  rate=widgets.IntSlider(10, min=5, max=20)
)
def plot_lif_neuron(n_steps=1000, alpha=0.01, rate=10):
  v, spike_times = _lif_neuron(int(n_steps), alpha, rate)
  plot_neuron_stats(v, spike_times)
```

    interactive(children=(FloatLogSlider(value=1000.0, description='n_steps', min=2.0), FloatLogSlider(value=0.01,…

```python
#@title Video 2: Linear-IF models
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='QBD7kulhg4U', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=QBD7kulhg4U

# Section 2: Inhibitory signals

Our linear integrate-and-fire neuron from the previous section was indeed able to produce spikes. However, our ISI histogram doesn't look much like empirical ISI histograms seen in Tutorial 1, which had an exponential-like shape. What is our model neuron missing, given that it doesn't behave like a real neuron?

In the previous model we only considered excitatory behavior -- the only way the membrane potential could decrease was upon a spike event. We know, however, that there are other factors that can drive $V_m$ down. First is the natural tendency of the neuron to return to some steady state or resting potential. We can update our previous model as follows:

\begin{align}
  dV_m = -{\beta}V_m + {\alpha}I
\end{align}

where $V_m$ is the current membrane potential and $\beta$ is some leakage factor. This is a basic form of the popular Leaky Integrate-and-Fire model neuron (for a more detailed discussion of the LIF Neuron, see the Appendix).

We also know that in addition to excitatory presynaptic neurons, we can have inhibitory presynaptic neurons as well. We can model these inhibitory neurons with another Poisson random variable:

\begin{align}
I = I_{exc} - I_{inh} \\
I_{exc} \sim \mathrm{Poisson}(\lambda_{exc}) \\
I_{inh} \sim \mathrm{Poisson}(\lambda_{inh})
\end{align}

where $\lambda_{exc}$ and $\lambda_{inh}$ are the average spike rates (per timestep) of the excitatory and inhibitory presynaptic neurons, respectively.

### Exercise 2: Compute $dV_m$ with inhibitory signals

For your second exercise, you will again write the code to compute the change in voltage $dV_m$, though now of the LIF model neuron described above. Like last time, the rest of the code needed to handle the neuron dynamics are provided for you, so you just need to fill in a definition for `dv` below.

```python
def lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """

  # precompute Poisson samples for speed
  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []

  for i in range(1, n_steps):

    dv = -beta * v[i-1] + alpha * exc[i] - alpha * inh[i]

    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

# Uncomment these lines do make the plot once you've completed the function
v, spike_times = lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=5, inh_rate=5)
plot_neuron_stats(v, spike_times)

```

## Interactive Demo: LIF + inhibition neuron

```python
#@title
#@markdown **Run the cell** to enable the sliders.
def _lif_neuron_inh(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  """ Simulate a simplified leaky integrate-and-fire neuron with both excitatory
  and inhibitory inputs.

  Args:
    n_steps (int): The number of time steps to simulate the neuron's activity.
    alpha (float): The input scaling factor
    beta (float): The membrane potential leakage factor
    exc_rate (int): The mean rate of the incoming excitatory spikes
    inh_rate (int): The mean rate of the incoming inhibitory spikes
  """
  # precompute Poisson samples for speed
  exc = stats.poisson(exc_rate).rvs(n_steps)
  inh = stats.poisson(inh_rate).rvs(n_steps)

  v = np.zeros(n_steps)
  spike_times = []
  for i in range(1, n_steps):
    dv = -beta * v[i-1] + alpha * (exc[i] - inh[i])
    v[i] = v[i-1] + dv
    if v[i] > 1:
      spike_times.append(i)
      v[i] = 0

  return v, spike_times

@widgets.interact(n_steps=widgets.FloatLogSlider(1000.0, min=2.5, max=4),
                  alpha=widgets.FloatLogSlider(0.5, min=-1, max=1),
                  beta=widgets.FloatLogSlider(0.1, min=-1, max=0),
                  exc_rate=widgets.IntSlider(12, min=10, max=20),
                  inh_rate=widgets.IntSlider(12, min=10, max=20))
def plot_lif_neuron(n_steps=1000, alpha=0.5, beta=0.1, exc_rate=10, inh_rate=10):
  v, spike_times = _lif_neuron_inh(int(n_steps), alpha, beta, exc_rate, inh_rate)
  plot_neuron_stats(v, spike_times)
```

    interactive(children=(FloatLogSlider(value=1000.0, description='n_steps', min=2.5), FloatLogSlider(value=0.5, …

```python
#@title Video 3: LIF + inhibition
from IPython.display import YouTubeVideo
video = YouTubeVideo(id='Aq7JrxRkn2w', width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=Aq7JrxRkn2w

#Summary

In this tutorial we gained some intuition for the mechanisms that produce the observed behavior in our real neural data. First, we built a simple neuron model with excitatory input and saw that it's behavior, measured using the ISI distribution, did not match our real neurons. We then improved our model by adding leakiness and inhibitory input. The behavior of this balanced model was much closer to the real neural data.

# Bonus

### Why do neurons spike?

A neuron stores energy in an electric field across its cell membrane, by controlling the distribution of charges (ions) on either side of the membrane. This energy is rapidly discharged to generate a spike when the field potential (or membrane potential) crosses a threshold. The membrane potential may be driven toward or away from this threshold, depending on inputs from other neurons: excitatory or inhibitory, respectively. The membrane potential tends to revert to a resting potential, for example due to the leakage of ions across the membrane, so that reaching the spiking threshold depends not only on the amount of input ever received following the last spike, but also the timing of the inputs.

The storage of energy by maintaining a field potential across an insulating membrane can be modeled by a capacitor. The leakage of charge across the membrane can be modeled by a resistor. This is the basis for the leaky integrate-and-fire neuron model.

### The LIF Model Neuron

The full equation for the LIF neuron is

\begin{align}
C_{m}\frac{dV_m}{dt} = -(V_m - V_{rest})/R_{m} + I
\end{align}

where $C_m$ is the membrane capacitance, $R_M$ is the membrane resistance, $V_{rest}$ is the resting potential, and $I$ is some input current (from other neurons, an electrode, ...).

In our above examples we set many of these parameters to convenient values ($C_m = R_m = dt = 1$, $V_{rest} = 0$) to focus more on the general behavior of the model. However, these too can be manipulated to achieve different dynamics, or to ensure the dimensions of the problem are preserved between simulation units and experimental units (e.g. with $V_m$ given in millivolts, $R_m$ in megaohms, $t$ in milliseconds).


# A Brief Introduction to Clustering and Community Detection Methods
#### Author: [Erika Fille Legara](https://erikalegara.site)

[](https://github.com/eflegara/Network-Science-Lectures/blob/master/LICENSE.md)

For context, this notebook is accompanied by a lecture session under the Network Science course under [AIM's MSc in Data Science program](https://aim.edu/programs/degree-programs/master-science-data-science).

---

    Run in Google Colab

    View source on GitHub

---

## Description
Real-world networks reveal clustering behaviour, which is exhibited in the formation of communities/clusters/partitions (terms are used interchangeably in this notebook) in the graph structure. When studying networks, their structure and function, it is crucial to identify these bunchings. In this recipe, we explore these different structures (including **cliques** and **connected components**) with a strong focus on clustering and community detection. In particular, we explore two general methods, and three submethods:

1. Divisive Method
    - Girvan-Newman algorithm using _edge betweenness_ (**Section 3**)
2. Agglomerative Method
    - Ravasz (hierarchical clustering) algorithm using _distance matrix_ (**Section 4**)
    - Louvain method using _modularity_ (**Section 5**)

Note: I originally wrote this notebook for my lecture session on Community Structures at the [2016 NTU Winter School on Complex Systems](http://www.complexity.ntu.edu.sg/Programmes/SchoolsCourses/Pages/2016-Winter-School.aspx).

```python
try:
    ## For Network Analysis and Visualization
    import networkx as nx
    import numpy as np
    import matplotlib.pyplot as plt
    from collections import defaultdict
    import operator
    import warnings
    import pandas as pd

    ## For Hierarchical Clustering
    from scipy.cluster import hierarchy
    from scipy.spatial import distance

    ## For Community Detection (Louvain Method)
    import community

except:
    import traceback
    traceback.print_exc()
    raise ImportError('Something failed, see above.')

warnings.filterwarnings("ignore")
%matplotlib inline
```

## 1. Load the Network
Here, we use the Zachary's karate club network $K$ to illustrate some concepts on network structures. This network was first presented in [1].

[1] W. W. Zachary, An information flow model for conflict and fission in small groups, Journal of Anthropological Research 33, 452-473 (1977).

```python
K = nx.karate_club_graph()
```

## 2. Network Structure and Properties

1. Clique
2. Connected Components
3. Edge Betweenness

```python
print ("Number of nodes: ", K.size())
print ("Number of edges: ", K.order())
```

    Number of nodes:  78
    Number of edges:  34

```python
pos = nx.fruchterman_reingold_layout(K);
```

```python
plt.figure(figsize=(8,8));
plt.axis("off");
nx.draw_networkx_nodes(K, pos, node_size=300, node_color="black");
nx.draw_networkx_edges(K, pos, alpha=0.500);
nx.draw_networkx_labels(K, pos, font_color="white");
plt.show();
```

### Real-world partition

```python
mrhi = [0,1,2,3,4,5,6,7,8,10,11,12,13,16,17,19,21]
johna = [9,14,15,18,20,22,23,24,25,26,27,28,29,30,31,32,33]

for mem in K.nodes():
    if mem in mrhi:
        K.nodes()[mem]["group"] = "mrhi"
    else:
        K.nodes()[mem]["group"] = "johna"
```

```python
plt.figure(figsize=(8,8));
plt.axis("off");
nx.draw_networkx_nodes(K,pos,
                       nodelist=mrhi,
                       node_color='r',
                       node_size=300,
                   alpha=0.8);
nx.draw_networkx_nodes(K,pos,
                       nodelist=johna,
                       node_color='b',
                       node_size=300,
                   alpha=0.8);
nx.draw_networkx_edges(K, pos, alpha=0.5);
nx.draw_networkx_labels(K, pos, font_color="white");
```

### Cliques
Cliques are **complete subgraphs**; that is, subgraphs where the nodes are all _directly_ connected to each other. For example, using the Karate Club network $K$, the cliques are listed below. Note that any two connected nodes in an undirected network are considered to form a **clique**.

```python
print (list(nx.find_cliques(K)))
```

    [[0, 1, 17], [0, 1, 2, 3, 13], [0, 1, 2, 3, 7], [0, 1, 19], [0, 1, 21], [0, 4, 10], [0, 4, 6], [0, 5, 10], [0, 5, 6], [0, 8, 2], [0, 11], [0, 12, 3], [0, 31], [1, 30], [2, 32, 8], [2, 9], [2, 27], [2, 28], [5, 16, 6], [33, 32, 8, 30], [33, 32, 14], [33, 32, 15], [33, 32, 18], [33, 32, 20], [33, 32, 22], [33, 32, 23, 29], [33, 32, 31], [33, 9], [33, 13], [33, 19], [33, 26, 29], [33, 27, 23], [33, 28, 31], [24, 25, 31], [24, 27], [25, 23]]

## 3. Divisive Method: The Girvan-Newman Algorithm
### Edge Betweenness
The **betweenness** of an edge $e$ is the fraction of shortest-paths that course through it.

$$c_B(e)=\sum _{i,j \in V}\frac{\sigma (i,j)|e}{\sigma (i,j)},$$

where $\sigma(i,j)$ is the total number of shortest paths and $\sigma (i,j)|e$ is the number of shortest paths that pass through edge $e$. In `networkx`, the function is `edge_betweenness_centrality(G)`. Below, we illustrate the function using Zachary's karate club $K$.

```python
#order the dictionary by value, which is the edge betweeness of two nodes
ebet = nx.edge_betweenness_centrality(K)
sorted_ebet = sorted(ebet.items(), key=operator.itemgetter(1), reverse=True)
sorted_ebet[0:5]
```

    [((0, 31), 0.1272599949070537),
     ((0, 6), 0.07813428401663695),
     ((0, 5), 0.07813428401663694),
     ((0, 2), 0.0777876807288572),
     ((0, 8), 0.07423959482783014)]

Iteratively remove edges with the highest edge betweenness. For purpose of illustration, we first make a copy of the karate club network $K$. We then perform $i=10$ iterations of edge removal.

```python
K2 = K.copy()

def remove_top_ebet(K2):
    ebet = nx.edge_betweenness_centrality(K2)
    sorted_ebet = sorted(ebet.items(), key=operator.itemgetter(1), reverse=True)
    edge_to_remove = sorted_ebet[0]
    K2.remove_edge(*edge_to_remove[0])

    return K2

for i in range(10):
    K2=remove_top_ebet(K2)
```

```python
nx.draw(K2, pos)
nx.draw_networkx_labels(K2, pos)
plt.show()
```

```python
for i in range(5):
    K2=remove_top_ebet(K2)

nx.draw(K2, pos)
nx.draw_networkx_labels(K2, pos)
plt.show()
```

The algorithm for finding "communities" based on edge betweenness removal was presented by Girvan-Newmann in their [2002 PNAS paper](http://www.pnas.org/content/99/12/7821). Based on the method implemented, the algorithm is classified as a _dividive procedure_ (as opposed to _agglomerative_, which is discussed in the next section. A code snippet taken [here](http://www-rohan.sdsu.edu/~gawron/python_for_ss/course_core/book_draft/Social_Networks/Networkx.html) is shown below.

### Connected Components

**Connected components** are subgraphs where all the nodes are connected to each via a path. Unlike **cliques**, they don't have to be _directly_ connected. Moreover, the **Giant Component** refers to the network's largest component.

```python
print ("Number of components: ", len(list(nx.connected_components(K2))),"\n")
print (list(nx.connected_components(K2)))
```

    Number of components:  3

    [{0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 17, 19, 21}, {32, 33, 2, 8, 14, 15, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31}, {9}]

## 4. Agglomerative Method: Hierarchical Clustering
In this method, we start with a **distance matrix** $D$, which is a matrix that provides the distance between two nodes $i$ and $j$ in network $G$. Using $D$, we then perform [hierarchical clustering](http://docs.scipy.org/doc/scipy-0.17.0/reference/cluster.hierarchy.html#module-scipy.cluster.hierarchy) in a recursive manner, group _similar_ nodes at each step. There are various ways to perform the similarity grouping using _agglomerative procedures_ such as **single**, **complete**, and **average**, to name a few. In the example below, we implement the [average/UPGMA linkage](http://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.cluster.hierarchy.average.html#scipy.cluster.hierarchy.average), which is the one implemented in the Ravasz Algorithm published in [Science in 2002](http://science.sciencemag.org/content/297/5586/1551).

In the following cells, we perform hierarchical clustering and visualize the order in which the nodes are grouped together in communities.

```python
__author__ = """\n""".join(['Erika Fille Legara <legareft@ihpc.a-star.edu.sg>',
                            'Maksim Tsvetovat <maksim@tsvetovat.org',
                            'Drew Conway <drew.conway@nyu.edu>',
                            'Aric Hagberg <hagberg@lanl.gov>'])

'''
The original code was written by Drew Conway and Aric Hagberg and was later
modified by Maksim Tsvetovat for his book titled “Social Network Analysis for Startups”.
EF Legara slightly modified Tsvetovat's version to address some issues
on the ordering of keys within the path_length dictionary. Moreover,
in this code, a different agglomerative procedure is implemented.
'''

def create_hc(G, t):
    ## Set-up the distance matrix D
    labels = G.nodes()    # keep node labels
    path_length = nx.all_pairs_shortest_path_length(G)
    distances=np.zeros((len(G),len(G)))

    '''
    for u,p in path_length.items():
        for v,d in p.items():
            distances[G.nodes().index(u)][G.nodes().index(v)] = d
            distances[G.nodes().index(v)][G.nodes().index(u)] = d
            if u==v: distances[G.nodes().index(u)][G.nodes().index(u)]=0
    '''

    for node, info in path_length:
        for other_node, l in info.items():
            distances[node][other_node] = l
            distances[other_node][node] = l
            if node==other_node: distances[node][node]=0

    # Create hierarchical cluster (HC)
    # There are various other routines for agglomerative clustering,
    # but here we create the HCs using the complete/max/farthest point linkage
    Y = distance.squareform(distances) ## the upper triangular of the distance matrix
    Z = hierarchy.average(Y)

    # This partition selection (t) is arbitrary, for illustrive purposes
    membership=list(hierarchy.fcluster(Z,t=t))

    # Create collection of lists for blockmodel
    partition = defaultdict(list)
    for n,p in zip(list(range(len(G))),membership):
        #partition[p].append(labels[n])
        partition[p].append(n)

    return Z, membership, partition
```

```python
Z, membership, partition = create_hc(K, t=1.15)
partition.items()
```

    dict_items([(4, [0, 1, 2, 3, 7, 11, 12, 17, 21]), (1, [4, 6]), (2, [5, 10]), (5, [8, 9, 13, 14, 15, 18, 19, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]), (3, [16])])

```python
partition = {}
i = 0
for i in range(len(membership)):
    partition[i]=membership[i]
```

```python
plt.figure(figsize=(10,10))
plt.axis('off')

#nx.draw_networkx_nodes(K, pos, cmap=plt.cm.RdYlBu, node_color=partition.values())
nx.draw_networkx_nodes(K, pos, cmap=plt.cm.RdYlBu, node_color=list(partition.values()))
nx.draw_networkx_edges(K, pos, alpha=0.5)
nx.draw_networkx_labels(K, pos)
plt.show()
```

### Dendrogram
The dendrogram allows us to visualize the order in which nodes are grouped together in the clustering process.

```python
plt.figure(figsize=(10,8))
hierarchy.dendrogram(Z)
plt.show()
```

## 5. Modularity-Based Community Detection: The Louvain Method

The **community detection algorithm** that we implement in this notebook is a **modularity**-based algorithm. The **modularity** $M_c$ quantifies how good a "community" or partition is, and is given by

\begin{equation}
M_c = \sum _{c=1} ^{n_c} \left[ \frac{L_c}{L} - \left (\frac{k_c}{2L} \right)^2\right]
\end{equation}

where $n_c$ is the number of communiies, $L_c$ is the number of links within the community, $k_c$ is the total degree of nodes belonging to the community, and $L$ is the total number of links in the network. The illustration below that is taken from the slide deck shows how modularity is calculated.

The higher the network modularity is, the more optimal the partitionioning. And, if you only have a single community, $M=0$.

In this recipe, we implement a modularity-maximization algorithm called the Louvain algorithm. We go back to Zachary's karate club and find the best partitions.

```python
partition = community.best_partition(K)
```

Now, let's redraw the network; this time, highlighting the different communities by node color.

```python
plt.figure(figsize=(10,10))
plt.axis('off')

nx.draw_networkx_nodes(K, pos, cmap=plt.cm.RdYlBu, node_color=list(partition.values()))
nx.draw_networkx_edges(K, pos, alpha=0.5)
nx.draw_networkx_labels(K, pos)
plt.show()
```

The algorithm produces four (4) partitions. Let's compare this partition to the actual.

```python
plt.figure(figsize=(8,8));
plt.axis("off");
nx.draw_networkx_nodes(K,pos, nodelist=mrhi, node_color='r',
                       node_size=300, alpha=0.8);
nx.draw_networkx_nodes(K,pos, nodelist=johna, node_color='b',
                       node_size=300, alpha=0.8);
nx.draw_networkx_edges(K, pos, alpha=0.5);
nx.draw_networkx_labels(K, pos, font_color="white");
```

## 6. More Real-World Networks

Note: If you are running this on Google Colab, the `pd.read_csv()` in the input cell below will result to an error. To fix the issue, download the datasets first from the Github repository and then upload them on your own Google Drive. Then execute the code:

```python
from google.colab import drive
drive.mount("/content/drive")
```

Assuming you upload the data in a directory `MyData` on your drive, you may read the file as:

```python
data_dir = "/content/drive/MyDrive/MyData/"
marvel = pd.read_csv(data_dir + "hero-network.csv", header=None)
```

### Sample 1: The Marvel Universe
There are two types of Marvel Universe networks that is [available on the web](http://exposedata.com/marvel/), the Comic Network and the Hero Network. The Comic Network connects the characters to the commic issues they appeared in; on ther hand, the Hero Network represents the connections between characters as they appear in the same comic issue (weighted). In this notebook, we look at the Hero Network.

```python
marvel = pd.read_csv('./datasets/hero-network.csv', header=None)
```

```python
marvel.head()
```

      0
      1

      0
      LITTLE, ABNER
      PRINCESS ZANDA

      1
      LITTLE, ABNER
      BLACK PANTHER/T'CHAL

      2
      BLACK PANTHER/T'CHAL
      PRINCESS ZANDA

      3
      LITTLE, ABNER
      PRINCESS ZANDA

      4
      LITTLE, ABNER
      BLACK PANTHER/T'CHAL

```python
M = nx.from_pandas_edgelist(marvel, 0, 1)
```

To export this network for use in **Gephi**, use

```python
nx.write_gexf(M, "marvel_universe.gexf")
```

#### Graph Statistics

```python
M.size() #number of edges
```

    167219

```python
M.order() #number of nodes
```

    6426

```python
M_cliques = list(nx.find_cliques(M))
```

```python
print (M_cliques[0:2])
```

    [['MOLGOTHA', 'VILIVEN', 'BRUTUS KLOR', 'SIFFROR', 'THOG', 'NIGHTCRAWLER/KURT WA', 'SKARABROUS', 'DAYTRIPPER/AMANDA SE', "S'YM", 'MALEVOLENCE', 'HEDRON', 'PHEMOUS', 'PLUTO [OLYMPIAN]', "D'SPAYRE", 'MEPHISTO', 'UMAR', 'DORMAMMU'], ['CEAUSECU, IONEL', 'DIABLO/ESTEBAN DE AB', 'CAPTAIN HIP/FRED MAC', 'BLACK FOX/ROBERT W. ', 'ZORN', 'PIXIE/ [ETERNAL]', 'MACRAE, TRUTH', 'LOCKE, DR. CASSANDRA', 'HUGO, CAPTAIN EDVARD', 'DR. STRANGE/STEPHEN ', 'LOCKE, DR. ALEXANDER', 'SUNSHINE/AUTUMN MACR', 'NOCTURNE/']]

#### Finding Communities

```python
partition = community.best_partition(M)
```

```python
len(set(partition.values()))
```

    26

The code below may take a while to run since we are looking at a relatively large network, so we're commenting it out.'

```python
plt.figure(figsize=(10,10))
plt.axis('off')

pos = nx.spring_layout(M)
nx.draw_networkx_nodes(M, pos,cmap=plt.cm.RdYlBu, node_color=partition.values())
nx.draw_networkx_edges(M, pos, alpha=0.4)
plt.savefig("Marvel Universe.png")
```

But this is how it looks using Gephi.

What are these communities? Which community is the largest? Sneak a look into the communities.

```python
comm = defaultdict(list)
for k, v in partition.items():
    comm[v].append(k)
```

```python
# comm.items()
```

```python
# list(comm.items())[11]
```

### Sample 2: Word Co-occurrence Network
In this example, we look at the word co-occurrence dataset used in the paper titled "News Framing of Population and Family Planning Issues via Syntactic Network Analysis" by E.F. Legara et al. Using a community detection algorithm, we looked for word patterns (how the words were 'arranged' or connected in the sentences of each news article) to evaluate how the media framed the population issue in the Philippines through the use of the different labels to refer to it in public discourse.

```python
F = nx.read_gml("./datasets/word-net.gml", label="label")
```

```python
partition = community.best_partition(F, weight="weight")
```

```python
# nx.write_gexf(F, "framing.gexf")
```

Commenting this out because the network is so dense, it doesn't say much when plotted.
```python
plt.figure(figsize=(10,10))
plt.axis('off')

pos = nx.spring_layout(F, iterations=150)
nx.draw_networkx_nodes(F, pos,cmap=plt.cm.RdYlBu, node_color=partition.values(), node_size=10, label=True)
nx.draw_networkx_edges(F, pos, alpha=0.4)
plt.savefig("Frames.png")
```
Instead, let's view the communities by printing out the words.

```python
coms = defaultdict(list)
for k in partition.keys():
    coms[partition[k]].append(k)
```

```python
# print (coms[0], "\n", coms[1], "\n", coms[2], "\n", coms[3], "\n", coms[4])
```

### Sample 3: Political Blogs

Here, we use the dataset utilized in the paper titled "The political blogosphere and the 2004 US Election" by L. A. Adamic and N. Glance (2005). It's a directed network of hyperlinks between weblogs on US politics.

```python
P = nx.read_gml("./datasets/polblogs copy.gml")
```

#### Some Network Properties

```python
# P.nodes(data=True)
```

```python
P.order(), P.size()
```

    (1490, 19090)

```python
pos = nx.spring_layout(P)
nx.draw(P, pos, node_size=30)
plt.show()
```

#### Finding Communities
To perform the community detection algorithm, the **directed graph** needs to be made into an **undirected** one.

```python
PR = P.to_undirected()
PR = nx.Graph(PR)
```

```python
mypalette = ['blue', 'red', 'green', 'yellow', 'orange', 'violet'] + 5 * ['grey']
```

```python
pos = nx.spring_layout(PR)
```

```python
#colors = [mypalette[PR.node[i]['value']] for i in range(1,len(PR.nodes()))]
colors = [mypalette[PR.nodes()[i]['value']] for i in PR.nodes()]
nx.draw(PR, pos, node_color=colors, node_size=10)
plt.show()
```

##### Extract Giant Component

```python
GC_nodes = max(nx.connected_components(PR), key=len)
GC = PR.subgraph(GC_nodes).copy()
```

```python
type(GC)
```

    networkx.classes.graph.Graph

```python
partition = community.best_partition(GC)
```

```python
print ("Number of Communities: ", len(set(partition.values())))
```

    Number of Communities:  11

```python
mypalette = ['blue', 'green', 'red', 'yellow', 'orange', 'violet'] + 5 * ['grey']

for k, v in partition.items():
    GC.nodes()[k]["louvain-val"] = v
colors = [mypalette[GC.nodes()[node]["louvain-val"]] for node in GC.nodes()]
```

```python
plt.figure(figsize=(10,10))
plt.axis('off')
pos = nx.spring_layout(GC, scale=3)
nx.draw_networkx_nodes(GC, pos, node_color=colors, node_size=18, label=True)
nx.draw_networkx_edges(GC, pos, alpha=0.4)
plt.savefig("polblogs.png")
```

### Sample 4: Your Own Network Data

#### Load the Network

```python

```

#### Network Properties and Structures

```python

```

## Relevant References

* **Network Science** by Albert-László Barabási (2015)
* **Social Network Analysis for Startups: Finding connections on the social web** by Maksim Tsvetovat and  Alexander Kouznetsov (2011)
* **Practical Data Science Cookbook** by Tony Ojeda et al. (2014)

## Some Network Data Repositories

* [The UCI Network Data Repository](http://deim.urv.cat/~alexandre.arenas/data/welcome.htm)
* [CCNR Network Databases](http://www3.nd.edu/~networks/resources.htm)
* [Alex Arenas Website](http://deim.urv.cat/~alexandre.arenas/data/welcome.htm)
* [Stanford Large Network Dataset Collection](http://snap.stanford.edu/data/)


```python
%%time
import time
for _ in range(1000):
    time.sleep(0.01)# sleep for 0.01 seconds

from sympy import *
from sympy import init_printing; init_printing(use_latex = 'mathjax')
from sympy.plotting import plot

n = int(input('Qué número de valores de energía desea aproximar?'))

l, m, hbar, k = symbols('l m hbar k', real = True, constant = True)
var('x,W')

H = ones(n,n)
S = ones(n,n)
U = ones(n,n)
CC = ones(n,n)

#F = [sympify(input('Ingrese la función {0}: '.format(i+1))) for i in range(n)]
F = [x*(l - x),(x**2)*((l - x)**2),x*(l - x)*((l/2)-x),(x**2)*((l - x)**2)*((l/2)-x)]

fi = zeros(n)
c = ones(n,n)
for i in range(n):
    for j in range(n):
        c[i,j] = sympify('c%d%d' %(j+1,i+1))
        fi[j] = sympify('phi%d' %(j+1))

for j in range(1,n+1):                                                             #loop para llenar la matriz H
    for i in range(1,n+1):
        I = ((-hbar**2)/(2*m))
        integrando = I*(F[j-1])*diff(F[i-1], x, 2)
        A = integrate(integrando, (x, 0, l))
        integrandos = (F[j-1])*(F[i-1])
        B = integrate(integrandos, (x, 0, l))
        H[j-1,i-1] *= A
        S[j-1,i-1] *= B
        U[j-1,i-1] *= (H[j-1,i-1] -W*S[j-1,i-1])

E = U.det()
EE = solve(E,W)

a = 1/EE[0]

#truco para ordenar los W
for i in range(n):
    EE[i] = EE[i]*a
EE.sort()
for j in range(n):
    EE[j] = EE[j]*(1/a)

cc = Matrix(c)

for j in range(n):
    for i in range(n):
        C = U*cc.col(j)
        CC[i,j] *= C[i].subs(W, EE[j])
G = []
for i in range(n):
    D = solve(CC.col(i),cc)
    G.append(list(D.items()))

G = Matrix(G)
J = []

for i in range(len(G)):
    if G[i][1] != 0:
        J.append(factor(G[i]))

ceros = []
param = []

for i in range(len(G)):
    if G[i][1] != 0:
        param.append(G[i][0])
    elif G[i][1] == 0:
        ceros.append(G[i][0])
kas = [x for x in cc if x not in (ceros+param)]

finale = ones(n,n)

for j in range(n):
    for i in range(n):
        if sympify('c'+str(i+1)+str(j+1)) not in (ceros+param):
            finale[i,j] *= k
        elif sympify('c'+str(i+1)+str(j+1)) not in (kas+ceros):
            finale[i,j] *= J[i][1].subs(sympify('c'+str(i+1)+str(j+1+1)),k)
        else:
            finale[i,j] *= 0

Psi = factor(finale*Matrix(F))
integrand = []
Psis = []
for i in range(n):
    integrand.append(Psi[i]**2)
    Psis.append(integrate(integrand[i], (x, 0, l)))

normaliz = []
for i in range(n):
    normaliz.append(factor(Psis[i])*(1/k**2)-(1/k**2))
KKK = []
Figaro = []
for i in range(n):
    KKK.append(solve(normaliz[i],k**2))
    Figaro.append(Psi[i]**2)
    Figaro[i] = Figaro[i].subs(k**2,KKK[i][0])

for i in range(n):
    plot(Figaro[i].subs(l, 1), (x, 0,1))

```