###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2018  by D. Koehn, heterogeneous models are from [this Jupyter notebook](https://krischer.github.io/seismo_live_build/html/Computational%20Seismology/The%20Finite-Difference%20Method/fd_ac2d_heterogeneous_wrapper.html) by Heiner Igel ([@heinerigel](https://github.com/heinerigel)), Florian Wölfl and Lion Krischer ([@krischer](https://github.com/krischer)) which is a supplemenatry material to the book [Computational Seismology: A Practical Introduction](http://www.computational-seismology.org/), notebook style sheet by L.A. Barba, N.C. Clementi

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 2D acoustic FD modelling of the Marmousi-2 model

In this notebook, we will apply all the knowledge about FD modelling, we covered so far. While the modelling examples in the last lesson where quite simple, we now calculate the 2D acoustic wave propagation in a more realistic problem called the **Marmousi-2 model**.
Developed in the 1990s by the French Petroleum Institute (IFP) ([Versteeg, 1994](https://library.seg.org/doi/abs/10.1190/1.1437051)), the Marmousi model is a widely used benchmark problem for seismic imaging and inversion techniques. Beside the original acoustic version of the model, an elastic version was developed by [Martin et al. (2006)](https://library.seg.org/doi/abs/10.1190/1.2172306).

The Marmousi-2 model consists of a 460 m thick water layer above an elastic subseafloor model.  The sediment model is very simple near the left and right boundaries but rather complex in the centre. At both sides, the subseafloor is approximately horizontally layered, while steep thrust faults are disturbing the layers in the centre of the model. Embedded in the thrust fault system and layers are small scale hydrocarbon reservoirs.

In this notebook you will learn how to setup the basic parameters, required to model a seismic acquistion geometry:

Setup and model the 2D acoustic wave propagation in the Marmousi-2 model:

- Define the model discretization based on the given Marmousi-2 P-wave velocity model
- Calculate the central frequency $f_0$ of the source wavelet based on the grid dispersion criterion

\begin{equation}
dx \le \frac{vp_{min}}{N_\lambda f_0}, \nonumber
\end{equation}

which you can use for the FD modelling run, for the pre-defined $dx$ of the Marmousi-2 model , minimum P-wave velocity $vp_{min}$ and $N_\lambda = 4$ gridpoints per dominant wavelength.
- Starting a modelling run for the Marmousi-2 model. Place an airgun for a central shot at x = 5000 m at a depth = 40 m below the sea surface. Do not forget to calculate an appropriate time step $dt$
- Imagine you place an Ocean-Bottom-Cable (OBC) on the seafloor of the Marmousi-2 model. Calculate an OBC shot gather, by placing receivers at each gridpoint of the Cartesian model in x-direction in a depth of 460 m. Modify the FD code to record seismograms at each receiver position. Do not forget to return the seismograms from the modelling function.
- Plot the OBC-seismograms produced by the finite-difference modelling code.

```python
# Import Libraries
# ----------------
import numpy as np
from numba import jit
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

from mpl_toolkits.axes_grid1 import make_axes_locatable
```

The elastic version of the Marmousi-2 model, together with FD modelled high frequency shot gather data is available from [here](http://www.agl.uh.edu/downloads/downloads.htm). The central part of the P-wave velocity model of the Marmousi-2 with the spatial discretization:

$nx = 500$ gridpoints

$nz = 174$ gridpoints

$dx = dz = 20\; m$

is available as IEEE little endian binary file `marmousi_II_marine.vp` in the `marmousi-2` directory. It can be imported to Python with the following code snippet:

```python
# Import Marmousi-2 Vp model
# --------------------------

# DEFINE MODEL DISCRETIZATION HERE!
nx = 500       # number of grid points in x-direction
nz = 174       # number of grid points in z-direction
dx = 20.0      # spatial grid point distance in x-direction (m)
dz = dx        # spatial grid point distance in z-direction (m)

# Define model filename
name_vp = "marmousi-2/marmousi_II_marine.vp"

# Open file and write binary data to vp
f = open(name_vp)
data_type = np.dtype ('float32').newbyteorder ('<')
vp = np.fromfile (f, dtype=data_type)

# Reshape (1 x nx*nz) vector to (nx x nz) matrix
vp = vp.reshape(nx,nz)
```

After reading the model into Python, we can take a look at it ...

```python
# Plot Marmousi-2 vp-model
# ------------------------

# DEFINE XMAX, ZMAX AND MODEL EXTENSION HERE!
xmax = nx * dx
zmax = nz * dz
extent = [0, xmax, zmax, 0]

fig = plt.figure(figsize=(12,3))  # define figure size

image = plt.imshow((vp.T)/1000, cmap=plt.cm.viridis, interpolation='nearest',
                   extent=extent)

cbar = plt.colorbar(aspect=10, pad=0.02)
cbar.set_label('Vp [km/s]', labelpad=10)
plt.title('Marmousi-2 model')
plt.xlabel('x [m]')
plt.ylabel('z [m]')
```

```python
# Definition of modelling parameters
# ----------------------------------
# DEFINE MAXIMUM RECORDING TIME HERE!
tmax = 6.0    # maximum recording time of the seismogram (s)

# DEFINE YOUR SHOT POSITION HERE!
xsrc = 5000.0 # x-source position (m)
zsrc = 40.0   # z-source position (m)

# calculate maximum frequency according to grid dispersion criterion
N_lam = 4

# CALCULATE DOMINANT FREQUENCY OF THE SOURCE WAVELET HERE!
f0   = np.min(vp) / (dx * N_lam) # dominant frequency of the source (Hz)
print("f0 = ", f0, " Hz")
t0   = 4.0/f0   # source time shift (s)

isnap = 2  # snapshot interval (timesteps)
```

    f0 =  18.75  Hz

```python
@jit(nopython=True) # use JIT for C-performance
def update_d2px_d2pz_3pt(p, dx, dz, nx, nz, d2px, d2pz):

    for i in range(1, nx - 1):
        for j in range(1, nz - 1):

            d2px[i,j] = (p[i + 1,j] - 2 * p[i,j] + p[i - 1,j]) / dx**2
            d2pz[i,j] = (p[i,j + 1] - 2 * p[i,j] + p[i,j - 1]) / dz**2

    return d2px, d2pz
```

```python
# Define simple absorbing boundary frame based on wavefield damping
# according to Cerjan et al., 1985, Geophysics, 50, 705-708
def absorb(nx,nz):

    FW = 60     # thickness of absorbing frame (gridpoints)
    a = 0.0053

    coeff = np.zeros(FW)

    # define coefficients in absorbing frame
    for i in range(FW):
        coeff[i] = np.exp(-(a**2 * (FW-i)**2))

    # initialize array of absorbing coefficients
    absorb_coeff = np.ones((nx,nz))

    # compute coefficients for left grid boundaries (x-direction)
    zb=0
    for i in range(FW):
        ze = nz - i - 1
        for j in range(zb,ze):
            absorb_coeff[i,j] = coeff[i]

    # compute coefficients for right grid boundaries (x-direction)
    zb=0
    for i in range(FW):
        ii = nx - i - 1
        ze = nz - i - 1
        for j in range(zb,ze):
            absorb_coeff[ii,j] = coeff[i]

    # compute coefficients for bottom grid boundaries (z-direction)
    xb=0
    for j in range(FW):
        jj = nz - j - 1
        xb = j
        xe = nx - j
        for i in range(xb,xe):
            absorb_coeff[i,jj] = coeff[j]

    return absorb_coeff
```

```python
# FD_2D_acoustic code with JIT optimization
# -----------------------------------------
def FD_2D_acoustic_JIT(vp, dt,dx,dz,f0,xsrc,zsrc,op):

    # calculate number of time steps nt
    # ---------------------------------
    nt = (int)(tmax/dt)

    # locate source on Cartesian FD grid
    # ----------------------------------
    isrc = (int)(xsrc/dx)  # source location in grid in x-direction
    jsrc = (int)(zsrc/dz)  # source location in grid in x-direction

    # Source time function (Gaussian)
    # -------------------------------
    src  = np.zeros(nt + 1)
    time = np.linspace(0 * dt, nt * dt, nt)

    # 1st derivative of Gaussian
    src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

    # define clip value: 0.1 * absolute maximum value of source wavelet
    clip = 0.5 * max([np.abs(src.min()), np.abs(src.max())]) / (dx*dz) * dt**2

    # Define absorbing boundary frame
    # -------------------------------
    absorb_coeff = absorb(nx,nz)

    # Define squared vp-model
    # -----------------------
    vp2 = vp**2

    # Initialize empty pressure arrays
    # --------------------------------
    p    = np.zeros((nx,nz)) # p at time n (now)
    pold = np.zeros((nx,nz)) # p at time n-1 (past)
    pnew = np.zeros((nx,nz)) # p at time n+1 (present)
    d2px = np.zeros((nx,nz)) # 2nd spatial x-derivative of p
    d2pz = np.zeros((nx,nz)) # 2nd spatial z-derivative of p

    # Initialize seismograms
    # ----------------------
    seis = np.zeros((nx,nt+1))
    jr = 23

    # Initalize animation of pressure wavefield
    # -----------------------------------------
    fig = plt.figure(figsize=(7,3))  # define figure size
    extent = [0.0,xmax,zmax,0.0]     # define model extension

    # Plot Vp-model
    image = plt.imshow((vp.T)/1000, cmap=plt.cm.gray, interpolation='nearest',
                        extent=extent)

    # Plot pressure wavefield movie
    image1 = plt.imshow(p.T, animated=True, cmap="RdBu", alpha=.75, extent=extent,
                          interpolation='spline36', vmin=-clip, vmax=clip)
    plt.title('Pressure wavefield')
    plt.xlabel('x [m]')
    plt.ylabel('z [m]')

    plt.ion()
    plt.show(block=False)

    snap_count = 0

    # Calculate Partial Derivatives
    # -----------------------------
    for it in range(nt):

        # FD approximation of spatial derivative by 3 point operator
        if(op==3):
            d2px, d2pz = update_d2px_d2pz_3pt(p, dx, dz, nx, nz, d2px, d2pz)

        # Time Extrapolation
        # ------------------
        pnew = 2 * p - pold + vp2 * dt**2 * (d2px + d2pz)

        # Add Source Term at isrc
        # -----------------------
        # Absolute pressure w.r.t analytical solution
        pnew[isrc,jsrc] = pnew[isrc,jsrc] + src[it] / (dx * dz) * dt ** 2

        # Apply absorbing boundary frame
        # ------------------------------
        p *= absorb_coeff
        pnew *= absorb_coeff

        # Remap Time Levels
        # -----------------
        pold, p = p, pnew

        # WRITE SEISMOGRAMS HERE!
        seis[:, it] = p[:, jr]

        # display pressure snapshots
        if (it % isnap) == 0:
            image1.set_data(p.T)
            fig.canvas.draw()

            # write snapshots to Tiff files
            name_snap = "image_out/marmousi_2D_" + "%0.*f" %(0,np.fix(snap_count+1000)) + ".tiff"
            plt.savefig(name_snap, format='tiff', bbox_inches='tight', dpi=125)
            snap_count += 1

    # DO NOT FORGET TO RETURN THE SEISMOGRAM HERE!
    return seis
```

## Model wave propagation in the Marmousi-2 model with 2D acoustic FD code

Time to model acoustic wave propagation in the Marmousi-2 model using the 3-point operator. We only have to define the timestep $dt$:

```python
# Run 2D acoustic FD modelling with 3-point spatial operater
# ----------------------------------------------------------
%matplotlib notebook
op = 3  # define spatial FD operator (3-point)

# DEFINE TIME STEP HERE!
# calculate dt according to CFL criterion
dt = 0.85 * dx / (np.sqrt(2) * np.max(vp)) # time step (s)

seis_marm_3pt = FD_2D_acoustic_JIT(vp,dt,dx,dz,f0,xsrc,zsrc,op)
```

    <IPython.core.display.Javascript object>

```python
from IPython.display import YouTubeVideo
YouTubeVideo('IHF4vYeVKs8')
```

Notice, how the direct wave, excitated by the air gun in the water column is first reflected at the seafloor, and leads to a very complex seismic wavefield as soon as it propagates in the Marmousi-2 subsurface model. This wavefield consists of reflected, diffracted and refracted acoustic waves.

Finally, we can take a look at the OBC shot gather:

```python
%matplotlib notebook
# PLOT YOUR MODELLED OBC SHOT GATHER HERE!
clip_seis = 1e-7
extent_seis = [0.0,xmax/1000,tmax,0.0]
plt.imshow(seis_marm_3pt.T, cmap=plt.cm.gray, aspect=2, vmin=-clip_seis,
                   vmax=clip_seis, extent=extent_seis)

plt.title('OBC seismogram')
plt.xlabel('x [km]')
plt.ylabel('t [s]')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 't [s]')

## What we learned:

- How to model wave propgation in the complex Marmousi-2 model. Now you can model everything ...


# IPython Notebooks

## What are IPython Notebooks?

* IPython Notebooks are browser based files
* They contain Python code and output from code
* They can contain documentation written in Markdown amrkup as well as **mathematical equations** typeset in $\LaTeX{}$
* They can contain rich media - images, video, other web pages

## Cells
* An IPython Notebook consists of **cells**
* Cells can be of different types
  * **Markdown** - documentation with embedded $\LaTeX{}$ equations
  * **Code** - Python code
  * **Slide** - HTML presentation slides

## $\LaTeX{}$ in Markdown
* IPython Notebooks use **MathJax** Javascript library to render $\LaTeX$ equations
* MathJax Javascript library is downloaded from the Internet once per session
* If Internet connection is unavailable, $\LaTeX$ equations **will not be rendered**
* It is possible to install a local copy of MathJax so that $\LaTeX$ equations can be rendered without an Internet connection

## Installing Local Copy of MathJax to Render Equations when Offline
With your machine connected to the Internet, open IPython console and at the IPython prompt, type the following statements

    from IPython.external.mathjax import install_mathjax
    install_mathjax()

Wait for MathJax to be downloaded and installed. Now you have a local copy of MathJax and $\LaTeX$ equations will be rendered even when you are not connected to the Internet

## Converting IPython Notebooks to Other Formats

IPython Notebooks are live, that is code can be changed and executed to see the change to the output. Notebooks can be converted to other formats, but the Notebook is now static, that is, code can no longer be changed and executed to see the changed output.

Following output formats are supported:
* HTML
* $\LaTeX$ if **`latex`** is installed
* PDF if **`pdflatex`** is installed

Commands to convert Notebooks to other formats are:

* Convert to HTML: **`ipython nbconvert notebook.ipynb`**
* Convert to $\LaTeX$: **`ipython nbconvert notebook.ipynb --to latex`**
* Convert to PDF: **`ipython nbconvert notebook.ipynb --to latex --post PDF`**
* Convert to HTML slides and start slideshow: **`ipython notebook notebook.ipynb --to slides --post serve`**

## Starting the Notebook Server

* IPython Notebooks have the filename extension **`.ipynb`**
* Choose a directory where you want to put your IPython Notebooks
* Change over to the chosen directory and type the command at the command prompt

    ipython notebook

* This will start up the IPython Notebook server, which is a web server listening usually on **port 8888**
* Your default browser will be opened and will display the IPython Notebook dashboard
* Dashboard lists all Notebook files, if any, in the directory from where the Notebook server is started
* Dashboard will also show any directories that may be present in the Notebook directory

## Working with Notebooks

* You can create new Notebooks. New Notebooks are assigned a name starting with **Untitledx.ipynb**
* You can delete existing Notebooks from the dashboard
* You can rename an opened Notebook
* You can shutdown a running Notebook from the dashboard

## Quick Introduction to Markdown

### Headings

* Level 1 headings are created with the markup `# Level 1 Heading`
* Two `##` indicate a Level 2 heading, Three `###` indicate a Level 3 heading and so on

### Bold, Italic

* Anything marked up as `**Bold**` will show up as **Bold**
* Anything marked up as `*Italic*` shows up as *Italic*
* `monospaced` is typeset by enclosing monospaced within two backquotes, namely `

### $\LaTeX$ Equations
* Inline equations use `$` to demarcate a $\LaTeX$ equation. For example `$f(x) = a^2 x + b x + x$` will be typeset as $f(x) = a^2 x + b x + x$
* Displayed equations without numbers are enclosed within matching pair of `$$ f(x) = \frac{\sqrt{a^2 + b^2}}{2 \sin \theta} $$`

$$ f(x) = \frac{\sqrt{a^2 + b^2}}{2 \sin \theta} $$

* You can use `\begin{equation}`
* You can use `\begin{align}`

#### Source
    \begin{align} \dot{x} & = \sigma(y-x) \\ \dot{y} & = \rho x - y - xz \\ \dot{z} & = -\beta z + xy \end{align}

\begin{align} \dot{x} & = \sigma(y-x) \\ \dot{y} & = \rho x - y - xz \\ \dot{z} & = -\beta z + xy \end{align}

### Shutting Down Notebook Server

* Save and close all Notebooks
* Close the dashboard
* Go to the MS DOS prompt or Terminal from where the Notebook Server was started and stop it by pressing `Ctrl+C` twice. If server does not shutdown, close the MS DOS prompt window

```python
from IPython.display import Image
Image(filename='python-logo.png')
```

```python

```


# Tutorial-IllinoisGRMHD: eigen.C

## Authors: Leo Werneck & Zach Etienne

**This module is currently under development**

## In this tutorial module we explain how to obtain the eigenvalues of a $3\times3$ matrix. This module will likely be absorbed by another one once we finish documenting the code.

### Required and recommended citations:

* **(Required)** Etienne, Z. B., Paschalidis, V., Haas R., Mösta P., and Shapiro, S. L. IllinoisGRMHD: an open-source, user-friendly GRMHD code for dynamical spacetimes. Class. Quantum Grav. 32 (2015) 175009. ([arxiv:1501.07276](http://arxiv.org/abs/1501.07276)).
* **(Required)** Noble, S. C., Gammie, C. F., McKinney, J. C., Del Zanna, L. Primitive Variable Solvers for Conservative General Relativistic Magnetohydrodynamics. Astrophysical Journal, 641, 626 (2006) ([astro-ph/0512420](https://arxiv.org/abs/astro-ph/0512420)).
* **(Recommended)** Del Zanna, L., Bucciantini N., Londrillo, P. An efficient shock-capturing central-type scheme for multidimensional relativistic flows - II. Magnetohydrodynamics. A&A 400 (2) 397-413 (2003). DOI: 10.1051/0004-6361:20021641 ([astro-ph/0210618](https://arxiv.org/abs/astro-ph/0210618)).

# Table of Contents
$$\label{toc}$$

This module is organized as follows

0. [Step 0](#src_dir): **Source directory creation**
1. [Step 1](#introduction): **Introduction**
1. [Step 2](#eigen__c): **`eigen.C`**
    1. [Step 2.a](#eigen__c__variables): *The variables used in `eigen.C`*
    1. [Step 2.b](#eigen__c__phi): *Determining $\phi$*
    1. [Step 2.c](#eigen__c__eigenvalues): *The eigenvalues of a $3\times3$ symmetric matrix*
1. [Step 3](#code_validation): **Code validation**
1. [Step 4](#latex_pdf_output): **Output this module to $\LaTeX$-formatted PDF file**

# Step 0: Source directory creation \[Back to [top](#toc)\]
$$\label{src_dir}$$

We will now use the [cmdline_helper.py NRPy+ module](Tutorial-Tutorial-cmdline_helper.ipynb) to create the source directory within the `IllinoisGRMHD` NRPy+ directory, if it does not exist yet.

```python
# Step 0: Creation of the IllinoisGRMHD source directory
# Step 0a: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..","..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step 0b: Load up cmdline_helper and create the directory
import cmdline_helper as cmd
IGM_src_dir_path = os.path.join("..","src")
cmd.mkdir(IGM_src_dir_path)

# Step 0c: Create the output file path
outfile_path__eigen__C = os.path.join(IGM_src_dir_path,"eigen.C")
```

# Step 1: Introduction \[Back to [top](#toc)\]
$$\label{introduction}$$

In this tutorial notebook we will implement an algorithm to evaluate the eigenvalues of a $3\times3$ symmetric matrix. Our method will be analytical and will follow closely [this discussion](https://en.wikipedia.org/wiki/Eigenvalue_algorithm#3%C3%973_matrices).

Let $\mathcal{M}$ be a $3\times3$ symmetric matrix,

$$
\mathcal{M} =
\begin{pmatrix}
M_{11} & M_{12} & M_{13}\\
M_{12} & M_{22} & M_{23}\\
M_{13} & M_{23} & M_{33}
\end{pmatrix}\ .
$$

To obtain the eigenvalues of $\mathcal{M}$, we must solve the *characteristic equation*

$$
\det\left(\lambda I_{3\times3} - \mathcal{M}\right) = 0\ ,
$$

where $\lambda$ represents the eigenvalues of $\mathcal{M}$ and $I_{3\times3} = {\rm diag}\left(1,1,1\right)$ is the $3\times3$ identity matrix. For this particular case, the characteristic equation of $\mathcal{M}$ is then given by

$$
\lambda^{3} - {\rm tr}\left(\mathcal{M}\right)\lambda^{2} + \left[\frac{{\rm tr}\left(\mathcal{M}^{2}\right) - {\rm tr}\left(\mathcal{M}\right)^{2}}{2}\right]\lambda - \det\left(\mathcal{M}\right) = 0\ .
$$

Now let $\mathcal{M} = n\mathcal{N} + mI_{3\times3}$, so that the matrices $\mathcal{M}$ and $\mathcal{N}$ have the same eigenvectors. Then, $\kappa$ is an eigenvalue of $\mathcal{N}$ if, and only if, $\lambda = n\kappa + m$ is an eigenvalue of $\mathcal{M}$. Now, let us look at the following identities:

$$
\mathcal{N} = \frac{1}{n}\left(\mathcal{M} - mI_{3\times3}\right)\ .
$$

Choosing $m \equiv \frac{1}{3}{\rm tr}\left(\mathcal{M}\right)$, we get

$$
{\rm tr}\left(\mathcal{N}\right) = \frac{1}{n}\left(\mathcal{M} - 3m\right)=0\ .
$$

Also,

$$
{\rm tr}\left(\mathcal{N}^{2}\right) = \frac{1}{n^{2}}\left[N_{11}^{2}+N_{22}^{2}+N_{33}^{2}+2\left(N_{12}^{2}+N_{13}^{2}+N_{23}^{2}\right)\right]\ ,
$$

so that if we choose $n\equiv\sqrt{\frac{N_{11}^{2}+N_{22}^{2}+N_{33}^{2}+2\left(N_{12}^{2}+N_{13}^{2}+N_{23}^{2}\right)}{6}}$ we get

$$
{\rm tr}\left(\mathcal{N}^{2}\right) = 6\ .
$$

Then, if we look at the characteristic equation for the matrix $\mathcal{N}$,

$$
\kappa^{3} - {\rm tr}\left(\mathcal{N}\right)\kappa^{2} + \left[\frac{{\rm tr}\left(\mathcal{N}^{2}\right) - {\rm tr}\left(\mathcal{N}\right)^{2}}{2}\right]\kappa - \det\left(\mathcal{N}\right) = 0\ ,
$$

we see that it can be greatly simplified with our choices of $m$ and $n$,

$$
\kappa^{3} - 3\kappa - \det\left(\mathcal{N}\right) = 0\ .
$$

Further simplification of this characteristic equation can be obtained by using

$$
\begin{align}
\kappa &\equiv 2\cos\phi\ ,\\
\cos\left(3\phi\right) &= 4\cos^{3}\phi - 3\cos\phi\ ,
\end{align}
$$

so that

$$
\begin{align}
0 &= 8\cos^{3}\phi - 6\cos\phi - \det\left(\mathcal{N}\right)\\
  &= 2\cos\left(3\phi\right) - \det\left(\mathcal{N}\right)\\
\implies \phi &= \frac{1}{3}\arccos\frac{\det\left(\mathcal{N}\right)}{2} + \frac{2k\pi}{3}\ ,\ k=0,1,2\ ,
\end{align}
$$

which, finally, yields

$$
\boxed{\kappa\left(k\right) = 2\cos\left(\frac{1}{3}\arccos\frac{\det\left(\mathcal{N}\right)}{2}+\frac{2k\pi}{3}\right)}\ .
$$

Once we have $\kappa$, we can find the eigenvectors of $\mathcal{M}$ doing

$$
\boxed{
\begin{align}
\lambda_{1} &= m + 2n\kappa(0)\\
\lambda_{2} &= m + 2n\kappa(1)\\
\lambda_{3} &= 3m - \lambda_{1} - \lambda_{2}
\end{align}
}\ ,
$$

where we have used the fact that ${\rm tr}\left(\mathcal{M}\right)=\lambda_{1}+\lambda_{2}+\lambda_{3}$ to compute $\lambda_{3}$.

# Step 2: `eigen.C` \[Back to [top](#toc)\]
$$\label{eigen__c}$$

## Step 2.a: The variables used in `eigen.C` \[Back to [top](#toc)\]
$$\label{eigen__c__variables}$$

In the algorithm below, we define the following quantities

$$
\boxed{
\begin{align}
\mathcal{K} &= \mathcal{M} - mI_{3\times3}\\
m &= \frac{{\rm tr}\left(\mathcal{M}\right)}{3}\\
q &= \frac{\det\left(\mathcal{K}\right)}{2}\\
p &= n^{2} = \frac{{\rm tr}\left(\mathcal{K}^{2}\right)}{6}
\end{align}
}\ .
$$

We these definitions, we have the following quantities to be implemented:

$$
\boxed{ m = \frac{\left(M_{11} + M_{22} + M_{33}\right)}{3} }\ .
$$

The matrix $\mathcal{K}$ is simply

$$
\boxed{
\mathcal{K} =
\begin{pmatrix}
M_{11}-m & M_{12} & M_{13}\\
M_{12} & M_{22}-m & M_{23}\\
M_{13} & M_{23} & M_{33}-m
\end{pmatrix}
}\ .
$$

Straightforwardly, we have

$$
\boxed{q = \frac{K_{11}K_{22}K_{33} +
          K_{12}K_{23}K_{13} +
          K_{13}K_{12}K_{23} -
          K_{13}K_{22}K_{13} -
          K_{12}K_{12}K_{33} -
          K_{11}K_{23}K_{23}
          }{2}
          }\ .
$$

Since $\mathcal{K}$ is symmetric as well, we have

$$
\boxed{p = \frac{K_{11}^{2} + K_{22}^{2} + K_{33}^{2} + 2\left(K_{12}^{2} + K_{13}^{2} + K_{23}^{2}\right)}{6}}\ .
$$

```python
%%writefile $outfile_path__eigen__C
//
// This subroutine calcualtes the eigenvalues of a real, symmetric 3x3
// matrix M={{M11,M12,M13},{M12,M22,M23},{M13,M23,M33}} based on the
// algorithm described in
// http://en.wikipedia.org/wiki/Eigenvalue_algorithm#Eigenvalues_of_3.C3.973_matrices
// which simply solve the cubic equation Det( M - lamnda I)=0 analytically.
// The eigenvalues are stored in lam1, lam2 and lam3.
//
void eigenvalues_3by3_real_sym_matrix(CCTK_REAL & lam1, CCTK_REAL & lam2, CCTK_REAL & lam3,
                                      CCTK_REAL M11, CCTK_REAL M12, CCTK_REAL M13, CCTK_REAL M22, CCTK_REAL M23, CCTK_REAL M33)
{
  CCTK_REAL m = (M11 + M22 + M33)/3.0;
  CCTK_REAL K11 = M11 - m, K12 = M12, K13 = M13, K22 = M22-m, K23 = M23, K33=M33-m;
  CCTK_REAL q = 0.5* (K11*K22*K33 + K12*K23*K13 + K13*K12*K23 - K13*K22*K13
                      - K12*K12*K33 - K11*K23*K23);
  CCTK_REAL p = ( SQR(K11) + SQR(K22) + SQR(K33) + 2.0*(SQR(K12) + SQR(K13) + SQR(K23) ) )/6.0;
```

    Overwriting ../src/eigen.C

## Step 2.b: Determining $\phi$ \[Back to [top](#toc)\]
$$\label{eigen__c__phi}$$

We then employ the following criterion to determine $\phi$:

$$
\phi
=
\left\{
\begin{matrix}
0 &,\ {\rm if}\ \left|q\right| \geq p^{3/2}\ ,\\
\arccos\left(\frac{q}{p^{3/2}}\right) &,\ {\rm otherwise}\ .
\end{matrix}
\right.
$$

```python
%%writefile -a $outfile_path__eigen__C

  CCTK_REAL phi;
  CCTK_REAL p32 = sqrt(p*p*p);
  if (fabs(q) >= fabs(p32) ) {
    phi = 0.0;
  } else {
    phi = acos(q/p32)/3.0;
  }
  if (phi<0.0) phi += M_PI/3.0;
```

    Appending to ../src/eigen.C

## Step 2.c: The eigenvalues of a $3\times3$ symmetric matrix \[Back to [top](#toc)\]
$$\label{eigen__c__eigenvalues}$$

Finally, the eigenvalues are computed using

$$
\boxed{
\begin{align}
\lambda_{1} &= m + 2\sqrt{p}\cos\phi\\
\lambda_{2} &= m - \sqrt{p}\cos\phi - \sqrt{3p}\sin\phi\\
\lambda_{3} &= m - \sqrt{p}\cos\phi + \sqrt{3p}\sin\phi
\end{align}
}\ .
$$

```python
%%writefile -a $outfile_path__eigen__C

  CCTK_REAL sqrtp = sqrt(p);
  CCTK_REAL sqrtp_cosphi = sqrtp*cos(phi);
  CCTK_REAL sqrtp_sqrt3_sinphi = sqrtp*sqrt(3.0)*sin(phi);
  lam1 = m + 2.0*sqrtp_cosphi;
  lam2 = m - sqrtp_cosphi - sqrtp_sqrt3_sinphi;
  lam3 = m - sqrtp_cosphi + sqrtp_sqrt3_sinphi;
}

```

    Appending to ../src/eigen.C

# Step 3: Code validation \[Back to [top](#toc)\]
$$\label{code_validation}$$

First we download the original `IllinoisGRMHD` source code and then compare it to the source code generated by this tutorial notebook.

```python
# Verify if the code generated by this tutorial module
# matches the original IllinoisGRMHD source code

# First download the original IllinoisGRMHD source code
import urllib
from os import path

original_IGM_file_url  = "https://bitbucket.org/zach_etienne/wvuthorns/raw/5611b2f0b17135538c9d9d17c7da062abe0401b6/IllinoisGRMHD/src/eigen.C"
original_IGM_file_name = "eigen-original.C"
original_IGM_file_path = os.path.join(IGM_src_dir_path,original_IGM_file_name)

# Then download the original IllinoisGRMHD source code
# We try it here in a couple of ways in an attempt to keep
# the code more portable
try:
    original_IGM_file_code = urllib.request.urlopen(original_IGM_file_url).read()
    # Write down the file the original IllinoisGRMHD source code
    with open(original_IGM_file_path,"w") as file:
        file.write(original_IGM_file_code)
except:
    try:
        original_IGM_file_code = urllib.urlopen(original_IGM_file_url).read()
        # Write down the file the original IllinoisGRMHD source code
        with open(original_IGM_file_path,"w") as file:
            file.write(original_IGM_file_code)
    except:
        # If all else fails, hope wget does the job
        !wget -O $original_IGM_file_path $original_IGM_file_url

# Perform validation
Validation__eigen__C  = !diff $original_IGM_file_path $outfile_path__eigen__C

if Validation__eigen__C == []:
    # If the validation passes, we do not need to store the original IGM source code file
    !rm $original_IGM_file_path
    print("Validation test for eigen.C: PASSED!")
else:
    # If the validation fails, we keep the original IGM source code file
    print("Validation test for eigen.C: FAILED!")
    # We also print out the difference between the code generated
    # in this tutorial module and the original IGM source code
    print("Diff:")
    for diff_line in Validation__eigen__C:
        print(diff_line)
```

    Validation test for eigen.C: PASSED!

# Step 4: Output this module to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-IllinoisGRMHD__eigen.pdf](Tutorial-IllinoisGRMHD__eigen.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means).

```python
latex_nrpy_style_path = os.path.join(nrpy_dir_path,"latex_nrpy_style.tplx")
#!jupyter nbconvert --to latex --template $latex_nrpy_style_path Tutorial-IllinoisGRMHD__eigen.ipynb
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__eigen.tex
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__eigen.tex
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__eigen.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```


# Test against HMcode

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from scipy.interpolate import InterpolatedUnivariateSpline as spline
from pathlib import Path

%load_ext autoreload
%autoreload 2
```

## Run HMcode

First, we run HMCode. We use `ihm=2`, which is the simple halo model calculation (needs to be set in-code and recompiled). The precise version (git hash) we use is `a53bb0ea574bc19d45467e6847cb0753eca49e93` (Nov 6 2019).

Line numbers / function names used throughout refer to this version.

```python
hmcode_dir = Path("/home/steven/Documents/Projects/halos/HALOMOD/other-codes/HMcode")
```

```python
def read_power(fname):
    # Each column is the power at a different redshift.
    with open(fname, 'r') as fl:
        line = fl.readline().split("#####")[-1].split('        ')[1:]
        redshifts = [float(x) for x in line]

    data = np.genfromtxt(fname, skip_header=1)
    k = data[:, 0]
    return k, redshifts, data[:, 1:]
```

```python
k, redshifts, data = read_power(hmcode_dir / 'power.dat')
```

## Make halomod model

```python
from halomod import DMHaloModel
```

```python
hm = DMHaloModel(
    exclusion_model=None,
    sd_bias_model=None,
    transfer_model="EH_BAO",
    cosmo_params={
        "Tcmb0":2.725,  # Line 596
        'Om0': 0.3,     # Line 587
        'Ob0': 0.05,    # Line 589
        'H0': 70.0      # Line 591
    },
    hc_spectrum="linear",
    halo_concentration_model="Bullock01",
    halo_concentration_params={"K":4, "F":0.01}, # Line 376
    hmf_model="SMT",
    sigma_8 = 0.8,      # Line 593
    n = 0.96,           # Line 594
    Mmin=2,             # Line 795
    Mmax=18,            # Line 796,
    lnk_min = np.log(1e-3),        # Line 50
    lnk_max = np.log(1e2),         # Line 51
    dlnk = np.log(1e2/1e-3) / 129,  # Line 52
    dlog10m = 16 / 256,
    mdef_model='SOMean',
    disable_mass_conversion=True
)
```

### The big picture (1h+2h)

```python
fig,ax = plt.subplots(2,1,sharex=True,subplot_kw={"xscale":'log',"xlim":(1e-3,1e4)}, figsize=(13, 7))
ax[0].plot(hm.k,hm.power_auto_matter * hm.k**3/ (2*np.pi**2),label="tot")
ax[0].plot(hm.k,hm.power_1h_auto_matter * hm.k**3/ (2*np.pi**2),label='1h')
ax[0].plot(hm.k,hm.power_2h_auto_matter * hm.k**3/ (2*np.pi**2),label="2h")
ax[0].plot(k,data[:,0],label="hmcode")
ax[0].set_yscale('log')

ax[0].set_ylim((1e-16,1e5))
ax[0].legend(loc=0)

spl = spline(np.log(hm.k),np.log(hm.power_auto_matter*hm.k**3/(2*np.pi**2)))
ax[1].plot(k,np.exp(spl(np.log(k)))/data[:,0] -1)
ax[1].grid(True)
```

## Intermediate Products

Since we are using the forked version of HMCode which prints out intermediate results, we can compare.

```python
hmc_intermediate = np.genfromtxt("/home/steven/Documents/Projects/halos/HALOMOD/other-codes/HMcode/mass_data.dat")
```

```python
m = hmc_intermediate[0]
```

```python
def get(thing, iz=0):
    things = ['rv', 'nu', 'rr', 'sig', 'sigf', 'zc', 'c', 'gnu']
    indx = iz * len(things) + 1 + things.index(thing)
    return hmc_intermediate[indx]
```

### Mass to Radius

```python
# Redshift 0
r = spline(hm.m, hm.radii)(m)
plt.plot(m, r / get('rr')-1)
plt.xscale('log')
```

### sigma

```python
sig = spline(hm.m, hm.sigma)(m)
plt.plot(m, sig/get('sig') -1)
# There's an "rsplit" parameter of 1e-2 in HMcode where the integral is treated differently
# Maybe it corresponds to the break?
plt.axvline(m[np.where(r > 1e-2)[0][0]])
plt.xscale('log')
```

We see that $\sigma$ is quite different at low masses. HMCode uses a "rapidising" function (Line 1839) to make the integral faster -- potentially that is causing problems at low mass. On the other hand, maybe hmf is doing the wrong thing?

If we update the k-bounds for hmf (the thing we're integrating over here), let's see what happens:

```python
hm.update(lnk_max=np.log(1e5))
```

```python
sig = spline(hm.m, hm.sigma)(m)
plt.plot(m, sig/get('sig') -1)
# There's an "rsplit" parameter of 1e-2 in HMcode where the integral is treated differently
# Maybe it corresponds to the break?
plt.axvline(m[np.where(r > 1e-2)[0][0]])
plt.xscale('log')
```

This fixes the issue. So even though HMCode seems to be able to get away with using a smaller k-range, hmf cannot.

### nu

```python
nu = np.sqrt(spline(hm.m, hm.nu)(m))
plt.plot(m, nu/get('nu')-1)
plt.xscale('log')
```

### Growth Function

```python
hmc_growth = np.genfromtxt("/home/steven/Documents/Projects/halos/HALOMOD/other-codes/HMcode/growth_data.dat")
```

```python
zz = 1/hmc_growth[:,0] -1
```

```python
hm_growth_fn = hm.growth.growth_factor_fn()
```

```python
hm.update(growth_params={"dlna": 0.01, "amin": 1e-12})
```

```python
from scipy.integrate import quad
```

```python
intg = lambda z: (1 + z)/hm.cosmo.H(z).value**3
```

```python
g0 = quad(intg, 0, np.inf)[0]
```

```python
growths = []
for i, z in enumerate(zz):
    growths.append(quad(intg, z, np.inf)[0] * hm.cosmo.H(z).value/(g0 * hm.cosmo.H(0).value))
```

```python
hm_growths = np.array([hm.growth.growth_factor(z) for z in zz])
```

```python
plt.plot(zz, hm_growths/hmc_growth[:, 1] -1)
#plt.plot(zz, [hm.growth.growth_factor(z) for z in zz])
#plt.plot(zz, growths)
plt.xscale('log')
#plt.yscale('log')
```

Though there is some discrepancy for large redshifts (>100), the discrepancy seems to be in favour of `hmf` (when compared to pure quadrature integration) and anyway, discrepancy is well under 5% for redshifts actually used (as collapse redshift).

```python
plt.plot(zz, hm_growths/hmc_growth[:, 1] -1)
#plt.plot(zz, [hm.growth.growth_factor(z) for z in zz])
#plt.plot(zz, growths)
plt.xscale('log')
plt.xlim(1e-2, 10)
plt.ylim(-0.02, 0.02)
#plt.yscale('log')
```

### Sig-f

```python
nuf = 1.686 / get('sigf')
r = hm.halo_concentration.filter.mass_to_radius(hm.halo_concentration.params["F"] * m, hm.halo_concentration.mean_density0)
hm_nu = hm.halo_concentration.filter.nu(r, 1.686)

hm_nu = np.sqrt(spline(hm.m, hm_nu)(m))
```

```python
plt.plot(m, hm_nu/nuf - 1)
plt.xscale('log')
```

### Collapse Redshift

```python
zc = spline(hm.m, hm.halo_concentration.zc(hm.m))(m)
print("Maximum collapse redshift: ", zc.max())
plt.plot(m, zc/get('zc')-1)
plt.xscale('log')
plt.ylim(-0.01,0.01)
```

### Concentration

```python
c = spline(hm.m, hm.cmz_relation)(m)
plt.plot(m, c/get('c')-1)
plt.xscale('log')
```

### 2-halo

The 2-halo term in HMcode, with imead=0, is just the linear power, which means we should get it exactly if our transfer function is correct, and normalisation as well.

```python
k, _, data_lin = read_power(hmcode_dir / 'power_linear.dat')
```

```python
spl = spline(hm.k,hm.delta_k)
plt.plot(k, np.abs(spl(k)/data_lin[:,0] -1) )

plt.xscale('log')
plt.yscale('log')
```

I think we can be pretty confident that our linear power spectrum is lining up, to within 0.06%

```python
k, _, data_2h = read_power(hmcode_dir / 'power_2halo.dat')
```

```python
spl = spline(hm.k, hm.power_2h_auto_matter * hm.k**3 / (2*np.pi**2))
plt.plot(k,np.abs(spl(k)/data_2h[:, 0] -1 ) )
plt.xscale('log')
plt.yscale('log')
```

As we would hope, this is precisely the same plot as for the linear power. There are some weird things around the scale of the BAO peak, which may even come from our spline interpolation, but things are pretty close overall.

### Mass Function

```python
gnu = spline(hm.m, hm.fsigma / np.sqrt(hm.nu))(m)
plt.plot(m, gnu /get('gnu') - 1)
plt.xscale('log')
plt.ylim(-.01,.01)
```

### Virial Radius

```python
plt.plot(hm.m, hm.halo_profile._halo_mass_to_radius(m)/get('rv') - 1)
plt.xscale('log')
```

### Halo Profile (u)

```python
ukm = np.genfromtxt(hmcode_dir / 'ukm.dat')
with open(hmcode_dir / '1h_integrand.dat') as fl:
    kk = float(fl.readline().split('=')[-1].strip())

```

```python
hm_ukm = hm.halo_profile.u(kk, m)
plt.plot(m, hm_ukm/ukm[0] - 1)
plt.xscale('log')
```

### 1-halo

```python
k, _, data_1h = read_power(hmcode_dir / 'power_1halo.dat')
```

Now, the way that HMCode is written is a bit confusing on the face of it, but that's because it integrates over $\nu$ instead of $m$ directly. It turns out this is actually a little easier than integrating over $m$. Here's the math for posterity:

An integral of any function over mass with a factor of the mass function can be written:

\begin{equation}
    I = \int \frac{dn}{dm} g(m) dm.
\end{equation}

Now,

\begin{align}
    \frac{dn}{dm} &= - \frac{\bar{\rho}}{m^2} \nu f(\nu) \frac{d\ln\sigma}{d\ln m} \\
    &= - \frac{\bar{\rho}}{m \sigma} \nu f(\nu) \frac{d\sigma}{d m} \\
    &= - \frac{\bar{\rho}}{m \delta_c} \nu^2 f(\nu) \frac{d \sigma}{d\nu} \frac{d \nu}{dm}
    &= \frac{\bar{\rho}}{m} f(\nu) \frac{d \nu}{dm}
\end{align}

So we have

\begin{equation}
    I = \int \frac{\bar{\rho}}{m} f(\nu) g(m) d\nu.
\end{equation}

```python
spl = spline(hm.k,hm.power_1h_auto_matter * hm.k**3 / (2*np.pi**2))
plt.plot(k,np.abs(spl(k)/data_1h[:, 0]-1))

plt.xscale('log')
plt.yscale('log')
```

There is a 60% difference at small scales here... !!??

### Full Picture (with increased k range)

```python
fig,ax = plt.subplots(2,1,sharex=True,subplot_kw={"xscale":'log',"xlim":(1e-3,1e4)}, figsize=(13, 7))
ax[0].plot(hm.k,hm.power_auto_matter * hm.k**3/ (2*np.pi**2),label="tot")
ax[0].plot(hm.k,hm.power_1h_auto_matter * hm.k**3/ (2*np.pi**2),label='1h')
ax[0].plot(hm.k,hm.power_2h_auto_matter * hm.k**3/ (2*np.pi**2),label="2h")
ax[0].plot(k,data[:,0],label="hmcode")
ax[0].set_yscale('log')

ax[0].set_ylim((1e-16,1e5))
ax[0].legend(loc=0)

spl = spline(np.log(hm.k),np.log(hm.power_auto_matter*hm.k**3/(2*np.pi**2)))
ax[1].plot(k,np.exp(spl(np.log(k)))/data[:,0] -1)
ax[1].grid(True)
```

## High Redshift

```python
hm.update(z=4)

fig,ax = plt.subplots(2,1,sharex=True,subplot_kw={"xscale":'log',"xlim":(1e-3,1e4)}, figsize=(13, 7))
ax[0].plot(hm.k,hm.power_auto_matter * hm.k**3/ (2*np.pi**2),label="tot")
ax[0].plot(hm.k,hm.power_1h_auto_matter * hm.k**3/ (2*np.pi**2),label='1h')
ax[0].plot(hm.k,hm.power_2h_auto_matter * hm.k**3/ (2*np.pi**2),label="2h")

ax[0].plot(k,data[:,-1],label="hmcode")
ax[0].set_yscale('log')

ax[0].set_ylim((1e-16,1e5))
ax[0].legend(loc=0)

spl = spline(np.log(hm.k),np.log(hm.power_auto_matter*hm.k**3/(2*np.pi**2)))
ax[1].plot(k,np.exp(spl(np.log(k)))/data[:,-1] -1)
ax[1].grid(True)
```

Seems to be just the 1-halo term which hasn't evolved properly. Let's look at the bits again.

### Radii

```python
r = spline(hm.m, hm.radii)(m)
plt.plot(m, r / get('rr', iz=15)-1)
plt.xscale('log')
```

### Sigma

```python
sig = spline(hm.m, hm.sigma)(m)
plt.plot(m, sig/get('sig',iz=15) -1)
plt.xscale('log')
```

This is obviously off by a fraction of a percent... not sure if that's worth worrying about.

### nu

```python
nu = np.sqrt(spline(hm.m, hm.nu)(m))
plt.plot(m, nu/get('nu', iz=15)-1)
plt.xscale('log')
```

### Sig-f

```python
nuf = 1.686 / get('sigf', iz=15)
r = hm.halo_concentration.filter.mass_to_radius(hm.halo_concentration.params["F"] * m, hm.halo_concentration.mean_density0)
hm_nu = hm.halo_concentration.filter.nu(r, 1.686) / hm.growth_factor**2

hm_nu = np.sqrt(spline(hm.m, hm_nu)(m))
```

```python
plt.plot(m, hm_nu)
plt.plot(m, nuf)
plt.xscale('log')
```

```python
plt.plot(m, hm_nu/nuf - 1)
plt.xscale('log')
```

### Collapse Redshift

```python
zc = hm.halo_concentration.zc(m, z=4)
print("Maximum collapse redshift: ", zc.max())
plt.plot(m, zc/get('zc',iz=15)-1)
plt.xscale('log')
#plt.ylim(-0.05,0.05)
```

### Concentration

```python
c = spline(hm.m, hm.cmz_relation)(m)
plt.plot(m, c/get('c', iz=15)-1)
plt.xscale('log')
```

### 2-halo

```python
spl = spline(hm.k, hm.power_2h_auto_matter * hm.k**3 / (2*np.pi**2))
plt.plot(k,np.abs(spl(k)/data_2h[:, -1] -1 ) )
plt.xscale('log')
plt.yscale('log')
```

### Mass Function

```python
gnu = spline(hm.m, hm.fsigma / np.sqrt(hm.nu))(m)
plt.plot(m, gnu /get('gnu',iz=15) - 1)
plt.xscale('log')
plt.ylim(-.05,.05)
```

The difference here seems to be due to slight differences in $\nu$ (of 0.5%) which get blown up in the exponential of the mass function:

\begin{equation}
 \exp(-q \nu^2 / 2)/\exp(-q \nu^2 (1 + \delta)^2 / 2) \approx \exp(-q \nu^2 / 2 ( 1 - (1 + \delta)^2)) \approx \exp(-q \nu^2 \delta ).
\end{equation}

With $\delta \approx 0.005$ and $q = 0.707$, $\nu$ does not need to be very large before differences of a few percent will arise.

On the other hand, at these higher redshifts, we don't expect these larger masses to contribute very significantly to the 1-halo integral.

### Virial Radius

```python
plt.plot(hm.m, hm.halo_profile._halo_mass_to_radius(m)/get('rv',iz=15) - 1)
plt.xscale('log')
```

### Ukm

```python
hm_ukm = hm.halo_profile.u(kk, m)
plt.plot(m, hm_ukm/ukm[-1] - 1)
plt.xscale('log')
```

### 1-halo

```python
spl = spline(hm.k,hm.power_1h_auto_matter * hm.k**3 / (2*np.pi**2))
plt.plot(k,np.abs(spl(k)/data_1h[:, -1]-1))

plt.xscale('log')
plt.yscale('log')
```


# Tutorial 2: Convolutional Neural Networks
**Week 2, Day 1: Deep Learning**

**By Neuromatch Academy**

**Content creators**: Jorge A. Menendez, Carsen Stringer

**Content reviewers**: Roozbeh Farhoodi,  Madineh Sarvestani, Kshitij Dwivedi, Spiros Chavlis, Ella Batty, Michael Waskom

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

*Estimated timing of tutorial: 40 minutes*

In this short tutorial, we'll go through an introduction to 2D convolutions and apply a convolutional network to an image to prepare for creating normative models in tutorial 3.

In this tutorial, we will
* Understand the basics of 2D convolution
* Build a convolutional layer using PyTorch
* Visualize and analyze its outputs

```python
# @title Tutorial slides

# @markdown These are the slides for all videos in this tutorial.
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/s59jy/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
# Imports

import os
import numpy as np
import torch
from torch import nn
from torch import optim
from matplotlib import pyplot as plt
import matplotlib as mpl
```

```python
#@title Figure settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Data retrieval and loading

import hashlib
import requests

fname = "W3D4_stringer_oribinned6_split.npz"
url = "https://osf.io/p3aeb/download"
expected_md5 = "b3f7245c6221234a676b71a1f43c3bb5"

if not os.path.isfile(fname):
  try:
    r = requests.get(url)
  except requests.ConnectionError:
    print("!!! Failed to download data !!!")
  else:
    if r.status_code != requests.codes.ok:
      print("!!! Failed to download data !!!")
    elif hashlib.md5(r.content).hexdigest() != expected_md5:
      print("!!! Data download appears corrupted !!!")
    else:
      with open(fname, "wb") as fid:
        fid.write(r.content)
```

```python
# @title Plotting Functions

def show_stimulus(img, ax=None):
  """Visualize a stimulus"""
  if ax is None:
    ax = plt.gca()
  ax.imshow(img+0.5, cmap=mpl.cm.binary)
  ax.set_xticks([])
  ax.set_yticks([])
  ax.spines['left'].set_visible(False)
  ax.spines['bottom'].set_visible(False)

def plot_weights(weights, channels=[0], colorbar=True):
  """ plot convolutional channel weights
  Args:
        weights: weights of convolutional filters (conv_channels x K x K)
        channels: which conv channels to plot
  """
  wmax = torch.abs(weights).max()
  fig, axs = plt.subplots(1,len(channels), figsize=(12,2.5))
  for i, channel in enumerate(channels):
    im = axs[i].imshow(weights[channel,0], vmin=-wmax, vmax=wmax, cmap='bwr')
    axs[i].set_title('channel %d'%channel)

  if colorbar:
    ax = fig.add_axes([1, 0.1, 0.05, 0.8])
    plt.colorbar(im, ax=ax)
    ax.axis('off')

def plot_example_activations(stimuli, act, channels=[0]):
  """ plot activations act and corresponding stimulus
  Args:
        stimuli: stimulus input to convolutional layer (n x h x w) or (h x w)
        act: activations of convolutional layer (n_bins x conv_channels x n_bins)
        channels: which conv channels to plot
  """
  if stimuli.ndim>2:
    n_stimuli = stimuli.shape[0]
  else:
    stimuli = stimuli.unsqueeze(0)
    n_stimuli = 1

  fig, axs = plt.subplots(n_stimuli,1+len(channels),figsize=(12,12))

  # plot stimulus
  for i in range(n_stimuli):
    show_stimulus(stimuli[i].squeeze(), ax=axs[i, 0])
    axs[i, 0].set_title('stimulus')

    # plot example activations
    for k, (channel, ax) in enumerate(zip(channels, axs[i][1:])):
      img=ax.imshow(act[i,channel], vmin=-3, vmax=3, cmap='bwr')
      ax.set_xlabel('x-pos')
      ax.set_ylabel('y-pos')
      ax.set_title('channel %d'%channel)
  ax = fig.add_axes([1.05,0.8,0.01,0.1])
  plt.colorbar(img, cax=ax)
  ax.set_title('activation\n strength')
```

```python
# @title Helper Functions

def load_data_split(data_name=fname):
  """Load mouse V1 data from Stringer et al. (2019)

  Data from study reported in this preprint:
  https://www.biorxiv.org/content/10.1101/679324v2.abstract

  These data comprise time-averaged responses of ~20,000 neurons
  to ~4,000 stimulus gratings of different orientations, recorded
  through Calcium imaginge. The responses have been normalized by
  spontaneous levels of activity and then z-scored over stimuli, so
  expect negative numbers. The repsonses were split into train and
  test and then each set were averaged in bins of 6 degrees.

  This function returns the relevant data (neural responses and
  stimulus orientations) in a torch.Tensor of data type torch.float32
  in order to match the default data type for nn.Parameters in
  Google Colab.

  It will hold out some of the trials when averaging to allow us to have test
  tuning curves.

  Args:
    data_name (str): filename to load

  Returns:
    resp_train (torch.Tensor): n_stimuli x n_neurons matrix of neural responses,
        each row contains the responses of each neuron to a given stimulus.
        As mentioned above, neural "response" is actually an average over
        responses to stimuli with similar angles falling within specified bins.
    resp_test (torch.Tensor): n_stimuli x n_neurons matrix of neural responses,
        each row contains the responses of each neuron to a given stimulus.
        As mentioned above, neural "response" is actually an average over
        responses to stimuli with similar angles falling within specified bins
    stimuli: (torch.Tensor): n_stimuli x 1 column vector with orientation
        of each stimulus, in degrees. This is actually the mean orientation
        of all stimuli in each bin.

  """
  with np.load(data_name) as dobj:
    data = dict(**dobj)
  resp_train = data['resp_train']
  resp_test = data['resp_test']
  stimuli = data['stimuli']

  # Return as torch.Tensor
  resp_train_tensor = torch.tensor(resp_train, dtype=torch.float32)
  resp_test_tensor = torch.tensor(resp_test, dtype=torch.float32)
  stimuli_tensor = torch.tensor(stimuli, dtype=torch.float32)

  return resp_train_tensor, resp_test_tensor, stimuli_tensor

def filters(out_channels=6, K=7):
  """ make example filters, some center-surround and gabors
  Returns:
      filters: out_channels x K x K
  """
  grid = np.linspace(-K/2, K/2, K).astype(np.float32)
  xx,yy = np.meshgrid(grid, grid, indexing='ij')

  # create center-surround filters
  sigma = 1.1
  gaussian = np.exp(-(xx**2 + yy**2)**0.5/(2*sigma**2))
  wide_gaussian = np.exp(-(xx**2 + yy**2)**0.5/(2*(sigma*2)**2))
  center_surround = gaussian - 0.5 * wide_gaussian

  # create gabor filters
  thetas = np.linspace(0, 180, out_channels-2+1)[:-1] * np.pi/180
  gabors = np.zeros((len(thetas), K, K), np.float32)
  lam = 10
  phi = np.pi/2
  gaussian = np.exp(-(xx**2 + yy**2)**0.5/(2*(sigma*0.4)**2))
  for i,theta in enumerate(thetas):
    x = xx*np.cos(theta) + yy*np.sin(theta)
    gabors[i] = gaussian * np.cos(2*np.pi*x/lam + phi)

  filters = np.concatenate((center_surround[np.newaxis,:,:],
                            -1*center_surround[np.newaxis,:,:],
                            gabors),
                           axis=0)
  filters /= np.abs(filters).max(axis=(1,2))[:,np.newaxis,np.newaxis]
  filters -= filters.mean(axis=(1,2))[:,np.newaxis,np.newaxis]
  # convert to torch
  filters = torch.from_numpy(filters)
  # add channel axis
  filters = filters.unsqueeze(1)

  return filters

def grating(angle, sf=1 / 28, res=0.1, patch=False):
  """Generate oriented grating stimulus

  Args:
    angle (float): orientation of grating (angle from vertical), in degrees
    sf (float): controls spatial frequency of the grating
    res (float): resolution of image. Smaller values will make the image
      smaller in terms of pixels. res=1.0 corresponds to 640 x 480 pixels.
    patch (boolean): set to True to make the grating a localized
      patch on the left side of the image. If False, then the
      grating occupies the full image.

  Returns:
    torch.Tensor: (res * 480) x (res * 640) pixel oriented grating image

  """

  angle = np.deg2rad(angle)  # transform to radians

  wpix, hpix = 640, 480  # width and height of image in pixels for res=1.0

  xx, yy = np.meshgrid(sf * np.arange(0, wpix * res) / res, sf * np.arange(0, hpix * res) / res)

  if patch:
    gratings = np.cos(xx * np.cos(angle + .1) + yy * np.sin(angle + .1))  # phase shift to make it better fit within patch
    gratings[gratings < 0] = 0
    gratings[gratings > 0] = 1
    xcent = gratings.shape[1] * .75
    ycent = gratings.shape[0] / 2
    xxc, yyc = np.meshgrid(np.arange(0, gratings.shape[1]), np.arange(0, gratings.shape[0]))
    icirc = ((xxc - xcent) ** 2 + (yyc - ycent) ** 2) ** 0.5 < wpix / 3 / 2 * res
    gratings[~icirc] = 0.5

  else:
    gratings = np.cos(xx * np.cos(angle) + yy * np.sin(angle))
    gratings[gratings < 0] = 0
    gratings[gratings > 0] = 1

  gratings -= 0.5

  # Return torch tensor
  return torch.tensor(gratings, dtype=torch.float32)
```

---
# Section 1: Introduction to 2D convolutions

## Section 1.1: What is a 2D convolution?

A 2D convolution is an integral of the product of a filter $f$ and an input image $I$ computed at various positions as the filter is slid across the input. The output of the convolution operation at position $(x,y)$ can be written as follows, where the filter $f$ is size $(K,K)$:
\begin{equation}
C(x,y) = \sum_{k_x=-K/2}^{K/2} \sum_{k_y=-K/2}^{K/2} f(k_x,k_y) I(x+k_x,y+k_y)
\end{equation}
This **convolutional filter** is often called a **kernel**.

Here is an illustration of a 2D convolution from this [article](https://towardsdatascience.com/light-on-math-machine-learning-intuitive-guide-to-convolution-neural-networks-e3f054dd5daa):

```python
#@title
#@markdown Execute this cell to view convolution gif

from IPython.display import Image
Image(url='https://miro.medium.com/max/700/1*5BwZUqAqFFP5f3wKYQ6wJg.gif')
```

## Section 1.2: 2D convolutions in deep learning

*Estimated timing to here from start of tutorial: 6 min*

```python
# @title Video 1: 2D Convolutions

from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1jw411d7Kg", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="zgO9rHYbDxE", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers convolutions and how to implement them in Pytorch.

 Click here for text recap of video

Recall Aude Oliva’s discussion of convolutions in the [intro](https://www.youtube.com/watch?v=IZvcy0Myb3M). Convolutional neural networks with several layers revolutionized the deep learning field, and in particular AlexNet, depicted here, was the first deep neural network to excel on the ImageNet classification task. The first layer in the network takes as input an image, runs convolutional filters on the image, rectifies the output, them downsamples the output (the pooling layer). The next layers repeat this process, and then at the end fully connected linear layers are attached which output a label for the image.

The main advantages of convolutional layers over fully connected layers are the reduction in parameters through weight-sharing, which we will get into shortly, and also the fact that the units have local receptive fields. These local receptive fields allow the network to pool over units in spatial proximity and helps the network learn translation invariant representations.

A convolution is the integral of the product of two functions, one of which is a stimulus and the other which is a filter. This integral is computed at all positions by sliding the filter weights across the stimulus. If you want to perform a convolution and get the same output as the input you need to pad the input by half the filter size on each side. This is called “same” padding. Another parameter of this convolution computation is the stride -- how often the convolution is computed along the stimulus dimension. In this case, we use a stride of 1, but we can increase the stride and in turn have fewer units.

All the units of this filter are called a single output **channel**. A convolutional layer often consists of multiple output channels each with their own filter weights. We call the number of output convolutional channels $C_{out}$.

We will implement this convolutional layer in pytorch. We will create a convolutional layer `ConvolutionalLayer` which takes as input a stimulus, which in our case are the gratings images. The convolutional layer is initialized with a few different parameters - first is the # of input channels $C_{in}$, which is 1 in our case. Next the number of convolutional channels which we’ll call $C_{out}$ which we can set to 6. Then the size of the filter $K$ which we set by default to 7. There’s also an optional `filters` input which we use to initialize the convolutional weights.  We set them as the weights of the conv layer we just created, and set the bias terms for the conv layer to zero.

We declare an `nn.conv2d` variable to be an attribute of the class `ConvolutionalLayer` called `conv`. For this convolutional layer we set the padding to half the filter size and the stride to 1 to get the same size output as the input.

What are some example filters we might use? One is a center-surround filter. It is positive in the middle and negative in the surround. Another is a gabor filter, which has a positive section next to a negative section. Look at the responses of these filters given the image. Both of these filter types are inspired by neurons recorded in the brain.

In fact, convolutional neural networks are inspired by the brain. In the retina there are a variety of cell types which we can think of as filters and each of these cells tile the entire visual space. The picture here shows the part of visual space each one of these cells responds to. In barrel cortex we have a similar situation where each whisker’s activation corresponds to a single cortical column of activity and the functions computed in each of these columns are similar.

Object recognition was essentially an unsolved problem in machine learning until the [advent](https://en.wikipedia.org/wiki/AlexNet) of techniques for effectively training *deep* convolutional neural networks. See Bonus Section 1 for more information on why we use CNNs to model the visual system.

Convolutional neural networks consist of 2D convolutional layers, RELU non-linearities, 2D pooling layers, and at the output, a fully connected layer. We will see an example network with all these components in tutorial 3.

A 2D convolutional layer has multiple output channels. Each output **channel** is the result of a 2D convolutional filter applied to the input. In the gif below, the input is in blue, the filter is in gray, and the output is in green. The number of units in the output channel depends on the *stride* you set. In the gif below, the stride is 1 because the input image is sampled at each position, a stride of 2 would mean skipping over input positions. In most applications, especially with small filter sizes, a stride of 1 is used.

(*Technical note*: if filter size *K* is odd and you set the *pad=K//2* and *stride=1* (as is shown below), you get a **channel** of units that is the same size as the input. See a more detailed explanation of strides and pads [here](https://theano-pymc.readthedocs.io/en/latest/tutorial/conv_arithmetic.html) if interested).

```python
#@title
#@markdown Execute this cell to view convolution gif

from IPython.display import Image
Image(url='https://miro.medium.com/max/790/1*1okwhewf5KCtIPaFib4XaA.gif')
```

## Section 1.3: 2D convolutions in Pytorch

*Estimated timing to here from start of tutorial: 18 min*

In Tutorial 1, fully connected linear layers were used to decode stimuli from neural activity. Convolutional layers are different from their fully connected counterparts in two ways:
  * In a fully connected layer, each unit computes a weighted sum over all the input units. In a convolutional layer, on the other hand, each unit computes a weighted sum over only a small patch of the input, referred to as the unit's **receptive field**. When the input is an image, the receptive field can be thought of as a local patch of pixels.
  * In a fully connected layer, each unit uses its own independent set of weights to compute the weighted sum. In a convolutional layer, all the units (within the same channel) **share the same weights**. This set of shared weights is called the **convolutional filter or kernel**. The result of this computation is a convolution, where each unit has computed the same weighted sum over a different part of the input. This reduces the number of parameters in the network substantially.

We will compute the difference in the number of weights for a fully connected layer versus a convolutional layer in the Think exercise below.

First, let's visualize the stimuli in the dataset from tutorial 1. During the neural recordings from [Stringer et al 2019](https://www.biorxiv.org/content/10.1101/679324v2.abstract), mice were presented oriented gratings:

```python
# @markdown Execute this cell to plot example stimuli

orientations = np.linspace(-90, 90, 5)

h = 3
n_col = len(orientations)
fig, axs = plt.subplots(1, n_col, figsize=(h * n_col, h))

h, w  = grating(0).shape  # height and width of stimulus
print('stimulus size: %i x %i' % (h, w))

for i, ori in enumerate(orientations):
  stimulus = grating(ori)
  axs[i].set_title(f'{ori: .0f}$^o$')
  show_stimulus(stimulus, axs[i])
```

Now let's implement 2D convolutional operations. We will use multiple convolutional channels and implement this operation efficiently using pytorch. A *layer* of convolutional channels can be implemented with one line of code using the PyTorch class `nn.Conv2d()`, which requires the following arguments for initialization (see full documentation [here](https://pytorch.org/docs/master/generated/torch.nn.Conv2d.html)):
  * $C^{in}$: the number of input channels
  * $C^{out}$: the number of output channels (number of different convolutional filters)
  * $K$: the size of the $C^{out}$ different convolutional filters

When you run the network, you can input a stimulus of arbitrary size $(H^{in}, W^{in})$, but it needs to be shaped as a 4D input $(N, C^{in}, H^{in}, W^{in})$, where $N$ is the number of images. In our case, $C^{in}=1$ because there is only one color channel (our images are grayscale, but often $C^{in}=3$ in image processing).

```python
class ConvolutionalLayer(nn.Module):
  """Deep network with one convolutional layer
     Attributes: conv (nn.Conv2d): convolutional layer
  """
  def __init__(self, c_in=1, c_out=6, K=7, filters=None):
    """Initialize layer

    Args:
        c_in: number of input stimulus channels
        c_out: number of output convolutional channels
        K: size of each convolutional filter
        filters: (optional) initialize the convolutional weights

    """
    super().__init__()
    self.conv = nn.Conv2d(c_in, c_out, kernel_size=K,
                          padding=K//2, stride=1)
    if filters is not None:
      self.conv.weight = nn.Parameter(filters)
      self.conv.bias = nn.Parameter(torch.zeros((c_out,), dtype=torch.float32))

  def forward(self, s):
    """Run stimulus through convolutional layer

    Args:
        s (torch.Tensor): n_stimuli x c_in x h x w tensor with stimuli

    Returns:
        (torch.Tensor): n_stimuli x c_out x h x w tensor with convolutional layer unit activations.

    """
    a = self.conv(s)  # output of convolutional layer

    return a
```

See that `ConvolutionalLayer` takes as input `filters`. We have predesigned some filters that you can use by calling the `filters` function below. These are similar to filters we think are implemented in biological circuits such as the retina and the visual cortex. Some of them are **center-surround** filters and some of them are **gabor** filters. Check out this [website](http://www.cns.nyu.edu/~david/courses/perception/lecturenotes/ganglion/ganglion.html) for more details on center-surround filters, and this [website](https://en.wikipedia.org/wiki/Gabor_filter) for more details on gabor filters, if you're interested.

```python
# @markdown Execute this cell to create and visualize filters

example_filters = filters(out_channels=6, K=7)

plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plt.imshow(example_filters[0,0], vmin=-1, vmax=1, cmap='bwr')
plt.title('center-surround filter')
plt.axis('off')
plt.subplot(1,2,2)
plt.imshow(example_filters[4,0], vmin=-1, vmax=1, cmap='bwr')
plt.title('gabor filter')
plt.axis('off');
```

### Coding Exercise 1.3: 2D convolution in pytorch

We will now run the convolutional layer on our stimulus. We will use gratings stimuli made using the function `grating`, which returns a stimulus which is 48 x 64.

Reminder, `nn.Conv2d` takes in a tensor of size $(N, C^{in}, H^{in}, W^{in}$) where $N$ is the number of stimuli, $C^{in}$ is the number of input channels, and $(H^{in}, W^{in})$ is the size of the stimulus. We will need to add these first two dimensions to our stimulus, then input it to the convolutional layer.

We will plot the outputs of the convolution. `convout` is a tensor of size $(N, C^{out}, H^{in}, W^{in})$ where $N$ is the number of examples and $C^{out}$ are the number of convolutional channels. It is the same size as the input because we used a stride of 1 and padding that is half the kernel size.

```python
# Stimulus parameters
in_channels = 1 # how many input channels in our images
h = 48 # height of images
w = 64 # width of images

# Convolution layer parameters
K = 7 # filter size
out_channels = 6 # how many convolutional channels to have in our layer
example_filters = filters(out_channels, K) # create filters to use

convout = np.zeros(0) # assign convolutional activations to convout

################################################################################
## TODO for students: create convolutional layer in pytorch
# Complete and uncomment
raise NotImplementedError("Student exercise: create convolutional layer")
################################################################################

# Initialize conv layer and add weights from function filters
# you need to specify :
# * the number of input channels c_in
# * the number of output channels c_out
# * the filter size K
convLayer = ConvolutionalLayer(..., filters=example_filters)

# Create stimuli (H_in, W_in)
orientations = [-90, -45, 0, 45, 90]
stimuli = torch.zeros((len(orientations), in_channels, h, w), dtype=torch.float32)
for i,ori in enumerate(orientations):
  stimuli[i,0] = grating(ori)

convout = convLayer(...)
convout = convout.detach() # detach gradients

plot_example_activations(stimuli, convout, channels=np.arange(0, out_channels))
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D1_DeepLearning/solutions/W2D1_Tutorial2_Solution_712d995b.py)

*Example output:*

### Think! 1.3: Output and weight shapes in a convolutional layer

Let's think about the shape of the weights and outputs of `convLayer`:
  - How many convolutional activations are there in each channel, and why is it this size?
  - How many weights does `convLayer` have?
  - How many weights would it have if it were a fully connected layer?

Additionally, let's think about why the activations look the way they do. It seems like for all channels the activations are only non-zero for edges of the gratings (where the grating goes from white-to-black and from black-to-white).
  -  Channel 0 and 1 seem to respond to every edge regardless of the orientation, but their signs are different. What type of filter might produce these types of responses?
  - Channels 2-5 seem to respond differently depending on the orientation of the stimulus. What type of filter might produce these types of responses?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D1_DeepLearning/solutions/W2D1_Tutorial2_Solution_f48d6ecf.py)

Please see Bonus Section 2 to visualize the convolutional filter weights. See the Bonus Tutorial to use CNNs as encoding models of neurons (by fitting directly to neural responses).

---
# Summary

*Estimated timing of tutorial: 40 minutes*

In this notebook, we built a 2D convolutional layer which is meant to represent the responses of neurons in mouse visual cortex, or the responses of neurons
which are inputs to mouse visual cortex.

In Tutorial 3, we will add to this 2D convolutional layer a fully-connected layer and train this model to predict whether an orientation is left or right. We will see if the convolutional filters it learns are similar to mouse visual cortex. See Section 3 in the Bonus Tutorial to fit convolutional neural networks directly to neural activity.

---
# Bonus

## Bonus Section 1: Why CNN's?

CNN models are particularly [well-suited](https://www.nature.com/articles/nn.4244) to modeling the visual system for a number of reasons:

1. **Distributed computation**: like any other neural network, CNN's use distributed representations to compute -- much like the brain seems to do. Such models, therefore, provide us with a vocabulary with which to talk and think about such distributed representations. Because we know the exact function the model is built to perform (e.g. orientation discrimination), we can analyze its internal representations with respect to this function and begin to interpret why the representations look the way they do. Most importantly, we can then use these insights to analyze the structure of neural representations observed in recorded population activity. We can qualitatively and quantitatively compare the representations we see in the model and in a given population of real neurons to hopefully tease out the computations it performs.

2. **Hierarchical architecture**: like in any other deep learning architecture, each layer of a deep CNN comprises a non-linear transformation of the previous layer. Thus, there is a natural hierarchy whereby layers closer to the network output represent increasingly more abstract information about the input image. For example, in a network trained to do object recognition, the early layers might represent information about edges in the image, whereas later layers closer to the output might represent various object categories. This resembles the [hierarchical structure of the visual system](https://pubmed.ncbi.nlm.nih.gov/1822724/), where [lower-level areas](https://www.jneurosci.org/content/25/46/10577.short) (e.g. retina, V1) represent visual features of the sensory input and [higher-level areas](https://www.sciencedirect.com/science/article/pii/S089662731200092X) (e.g. V4, IT) represent properties of objects in the visual scene. We can then naturally use a single CNN to model multiple visual areas, using early CNN layers to model lower-level visual areas and late CNN layers to model higher-level visual areas.

  Relative to fully connected networks, CNN's, in fact, have further hierarchical structure built-in through the max pooling layers. Recall that each output of a convolution + pooling block is the result of processing a local patch of the inputs to that block. If we stack such blocks in a sequence, then the outputs of each block will be sensitive to increasingly larger regions of the initial raw input to the network: an output from the first block is sensitive to a single patch of these inputs, corresponding to its "receptive field"; an output from the second block is sensitive to a patch of outputs from the first block, which together are sensitive to a larger patch of raw inputs comprising the union of their receptive fields. Receptive fields thus get larger for deeper layers (see [here](http://colah.github.io/posts/2014-07-Conv-Nets-Modular/) for a nice visual depiction of this). This resembles primate visual systems, where neurons in higher-level visual areas respond to stimuli in wider regions of the visual field than neurons in lower-level visual areas.

3. **Convolutional layers**: through the weight sharing constraint, the outputs of each channel of a convolutional layer process different parts of the input image in exactly the same way. This architectural constraint effectively builds into the network the assumption that objects in the world typically look the same regardless of where they are in space. This is useful for modeling the visual system for two (largely separate) reasons:
  * Firstly, this assumption is generally valid in mammalian visual systems, since mammals tend to view the same object from many perspectives. Two neurons at a similar hierarchy in the visual system with different receptive fields could thus end up receiving statistically similar synaptic inputs, so that the synaptic weights developed over time may end up being similar as well.
  * Secondly, this architecture significantly improves object recognition ability. Object recognition was essentially an unsolved problem in machine learning until the [advent](https://en.wikipedia.org/wiki/AlexNet) of techniques for effectively training *deep* convolutional neural networks. Fully connected networks on their own can't achieve object recognition abilities anywhere close to human levels, making them bad models of human object recognition. Indeed, it is generally the case that [the better a neural network model is at object recognition, the closer the match between its representations and those observed in the brain](https://www.pnas.org/content/111/23/8619.short). That said, it is worth noting that our much simpler orientation discrimination task (in Tutorial 3) can be solved by relatively simple networks.

---
## Bonus Section 2: Understanding activations from weight

### Bonus Coding Exercise 2: Visualizing convolutional filter weights

Why do the activations look the way they do? Let's look at the weights (`convLayer.conv.weight.detach()`) of the convolutional filters and try to interpret them.

```python
################################################################################
## TODO for students: get weights
# Complete and uncomment
raise NotImplementedError("Student exercise: get weights")
################################################################################

# get weights of conv layer in convLayer
weights = ...
print(weights.shape) # can you identify what each of the dimensions are?

plot_weights(weights, channels=np.arange(0, out_channels))
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D1_DeepLearning/solutions/W2D1_Tutorial2_Solution_20c66403.py)

*Example output:*

In the function `filters` we premade center-surround filters and Gabor filters of various orientations. [Gabor filters](https://en.wikipedia.org/wiki/Gabor_filter) have a positive region next to a negative region and the orientation of these regions of the filter determine the orientation of edges to which they respond.

In visual cortex, Hubel and Wiesel discovered simple cells, which would correspond to a unit in the channels 2-5 above with Gabor filters. There are also some neurons with activity that resembles center-surround filters, which would correspond to the first two convolutional channels above.

There were additional cells discovered by Hubel and Wiesel - complex cells - that respond to an oriented grating regardless of where the bars are exactly (note that the responses we see are specific to where the bars are). These cells therefore have some level of translation invariance. This is something that convolutional neural networks try to replicate -- e.g. a grating is still oriented horizontal even if it moves slightly, and a cat is still a cat even if it's in a different position in the image.

### Bonus Think! 2
  - How might you create a complex cell and have responses that are translation invariant?
  - How might you create a cell that responds to multiple orientations?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D1_DeepLearning/solutions/W2D1_Tutorial2_Solution_d290c523.py)