# StellarDynamics_HW_Chap03

Sorry, I failed to write a reproducible homework based on programming. QAQ.

3.1

a.
modified from (3.23b), the rariated power is

$$
P = \frac{{2{{\left| {{\mathbf{\ddot m}}} \right|}^2}}}{{3{c^3}}}
$$

Only the component that perpendicular to the rotation axis contributes to the radiation, and the variation of dipole is contributed only by the component perpendicular to the rotation axis,

$$
{\mathbf{m}} \propto {e^{i\omega t}},
$$
so that
$$ \left| {{\mathbf{\ddot m}}} \right| = {\omega ^2}\left| {\mathbf{m}} \right|\sin \alpha $$

Comparing to electric field, we can get a similiar form of relation between field and dipole,

Electric potential:
$$
\phi  = \frac{e}{{\left| {\mathbf{r}} \right|}} - \frac{e}{{\left| {{\mathbf{r}} - {\mathbf{d}}} \right|}} = \frac{{e\left| {{\mathbf{r}} - {\mathbf{d}}} \right| - e\left| {\mathbf{r}} \right|}}{{\left| {\mathbf{r}} \right|\left| {{\mathbf{r}} - {\mathbf{d}}} \right|}} \approx \frac{{ed\cos \theta }}{{{r^2}}}
$$

Note $ed$ is the electric dipole.

$$
{\mathbf{E}} = \nabla \phi  = (\frac{\partial }{{\partial r}}\hat r + \frac{1}{r}\frac{\partial }{{\partial \theta }}\hat \theta  + \frac{1}{{r\sin \theta }}\frac{\partial }{{\partial \phi }}\hat \phi )\frac{{ed\cos \theta }}{{{r^2}}} \\
 =  - 2\frac{{ed\cos \theta }}{{{r^3}}}\hat r - \frac{{ed\sin \theta }}{{{r^3}}}\hat \theta
$$

So that,

$$
{{\mathbf{B}}_{\mathbf{0}}} =  - 2\frac{m}{{{r^3}}}\hat r
$$
and
$$
\left| {\mathbf{m}} \right| = \frac{{{B_0}R_0^3}}{2}
$$

So the answer is
$$
P = \frac{{2{\omega ^4}{{\left( {\frac{{{B_0}R_0^3}}{2}} \right)}^2}{{\sin }^2}\alpha }}{{3{c^3}}} = \frac{{{\omega ^4}B_0^2R_0^6{{\sin }^2}\alpha }}{{6{c^3}}}.
$$

b.

So we can build the connection between rotation energy and radiation power.

$$
P =  - {{\dot E}_{rot}},
$$
with
$$
{E_{rot}} = \frac{1}{5}M{R^2}{\omega ^2}
$$

So that
$$
P = \frac{{{\omega ^4}B_0^2R_0^6{{\sin }^2}\alpha }}{{6{c^3}}} =  - \frac{2}{5}M{R_0}^2\omega \dot \omega  =  - {{\dot E}_{rot}}.
$$

He have
$$
\tau  \equiv \frac{{ - \omega }}{{\dot \omega }} = \frac{{12M{c^3}}}{{5{\omega ^2}B_0^2R_0^4{{\sin }^2}\alpha }}
$$

c.

```python
from scipy import constants as c
import numpy as np
```

```python
def radiation_power(**kwargs):
    '''cgs unit radiation power'''
    return kwargs['w']**4 * kwargs['B']**2 * \
        kwargs['R']**6 * \
        np.sin(kwargs['a'] * np.pi / 180)**2 / \
        (6 * (c.speed_of_light * 100)**3)

values = {'M': 2*10**33.,
          'R': 10**6.,
          'B': 10**12.,
          'a': 90.}
ws = [10**4., 10**3., 10**2.]
Ps = map(lambda w:radiation_power(w=w,
    B=values['B'], R=values['R'], a=values['a']),
    ws)
for p,w in zip(Ps, ws):
    print('w = {:.2g}; P = {:.2g}'.format(w, p))
```

    w = 1e+04; P = 6.2e+43
    w = 1e+03; P = 6.2e+39
    w = 1e+02; P = 6.2e+35

```python
def time_scale(**kwargs):
    '''cgs unit time-scale'''
    return 12 * kwargs['M'] * (c.speed_of_light * 100)**3 /\
    (5 * kwargs['R']**4 * kwargs['w']**2 * \
     kwargs['B']**2 *\
     np.sin(kwargs['a'] * np.pi / 180)**2)

taus = list(map(lambda w:time_scale(w=w,
    M=values['M'], R=values['R'], a=values['a'], B=values['B']),
    ws))
for yr,w in zip((tau / 60 / 60 / 24 / 365 for tau in taus),
               ws):
    print('w = {:.2g}; {:.2g} yr'.format(w, yr))
```

    w = 1e+04; 41 yr
    w = 1e+03; 4.1e+03 yr
    w = 1e+02; 4.1e+05 yr

3.3
a.
Larmor Eq:
$$
{{\mathbf{E}}_{{\mathbf{rad}}}} = \left[ {(q/R{c^2}){\mathbf{n}} \times ({\mathbf{n}} \times {\mathbf{\dot n}})} \right]
$$

With:

$$
u = r\omega \cos \omega t
$$

$$
\dot u =  - r{\omega ^2}\cos \omega t\\
q\dot u =  - d{\omega ^2}\cos \omega t
$$

Larmor Eq becomes:
$$
\left| {{{\mathbf{E}}_{{\mathbf{rad}}}}} \right| =  - \frac{{{\omega ^2}}}{{R{c^2}}}\left[ {{d_1}\cos \omega t + {d_2}\cos (\omega (t - \Delta t))} \right]\sin \theta,
$$

where
$$
\Delta t = (L/c)\sin \theta.
$$

Use the triangular blabla eqaution:
$$
\cos (\omega (t - \Delta t)) = \cos \omega t\cos \delta  + \sin \omega t\sin \delta.
$$

So that:
$$
\left| {{{\mathbf{E}}_{{\mathbf{rad}}}}} \right| =  - \frac{{{\omega ^2}}}{{R{c^2}}}\left[ {{d_1}\cos \omega t + {d_2}\cos \omega t\cos \delta  + {d_2}\sin \omega t\sin \delta } \right]\sin \theta.
$$

The average radiation over time is:
$$
\left\langle {{{\left| {{{\mathbf{E}}_{{\mathbf{rad}}}}} \right|}^2}} \right\rangle  = \frac{{{\omega ^4}{{\sin }^2}\theta }}{{2{R^2}{c^4}}}\left[ {{d_1}^2 + {d_1}{d_2}{{\cos }^2}\delta  + {d_2}^2} \right]
$$

Note that cross terms of $\cos\omega t$ and $\sin\omega t$ are eliminated from the averaged.

For Power per solid angle:
$$
\left\langle {\frac{{dP}}{{d\Omega }}} \right\rangle  = \frac{{c{R^2}}}{{4\pi }}\left\langle {{{\left| {{{\mathbf{E}}_{{\mathbf{rad}}}}} \right|}^2}} \right\rangle  = \frac{{{\omega ^4}{{\sin }^2}\theta }}{{8\pi {c^3}}}\left[ {{d_1}^2 + {d_1}{d_2}{{\cos }^2}\delta  + {d_2}^2} \right]
$$

b.

$$
L/\lambda  = \omega L 2\pi \ll 1
$$

So that,

$$
\delta  = \omega L\sin \theta /c \ll 1.
$$

We got,
$$
\left\langle {\frac{{dP}}{{d\Omega }}} \right\rangle  = \frac{{{\omega ^4}{{\sin }^2}\theta }}{{8\pi {c^3}}}\left[ {{d_1}^2 + {d_1}{d_2} + {d_2}^2} \right] = \frac{{{\omega ^4}{{\sin }^2}\theta }}{{8\pi {c^3}}}{\left[ {{d_1} + {d_2}} \right]^2}
$$

3.5

This is dipole,
$$
d = P \times V = \alpha E'\frac{{3\pi {a^3}}}{4} = \alpha {E_0}\cos \omega t\frac{{3\pi {a^3}}}{4}{\left( {1 + \frac{{4\pi \alpha }}{3}} \right)^{ - 1}}
$$

(p.91) cross section can be accessed by
$$
\sigma  = \frac{{\left\langle P \right\rangle }}{{\left\langle S \right\rangle }}
$$

$$
\left\langle P \right\rangle  = \frac{{2{{{\mathbf{\ddot d}}}^2}}}{{3{c^3}}} = \frac{1}{{3{c^3}}}{\alpha ^2}{E_0}^2{\omega ^4}{\left( {\frac{{3\pi {a^3}}}{4}} \right)^2}{\left( {1 + \frac{{4\pi \alpha }}{3}} \right)^{ - 2}}
$$

$$
\left\langle S \right\rangle  = \frac{{cE_0^2}}{{8\pi }}
$$

we got cross section,
$$
\sigma  = \frac{{\left\langle P \right\rangle }}{{\left\langle S \right\rangle }} = \frac{{\frac{1}{{3{c^3}}}{\alpha ^2}{E_0}^2{\omega ^4}{{\left( {\frac{{4\pi {a^3}}}{3}} \right)}^2}{{\left( {1 + \frac{{4\pi \alpha }}{3}} \right)}^{ - 2}}}}{{\frac{{cE_0^2}}{{8\pi }}}}
$$

Holy crap, I need to rewrite this thing?? maybe I should try Mathmatica kernel on notebook next time.

$$\sigma  = \frac{{\left\langle P \right\rangle }}{{\left\langle S \right\rangle }} = \frac{{\frac{1}{{3{c^3}}}{\alpha ^2}{E_0}^2{\omega ^4}{{\left( {\frac{{4\pi {a^3}}}{3}} \right)}^2}{{\left( {1 + \frac{{4\pi \alpha }}{3}} \right)}^{ - 2}}}}{{\frac{{cE_0^2}}{{8\pi }}}} = \frac{{8\pi }}{3}{k^4}{a^6}{\left( {1 + \frac{3}{{4\pi \alpha }}} \right)^{ - 2}} \\
= \pi {a^2}\frac{8}{3}{(ka)^4}{\left( {1 + \frac{3}{{4\pi \alpha }}} \right)^{ - 2}} = \pi {a^2}{Q_{scatt}}
$$

The end.....

## Sympy Tutorial

### Symbolic Variables

```python
import sympy as sp
sp.init_printing()
x,y,z,t = sp.symbols('x y z t')
sp.simplify((x**3 + x**2 - x - 1)/(x**2 + 2*x + 1))
```

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
x = sp.Symbol('x', real=True)
x.is_imaginary
```

    False

```python
1+1*sp.I
```

```python
sp.I**2
```

```python
(x * sp.I + 1)**2
```

### Rational Numbers

```python
r1 = sp.Rational(4,5)
r2 = sp.Rational(5,4)
```

```python
r1
```

```python
r1 + r2
```

### Numerical

```python
sp.pi.evalf(n=50)
```

```python
y = (x + sp.pi)**2
sp.N(y)
```

```python

```

### Expand

```python
sp.expand((x + 1)**2)
```

```python
sp.expand((x + 1)*(x - 2) - (x - 1)*x)
```

### Factor

```python
sp.factor(x**3 - x**2 + x - 1)
```

```python
sp.factor(sp.expand( (sp.cos(x) + sp.sin(x))**2 ))
```

### Collect

```python
expr = x*y + x - 3 + 2*x**2 - z*x**2 + x**3
expr
```

```python
sp.collect(expr, x)
```

### Cancel

```python
expr = (x**2 + 2*x + 1) / (x**2 + x)
expr
```

```python
sp.cancel(expr)
```

### Apart

```python
expr = (4*x**3 + 21*x**2 + 10*x + 12)/(x**4 + 5*x**3 + 5*x**2 + 4*x)
expr
```

```python
sp.apart(expr)
```

### Trigonnometric Simplification

```python

```


# Kinematics: the geometry of motion

## Galileo's Physics: Kinematics

[Galileo Galilei](https://en.wikipedia.org/wiki/Galileo_Galilei)
1564-1642, created the foundations for classical engineering physics by
defining __kinematics__. __Kinematics__ is the study of the
geometry of motion. He observed the motion of planets, stars,
falling objects, etc. Galileo defined used the relationships between
position, velocity, and acceleration to make informed predictions.

One of the first kinematic experiments, was Galileo's inclined ramp.
Galileo's hypthosis: objects all fall at the same rate of change in
speed. Now, engineers take this for granted and use a gravitational
constant to describe acceleration, $g=9.81~\frac{m}{s^2}$.

### Inclined ramp experiment

Placing two round objects, of the same shape, on an inclined plane, you
can release them and see they both
- move the same distance in the same amount of time
- move slowly at first, then fastest at the end of the ramp
- the distance travelled is $d \propto t^2$ time-squared

The last obeservation is harder to _see_, but if you use a stopwatch and
meter stick you can create a table for distance and time.

Classical physics describes the position of an object using three
independent coordinates e.g.

$$\mathbf{r}_{P/O} = x\hat{i} + y\hat{j} + z\hat{k}$$

where $\mathbf{r}_{P/O}$ is the position of point $P$ _with respect to the point
of origin_ $O$, $x,~y,~z$ are magnitudes of distance along a
Cartesian coordinate system and $\hat{i},~\hat{j}$ and $\hat{k}$ are
unit vectors that describe three

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
```

## Velocity

The velocity of an object is the change in [position](#position) per length of time.

\begin{equation}
\mathbf{v}_{P/O} = \frac{d\mathbf{r}_{P/O}}{dt} = \dot{x}\hat{i} + \dot{y}\hat{j} +
\dot{z}\hat{k}
\end{equation}

> __Note:__ The notation $\dot{x}$ and $\ddot{x}$ is short-hand for writing out
> $\frac{dx}{dt}$ and $\frac{d^2x}{dt^2}$, respectively.

The definition of velocity in equation {eq}`velocity` depends upon the change in position of all
three independent coordinates, where
$\frac{d}{dt}(x\hat{i})=\dot{x}\hat{i}$.

## Example - GPS vs speedometer
You can find velocity based upon postion, but you can only find changes
in position with velocity. Consider tracking the motion of a car driving
down a road using GPS. You determine its motion and create the position,
$\mathbf{r} = x\hat{i} +y\hat{j}$, where

$x(t) = 4t + 3$ and $y(t) = 3t - 1$

To get the velocity, calculate $\mathbf{v} = \dot{\mathbf{r}}$

$\mathbf{v} = 4\hat{i} +3 \hat{j}$

```python
t = np.arange(0,5)
x = 4*t + 3
y = 3*t -1
plt.plot(x,y,'o')
plt.quiver(x,y,4,3)
plt.title('Position of car on road every 1 second'+
    '\nvelocity shown as arrow')
plt.xlabel('x-position (m)')
plt.ylabel('y-position (m)');
```

## Speed

The speed of an object is the
[magnitude](https://www.mathsisfun.com/algebra/vectors.html) of the
velocity,

$|\mathbf{v}_{P/O}| = \sqrt{\mathbf{v}\cdot\mathbf{v}} =
\sqrt{\dot{x}^2 + \dot{y}^2 + \dot{z}^2}$

### Remember the chain rule
In Dynamics, there are many moving parts, so it is usually safer to
assume that a variable is a function of time. Consider the derivative of
position,
\begin{equation}
\frac{d}{dt}(x\hat{i}+y\hat{j}) = \dot{x}\hat{i} + \dot{y}\hat{j} +
x\dot{\hat{i}} + y\dot{\hat{j}},
\end{equation}

in a fixed coordinate system (like the ground)
$\dot{\hat{i}}=\dot{\hat{j}}=0$ because these unit vectors do not change direction.

### Chain rule in dynamics

In calculus, many times the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) is shown as an example

- $$\frac{d}{dx}(\sin x^2) = \frac{d}{dx}(x^2)\cdot \cos x^2$$
- $$\frac{d}{dx}(\sin x^2)= 2x\cos x^2$$

When you do this calculation, you have used the chain rule, but there is
a hidden step _multiplying by a chain rule operator_
$$\frac{dx^2}{dx^2}$$:
- $$\frac{d}{dx}(\sin x^2) = \frac{dx^2}{dx}\frac{d}{dx^2}\cdot \cos
x^2$$
-  $$\frac{d}{dx}(\sin x^2) =
\frac{d}{dx}{x^2}\cdot\frac{d}{dx^2}(\cos x^2)$$
- $$\frac{d}{dx}(\sin x^2) = \frac{d}{dx}(x^2)\cdot \cos x^2$$
- $$\frac{d}{dx}(\sin x^2)= 2x\cos x^2$$

The chain rule in our case is used for functions of time, $$t$$
e.g. $$x(t),~y(t),~\theta(t)$$

So, when you take a derivative of a function:

- $$\frac{d}{dt}\sin\theta =
\frac{d\theta}{dt}\frac{d}{d\theta}\sin\theta$$
- $$\frac{d}{dt}\sin\theta =\frac{d\theta}{dt}\cos\theta$$
- $$\frac{d}{dt}\sin\theta =\dot{\theta}\cos\theta$$

## Acceleration

The acceleration of an object is the change in velocity per length of
time.

$\mathbf{a}_{P/O} = \frac{d \mathbf{v}_{P/O} }{dt} = \ddot{x}\hat{i} +
\ddot{y}\hat{j} + \ddot{z}\hat{k}$

where $\ddot{x}=\frac{d^2 x}{dt^2}$ and $\mathbf{a}_{P/O}$ is the
acceleration of point $P$ _with respect to the point of origin_ $O$.

## Rotation and Orientation

The definitions of position, velocity, and acceleration all describe a
single point, but dynamic engineering systems are composed of rigid
bodies is needed to describe the position of an object.

```python
from IPython.core.display import SVG

SVG(filename='./images/position_angle.svg')
```

_In the figure above, the center of the block is located at
$r_{P/O}=x\hat{i}+y\hat{j}$ in both the left and right images, but the
two locations are not the same. The orientation of the block is
important for determining the position of all the material points._

In general, a rigid body has a _pitch_, _yaw_, and _roll_ that describes
its rotational orientation, as seen in the animation below. We will
revisit 3D motion in Module_05

```python
from IPython.display import YouTubeVideo
vid = YouTubeVideo("li7t--8UZms?loop=1")
display(vid)
```

## Rotation in planar motion

Our initial focus is planar rotations e.g. yaw and roll are fixed. For a
body constrained to planar motion, you need 3 independent measurements
to describe its position e.g. $x$, $y$, and $\theta$


#Notebook ROHSA

### About this notebook

This is a notebook to show you how to install and run ROHSA. It was built to run on [Google Colab](https://colab.research.google.com) but should work on many systems. This [link](https://colab.research.google.com/drive/1i5ehl1VVpe0JKxFEq0lscIo-ml5uWWtm) should find a version of this notebook launched on colab.

#### Notebook Authors:
Joshua Peek, Antoine Marchal

## Installing ROHSA
[ROHSA](https://github.com/antoinemarchal/ROHSA) is a code for decomposing PPV emission cubes into Gaussians. It is written in FORTRAN90 and thus needs to be installed via make. We clone the repository from github, change to the correct directory and make.

##Model from ROHSA
\begin{equation}
  \tilde T_B\big(v_z, \theta(r)\big) = \sum_{n=1}^{N}
  G\big(v_z, \theta_n(r)\big)
\end{equation}

with $\theta(r) = \big(\theta_1(r), \dots, \theta_n(r)\big)$ and where

\begin{equation}
  G\big(v_z, \theta_n(r)\big) = a_n(r) \exp
  \left( - \frac{\big(v_z - \mu_n(r)\big)^2}{2 \sigma_n(r)^2} \right)
\end{equation}

is parametrized by $\theta_n = \big(a_n, \mu_n, \sigma_n\big)$ with
$a_n \geq 0$ being the amplitude, $\mu_{n}$ the position, and
$\sigma_{n}$ the standard deviation 2D maps of the $n$-th Gaussian
profile across the plan of sky.

The residual is

\begin{equation}
  L\big(v_z, \theta(r)\big) = \tilde T_B\big(v_z, \theta(r)\big)
  - T_B(v_z, r).
\end{equation}

The cost function that includes the sum of the squares of the residual is

\begin{equation}
  Q(\theta) = \frac{1}{2} \, \sum_{v_z, r}
  \left(\frac{L\big(v_z, \theta(r)\big)}{\Sigma(r)}\right)^2
\end{equation}

where $\Sigma$ is the standard deviation 2D map of the noise assumed
without spatial correlation.

The regularization terms are

\begin{equation}
  R(\theta,m) = \frac{1}{2} \, \sum_{n=1}^N \lambda_{a} \|D a_n\|_2^2 +
  \lambda_{\mu} \|D \mu_n\|_2^2 + \lambda_{\sigma} \|D \sigma_n\|_2^2 +
  \lambda'_{\sigma} \|\sigma_n - m_n\|_2^2
\end{equation}

with $m = (m_1, \dots, m_N)$ and $a_n \geq 0$, $\forall \, n \in [1, \dots, N]$ and where D is a matrix performing the 2D convolution using the kernel d and $\lambda_a$, $\lambda_{\mu}$, and $\lambda_{\sigma}$ are hyper-parameters than tune the balance between the different terms.

\begin{equation}
  d = \begin{bmatrix}
  0 & -1 & 0 \\
  -1 & 4 & -1 \\
  0 & -1 & 0 \\
  \end{bmatrix}
\end{equation}

The full cost function is then

\begin{align}
  J(\theta, m) = Q(\theta) + R(\theta,m).
\end{align}

Finally, the minimizer is

\begin{equation}
[\hat \theta, \hat m] = \underset{\theta,m}{\text{argmin}}\ J(\theta,m),
\ \text{wrt. } a_n \geq 0,\ \forall \, n \in [1, \dots, N]
\end{equation}

```
!git clone https://github.com/antoinemarchal/ROHSA.git ROHSA
import os
os.chdir('ROHSA/src')
!make
os.chdir('../../')
```

    fatal: destination path 'ROHSA' already exists and is not an empty directory.
    gfortran blas.o lbfgsb.o linpack.o timer.o mod_constants.o mod_convert.o mod_inout.o mod_array.o mod_optimize.o mod_optimize_lym.o mod_minimize.o mod_functions.o mod_start.o mod_rohsa.o main.o -lm -g -O2  -o ROHSA

## Getting Data
Our example data comes from the GHIGLS Survey from Martin et al. 2015.

```
!wget https://www.cita.utoronto.ca/GHIGLS/FITS/GHIGLS_DFN_Tb.fits
```

    --2020-12-18 02:57:28--  https://www.cita.utoronto.ca/GHIGLS/FITS/GHIGLS_DFN_Tb.fits
    Resolving www.cita.utoronto.ca (www.cita.utoronto.ca)... 128.100.76.44
    Connecting to www.cita.utoronto.ca (www.cita.utoronto.ca)|128.100.76.44|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 2485440 (2.4M) [image/fits]
    Saving to: ‘GHIGLS_DFN_Tb.fits.1’

    GHIGLS_DFN_Tb.fits. 100%[===================>]   2.37M  2.64MB/s    in 0.9s

    2020-12-18 02:57:30 (2.64 MB/s) - ‘GHIGLS_DFN_Tb.fits.1’ saved [2485440/2485440]

## Getting started with ROHSApy

### Installation

```
!pip install ROHSApy
```

    Requirement already satisfied: ROHSApy in /usr/local/lib/python3.6/dist-packages (0.2.0)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from ROHSApy) (3.2.2)
    Requirement already satisfied: astropy in /usr/local/lib/python3.6/dist-packages (from ROHSApy) (4.1)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from ROHSApy) (1.19.4)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->ROHSApy) (1.3.1)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->ROHSApy) (2.8.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->ROHSApy) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->ROHSApy) (0.10.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib->ROHSApy) (1.15.0)

 ### Load data + Visualizing the integrated column density map

```
import numpy as np
from astropy.io import fits
import matplotlib.pyplot as plt
from ROHSApy import ROHSA

fitsname = "GHIGLS_DFN_Tb.fits"
hdu = fits.open(fitsname)
hdr = hdu[0].header
cube = hdu[0].data[0][150:350,:16,:16]
```

```
plt.figure()
plt.imshow(np.sum(cube,0), origin="lower", cmap="inferno")
```

### Generate files for ROHSA with your data
Here we run cube2dat that converts the fits file to a dat file readable by FORTRAN and generate a parameters file to run the code.

```
filename = "GHIGLS_DFN_Tb.dat" #@param
fileout = "GHIGLS_DFN_Tb_gauss_run_0.dat" #@param
filename_noise = ''
n_gauss = 6           #@param {type:"slider", min:1, max:12, step:1}
lambda_amp = 100      #@param {type:"slider", min:0, max:1000, step:1}
lambda_mu = 100       #@param {type:"slider", min:0, max:1000, step:1}
lambda_sig = 100      #@param {type:"slider", min:0, max:1000, step:1}
lambda_var_sig = 0.   #@param {type:"slider", min:0, max:10000, step:1}
amp_fact_init = 0.66  # times max amplitude of Gaussians for the fit of the mean spectrum
sig_init = 4.         # dispersion of Gaussians for the fit of the mean spectrum
lb_sig_init = 1.      # lower limit on sigma for the fit of the mean spectrum
ub_sig_init = 12.     # upper limit on sigma for the fit of the mean spectrum
lb_sig = 1.
ub_sig = 100.
maxiter_init = 15000  # max iteration for L-BFGS-B alogorithm init mean
maxiter = 800         #@param {type:"slider", min:1, max:800, step:1}
noise = ".false."     # if false - STD map computed by ROHSA between lstd and ustd
lstd = 1              #@param {type:"slider", min:1, max:400, step:1}
ustd = 20             #@param {type:"slider", min:1, max:400, step:1}
iprint = -1           #@param ["-1", "0", "1"]
iprint_init = -1      # print option init
save_grid = ".false." #@param [".true.", ".false."]

core = ROHSA(cube)
core.cube2dat(filename=filename)
core.gen_parameters(filename=filename,
                    fileout=fileout,
                    n_gauss=n_gauss,
                    lambda_amp=lambda_amp,
                    lambda_mu=lambda_mu,
                    lambda_sig=lambda_sig,
                    lambda_var_sig=lambda_var_sig,
                    amp_fact_init=amp_fact_init,
                    sig_init=sig_init,
                    lb_sig_init=lb_sig_init,
                    ub_sig_init=ub_sig_init,
                    lb_sig=lb_sig,
                    ub_sig=ub_sig,
                    noise=noise,
                    maxiter=maxiter,
                    lstd=lstd,
                    ustd=ustd,
                    iprint_init=iprint_init,
                    iprint=iprint,
                    save_grid=save_grid)

```

    Generate GHIGLS_DFN_Tb.dat file readable by fortran
    Generate parameters.txt file

### Running ROHSA
Here we run ROHSA on the data. This is invoking the parameter file parameters.txt.
Note : If you want to use core.run() method, you have to add your executable in your own .bashrc -
Here we use directly ROHSA from src

```
!./ROHSA/src/ROHSA parameters.txt
```

     -------------------------------------------------------------------------
    18 December 2020   3:02:16.523 AM

       ____     ___    _   _   ____       _
      |  _ \   / _ \  | | | | / ___|     / \
      | |_) | | | | | | |_| | \___ \    / _ \
      |  _ <  | |_| | |  _  |  ___) |  / ___ \
      |_| \_\  \___/  |_| |_| |____/  /_/   \_\

      Version 1.2.1
      ROHSA is released as open source code
      Check out the documentation: https://antoinemarchal.github.io/ROHSA/

     run: ./ROHSA parameters.txt
     -------------------------------------------------------------------------
     filename = 'GHIGLS_DFN_Tb.dat'
     fileout = 'GHIGLS_DFN_Tb_gauss_run_0.dat'
     timeout = 'timestep.dat'

     ______Parameters_____
     n_gauss =            6
     n_gauss_add =            0
     lambda_amp =    100.00000000000000
     lambda_mu =    100.00000000000000
     lambda_sig =    100.00000000000000
     lambda_var_amp =    0.0000000000000000
     lambda_var_mu =    0.0000000000000000
     lambda_var_sig =    0.0000000000000000
     lambda_lym_sig =    0.0000000000000000
     amp_fact_init =   0.66000000000000003
     sig_init =    4.0000000000000000
     lb_sig_init =    1.0000000000000000
     ub_sig_init =    12.000000000000000
     lb_sig =    1.0000000000000000
     ub_sig =    100.00000000000000
     init_option = mean
     maxiter_init =        15000
     maxiter =          800
     lstd =            1
     ustd =           20
     noise =  F
     regul =  T
     descent =  T
     save_grid =  F
     lym =  F

     dim_v, dim_y, dim_x =          200          16          16

     number of los =          256
     nside =            4

     Reshape cube, new dimensions :
     dim_v, dim_y, dim_x =          200          16          16

     Compute mean and std spectrum
                         Start iteration

     Start hierarchical descent
     Init mean spectrum
     Update level           0

     Interpolate parameters level            0
     Update level           1 >           2

     Interpolate parameters level            1
     Update level           2 >           4

     Interpolate parameters level            2
     Update level           3 >           8

     Interpolate parameters level            3

     Reshape cube, restore initial dimensions :
     dim_v, dim_y, dim_x =          200          16          16

     Start updating last level.

     _____ Write output file _____

             200
     ##################################################################
    18 December 2020   3:02:55.940 AM
      Terminate
                             ROHSA ALGORITHM

     ##################################################################
    Time = 39.354 seconds.

### Reading the result of ROHSA
read_gaussian() method allows you to read the output file generated by ROHSA. The structure of the cube is the following :

* dim 1 = Gaussian parameters / Ordering : amplitude - position - dispersion

* dim 2 and dim 3 = spatial axis

Note that position and dispersion are in pixel unit

```
gaussian = core.read_gaussian("GHIGLS_DFN_Tb_gauss_run_0.dat")
print("dim cube = " + str(gaussian.shape))

model = core.return_result_cube(gaussian=gaussian)

amplitude = gaussian[0::3]
position = gaussian[1::3]
dispersion = gaussian[2::3]

integral = amplitude * dispersion
```

    Opening data file
    dim cube = (18, 16, 16)

### Plot model of the integrated column density map

```
plt.figure(figsize=(12,5))
plt.subplot(1,3,1)
plt.imshow(np.sum(cube,0), origin="lower", cmap="inferno")
plt.subplot(1,3,2)
plt.imshow(np.sum(model,0), origin="lower", cmap="inferno")
plt.subplot(1,3,3)
plt.imshow((np.sum(model,0)-np.sum(cube,0))/np.sum(model,0), origin="lower", cmap="inferno")
plt.colorbar()

```

### Plot the first Gaussian component
Here we plot the amplitude, the position and the dispersion of the first Gaussian + a random spectum of the cube.

```
plt.figure(figsize=(12,5))
plt.subplot(1,3,1)
plt.imshow(amplitude[0], origin="lower", cmap="inferno")
plt.subplot(1,3,2)
plt.imshow(position[0], origin="lower", cmap="coolwarm")
plt.subplot(1,3,3)
plt.imshow(dispersion[0], origin="lower", cmap="cubehelix")

```

```
#Plot mosaic spectra
pvalues = np.logspace(-1, 0, len(integral))
pmin = pvalues[0]
pmax = pvalues[-1]

def norm(pval):
    return (pval - pmin) / float(pmax - pmin)

ny = 4; nx = 4
center_y = int(cube.shape[2]/2); center_x = int(cube.shape[1]/2)
x = np.arange(cube.shape[0])
cb = "magenta"
cw = "crimson"
fig, axs = plt.subplots(4, 4, sharex=True, sharey=True, figsize=(10.,6.))
fig.subplots_adjust(hspace=0, wspace=0, left=0, right=1, top=1, bottom=0)
for i in np.arange(ny):
    for j in np.arange(nx):
        axs[i][j].step(x, cube[:,center_y+i,center_x+j], color='cornflowerblue', linewidth=2.)
        axs[i][j].plot(x, model[:,center_y+i,center_x+j], linestyle="-", linewidth=2., color="k")
        for k in range(len(integral)):
            axs[i][j].plot(x, core.gauss(x, gaussian[0::3][k][center_y+i,center_x+j],
                                            gaussian[1::3][k][center_y+i,center_x+j],
                                            gaussian[2::3][k][center_y+i,center_x+j]),
                           linewidth=2., color=plt.cm.inferno(pvalues[k]))
        if j == 0: axs[i][j].set_ylabel(r'T [k]')
        axs[i][j].set_xlabel(r'v [km s$^{-1}$]')
# plt.savefig("plot/" + 'mosaic_spectra_all.png', format='png', bbox_inches='tight', pad_inches=0.02)

```

###Enjoy ! And please contact me for any question related to ROHSA


# Tutorial 1: Decoding Neural Responses
**Week 2, Day 1: Deep Learning**

**By Neuromatch Academy**

**Content creators**: Jorge A. Menendez, Carsen Stringer

**Content reviewers**: Roozbeh Farhoodi,  Madineh Sarvestani, Kshitij Dwivedi, Spiros Chavlis, Ella Batty, Michael Waskom

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives
In this tutorial, we'll use deep learning to decode stimulus information from the responses of sensory neurons. Specifically, we'll look at the activity of ~20,000 neurons in mouse primary visual cortex responding to oriented gratings recorded in [this study](https://www.biorxiv.org/content/10.1101/679324v2.abstract). Our task will be to **decode the orientation of the presented stimulus from the responses of the whole population of neurons**. We could do this in a number of ways, but here we'll use deep learning. Deep learning is particularly well-suited to this problem for a number of reasons:
* The data are very high-dimensional: the neural response to a stimulus is a ~20,000 dimensional vector. Many machine learning techniques fail in such high dimensions, but deep learning actually thrives in this regime, as long as you have enough data (which we do here!).
* As you'll be able to see below, different neurons can respond quite differently to stimuli. **This complex pattern of responses will, therefore, require non-linear methods to be decoded**, which we can easily do with non-linear activation functions in deep networks.
* Deep learning architectures are highly flexible, meaning we can easily adapt the architecture of our decoding model to optimize decoding. Here, we'll focus on a single architecture, but you'll see that it can easily be modified with few changes to the code.

More concretely, our goal will be learn how to:
* Build a deep feed-forward network using PyTorch
* Evaluate the network's outputs using PyTorch built-in loss functions
* Compute gradients of the loss with respect to each parameter of the network using automatic differentiation
* Implement gradient descent to optimize the network's parameters

##  Tutorial slides

 These are the slides for all videos in this tutorial.

```python
# @title Tutorial slides

# @markdown These are the slides for all videos in this tutorial.
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/vb7c4/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

##  Video 1: Decoding from neural data using feed-forward networks in pytorch

```python
# @title Video 1: Decoding from neural data using feed-forward networks in pytorch
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Xa4y1a7Jz", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="SlrbMvvBOzM", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers the decoding task we will use in these tutorials, a linear network with one hidden layer, and how to build this in Pytorch.

 Click here for text recap of video

Generalized linear models were used as decoding and encoding models in W1D4 Machine Learning. A model that decodes a variable from neural activity can tell us *how much information* a brain area contains about that variable. An encoding model is a model from an input variable, like visual stimulus, to neural activity. The encoding model is meant to approximate the same transformation that the brain performs on input variables and therefore help us understand *how the brain represents information*. Today we will use deep neural networks to build these models because deep neural networks can approximate a wide range of non-linear functions and can be easily fit.

We will be exploring neural activity in mice while the mice is viewing oriented grating stimuli on a screen in front of it. We record neural activity using a technique called two-photon calcium imaging, which allows us to record many thousands of neurons simultanously. The neurons light up when they fire. We then convert this imaging data to a matrix of neural responses by stimuli presented. For the purposes of this tutorial we are going to bin the neural responses and compute each neuron’s tuning curve. We used bins of 1 degree. Here’s what the tuning curves look like for 3 example neurons. We will use the response of all neurons in a single bin to try to predict which stimulus was shown. So we are going to be using the responses of 24000 neurons to try to predict 360 different possible stimulus conditions corresponding to each degree of orientation. Now we’re in a regime of having large amounts of data so let’s try to build a deep network to predict stimuli from these 24000 neural responses.

First I will describe a linear network with no hidden layers, where the stimulus prediction $y$ is a product of weights $\mathbf{W}_{out}$ and neural responses $\mathbf{r}$ with an added term $\mathbf{b}$ which is called the bias term. When you fit a linear model such as this you minimize the squared error between the predicted stimulus $y$ and the true stimulus $\tilde{y}$, this is the “loss function”.
\begin{align}
    L &= (y - \tilde{y})^2 \\
     &= ((\mathbf{W}^{out} \mathbf{r} + \mathbf{b}) - \tilde{y})^2
\end{align}
The solution to minimizing this loss function in a linear model can be found in closed form, and you learned how to solve this linear regression problem in the first week if you remember. If we use a simple linear model for this data we are able to predict the stimulus within 2-3 degrees. Let’s see if we can predict the neural activity better with a deep network.

Let’s add a hidden layer with $M$ units to this linear model, where now the output $y$ is as follows:
\begin{align}
    \mathbf{h} &= \mathbf{W}^{in} \mathbf{r} + \mathbf{b}^{in}, && [\mathbf{W}^{in}: M \times N,\, \mathbf{b}^{in}: M \times 1], \\
    y &= \mathbf{W}^{out} \mathbf{h} + \mathbf{b}^{out},  && [\mathbf{W}^{out}: 1 \times M,\, \mathbf{b}^{in}: 1 \times 1],
\end{align}

Note this linear network with one hidden layer where $M$ hidden units is less than $N$ inputs is equivalent to performing [reduced rank regression](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444519/), a technique that is useful for regularizing your regression model.

Adding this hidden layer means the model now has a depth of $1$. The number of units $M$ is termed the width of the network. Increasing the depth and the width of the network can increase the expressivity of the model -- in other words how well it can fit complex non-linear functions. Many state-of-the-art models now have close to 100 layers! But for now let’s start with a model with a depth of $1$ and see if we can improve our prediction of the stimulus.

To build this network we are going to use pytorch. First we import the pytorch library called `torch` and its neural network module `nn`. Next we will create a class for the deep network called DeepNet. A class has functions which are called methods. A class in python is initialized using a method called `__init__`. In this case the init method is declared to takes two inputs (other than the `self` input which represents the class itself), which are `n_inputs` and `n_hidden`. In our case `n_inputs` is the number of neurons we are using to do the prediction, and `n_hidden` is the number of hidden units. We first call the super function to invoke the `nn.Module`’s init function. Next we add the hidden layer `in_layer` as an attribute of the class. It is a linear layer called `nn.Linear` with size `n_inputs` by `n_hidden`. Then we add a second linear layer `out_layer` of size `n_hidden` by `1`, because we are predicting one output - the orientation of the stimulus.

Next we add another method to the class called `forward`. This is the method that runs when you call the class as a function. It takes as input `r` which is the neural responses. Then `r` is sent through the linear layers `in_layer` and `out_layer` and returns our prediction `y`. Let’s create an instantiation of this class called `net` with 200 hidden units with `net = DeepNet(n_neurons, 200)`. Now we can run the neural response through the network to predict the stimulus (`net(r)`); running the “net” this way calls the forward method.

---
# Setup

```python
# Imports

import os
import numpy as np

import torch
from torch import nn
from torch import optim

import matplotlib as mpl
from matplotlib import pyplot as plt
```

##  Data retrieval and loading

```python
#@title Data retrieval and loading
import hashlib
import requests

fname = "W3D4_stringer_oribinned1.npz"
url = "https://osf.io/683xc/download"
expected_md5 = "436599dfd8ebe6019f066c38aed20580"

if not os.path.isfile(fname):
  try:
    r = requests.get(url)
  except requests.ConnectionError:
    print("!!! Failed to download data !!!")
  else:
    if r.status_code != requests.codes.ok:
      print("!!! Failed to download data !!!")
    elif hashlib.md5(r.content).hexdigest() != expected_md5:
      print("!!! Data download appears corrupted !!!")
    else:
      with open(fname, "wb") as fid:
        fid.write(r.content)
```

##  Figure Settings

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

##  Plotting Functions

```python
#@title Plotting Functions

def plot_data_matrix(X, ax):
  """Visualize data matrix of neural responses using a heatmap

  Args:
    X (torch.Tensor or np.ndarray): matrix of neural responses to visualize
        with a heatmap
    ax (matplotlib axes): where to plot

  """

  cax = ax.imshow(X, cmap=mpl.cm.pink, vmin=np.percentile(X, 1), vmax=np.percentile(X, 99))
  cbar = plt.colorbar(cax, ax=ax, label='normalized neural response')

  ax.set_aspect('auto')
  ax.set_xticks([])
  ax.set_yticks([])

def plot_decoded_results(train_loss, test_loss, test_labels, predicted_test_labels):
  """ Plot decoding results in the form of network training loss and test predictions

  Args:
    train_loss (list): training error over iterations
    test_labels (torch.Tensor): n_test x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians
    predicted_test_labels (torch.Tensor): n_test x 1 tensor with predicted orientations of the
      stimuli from decoding neural network

  """

  # Plot results
  fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))

  # Plot the training loss over iterations of GD
  ax1.plot(train_loss)
  # Plot the testing loss over iterations of GD
  ax1.plot(test_loss)
  ax1.legend(['train loss', 'test loss'])

  # Plot true stimulus orientation vs. predicted class
  ax2.plot(stimuli_test.squeeze(), predicted_test_labels, '.')

  ax1.set_xlim([0, None])
  ax1.set_ylim([0, None])
  ax1.set_xlabel('iterations of gradient descent')
  ax1.set_ylabel('negative log likelihood')
  ax2.set_xlabel('true stimulus orientation ($^o$)')
  ax2.set_ylabel('decoded orientation bin')
  ax2.set_xticks(np.linspace(0, 360, n_classes + 1))
  ax2.set_yticks(np.arange(n_classes))
  class_bins = [f'{i * 360 / n_classes: .0f}$^o$ - {(i + 1) * 360 / n_classes: .0f}$^o$' for i in range(n_classes)]
  ax2.set_yticklabels(class_bins);

  # Draw bin edges as vertical lines
  ax2.set_ylim(ax2.get_ylim())  # fix y-axis limits
  for i in range(n_classes):
    lower = i * 360 / n_classes
    upper = (i + 1) * 360 / n_classes
    ax2.plot([lower, lower], ax2.get_ylim(), '-', color="0.7", linewidth=1, zorder=-1)
    ax2.plot([upper, upper], ax2.get_ylim(), '-', color="0.7", linewidth=1, zorder=-1)

  plt.tight_layout()
```

##  Helper Functions

```python
#@title Helper Functions

def load_data(data_name=fname, bin_width=1):
  """Load mouse V1 data from Stringer et al. (2019)

  Data from study reported in this preprint:
  https://www.biorxiv.org/content/10.1101/679324v2.abstract

  These data comprise time-averaged responses of ~20,000 neurons
  to ~4,000 stimulus gratings of different orientations, recorded
  through Calcium imaging. The responses have been normalized by
  spontaneous levels of activity and then z-scored over stimuli, so
  expect negative numbers. They have also been binned and averaged
  to each degree of orientation.

  This function returns the relevant data (neural responses and
  stimulus orientations) in a torch.Tensor of data type torch.float32
  in order to match the default data type for nn.Parameters in
  Google Colab.

  This function will actually average responses to stimuli with orientations
  falling within bins specified by the bin_width argument. This helps
  produce individual neural "responses" with smoother and more
  interpretable tuning curves.

  Args:
    bin_width (float): size of stimulus bins over which to average neural
      responses

  Returns:
    resp (torch.Tensor): n_stimuli x n_neurons matrix of neural responses,
        each row contains the responses of each neuron to a given stimulus.
        As mentioned above, neural "response" is actually an average over
        responses to stimuli with similar angles falling within specified bins.
    stimuli: (torch.Tensor): n_stimuli x 1 column vector with orientation
        of each stimulus, in degrees. This is actually the mean orientation
        of all stimuli in each bin.

  """
  with np.load(data_name) as dobj:
    data = dict(**dobj)
  resp = data['resp']
  stimuli = data['stimuli']

  if bin_width > 1:
    # Bin neural responses and stimuli
    bins = np.digitize(stimuli, np.arange(0, 360 + bin_width, bin_width))
    stimuli_binned = np.array([stimuli[bins == i].mean() for i in np.unique(bins)])
    resp_binned = np.array([resp[bins == i, :].mean(0) for i in np.unique(bins)])
  else:
    resp_binned = resp
    stimuli_binned = stimuli

  # Return as torch.Tensor
  resp_tensor = torch.tensor(resp_binned, dtype=torch.float32)
  stimuli_tensor = torch.tensor(stimuli_binned, dtype=torch.float32).unsqueeze(1)  # add singleton dimension to make a column vector

  return resp_tensor, stimuli_tensor

def identityLine():
  """
  Plot the identity line y=x
  """
  ax = plt.gca()
  lims = np.array([ax.get_xlim(), ax.get_ylim()])
  minval = lims[:, 0].min()
  maxval = lims[:, 1].max()
  equal_lims = [minval, maxval]
  ax.set_xlim(equal_lims)
  ax.set_ylim(equal_lims)
  line = ax.plot([minval, maxval], [minval, maxval], color="0.7")
  line[0].set_zorder(-1)

def get_data(n_stim, train_data, train_labels):
  """ Return n_stim randomly drawn stimuli/resp pairs

  Args:
    n_stim (scalar): number of stimuli to draw
    resp (torch.Tensor):
    train_data (torch.Tensor): n_train x n_neurons tensor with neural
      responses to train on
    train_labels (torch.Tensor): n_train x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data, in radians

  Returns:
    (torch.Tensor, torch.Tensor): n_stim x n_neurons tensor of neural responses and n_stim x 1 of orientations respectively
  """
  n_stimuli = train_labels.shape[0]
  istim = np.random.choice(n_stimuli, n_stim)
  r = train_data[istim]  # neural responses to this stimulus
  ori = train_labels[istim]  # true stimulus orientation

  return r, ori

def stimulus_class(ori, n_classes):
  """Get stimulus class from stimulus orientation

  Args:
    ori (torch.Tensor): orientations of stimuli to return classes for
    n_classes (int): total number of classes

  Returns:
    torch.Tensor: 1D tensor with the classes for each stimulus

  """
  bins = np.linspace(0, 360, n_classes + 1)
  return torch.tensor(np.digitize(ori.squeeze(), bins)) - 1  # minus 1 to accomodate Python indexing
```

---
# Section 1: Load and visualize data

In the next cell, we have provided code to load the data and plot the matrix of neural responses.

Next to it, we plot the tuning curves of three randomly selected neurons. These tuning curves are the averaged response of each neuron to oriented stimuli within 1$^\circ$, and since there are 360$^\circ$ in total, we have 360 responses.

In the recording, there were actually thousands of stimuli shown, but in practice we often create these tuning curves because we want to visualize averaged responses with respect to the variable we varied in the experiment, in this case stimulus orientation.

 Execute this cell to load and visualize data

```python
#@title

#@markdown Execute this cell to load and visualize data

# Load data
resp_all, stimuli_all = load_data()  # argument to this function specifies bin width
n_stimuli, n_neurons = resp_all.shape

print(f'{n_neurons} neurons in response to {n_stimuli} stimuli')

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(2 * 6, 5))

# Visualize data matrix

plot_data_matrix(resp_all[:100, :].T, ax1)  # plot responses of first 100 neurons
ax1.set_xlabel('stimulus')
ax1.set_ylabel('neuron')

# Plot tuning curves of three random neurons
ineurons = np.random.choice(n_neurons, 3, replace=False)  # pick three random neurons
ax2.plot(stimuli_all, resp_all[:, ineurons])
ax2.set_xlabel('stimulus orientation ($^o$)')
ax2.set_ylabel('neural response')
ax2.set_xticks(np.linspace(0, 360, 5))

plt.tight_layout()
```

We will split our data into a training set and test set. In particular, we will have a training set of orientations (`stimuli_train`) and the corresponding responses (`resp_train`). Our testing set will have held-out orientations (`stimuli_test`) and the corresponding responses (`resp_test`).

 Execute this cell to split into training and test sets

```python
#@title
#@markdown Execute this cell to split into training and test sets

# Set random seeds for reproducibility
np.random.seed(4)
torch.manual_seed(4)

# Split data into training set and testing set
n_train = int(0.6 * n_stimuli)  # use 60% of all data for training set
ishuffle = torch.randperm(n_stimuli)
itrain = ishuffle[:n_train]  # indices of data samples to include in training set
itest = ishuffle[n_train:]  # indices of data samples to include in testing set
stimuli_test = stimuli_all[itest]
resp_test = resp_all[itest]
stimuli_train = stimuli_all[itrain]
resp_train = resp_all[itrain]
```

---
# Section 2: Deep feed-forward networks in *pytorch*

We'll now build a simple deep neural network that takes as input a vector of neural responses and outputs a single number representing the decoded stimulus orientation.

To keep things simple, we'll build a deep network with **one** hidden layer that is **fully connected**, which means *every* output is a weighted sum of *every* input. See [bonus section 1](#b1) for a deeper discussion of what this choice entails, and when one might want to use deeper/shallower and wider/narrower architectures.

Let $\mathbf{r}^{(n)} = \begin{bmatrix} r_1^{(n)} & r_2^{(n)} & \ldots & r_N^{(n)} \end{bmatrix}^T$ denote the vector of neural responses (of neurons $1, \ldots, N$) to the $n$th stimulus. The network we will use is described by the following set of equations:
\begin{align}
    \mathbf{h}^{(n)} &= \mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in}, && [\mathbf{W}^{in}: M \times N,\, \mathbf{b}^{in}: M \times 1], \\
    y^{(n)} &= \mathbf{W}^{out} \mathbf{h}^{(n)} + \mathbf{b}^{out},  && [\mathbf{W}^{out}: 1 \times M,\, \mathbf{b}^{in}: 1 \times 1],
\end{align}
where $y^{(n)}$ denotes the scalar output of the network: the decoded orientation of the $n$th stimulus.

The $M$-dimensional vector $\mathbf{h}^{(n)}$ denotes the activations of the **hidden layer** of the network. The blue components of this diagram denote the **parameters** of the network, which we will later optimize with gradient descent. These include all the weights and biases $\mathbf{W}^{in}, \mathbf{b}^{in}, \mathbf{W}^{out}, \mathbf{b}^{out}$. The **weights** are matrices of size (# of outputs, # of inputs) that are multiplied by the input of each layer, like the **regression coefficients** in linear regression. The **biases** are vectors of size (# of outputs, 1), like the **intercept term** in linear regression (see W1D3 for more details on multivariate linear regression).

### Section 2.1: Introduction to PyTorch

Here, we'll use the **PyTorch** package to build, run, and train deep networks of this form in Python. PyTorch uses a data type called a `torch.Tensor`. `torch.Tensor`'s are effectively just like a `numpy` arrays, except that they have some important attributes and methods needed for **automatic differentiation** (to be discussed below). They also come along with infrastructure for easily storing and computing with them on GPU's, a capability we won't touch on here but which can be really useful in practice.

The next cell contains code for building the deep network we defined above and in the video using the `nn.Module` base class for deep neural network models (documentation [here](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=nn%20module#torch.nn.Module)).

Note the number of hidden units `n_hidden` is a parameter that we are free to vary in deciding how to build our network. See [Bonus Section 1](#b1) for a discussion of **how this architectural choice affects the computations the network can perform**.

Each of the layers consists of an `nn.Linear` module, which is a built-in PyTorch class containing all the weights and biases for a given network layer (documentation [here](https://pytorch.org/docs/master/generated/torch.nn.Linear.html)). This class takes two arguments to initialize:
  * \# of inputs to that layer
  * \# of outputs from that layer

For the input layer, for example, we have:
  * \# of inputs = \# of neurons whose responses are to be decoded ($N$, specified by `n_inputs`)
  * \# of outputs = \# of hidden layer units ($M$, specified by `n_hidden`)

PyTorch will initialize all weights and biases randomly.

Finally we define the `forward()` method, which takes as argument an input to the network and returns the network output. In our case, this comprises computing the output $y$ from a given input $\mathbf{r}$ using the above two equations. See the next cell for code implementing this computation using the built-in PyTorch `nn.Linear` classes.

```python
class DeepNet(nn.Module):
  """Deep Network with one hidden layer

  Args:
    n_inputs (int): number of input units
    n_hidden (int): number of units in hidden layer

  Attributes:
    in_layer (nn.Linear): weights and biases of input layer
    out_layer (nn.Linear): weights and biases of output layer

  """

  def __init__(self, n_inputs, n_hidden):
    super().__init__()  # needed to invoke the properties of the parent class nn.Module
    self.in_layer = nn.Linear(n_inputs, n_hidden) # neural activity --> hidden units
    self.out_layer = nn.Linear(n_hidden, 1) # hidden units --> output

  def forward(self, neural_responses):
    """Decode stimulus orientation from neural responses

    Args:
      r (torch.Tensor): vector of neural responses to decode, must be of
        length n_inputs. Can also be a tensor of shape n_stimuli x n_inputs,
        containing n_stimuli vectors of neural responses

    Returns:
      torch.Tensor: network outputs for each input provided in r. If
        r is a vector, then y is a 1D tensor of length 1. If r is a 2D
        tensor then y is a 2D tensor of shape n_stimuli x 1.

    """
    h = self.in_layer(r)  # hidden representation
    y = self.out_layer(h)
    return y
```

### Section 2.2: Activation functions

####  Video 2: Nonlinear activation functions

```python
# @title Video 2: Nonlinear activation functions
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1m5411h7V5", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="JAdukDCQALA", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers adding a nonlinear activation funciton, specifically a Rectified Linear Unit (ReLU), to the linear network.

 Click here for text recap of video

Note that the deep network we constructed above comprises solely **linear** operations on each layer: each layer is just a weighted sum of all the elements in the previous layer. It turns out that linear hidden layers like this aren't particularly useful, since a sequence of linear transformations is actually essentially the same as a single linear transformation. We can see this from the above equations by plugging in the first one into the second one to obtain
\begin{equation}
    y^{(n)} = \mathbf{W}^{out} \left( \mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in} \right) + \mathbf{b}^{out} = \mathbf{W}^{out}\mathbf{W}^{in} \mathbf{r}^{(n)} + \left( \mathbf{W}^{out}\mathbf{b}^{in} + \mathbf{b}^{out} \right)
\end{equation}
In other words, the output is still just a weighted sum of elements in the input -- the hidden layer has done nothing to change this.

To extend the set of computable input/output transformations to more than just weighted sums, we'll incorporate a **non-linear activation function** in the hidden units. This is done by simply modifying the equation for the hidden layer activations to be
\begin{equation}
    \mathbf{h}^{(n)} = \phi(\mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in})
\end{equation}
where $\phi$ is referred to as the activation function. Using a non-linear activation function will ensure that the hidden layer performs a non-linear transformation of the input, which will make our network much more powerful (or *expressive*, see [Bonus Section 3](#b3)). In practice, deep networks *always* use non-linear activation functions.

The most common non-linearity used is the rectified linear unit (or ReLU), which is a max(0, x) function. At the beginning of neural network development, researchers experimented with different non-linearities such as sigmoid and tanh functions, but in the end they found that RELU activation functions worked the best. It works well because the gradient is able to back-propagate through the network as long as the input is positive - the gradient is 1 for all values of x greater than 0. If you use a saturating non-linearity then the gradients will be very small in the saturating regimes, reducing the effective computing regime of the unit.

#### Coding Exercise 2.2: Nonlinear Activations

Create a new class `DeepNetReLU` by modifying our above deep network model to add a **non-linear activation** function $\phi$:
\begin{equation}
    \mathbf{h}^{(n)} = \phi(\mathbf{W}^{in} \mathbf{r}^{(n)} + \mathbf{b}^{in})
\end{equation}

We'll use the linear rectification function:
\begin{equation}
  \phi(x) =
  \begin{cases}
    x & \text{if } x > 0 \\
    0 & \text{else}
  \end{cases}
\end{equation}
which can be implemented in PyTorch using `torch.relu()`. Hidden layers with this activation function are typically referred to as "**Re**ctified **L**inear **U**nits", or **ReLU**'s.

Initialize this network with 10 hidden units and run on an example stimulus.

**Hint**: you only need to modify the `forward()` method of the above `DeepNet()` class to include `torch.relu()`.

```python
class DeepNetReLU(nn.Module):
  """ network with a single hidden layer h with a RELU """

  def __init__(self, n_inputs, n_hidden):
    super().__init__()  # needed to invoke the properties of the parent class nn.Module
    self.in_layer = nn.Linear(n_inputs, n_hidden) # neural activity --> hidden units
    self.out_layer = nn.Linear(n_hidden, 1) # hidden units --> output

  def forward(self, r):

    ############################################################################
    ## TO DO for students: write code for computing network output using a
    ## rectified linear activation function for the hidden units
    # Fill out function and remove
    # raise NotImplementedError("Student exercise: complete DeepNetReLU forward")
    ############################################################################

    h = torch.relu(self.in_layer(r)) # h is size (n_inputs, n_hidden)
    y = self.out_layer(h) # y is size (n_inputs, 1)

    return y
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D1_DeepLearning/solutions/W2D1_Tutorial1_Solution_10319cbf.py)

The next cell contains code for initializing and running this network. We use it to decode stimulus orientation (true stimulus given by `ori`) from a vector of neural responses `r` to the very first stimulus. Note that when the initialized network class is called as a function on an input (e.g. `net(r)`), its `.forward()` method is called. This is a special property of the `nn.Module` class.

Note that the decoded orientations at this point will be nonsense, since the network has been initialized with random weights. Below, we'll learn how to optimize these weights for good stimulus decoding.

```python
# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Initialize a deep network with M=200 hidden units
net = DeepNet(n_neurons, 200)

# Get neural responses (r) to and orientation (ori) to one stimulus in dataset
r, ori = get_data(1, resp_train, stimuli_train)  # using helper function get_data

# Decode orientation from these neural responses using initialized network
out = net(r)  # compute output from network, equivalent to net.forward(r)

print('decoded orientation: %.2f degrees' % out)
print('true orientation: %.2f degrees' % ori)
```

    decoded orientation: 0.08 degrees
    true orientation: 139.00 degrees

You should see that the decoded orientation is 0.08 $^{\circ}$ while the true orientation is 139.00 $^{\circ}$.

---
# Section 3: Loss functions and gradient descent

##  Video 3: Loss functions & gradient descent

```python
# @title Video 3: Loss functions & gradient descent
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV19k4y1271n", width=730, height=410, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="aEtKpzEuviw", width=730, height=410, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers loss functions, gradient descent, and how to implement these in Pytorch.

 Click here for text recap of video

We need to define the loss function that we are optimizing the network to minimize. In this case we are minimizing the squared error between the prediction and the stimulus. So first we run the neural responses through the network `net` to get the output `out`. Then we declare our loss function, we will use the built in `nn.MSELoss` function for this purpose: `loss_fn = nn.MSELoss()`. This loss function takes two inputs, the network output `out` and the true stimulus orientations `ori` and finds the mean squared error: `loss = loss_fn(out, ori)`.

Next we minimize this loss function using gradient descent. In gradient descent we compute the gradient of the loss function with respect to each parameter (all W’s and b’s). We then update the parameters by subtracting the learning rate times the gradient.

Let’s visualize this loss function $L$ with respect to a weight $w$. If the gradient is positive (the slope $\frac{dL}{dw}$ > 0) as in this case then we want to move in the opposite direction which is negative. So we update the $w$ accordingly in the negative direction on each iteration. Once the iterations complete the weight will ideally be at a value that minimizes the cost function.

In reality these cost functions are not convex like this one and depend on hundreds of thousands of parameters. There are tricks to help navigate this rocky cost landscape such as adding momentum or changing the optimizer but we won’t have time to get into that today. There are also ways to change the architecture of the network to improve optimization, such as including skip connections. These skip connections are used in residual networks and allow for the optimization of many layer networks.

</detail>

### Section 3.1: Loss functions

Because the weights of the network are currently randomly chosen, the outputs of the network are nonsense: the decoded stimulus orientation is nowhere close to the true stimulus orientation. We'll shortly write some code to change these weights so that the network does a better job of decoding.

But to do so, we first need to define what we mean by "better". One simple way of defining this is to use the squared error
\begin{equation}
    L = (y - \tilde{y})^2
\end{equation}
where $y$ is the network output and $\tilde{y}$ is the true stimulus orientation. When the decoded stimulus orientation is far from the true stimulus orientation, $L$ will be large. We thus refer to $L$ as the **loss function**, as it quantifies how *bad* the network is at decoding stimulus orientation.

PyTorch actually carries with it a number of built-in loss functions. The one corresponding to the squared error is called `nn.MSELoss()`. This will take as arguments a **batch** of network outputs $y_1, y_2, \ldots, y_P$ and corresponding target outputs $\tilde{y}_1, \tilde{y}_2, \ldots, \tilde{y}_P$, and compute the **mean squared error (MSE)**
\begin{equation}
    L = \frac{1}{P}\sum_{n=1}^P \left(y^{(n)} - \tilde{y}^{(n)}\right)^2
\end{equation}
where $P$ is the number of different stimuli in a batch, called the *batch size*.

#### Computing MSE

Evaluate the mean squared error for a deep network with $M=10$ rectified linear units, on the decoded orientations from neural responses to 20 random stimuli.

```python
# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Initialize a deep network with M=10 hidden units
net = DeepNetReLU(n_neurons, 10)

# Get neural responses to first 20 stimuli in the data set
r, ori = get_data(20, resp_train, stimuli_train)

# Decode orientation from these neural responses
out = net(r)

# Initialize PyTorch mean squared error loss function (Hint: look at nn.MSELoss)
loss_fn = nn.MSELoss()

# Evaluate mean squared error
loss = loss_fn(out, ori)

print('mean squared error: %.2f' % loss)
```

    mean squared error: 42949.14

You should see a mean squared error of 42949.14.

### Section 3.2: Optimization with gradient descent

Our goal is now to modify the weights to make the mean squared error loss $L$ as small as possible over the whole data set. To do this, we'll use the **gradient descent (GD)** algorithm, which consists of iterating three steps.

 Execute this cell to view convolution gif

```python
#@markdown Execute this cell to view convolution gif

from IPython.display import Image
Image(url='https://github.com/NeuromatchAcademy/course-content/blob/W2D1_updates/tutorials/static/grad_descent.gif?raw=true')
```

1. **Evaluate the loss** on the training data,
```
out = net(train_data)
loss = loss_fn(out, train_labels)
```
where `train_data` are the network inputs in the training data (in our case, neural responses), and `train_labels` are the target outputs for each input (in our case, true stimulus orientations).
2. **Compute the gradient of the loss** with respect to **each of the network weights**. In PyTorch, we can do this with the `.backward()` method of the loss `loss`. Note that the gradients of each parameter need to be cleared before calling `.backward()`, or else PyTorch will try to accumulate gradients across iterations. This can again be done using built-in optimizers via the method `.zero_grad()`. Putting these together we have
```
optimizer.zero_grad()
loss.backward()
```
3. **Update the network weights** by descending the gradient. In Pytorch, we can do this using built-in optimizers. We'll use the `optim.SGD` optimizer (documentation [here](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD)) which updates parameters along the negative gradient, scaled by a learning rate. To initialize this optimizer, we have to tell it
  * which parameters to update, and
  * what learning rate to use

  For example, to optimize *all* the parameters of a network `net` using a learning rate of .001, the optimizer would be initialized as follows
  ```
  optimizer = optim.SGD(net.parameters(), lr=.001)
  ```
  where `.parameters()` is a method of the `nn.Module` class that returns a [Python generator object](https://wiki.python.org/moin/Generators) over all the parameters of that `nn.Module` class (in our case, $\mathbf{W}^{in}, \mathbf{b}^{in}, \mathbf{W}^{out}, \mathbf{b}^{out}$).

  After computing all the parameter gradients in step 2, we can then update each of these parameters using the `.step()` method of this optimizer,
  ```
  optimizer.step()
  ```
In the next exercise, we'll give you a code skeleton for implementing the GD algorithm. Your job will be to fill in the blanks.

For the mathematical details of the GD algorithm, see [bonus section 2.1](#b21).

In this case we are using gradient descent (not *stochastic* gradient descent) because we are computing the gradient over ALL training data at once. Normally there is too much training data to do this in practice, and for instance the neural responses may be divided into sets of 20 stimuli. An **epoch** in deep learning is defined as the forward and backward pass of all the training data through the network. We will run the forward and backward pass of the network here for 20 **epochs**, in practice training may require thousands of epochs.

See [bonus section 2.2](#b22) for a more detailed discussion of stochastic gradient descent.

#### Coding Exercise 3.2: Gradient descent in PyTorch

Complete the function `train()` that uses the gradient descent algorithm to optimize the weights of a given network. This function takes as input arguments
* `net`: the PyTorch network whose weights to optimize
* `loss_fn`: the PyTorch loss function to use to evaluate the loss
* `train_data`: the training data to evaluate the loss on (i.e. neural responses to decode)
* `train_labels`: the target outputs for each data point in `train_data` (i.e. true stimulus orientations)

We will then train a neural network on our data and plot the loss (mean squared error) over time. When we run this function, behind the scenes PyTorch is actually changing the parameters inside this network to make the network better at decoding, so its weights will now be different than they were at initialization.

```python
def train(net, loss_fn, train_data, train_labels,
          n_epochs=50, learning_rate=1e-4):
  """Run gradient descent to opimize parameters of a given network

  Args:
    net (nn.Module): PyTorch network whose parameters to optimize
    loss_fn: built-in PyTorch loss function to minimize
    train_data (torch.Tensor): n_train x n_neurons tensor with neural
      responses to train on
    train_labels (torch.Tensor): n_train x 1 tensor with orientations of the
      stimuli corresponding to each row of train_data
    n_epochs (int, optional): number of epochs of gradient descent to run
    learning_rate (float, optional): learning rate to use for gradient descent

  Returns:
    (list): training loss over iterations

  """

  # Initialize PyTorch SGD optimizer
  optimizer = optim.SGD(net.parameters(), lr=learning_rate)

  # Placeholder to save the loss at each iteration
  train_loss = []

  # Loop over epochs
  for i in range(n_epochs):

    ######################################################################
    ## TO DO for students: fill in missing code for GD iteration
    # raise NotImplementedError("Student exercise: write code for GD iterations")
    ######################################################################

    # compute network output from inputs in train_data
    out = net(train_data)  # compute network output from inputs in train_data

    # evaluate loss function
    loss = loss_fn(out, train_labels)

    # Clear previous gradients
    optimizer.zero_grad()

    # Compute gradients
    loss.backward()

    # Update weights
    optimizer.step()

    # Store current value of loss
    train_loss.append(loss.item())  # .item() needed to transform the tensor output of loss_fn to a scalar

    # Track progress
    if (i + 1) % (n_epochs // 5) == 0:
      print(f'iteration {i + 1}/{n_epochs} | loss: {loss.item():.3f}')

  return train_loss

# Set random seeds for reproducibility
np.random.seed(1)
torch.manual_seed(1)

# Initialize network with 10 hidden units
net = DeepNetReLU(n_neurons, 10)

# Initialize built-in PyTorch MSE loss function
loss_fn = nn.MSELoss()

# Run gradient descent on data
train_loss = train(net, loss_fn, resp_train, stimuli_train)

# Plot the training loss over iterations of GD
plt.plot(train_loss)
plt.xlim([0, None])
plt.ylim([0, None])
plt.xlabel('iterations of gradient descent')
plt.ylabel('mean squared error')
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D1_DeepLearning/solutions/W2D1_Tutorial1_Solution_fab9f5bd.py)

*Example output:*

**We can further improve our model - please see the Bonus Tutorial when you have time to dive deeper into this model by evaluating and improving it's performance by visualizing the weights, looking at performance on test data, switching to a new loss function and adding regularization.**

---
# Summary

We have now covered a number of common and powerful techniques for applying deep learning to decoding from neural data, some of which are common to almost any machine learning problem:
* Building and training deep networks using the **PyTorch** `nn.Module` class and built-in **optimizers**
* Choosing and evaluating **loss functions**
* Testing a trained model on unseen data via **cross-validation**, by splitting the data into a **training set and testing set**

An important aspect of this tutorial was the `train()` function we wrote in exercise 3. Note that it can be used to train *any* network to minimize *any* loss function (cf. advanced exercise 6) on *any* training data. This is the power of using PyTorch to train neural networks and, for that matter, **any other model**! There is nothing in the `nn.Module` class that forces us to use `nn.Linear` layers that implement neural network operations. You can actually put anything you want inside the `.__init__()` and `.forward()` methods of this class. As long as its parameters and computations involve only `torch.Tensor`'s, and the model is differentiable, you'll then be able to optimize the parameters of this model in exactly the same way we optimized the deep networks here.

What kinds of conclusions can we draw from these sorts of analyses? If we can decode the stimulus well from visual cortex activity, that means that there is information about this stimulus available in visual cortex. Whether or not the animal uses that information to make decisions is not determined from an analysis like this. In fact mice perform poorly in orientation discrimination tasks compared to monkeys and humans, even though they have information about these stimuli in their visual cortex. Why do you think they perform poorly in orientation discrimination tasks?

See [this paper](https://www.biorxiv.org/content/10.1101/679324v2) for some potential hypotheses, but this is totally an open question!

---
# Bonus

## Bonus Section 1: Neural network *depth*, *width* and *expressivity*

Two important architectural choices that always have to be made when constructing deep feed-forward networks like those used here are
* the number of hidden layers, or the network's **depth**
* the number of units in each layer, or the layer **widths**

Here, we restricted ourselves to networks with a single hidden layer with a width of $M$ units, but it is easy to see how this code could be adapted to arbitrary depths. Adding another hidden layer simply requires adding another `nn.Linear` module to the `__init__()` method and incorporating it into the `.forward()` method.

**The depth and width of a network determine the set of input/output transormations that it can perform**, often referred to as its **expressivity**. The deeper and wider the network, the more *expressive* it is; that is, the larger the class of input/output transformations it can compute. In fact, it turns out that an infinitely wide *or* infinitely deep networks can in principle [compute (almost) *any* input/output transformation](https://en.wikipedia.org/wiki/Universal_approximation_theorem).

A classic mathematical demonstration of the power of depth is given by the so-called [XOR problem](https://medium.com/@jayeshbahire/the-xor-problem-in-neural-networks-50006411840b#:~:text=The%20XOr%2C%20or%20%E2%80%9Cexclusive%20or,value%20if%20they%20are%20equal.). This toy problem demonstrates how even a single hidden layer can drastically expand the set of input/output transformations a network can perform, relative to a shallow network with no hidden layers. The key intuition is that the hidden layer allows you to represent the input in a new format, which can then allow you to do almost anything you want with it. The *wider* this hidden layer, the more flexibility you have in this representation. In particular, if you have more hidden units than input units, then the hidden layer representation of the input is higher-dimensional than the raw data representation. This higher dimensionality effectively gives you more "room" to perform arbitrary computations in. It turns out that even with just this one hidden layer, if you make it wide enough you can actually approximate any input/output transformation you want. See [here](http://neuralnetworksanddeeplearning.com/chap4.html) for a neat visual demonstration of this.

In practice, however, it turns out that **increasing depth seems to grant more expressivity with fewer units than increasing width does** (for reasons that are not well understood). It is for this reason that truly *deep* networks are almost always used in machine learning, which is why this set of techniques is often referred to as *deep* learning.

That said, there is a cost to making networks deeper and wider. The bigger your network, the more parameters (i.e. weights and biases) it has, which need to be optimized! The extra expressivity afforded by higher width and/or depth thus carries with it (at least) two problems:
* optimizing more parameters usually requires more data
* a more highly parameterized network is more prone to overfit to the training data, so requires more sophisticated optimization algorithms to ensure generalization

## Bonus Section 2: Gradient descent

### Bonus Section 2.1: Gradient descent equations

Here we provide the equations for the three steps of the gradient descent algorithm, as applied to our decoding problem:

1. **Evaluate the loss** on the training data. For a mean squared error loss, this is given by
\begin{equation}
    L = \frac{1}{P}\sum_{n=1}^P (y^{(n)} - \tilde{y}^{(n)})^2
\end{equation}
where $y^{(n)}$ denotes the stimulus orientation decoded from the population response $\mathbf{r}^{(n)}$ to the $n$th stimulus in the training data, and $\tilde{y}^{(n)}$ is the true orientation of that stimulus. $P$ denotes the total number of data samples in the training set. In the syntax of our `train()` function above, $\mathbf{r}^{(n)}$ is given by `train_data[n, :]` and $\tilde{y}^{(n)}$ by `train_labels[n]`.

2. **Compute the gradient of the loss** with respect to each of the network weights. In our case, this entails computing the quantities
\begin{equation}
    \frac{\partial L}{\partial \mathbf{W}^{in}}, \frac{\partial L}{\partial \mathbf{b}^{in}}, \frac{\partial L}{\partial \mathbf{W}^{out}}, \frac{\partial L}{\partial \mathbf{b}^{out}}
\end{equation}
Usually, we would require lots of math in order to derive each of these gradients, and lots of code to compute them. But this is where PyTorch comes to the rescue! Using a cool technique called [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), PyTorch automatically calculates these gradients when the `.backward()` function is called.

  More specifically, when this function is called on a particular variable (e.g. `loss`, as above), PyTorch will compute the gradients with respect to each network parameter. These are computed and stored behind the scenes, and can be accessed through the `.grad` attribute of each of the network's parameters. As we saw above, however, we actually never need to look at or call these gradients when implementing gradient descent, as this can be taken care of by PyTorch's built-in optimizers, like `optim.SGD`.

3. **Update the network weights** by descending the gradient:
\begin{align}
    \mathbf{W}^{in} &\leftarrow \mathbf{W}^{in} - \alpha \frac{\partial L}{\partial \mathbf{W}^{in}} \\
    \mathbf{b}^{in} &\leftarrow \mathbf{b}^{in} - \alpha \frac{\partial L}{\partial \mathbf{b}^{in}} \\
    \mathbf{W}^{out} &\leftarrow \mathbf{W}^{out} - \alpha \frac{\partial L}{\partial \mathbf{W}^{out}} \\
    \mathbf{b}^{out} &\leftarrow \mathbf{b}^{out} - \alpha \frac{\partial L}{\partial \mathbf{b}^{out}}
\end{align}
where $\alpha$ is called the **learning rate**. This **hyperparameter** of the SGD algorithm controls how far we descend the gradient on each iteration. It should be as large as possible so that fewer iterations are needed, but not too large so as to avoid parameter updates from skipping over minima in the loss landscape.

While the equations written down here are specific to the network and loss function considered in this tutorial, the code provided above for implementing these three steps is completely general: no matter what loss function or network you are using, exactly the same commands can be used to implement these three steps.

The way that the gradients are calculated is called **backpropagation**. We have a loss function:
\begin{align}
    L &= (y - \tilde{y})^2 \\
     &= (\mathbf{W}^{out} \mathbf{h} - \tilde{y})^2
\end{align}
where $\mathbf{h} = \phi(\mathbf{W}^{in} \mathbf{r} + \mathbf{b}^{in})$
You may see that $\frac{\partial L}{\partial \mathbf{W}^{out}}$ is simple to calculate as it is on the outside of the equation (it is also a vector in this case, not a matrix, so the derivative is standard):
\begin{equation}
\frac{\partial L}{\partial \mathbf{W}^{out}} = 2 (\mathbf{h} - \tilde{y})
\end{equation}
Now let's compute the derivative with respect to $\mathbf{W}^{in}$ using the chain rule. Note it is only positive if the output is positive due to the RELU $\phi$:
\begin{align}
\frac{\partial L}{\partial \mathbf{W}^{in}} &= \begin{cases}
    \frac{\partial L}{\partial \mathbf{h}} \frac{\partial \mathbf{h}}{\partial \mathbf{W}^{in}}  & \text{if }  \mathbf{h} > 0 \\
    0 & \text{else}
  \end{cases} \\
  &= \begin{cases}
    2 (\mathbf{W}^{out} - \tilde{y}) \mathbf{r}^\top  & \text{if }  \mathbf{h} > 0 \\
    0 & \text{else}
  \end{cases}
\end{align}
It is most efficient to compute the derivative once for the last layer, then once for the next layer and multiply by the previous layer's derivative and so on using the chain rule. Each of these operations is relatively fast, making training of deep networks feasible.

The command `loss.backward()` computes these gradients for the defined `loss` with respect to each network parameter. The computation is done using [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), which implements backpropagation. Note that this works no matter how big/small the network is, allowing us to perform gradient descent for any deep network model built using PyTorch.

### Bonus Section 2.2: *Stochastic* gradient descent (SGD) vs. gradient descent (GD)

In this tutorial, we used the gradient descent algorithm, which differs in a subtle yet very important way from the more commonly used **stochastic gradient descent (SGD)** algorithm. The key difference is in the very first step of each iteration, where in the GD algorithm we evaluate the loss *at every data sample in the training set*. In SGD, on the other hand, we **evaluate the loss only at a random subset of data samples from the full training set**, called a **mini-batch**. At each iteration, we randomly sample a mini-batch to perform steps 1-3 on. All the above equations still hold, but now the $P$ data samples $\mathbf{r}^{(n)}, \tilde{y}^{(n)}$ denote a mini-batch of $P$ random samples from the training set, rather than the whole training set.

There are several reasons why one might want to use SGD instead of GD. The first is that the training set might be too big, so that we actually can't actually evaluate the loss on every single data sample in it. In this case, GD is simply infeasible, so we have no choice but to turn to SGD, which bypasses the restrictive memory demands of GD by sub-sampling the training set into smaller mini-batches.

But, even when GD is feasible, SGD turns out to often be better. The stochasticity induced by the extra random sampling step in SGD effectively **adds some noise in the search for local minima of the loss function**. This can be really useful for **avoiding potential local minima**, and enforce that whatever minimum is converged to is a good one. This is particularly important when networks are wider and/or deeper, in which case the large number of parameters can lead to overfitting.

Here, we used only GD because (1) it is simpler, and (2) it suffices for the problem being considered here. Because we have so many neurons in our data set, decoding is not too challenging and doesn't require a particularly deep or wide network. The small number of parameters in our deep networks therefore can be optimized without a problem using GD.