# Music machine learning - Neural networks

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [quick introduction](#intro) on the code that we will adress and auxiliary functions
2. A simple implementation for a [single neuron](#single)
3. An implementation of [regression](#implem) using scikit-learn
4. Some [common good practices](#practices) in machine learning

# Introducing neural networks

In this tutorial, we will cover a more advanced classification algorithm through the use of *neural networks*. The tutorial starts by performing a simple **single neuron** discrimination of two random distributions. Then, we will study the typical **XOR problem** by using a more advanced 2-layer **perceptron**. Finally, we generalize the use of neural networks in order to perform classification on a given set of audio files.

We are going to use relatively _low-level_ libraries to perform the first exercises (implementing your own neurons)

```python
import time
import numpy as np
%matplotlib inline
from matplotlib import pyplot as plt
```

To simplify your work, we provide a first set of functions that provides simple plotting functionnalities (from the `helper_plot.py` file)

  |**File**|*Explanation*|
  |-------:|:---------|
  |`plot_boundary`|Plots the decision boundary of a single neuron with 2-dimensional inputs|
  |`plot_patterns`|Plots (bi-dimensionnal) input patterns|

```python
from helper_plot import plot_boundary
from helper_plot import plot_patterns
```

Remember that a single neuron is only able to learn _linearly separable_ problems. To produce such classes of problems, we provide a script that draw a set of random 2-dimensional points, then choose a random line in this space that will act as the linear frontier between 2 classes (hence defining a linear 2-class problem). The variables that will be used by your code are the following.

```Matlab
desired       % classes of the patterns
inputs        % 2 x n final matrix of random input patterns
weights       % 2 x 1 vector of neuron weights
bias          % 1 x 1 vector of bias
```

You can execute the code below to see our simple classification problem. (Note that running the same cell multiple times produces a different starting dataset).

```python
# Number of points to generate
nPats = 30 + np.floor(np.random.rand() * 30);
# Generate 2-dimensional random points
patterns = np.random.rand(2, int(nPats)) * 2 - 1;
# Slope of separating line
slope = np.log(np.random.rand() * 10);
yint = np.random.rand() * 2 - 1;
# Create the indexes for a two-class problem
desired = (patterns[1,:] - patterns[0,:] * slope - yint > 0) * 1;
# Plot the corresponding pattern
fig = plot_patterns(patterns,desired);
```

## Single neuron

For the first parts of the tutorial, we will perform the simplest classification model possible in a neural network setting, a single neuron. We briefly recall here that; given an input vector $ \mathbf{x} \in \mathbb{R}^{n} $, a single neuron computes the function

$$
\begin{equation}
y=\sigma\left(\sum_{i = 1}^{n}w_{i}.x_{i} + b\right)
\end{equation}
$$

with $ \mathbf{w} \in \mathbb{R}^{n} $ a weight vector, $ b $ a bias and $ \sigma\left(\right) $ an *activation function*. Therefore, if we consider the *threshold* activation function ($ \sigma_0\left(x\right)=1 $ if $ x \geq 0$), a single neuron simply performs an *affine transform* and then a *linear* discrimination of the space. A network will be composed of _layers_ of these neurons, which produce successive computations

Geometrically, a single neuron computes an hyperplane that separates the space. In order to learn, we have to adjust the weights and know "how much wrong we are". To do so, we consider that we know the desired output $ d $ of a system for a given example $ \mathbf{x} $ (eg. a predicted value for a regression system, a class value for a classification system). Therefore, we define the loss function $ \mathcal{L}_{\mathcal{D}} $ over a whole dataset as

$$
\begin{equation}
\mathcal{L}=\sum_{j=1}^{k_{\mathcal{D}}}\left\Vert d_{j}-y_{j}\right\Vert ^{2}
\end{equation}
$$

In order to know how to change the weights based on the value of the errors, we need to now "how to change it to make it better". Therefore, we should compute the sets of derivatives of the error given each parameter

$$
\begin{equation}
\Delta\bar{\mathbf{w}}=\left(\frac{\delta\mathcal{L}_{\mathcal{D}}}{\delta w_{1}},\ldots,\frac{\delta\mathcal{L}_{\mathcal{D}}}{\delta w_{n}}\right)
\end{equation}
$$

***

**Exercise**

  1. Perform the derivatives of the output given a single neuron
  2. Perform the derivatives for the bias as well

***

### Training your own neuron

We will start by training a single neuron to learn how to perform this discrimination with a linear problem (so that a single neuron is enough to solve it).

```python
# Inputs to use
inputs = patterns;
# Initialize the weights
weights = np.random.randn(1, 2);
print(weights)
bias = np.random.randn(1, 1)
# Learning rate
eta = 0.2;
# Weight decay
lambdaW = 0.001
```

    [[0.49535084 0.05333204]]

Now you need to update the following code loop to ensure that your neuron learns to separate between the classes
***

**Exercise**
  1. Update the loop so that it computes the forward propagation error
  2. Update the loop to perform learning (based on back-propagation)
  3. Run the learning procedure, which should produce a result similar to that displayed on the website
  4. Perform multiple re-runs by **tweaking the hyperparameters** (learning rate, weight decay)
  5. What observations can you make on the learning process?
  6. (Optional) Change the input patterns, and confirm your observations.
  6. (Optional) Incorporate the bias in the weights to obtain a **vectorized** code.

***

```python
beta=1
def T_func (x):
    return 1/(1+np.exp(-beta*x))
x=np.linspace(-10,10,500)
plt.figure()
plt.plot(x,T_func(x))
# Plot the corresponding pattern
fig = plot_patterns(patterns,desired);
plt.draw()
# Update loop

for i in range(400):

    ######################
    output=T_func(np.sum(weights.T*inputs+bias[0,0],axis=0))
    error=desired-output
    bias[0,0]=bias[0,0]+eta*np.mean(error*beta*output*(1-output))
    #for j in range(2):
    #    weights[0,j]=weights[0,j]+eta*np.mean(error*beta*output*(1-output)*inputs[j,:])#-lambdaW*weights[0,j]**2
    weights=weights+eta*np.mean(error*beta*output*(1-output)*inputs,axis=1)#-lambdaW*weights[0,j]**2
    ######################

    #print('%2d.  weights = %f, %f, %f' % (i, bias[0, 0], weights[0, 0], weights[0, 1]))
    if (i % 10)==0:
        plot_boundary(np.concatenate((bias, weights), axis=1), i, '--', fig)
        plt.draw()
    #time.sleep(0.2)
plot_boundary(np.concatenate((bias, weights), axis=1), i, '-', fig);
```

### 2-layer XOR problem

In most cases, classification problems are far from being linear. Therefore, we need more advanced methods to be able to compute non-linear class boundaries. The advantage of neural networks is that the same principle can be applied in a *layer-wise* fashion. This allows to further discriminate the space in sub-regions (as seen in the course). We will try to implement the 2-layer *perceptron* that can provide a solution to the infamous XOR problem. The idea is now to have the output of the first neurons to be connected to a set of other neurons. Therefore, if we take back our previous formulation, we have the same output for the first neuron(s) $y$, that we will now term as $y_{1}$. Then, we feed these outputs to a second layer of neurons, which gives

$$
\begin{equation}
y_{2}=\sigma\left(\sum_{i = 1}^{n}w_{i}.y_{1}^{i} + b\right)
\end{equation}
$$

Finally, we will rely on the same loss $\mathcal{L_{D}}$ as in the previous exercise, but the outputs used are $y_2$ instead of $y$. As in the previous case, we now need to compute the derivatives of the weights and biases for several layers . However, you should see that some form of generalization might be possible for any number of layer.

***

**Exercise**
  1. Perform the derivatives for the last layer specifically
  2. Define a generalized derivative for any previous layer

***

We can construct the prototypical set of XOR values by using the following code (note that this is the most simple case, but still this is typically a problem that cannot be solved by a _linear classifier_

```python
patterns = np.array([[-1, -1],[-1,  1],[1, -1],[1,  1]]).transpose() # Input patterns
desired = np.array([0, 1, 1, 0])                       # Corresponding classes
# Initialize based on their sizes
nInputs = patterns.shape[0]
nOutputs = 1
nPat = patterns.shape[1]
# First plot the patterns
fig = plot_patterns(patterns, desired);
```

The variables that will be used by your code are the following.

```Matlab
patterns          % 2 x n matrix of random points
desired           % classes of the patterns
inputs1           % 3 x n final matrix of inputs (accounting for bias)
nHiddens          % Number of hidden units
learnRate         % Learning rate parameter
momentum          % Momentum parameter (bonus)
weights1          % 1st layer weights
weights2          % 2nd layer weights
TSS_Limit         % Sum-squared error limit
```

```python
nHiddens = 2           # Number of hidden units
learnRate = 0.001       # Learning rate parameter
momentum = 0.1         # Momentum parameter
# Overall input patterns
inputs = patterns
TSS_Limit = 0.02                                           # Sum-squared error limit
# Learning rate
eta = 0.4
#options
sparse_coeff=0
lambda_coeff=0.0001
```

***

**Exercise**

  1. Update the forward propagation and error computation (compared to desired).
  2. Update the back-propagation part to learn the weights of both layers.
  3. Run the learning, which should produce a result similar to that displayed below.
  4. Perform multiple re-runs of the learning procedure (re-launching with different initializations)
  5. What observations can you make on the learning process?
  6. What happens if you initialize all weights to zeros?
  7. (Optional) Implement the *sparsity* constraint in your neural network.
  7. (Optional) Implement the *weight decay* constraint in your network.
  7. (Optional) Add the *momentum* to the learning procedure.

*For optional questions, please look after the first code box for more information*

***

```python
# Weights of first and second layer
weights1 = (np.random.randn(nHiddens, nInputs) - 0.5)      # 1st layer weights
weights2 = (np.random.randn(nOutputs, nHiddens) - 0.5)     # 2nd layer weights
bias1 = (np.random.randn(nHiddens, 1) - 0.5)               # 1st layer biases
bias2 = (np.random.randn(nOutputs, 1) - 0.5)               # 2nd layer biases
# First plot the patterns
fig = plot_patterns(patterns, desired);
# Iterate for a fixed number of iterations
deltaL_prev=0
delta_hidden_prev=0
loss=[0]*10000
for epoch in range(10000):

    ######################
    out_hidden=T_func(np.dot(weights1.T,inputs)+bias1)
    outputs=T_func(np.dot(weights2,out_hidden)+bias2)

    error=desired-outputs
    loss[epoch]=np.sum(error**2)
    #backpropagation
    deltaL=outputs*(1-outputs)*error+momentum*deltaL_prev
    delta_hidden=out_hidden*(1-out_hidden)*weights2.T*deltaL+momentum*delta_hidden_prev

    weights2+=eta*(np.mean(deltaL*out_hidden,axis=1)-2*lambda_coeff*weights2-sparse_coeff*np.sign(weights2))
    bias2+=eta*np.mean(deltaL,axis=1)
    weights1+=eta*(np.mean(delta_hidden*inputs,axis=1)-2*lambda_coeff*weights1-sparse_coeff*np.sign(weights1))
    bias1+=eta*np.mean(delta_hidden,axis=1).reshape(2,1)

    deltaL_prev=deltaL
    delta_hidden_prev=delta_hidden
    ######################

    #print('Epoch %3d:  Error = %f' % (epoch, TSS));
    #if TSS < TSS_Limit:
    #    break
    if (epoch % 1000)==0:
        plot_boundary(np.concatenate((bias2, weights2), axis=1), epoch, '--', fig)

plot_boundary(np.concatenate((bias1, weights1), axis=1), epoch, '-', fig)
plt.figure()
plt.plot(loss)
```

```python
print(outputs)
print(desired)
```

    [[0.07343813 0.92411483 0.92732521 0.07487704]]
    [0 1 1 0]

**Optional questions**

2. *Weight decay* constraint

As nothing constrains the weights in the network, we can note that usually all weights vector given a multiplicative factor might be equivalent, which can stall the learning (and lead to exploding weights). The *weight decay* allows to regularize the learning by penalizing weights with a too wide amplitude. The idea is to add this constraint as a term to the final loss (which leads to an indirect "pressure" on the learning process. Therefore, the final loss will be defined as

$$
\begin{equation}
\mathcal{L}_{final}=\mathcal{L_D} + \lambda \sum_{l} \sum_{i} \sum_{j} \left( W_{ij}^{l} \right)^{2}
\end{equation}
$$

where the parameter $\lambda$ controls the relative importance of the two terms.

3. *Momentum* in learning

Usually, in complex problems, the gradient can be very noisy and, therefore, the learning might oscillate widely. In order to reduce this problem, we can *smooth* the different gradient updates by retaining the values of the gradient at each iteration and then performing an update based on the latest gradient $\delta_{i}^{t}$ and the gradient at the previous iteration $\delta_{i}^{t-1}$. Therefore, a gradient update is applied as

$$
\begin{equation}
\delta_{final}^{t} = \delta_{i}^{t} + m.\delta_{i}^{t-1}
\end{equation}
$$

with $m$ the momentum parameter, which control the amount of gradient smoothing.

## 3-layer audio classification

Finally, we will attack a complete audio classification problem and try to perform neural network learning on a set of audio files. The data structure will be the same as the one used for parts 1 and 2. As discussed during the courses, even though a 2-layer neural network can provide non-linear boundaries, it can not perform "holes" inside those regions. In order to obtain an improved classification, we will now rely on a 3-layer neural network. The modification to the code of section 3.2 should be minimal, as the back-propagation will be similar for the new layer as one of the two others. We do not develop the math here as it is simply a re-application of the previous rules with an additional layer (which derivatives you should have generalized in the previous exercise).

However, up until now, we only performed *binary classification* problems, but this time we need to obtain a decision rule for multiple classes. Therefore, we cannot rely on simply computing the distance between desired patterns and the obtained binary value. The idea here is to rely on the *softmax regression*, by considering classes as a vector of probabilities. The desired answers will therefore be considered as a set of *probabilities*, where the desired class is $1$ and the others are $0$ (called *one-hot* representation). Then, the cost function will rely on the softmax formulation

$$
\begin{equation}
\mathcal{L_D}(\theta) = - \frac{1}{m} \left[ \sum_{i=1}^{m} \sum_{j=1}^{k} 1\left\{y^{(i)} = j\right\} log \frac{e^{\theta_{j}^{T} x^{(i)}}}{\sum_{l=1}^{k} e^{ \theta_{l}^{T} x^{(i)} }}  \right]
\end{equation}
$$

Therefore, we compute the output of the softmax by taking

$$
\begin{equation}
p(y^{(i)} = j | x^{(i)}; \theta) = \frac{e^{\theta_{j}^{T} x^{(i)}}}{\sum_{l=1}^{k} e^{ \theta_{l}^{T} x^{(i)}} }
\end{equation}
$$

By taking derivatives, we can show that the gradient of the softmax layer is

$$
\begin{equation}
\nabla_{\theta_{j}} \mathcal{L_D}(\theta) = - \frac{1}{m} \sum_{i=1}^{m}{ \left[ x^{(i)} \left( 1\{ y^{(i)} = j\}  - p(y^{(i)} = j \mid x^{(i)}, \theta) \right) \right]}
\end{equation}
$$

### Sweet activation functions

As discussed in the course, the interest of stacking layers is that there is an _activation function_, which allows non-linear interactions between the dimensions (and avoids to only compute a single huge affine transform). Although the `sigmoid` function has been historically the most used, there has been some large developments since. Notably the `ReLU` (Rectified Linear Unit) is one of the major difference in modern networks (we will see more about that in a later course)

```python
# Function for computing the Sigmoid activation
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def dsigmoid(a):
    return a * (1.0 - a)
# Function for computing the ReLU activation
def relu(x):
    return np.maximum(0, x)
def drelu(x):
    return 1 / (1 + np.exp(-x))
# Function for computing the Tanh activation
def tanh(x):
    return np.tanh(x);
def dtanh(x):
    return np.cosh(x) ^ -2
```

Here, we plot some simple examples of what these activation functions look like. You can try to rely on these functions in your previous training code and witness the differences in training.

```python
from helper_plot import prep_plots, finalize_plots
funcs = [('sigmoid',sigmoid,'red'), ('tanh',tanh,'orange'), ('relu',relu,'yellow')]
axes = prep_plots([funcs[0][0], funcs[1][0], funcs[2][0]], fig_size=(18,5), fig_num=1)
x_plot = np.linspace(-5,5,100)
np.maximum(0, x_plot)
for f in range(3):
    axes[f].plot(x_plot, funcs[f][1](x_plot), color=funcs[f][2], linewidth=4, label=funcs[f][0])
finalize_plots(axes, fig_title="Activation functions")
```

## Coding the whole network from scratch

You should now have all the tools necessary to apply neural networks from scratch to a more complex problem. In the following exercise, we simply removed any guideline code, and you need to code all the procedure for training a NN and **apply it to audio data**. You will use the spectral features discussed in the previous exercise as an input.

***

**Exercise**

  1. Based on the previous neural network, upgrade the code to a 3-layer neural network
  2. Implement the *softmax regression* on top of your 3-layer network
  3. Use the provided code to perform classification on a pre-defined set of features
  4. As previously, change the set of features to assess their different accuracies
  5. Evaluate the neural network accuracy for all features combinations
  6. What happens if the learning rate is too large ? What is this phenomenon ?
  7. (Optional) Perform a more advanced visualization of the learning process.

***

#### Structure du réseau

```python
#couche d'entrée
InputsS = 6 #nombre de neurones sur la couche d'entrée
nInputs = 5 #nombre de données d'entrée
#couches intermédiaires
nLayers = 3
nHiddens = 5 #nombre de neurones sur les couches intermédiaires
#couche de sortie
nOutputs = 16
```

#### DOnnées d'entrée

```python
inputs=np.ones((5,2))
```

#### Paramètres d'apprentissage

```python
learnRate = 0.001       # Learning rate parameter
momentum = 0.1         # Momentum parameter
# Overall input patterns
TSS_Limit = 0.02                                           # Sum-squared error limit
# Learning rate
eta = 0.4
#options
sparse_coeff=0.0001
lambda_coeff=0.0001
```

#### Initialisation

```python
# Weights of first and second layer
weightsIn = (np.random.randn(InputsS, nInputs) - 0.5)      # Weights layer d'entrée
weightsOut = (np.random.randn(nOutputs, nHiddens) - 0.5)     # Weights layer de sortie
weightsHidden1 = (np.random.randn(nHiddens, InputsS) - 0.5)   #Weights premier hidden layer
if nLayers>1:
    weightsHidden = [(np.random.randn(nHiddens, nHiddens) - 0.5) for i in range(nLayers-1)]  #Weights hidden layers

bias1 = (np.random.randn(1, InputsS) - 0.5)               # Entry layer biases
bias2 = (np.random.randn(1, nOutputs) - 0.5)               # Out layer biases
biasHidden = (np.random.randn(nLayers, nHiddens) - 0.5)     # Hidden layers biases
# First plot the patterns

# Iterate for a fixed number of iterations
deltaL_prev=0
delta_hidden_prev=0
loss=[0]*10000

out_hidden=np.zeros((nLayers,nHiddens))
```

```python

```

```python
for i in range(1,2):
    print(i)
```

    1

#### Entrainement

```python
# Iterate for a fixed number of iterations
for epoch in range(500):

    ######################
    #Forward propagation
    #Input Layer
    out_input = np.dot(weightsIn,inputs.T)+bias1 #Sortie avant fonction d'activation
    out_input = out_input #Après fonction d'activation

    #First Hidden layer
    out_line = np.dot(weightsHidden1,out_input.T)+biasHidden[0].reshape(nHiddens,1)
    out_hidden[0,:] = out_line.reshape(nHiddens)
    out_hidden[0,:] = out_hidden[0,:]
    #Other Hidden Layers
    for i in range(1,nLayers) :
        out_line = np.dot(weightsHidden[i-1],out_hidden[0,:].T)+biasHidden[i].reshape(1,nHiddens)
        out_hidden[i,:] = out_line.reshape(nHiddens)

    #Output Layer
    outputs=(np.dot(weightsOut,out_hidden[nLayers-1,:].reshape(nHiddens,1))+bias2.reshape(nOutputs,1)).T #Sortie avant fonction d'activation
    outputs=outputs #Après fonction d'activation

    """
    ##############
    error=desired-outputs
    loss[epoch]=np.sum(error**2)
    #backpropagation
    deltaL=outputs*(1-outputs)*error+momentum*deltaL_prev
    delta_hidden=out_hidden*(1-out_hidden)*weights2.T*deltaL+momentum*delta_hidden_prev

    weights2+=eta*(np.mean(deltaL*out_hidden,axis=1)-2*lambda_coeff*weights2-sparse_coeff*np.sign(weights2))
    bias2+=eta*np.mean(deltaL,axis=1)
    weights1+=eta*(np.mean(delta_hidden*inputs,axis=1)-2*lambda_coeff*weights1-sparse_coeff*np.sign(weights1))
    bias1+=eta*np.mean(delta_hidden,axis=1).reshape(2,1)

    deltaL_prev=deltaL
    delta_hidden_prev=delta_hidden
    ######################

    print('Epoch %3d:  Error = %f\n' % epoch, TSS);
    if TSS < TSS_Limit:
        break
plo_boundary(np.concatenate(bias1, weights1), epoch, '-');
    """
```

```python
outputs
```

    array([[ 70.42226608,   4.7112851 , -26.64462671,  92.34633111,
            154.81953653, 247.7670092 , -26.17720895,  27.6813103 ,
              8.32751212,  21.76950085,  81.78149506,  46.20283095,
            164.39043612,   5.53618167,  29.35867026, -35.35415959]])

```python
out_input
```

    array([[ 0.13385349, -5.85639205, -1.93544627, -1.99704253, -3.38893765,
            -0.84334717]])

```python
out_hidden[0]
```

    array([ 0.21383226, 21.63732873,  7.77252951,  1.5625695 , 13.05477784])

```python
weightsHidden1
```

    array([[ 0.23417693,  0.15465218, -2.20514202,  0.19922594,  0.43572309],
           [ 0.08814463,  0.0476187 ,  0.56392929, -2.69049748, -1.28021802],
           [ 1.02261729, -0.02580423, -0.17268853, -0.88280955,  0.25610718],
           [-0.93405185,  0.05106439, -0.12841387, -0.67851671, -1.26372586],
           [-1.59020093, -1.47292629, -0.13376435,  0.45529411,  0.15879279],
           [ 0.37104001, -1.01940478,  1.71356566,  0.50818289, -0.35745936]])

```python
np.dot(weightsIn,inputs.T)+bias1.T
```

    array([[-5.94042325, -0.91678248, -1.5795785 , -3.64397425, -2.99679312,
            -2.32534787]])

## Using Pytorch to enjoy life

Up to now, we have been writing every operations by ourselves (in order to better understand the mathematics behind NN). However, there exists of course some simplifying libraries that provide large simplifications to this question.

One of the most powerful and complete library of this sort is `Pytorch`, which has been developed for several years (even prior to the recent boom of deep learning). `Pytorch` provides a large set of pre-coded layers, but also **computational graphs** and **autograd**, which are very powerful paradigms allowing to define complex operators and automatically taking derivatives.

### Defining our network

When building neural networks we frequently think of arranging the computation into layers, some of which have learnable parameters which will be optimized during learning. In `PyTorch`, the `nn` package provides higher-level abstractions over raw computational graphs that are useful for building neural networks. The `nn` package defines a set of `Modules`, which are roughly equivalent to neural network layers. A `Module` receives input `Tensors` and computes output `Tensors`, but may also hold internal state such as `Tensors` containing learnable parameters. The nn package also defines a set of useful loss functions that are commonly used when training neural networks.

In the following example, we use the `nn` package to show how easy it is to instantiate our previous three-layers network

```python
import torch
# Define the input dimensions
in_size = 1000
# Number of neurons in a layer
hidden_size = 100
# Output (target) dimension
output_size = 10
# Use the nn package to define our model and loss function.
model = torch.nn.Sequential(
    torch.nn.Linear(in_size, hidden_size),
    torch.nn.ReLU(),
    torch.nn.Linear(hidden_size, hidden_size),
    torch.nn.ReLU(),
    torch.nn.Linear(hidden_size, output_size),
    torch.nn.Softmax()
)
```

### Optimizing the network

Up to this point we have updated the weights of our models by manually performing the gradient descent algorithm (changing the parameters vectors). Although this is not a huge burden for simple optimization algorithms like stochastic gradient descent, in practice we often train neural networks using more sophisticated optimizers like AdaGrad, RMSProp or Adam (that we will see later in this course)

The `optim` package in PyTorch abstracts the idea of an optimization algorithm and provides implementations of commonly used optimization algorithms, and greatly simplfies the training loop associated with training a neural network.

For the sake of presentation we will use random inputs $\mathbf{x}$ that should be matched with random outputs $\mathbf{y}$

```python
batch_size = 64
# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)
```

In the following example we optimize the model using the Adam algorithm provided by the `optim` package, based on a `MSE` loss.

```python
# Learning rate
learning_rate = 1e-4
# Loss function that we will use
loss_fn = torch.nn.MSELoss(reduction='sum')
# Optimizer to fit the weights of the network
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
for t in range(500):
    # Forward pass: compute predicted y by passing x to the model.
    y_pred = model(x)
    # Compute the loss.
    loss = loss_fn(y_pred, y)
    # Before the backward pass, zero all of the network gradients
    optimizer.zero_grad()
    # Backward pass: compute gradient of the loss with respect to parameters
    loss.backward()
    # Calling the step function to update the parameters
    optimizer.step()
```

## Using Pytorch to classify audio

Now that we know the main components of `Pytorch` to define and optimize networks, your assignement is to define a complete classification problem from audio data, by relying on this toolbox

***

**Exercise**

  1. Use `Pytorch` to define a model for audio classification
  2. Import the audio features dataset and check that your model produces an output
  3. Write the optimization loop (think carefully about the _loss function_
  4. As previously, change the set of features to assess their different accuracies
  5. Think of how you could use more complex features (time series, audio, STFT) to classify your data

***

```python

######################
# YOUR CODE GOES HERE
######################

```


# Axion-electron coupling limits vs axion mass

Axion mass-PQ scale relation:
\begin{equation}
m_{a}=5.70(7) \mu \mathrm{eV}\left(\frac{10^{12} \,\mathrm{GeV}}{f_{a}}\right)
\end{equation}

Axion-electron coupling
\begin{equation}
g_{ae} \equiv \frac{C_{ae} m_{e}}{f_{a}}=8.943 \times 10^{-11} C_{ae}\frac{m_a}{\mathrm{eV}}
\end{equation}

Model dependent constant:
\begin{equation}
C_{ae}  =
\begin{cases}
  2\times 10^{-4} & {\rm KSVZ} \\
  [0.024,\frac{1}{3}] & {\rm DFSZ\,I} \\
  [-\frac{1}{3},0] & {\rm DFSZ\,II}
\end{cases}
\end{equation}
In DFSZ the lepton mass can come from either coupling to $H_u$ or $H_d$, so $C_{ae} = -C_{au}$ or $C_{ad}$. The range of values for DFSZ I and II come from the perturbativity of the Yukawa couplings with sets the range $0.28<v_{u} / v_{d}<140$ for the Higgs vevs.

```python
% matplotlib inline
from numpy import loadtxt
import matplotlib.pyplot as plt
from PlotFuncs import FigSetup, BlackHoleSpins, AxionElectron, MySaveFig

fig, ax = FigSetup(Shape='Rectangular', ylab=r'$|g_{ae}|$', mathpazo=True,
                   g_min=1e-15, g_max=1e-10, m_min=1e-9, m_max=1e5, xtick_rotation=0)

AxionElectron.QCDAxion(ax)
AxionElectron.LUX(ax)
AxionElectron.PandaX(ax)
AxionElectron.XENON1T(ax)
AxionElectron.SolarBasin(ax)
AxionElectron.SuperCDMS(ax)
AxionElectron.EDELWEISS(ax)
AxionElectron.RedGiants(ax)
AxionElectron.SolarNu(ax)
AxionElectron.QUAX(ax)

MySaveFig(fig, 'AxionElectron')
```

```python
fig, ax = FigSetup(Shape='Rectangular', ylab=r'$|g_{ae}|$', mathpazo=True,
                   g_min=1e-15, g_max=1e-10, m_min=1e-9, m_max=1e5, xtick_rotation=0)

AxionElectron.QCDAxion(ax)
AxionElectron.LUX(ax)
AxionElectron.PandaX(ax)
AxionElectron.XENON1T(ax)
AxionElectron.SolarBasin(ax)
AxionElectron.SuperCDMS(ax)
AxionElectron.EDELWEISS(ax, text_on=False)
AxionElectron.RedGiants(ax)
AxionElectron.SolarNu(ax)
AxionElectron.QUAX(ax)

AxionElectron.DARWIN(ax)
AxionElectron.LZ(ax)
AxionElectron.Semiconductors(ax)

AxionElectron.Magnon(ax)
AxionElectron.MagnonScan(ax)

MySaveFig(fig, 'AxionElectron_with_Projections')

```

```python
fig, ax = FigSetup(Shape='Rectangular', ylab=r'$|g_{ae}|$', mathpazo=True,
                   g_min=1e-15, g_max=1e-10, m_min=1e0, m_max=5e5, xtick_rotation=0)

AxionElectron.QCDAxion(ax, DFSZ_on=True, KSVZ_on=False, Hadronic_on=False)

plt.gcf().text(0.16, 0.23, r'{\bf DFSZ axions}', fontsize=30, rotation=37, alpha=1)

AxionElectron.LUX(ax, text_pos=[1.2e0, 6e-12])
AxionElectron.PandaX(ax, fs=25, text_pos=[1.6e3, 1.04e-13], rotation=55, alpha=0.8, rotation_mode='anchor')
AxionElectron.XENON1T(ax, text_shift=[2, 1.4], fs=22)
AxionElectron.SolarBasin(ax)
AxionElectron.SuperCDMS(ax, text_pos=[5e1, 2.5e-11], rotation=-72)
AxionElectron.EDELWEISS(ax, text_pos=[1.3e5, 2.4e-11], fs=20, rotation=40)
AxionElectron.RedGiants(ax, text_pos=[1.2e0, 2e-13])
AxionElectron.SolarNu(ax, text_pos=[1.2e0, 4e-11])
AxionElectron.DARWIN(ax, text_pos=[1.2e3, 2.5e-14], rotation=10, rotation_mode='anchor')
AxionElectron.LZ(ax, fs=25, rotation=30, rotation_mode='anchor')

MySaveFig(fig, 'AxionElectron_UndergroundDetectorsCloseup')

```

```python

```


```python
import pickle
import sys
sys.path.insert(0, '../')
from AircraftIden.data_case.GeneralAircraftCase import GeneralAircraftCase, PX4AircraftCase
from AircraftIden import FreqIdenSIMO, TransferFunctionFit,TransferFunctionModel,TransferFunctionParamModel
import numpy as np
import matplotlib.pyplot as plt
import math
import sympy as sp
sp.init_printing()
pkl_name = "./log_34_2018-4-10-freq-res-ele.pkl"
ele_freqres = pickle.load(open("./log_34_2018-4-10-freq-res-ele.pkl","rb"))
```

```python
def process_status_freqres(freqres):
    freq, H, gamma2, gxx, gxy, gyy = freqres.get_freq_iden(0)
    a,b,c,d,tau,s = sp.symbols("a b c d tau s")

    num = d
    den = s*(b *s + c)
    tfpm = TransferFunctionParamModel(num,den,tau)
    fitter = TransferFunctionFit(freq, H, gamma2, tfpm,nw=20,iter_times=1000,reg = 0)
    tf = fitter.estimate(10,accept_J=20)
    return fitter
#fitter = process_status_freqres(ele_freqres["1m/s"])

```

```python
status_list = ["1m/s","3m/s","5m/s","8m/s","10m/s"]

def process_status(x):
    print("Status process flying status {}".format(x))
    return process_status_freqres(ele_freqres[x])
ret = map(process_status,status_list)
ret = dict(zip(status_list, ret))
```

    Status process flying status 1m/s
    Uknown number 4 [c, b, tau, d]
    Found new better 12.198545441564024
    Status process flying status 3m/s
    Uknown number 4 [c, b, tau, d]
    Found new better 11.527719083912041
    Status process flying status 5m/s
    Uknown number 4 [c, b, tau, d]
    Found new better 10.834112838031
    Status process flying status 8m/s
    Uknown number 4 [c, b, tau, d]
    Found new better 18.791264925795723
    Status process flying status 10m/s
    Uknown number 4 [c, b, tau, d]
    Found new better 17.690477673002434

```python
plt.rc('figure', figsize=(20.0, 10.0))
def plot_fitter(fitter, status_name):
    plt.figure(status_name)
    fitter.plot(status_name+":")
    trans_str = fitter.latex()
    plt.text(20, 0,trans_str, fontsize=20,color='red')
    plt.show()

def poly_latex(poly,cha = "s"):
    ret_str = ""
    ords = len(poly) - 1
    for i in range(ords+1):
        ordn = ords - i
        if ordn == 0:
            ret_str = ret_str + "{:4.2f}".format(poly[i])
        else:
            if (poly[i] == 1):
                ret_str = ret_str + "s^{:d}+".format(ordn)
            else:
                ret_str = ret_str + "{:4.2f} s^{:d}+".format(poly[i],ordn)
    return ret_str

def transfer_func_latex(num,den,tau):
    return r"$\frac{" + poly_latex(num) + "}{" + poly_latex(den) +"}" + "e^{-" + "{:4.3f}".format(tau) + "t}$"

for status in ret:
    plot_fitter(ret[status],status)
```

```python
plt.rc('figure', figsize=(20.0, 10.0))
fitter.plot()
plt.text(20,0,fitter.latex(), fontsize=20,color='red')
```

```python

```


# A study on spontaneous decay rate of an atom in presence of a square dielectric waveguide using BEM approach

In these notes, I calculate the Local Density of States (LDOS), or the imaginary part of the on-site Green's function and hence the modified spontaneous emission rate of an atom in presence of a square dielectric waveguide using the Boundary Element Method (BEM). The BEM code is from Prof. Alejandro Manjavacas's group.

This is an [IJulia notebook](https://github.com/JuliaLang/IJulia.jl), which provides a nice
browser-based [Jupyter](http://jupyter.org/) interface to the [Julia language](http://julialang.org/), a high-level dynamic language (similar to Matlab or Python+SciPy) for technical computing.  The notebook allows us to combine code and results in one place.

We are only manipulating the generated data from the simulation results in this notebook. As a brief recap of the simulation process, I have used a compiled BEM code by Alejandro's group written in C++ called `bem2D` on a cluster computing system. A configuration C++ code is defined in the file `scripts_ldos.cpp` which is in the same folder as `bem2D`. I compiled the script and put the generated executable into another folder called `p3`, for example, by
```
g++ scripts_ldos.cpp -lstdc++ -o ../p3/scripts_ldos
```
Then ran the executable and submitted the generated PBS script to the cluster system to run the simulation:
```
cd ../p3/
./scripts_ldos
qsub ldos_N_1_lam_894_eps_4_0_a_300_b_300_c_5_x_0_y_350_q_0_2_401.pbs
```
Notice that the name of the PBS script is automatically generated based on the configuration parameters for this simulation. The name will vary for different simulations. After running the script, I got a set of data files--one has the same name as the PBS script but with a `.dat` extension for the data table storing the calculated LDOS values; there are another two `*.dat` files for the geometry of boundary and dieletric function distribution. We will look into those data files in the following sections.

## 3D dielectric waveguide simulated in 2D

Just a little more detail on the simulation itself: By assuming the waveguide along z-axis or the light propagation direction is uniform, one can completely solve the boundary condition problem of dipole emission by simulating the field in one single layer of the xy cross section. The z-component of the field only adds in a phase factor. The data of the simulated result is stored in the /data/ folder of this repo.

We can read in this data as a matrix of numbers by the `readdlm` function in Julia with the `header=false` option meaning that it reads the first line as the beginning of the data entries without a list of strings describing each column.

Now, let's plot the results.  I'll use the [PyPlot](https://github.com/stevengj/PyPlot.jl) package in Julia, which is an interface to the sophisticated [matplotlib](http://matplotlib.org/) Python plotting library.   We'll plot three things:

* The waveguide structure in terms of $\epsilon$ and the interface boundary between two media.
* Plot the LDOS components for a fixed dipole position.
* Calculate the waveguide-modified spontaneous decay rate when the dipole varies its position outside of the waveguide.

### Plotting the boundary and index of refraction profile of the waveguide in the xy cross section

Our boundary points are meshed in the file `/data/p3/geom_regions_a_300_b_300_c_5.data`, which positions where the equivalent charges and current sources to be computed in the BEM simulation. The waveguide has a square cross-section of a width $a=b=300nm$ ($nm$ is the unit of length) and a index of refraction of $n_1=n_{core}=2$ for the waveguide material and $n_2=n_{clad}=1$ for the vacuum clad.

It is good to plot out the mesh of index of refraction in space and find out how good is the mesh resolution. This can be done by plotting out the output eps file in a simple data table format, which ends in `.dat`.

The following will first print out some of the data in order to figure out the physics meaning of the dimensions. They should contain the coordinate and index of refraction information for the simulation.

```julia
boundary = readdlm("data/geom_regions_a_300_b_300_c_5.dat", header=false);
boundarypoints = boundary[:,1:3]
```

    10201×3 Array{Float64,2}:
     -180.0  -180.0  1.0
     -180.0  -176.4  1.0
     -180.0  -172.8  1.0
     -180.0  -169.2  1.0
     -180.0  -165.6  1.0
     -180.0  -162.0  1.0
     -180.0  -158.4  1.0
     -180.0  -154.8  1.0
     -180.0  -151.2  1.0
     -180.0  -147.6  1.0
     -180.0  -144.0  1.0
     -180.0  -140.4  1.0
     -180.0  -136.8  1.0
        ⋮
      180.0   140.4  1.0
      180.0   144.0  1.0
      180.0   147.6  1.0
      180.0   151.2  1.0
      180.0   154.8  1.0
      180.0   158.4  1.0
      180.0   162.0  1.0
      180.0   165.6  1.0
      180.0   169.2  1.0
      180.0   172.8  1.0
      180.0   176.4  1.0
      180.0   180.0  1.0

```julia
epsilon3D = readdlm("data/geom_a_300_b_300_c_5.dat", header=false);
epsilon2Dpoints = epsilon3D[:,[1,2,4]]
```

    620×3 Array{Float64,2}:
        0.966667   150.0    2.0
        2.9        150.0    2.0
        4.83333    150.0    2.0
        6.76667    150.0    2.0
        8.7        150.0    2.0
       10.6333     150.0    2.0
       12.5667     150.0    2.0
       14.5        150.0    2.0
       16.4333     150.0    2.0
       18.3667     150.0    2.0
       20.3        150.0    2.0
       22.2333     150.0    2.0
       24.1667     150.0    2.0
        ⋮
     -150.0        142.1    2.0
     -150.0        144.033  2.0
     -149.938     -145.782  1.0
     -149.455     -147.27   1.0
     -148.536     -148.536  1.0
     -147.27      -149.455  1.0
     -145.782     -149.938  1.0
     -145.782      149.938  1.0
     -147.27       149.455  1.0
     -148.536      148.536  1.0
     -149.455      147.27   1.0
     -149.938      145.782  1.0

Now we plot out the data in a 2D (xy) plane.

```julia
using PyPlot
#println(convert(Int64,floor(lenz/2)))
x = boundarypoints[:,1];
y = boundarypoints[:,2];
v_regions = boundarypoints[:,3];
lenx = length(x)

fig = figure("Boundary points plot",figsize=(10,10))
ax = fig[:add_subplot](1,2,1)
c = get_cmap("PRGn")
rgbs = [c(norm(value/2.)) for value in v_regions]
scatter(x,y,c=rgbs,linewidths=0,marker=".",s=5)

xlabel(L"x/nm")
ylabel(L"y/nm")
axis("image")
xlim(-250,250)
ylim(-250,250)
tight_layout()
title("meshing points (x,y)")
display(maximum(abs.(x)))

subplot(1,2,2)
x_eps = epsilon2Dpoints[:,1]; y_eps = epsilon2Dpoints[:,2]; v_eps = epsilon2Dpoints[:,3];
ax = fig[:add_subplot](1,2,2)
c = get_cmap("RdBu")
rgbs = [c(norm(value)) for value in v_eps]
scatter(x_eps,y_eps,c=rgbs,s=1)

xlabel(L"x/ nm")
ylabel(L"y/ nm")
axis("image")
xlim(-250,250)
ylim(-250,250)
tight_layout()
title("boundary (x,y)");
```

The first plot only shows the 101$\times$101 data points, in which the plotted area is the computing region of a $180nm\times 180nm$ square with the waveguide region colored in green. The figure on the right covers the boundary points pretty densely, although the plot didn't resolve the image on the corners clearly.

# Plotting the LDOS components with a fixed dipole position

To calculate the modified decay rates, we need to use the LDOS value at the dipole position. The result is calculated at a series of $k$ points. I expect to see a continuous positive curve when $k\in [0,1]\omega/c$ or in the radiative mode regime and a single positive spark in the $[1,2]\omega/c$ or guided mode regime--given the waveguide is a single-mode waveguide.

```julia
ldos = readdlm("data/ldos_N_1_lam_894_eps_4_0_a_300_b_300_c_5_x_0_y_350_q_0_2_401.dat", header=false);
#ldos = readdlm("data/ldos_N_1_lam_894_eps_4_0.001_a_300_b_300_c_5_x_0_y_350_q_0.1_4_101.dat", header=false);
ldosqpoints = ldos[:,[2,5,6,7,8]]
using PyPlot
#println(convert(Int64,floor(lenz/2)))
q = ldosqpoints[:,1]
ldosx = ldosqpoints[:,2]
ldosy = ldosqpoints[:,3]
ldosz = ldosqpoints[:,4]
ldos_av = ldosqpoints[:,5]
lenx = length(q)

fig = figure("LDOS q plot",figsize=(10,10))
ax = fig[:add_subplot](1,2,1)
cp = ax[:plot](q, ldosx, "b-", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS_x")
axis("image")
xlim(-0.0,2)
ylim(-2,3)
tight_layout()
gcf() # Needed for IJulia to plot inline
display(maximum(abs.(ldosx)))

ax = fig[:add_subplot](1,2,2)
cp = ax[:plot](q, ldos_av, "r-", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS")
axis("image")
xlim(-0.0,2)
ylim(-2,3)
tight_layout()
gcf() # Needed for IJulia to plot inline
display(maximum(abs.(ldosx)))
```

As you can see, ***there are negative sparks in the guided mode regime***. It could be a numerical error in the code and may be removed using some tricks.

Now we can integrate the $\mathrm{LDOS}_i$ values over $k$ along the whole axis to obtain the total decay rate or from $0$ to $\omega/c$ to obtain the radiative mode contribution for the decay rate. Each integral can be performed as a sum over all discrete points along the $k$-axis using the Trapezoid approximation as below:
$$\begin{align}\int \mathrm{LDOS}_i(k)dk &= \sum_{j=1,\cdots,N-1} \frac{\mathrm{LDOS}_i(k_j)+\mathrm{LDOS}_i(k_j+1)}{2}\Delta k\\
&= \left(\frac{\mathrm{LDOS}_i(k_N)+\mathrm{LDOS}_i(k_N)}{2}+\sum_{j=2,\cdots,N-1}\mathrm{LDOS}_i(k_j)\right)\Delta k,\end{align}$$
where $N$ is the total number of data points along the $k$-axis, and $\Delta k=k_j-k_{j-1}=k_2-k_1$ as the interval of $k$-vector in a uniform distribution manner.

In our case, the integrand is discontinuous and breaks the continuity at $k=k_0=\omega/c$ point. Therefore, we divide our integration limit into the $[0,n_2)k_0$ and $(n_2,n_1]k_0$ two regions corresponding to radiation mode contribution and guided mode contribution parts, where $n_2=1$ is the index of refraction of the vacuum clad and $n_1=2$ is the index of refraction of the waveguide bulk material. Since there are surdden jumps in the guided mode regime, the integration may have some error using current method, but it shouldn't be too large as the jumps are in small intervals.

```julia
length_of_q=length(ldosqpoints[:,1])
breakpoint = Int(floor((length_of_q-1)/2)) # The breaking point of index is chosen under the fact that the radiation contribution part takes a half space for $n_1=2$.
using NumericalIntegration
ldos_x_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,2])
ldos_x_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,2])
ldos_x = ldos_x_rad+ldos_x_guide;
ldos_y_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,3])
ldos_y_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,3])
ldos_y = ldos_y_rad+ldos_y_guide;
ldos_z_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,4])
ldos_z_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,4])
ldos_z = ldos_z_rad+ldos_z_guide;
ldos_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,5])
ldos_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,5])
ldos_total = ldos_rad + ldos_guide;
# Print out the result.
@printf("\Delta k = %4f k_0.\n",ldosqpoints[2,1]-ldosqpoints[1,1])
@printf("LDOS_x_rad=%5f, LDOS_x_guide=%5f, LDOS_x=%5f;\n",ldos_x_rad,ldos_x_guide,ldos_x)
@printf("LDOS_y_rad=%5f, LDOS_y_guide=%5f, LDOS_y=%5f;\n",ldos_y_rad,ldos_y_guide,ldos_y)
@printf("LDOS_z_rad=%5f, LDOS_z_guide=%5f, LDOS_z=%5f;\n",ldos_z_rad,ldos_z_guide,ldos_z)
@printf("LDOS_rad = %5f, LDOS_guide = %5f, LDOS = %5f.",ldos_rad,ldos_guide,ldos_total)
```

    Delta k = 0.005000 k_0.

Notice that the result above was calculated using a $k$-resolution of $\Delta k= 0.005k_0$. We can compare the results above with a coarser gridding case with $\Delta k= 0.01k_0$. The LDOS values can be then calculated as below.

```julia
ldos = readdlm("data/ldos_N_1_lam_894_eps_4_0_a_300_b_300_c_5_x_0_y_350_q_0_2_201.dat", header=false);
ldosqpoints = ldos[:,[2,5,6,7,8]]
length_of_q=length(ldosqpoints[:,1])
breakpoint = Int(floor((length_of_q-1)/2)) # The breaking point of index is chosen under the fact that the radiation contribution part takes a half space for $n_1=2$.
using NumericalIntegration
ldos_x_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,2])
ldos_x_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,2])
ldos_x = ldos_x_rad+ldos_x_guide;
ldos_y_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,3])
ldos_y_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,3])
ldos_y = ldos_y_rad+ldos_y_guide;
ldos_z_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,4])
ldos_z_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,4])
ldos_z = ldos_z_rad+ldos_z_guide;
ldos_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,5])
ldos_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,5])
ldos_total = ldos_rad + ldos_guide;
# Print out the result.
display("\Delta k = $(ldosqpoints[2,1]-ldosqpoints[1,1]) k_0.")
@printf("LDOS_x_rad=%5f, LDOS_x_guide=%5f, LDOS_x=%5f;\n",ldos_x_rad,ldos_x_guide,ldos_x)
@printf("LDOS_y_rad=%5f, LDOS_y_guide=%5f, LDOS_y=%5f;\n",ldos_y_rad,ldos_y_guide,ldos_y)
@printf("LDOS_z_rad=%5f, LDOS_z_guide=%5f, LDOS_z=%5f;\n",ldos_z_rad,ldos_z_guide,ldos_z)
@printf("LDOS_rad = %5f, LDOS_guide = %5f, LDOS = %5f.",ldos_rad,ldos_guide,ldos_total)
#ldosqpoints[breakpoint,2]
```

    "Delta k = 0.01 k_0."

    LDOS_x_rad=0.623963, LDOS_x_guide=-0.004968, LDOS_x=0.618995;
    LDOS_y_rad=0.657902, LDOS_y_guide=-0.002447, LDOS_y=0.655455;
    LDOS_z_rad=0.525345, LDOS_z_guide=0.012674, LDOS_z=0.538019;
    LDOS_rad = 1.807210, LDOS_guide = 0.005258, LDOS = 1.812468.

We can see that there are considerable differences between the integrated LDOS values for the two cases. The main differences are from the guided mode contribution part to LDOS, and should be related to the negative sparks. Therefore, I suspect that I still need to find a way to rule out the spark errors in order to calculate the LDOS values accurate. The resolution of the later case seems fine for the radiation contribution part, at least.

I didn't plot the details of the LDOS components for this case, but there are also negative sparks in the guided mode regime. The sparks disappear in the case of $\Delta k=0.05k_0$, but the integrals may not be accurate enough.

Another set of data is taken at $r=330nm$ position with a slightly different setting--mainly the edges are having a larger radius. We can plot the result below.

```julia
ldos = readdlm("data/ldos_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_330_y_0_q_0_2_201.dat", header=false);
#ldos = readdlm("data/ldos_N_1_lam_894_eps_4_0.001_a_300_b_300_c_5_x_0_y_350_q_0.1_4_101.dat", header=false);
ldosqpoints = ldos[:,[2,5,6,7,8]]
using PyPlot
#println(convert(Int64,floor(lenz/2)))
q = ldosqpoints[:,1]
ldosx = ldosqpoints[:,2]
ldosy = ldosqpoints[:,3]
ldosz = ldosqpoints[:,4]
ldos_av = ldosqpoints[:,5]
lenx = length(q)

fig = figure("LDOS q plot",figsize=(10,10))
ax = fig[:add_subplot](1,2,1)
cp = ax[:plot](q, ldosx, "b-", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS_x")
axis("image")
xlim(-0.0,2)
#ylim(-300,300)
tight_layout()
gcf() # Needed for IJulia to plot inline
display(maximum(abs.(ldosx)))

ax = fig[:add_subplot](1,2,2)
cp = ax[:plot](q, ldos_av, "r-", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS")
axis("image")
xlim(-0.0,2)
#ylim(-300,300)
tight_layout()
gcf() # Needed for IJulia to plot inline
display(maximum(abs.(ldosx)))
```

## LDOS components as a function of dipole position

We can also plot out the LDOS's when the dipole is placed at different locations along the x-axis. Here we are using some rough parameters just for demonstration purpose.

As plotted below, the dipole is changing position from $225$nm to $420$nm from the origin (center of the waveguide) along the x-axis.

```julia
# Load the simulated data.
ldos = readdlm("data/ldos_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x0_225_x1_420_nx_14_y_0_q_0_2_201.dat", header=false)
ldosqpoints = ldos[:,[2,3,5,6,7,8]]
lenr = 14; rstart=225; rend=420;
lenq = 201;
breakpoint = Int(floor((lenq-1)/2)) # The breaking point of index is chosen under the fact that the radiation contribution part takes a half space for $n_1=2$.
rprime = linspace(rstart,rend,lenr);
ldos_rscan = zeros(lenq,5,lenr);
ldos_int = zeros(lenr,12);
ii=1;
using NumericalIntegration
for ri in rprime
    ind=find(ldosqpoints[:,2].==ri);
    ldos_rscan[:,:,ii]=ldosqpoints[ind,[1,3,4,5,6]];

    ldos_x_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,2,ii])
    ldos_x_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,2,ii])
    ldos_x = ldos_x_rad+ldos_x_guide;
    ldos_y_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,3,ii])
    ldos_y_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,3,ii])
    ldos_y = ldos_y_rad+ldos_y_guide;
    ldos_z_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,4,ii])
    ldos_z_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,4,ii])
    ldos_z = ldos_z_rad+ldos_z_guide;
    ldos_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,5,ii])
    ldos_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,5,ii])
    ldos_total = ldos_rad + ldos_guide;

    ldos_int[ii,1]=ldos_x_rad;
    ldos_int[ii,2]=ldos_x_guide;
    ldos_int[ii,3]=ldos_x;
    ldos_int[ii,4]=ldos_y_rad;
    ldos_int[ii,5]=ldos_y_guide;
    ldos_int[ii,6]=ldos_y;
    ldos_int[ii,7]=ldos_z_rad;
    ldos_int[ii,8]=ldos_z_guide;
    ldos_int[ii,9]=ldos_z;
    ldos_int[ii,10]=ldos_rad;
    ldos_int[ii,11]=ldos_guide;
    ldos_int[ii,12]=ldos_total;
    ii+=1;
end
```

```julia
# Plot the LDOS's.
using PyPlot
fig = figure("LDOS(r') plot",figsize=(10,5))
ax = fig[:add_subplot](1,2,1)
cp = ax[:plot](rprime, ldos_int[:,3], "b:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,6], "r:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,9], "m:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,12], "k-", linewidth=2.0)

xlabel(L"r\prime(nm)")
ylabel(L"LDOS")
ylim(-0.0,2.2)
xlim(205,420)
legend(["LDOS_x","LDOS_y","LDOS_z","LDOS_total"],loc="right")

ax = fig[:add_subplot](1,2,2)
cp = ax[:plot](rprime, ldos_int[:,10], "b:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,11], "r:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,12], "k-", linewidth=2.0)

xlabel(L"r\prime(nm)")
ylabel(L"LDOS")
ylim(-0.0,2.2)
xlim(205,420)
legend(["LDOS_rad","LDOS_guide","LDOS_total"],loc="right")
```

The guide mode contribution seems very small compared to the nanofiber case using the BEM approach. As discussed in other tests, the BEM calculation on the guided mode contribution part is not accurate due to the fact that the dielectric function has a zero imaginary part or loss in the frequency domain which will cause an infinite narrow peak of the LDOS function and leads to unphysical solution.
To avoid this difficulty, we will use other method to calculate the guided mode contribution part to the Green's function tensor.

# Calculation of Green's function tensor using BEM

BEM can output full local field components so that computing the radiative mode contribution to the full Green's function tensor is possible. With the Green's function tensor, one can then calculate the modified decay rates with dipoles orientated along arbitrary directions--including the dipole transitions corresponding to $\sigma_\pm$ and $\pi$ transitions.

```julia
# Load data for different dipole positions.
rp_BEM=160:10:600;#[170,190,210,230,250,270,290,310,330,350,370,390,410,430,450,470];
lenrp=length(rp_BEM);
lendr=201;
E_dipolex = readdlm("data/dipolex_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_330_y_0_q_0_2_201.dat", header=false);
k_vec=E_dipolex[:,2];
Ex_dx=zeros(Complex{Float64},lendr,lenrp); Ey_dx=zeros(Complex{Float64},lendr,lenrp); Ez_dx=zeros(Complex{Float64},lendr,lenrp);
Ex_dy=zeros(Complex{Float64},lendr,lenrp); Ey_dy=zeros(Complex{Float64},lendr,lenrp); Ez_dy=zeros(Complex{Float64},lendr,lenrp);
Ex_dz=zeros(Complex{Float64},lendr,lenrp); Ey_dz=zeros(Complex{Float64},lendr,lenrp); Ez_dz=zeros(Complex{Float64},lendr,lenrp);
for ii=2:2:(lenrp-1)
    E_dipolex = readdlm("data/dipolex_E_N_1_lam_894_eps_4_0.01_a_300_b_300_c_6_x_$(rp_BEM[ii])_y_0_q_0_2_201.dat", header=false); #",rp_list[ii],"
    Ex_dx[:,ii]=E_dipolex[:,5]+im*E_dipolex[:,6];
    Ey_dx[:,ii]=E_dipolex[:,7]+im*E_dipolex[:,8];
    Ez_dx[:,ii]=E_dipolex[:,9]+im*E_dipolex[:,10];

    E_dipoley = readdlm("data/dipoley_E_N_1_lam_894_eps_4_0.01_a_300_b_300_c_6_x_$(rp_BEM[ii])_y_0_q_0_2_201.dat", header=false); #",rp_list[ii],"
    Ex_dy[:,ii]=E_dipoley[:,5]+im*E_dipoley[:,6];
    Ey_dy[:,ii]=E_dipoley[:,7]+im*E_dipoley[:,8];
    Ez_dy[:,ii]=E_dipoley[:,9]+im*E_dipoley[:,10];

    E_dipolez = readdlm("data/dipolez_E_N_1_lam_894_eps_4_0.01_a_300_b_300_c_6_x_$(rp_BEM[ii])_y_0_q_0_2_201.dat", header=false); #",rp_list[ii],"
    Ex_dz[:,ii]=E_dipolez[:,5]+im*E_dipolez[:,6];
    Ey_dz[:,ii]=E_dipolez[:,7]+im*E_dipolez[:,8];
    Ez_dz[:,ii]=E_dipolez[:,9]+im*E_dipolez[:,10];
end
for ii=1:2:lenrp
    E_dipolex = readdlm("data/dipolex_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_$(rp_BEM[ii])_y_0_q_0_2_201.dat", header=false); #",rp_list[ii],"
    Ex_dx[:,ii]=E_dipolex[:,5]+im*E_dipolex[:,6];
    Ey_dx[:,ii]=E_dipolex[:,7]+im*E_dipolex[:,8];
    Ez_dx[:,ii]=E_dipolex[:,9]+im*E_dipolex[:,10];

    E_dipoley = readdlm("data/dipoley_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_$(rp_BEM[ii])_y_0_q_0_2_201.dat", header=false); #",rp_list[ii],"
    Ex_dy[:,ii]=E_dipoley[:,5]+im*E_dipoley[:,6];
    Ey_dy[:,ii]=E_dipoley[:,7]+im*E_dipoley[:,8];
    Ez_dy[:,ii]=E_dipoley[:,9]+im*E_dipoley[:,10];

    E_dipolez = readdlm("data/dipolez_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_$(rp_BEM[ii])_y_0_q_0_2_201.dat", header=false); #",rp_list[ii],"
    Ex_dz[:,ii]=E_dipolez[:,5]+im*E_dipolez[:,6];
    Ey_dz[:,ii]=E_dipolez[:,7]+im*E_dipolez[:,8];
    Ez_dz[:,ii]=E_dipolez[:,9]+im*E_dipolez[:,10];
end
# Calculate the diagonal elements of the Green's function tensor from the radiation mode contribution.
c=2.99792458e8;
au=1.72e7; # This is the atomic unit in the CGS-units: $q/a_0^2$ statvolts/cm. In SI units, it becomes $e/(4π\varepsilon_0a_0^2)$ = 5.2e11 V/m.
lambda0=0.895e-6;
ω=2.*pi*c/lambda0;
GFT_rad_ind=zeros(Complex{Float64},3,3,lenrp);
Gxx_rad_ind=zeros(Complex{Float64},lenrp);
Gxy_rad_ind=zeros(Complex{Float64},lenrp);
Gxz_rad_ind=zeros(Complex{Float64},lenrp);
Gyx_rad_ind=zeros(Complex{Float64},lenrp);
Gyy_rad_ind=zeros(Complex{Float64},lenrp);
Gyz_rad_ind=zeros(Complex{Float64},lenrp);
Gzx_rad_ind=zeros(Complex{Float64},lenrp);
Gzy_rad_ind=zeros(Complex{Float64},lenrp);
Gzz_rad_ind=zeros(Complex{Float64},lenrp);
G0=Inf + 2.0/3.*(ω/c)^3*im;
gamma_rad_BEM_rp_average=zeros(lenrp);
gamma_rad_BEM_rp_sigmap=zeros(lenrp);
gamma_rad_BEM_rp_sigmam=zeros(lenrp);
gamma_rad_BEM_rp_pi=zeros(lenrp);
# Define the unitary dipole orientation vector.
e_dipole_sigmap=[-1.;-1.0*im;0]/sqrt(2);
e_dipole_sigmam=[1.; -1.0*im;0]/sqrt(2);
e_dipole_pi=[0.; 0.; 1.];
using NumericalIntegration
for ii=1:lenrp
    Gxx_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ex_dx[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;#G0+
    Gyy_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ey_dy[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;#G0+
    Gzz_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ez_dz[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;#G0+
    Gyx_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ey_dx[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gzx_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ez_dx[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gxy_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ex_dy[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gzy_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ez_dy[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gxz_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ex_dz[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gyz_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ey_dz[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    GFT_rad_ind[:,:,ii]=[Gxx_rad_ind[ii] Gxy_rad_ind[ii] Gxz_rad_ind[ii];
        Gyx_rad_ind[ii] Gyy_rad_ind[ii] Gzy_rad_ind[ii];
        Gzx_rad_ind[ii] Gzy_rad_ind[ii] Gzz_rad_ind[ii]];
end

# Calculate the relative averaged decay rate at the given dipole located at x=405nm and y=0.
gamma0=imag(G0);
for ii =1:lenrp
    gamma_rad_BEM_rp_average[ii]=1+trace(imag(GFT_rad_ind[:,:,ii]))/gamma0/3.;
    gamma_rad_BEM_rp_sigmap[ii]=1+real((e_dipole_sigmap'*imag(GFT_rad_ind[:,:,ii])*e_dipole_sigmap)/gamma0)[1];
    gamma_rad_BEM_rp_sigmam[ii]=1+real((e_dipole_sigmam'*imag(GFT_rad_ind[:,:,ii])*e_dipole_sigmam)/gamma0)[1];
    gamma_rad_BEM_rp_pi[ii]=1+real((e_dipole_pi'*imag(GFT_rad_ind[:,:,ii])*e_dipole_pi)/gamma0)[1];
end

# Recalculate the diagonal GFT elements with a dipole placed at r'=405nm from the fiber axis with lower imaginary part of epsilon.
E_dipolex = readdlm(join(["data/dipolex_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_","330","_y_0_q_0_2_201.dat"]), header=false)
Ex_dx_r0=E_dipolex[:,5]+im*E_dipolex[:,6];
Ey_dx_r0=E_dipolex[:,7]+im*E_dipolex[:,8];
Ez_dx_r0=E_dipolex[:,9]+im*E_dipolex[:,10];
E_dipoley = readdlm(join(["data/dipoley_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_","330","_y_0_q_0_2_201.dat"]), header=false)
Ex_dy_r0=E_dipoley[:,5]+im*E_dipoley[:,6];
Ey_dy_r0=E_dipoley[:,7]+im*E_dipoley[:,8];
Ez_dy_r0=E_dipoley[:,9]+im*E_dipoley[:,10];
E_dipolez = readdlm(join(["data/dipolez_E_N_1_lam_894_eps_4_0_a_300_b_300_c_6_x_","330","_y_0_q_0_2_201.dat"]), header=false)
Ex_dz_r0=E_dipolez[:,5]+im*E_dipolez[:,6];
Ey_dz_r0=E_dipolez[:,7]+im*E_dipolez[:,8];
Ez_dz_r0=E_dipolez[:,9]+im*E_dipolez[:,10];
Gxx_rad_r0=integrate(k_vec[1:breakpoint],Ex_dx_r0[1:breakpoint],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
Gyy_rad_r0=integrate(k_vec[1:breakpoint],Ey_dy_r0[1:breakpoint],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
Gzz_rad_r0=integrate(k_vec[1:breakpoint],Ez_dz_r0[1:breakpoint],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
gamma_rad_BEM_r0=imag(Gxx_rad_r0+Gyy_rad_r0+Gzz_rad_r0)/3/gamma0;

# Plot out gamma_rad as a function of dipole position.
figure(figsize=(16,3));
subplot(1,2,1)
plot((1:breakpoint)/100.,real(Ex_dx[1:breakpoint,4]),"r-")
plot((1:breakpoint)/100.,imag(Ex_dx[1:breakpoint,4]),"b-")
ylabel(L"E(\beta)")
xlabel(L"\beta/k_0")
legend(["Real","Imag"],loc="upper left")

subplot(1,2,2)
a=300.;
#plot(rp0_test[1,:]/1.e-9, 1+sum(gamma_rad,2), "r-", linewidth=2.0)
plot(rp_BEM-a/2,real(gamma_rad_BEM_rp_average),"b-");
plot(330-a/2,real(gamma_rad_BEM_r0)+1.,"mo");
plot(rp_BEM-a/2,gamma_rad_BEM_rp_sigmap,"r--");
plot(rp_BEM-a/2,gamma_rad_BEM_rp_sigmam,"m.-");
plot(rp_BEM-a/2,gamma_rad_BEM_rp_pi,"k.");
xlabel("(r'-a/2)/nm");
ylabel(L"\Gamma_{rad}^{BEM}/\Gamma_0");
#ylim([0,1.2])
legend(["average",L"average_{no\,\,loss}",L"\sigma_+",L"\sigma_-",L"\pi"],loc="upper right",fontsize=12);
#gamma_rad_BEM_rp_sigmap
```

The first figure (on the left) shows the real and imaginary parts of the $E_x$ component changes as a function of $\beta=k_z$ in the radiative mode regime.

The second figure (on the right) shows the decay rates caused by non-guided modes and decomposed into the $\sigma_\pm$ and $\pi$ transitions and their average.
In calculating the averaged non-guided mode induced decay rates, we define
$$\begin{align}
\frac{\Gamma_{rad}^{ave}}{\Gamma_0} &= 1+ \frac{\sum_{i=x,y,z}\mathrm{Im}\left[\mathbf{e}_i^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_i\right]}{\sum_{i=x,y,z} \mathrm{Im}\left[\mathbf{e}_i^*\cdot \mathbf{G}_0(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_i\right]}\\
&=1+ \frac{\sum_{i=x,y,z}\mathrm{Im}\left[\mathbf{e}_i^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_i\right]}{3\mathrm{Im}\left[G_0(\mathbf{r}',\mathbf{r}')\right]}\\
&= 1+ \frac{\mathrm{Tr}\left\{\mathrm{Im}\left[ \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\right]\right\}}{3\mathrm{Im}\left[G_0(\mathbf{r}',\mathbf{r}')\right]}
\end{align}$$
with the free-space Green's function scaled by $G_0(\mathbf{r}',\mathbf{r}';\omega)=\frac{2}{3}k_0^3$ in the CGS units as $\mathbf{G}_0=G_0\mathbb{1}$ and the numerical result of the waveguide-induced Green's function tensor elements
$$ G_{ind,rad}^{ij}(\mathbf{r}',\mathbf{r}')=\frac{2k_0^2*a.u.}{3\pi^2}\int_{-k_0}^{k_0} d\beta E_j^i(\mathbf{r}')=\frac{4k_0^2*a.u.}{3\pi^2}\int_{0}^{k_0} d\beta E_j^i(\mathbf{r}')$$
calculated via BEM by putting a unit dipole in atomic units orientated along $j$ direction while the $i$-th electric field component is measured at the dipole position.
We have used $\varepsilon=4+0.01i$ (with a small loss) and $\varepsilon=4$ (without loss) to calculate the Green's function tensors as well as the averaged corresponding decay rate contributions.
These two cases doesn't show a noticeable difference with one sampling data point as shown in the purple dot points.

In calculating the corresponding contributions from the $\sigma_\pm$ and $\pi$ transitions of the atoms, we have defined
$$\begin{align}
\frac{\Gamma_{rad}^{\mathbf{e}_q}}{\Gamma_0} &= 1+ \frac{\mathrm{Im}\left[\mathbf{e}_q^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_q\right]}{ \mathrm{Im}\left[\mathbf{e}_q^*\cdot \mathbf{G}_0(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_q\right]}
=1+ \frac{\mathrm{Im}\left[\mathbf{e}_q^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_q\right]}{\mathrm{Im}\left[G_0(\mathbf{r}',\mathbf{r}')\right]},
\end{align}$$
where the three orthogonal dipole transition bases
$$\begin{align}
\mathbf{e}_\pm &=\mp \frac{\mathbf{e}_{\tilde{x}}\pm i\mathbf{e}_{\tilde{y}}}{\sqrt{2}}\\
\mathbf{e}_0 &=\mathbf{e}_{\tilde{z}}
\end{align}$$
correspond to the $\sigma_\pm$ and $\pi$ transitions of the atoms.
These basis vectors are quantization-axis dependent, but in our calculation above, we assume the $z$-axis or the waveguide axis is the quantization axis, and hence $\mathbf{e}_{\tilde{x}}=\mathbf{e}_x$, $\mathbf{e}_{\tilde{y}}=\mathbf{e}_y$ and $\mathbf{e}_{\tilde{z}}=\mathbf{e}_z$.

As you can see, when the atoms are placed around $200$nm from the nanofiber surface, different dipole transitions make a noticeable difference.

To wrap up, the total decay rates
$$\begin{align}
\Gamma \propto \mathbf{e}_d^*\cdot \mathrm{Im}\left[\mathbf{G}(\mathbf{r}',\mathbf{r}')\right]\cdot \mathbf{e}_d,
\end{align}$$
where $$\mathbf{G}=\mathbf{G}_{hom}+\mathbf{G}_{inhom}=\mathbf{G}_{rad,free-space}+\mathbf{G}_{rad,ind}+\mathbf{G}_{gyd}$$
with $\mathbf{G}_{rad,free-space}=\mathbf{G}_0$ and $\mathrm{Im}\left[ \mathbf{G}_0(\mathbf{r}',\mathbf{r}')\right]=\frac{2}{3}\mathbb{1}$.
In the end, the total decay rate
$$\begin{align}\Gamma = \Gamma_{rad}+\Gamma_{gyd}=\Gamma_{rad,free-space}+\Gamma_{rad,ind}+\Gamma_{gyd}\end{align}$$
with $\Gamma_{rad,free-space}=\Gamma_0$.

```julia
# Export data to a MAT file.
using MAT
matopen("data/Julia_swg_GFT_decayrates_rad_D1.mat", "w") do matfile
    write(matfile,"omega0",ω)
    write(matfile,"rp_BEM",collect(rp_BEM))
    write(matfile,"gamma0",gamma0)
    write(matfile,"e_dipole_sigmap",e_dipole_sigmap)
    write(matfile,"e_dipole_sigmam",e_dipole_sigmam)
    write(matfile,"e_dipole_pi",e_dipole_pi)
    write(matfile,"gamma_rad_BEM_rp_average",gamma_rad_BEM_rp_average)
    write(matfile,"gamma_rad_BEM_rp_sigmap",gamma_rad_BEM_rp_sigmap)
    write(matfile,"gamma_rad_BEM_rp_sigmam",gamma_rad_BEM_rp_sigmam)
    write(matfile,"gamma_rad_BEM_rp_pi",gamma_rad_BEM_rp_pi)
    write(matfile,"a",a)
    write(matfile,"GFT_rad_rp",GFT_rad_ind)
    write(matfile,"Gxx_rad_rp",Gxx_rad_ind)
    write(matfile,"Gxy_rad_rp",Gxy_rad_ind)
    write(matfile,"Gxz_rad_rp",Gxz_rad_ind)
    write(matfile,"Gyx_rad_rp",Gyx_rad_ind)
    write(matfile,"Gyy_rad_rp",Gyy_rad_ind)
    write(matfile,"Gyz_rad_rp",Gyz_rad_ind)
    write(matfile,"Gzx_rad_rp",Gzx_rad_ind)
    write(matfile,"Gzy_rad_rp",Gzy_rad_ind)
    write(matfile,"Gzz_rad_rp",Gzz_rad_ind)
end
```

```julia

```


# Tutorial 2: Differential Equations
**Week 0, Day 4: Calculus**

**By Neuromatch Academy**

__Content creators:__ John S Butler, Arvind Kumar with help from Rebecca Brady

__Content reviewers:__  Swapnil Kumar, Sirisha Sripada, Matthew McCann, Tessy Tom

__Production editors:__ Matthew McCann, Ella Batty

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives
*Estimated timing of tutorial: 45 minutes*

A great deal of neuroscience can be modelled using differential equations, from gating channels to single neurons to a network of neurons to blood flow, to behaviour. A simple way to think about differential equations is they are equations that describe how something changes.

The most famous of these in neuroscience is the Nobel Prize winning Hodgkin Huxley equation, which describes a neuron by modelling the gating of each axon. But we will not start there; we will start a few steps back.

Differential Equations are mathematical equations that describe how something like population or a neuron changes over time. The reason why differential equations are so useful is they can generalise a process such that one equation can be used to describe many different outcomes.
The general form of a first order differential equation is:

\begin{align*}
\frac{d}{dt}y(t)&=f(t,y(t))\\
\end{align*}

which can be read as "the change in a process $y$ over time $t$ is a function $f$ of time $t$ and itself $y$". This might initially seem like a paradox as you are using a process $y$ you want to know about to describe itself, a bit like the MC Escher drawing of two hands painting [each other](https://en.wikipedia.org/wiki/Drawing_Hands). But that is the beauty of mathematics - this can be solved some of time, and when it cannot be solved exactly we can use numerical methods to estimate the answer (as we will see in the next tutorial).

In this tutorial, we will see how __differential equations are motivated by observations of physical responses.__ We will break down the population differential equation, then the integrate and fire model, which leads nicely into raster plots and frequency-current curves to rate models.

**Steps:**
- Get an intuitive understanding of a linear population differential equation (humans, not neurons)
- Visualize the relationship between the change in population and the population
- Breakdown the Leaky Integrate and Fire (LIF) differential equation
- Code the exact solution of an LIF for a constant input
- Visualize and listen to the response of the LIF for different inputs

```python
# @title Video 1: Why do we care about differential equations?
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1v64y197bW", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="LhX-mUd8lPo", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

---
# Setup

```python
# Imports

import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import IPython.display as ipd
from matplotlib import gridspec

import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'

# use NMA plot style
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
my_layout = widgets.Layout()
```

```python
# @title Plotting Functions

def plot_dPdt(alpha=.3):
  """ Plots change in population over time
    Args:
      alpha: Birth Rate
    Returns:
      A figure two panel figure
      left panel: change in population as a function of population
      right panel: membrane potential as a function of time
  """

  with plt.xkcd():
    time=np.arange(0, 10 ,0.01)
    fig = plt.figure(figsize=(12,4))
    gs = gridspec.GridSpec(1, 2)

    ## dpdt as a fucntion of p
    plt.subplot(gs[0])
    plt.plot(np.exp(alpha*time), alpha*np.exp(alpha*time))
    plt.xlabel(r'Population $p(t)$ (millions)')
    plt.ylabel(r'$\frac{d}{dt}p(t)=\alpha p(t)$')

    ## p exact solution
    plt.subplot(gs[1])
    plt.plot(time, np.exp(alpha*time))
    plt.ylabel(r'Population $p(t)$ (millions)')
    plt.xlabel('time (years)')
    plt.show()

def plot_V_no_input(V_reset=-75):
  """
    Args:
      V_reset: Reset Potential
    Returns:
      A figure two panel figure
      left panel: change in membrane potential as a function of membrane potential
      right panel: membrane potential as a function of time
  """
  E_L=-75
  tau_m=10
  t=np.arange(0,100,0.01)
  V= E_L+(V_reset-E_L)*np.exp(-(t)/tau_m)
  V_range=np.arange(-90,0,1)
  dVdt=-(V_range-E_L)/tau_m

  with plt.xkcd():
    time=np.arange(0, 10, 0.01)
    fig = plt.figure(figsize=(12, 4))
    gs = gridspec.GridSpec(1, 2)

    plt.subplot(gs[0])
    plt.plot(V_range,dVdt)
    plt.hlines(0,min(V_range),max(V_range), colors='black', linestyles='dashed')
    plt.vlines(-75, min(dVdt), max(dVdt), colors='black', linestyles='dashed')
    plt.plot(V_reset,-(V_reset - E_L)/tau_m, 'o', label=r'$V_{reset}$')
    plt.text(-50, 1, 'Positive')
    plt.text(-50, -2, 'Negative')
    plt.text(E_L - 1, max(dVdt), r'$E_L$')
    plt.legend()
    plt.xlabel('Membrane Potential V (mV)')
    plt.ylabel(r'$\frac{dV}{dt}=\frac{-(V(t)-E_L)}{\tau_m}$')

    plt.subplot(gs[1])
    plt.plot(t,V)
    plt.plot(t[0],V_reset,'o')
    plt.ylabel(r'Membrane Potential $V(t)$ (mV)')
    plt.xlabel('time (ms)')
    plt.ylim([-95, -60])

    plt.show()

## LIF PLOT
def plot_IF(t, V,I,Spike_time):
  """
    Args:
      t  : time
      V  : membrane Voltage
      I  : Input
      Spike_time : Spike_times
    Returns:
      figure with three panels
      top panel: Input as a function of time
      middle panel: membrane potential as a function of time
      bottom panel: Raster plot
  """

  with plt.xkcd():
    fig = plt.figure(figsize=(12, 4))
    gs = gridspec.GridSpec(3, 1,  height_ratios=[1, 4, 1])

    # PLOT OF INPUT
    plt.subplot(gs[0])
    plt.ylabel(r'$I_e(nA)$')
    plt.yticks(rotation=45)
    plt.hlines(I,min(t),max(t),'g')
    plt.ylim((2, 4))
    plt.xlim((-50, 1000))

    # PLOT OF ACTIVITY
    plt.subplot(gs[1])
    plt.plot(t,V)
    plt.xlim((-50, 1000))
    plt.ylabel(r'$V(t)$(mV)')

    # PLOT OF SPIKES
    plt.subplot(gs[2])
    plt.ylabel(r'Spike')
    plt.yticks([])
    plt.scatter(Spike_time, 1 * np.ones(len(Spike_time)), color="grey", marker=".")
    plt.xlim((-50, 1000))
    plt.xlabel('time(ms)')
    plt.show()

##    Plotting the differential Equation
def plot_dVdt(I=0):
  """
    Args:
      I  : Input Current
    Returns:
      figure of change in membrane potential as a function of membrane potential
  """

  with plt.xkcd():
    E_L = -75
    tau_m = 10
    V = np.arange(-85, 0, 1)
    g_L = 10.
    fig = plt.figure(figsize=(6, 4))

    plt.plot(V,(-(V-E_L) + I*10) / tau_m)
    plt.hlines(0, min(V), max(V), colors='black', linestyles='dashed')
    plt.xlabel('V (mV)')
    plt.ylabel(r'$\frac{dV}{dt}$')
    plt.show()
```

```python
# @title Helper Functions

## EXACT SOLUTION OF LIF
def Exact_Integrate_and_Fire(I,t):
  """
    Args:
      I  : Input Current
      t : time
    Returns:
      Spike : Spike Count
      Spike_time : Spike time
      V_exact : Exact membrane potential
  """

  Spike = 0
  tau_m = 10
  R = 10
  t_isi = 0
  V_reset = E_L = -75
  V_exact = V_reset * np.ones(len(t))
  V_th = -50
  Spike_time = []

  for i in range(0, len(t)):

    V_exact[i] = E_L + R*I + (V_reset - E_L - R*I) * np.exp(-(t[i]-t_isi)/tau_m)

    # Threshold Reset
    if V_exact[i] > V_th:
        V_exact[i-1] = 0
        V_exact[i] = V_reset
        t_isi = t[i]
        Spike = Spike+1
        Spike_time = np.append(Spike_time, t[i])

  return Spike, Spike_time, V_exact
```

---
# Section 1: Population differential equation

```python
# @title Video 2: Population differential equation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1pg41137CU", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="czgGyoUsRoQ", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers our first example of a differential equation: a differential equation which models the change in population.

 Click here for text recap of video

To get an intuitive feel of a differential equations, we will start with a population differential equation, which models the change in population [1], that is human population not neurons, we will get to neurons later. Mathematically it is written like:
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= \alpha p(t),\\
\end{align*}

where $p(t)$ is the population of the world and $\alpha$ is a parameter representing birth rate.

Another way of thinking about the models is that the equation
\begin{align*}
\\
\frac{d}{dt}\,p(t) &= \alpha p(t),\\
\text{can be written as:}\\
\text{"Change in Population"} &= \text{ "Birth rate times Current population."}
\end{align*}

The equation is saying something reasonable maybe not the perfect model but a good start.

### Think! 1.1: Interpretating the behavior of a linear population equation
Using the plot below of change of population $\frac{d}{dt} p(t) $ as a function of population $p(t)$ with birth-rate $\alpha=0.3$, discuss the following questions:
1. Why is the population differential equation known as a linear differential equation?

2. How does population size affect the rate of change of the population?

```python
# @markdown Execute the code to plot the rate of change of population as a function of population
p = np.arange(0, 100, 0.1)

with plt.xkcd():

  dpdt = 0.3*p
  fig = plt.figure(figsize=(6, 4))
  plt.plot(p, dpdt)
  plt.xlabel(r'Population $p(t)$ (millions)')
  plt.ylabel(r'$\frac{d}{dt}p(t)=\alpha p(t)$')
  plt.show()
```

```python
# to_remove explanation

"""
  1. The plot of $\frac{dp}{dt}$ is a line, which is why the differential
  equation is known as a linear differential equation.

  2. As the population increases, the change of population increases. A
  population of 20 has a change of 6 while a population of 100 has a change of
  30. This makes sense - the larger the population the larger the change.
"""
```

## Section 1.1: Exact solution of the population equation

### Section 1.1.1: Initial condition
The linear population differential equation is known as an initial value differential equation because we need an initial population value to solve it. Here we will set our initial population at time 0 to 1:

\begin{align*}
&p(0)=1.\\
\end{align*}

Different initial conditions will lead to different answers, but they will not change the differential equation. This is one of the strengths of a differential equation.

###  Section 1.1.2: Exact Solution
To calculate the exact solution of a differential equation, we must integrate both sides. Instead of numerical integration (as you delved into in the last tutorial), we will first try to solve the differential equations using analytical integration. As with derivatives, we can find analytical integrals of simple equations by consulting [a list](https://en.wikipedia.org/wiki/Lists_of_integrals). We can then get integrals for more complex equations using some mathematical tricks - the harder the equation the more obscure the trick.

The linear population equation
\begin{align*}
\frac{d}{dt}\,p(t) &= \alpha p(t),\\\\
p(0)=P_0,\\
\end{align*}
has the exact solution:
\begin{align*}
p(t)&=P_0e^{\alpha t}.\\
\end{align*}

The exact solution written in words is:

\begin{align*}
\text{"Population"}&=\text{"grows/declines exponentially as a function of time and birth rate"}.\\
\end{align*}

Most differential equations do not have a known exact solution, so in the next tutorial on numerical methods we will show how the solution can be estimated.

A small aside: a good deal of progress in mathematics was due to mathematicians writing taunting letters to each other saying they had a trick that could solve something better than everyone else. So do not worry too much about the tricks.

####  Example Exact Solution of the Population Equation
Let's consider the population differential equation with a birth rate $\alpha=0.3$:

\begin{align*}
\frac{d}{dt}\,p(t) = 0.3 p(t),\\
\text{with the initial condition}\\
p(0)=1.\\
\end{align*}

It has an exact solution
\begin{align*}
\\
p(t)=e^{0.3 t}.
\end{align*}

```python
# @markdown Execute code to plot the exact solution
t = np.arange(0, 10, 0.1) # Time from 0 to 10 years in 0.1 steps

with plt.xkcd():

  p = np.exp(0.3 * t)

  fig = plt.figure(figsize=(6, 4))
  plt.plot(t, p)
  plt.ylabel('Population (millions)')
  plt.xlabel('time (years)')
  plt.show()
```

## Section 1.2: Parameters of the differential equation

*Estimated timing to here from start of tutorial: 12 min*

One of the goals when designing a differential equation is to make it generalisable. Which means that the differential equation will give reasonable solutions for different countries with different birth rates $\alpha$.

###  Interactive Demo 1.2:  Interactive Parameter Change
Play with the widget to see the relationship between $\alpha$ and the population differential equation as a function of population (left-hand side), and the population solution as a function of time (right-hand side). Pay close attention to the transition point from positive to negative.

How do changing parameters of the population equation affect the outcome?

1. What happens when $\alpha < 0$?
2. What happens when $\alpha > 0$?
3. What happens when $\alpha = 0$?

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
    alpha=widgets.FloatSlider(.3, min=-1., max=1., step=.1, layout=my_layout)
)
def Pop_widget(alpha):
  plot_dPdt(alpha=alpha)
  plt.show()
```

```python
# to_remove explanation

"""
  1. Negative values of alpha result in an exponential decrease to 0 a stable solution.
  2. Positive Values of alpha in an exponential increases to infinity.
  3. Alpha equal to 0 is a unique point known as an equilibrium point when the
  dp/dt=0 and there is no change in population. This is known as a stable point.
"""
```

The population differential equation is an over-simplification and has some very obvious limitations:
1. Population growth is not exponential as there are limited number of resources so the population will level out at some point.
2. It does not include any external factors on the populations like weather, predators and preys.

These kind of limitations can be addressed by extending the model.

While it might not seem that the population equation has direct relevance to neuroscience, a similar equation is used to describe the accumulation of evidence for decision making. This is known as the Drift Diffusion Model and you will see in more detail in the Linear System day in Neuromatch (W2D2).

Another differential equation that is similar to the population equation is the Leaky Integrate and Fire model which you may have seen in the python pre-course materials on W0D1 and W0D2. It will turn up later in Neuromatch as well. Below we will delve in the motivation of the differential equation.

---
# Section 2: The leaky integrate and fire model

```python
# @title Video 3: The leaky integrate and fire model
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1rb4y1C79n", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="ZfWO6MLCa1s", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video covers the Leaky Integrate and Fire model (a linear differential equation which describes the membrane potential of a single neuron).

 Click here for text recap of full LIF equation from video

The Leaky Integrate and Fire Model is a linear differential equation that describes the membrane potential ($V$) of a single neuron which was proposed by Louis Édouard Lapicque in 1907 [2].

The subthreshold membrane potential dynamics of a LIF neuron is described by
\begin{align}
\tau_m\frac{dV}{dt} = -(V-E_L) + R_mI\,
\end{align}

where $\tau_m$ is the time constant, $V$ is the membrane potential,  $E_L$ is the resting potential, $R_m$ is membrane resistance, and $I$ is the external input current.

In the next few sections, we will break down the full LIF equation and then build it back up to get an intuitive feel of the different facets of the differential equation.

## Section 2.1: LIF without input

*Estimated timing to here from start of tutorial: 18 min*

As seen in the video, we will first model an LIF neuron without input, which results in the equation:

\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}.\\
\end{align}

where $\tau_m$ is the time constant, $V$ is the membrane potential, and $E_L$ is the resting potential.

 Click here for further details (from video)

Removing the input gives the equation

\begin{align}
\tau_m\frac{dV}{dt} &= -V+E_L,\\
\end{align}

which can be written in words as:

\begin{align}
\begin{matrix}\text{"Time constant multiplied by the} \\ \text{change in membrane potential"}\end{matrix}&=\begin{matrix}\text{"Minus Current} \\ \text{membrane potential"} \end{matrix}+
\begin{matrix}\text{"resting potential"}\end{matrix}.\\
\end{align}

The equation can be re-arranged to look even more like the population equation:

\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}.\\
\end{align}

### Think! 2.1: Effect on membrane potential $V$ on the LIF model

The plot the below shows the change in membrane potential $\frac{dV}{dt}$ as a function of membrane potential $V$ with the parameters set as:
* `E_L = -75`
* `V_reset = -50`
* `tau_m = 10.`

1. What is the effect on $\frac{dV}{dt}$ when $V>-75$ mV?
2. What is the effect on $\frac{dV}{dt}$ when $V<-75$ mV
3. What is the effect on $\frac{dV}{dt}$ when $V=-75$ mV?

```python
# @markdown Make sure you execute this cell to plot the relationship between dV/dt and V
# Parameter definition
E_L = -75
tau_m = 10

# Range of Values of V
V = np.arange(-90, 0, 1)
dV = -(V - E_L) / tau_m

with plt.xkcd():

  fig = plt.figure(figsize=(6, 4))
  plt.plot(V, dV)
  plt.hlines(0, min(V), max(V), colors='black', linestyles='dashed')
  plt.vlines(-75, min(dV), max(dV), colors='black', linestyles='dashed')

  plt.text(-50, 1, 'Positive')
  plt.text(-50, -2, 'Negative')
  plt.text(E_L, max(dV) + 1, r'$E_L$')
  plt.xlabel(r'$V(t)$ (mV)')
  plt.ylabel(r'$\frac{dV}{dt}=\frac{-(V-E_L)}{\tau_m}$')
  plt.ylim(-8, 2)
  plt.show()
```

```python
# to_remove explanation

"""
  1. For $V>-75$ mV, the derivative is negative.
  2. For $V<-75$ mV, the derivative is positive.
  3. For $V=-75$ mV, the derivative is equal to $0$ is and a stable point when nothing changes.
"""
```

### Section 2.1.1: Exact Solution of the LIF model without input
The LIF model has the exact solution:
\begin{align*}
V(t)=&\ E_L+(V_{reset}-E_L)e^{\frac{-t}{\tau_m}}\\
\end{align*}

where $\tau_m$ is the time constant, $V$ is the membrane potential,  $E_L$ is the resting potential, and $V_{reset}$ is the initial membrane potential.

 Click here for further details (from video)

Similar to the population equation, we need an initial membrane potential at time $0$ to solve the LIF model.

With this equation
\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}\,\\
V(0)&=V_{reset},
\end{align}
where is $V_{reset}$ is called the reset potential.

The LIF model has the exact solution:
\begin{align*}
V(t)=&\ E_L+(V_{reset}-E_L)e^{\frac{-t}{\tau_m}}\\
\text{ which can be written as: }\\
\begin{matrix}\text{"Current membrane} \\ \text{potential}"\end{matrix}=&\text{"Resting potential"}+\begin{matrix}\text{"Reset potential minus resting potential} \\ \text{times exponential with rate one over time constant."}\end{matrix}\\
\end{align*}

#### Interactive Demo 2.1.1: Initial Condition $V_{reset}$
This exercise is to get an intuitive feel of how the different initial conditions $V_{reset}$ impacts the differential equation of the LIF and the exact solution for the equation:

\begin{align}
\frac{dV}{dt} &= \frac{-(V-E_L)}{\tau_m}\,\\
\end{align}
with the parameters set as:
* `E_L = -75,`
* `tau_m = 10.`

The panel on the left-hand side plots the change in membrane potential $\frac{dV}{dt}$ as a function of membrane potential $V$ and right-hand side panel plots the exact solution $V$ as a function of time $t,$ the green dot in both panels is the reset potential $V_{reset}$.

Pay close attention to when $V_{reset}=E_L=-75$mV.

1. How does the solution look with initial values of $V_{reset} < -75$?
2. How does the solution look with initial values of $V_{reset} > -75$?
3. How does the solution look with initial values of $V_{reset} = -75$?

```python
#@markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
  V_reset=widgets.FloatSlider(-77., min=-91., max=-61., step=2,
                              layout=my_layout)
)

def V_reset_widget(V_reset):
  plot_V_no_input(V_reset)
```

```python
# to_remove explanation

"""
  1. Initial Values of $V_{reset} < -75$ result in the solution increasing to
  -75mV because $\frac{dV}{dt} > 0$.
  2. Initial Values of $V_{reset} > -75$ result in the solution decreasing to
  -75mV because $\frac{dV}{dt}  < 0$.
  3. Initial Values of $V_{reset} = -75$ result in a constant $V = -75$ mV
  because $\frac{dV}{dt}  = 0$ (Stable point).
"""
```

## Section 2.2: LIF with input
*Estimated timing to here from start of tutorial: 24 min*

We will re-introduce the input $I$ and membrane resistance $R_m$ giving the original equation:

\begin{align}
\tau_m\frac{dV}{dt} = -(V-E_L) + \color{blue}{R_mI}\,
\end{align}

The input can be other neurons or sensory information.

### Interactive Demo 2.2: The Impact of Input
The interactive plot below manipulates $I$ in the differential equation.

- With increasing input, how does the $\frac{dV}{dt}$ change? How would this impact the solution?

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
  I=widgets.FloatSlider(3., min=0., max=20., step=2,
                                layout=my_layout)
)

def Pop_widget(I):
  plot_dVdt(I=I)
  plt.show()
```

```python
# to_remove explanation

"""
dV/dt becomes bigger and less of it is below 0. This means the solution will increase well beyond what is bioligically plausible
"""
```

### Section 2.2.1: LIF exact solution

The LIF with a constant input has a known exact solution:
\begin{align*}
V(t)=&\ E_L+R_mI+(V_{reset}-E_L-R_mI)e^{\frac{-t}{\tau_m}}\\
\text{which is written as:}\\
\begin{matrix}\text{"Current membrane} \\ \text{potential"}\end{matrix}=&\text{"Resting potential"}+\begin{matrix}\text{"Reset potential minus resting potential} \\ \text{times exponential with rate one over time constant." }\end{matrix}\\
\end{align*}

The plot below shows the exact solution of the membrane potential with the parameters set as:
* `V_reset = -75,`
* `E_L = -75,`
* `tau_m = 10,`
* `R_m = 10,`
* `I = 10.`

Ask yourself, does the result make biological sense? If not, what would you change? We'll delve into this in the next section

```python
# @markdown Make sure you execute this cell to see the exact solution
dt = 0.5
t_rest = 0

t = np.arange(0, 1000, dt)

tau_m = 10
R_m = 10
V_reset = E_L = -75

I = 10

V = E_L + R_m*I + (V_reset - E_L - R_m*I) * np.exp(-(t)/tau_m)

with plt.xkcd():

  fig = plt.figure(figsize=(6, 4))
  plt.plot(t,V)
  plt.ylabel('V (mV)')
  plt.xlabel('time (ms)')
  plt.show()
```

## Section 2.3: Maths is one thing, but neuroscience matters

*Estimated timing to here from start of tutorial: 30 min*

```python
# @title Video 4: Adding firing to the LIF
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1gX4y1P7pZ", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="rLQk-vXRaX0", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

This video first recaps the introduction of input to the leaky integrate and fire model and then delves into how we add spiking behavior (or firing) to the model.

 Click here for text recap of video

While the mathematics of the exact solution is exact, it is not biologically valid as a neuron spikes and definitely does not plateau at a very positive value.

To model the firing of a spike, we must have a threshold voltage $V_{th}$ such that if the voltage $V(t)$ goes above it, the neuron spikes
$$V(t)>V_{th}.$$
We must record the time of spike $t_{isi}$  and count the number of spikes
$$t_{isi}=t, $$
$$𝑆𝑝𝑖𝑘𝑒=𝑆𝑝𝑖𝑘𝑒+1.$$
Then reset the membrane voltage $V(t)$
$$V(t_{isi} )=V_{Reset}.$$

To take into account the spike the exact solution becomes:
\begin{align*}
V(t)=&\ E_L+R_mI+(V_{reset}-E_L-R_mI)e^{\frac{-(t-t_{isi})}{\tau_m}},&\qquad V(t)<V_{th} \\
V(t)=&V_{reset},&\qquad V(t)>V_{th}\\
Spike=&Spike+1,&\\
t_{isi}=&t,\\
\end{align*}
while this does make the neuron spike, it introduces a discontinuity which is not as elegant mathematically as it could be, but it gets results so that is good.
</detail>

### Interactive Demo 2.3.1: Input on spikes
This exercise show the relationship between firing rate and the Input for exact solution `V` of the LIF:
$$
V(t)=\ E_L+R_mI+(V_{reset}-E_L-R_mI)e^{\frac{-(t-t_{isi})}{\tau_m}},
$$
with the parameters set as:
* `V_reset = -75,`
* `E_L = -75,`
* `tau_m = 10,`
* `R_m = 10.`

Below is a figure with three panels;
* the top panel is the input, $I,$
* the middle panel is the membrane potential $V(t)$. To illustrate the spike, $V(t)$ is set to $0$ and then reset to $-75$ mV when there is a spike.
* the bottom panel is the raster plot with each dot indicating a spike.

First, as electrophysiologist normally listen to spikes when conducting experiments, listen to the music of the firing rate for a single value of $I$. (Note the audio doesn't work in some browsers so don't worry about it if you can't hear anything)

```python
# @markdown Make sure you execute this cell to be able to hear the neuron
I = 3
t = np.arange(0, 1000, dt)
Spike, Spike_time, V = Exact_Integrate_and_Fire(I, t)

plot_IF(t, V, I, Spike_time)
ipd.Audio(V, rate=len(V))
```

Manipulate the input into the LIF to see the impact of input on the firing pattern (rate).

*  What is the effect of $I$ on spiking?
* Is this biologically valid?

```python
# @markdown Make sure you execute this cell to enable the widget!
my_layout.width = '450px'
@widgets.interact(
  I=widgets.FloatSlider(3, min=2.0, max=4., step=.1,
                              layout=my_layout)
)

def Pop_widget(I):
  Spike, Spike_time, V = Exact_Integrate_and_Fire(I, t)
  plot_IF(t, V, I, Spike_time)
```

```python
# to_remove explanation

"""
  1. As $I$ increases, the number of spikes increases.
  2. No, as there is a limit to the number of spikes due to a refractory period, which is not accounted for in this model.
"""
```

## Section 2.4 Firing Rate as a function of Input

*Estimated timing to here from start of tutorial: 38 min*

The firing frequency of a neuron plotted as a function of current is called an input-output curve (F–I curve). It is also known as a transfer function, which you came across in the previous tutorial. This function is one of the starting points for the rate model, which extends from modelling single neurons to the firing rate of a collection of neurons.

By fitting this to a function, we can start to generalise the firing pattern of many neurons, which can be used to build rate models. This will be discussed later in Neuromatch.

```python
# @markdown *Execture this cell to visualize the FI curve*
I_range = np.arange(2.0, 4.0, 0.1)
Spike_rate = np.ones(len(I_range))

for i, I in enumerate(I_range):
  Spike_rate[i], _, _ = Exact_Integrate_and_Fire(I, t)

with plt.xkcd():
  fig = plt.figure(figsize=(6, 4))
  plt.plot(I_range,Spike_rate)
  plt.xlabel('Input Current (nA)')
  plt.ylabel('Spikes per Second (Hz)')
  plt.show()
```

The LIF model is a very nice differential equation to start with in computational neuroscience as it has been used as a building block for many papers that simulate neuronal response.

__Strengths  of LIF model:__
+ Has an exact solution;
+ Easy to interpret;
+ Great to build network of neurons.

__Weaknesses of the LIF model:__
- Spiking is a discontinuity;
- Abstraction from biology;
- Cannot generate different spiking patterns.

---
# Summary

*Estimated timing of tutorial: 45 min*

```python
# @title Video 5: Summary
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1jV411x7t9", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="VzwLAW5p4ao", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

In this tutorial, we have seen two differential equations, the population differential equations and the leaky integrate and fire model.

We learned about:
* The motivation for differential equations.
* An intuitive relationship between the solution and the form of the differential equation.
* How different parameters of the differential equation impact the solution.
* The strengths and limitations of the simple differential equations.

---
# Links to Neuromatch Days

Differential equations turn up in a number of different Neuromatch days:
* The LIF model is discussed in more details in Model Types (Week 1 Day 1) and Real Neurons (Week 2 Day 3).
* Drift Diffusion model which is a differential equation for decision making is discussed in Linear Systems (Week 2 Day 2).
* Systems of differential equations are discussed in Linear Systems (Week 2 Day 2) and Dynamic Networks (Week 2 Day 4).

---
# References
1. Lotka, A. L, (1920)   Analytical note on certain rhythmic  relations inorganic  systems.Proceedings of the National Academy of Sciences,6(7):410–415,1920.

2. Brunel N, van Rossum MC. Lapicque's 1907 paper: from frogs to integrate-and-fire. Biol Cybern. 2007 Dec;97(5-6):337-9. doi: 10.1007/s00422-007-0190-0. Epub 2007 Oct 30. PMID: 17968583.

# Bibliography
1. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series.
2. Strogatz, S. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering (studies in nonlinearity), Westview Press; 2 edition (29 July 2014)

## Supplemental Popular Reading List
1. Lindsay, G. (2021). Models of the Mind: How Physics, Engineering and Mathematics Have Shaped Our Understanding of the Brain. Bloomsbury Publishing.
2. Strogatz, S. (2004). Sync: The emerging science of spontaneous order. Penguin UK.

## Popular Podcast
1. Strogatz, S. (Host). (2020-), Joy of X  https://www.quantamagazine.org/tag/the-joy-of-x/ Quanta Magazine