# Weight Intilization
Weight and bias intialization is one of the important factor responsible for today's state of the art algorithm.

Weights intialization is done in random fashion but that randomness is to be tuned in various ways to get optimum result.

Pitfall: all zero initialization. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

Vanishing Gradient In deep nueral network if sigmoid neuron is used there will be a problem of vanishing gradient because the value of sigmoid is between 0 and 1 and derivative of sigmoid is $$sigmoid(x)*(1-sigmoid(x))$$ and  when gradient flows backward in the network it is multiplied by subsequent wights which decreases the gradient value so the gradient to the beginning layers is very close to zero that means there is no learning. Using ReLU solves this problem.

Exploding Gradient:- Vanishing gradient is not much of a problem but in the contrary Exploding gradient is something on which we can ponder. This problem occours when weight are intialized with huge number like 10 or 100 and input to the network is not normalized. In this scenario either of these three can happen.

    1. The model is unable to get traction on your training data (e.g. poor loss).
    2. The model is unstable, resulting in large changes in loss from update to update.
    3. The model loss goes to NaN during training.

To Solve this problem the solution given are:
 1. Redesign the network
In deep neural networks, exploding gradients may be addressed by redesigning the network to have fewer layers.
There may also be some benefit in using a smaller batch size while training the network.

2. Weight Regularizers
Another approach, if exploding gradients are still occurring, is to check the size of network weights and apply a penalty to the networks loss function for large weight values.

This is called weight regularization and often an L1 (absolute weights) or an L2 (squared weights) penalty can be used.

3. Gradient Clipping
Clip the gradient before updating the weights. Either the gradient can be clipped by using l2 norm or the easiest way to clip is by values. we define a threshold value with range `[-limit ,limit]` if the grad value exceeds the limit we replace it with the limit value

Small random numbers. Therefore, we still want the weights to be very close to zero, but as we have argued above, not identically zero. As a solution, it is common to initialize the weights of the neurons to small numbers and refer to doing so as symmetry breaking. The idea is that the neurons are all random and unique in the beginning, so they will compute distinct updates and integrate themselves as diverse parts of the full network. The implementation for one weight matrix might look like W = 0.01* np.random.randn(D,H), where randn samples from a zero mean, unit standard deviation gaussian. With this formulation, every neuron’s weight vector is initialized as a random vector sampled from a multi-dimensional gaussian, so the neurons point in random direction in the input space. It is also possible to use small numbers drawn from a uniform distribution, but this seems to have relatively little impact on the final performance in practice.

Warning: It’s not necessarily the case that smaller numbers will work strictly better. For example, a Neural Network layer that has very small weights will during backpropagation compute very small gradients on its data (since this gradient is proportional to the value of the weights). This could greatly diminish the “gradient signal” flowing backward through a network, and could become a concern for deep networks.

Calibrating the variances with 1/sqrt(n) Xavier/Glorot Initialization. One problem with the above suggestion is that the distribution of the outputs from a randomly initialized neuron has a variance that grows with the number of inputs. It turns out that we can normalize the variance of each neuron’s output to 1 by scaling its weight vector by the square root of its fan-in (i.e. its number of inputs). That is, the recommended heuristic is to initialize each neuron’s weight vector as: w = np.random.randn(n) / sqrt(n), where n is the number of its inputs. This ensures that all neurons in the network initially have approximately the same output distribution and empirically improves the rate of convergence.

The sketch of the derivation is as follows: Consider the inner product \(s = \sum_i^n w_i x_i\) between the weights \(w\) and input \(x\), which gives the raw activation of a neuron before the non-linearity. We can examine the variance of \(s\):

\begin{align}
\text{Var}(s) &= \text{Var}(\sum_i^n w_ix_i) \\\\
&= \sum_i^n \text{Var}(w_ix_i) \\\\
&= \sum_i^n [E(w_i)]^2\text{Var}(x_i) + E[(x_i)]^2\text{Var}(w_i) + \text{Var}(x_i)\text{Var}(w_i) \\\\
&= \sum_i^n \text{Var}(x_i)\text{Var}(w_i) \\\\
&= \left( n \text{Var}(w) \right) \text{Var}(x)
\end{align}

where in the first 2 steps we have used properties of variance. In third step we assumed zero mean inputs and weights, so \(E[x_i] = E[w_i] = 0\). Note that this is not generally the case: For example ReLU units will have a positive mean. In the last step we assumed that all \(w_i, x_i\) are identically distributed. From this derivation we can see that if we want \(s\) to have the same variance as all of its inputs \(x\), then during initialization we should make sure that the variance of every weight \(w\) is \(1/n\). And since \(\text{Var}(aX) = a^2\text{Var}(X)\) for a random variable \(X\) and a scalar \(a\), this implies that we should draw from unit gaussian and then scale it by \(a = \sqrt{1/(n)}\), to make its variance \(1/n\). This gives the initialization w = np.random.randn(n) / sqrt(nin+nout).

Sparse initialization. Another way to address the uncalibrated variances problem is to set all weight matrices to zero, but to break symmetry every neuron is randomly connected (with weights sampled from a small gaussian as above) to a fixed number of neurons below it. A typical number of neurons to connect to may be as small as 10.

Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.

In practice, the current recommendation is to use ReLU units and use the w = np.random.randn(n) * sqrt(2.0/n), as discussed in He et al..

Pytorch Supports various popular intialization type which can seen here --->> https://pytorch.org/docs/stable/nn.html#torch-nn-init

```python
from torch import nn

# a simple network
model=nn.Sequential(nn.Linear(2, 5),
                         nn.ReLU(),)
next(model.parameters())
```

    Parameter containing:
    tensor([[-0.4624,  0.1237],
            [ 0.3361,  0.3213],
            [ 0.0601, -0.5251],
            [ 0.0932, -0.6454],
            [-0.1530,  0.3163]], requires_grad=True)

```python

# initialization function, first checks the module type,
# then applies the desired changes to the weights
def init_normal(m):
    if type(m) == nn.Linear:
        nn.init.uniform_(m.weight)
    elif isinstance(m , nn.Conv2d):
        nn.init.kaiming_normal_(m.weight)

# use the modules apply function to recursively apply the initialization to every submodule.
model.apply(init_normal)
next(model.parameters())
```

    Parameter containing:
    tensor([[0.9828, 0.0986],
            [0.8059, 0.4303],
            [0.2424, 0.9537],
            [0.5189, 0.4018],
            [0.1175, 0.8017]], requires_grad=True)

# Additional Resources

https://pytorch.org/docs/stable/nn.html#torch-nn-init

http://cs231n.github.io/neural-networks-2/#init

```python

```


# ETS models

The ETS models are a family of time series models with an underlying state space model consisting of a level component, a trend component (T), a seasonal component (S), and an error term (E).

This notebook gives a very brief introduction to these models and shows how they can be used with statsmodels. For a more thorough treatment we refer to [1], chapter 8 (free online resource), on which the implementation in statsmodels and the examples used in this notebook are based.

[1] Hyndman, Rob J., and George Athanasopoulos. *Forecasting: principles and practice*, 3rd edition, OTexts, 2019. https://www.otexts.org/fpp3/7

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
from statsmodels.tsa.exponential_smoothing.ets import ETSModel
```

```python
plt.rcParams['figure.figsize'] = (12, 8)
```

## Simple exponential smoothing

The simplest of the ETS models is also known as *simple exponential smoothing*. In ETS terms, it corresponds to the (A, N, N) model, that is, a model with additive errors, no trend, and no seasonality. The state space formulation of Holt's method is:

\begin{align}
y_{t} &= y_{t-1} + e_t\\
l_{t} &= l_{t-1} + \alpha e_t\\
\end{align}

This state space formulation can be turned into a different formulation, a forecast and a smoothing equation (as can be done with all ETS models):

\begin{align}
\hat{y}_{t|t-1} &= l_{t-1}\\
l_{t} &= \alpha y_{t-1} + (1 - \alpha) l_{t-1}
\end{align}

Here, $\hat{y}_{t|t-1}$ is the forecast/expectation of $y_t$ given the information of the previous step. In the simple exponential smoothing model, the forecast corresponds to the previous level. The second equation (smoothing equation) calculates the next level as weighted average of the previous level and the previous observation.

```python
oildata = [
    111.0091, 130.8284, 141.2871, 154.2278,
    162.7409, 192.1665, 240.7997, 304.2174,
    384.0046, 429.6622, 359.3169, 437.2519,
    468.4008, 424.4353, 487.9794, 509.8284,
    506.3473, 340.1842, 240.2589, 219.0328,
    172.0747, 252.5901, 221.0711, 276.5188,
    271.1480, 342.6186, 428.3558, 442.3946,
    432.7851, 437.2497, 437.2092, 445.3641,
    453.1950, 454.4096, 422.3789, 456.0371,
    440.3866, 425.1944, 486.2052, 500.4291,
    521.2759, 508.9476, 488.8889, 509.8706,
    456.7229, 473.8166, 525.9509, 549.8338,
    542.3405
]
oil = pd.Series(oildata, index=pd.date_range('1965', '2013', freq='AS'))
oil.plot()
plt.ylabel("Annual oil production in Saudi Arabia (Mt)");
```

The plot above shows annual oil production in Saudi Arabia in million tonnes. The data are taken from the R package `fpp2` (companion package to prior version [1]).
Below you can see how to fit a simple exponential smoothing model using statsmodels's ETS implementation to this data. Additionally, the fit using `forecast` in R is shown as comparison.

```python
model = ETSModel(oil, error='add', trend='add', damped_trend=True)
fit = model.fit(maxiter=10000)
oil.plot(label='data')
fit.fittedvalues.plot(label='statsmodels fit')
plt.ylabel("Annual oil production in Saudi Arabia (Mt)");

# obtained from R
params_R = [0.99989969, 0.11888177503085334, 0.80000197, 36.46466837, 34.72584983]
yhat = model.smooth(params_R).fittedvalues
yhat.plot(label='R fit', linestyle='--')

plt.legend();
```

By default the initial states are considered to be fitting parameters and are estimated by maximizing log-likelihood. Additionally it is possible to only use a heuristic for the initial values. In this case this leads to better agreement with the R implementation.

```python
model_heuristic = ETSModel(oil, error='add', trend='add', damped_trend=True,
                          initialization_method='heuristic')
fit_heuristic = model_heuristic.fit()
oil.plot(label='data')
fit.fittedvalues.plot(label='estimated')
fit_heuristic.fittedvalues.plot(label='heuristic', linestyle='--')
plt.ylabel("Annual oil production in Saudi Arabia (Mt)");

# obtained from R
params = [0.99989969, 0.11888177503085334, 0.80000197, 36.46466837, 34.72584983]
yhat = model.smooth(params).fittedvalues
yhat.plot(label='with R params', linestyle=':')

plt.legend();
```

```python
fit.summary()
```

```python
fit_heuristic.summary()
```

## Holt-Winters' seasonal method

The exponential smoothing method can be modified to incorporate a trend and a seasonal component. In the additive Holt-Winters' method, the seasonal component is added to the rest. This model corresponds to the ETS(A, A, A) model, and has the following state space formulation:

\begin{align}
y_t &= l_{t-1} + b_{t-1} + s_{t-m} + e_t\\
l_{t} &= l_{t-1} + b_{t-1} + \alpha e_t\\
b_{t} &= b_{t-1} + \beta e_t\\
s_{t} &= s_{t-m} + \gamma e_t
\end{align}

```python
austourists_data = [
    30.05251300, 19.14849600, 25.31769200, 27.59143700,
    32.07645600, 23.48796100, 28.47594000, 35.12375300,
    36.83848500, 25.00701700, 30.72223000, 28.69375900,
    36.64098600, 23.82460900, 29.31168300, 31.77030900,
    35.17787700, 19.77524400, 29.60175000, 34.53884200,
    41.27359900, 26.65586200, 28.27985900, 35.19115300,
    42.20566386, 24.64917133, 32.66733514, 37.25735401,
    45.24246027, 29.35048127, 36.34420728, 41.78208136,
    49.27659843, 31.27540139, 37.85062549, 38.83704413,
    51.23690034, 31.83855162, 41.32342126, 42.79900337,
    55.70835836, 33.40714492, 42.31663797, 45.15712257,
    59.57607996, 34.83733016, 44.84168072, 46.97124960,
    60.01903094, 38.37117851, 46.97586413, 50.73379646,
    61.64687319, 39.29956937, 52.67120908, 54.33231689,
    66.83435838, 40.87118847, 51.82853579, 57.49190993,
    65.25146985, 43.06120822, 54.76075713, 59.83447494,
    73.25702747, 47.69662373, 61.09776802, 66.05576122,
]
index = pd.date_range("1999-03-01", "2015-12-01", freq="3MS")
austourists = pd.Series(austourists_data, index=index)
austourists.plot()
plt.ylabel('Australian Tourists');
```

```python
# fit in statsmodels
model = ETSModel(austourists, error="add", trend="add", seasonal="add",
                damped_trend=True, seasonal_periods=4)
fit = model.fit()

# fit with R params
params_R = [
    0.35445427, 0.03200749, 0.39993387, 0.97999997, 24.01278357,
    0.97770147, 1.76951063, -0.50735902, -6.61171798, 5.34956637
]
fit_R = model.smooth(params_R)

austourists.plot(label='data')
plt.ylabel('Australian Tourists')

fit.fittedvalues.plot(label='statsmodels fit')
fit_R.fittedvalues.plot(label='R fit', linestyle='--')
plt.legend();
```

```python
fit.summary()
```

```python
fit._rank
```


```python
%load_ext sympyprinting
%matplotlib inline
import matplotlib.pyplot as plt
import sympy
from IPython.display import display
sympy.init_printing(use_unicode=False, wrap_line=False, no_global=True)
import scipy.constants
import numpy as np
```

    /home/ash/anaconda2/envs/python3/lib/python3.5/site-packages/IPython/extensions/sympyprinting.py:31: UserWarning: The sympyprinting extension has moved to `sympy`, use `from sympy import init_printing; init_printing()`
      warnings.warn("The sympyprinting extension has moved to `sympy`, "

# Equation of motion - SDE to be solved

### $\ddot{q}(t) + \Gamma_0\dot{q}(t) + \Omega_0^2 q(t) -  \dfrac{1}{m} F(t) = 0 $

#### where q = x, y or z

Where $F(t) = \mathcal{F}_{fluct}(t) + F_{feedback}(t)$

Taken from page 46 of 'Dynamics of optically levitated nanoparticles in high vacuum' - Thesis by Jan Gieseler

Using $\mathcal{F}_{fluct}(t) = \sqrt{2m \Gamma_0 k_B T_0}\dfrac{dW(t)}{dt}$

and $F_{feedback}(t) = \Omega_0 \eta q^2 \dot{q}$

Taken from page 49 of 'Dynamics of optically levitated nanoparticles in high vacuum' - Thesis by Jan Gieseler

we get the following SDE:

$\dfrac{d^2q(t)}{dt^2} + (\Gamma_0 - \Omega_0 \eta q(t)^2)\dfrac{dq(t)}{dt} + \Omega_0^2 q(t) - \sqrt{\dfrac{2\Gamma_0 k_B T_0}{m}} \dfrac{dW(t)}{dt} = 0$

split into 2 first order ODE/SDE s

letting $v = \dfrac{dq}{dt}$

$\dfrac{dv(t)}{dt} + (\Gamma_0 - \Omega_0 \eta q(t)^2)v + \Omega_0^2 q(t) - \sqrt{\dfrac{2\Gamma_0 k_B T_0}{m}} \dfrac{dW(t)}{dt} = 0$

therefore

$\dfrac{dv(t)}{dt} = -(\Gamma_0 - \Omega_0 \eta q(t)^2)v - \Omega_0^2 q(t) + \sqrt{\dfrac{2\Gamma_0 k_B T_0}{m}} \dfrac{dW(t)}{dt} $

$v = \dfrac{dq}{dt}$ therefore $dq = v~dt$

\begin{align}
dq&=v\,dt\\
dv&=[-(\Gamma_0-\Omega_0 \eta q(t)^2)v(t) - \Omega_0^2 q(t)]\,dt + \sqrt{\frac{2\Gamma_0 k_B T_0}m}\,dW
\end{align}

### Apply Milstein Method to solve

Consider the autonomous Itō stochastic differential equation

${\mathrm  {d}}X_{t}=a(X_{t})\,{\mathrm  {d}}t+b(X_{t})\,{\mathrm  {d}}W_{t}$

Taking $X_t = q_t$ for the 1st equation above (i.e. $dq = v~dt$) we get:

$$ a(q_t) = v $$
$$ b(q_t) = 0 $$

Taking $X_t = v_t$ for the 2nd equation above (i.e. $dv = ...$) we get:

$$a(v_t) = -(\Gamma_0-\Omega_0\eta q(t)^2)v - \Omega_0^2 q(t)$$
$$b(v_t) = \sqrt{\dfrac{2\Gamma_0 k_B T_0}m}$$

${\displaystyle b'(v_{t})=0}$ therefore the diffusion term does not depend on ${\displaystyle v_{t}}$ , the Milstein's method in this case is therefore equivalent to the Euler–Maruyama method.

We then construct these functions in python:

```python
def a_q(t, v, q):
    return v

def a_v(t, v, q):
    return -(Gamma0 - Omega0*eta*q**2)*v - Omega0**2*q

def b_v(t, v, q):
    return np.sqrt(2*Gamma0*k_b*T_0/m)

```

Using values obtained from fitting to data from a real particle we set the following constant values describing the system. Cooling has been assumed to be off by setting $\eta = 0$.

```python
Gamma0 = 4000 # radians/second
Omega0 = 75e3*2*np.pi # radians/second
eta = 0.5e7
T_0 = 300 # K
k_b = scipy.constants.Boltzmann # J/K
m = 3.1e-19 # KG
```

partition the interval [0, T] into N equal subintervals of width $\Delta t>0$:
$ 0=\tau _{0}<\tau _{1}<\dots <\tau _{N}=T{\text{ with }}\tau _{n}:=n\Delta t{\text{ and }}\Delta t={\frac {T}{N}}$

```python
dt = 1e-10
tArray = np.arange(0, 100e-6, dt)
```

```python
print("{} Hz".format(1/dt))
```

    10000000000.0 Hz

set $Y_{0}=x_{0}$

```python
q0 = 0
v0 = 0
q = np.zeros_like(tArray)
v = np.zeros_like(tArray)
q[0] = q0
v[0] = v0
```

Generate independent and identically distributed normal random variables with expected value 0 and variance dt

```python
np.random.seed(88)

dwArray = np.random.normal(0, np.sqrt(dt), len(tArray)) # independent and identically distributed normal random variables with expected value 0 and variance dt
```

Apply Milstein's method (Euler Maruyama if $b'(Y_{n}) = 0$ as is the case here):

recursively define $Y_{n}$ for $ 1\leq n\leq N $ by

$ Y_{{n+1}}=Y_{n}+a(Y_{n})\Delta t+b(Y_{n})\Delta W_{n}+{\frac  {1}{2}}b(Y_{n})b'(Y_{n})\left((\Delta W_{n})^{2}-\Delta t\right)$

Perform this for the 2 first order differential equations:

```python
#%%timeit

for n, t in enumerate(tArray[:-1]):
    dw = dwArray[n]
    v[n+1] = v[n] + a_v(t, v[n], q[n])*dt + b_v(t, v[n], q[n])*dw + 0
    q[n+1] = q[n] + a_q(t, v[n], q[n])*dt + 0

```

We now have an array of positions, $v$, and velocities $p$ with time $t$.

```python
plt.plot(tArray*1e6, v)
plt.xlabel("t (us)")
plt.ylabel("v")
```

```python
plt.plot(tArray*1e6, q)
plt.xlabel("t (us)")
plt.ylabel("q")
```

Alternatively we can use a derivative-free version of Milsteins method as a two-stage kind-of Runge-Kutta method, documented in wikipedia (https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_method_%28SDE%29) or the original in arxiv.org https://arxiv.org/pdf/1210.0933.pdf.

```python
q0 = 0
v0 = 0
X = np.zeros([len(tArray), 2])
X[0, 0] = q0
X[0, 1] = v0
```

```python
def a(t, X):
    q, v = X
    return np.array([v, -(Gamma0 - Omega0*eta*q**2)*v - Omega0**2*q])

def b(t, X):
    q, v = X
    return np.array([0, np.sqrt(2*Gamma0*k_b*T_0/m)])
```

```python
%%timeit

S = np.array([-1,1])
for n, t in enumerate(tArray[:-1]):
    dw = dwArray[n]
    K1 = a(t, X[n])*dt + b(t, X[n])*(dw - S*np.sqrt(dt))
    Xh = X[n] + K1
    K2 = a(t, Xh)*dt + b(t, Xh)*(dw + S*np.sqrt(dt))
    X[n+1] = X[n] + 0.5 * (K1+K2)

```

    1 loop, best of 3: 2.33 s per loop

```python
q = X[:, 0]
v = X[:, 1]
```

```python
plt.plot(tArray*1e6, v)
plt.xlabel("t (us)")
plt.ylabel("v")
```

```python
plt.plot(tArray*1e6, q)
plt.xlabel("t (us)")
plt.ylabel("q")
```

The form of $F_{feedback}(t)$ is still questionable

On page 49 of 'Dynamics of optically levitated nanoparticles in high vacuum' - Thesis by Jan Gieseler he uses the form: $F_{feedback}(t) = \Omega_0 \eta q^2 \dot{q}$

On page 2 of 'Parametric feeedback cooling of levitated optomechancs in a parabolic mirror trap' Paper by Jamie and Muddassar they use the form: $F_{feedback}(t) = \dfrac{\Omega_0 \eta q^2 \dot{q}}{q_0^2}$ where $q_0$ is the amplitude of the motion: $q(t) = q_0(sin(\omega_0t)$

However it always shows up as a term $\delta \Gamma$ like so:

$\dfrac{d^2q(t)}{dt^2} + (\Gamma_0 + \delta \Gamma)\dfrac{dq(t)}{dt} + \Omega_0^2 q(t) - \sqrt{\dfrac{2\Gamma_0 k_B T_0}{m}} \dfrac{dW(t)}{dt} = 0$

By fitting to data we extract the following 3 parameters:

1) $A = \gamma^2 \dfrac{k_B T_0}{\pi m}\Gamma_0 $

Where:

- $\gamma$ is the conversion factor between Volts and nanometres. This parameterises the amount of light/ number of photons collected from the nanoparticle. With unchanged allignment and the same particle this should remain constant with changes in pressure.
- $m$ is the mass of the particle, a constant
- $T_0$ is the temperature of the environment
- $\Gamma_0$ the damping due to the environment only

2) $\Omega_0$ - the natural frequency at this trapping power

3) $\Gamma$ - the total damping on the system including environment and feedback etc...

By taking a reference save with no cooling we have $\Gamma = \Gamma_0$ and therefore we can extract $A' = \gamma^2 \dfrac{k_B T_0}{\pi m}$. Since $A'$ should be constant with pressure we can therefore extract $\Gamma_0$ at any pressure (if we have a reference save and therefore a value of $A'$) and therefore can extract $\delta \Gamma$, the damping due to cooling, we can then plug this into our SDE instead in order to include cooling in the SDE model.

For any dataset at any pressure we can do:

$\Gamma_0 = \dfrac{A}{A'}$

And then $\delta \Gamma = \Gamma - \Gamma_0$

Using this form and the same derivation as above we arrive at the following form of the 2 1st order differential equations:

\begin{align}
dq&=v\,dt\\
dv&=[-(\Gamma_0 + \delta \Gamma)v(t) - \Omega_0^2 v(t)]\,dt + \sqrt{\frac{2\Gamma_0 k_B T_0}m}\,dW
\end{align}

```python
def a_q(t, v, q):
    return v

def a_v(t, v, q):
    return -(Gamma0 + deltaGamma)*v - Omega0**2*q

def b_v(t, v, q):
    return np.sqrt(2*Gamma0*k_b*T_0/m)

```

values below are taken from a ~1e-2 mbar cooled save

```python
Gamma0 = 15 # radians/second
deltaGamma = 2200
Omega0 = 75e3*2*np.pi # radians/second
eta = 0.5e7
T_0 = 300 # K
k_b = scipy.constants.Boltzmann # J/K
m = 3.1e-19 # KG
```

```python
dt = 1e-10
tArray = np.arange(0, 100e-6, dt)
```

```python
q0 = 0
v0 = 0
q = np.zeros_like(tArray)
v = np.zeros_like(tArray)
q[0] = q0
v[0] = v0
```

```python
np.random.seed(88)

dwArray = np.random.normal(0, np.sqrt(dt), len(tArray)) # independent and identically distributed normal random variables with expected value 0 and variance dt
```

```python
for n, t in enumerate(tArray[:-1]):
    dw = dwArray[n]
    v[n+1] = v[n] + a_v(t, v[n], q[n])*dt + b_v(t, v[n], q[n])*dw + 0
    q[n+1] = q[n] + a_q(t, v[n], q[n])*dt + 0

```

```python
plt.plot(tArray*1e6, v)
plt.xlabel("t (us)")
plt.ylabel("v")
```

```python
plt.plot(tArray*1e6, q)
plt.xlabel("t (us)")
plt.ylabel("q")
```

```python

```

```python

```


# 14 Partial Differential Equations — 2

## Solving Laplace's or Poisson's equation

Still solving **Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

General solution by iteration:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

```python
def Laplace_Jacobi_slow(Phi):
    # Don't use, very slow AND inefficient
    Phi_new = Phi.copy()
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                  + Phi[xi, yj+1] + Phi[xi, yj-1])
    Phi[:, :] = Phi_new
    return Phi
```

Fast implementation using numpy array operations (vectorized, run at C speed, not Python speed) (see the [board notes (PDF)](14_PDEs-2-LectureNotes.pdf)) for an explanation for how to set up the numpy array operations that give you a 100-fold speed up over `Laplace_Jacobi_slow()`):

```python
def Laplace_Jacobi(Phi):
    """One update in the Jacobi algorithm"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2])
    return Phi
```

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

```python
def Laplace_Gauss_Seidel(Phi):
    """One update in the Gauss-Seidel algorithm"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
    return Phi
```

#### Fast Gauss-Seidel-like

Divide the lattice into a checkerboard of black and white cells. Update the odd cells first (like Jacobi) but then use the odd cells to update the even ones (Gauss-Seidel-like; see the [board notes (PDF)](14_PDEs-2-LectureNotes.pdf)). This leads to faster convergence *and* can be done with numpy array operations.

```python
def Laplace_Gauss_Seidel_odd_even(Phi):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2] + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2])
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2] + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2])

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2] + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2])
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2] + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2])
    return Phi
```

### *Converged* solution of the wire-in-a-box problem

Solve the box-wire problem and **make sure that the solution is converged to  `tol = 1e-3`.**

Check convergence with the [_Frobenius Norm_](http://mathworld.wolfram.com/FrobeniusNorm.html):

$$
||\mathsf{\Phi}|| := \sqrt{\sum_{i,j}|\Phi_{ij}|^2}
$$

which is implemented as [numpy.linalg.norm()](https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html) when the argument is a matrix.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Interactive plotting (with ipympl):

```python
# for interactive work
%matplotlib widget
```

Only execute the next line if you *don't* want interactive plotting (e.g., when exporting to LaTeX/PDF or html):

```python
%matplotlib inline
```

#### Convience plotting functions.
If you provide a filename then output is only written to a file and figures are close to conserve memory. This allows you to plot files in loops and later assemble them into movies using other programs such as ffmpeg, ImageMagick, mencoder, QuickTime 7, ...

```python
def plot_contour(Phi, filename=None, zlabel=r"potential $\Phi$ (V)",
                 cmap=plt.cm.coolwarm):
    """Plot Phi as a contour plot.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    cmap : colormap
          pick one from matplotlib.cm
    """
    fig = plt.figure(figsize=(5,4))
    ax = fig.add_subplot(111)

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]
    cset = ax.contourf(X, Y, Z, 20, cmap=cmap)
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_aspect(1)

    cb = fig.colorbar(cset, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax

def plot_surf(Phi, filename=None, offset=-20, zlabel=r'potential $\Phi$ (V)',
             elevation=40, azimuth=-65, cmap=plt.cm.coolwarm):
    """Plot Phi as a 3D plot with contour plot underneath.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    cmap : colormap
          pick one from matplotlib.cm
    """

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
    surf = ax.plot_surface(X, Y, Z, cmap=cmap, alpha=0.6)
    cset = ax.contourf(X, Y, Z, 20, zdir='z', offset=offset+Z.min(), cmap=cmap)

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Z.min(), Z.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

#### Wire problem

Use the fastest Poisson solver that we have at the moment: `Laplace_Gauss_Seidel_odd_even()`

(In order to get a cool progress bar, install tqdm with `conda install tqdm`.)

```python
from tqdm import tqdm
```

```python
Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in tqdm(range(Max_iter)):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

     25%|██████████████                                          | 7557/30000 [00:00<00:02, 10629.19it/s]

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

Plot the result and visualy compare to what we had before:

```python
plot_contour(Phi);
```

```python
plot_surf(Phi);
```

## Successive Over Relaxation (SOR)
Accelerate convergence with the scheme

\begin{align}
r_{i, j} &= \Phi_{i,j}^\text{new} - \Phi_{i, j}^\text{old}\\
\Phi_{i,j}^\text{new} &= \Phi_{i,j}^\text{old} + \omega r_{i,j}
\end{align}

where the new solution is computed with the Gauss-Seidel scheme.

Values of $1 \leq \omega \leq 2$ may work well, $\omega > 2$ can lead to numerical instabilities. Experiment!

### Solve Wire-in-a-box with SOR

Run to convergence (tol = 1e-03). Record the number of steps required (and visually check the solution).
* Start with $\omega = 1$. What should you get?
* Change $\omega$ and try to get faster convergence? Who can get it to converge in the fewest number of steps?

```python
Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in tqdm(range(Max_iter)):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual  # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

     38%|█████████████████████▋                                   | 3796/10000 [00:00<00:00, 9308.86it/s]

    SOR converged in 3797 iterations to 0.0009987117944452034

```python
plot_surf(Phi);
```

## Simple example of Poisson's equation: Wire and charge
Add a positive charge and a negative charge in the box.

Now we need to solve **Poisson's equation**.

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

Finite difference solution:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

### Poisson Solver
Modify the Laplace solvers to now solve Poisson's equation:

```python
import numpy as np

def Poisson_Jacobi(Phi, rho, Delta=1.):
    """One update in the Jacobi algorithm for Poisson's equation"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2]) \
                      + np.pi * Delta**2 * rho[1:-1, 1:-1]
    return Phi

def Poisson_Gauss_Seidel(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm for Poisson's equation"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1]) \
                            + np.pi * Delta**2 * rho[xi, yj]
    return Phi

def Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    a = np.pi * Delta**2
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2]
                                + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2]) + a * rho[1:-2:2, 1:-2:2]
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2]
                                + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2]) + a * rho[2:-1:2, 2:-1:2]

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2]
                                + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2]) + a * rho[1:-2:2, 2:-1:2]
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2]
                                + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2]) + a * rho[2:-1:2, 1:-2:2]
    return Phi
```

### Solve wire with dipole

Solve the Poisson problem with SOR (as above), but now employing the Poisson solvers (we are using the fastest one that we have):

```python
Nmax = 100
Max_iter = 10000
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in tqdm(range(Max_iter)):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

     38%|█████████████████████▊                                   | 3822/10000 [00:00<00:00, 7068.80it/s]

    SOR converged in 3823 iterations to 0.0009991605868815044

```python
plot_surf(Phi, elevation=40, azimuth=20);
```

## Charge density
Given a potential $\Phi$, Poisson's equation gives the charge density:
$$
\rho(\mathbf{x}) = -\frac{1}{4\pi} \nabla^2\Phi(\mathbf{x})
$$

Discretized Laplacian
$$
\nabla^2 \Phi \approx \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) +\, \Phi(x,y-\Delta y) - 4\Phi(x, y)}{\Delta^2}
$$

### Laplacian in Python

```python
def laplacian2d(f, Delta=1):
    """Finite difference approximation of Del^2 f.

    Arguments
    ---------
    f : M x N matrix
    Delta : float

    Returns
    -------
    M x N matrix, boundaries set to 0
    """

    L = np.zeros_like(f, dtype=np.float64)
    L[1:-1, 1:-1] = f[2:, 1:-1] + f[:-2, 1:-1] + f[1:-1, 2:] + f[1:-1, :-2] - 4*f[1:-1, 1:-1]
    return L/Delta**2

def laplacian2dsimple(f, Delta=1):
    L = np.zeros_like(f, dtype=np.float64)
    for i in range(1, L.shape[0]-1):
        for j in range(1, L.shape[1]-1):
            L[i, j] = f[i+1, j] + f[i-1, j] + f[i, j+1] + f[i, j-1] - 4*f[i, j]
    return L/Delta**2
```

#### Testing the implementation of Laplacian

Making sure that our fancy numpy-version of the Laplacian is producing the same results as the naive implementation with Python loops:

```python
import numpy as np

def test_laplacian2d():
    ftest = np.random.random((200, 200))
    assert np.allclose(laplacian2d(ftest), laplacian2dsimple(ftest))

test_laplacian2d()
```

The `test_laplacian2d()` function does _not_ raise `AssertionError` so the test passed.

### Charge density from potential

Compute the charge density from the converged potential
$$
\rho(x, y) = -\frac{1}{4\pi} \nabla^2 \Phi(x, y)
$$

```python
rhox = - laplacian2d(Phi)/(4*np.pi)
```

Does `rhox` show the charges of $+5$ and $-5$ that we introduced with the charge density $\rho$?

```python
print(rhox.min())
print(rhox.max())
```

    -5.0000041867249845
    5.000000000000001

Indeed, the point charges are recovered from the potential.

Plot the charge density:

```python
plot_contour(rhox, zlabel=r"charge density $\rho$");
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. This is expected because the Laplacian is not defined on the boundary. If you wanted to see the charge on the wire you would need to include it inside the integration domain and also assign a thickness (e.g., one or two grid cells).

The 3D plot shows the "point" charges as spikes of the size of one grid cell:

```python
plot_surf(rhox, zlabel=r"charge density $\rho$", elevation=20, azimuth=20, offset=-3);
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. The problem is that the Laplacian is not defined on the boundary.

Also, if the charges are only defined on a single grid cell then the 3D visualization is sometimes not able to show it correctly.

```python

```


# Imports and Utilities

```python
import copy
import math
import numpy as np
import matplotlib.pyplot as plt
```

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import Image
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from termcolor import colored
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.animation as animation
from matplotlib import colors as mcolors

from IPython.display import HTML

```

```python
def plot_schedule(schedule_fn, iterations=1500):
    # Iteration count starting at 1
    iterations = [i+1 for i in range(iterations)]
    lrs = [schedule_fn(i) for i in iterations]
    plt.scatter(iterations, lrs)
    plt.xlabel("Iteration")
    plt.ylabel("Learning Rate")
    plt.show()
```

```python
class TriangularSchedule():
    def __init__(self, min_lr, max_lr, cycle_length, inc_fraction=0.5):
        """
        min_lr: lower bound for learning rate (float)
        max_lr: upper bound for learning rate (float)
        cycle_length: iterations between start and finish (int)
        inc_fraction: fraction of iterations spent in increasing stage (float)
        """
        self.min_lr = min_lr
        self.max_lr = max_lr
        self.cycle_length = cycle_length
        self.inc_fraction = inc_fraction

    def __call__(self, iteration):
        if iteration <= self.cycle_length*self.inc_fraction:
            unit_cycle = iteration * 1 / (self.cycle_length * self.inc_fraction)
        elif iteration <= self.cycle_length:
            unit_cycle = (self.cycle_length - iteration) * 1 / (self.cycle_length * (1 - self.inc_fraction))
        else:
            unit_cycle = 0
        adjusted_cycle = (unit_cycle * (self.max_lr - self.min_lr)) + self.min_lr
        return adjusted_cycle
```

```python
class CosineAnnealingSchedule():
    def __init__(self, min_lr, max_lr, cycle_length):
        """
        min_lr: lower bound for learning rate (float)
        max_lr: upper bound for learning rate (float)
        cycle_length: iterations between start and finish (int)
        """
        self.min_lr = min_lr
        self.max_lr = max_lr
        self.cycle_length = cycle_length

    def __call__(self, iteration):
        if iteration <= self.cycle_length:
            unit_cycle = (1 + math.cos(iteration * math.pi / self.cycle_length)) / 2
            adjusted_cycle = (unit_cycle * (self.max_lr - self.min_lr)) + self.min_lr
            return adjusted_cycle
        else:
            return self.min_lr
```

```python
class LinearWarmUp():
    def __init__(self, schedule, start_lr, length):
        """
        schedule: a pre-initialized schedule (e.g. TriangularSchedule(min_lr=0.5, max_lr=2, cycle_length=500))
        start_lr: learning rate used at start of the warm-up (float)
        length: number of iterations used for the warm-up (int)
        """
        self.schedule = schedule
        self.start_lr = start_lr
        # calling mx.lr_scheduler.LRScheduler effects state, so calling a copy
        self.finish_lr = copy.copy(schedule)(0)
        self.length = length

    def __call__(self, iteration):
        if iteration <= self.length:
            return iteration * (self.finish_lr - self.start_lr)/(self.length) + self.start_lr
        else:
            return self.schedule(iteration - self.length)
```

```python
class OneCycleSchedule():
    def __init__(self, start_lr, max_lr, cycle_length, cooldown_length=0, finish_lr=None):
        """
        start_lr: lower bound for learning rate in triangular cycle (float)
        max_lr: upper bound for learning rate in triangular cycle (float)
        cycle_length: iterations between start and finish of triangular cycle: 2x 'stepsize' (int)
        cooldown_length: number of iterations used for the cool-down (int)
        finish_lr: learning rate used at end of the cool-down (float)
        """
        if (cooldown_length > 0) and (finish_lr is None):
            raise ValueError("Must specify finish_lr when using cooldown_length > 0.")
        if (cooldown_length == 0) and (finish_lr is not None):
            raise ValueError("Must specify cooldown_length > 0 when using finish_lr.")

        finish_lr = finish_lr if (cooldown_length > 0) else start_lr
        schedule = TriangularSchedule(min_lr=start_lr, max_lr=max_lr, cycle_length=cycle_length)
        self.schedule = LinearCoolDown(schedule, finish_lr=finish_lr, start_idx=cycle_length, length=cooldown_length)

    def __call__(self, iteration):
        return self.schedule(iteration)
```

```python
class LinearCoolDown():
    def __init__(self, schedule, finish_lr, start_idx, length):
        """
        schedule: a pre-initialized schedule (e.g. TriangularSchedule(min_lr=0.5, max_lr=2, cycle_length=500))
        finish_lr: learning rate used at end of the cool-down (float)
        start_idx: iteration to start the cool-down (int)
        length: number of iterations used for the cool-down (int)
        """
        self.schedule = schedule
        # calling mx.lr_scheduler.LRScheduler effects state, so calling a copy
        self.start_lr = copy.copy(self.schedule)(start_idx)
        self.finish_lr = finish_lr
        self.start_idx = start_idx
        self.finish_idx = start_idx + length
        self.length = length

    def __call__(self, iteration):
        if iteration <= self.start_idx:
            return self.schedule(iteration)
        elif iteration <= self.finish_idx:
            return (iteration - self.start_idx) * (self.finish_lr - self.start_lr) / (self.length) + self.start_lr
        else:
            return self.finish_lr
```

```python
class CyclicalSchedule():
    def __init__(self, schedule_class, cycle_length, cycle_length_decay=1, cycle_magnitude_decay=1, **kwargs):
        """
        schedule_class: class of schedule, expected to take `cycle_length` argument
        cycle_length: iterations used for initial cycle (int)
        cycle_length_decay: factor multiplied to cycle_length each cycle (float)
        cycle_magnitude_decay: factor multiplied learning rate magnitudes each cycle (float)
        kwargs: passed to the schedule_class
        """
        self.schedule_class = schedule_class
        self.length = cycle_length
        self.length_decay = cycle_length_decay
        self.magnitude_decay = cycle_magnitude_decay
        self.kwargs = kwargs

    def __call__(self, iteration):
        cycle_idx = 0
        cycle_length = self.length
        idx = self.length
        while idx <= iteration:
            cycle_length = math.ceil(cycle_length * self.length_decay)
            cycle_idx += 1
            idx += cycle_length
        cycle_offset = iteration - idx + cycle_length

        schedule = self.schedule_class(cycle_length=cycle_length, **self.kwargs)
        return schedule(cycle_offset) * self.magnitude_decay**cycle_idx
```

# Neural Network Architecture - Basic

# Checklist

We know:
1. What a Neural Network Architecture is
2. When input is feeded into the network we get an output(score) from the output layer of the network
3. Loss functions, which tells us how bad our output is as compared to the actual.
4. Backpropogation - Way to **nudge** the weights of neurons by calculating the gradient of the loss function.
5. Optimizers - Tells how to change the parameters of our model, by how much, and when?

# Optimization vs Optimizers

In a nutshell, optimization is a minimization problem.But the question is what do we need to minimize in our context.

In a Learning problem we have a **Loss-Function** or a **Target function** which is a mathematical way of measuring how wrong our predictions are.During the training process, we tweak and change the parameters (weights) of our model to try and minimize that loss function, and make our predictions as correct as possible.So we need a way that tells us **How do we change the parameters of our model, by how much, and when?** and that way is by using **Optimizers**

**Optimizers**

In simpler terms, optimizers shape and mold your model into its most accurate possible form by futzing with the weights. The loss function is the guide to the terrain, telling the optimizer whether it’s moving in the right or wrong direction

# Batch Gradient Descent

Gradient descent update:
        \begin{align}
        w_{t+1}=w_t− \eta \bigtriangledown f(x),
    \end{align}

where $ \eta$ is the Learning Rate

**In simple english**
1. Calculate what a small change in each individual weight would do to the loss function (i.e. which direction we should move in)
2. Adjust each individual weight based on its gradient (i.e. take a small step(determined by learning rate) in the determined direction)
3. Keep doing $ Steps-1$ and $Step-2$ until the loss function gets as low as possible

# Learning Rate

Ensures that we change our weights at the right pace, not making any changes that are too big or too small.

# Loss Function Curve in 3 dimension

# Critical Points

When $f'(x) = 0$, the derivative provides no information about which direction to move. Points where $f'(x) = 0$ are known as **critical points** or **stationary points**

**Types of Critical Points:**
1. **Local Minimum**-Point where $f(x)$ is lower than at all neighboring points, so it is no longer possible to decrease $f(x)$ by making infinitesimal steps.

2. **Local Maximum** Point where $f(x)$ is higher than at all neighboring points,so it is not possible to increase $f(x)$ by making infinitesimal steps.
3. **Saddle Points**-Slopes (derivatives) in orthogonal directions are all zero,but these points are neither maxima nor minima.
4. **Global Minimum**-Point that obtains the absolute lowest value of $f(x)$

## Saddle Point in 3D

```python
fig = plt.figure(figsize=(8,8))
ax = fig.gca(projection='3d')
X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
Z = X**2 - Y**2

ax.text2D(0,0,'*',fontdict={'ha': 'center', 'va': 'center', 'family': 'sans-serif','fontweight': 'bold',"fontsize":20})
ax.text2D(0,0.01,'Saddle Point',fontdict={'family': 'sans-serif','fontweight': 'bold'})
ax.set_xlabel('$X$')
ax.set_ylabel('$Y$')
ax.set_zlabel(r'$Z$')
ax.plot_surface(X, Y, Z,color="red");
```

## Pathological Curvature 3D

## Pathological Curvature 2D

# Optimizers with Adaptive Learning Rates

Other than SGD there are many more different optimization alogrithms which provide an efficient and faster convergence method to handle the pathelogical curve problems by updating the weights and biases.Some of them are lsited below:

1. Nestrov Acclerated Gradient
2. Adagrad
3. Adadelta
4. RMSprop
5. Adam
6. AdaMax
7. Nadam
8. AMSGrad

# Problem at Hand

1. Need to experimentally find the best learning rate values and schedule for the global learning rates

2. Adaptive Leanring rates like AdaGrad,RMSProp,AdaDelta are not computationally simple.
3. Difficulty in minimizing the loss arising from saddle points rather than poor local minima as saddle points have small gradients that slows the learning process.
4. Convex Loss Function is a Myth...

Highly non-convex loss functions for vgg56

# Our Objective

# Advanced Learning Rate Schedulers

Changing learning rate over Schedules

1. Schedules define how the learning rate changes over time
2. Specified for each epoch or each iteration of training

They are different from adaptive methods because they:
1. Change the global learning rate for the optimizers,rather than parameter-wise learning rates.
2. Don't take feedback from the training process and are specified beforehand.

## Different shapes for Learning Rate Schedules

Approaches:
1. Step Change

2. Smooth linear decrease

3. Cyclical

[Cyclical Learning Rates for Training Neural Networks-2015-2016](https://arxiv.org/pdf/1506.01186.pdf)

4. Cosine Form

[SGDR: STOCHASTIC GRADIENT DESCENT WITH WARM RESTARTS-2017](https://arxiv.org/pdf/1608.03983.pdf)

# Cyclical Learning Rate

[Cyclical Learning Rates for Training Neural Networks-2015-2016](https://arxiv.org/pdf/1506.01186.pdf)

**What they are and what they are not**

1. They are a framework to vary learning rate during training and hence helps in letting the global learning rate vary cyclically within a band of values instead of setting it to a fixed value.
2. They are fundamentally different from adaptive learning rate and they can combine with adaptive learning rates.
2. Cyclical Learning Rate is similar to Stocastic Gradient Descent  with Restarts Methods.

**Advantages**
1. No need to tune the learning rate
2. Obtain near Optimal Accuracy for any task
3. Implementing cyclical learning rate make sure rapid traversal of saddle points plateaus due to increasing learning rates.

# Different Functional Form for varying learning Rates

1. Trinagular Window (linear)
2. Welch Window (Parabolic)
3. Hann Window (Sinusoidal)

## Hyperparameters in Cyclical Learning Rate

1. Minimum Learning Rate
2. Maximum Learning Rate
3. Step Size : 2 to 10 times the number of iterations in an epoch
4. Cycle Length - 2 times the step size

```python
schedule = CyclicalSchedule(TriangularSchedule, min_lr=0.5, max_lr=2, cycle_length=500)
plot_schedule(schedule)
```

# Learning Rate Range test (LR Finder)

Used to determine if super-convergence is possible for an architecture

How we do it..

1. Choose a minimum and maximum learning rate
2. Set step size = max_iterations(total epochs)

generall epochs ranges between 1 to 5
3. Plot accuracy vs learning rate(lr finder plot) and choose min lr when accuracy starts to increase and max lr when accuracy starts to decrease.

# LR Modified

## SGD with Warm up

[Accurate, Large Minibatch SGD:
Training ImageNet in 1 Hour - 2017](https://arxiv.org/pdf/1706.02677.pdf)

Mostly focussed on demonstrating feasibility of large scale training with distributed Stochastic Gradient Descent approach

**Salient Points**

1. Trained ImageNet Data on ResNet50 in 1 hour.
2. Trained on 256 GPU with batchsize of 8192 as compared to 8 GPU with batch size of 256 for over 29 hours
3. Provided the same level of accuracy as the 256 minibatch baseline.
4. Present a new warmup strategy, i.e., a strategy of using lower learning rates at the start of training, to overcome early optimization difficulties
5. Came up with Linear Scaling rule i.e When the minibatch size is multiplied by k, multiply the learning rate by k

However in the initial training when the network is changing rapidly, linear scaling does not hold which can be addressed using **gradual warmup phase**.

Also  minibatch size cannot be scaled indefinitely: while results are stable for a large range of sizes,beyond a certain point accuracy degrades rapidly.For **Imagenet it was around 8K images**

**Gradual Warmup**
The idea is to start from a learning rate of η and increment it by a constant amount at each iteration such that it reaches ηˆ = kη after 5 epoch

# 1 Cycle Policy

[Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://arxiv.org/pdf/1708.07120.pdf)

1. We progressively increase our learning rate from $\frac{lr\_max}{div\_factor}$ to $lr\_max$ and at the same time we progressively decrease our momentum from $mom\_max$ to $mom\_min$.
2. We do the exact opposite : We progressively decrease our learning rate from $lr\_max$ to $\frac{lr\_max}{div\_factor}$ and at the same time we progressively increase our momentum from $mom\_min$ to $mom\_max$.
3. We further decrease our learning rate from  $\frac{lr\_max}{div\_factor}$  to  $\frac{lr\_max}{div\_factor x 100}$ and we keep momentum steady at mom_max.

```python
schedule = OneCycleSchedule(start_lr=0.05, max_lr=0.5, cycle_length=500, cooldown_length=550, finish_lr=0.001)
plot_schedule(schedule)
```

## SGD with Warm Restarts

[SGDR: STOCHASTIC GRADIENT DESCENT WITH WARM RESTARTS](https://arxiv.org/pdf/1608.03983.pdf)

Salient Points

1.  SGD with warm restarts requires 2× to 4× fewer epochs than the currently-used learning rate schedule chemes to achieve comparable or even better results.
2. The restarts are not performed from scratch but emulated by decreasing the learning rate $η_t$ by some factor to the original starting learning rate.
3. As we decrease the learning rate by some factor during restarts, we can increase the cycle length to have a better convergence.

```python
schedule = CyclicalSchedule(CosineAnnealingSchedule, min_lr=0.01, max_lr=2,
                            cycle_length=250, cycle_length_decay=2, cycle_magnitude_decay=0.5)
plot_schedule(schedule)
```

# Our Results

## Computer Vision Task

We did Benchmarking in 2 computer vision task.

###  1. Transfer Learning using Resnet-50 model.

Here the normal training gives 86.34% in 10 epochs and Super Convergence gives 89.80% in 5 epochs.

### 2. Traning DenseNet-BC 121 Architecture for Cifar10.

Here the normal training gives 86% in 8 epochs and Super Convergence gives 90% in 9 epochs.

## NLP Task

We trained OpenAI GPT model for 100 epochs on wikitext and here is the result.

Train Loss: 4.299 Val. Loss: 4.461

# References

1. [Code](https://colab.research.google.com/drive/1Q72MaPlrAp1ZyRhkI3RHDjIDo0ZcGP0a)
2. [Optimizers Visualization](https://gitlab.com/entirety.ai/meetup-intuition-to-implementation/blob/master/Phase%20-%201/Session-1%20and%202/7_Optimizers.ipynb)
1. [Super Convergence paper](https://arxiv.org/abs/1708.07120)
2. [1 Cycle Policy](https://docs.fast.ai/callbacks.one_cycle.html)
3. [LR Scheduler Keras](https://www.pyimagesearch.com/2019/07/22/keras-learning-rate-schedules-and-decay/)
4. [Mxnet](https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/training/learning_rates/learning_rate_schedules_advanced.html)
5. [Optimizers](https://blog.paperspace.com/intro-to-optimization-momentum-rmsprop-adam/)


# Gradient Descent :
There are three variants of gradient descent, which differ in how much data we use to compute the gradient of the objective function. Depending on the amount of data, we make a trade-off between the accuracy of the parameter update and the time it takes to perform an update.
- Batch Gradient Descent
- Stochastic Gradient Descent
- Mini-batch Gradient Descent

## Batch gradient descent

Vanilla gradient descent, aka batch gradient descent, computes the gradient of the cost function w.r.t. to the parameters
θ for the entire training dataset:

$\theta = \theta - \eta \cdot \nabla_\theta J( \theta)$

As we need to calculate the gradients for the whole dataset to perform just one update, batch gradient descent can be very slow and is intractable for datasets that don't fit in memory. Batch gradient descent also doesn't allow us to update our model online, i.e. with new examples on-the-fly.

## Stochastic gradient descent

Stochastic gradient descent (SGD) in contrast performs a parameter update for each training example
$x^{(i)}$ and label $y^{(i)}$

$\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i)}; y^{(i)})$

Batch gradient descent performs redundant computations for large datasets, as it recomputes gradients for similar examples before each parameter update. SGD does away with this redundancy by performing one update at a time. It is therefore usually much faster and can also be used to learn online.

## Mini-batch gradient descent

Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of
n  training examples:

$\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i:i+n)}; y^{(i:i+n)})$

This way, it
- Reduces the variance of the parameter updates, which can lead to more stable convergence; and
- Can make use of highly optimized matrix optimizations common to state-of-the-art deep learning libraries that make computing the gradient w.r.t. a mini-batch very efficient.
- Common mini-batch sizes range between 50 and 256, but can vary for different applications.
- Mini-batch gradient descent is typically the algorithm of choice when training a neural network and the term SGD usually is employed also when mini-batches are used.

## Conclusion

We have a dataset of size N  .
1. In Batch GD we consider all the data point (all N data points)
2. In SGD we consider only 1 datapoint per epoch
3. In Mini-Batch` SGD we consider k datapoints per epoch ( k < N)

# Gradient descent optimization algorithms

1. Momentum
2. Nesterov accelerated gradient
3. Adagrad
4. Adadelta
5. RMSprop
6. Adam
7. Nadam
8. AMSGrad
9. Weigth Decay

##  SGD with Momentum : Sochastic Gradient Descent

- It takes two steps one 0.9 times the history and 0.1 times the current gradient

SGD has trouble navigating ravines, i.e. areas where the surface curves much more steeply in one dimension than in another , which are common around local optima. In these scenarios, SGD oscillates across the slopes of the ravine while only making hesitant progress along the bottom towards the local optimum as in Image 2.

Momentum is 0.1 of the derivative and 0.9 of it is just the same direction I went last time.

Challenges : Overshots its objects and then had to take a U-turn.

Image 2: without Momentum

Image 3 : with Momentum

Momentum is a method that helps accelerate SGD in the relevant direction and dampens oscillations as can be seen in Image 3. It does this by adding a fraction γ  of the update vector of the past time step to the current update vector:

$v_t = \gamma v_{t-1} + \eta \nabla_\theta J( \theta)$

$\theta = \theta - v_t $

Note: Some implementations exchange the signs in the equations. The momentum term γ  is usually set to 0.9 or a similar value.

```python
from IPython.display import YouTubeVideo

def display_yotube_video(url, **kwargs):
    """
    Displays a Youtube video in a Jupyter notebook.

    Args:
        url (string): a link to a Youtube video.
        **kwargs: further arguments for IPython.display.YouTubeVideo

    Returns:
        YouTubeVideo: a video that is displayed in your notebook.
    """
    id_ = url.split("=")[-1]
    return YouTubeVideo(id_, **kwargs)

display_yotube_video("https://youtu.be/uQtTwhpv7Ew?t=6520", width=1200, height=800)
```

## Nesterov Momentum
Look before you leap , look ahead gradient

- It takes two steps one 0.9 times the history and 0.1 times the gradient step state.
- It takes shorter U-turns
- NAG first makes a big jump in the direction of the previous accumulated gradient (brown vector), measures the gradient and then makes a correction (red vector), which results in the complete NAG update (green vector).

- https://www.youtube.com/watch?v=sV9aiEsXanE

## Adagrad and RMSPROP

- In Adagrad we use different learning rates for every parameter at time step t.
- learning rate = η
- Backpropogation :
    $\theta_{t+1, i} = \theta_{t, i} - \dfrac{\eta}{\sqrt{G_{t, ii} + \epsilon}} \cdot g_{t, i}$
- $\epsilon$ is a smoothing term that avoids division by zero (usually on the order of 1e-8)
- $G_{t}$ contains the sum of the squares of the past gradients w.r.t. to all parameters $\theta$

## Adadelta

- Adadelta is an extension of Adagrad that seeks to reduce its aggressive, monotonically decreasing learning rate.
- Instead of accumulating all past squared gradients, Adadelta restricts the window of accumulated past gradients to some fixed size w
- prevent this $G_{t}$ to become very big , as this reduces the learning rate .
- $G_{t}$ in Adadelta is weighted moving average and not sum of gradeint sqaure like adagrad.
- RMSprop and Adadelta have both been developed independently around the same time stemming from the need to resolve Adagrad's radically diminishing learning rates. RMSprop in fact is identical to the first update vector of Adadelta that we derived above

## SGD and SGD+Momentum

## Momentum:
Accelerate SGD in the relevant direction and dampen side-to-side oscillations
It does this by combining a decaying average of previous gradients with the current gradient

 SGD

 SGD+Momentum

## Momentum and ADAGRAD

## ADAGRAD:
The idea is that you keep the running sum of squared gradients during optimization. In this case, we have no momentum term, but an expression g that is the sum of the squared gradients.

## ADAGRAD and RMSPROP

There is a slight variation of AdaGrad called RMSProp that addresses the problem that AdaGrad has. With RMSProp we still keep the running sum of squared gradients but instead of letting that sum grow continuously over the period of training we let that sum actually decay.
In RMSProp we multiply the sum of squared gradients by a decay rate α and add the current gradient weighted by (1- α). The update step in the case of RMSProp looks exactly the same as in AdaGrad where we divide the current gradient by the sum of squared gradients to have this nice property of accelerating the movement along the one dimension and slowing down the movement along the other dimension.

 SGD

 SGD+Momentum

 RMSPROP

##  Adam Optimizer

The main part of the algorithm consists of the following three equations.

The first equation looks a bit like the SGD with momentum. In the case, the term m would be the velocity and β1 the friction term. In the case of Adam, we call m the first momentum and β1 is just a hyperparameter.

The difference to SGD with momentum, however, is the factor (1- β1), which is multiplied with the current gradient.

The second part of the equations, on the other hand, can be regarded as RMSProp, in which we are keeping the running sum of squared gradients. Also, in this case, there is the factor (1-β2) which is multiplied with the squared gradient.

The term v in the equation is called as the second momentum, and β2 is also just a hyperparameter. The final update equation can be seen as a combination of RMSProp and SGD with Momentum.

So far, Adam has integrated the nice features of the two previous optimization algorithms, but here’s a little problem, and that’s the question of what happens in the beginning.

At the very first time step, the first and second momentum terms are initialized to zero. After the first update of the second momentum, this term is still very close to zero. When we update the weight parameters in the last equation, we divide by a very small second momentum term v, resulting in a very large first update step.

This first very large update step is not the result of the geometry of the problem, but it is an artifact of the fact that we have initialized the first and second momentum to zero. To solve the problems of large first update steps, Adam includes a correction clause:

You can see that after the first update of the first and second momentum m and v we make an unbiased estimate of these momentums by taking into account the current time step t. These correction terms make the values of the first and second momentum to be higher in the beginning than in the case without the bias correction.

As a result, the first update step of the neural network parameters does not get that large and we don’t mess up our training in the beginning. The additional bias corrections give us the full form of Adam Optimizer.

 SGD

 SGD+Momentum

 RMSPROP

 ADAM

- https://www.deeplearning-academy.com/p/ai-wiki-optimization-algorithms

## Nadam

As we have seen before, Adam can be viewed as a combination of RMSprop and momentum:
- RMSprop contributes the exponentially decaying average of past squared gradients vt,
- while momentum accounts for the exponentially decaying average of past gradients mt.
- We have also seen that Nesterov accelerated gradient (NAG) is superior to vanilla momentum.
- Nadam (Nesterov-accelerated Adaptive Moment Estimation) thus combines Adam and NAG. In order to incorporate NAG into Adam, we need to modify its momentum term mt.

## ADAGRAD

In settings where Adam converges to a suboptimal solution, it has been observed that some minibatches provide large and informative gradients, but as these minibatches only occur rarely, exponential averaging diminishes their influence, which leads to poor convergence. The authors provide an example for a simple convex optimization problem where the same behaviour can be observed for Adam.

To fix this behaviour, the authors propose a new algorithm, AMSGrad that uses the maximum of past squared gradients
vt rather than the exponential average to update the parameters. vt is defined the same as in Adam above:

$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$

Instead of using vt  (or its bias-corrected version ^vt) directly, we now employ the previous
vt−1 if it is larger than the current one:

$\hat{v}_t = \text{max}(\hat{v}_{t-1}, v_t)$

This way, **AMSGrad results in a non-increasing step size**, which avoids the problems suffered by Adam. For simplicity, the authors also remove the debiasing step that we have seen in Adam. The full AMSGrad update without bias-corrected estimates can be seen below:

$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t  $

$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 $

$\hat{v}_t = \text{max}(\hat{v}_{t-1}, v_t)  $

$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{\hat{v}_t} + \epsilon} m_t$

## Additional strategies for optimizing SGD

### Shuffling and Curriculum Learning

Generally, we want to avoid providing the training examples in a meaningful order to our model as this may bias the optimization algorithm. Consequently, it is often a good idea to shuffle the training data after every epoch.

On the other hand, for some cases where we aim to solve progressively harder problems, supplying the training examples in a meaningful order may actually lead to improved performance and better convergence. The method for establishing this meaningful order is called Curriculum Learning .

Zaremba and Sutskever were only able to train LSTMs to evaluate simple programs using Curriculum Learning and show that a combined or mixed strategy is better than the naive one, which sorts examples by increasing difficulty.

### Batch Normalization

Another good standard approach to reducing overfitting is Batch Normalization.

In general, the inputs to your neural network should always be normalized. Normalization is a process where given a set of data, you subtract from each element the mean value for that data set and divide it by the data set's standard deviation. By doing so, we put the input values onto the same "scale", in that all the values have been normalized into units of "# of standard deviations from the mean".

The reason we would want to put our inputs onto the same scale is because unbalanced inputs with a large range of magnitudes can typically cause instability in neural networks. An extremely large input can often cascade down through the layers. Typically such an imbalance creates gradients that are also wildly imbalanced, and this makes the optimization process difficult to prevent things like explosion. It also creates imbalanced weights. Normalizing inputs avoids this problem.

Often times with images, we don't worry about dividing by the standard deviation, but just subtract the mean.

Occasionally, these instabilities can arise during training. Imagine that at some point during training, we end up with one extremely large weight. This extremely large weight will then produce an extremely large output value for some element of the output vector, and this imbalance will again pass through the neural network and make it unstable.

One idea is to normalize the activation outputs. Unfortunately, this won't prevent SGD from trying to create an imbalanced weight again during the next back-propagation, and trying to solve the problem this way will just cause SGD to continuously try to undo this activation layer normalization.

Batch normalize extends this idea with two additions:

- after normalizing the activation layer, let's multiply the outputs by an arbitrarily set parameter, and add to that value an additional parameter, therefore setting a new standard deviation and mean

- Make all four of these values (the mean and standard deviation for normalization, and the two new parameters to set arbitrary mean and standard deviations after the normalization) trainable.

This ensures that the weights don't tend to push very high or very low (since the normalization is included in the gradient calculations, so the updates are aware of the normalization). But it also ensures that if a layer does need to change the overall mean or standard deviation in order to match the output scale, it can do so.

By default, you should always include batch normalization, and all modern neural networks do so because:

- Adding batchnorm to a model can result in 10x or more improvements in training speed
- Because normalization greatly reduces the ability of a small number of outlying inputs to over-influence the training, it also tends to reduce overfitting.

### Dropout
You'll notice in observing our metrics while training a Vgg model for Cats and Dogs that our training accuracy is usually lower than our validation. This points to underfitting, and the source of this in Vgg is relatively simple.

If you observe the Vgg layers in Keras, you'll notice a layer called Dropout. Dropout (which only happens during training) occurs after the activation layers, and randomly sets activations to zero. Why would we want to randomly delete parts of neural network? It turns out that this allows us to prevent overfitting. We can't overfit exactly to our training data when we're consistently throwing away information learned along the way. This allows our neural network to learn to generalize.

In the case of Vgg, the Dropout rate is set to 0.5. This seems quite large, and given the complexity of classification for Imagenet, it seems reasonable to want to make our dropout rate this high. However, for something much simpler like Cats and Dogs, we can see that we're underfitting, and we might have more success by retaining more information and lowering our dropout rate, and retraining.

### Gradient noise

Neelakantan add noise that follows a Gaussian distribution $N(0, \sigma^2_t)$ to each gradient update:

$g_{t, i} = g_{t, i} + N(0, \sigma^2_t)$

They anneal the variance according to the following schedule:

$\sigma^2_t = \dfrac{\eta}{(1 + t)^\gamma}$

## Weigth Decay

### What is Weigth Decay ? And why do we care ?

- We do a lot of feature engineering and one thing which we do very regularly is to reduce the paramters.
- So what is so wrong in this approach ? So why do we care? Why would I want to use more parameters?
- Because more parameters means more nonlinearities, more interactions, more curvy bits.
- Real life is full of curvy bits. Real life does not look like a straight line. But we don't want them to be more curvy than necessary or more interacting than necessary. Therefore let's use lots of parameters and then penalize complexity.
- So one way to penalize complexity is let's sum up the value of your parameters.Now that doesn't quite work because some parameters are positive and some are negative. So what if we sum up the square of the parameters, and that's actually a really good idea.
- Let's create a model, and in the loss function we're going to add the sum of the square of the parameters.
- Let's actually create a model, and in the loss function we're going to add the sum of the square of the parameters. Now here's a problem with that though. Maybe that number is way too big, and it's so big that the best loss is to set all of the parameters to zero. That would be no good.
- So we want to make sure that doesn't happen, so therefore let's not just add the sum of the squares of the parameters to the model but let's multiply that by some number that we choose.That number that we choose is called wd (weigth decay).
- loss function and we're going to add to it the sum of the squares of parameters multiplied by some number wd.

### Explanation with little Maths :

- we have a cost or error function **E(w)** that we want to minimize. Gradient descent tells us to modify the weights w in the direction of steepest descent in E. This is called backpropogation.
   -  $\begin{equation}w_i \leftarrow w_i-lr\frac{\partial E}{\partial w_i},\end{equation}$
- where lr is the learning rate, and if it's large you will have a correspondingly large modification of the weights wi (in general it shouldn't be too large, otherwise you'll overshoot the local minimum in your cost function).
- In order to effectively limit the number of free parameters in your model so as to avoid over-fitting, it is possible to regularize the cost function remember more parameters means more nonlinearities, more interactions, more curvy bits.Real life is full of curvy bits.
- An easy way to do that is by introducing a zero mean Gaussian prior over the weights, which is equivalent to changing the cost function to
    - $\widetilde{E}(\mathbf{w})=E(\mathbf{w})+\frac{wd}{2}\mathbf{w}^2$
- In practice this penalizes large weights and effectively limits the freedom in your model. The regularization parameter wd determines how you trade off the original cost E with the large weights penalization.
- Applying gradient descent to this new cost function we obtain: (its just diffrentiation of new loss **$\widetilde{E}(\mathbf{w})$**)
    - $\begin{equation}w_i \leftarrow w_i-lr\frac{\widetilde{E}(\mathbf{w})}{\partial w_i} \end{equation}$ Substitute the value of **$\widetilde{E}(\mathbf{w})$** from above
    - $\begin{equation}w_i \leftarrow w_i-lr\frac{\partial E}{\partial w_i}-lr . wd. w_i.\end{equation}$
- The new term **−lr.wd.wi** coming from the regularization causes the weight to decay in proportion to its size.
- When it's in this form ($wd . w^2$) where we add the square to the loss function, that's called L2 regularization.
- When it's in this form ($wd . w$) where we subtract  times weights from the gradients, that's called **weight decay**.

```python
from IPython.display import IFrame

IFrame(src="https://ruder.io/optimizing-gradient-descent/", width=800, height=400)
```

```python
IFrame(src="https://cs231n.github.io/neural-networks-3/", width=800, height=400)
```

```python
IFrame(src="http://wiki.fast.ai/index.php/Lesson_3_Notes", width=800, height=400)
```


# My First Univariate Linear Regression

##### Dataset from: https://www.kaggle.com/andonians/random-linear-regression
##### NoteBook by: [iArunava](https://github.com/iArunava)

## Introduction

This project is my first creating a univariate linear regression model. And in general, my first project :)

The Dataset is collected so as it is best suited for a univariate linear regression model.

That is all, its a quite simple project.

So, lets get Started!!

## Libraries Required

1) _Numpy_: NumPy is the fundamental package for scientific computing with Python.

2) _Pandas_: Pandas is an easy-to-use data structures and data analysis tools for the Python programming language.

3) _Matplotlib_: Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.

## Goal of the project

Create a Univariate Linear Regression Model **without the use of any readily available libraries.**

To get a better understanding of how the algorithm works under the hood.

### Step 1:

Get the Dataset. Its important to notice that the dataset is really small and, thus, is confirmed that there are no missing values. So, no data cleaning is done and the data is considered to be good for the goal of this project.

**Note**: The process of dividing the data into test and training set is already done by the creator of the dataset.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Getting the training and test data
df_test  = pd.read_csv('./DataSet/test.csv')
df_train = pd.read_csv('./DataSet/train.csv')

# Separating the training and test datasets into 'X' and 'Y' (i.e the features and output)
xtrain = df_train.iloc[:, 0]
ytrain = df_train.iloc[:, 1]

xtest = df_test.iloc[:, 0]
ytest = df_test.iloc[:, 1]

# Performing feature normalization (Min-Max Standardization)
xtrain /= (xtrain.max() - xtrain.min())
ytrain /= (ytrain.max() - ytrain.min())

# Adding the bias unit to the xtrain and xtest
xtrain = pd.concat([pd.Series(np.ones(len(xtrain))), xtrain], axis=1)
xtest  = pd.concat([pd.Series(np.ones(len(xtrain))), xtest], axis=1)

# Getting just the features without the bias unit
xtr = xtrain.iloc[:, 1]
xte = xtest.iloc[:, 1]
```

### Step 2:

So, now we have features 'X' and response 'Y' for the training data and the test data.
Next, lets randomly choose a hypothesis and lets train it! **Exciting !!**

```python
hypo = np.random.randn(2, 1)
```

Before starting to train the model, lets see what the initial cost is for the randomly chosen hypothesis.

**Note:** We are going to calculate SSE (Sum of Squared Errors) and provide the Accuracy percentage (for the randomly chosen hypothesis.

Lets have a look at the formula used for calculating the cost function.

```latex
%%latex
\begin{align}
J(\Theta) = \frac{1}{2 \times m} \times \sum_{i=1}^m ( h_{\Theta}(x^{(i)}) - y^{(i)})^{2}
\end{align}
```

\begin{align}
J(\Theta) = \frac{1}{2 \times m} \times \sum_{i=1}^m ( h_{\Theta}(x^{(i)}) - y^{(i)})^{2}
\end{align}

```python
# Cost Function
def get_hypo(x, hypo):
    return x.dot(hypo).iloc[:, 0]

def cost_function(x, y, hypo):
    cost = (((get_hypo(x, hypo) - y)**2).sum()) * (1/(2*len(x)))
    return cost

def plot_fit_line():
    f = lambda a : a.dot(hypo)
    plt.scatter(xtr, ytrain)
    plt.plot(xtr, f(xtrain), c='orange')
    plt.show()

print ('Initial Cost: ' + str(cost_function(xtrain, ytrain, hypo)))
plot_fit_line()
```

So, that is how it seems initially.

Now, as this is a practice work. I will try different learning rates to see how things work out.

All righty then, lets get started by declaring all the essential functions that will be requried.

So, for optimizing the model we will be using Gradient Descent.

Let's have a look at the formula used for gradient descent.

```latex
%%latex
\begin{align}
\Theta_{j} = \Theta_{j} - \alpha \frac{1}{m}\sum_{i=1}^m ( h_{\Theta}(x^{(i)}) - y^{(i)} ) x_{j}^{(i)}
\end{align}
```

\begin{align}
\Theta_{j} = \Theta_{j} - \alpha \frac{1}{m}\sum_{i=1}^m ( h_{\Theta}(x^{(i)}) - y^{(i)} ) x_{j}^{(i)}
\end{align}

This is to be performed for all `j` where `j` is a feature.

So, lets code it up!

```python
# Derivative of the cost function
def f_cost_func(x, y, hypo, j, ALPHA):
    return (((get_hypo(x, hypo) - y) * x.iloc[:, j]).sum()) * (1.0/len(x)) * ALPHA

# Gradient Descent
def gradient_descent(iter_num, alpha, reset_hypo=False):
    '''
    Passing just `iter_num` rest is read from the global variables
    '''
    if reset_hypo: # If `reset_hypo` is true, then the model is reset to a random
        hypo = np.random.randn(2, 1)

    ALPHA = alpha # Learning Rate
    for j in range(iter_num):
        for i in range(len(hypo)):
            t_hypo = hypo
            hypo[i] -= f_cost_func(xtrain, ytrain, t_hypo, i, ALPHA)
        plt.plot(j, cost_function(xtrain, ytrain, hypo), 'bo')
    plt.show()

    return hypo
```

Now, as I said, I will be testing with different learning rates. So, below I do three tests.

**Note:** I have a `reset_hypo` parameter in `gradient_descent()` which if true resets the model. You can use that, and play around with different learning rates.

```python
# Testing with Learning Rate = 0.01
hypo = gradient_descent(500, 0.01, True)
plot_fit_line()
print ('Initial Cost: ' + str(cost_function(xtrain, ytrain, hypo)))
```

```python
# Testing with Learning Rate = 0.1
hypo = gradient_descent(500, 0.1, True)
plot_fit_line()
print ('Initial Cost: ' + str(cost_function(xtrain, ytrain, hypo)))
```

```python
# Testing with Learning Rate = 0.5
hypo = gradient_descent(500, 0.5, True)
plot_fit_line()
print ('Initial Cost: ' + str(cost_function(xtrain, ytrain, hypo)))
```

```python
# Testing with Learning Rate = 4
hypo = gradient_descent(500, 4, True)
plot_fit_line()
print ('Initial Cost: ' + str(cost_function(xtrain, ytrain, hypo)))
```

There you go!!!

Model trained!!!

**Things to Notice:**

_**1)**_ Larger Learning Rate causes faster divergence

_**2)**_ Too much larger rate causes one to overshoot the minima and as can be seen from the above example which uses learning rate as `4`, that the cost increases and decreases alternatively. You can try with much larger rates, and see that the cost increases each iteration.

_**3)**_ Smaller learning rates, on the other hand converges alright if a convinient number of iterations are used and it takes time.

_**4)**_ With too small learning rates, one might never reach the global optima (Well one would for sure, with many more iterations)

<hr/>

Finally, lets train the model with learning rate = `0.5`

```python
hypo = gradient_descent(500, 0.5, True)
print ('\nHypothesis: \n' + str(hypo))
```

### Step 3:

With this, lets see the accuaracy of our model on the test data.

```python
# Accuarcy %
print ('Accuracy : ' + str(100 - (((ytest - xtest.dot(hypo).iloc[:, 0]) / ytest) *100).mean()) + '%')
```

    Accuracy : 89.22284928662384%

Well Well Well!!

We got a accuracy of `89.22%`.

Which is not bad!

## Conclusion

This is how we create a univariate linear regression model.

The data was generated randomly, for univariate linear regression purposes. So, cannot conclude anything specific.

In brief, had too much fun creating my first univariate linear regression model.