Presented Below are two plots in subplot configuration.  The upper plot is a sum of the cusp crossings.  The lower plot is a plot of the cusp latitude and the spacecraft latitude/longitude plotted against time.  Also included is the process of thoughts that led to that conclusion.

I used the Tsyganenko model of the cusp position.  This model describes the cusp position in the noon-midnight meridian using a cylindrical coordinate system.  The $r$ and $\theta$ components are occuring in the xz plane.  The neutral axis is therefore the $y$ axis.

Tsyganenko provides the following equation for a simplified cusp model
\begin{equation}
\phi_{c} = arcsin(\frac{\sqrt{\rho}}{\sqrt{\rho + \sin^{-2}(\phi_{1}) -1}}) + \psi(t)
\end{equation}

Here, $\rho$ is the radial of the satellite from the center of the earth.  $\phi_{c}$ is the colatitude of the cusp position.  $\psi$ is the dipole tilt of the cusp.  $\phi_{1}$ is provided by the equation $\phi_{c0} - (\alpha_{1} \psi + \alpha_{2}\psi^{2})$.  As in (Tsyganenko) the values of $\phi_{c}$, $\alpha_{1}$, and $\alpha_{2}$ are 0.24, 0.1287, and 0.0314 respectively.  ((note that that last sentence is almost a literal copy paste of the Tsyg paper)).  Note that $\psi (t)$ is my own addition and that the original notation is $\psi$.  I want to make it extremely clear that $\psi$ varies with time, requiring that the tsyganenko library be a function of time as well.

The dataset used here depicts the path of a satellite on a single day of its orbit 01-Jan-2019, at a $65^{\circ}$ inclined orbit.

```python
import tsyganenko as tsyg
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from spacepy import coordinates as coord
import spacepy.time as spt
from spacepy.time import Ticktock
import datetime as dt
from mpl_toolkits.mplot3d import Axes3D
import sys

#adding the year data here so I don't have to crush my github repo
pathname = '../../data-se3-path-planner/yearData/cuspCrossings2019/'
sys.path.append(pathname)

# find phi for a stationary cusp
r = np.linspace(0,10, 1000)
phi_c = tsyg.getPhi_c(r)
plt.plot(r,phi_c)
plt.ylabel('phi_c (radians)')
plt.xlabel('earth radii ($R_{e}$)')
plt.title('Tsyganenko Cusp Position')
plt.show()

```

Note that below, there are many ways that we could set up the subplot, but I just chose (programattically) to do it a certain way provided in http://matplotlib.org/examples/pylab_examples/subplots_demo.html

Note: The units for this system is solar magnetic degrees.  Some plots I did gave me that kind of bias to think that SM would be the best coordinate system for this. (That's probably not true but there are many ways to solve this problem).

```python
# the orbit is fairly easy to get
df = pd.read_csv('singleday.csv')
# df = pd.read_csv('5day.csv')
t = df['DefaultSC.A1ModJulian'] + 29999.5
x = df['DefaultSC.gse.X']
y = df['DefaultSC.gse.Y']
z = df['DefaultSC.gse.Z']

# set the "ticks"
cvals = coord.Coords([[i,j,k] for i,j,k in zip(x,y,z)], 'GSE', 'car')
cvals.ticks = Ticktock(t,'MJD')
# originally SM
sm = cvals.convert('SM', 'sph')

# t = np.asarray(t)
# plotting stuff
f, axarr = plt.subplots(2, sharex=True)
# axarr[0].plot(t, y)
axarr[0].set_title('Spacecraft Latitude')
axarr[0].set_ylabel('latitude (sm deg)')
axarr[0].plot(sm.ticks.MJD, sm.lati)
axarr[1].set_title('Spacecraft Longitude')
axarr[1].set_ylabel('longitude (sm deg)')
axarr[1].plot(sm.ticks.MJD, sm.long)

plt.xlabel('time (unit)')

plt.show()
```

The next thing to do is compute the cusp latitude and longitude using tsyganenko.  The cusp angular position is a function of radial distance from the earth as given in the Tsyganenko equation.  This means that different "tracks" will be given as we vary $R_{e}$.  Since I am using a very circular orbit, with $e \sim 0.02$, I will plot the cusp latitude and longitude below as a function of the spacecraft altitude in my GMAT file.  The semi major axis is given as $7191 km$, $\sim 1.127 R_{e}$.

```python
rs = 1.127 # average of the radius of the orbit
# rs = np.array([1,2,3])
phic = tsyg.getPhi_c(rs)
x,y,z= tsyg.tsygCyl2Car(phic,rs)
# print("x,y,z",xt,yt,zt)
print("x",x)
print("y",y)
print("z",z)

# next lets put these three coordinates into a spacepy coordinates object

singletrack = coord.Coords([[x,y,z]]*len(t), 'SM', 'car')
singletrack.ticks = Ticktock(t,'MJD')
singletrack = singletrack.convert('GEI','sph')
f, axarr2 = plt.subplots(2, sharex=True)
# axarr[0].plot(t, y)
axarr2[0].set_title('Cusp Longitude and Latitude')
axarr2[0].set_ylabel('latitude (sm deg)')
axarr2[0].scatter(singletrack.ticks.MJD, singletrack.lati)
axarr2[1].set_title('Cusp and Spacecraft Latitude')
axarr2[1].set_ylabel('longitude (sm deg)')
axarr2[1].scatter(singletrack.ticks.MJD, singletrack.long)
plt.xlabel('time (unit)')
f,axarr3 = plt.subplots(1)
axarr3.set_title('Lat vs Lon')
axarr3.set_ylabel('latitude')
axarr3.set_xlabel('longitude')
axarr3.plot(singletrack.long,singletrack.lati, label='r = 1.127 Re')
axarr3.scatter(sm.long,sm.lati,label='spacecraft orbit actual')
axarr3.legend(loc='lower left')
plt.show()
```

So I suppose this does show that for a region of the stationary cusp, that the satellite potentially crosses it.  However, the cusp location moves throughout the day so it's conceivable that the cusp avoids the orbit of the satellite, and I need a way to test for that.  The other big question is how do I integrate dipole tilt into this at the same time.  The first question I need to answer is whether or not the dipole tilt matters in SM.

One other approach I could take is to come up with the cusp position as $f(\rho,t)$, then compare that latitude and longitude to the spacecraft latitude and longitude. That's really what I need to do.

Step by Step what I will attempt to do:

1.  Move the cusp and prove that it is moving.  I'll use the r = 1.127Re track to do this and one years worth of data. x

2.  Once I have the cusp latitude and longitude, compare the satellite latitude and longitude.

3.  Do the cusp crossings count.

4.  Plot the lat/lon of the cusp/spacecraft and the cusp crossings subplots together.

From here one we will be using the data from a year's flight around the earth, because these trends are periodic over a year.

```python
# the orbit is fairly easy to get
#df = pd.read_csv(pathname+'zero.csv')
df = pd.read_csv('5day.csv')
# df =pd.read_csv(pathname+'zero_year.csv')
t = df['DefaultSC.A1ModJulian'] + 29999.5
x = df['DefaultSC.gse.X']
y = df['DefaultSC.gse.Y']
z = df['DefaultSC.gse.Z']

# set the "ticks"
cvals = coord.Coords([[i,j,k] for i,j,k in zip(x,y,z)], 'GSM', 'car')
cvals.ticks = Ticktock(t,'MJD')
sm = cvals.convert('SM','sph')
spacecraft_gei = cvals.convert('GEI', 'sph')
spacecraft = cvals.convert('GSM','sph')

# t = np.asarray(t)
# plotting stuff
f, axarr = plt.subplots(2, sharex=True)
# axarr[0].plot(t, y)
axarr[0].set_title('Spacecraft Latitude')
axarr[0].set_ylabel('latitude (sm deg)')
axarr[0].plot(sm.ticks.MJD, sm.lati)
axarr[1].set_title('Spacecraft Longitude')
axarr[1].set_ylabel('longitude (sm deg)')
axarr[1].plot(sm.ticks.MJD, sm.long)

plt.xlabel('time (unit)')

plt.show()

f, a = plt.subplots(1)
plt.title('Solar Magnetic Orbit')
plt.xlabel('Solar Magnetic Longitude')
plt.ylabel('Solar Magnetic Latitude')
a.scatter(sm.long,sm.lati)
plt.show()
```

```python
psi = tsyg.getTilt(t)
plt.plot(t,psi)
plt.title('Dipole Tilt')
plt.xlabel('MJD')
plt.ylabel('Dipole Tilt (gsm deg)')
plt.show()
# here the OUTPUT, psi is in degrees
```

```python
# output cusp position with the dipole tilt
r = 1.127
psi = np.deg2rad(psi)
phi_c = tsyg.getPhi_c(r,psi)
phi_c = np.rad2deg(phi_c)
plt.figure()
plt.plot(t,phi_c)
plt.title('Cusp location over time for $r = 1.127R_{e}$')
plt.xlabel('MJD')
plt.ylabel('Cusp Location (gsm deg)')
plt.show()
```

```python
# next lets get the spherical coordinates and plot the latitude and longitudes, which SHOULD
# allow us to get the cusp crossings.
xc,yc,zc = tsyg.tsygCyl2Car(phi_c,r)

#originally it was GSM
cusp_location = coord.Coords([[i,j,k] for i,j,k in zip(xc,yc,zc)], 'GSM', 'car')
cusp_location.ticks = Ticktock(t,'MJD')
cusp_location = cusp_location.convert('GEI','sph')
plt.scatter(cusp_location.long, cusp_location.lati,label='cusp')
plt.scatter(spacecraft_gei.long,spacecraft_gei.lati,color='green', label='satellite')
plt.xlabel('gei longitude (deg)')
plt.ylabel('gei latitude (deg)')
plt.title('Cusp Latitude vs. Longitude (gei)')
plt.legend()
plt.show()

```

The biggest issues of this plot are that we aren't able to see where the cusp is at a certain time, it's just a ground track.  Therefore, we need to also put in the plots of lat/lon vs. time.

Close to solving this.  I think my cusp actually behaves somewhat correctly. The above plot was actually really promising, but for some reason my  Now I need to use the Tsyganenko equation as an function of the satellite's location.  It really don't make much sense to me why the cusp wouldn't seem to change in latitude.  My guess is that there's tiny circles there if you look closely at the data.

I'll deal with what I suspect are innacuracies later I feel like I've done okay thus far.  Like I "feel" like the equations are wrong.
Open Questions at this point:
1.  should the latitude go so low?
2.

```python
# plot of the latitude and the longitude of the spacecraft at each timestep,
f,ax = plt.subplots(2)
# i can probably use a list comprehension here for the cusp location
# just feel really unmotivated rn
# xk,yk,zk = tsyg.tsygCyl2Car()
# longitude vs. time
ax[0].set_title('Cusp and Satellite Longitude')
ax[1].set_title('Cusp and Satellite Latitude')
ax[0].plot(spacecraft_gei.ticks.MJD, spacecraft_gei.long)
ax[1].plot(spacecraft_gei.ticks.MJD, spacecraft_gei.lati)
plt.show()
```

```python
# testing the function for
x,y,z = tsyg.orbitalCuspLocation(cvals,t)
gsm2 = spacecraft_gei.convert('GSM','car')
fig = plt.figure()
ax = fig.add_subplot(111,projection='3d')

# my present guess is again, a disagreement between the
# coordinate systems
ax.plot(x,y,z)
ax.plot(gsm2.x,gsm2.y,gsm2.z)
print("satx",gsm2.x)
print("saty",gsm2.y)
print("satz",gsm2.z)
ax.set_title('Single Plot of Cusp and Orbit Location')
plt.show()

# okay lets look at the coordinate systems here.  My initial
# thought is that if it's the cusp location it's probably okay for it
# to be really big, esp if the corresponding location is like in the
# nightside of the earth, so let's not panic because these values
# are really high.
```

So I think the main way we can test for cusp crossings is that I can plot the latitude and Longitude vs. time of each component, then, I can do the summation similar to how I was making the colormap earlier

```python
# gsm is the satellite orbit (terrible variable name)
# x,y,z is the coordinates output from the cusp model (i think it's in gsm)
# note that x,y,z are all np.ndarray typreddit.com/r/smashbroses

# instatiate the cusp coordinates to a spacepy coords object
cusp_location = coord.Coords([[i,j,k] for i,j,k in zip(x,y,z)], 'GSM', 'car')
cusp_location.ticks = Ticktock(t,'MJD')
cusp_location = cusp_location.convert('GSM','sph')

# put whatever conversions here to maintain consistency
system = 'GSM'
cusp_location = cusp_location.convert(system,'sph')
spacecraft = spacecraft.convert(system,'sph')

plt.plot(cusp_location.ticks.MJD,cusp_location.lati, label='cusp')
plt.plot(spacecraft.ticks.MJD, spacecraft.lati,label='spacecraft')
plt.title('Cusp and Spacecraft Latitude vs. MJD time')
plt.xlabel('MJD days')
plt.ylabel('latitude (gsm deg)')
plt.legend()
plt.show()

plt.plot(cusp_location.ticks.MJD, cusp_location.long,color='blue', label='cusp')
plt.plot(spacecraft.ticks.MJD, spacecraft.long, color='green', label='spacecraft')
plt.title('Cusp and Spacecraft Longitude vs. MJD time')
plt.xlabel('MJD days')
plt.ylabel('longitude (gsm deg)')
plt.legend()
plt.show()
```

```python
count = 0
thresh = 2 #degrees
for satlat, cusplat, satlong, cusplong in zip(spacecraft.lati, cusp_location.lati, spacecraft.long, cusp_location.long):
    if abs(cusplat-satlat)<=thresh and abs(satlong-cusplong)<=thresh:
        count+=1
    else:
        pass

print("count = count",count)
print("lenght of spacecraft.lati",len(spacecraft.lati))
```

    count = count 0
    lenght of spacecraft.lati 4341


# Tutorial: Dimensionality Reduction

---
# Objectives

In this notebook we'll explore how to use [Principal Components Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA) by projecting data from two neurons (two dimensions) onto the *eigenvectors of its covariance matrix* (a new basis).

Overview:
- Perform PCA (Get the eigenvectors of the activity of two neurons).
- Plot and explore the eigenvalues.

---
# Setup

```python
# @markdown Execute this cell to specify Imports and Functions

# Imports

import numpy as np
import matplotlib.pyplot as plt

# Figure Settings
import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")

# Plotting Functions

def plot_eigenvalues(evals):
  """
  Plots eigenvalues.

  Args:
      (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """
  plt.figure(figsize=(6, 4))
  plt.plot(np.arange(1, len(evals) + 1), evals, 'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  plt.xticks(np.arange(1, len(evals) + 1))
  plt.ylim([0, 2.5])

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  scatter plot of their joint activity. The title indicates the sample
  correlation calculated from the data.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(X[:, 0], color='k')
  plt.ylabel('Neuron 1')
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(X[:, 1], color='k')
  plt.xlabel('Sample Number (sorted)')
  plt.ylabel('Neuron 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
           markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))
  plt.show()

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data
  but with colors corresponding to projections onto basis 1 (red) and
  basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random
  variable.

  Args:
    Y (numpy array of floats) : Data matrix in new basis each column
                                corresponds to a different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(Y[:, 0], 'r')
  plt.ylabel('Projection \n basis vector 1')
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(Y[:, 1], 'b')
  plt.xlabel('Sample number')
  plt.ylabel('Projection \n basis vector 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:, 0], Y[:, 1], '.', color=[.5, .5, .5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:, 0], Y[:, 1])[0, 1]))
  plt.show()

def plot_basis_vectors(X, W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable
    W (numpy array of floats) : Square matrix representing new orthonormal
                                basis each column represents a basis vector

  Returns:
    Nothing.
  """

  plt.figure(figsize=[4, 4])
  plt.plot(X[:, 0], X[:, 1], '.', color=[.5, .5, .5])
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
           label='Basis vector 1')
  plt.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
           label='Basis vector 2')
  plt.legend()
  plt.show()

# Helper Functions

def sort_evals_descending(evals, evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats)    : Vector of eigenvalues
    evectors (numpy array of floats) : Corresponding matrix of eigenvectors
                                        each column corresponds to a different
                                        eigenvalue

  Returns:
    (numpy array of floats)          : Vector of eigenvalues after sorting
    (numpy array of floats)          : Matrix of eigenvectors after sorting
  """

  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:, index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:, 0],
                           1 / np.sqrt(2) * np.array([1, 1]))) > np.pi / 2:
      evectors[:, 0] = -evectors[:, 0]
    if np.arccos(np.matmul(evectors[:, 1],
                           1 / np.sqrt(2) * np.array([-1, 1]))) > np.pi / 2:
      evectors[:, 1] = -evectors[:, 1]
  return evals, evectors

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random
  variable.

  Args:
    var_1 (scalar)                     : variance of the first random variable
    var_2 (scalar)                     : variance of the second random variable
    cov_matrix (numpy array of floats) : desired covariance matrix

  Returns:
    (numpy array of floats)            : samples from the bivariate Gaussian,
                                          with each column corresponding to a
                                          different random variable
  """

  mean = np.array([0, 0])
  X = np.random.multivariate_normal(mean, cov_matrix, size=1000)
  indices_for_sorting = np.argsort(X[:, 0])
  X = X[indices_for_sorting, :]
  return X

def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and
  correlation coefficient.

  Args:
    var_1 (scalar)         :  variance of the first random variable
    var_2 (scalar)         :  variance of the second random variable
    corr_coef (scalar)     :  correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1, cov], [cov, var_2]])
  return cov_matrix

def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats) : arbitrary 2D vector used for new basis

  Returns:
    (numpy array of floats)   : new orthonormal basis columns correspond to
                                basis vectors
  """

  u = u / np.sqrt(u[0] ** 2 + u[1] ** 2)
  w = np.array([-u[1], u[0]])
  W = np.column_stack((u, w))
  return W

def change_of_basis(X, W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)   : Data matrix expressed in new basis
  """

  Y = np.matmul(X, W)
  return Y

def get_sample_cov_matrix(X):
  """
  Returns the sample covariance matrix of data X

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Covariance matrix
  """

  # Subtract the mean of X
  X = X - np.mean(X, 0)

  # Calculate the covariance matrix (hint: use np.matmul)
  cov_matrix = 1 / X.shape[0] * np.matmul(X.T, X)

  return cov_matrix

def pca(X):
  """
  Performs PCA on multivariate data.

  Args:
    X (numpy array of floats) : Data matrix each column corresponds to a
                                different random variable

  Returns:
    (numpy array of floats)   : Data projected onto the new basis
    (numpy array of floats)   : Vector of eigenvalues
    (numpy array of floats)   : Corresponding matrix of eigenvectors

  """

  # Calculate the sample covariance matrix
  cov_matrix = get_sample_cov_matrix(X)

  # Calculate the eigenvalues and eigenvectors
  evals, evectors = np.linalg.eigh(cov_matrix)

  # Sort the eigenvalues in descending order
  evals, evectors = sort_evals_descending(evals, evectors)

  # Project the data onto the new eigenvector basis
  score = change_of_basis(X, evectors)

  return score, evectors, evals
```

---
# Section 1: What does PCA do?

In the last tutorial (Geometric View of Data) you explored what it means to ***project*** data onto new *orthonormal* dimensions. The following plot shows data in two dimensions (the activity of two neurons) and an orthonormal basis that would:

1. Explain the most variance in the data with the first basis.
2. Reduce correlations in the data.

The goal of PCA is to find an orthonormal basis capturing the direction(s) of maximum variance of the data. More precisely, the $i$th basis vector is the direction that maximizes the projected variance, while being orthogonal to all previous basis vectors. Mathematically, these basis vectors are the eigenvectors of the covariance matrix (also called *loadings* or *weights*).

To perform PCA, we project the data onto the eigenvectors of the covariance matrix, i.e.:
\begin{align}
\bf S = X W
\end{align}
where $\bf S$ is an $N_\text{samples} \times N$ matrix representing the projected data (also called *scores*), and $\bf W$ is an $N\times N$ orthogonal matrix, each of whose columns represents the eigenvectors of the covariance matrix (also called *weights* or *loadings*).

Executing the following code cell will plot the eigenvectors of the covariance matrix for a set of random data with a correlation coefficient specified by the slider.

> Note that the sample number is sorted based on the value of neuron 1's activity for each datapoint.

```python
# @markdown Execute this cell to enable widget

def refresh(corr_coef=0.5):
  # corr_coef=0.5
  variance_1 = 1
  variance_2 = 1

  # Compute covariance matrix
  cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)

  # Generate data with this covariance matrix
  X = get_data(cov_matrix)

  # Perform PCA on the data matrix X
  score, evectors, evals = pca(X)

  W = evectors

  # Plot the data projected into the new basis
  with plt.xkcd():
      # plot_data_new_basis(Y)
    fig = plt.figure(figsize=[20, 10])
    gs = fig.add_gridspec(4, 4)

    ax1 = fig.add_subplot(gs[0, 0])
    ax1.plot(X[:, 0], color='k')
    plt.ylabel('Neuron 1')
    plt.title('Sample var 1: {:.1f}'.format(np.var(X[:, 0])))
    ax1.set_xticklabels([])

    ax2 = fig.add_subplot(gs[1, 0])
    ax2.plot(X[:, 1], color='k')
    plt.xlabel('Sample Number')
    plt.ylabel('Neuron 2')
    plt.title('Sample var 2: {:.1f}'.format(np.var(X[:, 1])))

    ax3 = fig.add_subplot(gs[0:2, 1])
    ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
            markeredgewidth=0)
    ax3.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
              label='Basis vector 1')
    ax3.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
              label='Basis vector 2')
    ax3.axis('equal')
    plt.xlabel('Neuron 1 activity')
    plt.ylabel('Neuron 2 activity')
    plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))

    # plot_data_new_basis(score)
    Y = score
    ax4 = fig.add_subplot(gs[2, 0])
    ax4.plot(Y[:, 0], 'r')
    plt.ylabel('Projection \n basis vector 1')
    ax5 = fig.add_subplot(gs[3, 0])
    ax5.plot(Y[:, 1], 'b')
    plt.xlabel('Sample number')
    plt.ylabel('Projection \n basis vector 2')
    ax6 = fig.add_subplot(gs[2:4, 1])
    ax6.plot(Y[:, 0], Y[:, 1], '.', color=[.5, .5, .5])
    ax6.axis('equal')
    plt.xlabel('Projection basis vector 1')
    plt.ylabel('Projection basis vector 2')
    plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:, 0], Y[:, 1])[0, 1]))
    plt.show()

_ = widgets.interact(refresh, corr_coef=(-1,1,0.1))
```

You should be able to see that the eigenvectors align with the geometry of the data.

PCA also has the useful property that the projected data (*scores*) are uncorrelated.

---
# Section 2: How much variance in the data is explained by each basis vector of the PCA?

Finally, we will examine the eigenvalues of the covariance matrix. Remember that each eigenvalue describes the variance of the data projected onto its corresponding eigenvector. This is an important concept because it allows us to rank the PCA basis vectors based on how much variance each one can capture.

First, run the code cell below to generate a random dataset and visualize the eigenvectors (orthonormal basis) of that dataset (computed by PCA). The first eigenvector is plotted in red and the second eigenvector is plotted in blue.

```python
# @markdown Execute this code cell to plot a new set of data (the activity of two neurons)
# @markdown with correlation coefficient 0.8

corr_coef=0.8
  # corr_coef=0.5
variance_1 = 1
variance_2 = 1

# Compute covariance matrix
cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)

# Generate data with this covariance matrix
X = get_data(cov_matrix)

# Perform PCA on the data matrix X
score, evectors, evals = pca(X)

W = evectors

# Plot the data projected into the new basis
with plt.xkcd():
    # plot_data_new_basis(Y)
  fig = plt.figure(figsize=[10, 6])
  gs = fig.add_gridspec(2, 2)

  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(X[:, 0], color='k')
  plt.ylabel('Neuron 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(X[:, 0])))
  ax1.set_xticklabels([])

  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(X[:, 1], color='k')
  plt.xlabel('Sample Number')
  plt.ylabel('Neuron 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(X[:, 1])))

  ax3 = fig.add_subplot(gs[0:2, 1])
  ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
          markeredgewidth=0)
  ax3.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
            label='Basis vector 1')
  ax3.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
            label='Basis vector 2')
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))
  plt.show()
```

Next, run the code below to plot the eigenvalues of the eigenvectors for thsi dataset (sometimes called the "scree plot"). Which eigenvalue is larger?

```python
# @markdown Execute this cell to plot the eigenvalues for the new basis calculated by PCA on the data above
plot_eigenvalues(evals)
```

## Interactive Demo 2: Exploration of the correlation coefficient

How do the eigenvectors and eigenvalues change as the geometry of the original dataset changes? The correlation coefficient between the activity of two neurons is one parameter that can change the geometry of that original data.

Run the following cell and use the slider to change the correlation coefficient in the data. You should see the scree plot and the plot of basis vectors update.

1.  What happens to the eigenvalues as you change the correlation coefficient?
2.  Can you find a value for which both eigenvalues are equal?
3.  Can you find a value for which only one eigenvalue is nonzero?

```python
# @markdown Execute this cell to enable the widget

def plot_eigenvalues(corr_coef=0.8):
  variance_1 = 1
  variance_2 = 1

  cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
  X = get_data(cov_matrix)
  score, evectors, evals = pca(X)

  fig = plt.figure(figsize=[10, 8]);
  gs = fig.add_gridspec(1,3)
  # plot_basis_vectors(X, evectors)

  ax1 = fig.add_subplot(gs[0, 0:2])
  ax1.plot(X[:, 0], X[:, 1], '.', color=[.5, .5, .5])
  ax1.axis('equal')
  ax1.set_xlabel('Neuron 1 activity')
  ax1.set_ylabel('Neuron 2 activity')
  ax1.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
           label='Basis vector 1')
  ax1.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
           label='Basis vector 2')
  plt.axis('equal')
  ax1.legend()

  # plot_eigenvalues(evals)
  ax2 = fig.add_subplot(gs[0, 2])
  ax2.plot(np.arange(1, len(evals) + 1), evals, 'o-k')
  ax2.set_xlabel('Component')
  ax2.set_ylabel('Eigenvalue')
  ax2.set_title('Scree plot')
  ax2.set_xticks(np.arange(1, len(evals) + 1))
  ax2.set_ylim([0, 2.5])

  plt.show()

_ = widgets.interact(plot_eigenvalues, corr_coef=(-1, 1, .1))
```

---
# Section 3: Using PCA to reduce the dimensionality of data.

Now let's consider data that has a higher dimensionality. For example, the activity of 1000 neurons. We can't visualize that like we have been because on a computer screen we can only visualize two dimensions (or 3 with an interactive rotating plot).
We can perform PCA on the 1000-dimensional data just like we have been on 2-dimensional data. Instead of 2 eigenvectors, PCA would result in up to 1000.

But here is where PCA becomes particularly significant.
In the example scree plot below, most of the eigenvalues are near zero, with fewer than 100 having large values (only the first 100 components are shown).

Based on the distribution of eigenvalues, we can understand something about the ***intrinsic dimensionality*** of the data. To determine the data's intrinsic dimensionality we can consider the variance explained (eigenvalues) by the components of the PCA (the eigenvectors). This can be examined with a cumulative plot of the fraction of the total variance (eigenvalues) explained by the top $K$ components

 Click here if you are interested in detials about the math behind calculating the total variance explained
\begin{align}
\text{var explained} = \frac{\sum_{i=1}^K \lambda_i}{\sum_{i=1}^N \lambda_i}
\end{align}

where $\lambda_i$ is the $i^{th}$ eigenvalue and $N$ is the total number of components (the original number of dimensions in the data).

The intrinsic dimensionality is often quantified by the eigenvectors necessary to explain a large proportion of the total variance of the data (often a defined threshold, e.g., 90%).

Based on the cummulative explained variance, we see that we could almost fully describe the 1000-dimensional data with just 100 components (eigenvectors from PCA).

In the next couple of weeks, we will encounter specific examples of the utility of this concept in neuroscience.

---
# Summary

- In this tutorial, we learned that The goal of PCA is to find an orthonormal basis capturing the directions of maximum variance of the data. More precisely, the $i$th basis vector is the direction that maximizes the projected variance, while being orthogonal to all previous basis vectors. Mathematically, these basis vectors are the eigenvectors of the covariance matrix (also called *loadings*).
- PCA also has the useful property that the projected data (*scores*) are uncorrelated.
- The projected variance along each basis vector is given by its corresponding eigenvalue. This is important because it allows us rank the "importance" of each basis vector in terms of how much of the data variability it explains. An eigenvalue of zero means there is no variation along that direction so it can be dropped without losing any information about the original data.
- Importantly, neuroscientists use this property to reduce the dimensionality of high-dimensional data.

---
# Notation

\begin{align}
\mathbf{x_i}  &\quad \text{all measurements of neuron } i\\
\bar{\bf x_i}  &\quad \text{mean across samples for neuron } i \\
\bf \Sigma &\quad \text{covariance matrix}\\
\bf \hat \Sigma &\quad \text{sample covariance matrix}\\
\bf W &\quad \text{weights, loadings matrix}\\
{\bf X} &\quad \text{original data matrix}\\
\bf S &\quad \text{projected matrix, scores}\\
N &\quad \text{data dimensionality}\\
N_\text{samples} &\quad \text{number of samples}\\
\end{align}

---
This tutorial was written by Krista Perks for BIOL358 Motor Systems taught at Wesleyan University. Based on content from **Neuromatch Academy 2020: Week 1, Day 5: Dimensionality Reduction** by Alex Cayco Gajic, John Murray


# Basic Neural Network from Scratch

>## Objective:

To understand and build a basic one hidden layer Neutral Network from scratch

> ## Approach :

- **Getting the Dataset**
- **Logistic Regression**
- **Neural Network : Understanding**

       - Neural Network structure
       - Activation functions : Softmax and Tanh and their necessity
       - Weight factors and Bias
       - Forward Propagation
       - Back Propagation
       - Shape of Vectors
       - Differentiation

- **Implementation**

       - Model Function
       - Predict Function
       - Loss Function

- **Varying hidden layer nodes**
- **Conclusion**

**Importing Libraries**

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.linear_model
import matplotlib
%matplotlib inline
matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)
```

**`Plot_decision_boundary` function is defined to conveniently see the decision boundary**

```python
# Helper function to plot a decision boundary.
# If you don't fully understand this function don't worry, it just generates the contour plot below.
def plot_decision_boundary(pred_func):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
```

## `1.` Getting the Dataset

Dataset is generated using sklearn's built in methods

```python
# Generate a dataset and plot it
import sklearn
from sklearn import datasets
np.random.seed(0)
X, y = sklearn.datasets.make_moons(200, noise=0.20)
plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.Spectral)
```

The dataset generated has two classes, plotted as red and blue points. You can think of the blue dots as male patients and the red dots as female patients, with the x- and y- axis being medical measurements.

Your goal is to train a Machine Learning classifier that predicts the correct class (male or female) given the x- and y- coordinates. Note that the data is not linearly separable, a straight line cant be drawn that separates the two classes. This means that linear classifiers, such as Logistic Regression, won't be able to fit the data unless you hand-engineer non-linear features (such as polynomials) that work well for the given dataset.

In fact, that's one of the major advantages of Neural Networks. You don't need to worry about feature engineering. The hidden layer of a neural network will learn features for you.

## `2.` Logistic Regression:

To understand the problem better , try to implement a logistic regression first on the dataset and plot the decision boundary using the earlier defined function

**TASK : Build a `logistic regression` model and fit it on the dataset**

```python
### START CODE HERE (~ 2 Lines of code)
rf = sklearn.linear_model.LogisticRegressionCV()
rf.fit(X, y)
### END CODE
```

    C:\Users\kusht\Anaconda3\lib\site-packages\sklearn\model_selection\_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.
      warnings.warn(CV_WARNING, FutureWarning)

    LogisticRegressionCV(Cs=10, class_weight=None, cv='warn', dual=False,
                         fit_intercept=True, intercept_scaling=1.0, l1_ratios=None,
                         max_iter=100, multi_class='warn', n_jobs=None,
                         penalty='l2', random_state=None, refit=True, scoring=None,
                         solver='lbfgs', tol=0.0001, verbose=0)

**TASK : Plot the `Decision boundary` using the earlier defined function**

```python
# For the argument of lambda x: use the predict function of the earlier defined classifier that was fit on dataset

### START CODE HERE (~2 Lines of code)
#(Write code where '#' is given)

plot_decision_boundary(lambda x: rf.predict(x))
plt.title("Logistic Regression")

### END CODE
```

The graph shows the decision boundary learned by our Logistic Regression classifier. It separates the data as good as it can using a straight line, but it's unable to capture the "moon shape" of our data.

## `3.`  Neural Networks :

### `3.1` Neural Networks Understanding

Let's now build a 3-layer neural network with one input layer, one hidden layer, and one output layer. The number of nodes in the input layer is determined by the dimensionality of our data, 2. Similarly, the number of nodes in the output layer is determined by the number of classes we have, also 2. (Because we only have 2 classes we could actually get away with only one output node predicting 0 or 1, but having 2 makes it easier to extend the network to more classes later on). The input to the network will be x- and y- coordinates and its output will be two probabilities, one for class 0 ("female") and one for class 1 ("male"). It looks something like this:

You can choose the dimensionality (the number of nodes) of the hidden layer. The more nodes we put into the hidden layer the more complex functions we will be able fit. But higher dimensionality comes at a cost. First, more computation is required to make predictions and learn the network parameters. A bigger number of parameters also means that the model become more prone to overfitting the data.

How to choose the size of the hidden layer? While there are some general guidelines and recommendations, it always depends on your specific problem and is more of an art than a science. Its best to play with the number of nodes in the hidden layer and see how it affects the output.

You also need to pick an activation function for the hidden layer. The activation function transforms the inputs of the layer into its outputs.  Common chocies for activation functions are tanh, the sigmoid function, or ReLUs. Here you'll use tanh, which performs quite well in many scenarios. A nice property of these functions is that their derivate can be computed using the original function value. For example, the derivative of $\tanh x$ is $1-\tanh^2 x$. This is useful because it allows to compute $\tanh x$ once and re-use its value later on to get the derivative.

The network is desired to output probabilities thus the activation function for the output layer will be the softmax, which is simply a way to convert raw scores to probabilities. If you're familiar with the logistic function you can think of softmax as its generalization to multiple classes.

### `3.2` Activation Functions

**Softmax :**

Where the function is defined as :

**Tanh :**

**Why are Activation Functions necessary?**

If there were no activation function or just linear activation function then it would work same as a linear classifier/regressor and the models applications would be very limited and thus a nonlinear activation function is what really allows to fit nonlinear hypotheses.

**Understanding the network**

The input $x$ contains two attributes namely $x1$ and $x2$ and has $m$ number of training examples , in this dataset $m$ = 200.
Neural networks are similar to logistic regression just that the latter is used for binary classification/regression and the former is used for multiclass classification/regression. Think of it this way that In logistic regression , attributes or inputs are given to a box or a node which multiplies the inputs with weight factors , adds bias and outputs the resultant similar to linear regression which is then transformed by a non linear activtion function , sigmoid in logistic regression to give the final output. Similarly , In neural networks this whole process happens alot of times , in this notebook with just one hidden layer the whole process occurs as many times as the number of nodes in the hidden layer , so each node's input-output can be visualised as logistic regression and many such singluar logistic regression units give birth to neural networks which shows generally why neural networks perform better than its singular counterpart. This whole process can be explained better with the following image :

### `3.3` Weight factors and Bias

**Bias** : The term bias is used to adjust the final output matrix as the y-intercept does. For instance, in the classic equation, $y = mx + c$, if $c$ = 0, then the line will always pass through 0. Adding the bias term provides more flexibility and better generalisation to our Neural Network model.

**Weight Factors** : Weight factors gives out the best linear combinations of attributes which would give the closest ouput

Weight factors and Bias values are first randomly given and then adjusted through `Back Propaagtion` and `Gradient Descent` to find the optimum combination

### `3.4` Forward Propagation : How the network makes predictions

The network makes predictions using forward propagation, which is just a bunch of matrix multiplications and the application of the activation function(s) as defined above. If $x$ is the 2-dimensional input to the network then predictions $\hat{y}$ (also two-dimensional) can be calculated as follows:

$$
\begin{aligned}
z_1 = xW_1 + b_1 \\
a_1 = \tanh(z_1) \\
z_2 = a_1W_2 + b_2 \\
a_2 = \hat{y} = \mathrm{softmax}(z_2)
\end{aligned}
$$

$z_i$ is the weighted sum of inputs of layer $i$ (bias included) so $z_1$ contains all the outputs from all the nodes in the first hidden layer so  $z_1[1]$ will be the output of first node of hidden layer , $z_1[2]$ the output of second node of hidden layer and so on. $a_i$ is the output of layer $i$ after applying the activation function, it forms a vector similar to $z_i$ and the values are tranformed from $z_i$ using an activation function.\
$W_1, b_1, W_2, b_2$ are parameters of the network, which are needed to learn from the training data. You can think of them as matrices transforming data between layers of the network. Looking at the matrix multiplications above you can figure out the dimensionality of these matrices. If 500 nodes are used for the hidden layer then $W_1 \in \mathbb{R}^{2\times500}$, $b_1 \in \mathbb{R}^{500}$, $W_2 \in \mathbb{R}^{500\times2}$, $b_2 \in \mathbb{R}^{2}$. Now you see why you'll have more parameters if we increase the size of the hidden layer.

### `3.5` Shapes of all the Vectors

Understand that all the variables like weight factors, bias , input x etc are all vectors. Vectorization is done to ease out processes like element wise multiplication and then its summation which would generally require a for loop but with a vectorization a simple dot product would suffice thus increasing the speed and also conveneient and short to code.
With neural networks , understanding the shapes of all the parameters is very important so lets look at the shapes :

Here , number of training examples are $m$=200 and **n_hid** is the number of nodes in hidden layer
1. $X$ has a shape of `(m,2)` : 2 is for the number of attributes i.e $x_1$ and $x_2$
2. $W_1$ has a shape of `(2,n_hid)` connecting each node of input layer and hidden layer and giving them weights : 2 is for the number of attributes i.e $x_1$ and $x_2$
3. $b_1$ has a shape of `(1,n_hid)` as there is one bias for each node in the hidden layer
4. $W_2$ has a shape of `(n_hid,2)` connecting each node from hidden layer to the output layer of 2 : 2 is for the output layer giving probabilities of `male` and `female`
5. $b_2$ has a shape of `(1,2)` as there is one bias for each node in the output layer thus 2 for 2 nodes (`male` and `female` probabilities)
6. $z_1$ has a shape of `(1,n_hid)` as it gives one output for each node and it does that $m$ such instances
7. $a_1$ has a shape of `(1,n_hid)` same as $z_1$ as only values inside are changed and not the shape
8. $ \hat{y}$ has a shape of `(1,2)` as its the final output and it has 2 probability values for each of **m** instances

### `3.6` Back Propagation : Learning the Parameters

Learning the parameters for the network means finding parameters ($W_1, b_1, W_2, b_2$) that minimize the error on the training data. But how to define the error? Especially for that a function that measures our error the loss function is called. Negative log neighbourhood would be used here . If you have $N$ training examples and $C$ classes then the loss for our prediction $\hat{y}$ with respect to the true labels $y$ is given by:

$$
\begin{aligned}
H(y,p) = - \sum_i y_i log(p_i)
\end{aligned}
$$
The formula looks complicated, but all it really does is sum over our training examples and the $p_{i}$ are the probabilities of the correct classes. So, the further away $y$ (the correct labels) and $\hat{y}$ (our predictions) are, the greater the loss will be. **Understand that if $p_{i}$ is the probability of the correct classes then $\hat{y}$ will always be equal to one. So this would be just summation of negative log of probabilities of the correct classes.**

Remember that the goal is to find the parameters that minimize the loss function. You can use gradient descent to find its minimum. Here the most basic version of Gradient Descent, also called batch gradient descent with a fixed learning rate is implemented.  Variations such as SGD (stochastic gradient descent) or minibatch gradient descent typically perform better in practice. So if you are serious you'll want to use one of these, and ideally you would also decay the learning rate over time but for now you can use this.

As an input, gradient descent needs the gradients (vector of derivatives) of the loss function with respect to the parameters: $\frac{\partial{L}}{\partial{W_1}}$, $\frac{\partial{L}}{\partial{b_1}}$, $\frac{\partial{L}}{\partial{W_2}}$, $\frac{\partial{L}}{\partial{b_2}}$. To calculate these gradients following formulas can be used,

Applying the backpropagation formula ,the following results are obtained :

$$
\begin{aligned}
\delta_3 = \hat{y} - y \\
\delta_2 = (1 - \tanh^2z_1) \circ \delta_3W_2^T \\
\frac{\partial{L}}{\partial{W_2}} = a_1^T \delta_3  \\
\frac{\partial{L}}{\partial{b_2}} = \delta_3\\
\frac{\partial{L}}{\partial{W_1}} = x^T \delta_2\\
\frac{\partial{L}}{\partial{b_1}} = \delta_2 \\
\end{aligned}
$$

$T$ signifies transpose of that matrix/vector.
**NOTE : in $\delta_3$ , $y$ would always be equal to 1 if $\hat{y}$ are the probabilities of the correct classes. So in the code its better to find the probabilities of the right classes and then subtract it by 1**
A More in-depth and better understanding can be given [here](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/ )

### `3.7` Differentiations :

The Softmax differentiation with the cross entropy loss can be given as :

$$
\begin{align}
L &= - \sum_i y_i log(p_i) \\
\frac{\partial L}{\partial o_i} &= - \sum_k y_k \frac{\partial log(p_k)}{\partial o_i } \\
&= - \sum_k y_k \frac{\partial log(p_k)}{\partial p_k} \times \frac{\partial p_k}{ \partial o_i} \\
&= - \sum y_k \frac{1}{p_k} \times \frac{\partial p_k}{\partial o_i} \\
\end{align}
$$

With the softmax derivative :

$$
\begin{align}
\frac{\partial L}{\partial o_i}  &= -y_i(1-p_i) - \sum_{k\neq i} y_k \frac{1}{p_k}(-p_k.p_i) \\
&= -y_i(1-p_i) + \sum_{k \neq 1} y_k.p_i \\
&= - y_i + y_ip_i + \sum_{k \neq 1} y_k.p_i \\
&= p_i\left( y_i +  \sum_{k \neq 1} y_k\right) - y_i \\
&= p_i\left( y_i +  \sum_{k \neq 1} y_k\right)  - y_i
\end{align}
$$

$y$ is a one hot encoded vector for the labels, so $\sum_k y_k = 1$ and $y_i +  \sum_{k \neq 1} y_k = 1$ , Thus simplifying it into :\

$\frac{\partial L}{\partial o_i} = p_i - y_i$

A much more intensive and indepth explanation can be provided [here](https://deepnotes.io/softmax-crossentropy) and [here](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)

**Forward and backward propagation can be understood with the help of following schematics**

**For single node :**

**For multiple :**

**These derivatives are calculated , multiplied by learning rate and then added to the weights so as these gradients tend to zero that is error is minimum, the final optimum weights are obtained**

## `4.` Implementation :

Now , that a proper understanding of neural networks has been obtained , its time to make a model implementing it

**Pre defining constant parameters**

```python
num_examples = len(X) # training set size
nn_input_dim = 2 # input layer dimensionality
nn_output_dim = 2 # output layer dimensionality

# Gradient descent parameters (I picked these by hand)
epsilon = 0.01 # learning rate for gradient descent
reg_lambda = 0.01 # regularization strength

```

### `4.1` Model function

**TASK : Build a function to implement the Neural Network**

```python
# This function learns parameters for the neural network and returns the model.
# - nn_hdim: Number of nodes in the hidden layer
# - num_passes: Number of passes through the training data for gradient descent
# - print_loss: If True, print the loss every 1000 iterations

# One completion of forward and backward pass happens in one pass and you'll be using 20000 such passes

### START CODE HERE : (Write code where '#' is given)

def build_model(nn_hdim, num_passes=20000, print_loss=False):

    # Initialize the parameters to random values.
    # With succesive gradient descents , the model will learn and will keep updating the parameters
    np.random.seed(0)
    W1 = np.random.randn(nn_input_dim, nn_hdim) / np.sqrt(nn_input_dim)
    b1 = np.zeros((1, nn_hdim))

    # Seeing the earlier definition , define 'W2' and 'b2' in the same way
    # Take help of the earlier section where shapes are defined
    W2 = np.random.randn(nn_hdim, nn_output_dim) / np.sqrt(nn_hdim)
    # Understand that while weight factors cant be randomly assigned to zeros , however biases can.
    b2 = np.zeros((1, nn_output_dim))

    # This is the model to return at the end
    model = {}

    # Gradient descent. For each batch...
    for i in range(0, num_passes):

        ## FORWARD PROPAGATION
        z1 = X.dot(W1) + b1
        a1 = np.tanh(z1)

        ## Seeing the earlier definition , define 'z2' and find 'probs' using softmax activation instead of tanh
        ## Take help of the section where forward propagation formulas are defined
        z2 = a1.dot(W2) + b2
        exp_scores = np.exp(z2)
        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

        ## BACK PROPAGATION
        delta3 = probs
        delta3[range(num_examples), y] -= 1
        dW2 = (a1.T).dot(delta3)
        db2 = np.sum(delta3, axis=0, keepdims=True)
        #keepdims =True keeps the earlier vectors dimensions which is very important

        ## Seeing the earlier definitions , define 'delta2' ,'dW1' ,'db1'
        ## Take help of the backward propagation formulas defined in the earlier sections

        delta2 = delta3.dot(W2.T) * (1 - np.power(a1, 2))
        dW1 = np.dot(X.T, delta2)
        db1 = np.sum(delta2, axis=0)

        # Regularisation terms are added (b1 and b2 don't have regularization terms)
        dW2 += reg_lambda * W2
        dW1 += reg_lambda * W1

        # Gradient descent parameter update
        W1 += -epsilon * dW1
        b1 += -epsilon * db1

        ## Seeing the earlier definitions , define 'W2' , 'b2'
        ## Understand how parameters are updated , epsilon is the contant learning rate
        W2 += -epsilon * dW2
        b2 += -epsilon * db2

        # Assigning new parameters to the model
        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

        # Optionally print the loss.
        # This is expensive because it uses the whole dataset, so don't want to do it too often.
        if print_loss and i % 1000 == 0:
             print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model
```

Now , The neural network is made which takes in input `nn_hdim` which is the number of nodes in hidden layer and returns the model keeping in mind the input and output parameters. The approach of this was to first randomly assign the parameters and with gradient descent , update it until it reaches the minimum.

Once model is made , its essential to make a `predict` function to predict the models outputs and then a `calculate_loss` function that calculates the loss between $\hat{y}$ and $y$

### `4.2` Predict function :

**TASK : Make a function `Predict` that takes in model and X as arguments and returns the maximum probabilities**

```python
# Helper function to predict an output (0 or 1)
def predict(model, x):
    W1, b1, W2, b2 = model['W1'], model['b1'], model['W2'], model['b2']

### START CODE HERE ()

     # FORWARD PROPAGATION
    ## Write the full steps of forward propagration as before using the updated parameters
    z1 = x.dot(W1) + b1
    a1 = np.tanh(z1)
    z2 = a1.dot(W2) + b2
    exp_scores = np.exp(z2)
    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

### END CODE
    return np.argmax(probs, axis=1)
# The maximum probabilites are returned
```

A good schematic to understand why the function returns the maximum probability is given below :

Now , that the output probabilites are calculated , The last step remaining is to calculate the loss between $\hat{y}$ and $y$ using `cross-entropy` function

**Remember it can also be written as summation of negative log of the correct class probabilities which will be implemented in the code**

### `4.3` Loss function:

**TASK : Make function to calculate loss between $\hat{y}$ and $y$**

```python
# Helper function to evaluate the total loss on the dataset
def calculate_loss(model):

    ### START CODE HERE

    ## Take parameters 'W1','b1','W2','b2' from 'model'
    ## Take help from the previously defined functions to understand how it can be done

    W1, b1, W2, b2 = model['W1'], model['b1'], model['W2'], model['b2']

    # Forward propagation to calculate our predictions
    ## Note : you can also use previously defined function predict instead

    # Write code to get all the parameters using forward propagation formulas
    z1 = X.dot(W1) + b1
    a1 = np.tanh(z1)
    z2 = a1.dot(W2) + b2
    exp_scores = np.exp(z2)
    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

    ## Calculating the loss
    corect_logprobs = -np.log(probs[range(num_examples), y])
    data_loss = np.sum(corect_logprobs)

    # Adding regulatization term to loss (optional)

    data_loss += reg_lambda/2 * (np.sum(np.square(W1)) + np.sum(np.square(W2)))
    return 1./num_examples * data_loss

### END CODE
```

**Now that all the helper functions and model functions are made its time to implement it**

**TASK : Using `build_model` function, take `nn_hdim=3` and `print_loss=True` and get the model in a variable called `model`**

```python
# Build a model with a 3-dimensional hidden layer
### START CODE HERE (~ 1 Line of code)
model = build_model(3, print_loss=True)
### END CODE HERE

```

    Loss after iteration 0: 0.432387
    Loss after iteration 1000: 0.068947
    Loss after iteration 2000: 0.068890
    Loss after iteration 3000: 0.071218
    Loss after iteration 4000: 0.071253
    Loss after iteration 5000: 0.071278
    Loss after iteration 6000: 0.071293
    Loss after iteration 7000: 0.071303
    Loss after iteration 8000: 0.071308
    Loss after iteration 9000: 0.071312
    Loss after iteration 10000: 0.071314
    Loss after iteration 11000: 0.071315
    Loss after iteration 12000: 0.071315
    Loss after iteration 13000: 0.071316
    Loss after iteration 14000: 0.071316
    Loss after iteration 15000: 0.071316
    Loss after iteration 16000: 0.071316
    Loss after iteration 17000: 0.071316
    Loss after iteration 18000: 0.071316
    Loss after iteration 19000: 0.071316

**TASK : Plot the decision boundary using `plot_decision_boundary` function**

```python
# Plot the decision boundary
plot_decision_boundary(lambda x: predict(model, x))
plt.title("Decision Boundary for hidden layer size 3")
```

## `5.` Varying the hidden layer size

In the example above you made a hidden layer size of 3. Let's now get a sense of how varying the hidden layer size affects the result.

**TASK : Using a `for_loop` and the functions build above. make models and plot decision boundaries of varying hidden layer sizes**

```python
plt.figure(figsize=(16, 32))
hidden_layer_dimensions = [1, 2, 3, 4, 5, 20, 50]
for i, nn_hdim in enumerate(hidden_layer_dimensions):
    plt.subplot(5, 2, i+1)
    plt.title('Hidden Layer size %d' % nn_hdim)
    model = build_model(nn_hdim)
    plot_decision_boundary(lambda x: predict(model, x))
plt.show()
```

## `6.` Conclusion

After trying out different activation functions with different sizes of hidden layer ,get an idea of why alot of nodes cant be used in a single hidden layer and which activation functions work best . Based on this , make the adjustments and choose the best final model

Basic Neural Network from Scratch/Basic Neural Network from Scratch (Solution) .ipynb


# Anomalous diffusion
Diffusion processes are mostly used to describe the random motion of particles in fluids, but also apply to biological systems, like animals that search for food or cancer cells migrating through tissue. In an experimental setup, we may simply track the random motions of a particle/animal/cell over time and thereby use it as a probe to measure the diffusion coefficient of the fluid/habitat/tissue. However, by averaging over all measured movements, one may loose important information, as the diffusion coefficient may change over time or depending on the current position of the probe. Changes in the diffusion coefficient may be externally triggered, e.g. by a change in temperature for diffusing particles or a change in terrain within the foraging habitat of an animal. Furthermore, changes in diffusivity may be internally triggered, e.g. by the cell cycle of an invading cancer cell. Here, we present two examples of anomalous diffusion, one with a gradually changing diffusivity, and one with discrete regions of varying diffusivity.

## Diffusion gradient
Suppose we have a diffusion gradient, for example due to a temperature gradient across our sample, and we want to determine the diffusion coefficient for different positions along the axis of the gradient. To infer the spatially changing diffusion coefficient, we assume that the amplitude of the random walk of our particle changes gradually over time, as the particle moves through regions of varying diffusivity.

For this example, we assume a positive, constant diffusion gradient, resulting a linear increase of the diffusion coefficient $D$:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

D = np.linspace(0.0, 15., 500)
x = np.arange(500)

plt.figure(figsize=(8,2))
plt.fill_between(x, D, 0)
plt.xlabel('x position [a.u.]')
plt.ylabel('D [a.u.]');
```

### Trajectory simulation
For this example, we simply simulate a random walk of a particle assuming a Gaussian random walk with the standard deviation equal to the diffusion coefficient at the current position:

```python
np.random.seed(12337) # reproducable results
trajectory = [[150., 150.]] # starting point

for t in range(750):
    try:
        dc = D[int(trajectory[-1][0])] # look up diffusion coefficient
    except:
        if int(trajectory[-1][0]) >= 500:
            dc = 15. # constant diffusion coefficient on the far right
        else:
            dc = 0. # constant diffusion coefficient on the far left
    trajectory.append([p + np.random.normal(0, dc) for p in trajectory[-1]])

trajectory = np.array(trajectory)
```

If we plot this simulated trajectory, we see how the distance between two subsequent positions of the particle is significantly smaller on the left as compared to the right of our diffusion gradient (visible as gray shading):

```python
plt.figure(figsize=(8, 10))

D_img = np.array([list(D)]*len(D))[:, :250]
plt.imshow(D_img, cmap='Greys', alpha=0.9, zorder=0, interpolation='nearest')

plt.scatter(*trajectory.T, lw=0.2, c='b', alpha=0.7, zorder=2, s=50)
plt.plot(*trajectory.T, lw=1.5, c='k', zorder=1)
plt.xlim([0, 250])
plt.ylim([80, 250])
plt.xlabel('x position [a.u.]')
plt.ylabel('y position [a.u.]');
```

### Modeling
To model this example of anomalous diffusion, we first generate a low-level model - a simple Gaussian random walk - using `SymPy` random variables. Here, we simply assume that the velocity components of our particle are normally distributed, with zero mean. As we further assume that the diffusion coefficient changes gradually across our sample, our particle should also gradually change its diffusivity as it moves across the sample. We therefore also use a Gaussian random walk as the high-level model. As we do not know the amplitude of the change of diffusivity, we choose a `HyperStudy` as it enables the inference of this hyper-parameter.

```python
import bayesloop as bl
import sympy.stats
from sympy import Symbol

S = bl.HyperStudy()

# load data
velocity = trajectory[1:] - trajectory[:-1] # compute velocity vectors from positions
S.load(velocity)

# create low-level model
std = Symbol('D', positive=True)
normal = sympy.stats.Normal('normal', 0, std)

# we assume the diffusivity to lie within the interval ]0, 15[
# within this interval, we create 5000 equally spaced parameter values
L = bl.om.SymPy(normal, 'D', bl.oint(0, 15, 5000))

# create high-level model
# we assume a Gaussian random walk of the parameter 'D' with a
# standard deviation within the interval [0, 0.3] (20 equally
# spaced hyper-parameter values)
T = bl.tm.GaussianRandomWalk('sigma', bl.cint(0, 0.3, 20), target='D')

S.set(L, T)
S.fit()
```

    + Created new study.
      --> Hyper-study
    + Successfully imported array.
        + Trying to determine Jeffreys prior. This might take a moment...
        + Successfully determined Jeffreys prior: sqrt(2)/D. Will use corresponding lambda function.
    + Observation model: normal. Parameter(s): ('D',)
    + Transition model: Gaussian random walk. Hyper-Parameter(s): ['sigma']
    + Set hyper-prior(s): ['uniform']
    + Started new fit.
        + 20 analyses to run.

    Widget Javascript not detected.  It may not be installed or enabled properly.

    C:\Anaconda3\lib\site-packages\scipy\ndimage\filters.py:139: RuntimeWarning: divide by zero encountered in double_scalars
      p = numpy.polynomial.Polynomial([0, 0, -0.5 / (sigma * sigma)])
    C:\Anaconda3\lib\site-packages\numpy\polynomial\polynomial.py:780: RuntimeWarning: invalid value encountered in multiply
      c0 = c[-i] + c0*x

        ! WARNING: Forward pass distribution contains only zeros, check parameter boundaries!
          Stopping inference process. Setting model evidence to zero.

        + Computed average posterior sequence
        + Computed hyper-parameter distribution
        + Log10-evidence of average model: -1583.15297
        + Computed local evidence of average model
        + Computed mean parameter values.
    + Finished fit.

### Analysis

After fitting is done, we may evaluate how the diffusion coefficient of our particle changed over time by plotting the mean values of the posterior distributions (black line) together with the posterior distributions of the amplitude of the particle motions (blue shading):

```python
plt.figure(figsize=(8,2))
S.plot('D')
plt.xlim([0, 750])
plt.ylim([0, 8]);
```

As we can see, the diffusion coefficient changes quite irregularly over time, but this irregularity is just the result of the irregular path of our particle across the sample. However, this plots confirms that the chosen parameter boundaries of $\sigma \in [0, 20]$ suffice. We may re-plot the particle path from above, but this time color-code the particle positions with the corresponding inferred diffusivity values:

```python
D_mean = S.getParameterMeanValues('D') # extract posterior mean values of first (and only) parameter

# prepare color coding
m = plt.cm.ScalarMappable(cmap='RdYlBu')
m.set_array(D_mean)
m.autoscale()

plt.figure(figsize=(8, 10))
plt.imshow(D_img, cmap='Greys', alpha=0.9, zorder=0, interpolation='nearest')
plt.scatter(*trajectory[1:].T, lw=0.2, c=[m.to_rgba(x) for x in D_mean], alpha=0.9, zorder=2, s=50)
plt.plot(*trajectory[1:].T, lw=1.5, c='k', zorder=1)
plt.xlim([0, 250])
plt.ylim([80, 250])
plt.xlabel('x position [a.u.]')
plt.ylabel('y position [a.u.]');
```

Finally, we can directly compare the inferred mean values to the true values of the diffusion coefficient used in the simulation:

```python
D = np.linspace(0.0, 15., 500)
x = np.arange(500)

plt.figure(figsize=(8,2))
plt.fill_between(x, D, 0)
plt.scatter(trajectory[1:, 0], D_mean, c='r', alpha=0.6)

plt.xlim([0, 250])
plt.ylim([0, 8])
plt.xlabel('x position [a.u.]')
plt.ylabel('D [a.u.]');
```

To assess our choice for the hyper-parameter boundaries, we may further plot the inferred distribution of the magnitude of parameter changes between time steps. In this case, the distribution falls off to (approximately) zero on both sides, therefore the boundaries suffice.

```python
plt.figure(figsize=(8,3))
S.plot('sigma', alpha=0.7, facecolor='g')
plt.xlim([0, 0.3]);
```

## Regime-switch diffusion process

While the first example investigated gradual variations, we now turn to abrupt jumps in diffusivity, as they may occur for proteins diffusing on a cell membrane. The binding and unbinding of the protein to other constituents of the membrane may inhibit or favor diffusion. In a very simplistic attempt to model this complex process, we assume only two different regimes of diffusivity, and arrange them spatially like a chessboard pattern. On the dark patches, the diffusivity reaches three times the value as on the light patches:

```python
import matplotlib.patches as mpatches

# helper function for chessboard pattern
def check_region(x, y):
    if (int(x) % 2 == 0 and int(y) % 2 == 0) or (int(x) % 2 == 1 and int(y) % 2 == 1):
        return True
    else:
        return False

# create chessboard pattern
n = 5
D_img = np.array([0, 1]*int(np.ceil(0.5*(n**2))))[:-1].reshape([n, n])

# plot chessboard pattern
plt.figure(figsize=(8,8))
plt.imshow(D_img, cmap='Greys', alpha=0.5, extent=[0, n, 0, n], zorder=0, interpolation='nearest')
plt.grid(b=False, which='major')
plt.xlabel('x position [a.u.]')
plt.ylabel('y position [a.u.]')

# legend
white_patch = mpatches.Patch(color='white', label='D = 0.03 a.u.')
gray_patch = mpatches.Patch(color='0.5', label='D = 0.09 a.u.')
legend = plt.legend(handles=[white_patch, gray_patch],
                    bbox_to_anchor=(0., 1.02, 1., .102),
                    loc=3,
                    borderaxespad=0.,
                    frameon = 1)
frame = legend.get_frame()
frame.set_facecolor('0.85')
```

### Trajectory simulation
Again, we simulate a random walk of a particle assuming a Gaussian random walk with the standard deviation equal to the diffusion coefficient at the current position:

```python
np.random.seed(1274)

D1 = 0.03
D2 = 0.09

trajectory = [[2.5, 2.5]]

for t in range(2000):
    dc = D1 if check_region(*trajectory[-1]) else D2
    trajectory.append([x + np.random.normal(0, dc) for x in trajectory[-1]])

trajectory = np.array(trajectory)

# plotting
plt.figure(figsize=(8,8))
plt.imshow(D_img, cmap='Greys', alpha=0.5, extent=[0, n, 0, n], zorder=0, interpolation='nearest')
plt.grid(b=False, which='major')
plt.xlabel('x position [a.u.]')
plt.ylabel('y position [a.u.]')

plt.scatter(*trajectory.T, lw=0.2, c='b', alpha=0.7, zorder=2, s=50)
plt.plot(*trajectory.T, lw=1.5, c='k', zorder=1)

plt.xlim([1, 5])
plt.ylim([0, 4]);
```

### Modeling
The low-level model stays the same as in the previous example, a Gaussian random walk. As the parameter changes are not gradual in this example, however, we assume a regime-switching process for the high-level model. Here, we assign a small probability to all possible parameter values in each time step to account for the rare, but possible event of a abrupt parameter change. As we do not know this minimal probability value a-priori, we again use a hyper-study to test different values. Note that we may keep our study-instance defined as above and just load new data, adjust the parameter boundaries and alter the high-level model:

```python
# load new data
velocity = trajectory[1:] - trajectory[:-1]
S.load(velocity)

# change parameter boudaries of low-level model
L = bl.om.SymPy(normal, 'D', bl.oint(0, 0.2, 2000))

# create high-level model
T = bl.tm.RegimeSwitch('log10pMin', bl.cint(-10, 3, 20))

S.set(L, T)
S.fit()
```

    + Successfully imported array.
        + Trying to determine Jeffreys prior. This might take a moment...
        + Successfully determined Jeffreys prior: sqrt(2)/D. Will use corresponding lambda function.
    + Observation model: normal. Parameter(s): ('D',)
    + Transition model: Regime-switching model. Hyper-Parameter(s): ['log10pMin']
    + Set hyper-prior(s): ['uniform']
    + Started new fit.
        + 20 analyses to run.

    Widget Javascript not detected.  It may not be installed or enabled properly.

        + Computed average posterior sequence
        + Computed hyper-parameter distribution
        + Log10-evidence of average model: 3236.20040
        + Computed local evidence of average model
        + Computed mean parameter values.
    + Finished fit.

### Analysis

After fitting is done, we may check whether we are indeed able to detect the abrupt parameter changes over time:

```python
plt.figure(figsize=(8,2))
S.plot('D')
plt.xlim([0, 2000])
plt.ylim([0, .2]);
```

The irregular spacing of the parameter jumps in time gets more regular once the parameter evolution is plotted on the chessboard pattern:

```python
D_mean = S.getParameterMeanValues('D')

# prepare color coding
m = plt.cm.ScalarMappable(cmap='RdYlBu')
m.set_array([0.03, 0.09])
m.autoscale()

# plotting
plt.figure(figsize=(8,8))
plt.imshow(D_img, cmap='Greys', alpha=0.5, extent=[0, n, 0, n], zorder=0, interpolation='nearest')
plt.grid(b=False, which='major')
plt.xlabel('x position [a.u.]')
plt.ylabel('y position [a.u.]')

plt.scatter(*trajectory[1:].T, lw=0.2, c=[m.to_rgba(x) for x in D_mean], alpha=0.9, zorder=2, s=50)
plt.plot(*trajectory[1:].T, lw=1.5, c='k', zorder=1)

cb = plt.colorbar(m, shrink=0.75)
cb.set_label('diffusivity [a.u.]')
cb.set_clim(0.03, 0.09)

plt.xlim([1, 5])
plt.ylim([0, 4]);
```

The figure above shows that we can nicely reconstruct the spatial pattern in diffusivity for the region covered by the diffusing particle. Finally, we may plot a histogram to confirm that the inferred values for the diffusion coefficient (green histogram) align with the true values used in the simulation (red lines):

```python
plt.figure(figsize=(8,3))
plt.hist(D_mean, 20, alpha=0.7, facecolor='g', normed=True)
plt.axvline(0.03, 0, 1600, lw=1, c='r')
plt.axvline(0.09, 0, 1600, lw=1, c='r')
plt.xlabel('diffusivity [a.u.]')
plt.ylabel('probability density [a.u.]');
```

### Model selection
In the example above, we a-priori assumed abrupt changes of the diffusion coefficient, because we knew the parameter dynamics from the simulation of our trajectory. In an experimental setup, however, one hardly knows the exact nature of the parameter dynamics before-hand. The big advantage of *bayesloop* is to compute the model evidence, i.e. the probability (density) of observing the data given the model one assumes. For a simple demonstration, we again choose the high-level model of the first example and assume gradual parameter fluctuations (we only adjust the scale for this example) and re-run the fit:

```python
T = bl.tm.GaussianRandomWalk('sigma', bl.cint(0, 0.01, 20), target='D')
S.set(T)

S.fit()
```

    + Transition model: Gaussian random walk. Hyper-Parameter(s): ['sigma']
    + Set hyper-prior(s): ['uniform']
    + Started new fit.
        + 20 analyses to run.

    Widget Javascript not detected.  It may not be installed or enabled properly.

    C:\Anaconda3\lib\site-packages\scipy\ndimage\filters.py:139: RuntimeWarning: divide by zero encountered in double_scalars
      p = numpy.polynomial.Polynomial([0, 0, -0.5 / (sigma * sigma)])
    C:\Anaconda3\lib\site-packages\numpy\polynomial\polynomial.py:780: RuntimeWarning: invalid value encountered in multiply
      c0 = c[-i] + c0*x

        ! WARNING: Forward pass distribution contains only zeros, check parameter boundaries!
          Stopping inference process. Setting model evidence to zero.

        + Computed average posterior sequence
        + Computed hyper-parameter distribution
        + Log10-evidence of average model: 3206.17483
        + Computed local evidence of average model
        + Computed mean parameter values.
    + Finished fit.

Here, we obtain a model evidence value of **3206** on a log$_{10}$ scale (these probability density values can get absurdly small or large if many data points are involved), compared to **3236** for the correct, regime-switching model. This means that it is **10**$^{\textbf{30}}$**-times more likely** that the 2000 data points of our trajectory are created by a regime-switching process, compared to gradual variations of the diffusion coefficient!


Simple SWM c) habr.ru

\begin{equation}
b = \mathbb{E}_{k,\xi_k \neq 0}[y_k - \displaystyle\sum_{i}\lambda_i y_i K(x_i;x_k)](1)
\end{equation}
\begin{equation}
class(x) = sign(f(x))
\end{equation}
\begin{equation}
f(x) = \sum_{i} \lambda_i y_i K(x_i;x) + b (2)
\end{equation}
\begin{equation}
K = \begin{pmatrix}
    y_1 y_1 K(x_1;x_1) & y_1 y_2 K(x_1;x_2) & \dots & y_1 y_N K(x_1;x_N)\\
    y_2 y_1 K(x_2;x_1) & y_2 y_2 K(x_2;x_2) & \dots & y_2 y_N K(x_2;x_N)\\
    \dots & \dots & \dots & \dots \\
    y_N y_1 K(x_N;x_1) & y_N y_2 K(x_N;x_2) & \dots & y_N y_N K(x_N; x_N)
    \end{pmatrix} (3)
\end{equation}

\begin{equation}
\max\limits_{\lambda} \underbrace{\displaystyle\sum_{i=1}^{n}\lambda_i - \frac{1}{2}\lambda^TK\lambda}_{\Lambda}
\end{equation}
\begin{equation}
\Lambda = \lambda_\color{red}M + \lambda_\color{blue}L - \displaystyle\sum_{j}\lambda_\color{red}M \lambda_j K_\color{red}M,_j
    - \displaystyle\sum_{i}\lambda_\color{blue}L \lambda_i K_\color{blue}L,_i + const +
    \frac{1}{2}\lambda^2_\color{red}M K_\color{red}M,_\color{red}M + \lambda_\color{red}M\lambda_\color{blue}L K_\color{red}M,_\color{blue}L +
    \frac{1}{2}\lambda^2_\color{blue}L K_\color{blue}L,_\color{blue}L=
\end{equation}
\begin{equation}
=\lambda_\color{red}M(1-\displaystyle\sum_{j}\lambda_j K_\color{red}M,_j)+\lambda_\color{blue}L(1-\displaystyle\sum_{i}\lambda_i K_\color{blue}L,_i)
    +\frac{1}{2}(\lambda^2_\color{red}M K_\color{red}M,_\color{red}M + 2\lambda_\color{red}M\lambda_\color{blue}L K_\color{red}M,_\color{blue}L
                + \lambda^2_\color{blue}L K_\color{blue}L,_\color{blue}L) + const =
\end{equation}
\begin{equation}
=k^T_0 v_0 + \frac{1}{2}v^2_0 Q v_0 + const (4)
\end{equation}
\begin{equation}
v_0 = (\lambda_\color{red}M, \lambda_\color{blue}L)^T (4a)
\end{equation}
\begin{equation}
k_0 = (1 - \lambda^T K_\color{red}M, 1 - \lambda^T K_\color{blue}L)^T (4b)
\end{equation}
\begin{equation}
Q = \begin{pmatrix}
    K_\color{red}M,_\color{red}M & K_\color{red}M,_\color{blue}L \\
    K_\color{blue}L,_\color{red}M & K_\color{blue}L,_\color{blue}L
    \end{pmatrix}
    (4c)
\end{equation}
\begin{equation}
u = (-y_\color{blue}L, y_\color{red}M)^T (4d)
\end{equation}
\begin{equation}
t_n = \frac{k^T_0 u}{u^T Q u} (5)
\end{equation}

```latex
%%latex
\begin{equation}
b = \mathbb{E}_{k,\xi_k \neq 0}[y_k - \displaystyle\sum_{i}\lambda_i y_i K(x_i;x_k)](1)
\end{equation}
\begin{equation}
class(x) = sign(f(x))
f(x) = \sum_{i} \lambda_i y_i K(x_i;x) + b (2)
\end{equation}
\begin{equation}
K = \begin{pmatrix}
    y_1 y_1 K(x_1;x_1) & y_1 y_2 K(x_1;x_2) & \dots & y_1 y_N K(x_1;x_N)\\
    y_2 y_1 K(x_2;x_1) & y_2 y_2 K(x_2;x_2) & \dots & y_2 y_N K(x_2;x_N)\\
    \dots & \dots & \dots & \dots \\
    y_N y_1 K(x_N;x_1) & y_N y_2 K(x_N;x_2) & \dots & y_N y_N K(x_N; x_N)
    \end{pmatrix} (3)
\end{equation}

\begin{equation}
\max\limits_{\lambda} \underbrace{\displaystyle\sum_{i=1}^{n}\lambda_i - \frac{1}{2}\lambda^TK\lambda}_{\Lambda}
\end{equation}
\begin{equation}
\Lambda = \lambda_\color{red}M + \lambda_\color{blue}L - \displaystyle\sum_{j}\lambda_\color{red}M \lambda_j K_\color{red}M,_j
    - \displaystyle\sum_{i}\lambda_\color{blue}L \lambda_i K_\color{blue}L,_i + const +
    \frac{1}{2}\lambda^2_\color{red}M K_\color{red}M,_\color{red}M + \lambda_\color{red}M\lambda_\color{blue}L K_\color{red}M,_\color{blue}L +
    \frac{1}{2}\lambda^2_\color{blue}L K_\color{blue}L,_\color{blue}L=
\end{equation}
\begin{equation}
=\lambda_\color{red}M(1-\displaystyle\sum_{j}\lambda_j K_\color{red}M,_j)+\lambda_\color{blue}L(1-\displaystyle\sum_{i}\lambda_i K_\color{blue}L,_i)
    +\frac{1}{2}(\lambda^2_\color{red}M K_\color{red}M,_\color{red}M + 2\lambda_\color{red}M\lambda_\color{blue}L K_\color{red}M,_\color{blue}L
                + \lambda^2_\color{blue}L K_\color{blue}L,_\color{blue}L) + const =
\end{equation}
\begin{equation}
=k^T_0 v_0 + \frac{1}{2}v^2_0 Q v_0 + const (4)
\end{equation}
\begin{equation}
v_0 = (\lambda_\color{red}M, \lambda_\color{blue}L)^T (4a)
\end{equation}
\begin{equation}
k_0 = (1 - \lambda^T K_\color{red}M, 1 - \lambda^T K_\color{blue}L)^T (4b)
\end{equation}
\begin{equation}
Q = \begin{pmatrix}
    K_\color{red}M,_\color{red}M & K_\color{red}M,_\color{blue}L \\
    K_\color{blue}L,_\color{red}M & K_\color{blue}L,_\color{blue}L
    \end{pmatrix}
    (4c)
\end{equation}
\begin{equation}
u = (-y_\color{blue}L, y_\color{red}M)^T (4d)
\end{equation}
\begin{equation}
t_n = \frac{k^T_0 u}{u^T Q u} (5)
\end{equation}
```

\begin{equation}
b = \mathbb{E}_{k,\xi_k \neq 0}[y_k - \displaystyle\sum_{i}\lambda_i y_i K(x_i;x_k)](1)
\end{equation}
\begin{equation}
class(x) = sign(f(x))
f(x) = \sum_{i} \lambda_i y_i K(x_i;x) + b (2)
\end{equation}
\begin{equation}
K = \begin{pmatrix}
    y_1 y_1 K(x_1;x_1) & y_1 y_2 K(x_1;x_2) & \dots & y_1 y_N K(x_1;x_N)\\
    y_2 y_1 K(x_2;x_1) & y_2 y_2 K(x_2;x_2) & \dots & y_2 y_N K(x_2;x_N)\\
    \dots & \dots & \dots & \dots \\
    y_N y_1 K(x_N;x_1) & y_N y_2 K(x_N;x_2) & \dots & y_N y_N K(x_N; x_N)
    \end{pmatrix} (3)
\end{equation}

\begin{equation}
\max\limits_{\lambda} \underbrace{\displaystyle\sum_{i=1}^{n}\lambda_i - \frac{1}{2}\lambda^TK\lambda}_{\Lambda}
\end{equation}
\begin{equation}
\Lambda = \lambda_\color{red}M + \lambda_\color{blue}L - \displaystyle\sum_{j}\lambda_\color{red}M \lambda_j K_\color{red}M,_j
    - \displaystyle\sum_{i}\lambda_\color{blue}L \lambda_i K_\color{blue}L,_i + const +
    \frac{1}{2}\lambda^2_\color{red}M K_\color{red}M,_\color{red}M + \lambda_\color{red}M\lambda_\color{blue}L K_\color{red}M,_\color{blue}L +
    \frac{1}{2}\lambda^2_\color{blue}L K_\color{blue}L,_\color{blue}L=
\end{equation}
\begin{equation}
=\lambda_\color{red}M(1-\displaystyle\sum_{j}\lambda_j K_\color{red}M,_j)+\lambda_\color{blue}L(1-\displaystyle\sum_{i}\lambda_i K_\color{blue}L,_i)
    +\frac{1}{2}(\lambda^2_\color{red}M K_\color{red}M,_\color{red}M + 2\lambda_\color{red}M\lambda_\color{blue}L K_\color{red}M,_\color{blue}L
                + \lambda^2_\color{blue}L K_\color{blue}L,_\color{blue}L) + const =
\end{equation}
\begin{equation}
=k^T_0 v_0 + \frac{1}{2}v^2_0 Q v_0 + const (4)
\end{equation}
\begin{equation}
v_0 = (\lambda_\color{red}M, \lambda_\color{blue}L)^T (4a)
\end{equation}
\begin{equation}
k_0 = (1 - \lambda^T K_\color{red}M, 1 - \lambda^T K_\color{blue}L)^T (4b)
\end{equation}
\begin{equation}
Q = \begin{pmatrix}
    K_\color{red}M,_\color{red}M & K_\color{red}M,_\color{blue}L \\
    K_\color{blue}L,_\color{red}M & K_\color{blue}L,_\color{blue}L
    \end{pmatrix}
    (4c)
\end{equation}
\begin{equation}
u = (-y_\color{blue}L, y_\color{red}M)^T (4d)
\end{equation}
\begin{equation}
t_n = \frac{k^T_0 u}{u^T Q u} (5)
\end{equation}

```python
import numpy as np
```

```python
class SVM:
    #constructor
    def __init__(self, kernel = 'linear', C = 5000.0, max_iteration = 5000, degree = 3, gamma = 1):
        self.kernel = {'poly' : lambda x,y: np.dot(x, y.T) ** degree,
                       'linear' : lambda x,y: np.dot(x, y.T),
                       'rbf' : lambda x,y: np.exp(-gamma*np.sum((y-x[:,np.newaxis])**2,axis=-1))}[kernel]
        self.C = C
        self.max_iteration = max_iteration

    #we want to limit our parameter t, that our lamda-parameter doesn't leaving the (square) borders

    def restrict_to_square(self, t, v0, u):
        t = (np.clip(v0 + t*u, 0, self.C) - v0)[1]/u[1]
        return (np.clip(v0 + t*u, 0, self.C) - v0)[0]/u[0]

    def fit(self, x, y):
        self.x = x.copy()
        #transformation of the classes 0,1 into -1,+1
        self.y = y * 2 - 1
        self.lambdas = np.zeros_like(self.y, dtype=float)
        self.solution_distances = []
        self.convergence_step = np.zeros_like(self.y, dtype=float)

        #formula (3). Count the matrix K
        self.K = self.kernel(self.x, self.x) * self.y[:,np.newaxis] * self.y

        #self.max_iteration
        for _ in range(self.max_iteration):
            #for all lambda
            for idxM in range(len(self.lambdas)):
                # choose random idxL
                idxL = np.random.randint(0, len(self.lambdas))
                # count formula (4c)
                Q = self.K[[[idxM, idxM], [idxL, idxL]], [[idxM, idxL], [idxM, idxL]]]
                # count formula (4a)
                v0 = self.lambdas[[idxM, idxL]]
                # count formula (4b)
                k0 = 1 - np.sum(self.lambdas * self.K[[idxM, idxL]], axis=1)
                # count formula (4d)
                u = np.array([-self.y[idxL], self.y[idxM]])
                # formula (5) if idxM = idxL
                t_max = np.dot(k0, u) /(np.dot(np.dot(Q, u), u) + 1E-15)
                self.lambdas[[idxM, idxL]] = v0 + u * self.restrict_to_square(t_max, v0, u)

            #self.solution_distances.append(np.linalg.norm(self.convergence_step - self.alpha, ord=2))
            self.solution_distances.append(self.lambdas)

        #support vektor index
        idx, = np.nonzero(self.lambdas > 1E-15)
        #count formula (1)
        self.b = np.mean((1.0-np.sum(self.K[idx]*self.lambdas,axis=1))*self.y[idx])

    def decision_function(self, x):
        return np.sum(self.kernel(x, self.x) * self.y * self.lambdas, axis=1) + self.b

    def predict(self, x):
            #transformation of the classes -1,+1 into 0,1
            return (np.sign(self.decision_function(x)) + 1) // 2
```

```python

from sklearn.svm import SVC
import matplotlib.pyplot as plt
import seaborn as snc; snc.set()
from sklearn.datasets import make_blobs, make_circles
from matplotlib.colors import ListedColormap

def test_plot(X, y, svm_model, axes, title):
    plt.axes(axes)
    xlim = [np.min(X[:, 0]), np.max(X[:, 0])]
    ylim = [np.min(X[:, 1]), np.max(X[:, 1])]
    xx, yy = np.meshgrid(np.linspace(*xlim, num=700), np.linspace(*ylim, num=700))
    rgb = np.array([[210, 0, 0], [0, 0, 150]])/255.0

    svm_model.fit(X, y)
    z_model = svm_model.decision_function(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='prism') # standard
    plt.contour(xx, yy, z_model, colors='k', levels=[-1, 0, 1], alpha=0.5, linestyles=['--', '-', '--'])
    plt.contourf(xx, yy, np.sign(z_model.reshape(xx.shape)), alpha=0.3, levels=2, cmap=ListedColormap(rgb), zorder=1)
    plt.title(title)

X, y = make_circles(100, factor=.1, noise=.1)
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12,4))
test_plot(X, y, SVM(kernel='rbf', C=10, max_iteration=1, gamma=1), axs[0], 'SVM Algorithm')
test_plot(X, y, SVC(kernel='rbf', C=10, gamma=1), axs[1], 'sklearn')#

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12,4))
test_plot(X, y, SVM(kernel='rbf', C=10, max_iteration=10, gamma=1), axs[0], 'SVM Algorithm')
test_plot(X, y, SVC(kernel='rbf', C=10, gamma=1), axs[1], 'sklearn')#

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12,4))
test_plot(X, y, SVM(kernel='rbf', C=10, max_iteration=1000, gamma=1), axs[0], 'SVM Algorithm')
test_plot(X, y, SVC(kernel='rbf', C=10, gamma=1), axs[1], 'sklearn')#

"""
X,y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=1.4)
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12,4))
test_plot(X, y, SVM(kernel='linear', C=10, max_iteration=1), axs[0], 'SVM Algorithm')
test_plot(X, y, SVC(kernel='linear', C=10), axs[1], 'sklearn')#

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(12,4))
test_plot(X, y, SVM(kernel='poly', C=10, max_iteration=1, degree=3), axs[0], 'SVM Algorithm')
test_plot(X, y, SVC(kernel='poly', C=10, degree=3), axs[1], 'sklearn')#
"""
```

```python
iterations = [10, 10, 10, 10]
lamb = []

for it in iterations:
    model_ = SVM(kernel='rbf', C=10, max_iteration=it, gamma=1)
    model_.fit(X, y)
    lamb.append(model_.lambdas)

print(np.linalg.norm(lamb[0] - lamb[1]))
print(np.linalg.norm(lamb[0] - lamb[2]))
print(np.linalg.norm(lamb[0] - lamb[3]))
```

    5.492239850541629
    3.593057564932249
    3.9917276099451136

```python

for sol in model_.solution_distances:
    dist_2 = np.linalg.norm(model_.lambdas - sol, ord=2)
    dist_max = np.linalg.norm(model_.lambdas - sol, ord=np.inf)
    print(dist_2, dist_max)
    #print(sol)
    #print(model_.lambdas)
    #print(sol - model_.lambdas)

```

    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0
    0.0 0.0