# Announcements
- Please familiarize yourself with the term projects, and sign up for your (preliminary) choice using [this form](https://forms.gle/ByLLpsthrpjCcxG89). _You may revise your choice, but I'd recommend settling on a choice well before Thanksgiving._
- Recommended reading on ODEs: [Lecture notes by Prof. Hjorth-Jensen (University of Oslo)](https://www.asc.ohio-state.edu/physics/ntg/6810/readings/hjorth-jensen_notes2013_08.pdf)
- Problem Set 4 posted on D2L, due Oct 5.
- __Outlook__: completing ODE examples this week, then Partial Differential Equations (PDEs)
- Conference for Undergraduate Women in Physics: online event in 2021, [applications accepted until 10/25](https://www.aps.org/programs/women/cuwip/)

# Special Considerations: Symplectic Integrators
Some systems of ODEs have special properties, for example constants of motion, that should be
conserved by the system. Simple examples in mechanics are the energy, linear momentum, and angular
momentum. Sometimes it is more important to build in to the integration scheme these conservation
laws than to achieve a high accuracy per step size. Normally, the error in the solution grows as the
solution evolves, bounded by the number of steps times the error per step.  For certain schemes,
however, while they still make errors, the errors will be of such a form that the conserved quantities
will be better preserved.

In Lecture 15 we derived the __velocity form__ of the Verlet algorithm
\begin{align}
x_{n+1} &= x_n + v_n\delta t + {1\over 2}a_n(\delta t)^2 + \mathcal{O}(\delta t^3) \tag{1a}\\
a_{n+1} &= a(x_{n+1}) \label{eq:Vv}\tag{1b}\\
v_{n+1} &= v_n + {1\over 2}\left(a_n + a_{n+1}\right)\delta t + \mathcal{O}(\delta t^2).\tag{1c}
\end{align}

While the velocity computed in this way is only
first-order accurate, if the acceleration depends only upon position, then the
velocity appears only where multiplied by $\delta t$ in determining the positions and so
second-order accuracy is preserved. Of course, if determining the velocity to
high accuracy is required, then one might turn to a different scheme. On the
other hand, if the acceleration depends upon the velocity (for example, a drag
term), there are fewer constants of the motion to be conserved anyway, and one
of the higher-order schemes developed previously would serve.
 Note that Eq.(\ref{eq:Vv}) evaluates $a_{n+1}$ before $v_{n+1}$ can be computed, so the acceleration should only be a function of position, not velocity. _This is by design, as the derivation of the Verlet algorithm was based on conservation of energy._

Here's one implementation of the velocity Verlet algorithm. Note that we compute the acceleration $a_n$ and $a_{n+1}$ in each call; this is not optimal computationally, but easier to read and use with our existing `odeSolve` code.

```python
import numpy as np
def vVerlet_step(g, t, x,v, h, *P):
    """Implements a full step h of the velocity Verlet algorithm
    Parameters:
        g: function. RHS of ODE, with signature g(t,y[0],*P).
        x: Current position x.
        v: Current velocity v.
        h: float. Step length.
        *P:    tuple, additional parameters for g function
    Returns:
        x,v: numpy arrays: next position, next velocity
    """
    a_current = g(x,*P)

    #advance the position: x_n -> x_n+1
    x += v*h+a_current*h**2
    #compute a_n+1
    a_next = g(x,*P)
    #advance the velocity v_n -> v_n+1
    v +=(a_current+a_next)*h**2
    return x,v
```

## Example: Planetary Motion
Today we will study the orbits of two masses interacting through gravity.

In general, the gravitational force on object $j$ with mass $m_j$ and position $\vec{r}_j$ through its gravitational interaction with objects $i$ with mass $m_i$ and position $\vec{x}_i$ is given by
$$
\vec F_j = -\sum_{i\ne j}\frac{G m_j m_i}{|\vec x_i - \vec x_j|^2} \times \frac{\left(\vec x_i - \vec x_j\right)}{|\vec x_i - \vec x_j|}\,
$$
with the unit vector $\hat{\vec{x}} = \frac{\vec{x}}{|x|}$. _Note the direction of this force, gravity should pull two particles closer to each other! If that's not the case in your implementation, this is a good point to start..._

In the __special case of a two-particle system__, we can rewrite this in terms of the separation vector $\vec r = \vec x_1 - \vec x_2$ and total mass $M= m_1+m_2$ such that
the gravitational acceleration $\vec{a}_i=\vec{F}_i/m_i$ is given by
\begin{align}
\vec{a}(\vec r) &= -\frac{G M}{|r|^2} \hat{\vec{r}}\,.\label{eq:aG}\tag{2}
\end{align}
In this case, in the center of mass reference frame, the kinetic energy is given by
$$T =  \frac{1}{2}M\dot{\vec r}^2$$
and the potential energy is given by
$$U(\vec r) =-\frac{G m_1 m_2 }{|r|} $$
_Compared to previous lectures, we will now work with vectorized positions and velocities (rather than writing a separate ODE for each coordinate component). This will later require a small adaption to our function `odeSolve`:_

Now use `vVerlet_step` to __integrate simplified planetary orbits__:
Set $GM = 1$ and try initial conditions
$$
x(0) = 0.5, \quad y(0)=0, \quad v_x(0)=0, \quad v_y(0)=1.63\,.
$$
You will need to define a function `a_gravity` to evaluate the gravitational force, and later on you will need a function to calculate the potential energy `U_gravity` as well.

## Velocity Verlet vs RK4: Energy conservation
Assess the stability of `RK4` and `Velocity Verlet` by checking energy conservation over longer simulation times.

 In order to use `R4K` for integrating this problem, turn Equation(\ref{eq:aG}) into a system of ODEs for the vectorized position and velocity, so that you can reuse the code below.

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

def a_gravity(r, G=1, M=1):
    rr = np.sum(r*r)
    rhat = r/np.sqrt(rr)
    return - G*M/rr * rhat

def U_gravity(r, m=1, G=1, M=1):
    #this assumes the r is a 1D vector, not a 2D array
    return -G*m*M/np.sqrt(np.sum(r*r))

def analyze_Energy(r,v, m=1, G=1, M=1):
    if (len(r.shape) == 1): # r is simple vector
        return 0.5*np.sum(v*v)*M+U_gravity(r,m=m,G=G,M=M)
    else:
        n_steps, ndim = r.shape
        E = np.zeros(n_steps)
        for i in range(n_steps):
            E[i] = 0.5*np.sum(v[i,:]*v[i,:])*M+U_gravity(r[i,:],m=m,G=G,M=M)
def simplified_verlet_wrapper(tmax =2000, h = 1):
    dim = 3

    r0 = np.array([0.5, 0,0])
    v0 = np.array([0, 1.63,0])
    mass = 1
    nsteps = int(t_max/h)

    r = np.zeros((nsteps, dim))
    v = np.zeros_like(r)

    r[0, :] = r0
    v[0, :] = v0
    for i in range(nsteps-1):
    #write your Verlet integration here, without worrying about consistency with odeSolve conventiosn
    return r,v
```

```python
r,v = simplified_verlet_wrapper()
```

```python
rx, ry = r[:,0],r[:,1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)

```

 Write down the system of ODEs that you want to solve with RK4, and define the corresponding `a_gravity_RK4` function.

```python

```

```python
# code copied from Lecture 12/13
# you don't need to change anything in this cell
def RK4_step(t, y, h, g, *P):
    """
    Implements a single step of a fourth-order, explicit Runge-Kutta scheme
    """
    thalf = t + 0.5*h
    k1 = h * g(t, y, *P)
    k2 = h * g(thalf, y + 0.5*k1, *P)
    k3 = h * g(thalf, y + 0.5*k2, *P)
    k4 = h * g(t + h, y + k3, *P)
    return y + (k1 + 2*k2 + 2*k3 + k4)/6
```

 Next, we will consider the conservation of energy of the velocity Verlet algorithm and RK4.
Plot the total energy for both solvers as a function of time, for varying step sizes.

```python

```


# Sympy : Symbolic Mathematics in Python

http://www.scipy-lectures.org/packages/sympy.html

```python
from sympy import *
x = Symbol('x')
y = Symbol('y')
```

# Integration

```python
integrate(x**3, (x, -1, 1))
```

    0

```python
integrate( 1/x )
```

    log(x)

```python
integrate( 1/(1 +2*x) )
```

    log(2*x + 1)/2

```python
integrate( 1/(1 - 2*x) )
```

    -log(2*x - 1)/2

# Equation solving

```python
solve([x + 5*y - 2, -3*x + 6*y - 15], [x, y])
```

    {x: -3, y: 1}

# Differential Equations

```python
f, g = symbols('f g', cls=Function)
```

```python
f(x).diff(x, x) + f(x)
```

    f(x) + Derivative(f(x), x, x)

```python
dsolve(f(x).diff(x, x) + f(x), f(x))
```

    Eq(f(x), C1*sin(x) + C2*cos(x))

```python
dsolve(sin(x)*cos(f(x)) + cos(x)*sin(f(x))*f(x).diff(x), f(x), hint='separable')
```

    [Eq(f(x), -asin(sqrt(C1/(sin(x)**2 - 1) + 1)) + pi),
     Eq(f(x), asin(sqrt(C1/(sin(x)**2 - 1) + 1)) + pi),
     Eq(f(x), -asin(sqrt(C1/(sin(x)**2 - 1) + 1))),
     Eq(f(x), asin(sqrt(C1/(sin(x)**2 - 1) + 1)))]

```python
diff(x, x)
```

    1

```python
diff(1/x, x)
```

    -1/x**2

```python
diff(1/(2*y), y)
```

    -1/(2*y**2)

```python
diff(1/x, x) - diff(1/(2*y), y)
```

    1/(2*y**2) - 1/x**2

```python
from __future__ import division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)

```

```python
eq = (Eq(Derivative(x(t),t), 12*t*x(t) + 8*y(t)), Eq(Derivative(y(t),t), 21*x(t) + 7*t*y(t)))
eq
```

    (Eq(Derivative(x(t), t), 12*t*x(t) + 8*y(t)),
     Eq(Derivative(y(t), t), 7*t*y(t) + 21*x(t)))

```python
dsolve(eq)
```

    [Eq(x(t), C1*x0 + C2*x0*Integral(8*exp(Integral(7*t, t))*exp(Integral(12*t, t))/x0**2, t)),
     Eq(y(t), C1*y0 + C2(y0*Integral(8*exp(Integral(7*t, t))*exp(Integral(12*t, t))/x0**2, t) + exp(Integral(7*t, t))*exp(Integral(12*t, t))/x0))]

```python
eq1 = (Eq(Derivative(x(t),t), 12*t*x(t) + 8*y(t)))
eq1
```

    Eq(Derivative(x(t), t), 12*t*x(t) + 8*y(t))

```python
dsolve(eq1)
```

    Eq(-4*Integral(2*y(t)*exp(-6*t**2), t) - 4*Integral(3*t*x(t)*exp(-6*t**2), t), C1)

```python
eq1 = (Eq(Derivative(1/x,x)- Derivative((2*y)**(-1), y)))
eq1
```

    Eq(Derivative(1/x, x) - Derivative(1/(2*y), y), 0)

```python
dsolve(eq1)
```

```python

```


# Heat diffusion

- toc:false
- branch: master
- badges: true
- comments: false
- categories: [mathematics, numerical recipes]
- hide: true

-----
Questions:
- How do I use the Forward-Time Centred-Space method (FTCS) to solve the diffusion equation?

Objectives:
- Apply the Forward-Time Centred-Space method (FTCS) to solve theat heat diffusion equation

-----

### The diffusion equation is an initial value problem

In the previous section we solved a boundary value problem in the form of Laplace's equation. In this section we will look at an initial value problem, which is a more complex type of PDE. An initial value problem is more complex as we are told the starting conditions and then have to predict future behaviour as a function of time.

The example we will use is the one-dimensional diffusion equation:

\begin{equation}
 \frac{\partial \phi}{\partial t} = D\frac{\partial^2 \phi}{\partial x^2}
\end{equation}

In this case we have a variable $\phi(x,t)$ that depends on position $x$ and time $t$ - so can we not solve it in the same way as finding the $\phi(x,y)$ Laplace's equation, which also had two independent variables?

The problem is that we  only have an *initial* condition in the time dimension - we know the value of $\phi(x,t)$ at $t=0$ but we do not typically know the value of $t$ at a later point. In the spatial dimensions we know the boundary conditions at either end of the grid.

Instead, we will use the <mark>Forward-Time Centred-Space method (FTCS)</mark>.

### There are two steps to the Forward-Time Centred-Space method

#### Step one
Use the finite difference method to express the 1D Laplacian as a set of simulatenous equations:

\begin{equation}
\frac{\partial^2\phi}{\partial x^2} = \frac{\phi(x+a,t)+\phi(x-a,t) - 2\phi(x,t)}{a^2}
\end{equation}

where $a$ is the grid spacing.

Substitute this back into the diffusion equation to give a set of simulataneous ODEs:

\begin{equation}
 \frac{d \phi}{d t} = \frac{D}{a^2}(\phi(x+a,t)+\phi(x-a,t)-2\phi(x,t))
\end{equation}

If there are $N$ grid points then Equation 3 corresponds to a set of $N$ equations. It is an ODE as there is derivative with respect to only one variable - time.

#### Step two
We now have a set of simultaneous ODEs for $\phi(x,t)$. So we can use Euler's method to evolve the system forward in time. Euler's method for solving an ODE of the form $\frac{d\phi}{dt} = f(\phi,t)$ has the general form:

\begin{equation}
\phi(t+h) \simeq \phi(t) + hf(\phi, t).
\end{equation}

Applying this to Equation 3 gives:

\begin{equation}
\phi(x,t+h) = \phi(x,t) + h\frac{D}{a^2}(\phi(x+a,t)+\phi(x-a,t)-2\phi(x,t))
\end{equation}

### The FTCS method can be applied using the Python skills we have developed

Consider a 10cm rod of stainless steel initially at a uniform temperature of 20$^\mathrm{o}$ Celsius. The rod is dipped in a hot water bath at 90$^\mathrm{o}$ Celsius at one end, and held in someone's hand at the other. Assume that the hand is at constant body temperature throughout (27$^\mathrm{o}$ Celsius).

The problem can be represented visually as follows:

Our goal is to calculate the temperature profile of the steel as a function of distance $x$ from the cold side to the hot side, and as a function of time. For simplicity let us assume that the rod is perfectly insulated so that heat only moves horizontally; as a result this problem can be modelled as 1-dimensional. Also assume that neighter the hot water bath or the hand change temperature appreciably.

Thermal conduction is described by the diffusion equation (or heat equation in this context)

\begin{equation}
 \frac{\partial \phi}{\partial t} = D\frac{\partial^2 \phi}{\partial x^2},
\end{equation}

where $D$ is the material dependent thermal diffusivity. For steel $D=4.25\times10^{-6}\mathrm{m}^2\mathrm{s}^{-1}$.

First, let's import the libraries we will be using

```python
import numpy as np
import matplotlib.pyplot as plt
```

Now let's fix some of the constants in the problem

```python
L = 0.1 # width of the rod in metres
D = 4.25e-6 # thermal diffusivity of steel
```

And some of the numerical parameters:

```python
N = 100 # number of divisions in the grid
a = L/N # grid spacing
h = 1e-4 # time step
```

Let's specify the boundary conditions and the initial condition

```python
T_left = 27.0  # temperature fixed on left side of rod
T_right = 90.0 # temperature fixed on right side of rod
T_middle = 20.0 # temperature of rod at the beginning
```

And now create an array $T$ to hold the temperature of the rod and an array $T_{\mathrm{new}}$ to calculate the temperature of the rod after evolving through time. Note that there are $N$ grid divisions but $N+1$ grid points (as we are evaluating at the boundary on each edge).

```python
T = np.empty(N+1, float)
T_new = np.empty(N+1, float)
```

Apply the boundary conditions and initial condition to our rod:

```python
T[0] = T_left
T[-1] = T_right
T[1:-1] = T_middle
T_new[0] = T_left
T_new[-1] = T_right
```

```python
def evolve(T,T_new):
    c = (h*D)/(a*a)
    T_new[1:N] = T[1:N] + c*(T[2:N+1]+T[0:N-1]-2*T[1:N])
    return T, T_new

```

Note that in the function `evolve` we implement Equation 5. We make use of Numpy's element-by-element array manipulation to evaluate the $N-1$ equations in a single step.

```python
epsilon = h/100

times=np.array([0.01,0.1,1,10,100])
t_end = times[-1]+epsilon
t=0.0

while t < t_end:

    T_new, T = evolve(T, T_new)
    t+=h

    for time in times:
        if abs(t-time)<epsilon:
            plt.plot(T,label="{}s".format(round(t,ndigits=3)))

plt.legend()
plt.title("Heat diffusion along a 1D rod")
plt.xlabel("Position (cm)")
plt.ylabel("Temperature (degrees centigrade)")
```

---
Keypoints:

- The diffusion equation is an initial value problem
- There are two steps to the Forward-Time Centred-Space (FTCS) method
- The FTCS method can be applied using the Python skills we have developed

---

Do [the quick-test](https://nu-cem.github.io/CompPhys/2021/08/02/Heat-Diffusion-Qs.html).

Back to [Modelling with Partial Differential Equations](https://nu-cem.github.io/CompPhys/2021/08/02/PDEs.html).

---

```python

```


### Introduction

A neuron's job is to fire. If it does not fire, or emit action potentials, synaptic modifications takes place until the neuron is activated more frequently. However, neurons have limited resources and must dedicate them to the most productive synapses. The numerous machanisms of synaptic plasticity, from synaptic growth and pruning to the subtle changes in neurotransmitter and receptor levels, all work in concert to ensure the neuron is activated. This results in 'pattern recognition' behavior. In other words, the neuron changes its synaptic preferences such that a certain frequently occuring stimulus, or collection of stimuli, activate the neuron more, relative to other stimuli. How does a neuron identify a frequently ocuring stimulus and subsequently tune its internal biochemistry such that exposure to the stimulus leads to activation?

Below we provide a proof that a traditional Hebbian plasticity learning rule is minimizing an upper bound of the alpha divergence between the neuron and the distribution generating the stimuli. Modeling a neuron as a radial basis function, we will see that Hebbian plasticity leads to 'mode seeking' behavior. In other words, a neuron is seeking a pocket of high probability density in the data generating distribution. This permits a rather natural definition of a 'feature' as being a local extrema in the data generating distribution. This definition also captures the intrinsic subjectivity of what a feature can be. Certain tasks operating on the same distribution may identify different features as being useful - in both cases though, these features may be characterized as being local extrema.

### $\alpha$-Divergence

Divergences are information theoretic measures for quantifying the similarity between distributions. The alpha divergence is a generalization of the popular Kullback-Leibler (KL) divergence. More precisely, it is a family of divergences specified by the parameter $\alpha$. The forward and reverse KL divergences are special cases of the $\alpha$-divergence for $\alpha=1$ and $\alpha=0$ respectively and $\alpha=0.5$ is the Hellinger distance. The $\alpha$-divergence between two distributions $P(x)$ and $Q(x)$ is calculated using their probability density functions (pdf) $p(x)$ and $q(x)$.

\begin{align}
D_\alpha(p||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q(x)-p(x)^{\alpha}q(x)^{1-\alpha}dx
\end{align}

For a thorough review of the properties of the $\alpha$-divergence and their associated proofs, see [Proofs of Alpha Divergence Properties](https://www.ece.rice.edu/~vc3/elec633/proof_alpha_divergence.pdf). Let us model a neuron as a radial basis function. In particular, we choose a gaussian pdf denoted by $q(x)$ rendering it more amenable to information theoretic analysis. Furthermore, assume we have a data generating distribution $P(x)$ where i.i.d. $x_i \sim P$ are presented to the neuron $q(x_i)$. After the neuron has observed the point $x_i$ it will update its parameters $\mu$, $\Sigma$ based on some learning rules. Following any observation $x_i$ we can measure how similar our neuron's model $q(x|\mu_i,\Sigma_i)\equiv q_i(x)$ is to the data generating distribution $P(x)$. This can be done with the $\alpha$-divergence.

\begin{align}
D_\alpha(p||q_i) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q_i(x)-p(x)^{\alpha}q_i(x)^{1-\alpha}dx
\end{align}

where $p(x)$ is the pdf of the data generating distribution. It is natural to ask which values of $\mu$ and $\Sigma$ allow $q(x)$ to minimize $D_\alpha(p||q)$. The answer depends on which value of $\alpha$ we choose. A range of possible minima are presnted in the figure below.

As we can see, minimization of $D_\alpha(p||q)$ with respect to $\mu$ and $\Sigma$ for $\alpha \le 0$ leads qualitatively to mode seeking behavior. Therefore a simple learning rule designed to minimize the $\alpha$-divergence for $\alpha \le 0$ would produce the desired long-term behavior. However, we are confronted with a problem. The data generating distribution $P(x)$, colloquially the world or universe, is unknown. As a result we do not know the density $p(x)$ and are unable to analytically minimize $D_\alpha(p||q)$. However, by identifying that $D_\alpha(p||q)$ is valid for unnormalized distributions and constraining the values of $\alpha$ we can derive an upper bound that is an expectation over $p(x)$ and minimize it as a surrogate with stochastic approximation techniques.

To see this, let us begin by extracting one $p(x)$ from the term $p(x)^{\alpha}$ in the $\alpha$-divergence above

$$D_\alpha(p||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q(x)-p(x)p(x)^{\alpha-1}q(x)^{1-\alpha}dx$$

Because $D_\alpha$ holds for all $\alpha$ and unnormalized $p(x)$ if we divide $p(x)$ by its maximum such that $\frac{p(x)}{p_{max}}\equiv \bar{p}(x)\le1$ and constrain $\alpha \lt 0$ then we have

$$\bar{p}(x)^{\alpha-1}\ge1$$

In addition, for $\alpha \lt 0$ the leading factor $\frac{1}{\alpha(1-\alpha)}\lt0$

The factor $\bar{p}(x)^{\alpha-1}$ is present in the only negative term in the $\alpha$-divergence. As it is no less than one, by removing it $D_\alpha(\bar{p}||q)$ will either increase or remain unchanged. This produces an upper bound on the $\alpha$-divergence.

$$D_\alpha(\bar{p}||q) \le \frac{1}{\alpha(1-\alpha)} \int_x \alpha \bar{p}(x) +(1-\alpha)q(x)-\bar{p}(x)q(x)^{1-\alpha}dx$$

As we will show later on, the final update equations we arrive at can also be found for $\alpha=0$ by minimizing a similar upper bound for the reverse KL divergence. At first glance this does not seem to have helped our situation as the unknown density $\bar{p}(x)$ is still present. However the aid it provided will become evident as we continue with traditional optimization. For notational brevity let us define $\theta = (\mu, \Sigma)$ and the upper bound as

\begin{align}
J_\alpha(\bar{p}||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha \bar{p}(x) +(1-\alpha)q(x)-\bar{p}(x)q(x)^{1-\alpha}dx
\end{align}

Then the derivative of the upper bound with respect to $\theta$ is

\begin{align}
\frac{d J_\alpha(\bar{p}||q)}{d\theta} = \frac{1}{\alpha} \bigg( \int_x \frac{d q(x)}{d \theta}dx - \int_x \bar{p}(x)q(x)^{-\alpha}\frac{d q(x)}{d \theta}dx \bigg)
\end{align}

For the first integral, we are able to analytically integrate the derivative of our gaussian which we will now show is equal to the zero vector.
We first recall the partials of the multivariate gaussian pdf for both the mean $\mu$ and the covariance $\Sigma$.

\begin{align}
\frac{\partial q}{\partial \mu} &= q(x) \Sigma^{-1} (x-\mu) \\
\frac{\partial q}{\partial \Sigma} &= q(x) \frac{1}{2}\big[\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1}-\Sigma^{-1}\big] \\
\end{align}

With these we can calculate $\int_x \frac{d q(x)}{d \theta}dx$.

\begin{align}
\int_x \frac{\partial q}{\partial \mu}dx &= \int_x q(x) \Sigma^{-1} (x-\mu)dx \\
&= \Sigma^{-1} \bigg(\int_x q(x) x dx - \mu \int_x q(x) dx \bigg)\\
&= \Sigma^{-1} \big(\mu - \mu \big) \\
&= 0\\
\end{align}

We next calculate the integral of $\frac{\partial q}{\partial \Sigma}$

\begin{align}
\int_x \frac{\partial q}{\partial \Sigma}dx &= \int_x q(x) \frac{1}{2}\big[\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1}-\Sigma^{-1}\big]dx \\
&=  \frac{1}{2}\int_x q(x) \Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1} dx -\frac{1}{2}\Sigma^{-1} \\
&= \frac{1}{2}\Sigma^{-1}\bigg(\int_x q(x) (x-\mu)(x-\mu)^T dx \bigg) \Sigma^{-1} -\frac{1}{2}\Sigma^{-1} \\
&=  \frac{1}{2}\Sigma^{-1}\Sigma \Sigma^{-1} -\frac{1}{2}\Sigma^{-1} \\
&=  \frac{1}{2}\Sigma^{-1} -\frac{1}{2}\Sigma^{-1}\\
&= 0
\end{align}

We now rewrite the upper bound with zero in place of the first integral and we replace $\bar{p}(x)$ with the original scaled $p(x)/p_{max}$.

$$\frac{d J_\alpha(\bar{p}||q)}{d\theta} = -\frac{1}{\alpha} \int_x \frac{p(x)}{p_{max}}q(x)^{-\alpha}\frac{d q(x)}{d \theta}dx$$

We now factor out the $\frac{1}{p_{max}}$ and use the law of the unconscious statistician (LOTUS) to rewrite the derivative of the upper bound as

\begin{align}
\frac{d J_\alpha(\bar{p}||q)}{d\theta} &= -\frac{1}{\alpha p_{max}} \int_x p(x)q(x)^{-\alpha}\frac{d q(x)}{d \theta}dx\\
&= -\frac{1}{\alpha p_{max}} \mathbb{E}_{P} \bigg[q(x)^{-\alpha}\frac{d q(x)}{d \theta}\bigg]\\
\end{align}

Lastly we substitute the partials of $q(x)$ to see both components

\begin{align}
\frac{\partial J_\alpha(\bar{p}||q)}{\partial \mu} &= -\frac{1}{\alpha p_{max}} \mathbb{E}_{P} \big[q(x)^{-\alpha}\frac{\partial q}{\partial \mu}\big] \\
&= -\frac{1}{\alpha p_{max}} \mathbb{E}_{P} \big[q(x)^{-\alpha}q(x) \Sigma^{-1} (x-\mu) \big] \\
&= -\frac{\Sigma^{-1}}{\alpha p_{max}} \mathbb{E}_{P} \big[q(x)^{1-\alpha}(x-\mu) \big]
\end{align}

If the expectation in the above expression equals zero, then then entire partial derivative is zero. Furthermore, if $\Sigma$ is invertible then it is the only way the partial can be zero.

\begin{align}
\frac{\partial J_\alpha(\bar{p}||q)}{\partial \Sigma} &= -\frac{1}{\alpha p_{max}} \mathbb{E}_{P} \big[q(x)^{-\alpha}\frac{\partial q}{\partial \Sigma}\big] \\
&= -\frac{1}{\alpha p_{max}} \mathbb{E}_{P} \big[q(x)^{-\alpha}(-q(x) \frac{1}{2}\big[\Sigma^{-1}-\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1})\big] \\
&= -\frac{\Sigma^{-1}}{2 \alpha p_{max}} \mathbb{E}_{P} \big[ q(x)^{1-\alpha} [(x-\mu)(x-\mu)^T-\Sigma] \big]\Sigma^{-1}
\end{align}

As it was for $\frac{\partial J_\alpha(\bar{p}||q)}{\partial \mu}$, provided $\Sigma$ is invertible the expectation being zero is the only way for $\frac{\partial J_\alpha(\bar{p}||q)}{\partial \Sigma}$ to be zero.

### The Upper Bound

Before we begin exploring convergence properties and the dynamics of the underlying system, let us examine the upper bound $J_\alpha(p||q)$.

$$J_\alpha(\bar{p}||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha \bar{p}(x) +(1-\alpha)q(x)-\bar{p}(x)q(x)^{1-\alpha}dx$$

To build intuition about how the upper bound behaves we will visualize the entire surface in a low dimensional and easy to understand example. Let us define the data generating distribution as

$$p(x) = \frac{2}{5}N(\frac{1}{2},0.1)+\frac{3}{5}N(\frac{7}{10}, 0.05)$$

We also define the magnitude constrained $p(x)$ as

$$\bar{p}(x) = \frac{1}{p_{max}}\bigg(\frac{2}{5}N(\frac{1}{2},0.1)+\frac{3}{5}N(\frac{7}{10}, 0.05)\bigg)$$

and the neuron as

$$q(x) = N(\frac{1}{5},0.005)$$

```python
import numpy as np
import matplotlib.pyplot as plt
from OldBrain import Neuron, GMM
import warnings
warnings.filterwarnings('ignore')

p_mean = 0.5
p_sig = 0.175
p = GMM([0.4,0.3,0.3], np.array([[0.15,0.05],[0.85,0.035],[p_mean,p_sig]]))
#p = GMM([1.0], np.array([[p_mean,p_sig]]))
q = GMM([1.0], np.array([[0.85,0.05]]))
lims = [0,1]
x_vals = np.linspace(lims[0],lims[1],500)
p_vals = p(x_vals)
q_vals = q(x_vals)
fig = plt.figure(figsize=(10,5))
plt.plot(x_vals, p_vals, label="$p(x)$")
plt.plot(x_vals, q_vals, label="$q(x)$")
plt.legend()
plt.show()
```

We will approximate the integral in $J_{\alpha_0}(\bar{p}||q(x|\mu,\sigma^2))$ and $D_{\alpha_0}(p||q(x|\mu,\sigma^2))$ with a gaussian quadrature.

```python
from scipy.integrate import quadrature
def D(mu,sig,a):
    q = GMM([1.0], np.array([[mu,sig]]))
    def d_i(x):
        global p
        q_i = q(x)
        p_i = p(x)
        return a*p_i + (1-a)*q_i - np.power(p_i,a)*np.power(q_i,1-a)
    quad_rslt, quad_err = quadrature(d_i, lims[0], lims[1], maxiter=200)
    return (quad_rslt/(a*(1-a)), quad_err)

def J_old(mu,sig,a):
    q = GMM([1.0], np.array([[mu,sig]]))
    def j_old(x):
        global p
        q_i = q(x)
        p_i = p(x)
        return a*p_i + (1-a)*q_i - p_i*np.power(q_i,1-a)
        #return p_i + q_i - p_i*np.power(q_i,1-a)
    quad_rslt, quad_err = quadrature(j_old, lims[0], lims[1], maxiter=200)
    return (quad_rslt/(a*(1-a)), quad_err)
    #return (quad_rslt, quad_err)

def J_new(mu,sig):
    q = GMM([1.0], np.array([[mu,sig]]))
    def j_new(x):
        global p
        q_i = q(x)
        #log_q_i = -0.5*(np.log(2*np.pi*sig*sig)+np.power((x-mu)/sig,2))
        p_i = p(x)
        #return a*p_i + (1-a)*q_i - p_i*np.power(q_i,1-a)
        #c = np.sqrt(2*np.pi*(p_sig+sig))*np.exp(-0.5*(p_mean-mu)**2/(p_sig+sig))
        return p_i*q_i - p_i + q_i
    quad_rslt, quad_err = quadrature(j_new, lims[0], lims[1], maxiter=200)
    #return (quad_rslt/(a*(1-a)), quad_err)
    return (quad_rslt, quad_err)

def KL(mu,sig):
    q = GMM([1.0], np.array([[mu,sig]]))
    def kl_i(x):
        global p
        q_i = q(x)
        log_q_i = -0.5*(np.log(2*np.pi*sig*sig)+np.power((x-mu)/sig,2))
        p_i = p(x)
        return -q_i*(np.log(p_i)-log_q_i)
    quad_rslt, quad_err = quadrature(kl_i, lims[0], lims[1], maxiter=200)
    return (quad_rslt, quad_err)
```

It appears that the "upper bound" is not at all an "upper bound". Let us plot all of the integrands along the $x$ axis and investigate

```python
def d_i(x,a):
        global p,q
        q_i = q(x)
        p_i = p(x)
        return (a*p_i + (1-a)*q_i - np.power(p_i,a)*np.power(q_i,1-a))/(a*(1-a))

def j_i(x,a):
        global p,q
        q_i = q(x)
        p_i = p(x)
        return (a*p_i + (1-a)*q_i - p_i*np.power(q_i,1-a))/(a*(1-a))
        #return (p_i + q_i - p_i*np.power(q_i,1-a))

def ce_i(x,a):
        global p,q
        q_i = q(x)
        p_i = p(x)
        return -p_i*np.log(q_i)

def kl_i(x,a):
        global p,q
        q_i = q(x)
        p_i = p(x)
        return -q_i*np.log(p_i/q_i)

all_as =[0.9,0.5,0.1,0.01,-0.01,-0.1,-0.25,-0.5,-0.75,-1]
for a in all_as:
    di_vals = d_i(x_vals,a)
    ji_vals = j_i(x_vals,a)
    plt.plot(x_vals, di_vals/np.abs(di_vals).max(), label='$d$')
    #plt.plot(x_vals, di_vals, label='$d$')
    plt.plot(x_vals, ji_vals/np.abs(ji_vals).max(), label='$j$')
    #plt.plot(x_vals, ji_vals, label='$j$')
    #plt.plot(x_vals, ce_i(x_vals,a)/np.abs(ce_i(x_vals,a)).max(), label='$ce$')
    #plt.plot(x_vals, kl_i(x_vals,a), label='$kl$')
    plt.plot(x_vals, p_vals/p_vals.max(), label="$p(x)$", ls='dashed')
    plt.title("alpha = {}".format(a))
    #plt.ylim([ji_vals.min(), ji_vals.max()])
    #plt.ylim([-0.5, 1.5])

    plt.legend()
    plt.show()

```

Correct. Unfortunately, I forgot that the lead coefficient in the $\alpha$-divergence $\frac{1}{\alpha(1-\alpha)}$ is always negative for $\alpha \lt 0$. Therefore, $J$ is actually a lower bound of $D$... If that is the case, then why is it that the local minima seem to coincide?

So far, there is still a very strange connection between the true alpha divergence and the pseudo divergence $J$. I would like to explore it further. Let us now look at how both divergences changes for different values of the mean, $\mu$.

```python
plt.clf()
mus = np.linspace(0,1,100)
sigmas = np.linspace(0.0001,0.01,100)
#all_as =[-0.00001,-0.0001,-0.001,-0.01,-0.1,-1,-1.5,-2,-4]
all_as =[-1,-0.7,-0.5,-0.1,0.1,0.5]
for a in all_as:
    fig = plt.figure(figsize=(15,5))
    jold_mu = []
    kl_mu = []
    for mu in mus:
        jold_mu.append(J_old(mu,p_sig,a)[0])
        kl_mu.append(KL(mu,p_sig)[0])
    ax1 = fig.add_subplot(121)
    ax1.plot(mus, jold_mu, label='$J_{old}$')
    ax1.plot(mus, kl_mu, label='$KL$')
    ax1.set_title("alpha = {}".format(a))
    ax1.set_xlabel('$\mu$')
    ax1.legend()
    jnew_s = []
    jold_s = []
    kl_s = []
    for s in sigmas:
        jold_s.append(J_old(p_mean+0.1,s,a)[0])
        #kl_s.append(KL(p_mean+0.1,s)[0])

    ax2 = fig.add_subplot(122)
    ax2.plot(sigmas, jold_s/np.max(jold_s), label='$J_{old}$')
    #ax2.plot(sigmas, kl_s/np.max(kl_s), label='$KL$')
    ax2.plot([p_sig*p_sig,p_sig*p_sig],[0,1],c='k',ls='dashed')
    ax2.set_title("alpha = {}".format(a))
    ax2.set_xlabel('$\sigma$')
    ax2.legend()
    plt.show()
```

```python
plt.clf()
sigmas = np.linspace(0.0001,0.2,100)
#all_as =[-0.00001,-0.0001,-0.001,-0.01,-0.1,-1,-1.5,-2,-4]
all_as = np.linspace(-0.8,0.9,20)
for a in all_as:
    jold_s = []
    for s in sigmas:
        jold_s.append(J_old(p_mean,s,a)[0])

    plt.plot(sigmas, jold_s/np.max(np.abs(jold_s)), label='$J_{old}$')
    plt.plot([p_sig,p_sig],[0,1],c='k',ls='dashed')
    plt.title("alpha = {}".format(a))
    plt.xlabel('$\sigma$')
    plt.legend()
    plt.show()
```

```python
sigmas[(np.diff(np.sign(np.diff(jold_s))) > 0).nonzero()[0] + 1]
```

    array([], dtype=float64)

```python
sigmas[(np.diff(np.sign(np.diff(kl_s))) < 0).nonzero()[0] + 1]
```

    array([], dtype=float64)

```python
p_sig*p_sig
```

    0.030624999999999996

Let us now see how close the extrema are between the KL divergence and my proxy divergence

```python
plt.clf()
mus = np.linspace(0,1,500)
a = -0.5
fig = plt.figure(figsize=(10,5))
js_mu = []
kls_mu = []
for mu in mus:
    #q = Neuron([1,1], np.array([mu]), 0.01, 0.025, 1)
    q = Neuron([1,1], np.array([mu]), p_sig*p_sig, 0.025, 1)
    js_mu.append(J_old(mu, p_sig, a)[0])
    kls_mu.append(D(mu,p_sig,a)[0])
p_vals = p(x_vals)
plt.plot(x_vals, p_vals/p_vals.max(), label="$p(x)$")
plt.plot(mus, js_mu/np.max(np.abs(js_mu)), label='$J$')
plt.plot(mus, kls_mu/np.max(np.abs(kls_mu)), label='$KL$')
plt.title(f"Comparison of KL divergence and J $\\alpha$ = {a}", fontsize=16)
plt.xlabel('$\mu$', fontsize=16)

plt.legend()
plt.show()
```

```python
js_zeros = mus[(np.diff(np.sign(np.diff(js_mu))) < 0).nonzero()[0] + 1]
kls_zeros = mus[(np.diff(np.sign(np.diff(kls_mu))) > 0).nonzero()[0] + 1]

print("js_zeros = ", js_zeros)
print("kls_zeros = ", kls_zeros)
```

    js_zeros =  [0.1503006 0.498998  0.8496994]
    kls_zeros =  [0.14829659 0.498998   0.8496994 ]

```python
p_sig**2
```

    0.0012250000000000002

Take the following two equations...

\begin{align}
J_\alpha(p||q) = \frac{1}{\alpha(1-\alpha)} \int_x \alpha p(x) +(1-\alpha)q(x)-p(x)q(x)^{1-\alpha}dx
 && \text{KL}(q || p) = \int_x q(x) \log \frac{q(x)}{p(x)} dx
\end{align}

Let $q(x)$ be the pdf for a Gaussian,
$\mathcal{N}(\mu, \Sigma)$, and $p(x)$ be unknown. I have demonstrated numerically that both equations have the same extrema for $\mu$, for $-1 \lt \alpha \lt 0$. (If you would like to see the notebook for that, I can send it too. For now just assume it is true)

I would now like to show it analytically. I thought it would be relatiely straight forward to take the partials and show that the same $\mu^*$ minimizes both.

\begin{align}
\frac{\partial J_\alpha}{\partial \mu} &= \frac{1}{\alpha} \int_x q_{\mu}(x)\bigg(1-\frac{p(x)}{q(x)^{\alpha}}\bigg)dx
 && \frac{\partial \text{KL}}{\partial \mu} = & \int_x q_{\mu}(x) \bigg(\log \frac{q(x)}{p(x)}+1 \bigg) dx \\
 \frac{\partial J_\alpha}{\partial \mu} &= \frac{1}{\alpha} \int_x q(x)\Sigma^{-1}(x-\mu)\bigg(1-\frac{p(x)}{q(x)^{\alpha}}\bigg)dx
 && \frac{\partial \text{KL}}{\partial \mu} = & \int_x q(x) \Sigma^{-1}(x-\mu) \bigg(\log \frac{q(x)}{p(x)}+1 \bigg) dx
\end{align}

After observing that $\int_x q_{\mu}(x) dx = 0$, via...

\begin{align}
\int_x q_{\mu}(x) dx &= \int_x q(x) \Sigma^{-1} (x-\mu)dx \\
&= \Sigma^{-1} \bigg(\int_x q(x) x dx - \mu \int_x q(x) dx \bigg)\\
&= \Sigma^{-1} \big(\mu - \mu \big) \\
&= 0\\
\end{align}

and setting both equations to zero, we can see...

\begin{align}
0 = \mathbb{E}_{q} \bigg[\big(1-\frac{p(x)}{q(x)^{\alpha}}\big)(x-\mu) \bigg]
 && 0 = \mathbb{E}_{q} \bigg[ \log \frac{p(x)}{q(x)}(x-\mu) \bigg]
\end{align}

And from here, setting them both equal to zero, we can derive closed form solutions...

\begin{align}
\mu_J = \frac{ \mathbb{E}_{q} \bigg[ \frac{p(x)}{q(x)^{\alpha}}x \bigg]}{\mathbb{E}_{q} \bigg[ \frac{p(x)}{q(x)^{\alpha}} \bigg]}
&& \mu_{KL} = \frac{ \mathbb{E}_{q} \bigg[ \log\big(\frac{p(x)}{q(x)}\big)x \bigg]}{\mathbb{E}_{q} \bigg[ \log\big(\frac{p(x)}{q(x)}\big) \bigg]}
\end{align}

Let us test our formulas as a function of a few parameters. First, $\alpha$...

```python
def mu_j(a):
    def numerator(x):
        global p,q
        q_i = q(np.array(x).reshape(-1,1),update=False)
        p_i = p(x)
        return p_i*np.power(q_i,1-a)*x
    def denominator(x):
        global p,q
        q_i = q(np.array(x).reshape(-1,1),update=False)
        p_i = p(x)
        return p_i*np.power(q_i,1-a)
    num_quad, quad_err = quadrature(numerator, lims[0], lims[1], maxiter=200)
    den_quad, quad_err = quadrature(denominator, lims[0], lims[1], maxiter=200)

    return (num_quad/den_quad, quad_err)
def mu_kl():
    def numerator(x):
        global p,q
        q_i = q(np.array(x).reshape(-1,1),update=False)
        p_i = p(x)
        return q_i*np.log(p_i/q_i)*x
    def denominator(x):
        global p,q
        q_i = q(np.array(x).reshape(-1,1),update=False)
        p_i = p(x)
        return q_i*np.log(p_i/q_i)
    num_quad, quad_err = quadrature(numerator, lims[0], lims[1], maxiter=200)
    den_quad, quad_err = quadrature(denominator, lims[0], lims[1], maxiter=200)

    return (num_quad/den_quad, quad_err)
```

```python
j_vals = []
a=-0.5
a_vals = np.linspace(-0.001,-0.1,10)
q = Neuron([1,1], np.array([0.85]), 0.0001, 0.025, 1)
for a in a_vals:
    j_vals.append(J(a)[0])
kl_vals = [mu_kl()[0]]*len(a_vals)

plt.plot(a_vals, kl_vals, label='$KL$')
plt.plot(a_vals, j_vals, label='$J$')
plt.legend()
plt.show()
```

```python
j_vals
```

    [58.228124241886064,
     5.64800159377656,
     3.353244849286156,
     2.537653057250618,
     2.1166368516408514,
     1.8577815070433679,
     1.6812981236654556,
     1.5524044651395237,
     1.4535120137924598,
     1.3747683320650408]

For a given $\alpha=\alpha_0$ we will calculate the value of $J_{\alpha_0}(\bar{p}||q(x|\mu,\sigma^2))$ for $0\le \mu \le 1$ and $0 \lt \sigma^2 \le 1$ and visualize the resulting surface, along with the location of the minimum of the $\alpha$-divergence and the upper bound. We will also plot the gaussians that result from those minima.

### Hebbian Learning Rules

Given a neuron $q(x)$ where $x \in {\rm I\!R}^k$ is the input to the neuron, the most fundamental learning rule from Hebb's theory is

$$\Delta w_i = q(x_i)\cdot x_i$$

where $\Delta w_i$ denotes the change made to the $i^{\text{th}}$ synaptic weight. The update is the product of the input and the output. Many generalizations of this learning rule have been studied. (Go into them more in the future). One valid update rule is the Hebb rule with postsynaptic or presynaptic LTP/LTD threshold as follows (chapter 19.2 of Gerstner's [book](https://neuronaldynamics.epfl.ch/online/Ch19.S2.html))

$$\Delta w_i = q(x_i)\cdot (x_i-c)$$

We observe that this is exactly the learning rule derived above from minimizing $J_\alpha(p||q)$, the upperbound of the $\alpha$-divergence.

dissertation/Alpha Divergence - Lower Bound.ipynb


# Sampled Softmax

For classification and prediction problems a typical criterion function is cross-entropy with softmax. If the number of output classes is high the computation of this criterion and the corresponding gradients could be quite costly. Sampled Softmax is a heuristic to speed up training in these cases. (see: [Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model](http://www.iro.umontreal.ca/~lisa/pointeurs/importance_samplingIEEEtnn.pdf), [Exploring the Limits of Language Modeling](https://arxiv.org/pdf/1602.02410v1.pdf), [What is Candidate Sampling](https://www.tensorflow.org/extras/candidate_sampling.pdf))

## Basics

The softmax function is used in neural networks if we want to interpret the network output as a probability distribution over a set of classes $C$ with $|C|=N_C$.

Softmax maps an $N_C$-dimensional vector $z$, which has unrestricted values, to an $N_C$ dimensional vector $p$ with non-negative values that sum up to 1 so that they can be interpreted as probabilities. More precisely:

$$
\begin{align}
p_i &= softmax(z, i)\\
    &= \frac{exp(z_i)}{\sum_{k\in C} exp(z_k)}\\
\end{align}
$$

In what follows we assume that the input $z$ to the softmax is computed from some hidden vector $h$ of dimension $N_h$  in a specific way, namely:

$$ z = W h + b $$

where $W$ is a learnable weight matrix of dimension $(N_c, N_h)$ and $b$ is a learnable bias vector.
We restrict ourselves to this specific choice of $z$ because it helps in implementing an efficient sampled softmax.

In a typical use-case like for example a recurrent language model, the hidden vector $h$ would be the output of the recurrent layers and $C$ would be the set of words to predict.

As a training criterion, we use cross-entropy which is a function of the expected (true) class $t\in C$ and the probability predicted for it:

$$cross\_entropy := -log(p_t)$$

## Sampled Softmax from the outside

For the normal softmax the CNTK Python-api provides the function [cross_entropy_with_softmax](https://cntk.ai/pythondocs/cntk.ops.html?highlight=softmax#cntk.ops.cross_entropy_with_softmax). This takes as input the $N_C$-dimensional vector $z$. As mentioned for our sampled softmax implementation we assume that this z is computed by $ z = W h + b $. In sampled softmax this has to be part of the whole implementation of the criterion.

Below we show the code for `cross_entropy_with_sampled_softmax_and_embedding`. Let’s look at the signature first.

One fundamental difference to the corresponding function in the Python-api (`cross_entropy_with_softmax`) is that in the Python api function the input corresponds to $z$ and must have the same dimension as the target vector, while in cross_entropy_with_full_softmax the input corresponds to our hidden vector $h$ can have any dimension (hidden_dim).
Actually, hidden_dim will be typically much lower than the dimension of the target vector.

We also have some additional parameters `num_samples, sampling_weights, allow_duplicates` that control the random sampling.
Another difference to the api function is that we return a triple (z, cross_entropy_on_samples, error_on_samples).

We will come back to the details of the implementation below.

```python
import numpy as np
import os
import cntk as C

# Creates a subgraph computing cross-entropy with sampled softmax.
def cross_entropy_with_sampled_softmax_and_embedding(
    hidden_vector,            # Node providing hidden input
    target_vector,            # Node providing the expected labels (as sparse vectors)
    num_classes,              # Number of classes
    hidden_dim,               # Dimension of the hidden vector
    num_samples,              # Number of samples to use for sampled softmax
    sampling_weights,         # Node providing weights to be used for the weighted sampling
    allow_duplicates = True, # Boolean flag to control whether to use sampling with replacemement
                              # (allow_duplicates == True) or without replacement.
    ):
    # define the parameters leanabe parameters
    b = C.Parameter(shape = (num_classes, 1), init = C.init_bias_default_or_0)
    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.init_default_or_glorot_uniform)

    # Define the node that generates a set of random samples per minibatch
    # Sparse matrix (num_samples * num_classes)
    sample_selector = C.random_sample(sampling_weights, num_samples, allow_duplicates)

    # For each of the samples we also need the probablity that it in the sampled set.
    inclusion_probs = C.random_sample_inclusion_frequency(sampling_weights, num_samples, allow_duplicates) # dense row [1 * vocab_size]
    log_prior = C.log(inclusion_probs) # dense row [1 * num_classes]

    # Create a submatrix wS of 'weights
    W_sampled = C.times(sample_selector, W) # [num_samples * hidden_dim]
    z_sampled = C.times_transpose(W_sampled, hidden_vector) + C.times(sample_selector, b) - C.times_transpose (sample_selector, log_prior)# [num_samples]

    # Getting the weight vector for the true label. Dimension hidden_dim
    W_target = C.times(target_vector, W) # [1 * hidden_dim]
    z_target = C.times_transpose(W_target, hidden_vector) + C.times(target_vector, b) - C.times_transpose(target_vector, log_prior) # [1]

    z_reduced = C.reduce_log_sum(z_sampled)

    # Compute the cross entropy that is used for training.
    # We don't check whether any of the classes in the random samples conincides with the true label, so it might
    # happen that the true class is counted
    # twice in the normalising demnominator of sampled softmax.
    cross_entropy_on_samples = C.log_add_exp(z_target, z_reduced) - z_target

    # For applying the model we also output a node providing the input for the full softmax
    z = C.times_transpose(W, hidden_vector) + b
    z = C.reshape(z, shape = (num_classes))

    zSMax = C.reduce_max(z_sampled)
    error_on_samples = C.less(z_target, zSMax)
    return (z, cross_entropy_on_samples, error_on_samples)

```

To give a better idea of what the inputs and outputs are and how this all differs from the normal softmax we give below a corresponding function using normal softmax:

```python
# Creates subgraph computing cross-entropy with (full) softmax.
def cross_entropy_with_softmax_and_embedding(
    hidden_vector,  # Node providing hidden input
    target_vector,  # Node providing the expected labels (as sparse vectors)
    num_classes,    # Number of classes
    hidden_dim      # Dimension of the hidden vector
    ):
    # Setup bias and weights
    b = C.Parameter(shape = (num_classes, 1), init = C.init_bias_default_or_0)
    W = C.Parameter(shape = (num_classes, hidden_dim), init = C.init_default_or_glorot_uniform)

    z = C.reshape( C.times_transpose(W, hidden_vector) + b, (1, num_classes))

    # Use cross_entropy_with_softmax
    cross_entropy = C.cross_entropy_with_softmax(z, target_vector)

    zMax = C.reduce_max(z)
    zT = C.times_transpose(z, target_vector)
    error_on_samples = C.less(zT, zMax)

    return (z, cross_entropy, error_on_samples)
```

As you can see the main differences to the api function `cross_entropy_with_softmax` are:
* We include the mapping $ z = W h + b $ into the function.
* We return a triple (z, cross_entropy, error_on_samples) instead of just returning the cross entropy.

## A toy example

To explain how to integrate sampled softmax let us look at a toy example. In this toy example we first transform one-hot input vectors via some random projection into a lower dimensional vector $h$. The modeling task is to reverse this mapping using (sampled) softmax. Well, as already said this is a toy example.

```python
from math import log, exp, sqrt
from cntk.utils import ProgressPrinter
import timeit

# A class with all parameters
class Param:
    # Learning parameters
    learning_rate = 0.03
    minibatch_size = 100
    num_minbatches = 100
    test_set_size = 1000
    momentum_time_constant = 5 * minibatch_size
    reporting_interval = 10
    allow_duplicates = False

    # Parameters for sampled softmax
    use_sampled_softmax = True
    use_sparse = True
    softmax_sample_size = 10

    # Details of data and model
    num_classes = 50
    hidden_dim = 10

    data_sampling_distribution = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)
    softmax_sampling_weights = lambda: np.repeat(1.0 / Param.num_classes, Param.num_classes)

# Creates random one-hot vectors of dimension 'num_classes'.
# Returns a tuple with a list of one-hot vectors, and list with the indices they encode.
def get_random_one_hot_data(num_vectors):
    indices = np.random.choice(
        range(Param.num_classes),
        size=num_vectors,
        p=Param.data_sampling_distribution()).reshape((1, num_vectors))
    list_of_vectors = C.one_hot(indices, Param.num_classes)
    return (list_of_vectors, indices.flatten())

# Create a network that:
# * Transforms the input one hot-vectors with a constant random embedding
# * Applies a linear decoding with parameters we want to learn
def create_model(labels):
    # random projection matrix
    random_data = np.random.normal(scale = sqrt(1.0/Param.hidden_dim), size=(Param.num_classes, Param.hidden_dim)).astype(np.float32)
    random_matrix = C.constant(shape = (Param.num_classes, Param.hidden_dim), value = random_data)

    h = C.times(labels, random_matrix)

    # Connect the latent output to (sampled/full) softmax.
    if Param.use_sampled_softmax:
        sampling_weights = np.asarray(Param.softmax_sampling_weights(), dtype=np.float32)
        sampling_weights.reshape((1, Param.num_classes))
        softmax_input, ce, errs = cross_entropy_with_sampled_softmax_and_embedding(
            h,
            labels,
            Param.num_classes,
            Param.hidden_dim,
            Param.softmax_sample_size,
            Param.softmax_sampling_weights(),
            Param.allow_duplicates)
    else:
        softmax_input, ce, errs = cross_entropy_with_softmax_and_embedding(
            h,
            labels,
            Param.num_classes,
            Param.hidden_dim)

    return softmax_input, ce, errs

def train(do_print_progress):
    labels = C.input_variable(shape = Param.num_classes, is_sparse = Param.use_sparse)
    z, cross_entropy, errs = create_model(labels)

    # Setup the trainer
    learning_rate_schedule = C.learning_rate_schedule(Param.learning_rate, C.UnitType.sample)
    momentum_schedule = C.momentum_as_time_constant_schedule(Param.momentum_time_constant)
    learner = C.momentum_sgd(z.parameters, learning_rate_schedule, momentum_schedule, True)
    trainer = C.Trainer(z, cross_entropy, errs, learner)

    minbatch = 0
    average_cross_entropy = compute_average_cross_entropy(z)
    minbatch_data = [0] # store minibatch values
    cross_entropy_data = [average_cross_entropy] # store cross_entropy values

    if do_print_progress:
        progress_printer = ProgressPrinter(freq=Param.reporting_interval, tag='Training')

    # Run training
    t_total= 0

    # Run training
    for minbatch in range(1,Param.num_minbatches):
        # Specify the mapping of input variables in the model to actual minibatch data to be trained with
        label_data, indices = get_random_one_hot_data(Param.minibatch_size)
        arguments = ({labels : label_data})

        t_start = timeit.default_timer()
        trainer.train_minibatch(arguments)
        t_end = timeit.default_timer()

        t_delta = t_end - t_start
        samples_per_second = Param.minibatch_size / t_delta

        # We ignore the time measurements of the first two minibatches
        if minbatch > 2:
            t_total += t_delta

        # Print the progress using progress printer
        # The prints numbers computed the the sampled softmax criterion
        if do_print_progress:
            progress_printer.update_with_trainer(trainer, with_metric=True)

        # For comparison also print result using the full criterion
        if minbatch % Param.reporting_interval == int(Param.reporting_interval/2):
            # memorize the progress data for plotting
            average_cross_entropy = compute_average_cross_entropy(z)
            minbatch_data.append(minbatch)
            cross_entropy_data.append(average_cross_entropy)

            if do_print_progress:
                print("\nMinbatch=%d Cross-entropy from full softmax = %.3f perplexity = %.3f samples/s = %.1f"
                    % (minbatch, average_cross_entropy, exp(average_cross_entropy), samples_per_second))

    # Number of samples we measured. First two minbatches were ignored
    samples_measured = Param.minibatch_size * (Param.num_minbatches - 2)
    overall_samples_per_second = samples_measured / t_total
    return (minbatch_data, cross_entropy_data, overall_samples_per_second)

def compute_average_cross_entropy(softmax_input):
    vectors, indices = get_random_one_hot_data(Param.test_set_size)
    total_cross_entropy = 0.0
    arguments = (vectors)
    z = softmax_input.eval(arguments).reshape(Param.test_set_size, Param.num_classes)

    for i in range(len(indices)):
        log_p = log_softmax(z[i], indices[i])
        total_cross_entropy -= log_p

    return total_cross_entropy / len(indices)

# Computes log(softmax(z,index)) for a one-dimensional numpy array z in an numerically stable way.
def log_softmax(z,    # numpy array
                index # index into the array
            ):
    max_z = np.max(z)
    return z[index] - max_z - log(np.sum(np.exp(z - max_z)))

np.random.seed(1)

print("start...")
train(do_print_progress = True)
print("done.")
```

    start...

    Minbatch=5 Cross-entropy from full softmax = 3.862 perplexity = 47.567 samples/s = 2957.0
     Minibatch[   1-  10]: loss = 2.351011 * 1000, metric = 82.1% * 1000

    Minbatch=15 Cross-entropy from full softmax = 3.496 perplexity = 32.989 samples/s = 4178.4
     Minibatch[  11-  20]: loss = 2.057759 * 1000, metric = 58.6% * 1000

    Minbatch=25 Cross-entropy from full softmax = 3.065 perplexity = 21.431 samples/s = 4192.3
     Minibatch[  21-  30]: loss = 1.676824 * 1000, metric = 38.8% * 1000

    Minbatch=35 Cross-entropy from full softmax = 2.742 perplexity = 15.519 samples/s = 4736.3
     Minibatch[  31-  40]: loss = 1.468397 * 1000, metric = 25.5% * 1000

    Minbatch=45 Cross-entropy from full softmax = 2.363 perplexity = 10.620 samples/s = 4793.9
     Minibatch[  41-  50]: loss = 1.197275 * 1000, metric = 14.1% * 1000

    Minbatch=55 Cross-entropy from full softmax = 2.139 perplexity = 8.494 samples/s = 4738.2
     Minibatch[  51-  60]: loss = 1.009922 * 1000, metric = 6.0% * 1000

    Minbatch=65 Cross-entropy from full softmax = 1.924 perplexity = 6.848 samples/s = 4748.2
     Minibatch[  61-  70]: loss = 0.886661 * 1000, metric = 5.3% * 1000

    Minbatch=75 Cross-entropy from full softmax = 1.749 perplexity = 5.751 samples/s = 2004.8
     Minibatch[  71-  80]: loss = 0.794563 * 1000, metric = 5.1% * 1000

    Minbatch=85 Cross-entropy from full softmax = 1.563 perplexity = 4.772 samples/s = 3132.4
     Minibatch[  81-  90]: loss = 0.684997 * 1000, metric = 4.0% * 1000

    Minbatch=95 Cross-entropy from full softmax = 1.444 perplexity = 4.238 samples/s = 4206.4
    done.

In the above code we use two different methods to report training progress:
1. Using a function that computes the average cross entropy on full softmax.
2. Using the built-in ProgressPrinter

ProgressPrinter reports how the value of the training criterion changes over time.
In our case the training criterion is cross-entropy from **sampled** softmax.
The same is true for the error rate computed by progress printer, this is computed only for true-class vs sampled-classes and will therefore underestimate the true error rate.

Therefore while ProgressPrinter already gives us some idea how training goes on, if we want to compare the behavior for different sampling strategies (sample size, sampling weights, ...) we should not rely on numbers that are computed only using the sampled subset of classes.

##Importance sampling

Often the we don't have uniform distribution for the classes on the output side. The typical example is when we have words as output classes. A typical example are words where e.g. 'the' will be much more frequent than most others.

In such cases one often uses a non uniform distribution for drawing the samples in sampled softmax but instead increases the sampling weight for the frequent classes. This is also called importane sampling.
In our example the sampling distribution is controlled by the weight array `Param.softmax_sampling_weights`.

As an example let's look at the case where the classes are distrubted according to zipf-distrubtion like:
$$
p[i] \propto \frac{1}{i+5},
$$
actually we use this distribution already in our example.

How does training behavior change if we switch uniform sampling to sampling with the zipfian distribution in sampled softmax?

```python
# We want to lot the data
import matplotlib.pyplot as plt
%matplotlib inline

# Define weights of zipfian distributuion
def zipf(index):
    return 1.0 / (index + 5)

# Use zipifian distribution for the classes
def zipf_sampling_weights():
    return np.asarray([ zipf(i) for i in range(Param.num_classes)], dtype=np.float32)

Param.data_sampling_distribution = lambda: zipf_sampling_weights() / np.sum(zipf_sampling_weights())

print("start...")

# Train using uniform sampling (like before)
np.random.seed(1)
Param.softmax_sampling_weights = lambda: np.repeat(1.0/Param.num_classes, Param.num_classes)
minibatch_data, cross_entropy_data, _ = train(do_print_progress = False)

# Train using importance sampling
np.random.seed(1)
Param.softmax_sampling_weights = zipf_sampling_weights
minibatch_data2, cross_entropy_data2, _ = train(do_print_progress = False)

plt.plot(minibatch_data, cross_entropy_data, 'r--',minibatch_data, cross_entropy_data2, 'b--')
plt.xlabel('number of mini-batches')
plt.ylabel('cross entropy')
plt.show()

```

In the example above we compare uniform sampling (red) vs sampling with the same distribution the classes have (blue).
You will need to experiment to find the best settings for all the softmax parameters.

## What speedups to expect?

The speed difference between full softmax and sampled softmax in terms of training instances depends strongly on the concrete settings, namely
* number of classes
* number of samples used in sampled softmax
* dimension of hiddlen layer input
* minibatch size
* hardware

Also you need to test how much you can reduce sample size without degradation of the result.

Below you can quickly run some tests to get a basic idea

```python
print("start...")

# Set the parameters that affect speed (besides sample size)
Param.num_classes = 50000
Param.hidden_dim = 200
Param.minibatch_size = 256
Param.use_sampled_softmax = False

sample_sizes = [5, 10, 100, 1000, 2000, 10000, 25000]
speed_with_sampled_softmax = []

# Get the speed with sampled softmax for different sizes
for sample_size in sample_sizes:
    print("Measuring speed of sampled softmax for sample size %d ..." % (sample_size))
    Param.use_sampled_softmax = True
    Param.softmax_sample_size = sample_size
    _, _,  samples_per_second = train(do_print_progress = False)
    speed_with_sampled_softmax.append(samples_per_second)

# Get the speed with full softmax
Param.use_sampled_softmax = False
print("Measuring speed of full softmax ...")
_, _,  samples_per_second = train(do_print_progress = False)
speed_without_sampled_softmax = np.repeat(samples_per_second, len(sample_sizes))

# Plot the speed of sampled softmax (blue) as a function of sample sizes
# and compare it to the speed with full softmax (red).
plt.plot(sample_sizes, speed_without_sampled_softmax, 'r--',sample_sizes, speed_with_sampled_softmax, 'b--')
plt.xlabel('softmax sample size')
plt.ylabel('speed: instances / second')
plt.ylim(ymin=0)
plt.show()

```


##Introduction

The Black-Scholes model was first introduced by Fischer Black and Myron Scholes in 1973 in the paper "The Pricing of Options and Corporate Liabilities". Since being published, the model has become a widely used tool by investors and is still regarded as one of the best ways to determine fair prices of options.

The purpose of the model is to determine the price of a vanilla European call and put options (option that can only be exercised at the end of its maturity) based on price variation over time and assuming the asset has a lognormal distribution.

## Assumptions

To determine the price of vanilla European options, several assumptions are made:

* European options can only be exercised at expiration
* No dividends are paid during the option's life
* Market movements cannot be predicted
* The risk-free rate and volatility are constant
* Follows a lognormal distribution

## Non-Dividend Paying Black-Scholes Formula

In Black-Scholes formulas, the following parameters are defined.

* $S$, the spot price of the asset at time $t$
* $T$, the maturity of the option. Time to maturity is defined as $T - t$
* $K$, strike price of the option
* $r$, the risk-free interest rate, assumed to be constant between $t$ and $T$
* $\sigma$, volatility of underlying asset, the standard deviation of the asset returns

#### $N(d)$ is the cumulative distribution of the standard normal variable Z

$$N(d) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^d e^{-\frac{1}{2}x^2} dx$$

$C(S,t)$ is the value at time $t$ of a call option and $P(S,t)$ is the value at time $t$ of a put option.

The Black-Scholes call formula is given as:

$$C(S,t) = SN(d_1) - Ke^{-r(T - t)} N(d_2)$$

The put formula is given:

$$P(S,t) = Ke^{-r(T - t)}N(-d_2) - SN(-d_1)$$

Where:

$$d_1 = \frac{\ln \left(\frac{S}{K} \right) + \left(r + \frac{\sigma^2}{2} \right)(T - t)}{\sigma \sqrt{T - t}}$$

$$d_2 = d_1 - \sigma \sqrt{T - t} = \frac{\ln \left(\frac{S}{K} \right) + \left(r - \frac{\sigma^2}{2}\right)(T - t)}{\sigma \sqrt{T - t}}$$

## Python Implementation of Black-Scholes formula for non-dividend paying options

```python
import numpy as np
import scipy.stats as si
import sympy as sy
import sympy.statistics as systats
```

```python
def euro_vanilla_call(S, K, T, r, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #sigma: volatility of underlying asset

    d1 = (np.log(S / K) + (r + 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))
    d2 = (np.log(S / K) + (r - 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))

    call = (S * si.norm.cdf(d1, 0.0, 1.0) - K * np.exp(-r * T) * si.norm.cdf(d2, 0.0, 1.0))

    return call
```

```python
euro_vanilla_call(50, 100, 1, 0.05, 0.25)
```

    0.027352509369436617

```python
def euro_vanilla_put(S, K, T, r, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #sigma: volatility of underlying asset

    d1 = (np.log(S / K) + (r + 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))
    d2 = (np.log(S / K) + (r - 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))

    put = (K * np.exp(-r * T) * si.norm.cdf(-d2, 0.0, 1.0) - S * si.norm.cdf(-d1, 0.0, 1.0))

    return put
```

```python
euro_vanilla_put(50, 100, 1, 0.05, 0.25)
```

    45.150294959440842

The next function can be called with 'call' or 'put' for the option parameter to calculate the desired option

```python
def euro_vanilla(S, K, T, r, sigma, option = 'call'):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #sigma: volatility of underlying asset

    d1 = (np.log(S / K) + (r + 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))
    d2 = (np.log(S / K) + (r - 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))

    if option == 'call':
        result = (S * si.norm.cdf(d1, 0.0, 1.0) - K * np.exp(-r * T) * si.norm.cdf(d2, 0.0, 1.0))
    if option == 'put':
        result = (K * np.exp(-r * T) * si.norm.cdf(-d2, 0.0, 1.0) - S * si.norm.cdf(-d1, 0.0, 1.0))

    return result
```

```python
euro_vanilla(50, 100, 1, 0.05, 0.25, option = 'put')
```

    45.150294959440842

Sympy implementation for Exact Results

```python
def euro_call_sym(S, K, T, r, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #sigma: volatility of underlying asset

    N = systats.Normal(0.0, 1.0)

    d1 = (sy.ln(S / K) + (r + 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))
    d2 = (sy.ln(S / K) + (r - 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))

    call = (S * N.cdf(d1) - K * sy.exp(-r * T) * N.cdf(d2))

    return call
```

```python
euro_call_sym(50, 100, 1, 0.05, 0.25)
```

    -25.0*erf(1.22379436111989*sqrt(2)) - 22.5614712250357 + 47.5614712250357*erf(1.34879436111989*sqrt(2))

```python
def euro_put_sym(S, K, T, r, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #sigma: volatility of underlying asset

    N = systats.Normal(0.0, 1.0)

    d1 = (sy.ln(S / K) + (r + 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))
    d2 = (sy.ln(S / K) + (r - 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))

    put = (K * sy.exp(-r * T) * N.cdf(-d2) - S * N.cdf(-d1))

    return put
```

Sympy implementation of the above function that enables one to specify a call or put result.

```python
def sym_euro_vanilla(S, K, T, r, sigma, option = 'call'):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #sigma: volatility of underlying asset

    N = systats.Normal(0.0, 1.0)

    d1 = (sy.ln(S / K) + (r + 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))
    d2 = (sy.ln(S / K) + (r - 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))

    if option == 'call':
        result = (S * N.cdf(d1) - K * sy.exp(-r * T) * N.cdf(d2))
    if option == 'put':
        result = (K * sy.exp(-r * T) * N.cdf(-d2) - S * N.cdf(-d1))

    return result
```

```python
sym_euro_vanilla(50, 100, 1, 0.05, 0.25, option = 'put')
```

    -25.0*erf(1.22379436111989*sqrt(2)) + 22.5614712250357 + 47.5614712250357*erf(1.34879436111989*sqrt(2))

## Dividend Paying Black-Scholes Formula

For assets that pay dividends, the Black-Scholes formula is rather similar to the non-dividend paying asset formula; however, a new parameter $q$, is added.

* $S$, the spot price of the asset at time $t$
* $T$, the maturity of the option. Time to maturity is defined as $T - t$
* $K$, strike price of the option
* $r$, the risk-free interest rate, assumed to be constant between $t$ and $T$
* $\sigma$, volatility of underlying asset, the standard deviation of the asset returns
* $q$, the dividend rate of the asset. This is assumed to pay dividends at a continuous rate

In this case, the $q$ parameter is now included in $C(S,t)$ and $P(S,t)$.

$$C(S,t) = Se^{-q(T - t)} N(d_1) - Ke^{-r(T - t)} N(d_2)$$

$$P(S,t) = Ke^{-r(T - t)} N(-d_2) - Se^{-q(T - t)} N(-d_1)$$

Then, $d_1$ and $d_2$ are slightly modified to include the continuous dividends

$$d_1 = \frac{ln \left(\frac{S}{K} \right) + \left(r - q + \frac{\sigma^2}{2} \right)(T - t)}{\sigma \sqrt{T - t}}$$

$$d_2 = d_1 - \sigma \sqrt{T - t} = \frac{ln (\frac{S}{K}) + (r - q - \frac{\sigma^2}{2})(T - t)}{\sigma \sqrt{T - t}}$$

## Python Implementation

```python
def black_scholes_call_div(S, K, T, r, q, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #q: rate of continuous dividend paying asset
    #sigma: volatility of underlying asset

    d1 = (np.log(S / K) + (r - q + 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))
    d2 = (np.log(S / K) + (r - q - 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))

    call = (S * np.exp(-q * T) * si.norm.cdf(d1, 0.0, 1.0) - K * np.exp(-r * T) * si.norm.cdf(d2, 0.0, 1.0))

    return call
```

```python
def black_scholes_put_div(S, K, T, r, q, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #q: rate of continuous dividend paying asset
    #sigma: volatility of underlying asset

    d1 = (np.log(S / K) + (r - q + 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))
    d2 = (np.log(S / K) + (r - q - 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))

    put = (K * np.exp(-r * T) * si.norm.cdf(-d2, 0.0, 1.0) - S * np.exp(-q * T) * si.norm.cdf(-d1, 0.0, 1.0))

    return put
```

Implementation that can be used to determine the put or call option price depending on specification

```python
def euro_vanilla_dividend(S, K, T, r, q, sigma, option = 'call'):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #q: rate of continuous dividend paying asset
    #sigma: volatility of underlying asset

    d1 = (np.log(S / K) + (r - q + 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))
    d2 = (np.log(S / K) + (r - q - 0.5 * sigma ** 2) * T) / (sigma * np.sqrt(T))

    if option == 'call':
        result = (S * np.exp(-q * T) * si.norm.cdf(d1, 0.0, 1.0) - K * np.exp(-r * T) * si.norm.cdf(d2, 0.0, 1.0))
    if option == 'put':
        result = (K * np.exp(-r * T) * si.norm.cdf(-d2, 0.0, 1.0) - S * np.exp(-q * T) * si.norm.cdf(-d1, 0.0, 1.0))

    return result
```

Sympy Implementation of Black-Scholes with Dividend-paying asset

```python
def black_scholes_call_div_sym(S, K, T, r, q, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #q: rate of continuous dividend paying asset
    #sigma: volatility of underlying asset

    N = systats.Normal(0.0, 1.0)

    d1 = (sy.ln(S / K) + (r - q + 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))
    d2 = (sy.ln(S / K) + (r - q - 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))

    call = S * sy.exp(-q * T) * N.cdf(d1) - K * sy.exp(-r * T) * N.cdf(d2)

    return call
```

```python
def black_scholes_call_put_sym(S, K, T, r, q, sigma):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #q: rate of continuous dividend paying asset
    #sigma: volatility of underlying asset

    N = systats.Normal(0.0, 1.0)

    d1 = (sy.ln(S / K) + (r - q + 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))
    d2 = (sy.ln(S / K) + (r - q - 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))

    put = K * sy.exp(-r * T) * N.cdf(-d2) - S * sy.exp(-q * T) * N.cdf(-d1)

    return put
```

Sympy implementation of pricing a European put or call option depending on specification

```python
def sym_euro_vanilla_dividend(S, K, T, r, q, sigma, option = 'call'):

    #S: spot price
    #K: strike price
    #T: time to maturity
    #r: interest rate
    #q: rate of continuous dividend paying asset
    #sigma: volatility of underlying asset

    N = systats.Normal(0.0, 1.0)

    d1 = (sy.ln(S / K) + (r - q + 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))
    d2 = (sy.ln(S / K) + (r - q - 0.5 * sigma ** 2) * T) / (sigma * sy.sqrt(T))

    if option == 'call':
        result = S * sy.exp(-q * T) * N.cdf(d1) - K * sy.exp(-r * T) * N.cdf(d2)
    if option == 'put':
        result = K * sy.exp(-r * T) * N.cdf(-d2) - S * sy.exp(-q * T) * N.cdf(-d1)

    return result
```

```python
from IPython.core.display import HTML
def css():
    style = open("css/custom.css", "r").read()
    return HTML(style)
css()
```

/*
Placeholder for custom user CSS

mainly to be overridden in profile/static/custom/custom.css

This will always be an empty file in IPython
*/
<style>
@import url(http://fonts.googleapis.com/css?family=Lato|Source+Code+Pro|Montserrat:400,700);

body {
	font-family: 'Lato', sans-serif !important;
	font-size: 1.25em !important;
	line-height: 30px !important;
	font-weight: 400 !important;
	color: #8d8d8d !important;
}

#notebook-container {
	-webkit-box-shadow: none;
	box-shadow: none;
}

.rendered_html h1 { font-size: 4rem !important;  }
.rendered_html h2 { font-size: 3rem !important; }
.rendered_html h3 { font-size: 2.5rem !important; }
.rendered_html h4 { font-size: 2rem !important; }
.rendered_html h5 { font-size: 1.5rem !important; }
.rendered_html h6 { font-size: 1.5rem !important; }

.rendered_html h1,
.rendered_html h2,
.rendered_html h3,
.rendered_html h4,
.rendered_html h5,
.rendered_html h6 {
	font-family: 'Montserrat', sans-serif !important;
	font-weight: 300 !important;
	line-height: 1.5em !important;
	color: rgb(221, 153, 51) !important;
}

h1 { font-size: 4.5rem !important; }
h2 { font-size: 4rem !important; }
h3 { font-size: 3.5rem !important; }
h4 { font-size: 3rem !important; }
h5 { font-size: 2.5rem !important; }
h6 { font-size: 2rem !important; }

h1, h2, h3, h4, h5, h6 {
	font-family: 'Montserrat', sans-serif !important;
	color: #e6ae48 !important;
	line-height: 150px !important;
}

p {
	font-family: 'Lato', sans-serif !important;
	font-size: 1.25em !important;
	line-height: 30px !important;
	font-weight: 400 !important;
	color: #8d8d8d !important;
}

code {
	font-family: 'Source Code Pro', sans-serif !important;
	font-size: 1em !important;
}

pre {
	font-family: 'Source Code Pro', sans-serif !important;
	font-size: 1em !important;
}

div.input_area {
	border: none !important;
	background: whitesmoke !important;
}
</style>


# Fast Training of Support Vector Machines for Survival Analysis

This document demonstrates how to use the efficient implementation of *Survival Support Vector Machines* as proposed in

> Pölsterl, S., Navab, N., and Katouzian, A.,
> *Fast Training of Support Vector Machines for Survival Analysis*,
> Machine Learning and Knowledge Discovery in Databases: European Conference,
> ECML PKDD 2015, Porto, Portugal,
> Lecture Notes in Computer Science, vol. 9285, pp. 243-259 (2015)

The source code and installation instructions are available at https://github.com/sebp/scikit-survival.

The main class of interest is ``sksurv.svm.FastSurvivalSVM``, which implements the different optimizers for training
a Survival Support Vector Machine. Training data consists of $n$ triplets $(\mathbf{x}_i, y_i, \delta_i)$, where
$\mathbf{x}_i$ is a $d$-dimensional feature vector, $y_i > 0$ the survival time or time of censoring, and $\delta_i \in \{0,1\}$ the binary event indicator. Using the training data, the objective is to minimize the following function:

\begin{equation}
 \arg \min_{\mathbf{w}, b} \frac{1}{2} \mathbf{w}^T \mathbf{w}+ \frac{\alpha}{2} \left[
 r \sum_{i,j \in \mathcal{P}}
  \max(0, 1 - (\mathbf{w}^T \mathbf{x}_i - \mathbf{w}^T \mathbf{x}_j))^2
+ (1 - r) \sum_{i=0}^n \left( \zeta_{\mathbf{w},b} (y_i, x_i, \delta_i) \right)^2
\right]
\end{equation}

\begin{equation}
\zeta_{\mathbf{w},b} (y_i, \mathbf{x}_i, \delta_i) =
\begin{cases}
  \max(0, y_i - \mathbf{w}^T \mathbf{x}_i - b) & \text{if $\delta_i = 0$,} \\
 y_i - \mathbf{w}^T \mathbf{x}_i - b & \text{if $\delta_i = 1$,} \\
\end{cases}
\end{equation}

\begin{equation}
\mathcal{P} = \{ (i, j)~|~y_i > y_j \land \delta_j = 1 \}_{i,j=1,\dots,n}
\end{equation}

The hyper-parameter $\alpha > 0$ determines the amount of regularization to apply: a smaller value increases the amount of regularization and a higher value reduces the amount of regularization. The hyper-parameter $r \in [0; 1]$ determines the trade-off between the ranking objective and the regresson objective. If $r = 1$ it reduces to the ranking objective, and if $r = 0$ to the regression objective. If the regression objective is used, it is advised to log-transform the survival/censoring time first.

In this example, I'm going to use the ranking objective ($r = 1$) and grid search to determine the best setting for the hyper-parameter $\alpha$.

The class ``sksurv.svm.FastSurvivalSVM`` adheres to interfaces used in [scikit-learn](http://scikit-learn.org) and thus it is possible to combine it with auxiliary classes and functions from scikit-learn. Here, I'm going to use [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.module_selection.GridSearchCV.html) to determine which set hyper-parameters performs best for the Veteran's Lung Cancer data. Since, we require an event indicator $\delta_i$, which is boolean, and the survival/censoring time $y_i$ for training, we have to create a structured array that contains both information.

But first, we have to import the classes we are going to use.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas
import seaborn as sns
from sklearn.model_selection import ShuffleSplit, GridSearchCV

from sksurv.datasets import load_veterans_lung_cancer
from sksurv.column import encode_categorical
from sksurv.metrics import concordance_index_censored
from sksurv.svm import FastSurvivalSVM

sns.set_style("whitegrid")
```

Next, load data of the *Veteran's Administration Lung Cancer Trial* from disk and convert it to numeric values. The data consists of 137 patients and 6 features. The  primary  outcome  measure  was  death (`Status`, `Survival_in_days`).
The original data can be retrieved from http://lib.stat.cmu.edu/datasets/veteran.

Note that it does not matter how you name the fields corresponding to the event indicator and time, as long as the event indicator comes first.

```python
data_x, y = load_veterans_lung_cancer()
x = encode_categorical(data_x)
```

Now, we are essentially ready to start training, but before let's determine what the amount of censoring for this data is and plot the survival/censoring times.

```python
n_censored = y.shape[0] - y["Status"].sum()
print("%.1f%% of records are censored" % (n_censored / y.shape[0] * 100))
```

    6.6% of records are censored

```python
plt.figure(figsize=(9, 6))
val, bins, patches = plt.hist((y["Survival_in_days"][y["Status"]],
                               y["Survival_in_days"][~y["Status"]]),
                              bins=30, stacked=True)
plt.legend(patches, ["Time of Death", "Time of Censoring"])
```

First, we need to create an initial model with default parameters that is subsequently used in the grid search. We are going to use a Red-Black tree to speed up optimization.

```python
estimator = FastSurvivalSVM(optimizer="rbtree", max_iter=1000, tol=1e-6, random_state=0)
```

Next, we define a function for evaluating the performance of models during grid search. We use Harrell's concordance index.

```python
def score_survival_model(model, X, y):
    prediction = model.predict(X)
    result = concordance_index_censored(y['Status'], y['Survival_in_days'], prediction)
    return result[0]
```

The last part of the setup specifies the set of parameters we want to try and how many repetitions of training and testing we want to perform for each parameter setting. In the end, the parameters that on average peformed best across all test sets (200 in this case) are selected. [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) can leverage multiple cores by evaluating multiple parameter settings concurrently (I use 4 jobs in this example).

```python
param_grid = {'alpha': 2. ** np.arange(-12, 13, 2)}
cv = ShuffleSplit(n_splits=200, test_size=0.5, random_state=0)
gcv = GridSearchCV(estimator, param_grid, scoring=score_survival_model,
                   n_jobs=4, iid=False, refit=False,
                   cv=cv)
```

```python
param_grid
```

    {'alpha': array([2.44140625e-04, 9.76562500e-04, 3.90625000e-03, 1.56250000e-02,
            6.25000000e-02, 2.50000000e-01, 1.00000000e+00, 4.00000000e+00,
            1.60000000e+01, 6.40000000e+01, 2.56000000e+02, 1.02400000e+03,
            4.09600000e+03])}

Finally, start the hyper-parameter search. This can take a while since a total of ``13 * 200 = 2600`` fits have to be evaluated.

```python
import warnings
warnings.filterwarnings("ignore", category=UserWarning)
gcv = gcv.fit(x, y)
```

Let's check what is the best average performance across 200 random train/test splits we got and the corresponding hyper-parameters.

```python
gcv.best_score_, gcv.best_params_
```

    (0.718221596570157, {'alpha': 0.00390625})

Finally, we retrieve all 200 test scores for each parameter setting and visualize their distribution by box plots.

```python
def plot_performance(gcv):
    n_splits = gcv.cv.n_splits
    cv_scores = {"alpha": [], "test_score": [], "split": []}
    order = []
    for i, params in enumerate(gcv.cv_results_["params"]):
        name = "%.5f" % params["alpha"]
        order.append(name)
        for j in range(n_splits):
            vs = gcv.cv_results_["split%d_test_score" % j][i]
            cv_scores["alpha"].append(name)
            cv_scores["test_score"].append(vs)
            cv_scores["split"].append(j)
    df = pandas.DataFrame.from_dict(cv_scores)
    _, ax = plt.subplots(figsize=(11, 6))
    sns.boxplot(x="alpha", y="test_score", data=df, order=order, ax=ax)
    _, xtext = plt.xticks()
    for t in xtext:
        t.set_rotation("vertical")
```

```python
plot_performance(gcv)
```

# Kernel Survival Support Vector Machine

This section demonstrates how to use the efficient implementation of *Kernel Survival Support Vector Machines* as proposed in

> Pölsterl, S., Navab, N., and Katouzian, A.,
> *An Efficient Training Algorithm for Kernel Survival Support Vector Machines*
> 4th Workshop on Machine Learning in Life Sciences,
> 23 September 2016, Riva del Garda, Italy

As kernel we are going to use the clinical kernel, because it distinguishes between continuous, ordinal, and nominal attributes.

```python
from sksurv.svm import FastKernelSurvivalSVM
from sksurv.kernels import clinical_kernel
```

To use `GridSearchCV` with a custom kernel, we need to pre-compute the squared kernel matrix and pass it to `GridSearchCV.fit` later. It would also be possible to construct `FastKernelSurvivalSVM` with `kernel="rbf"` (or any other built-in kernel), which does not require pre-computing the kernel matrix.

```python
kernel_matrix = clinical_kernel(data_x)
kssvm = FastKernelSurvivalSVM(optimizer="rbtree", kernel="precomputed", random_state=0)
```

```python
kgcv = GridSearchCV(kssvm, param_grid, score_survival_model,
                    n_jobs=4, iid=False, refit=False,
                    cv=cv)
```

```python
import warnings
warnings.filterwarnings("ignore", category=UserWarning)
kgcv = kgcv.fit(kernel_matrix, y)
```

Now, print the best average concordance index the corresponding parameters.

```python
kgcv.best_score_, kgcv.best_params_
```

    (0.7071426137273039, {'alpha': 0.015625})

Finally, we visualize the distribution of test scores obtained via cross-validation.

```python
plot_performance(kgcv)
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import Symbol, integrate
%matplotlib inline
```

### Smooth local paths
We will use cubic spirals to generate smooth local paths. Without loss of generality, as $\theta$ smoothly changes from 0 to 1, we impose a condition on the curvature as follows

$\kappa = f'(x) = K(x(1-x))^n $

This ensures curvature vanishes at the beginning and end of the path. Integrating, the yaw changes as
$\theta = \int_0^x f'(x')dx'$

With $n = 1$ we get a cubic spiral, $n=2$ we get a quintic spiral and so on. Let us use the sympy package to find the family of spirals

1. Declare $x$ a Symbol

2. You want to find Integral of $f'(x)$

3. You can choose $K$ so that all coefficients are integers

Verify if $\theta(0) = 0$ and $\theta(1) = 1$

```python
K =  30 #choose for cubic/quintic 6 for cubic, 30 for quintic
n =  2 #choose for cubic/ quintic 1 for cubic, 2 for quintic
x =  Symbol('x') #declare as Symbol
print(integrate(K*((x*(1-x))**n), x)) # complete the expression
```

    6*x**5 - 15*x**4 + 10*x**3

```python
x = np.linspace(0, 1, num=100)
thetas = -2*x**3 + 3*x**2
plt.plot(x,thetas,'.')
thetas = 6*x**5 - 15*x**4 + 10*x**3
plt.plot(x,thetas,'g.')
```

```python
#write function to compute a cubic spiral
#input/ output can be any theta
def cubic_spiral(theta_i, theta_f, n=100):
    x = np.linspace(0, 1, num=n)
    # -2*x**3 + 3*x**2
    return (theta_f - theta_i)*(-2*x**3 + 3*x**2) + theta_i

def quintic_spiral(theta_i, theta_f, n=100):
    x = np.linspace(0, 1, num=n)
    # 6*x**5 - 15*x**4 + 10*x**3
    return (theta_f - theta_i)*(6*x**5 - 15*x**4 + 10*x**3) + theta_i
```

### Plotting
Plot cubic, quintic spirals along with how $\theta$ will change when moving in a circular arc. Remember circular arc is when  $\omega $ is constant

```python
plt.figure()
num_pts = 100
plt.plot(np.pi/2*(1-np.linspace(0,1,num_pts)), label='Circular')
plt.plot(cubic_spiral(np.pi/2,0,num_pts), label='Cubic')
plt.plot(quintic_spiral(np.pi/2,0,num_pts),label='Quintic')
plt.grid()
plt.legend()
```

## Trajectory

Using the spirals, convert them to trajectories $\{(x_i,y_i,\theta_i)\}$. Remember the unicycle model

$dx = v\cos \theta dt$

$dy = v\sin \theta dt$

$\theta$ is given by the spiral functions you just wrote. Use cumsum() in numpy to calculate {}

What happens when you change $v$?

```python
v = 1
dt = 0.1
num_pts = 50
thetas = cubic_spiral(np.pi/2,0,num_pts)
x = np.cumsum(v*dt*np.cos(thetas))
y = np.cumsum(v*dt*np.sin(thetas))
# plot trajectories for circular/ cubic/ quintic
plt.figure()
plt.plot(x,y, label='Cubic')

thetas = quintic_spiral(np.pi/2,0,num_pts+2)
x = np.cumsum(v*dt*np.cos(thetas))
y = np.cumsum(v*dt*np.sin(thetas))
plt.plot(x,y, label='Quintic')

thetas = np.pi/2*(1-np.linspace(0,1,num_pts-2))
x = np.cumsum(v*dt*np.cos(thetas))
y = np.cumsum(v*dt*np.sin(thetas))
plt.plot(x,y, label='Circular')
plt.grid()
plt.legend()
```

## Symmetric poses

We have been doing only examples with $|\theta_i - \theta_f| = \pi/2$.

What about other orientation changes? Given below is an array of terminal angles (they are in degrees!). Start from 0 deg and plot the family of trajectories

```python
dt = 0.1
thetas = [15, 30, 45, 60, 90, 120, 150, 180] #convert to radians
plt.figure()
for tf in thetas:
    t = cubic_spiral(0, np.deg2rad(tf),50)
    x = np.cumsum(np.cos(t)*dt)
    y = np.cumsum(np.sin(t)*dt)
    plt.plot(x, y)

# On the same plot, move from 180 to 180 - theta
thetas = [180 - x for x in [15, 30, 45, 60, 90, 120, 150, 180]] #convert to radians
for tf in thetas:
    t = cubic_spiral(np.pi, np.deg2rad(tf),50)
    x = np.cumsum(np.cos(t)*dt)
    y = np.cumsum(np.sin(t)*dt)
    plt.plot(x, y)

plt.grid()
```

Modify your code to print the following for the positive terminal angles $\{\theta_f\}$
1. Final x, y position in corresponding trajectory: $x_f, y_f$
2. $\frac{y_f}{x_f}$ and $\tan \frac{\theta_f}{2}$

What do you notice?
What happens when $v$ is doubled?

```python
dt = 0.1
v = 1/6
thetas = [15, 30, 45, 60, 90, 120, 150, 180] #convert to radians
for tf in thetas:
    t = cubic_spiral(0, np.deg2rad(tf),100)
    x = np.cumsum(v*np.cos(t)*dt)
    y = np.cumsum(v*np.sin(t)*dt)
    print(f'thetaf:{tf:0.1f} xf:{x[-1]:0.3f} yf:{y[-1]:0.3f} yf/xf:{y[-1]/x[-1]:0.3f} tan(theta/2):{np.tan(np.deg2rad(tf/2)):0.3f}')
```

    thetaf:15.0 xf:1.645 yf:0.217 yf/xf:0.132 tan(theta/2):0.132
    thetaf:30.0 xf:1.583 yf:0.424 yf/xf:0.268 tan(theta/2):0.268
    thetaf:45.0 xf:1.482 yf:0.614 yf/xf:0.414 tan(theta/2):0.414
    thetaf:60.0 xf:1.348 yf:0.778 yf/xf:0.577 tan(theta/2):0.577
    thetaf:90.0 xf:1.007 yf:1.007 yf/xf:1.000 tan(theta/2):1.000
    thetaf:120.0 xf:0.624 yf:1.081 yf/xf:1.732 tan(theta/2):1.732
    thetaf:150.0 xf:0.268 yf:1.002 yf/xf:3.732 tan(theta/2):3.732
    thetaf:180.0 xf:-0.000 yf:0.802 yf/xf:-2568534073562342.000 tan(theta/2):16331239353195370.000

These are called *symmetric poses*. With this spiral-fitting approach, only symmetric poses can be reached.

In order to move between any 2 arbitrary poses, you will have to find an intermediate pose that is pair-wise symmetric to the start and the end pose.

What should be the intermediate pose? There are infinite possibilities. We would have to formulate it as an optimization problem. As they say, that has to be left for another time!

```python

```


# Part 1 - Scalars and Vectors

For the questions below it is not sufficient to simply provide answer to the questions, but you must solve the problems and show your work using python (the NumPy library will help a lot!) Translate the vectors and matrices into their appropriate python  representations and use numpy or functions that you write yourself to demonstrate the result or property.

## 1.1 Create a two-dimensional vector and plot it on a graph

```
# import math and graphing libraries
import matplotlib.pyplot as plt
import numpy as np
```

```
# Create a 2-d vector
two_d = [.1,.2]

# Plot 2-d vector:
fig,ax=plt.subplots()
ax.grid()
plt.xlim(-1,1)
plt.ylim(-1,1)
ax.vlines((0,1),1,-1)
ax.hlines((0,1),1,-1)
plt.arrow(0,0,
          two_d[0],
          two_d[1],
          head_width = .02,
          head_length = .02,
          linewidth = 3,
          color = 'blue',
          length_includes_head=True)

plt.show()
```

## 1.2 Create a three-dimensional vecor and plot it on a graph

```
# import 3d plotting tool:
from mpl_toolkits.mplot3d import Axes3D

# Create 3-d vector
three_d = [.2,.3,.8]

vectors = np.array([[0,0,0,three_d[0],three_d[1],three_d[2]]])
zip(*vectors)
X, Y, Z, U, V, W = zip(*vectors)
fig = plt.figure()
ax = fig.add_subplot(111,projection='3d')
ax.quiver(X,Y,Z,U,V,W, length=1,color='green',linewidth=4)
ax.set_xlim(0,1)
ax.set_ylim(0,1)
ax.set_zlim(0,1)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')

plt.show()
```

## 1.3 Scale the vectors you created in 1.1 by $5$, $\pi$, and $-e$ and plot all four vectors (original + 3 scaled vectors) on a graph. What do you notice about these vectors?

```
from math import e, pi
print(e)
print(pi)
```

    2.718281828459045
    3.141592653589793

```
A = np.array(two_d)
B = np.array(two_d)*5
C = np.array(two_d)*e
D = np.array(two_d)*pi

fig,ax = plt.subplots()
ax.grid()
plt.xlim(-1,1)
plt.ylim(-1,1)

plt.arrow(-1, -1, A[0], A[1], head_width=.1, head_length=0.05, color='blue')
plt.arrow(-1, -1, B[0], B[1], head_width=.1, head_length=0.05, color='green')
plt.arrow(-1, -1, C[0], C[1], head_width=.1, head_length=0.05, color = 'red')
plt.arrow(-1, -1, D[0], D[1], head_width=.1, head_length=0.05, color = 'purple')
plt.title('Scaled Vectors')
plt.show()
```

## 1.4 Graph vectors $\vec{a}$ and $\vec{b}$ and plot them on a graph

\begin{align}
\vec{a} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}
\qquad
\vec{b} = \begin{bmatrix} 3 \\4 \end{bmatrix}
\end{align}

```
# create vectors using numpy arrays:
vector_a, vector_b = np.array([5,7]),np.array([3,4])

# plot vectors a and b:
fig, ax = plt.subplots(figsize=(7,7))
ax.grid(color='green')
ax.set_axisbelow(True)
ax.minorticks_on()
ax.grid(which='minor',
        linestyle=':',
        linewidth='0.5',
        color='k')
ax.vlines(0,-5,10)
ax.hlines(0,-5,10)
plt.ylim(-5,10)
plt.xlim(-5,10,)
plt.arrow(0,0,
          vector_a[0],vector_a[1],
          length_includes_head=True,
          head_width=.3,head_length=.5, width=.1,
          color = 'blue')
plt.arrow(0,0,
          vector_b[0],vector_b[1],
          length_includes_head=True,
          head_width=.3,head_length=.5,width=.1,
          color='red')
vector_subtraction = vector_a - vector_b
vector_subtraction
plt.arrow(0,0,
          vector_subtraction[0],vector_subtraction[1],
          length_includes_head=True, head_width=.3,head_length=.5,
          width=.1,color='purple')
plt.show()
```

## 1.5 find $\vec{a} - \vec{b}$ and plot the result on the same graph as $\vec{a}$ and $\vec{b}$. Is there a relationship between vectors $\vec{a} \thinspace, \vec{b} \thinspace \text{and} \thinspace \vec{a-b}$

```
vector_subtraction = vector_a - vector_b
vector_subtraction
```

    array([2, 3])

## 1.6 Find $c \cdot d$

\begin{align}
\vec{c} = \begin{bmatrix}7 & 22 & 4 & 16\end{bmatrix}
\qquad
\vec{d} = \begin{bmatrix}12 & 6 & 2 & 9\end{bmatrix}
\end{align}

```
vector_c = np.array([7,22,4,16])
vector_d = np.array([12,6,2,9])
vector_dot_product = np.dot(vector_c,vector_d)
vector_dot_product
```

    368

##  1.7 Find $e \times f$

\begin{align}
\vec{e} = \begin{bmatrix} 5 \\ 7 \\ 2 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\4 \\ 6 \end{bmatrix}
\end{align}

```
vector_e = np.array([[5],[7],[2]])
vector_f = np.array([[3],[4],[6]])
vector_multiply = vector_e*vector_f
vector_multiply
```

    array([[15],
           [28],
           [12]])

## 1.8 Find $||g||$ and then find $||h||$. Which is longer?

\begin{align}
\vec{g} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 8 \end{bmatrix}
\qquad
\vec{h} = \begin{bmatrix} 3 \\3 \\ 3 \\ 3 \end{bmatrix}
\end{align}

```
g = np.array([[1],[1],[1],[8]])
h = np.array([[3],[3],[3],[3]])

norm_g = np.linalg.norm(g)
print(g,'\nNorm of g: ', norm_g)
norm_h = np.linalg.norm(h)
print(h,'\nNorm of h: ', norm_h)
print('\nNorm of g is longer.')
```

    [[1]
     [1]
     [1]
     [8]]
    Norm of g:  8.18535277187245
    [[3]
     [3]
     [3]
     [3]]
    Norm of h:  6.0

    Norm of g is longer.

# Part 2 - Matrices

## 2.1 What are the dimensions of the following matrices? Which of the following can be multiplied together? See if you can find all of the different legal combinations.
\begin{align}
A = \begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
\qquad
B = \begin{bmatrix}
2 & 4 & 6 \\
\end{bmatrix}
\qquad
C = \begin{bmatrix}
9 & 6 & 3 \\
4 & 7 & 11
\end{bmatrix}
\qquad
D = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\qquad
E = \begin{bmatrix}
1 & 3 \\
5 & 7
\end{bmatrix}
\end{align}

**Dimensions of:**
- A = 3X2
- B = 1X3
- C = 2X3
- D = 3X3
- E = 2X2

```
def dimensions():
    print('Dimension of A: ',A.shape)
    print('Dimension of B: ',B.shape)
    print('Dimension of C: ',C.shape)
    print('Dimension of D: ',D.shape)
    print('Dimension of E: ',E.shape)
dimensions()
```

    Dimension of A:  (3, 2)
    Dimension of B:  (1, 3)
    Dimension of C:  (2, 3)
    Dimension of D:  (3, 3)
    Dimension of E:  (2, 2)

```
A = np.array([[1,2],[3,4],[5,6]])
B = np.array([[2,4,6]])
C = np.array([[9,6,3],[4,7,11]])
D = np.array([[1,0,0],[0,1,0],[0,0,1]])
E = np.array([[1,3],[5,7]])
```

```
def multiDim(matrix_A,matrix_B):
    combo_list = []
    for i in matrix_A:
        for b in matrix_B:
            if i.shape[1] == b.shape[0]:
                combo_list.append(np.matmul(i,b))
    return combo_list

matrix_A = [A,B,C,D,E]
matrix_B = [A,B,C,D,E]
combs = ['AxC','AxE','BxA','BxD','CxA','CxD','DxA','DxD','ExC','ExE']
for i in range(0,10):
    print(combs[i],' = \n',multiDim(matrix_A,matrix_B)[i],'\n')
```

    AxC  =
     [[17 20 25]
     [43 46 53]
     [69 72 81]]

    AxE  =
     [[11 17]
     [23 37]
     [35 57]]

    BxA  =
     [[44 56]]

    BxD  =
     [[2 4 6]]

    CxA  =
     [[ 42  60]
     [ 80 102]]

    CxD  =
     [[ 9  6  3]
     [ 4  7 11]]

    DxA  =
     [[1 2]
     [3 4]
     [5 6]]

    DxD  =
     [[1 0 0]
     [0 1 0]
     [0 0 1]]

    ExC  =
     [[21 27 36]
     [73 79 92]]

    ExE  =
     [[16 24]
     [40 64]]

## 2.2 Find the following products: CD, AE, and BA. What are the dimensions of the resulting matrices? How does that relate to the dimensions of their factor matrices?

```
CD = np.dot(C,D)
AE = np.dot(A,E)
BA = np.dot(B,A)

print('Original Dimensions:')
dimensions()

print('CD dims: \n',CD.shape)
print('AE dims: \n',AE.shape)
print('BA dims: \n',BA.shape)
```

    Original Dimensions:
    Dimension of A:  (3, 2)
    Dimension of B:  (1, 3)
    Dimension of C:  (2, 3)
    Dimension of D:  (3, 3)
    Dimension of E:  (2, 2)
    CD dims:
     (2, 3)
    AE dims:
     (3, 2)
    BA dims:
     (1, 2)

## 2.3  Find $F^{T}$. How are the numbers along the main diagonal (top left to bottom right) of the original matrix and its transpose related? What are the dimensions of $F$? What are the dimensions of $F^{T}$?

\begin{align}
F =
\begin{bmatrix}
20 & 19 & 18 & 17 \\
16 & 15 & 14 & 13 \\
12 & 11 & 10 & 9 \\
8 & 7 & 6 & 5 \\
4 & 3 & 2 & 1
\end{bmatrix}
\end{align}

```
F = np.array([[20,19,18,17],
              [16,15,14,13],
              [12,11,10,9],
              [8,7,6,5],
              [4,3,2,1]])
print('Dimensions of F: ',F.shape)
print('Dimensions of F-transpose: ',F.T.shape)
```

    Dimensions of F:  (5, 4)
    Dimensions of F-transpose:  (4, 5)

# Part 3 - Square Matrices

## 3.1 Find $IG$ (be sure to show your work) 😃

You don't have to do anything crazy complicated here to show your work, just create the G matrix as specified below, and a corresponding 2x2 Identity matrix and then multiply them together to show the result. You don't need to write LaTeX or anything like that (unless you want to).

\begin{align}
G=
\begin{bmatrix}
13 & 14 \\
21 & 12
\end{bmatrix}
\end{align}

```
G = np.array([[13,14],
              [21,12]])
I = np.array([[1,0],
              [0,1]])
np.matmul(I,G)
```

    array([[13, 14],
           [21, 12]])

## 3.2 Find $|H|$ and then find $|J|$.

\begin{align}
H=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\qquad
J=
\begin{bmatrix}
0 & 1 & 2 \\
7 & 10 & 4 \\
3 & 2 & 0
\end{bmatrix}
\end{align}

```
H = np.array([[12,11],
             [7,10]])
J = np.array([[0,1,2],
             [7,10,4],
             [3,2,0]])
print('Determinant of H: \n',np.linalg.det(H),'\n')
print('Determinant of J: \n',np.linalg.det(J),'\n')
```

    Determinant of H:
     43.000000000000014

    Determinant of J:
     -19.999999999999996

## 3.3 Find $H^{-1}$ and then find $J^{-1}$

```
print('Inverse of H: \n',np.linalg.inv(H),'\n')
print('Inverse of J: \n',np.linalg.inv(J))
```

    Inverse of H:
     [[ 0.23255814 -0.25581395]
     [-0.1627907   0.27906977]]

    Inverse of J:
     [[ 0.4  -0.2   0.8 ]
     [-0.6   0.3  -0.7 ]
     [ 0.8  -0.15  0.35]]

## 3.4 Find $HH^{-1}$ and then find $J^{-1}J$. Is $HH^{-1} == J^{-1}J$? Why or Why not?

Please ignore Python rounding errors. If necessary, format your output so that it rounds to 5 significant digits (the fifth decimal place).

**Answer**
> They are not equal, because they are different dimensions.

```
print('H multiplied by its Inverse:\n',np.matmul(H,np.linalg.inv(H)),'\n')
print("J's Inverse multiplied by J:\n",np.matmul(np.linalg.inv(J),J))
```

    H multiplied by its Inverse:
     [[1.00000000e+00 5.55111512e-16]
     [2.22044605e-16 1.00000000e+00]]

    J's Inverse multiplied by J:
     [[ 1.00000000e+00  2.22044605e-16  0.00000000e+00]
     [-1.11022302e-16  1.00000000e+00  0.00000000e+00]
     [-1.66533454e-16 -1.11022302e-16  1.00000000e+00]]

# Stretch Goals:

A reminder that these challenges are optional. If you finish your work quickly we welcome you to work on them. If there are other activities that you feel like will help your understanding of the above topics more, feel free to work on that. Topics from the Stretch Goals sections will never end up on Sprint Challenges. You don't have to do these in order, you don't have to do all of them.

- Write a function that can calculate the dot product of any two vectors of equal length that are passed to it.

```
def calcDot(vec_a,vec_b):
    return np.dot(vec_a,vec_b)

r = np.random.randint(-20,20)
va = np.array([r,r])
vb = np.array([r,r])

calcDot(va,vb)
```

    392

- Write a function that can calculate the norm of any vector

```
def calcNorm(vec):
    return np.linalg.norm(vec)

calcNorm(va)
```

    19.79898987322333

- Prove to yourself again that the vectors in 1.9 are orthogonal by graphing them.
- Research how to plot a 3d graph with animations so that you can make the graph rotate (this will be easier in a local notebook than in google colab)
- Create and plot a matrix on a 2d graph.
- Create and plot a matrix on a 3d graph.
- Plot two vectors that are not collinear on a 2d graph. Calculate the determinant of the 2x2 matrix that these vectors form. How does this determinant relate to the graphical interpretation of the vectors?