# Sample Mean vs. Mode vs. Expected Value

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gamma
from scipy.stats import multivariate_normal
import textwrap
import math
%matplotlib inline

def sampleMeanEx():
	'''sampleMeanEx()'''
	nSamples = (5, 10, 100, 5000)
	a = 3
	b = 1
	for i in range(len(nSamples)):
		fig = plt.figure()
		ax = fig.add_subplot(*[1,1,1])
		draws = np.random.gamma(shape=a,scale=b,size=nSamples[i])
		mode = (a-1)*b
		expectedv = a*b
		samplemean = sum(draws)/len(draws)
		x = np.linspace(gamma.ppf(0.001, a, scale=b), gamma.ppf(0.999, a, scale=b), 100)
		ax.plot(x, gamma.pdf(x, a, scale=b), 'r-', lw=5)
		ax.scatter(draws, np.zeros(len(draws)), c='k')
		myTitle = 'Num Points: ' + str(nSamples[i]) + ' Mode: ' + str("%.2f"%mode) + ' E[x]: ' + str("%.2f"%expectedv) + ' Sample Mean: ' + str("%.2f"%samplemean)
		ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))
		plt.show()

sampleMeanEx()
```

# Conjugate Priors

* Last class we mentioned the concept of *conjugate priors*
* Two distributions have a conjugate prior relationship when the form of the posterior is the same as the form of the prior.
* For example, a Gaussian distribution is a conjugate prior for the mean of a Gaussian as shown in the following:

\begin{eqnarray}
p(\mu |\mathbf{X}) &\propto& p(\mathbf{X}|\mu)p(\mu)\\
&=& \prod_{i=1}^N\mathscr{N}(x_i|\mu, \sigma^2)\mathscr{N}(\mu|\mu_0, \sigma_0^2)\\
&=& \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left\{-\frac{1}{2}\frac{(x_i - \mu)^2}{\sigma^2} \right\}\frac{1}{\sqrt{2\pi\sigma_0^2}}\exp\left\{-\frac{1}{2}\frac{(\mu - \mu_0)^2}{\sigma_0^2} \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}}\frac{1}{\sqrt{2\pi\sigma_0^2}} \exp\left\{ \sum_{i=1}^N \left(-\frac{1}{2}\frac{(x_i - \mu)^2}{\sigma^2}\right) -\frac{1}{2}\frac{(\mu - \mu_0)^2}{\sigma_0^2} \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \sum_{i=1}^N\frac{(x_i - \mu)^2}{\sigma^2} + \frac{(\mu - \mu_0)^2}{\sigma_0^2} \right) \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{\sum_{i=1}^Nx_i^2 - 2\sum_{i=1}^Nx_i\mu + \mu^2N}{\sigma^2} + \frac{\mu^2 - 2\mu\mu_0 + \mu_0^2}{\sigma_0^2} \right) \right\}\\
&=&  \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \mu^2\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right) - 2\mu\left( \frac{\sum_{i=1}^Nx_i}{\sigma^2} + \frac{\mu_0}{\sigma_0^2}  \right) +  \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2} \right) \right\}\\
&=& \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu^2 - 2\mu\left( \frac{\sum_{i=1}^Nx_i}{\sigma^2} + \frac{\mu_0}{\sigma_0^2}  \right)\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)^{-1} \right) \right\}\\
& & \exp\left\{ \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2}  \right\} \nonumber\\
&=& \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu - \left( \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{\sigma^2\sigma_0^2} \right)\left( \frac{N\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2}\right)^{-1}  \right)^2 \right.\\
&+& \left. \frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)
\left(
\left( \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{\sigma^2\sigma_0^2} \right)\left( \frac{N\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2}\right)^{-1} \right)^2  \right\}
\exp\left\{ \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2}  \right\} \nonumber\\
%
&=& \frac{1}{\sqrt{2\pi\sigma^2}\sqrt{2\pi\sigma_0^2}} \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu -  \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }  \right)^2  \right\} \\
& &\exp\left\{  \frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)
\left(\frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }\right)^2 + \frac{\sum_{i=1}^Nx_i^2 }{\sigma^2} + \frac{ \mu_0^2}{\sigma_0^2}  \right\} \nonumber\\
%
&=& C \exp\left\{  -\frac{1}{2}\left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)\left( \mu -  \frac{\sum_{i=1}^Nx_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }  \right)^2  \right\} \\
&\propto& \mathscr{N}\left(\mu\left|\frac{\sum_{i=1}^N x_i\sigma_0^2 + \mu_0\sigma^2}{N\sigma_0^2 + \sigma^2  }, \left( \frac{N}{\sigma^2} + \frac{1}{\sigma_0^2}\right)^{-1}\right.\right)
\end{eqnarray}

* So, as shown above, the form of the posterior is also a Gaussian distribution.

* There are many conjugate prior relationships, e.g., Bernoulli-Beta, Gaussian-Gaussian, Gaussian-Inverse Wishart, Multinomial-Dirichlet

# Bayesian Regression

* Look back our polynomial regression:
\begin{equation}
\min E^{\ast}(\mathbf{w}) = \frac{1}{2}\sum_{n=1}^N\left( y(x_n, \mathbf{w}) - t_n \right)^2 + \frac{\lambda}{2}\left\| \mathbf{w} \right\|_2^2
\end{equation}
This is equivalent to:
\begin{equation}
\max \prod_{n=1}^N \exp\left\{-\frac{1}{2}\left( y(x_n, \mathbf{w}) - t_n \right)^2 \right\}\exp\left\{- \frac{\lambda}{2}\left\| \mathbf{w} \right\|_2^2 \right\}
\end{equation}
* As discussed, the first term is the likelihood and the second term is the prior on the weights
*  These are Gaussian distributions:
\begin{equation}
\mathscr{N}(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}}\exp\left\{ -\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2} \right\}
\end{equation}
* $\sigma^2$ is the variance OR $\frac{1}{\sigma^2}$ is the *precision*
* So, as $\lambda$ gets big, variance gets smaller/tighter.  As $\lambda$ gets small, variance gets larger/wider.

* Previously, we used:
\begin{equation}
y = \sum_{j=0}^M w_j x^j
\end{equation}
*  We can extend this, to make it more general and flexible:
\begin{equation}
y = \sum_{j=0}^M w_j \phi_j(\mathbf{x})
\end{equation}
where $\phi_j(\mathbf{x})$ is a *basis function*
* For example:
    * Basis function we were using previously: $\phi_j(x) = x^j$ (for univariate $x$)
    * Linear Basis Function: $\phi_j(\mathbf{x}) = x_j$
    * Radial Basis Function: $\phi_j(\mathbf{x}) = \exp\left\{ - \frac{(x - \mu_j)^2}{2s_j^2}\right\}$
    * Sigmoidal Basis Function: $\phi_j(\mathbf{x}) = \frac{1}{1 + \exp \left\{ \frac{\mathbf{x} - \mu_j}{s}\right\}}$

* As before:
\begin{equation}
t = y(\mathbf{x},\mathbf{w}) + \epsilon
\end{equation}
* However, now:
\begin{equation}
y = \mathbf{w}^T\boldsymbol{\Phi}(\mathbf{x}) = [w_0, w_1, \ldots, w_M][\phi_0(\mathbf{x}), \phi_1(\mathbf{x}), \ldots, \phi_M(\mathbf{x})]^T
\end{equation}
where $\epsilon \sim \mathscr{N}(\cdot|0, \beta^{-1})$

\begin{equation}
p(t | \mathbf{w}, \beta) = \prod_{n=1}^N \mathscr{N}(t_n | \mathbf{w}^T\boldsymbol{\Phi}(\mathbf{x}_n), \beta^{-1})
\end{equation}
* So, what is the ''trick'' to use to maximize this?
\begin{eqnarray}
\mathscr{L} = \frac{N}{2}\ln\beta - \frac{N}{2}\ln(2\pi) - \frac{1}{2}\beta E(\mathbf{w})\\
\frac{\partial \mathscr{L}}{\partial \mathbf{w}} = \beta \sum_{n=1}^N(t_n - \mathbf{w}^T\boldsymbol{\Phi}(\mathbf{x}_n))\boldsymbol{\Phi}(\mathbf{x}_n)^T = 0
\end{eqnarray}
* This results in:
\begin{equation}
\mathbf{w}_{ML} = \left(\boldsymbol{\Phi}^T\boldsymbol{\Phi} \right)^{-1}\boldsymbol{\Phi}^T\mathbf{t}
\end{equation}
where
\begin{equation}
\boldsymbol{\Phi} = \left[ \boldsymbol{\Phi}(x_1), \boldsymbol{\Phi}(x_2), ...\right]
\end{equation}

* What would you do if you want to include a prior? get the MAP solution?  If assuming zero-mean Gaussian noise, then Regularized Least Squares!

### Bayesian Linear Regression

* Recall: $E_D(\mathbf{w}) + \lambda E_W(\mathbf{w})$ where $\lambda$ is the trade-off regularization parameter
* A simple regularizer (and the one we used previously) is: $E_W(\mathbf{w}) = \frac{1}{2}\mathbf{w}^T\mathbf{w}$
* If we assume zero-mean Gaussian noise: $E_D(\mathbf{w}) = \frac{1}{2}\sum_{n=1}^N \left\{ t_n - \mathbf{w}^T\boldsymbol{\phi}(\mathbf{x}_n) \right\}^2$
* Then, the total error becomes: $\frac{1}{2}\sum_{n=1}^N \left\{ t_n - \mathbf{w}^T\boldsymbol{\phi}(\mathbf{x}_n) \right\}^2 +  \frac{\lambda}{2}\mathbf{w}^T\mathbf{w}$
* We can take the derivative, set it equal to zero and solve for the weights.  When we do, we get:
\begin{equation}
\mathbf{w} = \left( \lambda \mathbf{I} + \boldsymbol{\Phi}^T\boldsymbol{\Phi}\right)^{-1}\boldsymbol{\Phi}^T\mathbf{t}
\end{equation}
* Recall, we can interpret this as:
\begin{eqnarray}
\min_{\mathbf{w}} E^{\ast} &=& \min_{\mathbf{w}}\left\{ E_D(\mathbf{w}) + \lambda E_W(\mathbf{w}) \right\}\\
 &=& \max_{\mathbf{w}}\left\{ -E_D(\mathbf{w}) - \lambda E_W(\mathbf{w}) \right\}\\
 &=& \max_{\mathbf{w}}\exp\left\{ -E_D(\mathbf{w}) - \lambda E_W(\mathbf{w}) \right\}\\
 &=& \max_{\mathbf{w}}\exp\left\{ -E_D(\mathbf{w})\right\} \exp\left\{- \lambda E_W(\mathbf{w}) \right\}\\
 &\propto& \max_{\mathbf{w}} \prod_{n=1}^N \mathscr{N}\left(t\left|\mathbf{w}^T\mathbf{\Phi}(\mathbf{x}_n),  \beta^{-1}\mathbf{I}\right.\right) \mathscr{N}\left(\mathbf{w}\left|\mathbf{m}_0, \mathbf{S}_0\right.\right)\\
 &=& \max_{\mathbf{w}} p\left(\mathbf{t}\left|\mathbf{w}, \mathbf{X}\right.\right)p\left(\mathbf{w}\right)\\
 &\propto& \max_{\mathbf{w}} p(\mathbf{w}|\mathbf{m}_N, \mathbf{S}_N)= \mathscr{N}(\mathbf{w}|\mathbf{m}_N, \mathbf{S}_N)
\end{eqnarray}
where $\mathbf{m}_N = \mathbf{S}_N\left(\mathbf{S}_0\mathbf{m}_0 + \beta\boldsymbol{\Phi}^T\mathbf{t} \right)$ and $\mathbf{S}_N^{-1} = \mathbf{S}_0^{-1} + \beta\boldsymbol{\Phi}^T\boldsymbol{\Phi}$

* What happens with different values of $\beta$ and $\mathbf{S}_0$?

* To simplify, let us assume that $\mathbf{S}_0 = \alpha^{-1}\mathbf{I}$ and $\mathbf{m}_0 = \mathbf{0}$, thus, $\mathbf{m}_N = \beta\mathbf{S}_N\boldsymbol{\Phi}^T\mathbf{t}$ and $\mathbf{S}_N^{-1} = (\alpha^{-1}\mathbf{I})^{-1} + \beta\boldsymbol{\Phi}^T\boldsymbol{\Phi} =  \alpha\mathbf{I} + \beta\boldsymbol{\Phi}^T\boldsymbol{\Phi}$
* This results in the following Log Posterior:
\begin{eqnarray}
\ln p(\mathbf{w}|\mathbf{t}) = - \frac{\beta}{2}\sum_{n=1}^N\left( t_n - \mathbf{w}^T\mathbf{\Phi}(\mathbf{x}_n)\right)^2 - \frac{\alpha}{2}\mathbf{w}^T\mathbf{w} + const
\end{eqnarray}

* Let us suppose we are dealing with 1-D data, $\mathbf{X} = \left\{ x_1, \ldots, x_N \right\}$ and a linear form for y: $y(x, \mathbf{w}) = w_0 + w_1x$

* We are going to generate synthetic data from: $t = -0.3 + 0.5x + \epsilon$ where $\epsilon$ is from zero-mean Gaussian noise. The goal is to estimate the true values $w_0 = -0.3$ and $w_1 = 0.5$.

```python
def innerBasisFunc(dataX):
	return np.vstack([dataX[i,:]@dataX[i,:].T for i in range(dataX.shape[0])])

def prodBasisFunc(dataX):
	return np.vstack([dataX[i,0]*dataX[i,1] for i in range(dataX.shape[0])])

def xorExample():
	'''xorExample()'''
	class1X = np.vstack([np.array([-1,-1])+np.random.normal(0,.1,2 ) for i in range(100)])
	class1X = np.vstack((class1X,np.vstack([np.array([1,1])+np.random.normal(0,.1,2 ) for i in range(100)])))
	class2X = np.vstack([np.array([1,-1])+np.random.normal(0,.1,2 ) for i in range(100)])
	class2X = np.vstack((class2X,np.vstack([np.array([-1,1])+np.random.normal(0,.1,2 ) for i in range(100)])))
	phi1X = prodBasisFunc(class1X)
	phi2X = prodBasisFunc(class2X)

	fig = plt.figure()
	ax = fig.add_subplot(*[1,2,1])
	ax.scatter(class1X[:,0], class1X[:,1], c='r')
	ax.scatter(class2X[:,0], class2X[:,1])
	ax = fig.add_subplot(*[1,2,2])
	ax.scatter(phi1X, np.zeros(phi1X.shape)+0.001, c='r')
	ax.scatter(phi2X, np.zeros(phi2X.shape))
	plt.show()

xorExample()
```

```python
def likelihood_prior_func():
    fig = plt.figure(figsize=(18, 16), dpi= 80, facecolor='w', edgecolor='k')

    #set up variables
    a = 0.5
    b = -0.5
    rangeX = [-1, 1]
    step = 0.025
    X = np.mgrid[rangeX[0]:rangeX[1]:step]
    alpha = 10
    beta = 10
    S0 = (1/alpha)*np.eye(2)
    draw_num = (0,1,2,3,20)

    #initialize prior/posterior and sample data
    sigma = S0
    mean = [.5,-.5]
    draws = np.random.uniform(rangeX[0],rangeX[1],size=draw_num[-1])
    T = a + b*draws + np.random.normal(loc=0, scale=math.sqrt(1/beta))

    for i in range(len(draw_num)):
        if draw_num[i]>0: #skip first image
            #Show data likelihood
            Phi = np.vstack((np.ones(draws[0:draw_num[i]].shape), draws[0:draw_num[i]]))
            t = T[0:draw_num[i]]
            sigma = np.linalg.inv(S0 + beta*Phi@Phi.T)
            mean = beta*sigma@Phi@t

            w0, w1 = np.mgrid[rangeX[0]:rangeX[1]:step, rangeX[0]:rangeX[1]:step]
            p = multivariate_normal(t[draw_num[i]-1], 1/beta)
            out = np.empty(w0.shape)
            for j in range(len(w0)):
                out[j] = p.pdf(w0[j]+w1[j]*draws[draw_num[i]-1])

            ax = fig.add_subplot(*[len(draw_num),3,(i)*3+1])
            ax.pcolor(w0, w1, out)
            ax.scatter(a,b, c='c')
            myTitle = 'data likelihood'
            ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))

        #Show prior/posterior
        w0, w1 = np.mgrid[rangeX[0]:rangeX[1]:step, rangeX[0]:rangeX[1]:step]
        pos = np.empty(w1.shape + (2,))
        pos[:, :, 0] = w0; pos[:, :, 1] = w1
        p = multivariate_normal(mean, sigma)

        ax = fig.add_subplot(*[len(draw_num),3,(i)*3+2])
        ax.pcolor(w0, w1, p.pdf(pos))
        ax.scatter(a,b, c='c')
        myTitle = 'Prior/Posterior'
        ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))

        #Show data space
        for j in range(6):
            w0, w1 = np.random.multivariate_normal(mean, sigma)
            t = w0 + w1*X
            ax = fig.add_subplot(*[len(draw_num),3,(i)*3+3])
            ax.plot(X,t)
            if draw_num[i] > 0:
                ax.scatter(Phi[1,:], T[0:draw_num[i]])
            myTitle = 'data space'
            ax.set_title("\n".join(textwrap.wrap(myTitle, 100)))

    plt.show()

likelihood_prior_func()

```

```python

```

06_BayesianRegression/Lecture 06 - Conjugate Priors and Bayesian Linear Regression.ipynb


# Chapter 1 - Softmax from First Principles

## Language barriers between humans and autonomous systems

If our goal is to help humans and autnomous systems communicate, we need to speak in a common language. Just as humans have verbal and written languages to communicate ideas, so have we developed mathematical languages to communicate information. Probability is one of those languages and, thankfully for us, autonomous systems are pretty good at describing probabilities, even if humans aren't. This document shows one technique for translating a human language (English) into a language known by autonomous systems (probability).

Our translator is something called the **SoftMax classifier**, which is one type of probability distribution that takes discrete labels and translates them to probabilities. We'll show you the details on how to create a softmax model, but let's get to the punchline first: we can decompose elements of human language to represent a partitioning of arbitrary state spaces.

Say, for instance, we'd like to specify the location of an object in two dimensional cartesian coordinates. Our state space is all combinations of *x* and *y*, and we'd like to translate human language into some probability that our target is at a given combination of *x* and *y*. One common tactic humans use to communicate position is range (near, far, next to, etc.) and bearing (North, South, SouthEast, etc.). This already completely partitions our *xy* space: if something is north, it's not south; if it's east, it's not west; and so on.

A softmax model that translates range and bearing into probability in a state space is shown below:

Assuming that *next to* doesn't require a range, we see seventeen different word combinations we can use to describe something's position: two ranges (*nearby* and *far*) for each cardinal and intercardinal direction (eight total), and then one extra label for *next to*. This completely partitions our entire state space $\mathbb{R}^2$.

This range and bearing language is, by its nature, inexact. If I say, "That boat is far north.", you don't have a deterministic notion of exactly where the boat is -- but you have a good sense of where it is, and where it is not. We can represent that sense probabilistically, such that the probability of a target existing at a location described by a range and bearing label is nonzero over the entire state space, but that probability is very small if not in the area most associated with that label.

What do we get from this probabilistic interpretation of the state space? We get a two-way translation between humans and autonomous systems to describe anything we'd like. If our state space is one-dimensional relative velocity (i.e. the derivative of range without bearing), I can say, "She's moving really fast!", to give the autonomous system a probability distribution over my target's velocity with an expected value of, say, 4 m/s. Alternatively, if my autnomous system knows my target's moving at 0.04352 m/s, it can tell me, "Your target is moving slowly." Our labeled partitioning of the state space (that is, our classifier) is the mechanism that translates for us.

## Softmax model construction
The [SoftMax function](http://en.wikipedia.org/wiki/Softmax_function) goes by many names: normalized exponential, multinomial logistic function, log-linear model, sigmoidal function. We use the SoftMax function to develop a classification model for our state space:

\begin{equation}
P(L=i \vert \mathbf{x}) = \frac{e^{\mathbf{w}_i^T \mathbf{x} + b_i}}{\sum_{k=1}^M e^{\mathbf{w}_k^T\mathbf{x} + b_k}}
\end{equation}

Where $L = i$ is our random variable of class labels instantiated as class $i$, $\mathbf{x}$ is our state vector, $\mathbf{w}_i$ is a vector of parameters (or weights) associated with our class $i$, $b_i$ is a bias term for class $i$, and $M$ is the total number of classes.

The terms *label* and *class* require some distinction: a label is a set of words associated with a class (i.e. *far northwest*) whereas a class is a probability distribution over the entire state space. They are sometimes used interchangeably, and the specific meaning should be clear from context.

Several key factors come out of the SoftMax equation:
- The probabilities of all classes for any given point $\mathbf{x}$ sum to 1.
- The probability any single class for any given point $\mathbf{x}$ is bounded by 0 and 1.
- The space can be partitioned into an arbitrary number of classes (with some restrictions about those classes - more on this later).
- The probability of one class for a given point $\mathbf{x}$ is determined by that class' weighted exponential sum of the state vector *relative* to the weighted exponential sums of *all* classes.
- Since the probability of a class is conditioned on $\mathbf{x}$, we can apply estimators such as [Maximum Likelihood](http://en.wikipedia.org/wiki/Maximum_likelihood) to learn SoftMax models.
- $P(L=i \vert \mathbf{x})$ is convex in $\mathbf{w_i}$ for any $\mathbf{x}$.

Let's try to get some intuition about this setup. For a two-dimensional case with state $\mathbf{x} = \begin{bmatrix}x & y\end{bmatrix}^T$, each class $i$ has weights $\mathbf{w}_i = \begin{bmatrix}w_{i,x} & w_{i,y}\end{bmatrix}^T$. Along with the constant bias term $b_i$, we have  one weighted linear function of $x$ and one weighted linear function of $y$. Each class's probability is normalized with respect to the sum of all other classes, so the weights can be seen as a relative scaling of one class over another in any given state. The bias weight increases a class's probability in all cases, the $x$ weight increases the class's probability for greater values of $x$ (and positive weights), and the $y$ weight, naturally, increases the class's probability for greater values of $y$ (and positive weights).

We can get fancy with our state space, having states of the form $\mathbf{x} = \begin{bmatrix}x & y & x^2 & y^2 & 2xy\end{bmatrix}^T$, but we'll build up to states like that. Let's look at some simpler concepts first.

## Class boundaries

For any two classes, we can take the ratio of their probabilities to determine the **odds** of one class instead of the other:

$$
L(i,j) =\frac{P(L=i \vert \mathbf{x})}{P(L=j \vert \mathbf{x})} =
\frac{\frac{e^{\mathbf{w}_i^T \mathbf{x} + b_i}}{\sum_{k=i}^M e^{\mathbf{w}_k^T\mathbf{x} + b_k}}}{\frac{e^{\mathbf{w}_j^T \mathbf{x} + b_{j}}}{\sum_{k=i}^M e^{\mathbf{w}_k^T\mathbf{x} + b_k}}} = \frac{e^{\mathbf{w}_i^T \mathbf{x} + b_i}}{e^{\mathbf{w}_j^T\mathbf{x} + b_j}}
$$

When $L(i,j)=1$, the two classes have equal probability. This doesn't give us a whole lot of insight until we take the **log-odds** (the logarithm of the odds):

\begin{align}
L_{log}(i,j) &=
\log{\frac{P(L=i \vert \mathbf{x})}{P(L=j \vert \mathbf{x})}}
= \log{\frac{e^{\mathbf{w}_i^T \mathbf{x} + b_j}}{e^{\mathbf{w}_j^T\mathbf{x} + b_j}}}
= (\mathbf{w}_i^T\mathbf{x} + b_i)- (\mathbf{w}_j^T\mathbf{x} + b_j) \\
&= (\mathbf{w}_i - \mathbf{w}_j)^T\mathbf{x} + (b_i - b_j)
\end{align}

When $L_{log}(i,j) = \log{L(i,j)} = \log{1} = 0$, we have equal probability between the two classes, and we've also stumbled upon the equation for an n-dimensional affine hyperplane dividing the two classes:

\begin{align}
0 &= (\mathbf{w}_i - \mathbf{w}_j)^T\mathbf{x} + (b_i - b_j) \\
  &= (w_{i,x_1} - w_{j,x_1})x_1 + (w_{i,x_2} - w_{j,x_2})x_2 + \dots + (w_{i,x_n} - w_{j,x_n})x_n  + (b_i - b_j)
\end{align}

This follows from the general definition of an Affine Hyperplane (that is, an n-dimensional flat plane):

$$
a_1x_1 + a_2x_2 + \dots + a_nx_n + b = 0
$$

Where $a_1 = w_{i,x_1} - w_{j,x_1}$, $a_2 = w_{i,x_2} - w_{j,x_2}$, and so on. This gives us a general formula for the division of class boundaries -- that is, we can specify the class boundaries directly, rather than specifying the weights leading to those class boundaries.

### Example
Let's take a step back and look at an example. Suppose I'm playing Pac-Man, and I want to warn our eponymous hero of a ghost approaching him. Let's restrict my language to the four intercardinal directions: NE, SE, SW and NW. My state space is $\mathbf{x} = \begin{bmatrix}x & y\end{bmatrix}^T$ (one term for each cartesian direction in $\mathbb{R}^2$).

In this simple problem, we can expect our weights to be something along the lines of:

$$
\begin{align}
\mathbf{w}_{SW} &= \begin{bmatrix}-1 & -1 \end{bmatrix}^T \\
\mathbf{w}_{NW} &= \begin{bmatrix}-1 & 1 \end{bmatrix}^T \\
\mathbf{w}_{SE} &= \begin{bmatrix}1 & -1 \end{bmatrix}^T \\
\mathbf{w}_{NE} &= \begin{bmatrix}1 & 1 \end{bmatrix}^T \\
\end{align}
$$

If we run these weights in our SoftMax model, we get the following results:

```python
# See source at: https://github.com/COHRINT/cops_and_robots/blob/master/src/cops_and_robots/robo_tools/fusion/softmax.py
import numpy as np
from cops_and_robots.robo_tools.fusion.softmax import SoftMax
%matplotlib inline

labels = ['SW', 'NW', 'SE',' NE']
weights = np.array([[-1, -1],
                    [-1, 1],
                    [1, -1],
                    [1, 1],
                   ])
pacman = SoftMax(weights, class_labels=labels)
pacman.plot(title='Unshifted Pac-Man Bearing Model')
```

Which is along the right path, but needs to be shifted down to Pac-Man's location. Say Pac-Man is approximately one quarter of the map south from the center point, we can bias our model accordingly (assuming a $10m \times 10m$ space):

$$
\begin{align}
b_{SW} &= -2.5\\
b_{NW} &= 2.5\\
b_{SE} &= -2.5\\
b_{NE} &= 2.5\\
\end{align}
$$

```python
biases = np.array([-2.5, 2.5, -2.5, 2.5,])
pacman = SoftMax(weights, biases, class_labels=labels)
pacman.plot(title='Y-Shifted Pac-Man Bearing Model')
```

Looking good! Note that we'd get the same answer had we used the following weights:

$$
\begin{align}
b_{SW} &= -5\\
b_{NW} &= 0\\
b_{SE} &= -5\\
b_{NE} &= 0\\
\end{align}
$$
Because the class boundaries and probability distributions are defined by the *relative differences*.

But this simply shifts the weights in the $y$ direction. How do we go about shifting weights in any state dimension?

Remember that our biases will essentially scale an entire class, so, what we did was scale up the two classes that have a positive scaling for negative $y$ values. If we want to place the center of the four classes in the top-left, for instance, we'll want to bias the NW class less than the other classes.

Let's think of what happens if we use another coordinate system:

$$
\mathbf{x}' = \mathbf{x} + \mathbf{b}
$$

Where $\mathbf{x}'$ is our new state vector and $\mathbf{b}$ are offsets to each state in our original coordinate frame (assume the new coordinate system is unbiased). For example, something like:

$$
\mathbf{x}' = \begin{bmatrix}x & y\end{bmatrix}^T + \begin{bmatrix}2 & -3\end{bmatrix}^T = \begin{bmatrix}x + 2 & y -3\end{bmatrix}^T
$$

Can we represent this shift simply by adjusting our biases, instead of having to redefine our state vector? Assuming we're just shifting the distributions, the probabilities, and thus, the hyperplanes, will simply be shifted as well, so we have:

$$
0 = (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{x}' = (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{x} + (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{b}
$$

Which retains our original state and shifts only our biases. If we distribute the offset $\mathbf{b}$, we can define each class's bias term:

$$
\begin{align}
b_i - b_j &=  (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{b} \\
&= \mathbf{w}_i^T \mathbf{b} - \mathbf{w}_j^T \mathbf{b}
\end{align}
$$

Our bias for each class $i$ in our original coordinate frame is simply $\mathbf{w}_i^T \mathbf{b}$.

Let's try this out with $\mathbf{b} = \begin{bmatrix}2 & -3\end{bmatrix}^T$ (remembering that this will push the shifted origin negatively along the x-axis and positively along the y-axis):

$$
\begin{align}
b_{SW} &= \begin{bmatrix}-1 & -1 \end{bmatrix} \begin{bmatrix}2 \\ -3\end{bmatrix} = 1\\
b_{NW} &= \begin{bmatrix}-1 & 1 \end{bmatrix} \begin{bmatrix}2 \\ -3\end{bmatrix} =-5 \\
b_{SE} &= \begin{bmatrix}1 & -1 \end{bmatrix} \begin{bmatrix}2 \\ -3\end{bmatrix} = 5\\
b_{NE} &= \begin{bmatrix}1 & 1 \end{bmatrix} \begin{bmatrix}2 \\ -3\end{bmatrix} = -1 \\
\end{align}
$$

```python
biases = np.array([1, -5, 5, -1,])
pacman = SoftMax(weights, biases, class_labels=labels)
pacman.plot(title='Shifted Pac-Man Bearing Model')
```

One other thing we can illustrate with this example: how would the SoftMax model change if we multiplied all our weights and biases by 10?
We get:

```python
weights = np.array([[-10, -10],
                    [-10, 10],
                    [10, -10],
                    [10, 10],
                   ])
biases = np.array([10, -50, 50, -10,])
pacman = SoftMax(weights, biases, class_labels=labels)
pacman.plot(title='Steep Pac-Man Bearing Model')
```

Why does this increase in slope happen? Let's investigate.

## SoftMax slope for linear states

The [gradient](http://en.wikipedia.org/wiki/Gradient) of $P(L=i \vert \mathbf{x})$ will give us a function for the slope of our SoftMax model of class $i$. For a linear state space, such as our go-to $\mathbf{x} = \begin{bmatrix}x & y\end{bmatrix}$, our gradient is defined as:

$$
\nabla P(L=i \vert \mathbf{x}) = \nabla \frac{e^{\mathbf{w}_i^T \mathbf{x} + b_i}}{\sum_{k=1}^M e^{\mathbf{w}_k^T\mathbf{x} + b_k}} =
\frac{\partial}{\partial x} \frac{e^{\mathbf{w}_i^T \mathbf{x}}}{\sum_{k=1}^M e^{\mathbf{w}_k^T\mathbf{x}}} \mathbf{\hat{i}} +
\frac{\partial}{\partial y} \frac{e^{\mathbf{w}_i^T \mathbf{x}}}{\sum_{k=1}^M e^{\mathbf{w}_k^T\mathbf{x}}} \mathbf{\hat{j}}
$$

Where $\mathbf{\hat{i}}$ and $\mathbf{\hat{j}}$ are unit vectors in the $x$ and $y$ dimensions, respectively. Given the structure of our equation, the form of either partial derivative will be the same as the other, so let's look at the partial with respect to $x$, using some abused notation:

\begin{align}
\frac{\partial P(L = i \vert \mathbf{x})} {\partial x} &= \frac{d P(L = i \vert x)} {dx} =
\frac{\partial}{\partial x} \frac{e^{w_{i,x}x}}{\sum_{k=1}^M e^{w_{k,x}x}} \\
&= \frac{w_{i,x}e^{w_{i,x}x}\sum_{k=1}^M e^{w_{k,x}x} - e^{w_{i,x}x}(\sum_{k=1}^M w_{k,x}e^{w_{k,x}x})}{(\sum_{k=1}^M e^{w_{k,x}x})^2} \\
&= \frac{w_{i,x}e^{w_{i,x}x}\sum_{k=1}^M e^{w_{k,x}x}}{(\sum_{k=1}^M e^{w_{k,x}x})^2} -
\frac{e^{w_{i,x}x}(\sum_{k=1}^M w_{k,x}e^{w_{k,x}x})}{(\sum_{k=1}^M e^{w_{k,x}x})^2}\\
&= w_{i,x} \left( \frac{e^{w_{i,x}x}}{\sum_{k=1}^M e^{w_{k,x}x}}\right) -
\left( \frac{e^{w_{i,x}x}}{\sum_{k=1}^M e^{w_{k,x}x}}\right)\frac{\sum_{k=1}^M w_{k,x}e^{w_{k,x}x}}{\sum_{k=1}^M e^{w_{k,x}x}}\\
& = P(L = i \vert x) \left(w_{i,x} - \frac{\sum_{k=1}^M w_{k,x}e^{w_{k,x}x}}{\sum_{k=1}^M e^{w_{k,x}x}}\right) \\
& = P(L = i \vert x) \left(w_{i,x} - \sum_{k=1}^M w_{k,x}P(L = k \vert x) \right) \\
\end{align}

Where line 2 was found using the quotient rule. This is still hard to interpret, so let's break it down into multiple cases:

If $P(L = i \vert x) \approx 1$, the remaining probabilities are near zero, thus reducing the impact of their weights, leaving:

$$
\frac{\partial P(L = i \vert \mathbf{x})} {\partial x}
\approx P(L = i \vert x) \left(w_{i,x} - w_{i,x}P(L = i \vert x) \right)
= 0
$$

This makes sense: a dominating probability will be flat.

If $P(L = i \vert x) \approx 0$, we get:

$$
\frac{\partial P(L = i \vert \mathbf{x})} {\partial x}
\approx 0 \left(w_{i,x} - w_{i,x}P(L = i \vert x) \right)
= 0
$$

This also makes sense: a diminished probability will be flat.

We can expect the greatest slope of a [logistic function](http://en.wikipedia.org/wiki/Logistic_function) (which is simply a univariate SoftMax function) to appear at its midpoint  $P(L = i \vert x) = 0.5$. Our maximum slope, then, is:

$$
\frac{\partial P(L = i \vert \mathbf{x})} {\partial x}
= 0.5 \left(w_{i,x} - \sum_{k=1}^M w_{k,x}P(L = k \vert x) \right) \\
= 0.5 \left(w_{i,x} - \sum^M _{\substack{k = 1, \\ k \neq i}} w_{k,x}P(L = k \vert x) - 0.5w_{i,x}\right) \\
= 0.25w_{i,x} - 0.5\sum^M _{\substack{k = 1, \\ k \neq i}} w_{k,x}P(L = k \vert x) \\
$$

In general, we have the following ordinary differential equation:

$$
\frac{\partial P(L = i \vert \mathbf{x})} {\partial \mathbf{x}} =
P(L = i \vert \mathbf{x}) \left(\mathbf{w}_{i} - \sum_{k=1}^M \mathbf{w}_{k}P(L = k \vert \mathbf{x}) \right) \\
$$

NOTE: This section feels really rough, and possibly unnecessary. I need to work on it some more.

## Rotations

Just as we were able to shift our SoftMax distributions to a new coordinate origin, we can apply a [rotation](http://en.wikipedia.org/wiki/Rotation_matrix) to our weights and biases. Let's once again update our weights and biases through a new, rotated, coordinate scheme:

$$
R(\theta)\mathbf{x}' = R(\theta)(\mathbf{x} + \mathbf{b})
$$

As before, we examine the case at the linear hyperplane boundaries:

$$
0 = (\mathbf{w}_i - \mathbf{w}_j)^T \mathbf{x}' = (\mathbf{w}_i - \mathbf{w}_j)^T R(\theta)\mathbf{x} + (\mathbf{w}_i - \mathbf{w}_j)^T R(\theta) \mathbf{b}
$$

Our weights are already defined, so we simply need to multiply them by $R(\theta)$ to find our rotated weights. Let's find our biases:

$$
\begin{align}
b_i - b_j &=  (\mathbf{w}_i - \mathbf{w}_j)^T R(\theta) \mathbf{b} \\
&= \mathbf{w}_i^T R(\theta) \mathbf{b} - \mathbf{w}_j^T R(\theta) \mathbf{b}
\end{align}
$$

So, under rotation, $b_i = \mathbf{w}_i^T R(\theta) \mathbf{b}$.

Let's try this with a two-dimensional rotation matrix using $\theta = \frac{\pi}{4} rad$ and $\mathbf{b} = \begin{bmatrix}2 & -3\end{bmatrix}^T$:

$$
\begin{align}
b_{SW} &= \begin{bmatrix}-1 & -1 \end{bmatrix}
\begin{bmatrix}\frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\end{bmatrix}
\begin{bmatrix}2 \\ -3\end{bmatrix} = -2\sqrt{2} \\
b_{NW} &= \begin{bmatrix}-1 & 1 \end{bmatrix}
\begin{bmatrix}\frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\end{bmatrix}
\begin{bmatrix}2 \\ -3\end{bmatrix} = -3\sqrt{2} \\
b_{SE} &= \begin{bmatrix}1 & -1 \end{bmatrix}
\begin{bmatrix}\frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\end{bmatrix}
\begin{bmatrix}2 \\ -3\end{bmatrix} = 3\sqrt{2} \\
b_{NE} &= \begin{bmatrix}1 & 1 \end{bmatrix}\begin{bmatrix}\frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2}\end{bmatrix}
 \begin{bmatrix}2 \\ -3\end{bmatrix} = 2\sqrt{2} \\
\end{align}
$$

```python
# Define rotation matrix
theta = np.pi/4
R = np.array([[np.cos(theta), -np.sin(theta)],
              [np.sin(theta), np.cos(theta)]])

# Rotate weights
weights = np.array([[-1, -1],
                    [-1, 1],
                    [1, -1],
                    [1, 1],
                   ])
weights = np.dot(weights,R)

# Apply rotated biases
biases = np.array([-2 * np.sqrt(2),
                   -3 * np.sqrt(2),
                   3 * np.sqrt(2),
                   2 * np.sqrt(2),])

pacman = SoftMax(weights, biases, class_labels=labels)
pacman.plot(title='Rotated and Shifted Pac-Man Bearing Model')
```

##Summary

That should be a basic introduction to the SoftMax model. We've only barely scraped the surface of why you might want to use SoftMax models as a tool for aspects of HRI.

Let's move on to [Chapter 2](02_from_normals.ipynb) where we examine a more practical way of constructing SoftMax distributions.

```python
from IPython.core.display import HTML

# Borrowed style from Probabilistic Programming and Bayesian Methods for Hackers
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()

```

<link href='http://fonts.googleapis.com/css?family=Roboto:100,100italic,500,300,300italic,400' rel='stylesheet' type='text/css'>


# Derivation of the radial velocity field expansion

```python
%run notebook_setup.py
```

```python
from sympy import init_printing

init_printing()
```

In this notebook, we'll derive how to express the radial velocity field of a star at an arbitrary orientation on the sky in terms of spherical harmonics. This allows us to use `starry` to model the Rossiter-McLaughlin waveform **analytically**!

First, let's import some stuff. We'll use `sympy` for the derivations.

```python
import numpy as np
from IPython.display import display, Math
import sympy
from sympy import *
from sympy.functions.special.tensor_functions import KroneckerDelta

print("Using sympy version", sympy.__version__)

# Initialize the session
init_session(quiet=True)

# Define our symbols
x, y, z, mu, nu, l, m, j, k, p, q, n, A, B, C, alpha, omeq = symbols(
    "x y z mu nu l m j k p q n A B C alpha \omega_{eq}"
)
```

## The radial velocity field of a star

The polynomial describing the radial component of the velocity field on the projected disk of a star is given by ([Short et al. 2018](https://arxiv.org/abs/1810.09565))

$f(\mathbf{x}, \mathbf{y}, \mathbf{z}) = \omega_{eq}(A\mathbf{x} + B\mathbf{y})(1 - \alpha(-B\mathbf{x} + A\mathbf{y} + C\mathbf{z})^2)$

where

$A = \sin(i)\cos(\lambda)$

$B = \sin(i)\sin(\lambda)$

$C = \cos(i)$

and $i$ and $\lambda$ are the stellar inclination and obliquity, respectively. The constant $\alpha$ is the shear due to differential rotation. We are assuming a simple linear shear according to the following equation:

$\omega = \omega_{eq}(1 - \alpha \sin^2\theta)$,

where $\omega$ is the angular rotational velocity at a point on the surface, $\omega_{eq}$ is the velocity at the equator, and $\theta$ is the polar angle (latitude).

## Let's expand this function

If we expand all the products in $f$, we can write it as a dot product of a polynomial coefficient vector and the polynomial basis defined in Luger et al. (2018):

```python
def poly_basis(n, x, y):
    """Return the n^th term in the polynomial basis."""
    l = Rational(floor(sqrt(n)))
    m = Rational(n - l * l - l)
    mu = Rational(l - m)
    nu = Rational(l + m)
    if nu % 2 == 0:
        i = Rational(mu, 2)
        j = Rational(nu, 2)
        k = Rational(0)
    else:
        i = Rational(mu - 1, 2)
        j = Rational(nu - 1, 2)
        k = Rational(1)
    return x ** i * y ** j * sqrt(1 - x ** 2 - y ** 2) ** k

# Compute the polynomial basis
basis = Matrix([poly_basis(n, x, y) for n in range(16)]).T
```

Here's what the polynomial basis looks like (recall that $z = \sqrt{1 - x^2 - y^2}$):

```python
basis
```

And here's the function we wish to express:

```python
f = (
    omeq
    * (A * x + B * y)
    * (1 - alpha * (-B * x + A * y + C * sqrt(1 - x ** 2 - y ** 2)) ** 2)
)
f
```

We can use `sympy` to figure out the (exact) representation of `f` in the polynomial basis:

```python
def Coefficient(expression, term):
    """Return the coefficient multiplying `term` in `expression`."""
    # Get the coefficient
    coeff = expression.coeff(term)
    # Set any non-constants in this coefficient to zero. If the coefficient
    # is not a constant, this is not the term we are interested in!
    coeff = coeff.subs(sqrt(1 - x ** 2 - y ** 2), 0).subs(x, 0).subs(y, 0)
    return coeff

vec = Matrix([Coefficient(expand(f), term) for term in basis])
vec
```

We can check that dotting this vector with the polynomial basis yields the original function $f$:

```python
simplify(factor(basis.dot(vec)) - f) == 0
```

## Now let's represent it in terms of spherical harmonics

Now that we have the vector of polynomial coefficients `vec`, we want to apply a change-of-basis transformation to figure out their (exact) representation in terms of spherical harmonics.

As in Luger et al. (2018), let's compute the change of basis matrix from polynomials to spherical harmonic coefficients. This is the inverse of the $A_1$ matrix introduced in Luger et al. (2018). Note that it includes the normalization of $\frac{2}{\sqrt{\pi}}$ used internally by `starry`.

```python
def SA(l, m):
    """A spherical harmonic normalization constant."""
    return sqrt(
        (2 - KroneckerDelta(m, 0))
        * (2 * l + 1)
        * factorial(l - m)
        / (4 * pi * factorial(l + m))
    )

def SB(l, m, j, k):
    """Another spherical harmonic normalization constant."""
    try:
        ratio = factorial(Rational(l + m + k - 1, 2)) / factorial(
            Rational(-l + m + k - 1, 2)
        )
    except ValueError:
        ratio = 0
    res = (
        2 ** l
        * Rational(
            factorial(m),
            (factorial(j) * factorial(k) * factorial(m - j) * factorial(l - m - k)),
        )
        * ratio
    )
    return simplify(res)

def SC(p, q, k):
    """Return the binomial theorem coefficient `C`."""
    res = factorial(Rational(k, 2)) / (
        factorial(Rational(q, 2))
        * factorial(Rational(k - p, 2))
        * factorial(Rational(p - q, 2))
    )
    return simplify(res)

def Y(l, m, x, y):
    """Return the spherical harmonic of degree `l` and order `m`."""
    res = 0
    z = sqrt(1 - x ** 2 - y ** 2)
    if m >= 0:
        for j in range(0, m + 1, 2):
            for k in range(0, l - m + 1, 2):
                for p in range(0, k + 1, 2):
                    for q in range(0, p + 1, 2):
                        res += (
                            (-1) ** ((j + p) // 2)
                            * SA(l, m)
                            * SB(l, m, j, k)
                            * SC(p, q, k)
                            * x ** (m - j + p - q)
                            * y ** (j + q)
                        )
            for k in range(1, l - m + 1, 2):
                for p in range(0, k, 2):
                    for q in range(0, p + 1, 2):
                        res += (
                            (-1) ** ((j + p) // 2)
                            * SA(l, m)
                            * SB(l, m, j, k)
                            * SC(p, q, k - 1)
                            * x ** (m - j + p - q)
                            * y ** (j + q)
                            * z
                        )
    else:
        for j in range(1, abs(m) + 1, 2):
            for k in range(0, l - abs(m) + 1, 2):
                for p in range(0, k + 1, 2):
                    for q in range(0, p + 1, 2):
                        res += (
                            (-1) ** ((j + p - 1) // 2)
                            * SA(l, abs(m))
                            * SB(l, abs(m), j, k)
                            * SC(p, q, k)
                            * x ** (abs(m) - j + p - q)
                            * y ** (j + q)
                        )
            for k in range(1, l - abs(m) + 1, 2):
                for p in range(0, k, 2):
                    for q in range(0, p + 1, 2):
                        res += (
                            (-1) ** ((j + p - 1) // 2)
                            * SA(l, abs(m))
                            * SB(l, abs(m), j, k)
                            * SC(p, q, k - 1)
                            * x ** (abs(m) - j + p - q)
                            * y ** (j + q)
                            * z
                        )

    return res

def p_Y(l, m, lmax):
    """Return the polynomial basis representation of the spherical harmonic `Y_{lm}`."""
    ylm = Y(l, m, x, y)
    res = [ylm.subs(sqrt(1 - x ** 2 - y ** 2), 0).subs(x, 0).subs(y, 0)]
    for n in range(1, (lmax + 1) ** 2):
        res.append(Coefficient(ylm, poly_basis(n, x, y)))
    return res

def A1(lmax, norm=2 / sqrt(pi)):
    """Return the change of basis matrix A1. The columns of this matrix are given by `p_Y`."""
    res = zeros((lmax + 1) ** 2, (lmax + 1) ** 2)
    n = 0
    for l in range(lmax + 1):
        for m in range(-l, l + 1):
            res[n] = p_Y(l, m, lmax)
            n += 1
    return res * norm
```

We can now evaluate the change of basis matrix from spherical harmonic coefficients to polynomials, $A_1$ for $l_\mathrm{max} = 3$. We then take the inverse to go from polynomial coeffiecients to $Y_{lm}$ coefficients:

```python
M = Matrix(A1(3)).inv()
M
```

## The end result

We can finally compute the spherical harmonic coefficients of the function $f$:

```python
ycoeffs = simplify(M * vec)
ycoeffs
```

Note that this is the **exact** spherical harmonic representation of the function `f`.

## Interactive visualization

Below you can interactively see how different velocity profiles affect the Rossiter-McLaughlin waveform. We coded up a simple Jupyter widget to visualize the RM effect with `starry`. Try to find the parameters that best fit the measured RV data for the hot jupiter host HD 189733!

```python
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from ipywidgets import Layout
import matplotlib.pyplot as plt
import starry

starry.config.lazy = False
starry.config.quiet = True

vslider = widgets.FloatSlider(
    value=5.0,
    min=0.1,
    max=10.0,
    step=0.01,
    description=r"$v_\mathrm{eq}$ [km / s]:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".2f",
    layout=Layout(width="40%"),
)

oslider = widgets.FloatSlider(
    value=-0.50,
    min=-90,
    max=90.0,
    step=0.1,
    description=r"$\lambda$ [deg]:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".1f",
    layout=Layout(width="40%"),
)

islider = widgets.FloatSlider(
    value=80,
    min=1,
    max=179.0,
    step=0.1,
    description=r"$i$ [deg]:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".1f",
    layout=Layout(width="40%"),
)

aslider = widgets.FloatSlider(
    value=0.65,
    min=0,
    max=1.0,
    step=0.01,
    description=r"$\alpha$:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".2f",
    layout=Layout(width="40%"),
)

u1slider = widgets.FloatSlider(
    value=1.50,
    min=0.0,
    max=2.0,
    step=0.01,
    description=r"$u_1$:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".2f",
    layout=Layout(width="40%"),
)

u2slider = widgets.FloatSlider(
    value=-0.54,
    min=-1.0,
    max=1.0,
    step=0.01,
    description=r"$u_2$:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".2f",
    layout=Layout(width="40%"),
)

yslider = widgets.FloatSlider(
    value=-0.66,
    min=-1.0,
    max=1.0,
    step=0.01,
    description=r"$b$:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".2f",
    layout=Layout(width="40%"),
)

rslider = widgets.FloatSlider(
    value=0.158,
    min=0.01,
    max=0.5,
    step=0.001,
    description=r"$r / R_\star$:",
    disabled=False,
    continuous_update=False,
    orientation="horizontal",
    readout=True,
    readout_format=".3f",
    layout=Layout(width="40%"),
)

# RV data for HD 189733 from M. Bedell, corrected for the baseline
xo_189, rv_189 = np.array(
    [
        (-1.610613874961608571e00, -9.981225587125663878e-01),
        (-1.516819480458127867e00, -9.714511071633751271e-01),
        (-1.424288949323776032e00, 6.320045379397924989e-01),
        (-1.327854659861453346e00, 5.854148371854535071e-01),
        (-1.232414426212717684e00, -1.336544425081228837e00),
        (-1.135673041276108242e00, 8.028719313260808121e-01),
        (-1.039663090115350963e00, -7.792463171881536255e-01),
        (-9.455579835532572641e-01, 1.453737600349995773e00),
        (-8.467048139722169742e-01, 4.792210501625959296e00),
        (-7.477458584637551864e-01, 1.599068367822837899e01),
        (-6.553772659628724906e-01, 2.568976397998631001e01),
        (-5.591570578315693263e-01, 3.354680499304413388e01),
        (-4.625798890680289377e-01, 3.502869252636628516e01),
        (-3.650748292997558764e-01, 3.120010282560906489e01),
        (-2.704380165973445105e-01, 2.425808674314976088e01),
        (-1.722747673202078555e-01, 1.747718457640439738e01),
        (-7.671271240261887114e-02, 7.868697041703853756e00),
        (1.885820039767528364e-02, -1.486790492997897672e00),
        (1.155924921815155992e-01, -1.257290268435099279e01),
        (2.120216221213046814e-01, -2.145286194661639456e01),
        (3.092993853335966881e-01, -2.891529266112229379e01),
        (4.068310542953758047e-01, -3.485288236187807343e01),
        (5.022774536703960013e-01, -3.813336989657727827e01),
        (5.965030311267133234e-01, -3.469222671746319975e01),
        (6.955993594669751534e-01, -2.537275959802173020e01),
        (7.890948706217063569e-01, -1.200930395291834429e01),
        (8.859815670714117353e-01, -4.065734667423384963e00),
        (9.792870073717533375e-01, -1.655279022319518845e00),
        (1.078846540640843621e00, -2.328277573282321100e00),
        (1.172781349728483358e00, -1.592293655745379510e00),
        (1.266294524861562687e00, -8.531562584666971816e-01),
        (1.367439231357534979e00, -7.712511936460941797e-01),
        (1.658284285150540383e00, -1.439270159869920462e00),
        (1.761221193029112841e00, -1.678779727093569818e00),
        (1.853084956802548033e00, -2.067330139152545598e00),
        (1.948154374912242304e00, -1.369754796494817128e00),
        (2.037885209964362065e00, 1.118637971248972462e00),
        (2.137297896820141041e00, -5.354630210913455812e-01),
        (2.226274259361801011e00, 1.665095065392961260e00),
        (2.322326451882745779e00, 1.192561909507276141e00),
    ]
).T

# Create the global starry maps
map = starry.Map(ydeg=0, udeg=2, rv=True)

# Force compile some functions
map.rv()
map.render()

def visualize_func(veq=1, inc=90, obl=0, alpha=0, u1=0, u2=0, yo=0, ro=0.1):
    """Interactive visualization of the RM effect."""
    # Map resolution for plotting
    res = 300

    # Set the map coefficients
    map[1] = u1
    map[2] = u2
    map.inc = inc
    map.obl = obl
    map.alpha = alpha
    map.veq = veq * 1e3

    # Check if LD is physical
    if (u1 + u2) > 1 or (u1 + 2 * u2) < 0 or u1 < 0:
        u1slider.style.handle_color = "#FF0000"
        u2slider.style.handle_color = "#FF0000"
    else:
        u1slider.style.handle_color = "#FFFFFF"
        u2slider.style.handle_color = "#FFFFFF"

    # Plot the brightness-weighted velocity field
    img = map.render(rv=True, res=res) * (np.pi / 1.0e3)
    fig = plt.figure(figsize=(15, 8))
    axim = plt.axes((0, 0.05, 0.3, 0.8))
    axcb = plt.axes((0, 0.85, 0.3, 0.03))
    axrm = plt.axes((0.4, 0.20, 0.6, 0.5))
    im = axim.imshow(
        img, cmap="RdBu_r", origin="lower", vmin=-veq, vmax=veq, extent=(-1, 1, -1, 1)
    )
    cb = plt.colorbar(im, orientation="horizontal", cax=axcb)
    cb.ax.set_xlabel("Radial velocity [km / s]")
    axim.contour(
        img,
        origin="lower",
        levels=np.linspace(-veq, veq, 20),
        colors=["k" for i in range(20)],
        alpha=0.25,
        extent=(-1, 1, -1, 1),
    )
    axim.axis("off")
    axim.set_aspect(1)
    axim.axhline(yo, color="k", alpha=0.5)
    axim.axhline(yo + 0.5 * ro, color="k", ls="--", alpha=0.5)
    axim.axhline(yo - 0.5 * ro, color="k", ls="--", alpha=0.5)

    # Compute the RM effect amplitude
    xo = np.linspace(-1 - 2 * ro, 1 + 2 * ro, 1000)
    RM = map.rv(xo=xo, yo=yo, ro=ro)

    # Plot it
    axrm.plot(xo, RM)
    axrm.set_xlabel(r"Occultor x position [$R_\star$]", fontsize=16)
    axrm.set_ylabel("Radial velocity [m /s]", fontsize=16)
    axrm.set_title("The Rossiter-McLaughlin effect", fontsize=20)
    axrm.plot(xo_189, rv_189, ".")

def visualize():
    return interact(
        visualize_func,
        veq=vslider,
        inc=islider,
        obl=oslider,
        alpha=aslider,
        u1=u1slider,
        u2=u2slider,
        yo=yslider,
        ro=rslider,
    )

visualize();
```

## One possible solution

The following values should get you a pretty good fit. They are close to the means of the posterior distributions when we do a full MCMC fit of the dataset, but they are probably not the "true" values.

| Parameter | Value |
| --- | --- |
| $v_{eq}$  | 5.00  |
|$\lambda$  | -0.50 |
|$i$        | 80.0  |
|$\alpha$   | 0.65  |
|$u_1$      | 1.50  |
|$u_2$      | -0.54 |
|$b$        | -0.66 |
|$r/R_\star$|  0.158|


# Hamiltonian Monte Carlo (HMC)

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from functools import partial
```

## Use of auxiliary variables

Slice sampling is a simple MCMC algorithm that introduces the idea of auxiliary variables. The motivation for slice sampling is that if we can sample uniformly from the region under the graph of the target distribution, we will have random samples from the target distribution. In the univariate case, the algorithm is as follows

- start with some $x$ where $p(x) \ne 0$
- repeat
    - sample $y$ (auxiliary variable) uniformly from 0 to $f(x)$
    - draw a horizontal line at $y$ within $p(x)$ (this may consist of multiple intervals)
    - sample $x$ from the horizontal segments

The auxiliary $y$ variable allows us to sample $(x, y)$ points that are in the region under the graph of the target distribution. Only the $x$ variable is used for the Monte Carlo samples - the $y$ variables are simply discarded. This works because the joint distribution is constructed so that it factors $p(x, y) = p(y \mid x) p(x)$ and so projecting leaves just $p(x)$. The slice sampler is a Markov Chain Monte Carlo method since the next $(x, y)$ position depends only on the current position. Like Gibbs sampling, there is no tuning process and all proposals are accepted. For slice sampling, you either need the inverse distribution function or some way to estimate it. Later we will see that Hamiltonian Monte Carlo also uses auxiliary variables to generate a new proposal in an analogous way.

A toy example illustrates the process - Suppose we want to draw random samples from the posterior distribution $\mathcal{N}(0, 1)$ using slice sampling

Start with some value $x$
- sample $y$ from $\mathcal{U}(0, f(x))$ - this is the horizontal "slice" that gives the method its name
- sample the next $x$ from $f^{-1}(y)$ - this is typically done numerically
- repeat

Will sketch picture in class to show what is going on.

### A simple slice sampler example

```python
import scipy.stats as stats
```

```python
dist = stats.norm(5, 3)
w = 0.5
x = dist.rvs()

niters = 1000
xs = []
while len(xs) < niters:
    y = np.random.uniform(0, dist.pdf(x))
    lb = x
    rb = x
    while y < dist.pdf(lb):
        lb -= w
    while y < dist.pdf(rb):
        rb += w
    x = np.random.uniform(lb, rb)
    if y > dist.pdf(x):
        if np.abs(x-lb) < np.abs(x-rb):
            lb = x
        else:
            lb = y
    else:
        xs.append(x)
```

```python
plt.hist(xs, 20)
pass
```

Notes on the slice sampler:

- the slice may consist of disjoint pieces for multimodal distributions
- the slice can be a rectangular hyperslab for multivariable posterior distributions
- sampling from the slice (i.e. finding the boundaries at level $y$) is non-trivial and may involve iterative rejection steps - see figure below (from Wikimedia) for a typical approach - the blue bars represent disjoint pieces of the true slice through a bimodal distribution and the black lines are the proposal distribution approximating the true slice

## Hamiltonian Monte Carlo (HMC)

HMC uses an auxiliary variable corresponding to the momentum of particles in a potential energy well to generate proposal distributions that can make use of gradient information in the posterior distribution. For reversibility to be maintained, the total energy of the particle has to be conserved - hence we are interested in Hamiltonian systems. The main attraction of HMC is that it works much better than other methods when variables of interest are highly correlated. Because we have to solve problems involving momentum, we need to understand how to numerically solve differential equations in a way that is both accurate (i.e. second order) and preserves total energy (necessary for a Hamiltonian system).

Example adapted from [MCMC: Hamiltonian Monte Carlo (a.k.a. Hybrid Monte Carlo)](https://theclevermachine.wordpress.com/2012/11/18/mcmc-hamiltonian-monte-carlo-a-k-a-hybrid-monte-carlo/)

### Hamiltonian systems

In a Hamiltonian system, we consider particles with position $x$ and momentum (or velocity if we assume unit mass) $v$. The total energy of the system $H(x, v) = K(v) + U(x)$, where $K$ is the kinetic energy and $U$ is the potential energy, is conserved. Such a system satisfies the following Hamiltonian equations

$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta H}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta H}{dx}
\end{align}
$$

Since $K$ depends only on $v$ and $U$ depends only on $x$, we have
$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta K}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta U}{dx}
\end{align}
$$

#### Harmonic oscillator

We will consider solving a classical Hamiltonian system - that of a undamped spring governed by the second order differential equation

$$
x'' + x = 0
$$

We convert this to two first order ODEs by using a dummy variable $x' = v$ to get

$$
\begin{align}
x' &= v \\
v' &= -x
\end{align}
$$

From the Hamiltonian equations above, this is equivalent to a system with kinetic energy $K(v) = \frac{1}{2}v^2$ and potential energy $U(x) = \frac{1}{2}x^2$.

Writing in matrix form,

$$
A = \pmatrix{ x' \\ v' } = \pmatrix{0 & 1 \\ -1 & 0} \pmatrix{x \\ v}
$$

and in general, for the state vector $x$,

$$
x' = Ax
$$

We note that $A$ is anti- or skew-symmetric ($A^T = -A$), and hence has purely imaginary eigenvalues. Solving $|A - \lambda I = 0$, we see that the eigenvalues and eigenvectors are $i, \pmatrix{1\\i}$ and $-i, \pmatrix{1\\-i}$. Since the eigenvalues are pure imaginary, we see that the solution for the initial conditions $(x,v) = (1, 0)$ is $x(t) = e^{it}$ and the orbit just goes around a circle with a period of $2\pi$, neither growing nor decaying. Another weay of seeing this is that the Hamiltonian $H(u, v)$ or sum of potential ($U(x)) = \frac{1}{2}x^2$) and kinetic energy ($K(v) = \frac{1}{2}v^2$) is constant, i.e. in vector form, $(x^T x) = \text{constant}$.

### Finite difference methods

We want to find a finite difference approximation to $u' = Au$ that is **accurate** and **preserves total energy**. If total energy is not preserved, the orbit will either spiral in towards zero or outwards away from the unit circle. If the accuracy is poor, the orbit will not be close to its starting value after $t = 2\pi$. This gives us an easy way to visualize how good our numerical scheme is. We can also compare the numerical scheme to the Taylor series to evaluate its accuracy.

#### Forward Euler

The simplest finite difference scheme for integrating ODEs is the forward Euler

$$
\frac{u_{n+1} - u_n}{\Delta t} = A u_n
$$

Rearranging terms, we get

$$
u_{n+1} = u_n + \Delta t A u_n = \left( I + \Delta t A \right) u_n
$$

Since the eigenvalues of $A$ are $\pm i$, we see that the eigenvalues of the forward Euler matrix are $1 \pm i$. Since the absolute value of the eigenvalues is greater than 1, we expect **growing** solutions - i.e. the solution will spiral away from the unit circle.

```python
import scipy.linalg as la
```

```python
def f_euler(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u = u + dt * A @ u
        orbit[i] = u
    return orbit
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
orbit = f_euler(A, u, N)
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.3600318484671193

##### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

##### Accuracy and conservation of energy

We can see that forward Euler is not very accurate and also does not preserve energy since the orbit spirals away from the unit circle.

#### The trapezoidal method

The trapezoidal method uses the following scheme

$$
\frac{u_{n+1} - u_n}{\Delta t} = \frac{1}{2}  ( A u_{n+1} + A u_{n})
$$

This is an implicit scheme (because $u_{n+1}$ appears on the RHS) whose solution is

$$
u_{n+1} = \left(I - \frac{\Delta t}{2} A \right)^{-1} \left(I + \frac{\Delta t}{2} A \right) u_{n} = B u_n
$$

By inspection, we see that the eigenvalues are the complex conjugates of

$$
\frac{1 + \frac{\Delta t}{2} i}{1 - \frac{\Delta t}{2} i}
$$

whose absolute value is 1 - hence, energy is conserved. If we expand the matrix $B$ using the geometric series and compare with the Taylor expansion, we see that the trapezoidal method has local truncation error $O(h^3)$ and hence accuracy $O(h^2)$, where $h$ is the time step.

```python
def trapezoidal(A, u, N):
    p = len(u)
    orbit = np.zeros((N,p))

    dt = 2*np.pi/N
    for i in range(N):
        u = la.inv(np.eye(p) - dt/2 * A) @ (np.eye(p) + dt/2 * A) @ u
        orbit[i] = u
    return orbit
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
orbit = trapezoidal(A, u, N)
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.005039305635733781

##### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

#### The leapfrog method

The leapfrog method uses a second order difference to update $u_n$. The algorithm simplifies to the following explicit scheme:

- First take one half-step for v
- Then take a full step for u
- Then take one final half step for v

It performs as well as the trapezoidal method, with the advantage of being an explicit scheme and cheaper to calculate, so the leapfrog method is used in HMC.

```python
def leapfrog(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u[1] = u[1] + dt/2 * A[1] @ u
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt/2 * A[1] @ u
        orbit[i] = u
    return orbit
```

##### If we don't care about the intermediate steps, it is more efficient to just take 1/2 steps at the beginning and end

```python
def leapfrog2(A, u, N):
    dt = 2*np.pi/N

    u[1] = u[1] + dt/2 * A[1] @ u
    for i in range(N-1):
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt * A[1] @ u

    u[0] = u[0] + dt * A[0] @ u
    u[1] = u[1] + dt/2 * A[1] @ u
    return u
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
```

```python
orbit = leapfrog(A, u, N)
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.0025229913808033464

##### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

### From Hamiltonians to probability distributions

The physical analogy considers the negative log likelihood of the target distribution $p(x)$ to correspond to a potential energy well, with a collection of particles moving on the surface of the well. The state of each particle is given only by its position and momentum (or velocity if we assume unit mass for each particle). In a Hamiltonian system, the total energy $H(x,, v) = U(x) + K(v)$ is conserved. From statistical mechanics, the probability of each state is related to the total energy of the system

$$
\begin{align}
p(x, v) & \propto e^{-H(x, v)} \\
&= e^{-U(x) - K(v)} \\
&= e^{-P(x)}e^{-K(v)} \\
& \propto p(x) \, p(v)
\end{align}
$$

Since the joint distribution factorizes $p(x, v) = p(x)\, p(v)$, we can select an initial random $v$ for a particle, numerically integrate using a finite difference method such as the leapfrog and then use the updated $x^*$ as the new proposal. The acceptance ratio for the new $x^*$ is

$$
\frac{ e^{ -U(x^*)-K(v^*) }} { e^{-U(x)-K(v)} } = e^{U(x)-U(x^*)+K(x)-K(x^*)}
$$

If our finite difference scheme was exact, the acceptance ration would be 1 since energy is conserved with Hamiltonian dynamics. However, as we have seen, the leapfrog method does not conserve energy perfectly and an accept/reject step is still needed.

#### Example of HMC

We will explore how HMC works when the target distribution is bivariate normal centered at zero

$$
x \sim N(0, \Sigma)
$$

In practice of course, the target distribution will be the posterior distribution and depend on both data and distributional parameters.

The potential energy or negative log likelihood is proportional to
$$
U(x) = \frac{x^T\Sigma^{-1} x}{2}
$$

The kinetic energy is given by
$$
K(v) = \frac{v^T v}{2}
$$

where the initial $v_0$ is chosen at random from the unit normal at each step.

To find the time updates, we use the Hamiltonian equations and find the first derivatives of total energy with respect to $x$ and $v$

$$
\begin{align}
x' &= \frac{\delta K}{\delta v} &= v \\
v' &= -\frac{\delta U}{\delta x} &= -\Sigma^{-1} x \\
\end{align}
$$

giving us the block matrix

$$
A = \pmatrix{0 & 1 \\ -\Sigma^{-1} & 0}
$$

By using the first derivatives, we are making use of the gradient information on the log posterior to guide the proposal distribution.

##### This is what the target distribution should look like

```python
sigma = np.array([[1,0.8],[0.8,1]])
mu = np.zeros(2)
ys = np.random.multivariate_normal(mu, sigma, 1000)
sns.kdeplot(ys[:,0], ys[:,1])
plt.axis([-3.5,3.5,-3.5,3.5])
pass
```

##### This is the HMC posterior

```python
def E(A, u0, v0, u, v):
    """Total energy."""
    return (u0 @ tau @ u0 + v0 @ v0) - (u @ tau@u + v @ v)
```

```python
def leapfrog(A, u, v, h, N):
    """Leapfrog finite difference scheme."""
    v = v - h/2 * A @ u
    for i in range(N-1):
        u = u + h * v
        v = v - h * A @ u

    u = u + h * v
    v = v - h/2 * A @ u

    return u, v
```

```python
niter = 100
h = 0.01
N = 100

tau = la.inv(sigma)

orbit = np.zeros((niter+1, 2))
u = np.array([-3,3])
orbit[0] = u
for k in range(niter):
    v0 = np.random.normal(0,1,2)
    u, v = leapfrog(tau, u, v0, h, N)

    # accept-reject
    u0 = orbit[k]
    a = np.exp(E(A, u0, v0, u, v))
    r = np.random.rand()

    if r < a:
        orbit[k+1] = u
    else:
        orbit[k+1] = u0
```

```python
sns.kdeplot(orbit[:, 0], orbit[:, 1])
plt.plot(orbit[:,0], orbit[:,1], alpha=0.2)
plt.scatter(orbit[:1,0], orbit[:1,1],  c='red', s=30)
plt.scatter(orbit[1:,0], orbit[1:,1],  c=np.arange(niter)[::-1], cmap='Reds')
plt.axis([-3.5,3.5,-3.5,3.5])
pass
```

```python

```


#   Copyright statement:
#### The initial code was provided by Dr. Sandoval (Assitant professor of Electrical Engineering, New Mexico State University) as a starting point for project. https://web.nmsu.edu/~spsandov/
#### The algorithm in current form is developed by Sina Zamen.

#   Objective:
## In this problem, for finding minimum value of a quadratic cost function, “Gradient Decent” method is implemented. The initial weight vectors drawn from a Gaussian distribution with mean 0 and variance equal to 1.

### Loading Required Modules:

```python
import math
import numpy as np
import sympy as sym
from PIL import Image
from numpy import array
from matplotlib import style
from numpy.linalg import inv
import matplotlib.pyplot as plt
style.use('ggplot')
```

### Defining Gaussian Dataset Generator Function:

```python
def circGuass(N,Me,VAR):
    dimension = len(Me)
    a = np.array(Me)
    p = np.random.randn(dimension,N)
    h = np.tile(a, (2, N//2))
    x = math.sqrt(Var) * p + h
    return x
```

### Setting up countour map:

```python
HX ,HY  = 50,50 #number of x,y  points for countour
xmin,xmax = -15,15
ymin,ymax = -12,12
x1 = np.linspace(xmin,xmax,HX)
x2 = np.linspace(ymin,ymax,HY)
X1,X2 = np.meshgrid(x1,x2) # genertate mesh grid
w1=sym.Symbol('w1') # define symbols
w2=sym.Symbol('w2')
```

```python
bowl = 'bowl'
case2= ''
case = bowl #case select
if(case==bowl):
    j=(w1**2+w1*w2+3*w2**2)# define equation
elif(case==case2):
    #define other surfaces here
    pass
else:
    print('case not recognized')
```

```python
#compute gradient
j_grad1=sym.diff(j,w1)
j_grad2=sym.diff(j,w2)
#compute hessian
hess11=sym.diff(j_grad1,w1)
hess12=sym.diff(j_grad1,w2)
hess21=sym.diff(j_grad2,w1)
hess22=sym.diff(j_grad2,w2)

#generate contour map
ConMap=np.zeros((HX,HY))
for i in range(HX):
    for k in range(HY):
        ConMap[i,k]=j.subs({w1:x1[i],w2:x2[k]})

w=[-5,5] # starting point
wStar=[0,0] # ending point
ew=array([7])
jw=array([])
eta = 0.30 # learning rate
lim=200 # number of iterations
count= 0
line=[]
h = ew.astype(int)
h = ew[-1:].astype(int)
while (h > 0.05):
    h = ew[-1:].astype(int)
    #compute gradient matrix and hessian matrix
    g= np.array([float(j_grad1.subs({w1:w[0],w2:w[1]})),float(j_grad2.subs({w1:w[0],w2:w[1]}))])
    H= np.array([[float(hess11.subs({w1:w[0],w2:w[1]})),float(hess12.subs({w1:w[0],w2:w[1]}))],                 [float(hess21.subs({w1:w[0],w2:w[1]})),float(hess22.subs({w1:w[0],w2:w[1]}))]])

    wnew =  w-eta*g
    #loop check
    if( count>lim ):
        print('Count Break')
        break
    elif(np.isnan(g).any()):
        print('nan break')
        break
    else:
        count=count +1
        wprev=w.copy()

        line.append(w)
        w=wnew.copy()

        ew1 = (np.linalg.norm(w-wStar))
        jw1 = (j.subs({w1:w[0],w2:w[1]}))
        ew = np.append(ew, ew1)
        jw = np.append(jw, jw1)
line=np.array(line)
```

```python
print("Number of Iterations:")
print(count)
```

    Number of Iterations:
    11

```python
plt.figure(figsize=(10,10))
plt.contour(X1,X2,ConMap)
plt.plot(line[:,1],line[:,0])
plt.scatter(line[:,1],line[:,0])
plt.plot(wStar[1],wStar[0],'go')
plt.show()
```

```python
plt.figure(figsize=(10,10))
plt.title('J(w) vs iterations')
plt.ylabel('J(w)')
plt.xlabel('iteration')
plt.plot(range(len(jw)),jw)
plt.show()
```

```python
plt.figure(figsize=(10,10))
plt.title('E(w) vs iterations')
plt.ylabel('E(w)')
plt.xlabel('iteration')
plt.plot(range(len(ew)),ew)
plt.show()
```

```python
M1 = 0
Var = 3
N = 2
firstmean = [M1, M1]
Me = firstmean;
X = circGuass(N, Me, Var)
```

```python
w=X[0,:] # starting point
wStar=[0,0] # ending point
ew=array([7])
jw=array([])
eta = 0.31 # learning rate
lim=200 # number of iterations
count= 0
line=[]
h = ew.astype(int)
h = ew[-1:].astype(int)
while (h > 0.05):
    h = ew[-1:].astype(int)
    #compute gradient matrix and hessian matrix
    g= np.array([float(j_grad1.subs({w1:w[0],w2:w[1]})),float(j_grad2.subs({w1:w[0],w2:w[1]}))])
    H= np.array([[float(hess11.subs({w1:w[0],w2:w[1]})),float(hess12.subs({w1:w[0],w2:w[1]}))],                 [float(hess21.subs({w1:w[0],w2:w[1]})),float(hess22.subs({w1:w[0],w2:w[1]}))]])

    wnew =  w-eta*g
    #loop check
    if( count>lim ):
        print('Count Break')
        break
    elif(np.isnan(g).any()):
        print('nan break')
        break
    else:
        count=count +1
        wprev=w.copy()

        line.append(w)
        w=wnew.copy()

        ew1 = (np.linalg.norm(w-wStar))
        jw1 = (j.subs({w1:w[0],w2:w[1]}))
        ew = np.append(ew, ew1)
        jw = np.append(jw, jw1)
line=np.array(line)
```

```python
print("Number of Iterations:")
print(count)
```

    Number of Iterations:
    8

```python
plt.figure(figsize=(10,10))
plt.contour(X1,X2,ConMap)
plt.plot(line[:,1],line[:,0])
plt.scatter(line[:,1],line[:,0])
plt.plot(wStar[1],wStar[0],'go')
plt.show()
```

```python
plt.figure(figsize=(10,10))
plt.title('J(w) vs iterations')
plt.ylabel('J(w)')
plt.xlabel('iteration')
plt.plot(range(len(jw)),jw)
plt.show()
```

```python
plt.figure(figsize=(10,10))
plt.title('E(w) vs iterations')
plt.ylabel('E(w)')
plt.xlabel('iteration')
plt.plot(range(len(ew)),ew)
plt.show()
```

Gradient Decent/Gradient Decent With a Quadratic Cost Function .ipynb


```python
# Standard IPython notebook imports
%matplotlib inline

import os

from io import StringIO
import itertools
import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook

import scipy.stats as sstats
import scipy.optimize as sopt

import seaborn as sns
import sklearn.tree
import sklearn.ensemble

import graphviz
from numpy.linalg import inv
```

# Quantile regression

  The least squares method results in estimates that approximate
  the conditional mean of the response variable given certain values
  of the predictor variables.  However, for many applications we are interested
  in a median or other percentile rather than the mean of the response
  variable. An approximation of a percentile can be achieved
  indirectly by using the least squares method to fit a model,
  assuming a gaussian distribution on residuals of this model and
  calculating the required percentile of the gaussian distribution.
  The problems with this commonly used technique appear when the
  distribution of residuals does not follow a gaussian distribution.

  The quantile regression
  (https://en.wikipedia.org/wiki/Quantile_regression) aims at
  directly estimating a value of the conditional percentile of the
  response variable. It is often use in e.g. sales forecasting, where
  we are interested in e.g. keeping enough items to have a 90% chance
  to cover the demand. In quantile regression, instead of minimizing the mean squared error,
  quantile regression minimizes a different cost function namely:
  \begin{equation}
  f_{\tau} = \left\{
    \begin{array}{rl}
      \tau\cdot x & \text{if } x \geq 0,\\
      -(1-\tau)\cdot x & \text{if } x < 0,
    \end{array} \right.
  \end{equation}
  where $\tau$ is the precentile of interest.

   Download the house pricing data set from
    https://raw.githubusercontent.com/janchorowski/nn_assignments/nn18/assignment3/03-house-prices-outliers.csv. To
    load it into Python, you can use the `pandas.read_csv`
    function, similarly to how we read the height/weight table in the
    notebook for lecture 3.

    This data contains information about areas and prices of around
    5000 houses offered for sale. Imagine you want to buy a 60 squared
    meters flat. How much money do you need to have to be able to
    choose between $75\%$ of all flats offered for sale?

   Fit a quantile regression model by minimizing the $f_{\tau}$
    function. To fit the model write a function returning the cost,
    and its derivative (you can assume that the derivative is 0 at the
    singular points).  Use the L-BFGS solver (without step size
    control the GD algorithm does not converge to good solutions). You
    can start the solver from the least squares solution. Check the
    goodness of fit as described above.

   Now you can calculate the 75 percentile price value for a 60
    squared meters flat using the more adequate model.

   **Note**: Technically, we can not use a gradient-based
    optimization method to minimize the $f_{\tau}$ loss because it
    doesn't have a derivative at $x=0$. One solution is to use a
    subgradient method
    https://en.wikipedia.org/wiki/Subderivative. It is also
    possible to use a smooth loss function -- common variants are the
    Huber loss (https://en.wikipedia.org/wiki/Huber_loss) or a
    variant of the smooth approximation to the absolute value
    $\text{abs}(x) \approx \sqrt{x^2+\epsilon}$ with a small
    $\epsilon$. However, just ignoring the singular points tends to
    work well in practice.

```python
def powers_of_X(X, degree):
    powers = np.arange(degree + 1).reshape(-1,1)
    return X**powers

def compute_polynomial(X, Theta):
    XP = powers_of_X(X, len(Theta) - 1) # len(Theta) x N
    Y = XP * Theta
    Y = np.sum(Y, axis=0)
    return Y

def quantile_cost(Y, Theta, X, tau):
    '''
    Returns:
        pair (x, G), where
        x: mean of cost function values
        G: mean Gradient
    '''

    residuals = Y - Theta.T.dot(X)
    residuals = residuals.ravel()
    ans = []

    for i, r in enumerate(residuals):
        if r == 0:
            val = np.array([0, np.array([0, 0])])
            ans.append(val)
        elif r > 0:
            val = np.array([np.array(tau * r),
                            np.array(-tau * X[:, i])])
            ans.append(val)
        else:
            val = np.array([np.array((tau - 1) * r),
                            np.array((1 - tau) * X[:, i])])
            ans.append(val)

    ans = np.array(ans)
    N = residuals.size
    func_val, gradient_vals = 0, np.array([0, 0], dtype=np.float64)

    for f_val, gradient_val in ans:
        func_val += f_val
        gradient_vals += gradient_val

    func_val /= N
    gradient_vals /= N

    return (func_val, gradient_vals)

def get_theta(X, Y, Theta, tau=0.75):
    hist = []
    def save_hist(x):
        hist.append(np.array(x))

    lbfsgb_ret = sopt.fmin_l_bfgs_b(lambda theta: quantile_cost(Y, theta, X, tau),
                                    Theta, callback=save_hist)

    return hist[-1]
```

### Get data

```python
data = pd.read_csv(
    'https://raw.githubusercontent.com/janchorowski/nn_assignments/nn18/assignment3/03-house-prices-outliers.csv',
    index_col=0)
data.head()
```

      rooms
      area
      price

      0
      3
      66.80
      367.801

      1
      3
      55.03
      289.458

      2
      4
      70.80
      389.000

      3
      1
      29.02
      195.695

      4
      2
      45.76
      265.754

```python
X = np.vstack((np.ones_like(data.rooms), data.area))
X
```

    array([[ 1.  ,  1.  ,  1.  , ...,  1.  ,  1.  ,  1.  ],
           [66.8 , 55.03, 70.8 , ..., 46.7 , 50.41, 48.1 ]])

```python
Y = np.asarray(data.price)[None,:]
Y
```

    array([[367.801, 289.458, 389.   , ..., 266.   , 267.1  , 269.   ]])

```python
%%time
Theta = inv(X.dot(X.T)).dot(X).dot(Y.T)
taus = [0.25, 0.50, 0.75]
colors = ['red', 'green', 'blue']
x_space = np.linspace(0, np.max(X), 100)

for i, tau in enumerate(taus):
    theta = get_theta(X, Y, Theta, tau=tau)
    plt.plot(x_space, theta[0] + theta[1] * x_space,
             c=colors[i], label=tau)
    residuals = Y - compute_polynomial(X[1, :], theta[:, np.newaxis]).ravel()
    mu = residuals.mean()
    sigma = np.sqrt(np.mean((residuals - mu) ** 2))
    errors = residuals[residuals < 0].size / residuals.size

    print(f'Quant: {tau}\n{theta}')
    print(f'Fraction of negative errors: {errors}')
    print(f'mu: {mu}, sigma: {sigma}\n')

plt.legend(loc='lower right')
plt.scatter(X[1, :], Y, edgecolors='black', s=30, alpha=0.7)
plt.ylim([0, 2000])
plt.xlim([0, 200])
```

```python

```


# Zygote AutoDiff Works for Symboic Representation

## Check by Zygote for Composite Function

```julia
using Symbolics
using Zygote
```

```julia
f(x) = 3x^2
g(x) = 2x^2
h(x) = x^2
y(x) = f(g(h(x)))

@variables x
display(y(x))
```

\begin{equation}
12 x^{8}
\end{equation}

```julia
dy(x) = gradient(y,x)[1]
display(dy(x))
```

\begin{equation}
96 x^{7}
\end{equation}

## Check by Zygote for DiffProg-Type Code

```julia
function y(x, N)
    # N = 5
    y = 1
    for i=1:N
        y *= x
    end
    y
end

@variables x
display(y(x, 5))
```

\begin{equation}
x^{5}
\end{equation}

```julia
dy(x,N) = Zygote.gradient(y,x,N)[1]
dy(x,5)
```

\begin{equation}
5 x^{4}
\end{equation}

## Appendix

```julia
function y(x)
    N = 5
    y = 1
    for i=1:N
        y *= x
    end
    y
end

display(y(x))
dy(x) = Zygote.gradient(y,x)[1]
dy(x)
```

\begin{equation}
x^{5}
\end{equation}

\begin{equation}
5 x^{4}
\end{equation}


# Demystifying `meshgrid()`

We use the Numpy [meshgrid()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) function to plot 2D arrays but it can be a bit mysterious as to what the "meshgrid" actually does or look like.

The docs say:

_Return coordinate matrices from coordinate vectors._

_Make N-D coordinate arrays for vectorized evaluations of N-D scalar/vector fields over N-D grids, given one-dimensional coordinate arrays x1, x2,..., xn._

Things to keep in mind:
- NumPy functions evaluate input _element-wise_ : `f(v)` with 1D array `v` gives `[f(v[0]), f(v[1]), ...]` (a 1D array) or for a 2D array `a`, `f(a)` produces a 2D array `[[f(a[0,0]), f(a[0, 1]),...], [f(a[1,0]), f(a[1, 1]),...]]`
- matplotlib's 2D plotting functions take as input 2D arrays (together with optional arrays to specify coordinate values (as opposed to plotting "by index"))

## Visualization with a Gaussian
Testing with the initial condition, an isotropic 2D [Gaussian](http://mathworld.wolfram.com/GaussianFunction.html)

$$
g_2(x, y) = \frac{1}{2\pi\sigma^2} \exp\left[-\frac{(x-x_0)^2 + (y-y_0)^2}{2\sigma^2}\right]
$$

Note that the 2D Gaussian can be written as a product of two 1D Gaussians

\begin{align}
g_2(x, y) &= \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(x-x_0)^2}{2\sigma^2}\right] \cdot
          \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(y-y_0)^2}{2\sigma^2}\right]\\
        &= g_1(x) \cdot g_1(y)
\end{align}

Plot $g_2(x, y)$ by (1) constructing it from the product of $g_1$s and (2) from the explicit formula. Use [`np.meshgrid`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) to evaluate the function on a grid.

Use an *asymmetric* grid ($L_x=0.5, N_x=50$ and $L_y=1, N_y=100$) to clearly see the shape of the underlying arrays.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
```

```python
L = 0.5
Lx, Ly = 0.5, 1
Nx, Ny = 50, 100

x = np.linspace(0, Lx, Nx+1)  # need N+1!
y = np.linspace(0, Ly, Ny+1)
X, Y = np.meshgrid(x, y)

def gaussian2D(x, y, u0=0.05, x0=None, y0=None, sigma=0.1*L):
    x0 = np.mean(x) if x0 is None else x0
    y0 = np.mean(y) if y0 is None else y0
    return u0/(2*np.pi*sigma) * np.exp(-((x-x0)**2 + (y-y0)**2)/(2*sigma**2))

def gaussian(x, u0=0.05, x0=None, sigma=0.1*L):
    x0 = np.mean(x) if x0 is None else x0
    return u0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2 / (2*sigma**2))
```

#### Meshgrid

```python
print(x.shape, y.shape)
print(X.shape, Y.shape)
print(X)
print(Y)
```

    (51,) (101,)
    (101, 51) (101, 51)
    [[0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     ...
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]
     [0.   0.01 0.02 ... 0.48 0.49 0.5 ]]
    [[0.   0.   0.   ... 0.   0.   0.  ]
     [0.01 0.01 0.01 ... 0.01 0.01 0.01]
     [0.02 0.02 0.02 ... 0.02 0.02 0.02]
     ...
     [0.98 0.98 0.98 ... 0.98 0.98 0.98]
     [0.99 0.99 0.99 ... 0.99 0.99 0.99]
     [1.   1.   1.   ... 1.   1.   1.  ]]

Note that the two meshgrids have the same shape: `Nx x Ny`.

They also contain either constant rows or constant columns. We will see how this is useful — but you can already imagine what happens if you "overlay" `X` and `Y` and read of the entries in each "overlayed" cell: the coordinates of the cell:

```python
X[0, 0], Y[0, 0]
```

    (0.0, 0.0)

```python
X[1, 0], Y[1, 0]
```

    (0.0, 0.01)

#### Plot 1D Gaussians
Plot the 1D Gaussians using the meshgrid components.

```python
gaussian(X).shape
```

    (101, 51)

```python
fig = plt.figure()
ax = fig.add_subplot(111)
ax.contourf(X, Y, gaussian(X, sigma=0.1), 30)
ax.set_aspect(1)
plt.show()
```

```python
gaussian(Y).shape
```

    (101, 51)

```python
ax = plt.subplot(131)
ax.contourf(X, Y, gaussian(X, sigma=0.1), 30)
ax.set_aspect(1)
ax = plt.subplot(132)
ax.contourf(X, Y, gaussian(Y, sigma=0.1), 30)
ax.set_aspect(1)
ax = plt.subplot(133)
ax.contourf(X, Y, gaussian(X, sigma=0.1) * gaussian(Y, sigma=0.1), 30)
ax.set_aspect(1)
```

#### 2D gaussian

```python
gaussian2D(X, Y).shape
```

    (101, 51)

```python
ax = plt.subplot(111)
ax.contourf(X, Y, gaussian2D(X, Y, sigma=0.1), 30)
ax.set_aspect(1)
```

```python

```


# Systems of ODEs, and higher-order ODEs

Let's solve the **pendulum** equation
$$u'' + \omega_0^2\sin(u)=0,\quad u(0)=0.2, u'(0) = 0$$
where $\omega_0 = \sqrt{\frac{g}{L}}$. This models the angular displacement $u$ of a pendulum of length $L$ in Earth's gravitational field. This is not a first-order ODE, it is a **second-order** ODE.

As we did in the [notes](https://github.com/stephenbeckr/numerical-analysis-class/blob/master/Notes/Ch5_SystemsOfODEs.pdf), we can convert this to a system of **two** **first-order** ODEs.
\begin{align}
u' &= v \\
v' &= -\omega_0^2\sin(u) \\
u(0) &= 0.2 \\
v(0) &= 0
\end{align}
or more generally
\begin{align}
u_1' &= u_2 \\
u_2' &= -\omega_0^2\sin(u_1) \\
u_1(0) &= 0.2 \\
u_2(0) &= 0
\end{align}

Note that if we make the small angle approximation, then the ODE $u'' + \omega_0^2 u = 0$ with $u'(0)=0$ has the closed-form solution
$$u(t) = u(0)\cos(\omega_0 t).$$

```
import numpy as np
from numpy import sin, cos, exp
import matplotlib.pyplot as plt
import matplotlib as mpl
import scipy.interpolate # use for scipy.interpolate.CubicHermiteSpline
from scipy.integrate import solve_ivp
mpl.rcParams["figure.figsize"] = [8,6] # or 7, 4 or 10,8
mpl.rcParams["lines.linewidth"] = 2
mpl.rcParams["lines.markersize"] = 4
mpl.rcParams.update({'font.size': 20})
mpl.rcParams['mathtext.fontset'] = 'cm'
```

### Setup the RHS of the Pendulum ODE, call Scipy solver

```
omega = 2.
omegaSq = omega**2
def f_pendulum(t,uVector):
  # The RHS for the pendulum equation; this does not depend on t
  u, v = uVector
  f0 = v
  f1 = -omegaSq*np.sin(u)
  return [f0,f1]

tspan = [0,10] # solve for 10 seconds
y0    = [.2, 0]
ivp_soln = solve_ivp( f_pendulum, tspan, y0, dense_output=True )
y     = ivp_soln.sol

y_linearized = lambda t : y0[0]*np.cos(omega*t)
```

Let's plot it.  Is the solution similar to the closed form solution of the approximate/simpler equation?

```
t = np.linspace(*tspan,100)

plt.plot(t,y(t).T);
plt.plot(t,y_linearized(t),'--')
plt.legend(['$u(t)$ (angular displacement)','$v(t)$ (angular velocity)','closed form soln to simpler eq']);
```

Yes, the closed form solution to the simpler equation looks quite close. To see the differences better, let's plot the differences

```
t = np.linspace(*tspan,100)

plt.plot(t,y(t)[0].T - y_linearized(t) );
plt.ylabel('Error of linearized solution');
plt.title(f'Starting amplitude was {y0[0]}');
```

What if we do the same as above, for a larger starting amplitude? This should make the small angle approximation less valid

```
y0    = [.8, 0]
ivp_soln = solve_ivp( f_pendulum, tspan, y0, dense_output=True )
y     = ivp_soln.sol

y_linearized = lambda t : y0[0]*np.cos(omega*t)

plt.plot(t,y(t).T);
plt.plot(t,y_linearized(t),'--')
plt.legend(['$u(t)$ (angular displacement)','$v(t)$ (angular velocity)','closed form soln to simpler eq']);
```

```
plt.plot(t,y(t)[0].T - y_linearized(t) );
plt.ylabel('Error of linearized solution');
plt.title(f'Starting amplitude was {y0[0]}');
```

### Now let's define our own Euler's method
This the same code as last demo, but we modfied line 11, 20 and 22. These are just programming details

```
def forward_Euler( f, tspan,y0, h=0.1, dense_output=False):
  """ Solve IVP y' = f(t,y) on a <= t <= b, y(a)=y0, via improved Euler with stepsize h
  This code has NOT been tested to work with systems.
  Returns tHist, wHist where wHist[i] approximates y( tHist[i] )
  """
  if len(tspan) is not 2:
    raise ValueError("tspan must be array of length 2")
  a, b = tspan[0], tspan[1]
  # Initialize
  t    = a
  w    = np.asarray(y0)  # Changed this code
  # Update history
  tHist = [t]
  wHist = [w]  # w[i] approximates y( t[i] )
  dydtHist = []
  # Loop
  while t < b :
    tNext = min(t+h,b)  # don't go too far
    hh    = tNext - t   # hh is just h except perhaps at the last time step
    dydt  = np.asarray(f(t,w))   # changed this code
    # w    += hh*dydt     # doesn't play well with np.arrays
    w     = w + hh*dydt
    t     = tNext
    # Update history
    tHist.append(t)
    wHist.append(w)
    dydtHist.append(dydt)

  dydtHist.append( f(t,w) )# lagged

  if dense_output is True:
    sol = scipy.interpolate.CubicHermiteSpline(tHist,wHist,dydtHist)
    return np.array(tHist), np.array(wHist), sol
  else:
    return np.array(tHist), np.array(wHist)
```

and let's run it and see how it works

```
h = 0.5
tHist, wHist, yEuler = forward_Euler( f_pendulum, tspan, y0, h = h, dense_output=True )
```

```
t = np.linspace(*tspan,100)
plt.plot( t, yEuler(t) );
#plt.plot( tHist, wHist[:,0] ); # simple

plt.legend(['$u(t)$ (angular displacement)','$v(t)$ (angular velocity)']);
```

Well, for a large stepsize, we get a terrible answer. Perhaps not so unexpected. Let's do a smaller stepsize then

```
h = 0.01
tHist, wHist, yEuler = forward_Euler( f_pendulum, tspan, y0, h = h, dense_output=True )

t = np.linspace(*tspan,100)
plt.plot( t, yEuler(t) );

plt.legend(['$u(t)$ (angular displacement)','$v(t)$ (angular velocity)']);
```

```
plt.plot( t, y(t).T - yEuler(t) );
plt.title(f'Difference between scipy soln and Euler method, h is {h}');
```

```
h = 1e-5
tHist, wHist, yEuler = forward_Euler( f_pendulum, tspan, y0, h = h, dense_output=True )

t = np.linspace(*tspan,100)
plt.plot( t, yEuler(t) );

plt.legend(['$u(t)$ (angular displacement)','$v(t)$ (angular velocity)']);
```

```
plt.plot( t, y(t).T - yEuler(t) );
plt.title(f'Difference between scipy soln and Euler method, h is {h}');
```


# Synthetic data generation

We try to build a potential data scenario for the application of contextual multi-armed bandit strategies.

The scenario should be characterized by
1. **a context** (i.e. a collection of attributes describing the situation where a specific action is proposed). In this case the context is characterised by a collection of customer features, which could be either continuos, ordinal or categorical.
2. **a series of actions** to propose to the customer (e.g. no offer, mild offer and deep offer) given the context.

The dataset we build should represent a situation where we know in advance which are the actions that have been proposed to each customer and the value of the corresponding rewards. This would allow us to test bench any kind of contextual multi armed bandit solution via the execution of controlled experiments.

The notebook has the following structure

1. [Loading modules](#Loading-modules)
2. [Synthetic data generation](#Synthetic-data-generation)
3. [Input space inspection](#Input-space-inspection)
    * [Continuos features](#Continuos-features)
    * [Ordinal features](#Ordinal-features)
    * [Categorical features](#Categorical-features)
4. [Prediction power vs synthetic noise](#Prediction-power-vs-synthetic-noise)

## Loading modules

```python
import os
import sys
sys.path.append('/Users/fci02/Documents/GitHub/decisioning-analysis/test_and_learn/production_modules/')

from contextual_mab.data_utils.data_generation  import generate_synthetic_data
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from ccp_analysis.ccp_tree_analysis import compute_auc_metrics

```

## Synthetic data generation

We generate a synthetic dataset with the following shape

1. $3$ available actions
2. $n = 1000$ data points per action played
3. the **global** expected rewards of the actions are $[5\%,3\%,1\%]$
4. The features characterizing the context are
    * 3 continuos, normally distributed with mean vector $[0,10,-3]$ and standard deviations $[1,5.,.5]$
    * 2 ordinal features, with specified frequencies
        \begin{align}
        \text{O_1} &= [1,2,3,4,5,6,7,8,9,10] \quad p = [0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1]\\
        \text{O_2} &= [1,2,3] \quad p = [0.3,0.4,0.3]
        \end{align}
    * 3 categorical features, with specified frequencies
        \begin{align}
        \text{C_1} &= (\text{Yes,No}) \quad p = [0.4,.61]\\
        \text{C_2} &= (\text{Green,Yellow}) \quad p = [0.2,0.8]\\
        \text{C_3} &= (\text{A,B,C}) \quad p = [0.4,0.1,0.5]
        \end{align}
5. For each action $a$, the reward outcome are computed in the following way:
    * a random vector of weights is generated ${\bf w}_{a} \sim \mathcal{N}({\bf 0},{\bf 1})$
    * for a context matrix $X_a$, a probabilities vector is computed via sigmoid function
        \begin{align}
        {\bf z} = {\bf w}^T_a X_a+{\bf \epsilon}\\
        {\bf p} = \frac{1}{1+e^{-{\bf z}}}
        \end{align}
       where $\epsilon$ is a noise term
       \begin{equation}
       {\bf \epsilon} \sim \mathcal{N}(0, \alpha)
       \end{equation}
       and $\alpha$ is provided by the user. For each row $i$ of $X_a$, a reward value is then assigned using the following criterion
       \begin{align}
       r_i &= 1 \quad \text{if } p_i > (1-\bar{r}_a)\text{-quantile}\\
       r_i &= 0 \quad \text{if } p_i <= (1-\bar{r}_a)\text{-quantile}
       \end{align}
      where $\bar{r}_a$ is provided by the user. In this way we ensure the global expected reward of each action reflects the input value provided by the user.
      For example if $\bar{r}_a=0.1$ then the rule assigns a non zero reward only to those rows whith probabilities belonging to the last decile.

```python
n = 10000
nc = 3
cw = [.05,.03,.01]

ct_prm = {'X1':{'loc':0.,'scale':1.},
          'X2':{'loc':10.,'scale':5.},
          'X3':{'loc':-.3,'scale':.5}}

ord_prm = {'O1':{'start':1,'stop':11,'weights':[.1]*10},
           'O2':{'start':1,'stop':4,'weights':[.3,.4,.3]}}

catg_prm = {'C1':{'levels':['Yes','No'],'weights':[.4,.6]},
            'C2':{'levels':['Green','Yellow'],'weights':[.2,.8]},
            'C3':{'levels':['A','B','C'],'weights':[.4,.1,.5]}}
seed = 0
noise_scale=5.
```

```python
synthetic_data = generate_synthetic_data(n,nc,cw,
                       ct_prm,ord_prm,catg_prm,seed,noise_scale,
                       return_inputs=False,output_info=True)
```

    Total number of data: 30000
    Actions played: [1, 2, 3]
    Global expected rewards per action

      reward

      action_code

      1
      0.05

      2
      0.03

      3
      0.01

```python
synthetic_data.head()
```

      X1
      X2
      X3
      O1
      O2
      C1
      C2
      C3
      reward
      action_code

      0
      1.764052
      11.856161
      -0.838830
      8
      1
      Yes
      Green
      C
      0
      1

      1
      0.400157
      11.523919
      -0.721146
      6
      2
      No
      Green
      A
      0
      1

      2
      0.978738
      12.520623
      -0.428904
      1
      3
      Yes
      Yellow
      C
      0
      1

      3
      2.240893
      10.676498
      0.056038
      5
      2
      Yes
      Yellow
      C
      0
      1

      4
      1.867558
      13.268794
      -0.027104
      7
      3
      No
      Yellow
      C
      0
      1

## Input space inspection

### Continuos features

```python
fig, (ax1,ax2,ax3) = plt.subplots(1,3,figsize=(24,8))

for feature, ax in zip(['X1','X2','X3'],[ax1,ax2,ax3]):
    sns.distplot(synthetic_data[feature],ax=ax,label='mean: {0:.3f}\nstd: {1:.3f}'.format(synthetic_data[feature].mean(),synthetic_data[feature].std()))
    ax.set_title('Histogram of {}'.format(feature))
    ax.legend()
plt.tight_layout()
plt.show()
```

### Ordinal features

```python
fig, (ax1,ax2) = plt.subplots(1,2,figsize=(24,8))

for feature, ax in zip(['O1','O2'],[ax1,ax2]):
    sns.countplot(synthetic_data[feature],palette="rocket",ax=ax,)
    ax.set_title('Histogram of {}'.format(feature))
    ncount = len(synthetic_data[feature])
    for p in ax.patches:
        x=p.get_bbox().get_points()[:,0]
        y=p.get_bbox().get_points()[1,1]
        ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y),
            ha='center', va='bottom',fontsize=14)
plt.tight_layout()
plt.show()
```

### Categorical features

```python
fig, (ax1,ax2,ax3) = plt.subplots(1,3,figsize=(24,8))

for feature, ax in zip(['C1','C2','C3'],[ax1,ax2,ax3]):
    sns.countplot(synthetic_data[feature],palette="rocket",ax=ax,)
    ax.set_title('Histogram of {}'.format(feature))
    for p in ax.patches:
        x=p.get_bbox().get_points()[:,0]
        y=p.get_bbox().get_points()[1,1]
        ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y),
            ha='center', va='bottom',fontsize=14)
plt.tight_layout()
plt.show()
```

## Prediction power vs synthetic noise

We can define more or less difficult to learn reward distributions by tuning the value of the random noise term $\epsilon$. This would help us in mimicking more realistic scenarios.

```python
noise_scales = [.1,1.,5,10.,15]
fig, (ax1, ax2, ax3) = plt.subplots(1,3,figsize=(24,8))

for action, ax in zip(range(1,4),[ax1, ax2, ax3]):
    noise_perfs = {}
    for noise_scale in noise_scales:
        synthetic_data = generate_synthetic_data(n,nc,cw,
                                                 ct_prm,ord_prm,catg_prm,seed,noise_scale)
        subset = synthetic_data[synthetic_data['action_code']==action]
        X = pd.get_dummies(subset.drop(columns=['reward','action_code']),drop_first=True).values
        y = subset['reward'].values
        X_train, X_test, y_train, y_test =train_test_split(X,y,test_size=.3,stratify=y)
        model = LogisticRegression(solver='lbfgs')
        model.fit(X_train,y_train)
        noise_perfs[str(noise_scale)] = compute_auc_metrics(model,X_train, X_test, y_train, y_test)

    data = pd.DataFrame(noise_perfs)
    w=.1
    ax.bar(np.arange(1,6),data.loc['train',:],width=w,label='train AUC',align='center')
    ax.bar(np.arange(1,6)+w,data.loc['test',:],width=w,label='test AUC',align='center')
    ax.legend()
    ax.set_title('Action {}'.format(action),fontsize=20)
    ax.grid()
    ax.set_xticks(np.arange(1,6))
    ax.set_xticklabels([str(noise_scale) for noise_scale in noise_scales])
    ax.set_xlabel('$\sigma_\epsilon$', fontsize=20)
plt.tight_layout()
plt.show()
```

It seems we can capture the variability of the reward enough well even for large value of the noise term variance. Larger values have been tried, however in this case it is necessary to use larger sample size per action in order to be able to see non zero outcomes of the reward.