# Implementation of a Devito acoustic Least-square Reverse time migration

## This tutorial is contributed by SENAI CIMATEC (2021)

This tutorial is based on:

**LEAST-SQUARES REVERSE TIME MIGRATION (LSRTM) IN THE SHOT DOMAIN** (2016)

Antonio Edson Lima de Oliveira, Reynam da Cruz Pestana and Adriano Wagner Gomes dos Santos

Brazilian Journal of Geopyisics

http://dx.doi.org/10.22564/rbgf.v34i3.831

**Plane-wave least-squares reverse-time migration** (2013)

Wei Dai and Gerard T. Schuster

GEOPHYSICS Technical Papers

http://dx.doi.org/10.22564/rbgf.v34i3.831

**Two-point step size gradient method** (1988)

Barzilai, J. and Borwein, J.

IMA Journal of Numerical Analysis

https://doi.org/10.1093/imanum/8.1.141

## Introduction
The goal of this tutorial is to implement and validate the Least-squares reverse time migration (LSRTM) using a 2D three-layered velocity model with a square in the middle. The algorithm has been implemented using the Born's appoximation.

The acoustic wave equation for constant density is:

\begin{equation}
 m_{0} \dfrac{\partial^2 p_0 }{\partial t^2} - \nabla^2 p_0 = s (\mathbf{x},t)   \hspace{0.5cm}     (1)
\end{equation}

where $s (\mathbf{x},t) $ is the source, $p_{0}$ is the background wavefield and $m_{0}$ is the smoothed squared slowness.

A perturbation in the squared slowness $m = m_{0} + \delta m$ produces a background wavefield ( $p_{0}$ ) and scattered wavefield ( $\delta p$ ), so the wavefield $p$ is approximated to $ p = p_0 + \delta p$, that obeys the relation:

\begin{equation}
 m \dfrac{\partial^2 p}{\partial t^2} - \nabla^2 p = s (\mathbf{x},t)   \hspace{0.5cm}     (2)
\end{equation}

Using the approximation of $p$ and $m$ into equation (2),

\begin{equation}
 (m_{0} + \delta m) \dfrac{\partial^2 (p_0 + \delta p)}{\partial t^2} - \nabla^2 (p_0 + \delta p) = s (\mathbf{x},t)   \hspace{0.5cm}     (3)
\end{equation}

Expanding equation (3) we have:
\begin{equation}
m_{0} \dfrac{\partial^2 p_0 }{\partial t^2} - \nabla^2 p_0 + m_{0} \dfrac{\partial^2 \delta p }{\partial t^2} - \nabla^2\delta p + \delta m \dfrac{\partial^2 (p_{0} +\delta p) }{\partial t^2}= s (\mathbf{x},t) \hspace{0.5cm}    (4)
\end{equation}

Reordering the equation (4),
\begin{equation}
m_{0} \dfrac{\partial^2 p_0 }{\partial t^2} - \nabla^2 p_0 + m_{0} \dfrac{\partial^2 \delta p }{\partial t^2} - \nabla^2\delta p = s (\mathbf{x},t) - \delta m \dfrac{\partial^2 (p_{0} +\delta p) }{\partial t^2} \hspace{0.5cm}    (5)
\end{equation}

Considering that $ \delta m \dfrac{\partial^2 (p_{0} +\delta p) }{\partial t^2} = \delta m \dfrac{\partial^2 p }{\partial t^2} \approx \delta m \dfrac{\partial^2 p_{0} }{\partial t^2}  $( Via Born's approximation $p\approx p_{0}$ ):

\begin{equation}
m_{0} \dfrac{\partial^2 p_0 }{\partial t^2} - \nabla^2 p_0 + m_{0} \dfrac{\partial^2 \delta p }{\partial t^2} - \nabla^2\delta p = s (\mathbf{x},t) -\delta m \dfrac{\partial^2 p_{0} }{\partial t^2}\hspace{0.5cm}    (6)
\end{equation}

Now we get an equations system:

\begin{equation}
 \left\{
\begin{array}{lcl}
m_{0} \dfrac{\partial^2 p_0 }{\partial t^2} - \nabla^2 p_0 = s (\mathbf{x},t),\hspace{0.5cm} (7a) \\
m_{0} \dfrac{\partial^2 \delta p }{\partial t^2} - \nabla^2\delta p = - \delta m \dfrac{\partial^2 p_{0} }{\partial t^2} \hspace{0.5cm} (7b) \\
\end{array}
\right.
\end{equation}

Equations (7a) and (7b) are for Born modelling. The adjoint modelling is defined by the equation:

\begin{equation}
m_{0} \dfrac{\partial^2 v}{\partial t^2} - \nabla^2 v =\textbf{d}  \hspace{0.5cm}     (8)
\end{equation}
where $\textbf{d}$ is the shot recorded.

With all these equations, the migrated image can be constructed

\begin{equation}
\mathbf{m}_{mig}= \sum_{t} \dfrac{\partial^2 p_0}{\partial t^2}v \hspace{0.5cm}     (9)
\end{equation}

In this notebook the migration and gradient has been precoditioned by the source illumination, so equation (9) becomes:

\begin{equation}
\mathbf{m}_{mig}=  \frac{\sum_{t} \dfrac{\partial^2 p_0}{\partial t^2}v}{\sum_{t} p_{0}^2} \hspace{0.5cm}     (10)
\end{equation}

LSRTM aims to solve the reflectivity model $\mathbf{m}$ by minimizing the difference between the forward modeled data $L\mathbf{m}$ and the recorded data $d$ in a least-squares sense:

\begin{equation}
    f(\mathbf{m}) = \frac{1}{2}||L\mathbf{m}-\mathbf{d}||^{2} \hspace{0.5cm}     (11)
\end{equation}

To solve equation (11), it has been implemented here the steepest descent method with an appropriate step-length,

\begin{equation}
\textbf{m}_{k+1} = \textbf{m}_k -\alpha_k \textbf{g}_k \hspace{0.5cm}     (12)
\end{equation}
where $\textbf{g}_k$ is the gradient and $\alpha_k$ is the step-length. The gradient computation is simply taking equation (9) and instead of injecting the shot recorded $\textbf{d}$, injects the residue $\textbf{d}_{calc}-\textbf{d}_{obs}$

For now we are going to import the utilities.

```python
%matplotlib inline
import numpy as np

from devito import Operator,Eq,solve,Grid,SparseFunction,norm
from devito import TimeFunction,Function
from devito import gaussian_smooth
from devito import mmax

from devito.logger import info

from examples.seismic import Model
from examples.seismic import plot_velocity,plot_shotrecord
from examples.seismic import Receiver
from examples.seismic import PointSource
from examples.seismic import plot_image,AcquisitionGeometry
from examples.seismic import TimeAxis

from examples.seismic.self_adjoint import (setup_w_over_q)
from examples.seismic.acoustic import AcousticWaveSolver

import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import ImageGrid
from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable
import matplotlib.ticker as plticker

from devito import configuration
configuration['log-level'] = 'WARNING'
```

# Seismic modelling with Devito

Now let's import all the parameters needed and create the true 2D velocity model and the smoothed model to perform the Born's modelling.

```python
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # What is the location of the top left corner. This is necessary to define

fpeak = 0.025# Source peak frequency is 25Hz (0.025 kHz)
t0w = 1.0 / fpeak
omega = 2.0 * np.pi * fpeak
qmin = 0.1
qmax = 100000
npad=50
dtype = np.float32

nshots = 21
nreceivers = 101
t0 = 0.
tn = 1000.  # Simulation last 1 second (1000 ms)
filter_sigma = (5, 5) # Filter's length

v = np.empty(shape, dtype=dtype)

# Define a velocity profile. The velocity is in km/s
vp_top = 1.5

v[:] = vp_top # Top velocity
v[:, 30:65]= vp_top +0.5
v[:, 65:101]= vp_top +1.5
v[40:60, 35:55]= vp_top+1

init_damp = lambda func, nbl: setup_w_over_q(func, omega, qmin, qmax, npad, sigma=0)
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=8, bcs=init_damp,nbl=npad,dtype=dtype)
model0 = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=8, bcs=init_damp,nbl=npad,dtype=dtype)

dt = model.critical_dt
s = model.grid.stepping_dim.spacing
time_range = TimeAxis(start=t0, stop=tn, step=dt)
nt=time_range.num
```

```python
#NBVAL_IGNORE_OUTPUT
# Create initial model and smooth the boundaries
gaussian_smooth(model0.vp, sigma=filter_sigma)
# Plot the true and initial model
plot_velocity(model)
plot_velocity(model0)
```

Now we are going to set the source and receiver position.

```python
# First, position source centrally in all dimensions, then set depth
src_coordinates = np.empty((1, 2))
src_coordinates[0, :] = np.array(model.domain_size) * .5
src_coordinates[0, -1] = 30.

# Define acquisition geometry: receivers

# Initialize receivers for synthetic and imaging data
rec_coordinates = np.empty((nreceivers, 2))
rec_coordinates[:, 0] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec_coordinates[:, 1] = 30.

# Geometry
geometry = AcquisitionGeometry(model, rec_coordinates, src_coordinates, t0, tn, f0=fpeak, src_type='Ricker')

solver = AcousticWaveSolver(model, geometry, space_order=8)
```

```python
source_locations = np.empty((nshots, 2), dtype=dtype)
source_locations[:, 0] = np.linspace(0., 1000, num=nshots)
source_locations[:, 1] = 30. # Depth is 30m
```

We are going to compute the LSRTM gradient.

```python
def lsrtm_gradient(dm):

    residual = Receiver(name='residual', grid=model.grid, time_range=geometry.time_axis,
                        coordinates=geometry.rec_positions)

    d_obs = Receiver(name='d_obs', grid=model.grid,time_range=geometry.time_axis,
                         coordinates=geometry.rec_positions)

    d_syn = Receiver(name='d_syn', grid=model.grid,time_range=geometry.time_axis,
                         coordinates=geometry.rec_positions)

    grad_full = Function(name='grad_full', grid=model.grid)

    grad_illum = Function(name='grad_illum', grid=model.grid)

    src_illum = Function (name ="src_illum", grid = model.grid)

    # Using devito's reference of virtual source
    dm_true =  (solver.model.vp.data**(-2) - model0.vp.data**(-2))

    objective = 0.
    for i in range(nshots):

        #Observed Data using Born's operator
        geometry.src_positions[0, :] = source_locations[i, :]

        _, u0, _ = solver.forward(vp=model0.vp, save=True)

        _, _, _,_ = solver.jacobian(dm_true, vp=model0.vp, rec = d_obs)

        #Calculated Data using Born's operator
        solver.jacobian(dm, vp=model0.vp, rec = d_syn)

        residual.data[:] = d_syn.data[:]- d_obs.data[:]

        grad_shot,_ = solver.gradient(rec=residual, u=u0, vp=model0.vp)

        src_illum_upd =  Eq(src_illum, src_illum + u0**2)
        op_src = Operator([src_illum_upd])
        op_src.apply()

        grad_sum = Eq(grad_full, grad_full  + grad_shot)
        op_grad = Operator([grad_sum])
        op_grad.apply()

        grad_f = Eq(grad_illum, grad_full/(src_illum+10**-9))
        op_gradf = Operator([grad_f])
        op_gradf.apply()

        objective += .5*norm(residual)**2

    return objective,grad_illum,d_obs,d_syn
```

For the first LSRTM iteration, we used a quite simple step-length using the maximum value of the gradient. For the other LSRTM iterations we used the step-length proposed by Barzilai Borwein.
\begin{equation}
\alpha_{k}^{BB1} = \frac{\mathbf{s}_{k-1}^{T}\mathbf{s}_{k-1}}{\mathbf{s}_{k-1}^{T}\mathbf{y}_{k-1}}
\end{equation}
where $\textbf{s}_{k-1} = \textbf{m}_{k}-\textbf{m}_{k-1}$ and $\textbf{y}_{k-1} = \textbf{g}_{k}-\textbf{g}_{k-1}$

A second option is:
\begin{equation}
\alpha_{k}^{BB2} = \frac{\mathbf{s}_{k-1}^{T}\mathbf{y}_{k-1}}{\mathbf{y}_{k-1}^{T}\mathbf{y}_{k-1}}
\end{equation}

\begin{equation}
\alpha_{k} =
\begin{cases}
\alpha_{k}^{BB2},& \text{if}\ 0 < \frac{\alpha_{k}^{BB2}}{\alpha_{k}^{BB1}} < 1 \\
\alpha_{k}^{BB1},& \text{else}
\end{cases}
\end{equation}

```python
def get_alfa(grad_iter,image_iter,niter_lsrtm):

    term1 = np.dot(image_iter.reshape(-1), image_iter.reshape(-1))

    term2 = np.dot(image_iter.reshape(-1), grad_iter.reshape(-1))

    term3 = np.dot(grad_iter.reshape(-1), grad_iter.reshape(-1))

    if niter_lsrtm == 0:

        alfa = .05 / mmax(grad_full)

    else:
        abb1 = term1 / term2

        abb2 = term2 / term3

        abb3 = abb2 / abb1

        if abb3 > 0 and abb3 < 1:
            alfa = abb2
        else:
            alfa = abb1

    return alfa
```

Now is the kernel of the LSRTM. The migration will be updated iteratively, using the step-length and the gradient.

```python
#NBVAL_IGNORE_OUTPUT
image_up_dev = np.zeros((model0.vp.shape[0], model0.vp.shape[1]),dtype)

image = np.zeros((model0.vp.shape[0], model0.vp.shape[1]))

nrec=101
niter=20 # number of iterations of the LSRTM
history = np.zeros((niter, 1)) #objective function

image_prev = np.zeros((model0.vp.shape[0],model0.vp.shape[1]))

grad_prev  = np.zeros((model0.vp.shape[0],model0.vp.shape[1]))

yk  = np.zeros((model0.vp.shape[0],model0.vp.shape[1]))

sk = np.zeros((model0.vp.shape[0],model0.vp.shape[1]))

for k in range(niter) :

    dm =  image_up_dev # Reflectivity for Calculated data via Born

    print('LSRTM Iteration',k+1)

    objective,grad_full,d_obs,d_syn = lsrtm_gradient(dm)

    history[k] = objective

    yk = grad_full.data - grad_prev

    sk = image_up_dev - image_prev

    alfa = get_alfa(yk,sk,k)

    grad_prev = grad_full.data

    image_prev = image_up_dev

    image_up_dev = image_up_dev - alfa*grad_full.data

    if k == 0: # Saving the first migration using Born operator.

        image = image_up_dev
```

    LSRTM Iteration 1
    LSRTM Iteration 2
    LSRTM Iteration 3
    LSRTM Iteration 4
    LSRTM Iteration 5
    LSRTM Iteration 6
    LSRTM Iteration 7
    LSRTM Iteration 8
    LSRTM Iteration 9
    LSRTM Iteration 10
    LSRTM Iteration 11
    LSRTM Iteration 12
    LSRTM Iteration 13
    LSRTM Iteration 14
    LSRTM Iteration 15
    LSRTM Iteration 16
    LSRTM Iteration 17
    LSRTM Iteration 18
    LSRTM Iteration 19
    LSRTM Iteration 20

```python
#NBVAL_SKIP
plt.figure()
plt.plot(history)
plt.xlabel('Iteration number')
plt.ylabel('Misift value Phi')
plt.title('Convergence')
plt.show()
```

```python
def plot_image(data, vmin=None, vmax=None, colorbar=True, cmap="gray"):
    """
    Plot image data, such as RTM images or FWI gradients.

    Parameters
    ----------
    data : ndarray
        Image data to plot.
    cmap : str
        Choice of colormap. Defaults to gray scale for images as a
        seismic convention.
    """
    domain_size = 1.e-3 * np.array(model.domain_size)
    extent = [model.origin[0], model.origin[0] + domain_size[0],
              model.origin[1] + domain_size[1], model.origin[1]]
    plot = plt.imshow(np.transpose(data),
                      vmin=-.05,
                      vmax=.05,
                      cmap=cmap,extent=extent)

    plt.xlabel('X position (km)')
    plt.ylabel('Depth (km)')

    # Create aligned colorbar on the right
    if colorbar:
        ax = plt.gca()
        divider = make_axes_locatable(ax)
        cax = divider.append_axes("right", size="5%", pad=0.05)
        plt.colorbar(plot, cax=cax)
    plt.show()

```

The image below is our first migration. The reflectors are not well focused and backscaterring noise is very strong.

```python
#NBVAL_IGNORE_OUTPUT
slices=tuple(slice(model.nbl,-model.nbl) for _ in range(2))
rtm = image[slices]
plot_image(np.diff(rtm, axis=1))
```

So here we have the LSRTM migration after 20 iterations, it is clear that the reflector is well focused and backscaterring noise has been strongly attenuated.

```python
#NBVAL_SKIP
slices=tuple(slice(model.nbl,-model.nbl) for _ in range(2))
lsrtm = image_up_dev[slices]
plot_image(np.diff(lsrtm, axis=1))
```

Here we have the true reflectivity. The idea in showing it is to demonstrate that the amplitude range of the LSRTM is in the same amplitude range of the true reflectivity.

```python
#NBVAL_IGNORE_OUTPUT
slices=tuple(slice(model.nbl,-model.nbl) for _ in range(2))
dm_true = (solver.model.vp.data**(-2) - model0.vp.data**(-2))[slices]
plot_image(np.diff(dm_true, axis=1))
```

```python
#NBVAL_SKIP
plt.figure(figsize=(8,9))
x = np.linspace(0,1,101)
plt.plot(rtm[50,:],x,color=plt.gray(),linewidth=2)
plt.plot(lsrtm[50,:],x,'r',linewidth=2)
plt.plot(dm_true[50,:],x, 'k--',linewidth=2)

plt.legend(['Initial reflectivity', 'Reflectivity via LSRTM','True Reflectivity'],fontsize=15)
plt.ylabel('Depth (Km)')
plt.xlabel('Amplitude')
plt.gca().invert_yaxis()
plt.show()
```

```python
#NBVAL_SKIP
time = np.linspace(t0, tn, nt)
plt.figure(figsize=(8,9))
plt.ylabel('Time (ms)')
plt.xlabel('Amplitude')
plt.plot(d_syn.data[:, 20],time, 'y', label='Calculated data (Last Iteration)')
plt.plot(d_obs.data[:, 20],time, 'm', label='Observed data')
plt.legend(loc="upper left",fontsize=12)
ax = plt.gca()
ax.invert_yaxis()
plt.show()
```


# Homework 1

### Due Date: Thursday, July 30, PST, 2020

### Policy Gradient

In this assignment, we will implement vanilla policy gradient algorithm (REINFORCE) covered in the lecture. You will work on i) a function approximator, ii) computing action, iii) collecting samples, iV) training the agent, V) plotting the resutls.

***Complete the missing operations and test your implemented algorithm on the Gym environment.***

***Software requirements:***
- Python >= 3.6
- Tensorflow version <= 1.15.3 (1.X version)
- OpenAI Gym

- Training the agent (policy) can take long time. It is recomended to start solving the problems earlier.

- Save any plots you generated in this notebook. The grade will be given based on the plots you showed.

Make sure the packages you installed meet the requirements.

```python
import tensorflow as tf
tf.__version__
```

```python
import gym
gym.__version__
```

## 1.1 Tensorflow Implementation

We will be implementing policy gradient algorithm using Tensorflow 1.X., which simply updates the parameters of policy from obtaining gradient estimates. The core of policy gradient is to design a function approximator, computing actions, collecting samples, and training the policy. In the below cell, you are encouraged to fill in the components that are missing. ***Your tasks*** are

1. Complete the 'create_model' method to output the mean value for diagonal Guassian policy. Covariance is already defined in the model, so focus on creating neural network model.

2. Complete the 'action_op' method to calculate and return the actions for diagonal Gaussian policy. The applied action should be $\pi(s) = \pi_{\text{mean}}(s) + exp(logstd) * \mathcal{N}(0,1)$

***Hints***:
- Some useful tensorflow classes and methods include: 'tf.exp', 'tf.random_normal'

***IF you are using MAC, please run below box***

```python
import os
# MAC user only
os.environ['KMP_DUPLICATE_LIB_OK']='True'
```

```python
import tensorflow as tf
import numpy as np
import os
import ipdb

class PolicyOpt(object):

    def __init__(self, env, linear=False, stochastic=True, hidden_size=32, nonlinearity=tf.nn.relu):
        """Instantiate the policy iteration class.

        This initializes the policy optimization with a set of trainable
        parameters, and creates a tensorflow session.

        Attributes
        ----------
        env : gym.Env
            the environment that the policy will be trained on
        linear : bool, optional
            specifies whether to use a linear or neural network
            policy, defaults to False (i.e. Fully-Connected-Neural-Network)
        stochastic : bool, optional
            specifies whether to use a stochastic or deterministic
            policy, defaults to True
        hidden_size : list of int, optional
            list of hidden layers, with each value corresponding
            to the number of nodes in that layer
        nonlinearity : tf.nn.*
            activation nonlinearity
        """

        # clear computation graph
        tf.reset_default_graph()

        # set a random seed
        tf.set_random_seed(1234)

        # start a tensorflow session
        self.sess = tf.Session()

        # environment to train on
        self.env = env

        # number of elements in the action space
        self.ac_dim = env.action_space.shape[0]

        # number of elements in the observation space
        self.obs_dim = env.observation_space.shape[0]

        # actions placeholder
        self.a_t_ph = tf.placeholder(dtype=tf.float32,
                                     shape=[None, self.ac_dim])
        # state placeholder
        self.s_t_ph = tf.placeholder(dtype=tf.float32,
                                     shape=[None, self.obs_dim])
        # expected reward placeholder
        self.rew_ph = tf.placeholder(dtype=tf.float32,
                                     shape=[None])

        # specifies whether the policy is stochastic
        self.stochastic = stochastic

        # policy that the agent executes during training/testing
        self.policy = self.create_model(
            args={
                "num_actions": self.ac_dim,
                "hidden_size": hidden_size,
                "linear": linear,
                "nonlinearity": nonlinearity,
                "stochastic": stochastic,
                "scope": "policy",
            }
        )

        # define symbolic action
        self.symbolic_action = self.action_op()

        # initialize all variables
        self.sess.run(tf.global_variables_initializer())

        # create saver to save model variables
        self.saver = tf.train.Saver()

    def create_model(self, args):
        """Create a model for your policy or other components.

        Parameters
        ----------
        args : dict
            model-specific arguments, with keys:
              - "stochastic": True by default
              - "hidden_size": Number of neurons in hidden layer
              - "num_actions" number of output actions
              - "scope": scope of the model

        Returns
        -------
        tf.Variable
            mean actions of the policy
        tf.Variable
            logstd of the policy actions
        """

#################### Build Your Neural Network Here! ####################
        ??
        ??

##########################################################################

        if args["stochastic"]:
            output_logstd =  tf.get_variable(name="action_logstd",shape=[self.ac_dim],trainable=True)
        else:
            output_logstd = None

        return output_mean, output_logstd

    def action_op(self):
        """
        Create a symbolic expression that will be used to compute actions from observations.

        When the policy is stochastic, the action follows

            a_t = output_mean + exp(output_logstd) * z; z ~ N(0,1)
        """
        if self.stochastic:
            output_mean, output_logstd = self.policy

            #################### Implement a stochastic policy here ####################
            # Implement a stochastic version of computing actions.       #
            #                                                            #
            # The action in a stochastic policy represented by           #
            # a diagonal Gaussian distribution with mean "M" and log     #
            # standard deviation "logstd" is computed as follows:        #
            #                                                            #
            #     a = M + exp(logstd) * z                                #
            #                                                            #
            # where z is a random normal value, i.e. z ~ N(0,1)          #
            #                                                            #
            # In order to generate numbers from a normal distribution,   #
            # use the `tf.random_normal` function.                       #
            ############################################################################
            symbolic_action = ?

        else:
            symbolic_action, _ = self.policy

        return symbolic_action

    def compute_action(self, obs):
        """Returns a list of actions for a given observation.

        Parameters
        ----------
        obs : np.ndarray
            observations

        Returns
        -------
        np.ndarray
            actions by the policy for a given observation
        """
        return self.sess.run(self.symbolic_action,feed_dict={self.s_t_ph: obs})

    def rollout(self, s_mean=None, s_std=None):
        """Collect samples from one rollout of the policy.

        Returns
        -------
        dict
            dictionary containing trajectory information for the rollout,
            specifically containing keys for "state", "action", "next_state", "reward", and "done"
        """
        states = []
        next_states = []
        actions = []
        rewards = []
        dones = []

        # start a new rollout by re-setting the environment and collecting the initial state
        state =  self.env.reset()

        steps = 0
        while True:
            steps += 1

            # compute the action given the state
            if s_mean is not None and s_std is not None:
                action = self.compute_action([(state - s_mean) / s_std])
            else:
                action = self.compute_action([state])
            action = action[0]

            # advance the environment once and collect the next state, reward, done, and info parameters from the environment
            next_state, reward, done, info =  self.env.step(action)

            # add to the samples list
            states.append(state)
            actions.append(action)
            next_states.append(next_state)
            rewards.append(reward)
            dones.append(done)

            state = next_state

            # if the environment returns a True for the done parameter,
            # end the rollout before the time horizon is met
            if done or steps > env._max_episode_steps:
                break

        # create the output trajectory
        trajectory = {"state": np.array(states, dtype=np.float32),
                      "reward": np.array(rewards, dtype=np.float32),
                      "action": np.array(actions, dtype=np.float32),
                      "next_state": np.array(next_states, dtype=np.float32),
                      "done": np.array(dones, dtype=np.float32)}

        return trajectory

    def train(self, args):
        """Abstract training method.

        This method will be filled in by algorithm-specific
        training operations in subsequent problems.

        Parameters
        ----------
        args : dict
            algorithm-specific hyperparameters
        """
        raise NotImplementedError

```

## 1.2 Tensorflow Interpretation

In order to test your implementation of the **stochastic policy**, run the below cell. The task is to interpret the code you implemented in previous section. If you implement correctly, you can see the value_1 and value_2.

***Question: How do you interpret value_1 and value_2 below cell?***

```python
import gym
TEST_ENV = gym.make("Pendulum-v0")

alg = PolicyOpt(TEST_ENV, linear=False)
input_1 = [[0, 1, 2]]
value_1 = alg.sess.run(alg.policy[0], feed_dict={alg.s_t_ph: input_1})
value_2 = alg.compute_action(input_1)
```

```python
value_1

```

Answer:

```python
value_2
```

Answer:

## 1.3 Implement Policy Gradient

In this section, we will implement REINFORCE algorithm presented in the lecture. As a review, the objective is optimize the parameters $\theta$ of some policy $\pi_\theta$ so that the expected return

\begin{equation}
J(\theta) = \mathbb{E} \bigg\{ \sum_{t=0}^T \gamma^t r(s_{t},a_{t}) \bigg\}
\end{equation}

is optimized. In this algorithm, this is done by calculating the gradient $\nabla_\theta J$ and applying a gradient descent method to find a better policy.

\begin{equation}
\theta ' = \theta + \alpha \nabla_\theta J(\theta)
\end{equation}

In the lecture, we derive how we compute $\nabla_{\theta} J(\theta)$. We can rewrite our policy gradient as:

\begin{equation}
\nabla_\theta J (\theta) \approx \frac{1}{N} \sum_{i=0}^{N} \bigg( \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta (a_{it} | s_{it}) \bigg) \bigg( \sum_{t=0}^T \gamma^{t}r_i(t) \bigg)
\end{equation}

Finally, taking into account the causality principle discussed in class, we are able to simplifiy the gradient estimate such as:

\begin{equation}
\nabla_\theta J (\theta) \approx \frac{1}{N} \sum_{i=0}^{N} \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta (a_{it} | s_{it}) \sum_{t'=t}^T \gamma^{t'-t}r_i(t')
\end{equation}

You will be implementing final expression in this assignment!

The process of REINFOCE algorithm follows:

1. Collect samples from current policy $\pi_\theta(s)$ by executing rollouts of the environment.
2. Calculate an estimate for the expected return at state $s_t$.
3. Compute the log-likelihood of each action that was performed by the policy at every given step.
4. Estimate the gradient and update the parameters of policy using gradient-based technique.
5. Repeat steps 1-4 for a number of training iterations.

***Your task*** is to fill out the skeleton code for REINFORCE algorithm,

1. Complete the 'log_likelihoods' method to compute gradient of policy, $\nabla_{\theta}\pi_{\theta}$ for diagonal Guassian policy.

2. Complete the 'compute_expected_return' method to calculate the reward-to-go, $\sum_{t^{\prime}=t}^{T}$.

```python
import tensorflow as tf
import numpy as np
import time
import tensorflow_probability as tfp

class REINFORCE(PolicyOpt):

    def train(self, num_iterations=1000, steps_per_iteration=1000, learning_rate=int(1e-4), gamma=0.95,
              **kwargs):
        """Perform the REINFORE training operation.

        Parameters
        ----------
        num_iterations : int
            number of training iterations
        steps_per_iteration : int
            number of individual samples collected every training iteration
        learning_rate : float
            optimizer learning rate
        gamma : float
            discount rate
        kwargs : dict
            additional arguments

        Returns
        -------
        list of float
            average return per iteration
        """
        # set the discount as an attribute
        self.gamma = gamma

        # set the learning rate as an attribute
        self.learning_rate = learning_rate

        # create a symbolic expression to compute the log-likelihoods
        log_likelihoods = self.log_likelihoods()

        # create a symbolic expression for updating the parameters of your policy
        self.opt, self.opt_baseline = self.define_updates(log_likelihoods)

        # initialize all variables
        self.sess.run(tf.global_variables_initializer())

        # average return per training iteration
        ret_per_iteration = []

        samples = []
        for i in range(num_iterations):

            # collect samples from the current policy
            samples.clear()
            steps_so_far = 0
            while steps_so_far < steps_per_iteration:
                new_samples = self.rollout()
                steps_so_far += new_samples["action"].shape[0]
                samples.append(new_samples)

            # compute the expected returns
            v_s = self.compute_expected_return(samples)

            # compute and apply the gradients
            self.call_updates(log_likelihoods, samples, v_s, **kwargs)

            # compute the average cumulative return per iteration
            average_rew = np.mean([sum(s["reward"]) for s in samples])

            # display iteration statistics
            print("Iteration {} return: {}".format(i, average_rew))
            ret_per_iteration.append(average_rew)

        return ret_per_iteration

    def log_likelihoods(self):
        """Create a tensorflow operation that computes the log-likelihood
        of each performed action.
        """

        output_mean, output_logstd = self.policy

        ##############################################################
        # Create a tf operation to compute the log-likelihood of     #
        # each action that was performed by the policy               #
        #                                                            #
        # The log likelihood in the continuous case where the policy #
        # is expressed by a multivariate gaussian can be computing   #
        # using the tensorflow object:                               #
        #                                                            #
        #    p = tfp.distributions.MultivariateNormalDiag(           #
        #        loc=...,                                            #
        #        scale_diag=...,                                     #
        #    )                                                       #
        #                                                            #
        # This method takes as input a mean (loc) and standard       #
        # deviation (scale_diag), and then can be used to compute    #
        # the log-likelihood of a variable as follows:               #
        #                                                            #
        #    log_likelihoods = p.log_prob(...)                       #
        #                                                            #
        # For this operation, you will want to use placeholders      #
        # created in the __init__ method of problem 1.               #
        ##############################################################

        p = ?
        log_likelihoods = ?

        return log_likelihoods

    def compute_expected_return(self, samples):
        """Compute the expected return from a given starting state.
        This is done by using the reward-to-go method.

        Parameters
        ----------
        rewards : list of list of float
            a list of N trajectories, with each trajectory contain T
            returns values (one for each step in the trajectory)

        Returns
        -------
        list of float, or np.ndarray
            expected returns for each step in each trajectory
        """
        rewards = [s["reward"] for s in samples]

        ##############################################################
        # Estimate the expected return from any given starting state #
        # using the reward-to-go method.                             #
        #                                                            #
        # Using this method, the reward is estimated at every step   #
        # of the trajectory as follows:                              #
        #                                                            #
        #   r = sum_{t'=t}^T gamma^(t'-t) * r_{t'}                   #
        #                                                            #
        # where T is the time horizon at t is the index of the       #
        # current reward in the trajectory. For example, for a given #
        # set of rewards r = [1,1,1,1] and discount rate gamma = 1,  #
        # the expected reward-to-go would be:                        #
        #                                                            #
        #   v_s = [4, 3, 2, 1]                                       #
        #                                                            #
        # You will be able to test this in one of the cells below!   #
        ##############################################################
        v_s = ?

        return v_s

    def define_updates(self, log_likelihoods):
        """Create a tensorflow operation to update the parameters of
        your policy.

        Parameters
        ----------
        log_likelihoods : tf.Operation
            the symbolic expression you created to estimate the log
            likelihood of a set of actions

        Returns
        -------
        tf.Operation
            a tensorflow operation for computing and applying the
            gradients to the parameters of the policy
        None
            the second component is used in problem 2.b, please ignore
            for this problem
        """

        loss = - tf.reduce_mean(tf.multiply(log_likelihoods, self.rew_ph))
        opt = tf.train.AdamOptimizer(self.learning_rate).minimize(loss)

        return opt, None

    def call_updates(self, log_likelihoods, samples, v_s, **kwargs):
        """Apply the gradient update methods in a tensorflow session.

        Parameters
        ----------
        log_likelihoods: tf.Operation
            the symbolic expression you created to estimate the log
            likelihood of a set of actions
        samples : list of dict
            a list of N trajectories, with each trajectory containing
            a dictionary of trajectory data (see self.rollout)
        v_s : list of float, or np.ndarray
            the estimated expected returns from your
            `comput_expected_return` function
        kwargs : dict
            additional arguments (used in question 3)
        """
        # concatenate the states
        states = np.concatenate([s["state"] for s in samples])

        # concatenate the actions
        actions = np.concatenate([s["action"] for s in samples])

        # execute the optimization step
        self.sess.run(self.opt, feed_dict={self.s_t_ph: states,
                                           self.a_t_ph: actions,
                                           self.rew_ph: v_s})
```

Check your 'log_likelihoods' method by running below cell:

```python
alg = REINFORCE(TEST_ENV, stochastic=True)

log_likelihoods = alg.log_likelihoods()

# collect a sample output for a given input state
input_s = [[0, 0, 0], [0, 1, 2], [1, 2, 3]]
input_a = [[0], [1], [2]]

# Check
computed = alg.sess.run(log_likelihoods, feed_dict={alg.a_t_ph: input_a, alg.s_t_ph: input_s})
```

Test your 'compute_expected_return' by running below cell:

```python
# 1. Test the non-normalized case
alg = REINFORCE(TEST_ENV, stochastic=True)
alg.gamma = 1.0

input_1 = [{"reward": [1, 1, 1, 1]},
           {"reward": [1, 1, 1, 1]}]
vs_1 = alg.compute_expected_return(samples=input_1)
ans_1 = np.array([4, 3, 2, 1, 4, 3, 2, 1])

if np.linalg.norm(vs_1 - ans_1) < 1e-3:
    print('Great job!')
else:
    print('Check your implementation (compute_expected_return)')

```

## 1.4 Testing your algorithm

When you are ready, test your policy gradient algorithms on the *Pendulum-v0* environment in the cell below. *Pendulum-v0* environment is similar to *off-shore wind power*, the goal here is to maintain the Pendulum is upright using control input. The best policy should get around -200 scores. ***Your task*** is to run your REINFORCE algorithm and plot the result!

```python
# set this number as 1 for testing your algorithm, and 3 for plotting
NUM_TRIALS = 3

# ===========================================================================
# Do not modify below line
# ===========================================================================

# we will test the algorithms on the Pendulum-v0 gym environment
import gym
env = gym.make("Pendulum-v0")

# train on the REINFORCE algorithm
import numpy as np
r = []
for i in range(NUM_TRIALS):
    print("\n==== Training Run {} ====".format(i))
    alg = REINFORCE(env, stochastic=True)
    res = alg.train(learning_rate=0.005, gamma=0.95, num_iterations=500, steps_per_iteration=15000)
    r.append(np.array(res))
    alg = None

# save results
np.savetxt("InvertedPendulum_results.csv", np.array(r), delimiter=",")
```

```python
# collect saved results
import numpy as np
r1 = np.genfromtxt("InvertedPendulum_results.csv", delimiter=",")
all_results = [r1]
labels = ["REINFORCE"]

##############################################################
# Plot your Policy Gradient results below
##############################################################
?
```

The Policy gradient result looks like this:

```python

```


# 12 Root Finding

An important tool in the computational tool box is to find roots of equations for which no closed form solutions exist:

We want to find the roots $x_0$ of

$$
f(x_0) = 0
$$

## Problem: Projectile range
The equations of motion for the projectile with linear air resistance (see *12 ODE applications*) can be solved exactly.

As a reminder: the linear drag force is
$$
\mathbf{F}_1 = -b_1 \mathbf{v}\\
b := \frac{b_1}{m}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -b \mathbf{v}
\end{align}

### Analytical solution of the equations of motion
(Following Wang Ch 3.3.2)

Solve $x$ component of the velocity

$$
\frac{dv_x}{dt} = -b v_x
$$

by integration:

$$
v_x(t) = v_{0x} \exp(-bt)
$$

The drag force reduces the forward velocity to 0.

Integrate again to get the $x(t)$ component

$$
x(t) = x_0 + \frac{v_{0x}}{b} \left[1 - \exp(-bt)\right]
$$

Integrating the $y$ component of the velocity

$$
\frac{dv_y}{dt} = -g - b v_y
$$

gives

$$
v_y = \left(v_{0y} + \frac{g}{b}\right) \exp(-bt) - \frac{g}{b}
$$

and integrating again

$$
y(t) = y_0 + \frac{v_{0y} + \frac{g}{b}}{b} \left[1 - \exp(-bt)\right] - \frac{g}{b} t
$$

(Note: This shows immediately that the *terminal velocity* is

$$
\lim_{t\rightarrow\infty} v_y(t) = - \frac{g}{b},
$$

i.e., the force of gravity is balanced by the drag force.)

#### Analytical trajectory

To obtain the **trajectory $y(x)$** eliminate time (and for convenience, using the origin as the initial starting point, $x_0 = 0$ and $y_0 = 0$. Solve $x(t)$ for $t$

$$
t = -\frac{1}{b} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

and insert into $y(t)$:

$$
y(x) = \frac{x}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bx}{v_{0x}}\right)
$$

#### Plot

Plot the analytical solution for $\theta = 30^\circ$ and $v_0 = 100$~m/s.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
```

```python
def y_lindrag(x, v0, b1=0.2, g=9.81, m=0.5):
    b = b1/m
    v0x, v0y = v0
    return x/v0x * (v0y + g/b) + g/(b*b) * np.log(1 - b*x/v0x)

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

The analytical function drops *very* rapidly towards the end ($> 42$ m – found by manual trial-and-error plotting...) so in order to nicely plot the function we use a fairly coarse sampling of points along $x$ for the range $0 \le x < 42$ and very fine sampling for the last 2 m ($42 \le x < 45$):

```python
X = np.concatenate([np.linspace(0, 42, 100), np.linspace(42, 45, 10000)])
```

Evaluate the function for all $x$ values:

```python
Y = y_lindrag(X, initial_v(100, 30), b1=1)
```

    /Users/oliver/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:4: RuntimeWarning: invalid value encountered in log

(The warning can be ignored, it just means that some of our `X` values were not approriate and outside the range of validity – when the argument of the logarithm becomes ≤0).

To indicate the ground we also plot a dashed black line: note that the analytical solution goes below the dashed line.

```python
plt.plot(X, Y)
plt.xlabel("$x$ (m)")
plt.ylabel("$y$ (m)")
plt.hlines([0], X[0], X[-1], colors="k", linestyles="--");
```

Compare to the numerical solution (from 12 ODE Applications):

```python
import ode

def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):
    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)
```

```python
r = simulate(initial_v(100, 30), h=0.01, b1=1)
```

```python
plt.plot(X, Y, lw=2, label="analytical")
plt.plot(r[:, 1], r[:, 2], '--', label="RK4")
plt.legend(loc="best")
plt.xlabel("$x$ (m)"); plt.ylabel("$y$ (m)")
plt.hlines([0], X[0], X[-1], colors="k", linestyles="--");
```

The RK4 solution tracks the analytical solution perfectly (and we also programmed it to not go below ground...)

### Predict the range $R$
How far does the ball or projectile fly, i.e., that value $x=R$ where $y(R) = 0$:

$$
\frac{R}{v_{0x}} \left( v_{0y} + \frac{g}{b} \right) + \frac{g}{b^2} \ln \left(1 - \frac{bR}{v_{0x}}\right) = 0
$$

This *transcendental equation* can not be solved in terms of elementary functions.

Use a **root finding** algorithm.

## Root-finding with the Bisection algorithm
**Bisection** is the simplest (but very robust) root finding algorithm that uses trial-and-error:

* bracket the root
* refine the brackets
* see [12_Root-finding-algorithms (PDF)](12_Root-finding-algorithms.pdf)

More specifically
1. determine a bracket that contains the root: $x_{-} < x_0 < x_{+}$
2. cut bracket in half: $x' = \frac{1}{2}(x_{+} + x_{-})$
3. determine in which half the root lies: $f(x_{-}) f(x') < 0$ then the root lies in the left half, otherwise the right half.
4. repeat until $|f(x)| < \epsilon$.

### Implementation of Bisection

- Test that the initial bracket contains a root; if not, return `None` (and possibly print a warning).
- If either of the bracket points is a root then return the bracket point.
- Allow `Nmax` iterations or until the convergence criterion `eps` is reached.
- Bonus: print a message if no root was found after `Nmax` iterations, but print the best guess and the error (but return `None`).

```python
def bisection(f, xminus, xplus, Nmax=100, eps=1e-14):
    fa, fb = f(xminus), f(xplus)
    if (fa*fb) > 0:
        print("bisect: Initial bracket [{0}, {1}] "
              "does not contain a single root".format(xminus, xplus))
        return None
    if np.abs(fa) < eps:
        return xminus
    if np.abs(fb) < eps:
        return xplus

    for iteration in range(Nmax):
        x = (xplus + xminus)/2
        fx = f(x)
        if f(xplus) * fx > 0:
            # root is not between xplus and x
            xplus = x
        else:
            xminus = x
        if np.abs(fx) < eps:
            break
    else:
        print("bisect: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x
```

### Finding the range with the bisection algorithm

Define the trial function `f`.

Note that our `y_lindrag()` function depends on `x` **and** `v` but `bisect()` only accepts functions `f` that depend on a *single variable*, $f(x)$. We therefore have to wrap `y_lindrag(x, v)` into a function `f(x)` that sets `v` already to a value *outside* the function: [Python's scoping rules](https://stackoverflow.com/questions/291978/short-description-of-the-scoping-rules#292502) say that inside the function `f(x)`, the variable `x` has the value assigned to the argument of `f(x)` but any other variables such as `v` or `b1`, which were *not defined inside `f`*, will get the value that they had *outside `f`* in the *enclosing code*.

```python
v = initial_v(100, 30)

def f(x):
    return y_lindrag(x, v, b1=b1)
```

The initial bracket is a little bit difficult for this function: choose the right bracket near the point where the argument of the logarithm becomes 0 (which is actually the maximum $x$ value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$):

$$
x_{+} = \frac{v_{0x}}{b} - \epsilon
$$

```python
b1 = 1.
m = 0.5
b = b1/m
bisection(f, 0.1, v[0]/b - 1e-12, eps=1e-6)
```

    43.300674233470772

Note that this solution is *not* the maximum value $\lim_{t\rightarrow +\infty} x(t) = \frac{v_{0x}}{b}$:

```python
v[0]/b
```

    43.301270189221938

### Find the range as a function of the initial angle

```python
b1 = 1.
m = 0.5
b = b1/m
v0 = 100
u = []
for theta in np.arange(1, 90):
    v = initial_v(v0, theta)
    def f(x):
        return y_lindrag(x, v, b1=b1)
    R = bisection(f, 0.1, v[0]/b - 1e-16, eps=1e-5)
    if R is not None:
        u.append((theta, R))
u = np.array(u)
```

    /Users/oliver/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:4: RuntimeWarning: divide by zero encountered in log

```python
plt.plot(u[:, 0], u[:, 1])
plt.xlabel(r"launch angle $\theta$ ($^\circ$)")
plt.ylabel(r"range $R$ (m)");
```

Write a function `find_range()` to calculate the range for a given initial velocity $v_0$ and plot $R(\theta)$ for $10\,\text{m/s} ≤ v_0 ≤ 100\,\text{m/s}$.

```python
def find_range(v0, b1=1, m=0.5):
    b = b1/m
    u = []
    for theta in np.arange(1, 90):
        v = initial_v(v0, theta)
        def f(x):
            return y_lindrag(x, v, b1=b1)
        R = bisection(f, 0.1, v[0]/b - 1e-16, eps=1e-5)
        if R is not None:
            u.append((theta, R))
    return np.array(u)
```

```python
for v0 in (10, 25, 50, 75, 100):
    u = find_range(v0)
    plt.plot(u[:, 0], u[:, 1], label="{} m/s".format(v0))
plt.legend(loc="best")
plt.xlabel(r"$\theta$ (degrees)")
plt.ylabel(r"range $R$ (m)")
```

As a bonus, find the dependence of the *optimum launch angle* on the initial velocity, i.e., that angle that leads to the largest range:

```python
np.argmax(u[:, 1])
```

    10

```python
velocities = np.linspace(5, 100, 100)
results = []  # (v0, theta_opt)
for v0 in velocities:
    u = find_range(v0)
    thetas, ranges = u.transpose()
    # find index for the largest range and pull corresponding theta
    theta_opt = thetas[np.argmax(ranges)]
    results.append((v0, theta_opt))
results = np.array(results)

plt.plot(results[:, 0], results[:, 1])
plt.xlabel(r"velocity $v_0$ (m/s)")
plt.ylabel(r"$\theta_\mathrm{best}$ (degrees)")
```

The launch angle decreases with the velocity.

The steps in the graph are an artifact of choosing to only calculate the trajectories for integer angles (see `for theta in np.arange(1, 90)` in `find_range()`).

## Newton-Raphson algorithm
(see derivation in class and in the PDF or [Newton's Method](http://mathworld.wolfram.com/NewtonsMethod.html) on MathWorld)

### Activity: Implement Newton-Raphson
1. Implement the Newton-Raphson algorithm
2. Test with $g(x)$.

   $$
   g(x) = 2 \cos x - x
   $$

3. Bonus: test performance of `newton_raphson()` against `bisection()`.

```python
def g(x):
    return 2*np.cos(x) - x
```

```python
xvals = np.linspace(0, 7, 30)
plt.plot(xvals, np.zeros_like(xvals), 'k--')
plt.plot(xvals, g(xvals))
```

```python
def newton_raphson(f, x, h=1e-3, Nmax=100, eps=1e-14):
    """Find root x0 so that f(x0)=0 with the Newton-Raphson algorithm"""
    for iteration in range(Nmax):
        fx = f(x)
        if np.abs(fx) < eps:
            break
        df = (f(x + h/2) - f(x - h/2))/h
        Delta_x = -fx/df
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x
```

```python
x0 = newton_raphson(g, 2)
print(x0)
```

    1.02986652932

```python
g(x0)
```

    6.6613381477509392e-16

But note that the algorithm only converges well near the root. With other values it might not converge at all:

```python
newton_raphson(g, 3)
```

    Newton-Raphson: no root found after 100 iterations (eps=1e-14); best guess is -9322690.425062027 with error 16607175.892541457

```python
newton_raphson(g, 10)
```

    1.0298665293222589

```python
newton_raphson(g, 15)
```

    Newton-Raphson: no root found after 100 iterations (eps=1e-14); best guess is 52.96931452309739 with error -117.5878129311007

Let's look how Newton-Raphson iterates: also return all intermediate $x$ values:

```python
def newton_raphson_with_history(f, x, h=1e-3, Nmax=100, eps=1e-14):
    xvals = []
    for iteration in range(Nmax):
        fx = f(x)
        if np.abs(fx) < eps:
            break
        df = (f(x + h/2) - f(x - h/2))/h
        Delta_x = -fx/df
        x += Delta_x
        xvals.append(x)
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
              "best guess is {2} with error {3}".format(Nmax, eps, x, fx))
        x = None
    return x, np.array(xvals)
```

```python
x = {}

x0, xvals = newton_raphson_with_history(g, 1.5)
x[1.5] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))

x0, xvals = newton_raphson_with_history(g, 5)
x[5] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))

x0, xvals = newton_raphson_with_history(g, 10)
x[10] = xvals
print("root x0 = {} after {} iterations".format(x0, len(xvals)))
```

    root x0 = 1.0298665293222589 after 4 iterations
    root x0 = 1.0298665293222589 after 58 iterations
    root x0 = 1.0298665293222589 after 21 iterations

```python
for xstart in sorted(x.keys()):
    plt.semilogx(x[xstart], label=str(xstart))
plt.legend(loc="best")
plt.xlabel("iteration")
plt.ylabel("current guess for root $x_0$")
```

```python

```


(c) October 2016 - This notebook was created by [Oriol Pujol Vila](http://www.maia.ub.es/~oriol).

# Probabilistical models

# Parametric versus non parametric

Does the model have a finite set of parameteres or do the parameters grow with the amount of data? The kind of models that behave as the first type are called **parametric models**. On the contrary, those models with a number of parameters depending on the size of the data set are called **non-parametric models**. Parametric models are faster to use but make stronger assumptions about the nature of the data distributions. Non-parametric models are more flexible but may be computationally intractable when the amount of data is large.

# A simple non-parametric model

A simple example of a non-parametric classifier is the K nearest neighbor. This simple model looks for the most similar K points in the training set and counts the amount of elements for each class and returns the empirical fraction as teh estimate,

$$\mathbb{P}(y=c|x,D,K) = \frac{1}{K}\sum_{i\in N_K(x,D)} \mathbb{1}(y_i=c)$$

where $N_K(x,D)$ are the inidces of the $K$ nearest points to $x$ in $D$ and $\mathbb{1}(c)$ is the indicator funtion. This is an example of **memory-based** or **instance-based learning**.

# Probabilistic models

Probabilistic models are based on the modeling de distribution of data $p(x)$. In this sense they are a natural choice for unsupervised data analysis. As we will see later the key concept in probabilistic models is going to be the factorization of the joint probability density function by assuming or explicitly showing independences.

In the supervised setting we are looking for modeling the conditional distribution, $p(y|x)$. This is the probability of the labels having observed a certain sample. In this setting the joint probability of labels and data $p(x,y)$ is not necessary and we will be using Bayes Rule, that states the following,

$$p(Y=c|X=x_a) = \frac{p(X=x_a|Y=c)p(Y=c)}{p(X=x_a)} = \frac{p(X=x_a|Y=c)p(Y=c)}{\int p(X=x|Y=y)p(Y=y) dy}$$

A classifier that takes into account the former rule is called **generative classifier** since it specifies how to generate data using **class conditional density or likelihood** $p(X=x_a|Y=c)$ and the **class prior**, $p(Y=c)$.

Observe that the likelihood $p(X=x_a|Y=c)$, of $X=x_a$ does not necessarily adds to one when considered accross the all the models, i.e. $\sum\limits_{i=0}^{|Y|} p(X=x_a|Y=c_i) \neq 1$ because $p(X=x_a|Y=c_i)$ accounts for the probability the model for class $c_i$ assigns to example $x_a$. It may happen that all models have very small probability value for that sample and not necessarily add to one. Note however that given a certain model for class $c_i$, i.e. $p(X=x|Y=c_i)$ then $\sum\limits_{k=1}^{N} p(X=x_k|Y=c_i) = 1$

An alternative probabilistic approach to generative classifiers are **discriminative classifiers** that directly fit the **class posterior** $p(y = c|x)$.

# Generative models for discrete data

Let us start our discusion on probabilistic models

Recall that by applying Bayes rule we have that the probability of a sample to belong to one class is

$$p(y=c|x,\theta) \propto p(x|y = c,\theta)p(y = c|\theta)$$

The key to using these models is specifying a suitable form for the class-conditional density $p(x|y = c,\theta)$ which defines what kind of data we expect to see in each class according to some parameters $\theta$ governing the model.

In this context the **likelihood** accounts for the probability of generating that particular data assuming it belongs to a certain model or class. The **prior** is a subjective belief of probable is this model.

 **EXERCISE: **

Following Murphy's example, suppose that we are hypothesizing about a number generating function. Up to this point we don't have much information. We can, however, hypothesize about the process generating the values. We could think about many different hypothesis, such as `number generator`, `even number generator`, `odd number generator` , `numbers ended with 9`, `even numbers except 32`, `powers of 2`. With this information however we can check the different concepts involved.

For example:

What is the likelihood for all these hypotheses?

What is the prior for these hypotheses?

How is our prior belief after seeing example $\{16\}$?

and after seeing examples $\{16, 64, 8\}$?

After applying the Bayesian rule we obtain a probability density function on the hypotheses space. If we have to report one we could proceed in different ways. We can report the **Maximum A Posteriori (MAP)**, which corresponds to the mode of the posterior and can be written as,

$$\hat{h}^{MAP} = \arg\max_h p(D|h)p(h) = \arg\max_h [\log(p(D|h)) + \log(p(h))]$$

Usually the likelihood term depends on the number of samples. Then, as we get more data the MAP converges towards the **Maximum Likelihood Estimate (MLE)**

$$\hat{h}^{MLE} = \arg\max_h p(D|h) = \arg\max_h \log(p(D|h))$$

# The Beta-Binomial model

Suppose a random variable $X_i\sim \text{Ber}(\theta)$, where $X_i = 1$ represents "heads", $X_i = 0$ represents "tails", and $\theta\in [0,1]$ is the rate parameter (probability of heads). If data are iid, the likelihood has the form

$$p(D|\theta) = \theta^{N_1}(1-\theta)^{N_2}$$

where $N_1$, and $N_2$ are the heads and tails counts. What we are working out in this exercise works also for Binomial distributions, i.e. the probability of observing $N_1$ heads in $N$ tosses

$$\text{Bin}(k|N,D) = \big(\begin{align}N\\k \end{align}\big) \theta^k(1-\theta)^{N-k}$$.

We need a prior that has support over the interval $[0,1]$. We could use the same form as the likelihood. The prior could look like,

$$p(\theta) \propto \theta^{\gamma_1}(1-\theta)^{\gamma_2}$$

When the prior and the posterior have the same form, such as in this case, we say the prior is a **conjugate prior** for the corresponding likelihood. In the case of Bernoulli or Binomial, the conjugate prior is the beta distribution, that has the following form

$$\text{Beta}(\theta|a,b) \propto \theta^{a-1}(1-\theta)^{b-1}$$

The parameters of the prior are called **hyperparameters** and we can set them to encode prior belief.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

theta = np.linspace(0,1,30)

a=2
b=2

beta = theta**(a-1)*(1-theta)**(b-1)
prior= beta/np.sum(beta)

plt.plot(theta,prior)
plt.gca().set_ylim([0,1])
```

If we multipy the likelihood by the beta prior we get the following posterior. Suppose that we observe one heads. Then the likehood is

```python
likelihood = theta

plt.plot(theta,likelihood)
```

Observe that the likelihood does not necessarily is a probability density function and does not need to sum up to one when considered over all potential parameters, i.e. $\int p(X=\text{'heads'}|\theta) d\theta$.

```python
posterior = likelihood*prior

posterior = posterior/np.sum(posterior)
plt.plot(theta,posterior)
```

Interestingly enough we can now consider this last result as a new prior and repeat the process with a new observation. Consider a new head.

```python
prior = posterior
# a new head
likelihood = theta

posterior = likelihood * prior
posterior = posterior/np.sum(posterior)

accum_likelihood = theta*theta
accum_likelihood_normalized = accum_likelihood/np.sum(accum_likelihood)

plt.plot(theta,posterior, label ="posterior")
plt.plot(theta,accum_likelihood_normalized, label="Accum likelihood")

```

This resembles the effect of *online methods*. Observe that the MAP value is different than the mean posterior and the MLE. They correspond to

$$\hat{\theta}_{MAP} = \frac{a+N_1-1}{a+b+N-2} $$

with a uniform prior, MAP is reduced to MLE,

$$\hat{\theta}_{MLE} = \frac{N_1}{N} $$

By contrast the posterior mean is

$$\bar{\theta} = \frac{a+N_1}{a+b+N}$$

This example can be used to assess the probability and the credibility interval for a classifier comparison. When the outcomes are `classifier A beats classifier B` or the other way around.

# The Diritchlet-multinomial model

In the case we have a multinomial random variable, for example, topics in a document text we have a probability simplex. The probability density function will operate over that simplex.

## Likelihood

Suppose we observe $N$ dice rolls, $D = \{x_1,\dots,x_N\}$ if we assume the data is iid, the likelihood has the form

$$p(D|\theta) = \prod_{k=1}^K \theta_k^{N_k}$$

where $N_k = \sum_{i=1}^N \mathbb{1}(y_i=k)$ is the number of times event $k$ occurred

## Prior

Since the parameter vector lives in the K-dimensional probability simplex, we need a prior that has support over this simplex. The Dirichlet distribution ahs the following prior

$$\text{Dir}(\theta|\alpha) = \frac{1}{B(\alpha)}\prod_{k=1}^K \theta_k^{\alpha_k-1}\mathbb{1}(x\in S_k)$$

## Posterior

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.tri as tri
import functools

corners = np.array([[0, 0], [1, 0], [0.5, 0.75**0.5]])
triangle = tri.Triangulation(corners[:, 0], corners[:, 1])

refiner = tri.UniformTriRefiner(triangle)
trimesh = refiner.refine_triangulation(subdiv=4)

# Mid-points of triangle sides opposite of each corner
midpoints = [(corners[(i + 1) % 3] + corners[(i + 2) % 3]) / 2.0 for i in range(3)]
def xy2bc(xy, tol=1.e-3):
    '''Converts 2D Cartesian coordinates to barycentric.'''
    s = [(corners[i] - midpoints[i]).dot(xy - midpoints[i]) / 0.75 for i in range(3)]
    return np.clip(s, tol, 1.0 - tol)

class Dirichlet(object):
    def __init__(self, alpha):
        from math import gamma
        from operator import mul
        self._alpha = np.array(alpha)
        self._coef = gamma(np.sum(self._alpha)) / functools.reduce(mul, [gamma(a) for a in self._alpha])
    def pdf(self, x):
        '''Returns pdf value for `x`.'''
        from operator import mul
        return self._coef * functools.reduce(mul, [xx ** (aa - 1) for (xx, aa)in zip(x, self._alpha)])

def draw_pdf_contours(dist, nlevels=200, subdiv=8, **kwargs):
    import math

    refiner = tri.UniformTriRefiner(triangle)
    trimesh = refiner.refine_triangulation(subdiv=subdiv)
    pvals = [dist.pdf(xy2bc(xy)) for xy in zip(trimesh.x, trimesh.y)]

    plt.tricontourf(trimesh, pvals, nlevels, **kwargs)
    plt.axis('equal')
    plt.xlim(0, 1)
    plt.ylim(0, 0.75**0.5)
    plt.axis('off')

draw_pdf_contours(Dirichlet([3,2,5]))
```

# Naive Bayes

## 2.1 Basic document representation

In text classification, we are given a description $x \in {\bf R}^d$ of a document $\delta$ and a fixed set of classes $y \in \{c_1, \dots, c_K\}$, for example the document topic. Given a new document, our goal is to predict the most probable class.

A very simple description of a document is the **bag-of-words** description. This representation transforms a complete text to a vector of $d$ predefined words. The set of predefined words is selected by the practicioner. For example, the list can consist of the set of all words in a given language.

Example 1:
Suppose we are given four different documents belonging to the topics $y=\{\text{'economics'},\text{'technology'}\}$ and we select as our representation the following bag-of-words $x = \{\text{'market'}, \text{'stock'}, \text{'price'}, \text{'application'}, \text{'mobile'}, \text{'google'}\}$. We can count the number of times a certain term appears in that document and expect that this description is discriminative enough for identifying the document topic. Check the following example:

market
stock
price
application
mobile
google

document 1($\text{'economics'}$)
1
2
3
0
0
0

document 2($\text{'economics'}$)
0
1
2
0
0
1

document 3($\text{'technology'}$)
0
0
0
2
3
1

document 4($\text{'technology'}$)
1
0
1
2
3
0

In this representation, document 2 is represented by the vector (0,1,2,0,0,1). We can alternatively use a binary value representing whether a term appears or not in the document. In this last case document would be represesnted by (0,1,1,0,0,1).

Observe that this is a context free representation, i.e. the order of the words is not considered. Consider the sentences "Google reduces the prices of applications in App market" and "The number of aplications in Google App market with cheap prices is reduced by 20%". The representation for both sentences is the same, though the exact meaning of both sentences is completely different. However, this kind of representation may be enough for identifying that both refers to $\text{'technology'}$.

## 2.2 The Naive Bayes classifier

Naive Bayes is an instance of a Bayessian classifier. In this framework, the problem of classification consists of selecting the class with Maximum A Posteriori (MAP) probability, i.e. $$\hat{y} = \arg\max_y p(y|x).$$

In order to find this quantity we use the Bayes equation,

$$ p(x,y) = p(x|y)p(y) = p(y|x)p(x),$$

and

$$ p(y|x) = \frac{p(x|y)p(y)}{p(x)}.$$

In order to compute the MAP the quantities $p(x|y)$, $p(y)$, $p(x)$ have to be estimated from observed data.

In the problem of document classification, our goal is to select the class with MAP probability. For example, we will select the cathegory $\text{'economics'}$ for a text with description (1,1,1,0,0,0) only if $P(y = \text{'economics'}|x = (1,1,1,0,0,0)) > P(y = \text{'technology'}|x = (1,1,1,0,0,0))$.

Note that $p(x)$ is a constant value and it does not affect the decision, thus we just need to compute

$$P(y|x) \propto P(y)P(x|y)$$

Estimating the likelihood term, $P(x|y)$, accounts for computing the probability of certain description vector in a given class, e.g. the probability of a text in $\text{'economics'}$ having a description $x = (1,1,1,0,0,0)$ (the value of the probability that a description x = (1,1,1,0,0,0) has inside the category $\text{'economics'}$), $p(x = (1,1,1,0,0,0)|y = \text{'economics'})$

Up to this point, the description of the classifier is general for any Bayessian classifier. *Naive Bayes additionally assumes that $x$ is composed of a set of $d$ independent variables.* This allows to rewrite the likelihood term as
$$p(x_1,x_2,...,x_N | y) = p(x_1|y)p(x_2|y)...p(x_N|y) = \prod\limits_{i=1}^N p(x_i|y)$$

For example, in our case
$$P(x = (1,1,1,0,0,0)|y = \text{'tech'}) = P(x_1=1|y = \text{'tech'})P(x_2=1|y = \text{'tech'})P(x_3=1|y = \text{'tech'})P(x_4=0|y = \text{'tech'})P(x_5=0|y = \text{'tech'})P(x_6=0|y = \text{'tech'})$$

This is understood as the fact that the probability of a document described as x = (1,1,1,0,0,0) is described by the product of the probilities that the first to the third word are present, and the fourth to the sixth word are not.

In the end, the Naive Bayes classifier has the following form,
$$p(y|x) \propto p(y)\prod\limits_{i=1}^N p(x_i|y)$$

In many cases the prior $p(y)$ is unknown or simply we prefer to use a non-informative prior (all documents have the same probability of appearance in our context ($p(y)$)). In that case the formulation is simplified to the Maximum Likelihood Estimate.

## 2.3 Estimating conditioned probabilities

The last remaining step is the estimation of the individual conditional probabilities. There are two classical variants the **Multinomial Naive Bayes** and the **Bernoulli Naive Bayes**. The difference between both lies in the goal of what they are modeling. **In Multinomial NB we compute the probability of generating the observed document.** In this sense, we multiply the conditional probability of each word in the document for all words present in the document. An alternative view is the *Bernoulli model*. **In the Bernoulli Naive Bayes we compute the probability of the binary bag-of-words descriptor.** Observe that in the Bernouilli Naive Bayes the final probability depends on the words that appear in the document but also on the words that do not appear while in the multinomial NB it only depends on the words that appear. On the contrary, multinomial naive bayes takes into account the multiplicity of the words in the document while Bernoulli does not. Let us consider in this example the *Bernoulli model* that is consistent with our representation where a zero indicates a word is not present in the document and a one represents it is present. In order to estimate this probability we can use a frequentist approximation to probability, i.e. we will estimate the probability as the frequency of appearance of each term in each category. This computation divides the number documents where the word appears over the total number of documents.

In our previous example, $p(x_3=1 (\text{the word 'price' appears})|y =\text{'tech'}) = 1/2$ and $p(x_3=1 (\text{the word 'price' appears})|y =\text{'eco'}) = 2/2$. This is computed by dividing the number of documents where the word price appear in a given category over the number of documents of that category.

### 2.3.1 The zero probability effect
In the former example the probability $p(x_5=1|y=\text{'eco'}) = 0$. This implies that if the word 'mobile' appears the document can not belong to the class $\text{'economy'}$. It is unreasonable to completely penalize a whole class by the appearance or not appearance of a single word. It is customary to assign to those cases a very low probability value instead. One well known approach to correct this effect is the so called **Laplace correction**. It is computed as follows,

$$p(x_i=1 | y=c_k ) = \frac{\text{# of documents of class } c_k \text{ where word } x_i \text{ appears} + 1}{\text{# of documents of class } c_k + M}$$

where $M$ is the amount of words in the description.

### 2.3.2 Underflow effect

As the number of words in the description increase there is a higher probability that many of those words will not be present in the document. The product of many very small values may lead to floating point underflow effects. For this reason it is usual to use the log probability instead. This transformation does not change the decision boundary. In our simplified case

$$\log p(x|y) = \sum\limits_{i=1}^N \log p(x_i|y)$$

Let us code an even more simpler version of Naive Bayes, a Bernoulli generative version. This is just to even simplify more our code, so don't expect to work well.

```python
import numpy as np

class SimpleNaiveBayes():
    def __init__(self):
        self.prob_vec={}
        self.classes =[]

    def fit(self,X,y):
        # extract the probabilities for each element for different classes
        self.classes = np.unique(y)
        for c in self.classes:
            idx=np.where(y==c)[0]
            N = len(idx)
            self.prob_vec[c] = np.log(np.sum(X[idx,:],axis=0)/N+1e-10)

    def predict_proba(self,X):
        #Output matrix N_documents X N_classes
        probs = np.zeros((X.shape[0],len(self.classes)))
        i=0
        for k in self.classes:

            probs[:,i]=np.dot(X,self.prob_vec[k].T).ravel()
            i+=1
        return probs

    def predict(self,X):
        return self.classes[np.argmax(self.predict_proba(X),axis =1)]
```

Let us try

```python
X=np.array([[1,1,1,0,0,0],[0,1,1,0,0,1],[0,0,0,1,1,1],[1,0,1,1,1,0]],dtype=np.float)
```

```python
y = [0,0,1,1]
```

```python
clf = SimpleNaiveBayes()
```

```python
clf.fit(X,y)
```

```python
clf.classes
```

    array([0, 1])

```python
clf.prob_vec
```

    {0: array([ -6.93147180e-01,   1.00000008e-10,   1.00000008e-10,
             -2.30258509e+01,  -2.30258509e+01,  -6.93147180e-01]),
     1: array([ -6.93147180e-01,  -2.30258509e+01,  -6.93147180e-01,
              1.00000008e-10,   1.00000008e-10,  -6.93147180e-01])}

```python
clf.predict_proba(X)
```

    array([[ -0.69314718, -24.41214529],
           [ -0.69314718, -24.41214529],
           [-46.74484904,  -0.69314718],
           [-46.74484904,  -1.38629436]])

```python
clf.predict(X)
```

    array([0, 0, 1, 1])

```python
#load data
import pandas as pd
data=pd.read_csv('./files/Boydstun_NYT_FrontPage_Dataset_1996-2006_0.csv')
data.head()

```

      Article_ID
      Date
      Article_Sequence
      Title
      Summary
      Topic_6digit
      Topic_4digit
      Topic_2digit
      War on Terror
      Katrina
      Israel/Palestine
      Immigration
      Presidential Elections
      Clinton Impeachment
      Enron
      Darfur
      Race/Ethnicity
      Schiavo

      0
      1
      1/1/1996
      a
      Nation's Smaller Jails Struggle To Cope With S...
      Jails overwhelmed with hardened criminals
      120500
      1205
      12
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1
      2
      1/1/1996
      b
      Dancing (and Kissing) In the New Year
      new years activities
      280000
      2800
      28
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2
      3
      1/1/1996
      c
      Forbes's Silver Bullet for the Nation's Malaise
      Steve Forbes running for President
      201201
      2012
      20
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0

      3
      4
      1/1/1996
      d
      Up at Last, Bridge to Bosnia Is Swaying Gatewa...
      U.S. military constructs bridge to help their ...
      160200
      1602
      16
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      4
      5
      1/1/1996
      e
      2 SIDES IN SENATE DISAGREE ON PLAN TO END FURL...
      Democrats and Republicans can't agree on plan ...
      201206
      2012
      20
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

Let us split the data set in two set:

+ We will train the classifier with news up to 2004.
+ We will test the classifier in news from 2005 and 2006.

```python
import numpy as np
#Let us train the classifier with data up to 1/1/2004 and test its performnace in data from 2004-2006
split = pd.to_datetime(pd.Series(data['Date']))<pd.datetime(2004, 1, 1)
raw_data = data['Title']
raw_train = raw_data[split]
raw_test = raw_data[np.logical_not(split)]
y = data['Topic_2digit']
y_train = y[split]
y_test = y[np.logical_not(split)]
print ('Check the split sizes, train, test and total amount of data:')
print (raw_train.shape, raw_test.shape, raw_data.shape)
print ('Display the labels:')
print (np.unique(y))

```

    Check the split sizes, train, test and total amount of data:
    (23584,) (7450,) (31034,)
    Display the labels:
    [ 1  2  3  4  5  6  7  8 10 12 13 14 15 16 17 18 19 20 21 24 26 27 28 29 30
     31 99]

```python
# Let us tokenize the data
from sklearn.feature_extraction.text import CountVectorizer

# We use the count number of instances considering that a word has a minimum support of two documents
vectorizer = CountVectorizer(min_df=2,
# stop words such as 'and', 'the', 'of' are removed
 stop_words='english',
 strip_accents='unicode')

#example of the tokenization
test_string = raw_train[0]
print ("Example: " + test_string +"\n")
print ("Preprocessed: " + vectorizer.build_preprocessor()(test_string)+"\n")
print ("Tokenized:" + str(vectorizer.build_tokenizer()(test_string))+"\n")
print ("Analyzed data string:" + str(vectorizer.build_analyzer()(test_string))+"\n")

#Process and convert data
X_train = vectorizer.fit_transform(raw_train)
X_test = vectorizer.transform(raw_test)

print ("Number of tokens: " + str(len(vectorizer.get_feature_names())) +"\n")
print ("Extract of tokens:")
print (vectorizer.get_feature_names()[1000:1100])

```

    Example: Nation's Smaller Jails Struggle To Cope With Surge in Inmates

    Preprocessed: nation's smaller jails struggle to cope with surge in inmates

    Tokenized:['Nation', 'Smaller', 'Jails', 'Struggle', 'To', 'Cope', 'With', 'Surge', 'in', 'Inmates']

    Analyzed data string:['nation', 'smaller', 'jails', 'struggle', 'cope', 'surge', 'inmates']

    Number of tokens: 8950

    Extract of tokens:
    ['boeing', 'boiling', 'boils', 'bold', 'bolster', 'bolsters', 'bolt', 'bolts', 'bomb', 'bombay', 'bombed', 'bomber', 'bombers', 'bombing', 'bombings', 'bombs', 'bonanza', 'bond', 'bondage', 'bonds', 'bone', 'bones', 'bonn', 'bono', 'bonus', 'bonuses', 'book', 'books', 'booksellers', 'bookstore', 'boom', 'boomers', 'booming', 'booms', 'boost', 'boot', 'bora', 'border', 'borders', 'born', 'borough', 'boroughs', 'borrow', 'borrowing', 'bosnia', 'bosnian', 'bosnians', 'boss', 'bosses', 'boston', 'botched', 'bottle', 'bought', 'bounce', 'bound', 'bounty', 'bout', 'bow', 'bowing', 'bowl', 'bows', 'box', 'boxes', 'boxing', 'boy', 'boycott', 'boys', 'brace', 'braced', 'braces', 'bracket', 'bradley', 'brain', 'brains', 'branches', 'brand', 'brash', 'brave', 'bravery', 'braves', 'brawl', 'brawley', 'brazil', 'brazilian', 'breach', 'breaches', 'bread', 'break', 'breakdown', 'breaking', 'breaks', 'breakthrough', 'breakup', 'breast', 'breather', 'breed', 'brewers', 'brewing', 'bribery', 'bricks']

```python
%matplotlib inline

X_train = X_train.todense()
X_train = X_train.astype(np.float)
X_test = X_test.todense()
X_test = X_test.astype(np.float)
y_train = np.array(y_train.tolist())

clf = SimpleNaiveBayes()
clf.fit(X_train,y_train)
y_hat = clf.predict(X_test)

#from sklearn.naive_bayes import BernoulliNB
#nb = BernoulliNB()
#nb.fit(X_train,y_train)
#y_hat = nb.predict(X_test)

from sklearn import metrics
import matplotlib.pyplot as plt
def plot_confusion_matrix(y_pred, y):
    plt.imshow(metrics.confusion_matrix(y, y_pred), interpolation='nearest',cmap='gray')
    plt.colorbar()
    plt.ylabel('true value')
    plt.xlabel('predicted value')
    fig = plt.gcf()
    fig.set_size_inches(9,9)

print ("classification accuracy:", metrics.accuracy_score(y_hat, y_test))
plot_confusion_matrix(y_hat, y_test)
print ("Classification Report:")
print (metrics.classification_report(y_hat,np.array(y_test)))
```

```python
%matplotlib inline
#Fit a Bernoulli Naive Bayes

from sklearn.naive_bayes import BernoulliNB
nb = BernoulliNB()
nb.fit(X_train,y_train)
y_hat = nb.predict(X_test)

from sklearn import metrics
import matplotlib.pyplot as plt
def plot_confusion_matrix(y_pred, y):
    plt.imshow(metrics.confusion_matrix(y, y_pred), interpolation='nearest',cmap='gray')
    plt.colorbar()
    plt.ylabel('true value')
    plt.xlabel('predicted value')
    fig = plt.gcf()
    fig.set_size_inches(9,9)

print ("classification accuracy:", metrics.accuracy_score(y_hat, y_test))
plot_confusion_matrix(y_hat, y_test)
print ("Classification Report:")
print (metrics.classification_report(y_hat,np.array(y_test)))
```

**QUESTION:** Identify the three most simple classes.

```python
#What are the top N most predictive features per class?
N = 5
voc = vectorizer.get_feature_names()
for i, label in enumerate(np.unique(y)):
    topN = np.argsort(nb.coef_[i])[-N:]
    print ('Code: '+ str(label) + ' Terms : '+ str([voc[i] for i in topN]))
```

    Code: 1 Terms : ['cut', 'bush', 'economy', 'budget', 'tax']
    Code: 2 Terms : ['race', 'gay', 'new', 'court', 'abortion']
    Code: 3 Terms : ['care', 'medicare', 'drug', 'health', 'new']
    Code: 4 Terms : ['farm', 'safety', 'new', 'farmers', 'food']
    Code: 5 Terms : ['workers', 'strike', 'union', 'immigrants', 'new']
    Code: 6 Terms : ['students', 'city', 'new', 'school', 'schools']
    Code: 7 Terms : ['rules', 'warming', 'air', 'new', 'pollution']
    Code: 8 Terms : ['blackout', 'california', 'power', 'energy', 'oil']
    Code: 10 Terms : ['new', 'security', '800', 'flight', 'crash']
    Code: 12 Terms : ['drug', 'case', 'death', 'new', 'police']
    Code: 13 Terms : ['plan', 'security', 'new', 'social', 'welfare']
    Code: 14 Terms : ['city', 'homeless', 'york', 'rent', 'new']
    Code: 15 Terms : ['new', 'billion', 'deal', 'enron', 'microsoft']
    Code: 16 Terms : ['bush', 'challenged', 'war', 'iraq', 'nation']
    Code: 17 Terms : ['space', 'nasa', 'loss', 'new', 'shuttle']
    Code: 18 Terms : ['business', 'bush', 'clinton', 'china', 'trade']
    Code: 19 Terms : ['mideast', 'war', 'israel', 'new', 'china']
    Code: 20 Terms : ['2000', 'clinton', 'bush', 'president', 'campaign']
    Code: 21 Terms : ['park', 'plan', 'zero', 'ground', 'new']
    Code: 24 Terms : ['mayor', 'giuliani', 'city', 'budget', 'new']
    Code: 26 Terms : ['blizzard', 'new', 'overview', 'hurricane', 'storm']
    Code: 27 Terms : ['blaze', 'ferry', 'killed', 'crash', 'fires']
    Code: 28 Terms : ['arts', 'tv', 'broadway', 'art', 'new']
    Code: 29 Terms : ['world', 'playoffs', 'series', 'yankees', 'baseball']
    Code: 30 Terms : ['87', '79', 'crash', 'dead', 'dies']
    Code: 31 Terms : ['faith', 'bishop', 'new', 'church', 'pope']
    Code: 99 Terms : ['editors', 'today', 'readers', 'note', 'special']

Let us check what would happen if we enrich the data set with the summary of the article.

```python
raw_data = data['Title']+data['Summary']
raw_train = raw_data[split]
raw_test = raw_data[np.logical_not(split)]
y = data['Topic_2digit']
y_train = y[split]
y_test = y[np.logical_not(split)]
```

```python
# Let us tokenize the data
from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(min_df=2,
 stop_words='english',
 strip_accents='unicode')

#example
test_string = raw_train[0]
print ("Example: " + test_string +"\n")
print ("Preprocessed: " + vectorizer.build_preprocessor()(test_string)+"\n")
print ("Tokenized:" + str(vectorizer.build_tokenizer()(test_string))+"\n")
print ("Analyzed data string:" + str(vectorizer.build_analyzer()(test_string))+"\n")

#Fit and convert data
X_train = vectorizer.fit_transform(raw_train)
X_test = vectorizer.transform(raw_test)

print ("\n")
print ("Number of tokens: " + str(len(vectorizer.get_feature_names())) +"\n")
print ("Extract of tokes:")
print (vectorizer.get_feature_names()[1000:1100])
```

    Example: Nation's Smaller Jails Struggle To Cope With Surge in Inmates Jails overwhelmed with hardened criminals

    Preprocessed: nation's smaller jails struggle to cope with surge in inmates jails overwhelmed with hardened criminals

    Tokenized:['Nation', 'Smaller', 'Jails', 'Struggle', 'To', 'Cope', 'With', 'Surge', 'in', 'Inmates', 'Jails', 'overwhelmed', 'with', 'hardened', 'criminals']

    Analyzed data string:['nation', 'smaller', 'jails', 'struggle', 'cope', 'surge', 'inmates', 'jails', 'overwhelmed', 'hardened', 'criminals']

    Number of tokens: 11354

    Extract of tokes:
    ['banned', 'banner', 'banning', 'bans', 'bansenate', 'banus', 'baptist', 'baptists', 'bar', 'barak', 'barbie', 'bare', 'barely', 'bares', 'bargain', 'bargaining', 'bargains', 'barnes', 'barney', 'baron', 'barons', 'barrage', 'barred', 'barrel', 'barren', 'barrier', 'barriers', 'barring', 'bars', 'barter', 'base', 'baseball', 'based', 'basement', 'bases', 'basespentagon', 'bashing', 'basic', 'basis', 'baskeball', 'basketball', 'basks', 'basra', 'bastion', 'bat', 'bath', 'batter', 'battered', 'battering', 'batters', 'battery', 'battle', 'battledemocrats', 'battlefield', 'battleground', 'battles', 'battleus', 'battling', 'bay', 'bayer', 'bbc', 'beach', 'beaches', 'bear', 'bearing', 'bears', 'beat', 'beaten', 'beating', 'beats', 'beautiful', 'beauty', 'beckons', 'bed', 'bedevil', 'bedeviled', 'beef', 'beetle', 'began', 'begin', 'beginning', 'begins', 'beginsphoto', 'begun', 'behavior', 'behemoth', 'beijing', 'beijingus', 'beirut', 'belarus', 'belated', 'belfast', 'belgrade', 'beliefs', 'belies', 'believe', 'believed', 'believes', 'belittles', 'bell']

```python
from sklearn.naive_bayes import BernoulliNB
nb = BernoulliNB()
nb.fit(X_train,y_train)

y_hat = nb.predict(X_test)
from sklearn import metrics
import matplotlib.pyplot as plt
def plot_confusion_matrix(y_pred, y):
    plt.imshow(metrics.confusion_matrix(y, y_pred), interpolation='nearest')
    plt.colorbar()
    plt.ylabel('true value')
    plt.xlabel('predicted value')
    fig = plt.gcf()
    fig.set_size_inches(9,9)

print ("classification accuracy:", metrics.accuracy_score(y_hat, y_test))
plot_confusion_matrix(y_hat, y_test)
print ("Classification Report:")
print (metrics.classification_report(y_hat,np.array(y_test)))
```

```python
#What are the top N most predictive features per class?
N = 5
voc = vectorizer.get_feature_names()
for i, label in enumerate(np.unique(y)):
    topN = np.argsort(nb.coef_[i])[-N:]
    print ('Code: '+ str(label) + ' Terms : '+ str([voc[i] for i in topN]))
```

    Code: 1 Terms : ['cut', 'economy', 'market', 'budget', 'tax']
    Code: 2 Terms : ['gay', 'race', 'new', 'court', 'abortion']
    Code: 3 Terms : ['medicare', 'care', 'drug', 'new', 'health']
    Code: 4 Terms : ['disease', 'farm', 'new', 'farmers', 'food']
    Code: 5 Terms : ['new', 'workers', 'strike', 'union', 'immigrants']
    Code: 6 Terms : ['education', 'students', 'new', 'schools', 'school']
    Code: 7 Terms : ['water', 'pollution', 'new', 'global', 'warming']
    Code: 8 Terms : ['gas', 'prices', 'energy', 'oil', 'power']
    Code: 10 Terms : ['investigation', '800', 'twa', 'flight', 'crash']
    Code: 12 Terms : ['death', 'scandal', 'abuse', 'new', 'police']
    Code: 13 Terms : ['security', 'clinton', 'social', 'new', 'welfare']
    Code: 14 Terms : ['housing', 'york', 'rent', 'nyc', 'new']
    Code: 15 Terms : ['new', 'merger', 'scandal', 'antitrust', 'microsoft']
    Code: 16 Terms : ['bush', 'challenged', 'war', 'nation', 'iraq']
    Code: 17 Terms : ['loss', 'columbia', 'space', 'shuttle', 'new']
    Code: 18 Terms : ['deal', 'sanctions', 'clinton', 'china', 'trade']
    Code: 19 Terms : ['war', 'peace', 'new', 'china', 'israel']
    Code: 20 Terms : ['2000', 'bush', 'president', 'clinton', 'campaign']
    Code: 21 Terms : ['memorial', 'zero', 'ground', 'indian', 'new']
    Code: 24 Terms : ['city', 'budget', 'governor', 'mayor', 'new']
    Code: 26 Terms : ['hurricane', 'york', 'storm', 'new', 'weather']
    Code: 27 Terms : ['fires', 'killed', 'accident', 'crash', 'new']
    Code: 28 Terms : ['york', 'day', 'museum', 'art', 'new']
    Code: 29 Terms : ['world', 'playoffs', 'series', 'yankees', 'baseball']
    Code: 30 Terms : ['plane', 'death', 'crash', 'dead', 'dies']
    Code: 31 Terms : ['catholic', 'religious', 'new', 'pope', 'church']
    Code: 99 Terms : ['park', 'editors', 'new', 'special', 'note']

And now try with our own Simple Naive Bayes

```python
%matplotlib inline

X_train = X_train.todense()
X_train = X_train.astype(np.float)
X_test = X_test.todense()
X_test = X_test.astype(np.float)
y_train = np.array(y_train.tolist())

clf = SimpleNaiveBayes()
clf.fit(X_train,y_train)
y_hat = clf.predict(X_test)

#from sklearn.naive_bayes import BernoulliNB
#nb = BernoulliNB()
#nb.fit(X_train,y_train)
#y_hat = nb.predict(X_test)

from sklearn import metrics
import matplotlib.pyplot as plt
def plot_confusion_matrix(y_pred, y):
    plt.imshow(metrics.confusion_matrix(y, y_pred), interpolation='nearest',cmap='gray')
    plt.colorbar()
    plt.ylabel('true value')
    plt.xlabel('predicted value')
    fig = plt.gcf()
    fig.set_size_inches(9,9)

print ("classification accuracy:", metrics.accuracy_score(y_hat, y_test))
plot_confusion_matrix(y_hat, y_test)
print ("Classification Report:")
print (metrics.classification_report(y_hat,np.array(y_test)))
```

Observe that adding the small summary improves the recognition rate by $10\%$.

As a side note, Naive Bayes with these models creates a linear decision boundary. For this reason, sometimes NB is called a linear classifier.

**DELIVERABLE: ** Modify the SimpleBayes class to consider the Multinomial case. What is the performance achieved in the NY Times dataset?

```python

```


# Homogenize for representative N4 in slices of parameter space

```python
import planarfibers
import matplotlib.pyplot as plt
import pandas as pd
import vofotensors
from vofotensors.abc import la1, d1, d8
import sympy as sp
import numpy as np
```

```python
pd.set_option("display.max_columns", 100)
pd.set_option("display.width", 1000)
```

```python
# User input
SCALE_HOMOGENEOUS = False
key_quantity = "E_modulus"
homogenization_function = planarfibers.approximation.calc_MTOA
la1_values = ["3 / 6", "4 / 6", "5 / 6"]
```

```python
# Get points on views
df = planarfibers.discretization.get_points_on_slices(
    radii=["0", "1/2", "9/10"],
    la1s=list(map(eval, la1_values)) + ["1"],
    numeric=True,
)
```

```python
# Get fiber orientation tensors
parameterizations = vofotensors.fabric_tensors.N4s_parametric
parameterization = parameterizations["planar"]["la1_d1_d8"]
N4_func = sp.lambdify([la1, d1, d8], parameterization)
df["N4"] = df.apply(
    lambda row: N4_func(la1=row["la1"], d1=row["d_1"], d8=row["d_8"]), axis=1
)
```

```python
# Define angle discretization
angles = np.linspace(0, 2 * np.pi, 120)
```

```python
# Homogenize
df["stiffness"] = df.apply(
    lambda row: homogenization_function(N4=row["N4"], inp=None),
    axis=1,
)
```

```python
# Define helper func to explicitly select either Youngs or generalized compression mod.
def get_Youngs_modulus(stiffness, angles):
    projector = planarfibers.utils.PlanarStiffnesProjector()
    E, K = projector.get_planar_E_K(stiffness=stiffness, angles=angles)
    if key_quantity == "E_modulus":
        return E
    elif key_quantity == "K_modulus":
        return K
```

```python
# Get Youngs-Modulus for direction in plane
df[key_quantity] = df.apply(
    lambda row: get_Youngs_modulus(stiffness=row["stiffness"], angles=angles),
    axis=1,
)
```

```python
# Define layout
la1_key_extensions = {f"-la1-{index}": value for index, value in enumerate(la1_values)}
# la1_key_extensions = {
# "-la1-0": "3 / 6",
# "-la1-1": "4 / 6",
# "-la1-2": "5 / 6",
# }
grid_indices = {
    "vshc-central": (2, 0),
    "vshc-m90-0": (3, 0),
    "vshc-m90-1": (4, 0),
    "vshc-m45-0": (3, 1),
    "vshc-m45-1": (4, 2),
    "vshc-0-0": (2, 1),
    "vshc-0-1": (2, 2),
    "vshc-45-0": (1, 1),
    "vshc-45-1": (0, 2),
    "vshc-90-0": (1, 0),
    "vshc-90-1": (0, 0),
}
legend_axis_indices = (4, 1)
empty_axes_indices = [(0, 1), (1, 2), (3, 2), legend_axis_indices]
```

```python
# Plot first view
nbr_slices = len(la1_key_extensions)
fig = plt.figure(figsize=(6 * nbr_slices, 10))
subfigs = fig.subfigures(1, nbr_slices, wspace=0.0)
for index, (key_extension, la1val) in enumerate(la1_key_extensions.items()):
    subfig = subfigs[index]
    subfig.suptitle(f"la1 = {la1val}")
    axs = subfig.subplots(ncols=3, nrows=5, subplot_kw={"projection": "polar"})
    lines = []  # Initialize legend lines
    for key_N4_start, grid_index in grid_indices.items():
        ax = axs[grid_index]
        key = key_N4_start + key_extension

        ax.plot(angles, df.loc[key][key_quantity], label=key_quantity, color="red")

        # Update legend if something has been plotted
        lines_tmp, labels_tmp = ax.get_legend_handles_labels()
        if len(lines_tmp) > len(lines):
            lines, labels = lines_tmp, labels_tmp

    for ax in axs.flatten():
        ax.set_xticklabels([])
        if SCALE_HOMOGENEOUS:
            ax.set_ylim(
                bottom=0,
                top=1.2,
            )
            ax.set_yticks([0, 0.5, 1])

    legend_axis = axs[legend_axis_indices]
    legend_axis.legend(lines, labels, loc="center")

    for indice in empty_axes_indices:
        ax = axs[indice]
        ax.axis("off")
fig.tight_layout()
```

```python
# Define layout second view
grid_indices = {
    "v00-upper-0": (0, 0),
    "v00-upper-1": (0, 1),
    "v00-upper-2": (0, 2),
    #
    "v00-mid-0": (1, 0),
    "v00-mid-1": (1, 1),
    "v00-mid-2": (1, 2),
    #
    "v00-lower-0": (2, 0),
    "v00-lower-1": (2, 1),
    "v00-lower-2": (2, 2),
    #
    "v00-mid-3": (1, 3),
    #
}
legend_axis_indices = (0, 3)
empty_axes_indices = [(2, 3), legend_axis_indices]
```

```python
# Plot second dview
fig, axs = plt.subplots(
    figsize=(12, 9), ncols=4, nrows=3, subplot_kw={"projection": "polar"}
)
lines = []  # Initialize legend lines
for key_N4_start, grid_index in grid_indices.items():
    ax = axs[grid_index]
    key = key_N4_start

    ax.plot(angles, df.loc[key][key_quantity], label=key_quantity, color="red")

    # Update legend if something has been plotted
    lines_tmp, labels_tmp = ax.get_legend_handles_labels()
    if len(lines_tmp) > len(lines):
        lines, labels = lines_tmp, labels_tmp
for ax in axs.flatten():
    ax.set_xticklabels([])
    if SCALE_HOMOGENEOUS:
        ax.set_ylim(
            bottom=0,
            top=1.2,
        )
        ax.set_yticks([0, 0.5, 1])
legend_axis = axs[legend_axis_indices]
legend_axis.legend(lines, labels, loc="center")
for indice in empty_axes_indices:
    ax = axs[indice]
    ax.axis("off")
fig.tight_layout()
```

plt.show()