# 03 - Full-Waveform Inversion (FWI)

This notebook is the third in a series of tutorial highlighting various aspects of seismic inversion based on Devito operators. In this second example we aim to highlight the core ideas behind seismic inversion, where we create an image of the subsurface from field recorded data. This tutorial follows on the modelling tutorial and will reuse the modelling and velocity model.

## Inversion requirement

Seismic inversion relies on two known parameters:

- **Field data** - or also called **recorded data**. This is a shot record corresponding to the true velocity model. In practice this data is acquired as described in the first tutorial. In order to simplify this tutorial we will fake field data by modelling it with the true velocity model.

- **Initial velocity model**. This is a velocity model that has been obtained by processing the field data. This model is a rough and very smooth estimate of the velocity as an initial estimate for the inversion. This is a necessary requirement for any optimization (method).

## Inversion computational setup

In this tutorial, we will introduce the gradient operator. This operator corresponds to the imaging condition introduced in the previous tutorial with some minor modifications that are defined by the objective function (also referred to in the tutorial series as the *functional*, *f*) and its gradient, *g*. We will define these two terms in the tutorial too.

## Notes on the operators

As we have already described the creation of a forward modelling operator, we will only call a wrapper function here. This wrapper already contains all the necessary operators for seismic modeling, imaging and inversion. Operators introduced for the first time in this tutorial will be properly described.

```python
import numpy as np
%matplotlib inline

from devito import configuration
configuration['log-level'] = 'WARNING'
```

## Computational considerations

As we will see, FWI is computationally extremely demanding, even more than RTM. To keep this tutorial as lightwight as possible we therefore again use a very small demonstration model. We also define here a few parameters for the final example runs that can be changed to modify the overall runtime of the tutorial.

```python
nshots = 9  # Number of shots to create gradient from
nreceivers = 101  # Number of receiver locations per shot
fwi_iterations = 5  # Number of outer FWI iterations
```

# True and smooth velocity models

We will use a very simple model domain, consisting of a circle within a 2D domain. We will again use the "true" model to generate our synthetic shot data and use a "smooth" model as our initial guess. In this case the smooth model is very smooth indeed - it is simply a constant background velocity without any features.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import demo_model, plot_velocity, plot_perturbation

# Define true and initial model
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # Need origin to define relative source and receiver locations

model = demo_model('circle-isotropic', vp=3.0, vp_background=2.5,
                    origin=origin, shape=shape, spacing=spacing, nbl=40)

model0 = demo_model('circle-isotropic', vp=2.5, vp_background=2.5,
                     origin=origin, shape=shape, spacing=spacing, nbl=40,
                    grid = model.grid)

plot_velocity(model)
plot_velocity(model0)
plot_perturbation(model0, model)
```

## Acquisition geometry

In this tutorial, we will use the easiest case for inversion, namely a transmission experiment. The sources are located on one side of the model and the receivers on the other side. This allows to record most of the information necessary for inversion, as reflections usually lead to poor inversion results.

```python
#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: source
from examples.seismic import AcquisitionGeometry

t0 = 0.
tn = 1000.
f0 = 0.010
# First, position source centrally in all dimensions, then set depth
src_coordinates = np.empty((1, 2))
src_coordinates[0, :] = np.array(model.domain_size) * .5
src_coordinates[0, 0] = 20.  # Depth is 20m

# Define acquisition geometry: receivers

# Initialize receivers for synthetic and imaging data
rec_coordinates = np.empty((nreceivers, 2))
rec_coordinates[:, 1] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec_coordinates[:, 0] = 980.

# Geometry

geometry = AcquisitionGeometry(model, rec_coordinates, src_coordinates, t0, tn, f0=f0, src_type='Ricker')
# We can plot the time signature to see the wavelet
geometry.src.show()
```

```python
# Plot acquisition geometry
plot_velocity(model, source=geometry.src_positions,
              receiver=geometry.rec_positions[::4, :])
```

## True and smooth data

We can generate shot records for the true and smoothed initial velocity models, since the difference between them will again form the basis of our imaging procedure.

```python
# Compute synthetic data with forward operator
from examples.seismic.acoustic import AcousticWaveSolver

solver = AcousticWaveSolver(model, geometry, space_order=4)
true_d, _, _ = solver.forward(vp=model.vp)
```

```python
# Compute initial data with forward operator
smooth_d, _, _ = solver.forward(vp=model0.vp)
```

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

# Plot shot record for true and smooth velocity model and the difference
plot_shotrecord(true_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data - true_d.data, model, t0, tn)
```

# Full-Waveform Inversion

## Formulation

Full-waveform inversion (FWI) aims to invert an accurate model of the discrete wave velocity, $\mathbf{c}$, or equivalently the square slowness of the wave, $\mathbf{m} = \frac{1}{\mathbf{c}^2}$, from a given set of measurements of the pressure wavefield $\mathbf{u}$. This can be expressed as the following optimization problem [1, 2]:

\begin{aligned}
	\mathop{\hbox{minimize}}_{\mathbf{m}} \Phi_s(\mathbf{m})&=\frac{1}{2}\left\lVert\mathbf{P}_r
	\mathbf{u} - \mathbf{d}\right\rVert_2^2 \\
	\mathbf{u} &= \mathbf{A}(\mathbf{m})^{-1} \mathbf{P}_s^T \mathbf{q}_s,
\end{aligned}

where $\mathbf{P}_r$ is the sampling operator at the receiver locations, $\mathbf{P}_s^T$ is the injection operator at the source locations, $\mathbf{A}(\mathbf{m})$ is the operator representing the discretized wave equation matrix, $\mathbf{u}$ is the discrete synthetic pressure wavefield, $\mathbf{q}_s$ is the corresponding pressure source and $\mathbf{d}$ is the measured data. It is worth noting that $\mathbf{m}$ is the unknown in this formulation and that multiple implementations of the wave equation operator $\mathbf{A}(\mathbf{m})$ are possible.

We have already defined a concrete solver scheme for $\mathbf{A}(\mathbf{m})$ in the first tutorial, including appropriate implementations of the sampling operator $\mathbf{P}_r$ and source term $\mathbf{q}_s$.

To solve this optimization problem using a gradient-based method, we use the
adjoint-state method to evaluate the gradient $\nabla\Phi_s(\mathbf{m})$:

\begin{align}
 \nabla\Phi_s(\mathbf{m})=\sum_{\mathbf{t} =1}^{n_t}\mathbf{u}[\mathbf{t}] \mathbf{v}_{tt}[\mathbf{t}] =\mathbf{J}^T\delta\mathbf{d}_s,
\end{align}

where $n_t$ is the number of computational time steps, $\delta\mathbf{d}_s = \left(\mathbf{P}_r \mathbf{u} - \mathbf{d} \right)$ is the data residual (difference between the measured data and the modelled data), $\mathbf{J}$ is the Jacobian operator and $\mathbf{v}_{tt}$ is the second-order time derivative of the adjoint wavefield solving:

\begin{align}
 \mathbf{A}^T(\mathbf{m}) \mathbf{v} = \mathbf{P}_r^T \delta\mathbf{d}.
\end{align}

We see that the gradient of the FWI function is the previously defined imaging condition with an extra second-order time derivative. We will therefore reuse the operators defined previously inside a Devito wrapper.

## FWI gradient operator

To compute a single gradient $\nabla\Phi_s(\mathbf{m})$ in our optimization workflow we again use `solver.forward` to compute the entire forward wavefield $\mathbf{u}$ and a similar pre-defined gradient operator to compute the adjoint wavefield `v`. The gradient operator provided by our `solver` utility also computes the correlation between the wavefields, allowing us to encode a similar procedure to the previous imaging tutorial as our gradient calculation:

- Simulate the forward wavefield with the background velocity model to get the synthetic data and save the full wavefield $\mathbf{u}$
- Compute the data residual
- Back-propagate the data residual and compute on the fly the gradient contribution at each time step.

This procedure is applied to multiple source positions and summed to obtain a gradient image of the subsurface. We again prepare the source locations for each shot and visualize them, before defining a single gradient computation over a number of shots as a single function.

```python
#NBVAL_IGNORE_OUTPUT

# Prepare the varying source locations sources
source_locations = np.empty((nshots, 2), dtype=np.float32)
source_locations[:, 0] = 30.
source_locations[:, 1] = np.linspace(0., 1000, num=nshots)

plot_velocity(model, source=source_locations)
```

```python
# Create FWI gradient kernel
from devito import Function, TimeFunction
from examples.seismic import Receiver

import scipy
def fwi_gradient(vp_in):
    # Create symbols to hold the gradient and residual
    grad = Function(name="grad", grid=model.grid)
    residual = Receiver(name='rec', grid=model.grid,
                        time_range=geometry.time_axis,
                        coordinates=geometry.rec_positions)
    objective = 0.
    for i in range(nshots):
        # Update source location
        geometry.src_positions[0, :] = source_locations[i, :]

        # Generate synthetic data from true model
        true_d, _, _ = solver.forward(vp=model.vp)

        # Compute smooth data and full forward wavefield u0
        smooth_d, u0, _ = solver.forward(vp=vp_in, save=True)

        # Compute gradient from data residual and update objective function
        residual.data[:] = smooth_d.data[:] - true_d.data[:]

        objective += .5*np.linalg.norm(residual.data.flatten())**2
        solver.gradient(rec=residual, u=u0, vp=vp_in, grad=grad)

    return objective, -grad.data
```

Having defined our FWI gradient procedure we can compute the initial iteration from our starting model. This allows us to visualize the gradient alongside the model perturbation and the effect of the gradient update on the model.

```python
#NBVAL_IGNORE_OUTPUT

# Compute gradient of initial model
ff, update = fwi_gradient(model0.vp)
print('Objective value is %f ' % ff)
```

    Objective value is 57242.092078

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_image

# Plot the FWI gradient
plot_image(update, vmin=-1e4, vmax=1e4, cmap="jet")

# Plot the difference between the true and initial model.
# This is not known in practice as only the initial model is provided.
plot_image(model0.vp.data - model.vp.data, vmin=-1e-1, vmax=1e-1, cmap="jet")

# Show what the update does to the model
alpha = .5 / np.abs(update).max()
plot_image(model0.vp.data - alpha*update, vmin=2.5, vmax=3.0, cmap="jet")
```

We see that the gradient and the true perturbation have the same sign, therefore, with an appropriate scaling factor, we will update the model in the correct direction.

```python
# Define bounding box constraints on the solution.
def apply_box_constraint(vp):
    # Maximum possible 'realistic' velocity is 3.5 km/sec
    # Minimum possible 'realistic' velocity is 2 km/sec
    return np.clip(vp, 2.0, 3.5)
```

```python
#NBVAL_SKIP

# Run FWI with gradient descent
history = np.zeros((fwi_iterations, 1))
for i in range(0, fwi_iterations):
    # Compute the functional value and gradient for the current
    # model estimate
    phi, direction = fwi_gradient(model0.vp)

    # Store the history of the functional values
    history[i] = phi

    # Artificial Step length for gradient descent
    # In practice this would be replaced by a Linesearch (Wolfe, ...)
    # that would guarantee functional decrease Phi(m-alpha g) <= epsilon Phi(m)
    # where epsilon is a minimum decrease constant
    alpha = .05 / np.abs(direction).max()

    # Update the model estimate and enforce minimum/maximum values
    model0.vp = apply_box_constraint(model0.vp.data - alpha * direction)

    # Log the progress made
    print('Objective value is %f at iteration %d' % (phi, i+1))
```

    Objective value is 57242.092078 at iteration 1
    Objective value is 35760.595247 at iteration 2
    Objective value is 21002.159605 at iteration 3
    Objective value is 11552.015849 at iteration 4
    Objective value is 5791.144152 at iteration 5

```python
#NBVAL_IGNORE_OUTPUT

# Plot inverted velocity model
plot_velocity(model0)
```

```python
#NBVAL_SKIP
import matplotlib.pyplot as plt

# Plot objective function decrease
plt.figure()
plt.loglog(history)
plt.xlabel('Iteration number')
plt.ylabel('Misift value Phi')
plt.title('Convergence')
plt.show()
```

## References

[1] _Virieux, J. and Operto, S.: An overview of full-waveform inversion in exploration geophysics, GEOPHYSICS, 74, WCC1–WCC26, doi:10.1190/1.3238367, http://library.seg.org/doi/abs/10.1190/1.3238367, 2009._

[2] _Haber, E., Chung, M., and Herrmann, F. J.: An effective method for parameter estimation with PDE constraints with multiple right hand sides, SIAM Journal on Optimization, 22, http://dx.doi.org/10.1137/11081126X, 2012._

This notebook is part of the tutorial "Optimised Symbolic Finite Difference Computation with Devito" presented at the Intel® HPC Developer Conference 2017.


# Effects of tangential velocity in the reactive relativistic Riemann problem

This notebook generates the results presented in the 2018 paper by A. Harpole and I. Hawke.

```python
from r3d2 import State, RiemannProblem, eos_defns, wave
from IPython.display import display_png
from matplotlib import pyplot
import matplotlib.gridspec as gridspec
%matplotlib inline
from matplotlib import rcParams, patches
from scipy.optimize import brentq
import numpy
```

```python
rcParams['figure.figsize'] = (10, 8)
rcParams['font.family'] = 'serif'
rcParams['mathtext.fontset'] = 'cm'
pyplot.rc("font", size=18)
```

```python
def rayleigh(Vs, u, q, gamma):
    def find_root(p, V):
        return p - 1 + u**2 * (V - 1 + gamma / (gamma - 1) * (p * V**2 - 1) - q)

    ps = numpy.zeros_like(Vs)
    pmin = 0
    pmax = 20
    for i, V in enumerate(Vs):
        try:
            ps[i] = brentq(find_root, pmin, pmax, args=(V))
        except ValueError:
            ps[i] = None
    return ps

def hugoniot(Vs, q, gamma):
    def find_root(p, V):
        return (gamma + 1) / (gamma-1) * (p*V - 1) - (p-V) - 2*q - \
            gamma/(gamma-1) * p*(1-V**2) + gamma / (gamma-1)**2 * (p**2*V**2 - 1) - \
            q*(p+(gamma+1)/(gamma-1)) - q**2

    ps = numpy.zeros_like(Vs)
    pmin = 0
    pmax = 20
    for i, V in enumerate(Vs):
        try:
            ps[i] = brentq(find_root, pmin, pmax, args=(V))

        except ValueError:
            ps[i] = None
    return ps
```

## Changing $q$

Pressure-volume plot for the relativistic reactive Riemann problem for normal 4-velocity $u_S = 0.35$, adiabatic index $\gamma=5/3$ and various values of $\hat{q}$. Plotted are the Rayleigh lines and the Hugoniot curves. Unlike for the Newtonian case, both the Rayleigh lines and the Hugoniot curves are functions of functions of $\hat{q}$, with only the lines for $\hat{q}$ intersecting the (1,1) point. As $\hat{q}$ increases, both the Rayleigh lines and Hugoniot curves move rightwards, away from the (1,1) point.

```python
gamma = 5/3
qs = [0, 1, 2]
Vs = numpy.linspace(0.1, 5)
r_ps = []
h_ps = []
u = 0.35

for q in qs:
    r_ps.append(rayleigh(Vs, u, q, gamma))
    h_ps.append(hugoniot(Vs, q, gamma))

fig, ax = pyplot.subplots()

for r, h, q in zip(r_ps, h_ps, qs):
    l = ax.plot(Vs, r, '-', linewidth=2, label=r'$\hat{q}'+f' = ${q}')[0]
    ax.plot(Vs, h, 'x:', color=l.get_color(), linewidth=2, label='_nolegend_')
ax.plot(Vs, numpy.ones_like(Vs), 'r:')
ax.plot([1,1], [0, 20], 'r:')
ax.set_xlabel(r'$\hat{v}$')
ax.set_ylabel(r'$\hat{p}$')
ax.set_xlim([Vs[0], Vs[-1]])
ax.set_ylim([0, 2.])

handles, labels = ax.get_legend_handles_labels()

simArtist = pyplot.Line2D((0,1),(0,0), color='k', linestyle='-')
anyArtist = pyplot.Line2D((0,1),(0,0), color='k', linestyle=':', marker='x')

#Create legend from custom artist/label lists
ax.legend([handle for i,handle in enumerate(handles)]+[simArtist,anyArtist],
          [label for i,label in enumerate(labels)]+['Rayleigh lines', 'Hugoniot curves'])
pyplot.show()
```

## Changing $v_t$

Pressure-volume plot for the relativistic reactive Riemann problem for normal 3-velocity $v_{x,S} = 0.25$ and $\hat{q}=0.5$. Plotted are the Hugoniot curve and Rayleigh lines for various values of tangential velocity $v_t$. Unlike for the Newtonian case, the coupling of the tangential velocity via the Lorentz factor means that the range of possible solutions changes with the tangential velocity.

$u = v_S W$, the product of the shock speed and the Lorentz factor. The Lorentz factor $W = (1 - v_S^2 - v_t^2)^{-\frac{1}{2}}$, where $v_t$ is the tangential velocity.

```python
gamma = 5/3
vts = [0, 0.8, 0.9, 0.96]
Vs = numpy.linspace(0.1, 6)
r_ps = []
Ws = []
vs = 0.25
q = 0.5

for vt in vts:
    Ws.append(1/numpy.sqrt(1 - vs**2 - vt**2))
    u = vs * Ws[-1]
    r_ps.append(rayleigh(Vs, u, q, gamma))

fig, ax = pyplot.subplots()
line_styles = ['--', '-x', '-.', '.-', ':']
ax.plot(Vs, hugoniot(Vs, q, gamma), 'k', linewidth=2, label='Hugoniot curve')
for r, vt, ls, W in zip(r_ps, vts, line_styles, Ws):
    ax.plot(Vs, r, ls, linewidth=2, label=f'$v_t = ${vt}, $W = {W:.3}$')
ax.plot(Vs, numpy.ones_like(Vs), 'r:')
ax.plot([1,1], [0, 20], 'r:')
ax.set_xlabel(r'$\hat{v}$')
ax.set_ylabel(r'$\hat{p}$')
ax.set_xlim([Vs[0], Vs[-1]])
ax.set_ylim([0, 2.5])
ax.legend()
pyplot.tight_layout()
pyplot.show()
```

## Inert relativistic Riemann problem, varying $v_t$

Exact solution of the inert relativistic Riemann problem for different initial tangential velocities. All cases share the initial conditions $(\rho, v_x, \varepsilon)_L = (1, 0, 10)$, $(\rho, v_x, \varepsilon)_R = (1, 0, 1)$. For the blue curves, $v_t (L, R) = (0,0)$, for the orange dashed curves, $v_t (L, R) = (0,0.9)$, and for the green dot-dashed curves, $v_t (L, R) = (0.9,0.9)$. The final state consists of a left-going rarefaction wave, a contact wave and a right-going shock wave: $\mathcal{R}_\leftarrow \,\mathcal{C}\, \mathcal{S}_\rightarrow$. Increasing the tangential velocity alters the intermediate state, reducing the normal velocity.

```python
vtls = [0., 0., 0.9]
vtrs = [0., 0.9, 0.9]
gamma = 5./3.

eos = eos_defns.eos_gamma_law(gamma)
rhos = []
ps = []
vs = []
vts = []
xis = []

for n, (vtl, vtr) in enumerate(zip(vtls, vtrs)):

    U_left = State(1.0, 0.0, vtl, 10.0, eos, label="L")
    U_right = State(1.0, 0, vtr, 1.0, eos, label="R")

    rp = RiemannProblem(U_left, U_right)

    data = rp.state_l.state()
    xi = [-1.05]

    for wave in rp.waves:
        xi_wave, data_wave = wave.plotting_data()
        xi = numpy.hstack((xi, xi_wave))
        data = numpy.vstack((data, data_wave))
    xi = numpy.hstack((xi, [1.05]))
    data = numpy.vstack((data, rp.state_r.state()))

    rhos.append([])
    rhos[-1] = data[:,0]
    ps.append([])
    ps[-1] = data[:,4]
    vs.append([])
    vs[-1] = data[:,1]
    vts.append([])
    vts[-1] = data[:,2]
    xis.append([])
    xis[-1] = xi
```

```python
rcParams['figure.figsize'] = (10, 8)
pyplot.rc("font", size=20)

fig, axs = pyplot.subplots(2,2)
lstyles = ['-', '--', '-.', ':']

# density
for n, r in enumerate(rhos):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$, ' + r'$v_{t,R}' + f' = {vtrs[n]}$'
    axs[0,0].plot(xis[n], r, lstyles[n], label=lab)
axs[0,0].set_xlim([-1.05, 1.05])
axs[0,0].set_xlabel(r'$\xi$')
axs[0,0].set_ylabel(r'$\rho$')
ylims = axs[0,0].get_ylim()
axs[0,0].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[0,0].get_lines(), 'linewidth', 3)

# pressure
for n, p in enumerate(ps):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$, ' + r'$v_{t,R}' + f' = {vtrs[n]}$'
    axs[0,1].plot(xis[n], p, lstyles[n], label=lab)
axs[0,1].set_xlim([-1.05, 1.05])
axs[0,1].set_xlabel(r'$\xi$')
axs[0,1].set_ylabel(r'$p$')
ylims = axs[0,1].get_ylim()
axs[0,1].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[0,1].get_lines(), 'linewidth', 3)

# v
for n, v in enumerate(vs):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$, ' + r'$v_{t,R}' + f' = {vtrs[n]}$'
    axs[1,0].plot(xis[n], v, lstyles[n], label=lab)
axs[1,0].set_xlim([-1.05, 1.05])
axs[1,0].set_xlabel(r'$\xi$')
axs[1,0].set_ylabel(r'$v$')
ylims = axs[1,0].get_ylim()
axs[1,0].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[1,0].get_lines(), 'linewidth', 3)

# vt
for n, vt in enumerate(vts):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$, ' + r'$v_{t,R}' + f' = {vtrs[n]}$'
    axs[1,1].plot(xis[n], vt, lstyles[n], label=lab)
axs[1,1].set_xlim([-1.05, 1.05])
axs[1,1].set_xlabel(r'$\xi$')
axs[1,1].set_ylabel(r'$v_t$')
ylims = axs[1,1].get_ylim()
axs[1,1].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[1,1].get_lines(), 'linewidth', 3)

fig.tight_layout()
lgd = axs[0,1].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

pyplot.show()
```

## Reactive relativistic Riemann problem, DDT varying $q$

Deflagration to detonation transition for different values of $q = (0.001, 0.02, 0.7)$, corresponding to the solid blue, dashed orange and dotted green curves respectively. The initial conditions are $(\rho, v_x, v_t, \varepsilon)_L = (1, 0, 0, 5.0)$, $(\rho, v_x, v_t, \varepsilon)_R = (1, -0.3, 0, 2)$. For $q=0.001$, the solution is a CJ deflagration, $\mathcal{(CJDF_\leftarrow\,R_\leftarrow)C\,S_\rightarrow}$. As $q$ is increased to $q=0.02$, the CJ deflagration becomes a weak deflagration, $\mathcal{WDF_\leftarrow\, C\,S_\rightarrow}$, and at $q=0.7$ this has transitioned to a CJ detonation, $\mathcal{(CJDT_\leftarrow\,R_\leftarrow)C\,S_\rightarrow}$.

```python
qs = [0.001, 0.02, 0.7]
gamma = 5/3

eos = eos_defns.eos_gamma_law(gamma)
rhos = []
ps = []
vs = []
cs = []
xis = []

for n, q in enumerate(qs):
    eos_reactive = eos_defns.eos_gamma_law_react(gamma, q, 1.0, 1.0, eos)

    U_reactive = State(1.0, 0.0, 0, 5.0, eos_reactive)
    U_burnt = State(1.0, -0.3, 0, 2, eos)

    rp = RiemannProblem(U_reactive, U_burnt)
    display(rp)

    data = rp.state_l.state()
    xi = [-1.05]

    for wave in rp.waves:
        xi_wave, data_wave = wave.plotting_data()
        xi = numpy.hstack((xi, xi_wave))
        data = numpy.vstack((data, data_wave))
    xi = numpy.hstack((xi, [1.05]))
    data = numpy.vstack((data, rp.state_r.state()))

    rhos.append([])
    rhos[-1] = data[:,0]
    ps.append([])
    ps[-1] = data[:,4]
    vs.append([])
    vs[-1] = data[:,1]
    cs.append([])
    cs[-1] = data[:,-1]
    # Convert speed of sound to be in the frame of wave 0.
    # For strong detonations this is unique; for other waves take
    # the maximum
    frame_speed = max(numpy.abs(rp.waves[0].wavespeed))
    cs[-1] = (cs[-1] - frame_speed) / (1 - cs[-1] * frame_speed)
    xis.append([])
    xis[-1] = xi
```

```python
rcParams['figure.figsize'] = (10, 8)
pyplot.rc("font", size=20)

fig, axs = pyplot.subplots(2,2)
lstyles = ['-', '--', '-.', ':']

# density
for n, r in enumerate(rhos):
    lab = r'$q_{L}' + f' = {qs[n]}$'
    axs[0,0].plot(xis[n], r, lstyles[n], label=lab)
axs[0,0].set_xlim([-1.05, 1.05])
axs[0,0].set_xlabel(r'$\xi$')
axs[0,0].set_ylabel(r'$\rho$')
ylims = axs[0,0].get_ylim()
axs[0,0].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[0,0].get_lines(), 'linewidth', 3)

# pressure
for n, p in enumerate(ps):
    lab = r'$q_{L}' + f' = {qs[n]}$'
    axs[0,1].plot(xis[n], p, lstyles[n], label=lab)
axs[0,1].set_xlim([-1.05, 1.05])
axs[0,1].set_xlabel(r'$\xi$')
axs[0,1].set_ylabel(r'$p$')
ylims = axs[0,1].get_ylim()
axs[0,1].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[0,1].get_lines(), 'linewidth', 3)

# v
for n, v in enumerate(vs):
    lab = r'$q_{L}' + f' = {qs[n]}$'
    axs[1,0].plot(xis[n], v, lstyles[n], label=lab)
axs[1,0].set_xlim([-1.05, 1.05])
axs[1,0].set_xlabel(r'$\xi$')
axs[1,0].set_ylabel(r'$v$')
ylims = axs[1,0].get_ylim()
axs[1,0].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[1,0].get_lines(), 'linewidth', 3)

# cs (in wave frame)
for n, c in enumerate(cs):
    lab = r'$q_{L}' + f' = {qs[n]}$'
    axs[1,1].plot(xis[n], c, lstyles[n], label=lab)
axs[1,1].set_xlim([-0.92, -0.68])
#axs[1,1].set_ylim([0.73, 0.785])
axs[1,1].set_ylim([-0.5, 0.1])
axs[1,1].set_xlabel(r'$\xi$')
axs[1,1].set_ylabel(r'$c_s$ (wave frame)')
ylims = axs[1,1].get_ylim()
axs[1,1].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[1,1].get_lines(), 'linewidth', 3)

fig.tight_layout()
lgd = axs[0,1].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
fig.savefig('vary_q.pdf', bbox_extra_artists=(lgd,), bbox_inches='tight')
```

## Reactive relativistic Riemann problem, DDT varying $v_t$

Deflagration to detonation transition for different values of $v_{t,L} = (0.0, 0.3, 0.6, 0.8)$, corresponding to the solid blue, dashed orange, dash-dotted green and dotted red lines respectively. The initial conditions are $(\rho, v_x, v_t, \varepsilon)_L = (1, 0, 0, 5.0)$, $(\rho, v_x, v_t, \varepsilon)_R = (1, -0.3, v_t, 2)$. For $v_{t,L} = 0$, the solution is $\mathcal{(CJDF_\leftarrow\,R_\leftarrow)\,C\,S_\rightarrow}$. At $v_{t,L} = 0.3$, the CJ deflagration has transitioned to a weak deflagration, producing the solution $\mathcal{WDF_\leftarrow\,C\,S_\rightarrow}$. At $v_t = 0.6$, the deflagration has transitioned to a CJ detonation, $\mathcal{(CJDT_\leftarrow\,R_\leftarrow)\,C\,S_\rightarrow}$, then at $v_t = 0.8$, this has transitioned to a strong detonation, $\mathcal{SDT_\leftarrow\,C\,S_\rightarrow}$.

```python
vtls = [0, 0.3, 0.6, 0.8]
q = 0.01
gamma = 5./3.

eos = eos_defns.eos_gamma_law(gamma)
rhos = []
ps = []
vs = []
vts = []
cs = []
xis = []

for n, vt in enumerate(vtls):
    eos_reactive = eos_defns.eos_gamma_law_react(gamma, q, 1.0, 1.0, eos)

    U_reactive = State(1.0, 0.0, 0, 5.0, eos_reactive)
    U_burnt = State(1, -0.3, vt, 2, eos)

    rp = RiemannProblem(U_reactive, U_burnt)
    display(rp)

    data = rp.state_l.state()
    xi = [-1.05]

    for wave in rp.waves:
        xi_wave, data_wave = wave.plotting_data()
        xi = numpy.hstack((xi, xi_wave))
        data = numpy.vstack((data, data_wave))
    xi = numpy.hstack((xi, [1.05]))
    data = numpy.vstack((data, rp.state_r.state()))

    rhos.append([])
    rhos[-1] = data[:,0]
    ps.append([])
    ps[-1] = data[:,4]
    vs.append([])
    vs[-1] = data[:,1]
    vts.append([])
    vts[-1] = data[:,2]
    cs.append([])
    cs[-1] = data[:,-1]
    # Convert speed of sound to be in the frame of wave 0.
    # For strong detonations this is unique; for other waves take
    # the maximum
    frame_speed = max(numpy.abs(rp.waves[0].wavespeed))
    cs[-1] = (cs[-1] - frame_speed) / (1 - cs[-1] * frame_speed)
    xis.append([])
    xis[-1] = xi
```

```python
fig = pyplot.figure(figsize=(15,13))
gs = gridspec.GridSpec(3,3, fig, wspace=0.7, hspace=0.4)
axs = []
axs.append(pyplot.subplot(gs[0,0]))
axs.append(pyplot.subplot(gs[0,1]))
axs.append(pyplot.subplot(gs[1,0]))
axs.append(pyplot.subplot(gs[1,1]))
axs.append(pyplot.subplot(gs[1,2]))

for n, r in enumerate(rhos):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$'
    axs[0].plot(xis[n], r, lstyles[n], label=lab)
axs[0].set_xlim([-1.05, 1.05])
axs[0].set_xlabel(r'$\xi$')
axs[0].set_ylabel(r'$\rho$')
ylims = axs[0].get_ylim()
axs[0].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[0].get_lines(), 'linewidth', 3)

# pressure
for n, p in enumerate(ps):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$'
    axs[1].plot(xis[n], p, lstyles[n], label=lab)
axs[1].set_xlim([-1.05, 1.05])
axs[1].set_xlabel(r'$\xi$')
axs[1].set_ylabel(r'$p$')
ylims = axs[1].get_ylim()
axs[1].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[1].get_lines(), 'linewidth', 3)

# v
for n, v in enumerate(vs):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$'
    axs[2].plot(xis[n], v, lstyles[n], label=lab)
axs[2].set_xlim([-1.05, 1.05])
axs[2].set_xlabel(r'$\xi$')
axs[2].set_ylabel(r'$v$')
ylims = axs[2].get_ylim()
axs[2].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[2].get_lines(), 'linewidth', 3)

# vt
for n, vt in enumerate(vts):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$'
    axs[3].plot(xis[n], vt, lstyles[n], label=lab)
axs[3].set_xlim([-1.05, 1.05])
axs[3].set_xlabel(r'$\xi$')
axs[3].set_ylabel(r'$v_t$')
ylims = axs[3].get_ylim()
axs[3].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[3].get_lines(), 'linewidth', 3)

# cs (in wave frame)
for n, vt in enumerate(cs):
    lab = r'$v_{t,L}' + f' = {vtls[n]}$'
    axs[4].plot(xis[n], vt, lstyles[n], label=lab)
axs[4].set_xlim([-0.9,-0.6])
#axs[4].set_ylim([0.7685, 0.775])
axs[4].set_ylim([-0.06, 0.06])
axs[4].set_xlabel(r'$\xi$')
axs[4].set_ylabel(r'$c_s$ (wave frame)')
ylims = axs[4].get_ylim()
axs[4].yaxis.set_ticks(numpy.linspace(ylims[0], ylims[1], num=4, endpoint=True))
pyplot.setp(axs[4].get_lines(), 'linewidth', 3)

lgd = axs[1].legend(bbox_to_anchor=(1.7, 1), loc=2, borderaxespad=0.)
fig.savefig('vary_vt.pdf', bbox_extra_artists=(lgd,), bbox_inches='tight')
```

To suggest that there is not a DDT as $v_t$ varies, we can plot the characteristic structure across the reactive wave. This compares the 'unrealistic' data above to 'realistic' data, where no DDT occurs.

```python
from r3d2.wave import UnphysicalSolution

# unphysical between 0.681 < vt < 0.85
vtrs_a = numpy.concatenate((numpy.arange(0, 0.65, 0.05), [0.681]))
vtrs_b = numpy.concatenate((numpy.arange(0.85, 0.975, 0.05), [0.975, 0.985, 0.99, 0.995]))
q = 1e-5
gamma = 5./3.

eos = eos_defns.eos_gamma_law(gamma)
eos_reactive = eos_defns.eos_gamma_law_react(gamma, q, 1e-4, 1, eos)
wavespeeds_no_ddt_a = numpy.zeros((len(vtrs_a), 3))
wavespeeds_no_ddt_b = numpy.zeros((len(vtrs_b), 3))

for n, vt in enumerate(vtrs_a):

    U_reactive = State(1.0, 0, 0, 1.5e-4, eos_reactive)
    U_burnt = State(0.1, 0, vt, 1.5e-4, eos)

    rp = RiemannProblem(U_reactive, U_burnt)
    wavespeeds_no_ddt_a[n, 0] = rp.waves[0].q_l.wavespeed(0)
    wavespeeds_no_ddt_a[n, 1] = rp.waves[0].wavespeed[0]
    wavespeeds_no_ddt_a[n, 2] = rp.waves[0].q_r.wavespeed(0)

for n, vt in enumerate(vtrs_b):

    U_reactive = State(1.0, 0, 0, 1.5e-4, eos_reactive)
    U_burnt = State(0.1, 0, vt, 1.5e-4, eos)

    rp = RiemannProblem(U_reactive, U_burnt)
    wavespeeds_no_ddt_b[n, 0] = rp.waves[0].q_l.wavespeed(0)
    wavespeeds_no_ddt_b[n, 1] = rp.waves[0].wavespeed[0]
    wavespeeds_no_ddt_b[n, 2] = rp.waves[0].q_r.wavespeed(0)

vtrs = numpy.concatenate((vtrs_a, [0.7], vtrs_b))
wavespeeds_no_ddt = numpy.zeros((len(vtrs), 3))
wavespeeds_no_ddt[:len(vtrs_a), :] = wavespeeds_no_ddt_a
wavespeeds_no_ddt[len(vtrs_a), :] = numpy.nan
wavespeeds_no_ddt[len(vtrs_a)+1:, :] = wavespeeds_no_ddt_b

# unphysical between 0.40128 < vt < 0.56673
vtrs2_a = numpy.arange(0, 0.45, 0.05)
vtrs2_b = numpy.concatenate(([0.57], numpy.arange(0.6, 0.9, 0.05), [0.9, 0.9125, 0.925, 0.9375, 0.95]))
q = 0.01
eos = eos_defns.eos_gamma_law(gamma)
eos_reactive = eos_defns.eos_gamma_law_react(gamma, q, 1.0, 1.0, eos)
wavespeeds_ddt_a = numpy.zeros((len(vtrs2_a), 3))
wavespeeds_ddt_b = numpy.zeros((len(vtrs2_b), 3))

for n, vt in enumerate(vtrs2_a):

    U_reactive = State(1, 0, 0, 5, eos_reactive)
    U_burnt = State(1, -0.3, vt, 2, eos)

    rp = RiemannProblem(U_reactive, U_burnt)
    wavespeeds_ddt_a[n, 0] = rp.waves[0].q_l.wavespeed(0)
    wavespeeds_ddt_a[n, 1] = rp.waves[0].wavespeed[0]
    wavespeeds_ddt_a[n, 2] = rp.waves[0].q_r.wavespeed(0)

for n, vt in enumerate(vtrs2_b):

    U_reactive = State(1, 0, 0, 5, eos_reactive)
    U_burnt = State(1, -0.3, vt, 2, eos)

    rp = RiemannProblem(U_reactive, U_burnt)
    wavespeeds_ddt_b[n, 0] = rp.waves[0].q_l.wavespeed(0)
    wavespeeds_ddt_b[n, 1] = rp.waves[0].wavespeed[0]
    wavespeeds_ddt_b[n, 2] = rp.waves[0].q_r.wavespeed(0)

vtrs2 = numpy.concatenate((vtrs2_a, [0.5], vtrs2_b))
wavespeeds_ddt = numpy.zeros((len(vtrs2), 3))
wavespeeds_ddt[:len(vtrs2_a), :] = wavespeeds_ddt_a
wavespeeds_ddt[len(vtrs2_a), :] = numpy.nan
wavespeeds_ddt[len(vtrs2_a)+1:, :] = wavespeeds_ddt_b

fig, axs = pyplot.subplots(1,2, figsize=(10,5))

axs[0].plot(vtrs, wavespeeds_no_ddt[:, 0],
            lstyles[0], linewidth=3, label=r"$\lambda^{(0)}_L$")
axs[0].plot(vtrs, wavespeeds_no_ddt[:, 1],
            lstyles[1], linewidth=3, label=r"$\lambda^{(0)}_{{\cal W}_L}$")
axs[0].plot(vtrs, wavespeeds_no_ddt[:, 2],
            lstyles[2], linewidth=3, label=r"$\lambda^{(0)}_{L_*}$")
axs[0].set_xlabel(r"$v_t$")
axs[0].set_ylabel("Characteristic speed")
axs[0].set_xlim(0, 1)

rect = patches.Rectangle((0.681, axs[0].get_ylim()[0]), 0.164, axs[1].get_ylim()[1]-axs[0].get_ylim()[0], color='grey', zorder=10)
axs[0].add_patch(rect)

axs[1].plot(vtrs2, wavespeeds_ddt[:, 0],
            lstyles[0], linewidth=3, label=r"$\lambda^{(0)}_L$")
axs[1].plot(vtrs2, wavespeeds_ddt[:, 1],
            lstyles[1], linewidth=3, label=r"$\lambda^{(0)}_{{\cal W}_\leftarrow}$")
axs[1].plot(vtrs2, wavespeeds_ddt[:, 2],
            lstyles[2], linewidth=3, label=r"$\lambda^{(0)}_{L_*}$")
axs[1].set_xlabel(r"$v_t$")
axs[1].set_xlim(0, 1)

rect = patches.Rectangle((0.4, axs[1].get_ylim()[0]), 0.17, axs[1].get_ylim()[1]-axs[1].get_ylim()[0], color='grey', zorder=10)
axs[1].add_patch(rect)
rect = patches.Rectangle((0.95, axs[1].get_ylim()[0]), 0.05, axs[1].get_ylim()[1]-axs[1].get_ylim()[0], color='grey', zorder=10)
axs[1].add_patch(rect)

fig.tight_layout()
lgd = axs[1].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
fig.savefig('vt_char_ns_ocean.pdf', bbox_extra_artists=(lgd,), bbox_inches='tight')
```


# Spectral Estimation of Random Signals

*This jupyter/Python notebook is part of a [collection of notebooks](../index.ipynb) in the masters module [Digital Signal Processing](http://www.int.uni-rostock.de/Digitale-Signalverarbeitung.48.0.html), Comunications Engineering, Universität Rostock. Please direct questions and suggestions to <mailto:Sascha.Spors@uni-rostock.de>.*

## Parametric Methods

### Motivation

Non-parametric methods for the estimation of the power spectral density (PSD), like the [periodogram](periodogram.ipynb) or [Welch's method](welch_method.ipynb), don't rely on a-priori information about the process generating the random signal. Often some a-priori information is available that can be used to formulate a parametric model of the random process. The goal is then to estimate these parameters in order to characterize the random signal. Such techniques are known as *[parametric methods](https://en.wikipedia.org/wiki/Spectral_density_estimation#Parametric_estimation)* or *model-based methods*. The incorporation of a-priori knowledge can improve the estimation of the PSD significantly, as long as the underlying model is a valid description of the random process. The parametric model of the random process can also be used to generate random signals with a desired PSD.

### Process Models

For the remainder we assume wide-sense stationary real-valued random processes. For many applications the process can be modeled by a linear-time invariant (LTI) system

where $n[k]$ is [white noise](../random_signals/white_noise.ipynb) and $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ denotes the transfer function of the system. In general, the random signal $x[k]$ will be correlated as a result of the processing of the uncorrelated input signal $n[k]$ by the system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. Due to the white noise assumption $\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0$, the PSD of the random process is given as

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0 \cdot | H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

Parametric methods model the system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ by a limited number of parameters. These parameters are then estimated from $x[k]$, providing an estimate $\hat{H}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the transfer function. This estimate is then used to calculate the desired estimate $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the PSD.

#### Autoregressive model

The [autoregressive](https://en.wikipedia.org/wiki/Autoregressive_model) (AR) model assumes a recursive system with a direct path. Its output relation is given as

\begin{equation}
x[k] = \sum_{n=1}^{N} a_n \cdot x[k-n] + n[k]
\end{equation}

where $a_n$ denote the coefficients of the recursive path and $N$ the order of the model. Its system function $H(z)$ is derived by $z$-transformation of the output relation

\begin{equation}
H(z) = \frac{1}{1 - \sum_{n=1}^{N} a_n z^{-n}}
\end{equation}

Hence, the AR model is a pole-only model of the system.

#### Moving average model

The [moving average](https://en.wikipedia.org/wiki/Moving-average_model) (MA) model assumes a non-recursive system. The output relation is given as

\begin{equation}
x[k] = \sum_{m=0}^{M-1} b_m \cdot n[k-m] = h[k] * n[k]
\end{equation}

with the impulse response of the system $h[k] = [ b_0, b_1, \dots, b_{M-1} ]$. The MA model is a finite impulse response (FIR) model of the random process. Its system function is given as

\begin{equation}
H(z) = \mathcal{Z} \{ h[k] \} = \sum_{m=0}^{M-1} b_m \; z^{-m}
\end{equation}

#### Autoregressive moving average model

The [autoregressive moving average](https://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-average_model) (ARMA) model is a combination of the AR and MA model. It constitutes a general linear process model. Its output relation is given as

\begin{equation}
x[k] = \sum_{n=1}^{N} a_n \cdot x[k-n] + \sum_{m=0}^{M-1} b_m \cdot n[k-m]
\end{equation}

Its system function reads

\begin{equation}
H(z) = \frac{\sum_{m=0}^{M-1} b_m \; z^{-m}}{1 - \sum_{n=1}^{N} a_n z^{-n}}
\end{equation}

### Parametric Spectral Estimation

The models above describe the synthesis of the samples $x[k]$ from the white noise $n[k]$. For spectral estimation only the random signal $x[k]$ is known and we are aiming at estimating the parameters of the model. This can be achieved by determining an analyzing system $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ such to decorrelate the signal $x[k]$

where $e[k]$ should be white noise. Due to its desired operation, the filter $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is also denoted as *whitening filter*. The optimal filter $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given by the inverse system $\frac{1}{H(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$. However, $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is in general not known. But this nevertheless implies that our linear process model of $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ also applies to $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. Various techniques have been developed to estimate the parameters of the filter $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ such that $e[k]$ becomes decorrelated. For instance, by expressing the auto-correlation function (ACF) $\varphi_{xx}[\kappa]$ in terms of the model parameters and solving with respect to these. The underlying set of equations are known as [Yule-Walker equations](https://en.wikipedia.org/wiki/Autoregressive_model#Yule-Walker_equations).

Once the model parameters have been estimated, these can be used to calculate an estimate $\hat{G}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the analysis system. The desired estimate of the PSD is then given as

\begin{equation}
\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{|\hat{G}(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2}
\end{equation}

where if $e[k]$ is white noise, $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0$.

### Example

In the following example $n[k]$ is drawn from normal distributed white noise with $N_0 = 1$. The Yule-Walker equations are used to estimate the parameters of an AR model of $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The implementation provided by `statsmodels.api.regression.yule_walker` returns the estimated AR coefficients of the system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. These parameters are then used to numerically evaluate the estimated transfer function, resulting in $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1 \cdot \vert \hat{H}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \vert^2$.

```python
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import scipy.signal as sig
%matplotlib inline

K = 4096  # length of random signal
N = 3  # order of AR model
a = np.array((1, -1, .5))  # coefficients of AR model

# generate random signal n[k]
np.random.seed(2)
n = np.random.normal(size=K)

# AR model for random signal x[k]
x = np.zeros(K)
for k in np.arange(3, K):
    x[k] = a[0]*x[k-1] + a[1]*x[k-2] + a[2]*x[k-3] + n[k]

# estimate AR parameters by Yule-Walker method
rho, sigma = sm.regression.yule_walker(x, order=N, method='mle')

# compute true and estimated transfer function
Om, H = sig.freqz(1, np.insert(-a, 0, 1))
Om, He = sig.freqz(1, np.insert(-rho, 0, 1))
# compute PSD by Welch method
Om2, Pxx = sig.welch(x, return_onesided=True)

# plot PSDs
plt.figure(figsize=(10, 5))
plt.plot(Om, np.abs(H)**2, label=r'$\Phi_{xx}(e^{j\Omega})$')
plt.plot(Om2*2*np.pi, .5*np.abs(Pxx), 'k-', alpha=.5,
         label=r'$\hat{\Phi}_{xx}(e^{j\Omega})$ (Welch)')
plt.plot(Om, np.abs(He)**2,
         label=r'$\hat{\Phi}_{xx}(e^{j\Omega})$ (parametric)')

plt.xlabel(r'$\Omega$')
plt.axis([0, np.pi, 0, 20])
plt.legend()
plt.grid()
```

**Exercise**

* Change the order `N` of the AR model used for estimation by the Yule-Walker equations. What happens if the order is smaller or higher than the order of the true system? Why?

* Change the number of samples `K`. Is the estimator consistent?

Solution: Choosing the order of the estimated AR model differently from the true process results in a mismatch of the model with the consequence of potentially large deviations between the estimated PSD and true PSD. These deviations are typically larger when choosing the order smaller since the model does definitely not fit to the true process. However, choosing the order larger is typically not that problematic since some of the AR coefficients are estimated as approximately zero due to the lower order of the process generating the random signal. Increasing the number of samples seems to lower the bias and variance of the estimated PSD, the estimator can therefore be assumed to be consistent.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# Spectral Estimation of Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Introduction

In the preceding sections various statistical measures have been introduced to characterize random processes and signals. For instance, the probability density function (PDF) $p_x(\theta)$, the mean value $\mu_x$, the auto-correlation function (ACF) $\varphi_{xx}[\kappa]$ and its Fourier transformation, the power spectral density (PSD) $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. For many random processes whose internal structure is known, these measures can be derived in closed-form. However, for practical random signals measures of interest have to be estimated from a limited number of samples. These estimated quantities can e.g. be used to fit a parametric model of the random process or as parameters in algorithms.

### Problem Statement

The estimation of the spectral properties of a random signal is of special interest for spectral analysis. The discrete Fourier transform (DFT) of a random signal is also random. It is not very well suited to gain insights into the average spectral structure of a random signal. We aim at estimating the PSD $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of a wide-sense stationary (WSS) and ergodic process from a limited number of samples. This is known as [*spectral (density) estimation*](https://en.wikipedia.org/wiki/Spectral_density_estimation). Many techniques have been developed for this purpose. They can be classified into

1. non-parametric and
2. parametric

techniques. Non-parametric techniques estimate the PSD of the random signal without assuming any particular structure for the generating random process. In contrary, parametric techniques assume that the generating random process can be modeled by a few parameters. Their aim is to estimate these parameters in order to characterize the spectral properties of the random signal.

### Evaluation

Various measures have been introduced in order to evaluate the performance of a particular estimation technique. The estimate $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can be regarded as a random signal itself. The performance of an estimator is therefore evaluated in a statistical sense. For the PSD, the following metrics are of interest

#### Bias

The [bias of an estimator](https://en.wikipedia.org/wiki/Estimator#Bias)

\begin{equation}
b_{\hat{\Phi}_{xx}}
= E\{ \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) - \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\}
= E\{ \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \} - \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

quantifies the difference between the estimated $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the true $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. An estimator is
* biased if $b_{\hat{\Phi}_{xx}} \neq 0$ and
* bias-free if $b_{\hat{\Phi}_{xx}} = 0$.

#### Variance

The [variance of an estimator](https://en.wikipedia.org/wiki/Estimator#Variance)

\begin{equation}
\sigma^2_{\hat{\Phi}_{xx}} = E \left\{ \left(\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) - E\{ \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\} \right)^2 \right\}
\end{equation}

quantifies its quadratic deviation from its mean value $E\{ \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\}$.

#### Consistency

A [consistent estimator](https://en.wikipedia.org/wiki/Estimator#Consistency) is an estimator for which the following conditions hold for a large number $N$ of samples:

1. the estimator is unbiased
    $$ \lim_{N \to \infty} b_{\hat{\Phi}_{xx}} = 0 $$

2. its variance converges towards zero
    $$ \lim_{N \to \infty} \sigma^2_{\hat{\Phi}_{xx}} = 0 $$

3. it converges in probability to the true $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$
    $$ \lim_{N \to \infty} \Pr \left\{ | \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) -  \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})| > \alpha \right\} = 0$$
    where $\alpha > 0$ denotes a (small) constant.

The last condition ensures that a consistent estimator does not generate outliers. Consistency is a desired property of an estimator. It ensures that if the number of samples $N$ increases towards infinity, the resulting estimates converges towards the true PSD.

#### Example - Discrete Fourier transform of sample functions

The following example computes and plots the magnitude of the discrete Fourier transform (DFT) $|X_n[\mu]|$ of an ensemble of random signals $x_n[k]$. In the plot, each color denotes one sample function and $\Omega[\mu] = \frac{2 \pi}{N} \mu$ where $N$ denotes the length of the sample function. The magnitude spectra are plotted as continuous signals for ease of illustration.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 256  # number of samples
M = 5  # number of sample functions

# generate random signal
np.random.seed(1)
x = np.random.normal(size=(M, N))
h = sig.firwin2(N, [0, .4, .42, .65, .67, 1], [0, 0, 1, 1, 0, 0])
x = [np.convolve(xi, h, mode='same') for xi in x]

# DFT of signal
X = np.fft.rfft(x, axis=1)
Om = np.linspace(0, np.pi, X.shape[1])

# plot signal and its spectrum
plt.figure(figsize=(10, 4))
plt.plot(Om, np.abs(X.T))
plt.title('Magnitude spectrum')
plt.xlabel(r'$\Omega[\mu]$')
plt.ylabel(r'$|X[\mu]|$')
plt.axis([0, np.pi, 0, 30])
plt.grid()
```

**Exercise**

* Increase the number `N` of samples. What changes? What does not change with respect to the evaluation criteria introduced above?
* Is the DFT of a single sample function a consistent estimator for the spectral properties of a random process?

Solution: Increasing the number of samples does only lead to an increase in the number of discrete frequencies $\mu$. The amplitude of the fluctuations (variance) of the spectra within $1.3 < \Omega < 2$ is not decreased when increasing the number of samples. The DFT of a single sample function is hence not a consistent estimator since at least the second condition is violated.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples.


## **Viscoelastic wave equation implementation on a staggered grid**

This is a first attempt at implementing the viscoelastic wave equation as described in [1]. See also the FDELMODC implementation by Jan Thorbecke [2].

In the following example, a three dimensional toy problem will be introduced consisting of a single Ricker source located at (100, 50, 35) in a 200 m $\times$ 100 m $\times$ 100 *m* domain.

```python
# Required imports:
import numpy as np
import sympy as sp

from devito import *
from examples.seismic.source import RickerSource, TimeAxis
from examples.seismic import ModelViscoelastic, plot_image
```

The model domain is now constructed. It consists of an upper layer of water, 50 m in depth, and a lower rock layer separated by a 4 m thick sediment layer.

```python
# Domain size:
extent = (200., 100., 100.) # 200 x 100 x 100 m domain
h = 1.0 # Desired grid spacing
shape = (int(extent[0]/h+1), int(extent[1]/h+1), int(extent[2]/h+1))

# Model physical parameters:
vp = np.zeros(shape)
qp = np.zeros(shape)
vs = np.zeros(shape)
qs = np.zeros(shape)
rho = np.zeros(shape)

# Set up three horizontally separated layers:
vp[:,:,:int(0.5*shape[2])+1] = 1.52
qp[:,:,:int(0.5*shape[2])+1] = 10000.
vs[:,:,:int(0.5*shape[2])+1] = 0.
qs[:,:,:int(0.5*shape[2])+1] = 0.
rho[:,:,:int(0.5*shape[2])+1] = 1.05

vp[:,:,int(0.5*shape[2])+1:int(0.5*shape[2])+1+int(4/h)] = 1.6
qp[:,:,int(0.5*shape[2])+1:int(0.5*shape[2])+1+int(4/h)] = 40.
vs[:,:,int(0.5*shape[2])+1:int(0.5*shape[2])+1+int(4/h)] = 0.4
qs[:,:,int(0.5*shape[2])+1:int(0.5*shape[2])+1+int(4/h)] = 30.
rho[:,:,int(0.5*shape[2])+1:int(0.5*shape[2])+1+int(4/h)] = 1.3

vp[:,:,int(0.5*shape[2])+1+int(4/h):] = 2.2
qp[:,:,int(0.5*shape[2])+1+int(4/h):] = 100.
vs[:,:,int(0.5*shape[2])+1+int(4/h):] = 1.2
qs[:,:,int(0.5*shape[2])+1+int(4/h):] = 70.
rho[:,:,int(0.5*shape[2])+1+int(4/h):] = 2.
```

Now create a Devito vsicoelastic model generating an appropriate computational grid along with absorbing boundary layers:

```python
# Create model
origin = (0, 0, 0)
spacing = (h, h, h)
so = 4 # FD space order (Note that the time order is by default 1).
nbl = 20 # Number of absorbing boundary layers cells
model = ModelViscoelastic(space_order=so, vp=vp, qp=qp, vs=vs, qs=qs,
                          rho=rho, origin=origin, shape=shape, spacing=spacing,
                          nbl=nbl)
```

    Operator `initdamp` run in 0.05 s
    Operator `padfunc` run in 0.02 s
    Operator `padfunc` run in 0.01 s
    Operator `padfunc` run in 0.01 s
    Operator `padfunc` run in 0.01 s
    Operator `padfunc` run in 0.02 s

The source frequency is now set along with the required model parameters:

```python
# Source freq. in MHz (note that the source is defined below):
f0 = 0.12

# Thorbecke's parameter notation
l = model.lam
mu = model.mu
ro = model.irho

k = 1.0/(l + 2*mu)
pi = l + 2*mu

t_s = (sp.sqrt(1.+1./model.qp**2)-1./model.qp)/f0
t_ep = 1./(f0**2*t_s)
t_es = (1.+f0*model.qs*t_s)/(f0*model.qs-f0**2*t_s)
```

```python
# Time step in ms and time range:
t0, tn = 0., 30.
dt = model.critical_dt
time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

Generate Devito time functions for the velocity, stress and memory variables appearing in the viscoelastic model equations. By default, the initial data of each field will be set to zero.

```python
# PDE fn's:
x, y, z = model.grid.dimensions
damp = model.damp

# Staggered grid setup:

# Velocity:
v = VectorTimeFunction(name="v", grid=model.grid, time_order=1, space_order=so)

# Stress:
tau = TensorTimeFunction(name='t', grid=model.grid, space_order=so, time_order=1)

# Memory variable:
r = TensorTimeFunction(name='r', grid=model.grid, space_order=so, time_order=1)

s = model.grid.stepping_dim.spacing # Symbolic representation of the model grid spacing
```

And now the source and PDE's are constructed:

```python
# Source
src = RickerSource(name='src', grid=model.grid, f0=f0, time_range=time_range)
src.coordinates.data[:] = np.array([100., 50., 35.])

# The source injection term
src_xx = src.inject(field=tau[0, 0].forward, expr=src*s)
src_yy = src.inject(field=tau[1, 1].forward, expr=src*s)
src_zz = src.inject(field=tau[2, 2].forward, expr=src*s)

# Particle velocity
u_v = Eq(v.forward, model.damp * (v + s*ro*div(tau)))

# Stress equations:
u_t = Eq(tau.forward,  model.damp *  (s*r.forward + tau +
                                      s * (l * t_ep / t_s * diag(div(v.forward)) +
                                           mu * t_es / t_s * (grad(v.forward) + grad(v.forward).T))))

# Memory variable equations:
u_r = Eq(r.forward, damp * (r - s / t_s * (r + l * (t_ep/t_s-1) * diag(div(v.forward)) +
                                           mu * (t_es/t_s-1) * (grad(v.forward) + grad(v.forward).T) )))
```

We now create and then run the operator:

```python
# Create the operator:
op = Operator([u_v, u_r, u_t] + src_xx + src_yy + src_zz,
               subs=model.spacing_map)
```

```python
#NBVAL_IGNORE_OUTPUT

# Execute the operator:
op(dt=dt)
```

    Operator `Kernel` run in 62.86 s

Before plotting some results, let us first look at the shape of the data stored in one of our time functions:

```python
v[0].data.shape
```

    (2, 241, 141, 141)

Since our functions are first order in time, the time dimension is of length 2. The spatial extent of the data includes the absorbing boundary layers in each dimension (i.e. each spatial dimension is padded by 20 grid points to the left and to the right).

The total number of instances in time considered is obtained from:

```python
time_range.num
```

    136

Hence 223 time steps were executed. Thus the final time step will be stored in index given by:

```python
np.mod(time_range.num,2)
```

    0

Now, let us plot some 2D slices of the fields `vx` and `szz` at the final time step:

```python
#NBVAL_SKIP

# Mid-points:
mid_x = int(0.5*(v[0].data.shape[1]-1))+1
mid_y = int(0.5*(v[0].data.shape[2]-1))+1

# Plot some selected results:

plot_image(v[0].data[1, :, mid_y, :], cmap="seismic")
plot_image(v[0].data[1, mid_x, :, :], cmap="seismic")

plot_image(tau[2, 2].data[1, :, mid_y, :], cmap="seismic")
plot_image(tau[2, 2].data[1, mid_x, :, :], cmap="seismic")
```

```python
#NBVAL_IGNORE_OUTPUT

assert np.isclose(norm(v[0]), 0.102959, atol=1e-4, rtol=0)
```

# References

[1] Johan O. A. Roberston, *et.al.* (1994). "Viscoelatic finite-difference modeling" GEOPHYSICS, 59(9), 1444-1456.

[2] https://janth.home.xs4all.nl/Software/fdelmodcManual.pdf


## 11.3 양자인공지능 알고리즘 구현
### 11.3.1 파라미터가 있는 양자컴퓨팅
#### 측정이 없는 가변양자회로
- 필요한 패키지를 불러옵니다

```python
import cirq
import sympy
import numpy as np
from cirq.contrib.svg import SVGCircuit
```

- 파라미터로 사용할 변수를 심볼릭 변수인 sympy로 정의

```python
W = sympy.symbols('W')
```

- 만들게 될 양자회로

```python
Qubit = cirq.NamedQubit("Qubit")
circuit = cirq.Circuit(
    cirq.ry(W).on(Qubit)) #, cirq.measure(Qubit, key='m'))

SVGCircuit(circuit)
```

- 회로의 동작을 시뮬레이션하는 함수를 만듭니다

```python
def simulation(circuit, w=np.pi/2):
    cirq_simulator = cirq.Simulator()
    resolver = cirq.ParamResolver({W: w})
    result = cirq_simulator.simulate(circuit, resolver)
    return result
```

- 시뮬레이션과 그 결과

```python
print(simulation(circuit, 0))
print(simulation(circuit, np.pi/2))
print(simulation(circuit, np.pi))
```

    measurements: (no measurements)
    output vector: |0⟩
    measurements: (no measurements)
    output vector: 0.707|0⟩ + 0.707|1⟩
    measurements: (no measurements)
    output vector: |1⟩

#### 측정이 있는 가변양자회로

```python
circuit = cirq.Circuit(
    cirq.ry(W).on(Qubit), cirq.measure(Qubit, key='m'))
SVGCircuit(circuit)
```

---
#### 전체코드

```python
import cirq
import sympy
import numpy as np
from cirq.contrib.svg import SVGCircuit

W = sympy.symbols('W')
# 측정이 없는 가변양자회로
Qubit = cirq.NamedQubit("Qubit")
circuit = cirq.Circuit(
    cirq.ry(W).on(Qubit))

def simulation(circuit, w=np.pi/2):
    cirq_simulator = cirq.Simulator()
    resolver = cirq.ParamResolver({W: w})
    result = cirq_simulator.simulate(circuit, resolver)
    return result
# 측정이 없는 가변양자회로
print(simulation(circuit, 0))
print(simulation(circuit, np.pi/2))
print(simulation(circuit, np.pi))

#
Qubit = cirq.NamedQubit("Qubit")
circuit = cirq.Circuit(
    cirq.ry(W).on(Qubit), cirq.measure(Qubit, key='m'))

def simulation(circuit, w=np.pi/2):
    cirq_simulator = cirq.Simulator()
    resolver = cirq.ParamResolver({W: w})
    result = cirq_simulator.simulate(circuit, resolver)
    return result

print(simulation(circuit, 0))
print(simulation(circuit, np.pi/2))
print(simulation(circuit, np.pi))
```

    measurements: (no measurements)
    output vector: |0⟩
    measurements: (no measurements)
    output vector: 0.707|0⟩ + 0.707|1⟩
    measurements: (no measurements)
    output vector: |1⟩
    measurements: m=0
    output vector: |0⟩
    measurements: m=1
    output vector: |1⟩
    measurements: m=1
    output vector: |1⟩

### 11.3.2 양자인공지능 학습하기

#### 양자인공지능 패키지 설치하기

```python
!conda install python=3.7
!pip install tensorflow==2.4.1
!pip install tensorflow-quantum
```

#### 관련 패키지 부르기

```python
import tensorflow_quantum as tfq
import cirq
from cirq.contrib.svg import SVGCircuit

import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras import Sequential, Input, Model

import sympy
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
```

#### 입력 데이터와 기대 출력 레이블 만들기

```python
X_Classic = np.array([[0], [1]], dtype=np.float32)
```

```python
Qubit = cirq.GridQubit(0, 0)

Initial_rotation_phase = np.random.uniform(0, 2 * np.pi)
X_Quantum_circuit = cirq.Circuit(
  cirq.ry(Initial_rotation_phase)(Qubit)
)
X_Quantum = tfq.convert_to_tensor(
  [X_Quantum_circuit]*len(X_Classic))

Y = np.array([[1], [-1]], dtype=np.float32)
```

#### 전통인공지능 부분 만들기: 파라미터 제어 뉴럴넷

```python
def make_classical_NN(x):
    model = Sequential()
    model.add(Dense(2, activation='relu'))
    model.add(Dense(1))
    return model(x)

Classical_NN_In = Input(shape=(1,), dtype=tf.dtypes.float32,
                         name='commands_input')
Classical_NN = make_classical_NN(Classical_NN_In)
```

#### 파라미터를 포함하는 양자회로 만들기

```python
W = sympy.symbols('W')
Quantum_circuit = cirq.Circuit(cirq.ry(W)(Qubit))
SVGCircuit(Quantum_circuit)
```

### 하이브리드 인공지능 만들기: 양자 및 전통 인공지능 복합형

```python
Quantum_In = Input(shape=(),dtype=tf.string, name='circuits_input')
Quantum_layer = tfq.layers.ControlledPQC(Quantum_circuit,
                                          operators = cirq.Z(Qubit))
expectation = Quantum_layer([Quantum_In, Classical_NN])
model = Model(inputs=[Quantum_In, Classical_NN_In],
                       outputs=expectation)
tf.keras.utils.plot_model(model, show_shapes=True, dpi=70)
```

```python
optimizer = tf.keras.optimizers.Adam(learning_rate=0.05)
loss = tf.keras.losses.MeanSquaredError()
model.compile(optimizer=optimizer, loss=loss)
```

```python
history = model.fit(x=[X_Quantum, X_Classic],
                    y=Y, epochs=50, verbose=0)
```

```python
plt.plot(history.history['loss'])
plt.title("Training for Quantum-Classic AI")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.grid()
plt.show()
```

- 모델의 예측 결과 확인

```python
Y_pred = model([X_Quantum, X_Classic]).numpy()
Err = Y - Y_pred
print('Predicted_Y =', Y_pred.flatten())
print('Y - Predicted_Y =', Err.flatten())
```

    Predicted_Y = [ 0.90674293 -0.99880725]
    Y - Predicted_Y = [ 0.09325707 -0.00119275]

---
#### 전체 코드

```python
import tensorflow_quantum as tfq
import cirq
from cirq.contrib.svg import SVGCircuit

import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras import Sequential, Input, Model

import sympy
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt

## 입력 데이터와 기대 출력 레이블 만들기
X_Classic = np.array([[0], [1]], dtype=np.float32)

Qubit = cirq.GridQubit(0, 0)
Initial_rotation_phase = np.random.uniform(0, 2 * np.pi)
X_Quantum_circuit = cirq.Circuit(
  cirq.ry(Initial_rotation_phase)(Qubit)
)
X_Quantum = tfq.convert_to_tensor(
  [X_Quantum_circuit]*len(X_Classic))

Y = np.array([[1], [-1]], dtype=np.float32)

## 전통인공지능 부분 만들기: 파라미터 제어 뉴럴넷
def make_classical_NN(x):
    model = Sequential()
    model.add(Dense(2, activation='relu'))
    model.add(Dense(1))
    return model(x)
Classical_NN_In = Input(shape=(1,), dtype=tf.dtypes.float32, name='commands_input')
Classical_NN = make_classical_NN(Classical_NN_In)

## 파라미터를 포함하는 양자회로 만들기
W = sympy.symbols('W')

Quantum_circuit = cirq.Circuit(cirq.ry(W)(Qubit))
SVGCircuit(Quantum_circuit)

## 하이브리드 인공지능 부분 만들기: 양자 및 전통 인공지능 복합형
Quantum_In = Input(shape=(),dtype=tf.string, name='circuits_input')
Quantum_layer = tfq.layers.ControlledPQC(Quantum_circuit,
                                          operators = cirq.Z(Qubit))
expectation = Quantum_layer([Quantum_In, Classical_NN])
model = Model(inputs=[Quantum_In, Classical_NN_In],
                       outputs=expectation)

## 모델 학습 및 결과 확인하기
optimizer = tf.keras.optimizers.Adam(learning_rate=0.05)
loss = tf.keras.losses.MeanSquaredError()
model.compile(optimizer=optimizer, loss=loss)
history = model.fit(x=[X_Quantum, X_Classic],
                    y=Y,
                    epochs=50,
                    verbose=0)

plt.plot(history.history['loss'])
plt.title("Training for Quantum-Classic AI")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.grid()
plt.show()

Y_pred = model([X_Quantum, X_Classic]).numpy()
Err = Y - Y_pred
print('Predicted_Y =', Y_pred.flatten())
print('Y - Predicted_Y =', Err.flatten())
```

```python

```


# Collapsed Gibbs sampler for binary logistic supervised latent Dirichlet allocation

  $
      \DeclareMathOperator{\dir}{Dirichlet}
      \DeclareMathOperator{\dis}{Discrete}
      \DeclareMathOperator{\normal}{Normal}
      \DeclareMathOperator{\ber}{Bernoulli}
      \DeclareMathOperator{\diag}{diag}
      \DeclareMathOperator{\Betaf}{B}
      \DeclareMathOperator{\Gammaf}{\Gamma}
      \DeclareMathOperator{\PG}{PG}
      \newcommand{\norm}[1]{\left\| #1 \right\|}
      \newcommand{\cp}[2]{p \left( #1 \middle| #2 \right)}
      \newcommand{\cN}[2]{\mathscr{N} \left( #1 \middle| #2 \right)}
      \newcommand{\cpsi}[2]{\psi \left( #1 \middle| #2 \right)}
      \newcommand{\cPsi}[2]{\Psi \left( #1 \middle| #2 \right)}
      \newcommand{\etd}[1]{\mathbf{z}^{(#1)}}
      \newcommand{\sumetd}{\mathbf{z}}
      \newcommand{\one}{\mathbf{1}}
  $

Here is the collapsed Gibbs sampler for Zhu et al.'s [improved Bayesian logistic supervised topic model with data augmentation](http://arxiv.org/pdf/1310.2408v1.pdf).  I am building on the [collapsed Gibbs sampler](http://nbviewer.savvysherpa.com/github/gp-0058-clt-at-toys-r-us/relational-topic-models/blob/master/slda.ipynb) I wrote for supervised latent Dirichlet allocation.

The generative model for BLSLDA is as follows:

$$\begin{align}
\theta^{(d)} &\sim \dir(\alpha) &\text{(topic distribution for document $d \in \{1, \ldots, D\}$)} \\
\phi^{(k)} &\sim \dir(\beta) &\text{(term distribution for topic $k \in \{1, \ldots, K\}$)} \\
z_n^{(d)} \mid \theta^{(d)} &\sim \dis \left( \theta^{(d)} \right) &\text{(topic of $n$th token of document $d$, $n \in \{1, \ldots, N^{(d)}\}$)} \\
w_n^{(d)} \mid \phi^{(z_n^{(d)})} &\sim \dis \left( \phi^{(z_n^{(d)})} \right) &\text{(term of $n$th token of document $d$, $n \in \{1, \ldots, N^{(d)}\}$)} \\
\eta_k &\sim \normal \left( \mu, \nu^2 \right) &\text{(regression coefficient for topic $k \in \{1, \ldots, K\}$)} \\
y^{(d)} \mid \eta, \etd{d} &\sim \ber \left( \frac{ \exp \left( \eta \cdot \etd{d} \right) }{ 1 + \exp \left( \eta \cdot \etd{d} \right) } \right) &\text{(response value of document $d \in \{1, \ldots, D\}$)}
\end{align}$$

where each token can be any one of $V$ terms in our vocabulary, and $\etd{d}$ is the empirical topic distribution of document $d$.

Plate notation for supervised latent Dirichlet allocation.

This diagram should replace $\beta_k$ with $\phi^{(k)}$, and each $\phi^{(k)}$ should be dependent on a single $\beta$.

Following [Zhu et al. 2013](http://arxiv.org/pdf/1310.2408v1.pdf), the regularized pseudo-likelihood for the response variable $y^{(d)}$, with regularization parameter $b \ge 0$, can be written

$$\begin{align}
\cpsi{y^{(d)}}{\eta, \etd{d}, b}
&= \cp{y^{(d)}}{\eta, \etd{d}}^b
\\ &= \left( \frac{\exp \left( \eta \cdot \etd{d} \right)^{y^{(d)}}}{ 1 + \exp \left( \eta \cdot \etd{d} \right)} \right)^b
\\ &= \frac{\exp \left( b y^{(d)} (\eta \cdot \etd{d}) \right)}
{ \left( \exp \left( -\frac{\eta \cdot \etd{d}}{2} \right) + \exp \left( \frac{ \eta \cdot \etd{d} }{2} \right) \right)^b \exp \left( \frac{b}{2} (\eta \cdot \etd{d}) \right) }
\\ &= 2^{-b} \exp \left( b \left( y^{(d)} - \frac{1}{2} \right) \left( \eta \cdot \etd{d} \right) \right) \cosh \left( \frac{ \eta \cdot \etd{d} }{2} \right)^{-b}
\\ &= 2^{-b} \exp \left( b \left( y^{(d)} - \frac{1}{2} \right) \left( \eta \cdot \etd{d} \right) \right)
\int_0^\infty \exp \left( -\frac{ \left( \eta \cdot \etd{d} \right)^2 }{2} \omega^{(d)} \right)
\cp{\omega^{(d)}}{b, 0} d\omega^{(d)}
\end{align}$$

where $\omega^{(d)}$ is a Polya-Gamma distributed variable with parameters $b = b$ and $c = 0$ (see [Polson et al. 2012](http://arxiv.org/pdf/1205.0310v3.pdf) for details).  This means that, for each document $d$, the pseudo-likelihood of $y^{(d)}$ is actually a mixture of Gaussians with respect to the Polya-Gamma distribution $\PG(b, 0)$.  Therefore, the joint pseudo-likelihood of $y^{(d)}$ and $\omega^{(d)}$ can be written

$$\cPsi{y^{(d)}, \omega^{(d)}}{\eta, \etd{d}, b}
= 2^{-b} \exp \left( \kappa^{(d)} \zeta^{(d)} - \frac{ \omega^{(d)} }{2} (\zeta^{(d)})^2 \right) \cp{\omega^{(d)}}{b, 0}.$$

where $\kappa^{(d)} = b(y^{(d)} - 1/2)$ and $\zeta^{(d)} = \eta \cdot \etd{d}$.  The joint probability distribution can therefore be factored as follows:

$$\begin{align}
\cp{\theta, \phi, z, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
&=
\prod_{k=1}^{K} \cp{\phi^{(k)}}{\beta}
\prod_{d=1}^{D} \cp{\theta^{(d)}}{\alpha}
\prod_{n=1}^{N^{(d)}} \cp{z_n^{(d)}}{\theta^{(d)}} \cp{w_n^{(d)}}{\phi^{(z_n^{(d)})}}
\\ & \quad \times \prod_{k'=1}^{K} \cp{\eta_{k'}}{\mu, \nu^2}
\prod_{d'=1}^D \cPsi{y^{(d')}, \omega^{(d')}}{\eta, \etd{d'}, b}
\\ &=
\prod_{k=1}^{K} \frac{\Betaf(b^{(k)} + \beta)}{\Betaf(\beta)} \cp{\phi^{(k)}}{b^{(k)} + \beta}
\prod_{d=1}^{D} \frac{\Betaf(a^{(d)} + \alpha)}{\Betaf(\alpha)} \cp{\theta^{(d)}}{a^{(d)} + \alpha}
\\ &\quad \times
\prod_{k'=1}^{K} \cN{\eta_{k'}}{\mu, \nu^2}
\prod_{d'=1}^{D} 2^{-b} \exp \left( \kappa^{(d')} \zeta^{(d')} - \frac{ \omega^{(d')} }{2} (\zeta^{(d')})^2 \right) \cp{\omega^{(d')}}{b, 0}
\end{align}$$

where $a_k^{(d)}$ is the number of tokens in document $d$ assigned to topic $k$, $b_v^{(k)}$ is the number of tokens equal to term $v$ and assigned to topic $k$, and $\Betaf$ is the [multivariate Beta function](https://en.wikipedia.org/wiki/Beta_function#Multivariate_beta_function). Marginalizing out $\theta$ and $\phi$ by integrating with respect to each $\theta^{(d)}$ and $\phi^{(k)}$ over their respective sample spaces yields

$$\begin{align}
\cp{z, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
&=
\prod_{k=1}^{K} \frac{\Betaf(b^{(k)} + \beta)}{\Betaf(\beta)}
\prod_{d=1}^{D} \frac{\Betaf(a^{(d)} + \alpha)}{\Betaf(\alpha)}
\prod_{k'=1}^{K} \cN{\eta_{k'}}{\mu, \nu^2}
\prod_{d'=1}^{D} 2^{-b} \exp \left( \kappa^{(d')} \zeta^{(d')} - \frac{ \omega^{(d')} }{2} (\zeta^{(d')})^2 \right) \cp{\omega^{(d')}}{b, 0}
\\ &=
\cp{w}{z, \beta} \cp{z}{\alpha} \cp{\eta}{\mu, \nu^2} \cPsi{y, \omega}{\eta, z, b}.
\end{align}$$

See my [LDA notebook](http://nbviewer.savvysherpa.com/github/bearnshaw/ml-demos/blob/master/lda_gibbs_sampling_cython.ipynb) for step-by-step details of the previous two calculations.

Our goal is to calculate the posterior distribution

$$\cp{z, \eta, \omega}{w, y, \alpha, \beta, \mu, \nu^2, b} =
\frac{\cp{z, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}}
{\sum_{z'} \iint \cp{z', w, \eta', y, \omega{'}}{\alpha, \beta, \mu, \nu^2, b} d\eta' d\omega{'}}$$

in order to infer the topic assignments $z$ and regression coefficients $\eta$ from the given term assignments $w$ and response data $y$.  Since calculating this directly is infeasible, we resort to collapsed Gibbs sampling.  The sampler is "collapsed" because we marginalized out $\theta$ and $\phi$, and will estimate them from the topic assignments $z$:

$$\hat\theta_k^{(d)} = \frac{a_k^{(d)} + \alpha_k}{\sum_{k'=1}^K \left(a_{k'}^{(d)} + \alpha_{k'} \right)},\quad
\hat\phi_v^{(k)} = \frac{b_v^{(k)} + \beta_v}{\sum_{v'=1}^V \left(b_{v'}^{(k)} + \beta_{v'} \right)}.$$

Gibbs sampling requires us to compute the full conditionals for each $z_n^{(d)}$, $\omega^{(d)}$ and $\eta_k$.  For example, we need to calculate, for all $n$, $d$ and $k$,

$$\begin{align}
\cp{z_n^{(d)} = k}{z \setminus z_n^{(d)}, w, \eta, y, \omega, \alpha, \beta, \mu, \nu^2, b}
&\propto
\cp{z_n^{(d)} = k, z \setminus z_n^{(d)}, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
\\ &\propto
\frac{b_{w_n^{(d)}}^{(k)} \setminus z_n^{(d)} + \beta_{w_n^{(d)}}}{ \sum_{v=1}^V \left( b_v^{(k)} \setminus z_n^{(d)} + \beta_v\right)}
\left( a_k^{(d)} \setminus z_n^{(d)} + \alpha_k \right)
\prod_{d'=1}^{D} \exp \left( \kappa^{(d')} \zeta^{(d')} - \frac{ \omega^{(d')} }{2} (\zeta^{(d')})^2 \right)
\\ &\propto
\frac{b_{w_n^{(d)}}^{(k)} \setminus z_n^{(d)} + \beta_{w_n^{(d)}}}{ \sum_{v=1}^V \left( b_v^{(k)} \setminus z_n^{(d)} + \beta_v\right)}
\left( a_k^{(d)} \setminus z_n^{(d)} + \alpha_k \right)
\\ &\quad \times \exp \left( \left( \kappa^{(d)} - \omega^{(d)} (\zeta^{(d)} \setminus z_n^{(d)}) \right) \frac{\eta_k}{N^{(d)}}
- \frac{\omega^{(d)}}{2} \left( \frac{\eta_k}{N^{(d)}} \right)^2 \right)
\end{align}$$

where the "set-minus" notation $\cdot \setminus z_n^{(d)}$ denotes the variable the notation is applied to with the entry $z_n^{(d)}$ removed (again, see my [LDA notebook](http://nbviewer.savvysherpa.com/github/bearnshaw/ml-demos/blob/master/lda_gibbs_sampling_cython.ipynb) for details).  This final proportionality is true since

$$\begin{align}
\prod_{d'=1}^{D} \exp \left( \kappa^{(d')} \zeta^{(d')} - \frac{ \omega^{(d')} }{2} (\zeta^{(d')})^2 \right)
&=
\prod_{d'=1}^{D} \exp \left( \kappa^{(d')} \left( \zeta^{(d')} \setminus z_n^{(d)} + \delta_{d, d'} \frac{\eta_k}{N^{(d)}} \right)
- \frac{ \omega^{(d')} }{2} \left( \zeta^{(d')} \setminus z_n^{(d)} + \delta_{d, d'} \frac{\eta_k}{N^{(d)}} \right)^2 \right)
\\ &\propto
\exp \left( \kappa^{(d)} \left( \zeta^{(d)} \setminus z_n^{(d)} + \frac{\eta_k}{N^{(d)}} \right)
- \frac{ \omega^{(d)} }{2} \left( \zeta^{(d)} \setminus z_n^{(d)} + \frac{\eta_k}{N^{(d)}} \right)^2 \right)
\\ &\propto
\exp \left( \left( \kappa^{(d)} - \omega^{(d)} (\zeta^{(d)} \setminus z_n^{(d)}) \right) \frac{\eta_k}{N^{(d)}}
- \frac{\omega^{(d)}}{2} \left( \frac{\eta_k}{N^{(d)}} \right)^2 \right)
\end{align}$$

where $\delta_{d, d'}$ is the [Kronecker delta](https://en.wikipedia.org/wiki/Kronecker_delta).

In order to calculate the full conditional for $\eta$, let $Z = (\etd{1} \cdots \etd{D})$ be the matrix whose $d$th column is $\etd{d}$, $\Omega = \diag(\omega^{(1)}, \ldots, \omega^{(D)})$ be the diagonal matrix whose $d$th diagonal entry is $\omega^{(d)}$, $I$ be the identity matrix, and $\one$ be the vector of ones, and note that

$$\prod_{k=1}^{K} \cN{\eta_{k}}{\mu, \nu^2} = \cN{\eta}{\mu \one, \nu^2 I}$$
$$\prod_{d=1}^{D} \exp \left( \kappa^{(d)} \zeta^{(d)} - \frac{ \omega^{(d)} }{2} (\zeta^{(d)})^2 \right)
= \exp \left( \eta^T Z \kappa - \frac{1}{2} \eta^T Z \Omega Z^T \eta \right)
\propto \cN{\eta}{(Z \Omega Z^T)^{-1} Z \kappa, (Z \Omega Z^T)^{-1}}.$$

Therefore

$$\begin{align}
\cp{\eta}{z, w, y, \omega, \alpha, \beta, \mu, \nu^2, b}
&\propto
\cp{z, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
\\ &\propto
\cN{\eta}{\mu \one, \nu^2 I} \cN{\eta}{(Z \Omega Z^T)^{-1} Z \kappa, (Z \Omega Z^T)^{-1}}
\\ &\propto
\cN{\eta}{\Sigma \left( \frac{\mu}{\nu^2} \one + Z \kappa \right), \Sigma}
\end{align}$$

where $\Sigma^{-1} = \nu^{-2} I + Z \Omega Z^T$ (see Section 8.1.8 of the [Matrix Cookbook](http://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)).

We also need to calculate the full conditional for $\omega$.  We calculate

$$\begin{align}
\cp{\omega}{z, w, \eta, y, \alpha, \beta, \mu, \nu^2, b}
&\propto

\cp{z, w, \eta, y, \omega}{\alpha, \beta, \mu, \nu^2, b}
\\ &\propto
\prod_{d=1}^D \exp \left( - \frac{ \omega^{(d)} }{2} (\zeta^{(d)})^2 \right) \cp{\omega^{(d)}}{b, 0}
\\ &=
\prod_{d=1}^D \cp{\omega^{(d)}}{b, \zeta^{(d)}}
\end{align}$$

that is, $\omega^{(d)} \sim \PG(b, \eta \cdot \etd{d})$ for each $d$.  We sample from the Polya-Gamma distribution according to the method of [Polson et al. 2012](http://arxiv.org/pdf/1205.0310.pdf), implemented for Python 3 in this [code repo](https://github.com/Savvysherpa/pypolyagamma).

## Graphical test

```python
%matplotlib inline

from modules.helpers import plot_images
from functools import partial
from sklearn.metrics import (roc_auc_score, roc_curve)
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np

imshow = partial(plt.imshow, cmap='gray', interpolation='nearest', aspect='auto')
sns.set(style='white')
```

### Generate topics

We assume a vocabulary of 25 terms, and create ten "topics", where each topic assigns exactly 5 consecutive terms equal probability.

```python
V = 25
K = 10
N = 100
D = 1000
```

```python
topics = []
topic_base = np.concatenate((np.ones((1, 5)) * 0.2, np.zeros((4, 5))), axis=0).ravel()
for i in range(5):
    topics.append(np.roll(topic_base, i * 5))
topic_base = np.concatenate((np.ones((5, 1)) * 0.2, np.zeros((5, 4))), axis=1).ravel()
for i in range(5):
    topics.append(np.roll(topic_base, i))
topics = np.array(topics)
plt.figure(figsize=(10, 5))
plot_images(plt, topics, (5, 5), layout=(2, 5), figsize=(10, 5))
```

### Generate documents from topics

We generate 1,000 documents from these 10 topics by sampling 1,000 topic distributions, one for each document, from a Dirichlet distribution with parameter $\alpha = (1, \ldots, 1)$.

```python
alpha = np.repeat(1., K)
np.random.seed(42)
thetas = np.random.dirichlet(alpha, size=D)
topic_assignments = np.array([np.random.choice(range(K), size=100, p=theta)
                              for theta in thetas])
word_assignments = np.array([[np.random.choice(range(V), size=1, p=topics[topic_assignments[d, n]])[0]
                              for n in range(N)] for d in range(D)])
doc_term_matrix = np.array([np.histogram(word_assignments[d], bins=V, range=(0, V - 1))[0] for d in range(D)])
imshow(doc_term_matrix)
```

### Generate responses

```python
# choose parameter values
mu = 0.
nu2 = 1.
np.random.seed(4)
eta = np.random.normal(loc=mu, scale=nu2, size=K)
print(eta)
# plot histogram of pre-responses
zeta = np.array([np.dot(eta, thetas[i]) for i in range(D)])
_ = plt.hist(zeta, bins=50)
```

```python
# choose parameter values
y = (zeta >= 0).astype(int)
# plot histogram of responses
print('positive examples {} ({:.1f}%)'.format(y.sum(), y.sum() / D * 100))
_ = plt.hist(y)
```

### Estimate parameters

```python
from slda.topic_models import BLSLDA
```

```python
_K = 10
_alpha = alpha
_beta = np.repeat(0.01, V)
_mu = mu
_nu2 = nu2
_b = 7.25
n_iter = 500
blslda = BLSLDA(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42)
```

```python
%%time
blslda.fit(doc_term_matrix, y)
```

    2016-06-16 13:43:44.065389 start iterations
    2016-06-16 13:43:44.376867 0:00:00.311478 elapsed, iter   10, LL -97726.0415, 37.08% change from last
    2016-06-16 13:43:44.652594 0:00:00.587205 elapsed, iter   20, LL -52290.2589, 46.49% change from last
    2016-06-16 13:43:44.945933 0:00:00.880544 elapsed, iter   30, LL -28731.1376, 45.05% change from last
    2016-06-16 13:43:45.244807 0:00:01.179418 elapsed, iter   40, LL -14008.1676, 51.24% change from last
    2016-06-16 13:43:45.515666 0:00:01.450277 elapsed, iter   50, LL -4592.6450, 67.21% change from last
    2016-06-16 13:43:45.780189 0:00:01.714800 elapsed, iter   60, LL 454.3363, 109.89% change from last
    2016-06-16 13:43:46.045411 0:00:01.980022 elapsed, iter   70, LL 3724.5545, 719.78% change from last
    2016-06-16 13:43:46.312595 0:00:02.247206 elapsed, iter   80, LL 5336.0692, 43.27% change from last
    2016-06-16 13:43:46.573980 0:00:02.508591 elapsed, iter   90, LL 6967.8111, 30.58% change from last
    2016-06-16 13:43:46.837874 0:00:02.772485 elapsed, iter  100, LL 11490.1673, 64.90% change from last
    2016-06-16 13:43:47.104985 0:00:03.039596 elapsed, iter  110, LL 14289.1681, 24.36% change from last
    2016-06-16 13:43:47.368892 0:00:03.303503 elapsed, iter  120, LL 16414.6325, 14.87% change from last
    2016-06-16 13:43:47.636476 0:00:03.571087 elapsed, iter  130, LL 19064.5871, 16.14% change from last
    2016-06-16 13:43:47.901682 0:00:03.836293 elapsed, iter  140, LL 20426.6905, 7.14% change from last
    2016-06-16 13:43:48.164834 0:00:04.099445 elapsed, iter  150, LL 21874.6680, 7.09% change from last
    2016-06-16 13:43:48.431032 0:00:04.365643 elapsed, iter  160, LL 22503.9176, 2.88% change from last
    2016-06-16 13:43:48.697741 0:00:04.632352 elapsed, iter  170, LL 23142.6131, 2.84% change from last
    2016-06-16 13:43:48.962917 0:00:04.897528 elapsed, iter  180, LL 23669.6861, 2.28% change from last
    2016-06-16 13:43:49.224741 0:00:05.159352 elapsed, iter  190, LL 23644.1627, -0.11% change from last
    2016-06-16 13:43:49.488091 0:00:05.422702 elapsed, iter  200, LL 24166.6925, 2.21% change from last
    2016-06-16 13:43:49.749972 0:00:05.684583 elapsed, iter  210, LL 23943.6736, -0.92% change from last
    2016-06-16 13:43:50.013902 0:00:05.948513 elapsed, iter  220, LL 23830.6447, -0.47% change from last
    2016-06-16 13:43:50.277361 0:00:06.211972 elapsed, iter  230, LL 24044.6142, 0.90% change from last
    2016-06-16 13:43:50.538762 0:00:06.473373 elapsed, iter  240, LL 24733.6069, 2.87% change from last
    2016-06-16 13:43:50.805555 0:00:06.740166 elapsed, iter  250, LL 24641.0416, -0.37% change from last
    2016-06-16 13:43:51.072573 0:00:07.007184 elapsed, iter  260, LL 24981.5228, 1.38% change from last
    2016-06-16 13:43:51.337118 0:00:07.271729 elapsed, iter  270, LL 25366.1493, 1.54% change from last
    2016-06-16 13:43:51.603452 0:00:07.538063 elapsed, iter  280, LL 24767.5741, -2.36% change from last
    2016-06-16 13:43:51.867107 0:00:07.801718 elapsed, iter  290, LL 25009.9327, 0.98% change from last
    2016-06-16 13:43:52.128730 0:00:08.063341 elapsed, iter  300, LL 25115.5275, 0.42% change from last
    2016-06-16 13:43:52.404560 0:00:08.339171 elapsed, iter  310, LL 24764.2753, -1.40% change from last
    2016-06-16 13:43:52.702307 0:00:08.636918 elapsed, iter  320, LL 25091.8952, 1.32% change from last
    2016-06-16 13:43:52.985886 0:00:08.920497 elapsed, iter  330, LL 25515.7933, 1.69% change from last
    2016-06-16 13:43:53.266369 0:00:09.200980 elapsed, iter  340, LL 25604.3514, 0.35% change from last
    2016-06-16 13:43:53.547447 0:00:09.482058 elapsed, iter  350, LL 25592.7337, -0.05% change from last
    2016-06-16 13:43:53.827423 0:00:09.762034 elapsed, iter  360, LL 25711.6631, 0.46% change from last
    2016-06-16 13:43:54.096971 0:00:10.031582 elapsed, iter  370, LL 25637.3562, -0.29% change from last
    2016-06-16 13:43:54.373274 0:00:10.307885 elapsed, iter  380, LL 25230.5460, -1.59% change from last
    2016-06-16 13:43:54.664293 0:00:10.598904 elapsed, iter  390, LL 25150.3624, -0.32% change from last
    2016-06-16 13:43:54.974061 0:00:10.908672 elapsed, iter  400, LL 25388.1922, 0.95% change from last
    2016-06-16 13:43:55.258373 0:00:11.192984 elapsed, iter  410, LL 25751.2618, 1.43% change from last
    2016-06-16 13:43:55.538874 0:00:11.473485 elapsed, iter  420, LL 26075.1021, 1.26% change from last
    2016-06-16 13:43:55.820955 0:00:11.755566 elapsed, iter  430, LL 25751.7570, -1.24% change from last
    2016-06-16 13:43:56.091240 0:00:12.025851 elapsed, iter  440, LL 26197.8723, 1.73% change from last
    2016-06-16 13:43:56.363768 0:00:12.298379 elapsed, iter  450, LL 26390.4878, 0.74% change from last
    2016-06-16 13:43:56.653482 0:00:12.588093 elapsed, iter  460, LL 25805.8916, -2.22% change from last
    2016-06-16 13:43:56.936218 0:00:12.870829 elapsed, iter  470, LL 26304.9692, 1.93% change from last
    2016-06-16 13:43:57.217246 0:00:13.151857 elapsed, iter  480, LL 26700.6544, 1.50% change from last
    2016-06-16 13:43:57.496533 0:00:13.431144 elapsed, iter  490, LL 26238.4650, -1.73% change from last
    CPU times: user 13.6 s, sys: 54.7 ms, total: 13.7 s
    Wall time: 13.7 s

```python
plot_images(plt, blslda.phi, (5, 5), (2, 5), figsize=(10, 5))
```

```python
topic_order = [1, 7, 0, 3, 6, 4, 9, 5, 2, 8]
plot_images(plt, blslda.phi[topic_order], (5, 5), (2, 5), figsize=(10, 5))
```

```python
imshow(blslda.theta)
```

```python
plt.plot(blslda.loglikelihoods)
```

```python
burn_in = 300 #max(n_iter - 100, int(n_iter / 2))
blslda.loglikelihoods[burn_in:].mean()
```

    25635.861229760965

```python
eta_pred = blslda.eta[burn_in:].mean(axis=0)
print(eta_pred)
```

    [-10.2046844   -0.32919142   1.7715393    7.69770083 -14.95961328
       5.38967261  -4.70075113   5.50790398 -13.1675921  -10.99316695]

```python
print(eta_pred[topic_order])
print(eta)
```

    [ -0.32919142   5.50790398 -10.2046844    7.69770083  -4.70075113
     -14.95961328 -10.99316695   5.38967261   1.7715393  -13.1675921 ]
    [ 0.05056171  0.49995133 -0.99590893  0.69359851 -0.41830152 -1.58457724
     -0.64770677  0.59857517  0.33225003 -1.14747663]

### Predict response of test documents

Create 1,000 test documents using the same generative process as our training documents, and compute their actual responses.

```python
np.random.seed(42^2)
thetas_test = np.random.dirichlet(alpha, size=D)
topic_assignments_test = np.array([np.random.choice(range(K), size=100, p=theta)
                                   for theta in thetas_test])
word_assignments_test = np.array([[np.random.choice(range(V), size=1, p=topics[topic_assignments_test[d, n]])[0]
                                   for n in range(N)] for d in range(D)])
doc_term_matrix_test = np.array([np.histogram(word_assignments_test[d], bins=V, range=(0, V - 1))[0] for d in range(D)])
y_test = np.array([np.dot(eta, thetas_test[i]) >= 0 for i in range(D)], dtype=int)
imshow(doc_term_matrix_test)
```

Estimate their topic distributions using the trained model, then calculate the predicted responses using the mean of our samples of $\eta$ after burn-in as an estimate for $\eta$.

```python
def bern_param(eta, theta):
    return np.exp(np.dot(eta, theta)) / (1 + np.exp(np.dot(eta, theta)))
```

```python
thetas_test_blslda = blslda.transform(doc_term_matrix_test)
y_blslda = [bern_param(eta_pred, thetas_test_blslda[i]) for i in range(D)]
_ = plt.hist(y_blslda, bins=30)
```

Measure the goodness of our prediction using [area under the ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic).

```python
fpr, tpr, _ = roc_curve(y_test, y_blslda)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_blslda))))
_ = plt.legend(loc='best')
```

### Find best b

```python
_b_grid = np.arange(1, 10, 1)
roc_auc_scores = []
for _b in _b_grid:
    print('Training BLSLDA with b = {}'.format(_b))
    _blslda = BLSLDA(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42, verbose=False)
    _blslda.fit(doc_term_matrix, y)
    _thetas_test_blslda = _blslda.transform(doc_term_matrix_test)
    _eta_pred = _blslda.eta[burn_in:].mean(axis=0)
    _y_blslda = [bern_param(_eta_pred, _thetas_test_blslda[i]) for i in range(D)]
    roc_auc_scores.append(roc_auc_score(y_test, _y_blslda))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_b_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_b_grid, roc_auc_scores)
```

```python
_b_grid = np.arange(7, 8, .1)
roc_auc_scores = []
for _b in _b_grid:
    print('Training BLSLDA with b = {}'.format(_b))
    _blslda = BLSLDA(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42, verbose=False)
    _blslda.fit(doc_term_matrix, y)
    _thetas_test_blslda = _blslda.transform(doc_term_matrix_test)
    _eta_pred = _blslda.eta[burn_in:].mean(axis=0)
    _y_blslda = [bern_param(_eta_pred, _thetas_test_blslda[i]) for i in range(D)]
    roc_auc_scores.append(roc_auc_score(y_test, _y_blslda))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_b_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_b_grid, roc_auc_scores)
```

### Two-step learning: learn topics, then learn regression

```python
from slda.topic_models import LDA
```

```python
lda = LDA(_K, _alpha, _beta, n_iter, seed=42)
```

```python
%%time
lda.fit(doc_term_matrix)
```

```python
plot_images(plt, lda.phi, (5, 5), (2, 5), figsize=(10, 5))
```

```python
imshow(lda.theta)
```

```python
plt.plot(lda.loglikelihoods)
```

```python
thetas_test_lda = lda.transform(doc_term_matrix_test)
```

#### L2-regularized logistic regression

- train logistic regression on training data
- calculate response on test data
- measure the goodness of prediction using area under the ROC curve

```python
from sklearn.linear_model import LogisticRegression
```

```python
_C_grid = np.arange(0.001, 100, 1)
roc_auc_scores = []
for _C in _C_grid:
    print('Training Logistic Regression with C = {}'.format(_C))
    _lr = LogisticRegression(fit_intercept=False, C=_C)
    _lr.fit(lda.theta, y)
    _y_lr = _lr.predict_proba(thetas_test_lda)[:, 1]
    roc_auc_scores.append(roc_auc_score(y_test, _y_lr))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_C_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_C_grid, roc_auc_scores)
```

```python
lr = LogisticRegression(fit_intercept=False, C=39)
lr.fit(lda.theta, y)
y_lr = lr.predict_proba(thetas_test_lda)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_lr)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_lr))))
_ = plt.legend(loc='best')
```

#### Gradient boosted trees

- train gradient boosted trees on training data
- calculate response on test data
- measure the goodness of prediction using area under the ROC

```python
from sklearn.ensemble import GradientBoostingClassifier
```

```python
gbr = GradientBoostingClassifier()
gbr.fit(lda.theta, y)
y_gbr = gbr.predict_proba(thetas_test_lda)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_gbr)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_gbr))))
_ = plt.legend(loc='best')
```

### Conclusion

The best BLSLDA is slightly better than the best logistic regression, and better than gradient boosted regression trees.  The similar performance to logistic regression is likely due to the fact that this test was set up as an exact problem - all parameters used in training, except $\beta$ (because we hand-picked the topics) and $\eta$ (because that was the one parameter we wanted to learn), were those used to generate documents.

## Test with fewer topics

We now redo the previous test, but this time with fewer topics, in order to determine whether

1. the supervised portion of BLSLDA will produce topics different from those produced by LDA, and
1. prediction with SLDA is improved over LDA-and-a-regression.

Because we will no longer be solving an exact problem (the number of topics, and hence $\alpha$, will both be different from the document generation process), we expect BLSLDA to do better than LDA-and-a-regression, including unregularized linear regression.

```python
_K = 5
_alpha = np.repeat(1., _K)
```

```python
_b_grid = np.arange(1, 10, 1)
roc_auc_scores = []
for _b in _b_grid:
    print('Training BLSLDA with b = {}'.format(_b))
    _blslda = BLSLDA(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42, verbose=False)
    _blslda.fit(doc_term_matrix, y)
    _thetas_test_blslda = _blslda.transform(doc_term_matrix_test)
    _eta_pred = _blslda.eta[burn_in:].mean(axis=0)
    _y_blslda = [bern_param(_eta_pred, _thetas_test_blslda[i]) for i in range(D)]
    roc_auc_scores.append(roc_auc_score(y_test, _y_blslda))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_b_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_b_grid, roc_auc_scores)
```

```python
_b_grid = np.arange(9, 11, 0.2)
roc_auc_scores = []
for _b in _b_grid:
    print('Training BLSLDA with b = {}'.format(_b))
    _blslda = BLSLDA(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42, verbose=False)
    _blslda.fit(doc_term_matrix, y)
    _thetas_test_blslda = _blslda.transform(doc_term_matrix_test)
    _eta_pred = _blslda.eta[burn_in:].mean(axis=0)
    _y_blslda = [bern_param(_eta_pred, _thetas_test_blslda[i]) for i in range(D)]
    roc_auc_scores.append(roc_auc_score(y_test, _y_blslda))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_b_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_b_grid, roc_auc_scores)
```

```python
_b = 9
n_iter = 1000
blslda1 = BLSLDA(_K, _alpha, _beta, _mu, _nu2, _b, n_iter, seed=42)
```

```python
%%time
blslda1.fit(doc_term_matrix, y)
```

```python
plot_images(plt, blslda1.phi, (5, 5), (1, 5), figsize=(10, 5))
```

```python
imshow(blslda1.theta)
```

```python
plt.plot(blslda1.loglikelihoods)
```

```python
burn_in1 = 600
blslda1.loglikelihoods[burn_in1:].mean()
```

```python
eta_pred1 = blslda1.eta[burn_in1:].mean(axis=0)
eta_pred1
```

```python
thetas_test_blslda1 = blslda1.transform(doc_term_matrix_test)
y_blslda1 = [bern_param(eta_pred1, thetas_test_blslda1[i]) for i in range(D)]
```

```python
fpr, tpr, _ = roc_curve(y_test, y_blslda1)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_blslda1))))
_ = plt.legend(loc='best')
```

```python
lda1 = LDA(_K, _alpha, _beta, n_iter, seed=42)
```

```python
%%time
lda1.fit(doc_term_matrix)
```

```python
plot_images(plt, lda1.phi, (5, 5), (1, 5), figsize=(10, 5))
```

We plot the SLDA topics again and note that they are indeed different!

```python
plot_images(plt, blslda1.phi, (5, 5), (1, 5), figsize=(10, 5))
```

```python
imshow(lda1.theta)
```

```python
plt.plot(lda1.loglikelihoods)
```

```python
thetas_test_lda1 = lda1.transform(doc_term_matrix_test)
```

#### L2-regularized linear regression

```python
_C_grid = np.arange(0.001, 100, 1)
roc_auc_scores = []
for _C in _C_grid:
    print('Training Logistic Regression with C = {}'.format(_C))
    _lr = LogisticRegression(fit_intercept=False, C=_C)
    _lr.fit(lda1.theta, y)
    _y_lr = _lr.predict_proba(thetas_test_lda1)[:, 1]
    roc_auc_scores.append(roc_auc_score(y_test, _y_lr))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_C_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_C_grid, roc_auc_scores)
```

```python
lr1 = LogisticRegression(fit_intercept=False, C=93)
lr1.fit(lda1.theta, y)
y_lr1 = lr1.predict_proba(thetas_test_lda1)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_lr1)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_lr1))))
_ = plt.legend(loc='best')
```

#### Gradient boosted trees

```python
gbr1 = GradientBoostingClassifier()
gbr1.fit(lda1.theta, y)
y_gbr1 = gbr1.predict_proba(thetas_test_lda1)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_gbr1)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_gbr1))))
_ = plt.legend(loc='best')
```

#### L2-regularized logistic regression with SLDA topics

```python
_C_grid = np.arange(0.001, 100, 1)
roc_auc_scores = []
for _C in _C_grid:
    print('Training Logistic Regression with C = {}'.format(_C))
    _lr = LogisticRegression(fit_intercept=False, C=_C)
    _lr.fit(blslda1.theta, y)
    _y_lr = _lr.predict_proba(thetas_test_blslda1)[:, 1]
    roc_auc_scores.append(roc_auc_score(y_test, _y_lr))
    print('  roc_auc_score = {}'.format(roc_auc_scores[-1]))
print(_C_grid[np.argmax(roc_auc_scores)], np.max(roc_auc_scores))
plt.plot(_C_grid, roc_auc_scores)
```

```python
lr1_0 = LogisticRegression(fit_intercept=False, C=34)
lr1_0.fit(blslda1.theta, y)
y_lr1_0 = lr1_0.predict_proba(thetas_test_blslda1)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_lr1_0)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_lr1_0))))
_ = plt.legend(loc='best')
```

#### Gradient boosted trees with SLDA topics

```python
gbr1_0 = GradientBoostingClassifier()
gbr1_0.fit(blslda1.theta, y)
y_gbr1_0 = gbr1_0.predict_proba(thetas_test_blslda1)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_gbr1_0)
plt.plot(fpr, tpr, label=('AUC = {:.3f}'.format(roc_auc_score(y_test, y_gbr1_0))))
_ = plt.legend(loc='best')
```

### Conclusion for test with fewer topics

BLSLDA performs at least 20% better in AUC than any of the LDA-and-a-regression methods, and similar to the BLSLDA-and-a-logistic-regression methods.