#SHDOM single-scattering adjoint

This folder conatains the most recent (3 May 2015) single-scattering adjoint calculations from frank.

>This has the log output files (.log), radiance output files (.arad),
and optical property adjoint files (.adj) for the "true" and "guess"
cases. The first set of output is for the single scattering forward radiance calculations

    >>les0822nh15t13y135_ocaer1_w0.646_ns1true.*,

    >>les0822nh15t13y135_ocaer1_w0.646_ns1guess.*,

>and the second set of outputs is for the full multiple scattering forward radiance calculations,

    >>les0822nh15t13y135_ocaer1_w0.646_ns1true2.*,

    >>les0822nh15t13y135_ocaer1_w0.646_ns1guess2.*.

The cloud field used in these simulations is a 2D slice taken at $y_{(i_y=135)}=8.375 \mathrm{km}$ of the 320x320 high resolution simulation shown below.

The adjoint calculation is for the gradient of the misfit function,

\begin{align}
\frac{\partial \Phi(\sigma, \omega)}{\partial(\sigma)} &= \left< \Delta p, \mathcal{U}_{\sigma}[\Delta f] \right>
\end{align}

Only the first order of scattering completed so the results are approximate.  In the calculation labelled "1" the forward and adjoint calculations are only single-scattering.  In the calculation labelled "2", the Forward calculation includes multiple scatterng while the adjoint does not.

```python
%pylab inline

# Imports
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm

import pandas as pd

```

    Populating the interactive namespace from numpy and matplotlib

    WARNING: pylab import has clobbered these variables: ['fmax']
    `%matplotlib` prevents importing * from pylab and numpy

```python

```

```python
# Utility functions

def print_head(fname, Nlines=10, indent="\t"):
    "Print the head of the file."

    # Print a message and then the first N lines
    print("Showing head: {}".format(fname))
    for i, line in zip(range(Nlines), open(fname, 'r')):
        print(indent + line.strip())

    print('\n')

def read_log(flog, start_line="X      Z   Extinct", max_lines=10**8):
    """
    Read in a log file but save lines only after reaching a line
    matching start_line.
    """

    # Loop over lines record after header and before footer
    ncols = None
    lines = []
    for i, line in zip(range(max_lines), open(flog, 'r')):
        new = line.strip().split()

        # Check if the line is the header
        if ncols is None:
            if line.strip()[:len(start_line)]==start_line:
                ncols = len(new[:9])
                cols = new[:9]
        else:
            if ncols == len(new[:9]):
                lines.append(new[:9])
            else:
                print(new)
                break
    # Raise an error if the file had no line with the specified columns
    if ncols is None:
        raise ValueError("No column line found containing {}".format(start_line))

    # Make dictionary of arrays from keys array with given lines
    N = len(lines) - 1
    value_array = np.array(lines[:], dtype='f8')

    log_dict = dict(zip(cols, list(value_array.T)))
    return log_dict

```

```python
ls
```

    exploring_adjoint.ipynb
    exploring_adjoint_with_aerosol.ipynb
    les0822nh15t13y135_ocaer1_w0.646_ns1guess2.adj
    les0822nh15t13y135_ocaer1_w0.646_ns1guess2.arad
    les0822nh15t13y135_ocaer1_w0.646_ns1guess2.log
    les0822nh15t13y135_ocaer1_w0.646_ns1true2.adj
    les0822nh15t13y135_ocaer1_w0.646_ns1true2.arad
    les0822nh15t13y135_ocaer1_w0.646_ns1true2.log
    les0822nh15t13y135_w0.646_ns1guess2.adj
    les0822nh15t13y135_w0.646_ns1guess2.arad
    les0822nh15t13y135_w0.646_ns1guess2.log
    les0822nh15t13y135_w0.646_ns1true2.adj
    les0822nh15t13y135_w0.646_ns1true2.arad
    les0822nh15t13y135_w0.646_ns1true2.log

# Looking at the data files

```python
# The log file
flog = "les0822nh15t13y135_ocaer1_w0.646_ns1true2.log"
print_head(flog, Nlines=10)

# The radiance file
frad = "les0822nh15t13y135_ocaer1_w0.646_ns1true2.arad"
print_head(frad, Nlines=25)

# The adjoint file
fadj_true = "les0822nh15t13y135_ocaer1_w0.646_ns1true2.adj"
fadj = "les0822nh15t13y135_ocaer1_w0.646_ns1guess2.adj"

print_head(fadj, Nlines=5)

# The adjoint Radiance file
farad = "les0822nh15t13y135_ocaer1_w0.646_ns1guess2.arad"
farad_true = "les0822nh15t13y135_ocaer1_w0.646_ns1true2.arad"
print_head(farad, Nlines=35)
```

    Showing head: les0822nh15t13y135_ocaer1_w0.646_ns1true2.log

    	Name of this SHDOMADJ run:
    	les0822nh15t13y135_ocaer1_w0.646_ns1true2
    	Input particle properties filename (or NONE for Rayleigh only)
    	les0822nh15t13y135_ocaer1t.part
    	Number of input scattering tables
    	2
    	Name of each scattering table
    	water_w0.646.scat
    	aero_oc_w0.646.scat

    Showing head: les0822nh15t13y135_ocaer1_w0.646_ns1true2.arad
    	! Polarized Spherical Harmonic Discrete Ordinate Radiative Transfer Radiance Output
    	!  L= 15  M= 15  NLM=  256   NMU= 16  NPHI= 32  NANG=  354   NSH=   3059599
    	!  NSTOKES=1   NX= 320   NY=   1   NZ=  86    NPTS=   36912   NCELLS=   42822
    	!  PARTICLE_FILE=les0822nh15t13y135_ocaer1t.part
    	!  CORRELATED_K-DIST_FILE=NONE   NUM_G= 1
    	!  SOURCE_TYPE=SOLAR               DELTA-M METHOD
    	!  GRID_TYPE=EVEN (X,Y)    PROPERTY-FILE (Z)    INDEPENDENT_PIXEL=2
    	!  SURFACE_TYPE=FIXED LAMBERTIAN         HORIZ_BOUNDARY_COND=0
    	!  GROUND_ALBEDO=0.0500000  SKY_RAD= 0.00000E+00
    	!  SOLAR_FLUX= 0.100000E+01   SOLAR_MU=-0.9221000   SOLAR_AZ= 339.400
    	!  UNITS=WATTS/(M^2 MICRON STER)    WAVELENGTH=      0.65
    	!  SPLITTING_ACCURACY= 0.300E-01   SPHERICAL_HARMONIC_ACCURACY= 0.300E-02
    	!  SOLUTION_ACCURACY= 0.300E-04
    	!  MAXIMUM_ITERATIONS= 200   NUMBER_ITERATIONS= 131
    	!  RADIANCE AT Z= 15.000
    	!  320    1    1 ! NXOUT, NYOUT, NRADDIR
    	!  ID      mu       phi
    	!   1  1.0000000    0.00
    	!
    	ID  IY  IX    RADIANCE (STOKES I, Q, U, V)
    	1   1   1   0.204325E+00
    	1   1   2   0.196358E+00
    	1   1   3   0.174226E+00
    	1   1   4   0.193639E+00
    	1   1   5   0.205518E+00

    Showing head: les0822nh15t13y135_ocaer1_w0.646_ns1guess2.adj
    	! SHDOMADJ optical property adjoint output
    	!
    	X       Y       Z      Extinct  SSalbedo  dMF/dExt  dMF/dSSalb  dMF/dP(Theta_j)
    	0.0000  0.0000 -0.0087    0.07041 0.970492  0.000E+00  0.000E+00  0.000E+00
    	0.0000  0.0000  0.0095    0.07018 0.970491  0.000E+00  0.000E+00  0.000E+00

    Showing head: les0822nh15t13y135_ocaer1_w0.646_ns1guess2.arad
    	! Polarized Spherical Harmonic Discrete Ordinate Radiative Transfer Radiance Output
    	!  L= 15  M= 15  NLM=  256   NMU= 16  NPHI= 32  NANG=  354   NSH=   2981886
    	!  NSTOKES=1   NX= 320   NY=   1   NZ=  86    NPTS=   36093   NCELLS=   41398
    	!  PARTICLE_FILE=les0822nh15t13y135_ocaer1g.part
    	!  CORRELATED_K-DIST_FILE=NONE   NUM_G= 1
    	!  SOURCE_TYPE=SOLAR               DELTA-M METHOD
    	!  GRID_TYPE=EVEN (X,Y)    PROPERTY-FILE (Z)    INDEPENDENT_PIXEL=2
    	!  SURFACE_TYPE=FIXED LAMBERTIAN         HORIZ_BOUNDARY_COND=0
    	!  GROUND_ALBEDO=0.0500000  SKY_RAD= 0.00000E+00
    	!  SOLAR_FLUX= 0.100000E+01   SOLAR_MU=-0.9221000   SOLAR_AZ= 339.400
    	!  UNITS=WATTS/(M^2 MICRON STER)    WAVELENGTH=      0.65
    	!  SPLITTING_ACCURACY= 0.300E-01   SPHERICAL_HARMONIC_ACCURACY= 0.300E-02
    	!  SOLUTION_ACCURACY= 0.300E-04
    	!  MAXIMUM_ITERATIONS= 200   NUMBER_ITERATIONS= 140
    	!  RADIANCE AT Z= 15.000
    	!  320    1    1 ! NXOUT, NYOUT, NRADDIR
    	!  ID      mu       phi
    	!   1  1.0000000    0.00
    	!
    	ID  IY  IX    RADIANCE (STOKES I, Q, U, V)
    	1   1   1   0.196312E+00
    	1   1   2   0.188236E+00
    	1   1   3   0.167071E+00
    	1   1   4   0.184278E+00
    	1   1   5   0.195281E+00
    	1   1   6   0.198276E+00
    	1   1   7   0.190030E+00
    	1   1   8   0.169683E+00
    	1   1   9   0.137259E+00
    	1   1  10   0.113736E+00
    	1   1  11   0.767046E-01
    	1   1  12   0.606018E-01
    	1   1  13   0.578038E-01
    	1   1  14   0.571656E-01
    	1   1  15   0.106376E+00

```python
pd.read_csv?
```

```python
# Load the adjoint files into memory
adj_frame = pd.read_csv(fadj, quoting=3, delim_whitespace=True, skiprows=2)
adj_true_frame = pd.read_csv(fadj_true, quoting=3, delim_whitespace=True, skiprows=2)
adj_arad_frame = pd.read_csv(farad, quoting=3, delim_whitespace=True, skiprows=19)
adj_arad_true_frame = pd.read_csv(farad_true, quoting=3, delim_whitespace=True, skiprows=19)

# Get variables from the array
nx = adj_frame['X'].unique().size
nz = adj_frame["Z"].unique().size

# Get the guess case adjoint field properties
adj_x = np.array(adj_frame["X"], dtype='f8').reshape(nx, nz)
adj_z = np.array(adj_frame["Z"], dtype='f8').reshape(nx, nz)
adj_ext = np.array(adj_frame["Extinct"], dtype='f8').reshape(nx, nz)
adj_alb = np.array(adj_frame["SSalbedo"], dtype='f8').reshape(nx, nz)
adj_dMFdext = np.array(adj_frame["dMF/dExt"], dtype='f8').reshape(nx, nz)
adj_dMFdalb = np.array(adj_frame["dMF/dSSalb"], dtype='f8').reshape(nx, nz)

# Get the True case adjoint extinction field
adj_ext_true = np.array(adj_true_frame["Extinct"], dtype='f8').reshape(nx, nz)

# Define the adjoint radiance parameters
adj_arad = np.array(adj_arad_frame['RADIANCE'], dtype='f8').reshape(nx)
adj_arad_true = np.array(adj_arad_true_frame['RADIANCE'], dtype='f8').reshape(nx)

```

```python
# Plot uniformity parameters
H = 15.5   # top of the atmosphere plots

mask_clouds = (adj_ext>= .75e-5) #* (adj_z <= 2.83)
cloud_outline = (adj_ext>= .75e-1) #* (adj_z <= 2.83)

# Scaling of the verticle coordinate

Zscale = adj_z[0,:]
DZscale = 0 * Zscale
DZscale_other = 0 * Zscale
DZscale[:-1] += 1 / (Zscale[1:] - Zscale[:-1])
DZscale_other[1:] += 1 / (Zscale[1:] - Zscale[:-1])

alpha = .99725
DZscale = alpha * DZscale + (1-alpha) * DZscale_other

# Compute the residual
adj_residual = adj_arad_true - adj_arad

#mask_clouds = adj_ext_true >= 1e-5

```

```python
# Make a figure to plot the true extinction field
f0 = plt.figure(0, (15,8), facecolor='white')
ax0 = f0.add_axes([.1, .1, .8, .8])
contour0 = ax0.contourf(adj_x, adj_z, adj_ext_true, 100, cmap=cm.gray)
ax0.set_ybound((0,8))
ax0.set_title(" Guess for the extinction field at y=8.35")
f0.colorbar(contour0, ax=ax0)
```

```python
# Make a figure to plot the true extinction field
f0 = plt.figure(0, (15,8), facecolor='white')
ax0 = f0.add_axes([.1, .1, .8, .8])

fvals = adj_ext_true - adj_ext

contour0 = ax0.contourf(adj_x, adj_z, fvals, 100, cmap=cm.gray)
ax0.set_ybound((0,8))
ax0.set_title("Difference $\sigma_{\mathrm{true}} - \sigma_{\mathrm{guess}}$")
f0.colorbar(contour0, ax=ax0)
```

```python

```

```python
# Make a figure to plot the true single-scattering albedo field
f1 = plt.figure(1, (15,8), facecolor='white')
ax1 = f1.add_axes([.1, .1, .8, .8])

levels = np.linspace(.999995, 1, 100)
contour1 = ax1.contourf(adj_x, adj_z, adj_alb, levels=levels, cmap=cm.gray)
ax1.set_ybound((0,13))
ax1.set_title("SSAlbedo field at y=8.35")
f1.colorbar(contour1, ax=ax1)

```

```python
# Make a figure to plot the true single-scattering albedo field
f2 = plt.figure(1, (13,6), facecolor='white')
ax2 = f2.add_axes([.1, .1, .8, .8])

fvals = adj_dMFdext * mask_clouds * DZscale #* .1 * adj_ext
fmax = abs(fvals).max()
levels = np.linspace(-fmax, fmax, 100)
contour2 = ax2.contourf(adj_x, adj_z, fvals, levels=levels, cmap=cm.RdBu_r) #levels=linspace(-16, 16, 100),
contour2a = ax2.contour(adj_x, adj_z, cloud_outline, 1, cmap=cm.gray) #levels=linspace(-16, 16, 100),

# Add the residual to the plot
ftrans = lambda y: 100 * y + .8*H
plot_overlay_origin = ftrans(0 * adj_residual)
plot_overlay_arad = ftrans(adj_residual)
plot_arad = ax2.fill_between(adj_x[:,0], plot_overlay_origin, plot_overlay_arad,
                             where=(plot_overlay_origin<plot_overlay_arad),
                             color='b')#, linewidth=3)
plot_arad = ax2.fill_between(adj_x[:,0], plot_overlay_origin, plot_overlay_arad,
                             where=(plot_overlay_origin>plot_overlay_arad),
                             color='r')#, linewidth=3)
plot_origin = ax2.plot(adj_x[:,0], plot_overlay_origin, 'k,', linewidth=1)
ax2.set_ybound((0,H))
ax2.set_title("dMF/dExt * DZscale; GUESS y=8.35")
cb = f2.colorbar(contour2, ax=ax2)
#cb.set_ticks([-2, -1, 0, 1, 2])
#cb.set_ticklabels(['-2', '-1', '0', '1', '2'])
```

```python
plot(Zscale, DZscale)
```

```python
# Make a figure to plot the true single-scattering albedo field
f3 = plt.figure(1, (15,8), facecolor='white')
ax3 = f3.add_axes([.1, .1, .8, .8])

# plot parameters
fvals = adj_dMFdalb * mask_clouds * DZscale # * .1 * adj_ext
fmax = abs(fvals).max()
levels = np.linspace(-fmax, fmax, 100)

#levels = np.linspace(-.05, .05, 100)

#fvals = adj_dMFdalb * mask_clouds * DZscale

contour3 = ax3.contourf(adj_x, adj_z, fvals, levels=levels, cmap=cm.RdBu_r)
contour3a = ax3.contour(adj_x, adj_z, cloud_outline, 1, cmap=cm.gray) #levels=linspace(-16, 16, 100),

# Add the residual to the plot
ftrans = lambda y: 2. * y + .8*H
plot_overlay_origin = ftrans(0 * adj_arad)
plot_overlay_arad = ftrans(adj_arad)

plot_arad = ax3.plot(adj_x[:,0], plot_overlay_arad, 'r', linewidth=3)
plot_origin = ax3.plot(adj_x[:,0], plot_overlay_origin, 'k,', linewidth=1)

ax3.set_ybound((0,H))
ax3.set_title("dMF/dalb (GUESS) field at y=8.35")

f3.colorbar(contour3, ax=ax3)
```

```python
# Plot the Adjoint radiance at the top of the atmosphere
f4 = plt.figure(4, (15,8), facecolor='white')
ax4 = f4.add_axes([.1, .1, .8, .8])
ax4.plot(np.unique(adj_x), adj_arad, 'k', linewidth=4)
ax4.set_xbound(0,25.5)
#ax4.contourf(adj_x, adj_z, adj_arad, 100, cmap=cm.RdBu_r)
print(adj_x.shape, adj_z.shape, adj_arad.shape)

```

```python
# Read the adjoint file

cols = ['X', 'Y', 'Z', 'ADEXTINCT', "ADALBEDO"]
value_list = []
for line in open(fadj):
    if line[0] =='!':
        pass
    else:
        new = line.strip().split()
        value_list.append(new[:len(cols)])

values = np.array(value_list, dtype="f8").T
adjoint_dict = dict(zip(cols, values))

```

```python
# Get values to plot
nx = np.unique(adjoint_dict['X']).size
nz = np.unique(adjoint_dict['Z']).size
x = adjoint_dict['X'].reshape(nx, nz)
z = adjoint_dict['Z'].reshape(nx, nz)
adj_ext = adjoint_dict['ADEXTINCT'].reshape(nx, nz)
adj_alb = adjoint_dict['ADALBEDO'].reshape(nx, nz)

print("maxes: {}, {}.".format(abs(adj_ext).max(),abs(adj_alb).max()))

```

    maxes: 16.91, 0.501.

```python
#plot the adjoint of the extinction
f1 = plt.figure(1, (15, 8), facecolor='white')
ax1 = f1.add_axes([.1, .1, .8, .8])
contour1 =ax1.contourf(x, z, adj_alb, levels=np.linspace(0, .6,50), cmap=cm.gray)
ax1.set_ybound(0,5)
f1.colorbar(contour1, ax=ax1)

ax1.set_title("Single-scattering adjoint derivative w.r.t. albedo", fontsize="x-large", fontweight="bold")
```

```python
# The first adjoint radiance field
farad1 = "radobs_les0822nh15t13y89_ocaer1_w0.646_ns1ss1.arad"
print_head(farad1, Nlines=100)

```

    Showing head: radobs_les0822nh15t13y89_ocaer1_w0.646_ns1ss1.arad
    	! Polarized Spherical Harmonic Discrete Ordinate Radiative Transfer Radiance Output
    	!  L= 15  M= 15  NLM=  256   NMU= 16  NPHI= 32  NANG=  354   NSH=   1757885
    	!  NSTOKES=1   NX= 320   NY=   1   NZ=  85    NPTS=   27285   NCELLS=   26880
    	!  PARTICLE_FILE=les0822nh15t13y89_ocaer1.part
    	!  CORRELATED_K-DIST_FILE=NONE   NUM_G= 1
    	!  SOURCE_TYPE=SOLAR               DELTA-M METHOD
    	!  GRID_TYPE=EVEN (X,Y)    PROPERTY-FILE (Z)    INDEPENDENT_PIXEL=2
    	!  SURFACE_TYPE=FIXED LAMBERTIAN         HORIZ_BOUNDARY_COND=0
    	!  GROUND_ALBEDO=0.0500000  SKY_RAD= 0.00000E+00
    	!  SOLAR_FLUX= 0.100000E+01   SOLAR_MU=-0.9221000   SOLAR_AZ= 339.400
    	!  UNITS=WATTS/(M^2 MICRON STER)    WAVELENGTH=      0.65
    	!  SPLITTING_ACCURACY= 0.300E-01   SPHERICAL_HARMONIC_ACCURACY= 0.300E-02
    	!  SOLUTION_ACCURACY= 0.300E-04
    	!  MAXIMUM_ITERATIONS=   0   NUMBER_ITERATIONS=   0
    	!  RADIANCE AT Z= 15.000
    	320    1    3 ! NXOUT, NYOUT, NRADDIR
    	!  ID      mu       phi
    	!   1  0.8660300  180.00
    	!   2  1.0000000    0.00
    	!   3  0.8660300    0.00
    	! ID  IY  IX    RADIANCE (STOKES I, Q, U, V)
    	1   1   1   0.134613E-01
    	1   1   2   0.115650E-01
    	1   1   3   0.125559E-01
    	1   1   4   0.125286E-01
    	1   1   5   0.121647E-01
    	1   1   6   0.117333E-01
    	1   1   7   0.124280E-01
    	1   1   8   0.115808E-01
    	1   1   9   0.988294E-02
    	1   1  10   0.123670E-01
    	1   1  11   0.118903E-01
    	1   1  12   0.117026E-01
    	1   1  13   0.114063E-01
    	1   1  14   0.110016E-01
    	1   1  15   0.109395E-01
    	1   1  16   0.111800E-01
    	1   1  17   0.119092E-01
    	1   1  18   0.124901E-01
    	1   1  19   0.131205E-01
    	1   1  20   0.126538E-01
    	1   1  21   0.138799E-01
    	1   1  22   0.130093E-01
    	1   1  23   0.133729E-01
    	1   1  24   0.134214E-01
    	1   1  25   0.127005E-01
    	1   1  26   0.161253E-01
    	1   1  27   0.191886E-01
    	1   1  28   0.176198E-01
    	1   1  29   0.179799E-01
    	1   1  30   0.172136E-01
    	1   1  31   0.174579E-01
    	1   1  32   0.181361E-01
    	1   1  33   0.185459E-01
    	1   1  34   0.184355E-01
    	1   1  35   0.184394E-01
    	1   1  36   0.180923E-01
    	1   1  37   0.178597E-01
    	1   1  38   0.178598E-01
    	1   1  39   0.178598E-01
    	1   1  40   0.178598E-01
    	1   1  41   0.178598E-01
    	1   1  42   0.178598E-01
    	1   1  43   0.178598E-01
    	1   1  44   0.178598E-01
    	1   1  45   0.178598E-01
    	1   1  46   0.178598E-01
    	1   1  47   0.178598E-01
    	1   1  48   0.178598E-01
    	1   1  49   0.178598E-01
    	1   1  50   0.178598E-01
    	1   1  51   0.178598E-01
    	1   1  52   0.178598E-01
    	1   1  53   0.178598E-01
    	1   1  54   0.178598E-01
    	1   1  55   0.178598E-01
    	1   1  56   0.178598E-01
    	1   1  57   0.178598E-01
    	1   1  58   0.178598E-01
    	1   1  59   0.178598E-01
    	1   1  60   0.178598E-01
    	1   1  61   0.178598E-01
    	1   1  62   0.178598E-01
    	1   1  63   0.178598E-01
    	1   1  64   0.178598E-01
    	1   1  65   0.178598E-01
    	1   1  66   0.178598E-01
    	1   1  67   0.178598E-01
    	1   1  68   0.174253E-01
    	1   1  69   0.151741E-01
    	1   1  70   0.132923E-01
    	1   1  71   0.148204E-01
    	1   1  72   0.169102E-01
    	1   1  73   0.174304E-01
    	1   1  74   0.174491E-01
    	1   1  75   0.174725E-01
    	1   1  76   0.184029E-01
    	1   1  77   0.181126E-01
    	1   1  78   0.179639E-01
    	1   1  79   0.178554E-01

```python
# The second adjoint radiance field
farad2 = "radobs_les0822nh15t13y89_ocaer1_w0.646_ns1ss2.arad"
print_head(farad2)

```

    Showing head: radobs_les0822nh15t13y89_ocaer1_w0.646_ns1ss2.arad
    	! Polarized Spherical Harmonic Discrete Ordinate Radiative Transfer Radiance Output
    	!  L= 15  M= 15  NLM=  256   NMU= 16  NPHI= 32  NANG=  354   NSH=   2422485
    	!  NSTOKES=1   NX= 320   NY=   1   NZ=  85    NPTS=   30052   NCELLS=   31290
    	!  PARTICLE_FILE=les0822nh15t13y89_ocaer1.part
    	!  CORRELATED_K-DIST_FILE=NONE   NUM_G= 1
    	!  SOURCE_TYPE=SOLAR               DELTA-M METHOD
    	!  GRID_TYPE=EVEN (X,Y)    PROPERTY-FILE (Z)    INDEPENDENT_PIXEL=2
    	!  SURFACE_TYPE=FIXED LAMBERTIAN         HORIZ_BOUNDARY_COND=0
    	!  GROUND_ALBEDO=0.0500000  SKY_RAD= 0.00000E+00
    	!  SOLAR_FLUX= 0.100000E+01   SOLAR_MU=-0.9221000   SOLAR_AZ= 339.400

```python
# A sample modis file
fmodis_blue = "les0822nh15t13_aer008_w0.646rad.modis"
print_head(fmodis_blue)
```

    Showing head: les0822nh15t13_aer008_w0.646rad.modis
    	40  40  23
    	! Simulated MODIS reflectances from SHDOM radiance output
    	! IA   Rad(1:Ncross)
    	1 1.0000  0.00   0.00 20.000 20.000 ! idir, mu, theta, phi_shdom, size_cross, size_along
    	1 0.0765 0.1141 0.2162 0.1455 0.1407 0.0961 0.0401 0.0496 0.0690 0.0696 0.0722 0.0709 0.0778 0.0677 0.0694 0.0692 0.0725 0.1055 0.0934 0.0724 0.0701 0.0680 0.0680 0.0679 0.0680 0.0682 0.0684 0.0691 0.0747 0.0888 0.1230 0.0937 0.1201 0.0896 0.0944 0.0857 0.0592 0.0672 0.0700 0.0722
    	2 0.0702 0.0723 0.0972 0.0969 0.2869 0.2755 0.0567 0.0559 0.0687 0.0636 0.0761 0.0977 0.0996 0.1314 0.0961 0.0598 0.0686 0.0689 0.0840 0.0941 0.0648 0.0692 0.0710 0.0670 0.0679 0.0678 0.0682 0.0687 0.0697 0.0782 0.1025 0.0723 0.0575 0.0718 0.0841 0.0720 0.0803 0.0733 0.0694 0.0699
    	3 0.0697 0.0700 0.0712 0.0723 0.1212 0.1625 0.0773 0.0977 0.0573 0.0622 0.0979 0.0866 0.1896 0.2539 0.1146 0.1198 0.1418 0.0673 0.0754 0.1100 0.0931 0.0671 0.0703 0.0683 0.0682 0.0679 0.0679 0.0683 0.0690 0.0737 0.0956 0.0804 0.0713 0.0749 0.0772 0.0540 0.0561 0.0685 0.0757 0.0685
    	4 0.0694 0.0696 0.0701 0.0710 0.0913 0.1209 0.3438 0.3862 0.1215 0.0807 0.2028 0.1568 0.0961 0.1207 0.1067 0.0881 0.0820 0.0575 0.0992 0.0793 0.0712 0.0679 0.0682 0.0682 0.0680 0.0680 0.0680 0.0681 0.0685 0.0693 0.0719 0.0790 0.0823 0.1356 0.1346 0.0854 0.0563 0.0640 0.0692 0.0697
    	5 0.0604 0.0688 0.0695 0.0700 0.0711 0.0791 0.2566 0.4028 0.1625 0.1041 0.2124 0.2489 0.0943 0.0469 0.0659 0.0868 0.1006 0.1008 0.0864 0.0592 0.0674 0.0679 0.0683 0.0682 0.0680 0.0680 0.0681 0.0680 0.0681 0.0688 0.0703 0.0903 0.1479 0.1072 0.1253 0.0785 0.0636 0.0553 0.0680 0.0623
    	6 0.0458 0.0667 0.0691 0.0695 0.0700 0.0715 0.0777 0.1253 0.0972 0.0941 0.2480 0.1939 0.0607 0.0509 0.0740 0.0917 0.0764 0.0891 0.0833 0.0703 0.0682 0.0671 0.0684 0.0683 0.0681 0.0681 0.0682 0.0684 0.0680 0.0683 0.0696 0.0764 0.1183 0.1811 0.1659 0.1587 0.0729 0.0854 0.1028 0.0425

```python
# Reflectance from the MODIS case, but at full 320 x 320 resolution
fhres_blue = "les0822nh15t13_aer008_w0.646.ref"
print_head(fhres_blue)
```

    Showing head: les0822nh15t13_aer008_w0.646.ref
    	0   0 0.0499 0.1004 0.2231 0.0945 0.0426 0.1586 0.0464
    	1   0 0.0495 0.1004 0.2272 0.0976 0.0393 0.2537 0.0463
    	2   0 0.0493 0.1004 0.2193 0.0935 0.0390 0.2863 0.0463
    	3   0 0.0493 0.1004 0.2151 0.0913 0.0391 0.2772 0.0463
    	4   0 0.0498 0.1004 0.2003 0.0937 0.0394 0.2357 0.0462
    	5   0 0.0533 0.1003 0.1858 0.0979 0.0399 0.1731 0.0462
    	6   0 0.0583 0.1003 0.1592 0.1063 0.0407 0.1175 0.0461
    	7   0 0.0602 0.1002 0.1362 0.1148 0.0413 0.0771 0.0460
    	8   0 0.0602 0.1002 0.1257 0.1197 0.0413 0.0479 0.0459
    	9   0 0.0602 0.1001 0.1255 0.1194 0.0413 0.0425 0.0459

```python

```

```python

```

```python

```


MIT License

Copyright (c) 2021 Taiki Miyagawa and Akinori F. Ebihara

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

# <font color=red> Lite version reduces GPU memory consumption at the expense of CPU memory and runtime. </font> Take care of  your CPU memory.

# Plot Speed-Accuracy Tradeoff (SAT) Curve of Validation dataset

```python
from __future__ import absolute_import, division, print_function

import os, sys, glob, time
from itertools import zip_longest
import optuna
import matplotlib.pyplot as plt
from matplotlib.ticker import LogFormatter, ScalarFormatter, NullFormatter
import numpy as np
import tensorflow as tf

from datasets.data_processing import read_tfrecords_UCF101,\
    decode_feat, sequential_slice, sequential_concat
from models.backbones_ti import LSTMModelLite
from models.optimizers import get_optimizer
from models.losses import get_gradient_lstm
from utils.misc import load_yaml, set_gpu_devices, fix_random_seed,\
    restrict_classes, extract_positive_row, add_max_to_diag
from utils.performance_metrics import multiplet_sequential_confmx,\
    llr_sequential_confmx,\
    truncated_MSPRT, threshold_generator,\
    calc_llrs, calc_oblivious_llrs, threshold_generator, thresh_sanity_check,\
    NP_test,\
    get_LLR_min_and_max, get_linspace, threshold_generator_with_values,\
    seqconfmx_to_metrics
```

# User-defined params

```python
# User-defined
order_sprt = 10 # order of MSPRT-TANDEM
oblivious = False # M-TANDEM (False) or M-TANDEMwO (True)
name_model = "./data-directory/sprt_multiclass/UCF101/ckptlogs/MSPRT-TANDEM_ver20210331_stat/__20210424_150952523/" # for example
```

```python
# User-defined
num_thresh = 10 # Total number of thresholds to be used to plot the SAT curve
batch_thresh = 5 # Memory consuming!! How many thresholds to be feeded at a time. < 10 is recommended
sparsity = "linspace" # "linspace" or logspace". Used for threshold generation.
class_plot = -2 # -1:micro-averaged recall, -2:macro-averaged recall.
dataname = "UCF101-50-240-320" # "UCF101-150-240-320", "UCF101-50-256-256", ...

tfr_train = "DATADIR/UCF101TFR-50-240-320/train01.tfrecords"
tfr_valid = "DATADIR/UCF101TFR-50-240-320/valid01.tfrecords"
tfr_test = "DATADIR/UCF101TFR-50-240-320/test01.tfrecords"
batch_size = 100 # Must be a divisor of the num of test datapoints. Memory consumption << batch_thresh.
batch_size2 = 50 # used for calc_llrs, NP_test, and trundated_MSPRT. Not necessarily a divisor of the sample size.
gpu = 0 # Which GPU to be used

assert num_thresh % batch_thresh == 0 # Make sure
```

```python
# Must be consistent with the model (name_model).
duration = 50
feat_dim = 2048 # input feature dimension
num_classes = 101
width_lstm = 256 # dim of hidden layers of LSTM
dropout = 0.
activation = "tanh"
```

```python
# Which GPU to be used
set_gpu_devices(gpu)
```

# 0. Start calc logits and LLRs

```python
# Load data
##################################
# Reed tfr and make
dataset_tr, dataset_va,\
    dataset_te = \
    read_tfrecords_UCF101(
        record_file_tr=tfr_train,
        record_file_va=tfr_valid,
        record_file_te=tfr_test,
        batch_size=batch_size,
        flag_shuffle=False,
        shuffle_buffer_size=10000)

# Model
##################################
model = LSTMModelLite(
    num_classes=num_classes,
    width_lstm=width_lstm,
    dropout=dropout,
    activation=activation)

# Restore parameters
#################################
ckpt = tf.train.Checkpoint(net=model)
ckpt_manager_restore = tf.train.CheckpointManager(ckpt,
    name_model, max_to_keep=3)
ckpt.restore(ckpt_manager_restore.latest_checkpoint)
print("Restored latest model\n{}".format(
    ckpt_manager_restore.latest_checkpoint))

# Evaluaton loop
##################################
for iter_b, feats in enumerate(dataset_va):
    cnt = iter_b + 1

    x_batch = feats[0]
    x_batch = tf.reshape(x_batch, (-1, duration, feat_dim))
        # (batch, duration, feat dims)
    y_batch = feats[1]
        # (batch, )

    x_slice, y_slice = sequential_slice(x_batch, y_batch, order_sprt)

    # Calc logits
    if iter_b == 0:
        ls_logits = []
        ls_labels = []

    # Calc loss
    logits_tmp = model(x_slice, False)
    logits_tmp, _ = sequential_concat(logits_tmp, y_slice, duration)

    ls_logits.append(logits_tmp.numpy())
    ls_labels.append(y_batch.numpy())

    # Verbose
    if ((iter_b+1)%10 == 0) or (iter_b == 0):
        sys.stdout.write("\rEvaluation Iter: {:3d}".format(iter_b+1))
        sys.stdout.flush()

print()
logits_all = np.concatenate(ls_logits, axis=0)
labels_all = np.concatenate(ls_labels, axis=0)
```

    Restored latest model
    /data/t-miyagawa/sprt_multiclass/UCF101/ckptlogs/50-240-320_ME_CSL_ver20210331_stat/__20210424_150952523/ckpt_step2600_macrec0.83303-10
    Evaluation Iter:  40

```python
# Calc LLRs
num_data = labels_all.shape[0]
num_iter = num_data // batch_size2 if num_data % batch_size2 == 0 else num_data // batch_size2 + 1
llrs_all = []
start = -1
stop = -1
for i in range(num_iter):
    itr_logits = logits_all[i * batch_size2 : (i + 1) * batch_size2]

    if oblivious:
        itr_llrs = calc_oblivious_llrs(itr_logits)
    else:
        itr_llrs = calc_llrs(itr_logits)

    llrs_all.append(itr_llrs.numpy())

    # Used to generate thresholds
    tmp_start, tmp_stop = get_LLR_min_and_max(itr_llrs) # A bit memory consuming.
    tmp_start = tmp_start.numpy()
    tmp_stop = tmp_stop.numpy()

    if i == 0:
        start = tmp_start
        stop = tmp_stop
        continue

    if tmp_start < start:
        start = tmp_start
    if tmp_stop > stop:
        stop = tmp_stop

    # Verbose
    if ((iter_b + 1) % 10 == 0) or (iter_b == 0):
        sys.stdout.write("\rEvaluation Iter: {:3d}/{}".format(iter_b+1, num_iter))
        sys.stdout.flush()

values = get_linspace(start, stop, num_thresh, sparsity).numpy()
llrs_all = np.concatenate(llrs_all, axis=0)
```

```python
tic = time.time()
print("This cell takes a few minutes because of the two mini-batch processes...")

# Batch processing for thresh axis to save GPU memory
######################################################
num_data = labels_all.shape[0]
ls_confmx_th = []
ls_mht_th = []
ls_vht_th = []
ls_trt_th = []
for i in range(num_thresh // batch_thresh):
    print("Thresh Iter {}/{} ...".format(i+1, num_thresh // batch_thresh))

    # Generate Thresholds for SPRT
    idx = batch_thresh * i
    itr_values = values[idx : idx + batch_thresh]
    itr_thresh = threshold_generator_with_values(itr_values, duration, num_classes) # memory consuming
    thresh_sanity_check(itr_thresh)

    # Batch processing for data axis to save GPU memory
    ####################################################
    num_iter = num_data // batch_size2 if num_data % batch_size2 == 0 else num_data // batch_size2 + 1
    tmp_confmx_th = 0
    tmp_mht_th = []
    tmp_sqr_th = []
    tmp_trt_th = []
    for i in range(num_iter):
        itr_llrs = llrs_all[i * batch_size2 : (i + 1) * batch_size2]
        itr_labels = labels_all[i * batch_size2 : (i + 1) * batch_size2]
        _len = itr_labels.shape[0]

        # Confusion matrix of SPRT, mean/var hitting time,
        # and truncation rate
        _tmp_confmx_th, _tmp_mht_th, _tmp_vht_th, _tmp_trt_th = \
            truncated_MSPRT(
                llr_mtx=itr_llrs,
                labels_concat=itr_labels,
                thresh_mtx=itr_thresh)
                # Super GPU memory super-consuming
                # if batch_thresh or batch_size2 is large.
            # The shapes of the outputs are:
            # (batch_thresh, num classes, num classes)
            # (batch_thresh,)
            # (batch_thresh,)
            # (batch_thresh,)
        _tmp_sqr_th = _tmp_vht_th + _tmp_mht_th ** 2

        tmp_confmx_th += _tmp_confmx_th.numpy()
        tmp_mht_th.append(np.expand_dims(_tmp_mht_th.numpy(), axis=0) * _len)
        tmp_sqr_th.append(np.expand_dims(_tmp_sqr_th.numpy(), axis=0) * _len)
        tmp_trt_th.append(np.expand_dims(_tmp_trt_th.numpy(), axis=0) * _len) #<- should be integers

    tmp_mht_th = np.concatenate(tmp_mht_th, axis=0) # (num_iter, batch_thresh)
    tmp_mht_th = np.sum(tmp_mht_th, axis=0) / num_data # (batch_thresh, )
    tmp_sqr_th = np.concatenate(tmp_sqr_th, axis=0) # (num_iter, batch_thresh)
    tmp_sqr_th = np.sum(tmp_sqr_th, axis=0) / num_data # (batch_thresh, )
    tmp_vht_th = tmp_sqr_th - tmp_mht_th ** 2 # (batch_thresh, )
    tmp_trt_th = np.concatenate(tmp_trt_th, axis=0) # (num_iter, batch_thresh)
    tmp_trt_th = np.sum(tmp_trt_th, axis=0) / num_data # (batch_thresh, )

    ls_confmx_th.append(tmp_confmx_th)
    ls_mht_th.append(tmp_mht_th)
    ls_vht_th.append(tmp_vht_th)
    ls_trt_th.append(tmp_trt_th)

confmx_th = np.concatenate(ls_confmx_th, axis=0) # (num thresh, num cls, num cls)
mht_th = np.concatenate(ls_mht_th, axis=0) # (num thresh, )
vht_th = np.concatenate(ls_vht_th, axis=0) # (num thresh, )
trt_th = np.concatenate(ls_trt_th, axis=0) # (num thresh, )

print(time.time() - tic, "seconds")
```

    This cell takes a few minutes because of the two mini-batch processes...OMG
    Thresh Iter 1/2 ...
    Thresh Iter 2/2 ...
    25.820960521697998 seconds

```python
# SPRT results
###########################
# Extract recalls (sensitivities)
dc_mtr = seqconfmx_to_metrics(confmx_th)

# Output
ls_sns = dc_mtr["SNS"].numpy()
    # (num thresh, num classes + 2)
    # [[recall of class 0, 1, 2, ..., recall of the last class, balanced accuracy, accuracy], ...]

ls_mht = mht_th#.numpy()
    # (num thresh,)
ls_vht = vht_th#.numpy()
    # (num thresh,)
ls_trt = trt_th#.numpy()
    # (num thresh,)

# NPT confmx
############################
num_iter = num_data // batch_size2 if num_data % batch_size2 == 0 else num_data // batch_size2 + 1
seqconfmx_llr = 0

for i in range(num_iter):
    if (i + 1) % 50 == 0:
        print("Iter {}/{}".format(i+1, num_iter))

    itr_llrs = llrs_all[i * batch_size2 : (i + 1) * batch_size2]
    itr_labels = labels_all[i * batch_size2 : (i + 1) * batch_size2]

    _tmp_seqconfmx_llr = NP_test(itr_llrs, itr_labels)
        # (duration, num classes, num classes)
    seqconfmx_llr += _tmp_seqconfmx_llr.numpy()
        # (duration, num classes, num classes)

# NPT results
###########################
dc_mtr_llr = seqconfmx_to_metrics(seqconfmx_llr)

# Output
ls_sns_NPT = dc_mtr_llr["SNS"].numpy()
    # (duration, num classes + 2)
    # [[recall of class 0, 1, 2, ..., recall of the last class, balanced accuracy, accuracy], ...]
################## end TMmod 20210106 ###############
```

    Iter 50/90

```python
ls_trt # truncation rate. All 0 or all 1 signal a problem.
```

    array([0.       , 0.3405927, 0.6012573, 0.8621464, 0.9986529, 1.       ,
           1.       , 1.       , 1.       , 1.       ], dtype=float32)

__Materials__
1. `ls_mht`

2. `ls_vht`

3. `ls_trt`

4. `ls_sns`

5. `ls_sns_NPT`

# 1. Plot SAT (Speed-Accuracy Tradeoff) Curve

```python
# Start plotting SP curve
######################################
# Parameters
x = ls_mht
y = 100 * (1 - ls_sns[:, class_plot]) # macro-averaged recall

title = "SAT curve, data={}, order={}".format(dataname, order_sprt)
xlabel = "Mean hitting time (# frames)"
if class_plot < num_classes:
    _tmp = "(Class={})".format(class_plot)
elif class_plot == num_classes:
    _tmp = "(Mac-ave)"
elif class_plot == num_classes + 1:
    _tmp = "(Mic-ave)"
else:
    raise ValueError
ylabel = "100 - Recall ({}) (%)".format(_tmp)

# Size
plt.rcParams["font.size"] = 25
fig, ax = plt.subplots(figsize=(13,8))
fig.patch.set_facecolor('white')
# Scale
ax.set_xscale('log')
# Grid
major_ticks = np.arange(0, duration + 1, 10)
major_ticks[0] += 1
minor_ticks = np.arange(0, duration + 1, 1)
ax.set_xticks(major_ticks)
ax.set_xticks(minor_ticks, minor=True)
ax.set_xticklabels(['0, 10, 20, 30, 40, 50'])
for axis in [ax.xaxis, ax.yaxis]: # ?
    axis.set_major_formatter(ScalarFormatter())
    axis.set_minor_formatter(NullFormatter())

# Plot
plt.scatter(
    x, y,
    s=10,
    color="red",
    marker='o',
    vmin=None,
    vmax=None,
    alpha=None,
    linewidths=None,
    edgecolors=None
    )

plt.ylabel(ylabel)
plt.xlabel(xlabel)
plt.grid(which='both')
#plt.legend(loc=None, fontsize='15')
plt.title(title)
plt.xlim(0.95, duration+0.3)# 1stO, 19thO
#plt.xlim(3.95, duration+0.3)# 1stO, 19thO
#plt.ylim(99.08, 99.65) # 1stO, 19thO
#plt.xlim(4.-0.3, duration+0.3)# 1stO, 19thO
#plt.ylim(70.08, 80.65) # 1stO, 19thO
plt.tight_layout()

# Save figure
#plt.savefig(***)
```

## 1-1. SPRT vs NPT

```python
# Plot NPT vs SPRT
###############################
# Parameters
x = ls_mht
y = 100 * (1 - ls_sns[:, class_plot])
x_np = np.array([i + 1 for i in range(duration)])
y_np = 100 * (1 - ls_sns_NPT[:, class_plot])
label = "SPRT"
label_np = "NPT"
title = "SAT curve, data={}, order={}".format(dataname, order_sprt)
xlabel = "Mean hitting time (# frames)"
if class_plot < num_classes:
    _tmp = "(Class={})".format(class_plot)
elif class_plot == num_classes:
    _tmp = "(Mac-ave)"
elif class_plot == num_classes + 1:
    _tmp = "(Mic-ave)"
else:
    raise ValueError
ylabel = "100 - Recall ({}) (%)".format(_tmp)

# Size
plt.rcParams["font.size"] = 25
fig, ax = plt.subplots(figsize=(13,8))
# Scale
ax.set_xscale('log')
# Grid
major_ticks = np.arange(0, duration + 1, 10)
major_ticks[0] += 1
minor_ticks = np.arange(0, duration + 1, 1)
ax.set_xticks(major_ticks)
ax.set_xticks(minor_ticks, minor=True)
ax.set_xticklabels(['0, 10, 20, 30, 40, 50'])
for axis in [ax.xaxis, ax.yaxis]: # ?
    axis.set_major_formatter(ScalarFormatter())
    axis.set_minor_formatter(NullFormatter())

# Plot SPRT
plt.scatter(
    x, y,
    s=10,
    color="red",
    marker='o',
    vmin=None,
    vmax=None,
    alpha=None,
    linewidths=None,
    edgecolors=None,
    label="SPRT"
    )

# Plot NPT
plt.scatter(
    x_np, y_np,
    s=10,
    color="blue",
    marker='o',
    vmin=None,
    vmax=None,
    alpha=None,
    linewidths=None,
    edgecolors=None,
    label="NPT"
    )

plt.ylabel(ylabel)
plt.xlabel(xlabel)
plt.grid(which='both')
plt.legend(loc=1, fontsize='15')
plt.title(title)
plt.xlim(0.95, duration+0.3)
#plt.xlim(10.95, duration+0.3)
#plt.ylim(88., 99.7) # v1
#plt.ylim(95, 99.7) # v2
plt.tight_layout()

# Save figure
# plt.savefig(***)
```

# 2. Plot SAT Curve with Standard Deviation of Hitting Times

```python
# Start plotting SP curve with variance
######################################
# Parameters
x = ls_mht
y = 100. * ls_sns[:, -2] # macro-averaged recall
xerr =  np.sqrt(ls_vht)
title = "SAT Curve, data=NMNIST, order={}".format(order_sprt)
xlabel = "Mean Hitting Time (# frames)"
ylabel = "Macro-averaged recall (%)"

# Size
plt.rcParams["font.size"] = 25
fig, ax = plt.subplots(figsize=(13,8))
fig.patch.set_facecolor('white')
# Scale
#ax.set_xscale('log')
# Grid
major_ticks = np.arange(0, duration + 1, 10)
major_ticks[0] += 1
minor_ticks = np.arange(0, duration + 1, 1)
ax.set_xticks(major_ticks)
ax.set_xticks(minor_ticks, minor=True)
ax.set_xticklabels(['0, 10, 20, 30, 40, 50'])
for axis in [ax.xaxis, ax.yaxis]: # ?
    axis.set_major_formatter(ScalarFormatter())
    axis.set_minor_formatter(NullFormatter())

# Plot
plt.errorbar(
    x, y,
    xerr = xerr,
    capsize=3,
    fmt='o',
    markersize=2,
    ecolor='black',
    markeredgecolor = "black",
    color='red',
    elinewidth=0.5)

plt.ylabel(ylabel)
plt.xlabel(xlabel)
plt.grid(which='both')
#plt.legend(loc=4, fontsize='15')
plt.title(title)
#plt.xlim(0.95, duration+0.3)
#plt.ylim(88., 99.7) # v1
#plt.ylim(95, 99.7) # v2
plt.tight_layout()

# Save figure
# plt.savefig(***)
```

# 3. Plot Trajectories of LLRs

- min LLR.
    - $\mathrm{max}_k \mathrm{min}_{l \neq k} \mathrm{LLR}_{kl} (t)$ on a single plane
- min LLR for each class ($K$ plots).
    - $\mathrm{min}_{l \neq k} \mathrm{LLR}_{kl} (t)$. One $k$, one plane
- all LLRs for each class ($K$ plots in each of which there are $K$ - 1 types of trajectories).
    - $\mathrm{LLR}_{kl} (t)$, where $k \neq l$. One $k$, one plane.
- all LLRs, positive class mins (red) and negative class mins (blue).
    - $\mathrm{min}_{l \neq k} \mathrm{LLR}_{kl} (t)$, where `color = red` if $k = y_i$ else `color = blue`.
- positive LLR row. All and each k.
    - $\mathrm{LLR}_{y_{i}\, l}$, where $l \neq y_i$

\begin{equation}
    \lambda(t) :=
    \begin{pmatrix}
    0                & \cdots &  \lambda_{1l}(t) & \cdots & \lambda_{1K}(t)\\
    \vdots           & \ddots &                  &        & \vdots \\
    -\lambda_{1l}(t) &        &                0 &        & \lambda_{kK}(t) \\
    \vdots           &        &                  & \ddots & \vdots \\
    -\lambda_{1K}(t) & \cdots & -\lambda_{kK}(t) & \cdots & 0
    \end{pmatrix}
\end{equation}

\begin{equation}
    \lambda_{kl}(t) := \log(\frac{p(X^{(1,t)}\,\,\,|y=k)}{p(X^{(1,t)}\,\,\,|y=l)})\,\,\, (k, l \in [K], K\in\mathbb{N})
\end{equation}

## 3-4-1. All LLR rows with minimum $l

- $\mathrm{min}_{l\neq k} \mathrm{LLR}_{k\, l}$

```python
num_trj = 1
plot_classes = [] # optional: default=[]
shuffle = True # currently not supported
```

```python
# Initialization
llrs_trj = llrs_all[:num_trj]
labels_trj = labels_all[:num_trj]

# Restrict classes (optional)
llrs_trj, labels_trj = restrict_classes(llrs_trj, labels_trj, plot_classes)
labels_trj = labels_trj.numpy() if len(plot_classes) != 0 else labels_trj

### Add max LLR to diag
llrs_trj = add_max_to_diag(llrs_trj)

### Extract min column: shape change (batch, duration, num classes num classes) -> (batch, duration, num classes)
llrs_trj = tf.reduce_min(llrs_trj, axis=3)

# Shuffle (optional)
if shuffle:
    idx_perm = np.random.permutation(labels_trj.shape[0]) # randomly take some LLRs
    llrs_trj = tf.constant(llrs_trj.numpy()[idx_perm], dtype=tf.float32)
    labels_trj = tf.constant(labels_trj[idx_perm], dtype=tf.int32)

# Pick some out
llrs_trj = llrs_trj[:num_trj].numpy()
labels_trj = labels_trj[:num_trj].numpy()
```

```python
# Color definitions
ls_color = [
    "red",
    "blue",
]
ls_plotLabel =[
    "min_l {LLR_k =yi l}",
    "min_l {LLR_k!=yi l}",
]
```

```python
# Start plotting trajectory
######################################
# Parameters
x = [i for i in range(0, duration+1)]
title = "Random example LLR trajectory,\ndata=NMNIST-100f15pix, order={}".format(order_sprt)
xlabel = "Frame"
ylabel = "Log-likelihood ratio"

# Size
plt.rcParams["font.size"] = 25
plt.figure(figsize=(13,8))

# plt.xticks([1, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100],
#     [])

_tmp = set()
for itr_llrs, itr_label in zip(llrs_trj, labels_trj):
    itr_llrs = np.transpose(itr_llrs, [1, 0])
        # (num classes, duration)
    for itr_i, itr_trj in enumerate(itr_llrs):
        y = np.append(0, itr_trj)

        # Color and plot label
        if itr_i == itr_label:
            color = ls_color[0]
            if not (_tmp >= {ls_plotLabel[0]}):
                plot_label = ls_plotLabel[0]
                _tmp.add(plot_label)
            else:
                plot_label = None
        else:
            color = ls_color[1]
            if not (_tmp >= {ls_plotLabel[1]}):
                plot_label = ls_plotLabel[1]
                _tmp.add(plot_label)
            else:
                plot_label = None

        # Plot
        plt.plot(
            x, y,
            color=color,
            marker='o',
            linewidth=1,
            markersize=0,
            label=plot_label,
            alpha=0.7
            )
        cnt += 1

plt.xlabel(xlabel)
plt.ylabel(ylabel)
plt.yscale("symlog")
plt.grid(True)
plt.legend(fontsize='15')
plt.title(title)
plt.xticks([i for i in range(0, duration + 1)])
#plt.xlim(-0.1, 20.1)
plt.locator_params(axis='x', nbins=11)
plt.tight_layout()

# Save figure
# plt.savefig(
#     "./traj_order{}.svg".format(order_sprt),
#     format="svg",
#     dpi=1200)
```

## 3-1. Positive LLR row

- $\mathrm{LLR}_{y_{i}\, l}$, where $l \neq y_i$

```python
num_trj = 10
plot_classes = [] # optional: default=[]
shuffle = True # currently not supported
```

```python
# Initialization
llrs_trj = llrs_all[:50]
labels_trj = labels_all[:50]

# Restrict classes (optional)
llrs_trj, labels_trj = restrict_classes(llrs_trj, labels_trj, plot_classes)
labels_trj = labels_trj.numpy() if len(plot_classes) != 0 else labels_trj ################### TMmod 20210106 ###############

### Extract positive rows: shape change -> (batch, duration, num classes)
llrs_trj = extract_positive_row(llrs_trj, labels_trj)

# Shuffle (optional)
if shuffle:
    idx_perm = np.random.permutation(labels_trj.shape[0]) # randomly take some LLRs
    llrs_trj = tf.constant(llrs_trj.numpy()[idx_perm], dtype=tf.float32)
    labels_trj = tf.constant(labels_trj[idx_perm], dtype=tf.int32)

# Pick some up
llrs_trj = llrs_trj[:num_trj].numpy()
labels_trj = labels_trj[:num_trj].numpy()
```

```python
# Color definitions
ls_color = [
    None
] * num_classes
ls_plotLabel =[
    "_"] * num_classes
```

```python
# Start plotting trajectory
######################################
# Parameters
x = [i for i in range(0, duration+1)]
title = "Random example LLR trajectory, data=NMNIST, order={}".format(order_sprt)
xlabel = "Frame"
ylabel = "Log-likelihood ratio"

# Size
plt.rcParams["font.size"] = 25
plt.figure(figsize=(13,8))

_tmp = set()
for itr_llrs, itr_label in zip(llrs_trj, labels_trj):
    itr_llrs = np.transpose(itr_llrs, [1, 0])
    itr_llrs = np.delete(itr_llrs, obj=itr_label, axis=0)
        # (num classes - 1, duration)
    for itr_trj in itr_llrs:
        y = np.append(0, itr_trj)

        # Color
        color = ls_color[itr_label]

        # Label
        if _tmp >= {itr_label}:
            plot_label = None
        else:
            _tmp.add(itr_label)
            plot_label = ls_plotLabel[itr_label]

        # Plot
        plt.plot(
            x, y,
            color=color,
            marker='o',
            linewidth=1,
            markersize=0,
            label=plot_label
            )
        cnt += 1

plt.xlabel(xlabel)
plt.ylabel(ylabel)
plt.yscale("symlog")
plt.grid(True)
#plt.legend(fontsize='15')
plt.title(title)
plt.xticks([i for i in range(0, duration + 1)])
#plt.xlim(-0.1, 20.1)

plt.tight_layout()

# Save figure
# plt.savefig(
#     "./traj_order{}.svg".format(order_sprt),
#     format="svg",
#     dpi=1200)
```

## 3-3. Positive LLR row with minimum $l$
- $\mathrm{min}_{l (\neq y_i)} \mathrm{LLR}_{y_{i}\, l}$

```python
num_trj = 20
plot_classes = [] # optional: default=[]
shuffle = True # currently not supported
```

```python
# Initialization
llrs_trj = llrs_all[:100]
labels_trj = labels_all[:100]

# Restrict classes (optional)
llrs_trj, labels_trj = restrict_classes(llrs_trj, labels_trj, plot_classes)
labels_trj = labels_trj.numpy() if len(plot_classes) != 0 else labels_trj ################### TMmod 20210106 ###############
### Add max LLR to diag
llrs_trj = add_max_to_diag(llrs_trj)

### Extract positive rows: shape change -> (batch, duration, num classes)
llrs_trj = extract_positive_row(llrs_trj, labels_trj)

### Extract min column: shape change (batch, duration, num classes) -> (batch, duration)
llrs_trj = tf.reduce_min(llrs_trj, axis=2)

# Shuffle (optional)
if shuffle:
    idx_perm = np.random.permutation(labels_trj.shape[0]) # randomly take some LLRs
    llrs_trj = tf.constant(llrs_trj.numpy()[idx_perm], dtype=tf.float32)
    labels_trj = tf.constant(labels_trj[idx_perm], dtype=tf.int32)

# Pick some up
llrs_trj = llrs_trj[:num_trj].numpy()
labels_trj = labels_trj[:num_trj].numpy()
```

```python
# Color definitions
ls_color = [
    None
] * num_classes
ls_plotLabel =[
    "_"] * num_classes
```

```python
# Start plotting trajectory
######################################
# Parameters
x = [i for i in range(0, duration+1)]
title = "Random example LLR trajectory, data=NMNIST, order={}".format(order_sprt)
xlabel = "Frame"
ylabel = "Log-likelihood ratio"

# Size
plt.rcParams["font.size"] = 25
plt.figure(figsize=(13,8))

_tmp = set()
for itr_trj, itr_label in zip(llrs_trj, labels_trj):
    y = np.append(0, itr_trj)

    # Color
    color = ls_color[itr_label]

    # Label
    if _tmp >= {itr_label}:
        plot_label = None
    else:
        _tmp.add(itr_label)
        plot_label = ls_plotLabel[itr_label]

    # Plot
    plt.plot(
        x, y,
        color=color,
        marker='o',
        linewidth=2,
        markersize=0,
        label=plot_label
        )

plt.xlabel(xlabel)
plt.ylabel(ylabel)
plt.yscale("symlog")
plt.grid(True)
#plt.legend(fontsize='15')
#plt.title(title)
plt.xticks([10 * i for i in range(0, 6)])
#plt.xlim(-0.1, 20.1)

plt.tight_layout()

# Save figure
# filename = "/data/t-miyagawa/sprt_multiclass/nosaic_mnist-1000f/graphs/LLRtrajectory_UCF101v4"
# plt.savefig(
#     "{}.svg".format(filename),
#     format="svg",
#     dpi=1200)
```


# Jupyter Test Drive - Typesetting with Markdown

In this notebook we'll be walking through some activities so you can familiarize yourself with using a Jupyter notebook for typesetting with markdown. This can be useful if you want to modify an existing notebook, or even to help your students use Jupyter for typesetting answers to any activities they do using Callysto.

## The _very_ Basics

Note that you can view the source code behind any cell by double clicking it. To re-render or re-run the cell, use any of these options:
1. Select `Cell` $\rightarrow$ `Run Cells`
2. Hit `Shift-Enter`
3. Click the `Run` button above

## Creating and Deleting Markdown Cells
The first step you may want do do is actually create a new cell for typesetting. In order to do that, while you have this cell selected, from the top menu click the plus (+) sign next to the save button.

Once you have created the new cell, you will get a blank `Code` cell above. In order to change that cell into one that you can use for typesetting, select it, and from the dropdown menu at the top, change `Code` $\rightarrow$ `Markdown`. This changes that cell type into a markdown cell, and it can now be used for typesetting!

The next step will be learning how to delete cells. Select the cell above that you just converted to a markdown cell. In order to remove this cell from your Jupyter notebook, from the top menu click `Edit` $\rightarrow$ `Delete Cells`. If you accidentally deleted an incorrect cell, you can always bring it back by clicking `Edit` $\rightarrow$ `Undo Delete Cells`.

## Creating Titles

Creating large bold title text in a Jupyter notebook with markdown is very simple! All you need to do is on a new line, type the pound sign and then a space. After the space, simply add your title. It's that simple. By adding additional pound signs, you will create different title levels. The cell below is unformatted markdown text demonstrating this.
# I am a level one title
## I am a level two title
### I am a level three title
#### I am a level four title
##### I am a level five title
###### I am a level six title
####### I would be a level seven title if such a thing existed

If the # isn't the first line, it does not get rendered as a title.
To see how those titles look when they're rendered in the Jupyter notebook, select the cell above, and from the top menu select `Raw NBConvert` $\rightarrow$ `Markdown` then type `Ctrl + Enter` to render the cell. Feel free to create a new cell and play with those, or simply modify the one above directly.

## Creating Itemized Lists
Creating lists in markdown in a Jupyter notebook is also fairly straight forward! To do so you simply need to start a new line and begin counting. The markdown syntax is shown in the code below. Again, to see how it renders, select the cell below then from the top menu select `Raw NBConvert` $\rightarrow$ `Markdown` then type `Ctrl + Enter`.
1. This is the first item of a list
1. This is the second
50. Notice how the numbers we use don't matter once it's rendered
    1. By pressing the `Tab` key we can add sub items
    45. The numbers don't matter here either
        - We can add many sub-lists
            - But at some point it starts to get redundant
                - But I guess that's okay sometimes
                    - Although, nested lists like this are generally a sign your list should be reorganized
                        - Notice how the color changes of the text before we render the markdown to help you
                        know where in the list you're nested (if cell is set to markdown)
        - We can also go back a few levels of nesting by lining the text up with the line above
2346. By going back to the original line, we leave our nested lists
23236235672456. The numbers really don't matter.
* In fact, you can give up on numbers at any time.
- It still counts for you
1. Even if you start counting again

- We also don't need to have numbers if we don't want to!
    * We have plenty of options
-Notice how if we don't put a space between the dash and the first term it doesn't become a new item in the list!

---
### Exercises 1
1. In the cell above, try fixing the last line so that the last item in the list becomes properly formatted once it's rendered.
2. Create a new cell below, add a title, and an ordered list of your five favourite fruits.
---

## Links
You can add hyper links to webpages anywhere you'd like right in the text. The syntax is as follows
```markdown
[text you want to display](link to a file or url page)
```
Change the cell below to markdown, run it, and see where the link takes you!
[this is the text that becomes the hyperlink](https://media.giphy.com/media/Vs44a88Dvbnkk/giphy.gif)
## Images/gifs
Images and gifs can be emebedded in a markdown cell easily using a syntax near identical to links by simply adding an explaination point before the link. For example
```markdown

```
would display an image/gif. It can either be a url link to an image, or one you have saved on the hub/locally. Change the cell below to markdown and run it to see it in action, or even embed your own images in this notebook.

---
### Exercises 2
1. In a new cell below, embed an image of a cat.
2. In the same cell, add a hyperlink below the image that links to the original image.
3. Change the text displayed in the hyperlink to say "Kitty Kitty".
---

# Block Quotes
Block quotes are a way to indent text/images with a grey bar in order to distinguish them from the main text. It is quite simlly done by adding a `>` at the beginning of a line. Change the cell below to markdown to see some examples
> This is an example of a block quote. You can type forever and the whole text will be indented. You could literally type forever and it would render as a block quote, but I don't recommend it because that would be a huge waste of everyone's time.

You can also use block quotes to easily format images with captions!

>
>
> Now we can include a caption for images. This is nice because the image is now distinguishable from the main text. This makes it easy to include images without necessarily interrupting any narrative you've created. Notice how I have included a blank `>` under the image. That is simply so the spacing of this text looks nice underneath. Try removing it and see what happens
## Tables

We can also create tables of data in Markdown. However, it's not the most convenient and for large tables I would recommend using online tools such as [here](https://jakebathman.github.io/Markdown-Table-Generator/) where you can copy-paste excel tables or csv files. This will then output syntax to properly render those tables. However, for smaller tables, it's quite easy to simply type out your own. See the cell below.
| This is the first column name| Second Column| Third|
|---|---|-------------|
|Notice how nothing is lined up when we type it | But that it looks nice when rendered| how convenient!|
|shorter lines are good too| numbers?| 343453|
|symbols| `yes`| $\int e^{-x^2} dx$ |
|also notice that the number of dashes in the second line don't matter | it just needs to be at least three| the rest are really only for you to make pre-rendering look nice |
The basic table syntax is as follows:
```markdown
|initialize| your| column| names|
|----------|-----|-------|------| <- this line sets up the headers, need at least three `---` each line
|     data | data| data  | data | <- any kind of text data can be put in a table. Feel free to mix too!
| numbers  | text| text  | data   <- you don't need to include the right-most pipe if you don't want to
```

essentially the syntax can be thought of as how you would try to write a table directly using symbols on your keyboard. Play with the table in the cell above, or create a new cell and try making your own! You could put one in this cell as well and put a table here. Simply double click anywhere on this cell to open edit mode.

---
### Exercises 3
1. Using sample notebooks, teachers may like or dislike various sections of the sample provided. Luckily, it's easy to make these minor changes in notebooks. In the two cells below:
    * remove the second cell because you won't need it in your lesson
    * remove the last paragraph in the first cell
    * change the date in the first paragraph
2. In the first cell below, take the ordered list out of the block quote.

---

> ## Properties of Random Numbers
>
> May 22, 2018
> Suppose you have a sequence of $N$ random numbers $\{R\}$ with contents $\{r_1, r_2, ... , r_N\}$ where each element $r_i$ is a random number. What sort of properties should this sequence of numbers have? If this is truly a sequence of random numbers, it _must_ satisfy the following properties, which we will explain in greater detail:
>
> 1. Drawing any $r_i$ is equally probable and independent.
> 2. The sequence of random numbers is uniformly distributed.
>
> "Drawing a value" in this scope means we're picking a number from our sequence of random numbers, but not removing it from the sequence (the sequence remains unchanged, we're simply "observing" the random number).
Let's look at these two properties in a little more detail.

> ### All Values Are Equally Probable and Independent
>
> What this means is that if you were to select (but not remove) a number from your sequence of random numbers $\{r_1, r_2, ... , r_N\}$ at random, the probability of drawing any of those numbers is
\begin{equation}
p(r_i) = \frac{1}{N}
\end{equation}
where $p(r_i)$ is the probability of selecting a number $r_i$. This probability is identical for all numbers within your set. More explicitly:
\begin{equation}
p(r_1) = p(r_2) = ... = p(r_N) = \frac{1}{N}
\end{equation}
>
> The independence property means that the if you draw a number from the set, it does not affect the probability of drawing other numbers, or even itself at a later time. This is because the sequence remains unchanged after you draw (observe) a number. This property leads directly to the second important properties of random numbers, discussed below.

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)