```python
# -*- coding: utf-8 -*-
#   This work is part of the Core Imaging Library (CIL) developed by CCPi
#   (Collaborative Computational Project in Tomographic Imaging), with
#   substantial contributions by UKRI-STFC and University of Manchester.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#   http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#   Copyright 2019 UKRI-STFC, The university of Manchester
#   Authored by:    Ryan Warr (UoM)
#                   Evangelos Papoutsellis (UKRI-STFC)
```

Hyperspectral CT Reconstruction

Learning Objectives

By the end of this notebook, you will be able to:

- Identify the key differences in building Image/Acquisition Geometries and Operators for hyperspectral datasets
- Build your own reconstructions using FDK, CGLS and PDHG
- Determine optimum regularisation parameters based on reconstruction method
- Evaluate the effectiveness of each reconstruction routine using spatial and energy profiles.

### Prerequisites:

- Acquisition/Image Geometry, Acquisition Data
- ProjectionOperator
- FDK, CGLS, PDHG, TV
- BlockFramework

### Background:

Conventional X-ray detectors only measure the variable density of objects they pass through, giving no insight into what materials are actually inside. This is because standard detectors measure only the number of photons that arrive at each point on the detector.

For hyperspectral imaging, one can use an energy-sensitive X-ray detector, which measures the energy of every X-ray photon that arrives at each individual pixel. This provides an additional layer of information which can provide important insight on a sample's composition or structure. However, adapted reconstruction routines are required to account for the extra energy-based dimension.

The additional energy dimension is stored as a histogram of energy 'channels', indicating the number of X-ray photons detected by a pixel within a fine energy range. Typically 200+ channels are acquired, however in order to speed up computation time, we will restrict our dataset to just 40 channels, where the dominant energy signals are known to appear.

```python
# Import libraries
from cil.framework import AcquisitionGeometry, BlockDataContainer

from cil.optimisation.algorithms import PDHG, CGLS
from cil.optimisation.operators import BlockOperator, GradientOperator
from cil.optimisation.functions import L2NormSquared, L1Norm, MixedL21Norm, BlockFunction, IndicatorBox

from cil.io import NEXUSDataReader, NEXUSDataWriter

from cil.plugins.astra.processors import FBP
from cil.plugins.astra.operators import ProjectionOperator

from cil.utilities.display import show2D, show_geometry
from cil.utilities.jupyter import islicer, link_islicer

import numpy as np
import matplotlib.pyplot as plt
import h5py
import scipy.io as sio
import os

from mpl_toolkits.axes_grid1 import AxesGrid
```

## Load data using scipy

```python
# Read h5 data file
pathname = os.path.abspath("/mnt/materials/SIRF/Fully3D/CIL/Lizard")
filename = 'Lizard_Sino_100_140.h5'

path = os.path.join(pathname , filename)
arrays = {}

with h5py.File(path, 'r') as f:
    for k, v in f.items():
        arrays[k] = np.array(v)
    X = arrays['SS']
f.close()

# Read Energy-Channel conversion
tmp_energy_channels = sio.loadmat(pathname + "/Energy_axis.mat")
ekeV = tmp_energy_channels['E_axis']
ekeV_crop = ekeV[0][99:139]
```

The sample we will look at in this notebook is an iodine-stained lizard head. The use of elemental staining is common in biology and medicine, by acting as a contrast agent to provide improved visibility of internal structures for X-ray imaging. Iodine is a popular choice in the clinical and research fields, due to its energy-based properties falling in the typical range for diagnostic X-ray imaging.

The sample was scanned in a lab-based X-ray CT cone-beam system at 50keV, 0.65W, using an energy-sensitive detector to acquire a full 4D dataset. The detector consisted of an 80 x 80 pixel array, with pixel sizes of 250 $\mu$m x 250 $\mu$m. A source-sample distance of 332.0 mm and a sample-detector distance of 270.0 mm gave a geometric magnification of 1.8x, and a reconstructed voxel size of 137$\mu$m. The sample was scanned for 60 projections over a full 360$^{\circ}$ rotation, with 120s exposure time per projection.

A diagram of the style of setup used for spectral imaging is shown below from a paper by [C.K.Egan *et al*, 2015](https://www.nature.com/articles/srep15979#):

As shown by the diagram, each pixel stores its own energy channel histogram, with characteristic signals such as 'Absorption edges' (caused by photoelectric absorption of X-ray photons by the iodine in our sample) producing sharp rises in signal. These are the key features we look for when analysing spectral data.

Setting up Acquisition/Image Geometry

First we need to setup the geometries based on the acquisition system.
These are currently ordered based on the 4D raw dataset.

Run the code to see the current dimensions.

```python
print(X.shape)
```

We can allocate these separately

```python
num_channels = X.shape[0]
vertical = X.shape[1]
num_angles = X.shape[2]
horizontal = X.shape[3]
```

We set the angles used based on the parameters chosen for data acquisition. An offset is also applied (this just gives us the freedom to adjust the rotation of our reconstructed images).

```python
angles = np.linspace(-180-160,180-160,num_angles,endpoint=False)*np.pi/180
```

## 4D Acquisition and Image Geometries

We encode all this information into `AcquisitionGeometry` and `ImageGeometry`. We first define the scan geometry parameters, including source-sample-detector distances. These are then used to define our `AcquisitionGeometry`. See **Week1/00_CIL_geometry.ipynb** for more information.

```python
# Define the system scan geometry and detector parameters
distance_source_center = 332.0  # [mm]
distance_center_detector = 270.0  # [mm]
detector_pixel_size = 0.250  # [mm]
```

```python
# Define acquisition geometry from the detector system acquisition settings
# with reordering of the dimension labels to match the raw data
ag = AcquisitionGeometry.create_Cone3D(source_position = [0,-distance_source_center,0],
                                       detector_position = [0,distance_center_detector,0])\
                                     .set_panel([horizontal,vertical],[detector_pixel_size,detector_pixel_size])\
                                     .set_channels(num_channels)\
                                     .set_angles(-angles,angle_unit="radian")\
                                     .set_labels(['channel', 'vertical', 'angle', 'horizontal'])

# Create the 4D acquisition data
data = ag.allocate()
data.fill(X)
```

## Show acquisition geometry

```python
show_geometry(ag, view_distance = 4.5, elevation=10, azimuthal=-65, figsize=(10,10))
```

```python
print(data)
```

Get the `ImageGeometry` directly from the `AcquisitionGeometry` using `ig = ag.get_ImageGeometry()`

```python
ig = ag.get_ImageGeometry()
```

We can use an interactive image slicer `islicer` to provide a visualisation of the `AcquisitionData` in any dimension below. Simply by taking a data slice in a particular dimension, we can then visualise the data in any other given dimension.

Run the code below to see three such examples:

1) Projection radiographs for each of the 60 rotation angles acquired in a single channel

2) The sinogram for each energy channel for the central slice

3) The spectral signals acquired in each energy channel for a single projection angle

**Note: You can adjust the look of your reconstructions by varying certain parameters**
 - by removing `cmap`, you return to the default colour map of 'gray'
 - the default scaling runs through the full data range, set initial limits using e.g. `minmax = (0.0 , 2.0)`

```python
islicer(data.get_slice(channel=20), direction='angle', title = 'Projection Angle', cmap='inferno',minmax=(0.0,2.0))
```

```python
islicer(data.get_slice(vertical=40), direction='channel', title = 'Sinogram Channel', cmap='inferno',minmax=(0.0,1.0))
```

```python
islicer(data.get_slice(angle=42), direction='channel', title = 'Channel', cmap='inferno',minmax=(0.0,2.0))
```

We setup the tomography operator for 3D hyperspectral data using the `AcquisitionGeometry` and `ImageGeometry`

```python
ag3D = ag.get_slice(channel=0)
ig3D = ag3D.get_ImageGeometry()
```

# FDK Reconstruction

One of the simplest, and most common, means of image reconstruction for X-ray CT is the use of Filtered BackProjection (FBP). In the case of many lab-based X-ray sources, which utilise a cone-beam rather than parallel- or fan-beams, we use a specific case of FBP: The [Feldkamp-Davis-Kress (FDK)](https://www.osapublishing.org/josaa/abstract.cfm?uri=josaa-1-6-612) algorithm.

The function `FBP` is capable of handling reconstructions for both parallel-beam and cone-beam geometries in 2D and 3D. To apply this to 4D data, we loop over all channels in our dataset, performing a 3D FDK reconstruction for each channel, and fill this into a 4D (3D + Channels) image data array.

```python
# allocate space for the FBP_4D recon
FBP_recon_4D = ig.allocate()

# FBP reconstruction per channel
for i in range(ig.channels):

    FBP_recon_3D = FBP(ig3D, ag3D, 'gpu')(data.get_slice(channel=i))
    FBP_recon_4D.fill(FBP_recon_3D, channel=i)

    print("Finish FBP recon for channel {}".format(i), end='\r')

print("\nFDK Reconstruction Complete!")
```

We can use the `show2D` function to observe a 2D reconstructed slice of our 4D volume. Using the `slice_list` argument, we can set specific values for each dimension to choose what reconstructed slice we wish to see. Here we look at a single slice (35) in the vertical ('axial') plane, shown for a single energy channel (20). As expected, the data is quite noisy, with poor feature definition in the majority of the soft tissue.

```python
show2D(FBP_recon_4D,
       slice_list=[[("channel",20),("vertical",35)]],
       cmap="inferno", origin="upper",
       fix_range=(0,0.5), size=(10,10))
```

While some features of the lizard head can be seen in the reconstructed images, much of the signal is shrouded by noise.
In the next section, we will explore the first iterative algorithm - CGLS.

## Running the CGLS algorithm on a 4D dataset

As the next step, we will begin with a standard CGLS algorithm, applied to our 4D dataset

This is very simple to set up, we simply require a `ProjectionOperator`, A, which we can create using our `AcquisitionGeometry` and `ImageGeometry`. For a 4D dataset, we require the use of `gpu`.

After initialising the reconstruction with `x0 = ig.allocate()`, we can set up our CGLS algorithm.

Let's test the CGLS algorithm for just 10 iterations.

```python
# Create Projection Operator
A = ProjectionOperator(ig, ag, 'gpu')

# Initialise
x0 = ig.allocate()
cgls = CGLS(initial = x0, operator = A, data = data,
               max_iteration = 10, update_objective_interval = 2)
cgls.run(10,verbose=1)
cgls_10 = cgls.solution
```

We can evaluate the quality of our reconstruction, by looking at different spatial planes, as well as varying the single energy channel we observe. We access these by selecting values from our reconstruction with `cgls_10.as_array()[channel_num,slice_z,slice_y,slice_x]`.

Run the code below to see reconstructions for the 10th, 20th and 30th channel in our truncated dataset, with the X-ray energies these correspond to, for three different spatial planes (Axial, Coronal, Sagittal). As you will see, the attenuation level rises sharply between the 10th and 20th energy channel, indicative of a spectral marker (absorption edge) being crossed.

## Show CGLS reconstruction with 10 iterations at different energy channels and views: Axial, Coronal, Sagittal

```python
recons = [cgls_10.get_slice(channel=10, vertical=35), cgls_10.get_slice(channel=10, horizontal_y=35), cgls_10.get_slice(channel=10, horizontal_x=25),\
          cgls_10.get_slice(channel=20, vertical=35), cgls_10.get_slice(channel=20, horizontal_y=35), cgls_10.get_slice(channel=20, horizontal_x=25),\
          cgls_10.get_slice(channel=30, vertical=35), cgls_10.get_slice(channel=30, horizontal_y=35), cgls_10.get_slice(channel=0, horizontal_x=25)]

labels_x = ["Axial", "Coronal", "Sagittal"]
labels_y = ["{:.2f} keV ".format(ekeV_crop[10]),
            "{:.2f} keV ".format(ekeV_crop[20]),
            "{:.2f} keV ".format(ekeV_crop[30])]

plt.rcParams['xtick.labelsize']=15
plt.rcParams['ytick.labelsize']=15

fig = plt.figure(figsize=(12, 14))

grid = AxesGrid(fig, 111,
                nrows_ncols=(3, 3),
                axes_pad=0.05,
                cbar_mode='single',
                cbar_location='right',
                cbar_size = 0.5,
                cbar_pad=0.1
                )
k = 0

for ax in grid:
    im = ax.imshow(recons[k].as_array(), cmap="inferno", vmin = 0.0, vmax = 0.5)

    if k==0:
        ax.set_title(labels_x[0],fontsize=20)
        ax.set_ylabel(labels_y[0],fontsize=20)
    if k==1:
        ax.set_title(labels_x[1],fontsize=20)
    if k==2:
        ax.set_title(labels_x[2],fontsize=20)
    if k==3:
         ax.set_ylabel(labels_y[1],fontsize=20)
    if k==6:
         ax.set_ylabel(labels_y[2],fontsize=20)

    ax.set_xticks([])
    ax.set_yticks([])
    k+=1

plt.colorbar(im, cax=grid.cbar_axes[0]).set_label(label='Optical Density', size = 20)
```

Let us now run the reconstruction for 30 iterations, and compare the two results in the cells below.

```python
cgls = CGLS(initial = x0, operator = A, data = data,
               max_iteration = 30, update_objective_interval = 10)
cgls.run(verbose=1)
cgls_30 = cgls.solution
```

## Show CGLS reconstruction 10 iterations vs 30 iterations at the 20th energy channel and views: Axial, Coronal, Sagittal

```python
recons = [cgls_10.get_slice(channel=20, vertical=35), cgls_10.get_slice(channel=20, horizontal_y=35), cgls_10.get_slice(channel=20, horizontal_x=25),\
          cgls_30.get_slice(channel=20, vertical=35), cgls_30.get_slice(channel=20, horizontal_y=35), cgls_30.get_slice(channel=20, horizontal_x=25)]

labels_x = ["Axial", "Coronal", "Sagittal"]
labels_y = ["10 Iter.","30 Iter."]

plt.rcParams['xtick.labelsize']=15
plt.rcParams['ytick.labelsize']=15

fig = plt.figure(figsize=(12, 14))

grid = AxesGrid(fig, 111,
                nrows_ncols=(2, 3),
                axes_pad=0.05,
                cbar_mode='single',
                cbar_location='right',
                cbar_size = 0.5,
                cbar_pad=0.1
                )

k = 0

for ax in grid:
    im = ax.imshow(recons[k].as_array(), cmap="inferno", vmin = 0.0, vmax = 0.5)

    if k==0:
        ax.set_title(labels_x[0],fontsize=20)
        ax.set_ylabel(labels_y[0],fontsize=20)
    if k==1:
        ax.set_title(labels_x[1],fontsize=20)
    if k==2:
        ax.set_title(labels_x[2],fontsize=20)
    if k==3:
        ax.set_ylabel(labels_y[1],fontsize=20)

    ax.set_xticks([])
    ax.set_yticks([])
    k+=1

plt.colorbar(im, cax=grid.cbar_axes[0]).set_label(label='Optical Density', size = 20)
```

These images highlight the instability of a basic CGLS algorithm, where the **number of iterations** effectively acts as the algorithm's **regularisation parameter**. As a result, too many iterations leads to a divergence from an optimum solution, with only additional noise being contributed beyond a certain point.

Next we will look at an extension of the CGLS algorithm, where we now apply a specific regularisation factor. Here, we will use **Tikhonov** regularisation, and show how it can be applied to improve reconstruction quality in both the spatial and spectral dimensions.

## Tikhonov Regularisation for CGLS

Here we will expand upon what you learned about BlockFrameworks. In particular, we will cover both `BlockOperators` and `BlockDataContainers`, which were covered in **Week2/02_tikhonov_block_framework**.

Below gives a brief definition of each type:

`BlockDataContainer` holds datacontainers as a column vector.

`BlockOperator` is a matrix of operators.

### Setting up "Regularised CGLS"

For our regularisation, we wish to solve:

$$\underset{u}{\mathrm{argmin}}\begin{Vmatrix}\binom{\alpha \nabla}{A} u - \binom{0}{g}\end{Vmatrix}^2_2$$
With the definitions:

$\tilde{A} = \binom{\alpha \nabla}{A} \quad\Longleftrightarrow\quad$ `BlockOperator(alpha*Grad(ig),A)`
(where $\alpha$ is the regularisation parameter)

$\tilde{g} = \binom{0}{g} \quad\Longleftrightarrow\quad$ `BlockDataContainer(Grad.domain.allocate(0),data)`

this can now be recognised as a least squares problem:

$$\underset{u}{\mathrm{argmin}}\begin{Vmatrix}\tilde{A} u - \tilde{g}\end{Vmatrix}^2_2$$
and being a least squares problem, it can be solved using CGLS with $\tilde{A}$ as operator and $\tilde{g}$ as data.

### Build `BlockOperator`, $\tilde{A} = \binom{\alpha \nabla}{A}$

Using the 2D hyperspectral data and geometry constructed above, build a `BlockOperator`, $\tilde{A}$, applying a "Space-Channel" correlation. This means that our regularisation will be applied across both the spatial and spectral domains.
Choose a regularisation value around $\alpha$ = 0.1 as a starting point.

```python
# Set up operators for BlockOperator
Grad = GradientOperator(ig, correlation='SpaceChannels')

alpha = 0.1
A_block = BlockOperator(alpha*Grad,A)
```

### Build `BlockDataContainer`, $\tilde{g} = \binom{0}{g}$

Next build a `BlockDataContainer`, $\tilde{g}$, containing an array with the range of the regularising operator, $\alpha \nabla$, and our `AcquisitionData` (`data`).

```python
# Create BlockDataContainer
g_block = BlockDataContainer(Grad.domain.allocate(0),data)
```

### Initialise and Run

Now we can initialise the `BlockOperator` and run the algorithm for 10 iterations.

```python
# Initialise the BlockOperator
x0 = ig.allocate(0)

# Setup and run the regularised CGLS algorithm

cgls_reg = CGLS(x_init = x0, operator = A_block, data = g_block,
                max_iteration = 100, update_objective_interval = 2)
cgls_reg.run(10, verbose=1)
cgls_reg_10 = cgls_reg.solution
```

Use `show2D` in the cell below to compare 10 iterations of the basic CGLS algorithm, with the 10 iteration result of CGLS with Tikhonov regularisation.

```python
show2D([cgls_10,cgls_reg_10],
       slice_list=[[("channel",20),("vertical",35)]]*2,
       cmap="inferno", origin="upper",title=["CGLS 10 iterations", "Tikhonov 10 iterations"],
       fix_range=(0,0.5), size=(10,10))
```

We can see that the main effect of the **Tikhonov** regularisation is to smooth the fetaures, such that the noise in the image is suppressed. However this is still not perfect for restoring strong feature definition.

In the next section, we will look at a different algorithm: Primal-Dual Hybrid Gradient (PDHG), combined with a Total Variation (TV) regularising factor, to see if this improves our reconstructed data quality any further.

## Total Variation Regularisation in 4D Volume using PDHG

The PDHG algorithm and Total Variation regularisation were covered extensively in **Week2/03_PDHG** notebook, but below gives a brief recap of the basics behind each.

### Recap on PDHG

PDHG aims to solve problems of the form:

$$ \begin{equation} \min_{u} \mathcal{F}(K u) + \mathcal{G}(u) \label{min_problem} \end{equation} $$

In order to setup and run PDHG, we need to define the following:

 - The operator $K$.
 - The function $\mathcal{F}$ and $\mathcal{G}$.
 - Step-sizes $\sigma$ and $\tau$ such that $\sigma\tau\|K\|^{2}<1$.

Then we can setup PDHG:

`pdhg = PDHG(f = F, g = G, operator = K, tau = tau, sigma = sigma, max_iteration = maxiter)`

### Applying Total Variation Regularisation

For this notebook, we will be applying Total Variation (TV) regularisation to our PDHG algorithm. TV is a non-smooth regulariser,

$$ \underset{u}{\operatorname{argmin}} \alpha\,\mathrm{TV}(u) + \frac{1}{2} \| \mathcal{A} u - g\|^{2}_{2} + \mathbb{I}_{\{u>0\}}(u) $$

where,

- The Total Variation is taken as a `MixedL21Norm()`, such that $$\mathrm{TV}(u) = \|\nabla u \|_{2,1} = \sum \sqrt{ (\partial_{y}u)^{2} + (\partial_{x}u)^{2} }$$
- $g$ is the Acquisition data obtained from the detector
- $\mathcal{A}$ is the projection operator ( Radon transform ) that maps from an image-space to an acquisition space, i.e.,
$\mathcal{A} : X \rightarrow Y$.
- $\alpha$ is the regularising parameter that measures a trade-off between the fidelity and the regulariser terms
- $\mathbb{I}_{\{u>0\}}(u) : =
\begin{cases}
0,\quad u>0\\
\infty,\quad \mbox{otherwise}
\quad
\end{cases},\quad
$ is a positivity constraint for the minimiser $u$.

## Setting up PDHG for TV

We define K as a `BlockOperator`, containing the Gradient and Projection operator:

$$ K =
\begin{bmatrix}
\nabla\\
A
\end{bmatrix}
$$
K = `BlockOperator(Grad, A)`

The function $\mathcal{F}$, is a `BlockFunction` with

a function $\alpha\|\cdot\|_{2,1}\quad\Longleftrightarrow\quad$ `MixedL21Norm()` term that represents the Total variation regularisation ,

a function $\|\cdot -g \|_{2}^{2}\quad\Longleftrightarrow\quad$ `L2NormSquared(data)` term that represents the data fitting.

Hence, $\mathcal{F} = [f_{1}, f_{2}] \quad\Longleftrightarrow\quad $ `F = BlockFunction(MixedL21Norm(), L2NormSquared(data))`

Finally, we have the function $\mathcal{G} = \mathbb{I}_{\{u>0\}}(u) \quad\Longleftrightarrow\quad$ `G = IndicatorBox(lower=0)`

Again, we can verify that with the above setting we can express our problem into this form, for $x=u$

$$ \begin{align} \underset{u}{\operatorname{argmin}}\alpha\|\nabla u\|_{2,1} + \frac{1}{2}\|\mathcal{A} u - g\|^{2}_{2} + \mathbb{I}_{\{u>0\}}(u) \\= \underset{u}{\operatorname{argmin}} f_{1}(\nabla u) + f_{2}(\mathcal{A}u) + \mathbb{I}_{\{u>0\}}(u) \\ = \underset{u}{\operatorname{argmin}} \mathcal{F}( \begin{bmatrix} \nabla \\ \mathcal{A} \end{bmatrix}u) + \mathbb{I}_{\{u>0\}}(u)\\ = \underset{u}{\operatorname{argmin}} \mathcal{F}(Ku) + \mathcal{G}(u)\\ = \underset{x}{\operatorname{argmin}} \mathcal{F}(Kx) + \mathcal{G}(x) \end{align} $$

We will start by building our `BlockOperator`, K. This requires a `GradientOperator`, which to start with we will create with only a _Space_ correlation, therefore only applying across the spatial dimensions.

```python
# The operator K is a Block Operator that contains the gradient and the tomography operator (as created previously)
Grad = GradientOperator(ig, correlation="Space")

# Set up a BlockOperator K
K = BlockOperator(Grad,A)
```

$\mathcal{F}$ is a `BlockFunction` of a fidelity term and our TV regularising term:

- $f_{1}$ = $\alpha\|\cdot\|_{2,1}\quad\Longleftrightarrow\quad$ `MixedL21Norm()` term that represents the Total variation regularisation ,

- $f_{2}$ = $\|\cdot -g \|_{2}^{2}\quad\Longleftrightarrow\quad$ `L2NormSquared(data)` term that represents the data fitting.

Therefore as $f_{1}$ and $f_{2}$ act on each element of $K$, we end up with
$$ \mathcal{F}(Ku) = \mathcal{F}(
\begin{bmatrix}
\nabla u \\
A u
\end{bmatrix}) = ( f_{1}(\nabla u), f_{2}(Au) ) $$

Our regularising parameter, $\alpha$ adjusts the wighting of the TV regularisation. THe greater impact you want the regularisation to have on your data, the larger you make $\alpha$. Typically, the optimum value will vary for each dataset, and some trial and error may be required to find the best result.
Let us start here by using $\alpha$ = 0.004, and then build our `BlockFunction` as described above.

```python
# Select the weighting of the regularising parameter, alpha
alpha = 0.004

# List of BlockFunctions
f1 = alpha * MixedL21Norm()
f2 = 0.5 * L2NormSquared(b = data)

F = BlockFunction(f1, f2)
```

Next, we have our positivity constraint function $\mathcal{G} = \mathbb{I}_{\{u>0\}}(u) \quad\Longleftrightarrow\quad$ `G = IndicatorBox(lower=0)`.

```python
G = IndicatorBox(lower = 0)
```

Finally, we compute the operator norm of $K$ (`normK = K.norm()`), and set our step sizes, $\sigma$ and $\tau$.

```python
# Compute the operator norm for K
normK = K.norm()

# Define the step sizes sigma and tau
sigma = 1./normK
tau = 1./normK
```

Now we can setup and run the PDHG algorithm. Here it will run for 100 iterations.

Due to the increased complexity of the algorithm, combined with the reconstruction of a 4D dataset, the runtime for 100 iterations alone will be around **3 minutes**. However, we may require many more than 100 iterations to reach an optimum solution. Further, we may wish to reconstruct hundreds of energy channels, rather than just the 40 we use here.

Therefore, consideration must be taken on the number of iterations you perform based on your choice of algorithm and size of your dataset.

In the following cells, we have also provided a final reconstruction after 1000 iterations of PDHG TV, which required around 30 minutes of runtime.

```python
PDHG_TV = PDHG(f = F, g = G, operator = K, tau = tau, sigma = sigma,
            max_iteration = 100, update_objective_interval = 25)
PDHG_TV.run(verbose=1)
PDHG_TV_100_Sp = PDHG_TV.solution
```

Below, we directly compare the reconstructions of FDK with PDHG TV. As you will see, even at 100 iterations, we get a reasonable reconstructed volume, with already much improved noise suppression compared to the FDK result.

## Show FBP reconstruction vs PDHG TV reconstruction with no regularisation on channels

```python
recons = [FBP_recon_4D.get_slice(channel=20, vertical=35), PDHG_TV_100_Sp.get_slice(channel=20, horizontal_y=35),\
          FBP_recon_4D.get_slice(channel=20, horizontal_x=35), PDHG_TV_100_Sp.get_slice(channel=20, horizontal_x=35)]

labels_text = ["FDK", "PDHG TV - 100 Iter."]

plt.rcParams['xtick.labelsize']=15
plt.rcParams['ytick.labelsize']=15

fig = plt.figure(figsize=(12, 14))

grid = AxesGrid(fig, 111,
                nrows_ncols=(2, 2),
                axes_pad=0.05,
                cbar_mode='single',
                cbar_location='right',
                cbar_size = 0.5,
                cbar_pad=0.1
                )

k = 0

for ax in grid:
    im = ax.imshow(recons[k].as_array(), cmap="inferno", vmin = 0.0, vmax = 0.5)

    if k==0:
        ax.set_title(labels_text[0],fontsize=20)
    if k==1:
        ax.set_title(labels_text[1],fontsize=20)

    ax.set_xticks([])
    ax.set_yticks([])
    k+=1

plt.colorbar(im, cax=grid.cbar_axes[0]).set_label(label='Optical Density', size = 20)
```

###  PDHG TV: 1000 Iterations

You may notice that, for 100 iterations, the PDHG TV result looks slightly __blocky__ in places, despite significant noise suppression. This may be because 100 iterations is insufficient for convergence, therefore we include below a reconstructed dataset after 1000 iterations, stored as a Nexus (`.nxs`) file.

We load in the 1000 iterations result using the `NEXUSDataReader()`. We can now directly compare this result to all our previous reconstructions, including our Tikhonov CGLS result, and the PDHG TV result for 100 iterations.

As you can see, 1000 iterations were sufficient for convergence of the PDHG TV reconstruction. We now see smoothing across all spatial features, reducing all noise such that boundaries of soft tissue features are more clearly defined. In particular, the noisy __haze__ around the main soft tissue is considerably suppressed.

```python
reader = NEXUSDataReader(file_name = "/mnt/materials/SIRF/Fully3D/CIL/Lizard/PDHG_TV_1000_Sp_alpha_0.004.nxs")
PDHG_TV_1000_Sp = reader.read()

recons = [FBP_recon_4D.get_slice(channel=20, vertical=35), cgls_reg_10.get_slice(channel=20, vertical=35),\
          PDHG_TV_100_Sp.get_slice(channel=20, vertical=35), PDHG_TV_1000_Sp.get_slice(channel=20, vertical=35),\
          FBP_recon_4D.get_slice(channel=20, horizontal_x=25), cgls_reg_10.get_slice(channel=20, horizontal_x=25),\
          PDHG_TV_100_Sp.get_slice(channel=20, horizontal_x=25), PDHG_TV_1000_Sp.get_slice(channel=20, horizontal_x=25)]

labels_text = ["FDK","CGLS Tik.- 10 Iter.","PDHG TV - 100 Iter.", "PDHG TV - 1000 Iter."]

plt.rcParams['xtick.labelsize']=15
plt.rcParams['ytick.labelsize']=15

fig = plt.figure(figsize=(20, 14))

grid = AxesGrid(fig, 111,
                nrows_ncols=(2, 4),
                axes_pad=0.05,
                cbar_mode='single',
                cbar_location='right',
                cbar_size = 0.5,
                cbar_pad=0.1
                )

k = 0

for ax in grid:
    im = ax.imshow(recons[k].as_array(), cmap="inferno", vmin = 0.0, vmax = 0.5)

    if k==0:
        ax.set_title(labels_text[0],fontsize=20)
    if k==1:
        ax.set_title(labels_text[1],fontsize=20)
    if k==2:
        ax.set_title(labels_text[2],fontsize=20)
    if k==3:
        ax.set_title(labels_text[3],fontsize=20)

    ax.set_xticks([])
    ax.set_yticks([])
    k+=1

plt.colorbar(im, cax=grid.cbar_axes[0]).set_label(label='Optical Density', size = 20)
```

## Exercise: Set up PDHG with Space-Channel TV Regularisation

We can make some adjustments to our framework to see how these affect the reconstruction.
Our gradient component is capable of smoothing our reconstruction both in the spatial domain, but also in the energy domain, by using neighbouring channels to aid the iterative reconstruction process. We will now test this by setting up PDHG with a Space-Channel TV regularisation.

**Step 1)** Adjust the `GradientOperator` to correlate in both the spatial and spectral domains. Then build your `BlockOperator` K using the `ProjectionOperator` we created earlier for our CGLS reconstruction.

Hint: For TV regularisation, we have the ability to split our regularisation, such that we can adjust the weighting of TV for the spatial and spectral dimensions separately. If you want to do this, add a third argument to your `GradientOperator` called `split=True`.

```python
# Adjust correlation of gradient component
Grad = GradientOperator(ig, correlation = "SpaceChannels", split=True)

# Construct new BlockOperator
K = BlockOperator(Grad, A)
```

**Step 2)** Set the regularisation parameters.

If you decided to split your TV regularisation, then now instead of **one** parameter, we now need to create **two**, $\alpha$ and $\beta$, which will act on the spatial and spectral domains respectively.

For our original PDHG reconstruction, our $\alpha$ parameter was:
$\alpha$ = 0.004.

Now that we have two parameters, $\alpha$ across the spatial domain and $\beta$ across the energy domain, you can try different combinations, for example:
$\alpha$ = 0.003,
$\beta$ = 0.5

```python
# Define regularising parameter alpha
alpha = 0.003
beta = 0.5
```

**Step 3)** Build the `BlockFunction`

Previously our `BlockFunction` contained two terms: the TV regularisation, and the data fitting. As our TV term is now split, we must build this term as a `BlockFunction` first, combining the spatial and spectral regularisation elements, before combining this with our data fitting term.

Our first term will now look like the following:

$$ \mathcal{f_{1}} = \begin{bmatrix}
\beta\|\cdot\|_{1} \\
\alpha\|\cdot\|_{2,1}
\end{bmatrix}
$$
So we use the `L1Norm()` for our $\beta$ parameter, and the `MixedL21Norm()` for our $\alpha$ parameter.

Then we can simply combine this new term with the `L2NormSquared` term we used previously for our data fitting, and build our final `BlockFunction`, F.

Have a go at building this, referring back to our past constructions of `BlockFunctions` if you need.

```python
# Fidelity terms f1,f2
f1 = BlockFunction(beta*L1Norm(), alpha * MixedL21Norm())
f2 = 0.5 * L2NormSquared(b = data)

# Construct BlockFunction
F = BlockFunction(f1,f2)
```

**Step 4)** Apply a positivity constraint function, G

This will be identical to before, enabling us to restrict our reconstruction to non-negative values.

```python
# Positivity Constraint Function
G = IndicatorBox(lower = 0)
```

**Step 5)** Compute our operator norm, K, and define the step sizes, $\sigma$ and $\tau$.

These will again be the same as we have used previously.

```python
# Compute the operator norm for K
normK = K.norm()

# Define the step sizes sigma and tau
sigma = 1./normK
tau = 1./normK
```

**Step 6)** Run the reconstruction for 100 iterations.

We have provided below the skeleton of the `PDHG` algorithm, you just need to finish adding in the correct arguments, and then run the code!

```python
PDHG_TV_SpCh = PDHG(f = F, g = G, operator = K, tau = tau, sigma = sigma,
            max_iteration = 100, update_objective_interval = 25)
PDHG_TV_SpCh.run(verbose=1)
PDHG_TV_100_SpCh = PDHG_TV_SpCh.solution
```

Use the `show2D` function below to see how your reconstruction looks in each view. Feel free to adjust the channel or slice numbers as well.

```python
show2D(PDHG_TV_100_SpCh,
       slice_list=[[("channel",20),("vertical",35)],
                   [("channel",20),("horizontal_y",35)],
                   [("channel",20),("horizontal_x",25)]],
                   cmap="inferno", num_cols=3,
                   origin="upper", fix_range=(0,0.5))
```

We can directly compare the two types of PDHG reconstruction using `islicer`. Once more, we have provided a 1000 iteration reconstruction of the Space-Channel regularised result as a Nexus file. As you move across energy channels, see if there are any major differences between the constructions.
 1) PDHG TV using a Spatial correlation for 1000 iterations
 2) PDHG TV using a Space-Channel correlation for 1000 iterations

**Note:** If you wish, you can also compare the 100 iteration results for each method, simply by adding extra lines in the cell below, such as:
`islicer(PDHG_TV_100_SpCh.get_slice(vertical=35), direction='channel', title = 'Sinogram Channel', cmap='inferno',minmax=(0.0,0.5))`

Look for any differences between the reconstructions as you move across energy channels. For example, do you notice the smoother transition from one channel to the next in our **Space-Channel** reconstructed dataset? This is because the additional spectral regularisation, with its own weighted parameter, helped to reduce noise fluctuations from channel to channel.

```python
reader = NEXUSDataReader(file_name = "/mnt/materials/SIRF/Fully3D/CIL/Lizard/PDHG_TV_1000_Sp_alpha_0.004.nxs")
PDHG_TV_1000_Sp = reader.read()

reader = NEXUSDataReader(file_name = "/mnt/materials/SIRF/Fully3D/CIL/Lizard/PDHG_TV_1000_SpCh_alpha_0.003_beta_0.5.nxs")
PDHG_TV_1000_SpCh = reader.read()

h1 = islicer(PDHG_TV_1000_Sp.get_slice(vertical=35), slice_number = 28, direction='channel', title = 'PDHG TV (Space) Channel', cmap='inferno',minmax=(0.0,0.5), origin="upper")
h2 = islicer(PDHG_TV_1000_SpCh.get_slice(vertical=35), slice_number = 28, direction='channel', title = 'PDHG TV (Space & Channel) Channel', cmap='inferno',minmax=(0.0,0.5),origin="upper")
link_islicer(h1,h2)
```

## Comparison of 4D reconstruction algorithms

In the cell below, we can compare all reconstruction algorithms we have tried so far:
1) FDK
2) Basic CGLS
3) CGLS with Tikhonov Regularisation ($\alpha$ = 0.1)
4) PDHG with Space-Channel Total Variation Regularisation ($\alpha$ = 0.003, $\beta$ = 0.5)

```python
recons = [FBP_recon_4D.get_slice(channel=20, vertical=35), cgls_reg_10.get_slice(channel=20, vertical=35),\
          cgls_reg_10.get_slice(channel=20, vertical=35), PDHG_TV_1000_SpCh.get_slice(channel=20, vertical=35),\
          FBP_recon_4D.get_slice(channel=20, horizontal_x=25), cgls_reg_10.get_slice(channel=20, horizontal_x=25),\
          cgls_reg_10.get_slice(channel=20, horizontal_x=25), PDHG_TV_1000_SpCh.get_slice(channel=20, horizontal_x=25)]

labels_text = ["FDK","CGLS - 10 Iter.","Tikhonov CGLS - 10 Iter.", "SpCh. PDHG TV - 1000 Iter."]

plt.rcParams['xtick.labelsize']=15
plt.rcParams['ytick.labelsize']=15

fig = plt.figure(figsize=(20, 14))

grid = AxesGrid(fig, 111,
                nrows_ncols=(2, 4),
                axes_pad=0.05,
                cbar_mode='single',
                cbar_location='right',
                cbar_size = 0.5,
                cbar_pad=0.1
                )

k = 0

for ax in grid:
    im = ax.imshow(recons[k].as_array(), cmap="inferno", vmin = 0.0, vmax = 0.5)

    if k==0:
        ax.set_title(labels_text[0],fontsize=20)
    if k==1:
        ax.set_title(labels_text[1],fontsize=20)
    if k==2:
        ax.set_title(labels_text[2],fontsize=20)
    if k==3:
        ax.set_title(labels_text[3],fontsize=20)

    ax.set_xticks([])
    ax.set_yticks([])
    k+=1

plt.colorbar(im, cax=grid.cbar_axes[0]).set_label(label='Optical Density', size = 20)
```

# Spatial/Energy Profiles

One way we can directly observe the effect of applying Spatial/Spectral correlation to our algorithms is through the use of pixel profiles across our reconstructed images in each domain respectively.

By plotting signal values across spatial pixels, or across energy channels, the smoothing effects of our regularisation are seen. We will demonstrate each of these below.

## Extracting Spatial Profiles

By identifying a region of our reconstructed slices were the Iodine signals are strong, we can plot along the pixels in this region and compare the line profiles for each algorithm covered here.

Given the strong signals given by the eyes of the lizard head sample, we will look across this pixel row for a single channel, given by slice indices of:
 - `vertical = 35`
 - `horizontal_y = 35`
 - `channel = 20`

**Feel free to adjust these values in the cell below if you wish to explore different regions of the reconstructed images.**

```python
plt.figure(figsize=[14,8])

# Extract line profiles for each algorithm
plt.plot(FBP_recon_4D.get_slice(vertical = 35, horizontal_y = 35, channel = 20).as_array(),label='FDK')
plt.plot(cgls_10.get_slice(vertical = 35, horizontal_y = 35, channel = 20).as_array(),label='CGLS')
plt.plot(PDHG_TV_1000_Sp.get_slice(vertical = 35, horizontal_y = 35, channel = 20).as_array(),label='Space PDHG TV')
plt.plot(PDHG_TV_1000_SpCh.get_slice(vertical = 35, horizontal_y = 35, channel = 20).as_array(),label='Space-Channel PDHG TV')

plt.ylim((-0.1,0.7))
# Label and add key
plt.xlabel('Pixels X', fontsize=20)
plt.ylabel('Optical Density (no units)', fontsize=20)
plt.legend(loc='upper right',fontsize=15)
plt.show()
```

We can see the reduced noise fluctuations for both PDHG TV reconstructions, compared to the significant noise seen in both FDK and CGLS reconstructions. **Note** that the parameters used for both TV reconstructions are not optimised.

## Extracting Energy Profiles

Once we have chosen and optimised our reconstruction routine, we can exploit this additional, energy dimension further by identifying the spectral signals that lie within.

First we can plot out energy profiles for any area within our reconstructed slices. This allows us to find, for instance, an absorption edge.
Iodine has a known absorption 'K-edge' of 33.169 keV, so we can test the accuracy of reconstruction algorithms by seeing where this edge falls in each case. A plot of an idealised, theoretical Iodine K-edge is shown below, occurring precisely at 33.169 keV.

Given that the data will be noisiest for a single voxel, we will extract spectral profiles from one voxel within the eye of the lizard, allowing us to compare the noise level of each reconstruction along the energy axis.

```python
# Select single voxel in lizard eye region

# FDK Recon
fdk_vox = FBP_recon_4D.as_array()[:,35,35,55]

# 3D multi-channel 10 iteration CGLS
cgls_vox = cgls_reg_10.as_array()[:,35,35,55]

# 3D multi-channel space correlated PDHG
pdhg_Sp_vox = PDHG_TV_1000_Sp.as_array()[:,35,35,55]

# 3D multi-channel space-channel correlated PDHG TV
pdhgSpCh_vox = PDHG_TV_1000_SpCh.as_array()[:,35,35,55]
```

```python
# Set channel numbers for reduced dataset (here 100 to 140)
channel_no = np.linspace(100,140,num_channels)

plt.figure(figsize=(12,8))

plt.plot(ekeV_crop,fdk_vox)
plt.plot(ekeV_crop,cgls_vox)
plt.plot(ekeV_crop,pdhg_Sp_vox)
plt.plot(ekeV_crop,pdhgSpCh_vox)

plt.plot((33.169,33.169),plt.ylim(0.1,0.6))

plt.legend(labels=['FDK', 'Tik. CGLS', 'Space PDHG TV',
                   'Sp-Ch. PDHG TV', 'I K-edge 33.169 keV'],fontsize=15)
plt.xlim([29,39])

plt.ylabel('Optical Density (no units)',fontsize=20)
plt.xlabel('Energy (keV)',fontsize=20)
```

From our plots, we can see all algorithms experience a sharp rise in signal value due to the iodine absorption K-edge. Compared to the theoretical value line, each of the cases match well with the expected position of the edge.
We can very clearly see the impact of the spectral regularisation using PDHG TV, with all noise entirely removed, and an almost perfectly smooth transition across energy channels. We still clearly see the rising edge at the iodine K-edge position. The use of spectral correlation enforces our operator to use the previous energy channel as a reference point for reconstructing data in the next, resulting in a smoother transition across channels.

## Constructing Elemental Maps

Once we know have identified the position of this edge in our energy profile, we can narrow our dataset and produce an **Iodine map**. That is, we can select only the energy channels occupied by the absorption edge, so all reconstructed signal is now due only to the iodine contrast agent. This method is known as **'K-edge subtraction'**, which you can read about in more detail in papers such as that by [C.K.Egan *et al*, 2015](https://www.nature.com/articles/srep15979#).
A basic concept is shown below for an energy profile plot. The hashed area highlights the energy range we are interested in, corresponding to the absorption edge.

Based on our plots, we will estimate the start and end of the edge to occur at approximately 32 keV and 34.5 keV respectively.

```python
# Calculate energy channels corresponding to start and end of the K-edge
e_keV = np.array([32,34.5])
# Convert back to channel number based on calibration values
# (these are specific to the dataset, used here just to show the method.)
channel_no = ((e_keV-0.8575)/0.2786)-100

# Display the channels corresponding to the start and end of the K-edge (within our truncated dataset)
print("Start of edge = channel",int(channel_no[0]))
print("End of edge = channel",int(channel_no[1]))
```

Now we need to average over all the channels in this channel range for each reconstruction, isolating the signal around the K-edge that only corresponds to Iodine. For simplicity, we will perform this over a single slice, in this case the 25th slice in the `horizontal_x` plane that we have used previously for comparing images.

```python
# 4D FBP
# Sum over all pixels for channels of interest
FBP_chan = np.mean(FBP_recon_4D.as_array()[int(channel_no[0]):int(channel_no[1]),:,:,25],0)

# 4D Tikhonov CGLS
CGLS_Tik_chan = np.mean(cgls_reg_10.as_array()[int(channel_no[0]):int(channel_no[1]),:,:,25],0)

# 4D Space-Channel PDHG TV
PDHG_SpCh_chan = np.mean(PDHG_TV_1000_Sp.as_array()[int(channel_no[0]):int(channel_no[1]),:,:,25],0)
```

We can use `islicer` once more to compare our results, which shows the iodine maps for each reconstruction. By stacking the results as an array, `islicer` allows you to switch from one reconstruction to the next. You may notice that a lot of the same signal is present to before! This is because the iodine is a high concentration contrast agent which diffuses into nearly all soft tissue structures, increasing the attenuation levels of many areas except the bone regions.

```python
# Collect together iodine maps
stack_maps = (np.array([FBP_chan, CGLS_Tik_chan, PDHG_SpCh_chan]))
titles = ['Simple CGLS', 'Regularised CGLS', 'Space-Channel PDHG TV']
islicer(stack_maps, title=titles, direction=0, cmap='inferno',minmax = (0.0,0.3))
```

Conclusions

This notebook focused on bringing together the core elements of the CIL framework you have learned in the previous notebooks, and seeing how these may be applied to a hyperspectral dataset.

We looked at the key differences in building Image/Acquisition Geometries and Operators for hyperspectral datasets.

We have covered three key algorithms (FDK, CGLS and PDHG) and shown the various ways in which we can build them, both with and without the addition of regularisation terms. We have also explored how each type of regularisation impacts our resulting reconstruction, highlighting why optimisation of our regularisation parameters is so important to produce the highest quality data. This is further highlighted through the extraction of both spatial and spectral profiles, as well as extraction of elemental maps using spectral analyses like K-edge subtraction. For more information on hyperspectral tomography reconstruction we refer to [Warr et al](https://arxiv.org/pdf/2103.04796), [Ametova et al](https://iopscience.iop.org/article/10.1088/1361-6463/ac02f9/pdf) and [Papoutsellis et al](https://arxiv.org/pdf/2102.06126).

**Click the link below to see the potential for full volume, white-beam reconstructions using both CGLS (Left) and PDHG TV (Right)**
[4D Lizard Full Volume Reconstruction](Images/Lizard_gif.gif "segment")

```python

```


# Goal:

* Modeling a theoretical diffusive boundary layer (DBL).
  * A DBL may be contributing to 'smearing' observed in 16S rRNA MiSeq data from real experiments.

# Init

```python
import os
import numpy as np
from scipy.integrate import quad
%load_ext rpy2.ipython

workDir = '/home/nick/notebook/SIPSim/dev/theory/'
```

    The rpy2.ipython extension is already loaded. To reload it, use:
      %reload_ext rpy2.ipython

```r
%%R
library(readxl)
library(dplyr)
library(tidyr)
library(ggplot2)
library(rootSolve)
```

```python
if not os.path.isdir(workDir):
    os.makedirs(workDir)
%cd $workDir
```

    /home/nick/notebook/SIPSim/dev/theory

# Setting parameters

```r
%%R
# tube characteristics (cm)
tube_diam = 1.3
tube_height = 4.8
tube_round_bottom_height = 0.65
tube_capacity__ml = 4.7
tube_composition = 'polypropylene'

# rotor (cm)
rotor_id = 'TLA-110'
r_min = 2.6
r_ave = 3.72
r_max = 4.85
frac_tube_angle = 90

# cfg run
## rpm of run
rpm = 55000
## angular velocity (w^2)
angular_velocity = 17545933.74
## average particle density
ave_gradient_density = 1.70
## beta^o
BetaO = 1.14e9  # CsCl at density of 1.70
## position of particle at equilibrium
particle_at_eq = 3.78
## max 13C shift
max_13C_shift_in_BD = 0.036
## min BD (that we care about)
min_GC = 13.5
min_BD = min_GC/100.0 * 0.098 + 1.66
## max BD (that we care about)
max_GC = 80
max_BD = max_GC / 100.0 * 0.098 + 1.66    # 80.0% G+C
max_BD = max_BD + max_13C_shift_in_BD

# diffusive boundary layer (DBL)
DBL_size_range__micron = c(10,100)

# misc
fraction_vol__cm3 = 0.1
```

```r
%%R
# rotor angle
## sin(x) = opp / hypo
## x = sin**-1(opp/hypo)

rad2deg = function(rad) {
    return((180 * rad) / pi)
}
deg2rad = function(deg){
    return(deg * pi / 180)
}

x = r_max - r_min
hyp = tube_height
rotor_tube_angle = rad2deg(asin(x / hyp))
cat("Tube angle from axis of rotation:", rotor_tube_angle, "\n")
```

    Tube angle from axis of rotation: 27.95319

```r
%%R
rad2deg = function(rad) {
    return((180 * rad) / pi)
}
deg2rad = function(deg){
    return(deg * pi / 180)
}

x = r_max - r_min
hyp = tube_height
rotor_tube_angle = rad2deg(asin(x / hyp))
cat("Tube angle from axis of rotation:", rotor_tube_angle, "\n")
```

    Tube angle from axis of rotation: 27.95319

```r
%%R
# calc tube angle from tube params
calc_tube_angle = function(r_min, r_max, tube_height){
    x = r_max - r_min
    hyp = tube_height
    rotor_angle = rad2deg(asin(x / hyp))
    return(rotor_angle)
    }

# test
## angled tube
ret = calc_tube_angle(r_min, r_max, tube_height)
print(ret)
## vertical tube
r_min_v = 7.47
r_max_v = 8.79
ret = calc_tube_angle(r_min_v, r_max_v, tube_height)
print(ret)
```

    [1] 27.95319
    [1] 15.96201

```r
%%R
# isoconcentration point
## Formula 6.7 in Birnine and Rickwood 1978
I = sqrt((r_min**2 + r_min * r_max + r_max**2)/3)

cat('Isoconcentration point:', I, '(cm)\n')
```

    Isoconcentration point: 3.781204 (cm)

# ratio of DBL size : fraction size as a function of DBL size

## Rough approximation

```r
%%R

DBL_rel_size = function(DBL_size, tube_diam, frac_size){
    # sizes in cm
    tube_radius = tube_diam / 2
    frac_vol = pi * tube_radius**2 * frac_size
    nonDBL_vol = pi * (tube_radius - DBL_size)**2 * frac_size
    DBL_vol = frac_vol - nonDBL_vol
    DBL_to_frac = DBL_vol / frac_vol * 100
    return(DBL_to_frac)
}

# in cm
frac_size = 0.01
tube_diam = 1.3
#DBL_size = 0.01
DBL_sizes = seq(0, 0.07, 0.005)

DBL_perc = sapply(DBL_sizes, DBL_rel_size, tube_diam=tube_diam, frac_size=frac_size)

df = data.frame('DBL_size' = DBL_sizes, 'DBL_perc' = DBL_perc)

ggplot(df, aes(DBL_size, DBL_perc)) +
    geom_point() +
    geom_line() +
    labs(x='DBL size (cm)', y='% tube volume that is DBL') +
    theme_bw() +
    theme(
        text = element_text(size=16)
    )
```

#### Notes

* Assuming cfg tube is just a cylinder

# Determining DBL from fragment G+C content

* fragment GC -->
  * BD (diffusive boundary layer) -->
    * angled tube position of DBL -->
      * vertical tube position range of DBL (min, mid, max)

# Functions for calculating DBL

## GC to BD

```r
%%R
GC2BD = function(GC){
    # GC = percent G+C
    GC / 100.0 * 0.098 + 1.66
}

# test
GC = seq(0, 100, 10)
sapply(GC, GC2BD)
```

     [1] 1.6600 1.6698 1.6796 1.6894 1.6992 1.7090 1.7188 1.7286 1.7384 1.7482
    [11] 1.7580

## BD to distance from the axis of rotation

\begin{align}
x = \sqrt{( ({\rho}-p_m) \frac{2B^{\circ}}{w^2}) + r_c^2}
\end{align}

```r
%%R
BD2distFromAxis = function(BD, D, BetaO, w2, I){
    # converting BD to distance from axis of rotation
    # BD  = density at a given radius
    # w^2 = angular velocity
    # \beta^o = beta coef
    # I = isocencentration point (cm)
    # D = average density of gradient
    sqrt(((BD-D)*2*BetaO/w2) + I^2)
}

# test
min_BD_r = BD2distFromAxis(min_BD, ave_gradient_density, BetaO, angular_velocity, I)
max_BD_r = BD2distFromAxis(max_BD, ave_gradient_density, BetaO, angular_velocity, I)

cat('radius range for BD-min to BD-max: ', min_BD_r, 'to', max_BD_r, '\n')
```

    radius range for BD-min to BD-max:  3.289207 to 4.895445

## distance from axis of rotation to tube height of BD 'band'

* The band is angled in the tube, so the BD band in the gradient (angled tube) will touch the wall of the tube at a min/max height of h1 and h2. This function determines those tube height values.

\begin{align}
    y_t =
\end{align}

* x = a distance from the axis of rotation
* r = radius of cfg tube
* D = max tube distance from axis of rotation
* A = angle of tube to axis of rotation (degrees)

```r
%%R

distFromAxis2angledTubePos = function(x, r, D, A){
  # converting distance from axis of rotation to cfg tube position (min & max of tube height)
  # x = a distance from the axis of rotation
  # r = radius of cfg tube
  # D = max tube distance from axis of rotation
  # A = angle of tube to axis of rotation (degrees)

  # Equation for finding the lower point of the band
  if(x >= D-(r*aspace::cos_d(A))-r) {
    d = x-(D-r)
    a = A-aspace::asin_d(d/r)
    LowH = r-r*aspace::cos_d(a)
    #print(LowH)                ## This band will be in the rounded part
  }else{
    d = D-(r*aspace::cos_d(A))-r-x
    hc = d/aspace::sin_d(A)
    LowH = r+hc
  # print(LowH)                 ## This band will be in the cylinder part
  }

  # Equation for finding the upper band
  if(x > D-(r-r*aspace::cos_d(A))) {
    d = x-(D-r)
    a = (A)-(180-aspace::asin_d(d/r))
    HighH = r-r*aspace::cos_d(a)
    #print(HighH)                ## This band will be in the rounded part
  }else{
    d = D-(r-r*aspace::cos_d(A))-x
  hc = d/aspace::sin_d(A)
  HighH = r+hc
  #print(HighH)                ## This band will be in the cylinder part
  }

  return(c(LowH, HighH))
}

# test
r = 0.65   # radius of tube (cm)
D = 4.85   # distance from axis of rotation to furthest part of tube (cm)
A = 27.95  # angle of tube to axis of rotation (degrees)
x = 3.5    # some distance from axis of rotation (from equation)

pos = distFromAxis2angledTubePos(x, r, D, A)
pos %>% print
delta = pos[2] - pos[1]
delta %>% print
```

    [1] 0.9184398 3.3685399
    [1] 2.4501

### Python version

```python
sin_d = lambda d : np.sin(np.deg2rad(d))
cos_d = lambda d : np.cos(np.deg2rad(d))
asin_d = lambda x : np.arcsin(x) * 180/np.pi  #np.arcsin(np.deg2rad(d))
acos_d = lambda x : np.arccos(x) * 180/np.pi  #np.arccos(np.deg2rad(d))

def axisDist2angledTubePos(x, tube_radius, r_max, A):
    if np.isnan(x):
        return (x, x)

    if(x >= r_max - (tube_radius * cos_d(A)) - tube_radius):
        # band in rounded bottom of cfg tube
        d = x - (r_max - tube_radius)
        a = A - asin_d(d / tube_radius)
        LowH = tube_radius - tube_radius * cos_d(a)
        #print LowH
    else:
        # band in cylinder of cfg tube
        d = r_max - (tube_radius * cos_d(A)) - tube_radius - x
        h_c = d/sin_d(A)
        LowH = tube_radius + h_c
        # print LowH

    if(x > r_max - (tube_radius - tube_radius * cos_d(A))):
        # Equation for finding the upper band
        d = x - (r_max - tube_radius)
        a = A - (180 - asin_d(d/tube_radius))
        HighH = tube_radius - tube_radius * cos_d(a)
        #print HighH
    else:
        # This band will be in the cylinder part
        d = r_max - (tube_radius - tube_radius * cos_d(A)) - x
        h_c = d/sin_d(A)
        HighH = tube_radius + h_c
        #print(HighH)
    return(LowH, HighH)

# test
r = 0.65   # radius of tube (cm)
D = 4.85   # distance from axis of rotation to furthest part of tube (cm)
A = 27.95  # angle of tube to axis of rotation (degrees)
x = 3.5    # some distance from axis of rotation (from equation)

ret = axisDist2angledTubePos(x, r, D, A)
print(ret)
delta = ret[1] - ret[0]
print(delta)
```

    (0.91843983608861524, 3.3685399170895249)
    2.450100081

## Converting distance from axis of rotation to angled tube volume

### Python

```python
def _SphVol(t, r, p2, R12):
    # helper function for axisDist2angledTubeVol
    v1 = t*((2*r)-t)/2
    v2 = 2*np.pi*((p2-t)/R12)
    v3 = np.sin(2*np.pi*((p2-t)/R12))
    return v1 * (v2 - v3)

def _CylWedVol(t, r, b, h):
    # helper function for axisDist2angledTubeVol
    return 2*(h*(t-r+b)/ b) * np.sqrt(r**2-t**2)

def axisDist2angledTubeVol(x, r, D, A):
    """Convert distance from axis of rotation to volume of gradient
    where the BD is >= to the provided BD.

    Parameters
    ----------
    x : float
        distance from axis of rotation (cm)
    r : float
        cfg tube radius (cm)
    D : float
        max distance from axis of rotation (cm)
    A : float
        cdf tube angle in rotor (degrees)

    Returns
    -------
    volume (ml) occupied by gradient heavier or as heavy as at that point.
    Note: nan returned if x = nan
    """
    # return nan if nan provided
    if np.isnan(x):
        return x

    a = np.deg2rad(A)
    p1 = r-(r*np.cos(a))
    p2 = r+(r*np.cos(a))
    R12 = p2-p1
    d = D-x
    D1 = D-p1
    D2 = D-p2

    if x < D2:
        if a == 0:
            z = 1
        else:
            z = np.sin(a)
        h1 = (D2-x)/z
        h2 = (D1-x)/z
        volume1 = (2/3.0)*np.pi*r**3
        volume2 = (0.5)*np.pi*r**2*(h1+h2)
        volume = volume1+volume2
    elif D1 >= x >= D2:
        volume1 = (1/3.0)*np.pi*p1**2*(3*r-p1)
        volume2 = quad(_SphVol, p1, d, args=(r, p2, R12))
        b = (d-p1)/np.cos(a)
        if a == 0:
            h = b
        else:
            h = b/np.tan(a)
        volume3 = quad(_CylWedVol, r-b, r, args=(r, b, h))
        volume = volume1+volume2[0]+volume3[0]
    elif D >= x > D1:
        volume = (1/3.0)*np.pi*d**2*(3*r-d)
    elif x > D:
        volume = np.nan
    else:
        volume = np.nan

    # status
    if np.isnan(volume):
        lmsg = 'axisDist2angledTubeVol: nan returned for x value: {}\n'
        sys.stderr.write(lmsg.format(x))

    return volume

# test
## fixed-angle rotor
r = 0.65   # radius of tube (cm)
D = 4.85   # distance from axis of rotation to furthest part of tube
A = 27.95  # angle of tube to axis of rotation (degrees)
x = 3.5    # some distance from axis of rotation (from equation)

ret = axisDist2angledTubeVol(x, r, D, A)
print(ret)

## vertical rotor
#x = 7.66
x = 8.5
r = 0.65
D = 8.79
A = 0

ret = axisDist2angledTubeVol(x, r, D, A)
print(ret)
```

    2.55751656335
    0.168563196908

## Converting tube volume to vertical tube height

### Python

```python
# converting cylinder volume to height
def cylVol2height(v, r):
    # v = volume (ml)
    # r = tube radius (cm)
    h = v / (np.pi * r**2)
    return h

# test
cylVol2height(0.1, 0.65)
```

    0.07533961803166643

```python
# converting sphere cap volume to sphere height
from scipy import optimize
def sphereCapVol2height(v, r):
    # v = volume (ml)
    # r = tube radius (cm)
    # h**3 - 3*r*h**2 + (3v / pi) = 0
    f = lambda x : x**3 - 3*r*x**2 + 3*v/np.pi
    try:
        root = optimize.brentq(f, 0, r*2, maxiter=1000)
    except ValueError:
        msg = 'WARNING: not roots for volume {}\n'
        sys.stderr.write(msg.format(v))
        root = np.nan
    return(root)

# test
sphereCapVol2heightV = np.vectorize(sphereCapVol2height)
heights = np.arange(0, 0.65**2, 0.1)
sphereCapVol2heightV(heights, 0.65)
```

    array([ 0.        ,  0.23603985,  0.34495023,  0.43482548,  0.51613246])

```python
# convert liquid volume in vertical cfg tube to tube height
def tubeVol2height(v, r):
    # v = volume (ml)
    # r = tube radius (cm)
    sphere_cap_vol = (4/3 * np.pi * r**3)/2

    if v <= sphere_cap_vol:
        # height does not extend to cylinder
        h = sphereCapVol2height(v, r)
    else:
        # height = sphere_cap + cylinder
        sphere_cap_height = sphereCapVol2height(sphere_cap_vol, r)
        h =  sphere_cap_height + cylVol2height(v - sphere_cap_vol, r)

    return(h)

# test
vol = 0.1   # 100 ul
vols = np.arange(0, 4+vol, vol)
tubeVol2heightV = np.vectorize(tubeVol2height)
tubeVol2heightV(vols, r=0.65)
```

    array([ 0.        ,  0.23603985,  0.34495023,  0.43482548,  0.51613246,
            0.59233273,  0.66767235,  0.74301197,  0.81835158,  0.8936912 ,
            0.96903082,  1.04437044,  1.11971006,  1.19504967,  1.27038929,
            1.34572891,  1.42106853,  1.49640815,  1.57174776,  1.64708738,
            1.722427  ,  1.79776662,  1.87310624,  1.94844585,  2.02378547,
            2.09912509,  2.17446471,  2.24980433,  2.32514394,  2.40048356,
            2.47582318,  2.5511628 ,  2.62650242,  2.70184203,  2.77718165,
            2.85252127,  2.92786089,  3.00320051,  3.07854012,  3.15387974,
            3.22921936])

# Test run of SIPSim DBL

## Angled rotor

```python
runDir = '/home/nick/notebook/SIPSim/t/genome100/'
!cd $runDir; \
    SIPSim DBL \
        --np 4 \
        ampFrag_skewN90-25-n5-nS_dif_kde.pkl \
        > ampFrag_skewN90-25-n5-nS_dif_DBL_kde.pkl
```

    DBL_index file written: "DBL_index.txt"
    Processing: Nocardia_nova_SH22a
    Processing: Myxococcus_stipitatus_DSM_14675
    Processing: Aeromonas_salmonicida_subsp_salmonicida_A449
    Processing: Bradyrhizobium_diazoefficiens_USDA_110
    Processing: Geobacillus_thermoleovorans_CCB_US3_UF5
    Processing: Sulfurihydrogenibium_azorense_Az-Fu1
    Processing: Vibrio_furnissii_NCTC_11218
    Processing: Thermincola_potens_JR
    Processing: Spiroplasma_culicicola_AES-1
    Processing: Kangiella_koreensis_DSM_16069
    Processing: Oceanithermus_profundus_DSM_14977
    Processing: Haemophilus_influenzae_PittEE
    Processing: Mycobacterium_abscessus_subsp_bolletii_50594
    Processing: Mycoplasma_bovoculi_M165_69
    Processing: Leptospira_borgpetersenii_serovar_Hardjo-bovis_str_L550
    Processing: Aeromonas_veronii_B565
    Processing: Capnocytophaga_canimorsus_Cc5
    Processing: Mycoplasma_yeatsii_GM274B
    Processing: Kosmotoga_olearia_TBF_19_5_1
    Processing: Planktomarina_temperata_RCA23
    Processing: Pseudonocardia_dioxanivorans_CB1190
    Processing: Dehalogenimonas_lykanthroporepellens_BL-DC-9
    Processing: Bacillus_subtilis_PY79
    Processing: Candidatus_Liberibacter_asiaticus_str_Ishi-1
    Processing: Clostridium_perfringens_str_13
    Processing: Caldicellulosiruptor_saccharolyticus_DSM_8903
    Processing: Vibrio_tasmaniensis_LGP32
    Processing: Desulfosporosinus_orientis_DSM_765
    Processing: Cronobacter_turicensis_z3032
    Processing: Rhodoferax_ferrireducens_T118
    Processing: Haemophilus_somnus_129PT
    Processing: Oscillibacter_valericigenes_Sjm18-20
    Processing: Desulfosporosinus_acidiphilus_SJ4
    Processing: Desulfovibrio_africanus_str_Walvis_Bay
    Processing: _Cellvibrio_gilvus_ATCC_13127
    Processing: Propionibacterium_avidum_44067
    Processing: Desulfotomaculum_carboxydivorans_CO-1-SRB
    Processing: Sulfuricurvum_kujiense_DSM_16994
    Processing: Halobacillus_halophilus_DSM_2266
    Processing: Candidatus_Zinderia_insecticola_CARI
    Processing: Campylobacter_fetus_subsp_venerealis_cfvi03_293
    Processing: Denitrovibrio_acetiphilus_DSM_12809
    Processing: Chromohalobacter_salexigens_DSM_3043
    Processing: Thermodesulfobium_narugense_DSM_14796
    Processing: Rickettsia_australis_str_Cutlack
    Processing: Eubacterium_rectale_ATCC_33656
    Processing: Halomonas_elongata_DSM_2581
    Processing: Lactobacillus_ruminis_ATCC_27782
    Processing: Shewanella_baltica_OS117
    Processing: Thermodesulfovibrio_yellowstonii_DSM_11347
    Processing: Borrelia_garinii_PBi
    Processing: Granulicella_mallensis_MP5ACTX8
    Processing: Cardinium_endosymbiont_cEper1_of_Encarsia_pergandiella
    Processing: Vibrio_cholerae_MJ-1236
    Processing: Nocardiopsis_alba_ATCC_BAA-2165
    Processing: Streptococcus_anginosus_C238
    Processing: Lysinibacillus_sphaericus_C3-41
    Processing: Brevibacillus_brevis_NBRC_100599
    Processing: Melissococcus_plutonius_DAT561
    Processing: Tropheryma_whipplei_str_Twist
    Processing: Corynebacterium_pseudotuberculosis_FRC41
    Processing: Clostridiales_genomosp_BVAB3_str_UPII9-5
    Processing: Clostridium_cellulovorans_743B
    Processing: Marinithermus_hydrothermalis_DSM_14884
    Processing: Herbaspirillum_seropedicae_SmR1
    Processing: Acidovorax_citrulli_AAC00-1
    Processing: Desulfocapsa_sulfexigens_DSM_10523
    Processing: Serratia_symbiotica_str_Cinara_cedri
    Processing: Kocuria_rhizophila_DC2201
    Processing: Gordonia_polyisoprenivorans_VH2
    Processing: Bartonella_grahamii_as4aup
    Processing: Alicyclobacillus_acidocaldarius_subsp_acidocaldarius_DSM_446
    Processing: Desulfovibrio_magneticus_RS-1
    Processing: Candidatus_Solibacter_usitatus_Ellin6076
    Processing: Campylobacter_coli_RM1875
    Processing: Bifidobacterium_dentium_Bd1
    Processing: Stackebrandtia_nassauensis_DSM_44728
    Processing: Edwardsiella_ictaluri_93-146
    Processing: Ignavibacterium_album_JCM_16511
    Processing: Granulicella_tundricola_MP5ACTX9
    Processing: Cyanobacterium_stanieri_PCC_7202
    Processing: Ilumatobacter_coccineus_YM16-304
    Processing: Caldicellulosiruptor_kristjanssonii_I77R1B
    Processing: Cronobacter_sakazakii_SP291
    Processing: Sulfurimonas_denitrificans_DSM_1251
    Processing: Pseudomonas_mandelii_JR-1
    Processing: Staphylococcus_haemolyticus_JCSC1435
    Processing: Croceibacter_atlanticus_HTCC2559
    Processing: Mycobacterium_avium_subsp_avium_2285_R
    Processing: Isoptericola_variabilis_225
    Processing: Nocardia_farcinica_IFM_10152
    Processing: Anaplasma_centrale_str_Israel
    Processing: Edwardsiella_piscicida_C07-087
    Processing: Mycobacterium_intracellulare_1956
    Processing: Corynebacterium_casei_LMG_S-19264
    Processing: Dyadobacter_fermentans_DSM_18053
    Processing: Mycobacterium_africanum_GM041182
    Processing: Crinalium_epipsammum_PCC_9333
    Processing: Polymorphum_gilvum_SL003B-26A1
    Processing: Erwinia_amylovora_LA636

```r
%%R -w 600 -h 450
inFile = '/home/nick/notebook/SIPSim/t/genome100/DBL_index.txt'
df = read.delim(inFile, sep='\t') %>%
    gather(pos, vert_grad_BD, vert_gradient_BD_low, vert_gradient_BD_high)

# example
df.ex = data.frame('DBL_BD' = c(1.675, 1.769), 'vert_grad_BD' = c(1.75, 1.75))

# plot
p.TLA = ggplot(df, aes(DBL_BD, vert_grad_BD, color=pos, group=DBL_BD)) +
    geom_line(color='black', size=1) +
    geom_point(data=df.ex, color='red', size=4) +
    geom_line(data=df.ex, aes(group=vert_grad_BD), color='red', linetype='dashed', size=1.2) +
    #geom_vline(xintercept=1.774, linetype='dashed', alpha=0.5, color='blue') +  # theoretical max fragment BD
    #scale_y_reverse(limits=c(1.85, 1.50)) +
    scale_y_reverse() +
    scale_x_continuous(limits=c(1.63, 1.77)) +
    labs(x='BD of DBL',
         y='BD of vertical gradient\n(during fractionation)',
         title='TLA-110, Beckman fixed-angle rotor') +
    theme_bw() +
    theme(
        text = element_text(size=16)
    )
p.TLA
```

```r
%%R -i workDir
# saving figure
F = file.path(workDir, 'DBL_TLA110.pdf')
ggsave(F, p.TLA, width=6, height=4.5)
cat('File written:', F, '\n')
```

    File written: /home/nick/notebook/SIPSim/dev/theory//DBL_TLA110.pdf

#### Notes

* The dashed line provides an example of the 'true' BD of fragments contained in the DBL at the gradient density of 1.7 when the gradient is in the vertically oriented during fractionation.

## Vertical rotor

* VTi 65.2, Beckman rotor
  * Refs:
    * >http://www.nature.com/ismej/journal/v1/n6/full/ismej200765a.html
    * >Neufeld JD, Vohra J, Dumont MG, Lueders T, Manefield M, Friedrich MW, et al. (2007). DNA stable-isotope probing. Nat Protocols 2: 860–866.
  * params:
    * tube width = 1.3 cm
    * tube height = 5.1 cm
    * tube volume = 5.1 ml
    * r_min = 7.47 cm
    * r_max = 8.79 cm
    * final density = 1.725
    * speed = 177000 g_av (42500 rpm)
    * angular velocity = $((2 * 3.14159 * R)/60)^2$ = 19807714
    * time = 40 hr

```python
runDir = '/home/nick/notebook/SIPSim/t/genome100/'
!cd $runDir; \
    SIPSim DBL \
        -D 1.725 \
        -w 19807714 \
        --tube_height 5.1 \
        --r_min 7.47 \
        --r_max 8.79 \
        --vertical \
        --np 4 \
        ampFrag_skewN90-25-n5-nS_dif_kde.pkl \
        > ampFrag_skewN90-25-n5-nS_dif_DBL_kde_VERT.pkl
```

    DBL_index file written: "DBL_index.txt"
    Processing: Nocardia_nova_SH22a
    Processing: Myxococcus_stipitatus_DSM_14675
    Processing: Aeromonas_salmonicida_subsp_salmonicida_A449
    Processing: Bradyrhizobium_diazoefficiens_USDA_110
    Processing: Sulfurihydrogenibium_azorense_Az-Fu1
    Processing: Geobacillus_thermoleovorans_CCB_US3_UF5
    Processing: Vibrio_furnissii_NCTC_11218
    Processing: Kangiella_koreensis_DSM_16069
    Processing: Spiroplasma_culicicola_AES-1
    Processing: Thermincola_potens_JR
    Processing: Haemophilus_influenzae_PittEE
    Processing: Mycobacterium_abscessus_subsp_bolletii_50594
    Processing: Oceanithermus_profundus_DSM_14977
    Processing: Mycoplasma_bovoculi_M165_69
    Processing: Aeromonas_veronii_B565
    Processing: Capnocytophaga_canimorsus_Cc5
    Processing: Leptospira_borgpetersenii_serovar_Hardjo-bovis_str_L550
    Processing: Mycoplasma_yeatsii_GM274B
    Processing: Planktomarina_temperata_RCA23
    Processing: Dehalogenimonas_lykanthroporepellens_BL-DC-9
    Processing: Kosmotoga_olearia_TBF_19_5_1
    Processing: Pseudonocardia_dioxanivorans_CB1190
    Processing: Candidatus_Liberibacter_asiaticus_str_Ishi-1
    Processing: Clostridium_perfringens_str_13
    Processing: Bacillus_subtilis_PY79
    Processing: Vibrio_tasmaniensis_LGP32
    Processing: Caldicellulosiruptor_saccharolyticus_DSM_8903
    Processing: Desulfosporosinus_orientis_DSM_765
    Processing: Cronobacter_turicensis_z3032
    Processing: Rhodoferax_ferrireducens_T118
    Processing: Haemophilus_somnus_129PT
    Processing: Oscillibacter_valericigenes_Sjm18-20
    Processing: Desulfosporosinus_acidiphilus_SJ4
    Processing: Desulfovibrio_africanus_str_Walvis_Bay
    Processing: _Cellvibrio_gilvus_ATCC_13127
    Processing: Desulfotomaculum_carboxydivorans_CO-1-SRB
    Processing: Propionibacterium_avidum_44067
    Processing: Sulfuricurvum_kujiense_DSM_16994
    Processing: Halobacillus_halophilus_DSM_2266
    Processing: Campylobacter_fetus_subsp_venerealis_cfvi03_293
    Processing: Denitrovibrio_acetiphilus_DSM_12809
    Processing: Candidatus_Zinderia_insecticola_CARI
    Processing: Chromohalobacter_salexigens_DSM_3043
    Processing: Thermodesulfobium_narugense_DSM_14796
    Processing: Eubacterium_rectale_ATCC_33656
    Processing: Rickettsia_australis_str_Cutlack
    Processing: Lactobacillus_ruminis_ATCC_27782
    Processing: Halomonas_elongata_DSM_2581
    Processing: Shewanella_baltica_OS117
    Processing: Borrelia_garinii_PBi
    Processing: Thermodesulfovibrio_yellowstonii_DSM_11347
    Processing: Cardinium_endosymbiont_cEper1_of_Encarsia_pergandiella
    Processing: Granulicella_mallensis_MP5ACTX8
    Processing: Vibrio_cholerae_MJ-1236
    Processing: Nocardiopsis_alba_ATCC_BAA-2165
    Processing: Streptococcus_anginosus_C238
    Processing: Brevibacillus_brevis_NBRC_100599
    Processing: Melissococcus_plutonius_DAT561
    Processing: Lysinibacillus_sphaericus_C3-41
    Processing: Tropheryma_whipplei_str_Twist
    Processing: Clostridiales_genomosp_BVAB3_str_UPII9-5
    Processing: Corynebacterium_pseudotuberculosis_FRC41
    Processing: Clostridium_cellulovorans_743B
    Processing: Herbaspirillum_seropedicae_SmR1
    Processing: Marinithermus_hydrothermalis_DSM_14884
    Processing: Acidovorax_citrulli_AAC00-1
    Processing: Serratia_symbiotica_str_Cinara_cedri
    Processing: Gordonia_polyisoprenivorans_VH2
    Processing: Desulfocapsa_sulfexigens_DSM_10523
    Processing: Bartonella_grahamii_as4aup
    Processing: Kocuria_rhizophila_DC2201
    Processing: Candidatus_Solibacter_usitatus_Ellin6076
    Processing: Alicyclobacillus_acidocaldarius_subsp_acidocaldarius_DSM_446
    Processing: Desulfovibrio_magneticus_RS-1
    Processing: Edwardsiella_ictaluri_93-146
    Processing: Campylobacter_coli_RM1875
    Processing: Bifidobacterium_dentium_Bd1
    Processing: Stackebrandtia_nassauensis_DSM_44728
    Processing: Granulicella_tundricola_MP5ACTX9
    Processing: Cyanobacterium_stanieri_PCC_7202
    Processing: Ignavibacterium_album_JCM_16511
    Processing: Caldicellulosiruptor_kristjanssonii_I77R1B
    Processing: Cronobacter_sakazakii_SP291
    Processing: Ilumatobacter_coccineus_YM16-304
    Processing: Sulfurimonas_denitrificans_DSM_1251
    Processing: Pseudomonas_mandelii_JR-1
    Processing: Staphylococcus_haemolyticus_JCSC1435
    Processing: Croceibacter_atlanticus_HTCC2559
    Processing: Mycobacterium_avium_subsp_avium_2285_R
    Processing: Isoptericola_variabilis_225
    Processing: Anaplasma_centrale_str_Israel
    Processing: Edwardsiella_piscicida_C07-087
    Processing: Corynebacterium_casei_LMG_S-19264
    Processing: Nocardia_farcinica_IFM_10152
    Processing: Mycobacterium_intracellulare_1956
    Processing: Crinalium_epipsammum_PCC_9333
    Processing: Dyadobacter_fermentans_DSM_18053
    Processing: Mycobacterium_africanum_GM041182
    Processing: Polymorphum_gilvum_SL003B-26A1
    Processing: Erwinia_amylovora_LA636

```r
%%R -w 600
inFile = '/home/nick/notebook/SIPSim/t/genome100/DBL_index.txt'
df = read.delim(inFile, sep='\t') %>%
    gather(pos, vert_grad_BD, vert_gradient_BD_low, vert_gradient_BD_high)

# example
df.ex = data.frame('DBL_BD' = c(1.638, 1.769), 'vert_grad_BD' = c(1.75, 1.75))

# plot
p.VTi = ggplot(df, aes(DBL_BD, vert_grad_BD, color=pos, group=DBL_BD)) +
    geom_line(color='black', size=1) +
    geom_point(data=df.ex, color='red', size=4) +
    geom_line(data=df.ex, aes(group=vert_grad_BD), color='red', linetype='dashed', size=1.2) +
    #scale_y_reverse(limits=c(1.85, 1.50)) +
    scale_y_reverse() +
    scale_x_continuous(limits=c(1.63, 1.77)) +
    labs(x='BD of DBL', y='BD of vertical gradient\n(during fractionation)',
         title='VTi 65.2, Beckman vertical rotor') +
    theme_bw() +
    theme(
        text = element_text(size=16)
    )
p.VTi
```

```r
%%R -i workDir
# saving figure
F = file.path(workDir, 'DBL_VTi65.2.pdf')
ggsave(F, p.VTi, width=6, height=4.5)
cat('File written:', F, '\n')
```

    File written: /home/nick/notebook/SIPSim/dev/theory//DBL_VTi65.2.pdf

#### Notes

* The dashed line provides an example of the 'true' BD of fragments contained in the DBL at the gradient density of 1.7 when the gradient is in the vertically oriented during fractionation.
* WARNING: the DBL simulation makes the simplifying assumption of a 2d tube object and finds the vertical distance that a band spans in the tube, which sets the span of DBL contamination in a fixed-angle rotor. However, for vertical tubes, the DBL would probably be more accurately modeled from a 3d representation of the tube.
  * Regardless, there would be substantially more DBL 'smearing' with a vertical rotor than a fixed-angle rotor.

***
# Misc

## DNA diffusion

* sedimentation coefficient of DNA (S)
  * $S = 2.8 + (0.00834 * M^{0.479})$
    * where
      * M = molecular weight of DNA
  * OR $S = 2.8 + (0.00834 * (L*666)^{0.479})$
    * where
      * L = length of DNA
* Svedberg's equation
  * $s/D = \frac{M(1-\bar{V}p)}{RT}$
  * where
    * s = sedimentation coefficient
    * D = diffusion coefficient
    * M = molecular weight
    * $\bar{V} = 1/\rho_p$
      * $\rho_p$ = density of the sphere
    * p = density of the liquid
    * R = universal gas constant
    * T = absolute temperature
* Finding diffusion coefficient of DNA in CsCl ($\mu m^2 / s$)
  * $D = \frac{RT}{M(1-\bar{V}p)}*s$
    * where
      * R = 8.3144598 (J mol^-1 K^-1)
      * T = 293.15 (K)
      * p = 1.7  (Buckley lab gradients)
      * $\bar{V} = 1/\rho_p$
        * $\rho_p$ = 1.99
      * $s = 2.8 + (0.00834 * (L*666)^{0.479})$
        * L = DNA length (bp)

```r
%%R -h 300

length2MW = function(L){ L * 666 }

length2sedCoef = function(L){
    2.8 + (0.00834 * (L*666)**0.479)
}

MW2diffuseCoef = function(L, p, R=8.3144598, T=293.15){
    V = 1/1.99
    M = length2MW(L)
    s = length2sedCoef(L)
    (R*T)/(M*(1-V*p)) * s
}

# test
L = seq(100, 50000, 100)
p = 1.7
D = sapply(L, MW2diffuseCoef, p=p)
df = data.frame('L' = L, 'D' = D)

# plotting
ggplot(df, aes(L, D)) +
    geom_point() +
    geom_line(alpha=0.5) +
    theme_bw() +
    theme(
        text = element_text(size=16)
    )
```

## Calculating diffusion from DBL

* Einstein-Smoluchowski relation
  * $t = \frac{z^2}{0.9 * D}$
    * where
      * t = time (sec)
      * z = mean deviation of molecules from starting position
      * D = diffusion coefficient (cm^2 s^-1)
  * rewritten: $z = \sqrt{0.9*D*t}$

```r
%%R

# converting D to cm^2/s
df$D_cm = df$D * 1e-5

# time periods (sec)
t = seq(1, 300, 10)

# calculating z (cm)
ES = function(D, t){
    sqrt(0.9 * D * t)
}
df2 = expand.grid(df$D_cm, t)
colnames(df2) = c('D_cm', 't')
df2$z = mapply(ES, df2$D_cm, df2$t)
tmp = expand.grid(df$L, t)

# adding variable
df2$L = tmp$Var1
df2$t_min = df2$t / 60
df2$z_uM = df2$z / 1e-5

## plotting
ggplot(df2, aes(t_min, z_uM, color=L, group=L)) +
    #geom_point(size=1.5) +
    geom_line() +
    labs(x='Time (minutes)',
         y='mean deviation of molecules\nfrom starting position (uM)') +
    scale_color_continuous('DNA fragment\nlength (bp)') +
    theme_bw() +
    theme(
        text = element_text(size=16)
    )
```

```r
%%R -w 800
## plotting
ggplot(df2, aes(L, z_uM, color=t_min, group=t_min)) +
    #geom_point(size=1.5) +
    geom_line() +
    labs(x='DNA fragment length (bp)',
         y='mean deviation of molecules\nfrom starting position (uM)') +
    scale_color_continuous('Time\n(minutes)') +
    theme_bw() +
    theme(
        text = element_text(size=16)
    )
```