# Multi-TX/TL

Matthieu Kratz

## Why this model?

Like most genetic circuit models, you typically start with a model that captures the two fundamental processes of transcription and translation. These processes can modelled with varying degrees of complexity, ranging from a basic tx/tl model (__Equation 1__) to complex models that simulate tx/tl at a base pair level. For my project, it was of particular importance that I accurately model the sequestration of transcriptional machinery. Further, I was working without RPU data, making typical models that assume some maximum steady state saturation e.g. positive proportional hill intractable for my system.

\begin{align}
\\ \\
&G \xrightarrow{ktx} T + G \\ \\
&\textbf{Equation 1.} \ \ \text{Simple Transcription}
\\
\end{align}

Hence, I needed a model that gave reasonable steady state transcription and translation dynamics from non-RPU parameters, all while accurately tracking the sequestration of the relevant machinery.

## The Multi-TX Model

The model CRN is below and the mechanism effectively relies on accounting for all possible RNAp occupancy states, including the various transitions between these occupancy states.

\begin{align}
\\ \\
&\textbf{1. Binding} \\
&T7_p + T7_p:G_n  \underset{kT72}{\overset{kT71}\rightleftharpoons} T7_p:G_{n\alpha}  \ \ , \   \text{n $\neq$ $n_{max}$} \\ \\
&\textbf{2. Closed --> Open} \\
&T7_p:G_{n\alpha}  \xrightarrow{k_{iso}} T7_p:G_{n+1}  \ \ , \   \text{n $\neq$ $n_{max}$} \\ \\
&\textbf{3. TX} \\
&T7_p:G_{n\alpha} \xrightarrow{ktx} nT7_m + nT7_p + T7_p:G_{0\alpha} \ \ , \  \text{n $\neq$ 0} \\
&T7_p:G_n \xrightarrow{ktx} nT7_m + nT7_p + T7_p:G_{0} \ \ , \  \text{n $\neq$ 0} \\
\end{align}

kT71 --> Promoter binding rate constant (bimolecular)

kT72 --> Promoter unbinding rate constant (unimolecular)

k_iso --> Closed to open complex transition rate constant (unimolecular)

ktx --> Single polymerase mRNA synthesis rate constant (unimolecular)

A couple of comments:
- We explicitly model the process of transitioning from the closed ($T7_p:G_{n\alpha}$) to open complex ($T7_p:G_{n+1}$). This a pretty slow reaction in vivo, and in the various iterations of this model, including this seemed to be key in accurately reflecting transcription dynamics.

- $n_{max}$ refers to the maximum possible occupancy of the gene. This is the physical limit which is ultimately determined by the footprint of the polymerase and the length of the gene. In relation to the point above, without isomerization, this physical limit is always met. With the explicit isomerization, this physical saturation is not met (with my parameter set at least).

- Polymerase can only be added one at a time to existing genes i.e. cannot have multiple binding events or closed complexes simultaneously.

- We have two TX reactions, one from the closed state and the other from the open state, both with $N$ actively transcribing polymerases. We decided to allow release from the closed state as there is no reason why one polymerase in the closed form should inhibit of other activitely transcribing polymerases. Further, at a modelling level, not allowing release from the closed state results in excessive sequestration of polymerase due to the long time scale of isomerization.

- In both TX reactions, the entirety of the $N$ polymerases (bar the one in the closed state) are simultaneously release, along with $N$ transcripts and the unoccupied gene ($T7_p:G_{0}$)

## Biocrnpyler multi_tx Mechanism subclass

Subclass should be availabe via `from biocrnpyler import mechanism`, code is below for clarity

Couple of comments:
- The subclass has been designed to be used in concert with the Promoter and DNA assemblies subclasses

- Have to define a maximum occupancy (int) and cognate polymerase (species or str) when instantiating

- The various complex species are generated within the subclass from the names of the polymerase and dna objects in the DNAassembly object. They are defined as species with DNA material types.

## Example: T7 Polymerase Transcription of GFP mRNA

Now we define a DNA assembly that use our mechanism in the following steps:
- Create a species for the relevant polymerase
- Create multi_tx Mechanism, give a maximum occupancy and polymerase (must be species)
- Associate this mechanism with a promoter
- Place this promoter into a DNA assembly

And voila, the cassette regulated by T7p is ready to use!

This DNAassembly will be placed in a SimpleTxTlExtract which only dilutes mRNA, not proteins. I haven't used protein dilution as it makes it easier to glean the degree of sequestration present with this mechanism. In practical use, you would of course allow your polymerase to be diluted.

```python
from biocrnpyler import *

#the most important parameters for multi_tx
#max_occ is the number of ribosomes that can bind to a transcript
#k_iso is the rate of isomerization
parameters = {"max_occ":10, "k_iso":10}

# Define Polymerase, and max occupancy and instatiate Mechanism Object
T7P = Species('T7p','protein')
#Multi-tx mechanism
MX = multi_tx(pol=T7P,name='MX')

# create promoter object, associated MX and params with it
pT7_mx = Promoter(name='pT7',mechanisms={'transcription':MX})

# place promoter object into DNA assembly with GFP reporter
GFP = Species("GFP", material_type = "protein")
PFL_mx = DNAassembly('PFL', promoter=pT7_mx, rbs = "weak", protein=GFP)

# create simple promoter and DNA assembly objects that synthesize polymerase
SC = DNAassembly('T7_const', promoter="weak", rbs = "weak", protein=T7P)

# make extract with T7p source and GFP and compile CRN
mixture = SimpleTxTlDilutionMixture(components=[PFL_mx, SC], parameter_file = "default_parameters.txt", parameters = parameters)

CRN1 = mixture.compile_crn()

print(CRN1.pretty_print(show_rates = False))
print(mixture.components)
```

    Species (26) = {0. dna[PFL] init_conc = 0, 1. complex[dna[PFL]:protein[T7p]] init_conc = 0, 2. complex[dna[PFL]:2x_protein[T7p]] init_conc = 0, 3. complex[dna[PFL]:3x_protein[T7p]] init_conc = 0, 4. complex[dna[PFL]:4x_protein[T7p]] init_conc = 0, 5. complex[dna[PFL]:5x_protein[T7p]] init_conc = 0, 6. complex[dna[PFL]:6x_protein[T7p]] init_conc = 0, 7. complex[dna[PFL]:7x_protein[T7p]] init_conc = 0, 8. complex[dna[PFL]:8x_protein[T7p]] init_conc = 0, 9. complex[dna[PFL]:9x_protein[T7p]] init_conc = 0, 10. complex[dna[PFL]:10x_protein[T7p]] init_conc = 0, 11. complex[dna[PFL]:protein[T7p](closed)] init_conc = 0, 12. complex[dna[PFL]:2x_protein[T7p](closed)] init_conc = 0, 13. complex[dna[PFL]:3x_protein[T7p](closed)] init_conc = 0, 14. complex[dna[PFL]:4x_protein[T7p](closed)] init_conc = 0, 15. complex[dna[PFL]:5x_protein[T7p](closed)] init_conc = 0, 16. complex[dna[PFL]:6x_protein[T7p](closed)] init_conc = 0, 17. complex[dna[PFL]:7x_protein[T7p](closed)] init_conc = 0, 18. complex[dna[PFL]:8x_protein[T7p](closed)] init_conc = 0, 19. complex[dna[PFL]:9x_protein[T7p](closed)] init_conc = 0, 20. complex[dna[PFL]:10x_protein[T7p](closed)] init_conc = 0, 21. protein[T7p] init_conc = 0, 22. rna[PFL] init_conc = 0, 23. protein[GFP] init_conc = 0, 24. dna[T7_const] init_conc = 0, 25. rna[T7_const] init_conc = 0}

    Reactions (48) = [
    0. protein[T7p]+complex[dna[PFL]:protein[T7p]] <--> complex[dna[PFL]:2x_protein[T7p](closed)]
    1. protein[T7p]+complex[dna[PFL]:2x_protein[T7p]] <--> complex[dna[PFL]:3x_protein[T7p](closed)]
    2. protein[T7p]+complex[dna[PFL]:3x_protein[T7p]] <--> complex[dna[PFL]:4x_protein[T7p](closed)]
    3. protein[T7p]+complex[dna[PFL]:4x_protein[T7p]] <--> complex[dna[PFL]:5x_protein[T7p](closed)]
    4. protein[T7p]+complex[dna[PFL]:5x_protein[T7p]] <--> complex[dna[PFL]:6x_protein[T7p](closed)]
    5. protein[T7p]+complex[dna[PFL]:6x_protein[T7p]] <--> complex[dna[PFL]:7x_protein[T7p](closed)]
    6. protein[T7p]+complex[dna[PFL]:7x_protein[T7p]] <--> complex[dna[PFL]:8x_protein[T7p](closed)]
    7. protein[T7p]+complex[dna[PFL]:8x_protein[T7p]] <--> complex[dna[PFL]:9x_protein[T7p](closed)]
    8. protein[T7p]+complex[dna[PFL]:9x_protein[T7p]] <--> complex[dna[PFL]:10x_protein[T7p](closed)]
    9. complex[dna[PFL]:protein[T7p](closed)] --> complex[dna[PFL]:protein[T7p]]
    10. complex[dna[PFL]:2x_protein[T7p](closed)] --> complex[dna[PFL]:2x_protein[T7p]]
    11. complex[dna[PFL]:3x_protein[T7p](closed)] --> complex[dna[PFL]:3x_protein[T7p]]
    12. complex[dna[PFL]:4x_protein[T7p](closed)] --> complex[dna[PFL]:4x_protein[T7p]]
    13. complex[dna[PFL]:5x_protein[T7p](closed)] --> complex[dna[PFL]:5x_protein[T7p]]
    14. complex[dna[PFL]:6x_protein[T7p](closed)] --> complex[dna[PFL]:6x_protein[T7p]]
    15. complex[dna[PFL]:7x_protein[T7p](closed)] --> complex[dna[PFL]:7x_protein[T7p]]
    16. complex[dna[PFL]:8x_protein[T7p](closed)] --> complex[dna[PFL]:8x_protein[T7p]]
    17. complex[dna[PFL]:9x_protein[T7p](closed)] --> complex[dna[PFL]:9x_protein[T7p]]
    18. complex[dna[PFL]:10x_protein[T7p](closed)] --> complex[dna[PFL]:10x_protein[T7p]]
    19. complex[dna[PFL]:protein[T7p]] --> protein[T7p]+rna[PFL]+dna[PFL]
    20. complex[dna[PFL]:2x_protein[T7p]] --> 2protein[T7p]+2rna[PFL]+dna[PFL]
    21. complex[dna[PFL]:3x_protein[T7p]] --> 3protein[T7p]+3rna[PFL]+dna[PFL]
    22. complex[dna[PFL]:4x_protein[T7p]] --> 4protein[T7p]+4rna[PFL]+dna[PFL]
    23. complex[dna[PFL]:5x_protein[T7p]] --> 5protein[T7p]+5rna[PFL]+dna[PFL]
    24. complex[dna[PFL]:6x_protein[T7p]] --> 6protein[T7p]+6rna[PFL]+dna[PFL]
    25. complex[dna[PFL]:7x_protein[T7p]] --> 7protein[T7p]+7rna[PFL]+dna[PFL]
    26. complex[dna[PFL]:8x_protein[T7p]] --> 8protein[T7p]+8rna[PFL]+dna[PFL]
    27. complex[dna[PFL]:9x_protein[T7p]] --> 9protein[T7p]+9rna[PFL]+dna[PFL]
    28. complex[dna[PFL]:10x_protein[T7p]] --> 10protein[T7p]+10rna[PFL]+dna[PFL]
    29. complex[dna[PFL]:2x_protein[T7p](closed)] --> protein[T7p]+rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    30. complex[dna[PFL]:3x_protein[T7p](closed)] --> 2protein[T7p]+2rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    31. complex[dna[PFL]:4x_protein[T7p](closed)] --> 3protein[T7p]+3rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    32. complex[dna[PFL]:5x_protein[T7p](closed)] --> 4protein[T7p]+4rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    33. complex[dna[PFL]:6x_protein[T7p](closed)] --> 5protein[T7p]+5rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    34. complex[dna[PFL]:7x_protein[T7p](closed)] --> 6protein[T7p]+6rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    35. complex[dna[PFL]:8x_protein[T7p](closed)] --> 7protein[T7p]+7rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    36. complex[dna[PFL]:9x_protein[T7p](closed)] --> 8protein[T7p]+8rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    37. complex[dna[PFL]:10x_protein[T7p](closed)] --> 9protein[T7p]+9rna[PFL]+complex[dna[PFL]:protein[T7p](closed)]
    38. dna[PFL]+protein[T7p] <--> complex[dna[PFL]:protein[T7p](closed)]
    39. rna[PFL] --> rna[PFL]+protein[GFP]
    40. dna[T7_const] --> dna[T7_const]+rna[T7_const]
    41. rna[T7_const] --> rna[T7_const]+protein[T7p]
    42. protein[T7p] -->
    43. rna[PFL] -->
    44. protein[GFP] -->
    45. rna[T7_const] -->
    46. rna[PFL] -->
    47. rna[T7_const] -->
    ]
    [DNAassembly: PFL
    	Promoter: pT7
    	transcript = rna_PFL
    	RBS: weak
    	protein = protein_GFP, DNAassembly: T7_const
    	Promoter: weak
    	transcript = rna_T7_const
    	RBS: weak
    	protein = protein_T7p]

```python
try:
    import numpy as np
    import matplotlib.pyplot as plt

    ts = np.arange(0,5000,1)

    x0_dict = {repr(PFL_mx.dna):1.,repr(SC.dna):.1}

    R1 = CRN1.simulate_with_bioscrape_via_sbml(ts, initial_condition_dict = x0_dict, stochastic = False)
    if R1 is not None:
        fig, ax = plt.subplots(1,2,figsize=(18,8))
        ax[0].set_title('Polymerase Levels',pad=20,fontdict={'fontsize':18})
        ax[0].plot(R1[str(T7P)],linewidth=3)
        ax[0].set_xlabel('Time (sec)',labelpad=15,fontdict={'fontsize':14})
        ax[0].set_ylabel('Polymerase Count',labelpad=15,fontdict={'fontsize':14})

        ax[1].set_title('GFP Levels',pad=20,fontdict={'fontsize':18})
        ax[1].plot(R1[str(GFP)],linewidth=3,c='k')
        ax[1].set_xlabel('Time (sec)',labelpad=15,fontdict={'fontsize':14})
        ax[1].set_ylabel('[GFP]',labelpad=15,fontdict={'fontsize':14})
except ModuleNotFoundError:
    print('please install the plotting libraries: pip install biocrnpyler[all]')

```

## Comparison of Transcript SS with RPU Data

Here we will do a head to head comparison with a simple transcription model built using RPU data. I consider this to effectively be the ground truth for the SS Transcript Count, although it should be noted this comparison does not necessary validate any of the pre-SS dynamics as the simple transcription model assumes immediate saturation. RPU data is from Qi et al.(2012) and RPU standard is from supplement of Nielsen et al. (2016).

```python
# place promoter object into DNA assembly
mech_default = Transcription_MM(rnap=T7P,name='MX')
pT7 = Promoter("pT7", mechanisms = [mech_default])
PFL_default = DNAassembly('PFL',dna='T7',promoter=pT7,rbs = "weak", protein = GFP)

# make extract with T7p source and GFP and compile CRN
Test_EX = SimpleTxTlDilutionMixture(components=[PFL_default, SC],parameter_file = "default_parameters.txt")
CRN2 = Test_EX.compile_crn()
```

```python
try:
    import numpy as np
    import matplotlib.pyplot as plt

    ts = np.arange(0,5000,1)
    x0_dict = {PFL_default.dna:1, SC.dna:.1}

    R2 = CRN2.simulate_with_bioscrape_via_sbml(ts, initial_condition_dict = x0_dict, stochastic = False,)
    if R2 is not None:
        fig = plt.figure(figsize=(10,6))
        plt.title('GFP Levels',pad=20,fontdict={'fontsize':18})
        plt.plot(R1[str(GFP)],linewidth=3,c='k',label='MTX Model')
        plt.plot(R2[str(GFP)],linewidth=3,c='b', label='RPU Model')
        plt.xlabel('Time (sec)',labelpad=15,fontdict={'fontsize':14})
        plt.ylabel('[GFP]',labelpad=15,fontdict={'fontsize':14})
        plt.legend(fontsize=14)

        print('\n \n')
        print(f"MX Model predicts {np.round(R1[str(str(GFP))].iloc[-1])} GFP and RPU Model predicts {np.round(R2[str(str(GFP))].iloc[-1])} GFP at SS \n \n")
except ModuleNotFoundError:
    print('please install the plotting libraries: pip install biocrnpyler[all]')

```

The two models seem to agree pretty well, with some minor difference. This is pretty cool given the fact that MX model uses indirect parameters (promoter affinity, isomerization and tx rates) to predict this SS value. Of course, it's also important to keep in mind that alternative models typically do to not account for the constant sequestration of the polymerase machinery to sustain this SS transcript level. As such, they may not reflect key dynamics resulting from machinery allocation/sharing e.g. retroactivity in tx/tl.

## Putative Multi-TL Model

The general model of binding, isomerization and production can be readily extended to the process of translation. However, there are several caveats for translation.

- First of all, I don't have any direct data like RPU experiments to compare my model with. All I can say is that with biologically reasonable parameter sets, you get something like a few 100 proteins per mRNA (RBZ in excess) and e. coli has a protein to mRNA ratio in that range (100:1 - 1000:1) (Taniguchi et al.(2010) or bionumbers book).

- Second of all, since we're no longer working with DNA complexes, the mRNA-RBZ complexes should be subject to degredation/dilution. However, it seems that at the level of the biology, initiation can have a stabilizing effect and for elongation it is uncertain (Roy et al. (2013)). In my simulations, I've effectively done as if only dilution is applied to these complexes, but there is an argument to include some form of active degredation (at a reduced rate). Depending on how this implemented, one may need to make a new degredation mechanism where the complexes effectively releases the ribosomes i.e. only mRNA degraded. This is also complicated by the fact that using subclasses like ComplexSpecies() results in inheritance of degredation/material properties.

## Quick Example
PFL is transcribed through simple transcription and translated through my multi_tl mechanism. We also have a constitutive source of our RBZ being produced to provide a constant saturating concentration of RBZ.

```python
#the most important parameters for multi_tl
#max_occ is the number of ribosomes that can bind to a transcript
#k_iso is the rate of isomerization
parameters = {"max_occ":2, "k_iso":10}

#Create a DNA assembly and Mixture
PFL = DNAassembly('PFL', dna='T7', rbs='medium', promoter='medium',transcript='GFP',protein='GFP')
EM = TxTlDilutionMixture(name = 'e coli',components=[PFL], parameter_file = "default_parameters.txt", parameters = parameters)

# Instantiate mechanism and pass it the ribosome
ML = multi_tl(EM.ribosome.get_species())

#Overwrite the translation mechanism
EM.add_mechanism(ML, overwrite = True)

CRN3 = EM.compile_crn()
print(CRN3.pretty_print(show_rates = False))
```

    Species (29) = {0. dna[T7] init_conc = 0, 1. protein[RNAP(machinery)] init_conc = 0, 2. rna[GFP] init_conc = 0, 3. complex[dna[T7]:protein[RNAP]] init_conc = 0, 4. complex[protein[Ribo]:rna[GFP]] init_conc = 0, 5. complex[2x_protein[Ribo]:rna[GFP]] init_conc = 0, 6. complex[protein[Ribo]:rna[GFP](closed)] init_conc = 0, 7. complex[2x_protein[Ribo]:rna[GFP](closed)] init_conc = 0, 8. protein[Ribo(machinery)] init_conc = 0, 9. protein[GFP] init_conc = 0, 10. protein[RNAase(machinery)] init_conc = 0, 11. dna[cellular_processes] init_conc = 0, 12. rna[cellular_processes] init_conc = 0, 13. complex[dna[cellular_processes]:protein[RNAP]] init_conc = 0, 14. complex[protein[Ribo]:rna[cellular_processes]] init_conc = 0, 15. complex[2x_protein[Ribo]:rna[cellular_processes]] init_conc = 0, 16. complex[protein[Ribo]:rna[cellular_processes](closed)] init_conc = 0, 17. complex[2x_protein[Ribo]:rna[cellular_processes](closed)] init_conc = 0, 18. protein[cellular_processes] init_conc = 0, 19. complex[protein[RNAase]:rna[GFP]] init_conc = 0, 20. complex[complex[protein[Ribo]:rna[GFP]]:protein[RNAase]] init_conc = 0, 21. complex[complex[2x_protein[Ribo]:rna[GFP]]:protein[RNAase]] init_conc = 0, 22. complex[complex[protein[Ribo]:rna[GFP]]:protein[RNAase]] init_conc = 0, 23. complex[complex[2x_protein[Ribo]:rna[GFP]]:protein[RNAase]] init_conc = 0, 24. complex[protein[RNAase]:rna[cellular_processes]] init_conc = 0, 25. complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] init_conc = 0, 26. complex[complex[2x_protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] init_conc = 0, 27. complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] init_conc = 0, 28. complex[complex[2x_protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] init_conc = 0}

    Reactions (42) = [
    0. dna[T7]+protein[RNAP(machinery)] <--> complex[dna[T7]:protein[RNAP]]
    1. complex[dna[T7]:protein[RNAP]] --> dna[T7]+rna[GFP]+protein[RNAP(machinery)]
    2. protein[Ribo(machinery)]+complex[protein[Ribo]:rna[GFP]] <--> complex[2x_protein[Ribo]:rna[GFP](closed)]
    3. complex[protein[Ribo]:rna[GFP](closed)] --> complex[protein[Ribo]:rna[GFP]]
    4. complex[2x_protein[Ribo]:rna[GFP](closed)] --> complex[2x_protein[Ribo]:rna[GFP]]
    5. complex[protein[Ribo]:rna[GFP]] --> protein[Ribo(machinery)]+protein[GFP]+rna[GFP]
    6. complex[2x_protein[Ribo]:rna[GFP]] --> 2protein[Ribo(machinery)]+2protein[GFP]+rna[GFP]
    7. complex[2x_protein[Ribo]:rna[GFP](closed)] --> protein[Ribo(machinery)]+protein[GFP]+complex[protein[Ribo]:rna[GFP](closed)]
    8. rna[GFP]+protein[Ribo(machinery)] <--> complex[protein[Ribo]:rna[GFP](closed)]
    9. dna[cellular_processes]+protein[RNAP(machinery)] <--> complex[dna[cellular_processes]:protein[RNAP]]
    10. complex[dna[cellular_processes]:protein[RNAP]] --> dna[cellular_processes]+rna[cellular_processes]+protein[RNAP(machinery)]
    11. protein[Ribo(machinery)]+complex[protein[Ribo]:rna[cellular_processes]] <--> complex[2x_protein[Ribo]:rna[cellular_processes](closed)]
    12. complex[protein[Ribo]:rna[cellular_processes](closed)] --> complex[protein[Ribo]:rna[cellular_processes]]
    13. complex[2x_protein[Ribo]:rna[cellular_processes](closed)] --> complex[2x_protein[Ribo]:rna[cellular_processes]]
    14. complex[protein[Ribo]:rna[cellular_processes]] --> protein[Ribo(machinery)]+protein[cellular_processes]+rna[cellular_processes]
    15. complex[2x_protein[Ribo]:rna[cellular_processes]] --> 2protein[Ribo(machinery)]+2protein[cellular_processes]+rna[cellular_processes]
    16. complex[2x_protein[Ribo]:rna[cellular_processes](closed)] --> protein[Ribo(machinery)]+protein[cellular_processes]+complex[protein[Ribo]:rna[cellular_processes](closed)]
    17. rna[cellular_processes]+protein[Ribo(machinery)] <--> complex[protein[Ribo]:rna[cellular_processes](closed)]
    18. rna[GFP]+protein[RNAase(machinery)] <--> complex[protein[RNAase]:rna[GFP]]
    19. complex[protein[RNAase]:rna[GFP]] --> protein[RNAase(machinery)]
    20. complex[protein[Ribo]:rna[GFP]]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[GFP]]:protein[RNAase]]
    21. complex[complex[protein[Ribo]:rna[GFP]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
    22. complex[2x_protein[Ribo]:rna[GFP]]+protein[RNAase(machinery)] <--> complex[complex[2x_protein[Ribo]:rna[GFP]]:protein[RNAase]]
    23. complex[complex[2x_protein[Ribo]:rna[GFP]]:protein[RNAase]] --> 2protein[Ribo(machinery)]+protein[RNAase(machinery)]
    24. complex[protein[Ribo]:rna[GFP](closed)]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[GFP]]:protein[RNAase]]
    25. complex[complex[protein[Ribo]:rna[GFP]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
    26. complex[2x_protein[Ribo]:rna[GFP](closed)]+protein[RNAase(machinery)] <--> complex[complex[2x_protein[Ribo]:rna[GFP]]:protein[RNAase]]
    27. complex[complex[2x_protein[Ribo]:rna[GFP]]:protein[RNAase]] --> 2protein[Ribo(machinery)]+protein[RNAase(machinery)]
    28. rna[cellular_processes]+protein[RNAase(machinery)] <--> complex[protein[RNAase]:rna[cellular_processes]]
    29. complex[protein[RNAase]:rna[cellular_processes]] --> protein[RNAase(machinery)]
    30. complex[protein[Ribo]:rna[cellular_processes]]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]]
    31. complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
    32. complex[2x_protein[Ribo]:rna[cellular_processes]]+protein[RNAase(machinery)] <--> complex[complex[2x_protein[Ribo]:rna[cellular_processes]]:protein[RNAase]]
    33. complex[complex[2x_protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] --> 2protein[Ribo(machinery)]+protein[RNAase(machinery)]
    34. complex[protein[Ribo]:rna[cellular_processes](closed)]+protein[RNAase(machinery)] <--> complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]]
    35. complex[complex[protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] --> protein[Ribo(machinery)]+protein[RNAase(machinery)]
    36. complex[2x_protein[Ribo]:rna[cellular_processes](closed)]+protein[RNAase(machinery)] <--> complex[complex[2x_protein[Ribo]:rna[cellular_processes]]:protein[RNAase]]
    37. complex[complex[2x_protein[Ribo]:rna[cellular_processes]]:protein[RNAase]] --> 2protein[Ribo(machinery)]+protein[RNAase(machinery)]
    38. rna[GFP] -->
    39. protein[GFP] -->
    40. rna[cellular_processes] -->
    41. protein[cellular_processes] -->
    ]

```python
try:

    import numpy as np
    import matplotlib.pyplot as plt

    ts = np.arange(0,10000,1)
    x0_dict = {PFL.dna:1, EM.ribosome.get_species():100,  EM.rnap.get_species():20}

    R3 = CRN3.simulate_with_bioscrape_via_sbml(ts, initial_condition_dict = x0_dict, stochastic = False,)

    if R3 is not None:
        fig, ax = plt.subplots(1,2,figsize=(18,8))
        ax[0].set_title('GFP Protein Levels',pad=20,fontdict={'fontsize':18})
        ax[0].plot(R3[str(PFL.protein)],linewidth=3)
        #ax[0].plot(R3[str(EM.ribosome.get_species())],linewidth=3)
        ax[0].set_xlabel('Time (sec)',labelpad=15,fontdict={'fontsize':14})
        ax[0].set_ylabel('GFP Protein Count',labelpad=15,fontdict={'fontsize':14})

        ax[1].set_title('GFP Transcript Levels',pad=20,fontdict={'fontsize':18})
        ax[1].plot(R3[str(PFL.transcript)],linewidth=3,c='k')
        ax[1].set_xlabel('Time (sec)',labelpad=15,fontdict={'fontsize':14})
        ax[1].set_ylabel('GFP Transcript Count',labelpad=15,fontdict={'fontsize':14})
except ModuleNotFoundError:
    print('please install the plotting libraries: pip install biocrnpyler[all]')

```

# Future Work

- Want to do more rigorous validation of multi-tx mechanism. As part of that, I first plan on comparing RPU data of existing consitutive promoters and non-native polymerase expression systems e.g. T5 with what the multi-tx model would predict.
- Is there data out there that would help validate the predicted RNAp occupancy of genes from the multi-tx model (same story for multi-tl model)?
- Eventually want to develop a multi-tx mechanism that can be used for TF-mediated transcription.
- Start looking towards validating multi-tl model, currently plan on seeing what data and parameters I could scrape from existing resource e.g. BCD RBS binding rates from biocrnpyler.
- Do some model comparison using bioscrape inference, where I generate parameters with RPU data model and fit parameters (vary known and unknown parameters) from the MTX model. Definitely very interesting for deriving isomerization rates as that seems hard to come by in literature and seems to be a key part of the model in determining system behaviour (Pre-SS and SS dynamics).

# Mentioned Papers and Parameter Resources

T7 parameters:
- Promoter Binding and Unbinding: Jia, Y., Kumar, A., & Patel, S. S. (1996). Equilibrium and Stopped-flow Kinetic Studies of Interaction between T7 RNA Polymerase and Its Promoters Measured by Protein and 2-Aminopurine Fluorescence Changes. Journal of Biological Chemistry , 271(48), 30451–30458. https://doi.org/10.1074/jbc.271.48.30451
- Isomerization Rate: Skinner, G. M., Baumann, C. G., Quinn, D. M., Molloy, J. E., & Hoggett, J. G. (2004). Promoter Binding, Initiation, and Elongation By Bacteriophage T7 RNA Polymerase: A SINGLE-MOLECULE VIEW OF THE TRANSCRIPTION CYCLE . Journal of Biological Chemistry , 279(5), 3239–3244. https://doi.org/10.1074/jbc.M310471200
- Translation Rate (T7p is VERY fast): Kochetkov, S. N., Rusakova, E. E., & Tunitskaya, V. L. (1998). Recent studies of T7 RNA polymerase mechanism. FEBS Letters, 440(3), 264–267. https://doi.org/https://doi.org/10.1016/S0014-5793(98)01484-7

Translation Parameters:
- RBS Binding and Unbinding: Chandra, F., & Del Vecchio, D. (2016). The Effects of Ribosome Autocatalysis and Negative Feedback in Resource Competition. bioRxiv. https://doi.org/10.1101/042127
- Isomerization Rate: Draper, D. E. (1993). Mechanisms of Translational Initiation and Repression in Prokaryotes BT  - The Translational Apparatus: Structure, Function, Regulation, Evolution. In K. H. Nierhaus, F. Franceschi, A. R. Subramanian, V. A. Erdmann, & B. Wittmann-Liebold (Eds.) (pp. 197–207). Boston, MA: Springer US. https://doi.org/10.1007/978-1-4615-2407-6_19

Misc:
- mRNA Degredation: Roy, B., & Jacobson, A. (2013). The intimate relationships of mRNA decay and translation. Trends in Genetics : TIG, 29(12), 691–699. https://doi.org/10.1016/j.tig.2013.09.002
- RPU Data: Nielsen, A. A. K., Der, B. S., Shin, J., Vaidyanathan, P., Paralanov, V., Strychalski, E. A., … Voigt, C. A. (2016). Genetic circuit design automation. Science, 352(6281), aac7341. https://doi.org/10.1126/science.aac7341 and Qi, L., Haurwitz, R. E., Shao, W., Doudna, J. A., & Arkin, A. P. (2012). RNA processing enables predictable programming of gene expression. Nature Biotechnology, 30(10), 1002–1006. https://doi.org/10.1038/nbt.2355
- Protein:mRNA ratios: Taniguchi, Y., Choi, P. J., Li, G.-W., Chen, H., Babu, M., Hearn, J., … Xie, X. S. (2010). Quantifying E. coli Proteome and Transcriptome with Single-Molecule Sensitivity in Single Cells. Science, 329(5991), 533 LP – 538. https://doi.org/10.1126/science.1188308


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)

# Hide the code completely

# from IPython.display import HTML
# tag = HTML('''<style>
# div.input {
#     display:none;
# }
# </style>''')
# display(tag)
```

Toggle cell visibility here.

```python
%matplotlib notebook
import pylab
import matplotlib.pyplot as plt
import math
import sympy as sym
import numpy as np
import ipywidgets as widgets
import control as control
import math as math
from ipywidgets import interact
from IPython.display import Latex, display, Markdown
```

## Linearization of a function

### Introduction

> Linearization is defined as a process of finding a linear approximation of a function at a certain point. The linear approximation of a function is obtained by the Taylor expansion around the point of interest in which only the first two terms are kept. Linearization is an effective method for approximating the output of a function $y=f(x)$ at any $x=x_0+\Delta x$ based on the value and the slope of the function at $x=x_0+\Delta x$, given that $f(x)$ is differentiable on $[x_0,x_0+\Delta x]$ (or $[x_0+\Delta x,x_0]$) and that $x_0$ is close to $x_0+\Delta x$. In short, linearization approximates the output of a function near $x=x_0$. (source: [Wikipedia](https://en.wikipedia.org/wiki/Linearization))

In this example, linearization is defined as:

\begin{equation}
    f(x)\approx f(x_0)+f^{\prime}(x_0) \cdot (x-x_0),
\end{equation}

where $f^{\prime}=\frac{f(x_0+h)-f(x_0)}{h}$ ($h$ is set to $0.01$ in order to calculate the derivative).

Unit step function is defined as:

\begin{equation}
    u(x) =
    \begin{cases}
      0; & \text{$x<0$}\\
      1; & \text{$x\geq0$}
    \end{cases},
\end{equation}

and unit ramp function:

\begin{equation}
    r(x) =
    \begin{cases}
      0; & \text{$x<0$}\\
      x; & \text{$x\geq0$}
    \end{cases}.
\end{equation}

---

### How to use this notebook?
Move the slider to change the value of $x_0$, i.e. the $x$ value at which you want to linearize the function.

```python
# sinus, step, ramp, x^2, sqrt(x)
functionSelect = widgets.ToggleButtons(
    options=[('sine function', 0), ('unit step function', 1), ('unit ramp function', 2), ('parabolic function', 3), ('square root function', 4)],
    description='Select: ')

fig = plt.figure(num='Linearization of a function')
fig.set_size_inches((9.8, 3))
fig.set_tight_layout(True)
f1 = fig.add_subplot(1, 1, 1)

f1.grid(which='both', axis='both', color='lightgray')

f1.set_xlabel('$x$')
f1.set_ylabel('$f(x)$')

f1.axhline(0,Color='black',linewidth=0.5)
f1.axvline(0,Color='black',linewidth=0.5)

func_plot, = f1.plot([],[])
tang_plot, = f1.plot([],[])
point_plot, = f1.plot([],[])

f1.set_xlim((-5,5))
f1.set_ylim((-6,6))

def create_draw_functions(x0,index):
    x=np.linspace(-5,5,1001)
    h=0.001 # equal to \Delta x

    global func_plot, tang_plot, point_plot

    if index==0:
        y=np.sin(x)
        fprime=(np.sin(x0+h)-np.sin(x0))/h
        tang=np.sin(x0)+fprime*(x-x0)
        fx0=np.sin(x0)
    elif index==1:
        y=np.zeros(1001)
        y[510:1001]=1
    elif index==2:
        y=np.zeros(1001)
        y[500:1001]=np.linspace(0,5,501)
    elif index==3:
        y=x*x
        fprime=((x0+h)*(x0+h)-(x0*x0))/h
        tang=x0*x0+fprime*(x-x0)
        fx0=x0*x0
    elif index==4:
        x1=np.linspace(0,5,500)
        y=np.sqrt(x1)
        if x0>=0:
            fprime=(np.sqrt(x0+h)-np.sqrt(x0))/h
            tang=np.sqrt(x0)+fprime*(x-x0)
            fx0=np.sqrt(x0)

    f1.lines.remove(func_plot)
    f1.lines.remove(tang_plot)
    f1.lines.remove(point_plot)

    if index == 0:
        func_plot, = f1.plot(x,y,label='$f(x)=sin(x)$',color='C0')
        tang_plot, = f1.plot(x,tang,'--r',label='tangent')
        point_plot, = f1.plot(x0,fx0,'om',label='$x_0$')
        for txt in f1.texts:
                txt.set_visible(False)
    elif index == 1: # in case of the unit step function
        if x0==0:
            func_plot, = f1.step(x,y,label='$f(x)=u(x)$',color='C0')
            tang_plot, = f1.plot([],[])
            point_plot, = f1.plot([],[])
            f1.text(0.1,1.3,'Linearization at $x_0=0$ is not possible!',fontsize=14)
        elif x0<0:
            tang=np.zeros(1001)
            func_plot, = f1.step(x,y,label='$f(x)=u(x)$',color='C0')
            tang_plot, = f1.plot(x,tang,'--r',label='tangent')
            point_plot, = f1.plot(x0,[0],'om',label='$x_0$')
            for txt in f1.texts:
                txt.set_visible(False)
        elif x0>0:
            tang=np.ones(1001)
            func_plot, = f1.step(x,y,label='$f(x)=u(x)$',color='C0')
            tang_plot, = f1.plot(x,tang,'--r',label='tangent')
            point_plot, = f1.plot(x0,[1],'om',label='$x_0$')
            for txt in f1.texts:
                txt.set_visible(False)
    elif index==2: # in case of the ramp
        if x0<0:
            tang=np.zeros(1001)
            func_plot, = f1.plot(x,y,label='$f(x)=R(x)$',color='C0')
            tang_plot, = f1.plot(x,np.zeros(1001),'--r',label='tangent')
            point_plot, = f1.plot(x0,[0],'om',label='$x_0$')
            for txt in f1.texts:
                txt.set_visible(False)
        elif x0>=0:
            tang=x
            func_plot, = f1.plot(x,y,label='$f(x)=R(x)$',color='C0')
            tang_plot, = f1.plot(x,tang,'--r',label='tangent')
            point_plot, = f1.plot(x0,x0,'om',label='$x_0$')
            for txt in f1.texts:
                txt.set_visible(False)
    elif index==3:
        func_plot, = f1.plot(x,y,label='$f(x)=x^2$',color='C0')
        tang_plot, = f1.plot(x,tang,'--r',label='tangent')
        point_plot, = f1.plot(x0,fx0,'om',label='$x_0$')
        for txt in f1.texts:
                txt.set_visible(False)
    elif index==4: #in case of the square root function
        if x0<0:
            for txt in f1.texts:
                txt.set_visible(False)
            func_plot, = f1.plot(x1,y,label='$f(x)=\sqrt{x}$',color='C0')
            tang_plot, = f1.plot([],[])
            point_plot, = f1.plot([],[])
            f1.text(-4.9,1.3,'Square root function is not defined for $x<0$!',fontsize=14)
        else:
            func_plot, = f1.plot(x1,y,label='$f(x)=\sqrt{x}$',color='C0')
            tang_plot, = f1.plot(x,tang,'--r',label='tangent')
            point_plot, = f1.plot(x0,fx0,'om',label='$x_0$')
            for txt in f1.texts:
                txt.set_visible(False)

    if (index==1) and x0==0 or (index==4 and x0<0):
        display(Markdown('See comment on the figure.'))
    else:
        k=round(((tang[-1]-tang[0])/(x[-1]-x[0])),3)
        n=round(((tang[-1]-k*x[-1])),3)
        display(Markdown('Equation of the tangent: $y=%.3fx+%.3f$.'%(k,n)))

    f1.legend()

    f1.relim()
    f1.relim()
    f1.autoscale_view()
    f1.autoscale_view()

x0_slider = widgets.FloatSlider(value=1, min=-5, max=5, step=0.2, description='$x_0$',
                          continuous_update=True, layout=widgets.Layout(width='auto', flex='5 5 auto'),readout_format='.1f')

input_data = widgets.interactive_output(create_draw_functions, {'x0':x0_slider, 'index':functionSelect})

def update_sliders(index):
    global x0_slider

    x0val = [0.5, 0.5, 1, 1, 5, 10]
    x0slider.value = x0val[index]

input_data2 = widgets.interactive_output(update_sliders, {'index':functionSelect})

display(functionSelect)

display(x0_slider,input_data)

# display(Markdown("The system can be represented as $f(x)=5$ for small excursions of x about x0."))
```

    <IPython.core.display.Javascript object>

    ToggleButtons(description='Select: ', options=(('sine function', 0), ('unit step function', 1), ('unit ramp fu…

    FloatSlider(value=1.0, description='$x_0$', layout=Layout(flex='5 5 auto', width='auto'), max=5.0, min=-5.0, r…

    Output()

```python

```


```python
%run notebook_setup
```

# Getting started with The Joker

*The Joker* (pronounced Yo-kurr) is a highly specialized Monte Carlo (MC) sampler that is designed to generate converged posterior samplings for Keplerian orbital parameters, even when your data are sparse, non-uniform, or very noisy. This is *not* a general MC sampler, and this is *not* a Markov Chain MC sampler like `emcee`, or `pymc3`: This is fundamentally a [rejection sampler](https://en.wikipedia.org/wiki/Rejection_sampling) with some tricks that help improve performance for the two-body problem.

*The Joker* shines over more conventional MCMC sampling methods when your radial velocity data is imprecise, non-uniform, sparse, or has a short baseline: In these cases, your likelihood function will have many, approximately equal-height modes that are often spaced widely, all properties that make conventional MCMC bork when applied to this problem. In this tutorial, we will not go through the math behind the sampler (most of that is covered [in the original paper](https://arxiv.org/abs/1610.07602)). However, some terminology is important to know for the tutorial below or for reading the documentation. Most relevant, the parameters in the two-body problem (Kepler orbital parameters) split into two sets: nonlinear and linear parameters. The nonlinear parameters are always the same in each run of The Joker: period $P$, eccentricity $e$, argument of pericenter $\omega$, and a phase $M_0$. The default linear parameters are the velocity semi-ampltude $K$, and a systemtic velocity $v_0$. However, there are ways to add additional linear parameters into the model (as described in other tutorials).

For this tutorial, we will set up an inference problem that is common to binary star or exoplanet studies, show how to generate posterior orbit samples from the data, and then demonstrate how to visualize the samples. Other tutorials demonstrate more advanced or specialized functionality included in *The Joker*, like:
- [fully customizing the parameter prior distributions](2-Customize-prior.ipynb),
- [allowing for a long-term velocity trend in the data](3-Polynomial-velocity-trend.ipynb),
- [continuing sampling with standard MCMC methods](4-Continue-sampling-mcmc.ipynb) when *The Joker* returns one or few samples,
- [simultaneously inferring constant offsets between data sources](5-Calibration-offsets.ipynb) (i.e. when using data from multiple instruments that may have calibration offsets)

But let's start here with the most basic functionality!

First, imports we will need later:

```python
import astropy.table as at
from astropy.time import Time
import astropy.units as u
from astropy.visualization.units import quantity_support
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

import thejoker as tj
```

```python
# set up a random generator to ensure reproducibility
rnd = np.random.default_rng(seed=42)
```

## Loading radial velocity data

To start, we need some radial velocity data to play with. Our ultimate goal is to construct or read in a `thejoker.RVData` instance, which is the main data container object used in *The Joker*. For this tutorial, we will use a simulated RV curve that was generated using a separate script and saved to a CSV file, and we will create an `RVData` instance manually.

Because we previously saved this data as an Astropy [ECSV](http://docs.astropy.org/en/latest/api/astropy.io.ascii.Ecsv.html#astropy.io.ascii.Ecsv) file, the units are provided with the column data and read in automatically using the [astropy.table read/write interface](http://docs.astropy.org/en/latest/table/index.html):

```python
data_tbl = at.QTable.read('data.ecsv')
data_tbl[:2]
```

The full simulated data table has many rows (256), so let's randomly grab 4 rows to work with:

```python
sub_tbl = data_tbl[rnd.choice(len(data_tbl), size=4, replace=False)]
```

```python
sub_tbl
```

It looks like the time column is given in Barycentric Julian Date (BJD), so in order to create an `RVData` instance, we will need to create an `astropy.time.Time` object from this column:

```python
t = Time(sub_tbl['bjd'], format='jd', scale='tcb')
data = tj.RVData(t=t, rv=sub_tbl['rv'], rv_err=sub_tbl['rv_err'])
```

We now have an `RVData` object, so we could continue on with the tutorial. But as a quick aside, there is an alternate, more automatic (automagical?) way to create an `RVData` instance from tabular data: `RVData.guess_from_table`. This classmethod attempts to guess the time format and radial velocity column names from the columns in the data table. It is very much an experimental feature, so if you think it can be improved, please open an issue in the [GitHub repo for The Joker](https://github.com/adrn/thejoker/issues). In any case, here it successfully works:

```python
data = tj.RVData.guess_from_table(sub_tbl)
```

One of the handy features of `RVData` is the `.plot()` method, which generates a quick view of the data:

```python
_ = data.plot()
```

The data are clearly variable! But what orbits are consistent with these data? I suspect many, given how sparse they are! Now that we have the data in hand, we need to set up the sampler by specifying prior distributions over the parameters in *The Joker*.

## Specifying the prior distributions for The Joker parameters

The prior *pdf* (probability distribution function) for *The Joker* is controlled and managed through the `thejoker.JokerPrior` class. The prior for *The Joker* is fairly customizable and the initializer for `JokerPrior` is therefore pretty flexible; usually too flexible for typical use cases. We will therefore start by using an alternate initializer defined on the class, `JokerPrior.default()`, that provides a simpler interface for creating a `JokerPrior` instance that uses the default prior distributions assumed in *The Joker*. In the default prior:

$$
\begin{align}
&p(P) \propto \frac{1}{P} \quad ; \quad P \in (P_{\rm min}, P_{\rm max})\\
&p(e) = B(a_e, b_e)\\
&p(\omega) = \mathcal{U}(0, 2\pi)\\
&p(M_0) = \mathcal{U}(0, 2\pi)\\
&p(K) = \mathcal{N}(K \,|\, \mu_K, \sigma_K)\\
&\sigma_K = \sigma_{K, 0} \, \left(\frac{P}{P_0}\right)^{-1/3} \, \left(1 - e^2\right)^{-1/2}\\
&p(v_0) = \mathcal{N}(v_0 \,|\, \mu_{v_0}, \sigma_{v_0})\\
\end{align}
$$

where $B(.)$ is the beta distribution, $\mathcal{U}$ is the uniform distribution, and $\mathcal{N}$ is the normal distribution.

Most parameters in the distributions above are set to reasonable values, but there are a few required parameters for the default case: the range of allowed period values (``P_min`` and ``P_max``), the scale of the ``K`` prior variance ``sigma_K0``, and the standard deviation of the $v_0$ prior ``sigma_v``. Let's set these to some arbitrary numbers. Here, I chose the value for ``sigma_K0`` to be typical of a binary star system; if using The Joker for exoplanet science, you will want to adjust this correspondingly.

```python
prior = tj.JokerPrior.default(
    P_min=2*u.day, P_max=1e3*u.day,
    sigma_K0=30*u.km/u.s,
    sigma_v=100*u.km/u.s)
```

Once we have the prior instance, we need to generate some prior samples that we will then use *The Joker* to rejection sample down to a set of posterior samples. To generate prior samples, use the `JokerSamples.sample()` method. Here, we'll generate a lare number of samples to use:

```python
prior_samples = prior.sample(size=250_000,
                             random_state=rnd)
prior_samples
```

This object behaves like a Python dictionary in that the parameter values can be accessed via their key names:

```python
prior_samples['P']
```

```python
prior_samples['e']
```

They can also be written to disk or re-loaded using this same class. For example, to save these prior samples to the current directory to the file "prior_samples.hdf5":

```python
prior_samples.write("prior_samples.hdf5", overwrite=True)
```

We could then load the samples from this file using:

```python
tj.JokerSamples.read("prior_samples.hdf5")
```

## Running The Joker

Now that we have a set of prior samples, we can create an instance of The Joker and use the rejection sampler:

```python
joker = tj.TheJoker(prior, random_state=rnd)
joker_samples = joker.rejection_sample(data, prior_samples,
                                       max_posterior_samples=256)
```

This works by either passing in an instance of `JokerSamples` containing the prior samples, or by passing in a filename that contains `JokerSamples` written to disk. So, for example, this is equivalent:

```python
joker_samples = joker.rejection_sample(data, "prior_samples.hdf5",
                                       max_posterior_samples=256)
```

The ``max_posterior_samples`` argument above specifies the maximum number of posterior samples to return. It is often helpful to set a threshold here in cases when your data are very uninformative to avoid generating huge numbers of samples (which can slow down the sampler considerably).

In either case above, the ``joker_samples`` object returned from ``rejection_sample()`` is also an instance of the ``JokerSamples`` class, but now contains posterior samples for all nonlinear and linear parameters in the model:

```python
joker_samples
```

## Plotting The Joker orbit samples over the input data

With posterior samples in Keplerian orbital parameters in hand for our data set, we can now plot the posterior samples over the input data to get a sense for how constraining the data are. *The Joker* comes with a convenience plotting function, ``plot_rv_curves``, for doing just this:

```python
_ = tj.plot_rv_curves(joker_samples, data=data)
```

It has various options to allow customizing the style of the plot:

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
_ = tj.plot_rv_curves(joker_samples, data=data,
                      plot_kwargs=dict(color='tab:blue'),
                      data_plot_kwargs=dict(color='tab:red'),
                      relative_to_t_ref=True, ax=ax)
ax.set_xlabel(f'BMJD$ - {data.t.tcb.mjd.min():.3f}$')
```

Another way to visualize the samples is to plot 2D projections of the sample values, for example, to plot period against eccentricity:

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))

with quantity_support():
    ax.scatter(joker_samples['P'],
               joker_samples['e'],
               s=20, lw=0, alpha=0.5)

ax.set_xscale('log')
ax.set_xlim(prior.pars['P'].distribution.a,
            prior.pars['P'].distribution.b)
ax.set_ylim(0, 1)

ax.set_xlabel('$P$ [day]')
ax.set_ylabel('$e$')
```

But is the true period value included in those distinct period modes returned by *The Joker*? When generating the simulated data, I also saved the true orbital parameters used to generate the data, so we can load and over-plot it:

```python
import pickle
with open('true-orbit.pkl', 'rb') as f:
    truth = pickle.load(f)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))

with quantity_support():
    ax.scatter(joker_samples['P'],
               joker_samples['e'],
               s=20, lw=0, alpha=0.5)

    ax.axvline(truth['P'], zorder=-1, color='tab:green')
    ax.axhline(truth['e'], zorder=-1, color='tab:green')
    ax.text(truth['P'], 0.95, 'truth', fontsize=20,
            va='top', ha='left', color='tab:green')

ax.set_xscale('log')
ax.set_xlim(prior.pars['P'].distribution.a,
            prior.pars['P'].distribution.b)
ax.set_ylim(0, 1)

ax.set_xlabel('$P$ [day]')
ax.set_ylabel('$e$')
```

It indeed looks like there are posterior samples from *The Joker* in the vicinity of the true value. Deciding what to do next depends on the problem you would like to solve. For example, if you just want to get a sense of how multi-modal the posterior *pdf* over orbital parameters is, you might be satisfied with the number of samples we generated and the plots we made in this tutorial. However, if you want to fully propagate the uncertainty in these orbital parameters through some other inference (for example, to transform the samples into constraints on companion mass or other properties), you may want or need to generate a lot more samples. To start, you could change ``max_posterior_samples`` to be a much larger number in the ``rejection_sample()`` step above. But I have found that in many cases, you need to run with many, many more (e.g., 500 million) prior samples. To read more, check out the next tutorial!


```python
import sys
sys.path.append('..') # Add src to path
import os
os.environ["HDF5_USE_FILE_LOCKING"]='FALSE'
import datetime
import h5py
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from matplotlib.animation import FuncAnimation
from IPython.display import Image

import tensorflow as tf

from src.display import get_cmap
from src.utils import make_log_dir

# comment these out if you don't have cartopy
import cartopy.feature as cfeature
from src.display.cartopy import make_ccrs,make_animation

from make_dataset import NowcastGenerator,get_nowcast_train_generator,get_nowcast_test_generator

from unet_benchmark import create_model
from unet_benchmark import nowcast_mae, nowcast_mse
from IPython.core.debugger import set_trace
```

# Radar Nowcast Challenge

This notebook will describe the SEVIR radar nowcasting challenge.  Nowcasts (https://en.wikipedia.org/wiki/Nowcasting_(meteorology)) are short-term forecast of weather variables typically measured by weather radar or satellite.   Nowcasts are different from traditional weather forecasts (like those you see on the news, or on your phones weather app) in that they are based (mostly) on statistical extrapolations of recent data, rather than full physics-based numerical weather prediction (NWP) models.   The advantages of using statistical extrapolation techniques is that they generally run much faster than NWP models (seconds compared to hours), and because of this, nowcasts are able to  leverage the most recently observed data and use it for forecasting.  Because of this extremely low latency, nowcast out-perform traditional weather models in terms of accuracy and precision for short look-aheads (typically 1-2 hours at most).

Below is an example that a nowcast generated using the baseline model created in this notebook.  The first half of the movie is observed radar VIL.  The second half of the movie is the forecast generated by a model trained in this notebook.   The challenge is to create the best possible nowcast algorithm given the data in SEVIR.

### Contents:
* [Python environment](#env)
* [Problem Statement](#problem)
* [Training & testing dataset](#datasets)
    * [Visualizing training samples](#vistraining)
* [Model Training](#training)
* [Visualize Result](#visualize)
* [Forecast Scoring](#scoring)
* [Applying to full-sized images](#fillimgs)

## Python Environment

Running this notebook requires a python environment that includes the modules described here.  You can setup the environment with anaconda via

```
conda create --name sevir_challenges
conda activate sevir_challenges
conda install --file requirements.txt
```

## Problem Statement

The goal of this challenge is to predict 12 future frames of radar images in 5 minute steps (`vil` in this case) given the previous 13 images of `vil` (also in 5 minute steps) in addition to other modalities in SEVIR.

## Training and testing datasets

### MIT Supercloud

If you have access to MIT supercloud, the data for this challenge can be accessed directly:

### AWS Download

Sample training and testing datasets are provided on the SEVIR S3 bucket.  These can be obtained by running the following cell.

**This downloads and decompresses files that will take up roughly 40 GB so make sure you have sufficient disk space**

```python
data_path="/scratch/li.baol/SEVIR"
```

```python
# Target locations of sample training & testing data
DEST_TRAIN_FILE= os.path.join(data_path,'data/processed/nowcast_training_000.h5')
DEST_TRAIN_META=os.path.join(data_path, 'data/processed/nowcast_training_000_META.csv')
DEST_TEST_FILE= os.path.join(data_path, 'data/processed/nowcast_testing_000.h5')
DEST_TEST_META= os.path.join(data_path, 'data/processed/nowcast_testing_000_META.csv')
```

```python
# THIS DOWNLOADS APPROXIMATELY 40 GB DATASETS (AFTER DECOMPRESSION)
import boto3
from botocore.handlers import disable_signing
import tarfile
resource = boto3.resource('s3')
resource.meta.client.meta.events.register('choose-signer.s3.*', disable_signing)
bucket=resource.Bucket('sevir')

print('Dowloading sample training data')
if not os.path.exists(DEST_TRAIN_FILE):
    bucket.download_file('data/processed/nowcast_training_000.h5.tar.gz',DEST_TRAIN_FILE+'.tar.gz')
    bucket.download_file('data/processed/nowcast_training_000_META.csv',DEST_TRAIN_META)
    with tarfile.open(DEST_TRAIN_FILE+'.tar.gz') as tfile:
        tfile.extract('data/processed/nowcast_training_000.h5',data_path)
else:
    print('Train file %s already exists' % DEST_TRAIN_FILE)
print('Dowloading sample testing data')
if not os.path.exists(DEST_TEST_FILE):
    bucket.download_file('data/processed/nowcast_testing_000.h5.tar.gz',DEST_TEST_FILE+'.tar.gz')
    bucket.download_file('data/processed/nowcast_testing_000_META.csv',DEST_TEST_META)
    with tarfile.open(DEST_TEST_FILE+'.tar.gz') as tfile:
        tfile.extract('data/processed/nowcast_testing_000.h5',data_path)
else:
    print('Test file %s already exists' % DEST_TEST_FILE)
```

    Dowloading sample training data
    Train file /scratch/li.baol/SEVIR/data/processed/nowcast_training_000.h5 already exists
    Dowloading sample testing data
    Test file /scratch/li.baol/SEVIR/data/processed/nowcast_testing_000.h5 already exists

See the contents of the training data file using `h5ls`:

```python
#!h5ls /scratch/li.baol/SEVIR/data/processed/nowcast_training_000.h5
```

This file contains four modalities you can use as input to a nowcast model (keys start with `IN_` -- `vil`,`ir_ir107`,`ir069`, or `lght`).  The challenge is to use these inputs to predict the output vil `OUT_vil`.

If you have downloaded the full SEVIR dataset, you can extract even larger datasets of this format, or even modify the dataset for your puprose.  We assume SEVIR is located in the directory `$SEVIR_ROOT`.  To make your own dataset, run the script `make_dataset.py`:

```
python make_dataset.py --input_types vil --output_types vil --sevir_data $SEVIR_ROOT/data/ --sevir_catalog $SEVIR_ROOT/CATALOG.csv --output_location ../data/processed/
```

The rest of this notebook assumes you are using the sample training and test files located in the directory `../data/processed/` relative to this notebook.  Adjust as needed.

### Exploring the training dataset

First load some samples from the training data and assocaited metadata for georeferencing and plotting.  If the full training set is too large to fit into memory, adjust `N_TRAIN` and `N_TEST` below.  This will use `TRAIN_VAL_FRAC * N_TRAIN` training samples for a validation set.

NOTE:  For this baseline we will only use `IN_vil` frames as inputs.  However the other modalities may be useful for nowcasting.

```python
# Control how many samples are read.   Set to -1 to read all 5000 samples.
N_TRAIN=-1
TRAIN_VAL_FRAC=0.8
#set_trace()
N_TEST=-1
```

```python
# Loading data takes a few minutes
with h5py.File(DEST_TRAIN_FILE,'r') as hf:
    Nr = N_TRAIN if N_TRAIN>=0 else hf['IN_vil'].shape[0]
    X_train = hf['IN_vil'][:Nr]
    Y_train = hf['OUT_vil'][:Nr]
    training_meta = pd.read_csv(DEST_TRAIN_META).iloc[:Nr]
    X_train,X_val=np.split(X_train,[int(TRAIN_VAL_FRAC*Nr)])
    Y_train,Y_val=np.split(Y_train,[int(TRAIN_VAL_FRAC*Nr)])
    training_meta,val_meta=np.split(training_meta,[int(TRAIN_VAL_FRAC*Nr)])
#set_trace()
with h5py.File(DEST_TEST_FILE,'r') as hf:
    Nr = N_TEST if N_TEST>=0 else hf['IN_vil'].shape[0]
    X_test = hf['IN_vil'][:Nr]
    Y_test = hf['OUT_vil'][:Nr]
    testing_meta=pd.read_csv(DEST_TEST_META).iloc[:Nr]
```

Get a sample batch of image sequences.  Note the shape`[N,L,L,T]` where `N` is the batch size, `L` is the size of the image patch, and `T` is the number of time frames in the video (1 time step = 5 minutes).  The VIL images in SEVIR are scaled to the range [0-255], and can be decoded back to orginal units following the [SEVIR Tutorial](https://nbviewer.jupyter.org/github/MIT-AI-Accelerator/eie-sevir/blob/master/examples/SEVIR_Tutorial.ipynb).

```python
batch_size,batch_num=8,0
bs,be=batch_size*batch_num,batch_size*(batch_num+1)
X,Y,meta = X_train[bs:be],Y_train[bs:be],training_meta.iloc[bs:be]
print('Input X:',X.shape)
print('Output Y:',Y.shape)
print('META:',meta.shape)
```

    Input X: (8, 384, 384, 13)
    Output Y: (8, 384, 384, 12)
    META: (8, 14)

We'll start by visualizing some of the data.  This includes some colormaps for visualization.  The following plots a few frames with the custom colormap provided for the `vil` type:

```python
batch_index=1
cmap,norm,vmin,vmax=get_cmap('vil')
fig,axs=plt.subplots(1,4,figsize=(12,4))
for i in range(4):
    axs[i].imshow(X[batch_index,:,:,4*i],origin='lower',cmap=cmap,norm=norm,vmin=vmin,vmax=vmax) # every 4th frame
    axs[i].set_axis_off()
```

The variable `meta` contains additional information about the event pictured:

```python
meta.iloc[batch_index]
```

    Unnamed: 0                                                    19989
    id                                                          S819072
    time_utc                                        2019-04-09 17:10:00
    episode_id                                                 136472.0
    event_id                                                   819072.0
    event_type                                        Thunderstorm Wind
    minute_offsets    -60:-55:-50:-45:-40:-35:-30:-25:-20:-15:-10:-5...
    llcrnrlat                                                 41.434316
    llcrnrlon                                                -77.367946
    urcrnrlat                                                 43.830965
    urcrnrlon                                                -71.510706
    proj              +proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63...
    height_m                                                   384000.0
    width_m                                                    384000.0
    Name: 1, dtype: object

Since this event has an associated `event_id`, more information can be obtained from NCEI storm event database associated to the year of the event (2019 in this case).  Look for the files named `StormEvents_details` for the particular years of interest here:

[https://www1.ncdc.noaa.gov/pub/data/swdi/stormevents/csvfiles](https://www1.ncdc.noaa.gov/pub/data/swdi/stormevents/csvfiles)

If you download relevant files, additional information for the event can be obtained by searching these files for the `episode_id` or `event_id` found above.

The metadata can be used to georeference the patches shown above.  If cartopy is installed in your environment, map features can be added to these plots:

```python
# REQUIRES CARTOPY (and an internet connection if this is your first time using cartopy)
proj,img_extent = make_ccrs(meta.iloc[batch_index])
xll,xur=img_extent[0],img_extent[1]
yll,yur=img_extent[2],img_extent[3]
fig,axs=plt.subplots(1,4,figsize=(12,4), subplot_kw={'projection':proj})
for i in range(4):
    axs[i].set_xlim((xll,xur))
    axs[i].set_ylim((yll,yur))
    axs[i].imshow(X[batch_index,:,:,4*i],interpolation='nearest',
                  origin='lower',transform=proj,extent=[xll,xur,yll,yur],
                  cmap=cmap,norm=norm,vmin=vmin,vmax=vmax) # every 4th frame
    axs[i].add_feature(cfeature.STATES)

```

The function `make_animation` creates a short movie loop:

```python
%%capture off
anim = make_animation(X[batch_index],meta.iloc[batch_index],title='Inputs')
anim.save('imgs/input_animation.gif', writer='imagemagick', fps=6)
anim = make_animation(Y[batch_index],meta.iloc[batch_index],title='Target')
anim.save('imgs/output_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/input_animation.gif')
```

    <IPython.core.display.Image object>

```python
Image('imgs/output_animation.gif')
```

    <IPython.core.display.Image object>

## Create baseline model

In this section we will initialize a baseline model and train it.

### Initialize U-Net model

The baseline model used in this notebook is the U-Net model based on the model described [here](https://proceedings.neurips.cc//paper/2020/hash/fa78a16157fed00d7a80515818432169-Abstract.html).

This model has the following structure, which takes a sequence of VIL images, and outputs another sequence that covers the next hour of time:

Code for the model is defined in `unet_benchmark.py`.

The following cell defines training hyperparamters we'll use to start.  These can be adjusted as needed.

```python
# Add more as needed
params={
    'start_neurons'   :16,      # Controls size of hidden layers in CNN, higher = more complexity
    'activation'      :'relu',  # Activation used throughout the U-Net,  see https://www.tensorflow.org/api_docs/python/tf/keras/activations
    'loss'            :'mae',   # Either 'mae' or 'mse', or others as https://www.tensorflow.org/api_docs/python/tf/keras/losses
    'loss_weights'    :0.021,    # Scale for loss.  Recommend squaring this if using MSE
    'opt'             :tf.keras.optimizers.Adam,  # optimizer, see https://www.tensorflow.org/api_docs/python/tf/keras/optimizers
    'learning_rate'   :0.001,   # Learning rate for optimizer
    'num_epochs'      :10,       # Number of epochs to train for
    'batch_size'      :8        # Size of batches during training
}
```

```python
unet = create_model(start_neurons=params['start_neurons'],activation=params['activation'])
unet.summary()
```

For a loss function, we'll start with mean absolute error scaled by an estimate of the standard deviation computed over the training data ($\sigma^{-1} \approx 0.021$)
\begin{equation}
L(Y,\hat{Y})=\frac{1}{\sigma}\sum_{i} ||Y-\hat{Y}||_1
\end{equation}

```python
opt=params['opt'](learning_rate=params['learning_rate'])
unet.compile(optimizer=opt, loss=params['loss'],loss_weights=[params['loss_weights']])
```

### Train Model

We'll set up two callbacks.  The first saves checkpoints whenever `val_loss` is minimized.  The other sets up tensorboard.

**NOTE:  Model checkpoints will be saved under the `experiments` directory in a newly created time-stamped directory.  A link named `experiments/latest` is also created that points to the most recent experiment.**

```python
# Training 10 epochs takes around 10-20 minutes on GPU
num_epochs=params['num_epochs']
batch_size=params['batch_size']
exprmt_dir=make_log_dir('experiments')

callbacks=[
    tf.keras.callbacks.ModelCheckpoint(exprmt_dir+'/nowcast-unet-{epoch:04d}-{val_loss:04f}.hdf5',
                    monitor='val_loss',save_best_only=True),
    tf.keras.callbacks.TensorBoard(log_dir=exprmt_dir+'/tboardlogs')
]

history = unet.fit(x=X_train, y=Y_train,
                  batch_size=batch_size,
                  epochs=num_epochs,
                   callbacks=callbacks,
                  validation_data=(X_val, Y_val))
```

Plot model training performance for the first 10 epochs

```python
plt.plot(history.history['loss'],label='Train loss')
plt.plot(history.history['val_loss'],label='Val loss')
plt.legend()
```

```python
# save model for later use
unet.save('unet_10epochs.h5')
```

```python
# Reload previously saved model
# Make sure to use `custom_objects` for custom loss functions

# NOTE: The best performing model (on val set) across all epochs was saved under experiments/latest,
# so you can also load that instead

unet = tf.keras.models.load_model('unet_10epochs.h5',custom_objects={})
```

## Visualize model on test samples

Start by visualizing the resulting model on test samples

```python
batch_size,batch_num=8,4
bs,be=batch_size*batch_num,batch_size*(batch_num+1)
x_test,y_test,meta = X_test[bs:be],Y_test[bs:be],testing_meta.iloc[bs:be]
```

```python
%%capture off
bidx=5
anim = make_animation(x_test[bidx],meta.iloc[bidx],title='Inputs')
anim.save('imgs/input_test_animation.gif', writer='imagemagick', fps=6)
anim = make_animation(y_test[bidx],meta.iloc[bidx],title='Target')
anim.save('imgs/output_test_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/input_test_animation.gif')
```

```python
Image('imgs/output_test_animation.gif')
```

Now compute the prediction

```python
%%capture off
y_pred = unet.predict(x_test)
anim = make_animation(y_pred[bidx],meta.iloc[bidx],title='Prediction')
anim.save('imgs/pred_test_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/pred_test_animation.gif')
```

What do you think?  You can adjust `batch_num` and `bidx` to view other cases from the test set.

The good:
* Seems to be moving the weather in the correct direction, and retaining some of the structure

The bad:
* A lot of the detail gets washed out, and for longer leads, the more intense sections of the storm wash away.

## Forecast Scoring

Let's apply some more quantitative verifications to establish baseline performance.

To define metrics used, let $T_i(t)$ and $F_i(t)$ represent target and forecast images, respectively, for test sample $i$ at lead times $t=5,10,...,60$ minutes.  The first metric to consider is mean-absolute error (MAE):

\begin{equation}
MAE(t) = \frac{1}{L^2 N_{test}}\sum_{i=1}^{N_{test}} ||T_i(t)-F_i(t)||_1
\end{equation}

MAE is nice because it is simple, however it often fails to capture forecast skill in predicting fine detail in storm, such as severe storm cores.  Therefore, we'll also apply more standard forecast verification metrics (see [https://www.cawcr.gov.au/projects/verification/](https://www.cawcr.gov.au/projects/verification/)).

Given a threshold $\tau$, we binarize both the target and forecast images, and label each pixel as a "hit" ($H$) if both target and forecast are $\geq \tau$, a "miss" ($M$) if $T\geq\tau$ and $F<\tau$, a "false alarm" ($FA$) if $T<\tau$ and $F\geq \tau$, and a correct rejection if both $T$ and $F$ are less than $\tau$.

These pixel scores are rolled up into Critical Success Index (CSI) (which is similar to Intersection over Union, which is another common name for this metric) and is computed as

\begin{equation}
CSI(t;\tau) = \frac{\#H}{\#H + \#M + \#FA}
\end{equation}

A CSI of 1 represents a perfect pixel-to-pixel forecast.  For scoring VIL nowcasts, we'll average this over three thresholds that are used in practice [16,74,133] representing low, medium, and high strom intensity.  We'll refer to this average as "mCSI", computed for each lead time $t$.

\begin{equation}
mCSI(t) = \frac{1}{3}(CSI(t;16) + CSI(t;74) + CSI(t;133))
\end{equation}

A "final" test scores can be computed by averaging $MAE$ and $mCSI$ over all lead times in 0-60 minutes.

This repo contains some code to compute CSI for the test set.

```python
# Run unet over the test set
y_pred = unet.predict(X_test,batch_size=4)
```

```python
# Compute metrics over the test set separately for each lead (takes a few minutes)
from tqdm import tqdm
from src.metrics import critical_success_index
CSI = lambda yt,yp,tau: critical_success_index(yt.astype(np.float32),
                                               yp.astype(np.float32),np.array([tau],dtype=np.float32)).numpy()

THRESHOLDS=np.array([16,74,133],dtype=np.float32)
nT=Y_test.shape[3]
mae_t=[]
csi_t = []
for t in tqdm(range(nT)):
    mae_t.append(np.mean(tf.keras.losses.MAE(Y_test[:,:,:,t:t+1],y_pred[:,:,:,t:t+1])))
    csi_t.append(np.mean([CSI(Y_test[:,:,:,t:t+1], y_pred[:,:,:,t:t+1],tau) for tau in THRESHOLDS]))

```

We'll also include another standard baseline model in nowcasting, the *persistence model*.  This model represents the "do nothing" forecast where the last image in the input sequence is just repeated for every future frame.

```python
# Also compute scores of persistence model
nT=Y_test.shape[3]
pers_mae_t=[]
pers_csi_t = []
for t in tqdm(range(nT)):
    pers_mae_t.append(np.mean(tf.keras.losses.MAE(Y_test[:,:,:,t:t+1],X_test[:,:,:,-1:].astype(np.float32))))
    pers_csi_t.append(np.mean([CSI(Y_test[:,:,:,t:t+1], X_test[:,:,:,-1:],tau) for tau in THRESHOLDS]))
```

```python
t=np.arange(5,65,5)
fig,ax=plt.subplots(1,2,figsize=(15,5))
ax[0].plot(t,mae_t,label='U-Net Baseline')
ax[0].plot(t,pers_mae_t,label='Persistence')
ax[0].set_xlabel('Lead time (minutes)')
ax[0].set_ylabel('Nowcast MAE (lower is better)')
ax[0].set_title('MAE')
ax[0].legend()

ax[1].plot(t,csi_t,label='U-Net Baseline')
ax[1].plot(t,pers_csi_t,label='Persistence')
ax[1].set_xlabel('Lead time (minutes)')
ax[1].set_ylabel('Nowcast mCSI (higher is better)')
ax[1].set_title('mCSI')
ax[1].legend()
plt.show()

print('Average MAE of U-Net on test set: ',np.mean(mae_t))
print('Average mCSI of U-Net on test set: ',np.mean(csi_t))

```

The baseline model beats persistence (phew!) but it can still be better!

##### Can you create a model that improves upon the above test scores?

## (Optional) Applying to US-sized images

The model trained above is fully convolutional and can be apllied to full-sized radar mosaics.  Sample US radar mosaics of VIL can be obtained from the MRMS system [https://mrms.ncep.noaa.gov/data/2D/VIL/](https://mrms.ncep.noaa.gov/data/2D/VIL/).  The past 24 hours of files are avaialble online for download.  The model *may* also work for other radar fields, e.g. reflectivity, so long as they are first normalized to the range [0-255] first.

MRMS data is not a part of this repo so you'll need to download and prepared this data yourself.  Before feeding this data to the CNN, some preparation is required first.  First, the MRMS files are available in `grib2` format and need to be converted to a numpy array.  Second, the units need to be converted to the [0-255] scale.  Code for doing these steps can be found in [this gist](https://gist.github.com/markveillette/048d7e6b6be35d26bac37378bd66ae01), which provides a script for doing this preprocessing.

```python
# MRMS updates every 2 minutes.  We will approximate 5 minute updates by switching between 4 and 6 minute
# steps.

# UPDATE WITH YOUR OWN LOCATION AND FILES
mrms_root='../data/MRMS'
files=[
    'MRMS_VIL_00.50_20201125-160039.grib2.npy','MRMS_VIL_00.50_20201125-160436.grib2.npy',
    'MRMS_VIL_00.50_20201125-161039.grib2.npy','MRMS_VIL_00.50_20201125-161442.grib2.npy',
    'MRMS_VIL_00.50_20201125-162039.grib2.npy','MRMS_VIL_00.50_20201125-162434.grib2.npy',
    'MRMS_VIL_00.50_20201125-163037.grib2.npy','MRMS_VIL_00.50_20201125-163436.grib2.npy',
    'MRMS_VIL_00.50_20201125-164041.grib2.npy','MRMS_VIL_00.50_20201125-164438.grib2.npy',
    'MRMS_VIL_00.50_20201125-165039.grib2.npy','MRMS_VIL_00.50_20201125-165436.grib2.npy',
    'MRMS_VIL_00.50_20201125-170042.grib2.npy']
vil=[]
for f in files:
    vil.append(np.load(mrms_root+'/'+f)[np.newaxis,:,:,np.newaxis])
vil=np.concatenate(vil,axis=3)
vil = np.pad(vil, ((0,0),(2,2),(4,4),(0,0))) # needed to make sure downsample layers can be done cleanly
m=vil[0:1,2:-2,4:-4,0]==255
vil[vil==255]=0
```

```python
# Load weights of your own saved model, but make sure input_shape is None so it works on full sized images
unet = create_model(start_neurons=params['start_neurons'],
                    activation=params['activation'],input_shape=(None,None,13))
unet.load_weights('experiments/unet_baseline/nowcast-unet-0039-0.133934.hdf5')
```

```python
# If this overwhelms your GPU, you might need to compute forecast in smaller subsections
forecast = unet.predict(vil)
```

```python
%%capture off
import cartopy.crs as crs
from cartopy.crs import Globe
import cartopy.feature as cfeature
from matplotlib import animation, rc
def make_mrms_animation(frames,img_type='vil',fig=None,
                   interval=100,title=None,**kwargs):
    """
    Makes animation of MRMS data
    """
    ellps='WGS84'
    globe=Globe(ellipse=ellps)
    proj=crs.PlateCarree(globe=globe)
    img_extent=(-14471586.0, -6679194.0322265625, 2226397.75, 6122594.1611328125) # MRMS extent
    xll,xur=img_extent[0],img_extent[1]
    yll,yur=img_extent[2],img_extent[3]
    if fig is None:
        fig=plt.gcf()
    ax=fig.add_subplot(1,1,1,projection=proj)
    ax.set_xlim((xll,xur))
    ax.set_ylim((yll,yur))
    cmap,norm,vmin,vmax=get_cmap(img_type)
    ax.add_feature(cfeature.STATES)
    ax.add_feature(cfeature.LAKES, alpha=0.5)
    ax.add_feature(cfeature.RIVERS, alpha=0.5)
    ax.add_feature(cfeature.COASTLINE)
    ax.add_feature(cfeature.BORDERS )

    im=ax.imshow(frames[:,:,0], interpolation='nearest',
          origin='upper', extent=[xll,xur,yll,yur],
          transform=proj,cmap=cmap,norm=norm,vmin=vmin,vmax=vmax);

    def init():
        return (im,)
    def animate(i):
        im.set_data(frames[:,:,i]);
        ftype='Analysis' if i<13 else 'Forecast'
        ax.set_title(ftype)
        return (im,)
    return animation.FuncAnimation(fig, animate, init_func=init,
                                   frames=range(frames.shape[2]),
                                   interval=interval, blit=True);

frames = np.concatenate((vil[:,2:-2,4:-4,:],forecast[:,2:-2,4:-4,:]),axis=3)
for i in range(frames.shape[3]):
    frames[:,:,:,i][m]=np.nan
plt.figure(figsize=(15,7))
anim = make_mrms_animation(frames[0])
anim.save('imgs/mrms_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/mrms_animation.gif')
```


#  Pawnee Fire Analysis

> * 🔬 Data Science
* 🖥 Requires RasterAnalytics Portal Configuration
* 🖥 Requires GeoEnrichment Portal Configuration
* 🖥 Requires GeoAnalytics Portal Configuration

The Pawnee Fire was a large wildfire that burned in Lake County, California. The fire started on June 23, 2018 and burned a total of 15,185 acres (61 km2) before it was fully contained on July 8, 2018.

Table of Contents

Pawnee Fire Analysis

Remote Sensing using Sentinel-2 data

Data Preparation

Visual Assessment

Quantitative Assessment

Impact Assessment

Conclusion

## Remote Sensing using Sentinel-2 data

```python
from datetime import datetime
import warnings

from IPython.display import HTML
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import arcgis
from arcgis import GIS
from arcgis.raster.functions import *
from arcgis.geoanalytics.use_proximity import create_buffers
from arcgis.geoenrichment import enrich
from arcgis.features import SpatialDataFrame
from arcgis.raster.analytics import create_image_collection
from arcgis.raster.analytics import list_datastore_content

gis= GIS("home")
```

### Data Preparation

In this analysis, we will be using Sentinel-2 data.

Sentinel-2 is an Earth observation mission developed by ESA as part of the Copernicus Programme to perform terrestrial observations in support of services such as forest monitoring, land cover change detection, and natural disaster management.

In this analysis data downloaded from https://earthexplorer.usgs.gov/ is used for creating hosted image service.
We add the data to the datastore and we then run the create_image_collection function which creates a collection with the input_rasters specified and publishes the collection as an image service.

We use list_datasore_content() in order to see the contents in the rasterstore.

```python
list_datastore_content("/rasterStores/LocalRasterStore")
```

    ['/rasterStores/LocalRasterStore/Hosted_GeneratedRasterProduct_AANS52.crf/',
     '/rasterStores/LocalRasterStore/L1C_T10SEJ_A015697_20180624T190108.zip',
     '/rasterStores/LocalRasterStore/me.txt',
     '/rasterStores/LocalRasterStore/m_7013bfcd7273e0a4a779fce061167d5c/',
     '/rasterStores/LocalRasterStore/m_b5f745ad6ef601d5e6adf104c8b4ef70/',
     '/rasterStores/LocalRasterStore/m_d0221b176ab5ed2398828b8079d62ef8/',
     '/rasterStores/LocalRasterStore/pawnee_fire_multispectral/',
     '/rasterStores/LocalRasterStore/pool_chips_1/',
     '/rasterStores/LocalRasterStore/pool_chips_2/',
     '/rasterStores/LocalRasterStore/S2A_MSIL1C_20180624T184921_N0206_R113_T10SEJ_20180624T234856.SAFE/',
     '/rasterStores/LocalRasterStore/S2B_MSIL1C_20180622T185919_N0206_R013_T10SEJ_20180622T205930.SAFE/',
     '/rasterStores/LocalRasterStore/sentinel_data/']

```python
sentinel_collection = create_image_collection(image_collection="pawnee_fire_multispectral",
                      input_rasters=["/rasterStores/LocalRasterStore/S2A_MSIL1C_20180624T184921_N0206_R113_T10SEJ_20180624T234856.SAFE",
                                     "/rasterStores/LocalRasterStore/S2B_MSIL1C_20180622T185919_N0206_R013_T10SEJ_20180622T205930.SAFE"],
                      raster_type_name="Sentinel-2",
                      raster_type_params={"productType":"All","processingTemplate":"Multispectral"},
                      context={"image_collection_properties":{"imageCollectionType":"Satellite"},"byref":True}, gis = gis)
```

```python
sentinel = sentinel_collection.layers[0]
```

### Select before and after rasters

```python
aoi = {'spatialReference': {'latestWkid': 3857, 'wkid': 102100},
 'xmax': -13643017.100720055,
 'xmin': -13652113.10708598,
 'ymax': 4739654.477447927,
 'ymin': 4731284.622850712}
arcgis.env.analysis_extent = aoi
sentinel.extent = aoi
```

```python
selected = sentinel.filter_by(where="acquisitiondate BETWEEN timestamp '2018-06-15 00:00:00' AND timestamp '2018-06-24 19:59:59'",
                              geometry=arcgis.geometry.filters.intersects(aoi))

df = selected.query(out_fields="*", order_by_fields="OBJECTID ASC").df
df['AcquisitionDate'] = pd.to_datetime(df['AcquisitionDate'], unit='ms')
df.tail(40)
```

      AcquisitionDate
      Category
      CenterX
      CenterY
      CloudCover
      CreationTime
      Creator
      GroupName
      HighPS
      LowPS
      ...
      SOrder
      SensingOrbit
      SensorName
      Shape_Area
      Shape_Length
      StereoID
      Tag
      Thumbnail
      ZOrder
      SHAPE

      0
      2018-06-24 18:49:21.024
      1
      -1.362127e+07
      4.758219e+06
      0
      1548778472000
      rjackson
      MTD_MSIL1C
      60
      60
      ...
      None
      113
      Sentinel-2A
      2.008228e+10
      562920.108277
      None
      MS
      None
      None
      {'rings': [[[-13549665.930399999, 4828694.125]...

      1
      2018-06-22 18:59:19.024
      1
      -1.364188e+07
      4.762904e+06
      0
      1548778472000
      rjackson
      MTD_MSIL1C
      60
      60
      ...
      None
      13
      Sentinel-2B
      1.417943e+10
      488606.729731
      None
      MS
      None
      None
      {'rings': [[[-13611168.835900001, 4689604.0966...

2 rows × 26 columns

```python
prefire = sentinel.filter_by('OBJECTID=2')
midfire = sentinel.filter_by('OBJECTID=1')
```

## Visual Assessment

```python
truecolor = extract_band(midfire, [4,3,2])
truecolor
```

### Visualize Burn Scars

We extract the [13, 12, 4] bands to improve visibility of fire and burn scars. This band combination pushes further into the SWIR range of the electromagnetic spectrum, where there is less susceptibility to smoke and haze generated by a burning fire.

```python
extract_band(midfire, [13,12,4])

```

For comparison, the same area before the fire started shows no burn scar.

```python
extract_band(prefire, [13,12,4])

```

## Quantitative Assessment

The **Normalized Burn Ratio (NBR)** can be used to delineate the burnt areas and identify the severity of the fire.

The formula for the NBR is very similar to that of NDVI except that it uses near-infrared band 9 and the short-wave infrared band 13:
\begin{align}
{\mathbf{NBR}} = \frac{\mathbf{B9} - \mathbf{B13}}{\mathbf{B9} + \mathbf{B13} + \mathbf{WS}} \\
\end{align}

The NBR equation was designed to be calcualted from reflectance, but it can be calculated from radiance and digital_number_(dn) with changes to the burn severity table below. The WS parameter is used for water suppression, and is typically 2000.

For a given area, NBR is calculated from an image just prior to the burn and a second NBR is calculated for an image immediately following the burn. Burn extent and severity is judged by taking the difference between these two index layers:

\begin{align}
{\Delta \mathbf{NBR}} = \mathbf{NBR_{prefire}} - \mathbf{NBR_{postfire}} \\
\end{align}

The meaning of the ∆NBR values can vary by scene, and interpretation in specific instances should always be based on some field assessment. However, the following table from the USGS FireMon program can be useful as a first approximation for interpreting the NBR difference:

| \begin{align}{\Delta \mathbf{NBR}}  \end{align}      | Burn Severity |
| ------------- |:-------------:|
| 0.1 to 0.27   | Low severity burn |
| 0.27 to 0.44  | Medium severity burn |
| 0.44 to 0.66 | Moderate severity burn |
| > 0.66 | High severity burn |

[Source: http://wiki.landscapetoolbox.org/doku.php/remote_sensing_methods:normalized_burn_ratio]

### Use Band Arithmetic and Map Algebra

In order to perform raster analysis on raw pixel value, we filter out the scenes from the sentinel image service again and create new layers

```python
nbr_prefire  = band_arithmetic(prefire, "(b9 - b13) / (b9 + b13 + 2000)")
nbr_postfire = band_arithmetic(midfire, "(b9 - b13) / (b9 + b13 + 2000)")

nbr_diff = nbr_prefire - nbr_postfire
```

```python
burnt_areas = colormap(remap(nbr_diff,
                             input_ranges=[0.1,  0.27,  # low severity
                                           0.27, 0.44,  # medium severity
                                           0.44, 0.66,  # moderate severity
                                           0.66, 1.00], # high severity burn
                             output_values=[1, 2, 3, 4],
                             no_data_ranges=[-1, 0.1], astype='u8'),
                       colormap=[[4, 0xFF, 0xC3, 0], [3, 0xFA, 0x8E, 0], [2, 0xF2, 0x55, 0], [1, 0xE6, 0,    0]])
```

```python
# Visualize burnt areas
burnt_areas
```

With this, we have computed the NBR on scenes from before and after the burn, and computed the NBR difference to identify places that have been affected by the fire. We've also normalized the values to match a burn severity index, and applied a color map that brings out the extent of fire damage.

### Area calculation

```python
pixx = (aoi['xmax'] - aoi['xmin']) / 1200.0
pixy = (aoi['ymax'] - aoi['ymin']) / 450.0

res = burnt_areas.compute_histograms(aoi, pixel_size={'x':pixx, 'y':pixy})

numpix = 0
histogram = res['histograms'][0]['counts'][1:]
for i in histogram:
    numpix += i
```

### Report burnt area

```python
sqmarea = numpix * pixx * pixy # in sq. m
acres = 0.00024711 * sqmarea   # in acres

HTML('<h3>Fire has consumed <font color="red">{:,} acres</font>  till {}</h3>.' \
     .format(int(acres), df.iloc[-1]['AcquisitionDate'].date()))
```

Fire has consumed 3,569 acres  till 2018-06-22
.

```python
%matplotlib inline

plt.title('Distribution by severity', y=-0.1)
plt.pie(histogram, labels=['Low Severity', 'Medium Severity', 'Moderate Severity', 'High Severity']);
plt.axis('equal');
```

### Visualize burnt areas

```python
firemap = gis.map()
firemap.extent = aoi
firemap.add_layer([truecolor, burnt_areas])

firemap
```

### Persist the burnt areas layer in the GIS

If required, using the save(), we can persist the output in the gis as a new layer. This uses distributed raster analysis to perform the analysis at the source resolution.

```python
burnt_areas = burnt_areas.save()
```

## Raster to Feature layer conversion

Use Raster Analytics and Geoanalytics to convert the burnt area raster to a feature layer. The `to_features()` method converts the raster to a feature layer and `create_buffers()` fills holes in the features and dissolves them to output one feature that covers the extent of the Pawnee Fire.

```python
burnt_areas = burnt_areas.layers[0]
fire_item = burnt_areas.to_features(output_name='Pawnee_Fire_Feature_Layer', gis=gis)
fire_layer = fire_item.layers[0]

```

```python
fire = create_buffers(fire_layer, 100, 'Meters', dissolve_option='All', multipart=True, output_name='PawneeFireArea_Buffer')
fire = fire.layers[0]
```

## Visualize Feature Layer

```python
vectormap = gis.map()
vectormap.basemap = 'dark-gray'
vectormap.extent  = aoi

vectormap.add_layer(fire)
vectormap
```

## Impact Assessment

### Assess Human Impact

```python
from arcgis import geometry

sdf = SpatialDataFrame.from_layer(fire)

fire_geometry = sdf.iloc[0].SHAPE
sa_filter = geometry.filters.intersects(geometry=fire_geometry, sr=4326)

def age_pyramid(df):
    %matplotlib inline
    warnings.simplefilter(action='ignore', category=FutureWarning)
    pd.options.mode.chained_assignment = None
    plt.style.use('ggplot')

    df = df[[x for x in impacted_people.columns if 'MALE' in x or 'FEM' in x]]
    sf = pd.DataFrame(df.sum())
    age = sf.index.str.extract('(\d+)').astype('int64')
    f = sf[sf.index.str.startswith('FEM')]
    m = sf[sf.index.str.startswith('MALE')]
    sf = sf.reset_index(drop = True)
    f = f.reset_index(drop = True)
    m = m.reset_index(drop = True)
    sf['age'] = age
    f["age"] = age
    m["age"] = age
    f = f.sort_values(by='age', ascending=False).set_index('age')
    m = m.sort_values(by='age', ascending=False).set_index('age')

    popdf = pd.concat([f, m], axis=1)
    popdf.columns = ['F', 'M']
    popdf['agelabel'] = popdf.index.map(str) + ' - ' + (popdf.index+4).map(str)
    popdf.M = -popdf.M

    sns.barplot(x="F", y="agelabel", color="#CC6699", label="Female", data=popdf, edgecolor='none')
    sns.barplot(x="M",  y="agelabel", color="#008AB8", label="Male",   data=popdf,  edgecolor='none')
    plt.ylabel('Age group')
    plt.xlabel('Number of people');
    return plt;
```

### Age Pyramid of Affected Population

```python
impacted_people = enrich(sdf, 'Age')
age_pyramid(impacted_people);
```

# Conclusion

In this notebook example, we used Sentinel-2 data in order to perform remote sensing. For this we filtered out pre and post fire scenes. Using extract_band() we carried out visual assessment of the burnt area. We then computed the NBR on these scenes and computed the NBR difference to identify places that have been affected by the fire, using raster functions. We also normalized the values to match the burn severity index, applied a color map raster function that brings out the extent of fire damage and calculated the burnt area. Finally, we carried out a human impact assessment by plotting the age pyramid of affected population