```python
import matplotlib.pylab as plt
import numpy as np
from scipy.integrate import solve_ivp, simps

font = {'font.family' : 'normal',
        'font.weight' : 'normal',
        'font.size'   : 20}
plt.rcParams.update(font)
```

```python
def plot_dose_tumor(t, y, administered_dose_rate=None, received_dose_rate=None,
                    prolif_fraction=None, show_hlines=True, labels=[]):
    subplot_cnt = 0
    has_dose_rate = isinstance(administered_dose_rate, np.ndarray) or isinstance(received_dose_rate, np.ndarray)
    has_prolif_fraction = isinstance(prolif_fraction, np.ndarray)
    if has_dose_rate and has_prolif_fraction and y.ndim==1:
        fig, axes = plt.subplots(3, 1, figsize=(15, 10), sharex=True, gridspec_kw = {'height_ratios':[1, 3, 1]})
        N = y
    elif has_dose_rate and y.ndim==1:
        fig, axes = plt.subplots(2, 1, figsize=(15, 10), sharex=True, gridspec_kw = {'height_ratios':[2, 3]})
        N = y
    elif has_dose_rate and has_prolif_fraction and y.ndim>1:
        fig, axes = plt.subplots(3, 1, figsize=(15, 10), sharex=True, gridspec_kw = {'height_ratios':[1, 3, 1]})
        N = y[0,:]
        N2= y[1,:]
    elif has_dose_rate and y.ndim>1:
        fig, axes = plt.subplots(2, 1, figsize=(15, 10), sharex=True, gridspec_kw = {'height_ratios':[2, 3]})
        N = y[0,:]
        N2= y[1,:]

    # plot dose
    if has_dose_rate:
        if isinstance(administered_dose_rate, np.ndarray):
            axes[0].plot(t, administered_dose_rate, label='administered dose I(t)')
        if isinstance(received_dose_rate, np.ndarray):
            axes[0].plot(t, received_dose_rate, label='dose acting on tumor C(t)')
        axes[0].set_ylabel('dose rate')
        axes[0].legend()
        axes[0].grid()
        axes[0].axhline(y=0, c='k', linestyle='--')

    # plot tumor cell population
    if y.ndim==1:
        axes[1].plot(t, N)
    else:
        for dim in range(y.shape[0]):
            if len(labels)==y.shape[0]:
                axes[1].plot(t, y[dim, :], label=labels[dim])
            else:
                axes[1].plot(t, y[dim, :])
    axes[1].set_ylabel('tumor cells $N(t)$')
    axes[1].set_yscale('log')
    N_min = np.min(N)
    N_max = np.max(N)
    axes[1].set_ylim(0.1, 5*N_max )
    axes[1].set_yticks(np.logspace(0, int(np.log10(N_max)), num=(int(np.log10(N_max))+1)))
    axes[1].axhline(y=1, c='k', linestyle='--')
    axes[1].legend()
    axes[1].grid()

    if has_prolif_fraction:
        axes[2].plot(t, prolif_fraction)
        axes[2].set_ylabel('Gompertz Growth $G$')
        axes[2].set_xlabel('time [days]')
        axes[2].grid()
    else:
        axes[1].set_xlabel('time [days]')

    if isinstance(dose_rate, np.ndarray) and show_hlines:
        for i, d in enumerate(dose_rate):
            if d>0:
                for ax in axes:
                    ax.axvline(x=t[i], color='r', linestyle=':')
    plt.subplots_adjust(bottom=0.1, right=0.8, top=0.9, hspace=0.05)

    return axes
```

# Cancer Growth & Therapy Models

An important application of cancer growth models is for treatment optimization.
Here we will review and implement some of the treatment models that lead to the development of important treatment paradigms.

Some of the content of this notebook is based on [[1]](#Benzekry_2015) which provides an overview of the historic development of mathematical models for chemotherapy.

## Log-Kill Hypothesis

Skipper, Schable and Wilcox introduced the first theoretical concepts for the optimal design of chemotherapy schedules in the 1960.
They observed that L1210 leukemic cells in mice grow *exponentially*,  until a lethal tumor volume of about $10^9$ cells [[2]](#SKIPPER_1964).
Exponential growth implies a constant doubling time, or more generally, a constant fractional increase, regardless of tumor size.

The Skipper-Schabel-Wilcox model postulates that **"a given dose of a given drug will kill approximately the same fraction, not the same number, of widely different-sized tumor cell populations -- so long as they are similarly exposed and both the growth fraction and the proportion of drug-resistant phenotypes are the same."**

This model resulted in development of the **log kill hypothesis** which  states that the same fraction of tumor cells is killed by a given dose of chemotherapy kills, regardless of the size of the tumor at the time of treatment.

Reduction of the cell population by a constant fraction, corresponds to a reduction by a constant amount on a logarithmic scale.
For instance, reducing a population of $N=10^4$ cells to $N'=10^3$ cells corresponds to a 90% decrease in cell number, or a decrease by 1 on a logarithmic scale:

\begin{align}
\frac{N}{N'}&=10  &\rightarrow & &\text{log kill}&=\log_{10}\frac{N}{N'}=1 \\
&&&&\frac{N-N'}{N} &=1-\frac{1}{10}=90\%
\end{align}
This is called a "one log-kill" effect.

### Model

Consider the following model:

$$\frac{d N}{d t}= \rho N - \kappa\, C(t)\, N $$

where $N$ is the *number of tumor cells*, $\rho$ is the *exponential growth rate*, $C(t)$ the rate by which a dose $D$ of some therapeutic agent is acting on the the tumor and $\kappa$ the *sensitivity* of tumor cells to this agent.

Here, we assume that the effect of the agent is immediate and not affected by metabolism.
In this case, $C(t)$ is identical to the instantaneous dose rate $I(t)$ at which the agent enters the body:
$$C(t)=I(t)=\frac{d D}{d t}$$

Units:
\begin{align}
[N] &= 1  & [\rho] &= \frac{1}{T}  & [C]&=\frac{[D]}{T} & [\kappa] &= \frac{1}{[D]}
\end{align}

If we assume that $C(t)$ is non-zero only in some time interval $t\in [t_{\text{start}}, t_{\text{end}}]$ during which treatment is given, above model becomes

$$\frac{d N}{d t}=
\begin{cases}
\rho N & \text{for } t < t_{\text{start}}, \, t >t_\text{end}\\
\rho N - \kappa\, C(t)\, N  & \text{for } t_{\text{start}} \leq t \leq t_\text{end}\\
\end{cases}
$$

and integration yields:

$$N(t)=
\begin{cases}
N_0 \exp(\rho\, t)& \text{for } t < t_{\text{start}}, \, t > t_\text{end}\\
N_0 \exp(\rho\, t - \kappa\int_{t_\text{start}}^{t} C(\tau) \,d\tau) & \text{for } t_{\text{start}} \leq t \leq t_\text{end}\\
\end{cases}
$$

Let's examine how the 'log kill' depends on the time of the treatment relative to the growth stage of the tumor.

\begin{align}
\frac{N(t_\text{start})}{N(t_\text{end})}&=\frac{N_0 \exp(\rho\, t_\text{start})}{N_0 \exp(\rho\, t_\text{end} - \kappa\int_{t_\text{start}}^{t_\text{end}} C(\tau) \,d\tau)}
= \frac{\exp(\kappa\int_{t_\text{start}}^{t} C(\tau) \,d\tau)}{\exp(\rho\, (t_\text{end} - t_\text{start}))}\\
\log_{e}\frac{N(t_\text{start})}{N(t_\text{end})}&= \kappa\,\int_{t_\text{start}}^{t_\text{end}}C(\tau)\, d\tau - \rho\,(t_{\text{end}}-t_{\text{start}})
\end{align}

Assume that total dose $D$ can be delivered instantaneously (relative to the relevant time scale), i.e. $C(t)=\delta(t-t_{treat})\, D$ and $t_{\text{start}} \leq t_\text{treat} \leq t_{\text{end}}$:

$$\log_{e}\frac{N(t_\text{start})}{N(t_\text{end})}= \kappa\, D - \rho\,(t_{\text{end}}-t_{\text{start}})$$

Since $\log_{b}(a)=\frac{\log_{x}(a)}{\log_x(b)}$:

$$\text{log kill} = \log_{10}\frac{N(t_\text{start})}{N(t_\text{end})}= \frac{\kappa\, D - \rho\,(t_{\text{end}}-t_{\text{start}})}{\log_e(10)}\, ,$$

which means that the killing ratio *log kill* depends on the duration of the treatment (regrowth!) and the total dose given during treatment, but not on the time at which treatment is given.

### Implementation

```python
def model_log_kill(t, y, r=0.1, k=1, C=0):
    """
    y[0] corresponds to tumor cell number N(t)
    y_p[0] corresponds to change in tumor cell number dN/dt
    r = exponential growth factor
    C = current dose rate experienced by tumor
    k = killing efficiency
    """
    y_p = np.zeros(y.shape)
    y_p[0] = r * y[0] -  k * y[0] * C
    return y_p
```

Let's assume that the dose $D$ of the drug reaches tumor cells at constant rate throughout the duration of treatment $T$:

$$C(t) = C = \frac{D}{T}\qquad \text{for  }\, t_{\text{start}} \leq t \leq t_\text{end}$$

For a given time discretization of the simulation time period, we construct an array contains the the current rate at which the treatment is administered at each time point $t_i$.

```python
# this function and its parameters are specific to the type of dosing schedule chosen
def dose_administration_rate(t, t_start, T, D):
    """
    t: discretization of simulation time period
    t_start: start of treatment
    T: duration of treatment
    D: total dose pere treatment (given at constant rate D/T)
    """
    delta_t = t[1]-t[0]
    t_end = t_start+T
    dose_rate = np.zeros(t.shape)
    for i, ti in enumerate(t):
        if (ti>=t_start) and (ti<t_end):
            dose_rate[i] = D
    dose_rate = dose_rate/len(dose_rate[dose_rate>0])/delta_t
    return dose_rate

# discretization of simulation time period
sim_time = 100
delta_t  = 1
t = np.arange(0, sim_time+delta_t, delta_t)

# treatment schedule
t_start  = 10.0
T = 5
t_end    = t_start + T
dose_rate = dose_administration_rate(t, t_start, t_end, 1)

plt.plot(t, dose_rate, 'x')
```

```python
# Solve model

# time period
sim_time = 100
delta_t  = 0.1
t = np.arange(0, sim_time+delta_t, delta_t)

# treatment schedule
t_start  = 10.0
T = 0.1

total_dose = 1 * np.log(10)
dose_rate = dose_administration_rate(t, t_start, T, total_dose)

# other model parameters
log_growth = 0.1
kill_eff = 1
initial_tumor_cell_population = 1E6

# solution
y = np.ones(t.shape[0])*np.nan                       # initialize solution array
y[0] = np.array(initial_tumor_cell_population)       # initial values
for i in range(len(t)-1):
    sol = solve_ivp(fun=lambda t, y: model_log_kill(t, y, r=log_growth, k=kill_eff, C=dose_rate[i]),
                    t_span=[t[i],t[i+1]], y0=[y[i]])
    y[i+1] = sol.y[-1][-1]

# plot results
plot_dose_tumor(t, y, received_dose_rate=dose_rate)

# compute log kill
t_end = t_start + T
i_start = (np.abs(t-t_start)).argmin()
i_end = (np.abs(t-t_end)).argmin()
log_kill = np.log10(y[i_start]/y[i_end+1])
print("log kill between time '%.2f' and '%.2f': %f"%(t_start, t_end, log_kill))

```

Assume that the entire dose $D$ is delivered in a single simulation time step.
If the simulation time step is large compared to the time scale of tumor regrowth, we can assume that

$$\text{log kill} = \log_{10}\frac{N(t_\text{start})}{N(t_\text{stop})} \approx \frac{\kappa\, D}{\log_e(10)} $$
and thus
$$\kappa\, D \approx \text{log kill} \cdot \log_e(10)$$

So to achieve *1 log kill*, $\kappa\, D = 1 \cdot \log_e(10)$

### Repeated Treatment Administration

Often, therapy is administered in cycles with a certain repetition period.

```python
def dose_administration_rate_cycles(t, t_start, T, n_cycles, cycle_period, D_cycle):
    delta_t = t[1]-t[0]
    steps_per_cycle = int(cycle_period/delta_t)
    dose_rate = np.zeros(t.shape)
    # find start time of first cycle
    start_time_index = (np.abs(t-t_start)).argmin()
    # compute dose rate for each cycle
    for i_cycle in range(n_cycles):
        i_start = start_time_index+i_cycle*steps_per_cycle
        i_end   = start_time_index+(i_cycle+1)*steps_per_cycle
        for i in range(i_start, i_end):
            if (t[i]>=t[i_start]) and (t[i]<t[i_start]+T):
                dose_rate[i] = D_cycle
        # normalize total dose given in cycle to D_cycle
        dose_rate_cycle = dose_rate[i_start:i_end]
        #print(len(dose_rate_cycle[dose_rate_cycle>0]))
        dose_rate_cycle = dose_rate_cycle/len(dose_rate_cycle[dose_rate_cycle>0])/delta_t
        dose_rate[i_start:i_end] = dose_rate_cycle
    return dose_rate
```

```python
# Solve model

# time period
sim_time = 300
delta_t  = 0.1
t = np.arange(0, sim_time+delta_t, delta_t)

# treatment schedule
t_start  = 0.0
T = 0.1

n_cycles = 15
cycle_period = 14
total_dose = 1 * np.log(10)
# n_cycles = 30
# cycle_period = 7
# total_dose = 0.5 * np.log(10)

print("accumulative dose: %f"%(n_cycles*total_dose/np.log(10)))

dose_rate = dose_administration_rate_cycles(t, t_start, T, n_cycles, cycle_period, total_dose)

# other model parameters
log_growth = 0.1
kill_eff = 1
initial_tumor_cell_population = 1E6

# solution
y = np.ones(t.shape[0])*np.nan                       # initialize solution array
y[0] = np.array(initial_tumor_cell_population)       # initial values
for i in range(len(t)-1):
    sol = solve_ivp(fun=lambda t, y: model_log_kill(t, y, r=log_growth, k=kill_eff, C=dose_rate[i]),
                    t_span=[t[i],t[i+1]], y0=[y[i]])
    y[i+1] = sol.y[-1][-1]

# plot results
plot_dose_tumor(t, y, administered_dose_rate=dose_rate, received_dose_rate=None)
```

Tumor eradication and *cure* is assumed when $N(t)<1$ at some time point $t$.

**Note:**

ODE-based models describe average quantities.
When the magnitudes of those quantities assume values that correspond to (few) single objects, e.g. single cells, or fractions, this description may become unsuitable.

First, fractions of single objects may not be meaningful. For example, the ode model here predicts recurrence from tumor cell numbers $N<<1$.
Second, stochastic effects gain importance and may critically influence the systems' behavior.

---
**Exercise**

For an initial tumor burden of $10^6$ cells and $\rho=0.1$:
- What single dose of therapeutic agent (in units of 'log kill') would need to be given to eradicate the tumor immediately?
- Due to toxicity, the maximum instantaneous dose is limited to the 'maximum tolerable dose' (MTD). Suppose that the MTD can be given periodically with recovery breaks of 14 days or more in between doses and that 1 MTD corresponds to an effect of 1 log-kill.
  How would you schedule drug adminstrations (start of treatment, period and number of cycles) to achieve *cure* with as little total dose ( MTD * number of cycles ) possible.
- Would *more frequent* adminstration of *doses below the MTD* result in better predicted outcome at similar total dose?

---

Indeed, it was demonstrated [[3]](#Skipper_1965) (for Leukemia)  that a large-dose / short time shedule was superior to a chronic low-dose schedule with similar or larger dose.

However, the experimental system was limited to a specific leukemic cell line, and when applied to other (solid) tumors, the log-kill hypothesis did not lead to the expected results.
Also, this model assumes the cancer cell population to be homogeneously sensitive to the drug.

## Norton-Simon Hypothesis

Studies of tumor kinetics in the 1960 led to the realization that the growth rate of solid tumors was not constant but rather decreased with tumor volume.
It was shown that the growth curve patterns could be well described by several growth models, the most famous one of those being the Gompertz model; [[4]](#Benzekry_2014) provides a recent overview of different models and their predictive properties for experimental data.

Norton and Simon [[5]](#Norton_1977) revisited the log-kill hypothesis in the 1970 in order to extend it to growing tumors with non-constant specific growth rate.
As chemotherapy is mostly based on anti-mitotic agents, they suggest that only actively proliferating cells should be affected by treatment.

### Model

Recall the Gompertz Model:

\begin{align}
      \frac{dN}{dt} &=G(t)N(t) \tag{1}\\
      \frac{dG}{dt} &=-\gamma\, G(t) \tag{2}
\end{align}

where the population $N(t)$ grows or shrinks over time with time-dependent growth factor $G(t)$, and the growth factor $G(t)$ decays exponentially over time with constant rate $\gamma$.

The solution as derived in the lecture notes:

$$N(t) = N_0\,e^{(-\frac{G_0}{\gamma}(e^{-\gamma\, t}-1))} \tag{3}$$

with initial cell number $N_0$ and initial growth fraction $G_0$.
Imposing that the system reaches its carrying capacity $K$ as $t\rightarrow +\infty$ allows to interpret the parameters $G_0$ and $\gamma$ of this model in terms of $K$ and $N_0$:

$$\frac{G_0}{\gamma}=\ln\left(\frac{K}{N_0}\right) \tag{4}$$

We implement Norton and Simon's hypothesis by extending equation (1) of this model by a treatment term ($-\kappa\, C(t)$, as before) that only affects the proliferative fraction $G(t)$ of the tumor.

$$\frac{dN}{dt} =G(t)\, N(t)\, (1-\kappa\, C(t)) \tag{5}$$

Instead of modeling the decrease of the proliferative fraction $G$ over time, here we are interested in the relation between proliferative fraction and tumor population.

From (4) and (5):
\begin{align}
\ln\left(\frac{N}{N_0}\right) &= -\frac{G_0}{\gamma}\, e^{-\gamma\, t} + \frac{G_0}{\gamma}\\
\ln\left(\frac{N}{N_0}\right) - \ln\left(\frac{K}{N_0}\right)&= -\frac{G_0}{\gamma}\, e^{-\gamma\, t} \\
\gamma\ln\left(\frac{K}{N}\right) &= G_0\, e^{-\gamma\, t} = G(t) \tag{6}
\end{align}

Substituting (6) into (5) yields the final expression:
$$\frac{dN}{dt} =\gamma\ln\left(\frac{K}{N}\right)\, N(t)\, (1-\kappa\, C(t)) \tag{5}$$

### Implementation

```python
def model_norton_simon(t, y, r=1, K=1, k=1, C=0):
    """
    y[0] corresponds to tumor cell number N(t)
    y_p[0] corresponds to change in tumor cell number dN/dt
    r = exponential growth factor
    C = current dose rate experienced by tumor
    k = killing efficiency
    """
    y_p = np.zeros(y.shape)
    y_p[0] = r*np.log(K/y[0])* y[0] * (1 -  k * C)
    return y_p
```

```python
# Solve model

# time period
sim_time = 300
delta_t  = 0.1
t = np.arange(0, sim_time+delta_t, delta_t)

# treatment schedule
t_start  = 10.0
T = 0.1

n_cycles = 8
cycle_period = 13
total_dose = 16

dose_rate = dose_administration_rate_cycles(t, t_start, T, n_cycles, cycle_period, total_dose)

# other model parameters
gomp_growth = 0.05

kill_eff = 1
initial_tumor_cell_population = 1E7
carrying_cap = 1E10

# solution
y = np.ones(t.shape[0])*np.nan                       # initialize solution array
y[0] = np.array(initial_tumor_cell_population)       # initial values
for i in range(len(t)-1):
    sol = solve_ivp(fun=lambda t, y: model_norton_simon(t, y, r=gomp_growth, K=carrying_cap, k=kill_eff, C=dose_rate[i]),
                    t_span=[t[i],t[i+1]], y0=[y[i]])
    y[i+1] = sol.y[-1][-1]

# plot results
prolif_fraction = gomp_growth*np.log(carrying_cap/y)
axes = plot_dose_tumor(t, y, administered_dose_rate=dose_rate, received_dose_rate=None, prolif_fraction=prolif_fraction)

# add different dosing schedule
# cycle_period = 15
# dose_rate = dose_administration_rate_cycles(t, t_start, T, n_cycles, cycle_period, total_dose)
# # solution
# y2 = np.ones(t.shape[0])*np.nan                       # initialize solution array
# y2[0] = np.array(initial_tumor_cell_population)       # initial values
# for i in range(len(t)-1):
#     sol = solve_ivp(fun=lambda t, y: model_norton_simon(t, y, r=gomp_growth, K=carrying_cap, k=kill_eff, C=dose_rate[i]),
#                     t_span=[t[i],t[i+1]], y0=[y2[i]])
#     y2[i+1] = sol.y[-1][-1]

# axes[1].plot(t, y2, label='15 day cycle')

```

The Norton-Simon Hypothesis predicts superiority of densified dosing regimens:
Less intercycle time implies less regrowth, smaller size at beginning of next cycle implies larger proliferating fraction and thus larger fraction killed by the drug.
This has been confirmed in clinical trials that reduced inter-cycle time from 21 days to 14 days and showed benefit of the applying the denser regimen [[6]](#Held_2006).

Another result from this model is that multidrug combination therapies are most effective when drugs are administered sequentially rather than alternating, so that the treatment schedule for each of the drugs is 'densified'. This prediction was confirmed in clinical trials in e.g. breast cancer [[7]](#Citron_2003).

The Norton-Simon model implies that larger absolute kill of cancer cells resulting from early therapy might be counter-balanced by faster regrowth due to the higher growth rates of those smaller tumors.
This may explain observations that adjuvant chemotherapy of breast cancer has less impact on overall survival (a function of eventual tumor body burden) than on disease-free survival (time to relapse), see this [figure](https://www.ncbi.nlm.nih.gov/books/NBK20771/figure/A8245/?report=objectonly) in [[8]](#Gilewski).

## Tumor Heterogeneity

Both, the log-kill and Norton-Simon hypotheses assume the population of cancer cells to be homogeneous, with equal sensitivity to the treatment drug.
We now consider the problem of dosing for heterogeneous tumors, consisting of subpopulations of cancer cells with distinctive sensitivities to a given drug.
The following model is based on [[8]](#Hahnfeldt_2003).

\begin{align}
\frac{dN_1}{dt} & = \left(\rho_1 - \gamma_1 - \alpha\, C(t)\right)\, N_1(t) + \gamma_2\, N_2(t) \\
\frac{dN_2}{dt} & = \left(\rho_2 - \gamma_2 - \beta\, C(t)\right)\, N_2(t) + \gamma_1\, N_1(t) \\
\end{align}

Cells of subpopulation $N_1$ and $N_2$ have net proliferations $\rho_1$, $\rho_2$ and experience exposure to the treatment drug $C(t)$ different with different sensitivities $\alpha$ and $\beta$.
Cells from one subpopulation can move to the other subpopulation with transition rates $\gamma_1$ and $\gamma_2$.
This transition may for example correspond to cells moving from a non-mitotic compartment to a mitotic one and vice versa.

Due to different sensitivities, exposure to treatment will alter the relative proportion of cells in each of the subpopulations from their equilibrium proportions.
Transition rates between these subpopulations capture the tendency of these subpopulations to return to their equilibrium proportions once the toxic insult ceases.

### 'Metronomic' and irregular dosing schedules

```python
def model_heterogeneity(t, y, r1=1, r2=1, g1=0.5, g2=0.5, a=0.1, b=0.2, C=0):
    """
    y[0] corresponds to tumor cell number N1(t)
    y[1] corresponds to tumor cell number N2(t)
    y_p[0] corresponds to change in tumor cell number dN1/dt
    y_p[1] corresponds to change in tumor cell number dN2/dt
    r1, r2 = exponential growth factors
    g1, g2 = conversion rates
    a, b = drug sensitivities
    """
    y_p = np.zeros(y.shape)
    y_p[0] = (r1 - g1 - a*C) * y[0] + g2 * y[1]
    y_p[1] = (r2 - g2 - b*C) * y[1] + g1 * y[0]
    return y_p
```

```python
# time period
sim_time = 15
delta_t  = 0.1
t = np.arange(0, sim_time+delta_t, delta_t)

# treatment schedule
t_start  = 1.0
T = 0.1

dose_rate_metronomic = dose_administration_rate_cycles(t, t_start=1, T=0.1, n_cycles=9,
                                                  cycle_period=1, D_cycle=1)
dose_rate_irreg_high = dose_administration_rate_cycles(t, t_start=1, T=0.1, n_cycles=3,
                                                  cycle_period=3, D_cycle=2)
dose_rate_irreg_low  = dose_administration_rate_cycles(t, t_start=3, T=0.1, n_cycles=3,
                                                  cycle_period=3, D_cycle=1)
dose_rate_irreg = dose_rate_irreg_low + dose_rate_irreg_high

plt.plot(t, dose_rate_metronomic)
plt.plot(t, dose_rate_irreg)
# plt.plot(t, dose_rate_irreg_high)
# plt.plot(t, dose_rate_irreg_low)
```

## Solving the model

```python
# define solver for reuse
def solve(t, N0, r1, r2, g1, g2, a, b, dose_rate):
    y = np.ones((2,t.shape[0]))*np.nan                     # initialize solution array
    y[:,0] = np.array([N0,N0])                             # initial values
    for i in range(len(t)-1):
        sol = solve_ivp(fun=lambda t, y: model_heterogeneity(t, y, r1=r1, r2=r2, g1=g1, g2=g2,
                                                             a=a, b=b, C=dose_rate[i]),
                        t_span=[t[i],t[i+1]], y0=y[:,i])
        y[:,i+1] = sol.y[:,-1]

    y_sum = np.sum(y, axis=0)
    y_all = np.vstack([y,y_sum])
    return y_all
```

```python
r1 = r2 = 1.0
a = 1.0
b = 0.1
# slow resensitization
g1 = 0.2
g2 = 1.8

dose_factor = 2
N0 = 10

# solution metronomic
y_metr_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_metronomic*dose_factor)

labels = ['more sensitive', 'more resistant', 'all']

axes = plot_dose_tumor(t, y_metr_all, administered_dose_rate=dose_rate_metronomic, received_dose_rate=None,
                       labels=['metronomic -- '+label for label in labels], show_hlines=False)
```

### Response to these dosing schedules

- Two subpopulations, with high and low sensitivity, respectively.
- Treated by 'irregular' and metronomic schedule

'Irregular' schedule with higher dose upfront is competitive early on, but not in the long term.

```python
# params from fig 5
r1 = r2 = 1.0
a = 1.0
b = 0.1
g1 = 0.2
g2 = 1.8

dose_factor = 4
N0 = 10

# solution metronomic
y_metr_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_metronomic*dose_factor)

# solution irregular
y_irr_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_irreg*dose_factor)

labels = ['more sensitive', 'more resistant', 'all']
# axes = plot_dose_tumor(t, y_metr_all, administered_dose_rate=dose_rate_metronomic, received_dose_rate=None,
#                        labels=['metronomic -- '+label for label in labels], show_hlines=False)

# for i, label in enumerate(labels):
#     axes[1].plot(t, y_irr_all[i,:], label='irregular -- '+ label, line_style = '--')
# axes[1].legend()
```

```python
fig, ax = plt.subplots(1, 1, figsize=(15, 7))
# metronomic -- blue
# irregular -- red
# total population -- solid
ax.plot(t, y_metr_all[2,:], color='b', linestyle='-', label='metronomic, total population')
ax.plot(t, y_irr_all[2,:], color='r', linestyle='-', label='irregular, total population')
# more resistant -- dashed
ax.plot(t, y_metr_all[1,:], color='b', linestyle='--', label='metronomic, more resistant')
ax.plot(t, y_irr_all[1,:], color='r', linestyle='--', label='irregular, more resistant')
ax.axhline(y=1, c='k', linestyle='--')
ax.set_yscale('log')
ax.grid()
ax.legend()
```

### Effect of resensitization rate on treatment effect

- Two subpopulations, with high and low sensitivity, respectively
- Treated by 'irregular' and metronomic schedule
- two scenarios: slow vs quick resensitization

```python
# params from fig 6
r1 = r2 = 1.0
a = 1.0
b = 0.1

dose_factor = 4
N0 = 10

#== slow resensitization
g1 = 0.1
g2 = 0.9
y_metr_slow_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_metronomic*dose_factor)
y_irr_slow_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_irreg*dose_factor)

#== quick resensitization
g1 = 0.4
g2 = 3.6
y_metr_quick_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_metronomic*dose_factor)
y_irr_quick_all = solve(t, N0, r1, r2, g1, g2, a, b, dose_rate_irreg*dose_factor)

```

```python
fig, ax = plt.subplots(1, 1, figsize=(15, 7))
# metronomic -- blue
# irregular -- red
# quickly resentizing -- dashed
ax.plot(t, y_metr_quick_all[2,:], color='b', linestyle='--', label='metronomic, quickly resensitizing')
ax.plot(t, y_irr_quick_all[2,:], color='r', linestyle='--', label='irregular, quickly resensitizing')
# slowly resensitizing -- solid
ax.plot(t, y_metr_slow_all[2,:], color='b', linestyle='-', label='metronomic, slowly resensitizing')
ax.plot(t, y_irr_slow_all[2,:], color='r', linestyle='-', label='irregular, slowly resensitizing')
ax.axhline(y=1, c='k', linestyle='--')
ax.set_yscale('log')
ax.grid()
ax.legend()

```

Differential response between irregular (red) and metronomic (blue) is small in the slowly resensitizing scenario (solid) but large in the quickly resensitizing scenario (dashed).

The slow resensitizing case may be representative of tumor cell populations, whereas the quickly resensitizing case may be more representative of endothelial cells which are also affected by chemotherapy.

This differential response of quickly resensitizing populations under irregular vs metronomic treatment schedules has been proposed as explanation for the observed increased therapeutic gain under metronomic dosing:
Metronomic dosing may potentiate the anti-angiogenic properties of chemotherapeutic drugs.

# References

[1]
Benzekry, S., Pasquier, E., Barbolosi, D., Lacarelle, B., Barlési, F., André, N., Ciccolini, J., 2015. Metronomic reloaded: Theoretical models bringing chemotherapy into the era of precision medicine. Semin. Cancer Biol. 35, 53–61. https://doi.org/10.1016/j.semcancer.2015.09.002

[2]
SKIPPER, H.E., SCHABEL Jr., F.M., WILCOX, W.S., 1964. EXPERIMENTAL EVALUATION OF POTENTIAL ANTICANCER AGENTS. XIII. ON THE. Cancer Chemother. Rep. 35, 1–111.

[3]
Skipper, H.E., 1965. The effects of chemotherapy on the kinetics of leukemic cell behavior. Cancer Res. 25, 1544–1550.

[4]
Benzekry, S., Lamont, C., Beheshti, A., Tracz, A., Ebos, J.M.L., Hlatky, L., Hahnfeldt, P., 2014. Classical Mathematical Models for Description and Prediction of Experimental Tumor Growth. PLoS Comput. Biol. 10, e1003800. https://doi.org/10.1371/journal.pcbi.1003800

[5]
Norton, L., Simon, R., 1977. Tumor size, sensitivity to therapy, and design of treatment schedules. Cancer Treat. Rep. 61, 1307–1317.

[6]
Held, G., Schubert, J., Reiser, M., Pfreundschuh, M., 2006. Dose-Intensified Treatment of Advanced-Stage Diffuse Large B-Cell Lymphomas. Seminars in Hematology 43, 221–229. https://doi.org/10.1053/j.seminhematol.2006.07.003

[7]
Citron, M.L., Berry, D.A., Cirrincione, C., Hudis, C., Winer, E.P., Gradishar, W.J., Davidson, N.E., Martino, S., Livingston, R., Ingle, J.N., Perez, E.A., Carpenter, J., Hurd, D., Holland, J.F., Smith, B.L., Sartor, C.I., Leung, E.H., Abrams, J., Schilsky, R.L., Muss, H.B., Norton, L., 2003. Randomized Trial of Dose-Dense Versus Conventionally Scheduled and Sequential Versus Concurrent Combination Chemotherapy as Postoperative Adjuvant Treatment of Node-Positive Primary Breast Cancer: First Report of Intergroup Trial C9741/Cancer and Leukemia Group B Trial 9741. Journal of Clinical Oncology 21, 1431–1439. https://doi.org/10.1200/JCO.2003.09.081

[8] Gilewski TA, Dang C, Surbone A, et al. Cytokinetics. In: Bast RC Jr, Kufe DW, Pollock RE, et al., editors. Holland-Frei Cancer Medicine. 5th edition. Hamilton (ON): BC Decker; 2000. Chapter 38. Available from: https://www.ncbi.nlm.nih.gov/books/NBK20771/

[9]
Hahnfeldt, P., Folkman, J., Hlatky, L., 2003. Minimizing Long-Term Tumor Burden: The Logic for Metronomic Chemotherapeutic Dosing and its Antiangiogenic Basis. J. Theor. Biol. 220, 545–554. https://doi.org/10.1006/jtbi.2003.3162

[10]
Traina, T.A., Norton, L., 2011a. Norton-Simon Hypothesis, in: Schwab, M. (Ed.), Encyclopedia of Cancer. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 2557–2559. https://doi.org/10.1007/978-3-642-16483-5_4130

[11]
Traina, T.A., Norton, L., 2011b. Log-Kill Hypothesis, in: Schwab, M. (Ed.), Encyclopedia of Cancer. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 2074–2075. https://doi.org/10.1007/978-3-642-16483-5_3409

###### About
This notebook is part of the *biosci670* course on *Mathematical Modeling and Methods for Biomedical Science*.
See https://github.com/cohmathonc/biosci670 for more information and material.


```python
import matplotlib
from matplotlib.axes import Axes
from matplotlib.patches import Polygon
from matplotlib.path import Path
from matplotlib.ticker import NullLocator, Formatter, FixedLocator
from matplotlib.transforms import Affine2D, BboxTransformTo, IdentityTransform
from matplotlib.projections import register_projection
import matplotlib.spines as mspines
import matplotlib.axis as maxis
import matplotlib.pyplot as plt

import numpy as np

class TriangularAxes(Axes):
    """
    A custom class for triangular projections.
    """

    name = 'triangular'

    def __init__(self, *args, **kwargs):
        Axes.__init__(self, *args, **kwargs)
        self.set_aspect(1, adjustable='box', anchor='SW')
        self.cla()

    def _init_axis(self):
        self.xaxis = maxis.XAxis(self)
        self.yaxis = maxis.YAxis(self)
        self._update_transScale()

    def cla(self):
        """
        Override to set up some reasonable defaults.
        """
        # Don't forget to call the base class
        Axes.cla(self)

        x_min = 0
        y_min = 0
        x_max = 1
        y_max = 1
        x_spacing = 0.1
        y_spacing = 0.1
        self.xaxis.set_minor_locator(NullLocator())
        self.yaxis.set_minor_locator(NullLocator())
        self.xaxis.set_ticks_position('bottom')
        self.yaxis.set_ticks_position('left')
        Axes.set_xlim(self, x_min, x_max)
        Axes.set_ylim(self, y_min, y_max)
        self.xaxis.set_ticks(np.arange(x_min, x_max+x_spacing, x_spacing))
        self.yaxis.set_ticks(np.arange(y_min, y_max+y_spacing, y_spacing))

    def _set_lim_and_transforms(self):
        """
        This is called once when the plot is created to set up all the
        transforms for the data, text and grids.
        """
        # There are three important coordinate spaces going on here:
        #
        #    1. Data space: The space of the data itself
        #
        #    2. Axes space: The unit rectangle (0, 0) to (1, 1)
        #       covering the entire plot area.
        #
        #    3. Display space: The coordinates of the resulting image,
        #       often in pixels or dpi/inch.

        # This function makes heavy use of the Transform classes in
        # ``lib/matplotlib/transforms.py.`` For more information, see
        # the inline documentation there.

        # The goal of the first two transformations is to get from the
        # data space (in this case longitude and latitude) to axes
        # space.  It is separated into a non-affine and affine part so
        # that the non-affine part does not have to be recomputed when
        # a simple affine change to the figure has been made (such as
        # resizing the window or changing the dpi).

        # 1) The core transformation from data space into
        # rectilinear space defined in the HammerTransform class.
        self.transProjection = IdentityTransform()
        # 2) The above has an output range that is not in the unit
        # rectangle, so scale and translate it so it fits correctly
        # within the axes.  The peculiar calculations of xscale and
        # yscale are specific to a Aitoff-Hammer projection, so don't
        # worry about them too much.
        self.transAffine = Affine2D.from_values(
                1., 0, 0.5, np.sqrt(3)/2., 0, 0)
        self.transAffinedep = Affine2D.from_values(
                1., 0, -0.5, np.sqrt(3)/2., 0, 0)
        #self.transAffine = IdentityTransform()

        # 3) This is the transformation from axes space to display
        # space.
        self.transAxes = BboxTransformTo(self.bbox)

        # Now put these 3 transforms together -- from data all the way
        # to display coordinates.  Using the '+' operator, these
        # transforms will be applied "in order".  The transforms are
        # automatically simplified, if possible, by the underlying
        # transformation framework.
        self.transData = \
            self.transProjection + \
            self.transAffine + \
            self.transAxes

        # The main data transformation is set up.  Now deal with
        # gridlines and tick labels.

        # Longitude gridlines and ticklabels.  The input to these
        # transforms are in display space in x and axes space in y.
        # Therefore, the input values will be in range (-xmin, 0),
        # (xmax, 1).  The goal of these transforms is to go from that
        # space to display space.  The tick labels will be offset 4
        # pixels from the equator.

        self._xaxis_pretransform = IdentityTransform()
        self._xaxis_transform = \
            self._xaxis_pretransform + \
            self.transData
        self._xaxis_text1_transform = \
            Affine2D().scale(1.0, 0.0) + \
            self.transData + \
            Affine2D().translate(0.0, -20.0)
        self._xaxis_text2_transform = \
            Affine2D().scale(1.0, 0.0) + \
            self.transData + \
            Affine2D().translate(0.0, -4.0)

        # Now set up the transforms for the latitude ticks.  The input to
        # these transforms are in axes space in x and display space in
        # y.  Therefore, the input values will be in range (0, -ymin),
        # (1, ymax).  The goal of these transforms is to go from that
        # space to display space.  The tick labels will be offset 4
        # pixels from the edge of the axes ellipse.

        self._yaxis_transform = self.transData
        yaxis_text_base = \
            self.transProjection + \
            (self.transAffine + \
             self.transAxes)
        self._yaxis_text1_transform = \
            yaxis_text_base + \
            Affine2D().translate(-8.0, 0.0)
        self._yaxis_text2_transform = \
            yaxis_text_base + \
            Affine2D().translate(8.0, 0.0)

    def get_xaxis_transform(self,which='grid'):
        assert which in ['tick1','tick2','grid']
        return self._xaxis_transform

    def get_xaxis_text1_transform(self, pad):
        return self._xaxis_text1_transform, 'bottom', 'center'

    def get_xaxis_text2_transform(self, pad):
        return self._xaxis_text2_transform, 'top', 'center'

    def get_yaxis_transform(self,which='grid'):
        assert which in ['tick1','tick2','grid']
        return self._yaxis_transform

    def get_yaxis_text1_transform(self, pad):
        return self._yaxis_text1_transform, 'center', 'right'

    def get_yaxis_text2_transform(self, pad):
        return self._yaxis_text2_transform, 'center', 'left'

    def _gen_axes_spines(self):
        dep_spine = mspines.Spine.linear_spine(self,
                                                   'right')
        # Fix dependent axis to be transformed the correct way
        dep_spine.set_transform(self.transAffinedep + self.transAxes)
        return {'left':mspines.Spine.linear_spine(self,
                                                   'left'),
                'bottom':mspines.Spine.linear_spine(self,
                                                   'bottom'),
		'right':dep_spine}

    def _gen_axes_patch(self):
        """
        Override this method to define the shape that is used for the
        background of the plot.  It should be a subclass of Patch.
        Any data and gridlines will be clipped to this shape.
        """

        return Polygon([[0,0], [0.5,np.sqrt(3)/2], [1,0]], closed=True)

    # Interactive panning and zooming is not supported with this projection,
    # so we override all of the following methods to disable it.
    def can_zoom(self):
        """
        Return True if this axes support the zoom box
        """
        return False
    def start_pan(self, x, y, button):
        pass
    def end_pan(self):
        pass
    def drag_pan(self, button, key, x, y):
        pass

# Now register the projection with matplotlib so the user can select
# it.
register_projection(TriangularAxes)
```

```python
import pycalphad.io.tdb_keywords
pycalphad.io.tdb_keywords.TDB_PARAM_TYPES.extend(['EM', 'BULK', 'SHEAR', 'C11', 'C12', 'C44'])
from pycalphad import Database, Model, calculate, equilibrium
import numpy as np
import pycalphad.variables as v
import sympy
from tinydb import where

class ElasticModel(Model):
    def build_phase(self, dbe, phase_name, symbols, param_search):
        phase = dbe.phases[phase_name]
        self.models['ref'] = self.reference_energy(phase, param_search)
        self.models['idmix'] = self.ideal_mixing_energy(phase, param_search)
        self.models['xsmix'] = self.excess_mixing_energy(phase, param_search)
        self.models['mag'] = self.magnetic_energy(phase, param_search)

        # Here is where we add our custom contribution
        # EM, BULK, SHEAR, C11, C12, C44
        for prop in ['EM', 'BULK', 'SHEAR', 'C11', 'C12', 'C44']:
            prop_param_query = (
            (where('phase_name') == phase.name) & \
            (where('parameter_type') == prop) & \
            (where('constituent_array').test(self._array_validity))
            )
            prop_val = self.redlich_kister_sum(phase, param_search, prop_param_query)
            setattr(self, prop, prop_val)

        # Extra code necessary for compatibility with order-disorder model
        ordered_phase_name = None
        disordered_phase_name = None
        try:
            ordered_phase_name = phase.model_hints['ordered_phase']
            disordered_phase_name = phase.model_hints['disordered_phase']
        except KeyError:
            pass
        if ordered_phase_name == phase_name:
            self.models['ord'] = self.atomic_ordering_energy(dbe,
                                                             disordered_phase_name,
                                                             ordered_phase_name)
```

```python
dbf = Database('ElasticTi.tdb')
mod = ElasticModel(dbf, ['TI', 'MO', 'NB', 'VA'], 'BCC_A2')
symbols = dict([(sympy.Symbol(s), val) for s, val in dbf.symbols.items()])
mod.EM = mod.EM.xreplace(symbols)
x1 = np.linspace(0,1, num=100)
x2 = np.linspace(0,1, num=100)
mesh = np.meshgrid(x1, x2)
X = mesh[0]
Y = mesh[1]
mesh_arr = np.array(mesh)
mesh_arr = np.moveaxis(mesh_arr, 0, 2)
dep_col = 1 - np.sum(mesh_arr, axis=-1, keepdims=True)
mesh_arr = np.concatenate((mesh_arr, dep_col), axis=-1)
mesh_arr = np.concatenate((mesh_arr, np.ones(mesh_arr.shape[:-1] + (1,))), axis=-1)
orig_shape = tuple(mesh_arr.shape[:-1])
mesh_arr = mesh_arr.reshape(-1, mesh_arr.shape[-1])
mesh_arr[np.any(mesh_arr < 0, axis=-1), :] = np.nan
res = calculate(dbf, ['TI', 'MO', 'NB', 'VA'], 'BCC_A2', T=300, P=101325,
                model=mod, output='EM', points=mesh_arr)
res_EM = res.EM.values.reshape(orig_shape)
```

```python
%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(12,12))
ax = fig.gca(projection='triangular')
CS = ax.contour(X, Y, res_EM, levels=list(range(-10, 310, 10)), linewidths=4, cmap='cool')
ax.clabel(CS, inline=1, fontsize=13, fmt='%1.0f')
#PCM=ax.get_children()[0] #get the mappable, the 1st and the 2nd are the x and y axes
#plt.colorbar(PCM, ax=ax)
ax.set_xlabel('Mole Fraction Mo', fontsize=18)
ax.set_ylabel('Mole Fraction Nb', fontsize=18, rotation=60, labelpad=-180)
ax.tick_params(axis='both', which='major', labelsize=18)
ax.tick_params(axis='both', which='minor', labelsize=18)
fig.savefig('TiMoNb-EM.pdf')
```

```python

```


# Embedding and Filtering Inference

Set default input and output directories according to local paths for data

```python
import os
os.environ['TRKXINPUTDIR']="/global/cfs/cdirs/m3443/data/trackml-kaggle/train_10evts"
os.environ['TRKXOUTPUTDIR']= "/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/outtest"
```

Import necessary libraries and functions

```python
import pkg_resources
import yaml
import pprint
import random
random.seed(1234)
import numpy as np
import pandas as pd
import itertools
import matplotlib.pyplot as plt
import tqdm
from os import listdir
from os.path import isfile, join
import matplotlib.cm as cm
import sys
# import sympy
# from sympy import S, symbols, printing
# %matplotlib widget

sys.path.append('/global/homes/c/caditi97/exatrkx-iml2020/exatrkx/src/')

# 3rd party
import torch
import torch.nn.functional as F
from torch_geometric.data import Data
from trackml.dataset import load_event
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import ModelCheckpoint

# local import
from exatrkx import config_dict # for accessing predefined configuration files
from exatrkx import outdir_dict # for accessing predefined output directories
from exatrkx.src import utils_dir
from exatrkx.src import utils_robust
from utils_robust import *

# for preprocessing
from exatrkx import FeatureStore
from exatrkx.src import utils_torch

# for embedding
from exatrkx import LayerlessEmbedding
from exatrkx.src import utils_torch
from torch_cluster import radius_graph
from utils_torch import build_edges
from embedding.embedding_base import *

# for filtering
from exatrkx import VanillaFilter

# for GNN
import tensorflow as tf
from graph_nets import utils_tf
from exatrkx import SegmentClassifier
import sonnet as snt

# for labeling
from exatrkx.scripts.tracks_from_gnn import prepare as prepare_labeling
from exatrkx.scripts.tracks_from_gnn import clustering as dbscan_clustering

# track efficiency
from trackml.score import _analyze_tracks
from exatrkx.scripts.eval_reco_trkx import make_cmp_plot, pt_configs, eta_configs
from functools import partial

```

```python
noise_keep = ['0','0.2','0.4','0.6','0.8','1']
plt.rcParams.update({'axes.titlesize' : 16, 'axes.labelsize' : 16, 'lines.linewidth' : 2, 'lines.markersize' : 10,
                    'xtick.labelsize' : 14, 'xtick.major.width' : 2,
                    'ytick.labelsize' : 14, 'ytick.major.width' : 2,
                    'grid.alpha' : 0.5, "legend.frameon" : False, 'legend.fontsize' : 16})
noise_labels = [f'{float(n)*0.2*100:.0f}' for n in noise_keep]
```

## Adding Noise

```python
def plot_noise_dist(noise_keeps):
    noise = []
    not_noise = []
    for i in noise_keeps:
        data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n{i}/feature_store/1000"
        #data = torch.load(f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/feature_store_endcaps/n{i}/1000")
        data = torch.load(data_path)
        arr = data['pid']
        n_count = np.count_nonzero(arr==0)
        not_n = np.count_nonzero(arr)
        noise.append(n_count)
        not_noise.append(not_n)
        print("-----")
        print(data)

    x = np.arange(len(noise_keeps))  # the label locations
    width = 0.35  # the width of the bars

    fig, ax = plt.subplots(figsize=(10,5))
    rects1 = ax.bar(x - width/2, noise, width, label='noise')
    rects2 = ax.bar(x + width/2, not_noise, width, label='not noise')

    # Add some text for labels, title and custom x-axis tick labels, etc.
    ax.set_ylabel('# of hits')
    ax.set_xlabel('Percentage Noise')
    ax.set_xticks(x)
    ax.set_xticklabels(noise_labels)
    ax.legend()

    fig2, ax2 = plt.subplots(figsize=(10,5))
    ax2.plot(x,noise, label='noise')
    ax2.set_ylabel('# of hits')
    ax2.set_xlabel('Percentage Noise')
    ax2.set_xticks(x)
    ax2.set_xticklabels(noise_labels)
    ax2.legend()

    fig3, ax3 = plt.subplots(figsize=(10,5))
    ax3.plot(x,not_noise, label='not noise')
    ax3.set_ylabel('# of hits')
    ax3.set_xlabel('Percentage Noise')
    ax3.set_xticks(x)
    ax3.set_xticklabels(noise_labels)
    ax3.legend()
```

```python
noise_keep = ["0", "0.2", "0.4", "0.6", "0.8", "1"]
plot_noise_dist(noise_keep)
```

```python
data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n1/feature_store/1000"
data = torch.load(data_path)
data['cell_data'][3]
```

    tensor([10.0000,  0.4211,  0.6767,  0.7532,  0.3000,  0.2812,  0.3000,  0.6220,
             1.8316])

## Embedding

Set checkpoint and other directory paths

```python
embed_ckpt_dir = '/global/cfs/cdirs/m3443/data/lightning_models/embedding/checkpoints/epoch=10.ckpt'
filter_ckpt_dir = '/global/cfs/cdirs/m3443/data/lightning_models/filtering/checkpoints/epoch=92.ckpt'
gnn_ckpt_dir = '/global/cfs/cdirs/m3443/data/lightning_models/gnn'
plots_dir = '/global/homes/c/caditi97/exatrkx-iml2020/exatrkx/src/plots/run1' # needs to change...
```

Fixing some parameters

```python
ckpt_idx = -1 # which GNN checkpoint to load
dbscan_epsilon, dbscan_minsamples = 0.25, 2 # hyperparameters for DBScan
min_hits = 5 # minimum number of hits associated with a particle to define "reconstructable particles"
frac_reco_matched, frac_truth_matched = 0.5, 0.5 # parameters for track matching
```

Load embedding model from checkpoint

```python
emb_ckpt = torch.load(embed_ckpt_dir, map_location='cpu')
emb_ckpt['hyper_parameters']['train_split'] = [8,1,1]
emb_ckpt['hyper_parameters']['clustering'] = 'build_edges'
emb_ckpt['hyper_parameters']['knn_val'] = 500
emb_ckpt['hyper_parameters']['r_val'] = 1.7
emb_ckpt['hyper_parameters']
```

    "adjacent":       False
    "clustering":     build_edges
    "emb_dim":        8
    "emb_hidden":     512
    "endcaps":        True
    "factor":         0.3
    "in_channels":    12
    "input_dir":      /global/cscratch1/sd/danieltm/ExaTrkX/trackml/feature_store_endcaps
    "knn":            20
    "knn_val":        500
    "layerless":      True
    "layerwise":      False
    "lr":             0.002
    "margin":         1
    "max_epochs":     100
    "nb_layer":       6
    "noise":          False
    "output_dir":     global/cscratch1/sd/danieltm/ExaTrkX/trackml_processed/embedding_processed/0_pt_cut_endcaps
    "overwrite":      True
    "patience":       5
    "project":        EmbeddingStudy
    "pt_min":         0
    "r_train":        1
    "r_val":          1.7
    "randomisation":  2
    "regime":         ['rp', 'hnm', 'ci']
    "train_split":    [8, 1, 1]
    "wandb_save_dir": /global/cscratch1/sd/danieltm/ExaTrkX/wandb_data
    "warmup":         500
    "weight":         4

```python
emb_model = LayerlessEmbedding(emb_ckpt['hyper_parameters'])
emb_model.load_state_dict(emb_ckpt["state_dict"])
```

    <All keys matched successfully>

Put model in evaluation state

```python
emb_model.eval()
```

    LayerlessEmbedding(
      (layers): ModuleList(
        (0): Linear(in_features=12, out_features=512, bias=True)
        (1): Linear(in_features=512, out_features=512, bias=True)
        (2): Linear(in_features=512, out_features=512, bias=True)
        (3): Linear(in_features=512, out_features=512, bias=True)
        (4): Linear(in_features=512, out_features=512, bias=True)
        (5): Linear(in_features=512, out_features=512, bias=True)
      )
      (emb_layer): Linear(in_features=512, out_features=8, bias=True)
      (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      (act): Tanh()
    )

Define embedding metrics (function gets metrics for one event)

```python
def get_emb_metrics(data_path, emb_model,r_val=1.7,knn_val=500):
    ## exclude loading time
    data = torch.load(data_path)
    spatial = emb_model(torch.cat([data.cell_data, data.x], axis=-1))

    if(torch.cuda.is_available()):
        spatial = spatial.cuda()

    e_spatial = utils_torch.build_edges(spatial, r_val, knn_val)

    ## takes time
    e_spatial_np = e_spatial.cpu().numpy()

    # remove R dist from out to in
    R_dist = torch.sqrt(data.x[:,0]**2 + data.x[:,2]**2)

    e_spatial_np = e_spatial_np[:, (R_dist[e_spatial_np[0]] <= R_dist[e_spatial_np[1]])]
    e_bidir = torch.cat([data.layerless_true_edges,torch.stack([data.layerless_true_edges[1],
                        data.layerless_true_edges[0]], axis=1).T], axis=-1)
    e_spatial_n, y_cluster = graph_intersection(torch.from_numpy(e_spatial_np), e_bidir)

    cluster_true = len(data.layerless_true_edges[0])
    cluster_true_positive = y_cluster.sum()
    cluster_positive = len(e_spatial_n[0])
    purity = cluster_true_positive/cluster_positive
    eff = cluster_true_positive/cluster_true

    print("-----------")
    print(f"cluster true = {cluster_true}")
    print(f"cluste true positive = {cluster_true_positive}")
    print(f"cluster positive = {cluster_positive}")
    print(f"purity = {purity}")
    print(f"efficiency = {eff}")

    return purity, eff
```

Metrics for one event and all noise configurations

```python
# %%time
# p_all = []
# e_all = []

# for i in noise_keep:
#     mypath = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n{i}/feature_store/1000"
#     p, e = get_emb_metrics(mypath, emb_model)
#     p_all.append(p)
#     e_all.append(e)
```

```python
# e_all
```

```python
# x = np.arange(len(noise_keep))
# fig1, (ax1,ax2) = plt.subplots(1,2, figsize=(15,7))
# fig1.suptitle('Evt 1000')
# fig1.tight_layout(w_pad = 10)
# ax1.plot(x, p_all, label = 'True Positive / Positive')
# ax1.set_xlabel('Noise Percentage')
# ax1.set_ylabel('Purity')
# ax1.set_xticks(x)
# ax1.set_xticklabels(noise_labels)
# ax1.legend()

# ax2.plot(x, e_all, label = 'True Positive / True')
# ax2.set_xlabel('Noise Percentage')
# ax2.set_ylabel('Efficiency')
# ax2.set_xticks(x)
# ax2.set_xticklabels(noise_labels)
# ax2.legend()

```

```python
# fig2, ax = plt.subplots(figsize=(10,10))
# fig2.suptitle(f'Eventid 1000, All noise, rval=1.7, knn=500')
# colors = cm.viridis(np.linspace(0,5))
# for i,noise in enumerate(noise_keep):
#     ax.scatter(p_all[i],e_all[i], color = colors[i], label = f'{float(noise)*0.2*100:.0f}')

# z = np.polyfit(p_all, e_all, 2)
# f = np.poly1d(z)

# ax.plot(p_all,f(p_all), label='fit')

# ax.set_xlabel('Purity')
# ax.set_ylabel('Efficiency')
# ax.legend()
# z
```

```python
# z

```

Average metrics for all events and all noise configurations

```python
def get_lvl_emb(mypath,emb_model,r_val=1.7,knn_val=500):
    events = [f for f in listdir(mypath) if isfile(join(mypath, f))][:5]
    p_lvl = []
    e_lvl = []
    for evt in events:
        data_path = join(mypath,evt)
        p, e = get_emb_metrics(data_path, emb_model,r_val,knn_val)
        p_lvl.append(p)
        e_lvl.append(e)

    return np.mean(p_lvl), np.mean(e_lvl)
```

```python
p_avg = []
e_avg = []
for i in noise_keep:
    print(f"--------- {i} ---------")
    data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n{i}/feature_store/"
    p, e = get_lvl_emb(data_path,emb_model)
    p_avg.append(p)
    e_avg.append(e)
```

    --------- 0 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95730
    cluster positive = 6626809
    purity = 0.014445866781432812
    efficiency = 0.9757713516874433
    -----------
    cluster true = 104909
    cluste true positive = 102231
    cluster positive = 7849250
    purity = 0.01302430168487435
    efficiency = 0.9744731147947269
    -----------
    cluster true = 116478
    cluste true positive = 113666
    cluster positive = 9160203
    purity = 0.012408676969276772
    efficiency = 0.9758581019591682
    -----------
    cluster true = 118662
    cluste true positive = 115783
    cluster positive = 9178018
    purity = 0.01261525091800866
    efficiency = 0.975737809913873
    -----------
    cluster true = 103195
    cluste true positive = 100664
    cluster positive = 7188799
    purity = 0.014002895337593944
    efficiency = 0.9754736179078444
    -----------
    cluster true = 101780
    cluste true positive = 99429
    cluster positive = 7227341
    purity = 0.013757341738822065
    efficiency = 0.9769011593633327
    -----------
    cluster true = 134776
    cluste true positive = 131302
    cluster positive = 11623251
    purity = 0.011296495274858988
    efficiency = 0.9742238974298094
    -----------
    cluster true = 108533
    cluste true positive = 105828
    cluster positive = 8113298
    purity = 0.013043770856191897
    efficiency = 0.9750767047810344
    -----------
    cluster true = 108630
    cluste true positive = 106181
    cluster positive = 7973216
    purity = 0.013317211022503342
    efficiency = 0.977455583172236
    -----------
    cluster true = 123250
    cluste true positive = 119961
    cluster positive = 10108715
    purity = 0.011867086964070113
    efficiency = 0.9733144016227181
    --------- 0.2 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95709
    cluster positive = 7413952
    purity = 0.012909309366988078
    efficiency = 0.9755572996829992
    -----------
    cluster true = 104909
    cluste true positive = 102173
    cluster positive = 8668811
    purity = 0.01178627611099146
    efficiency = 0.9739202546969278
    -----------
    cluster true = 116478
    cluste true positive = 113625
    cluster positive = 10007484
    purity = 0.01135400266440596
    efficiency = 0.9755061041570082
    -----------
    cluster true = 118662
    cluste true positive = 115732
    cluster positive = 10036932
    purity = 0.011530615132193781
    efficiency = 0.9753080177310344
    -----------
    cluster true = 103195
    cluste true positive = 100639
    cluster positive = 7986226
    purity = 0.012601571756171187
    efficiency = 0.9752313581084355
    -----------
    cluster true = 101780
    cluste true positive = 99393
    cluster positive = 8011360
    purity = 0.012406507758982246
    efficiency = 0.9765474552957359
    -----------
    cluster true = 134776
    cluste true positive = 131251
    cluster positive = 12491188
    purity = 0.010507487358288098
    efficiency = 0.9738454917789517
    -----------
    cluster true = 108533
    cluste true positive = 105780
    cluster positive = 8929507
    purity = 0.011846118716296431
    efficiency = 0.974634442980476
    -----------
    cluster true = 108630
    cluste true positive = 106148
    cluster positive = 8811137
    purity = 0.012047026393982979
    efficiency = 0.9771517996870109
    -----------
    cluster true = 123250
    cluste true positive = 119904
    cluster positive = 10970789
    purity = 0.010929387120652855
    efficiency = 0.9728519269776876
    --------- 0.4 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95684
    cluster positive = 8227003
    purity = 0.011630480747363286
    efficiency = 0.9753024758681847
    -----------
    cluster true = 104909
    cluste true positive = 102135
    cluster positive = 9547157
    purity = 0.010697949138157045
    efficiency = 0.973558036012163
    -----------
    cluster true = 116478
    cluste true positive = 113588
    cluster positive = 10894398
    purity = 0.010426275963114254
    efficiency = 0.9751884476038394
    -----------
    cluster true = 118662
    cluste true positive = 115686
    cluster positive = 10899748
    purity = 0.010613639874976925
    efficiency = 0.9749203620367093
    -----------
    cluster true = 103195
    cluste true positive = 100597
    cluster positive = 8854306
    purity = 0.011361364741629666
    efficiency = 0.9748243616454285
    -----------
    cluster true = 101780
    cluste true positive = 99355
    cluster positive = 8851684
    purity = 0.01122441786218306
    efficiency = 0.9761741010021615
    -----------
    cluster true = 134776
    cluste true positive = 131185
    cluster positive = 13399736
    purity = 0.009790118253076031
    efficiency = 0.97335579034843
    -----------
    cluster true = 108533
    cluste true positive = 105730
    cluster positive = 9777475
    purity = 0.010813630308438528
    efficiency = 0.9741737536048943
    -----------
    cluster true = 108630
    cluste true positive = 106104
    cluster positive = 9673556
    purity = 0.010968458754981106
    efficiency = 0.9767467550400442
    -----------
    cluster true = 123250
    cluste true positive = 119847
    cluster positive = 11881637
    purity = 0.010086741414503742
    efficiency = 0.9723894523326572
    --------- 0.6 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95648
    cluster positive = 9096481
    purity = 0.010514835352264244
    efficiency = 0.974935529574852
    -----------
    cluster true = 104909
    cluste true positive = 102090
    cluster positive = 10454898
    purity = 0.009764801148705612
    efficiency = 0.9731290928328361
    -----------
    cluster true = 116478
    cluste true positive = 113550
    cluster positive = 11797471
    purity = 0.009624944193547922
    efficiency = 0.9748622057384227
    -----------
    cluster true = 118662
    cluste true positive = 115617
    cluster positive = 11805421
    purity = 0.009793551623444856
    efficiency = 0.9743388784952217
    -----------
    cluster true = 103195
    cluste true positive = 100560
    cluster positive = 9726866
    purity = 0.010338376204627473
    efficiency = 0.9744658171423034
    -----------
    cluster true = 101780
    cluste true positive = 99319
    cluster positive = 9717169
    purity = 0.010220981028527958
    efficiency = 0.9758203969345648
    -----------
    cluster true = 134776
    cluste true positive = 131126
    cluster positive = 14317848
    purity = 0.009158219866560952
    efficiency = 0.9729180269484181
    -----------
    cluster true = 108533
    cluste true positive = 105685
    cluster positive = 10652544
    purity = 0.009921104292082717
    efficiency = 0.9737591331668709
    -----------
    cluster true = 108630
    cluste true positive = 106050
    cluster positive = 10566202
    purity = 0.01003671896486552
    efficiency = 0.9762496547914941
    -----------
    cluster true = 123250
    cluste true positive = 119793
    cluster positive = 12797043
    purity = 0.009360990660107964
    efficiency = 0.9719513184584179
    --------- 0.8 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95616
    cluster positive = 10004165
    purity = 0.009557619251581715
    efficiency = 0.9746093550918895
    -----------
    cluster true = 104909
    cluste true positive = 102044
    cluster positive = 11373894
    purity = 0.008971773431333192
    efficiency = 0.9726906175828576
    -----------
    cluster true = 116478
    cluste true positive = 113494
    cluster positive = 12736799
    purity = 0.008910716106927652
    efficiency = 0.9743814282525456
    -----------
    cluster true = 118662
    cluste true positive = 115555
    cluster positive = 12742041
    purity = 0.009068798318887846
    efficiency = 0.9738163860376532
    -----------
    cluster true = 103195
    cluste true positive = 100511
    cluster positive = 10632420
    purity = 0.009453257113620418
    efficiency = 0.973990987935462
    -----------
    cluster true = 101780
    cluste true positive = 99274
    cluster positive = 10622637
    purity = 0.009345513736372617
    efficiency = 0.9753782668500688
    -----------
    cluster true = 134776
    cluste true positive = 131064
    cluster positive = 15255514
    purity = 0.008591254283533155
    efficiency = 0.9724580043924734
    -----------
    cluster true = 108533
    cluste true positive = 105642
    cluster positive = 11565890
    purity = 0.009133927436626148
    efficiency = 0.9733629403038707
    -----------
    cluster true = 108630
    cluste true positive = 106000
    cluster positive = 11503171
    purity = 0.009214850409508821
    efficiency = 0.9757893767835772
    -----------
    cluster true = 123250
    cluste true positive = 119738
    cluster positive = 13740747
    purity = 0.008714082283881656
    efficiency = 0.9715050709939148
    --------- 1 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95574
    cluster positive = 10927804
    purity = 0.008745947493201746
    efficiency = 0.9741812510830012
    -----------
    cluster true = 104909
    cluste true positive = 102004
    cluster positive = 12308650
    purity = 0.008287180153794283
    efficiency = 0.9723093347567893
    -----------
    cluster true = 116478
    cluste true positive = 113438
    cluster positive = 13685912
    purity = 0.008288669399598653
    efficiency = 0.9739006507666684
    -----------
    cluster true = 118662
    cluste true positive = 115504
    cluster positive = 13693867
    purity = 0.008434724829735823
    efficiency = 0.9733865938548145
    -----------
    cluster true = 103195
    cluste true positive = 100463
    cluster positive = 11564939
    purity = 0.008686859481057359
    efficiency = 0.9735258491205969
    -----------
    cluster true = 101780
    cluste true positive = 99229
    cluster positive = 11534319
    purity = 0.008602935292495378
    efficiency = 0.9749361367655728
    -----------
    cluster true = 134776
    cluste true positive = 130993
    cluster positive = 16208678
    purity = 0.00808165847948858
    efficiency = 0.9719312043687304
    -----------
    cluster true = 108533
    cluste true positive = 105581
    cluster positive = 12492923
    purity = 0.008451264768061085
    efficiency = 0.9728008992656612
    -----------
    cluster true = 108630
    cluste true positive = 105954
    cluster positive = 12447702
    purity = 0.008511932563938307
    efficiency = 0.9753659210162938
    -----------
    cluster true = 123250
    cluste true positive = 119666
    cluster positive = 14697926
    purity = 0.008141692916401946
    efficiency = 0.970920892494929

```python
x = np.arange(len(noise_keep))
fig2, (ax1,ax2) = plt.subplots(1,2, figsize=(15,7))
fig2.tight_layout(w_pad = 10)
fig2.suptitle('10 Events')
ax1.plot(x, p_avg, label = 'True Positive / Positive')
ax1.set_xlabel('noise_level')
ax1.set_ylabel('Purity')
ax1.set_xticks(x)
ax1.set_xticklabels(noise_keep)
ax1.legend()

ax2.plot(x, e_avg, label = 'True Positive / True')
ax2.set_xlabel('noise_level')
ax2.set_ylabel('Efficiency')
ax2.set_xticks(x)
ax2.set_xticklabels(noise_keep)
ax2.legend()
```

```python
import scipy.optimize as opt

# calculate polynomial
z = np.polyfit(p_avg, e_avg, 2)
f = np.poly1d(z)

# calculate new x's and y's
x_new = np.linspace(p_avg[0], p_avg[-1], 50)
y_new = f(x_new)

plt.plot(p_avg,e_avg,'o', x_new, y_new)
plt.show()
```

Tuning Hyperparameters `r_val = 1.7` and `knn_val = 500` with a set level of noise for a specific event.

* `r_val` defines the radius of the clustering method
* `knn_val` defines the number of maximum neighbors in the embedding space

```python
# noise_lvl = 1
# evtid = 1000
# set_event = f'/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/outtest/out{noise_lvl}/feature_store/{evtid}'
```

```python
# import time
# from tqdm import tqdm
# rvals = np.arange(1,2,0.1)
# p_val = []
# e_val = []
# build_time = []
# for i in tqdm(range(len(rvals))):
#     data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n1/feature_store/1000"
#     start = time.time()
#     pv, ev = get_emb_metrics(data_path, emb_model,rvals[i],500)
#     end = time.time()
#     print(end - start)
#     build_time.append(end-start)
#     p_val.append(pv)
#     e_val.append(ev)
```

```python
%%time
import time
rvals = np.arange(1,2,0.1)
p_val = []
e_val = []
build_time = []
data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n1/feature_store/"
events = [f for f in listdir(data_path) if isfile(join(data_path, f))][:5]

for n in rvals:
    print(f"--------- {n} ---------")
    #data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n1/feature_store/1000"
    b = []
    p = []
    e = []
    for evt in events:
        evtpath = join(data_path,evt)
        start = time.time()
        pv, ev = get_emb_metrics(evtpath, emb_model,n,500)
        end = time.time()
        print(end - start)
        b.append(end-start)
        p.append(pv)
        e.append(ev)
    p_val.append(np.mean(p))
    e_val.append(np.mean(e))
    build_time.append(np.mean(b))
```

    --------- 1.0 ---------
    -----------
    cluster true = 98107
    cluste true positive = 85829
    cluster positive = 723053
    purity = 0.11870360817256827
    efficiency = 0.8748509280683335
    89.38341641426086
    -----------
    cluster true = 104909
    cluste true positive = 91255
    cluster positive = 816563
    purity = 0.11175500236968856
    efficiency = 0.8698491073215835
    92.58813261985779
    -----------
    cluster true = 116478
    cluste true positive = 101816
    cluster positive = 937575
    purity = 0.10859504572967496
    efficiency = 0.8741221518226618
    114.14114332199097
    -----------
    cluster true = 118662
    cluste true positive = 103575
    cluster positive = 922461
    purity = 0.11228116961042256
    efficiency = 0.8728573595590838
    116.1138174533844
    -----------
    cluster true = 103195
    cluste true positive = 90518
    cluster positive = 770916
    purity = 0.11741616466644875
    efficiency = 0.877154900915742
    93.91513848304749
    --------- 1.1 ---------
    -----------
    cluster true = 98107
    cluste true positive = 88958
    cluster positive = 1118933
    purity = 0.07950252606724442
    efficiency = 0.9067446767305085
    90.38384246826172
    -----------
    cluster true = 104909
    cluste true positive = 94608
    cluster positive = 1273940
    purity = 0.07426409407036438
    efficiency = 0.9018101402167593
    101.65235471725464
    -----------
    cluster true = 116478
    cluste true positive = 105450
    cluster positive = 1461638
    purity = 0.07214508653989565
    efficiency = 0.9053211765311905
    119.25926971435547
    -----------
    cluster true = 118662
    cluste true positive = 107374
    cluster positive = 1429414
    purity = 0.07511749570103553
    efficiency = 0.9048726635317119
    122.3467493057251
    -----------
    cluster true = 103195
    cluste true positive = 93610
    cluster positive = 1190743
    purity = 0.0786147808553147
    efficiency = 0.9071175929066331
    98.98018193244934
    --------- 1.2000000000000002 ---------
    -----------
    cluster true = 98107
    cluste true positive = 91124
    cluster positive = 1746593
    purity = 0.05217242940971365
    efficiency = 0.9288226120460313
    91.74931025505066
    -----------
    cluster true = 104909
    cluste true positive = 96960
    cluster positive = 2000532
    purity = 0.04846710774933868
    efficiency = 0.9242295703895758
    110.47075295448303
    -----------
    cluster true = 116478
    cluste true positive = 108035
    cluster positive = 2292910
    purity = 0.0471169823499396
    efficiency = 0.9275142086917701
    124.01386785507202
    -----------
    cluster true = 118662
    cluste true positive = 110068
    cluster positive = 2237431
    purity = 0.04919391927616986
    efficiency = 0.927575803542836
    131.82907629013062
    -----------
    cluster true = 103195
    cluste true positive = 95734
    cluster positive = 1857906
    purity = 0.051527902918662195
    efficiency = 0.927699985464412
    110.4542989730835
    --------- 1.3000000000000003 ---------
    -----------
    cluster true = 98107
    cluste true positive = 92667
    cluster positive = 2720507
    purity = 0.034062400868661614
    efficiency = 0.9445503378963784
    102.65164804458618
    -----------
    cluster true = 104909
    cluste true positive = 98684
    cluster positive = 3125312
    purity = 0.03157572747936846
    efficiency = 0.9406628601931197
    114.96311259269714
    -----------
    cluster true = 116478
    cluste true positive = 109923
    cluster positive = 3580735
    purity = 0.030698445989440715
    efficiency = 0.9437232782156287
    135.55846786499023
    -----------
    cluster true = 118662
    cluste true positive = 111938
    cluster positive = 3488248
    purity = 0.0320900348828409
    efficiency = 0.9433348502469199
    139.161687374115
    -----------
    cluster true = 103195
    cluste true positive = 97329
    cluster positive = 2887113
    purity = 0.03371153120781902
    efficiency = 0.9431561606666989
    111.34342622756958
    --------- 1.4000000000000004 ---------
    -----------
    cluster true = 98107
    cluste true positive = 93778
    cluster positive = 4174492
    purity = 0.02246452981584346
    efficiency = 0.955874708226732
    104.24717497825623
    -----------
    cluster true = 104909
    cluste true positive = 99928
    cluster positive = 4800212
    purity = 0.02081741389755286
    efficiency = 0.9525207560838441
    115.53528094291687
    -----------
    cluster true = 116478
    cluste true positive = 111310
    cluster positive = 5473692
    purity = 0.020335451830318547
    efficiency = 0.9556311063033363
    136.09014773368835
    -----------
    cluster true = 118662
    cluste true positive = 113348
    cluster positive = 5339224
    purity = 0.021229302235680692
    efficiency = 0.9552173400077532
    141.02323865890503
    -----------
    cluster true = 103195
    cluste true positive = 98523
    cluster positive = 4424054
    purity = 0.022269845711648185
    efficiency = 0.9547264886864674
    111.68121910095215
    --------- 1.5000000000000004 ---------
    -----------
    cluster true = 98107
    cluste true positive = 94632
    cluster positive = 6149939
    purity = 0.015387469696853904
    efficiency = 0.9645794897407932
    104.19399547576904
    -----------
    cluster true = 104909
    cluste true positive = 100885
    cluster positive = 7029431
    purity = 0.014351801731889822
    efficiency = 0.9616429476975283
    116.22485136985779
    -----------
    cluster true = 116478
    cluste true positive = 112265
    cluster positive = 7933791
    purity = 0.014150234106242526
    efficiency = 0.9638300794999914
    137.42169761657715
    -----------
    cluster true = 118662
    cluste true positive = 114361
    cluster positive = 7793780
    purity = 0.014673367736836298
    efficiency = 0.9637541925806071
    139.71138739585876
    -----------
    cluster true = 103195
    cluste true positive = 99483
    cluster positive = 6510070
    purity = 0.01528140250412054
    efficiency = 0.9640292649837686
    112.55556845664978
    --------- 1.6000000000000005 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95217
    cluster positive = 8469043
    purity = 0.011242946812290362
    efficiency = 0.970542367007451
    104.23494458198547
    -----------
    cluster true = 104909
    cluste true positive = 101596
    cluster positive = 9595361
    purity = 0.010588033113084542
    efficiency = 0.9684202499308925
    116.55731558799744
    -----------
    cluster true = 116478
    cluste true positive = 112974
    cluster positive = 10735299
    purity = 0.010523600693376123
    efficiency = 0.9699170658836862
    137.65687942504883
    -----------
    cluster true = 118662
    cluste true positive = 115068
    cluster positive = 10650766
    purity = 0.010803729985242375
    efficiency = 0.9697122920564292
    141.84952664375305
    -----------
    cluster true = 103195
    cluste true positive = 100065
    cluster positive = 8949684
    purity = 0.011180841692287682
    efficiency = 0.9696690731140074
    111.19281721115112
    --------- 1.7000000000000006 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95574
    cluster positive = 10927804
    purity = 0.008745947493201746
    efficiency = 0.9741812510830012
    105.1227765083313
    -----------
    cluster true = 104909
    cluste true positive = 102004
    cluster positive = 12308650
    purity = 0.008287180153794283
    efficiency = 0.9723093347567893
    115.73218250274658
    -----------
    cluster true = 116478
    cluste true positive = 113438
    cluster positive = 13685912
    purity = 0.008288669399598653
    efficiency = 0.9739006507666684
    137.6022026538849
    -----------
    cluster true = 118662
    cluste true positive = 115504
    cluster positive = 13693867
    purity = 0.008434724829735823
    efficiency = 0.9733865938548145
    141.1349639892578
    -----------
    cluster true = 103195
    cluste true positive = 100463
    cluster positive = 11564939
    purity = 0.008686859481057359
    efficiency = 0.9735258491205969
    108.99670839309692
    --------- 1.8000000000000007 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95774
    cluster positive = 13424717
    purity = 0.007134154112894894
    efficiency = 0.9762198416015168
    92.78107786178589
    -----------
    cluster true = 104909
    cluste true positive = 102253
    cluster positive = 15039380
    purity = 0.006799016980753196
    efficiency = 0.9746828203490644
    105.54694032669067
    -----------
    cluster true = 116478
    cluste true positive = 113656
    cluster positive = 16668821
    purity = 0.006818478643450547
    efficiency = 0.9757722488366902
    123.96346402168274
    -----------
    cluster true = 118662
    cluste true positive = 115765
    cluster positive = 16794751
    purity = 0.006892927439055214
    efficiency = 0.975586118555224
    125.98211479187012
    -----------
    cluster true = 103195
    cluste true positive = 100716
    cluster positive = 14221014
    purity = 0.007082195404631484
    efficiency = 0.9759775182906149
    101.14534997940063
    --------- 1.9000000000000008 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95917
    cluster positive = 15858242
    purity = 0.006048400573026947
    efficiency = 0.9776774338222552
    93.67161703109741
    -----------
    cluster true = 104909
    cluste true positive = 102396
    cluster positive = 17649046
    purity = 0.005801786680141238
    efficiency = 0.9760459064522586
    105.3618757724762
    -----------
    cluster true = 116478
    cluste true positive = 113791
    cluster positive = 19510218
    purity = 0.005832379730457138
    efficiency = 0.976931265990144
    123.65620613098145
    -----------
    cluster true = 118662
    cluste true positive = 115939
    cluster positive = 19764671
    purity = 0.005865971662265463
    efficiency = 0.9770524683554971
    126.34974527359009
    -----------
    cluster true = 103195
    cluste true positive = 100847
    cluster positive = 16779187
    purity = 0.006010243523717806
    efficiency = 0.9772469596395175
    100.98152017593384
    CPU times: user 17h 16min 7s, sys: 6min 5s, total: 17h 22min 13s
    Wall time: 1h 35min 49s

```python
fig3, (ax1,ax2) = plt.subplots(1,2, figsize=(15,7))
fig3.tight_layout(w_pad = 10)
# fig3.suptitle(f'Eventid {evtid}')
ax1.plot(rvals, p_val, label = 'True Positive / Positive')
ax1.set_xlabel('r_val')
ax1.set_ylabel('Purity')
# ax1.set_xticks(rvals)
ax1.legend()

ax2.plot(rvals, e_val, label = 'True Positive / True')
ax2.set_xlabel('r_val')
ax2.set_ylabel('Efficiency')
# ax2.set_xticks(x)
ax2.legend()
```

```python
fig4, ax = plt.subplots(figsize=(10,10))
# fig4.suptitle(f'Eventid {evtid}, 100% noise, 1<rval<2, knn=500')
n = len(p_val)-1
colors = cm.viridis_r(np.linspace(0,n))
for i,val in enumerate(rvals):
    ax.scatter(p_val[i],e_val[i], color = colors[i], label = f'{val:.1f}')

z = np.polyfit(p_val, e_val, 3)
f = np.poly1d(z)

ax.plot(p_val,f(p_val), label='fit')

ax.set_xlabel('Purity')
ax.set_ylabel('Efficiency')
ax.legend()
z
```

```python
fig121, ax121 = plt.subplots(1,1,figsize=(8,8))
colors = cm.viridis_r(np.linspace(0,len(rvals)-1))
for i,r in enumerate(rvals):
    ax121.scatter(rvals[i],build_time[i],color = colors[i],label=f'{rvals[i]:.1f}')
    ax121.set_xlabel('r_val')
    ax121.set_ylabel('wall time (seconds)')
    ax121.legend()

z = np.polyfit(rvals,build_time, 3)
f = np.poly1d(z)

ax121.plot(rvals,f(rvals), label='fit')
ax121.legend()

plt.show()
z
```

```python
rvals
```

    array([1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9])

```python
build_time
```

    [101.2283296585083,
     106.52447962760925,
     113.70346126556396,
     120.73566842079163,
     121.71541228294373,
     122.02150006294251,
     122.29829668998718,
     121.7177668094635,
     109.883789396286,
     110.0041928768158]

```python
%%time
import time
step = np.arange(400,500,10)
p_kval = []
e_kval = []
k_time = []

data_path = f"/global/cfs/projectdirs/m3443/usr/caditi97/iml2020/layerless_check/n1/feature_store/"
events = [f for f in listdir(data_path) if isfile(join(data_path, f))][:5]

for n in step:
    print(f"--------- {n} ---------")
    k = []
    p = []
    e = []
    for evt in events:
        evtpath = join(data_path,evt)
        start = time.time()
        pv, ev = get_emb_metrics(evtpath, emb_model,1.7,n.item())
        end = time.time()
        print(end - start)
        k.append(end-start)
        p.append(pv)
        e.append(ev)
    p_kval.append(np.mean(p))
    e_kval.append(np.mean(e))
    k_time.append(np.mean(k))
```

    --------- 400 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95383
    cluster positive = 10012578
    purity = 0.009526317797474336
    efficiency = 0.9722343971378189
    106.71507549285889
    -----------
    cluster true = 104909
    cluste true positive = 101775
    cluster positive = 11235370
    purity = 0.009058446673318279
    efficiency = 0.9701264905775482
    116.26516890525818
    -----------
    cluster true = 116478
    cluste true positive = 113171
    cluster positive = 12470539
    purity = 0.009075068848267104
    efficiency = 0.9716083723965041
    137.12152552604675
    -----------
    cluster true = 118662
    cluste true positive = 115263
    cluster positive = 12535728
    purity = 0.009194759171545521
    efficiency = 0.9713556151084594
    139.9547417163849
    -----------
    cluster true = 103195
    cluste true positive = 100269
    cluster positive = 10606151
    purity = 0.00945385371186965
    efficiency = 0.971645913077184
    119.18040084838867
    --------- 410 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95399
    cluster positive = 10116420
    purity = 0.009430114605759744
    efficiency = 0.9723974843793002
    103.13444519042969
    -----------
    cluster true = 104909
    cluste true positive = 101801
    cluster positive = 11356847
    purity = 0.008963843573837
    efficiency = 0.9703743244144926
    116.2886860370636
    -----------
    cluster true = 116478
    cluste true positive = 113213
    cluster positive = 12607785
    purity = 0.00897961061360104
    efficiency = 0.9719689555109119
    136.2460880279541
    -----------
    cluster true = 118662
    cluste true positive = 115292
    cluster positive = 12666587
    purity = 0.009102057247149528
    efficiency = 0.9716000067418381
    141.47669053077698
    -----------
    cluster true = 103195
    cluste true positive = 100292
    cluster positive = 10715640
    purity = 0.009359403638046817
    efficiency = 0.9718687920926401
    111.18831419944763
    --------- 420 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95418
    cluster positive = 10217376
    purity = 0.009338796967049074
    efficiency = 0.9725911504785592
    103.4587049484253
    -----------
    cluster true = 104909
    cluste true positive = 101821
    cluster positive = 11474812
    purity = 0.008873435137760862
    efficiency = 0.9705649658275267
    116.37056469917297
    -----------
    cluster true = 116478
    cluste true positive = 113245
    cluster positive = 12740957
    purity = 0.008888264829714125
    efficiency = 0.9722436855028418
    137.57707047462463
    -----------
    cluster true = 118662
    cluste true positive = 115315
    cluster positive = 12793632
    purity = 0.009013468575616369
    efficiency = 0.9717938345890007
    140.72779393196106
    -----------
    cluster true = 103195
    cluste true positive = 100308
    cluster positive = 10821710
    purity = 0.009269145079659315
    efficiency = 0.9720238383642619
    111.44096231460571
    --------- 430 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95444
    cluster positive = 10315043
    purity = 0.009252894049981178
    efficiency = 0.9728561672459661
    102.99831247329712
    -----------
    cluster true = 104909
    cluste true positive = 101845
    cluster positive = 11589418
    purity = 0.00878775793573068
    efficiency = 0.9707937355231677
    115.22296333312988
    -----------
    cluster true = 116478
    cluste true positive = 113278
    cluster positive = 12870299
    purity = 0.008801504922302117
    efficiency = 0.9725270008070194
    135.3641128540039
    -----------
    cluster true = 118662
    cluste true positive = 115347
    cluster positive = 12917239
    purity = 0.00892969465069122
    efficiency = 0.9720635081154877
    140.45433568954468
    -----------
    cluster true = 103195
    cluste true positive = 100329
    cluster positive = 10924330
    purity = 0.009183995723307517
    efficiency = 0.9722273365957653
    102.73981213569641
    --------- 440 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95466
    cluster positive = 10410294
    purity = 0.009170346197715453
    efficiency = 0.9730804122030029
    79.47083520889282
    -----------
    cluster true = 104909
    cluste true positive = 101881
    cluster positive = 11700638
    purity = 0.008707302969291076
    efficiency = 0.9711368900666292
    91.05987906455994
    -----------
    cluster true = 116478
    cluste true positive = 113307
    cluster positive = 12996610
    purity = 0.008718196514321812
    efficiency = 0.9727759748622057
    106.40219259262085
    -----------
    cluster true = 118662
    cluste true positive = 115377
    cluster positive = 13037612
    purity = 0.008849550055639024
    efficiency = 0.9723163270465692
    122.39508008956909
    -----------
    cluster true = 103195
    cluste true positive = 100353
    cluster positive = 11024246
    purity = 0.009102935475133628
    efficiency = 0.9724599060031979
    98.7754898071289
    --------- 450 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95489
    cluster positive = 10502750
    purity = 0.009091809288043607
    efficiency = 0.9733148501126321
    92.507741689682
    -----------
    cluster true = 104909
    cluste true positive = 101900
    cluster positive = 11809008
    purity = 0.008629005924968465
    efficiency = 0.9713179994090116
    109.42996191978455
    -----------
    cluster true = 116478
    cluste true positive = 113338
    cluster positive = 13119297
    purity = 0.008639029972413918
    efficiency = 0.9730421195418877
    127.31442880630493
    -----------
    cluster true = 118662
    cluste true positive = 115403
    cluster positive = 13154782
    purity = 0.008772703340883946
    efficiency = 0.97253543678684
    123.82991647720337
    -----------
    cluster true = 103195
    cluste true positive = 100373
    cluster positive = 11121010
    purity = 0.009025529156074854
    efficiency = 0.9726537138427249
    98.85224103927612
    --------- 460 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95508
    cluster positive = 10592592
    purity = 0.009016490014908533
    efficiency = 0.973508516211891
    92.95209431648254
    -----------
    cluster true = 104909
    cluste true positive = 101922
    cluster positive = 11914104
    purity = 0.00855473479163855
    efficiency = 0.9715277049633492
    102.34207081794739
    -----------
    cluster true = 116478
    cluste true positive = 113362
    cluster positive = 13238617
    purity = 0.00856297904834017
    efficiency = 0.973248167035835
    121.45751142501831
    -----------
    cluster true = 118662
    cluste true positive = 115421
    cluster positive = 13268194
    purity = 0.008699073890538532
    efficiency = 0.9726871281454889
    123.994637966156
    -----------
    cluster true = 103195
    cluste true positive = 100391
    cluster positive = 11214876
    purity = 0.008951592509805727
    efficiency = 0.9728281408982994
    99.28727173805237
    --------- 470 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95521
    cluster positive = 10679798
    purity = 0.008944083024791293
    efficiency = 0.9736410245955947
    92.05063438415527
    -----------
    cluster true = 104909
    cluste true positive = 101944
    cluster positive = 12016591
    purity = 0.00848360404377581
    efficiency = 0.9717374105176868
    102.94772362709045
    -----------
    cluster true = 116478
    cluste true positive = 113382
    cluster positive = 13354516
    purity = 0.008490161680138764
    efficiency = 0.9734198732807913
    120.74683117866516
    -----------
    cluster true = 118662
    cluste true positive = 115443
    cluster positive = 13378892
    purity = 0.008628741453328124
    efficiency = 0.9728725286949487
    124.83322954177856
    -----------
    cluster true = 103195
    cluste true positive = 100412
    cluster positive = 11305967
    purity = 0.008881327886416084
    efficiency = 0.9730316391298028
    99.30784964561462
    --------- 480 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95542
    cluster positive = 10764785
    purity = 0.008875421106877657
    efficiency = 0.9738550766000388
    92.07490563392639
    -----------
    cluster true = 104909
    cluste true positive = 101963
    cluster positive = 12116397
    purity = 0.008415290453094266
    efficiency = 0.9719185198600692
    103.44334650039673
    -----------
    cluster true = 116478
    cluste true positive = 113401
    cluster positive = 13467556
    purity = 0.008420310262678692
    efficiency = 0.9735829942134996
    120.3219621181488
    -----------
    cluster true = 118662
    cluste true positive = 115463
    cluster positive = 13486661
    purity = 0.00856127398768309
    efficiency = 0.973041074649003
    124.57912063598633
    -----------
    cluster true = 103195
    cluste true positive = 100434
    cluster positive = 11394775
    purity = 0.008814039768227104
    efficiency = 0.9732448277532826
    111.5484983921051
    --------- 490 ---------
    -----------
    cluster true = 98107
    cluste true positive = 95558
    cluster positive = 10847458
    purity = 0.008809252822181934
    efficiency = 0.97401816384152
    105.17079615592957
    -----------
    cluster true = 104909
    cluste true positive = 101982
    cluster positive = 12213845
    purity = 0.008349704781745633
    efficiency = 0.9720996292024516
    116.53579831123352
    -----------
    cluster true = 116478
    cluste true positive = 113424
    cluster positive = 13577790
    purity = 0.008353642234855598
    efficiency = 0.9737804563951991
    138.56224942207336
    -----------
    cluster true = 118662
    cluste true positive = 115486
    cluster positive = 13591641
    purity = 0.008496840079869679
    efficiency = 0.9732349024961656
    142.80694222450256
    -----------
    cluster true = 103195
    cluste true positive = 100454
    cluster positive = 11481266
    purity = 0.008749383561011478
    efficiency = 0.9734386355928097
    113.12068748474121
    CPU times: user 17h 4min 16s, sys: 6min 44s, total: 17h 11min
    Wall time: 1h 35min 32s

```python
k_time
```

    [123.84738249778748,
     121.6668447971344,
     121.91501927375793,
     119.3559072971344,
     99.62069535255432,
     110.3868579864502,
     108.00671725273132,
     107.97725367546082,
     110.39356665611267,
     123.23929471969605]

```python
fig5, (ax1,ax2) = plt.subplots(1,2, figsize=(15,7))
# fig5.suptitle(f'Eventid {evtid}, 100% noise, rval=1.7, 400,knnval,500')
ax1.plot(step, p_kval, label = 'True Positive / Positive')
ax1.set_xlabel('knn_val')
ax1.set_ylabel('Purity')
ax1.legend()

ax2.plot(step, e_kval, label = 'True Positive / True')
ax2.set_xlabel('knn_val')
ax2.set_ylabel('Efficiency')
ax2.legend()
```

```python
fig6, ax = plt.subplots(figsize=(10,10))
# fig6.suptitle(f'Eventid {evtid}, 100% noise, 1<rval<2, knn=500')
n = len(p_kval)-1
colors = cm.viridis_r(np.linspace(0,n))
for i,val in enumerate(step):
    ax.scatter(p_kval[i],e_kval[i], color = colors[i], label = f'{val:.1f}')

z = np.polyfit(p_kval, e_kval, 2)
f = np.poly1d(z)

ax.plot(p_kval,f(p_kval), label='fit')

ax.set_xlabel('Purity')
ax.set_ylabel('Efficiency')
ax.legend()
z
```

```python
fig121, ax121 = plt.subplots(1,1,figsize=(8,8))
colors = cm.viridis_r(np.linspace(0,len(step)-1))
for i,r in enumerate(step):
    ax121.scatter(step[i],k_time[i],color = colors[i],label=f'{step[i]:.0f}')
    ax121.set_xlabel('knn_val')
    ax121.set_ylabel('wall time (seconds)')
    ax121.legend()

z = np.polyfit(step,k_time, 1)
f = np.poly1d(z)

ax121.plot(step,f(step), label='fit')
ax121.legend()

plt.show()
z
```

```python
torch.cuda.is_available()
```

    False

```python

```