Notebook by **Jakob R. Jürgens** - Final project for the courses **OSE - Data Science** in the summer semester 2021 and **OSE - Scientific Computing** in the winter semester 2021/2022 - Find me at [jakobjuergens.com](https://jakobjuergens.com)

# Outlier Detection in Sensor Data using Functional Depth Measures
---

The best way to access this project is to clone this repository and execute the jupyter notebook and the shiny app locally. Alternatively, on the main site of this repository, there are nbviewer and binder badges set up to directly look at them in the browser.

The following packages and their dependencies are necessary to execute the notebook and the shiny app. If you are executing the code locally, make sure that these packages are provided and that an R Kernel (like irkernel) is activated in the Jupyter notebook.

```R
suppressMessages(library(devtools))
suppressMessages(library(MASS))
suppressMessages(library(tidyverse))
suppressMessages(library(shiny))
suppressMessages(library(shinydashboard))
suppressMessages(library(largeList))
suppressMessages(library(parallel))
suppressMessages(library(Rcpp))
suppressMessages(library(repr))

options(repr.plot.width=30, repr.plot.height=8)

install.packages('Code/OutDetectR_1.0.tar.gz', repos = NULL, type = 'source')
library(OutDetectR)

source("Code/auxiliary/observation_vis.R")
source("Code/auxiliary/distribution_vis.R")
source("Code/auxiliary/updating_vis.R")
source("Code/auxiliary/generate_set_1.R")
source("Code/auxiliary/generate_set_2.R")
source("Code/auxiliary/generate_set_3.R")
```

If you are in posession of the original Endanzug data set, convert it to a .RDS object using the provided function **data_prep()** from the package written for this project. Set the path to the RDS-object here. As it is property of Daimler AG, it is not included in the repository.

```R
real_data <- TRUE
if(real_data){
    endanzug_path <- '~/F/data_local/Projekt_AMEIUS_Daten/schra.RDS'
    endanzug_data <- readRDS(file = endanzug_path)
}
```

## Table of Contents
---
1. [Introduction](#introduction)
2. [Observation Structure](#observation)
3. [The Algorithm](#algorithm)
4. [h-modal depth](#h-modal)
5. [Difficulties due to the Data](#difficulties)
6. [Sampling Approach](#sampling)
7. [Finding Comparable Sets of Observations](#comparables)
8. [Description of Full Procedure for Existing Data sets](#procedure)
9. [Updating](#updating)
10. [Implementation](#implementation)
    1. [Grid Approximation](#impl_grid)
    2. [Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008) for Observations on a Shared Grid](#impl_algorithm)
    3. [Sampling Approach](#impl_sampling)
    4. [Dynamic Splitting](#impl_splitting)
    5. [Full Procedure](#impl_full)
    6. [Updating](#impl_upd)
11. [How to use the package?](#how_to)
12. [Simulated Data](#simulated)
13. [Endanzug-Data](#application)
14. [Shiny App](#shiny)
15. [Outlook](#outlook)
16. [Sources](#sources)

## Introduction
---
This project is part of a cooperation with **Daimler AG** and deals with outlier detection in sensor data from production processes.

It serves as my final project for the courses **OSE - Data Science** (summer semester 2021) and **OSE - Scientific Computing for Economists** (winter semester 2021/2022).

The main data set that is used in this project deals with the relation of angle and torque during the process of tightening a bolt in a screwing connection. The corresponding data set will be called "Endanzugsproblem" in the following notebook and contains ~350000 observations of what can be imagined as a function that maps angles to torque. The following schematic will give an idea of what the data set represents and what the problem is:

To clarify some things about this simplified schematic:
* The so called Endanzug is only part of the tightening process, but the parts of the observation happening before it are not subject of this analysis
* The focus of this project lies on curves that are "In Ordnung", so observations that do not immediately disqualify themselves in some way by for example not reaching the fixed window of acceptable final values
* The observations typically have a high frequency of measuring torque, but the measuring points are not equidistant
* The angles where torque is measured are not shared between observations, but the measuring interval of angles might overlap between observations
* The Endanzug does not start at the same angle for every observation and also does not necessarily start at the same torque
* Outliers can be very general, so methods based on detecting only specific types of outliers may not be able to effectively filter out other suspicious observations. So the optimum would be to have some kind of Omnibus test for outliers

Especially due to the high frequency of measurement and the non-identical points where torque is measured, the idea of interpreting each observation as a function and therefore approaching the problem from a standpoint of functional data analysis comes to mind. One method that is used in functional data analysis to identify outliers is based on what is called a "functional depth measure". Gijbels and Nagy (2017) introduces the idea of depth as follows and then elaborate on the theoretical properties a depth function for functional data should possess:

> For univariate data the sample median is well known
to be appropriately describing the centre of a data
cloud. An extension of this concept for multivariate
data (say p-dimensional) is the notion of a point (in
$\mathbb{R}^p$) for which a statistical depth function is maximized.

The idea is to define an analogous concept to centrality measures (such as the distance from some central tendency like the median) in a scalar setting for functional data and then use those to determine which functions in a set are typical for the whole population. Due to the more applied nature of this project, I will not go into detail on the theoretical properties of the methods used, but focus on giving intuition why the chosen methods make sense in this context.

The main inspiration for my approach to the problem of detecting outliers in a data set such as the one described above is the paper "**Outlier detection in functional data by depth measures, with application to identify abnormal NOx levels**" by Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008). I am going to first describe their algorithm, then present my extensions and my implementation and finally apply it to three simulated data sets mimicking the "Endanzugsproblem" as the original data is property of Daimler AG, which I cannot make public.

## Observation Structure
---
For the sake of clarity, I will show the typical structure of one observation and define a couple of objects I will refer to later. To give context for the later choices of data generating processes, it is useful to know that the physical process of tightening a bolt is typically associated with a linear relationship between angle and torque (at least in the relevant parts of the tightening process) - this approximation is good for the parts of the tightening process that are part of the "Endanzug". Therefore, simulations in the later parts of this notebook typically assume an approximately linear process as the data generating process for non-outliers.

One observation in a data set might look as follows.

| Observation 	| 1    	| 2    	| 3    	| 4    	| 5    	| 6    	| 7    	| 8    	| 9    	| 10   	| 11   	| 12   	| 13   	| 14   	| 15   	| 16   	| 17   	| 18   	| 19   	| 20   	|
|:-----------:	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|------	|
|    Angle    	| 2.01 	| 2.21 	| 2.91 	| 3.00 	| 3.07 	| 3.95 	| 4.33 	| 4.35 	| 4.41 	| 4.74 	| 4.77 	| 5.06 	| 6.33 	| 6.37 	| 6.41 	| 6.57 	| 7.25 	| 7.32 	| 7.71 	| 7.94 	|
|    Torque   	| 2.02 	| 2.61 	| 3.05 	| 3.16 	| 2.99 	| 4.19 	| 4.24 	| 4.37 	| 4.73 	| 4.89 	| 5.03 	| 5.45 	| 6.32 	| 6.18 	| 6.22 	| 7.06 	| 7.30 	| 7.59 	| 7.99 	| 8.06 	|

* The red diamonds represent measurements of torque that were taken at a recorded angle.
* The blue lines are an example of **linear interpolation** between the points that were actually measured. This will become important later on.
* The **measuring interval** marked in green is the convex hull of angles where measurements were taken for this observation.

The data set contains many of these objects, that do not necessarily share these characteristics.

```R
# Generated using function from auxiliary/observation_vis.R
# source("auxiliary/observation_vis.R")
# obs_vis()
```

## The Algorithm
---
The idea of Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008) is an iterative process that classifies observations as outliers if their functional depth lies below a threshold C, which is determined using a bootstrapping procedure in each iteration.

The algorithm can be decomposed into two parts:

1. **The iterative process**: (quoted from Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008))
    1. Obtain the functional depths $D_n(x_i),\: \dots \: ,D_n(x_n)$ for one of the functional depths [...]
    2. Let $x_{i_1},\: \dots,\: x_{i_k}$ be the k curves such that $D_n(x_{i_k}) \leq C$, for a given cutoff $C$. Then, assume that $x_{i_1},\: \dots,\: x_{i_k}$ are outliers and delete them from the sample.
    3. Then, come back to step 1 with the new dataset after deleting the outliers found in step 2. Repeat this until no more outliers are found.

The underlying idea is that observations that are more central in the sense of the chosen depth will have higher depth assigned to them. So choosing the least deep observations from a data set is a reasonable way to search for atypical observations. The question of where to draw the border for observations to be classified as abnormal is done using a bootstrapping procedure described next.

2. **Determining C**: (quoted from Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008))
    1. Obtain the functional depths $D_n(x_i),\: \dots ,\:D_n(x_n)$ for one of the functional depths [...]
    2. Obtain B standard bootstrap samples of size n from the dataset of curves obtained after deleting the $\alpha \%$ less deepest curves. The bootstrap samples are denoted by $x_i^b$ for $i = 1,\: \dots,\: n$ and $b = 1,\: \dots,\: B$.
    3. Obtain smoothed bootstrap samples $y_i^b = x_i^b + z_i^b$, where $z_i^b$ is such that $(z_i^b(t_1), \dots, z_i^b(t_m))$ is normally distributed with mean 0 and covariance matrix $\gamma\Sigma_x$ where $\Sigma_x$ is the covariance matrix of $x(t_1),\: \dots,\: x(t_m)$ and $\gamma$ is a smoothing parameter. Let $y_i^b$, $i = 1,\: \dots,\:n$ and $b = 1,\:\dots,\: B$ be these samples. *
    4. For each bootstrap set $b = 1,\:\dots,\:B$, obtain $C^b$ as the empirical 1% percentile of the distribution of the depths $D(y_i^b)$, $i = 1,\: \dots,\: n$.
    5. Take $C$ as the median of the values of $C^b$, $b = 1,\: \dots,\: B$.

This is done, as a theoretical derivation of the distribution of depth values for data generated by a specific data generating process is often infeasible. Instead, one drops a fraction of observations $\alpha$ and uses a smoothed bootstrapping algorithm to approximate the corresponding threshold value to remove approximately that fraction of observations in the procedure listed under 1.

*At this point in the algorithm the assumption that the functional observations are observed at a set of discrete points $t_1,\:\dots,\:t_m$ has already been made.

Some important points:
* I decided to deviate from the original procedure proposed by the authors by allowing the user to decide whether to reestimate $C$ in each iteration of the process. The authors present good arguments, why keeping $C$ fixed is the better approach and my testing confirms those. But to keep the possibilities for experimentation open, I opted to implement it in this way nevertheless. In later stages of this project I will go into detail on why keeping $C$ constant also has some problems and try to introduce corrections to improve the method developped below.
* The authors recommend a choice of $\gamma = 0.05$ as a smoothing parameter for the bootstrap which I adopted in my applications.
* For the choice of $\alpha$ preexisting information on the data should be used if available. A good way to choose this is the expected fraction of outliers in the data. However, my testing showed that in some cases the choice of this parameter had to be lower than the actual fraction of observations generated by abnormal processes when using a sampling procedure to get better results.

The authors propose three functional depth measures and benchmark them in a simulation setting. Because of their results and the computational cost which are comparatively small, I chose to use **h-modal depth** for my implementation, which I will introduce in the following.

## h-modal depth
---

Introduced by Cuevas, Febrero-Bande, and Fraiman (2006) h-modal depth is one of three depth measures covered in Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008). I will follow the summary in the latter paper for my overview.
 The idea behind this depth is that a curve is central in a set of curves if it is closely surrounded by other curves.

In mathematical terms, the h-modal depth of a curve $x_i$ in relation to a set of curves $x_1, \dots, x_n$ is defined as follows:

\begin{equation}
    MD_m(x_i,h) = \sum_{k = 1}^{n} K(\frac{||x_i - x_k||}{h})
\end{equation}

where $K: \mathbb{R^{+}} \rightarrow \mathbb{R^{+}}$ is a kernel function and $h$ is a bandwidth.

The authors recommend using the truncated Gaussian kernel, which is defined as follows:
\begin{equation}
    K(t) = \frac{2}{\sqrt{2\pi}} \exp(-\frac{t^2}{2}) \quad \text{for} \quad t>0 \quad \text{and} \quad 0 \quad \text{otherwise}
\end{equation}

and to choose $h$ as the 15th percentile of the empirical distribution of $\{||x_i - x_k|| \, | \, i,k = 1,\:\dots,\:n\}$

I chose to implement the $L^2$ norm - one of the norms recommended by the authors - as it performed better than the $L^{\infty}$ norm (which was also recommended) in my preliminary tests. In a functional setting $L^2$ is defined by:
\begin{equation}
    ||x_i - x_k||_{2} = \sqrt{\int_a^b (x_i(t) - x_k(t))^2 dt}
\end{equation}

where a and b are the boundaries of the measurement interval. This can be replaced by its empirical version
\begin{equation}
    ||x_i - x_k||_{2} = \sqrt{\sum_{j = 2}^m \Delta_j(x_i(t_j) - x_k(t_j))^2} = \sqrt{\sum_{j = 2}^m (t_j - t_{j-1})(x_i(t_j) - x_k(t_j))^2}
\end{equation}

in case of a discrete set of $m$ observation points shared between observations.

The choice of the functional norm could be adjusted to deal with data resembling different functional forms. This could be part of a possible extension, where different norms are implemented and compared. This might become part of future iteratons of this project.

## Difficulties due to the Data
---
### The Endanzug does not start at the same angle for every observation.
In this setting the fact that the first measurement is taken at different angles is not of a problem, since the property of interest is the shape of the curve after the Endanzug has started. In real world terms, the beginning of the "Endanzug" is determined by the first angle where a specific torque is exceeded and the change in torque during the "Endanzug" is of interest and not the position of the "Endanzug" in the whole tightening process. Therefore, all observations can be modified by subtracting the first angle of the "Endanzug" from all angles, effectively **zeroing** the observations.

At this time, I am going to focus on problems where zeroing is admissible and elaborate on how to possibly extend the method to scenarios where it is not. Those generalizations will be part of further work on this project.

### After zeroing, the measurement intervals might still not be identical due to differing lengths.
I decided to try to remedy this remaining problem in the zeroed data set by stretching the measuring intervals to a shared interval while leaving the observed torques identical. Excessive stretching however is problematic, as it can lead to similar observations ending up very different. Putting a conservative limit on stretching should however conserve the quality of relationships between observations. This should lead to limited influence on the calculated depths of the functional observations. To do so, I define a parameter $\lambda \geq 1$ called **acceptable stretching** and make observations comparable by stretching their measuring intervals by a factor $\psi_i \in [1/\lambda, \lambda]$ before approximating them using linear interpolation. In an optimal setting stretching would not be necessary and employing it leads to trade offs, so this parameter should be chosen conservatively.

If zeroing as described in 1. is not appropriate, a combination of **acceptable stretching** and **acceptable shifting** could be implemented to increase the size of sets of pairwise comparable functions. This will be part of future extensions of this project.

As you can already see in this animation, the acceptable stretching introduces inaccuracies even in an approximately linear setting. Outlier classifications could be quite sensitive to this parameter.

### The angles where torque is measured are not shared between observations.
Assuming that the measuring intervals are identical, I decided to use **linear interpolation** to approximate the observations. This is done to treat them as if they had been observed at a **shared grid of angles** to make them compatible with the simplification of the **h-modal norm** described above. This is only an approximation, but choosing an appropriately fine grid to approximate the observations should limit the influence of this procedure on the calculated functional depths.

Espacially in case of the dataset under consideration in this project, performing this approximation by linear interpolation should not result in large distortions due to the linearity of the described physical process. In other settings this approximation could lead to bad performance. One example that came to my mind is if most observations are zero, the frequency of taking measurements is comparatively low and the relevant parts of the observations are instantaneous deviations from zero (or spikes that have infinitesimally small duration). In cases like the one described, it would probably be a better idea to choose a different method or at least choose a different functional norm more appropriate for the dgp due to the necessity of very fine grids to achieve a appropriate approximation of the data.
 Benchmarking of the method for different data generating processes will be part of future revisions to explore the applicability of the developed method.

For sufficiently smooth processes with a high measuring frequency this approximation should however not result in huge distortions.

Another possibility to approach this third problem would be to use a different version of the norm for discretized points I described above.

Instead of calculating
\begin{equation}
    ||x_i - x_k||_{2} = \sqrt{\sum_{j = 2}^m \Delta_j(x_i(t_j) - x_k(t_j))^2}
\end{equation}

for a set of points on a grid approximated by linear interpolation, one could instead define functions $\tilde{x}_i$ which are just the piece wise linear functions defined by connecting the observed points of $x_i$. A norm based on this could be constructed as:

\begin{equation}
    ||x_i - x_k||_{\tilde{2}} = \sqrt{\int_a^b (\tilde{x}_i(t) - \tilde{x}_k(t))^2 dt}
\end{equation}

For very fine grid approximations these criteria should result in similar depths, as under some mild conditions, the first approach will converge to the second for increasingly fine grids. In a sense the first approach is similar to a Riemann sum which for increasingly fine grids converges to the corresponding Riemann integral under the necessary assumptions.

### Runtime Complexity
The runtime complexity of this algorithm is at least $O(n^2)$ and I concluded that using my implementation it is infeasible to use it on a very large data set such as the "Endanzugsproblem" (assuming that all observations are comparable at once). Even when splitting up the observations as proposed above into comparable subsets, some of them will be too large to directly approach with this method.

To solve this problem, I instead opted to use a **sampling approach** which I explain in the next section.

```R
### The gifs have been rendered using function from auxiliary/observation_vis.R (needs more packages, than are available in this environment)
# source("auxiliary/observation_vis.R")
# stretching_vis()
# zeroing_vis()
# lin_approx_vis()
```

## Sampling Approach
---
Since it is infeasible to use this method on very large data sets at once, I chose to pursue a sampling-based approach instead. Intuitively this is reasonable as in many cases observations that are atypical in the full data set will be classified as atypical in its subsamples more often than "typical" observations. If this principle does not apply to the data set, a sampling-based approach is difficult to justify. In a case like that it is more reasonable to choose differnet methods to identify abnormal observations.

As the assumption seems reasonable in case of the "Endanzugsproblem", instead of performing the algorithm described above on the whole data set (or its comparable subsets), I devise a procedure as described in the following:

Let $\{x_1, \dots, x_L\}$ be a set of observations that are comparable using the algorithm but too large to perform this procedure in reality.
1. Define the following objects:
    * Let *num\_samples* $ = (a_1,\:\dots,\: a_L) \in \mathbb{N}_0^L$ where $a_i$ is the number of samples $x_i$ was part of $\quad \forall i \in \{1,\:\dots,\: L\}$.
(Initialize all entries as 0)
    * Let *num\_outliers* $ = (b_1,\:\dots,\: b_L) \in \mathbb{N}_0^L$ where $b_i$ is the number of samples $x_i$ was classified as an outlier in $\quad \forall i \in \{1,\:\dots,\: L\}$.
(Initialize all as entries 0)
    * Let *frac\_outliers* $ = (c_1,\:\dots,\: c_L) \in \mathbb{R}_{\geq 0}^L$ where $c_i = \begin{cases}1 & a_i = 0 \\ \frac{b_i}{a_i} & a_i > 0\end{cases} \quad \forall i \in \{1,\:\dots,\: L\}$
(Initialize all entries as 1)
2. Draw a sample of size K from $\{x_1,\: \dots,\: x_L\}$ without replacement
3. Perform the outlier detection procedure on this sample and update the vectors according to your results.
4. Go back to two and iterate this process until some condition is fulfilled.

Typical conditions could be:
* A specified number of iterations was reached
* Every observation was part of more than a specified number of samples
* The vector of certainties did not change enough according to some criterion over a specified number of iterations

In the end, the entries of *frac\_outliers* can be used as a metric for the outlyingness of an observation. If a binary decision rule is needed, every observation with an entry over some specified threshold could be classified as an outlier.

## Finding Comparable Sets of Observations
---
Assume for the sake of simplicity that **zeroing** is reasonable so that the minima of the measurement intervals of the observations are all zero. Therefore, the measurement intervals only differ in their end points or lengths which is identical in this case.

As an example to visualize possible ways to find comparable subsets assume that the empirical distribution of endpoints looks as follows:

In the following I describe three methods to find comparable subsets to perform the sampling procedure on and explain why I chose the method that I ultimately implemented.

### Static Splitting

For **Static Splitting** the idea is to partition the whole data set into pairwise disjunct subsets of pairwise comparable observations. This partition depends on the acceptable stretching parameter and is not necessarily unique for a choice of the acceptable stretching parameter. Therefore, a choice procedure would have to be introduced if this approach were to be taken.

Some possible partitions of the set above (not necessarily consistent with the same acceptable stretching parameter) could look like this:

However, this idea has some problems:
* The choice of subsets could introduce a new source of distortions in addition to the acceptable stretching parameter.
* Adding new observations could change the chosen subsets, making an updating procedure difficult to realize.
* In each individual subset, the observations that are changed due to stretching are identical over samples. This could lead to distortions since for some observations not the original but only the stretched observations are taken into account in the classification.

### Dynamic Splitting

In **Dynamic Splitting** the allocation of comparable subsets takes place dynamically for each realization of the observation interval. The procedure is as follows:
\
For each realization of the endpoint, determine the subset of comparable observations and perform the sampling approach on this subset but keep the parameter for acceptable stretching constant for all end points.

This approach has some advantages over the first one:
* The choice of subsets becomes only a question of the acceptable stretching parameter and not of the choice of algorithm that chooses the partition of the data set.
* Adding new observations is unproblematic, as new observations do not influence the allocation of comparable subsets. Therefore additional samples containing the new observation can be drawn without creating problems in the comparability to previous sampling-based results. A procedure like this is described in the next section.
* Each observation can enter the classification procedure undistorted in at least the comparable subset corresponding to its own endpoint. Additionally, it can enter the classification in samples, where it is comparable due to acceptable stretching. The latter can realize for different degrees of stretching - increasing or decreasing the length of the measuring interval. This solves the problem of observations being used for classification only in a specific distorted state.

### Dynamic Splitting with varying acceptable stretching parameter

As shown above, the difference in length of these comparable subsets changes quite substantially and does not react to the density of observations having similar measurement intervals. A deviation from this appraoch might be desirable for multiple reasons:
* If there are many observations with a very similar measuring interval, including observations with a more dissimilar interval (in terms of the animation above, a midpoint farther away) might be detrimental to the procedure's performance. If there are fewer observations close by one might want to allow for a bigger acceptable stretching parameter to allow for more comparisons.
* In tandem with the advantage above one could also change the sample size to suit specific needs of the procedure.

It would be possible to use a **local acceptable stretching parameter** that changes as a function of the estimated density of end points (since zeroing was admissible in this example).
Using a rather simple function determining the local acceptable stretching parameter to serve as an example leads to the following choice of comparable subsets.

In this example the effect is quite subtle, but in comparison to the previous animation, one can see that the expansion of the interval of end points of comparable observations is slower in regions, where the estimated density of end points is higher.

There are two reasons why I decided against making acceptable stretching a varying parameter in my implementation:
1. It introduces another complication as the function to determine the local acceptable stretching has to be chosen.
2. It makes the updating procedure described later more difficult, as adding more observations will change the estimated density of the end points and thereby change the comparable subsets.

### Choice for Implementation
Due to the advantages and disadvantages described above, I decided to implement Dynamic Splitting with a global acceptable stretching parameter.

This makes it easier to later justify an updating procedure and still does not create distorted results as described for Static Splitting.

```R
### The graphics and gifs have been rendered using function from auxiliary/observation_vis.R (needs more packages, than are available in this environment)
# source("auxiliary/distribution_vis.R")
# dist_vis()
# static_splits_vis()
# dynamic_splits_vis()
# dynamic_splits2_vis()
```

## Description of Full Procedure for Existing Data sets
---
Having explained
* The algorithm by Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008)
* The sampling approach
* The procedure for selecting comparable subsets with dynamic splitting of the data set

I am going to explain the full procedure applied to an existing data set assuming that zeroing is admissible.

### Definition of Objects
* Let $x_1,\: \dots,\: x_n$ be the full data set of observations as described in the beginning
* Let $I_1 = [s_1, e_1], \dots, I_n = [s_n, e_n]$ be the measuring intervals of $x_1, \dots, x_n$

Assuming that zeroing is admissible:
* Let $\bar{x}_1,\: \dots, \:\bar{x}_n$ be the corresponding zeroed observations
* Let $\bar{I}_1 = [0, \: \bar{e}_1], \: \dots, \bar{I}_n = [0, \bar{e}_n]$ be the corresponding zeroed measuring intervals
* Let $\bar{E} = \{\bar{e} \: | \: \exists k \in \{1, \dots, n\} \: \text{s.t.} \: \bar{e} = \max(\bar{I}_k)\}$ be the set of measuring interval end points occurring in $\bar{I}_1,\: \dots,\: \bar{I}_n$
* Let $\bar{\Lambda}(\bar{x}, \bar{e})$ be the resulting object when zeroed observation $\bar{x}$ is stretched to fit zeroed measuring interval $\bar{I} = [0, \bar{e}]$

Again assuming that zeroing is admissible, one can reasonably define the following objects.
* Let $\bar{Z}(\bar{e}, \lambda) = \{\bar{x}_k \: | \: \frac{\bar{e}_k}{\bar{e}} \in [\frac{1}{\lambda},\lambda]\}$ be the set of zeroed observations that can be compared to a zeroed observation with measuring interval $\bar{I} = [0, \bar{e}]$ with an acceptable stretching factor of $\lambda$.
* Let $\bar{S}(\bar{e}, \lambda) = \{\bar{\Lambda}(\bar{e}, \bar{x}) \: | \: \bar{x} \in \bar{Z}(\bar{e}, \lambda)\}$ be the set of zeroed observations that have been stretched with an admissible stretching factor to be comparable to a zeroed observation with zeroed measuring interval $\bar{I} = [0, \bar{e}]$.

If zeroing is not a valid approach, corresponding objects dependent on acceptable shifting and acceptable stretching have to be defined, which will be one challenge in generalizing this method.

### Procedure
1. Choose parameters:
    * $\lambda$ acceptable stretching
    * $L$ sample size in sampling procedure (may also be varying depending on chosen approach for sampling)*
    * $K$ number of equidistant points in grid for approximation by linear interpolation*
    * $\alpha$ fraction of observations to drop in approximation of cutoff value $C$ in outlier classification algorithm*
    * $B$ sample size in approximation of cutoff value $C$ in outlier classification algorithm*
    * $\gamma$ tuning parameter in approximation of cutoff value $C$ in outlier classification algorithm*
2. Initialize the following vectors:
    * *num\_samples* $ = (a_1,\:\dots,\: a_n)\in \mathbb{N}_0^n \quad$ with all entries being 0
    * *num\_outliers* $ = (b_1,\:\dots,\: b_n)\in \mathbb{N}_0^n \quad$ with all entries being 0
    * *frac\_outliers* $ = (c_1,\:\dots,\: c_n) \in \mathbb{R}_{\geq 0}^n \quad$ with all entries being 1
3. Iterate through the elements of $\bar{E}$ doing the following:
    * Let $\hat{e}$ be the element of $\bar{E}$ currently looked at
    * Determine $\bar{S}(\hat{e}, \lambda)$ and perform the sampling based outlier identification procedure on this set for some predetermined stopping condition
    * Update *num\_samples*, *num\_outliers*, and *frac\_outliers* for both the stretched and non-stretched observations in $\bar{S}(\hat{e}, \lambda)$ as described in the section about sampling
    * Go to the next element of $\bar{E}$ and repeat until all elements have been reached.
4. Report *frac\_outliers* as a measure of outlyingness for the observations.

*not explicitly mentioned in the procedure below

## Updating
---
The procedure described above is constructed to work for a full data set. In a day-to-day setting the data set will not be static. Instead, new observations will be added and it would be a problem, if all calculations had to be done all over again only to incorporate a comparatively small number of new observations.

Instead, I devise mechanisms to assign comparable values of *frac\_outliers* to the newly added observations and to possibly also update the pre-existing observations due to the presence of the newly added ones. In the following assume that new observations are added sequentially. Two ways came to my mind to approach this updating procedure, one more true to the original process (1) and the other one less computationally expensive (2).

### Version 1:
This procedure involves additional samples from all sets the new observation could have been part of. So both in a stretched form or in its original form

* Let $x'$ be the new observation and $\bar{x}'$ its zeroed counterpart. Define $I'$, $\bar{I'}$ and $\bar{e}'$ accordingly.
* Determine all elements $\bar{e} \in \bar{E}$ such that $\bar{x}' \in \bar{Z}(\bar{e}, \lambda)$. Define $\bar{U}(\bar{x}', \lambda) = \{\bar{e} \in \bar{E} \: | \: \bar{x}' \in \bar{Z}(\bar{e}, \lambda)\}$ as the subset of $\bar{E}$ called the Updating Window.
 In this setting where zeroing is admissible, this can be simplified to $\bar{U}(\bar{x}', \lambda) = \{\bar{e} \in \bar{E} \: | \: \bar{e} \in [\frac{1}{\lambda} \bar{e}', \lambda \bar{e}']\} = \bar{E} \cap [\frac{1}{\lambda} \bar{e}', \lambda \bar{e}']$

For all $\tilde{e} \in \bar{U}(\bar{x}', \lambda)$ perform a sampling procedure as follows:

* $\bar{\Lambda}(\bar{x}', \tilde{e})$ is part of each sample
* The size of each sample is identical to the one used in the original procedure
* The number of samples for each $\tilde{e}$ is the expected value of the number of samples the new observation would have been part of, if it had been in the original data set
* The updating procedure of the vectors works as before. Not only the entry of the new observation in each vector is added and updated, also the entries for the original observations included in the new samples are updated.

So the set of zeroed observations that is potentially updated during this procedure is the following:

$$\bigcup_{\tilde{e}\in \bar{U}(\bar{x}', \lambda)} \bar{Z}(\tilde{e}, \lambda) = \Big\{\bar{x} \: | \: \bar{e} \in \big[\frac{1}{\lambda}\min\{\bar{U}(\bar{x}', \lambda)\}, \lambda \max\{\bar{U}(\bar{x}', \lambda)\}\big]\Big\}$$

Which is visualized in the following animation. (The endpoints of the potentially updated zeroed observations are marked by the red rectangle.)

### Version 2:
In comparison the other procedure involves only additional sampling from the set where the new observation is non-stretched.
* Let $x'$ be the new observation and $\bar{x}'$ its zeroed counterpart. Define $I'$, $\bar{I'}$ and $\bar{e}'$ accordingly.
* Determine $\bar{S}(\bar{e}', \lambda)$ and perform additional sampling as follows:
    * $\bar{x}'$ is part of each sample
    * The number of samples drawn is the expected value of samples the new observation would have been part of, if it had been in the original data set
    * The updating procedure works as before. Not only the entry of the new observation in each vector is added and updated, also the entries for the original observations are updated.

The following graphic shows the equivalent objects of version 1:

This procedure is less true to the values calculated in the original data set, as the new observation is only taken into consideration in its non-stretched form. Additionally, pre-existing observations could be taken into consideration in their stretched form more frequently depending on the structure of new data being added which could lead to additional distortions. Therefore, I decided to start by implementing the first method and possibly include comparisons of both methods in future revisions.

```R
### The graphics and gifs have been rendered using functions from auxiliary/updating_vis.R (needs more packages, than are available in this environment)
# source("auxiliary/updating_vis.R")
# upd_1_vis()
# pot_upd_obs_vis()
# upd_2_vis()
```

## Implementation
---
I decided to implement these methods in **R** and **C++** and to employ parallelization where possible, to strike a balance between speed and ease of use. All of these functions are implemented in the [**Rcpp**](https://cran.r-project.org/web/packages/Rcpp/) package **OutDetectR** that is contained in the code folder and can be installed as shown in the first code block. Documentation in the package is realized with [**roxygen2**](https://cran.r-project.org/web/packages/roxygen2/) so help files for the functions can be accessed using the notation ?\<function_name\>.

The following section will follow a similar structure as the description of the procedure above ordered as follows:

1. [Grid Approximation](#impl_grid)
2. [Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008) for Observations on a Shared Grid](#impl_algorithm)
3. [Sampling approach](#impl_sampling)
4. [Dynamic Splitting and Finding Comparable Subsets](#impl_splitting)
5. [Full Procedure](#impl_full)
6. [Updating](#impl_updating)

All of these functions can also be found in */auxiliary/R_functions.R* and will in the future be available as an Rcpp-package the tar-ball of which will be included in the repo.

In future revisions there will be a secondary notebook with the details of the implementation. This notebook will instead focus on explaining the method and evaluating its perfomance.

### Grid Approximation

#### Finding a grid for Approximation
Assuming that the observations already share the measuring intervals.

```R
#' This function creates a grid that is used for the piece wise linear
#' approximation of functions. Currently this assumes that the observations
#' share a measuring interval and constructs an equdistant grid of specified
#' length.
#'
#' @param func_dat: Data used by functional procedures
#' @param length.out: number of points in the grid
#'
#' @return A vector that contains the grid points
#' @export
grid_finder <- function(func_dat, length.out = 100) {

  # Extract Measuring Interval from first functional observation
  measuring_interval <- c(min(func_dat[[1]]$args), max(func_dat[[1]]$args))
  # Create Equdistant grid
  grid <- seq(measuring_interval[1], measuring_interval[2], length.out = length.out)
  # Return grid
  return(grid)
}
```

#### Grid approximation by Linear Interpolation
double interpolate(double x_left, double x_right, double y_left, double y_right, double x){
  if(x == x_left) return(y_left);
  double m = (y_right - y_left) / (x_right - x_left);
  double y = y_left + m * (x - x_left);
  return(y);
}

double x_lneigh(NumericVector args, double x) {
  int len = args.size();
  int i = 0;

  while(i < len){
    if(x >= args[i]){
      if(x == args[i]) return(i);
      i++;
    }
    else{
      break;
    }
  }
  return(i-1);
}

// [[Rcpp::export]]
NumericVector grid_approx_obs(NumericVector args, NumericVector vals, NumericVector grid) {

  int n_points = grid.size();
  NumericVector output(n_points);
  int x_l = 0;

  for(int i = 0; i < n_points; i++){
    x_l = x_lneigh(args, grid[i]);
    output(i) = interpolate(args[x_l], args[x_l + 1], vals[x_l], vals[x_l + 1], grid[i]);
  }

  return(output);
}
This following function acts as a wrapper for the **C++** function grid_approx_obs() shown above, that given a vector of arguments, a vector of values and a vector that represents the grid to be used for approximation, performs the desired approximation and returns the vector of approximated values taken at the grid points.

```R
#' This function approximates the functional observations given to it using a piece
#' wise linear approximation on a specified grid. The format of the data is unchanged
#'
#' @param func_dat: list that contains the observations:
#' each observation is a list, that contains two vectors of identical length: args and vals
#' @param grid: grid to use for approximation
#'
#' @return A list of identical length to func_dat. Each entry corresponds to the
#' same entry in func_dat but is an approximation using linear interpolation on
#' the points given in grid.
#' @export
grid_approx_set_obs <- function(func_dat, grid) {
  # for each observation perform an approximation on the grid points
  approx_list <- map(
    .x = func_dat,
    .f = function(obs) {
      list(
        args = grid,
        vals = grid_approx_obs(obs$args, obs$vals, grid)
      )
    }
  )
  # return the approximated observations
  return(approx_list)
}
```

The second wrapper then combines those values to a matrix, where each row represents the approximated values of an observation at the given grid points.

```R
#' This function is a wrapper around grid_approx_set_obs.
#' It takes the approximated observations of identical length and transforms
#' them into a matrix
#'
#' @param func_dat: list that contains the observations:
#' each observation is a list, that contains two vectors of identical length: args and vals
#' @param grid: grid to use for approximation
#'
#' @return A matrix of dimensions length(func_dat)xlength(grid) that contains
#' the approximated values of the functional observations at the grid points
#' in each row.
#' @export
grid_approx_mat <- function(func_dat, grid){
  # call approximation in list form
  list_approx <- grid_approx_set_obs(func_dat = func_dat,
                                     grid = grid)
  # extract values
  approx_values <- map(.x = list_approx,
                       .f = function(obs) obs$vals)
  # combine in matrix
  mat_approx <- matrix(data = unlist(approx_values),
                       nrow = length(list_approx), ncol = length(grid),
                       byrow = TRUE)
  # return matrix
  return(mat_approx)
}
```

### Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008) for Observations on a Shared Grid
The following functions implement the algorithm described above for functional observations that are observed at a common set of discrete points. The grid approximation above serves as a preparation to make these functions applicable.

#### approx_C
The function *approx_C* implements the approximation of the cutoff value $C$ by bootstrapping described in Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008).

```R
#' Uses a bootstrapping procedure to approximate the cutoff value for the identification
#' of outliers
#'
#' @param matr_dat: data in matrix form - each row contains the grid approximations of one observation
#' @param fdepths: corresponding depths for the observations
#' @param alpha: quantile of least deep observations to drop before bootstrapping
#' @param B: number of smoothed bootstrap samples to use
#' @param gamma: tuning parameter for smoothed bootstrap
#' @param grid: grid used in approximation of matr_dat
#'
#' @return placeholder
#' @export
approx_C <- function(matr_dat, fdepths, alpha, B, gamma, grid) {

  # set smoothing mode to on
  smoothing <- TRUE
  # infer number of observations from length of depth vector
  n <- length(fdepths)
  # Get number of elements in grid
  grid_length <- length(grid)
  # determine threshold to drop observations with lowest depth values
  depth_thr <- tryCatch(
    {
      quantile(x = fdepths, probs = alpha, na.rm = FALSE)
    },
    error = function(cond){
      stop(fdepths, 'approx_C - I ran into this error!')
    }
  )
  # drop observations for bootstrapping
  matr_dat_red <- matr_dat[fdepths >= depth_thr, ]

  n_red <- dim(matr_dat_red)[1]

  # Determine vcov-matrix for smoothed bootstrapping
  Sigma_x <- cov(matr_dat_red)

  my_vcov <- gamma * Sigma_x

  # Draw bootstrap samples from data set
  fsamples <- map(
    .x = 1:B,
    .f = function(inds) matr_dat_red[sample(x = 1:n_red, size = n, replace = TRUE), ]
  )

  # Create smoothing components for bootstrapping
  smoothing_components <- map(
    .x = 1:B,
    .f = function(x) mvrnorm(n = n, mu = rep(0, times = grid_length), Sigma = my_vcov)
  )

  # Obtain smoothed bootstrap samples
  smoothed_BS_samples <- map(
    .x = 1:B,
    .f = function(b) fsamples[[b]] + smoothing_components[[b]]
  )

  # Calculate depths for each smoothed bootstrap sample
  bootstrap_depths <- map(
    .x = 1:B,
    .f = function(b) hM_depth(smoothed_BS_samples[[b]], grid)
  )

  # Calculate first percentile from depths of smoothed bootstrap samples
  one_perc_quantiles <- unlist(map(
    .x = bootstrap_depths,
    .f = function(sample) quantile(sample, probs = 0.01, na.rm = FALSE)
  ))

  # return median of first percentiles
  return(median(one_perc_quantiles))
}
```

#### outlier_iteration
This function performs one iteration of the algorithm, including the calculation of functional depths, the approximation of $C$ and the flagging of observations with depths lower than $C$

```R
#' This function performs one iteration of the algorithm, including the
#' calculation of functional depths, the approximation of C and the
#' flagging of observations with depths lower than C
#'
#' @param matr_dat: data in matrix form - each row contains the grid approximations of one observation
#' @param alpha: quantile of least deep observations to drop before bootstrapping
#' (in approximation of C - optional if C is specified)
#' @param: B: number of smoothed bootstrap samples to use
#' (in approximation of C - optional if C is specified)
#' @param: gamma: tuning parameter for smoothed bootstrap
#' @param: ids: identifiers of individual observations
#' @param: grid: grid used in approximation of matr_dat
#' @param: C: should be provided. Otherwise C will be approximated in each step of the iteration
#'
#' @return placeholder
#' @export
outlier_iteration <- function(matr_dat, alpha = 0.05, B = 50, gamma, ids, grid, C = NULL) {

  # Calculating functional depths using a function from ./auxiliary/Rcpp_functions.cpp
  fdepths <- hM_depth(matr_dat, grid)

  if (missing(C)) {
    # Approximating C
    C <- approx_C(
      matr_dat = matr_dat, fdepths = fdepths, alpha = alpha,
      B = B, gamma = gamma, grid = grid
    )
  }

  # Flagging observations with depths lower than the cutoff value C
  outliers <- which(fdepths < C)

  return(list(
    matr_dat = matr_dat[-outliers, ],
    ids = ids[-outliers],
    outlier_ids = ids[outliers]
  ))
}
```

 #### outlier_detection
 This function serves as a wrapper for outlier_iteration and iterates the process until no new observations are flagged.

```R
#'  This function serves as a wrapper for outlier_iteration and iterates the process until no new observations are flagged.
#'
#' @param matr_dat: data in matrix form - each row contains the grid approximations of one observation
#' @param alpha: quantile of least deep observations to drop before bootstrapping
#' (in approximation of C - optional if C is specified)
#' @param B: number of smoothed bootstrap samples to use
#' (in approximation of C - optional if C is specified)
#' @param gamma: tuning parameter for smoothed bootstrap
#' @param ids: identifiers of individual observations
#' @param grid: grid used in approximation of matr_dat
#' @param C: should be provided. Otherwise C will be approximated in each step of the iteration
#'
#' @return placeholder
#' @export
outlier_detection <- function(matr_dat, alpha = 0.05, B = 100, gamma = 0.05, ids, grid, C = NULL) {
  tmp_ids <- ids
  # Initialize empty vectors for position of flagged observations in func_dat
  # and ids of flagged observations
  outlier_ids <- c()

  # loop that continues until an iteration does not flag any new observations
  condition <- TRUE
  while (condition) {

    # perform iteration
    iter_res <- outlier_iteration(
      matr_dat = matr_dat, alpha = alpha, B = B, gamma = gamma,
      ids = tmp_ids, grid = grid, C = C)

    new_outliers <- iter_res$outlier_ids

    # if there are no new flagged observations stop loop
    if (length(new_outliers) == 0) {
      condition <- FALSE
    } else {
      # otherwise: add flagged observations to vector
      outlier_ids <- c(outlier_ids, new_outliers)
      # reduce data to non-flagged observations
      matr_dat <- iter_res$matr_dat
      # reduce ids to non-flagged observations
      tmp_ids <- iter_res$ids
    }
  }

  # return identifiers of flagged observations and position of these flagged observations in the data set
  return(list(
    outlier_ids = outlier_ids,
    outlier_ind = which(is.element(ids, outlier_ids))
  ))
}
```

#### Outlier Detection - Wrapper
The following function acts as a wrapper to the previous one in case, $C$ should not be recalculated in each iteration. My recommendation would be to use this function as recalculation of $C$ in each iteration can lead to classifying unreasonably many observations as outliers. Due to problems with the parallelization, non-reproducible errors appear in some cases, making a tryCatch necessary.
In case of a reproducible error a counter will reach a specified value after a number of iterations, throwing the error to the overarching procedure, terminating the outlier detection process.

```R
#' This function performs the identification procedure implemented in this package
#' in the simplest scenario. It serves as an important building block for the
#' more high-level functions.
#' Important assumptions are the following:
#' - observations in func_dat are zeroed and have the same measuring interval
#'
#' @param func_dat: list that contains the observations
#' each observation is a list, that contains two vectors of identical length: args and vals
#' @param ids: identifiers of individual observations
#' @param alpha: quantile of least deep observations to drop before bootstrapping
#' (in approximation of C - optional if C is specified)
#' @param ids: B: number of smoothed bootstrap samples to use
#' (in approximation of C - optional if C is specified)
#' @param gamma: tuning parameter for smoothed bootstrap
#'
#' @return Returns a list containing two objects:
#' 1. outlier_ids: a vector containing the ids of the observations identified as outliers
#' 2. outlier_ind: a vector containing the position of the the observations identified as outliers in the func_dat list
#' @export
detection_wrap <- function(func_dat, ids, alpha, B, gamma = 0.05) {

  # determine the grid for approximation
  grid <- grid_finder(func_dat = func_dat)

  # Approximate by linear interpolation
  matr_dat <- grid_approx_mat(func_dat = func_dat, grid = grid)

  # tmp redo variable
  redos <- 0

  while(redos < 10){
    # calculate h-modal depths
    fdepths <- hM_depth(valueMatrix = matr_dat, grid = grid)
    if(all(is.na(fdepths))){
      redos <- redos + 1
    } else{
      redos <- 10
    }
  }

  # create temporary variable indicating if error occured
  n_error <- 0

  # Approximate a value of C
  while (n_error < 10) {
    add <- FALSE
    C_appr <- tryCatch(
      {
        approx_C(
          matr_dat = matr_dat, fdepths = fdepths, alpha = alpha,
          B = B, gamma = gamma, grid = grid
        )
      },
      error = function(cond) {
        warning(cond)
        add <<- TRUE
      }
    )
    if(add == TRUE){
      n_error <- n_error + 1
    } else{
      n_error <- 10
    }
  }

  # Perform the outlier classification procedure for the approximated value of C
  flagged <- outlier_detection(
    matr_dat = matr_dat, ids = ids, grid = grid, C = C_appr
  )

  # Return the list of outlier ids and outlier indices - these are useful in different cases
  # contains the objects outlier_ids and outlier_ind
  return(flagged)
}

```

### Sampling Approach

#### Helper function for parallelization
Since data sets can get large very quickly, it is useful to perform parallelization in this sampling approach in a less memory intensive way. Therefore I decided to write the data set in its prepared form to the disk and use a format that supports random access in lists, to only read in the observations that are part of the sample. After reading in those observations, it performs the outlier detection procedure implemented above.

```R

```

```R

```

#### How to use this function?
Using the function *sampling_wrap()* is not as straight-forward as using the previous as multiple steps have to be performed before and after using this function to ensure a problem free experience.
I will explain the following code fragments in detail, as depending on which operating system a user employs modifications have to be made. For this reason, the code is commented out in these parts, in order not to cause technical problems that are difficult to replicate. The following code segments were written on a Linux machine and will work on UNIX systems. Since forking is not supported under windows, alternatives would have to be used, like using PSOCK-Clusters instead of ForkClusters.

```R

```

* *num_cores()* detects the number of logical cores
* *makeForkCluster()* creates a virtual cluster object that can serve as an argument to functions from the parallel package to perform parallelized computations
* the *clusterCall()* calls execute the command inside on each individual node of the virtual cluster to ensure that all necessary packages are loaded in each instance
* *clusterExport* this exports objects from an environment to each of the nodes. These can be functions or objects.

(These last to steps are technically not necessary in case of a fork cluster, but will navigate around some hard to troubleshoot problems that can occur.)

* *sampling_wrap()* performs the actions implemented above on the virtual cluster *cl*
* *stopCluster()* stops the cluster, to ensure that it does not clog up the system

### Dynamic Splitting

#### Zeroing
To use the dynamic splitting procedure in the previously described settings, it is first necessary to zero all observations. This is implemented for the functional observations and its results should be saved as a separate object for future use.

```R

```

#### Determine measuring intervals
This function returns a matrix containing in each row the beginning and end point of the measuring interval of the corresponding observation.

```R

```

#### Create a list of measuring intervals that occur in the data set
This set is needed later on for iterating through all possible realizations of the measuring interval to determine the comparable sets for each one.

```R

```

#### Find comparable observations for one measuring interval
Given a measuring interval that is currently under consideration and the matrix of all measuring intervals, determine the indices of the observations that are comparable given an acceptable stretching factor $\lambda$. Here zeroing is assumed, such that the condition simplifies to a condition on the endpoint of the intervals.

```R

```

This function can then be used for determining which sets to sample from using the sampling procedure implemented above while iterating through the measuring intervals that occur in the data set.

### Full Procedure

#### Stretching an observation
The first function needed for the full procedure is the ability to stretch an observation to be comparable to another.

```R

```

#### Stretching a set of observations to prepare sampling procedure
This function acts as a wrapper for the previous function and applies it to a set of observations that are stretched to the same measuring interval.

```R

```

#### Performing stretching and sampling procedure on set of observations
This is nearly identical to the functions for the sampling procedure itself and could easily be included as a subcase. For the sake of clarity, I nevertheless decided to make these separate functions.
Understanding the sampling procedure and the stretching functions will make these functions easy to understand.

```R

```

This function performing the parallelized sampling is again very similar. It only differs in four points:
* argument measuring_interval: interval the observations are stretched to
* argument comparable: ids of the observations to include in the process as now not all observations are to be sampled in the same process
* no argument n_obs: this can be replaced by comparable
* use of the appropriately changed helper function for the parallelization

Be careful when using the following function. The vectors that are returned only relate to the set of comparable observations currently under consideration.

A value of 3 in the first entry of *num_samples* means that the first observation of the whole data set occurred in 3 samples, not that the first observation from the set of comparable observations occurred in 3 samples. This has to be kept in mind while later using the output of this function.

```R

```

#### Putting the pieces together
Now that there are functions that
* Find the set of measuring intervals that occur in the data set
* Find comparable observations given a specific measuring interval and an acceptable stretching factor
* Perform the sampling procedure with acceptable stretching on comparable subsets of the data set

it is possible to assemble the pieces into one cohesive function that performs the outlier classification procedure described above on a data set where zeroing is admissible.

```R

```

### Updating
The updating procedure is not yet implemented. This is why this part is a description of the functions and their implementation.

#### Find measuring intervals with possible occurences of new observation

```R
# new_func_obs: list with two elements: vectors of equal length called args and vals
# new_id: identifier for the new observation (be careful not to create duplicates - for example just consecutive intergers would be fine)
# list_path: path to the largeList object, where data set is saved
# id_path: path to the RDS file with the identifiers

obs_append <- function(new_func_obs, new_id, list_path, id_path){

    # read in previous ids
    ids <- readRDS(file = id_path)
    # append new ID
    new_ids <- c(ids, new_id)
    # overwrite old ID file
    saveRDS(object = new_ids, file = id_path)

    # largeLists support appending
    saveList(object = list(new_func_obs), file = list_path, append = TRUE)
}
```

#### Appending observation to list of functional data and ids

```R
# new_func_obs: list with two elements: vectors of equal length called args and vals
# unique_intervals: matrix containing measuring intervals that occur in the data set (one in each row)
# lambda: acceptable stretching parameter

possible_occurences <- function(new_func_obs, unique_intervals, lambda){

    # determine measuring interval of new observation
    new_measuring_interval <- c(min(new_func_obs$args),  max(new_func_obs$args))

    # determine for all previously used measuring intervals if the new
    # observation could have been part of the stretching and sampling procedure
    occurs <- map(.x = 1:(dim(unique_intervals)[1]),
                      .f = function(i) ifelse(new_measuring_interval[2] >= unique_intervals[i,2]/lambda
                                              & new_measuring_interval[2] <= unique_intervals[i,2]*lambda,
                                              TRUE, FALSE))

    # return the indices and intervals that the new observation could have been a part of
    return(list(occurs = occurs,
                occurs_intervals = unique_intervals[occurs, ]))
}
```

#### Determine expected number of occurences in this measuring interval sampling run

```R
# orig_n_obs: number of observations in the original comparable set (use output from comparable_obs_finder to determine this)
# orig_n_samples: number of samples drawn in the original procedure (this can be updated by adding those drawn now for future updates)
# orig_sample_size: sample size used in the original prrocedure for this comparable subset

exp_num_samples <- function(orig_n_obs, orig_n_samples, orig_sample_size){

    # determine the probability that the observation would have been part of any original sample,
    # if it had been part of the data set
    # exp_per_sample <- choose(orig_n_obs, orig_sample_size - 1) / choose(orig_n_obs + 1, orig_sample_size)
    # simplifies to:
    exp_per_sample <- orig_sample_size / orig_n_obs

    # multiplay with the number of samples and return the number rounded to the next whole integer
    return(ceiling(orig_n_samples * exp_per_sample))
}
```

#### Update one measuring interval

work in progress

```R

```

#### Updating for one new observation

work in progress

```R
obs_update <- function(cl, new_func_obs, list_path, measuring_intervals, lambda, n_samples = NULL,
                       prev_num_samples, prev_num_outliers, sample_size = NULL, alpha = NULL, B = NULL,
                       gamma = 0.05){

    main_interval <- c(min(new_func_obs$args),  max(new_func_obs$args))

    comparable <- comparable_obs_finder

}
```

## How to use the package?
---

## Simulated Data
---
To show possible usecases of the implementation I create three datasets:
1. **No sampling**
 This data set shows the outlier classification procedure of Febrero-Bande, Galeano, and Gonzàlez-Manteiga (2008) in action. It is characterized by:
    * Observations with identical measuring intervals
    * Relatively few observations meaning that the algorithm can be applied without sampling

2. **Sampling**
A data set where observations share the measuring interval but which is large enough that sampling is necessary to use the algorithm

3. **Full procedure**
    * A dataset where observations share the starting point of the measuring interval but have different endpoints.
    * This is meant to show the full procedure on a data set that is effectively already zeroed.

The functional form of the non-outliers in these data sets is approximately **linear** which has two reasons:

1. Simplicity
2. The approximate linearity of the relationship that is fundamental to the main data set explored in this project (angle and torque when tightening a bolt).

All of these data sets and the results of the outlier classification procedures applied to them can be seen in the **shiny app** provided in the repository.

### No sampling:
This data set is based on a simple data generating process. Each of the 500 observations is generated as follows:

* Determine if the observation is generated as an outlier by drawing from a Bernoulli distributed random variable with parameter 0.05.
* Determine the number of points $k$ where the function is observed. (In the terms of the method above the number of points where measurements are taken) This is drawn from a discrete uniform with elements $10, \dots, 100$

The generation of dependent variables depends on whether the observation is generated as an outlier. The general process is as follows:

* Draw k-2 realizations of $p \sim U[0, 1] \quad \text{s.t.} \quad p_1, \dots, p_k \quad \text{i.i.d.}\quad$ Let $p_{(1)}, \dots, p_{(k-2)}$ be the sorted realizations and define $(0, \: p_{(1)},\: \dots, \: p_{(k-2)},\: 1)^T = \vec{p}$
(This is equivalent to our grid of measuring points.)

* Draw k realizations of $s \sim U[\underset{\bar{}}{s}, \bar{s}] \quad \text{s.t.} \quad s_1, \dots, s_k \quad \text{i.i.d.} \quad$ and define $(s_1,\: \dots, \: s_k)^T = \vec{s}$
* Draw k realizations of $\epsilon \sim \mathcal{N}[0, \sigma] \quad \text{s.t.} \quad \epsilon_1, \dots, \epsilon_k \quad \text{i.i.d.} \quad$ and define $(\epsilon_1,\: \dots, \: \epsilon_k)^T = \vec{\epsilon}$
* Let $\vec{y} = m \vec{s} \odot \vec{p} + \vec{\epsilon}$ be the vector of realizations of the dependent variable, where $\odot$ is the component-wise (or Hadamard) product.

The parameters are different for non-outliers and outliers:

For non-outliers:
* $\underset{\bar{}}{s} = 0.8, \quad \bar{s} = 1.2$
* $\sigma = 0.05$
* $m = 1.02$

and for outliers:
* $\underset{\bar{}}{s} = 1, \quad \bar{s} = 1.4$
* $\sigma = 0.1$
* $m = 1.2 * 1.02$

```R
# Data generated, transformed and visualized with functions from auxiliary/generate_set_1.R
# Data set is also saved to ./data/Set_1 and if existent read from there instead
# source("auxiliary/generate_set_1.R")

if(file.exists("./data/Set_1/functional.llo") &&
   file.exists("./data/Set_1/ids.RDS") &&
   file.exists("./data/Set_1/outliers.RDS")){
    data_set_1 <- list(data = readList(file = "./data/Set_1/functional.llo"),
                       ids = readRDS(file = "./data/Set_1/ids.RDS"),
                       outliers = readRDS(file = "./data/Set_1/outliers.RDS"))

} else{
    data_set_1 <- generate_set_1()
}

if(file.exists("./data/Set_1/tibble.RDS")){
   tidy_set_1 <- readRDS("./data/Set_1/tibble.RDS")
} else{
   tidy_set_1 <- tidify_1(data_set_1$data, data_set_1$ids)
   saveRDS(object = tidy_set_1, file = "./data/Set_1/tibble.RDS")
}

vis_1(tidy_set_1)
```

Prepare data for visualization in the shiny app.

```R
if(file.exists("./data/Set_1/detection.RDS")){
    set_1_detection <- readRDS("./data/Set_1/detection.RDS")
} else{
    set_1_detection <- detection_wrap(func_dat = data_set_1$data, alpha = 0.05, B = 50, gamma = 0.05, ids = data_set_1$ids)
    saveRDS(object = set_1_detection, file = "./data/Set_1/detection.RDS")
}

if(file.exists("./data/Set_1/summary.RDS")){
    set_1_summary <- readRDS("./data/Set_1/summary.RDS")
} else{

    missed_outliers = setdiff(data_set_1$outliers, set_1_detection$outlier_ind)
    false_outliers = setdiff(set_1_detection$outlier_ind, data_set_1$outliers)

    set_1_summary <- list(flagged = set_1_detection$outlier_ind,
                          original = data_set_1$outliers,
                          missed = missed_outliers,
                          false = false_outliers)

    saveRDS(object = set_1_summary, file = "./data/Set_1/summary.RDS")
}

if(file.exists("./data/Set_1/shiny_tibble.RDS")){
    shiny_tibble_1 <- readRDS("./data/Set_1/shiny_tibble.RDS")
} else{
    shiny_tibble_1 <- tidy_set_1 %>%
        mutate(outlier = ifelse(ids %in% data_set_1$outliers, TRUE, FALSE),
               flagged = ifelse(ids %in% set_1_detection$outlier_ind, TRUE, FALSE))
    saveRDS(object = shiny_tibble_1, file = "./data/Set_1/shiny_tibble.RDS")
}
```

### Sampling:
This second data set consists of 10000 observations where 95% are generated by the same process as the non-outlier variant in the previous data set.
So a vector of arguments is drawn from the continuous uniform on $[0,1]$ of random length. Call this vector *args* in the following description.

But the outliers take 5 different forms each appearing in about one percent of cases. I will describe their data generating processes in mathematical form in the following:

1. Exactly as the outliers in data set 1.

2. Sigmoid function: $\frac{1}{\mathbb{1} + \exp{(-3 * \text{args})}} + \epsilon$ where $\epsilon$ is a vector of appropriate length containing i.i.d. elements drawn from $\mathcal{N}(0, 0.05^2)$

3. Half sigmoid function: $\frac{2}{\mathbb{1} + \exp{(-3 * \text{args})}} - \mathbb{1} + \epsilon$ where $\epsilon$ is a vector of appropriate length containing i.i.d. elements drawn from $\mathcal{N}(0, 0.05^2)$

4. Exponential function: $\frac{\exp(args) - \mathbb{1}}{e - 1} + \epsilon$ where $\epsilon$ is a vector of appropriate length containing i.i.d. elements drawn from $\mathcal{N}(0, 0.05^2)$

5. Noise: i.i.d. draws from $U[0,1]$

```R
# Data generated, transformed and visualized with functions from auxiliary/generate_set_2.R
# Data set is also saved to ./data/Set_2 and if existent read from there instead
# source("auxiliary/generate_set_2.R")

if(file.exists("./data/Set_2/functional.llo") &&
   file.exists("./data/Set_2/ids.RDS") &&
   file.exists("./data/Set_2/outliers.RDS")){
    data_set_2 <- list(data = readList(file = "./data/Set_2/functional.llo"),
                       ids = readRDS(file = "./data/Set_2/ids.RDS"),
                       outliers = readRDS(file = "./data/Set_2/outliers.RDS"))

} else{
    data_set_2 <- generate_set_2()
}

if(file.exists("./data/Set_2/tibble.RDS")){
   tidy_set_2 <- readRDS("./data/Set_2/tibble.RDS")
} else{
   tidy_set_2 <- tidify_2(data_set_2$data, data_set_2$ids)
   saveRDS(object = tidy_set_2, file = "./data/Set_2/tibble.RDS")
}

vis_2(tidy_set_2)
```

Due to the computational cost of this classifying procedure I saved its results and only perform the classification, if those results could not be found. In the case of 10000 observations can still be done on the full set and I opted to use this lower number of observations not due to computational problems with larger data sets but due to overplotting in the visualizations, that has to be addressed, before large data sets can be appropriately visualized.

```R
if(file.exists("./data/Set_2/results.RDS")){
    set_2_results <- readRDS(file = "./data/Set_2/results.RDS")

} else{
    num_cores <- detectCores()
    cl <- makeForkCluster(nnodes = num_cores)

    clusterExport(cl, varlist = list("grid_approx_set_obs",
                                     "approx_C",
                                     "grid_finder",
                                     "outlier_iteration",
                                     "outlier_detection",
                                     "detection_wrap",
                                     "random_access_par_helper"),
                                      envir = .GlobalEnv)

    set_2_results <- sampling_wrap(cl = cl, n_obs = 10000, n_samples = 150,
                               sample_size = 500, alpha = 0.05, B = 100, gamma = 0.05,
                               list_path = "./data/Set_2/functional.llo")

    stopCluster(cl)

    saveRDS(object = set_2_results, file = "./data/Set_2/results.RDS")
}

if(file.exists("./data/Set_2/shiny_tibble.RDS")){
    shiny_tibble_2 <- readRDS("./data/Set_2/shiny_tibble.RDS")
} else{
    shiny_tibble_2 <- tidy_set_2 %>%
        mutate(outlier = ifelse(ids %in% data_set_2$outliers, TRUE, FALSE))

    lengths <- unlist(map(.x = 1:10000,
                          .f = function(i) length(data_set_2$data[[i]]$vals)))

    shiny_tibble_2$cert <- unlist(map(.x = 1:10000,
                                      .f = function(i) rep(set_2_results$certainties[i], times = lengths[i])))

    saveRDS(object = shiny_tibble_2, file = "./data/Set_2/shiny_tibble.RDS")
}
```

To see the results of this classification procedure for different values of the certainty threshold, you can use the shiny app provided in the repository.

### Full Procedure:
This data set is similar to the sampling data set in its data generating process. There are 30000 observations each generated by a similar process as in data set 2.

The main difference is, that the measuring intervals are not identical and are drawn from the following possibilities:

| Endpoint of Measuring Interval 	| 0.9  	| 1   	| 1.1  	| 1.5  	| 1.6  	| 1.7  	| 1.9 	| 2    	| 2.1  	|
|:------------------------------:	|------	|-----	|------	|------	|------	|------	|-----	|------	|------	|
|           Probability          	| 0.05 	| 0.2 	| 0.05 	| 0.07 	| 0.15 	| 0.08 	| 0.1 	| 0.25 	| 0.05 	|

Also the data generating processes for both the outliers and the non-outliers are scaled such that the expected realizations at the beginning and end of the data set are identical. (Especially important for the exponential and sigmoidal outliers.) An exception from this are the type 5 outliers, which stay uniformly distributed - but the upper border of the support is made to fit the expected realization of the non-outlier process at the end point of the measuring interval.

```R
# Data generated, transformed and visualized with functions from auxiliary/generate_set_3.R
# Data set is also saved to ./data/Set_3 and if existent read from there instead
source("auxiliary/generate_set_3.R")

if(file.exists("./data/Set_3/functional.llo") &&
   file.exists("./data/Set_3/ids.RDS") &&
   file.exists("./data/Set_3/outliers.RDS")){
    data_set_3 <- list(data = readList(file = "./data/Set_3/functional.llo"),
                       ids = readRDS(file = "./data/Set_3/ids.RDS"),
                       outliers = readRDS(file = "./data/Set_3/outliers.RDS"))

} else{
    data_set_3 <- generate_set_3()
}

if(file.exists("./data/Set_3/tibble.RDS")){
   tidy_set_3 <- readRDS("./data/Set_3/tibble.RDS")
} else{
   tidy_set_3 <- tidify_3(data_set_3$data, data_set_3$ids)
   saveRDS(object = tidy_set_3, file = "./data/Set_3/tibble.RDS")
}

### Due to a lengthy plotting time and overplotting, this plot is saved an inserted as a png
# vis_3(tidy_set_3)
```

Full procedure works - update this!!!

```R
if(file.exists("./data/Set_3/results.RDS")){
   set_3_results <- readRDS(file = "./data/Set_3/results.RDS")
} else{
    num_cores <- detectCores()
    cl <- makeForkCluster(nnodes = num_cores)

    clusterExport(cl, varlist = list("grid_approx_set_obs",
                                     "approx_C",
                                     "grid_finder",
                                     "outlier_iteration",
                                     "outlier_detection",
                                     "detection_wrap",
                                     "random_access_par_helper"),
                                      envir = .GlobalEnv)

    measuring_intervals <- measuring_int(func_dat = data_set_3$data)

    comp_test <- comparable_obs_finder(main_interval = c(0,1), measuring_intervals = measuring_intervals,  lambda = 1.2, ids = 1:30000)$ids

    #test <- stretch_and_sample(cl = cl, n_samples = 100, sample_size = 100, alpha = 0.02, B = 100, gamma = 0.05, n_obs = 30000,
    #                           list_path = "./data/Set_3/functional.llo", measuring_interval = c(0,1),
    #                           comparable = comp_test)

    set_3_results <- dectection_zr_smpl(cl = cl, list_path = "./data/Set_3/functional.llo", measuring_intervals = measuring_intervals,
                                        n_obs = 30000, lambda = 1.2, n_samples = 300, sample_size = 300, alpha = 0.02, B = 100, gamma = 0.05)

    stopCluster(cl)

   saveRDS(object = set_3_results, file = "./data/Set_3/results.RDS")
}

if(file.exists("./data/Set_3/shiny_tibble.RDS")){
    shiny_tibble_3 <- readRDS("./data/Set_3/shiny_tibble.RDS")
} else{
    shiny_tibble_3 <- tidy_set_3 %>%
        mutate(outlier = ifelse(ids %in% data_set_3$outliers, TRUE, FALSE))

    lengths <- unlist(map(.x = 1:30000,
                          .f = function(i) length(data_set_3$data[[i]]$vals)))

    shiny_tibble_3$cert <- unlist(map(.x = 1:30000,
                                      .f = function(i) rep(set_3_results$certainties[i], times = lengths[i])))

    saveRDS(object = shiny_tibble_3, file = "./data/Set_3/shiny_tibble.RDS")
}
```

## Endanzug-Data
---
When applying the procedure to the real world data from the Endanzug dataset, I decided to use a subset of 20000 randomly chosen observations. Using the whole dataset is unfeasible with a typical computer and would require further optimizations and the usage of high-performance computing.

## Shiny App
---
Instead of presenting the visualizations for the classification procedure in static graphics, I decided to use a shiny web app instead. This has multiple advantages in the setting of this final project, but the main motivation behind it was the use case described to me:
 **An engineer wants to get a preselection of suspicious observations they should have a look at.**

In this context, having a raw R file as output without an easy way to interact with it, would not be particularly useful. Instead, being able to run this shiny app on a local server with the precalculated values stored in a database, would be more in line with the idea of the job. This also motivates the features I implemented for the app (and plan to implement in future updates) like:
* setting the focus to single observations
* changing the plotting window
* changing the centrainty threshold for observations to be classified as atypical

etc.

To start the shiny app, I recommend cloning this repository and executing the file **app.R** locally. This will start the shiny app on a local server. Instead, one can choose to use the **binder** button on the repo site to start it. Due to a lengthy build process, this is not the recommended way to look at it.

The shiny app serves as the visualization for all results of the previously explained simulation studies and shows what a possible deployment of the method in a real world scenario could look like.

## Outlook
---
The current state of the project and the implementation can serve as an example for the capabilities of the functional depth approach described by

### Finalize Implementation of Updating Procedure
In the current state of the project, the implementation of the updating procedure is incomplete. Future revisions will fill this incompleteness and add simulations to show the process of updating and compare cases where observations were added in an updating procedure to data sets containing them from the beginning. This can serve as a device to check the validity of the updating procedure.

### Improvements of Implementation
The implementation above still has some other problems. The most striking is the tendency of the sampling procedure to mark a full sample as atypical, if it contains a lower than expected fraction of atypical observations.

I plan to address this by introducing a rescaling factor to the cutoff threshold $C$ that adjusts for the removal of observations from the sample currently under consideration. This seems reasonable, as adding a new observation to a data set will always increase calculated depths for all observations. Therefore, removing observations can lead to overall lower depths and incrementally lower all observations under the cutoff that determines if they are classified as atypical in a sample. This is not described in the paper this project is based on - so it will make some theoretical work necessary to develop an appropritae correction.

### Generalizations
As described in a previous section, the method described in this project for data sets which allow for zeroing of the observations could be generalized by introducing another parameter **acceptable shifting** in addition to **acceptable stretching**. This could then be used to define a method that finds comparable subsets in data sets that do not allow zeroing. As this introduces more variables and makes a more sophisticated splitting algorithm necessary, it was out of the scope of this project, but will be addressed in the future.

### Parameter Choice
An important part of this procedure will be the choice of its tuning parameters:
* In its purest form the algorithm needs a choice for $\alpha$, $\gamma$, $B$ and a grid to use for approximation purposes
* Adding sampling adds the choice of sample size and number of samples
* The full procedure then additionally needs $\lambda$

Some of these like sample size and number of samples could be made dependent on the structure of comparable subsets and even change when switching from one comparable subset to the next. One goal of this could be to make each observation appear in a similar number of samples overall. But other reasonable procedures are possible.

Others like $\alpha$ and $\lambda$ could be chosen by a simulation method. Constructing a similar but smaller data set with intentionally added outliers to perform cross validation or a similar procedure could be an approach for this case. A more sophisticated and detailed description of this method will be part of future revisions.

### Making Results reproducible
Currently, the results of the method are mostly non-reproducible when taking about exact numbers. Qualitative results of the procedure will be similar, but due to randomization in the method the replication of exact numbers is currently not possible. This can be addressed with some work to allow for manually setting seeds in the classification algorithm without sampling and in the choice of samples. Because of parallelization this cannot be done be setting the seed once in the beginning.

### Performance Measures for the Algorithm & Benchmarking
Tightly connected to the point of parameter choice is the question of how to measure the performance of the outlier detection procedure in different settings.
* First, I am going to look for existing data sets that are commonly used to benchmark outlier classification procedures. The performance of this algorithm in these preclassified settings can serve as grounds for determining in which cases and using which parameter choices the method performs well and compare it to existing methods that are applicable in comparable scenarios.

* Second, as can be seen by the previously generated data, an ex ante classification of realizations created by different dgps may not appropriately cover the idea of outlyingness. Some realizations in previous data sets look very typical for the non-outlier dgp and a classification as atypical due to the different dgp could lead to an underestimation of the procedures performance. Therefore, a comparison to established methods can serve as a better tool to judge the effectiveness.

### Identifying what contributes to outlyingness of an observation
Once atypical observations in a data set are identified, it is very interesting to see what contributes to their outlyingness in the eyes of the algorithm. To create some ex-post explanation for why a classification decision was made would be a useful tool to inform future real-world decisions or improve the procedure itself by incorporating that information into the mechanism. Some interesting approaches to create an ex-post explanation are the following:

1. Create slightly altered realizations of an observation that has been marked as an outlier and see what effect different alterations have on its classification (or certainty in case of the sampling-based methods).
2. Compare locally similar observations that have different outcomes in the overall procedure.

There are more ways to gain information on what features contribute to outlyingness, but due to the current scope of this project, more in-depth considerations of this approach will only be part of future revisions.

### Creation of an Rcpp-Package to improve usability
Further iterations of this project will be different in the way the functions are made available. Instead of implementing the functions directly in the main notebook, there will be an additional notebook explaining the implementation and an R package that can be installed directly that contains all functionality provided above. This will make it easier to use the functions in the future.

### Improvements to the Shiny App
The visualizations of the shiny app will also undergo considerable overhauls, starting with changes to which observations are plotted when focus is set to a single observation. In this case for example plotting more observations in the close vicinity of the observation might be of greater interest, whereas observations farther away may provide little information of use and could be plotted less frequently to avoid overplotting.

There are further improvements planned but due to the interactive process of designing an interface that is meant for direct interaction with the user, it is difficult to predict the exact nature of the changes.

## Sources
---
* Cuevas, A. & Febrero-Bande, M. & Fraiman, R. (2006). On the use of bootstrap for estimating functions with functional data. Computational Statistics & Data Analysis. 51. 1063-1074.
* Febrero-Bande, M. & Galeano, P. & Gonzàlez-Manteiga, W. (2008). Outlier detection in functional data by depth measures, with application to identify abnormal NOx levels. Environmetrics. 19. 331 - 345.
* Gijbels, I. & Nagy, S. (2017). On a General Definition of Depth for Functional Data. Statistical Science. 32. 630-639.

---

Notebook by **Jakob R. Jürgens**

Final project for the course **OSE - data science** in the summer semester 2021

Find me at [jakobjuergens.com](https://jakobjuergens.com)


# Parameter settings

The following code imports the BD wrapper class and initializes it:

```python
import BD_wrapper.BD_wrapper as bdw
b = bdw.BayesianDistance()
```

In the following we list all possible parameter settings for the BD wrapper with their default settings.

### General settings

```python
b.version = '2.4'
```

The `version` parameter defines the BDC version that should be used. Currently available options are `'2.4'` (corresponding to the BDC version discussed in [Reid et al. 2019](https://ui.adsabs.harvard.edu/abs/2019ApJ...885..131R/abstract)) and `'1.0'` (corresponding to the BDC version discussed in [Reid et al. 2016](https://ui.adsabs.harvard.edu/abs/2016ApJ...823...77R/abstract)).

```python
b.use_ncpus = None
```

Specify the number of CPUs used for multiprocessing.

NOTE: By default the BD_wrapper uses 75% of all CPUs on the machine where it is running on unless the `use_ncpus` parameter is specified.

```python
b.verbose = True
```

Set the `verbose` parameter to `False` if you don't want status messages to be printed to the terminal.

```python
b.save_temporary_files = False
```

If the `save_temporary_files` parameter is set to `True`, all temporary files produced by the BDC will be saved (by default these files are removed after the main distance results are read in by the `BD wrapper`). setting this parameter to `True` can be useful in case of debugging or trying to understand the distance estimation process in detail.

```python
b.plot_probability = False
```

If the `plot_probability` is set to `True` a plot of the probability density distribution of the distance results (similar to Figs. 2 and 3 in [Riener et al. 2020b](https://ui.adsabs.harvard.edu/abs/2020A%26A...640A..72R/abstract)) is produced.

Note: Do not set the `save_temporary_files` and `plot_probability` parameters to `True` in case of large input tables!

### Parameters controlling the weight of the priors

```python
b.prob_sa = None  # default values: 0.85 (v2.4), 0.5 (v1.0)
b.prob_kd = None  # default values: 1.0 (v2.4), 0.85 (v1.0)
b.prob_gl = None  # default values: 1.0 (v2.4), 0.85 (v1.0)
b.prob_ps = None  # default values: 0.25 (v2.4), 0.15 (v1.0)
b.prob_pm = None  # default values: 0.85 (v2.4), N/A (v1.0)
```

These five parameters define the probabilities for the different priors. If they are not specified default values as specified in [Reid et al. 2019](https://ui.adsabs.harvard.edu/abs/2019ApJ...885..131R/abstract) (if `b.version = '2.4'`) [Reid et al. 2016](https://ui.adsabs.harvard.edu/abs/2016ApJ...823...77R/abstract) (if `b.version = '1.0'`) are assumed. The priors are:
- `prob_sa`: the proximity to features from an assumed spiral arm model
- `prob_kd`: the kinematic distance
- `prob_gl`: the Galactic latitude value or displacement from the Galactic midplane
- `prob_ps`: the proximity to high mass star-forming regions with parallax measurements
- `prob_pm`: the proper motion of the source (only available in v2.4)

### Parameters for input and output tables

```python
b.path_to_input_table = None
b.path_to_output_table = None
```

These parameter define the path to the input table and the resulting output table containing the distance results.

```python
b.table_format = 'ascii'
```
Table format of the input and resulting output table. See this [list](https://docs.astropy.org/en/stable/io/unified.html#built-in-table-readers-writers) for supported table formats.

```python
b.colname_lon = None
b.colname_lat = None
b.colname_vel = None
b.colname_e_vel = None
b.colname_kda = None
b.colname_name = None
b.colname_vel_disp = None
```

With the preceding seven parameters users can specify the corresponding columns in the input table. These column names specify:
- `colname_lon`: the name of the column giving the Galactic longitude value
- `colname_lat`: the name of the column giving the Galactic latitude value
- `colname_vel`: the name of the column giving the radial velocity (VLSR) value
- `colname_e_vel` (optional): the name of the column giving the error in radial velocity
- `colname_kda` (optional): the name of the column giving solutions for the kinematic distance ambiguity ('N', or 'F')
- `colname_name` (optional): the name of the column giving the source name
- `colname_vel_disp` (optional): the name of the column giving the velocity dispersion value associated with the source; has to be specified if `prior_velocity_dispersion` is set to `True`

Only the `colname_lon`, `colname_lat`, and `colname_vel` parameters have to be specified. In case there are no column names available, alternatively also the column number can be specified with the following parameters instead:

```python
b.colnr_lon = None
b.colnr_lat = None
b.colnr_vel = None
b.colnr_e_vel = None
b.colnr_kda = None
b.colnr_name = None
b.colnr_vel_disp = None
```

```python
b.add_kinematic_distance = True
```

If the `add_kinematic_distance` parameter is set to `True`, the distance results will include two additional columns giving the solutions of the near and far kinematic distances. This essentially corresponds to a BDC run in which all priors apart from KD prior are switched off.

```python
b.add_galactocentric_distance = True
```

If the `add_galactocentric_distance` parameter is set to `True`, the distance results will include an additional column giving the Galactocentric distance.

```python
b.default_e_vel = 5.0
```

The `default_e_vel` parameter sets the default value for the uncertainty in the radial velocity (VLSR) in case neither `colname_e_vel` or `colnr_e_vel` is specified or if the corresponding value is not given.

```python
b.max_e_vel = 5.0
```

The `max_e_vel` parameter sets the maximum value for the uncertainty in the radial velocity (VLSR). e_VLSR values exceeding this threshold are set to `max_e_vel`.

```python
b.kda_weight = 1
```
If the `colname_kda` or `colnr_kda` parameters are specified, users can decide whether to modify their weight with the `kda_weight` parameter. For example, if `kda_weight = 1`, KDA input values of `'N'` and `'F'` would result in $P_{\text{far}} = 0$ and $P_{\text{far}} = 1$, respectively. If `kda_weight = 0.5`, these values reduce to $P_{\text{far}} = 0.25$ and $P_{\text{far}} = 0.75$.

### Parameters for the prior for the kinematic distance ambiguity

```python
b.check_for_kda_solutions = True
b.kda_info_tables = []
b.exclude_kda_info_tables = []
```

If the `check_for_kda_solutions` parameter is set to `True`, solutions of the kinematic distance ambiguity for sources in the literature will be crossmatched with the input source coordinates to inform the $P_{\text{far}}$ prior. See Sect. 3.2 and Appendix A in [Riener et al. 2020b](https://ui.adsabs.harvard.edu/abs/2020A%26A...640A..72R/abstract) for more information. By default all KDA info tables in the `KDA_info` directory are considered. Users can specify whether they want to include or exclude specific catalogues. For example `b.kda_info_tables = ['Roman-Duval+09', 'Urquhart+18']` would only consider these KDA tables, whereas specifying `b.exclude_kda_info_tables = ['Roman-Duval+09', 'Simon+06', 'Urquhart+18']` would exclude these three tables, but consider the remaining nine tables.

### Parameters for the size-linewidth prior

The following parameters define the size-linewidth prior. The size-linewidth prior is only used if the `prior_velocity_dispersion` value is set to `True`. This prior requires that either `colname_vel_disp` or `colnr_vel_disp` are specified by the user.

```python
b.prior_velocity_dispersion = False
b.beam = None
b.random_seed = 177
b.sample = 1000
b.size_linewidth_index = 0.5
b.size_linewidth_e_index = 0.1
b.size_linewidth_sigma_0 = 0.7
b.size_linewidth_e_sigma_0 = 0.1
```

The size-linewidth prior tries to solve for the KDA by considering the velocity dispersion associated with the input source. By default the size-linewidth relationship established by [Solomon et al. (1987)](https://ui.adsabs.harvard.edu/abs/1987ApJ...319..730S/abstract) is used which is defined as:

\begin{equation}
\sigma_{v,exp} = \sigma_{v,0}\cdot\left( \dfrac{L}{1 \text{pc}} \right)^{\gamma}
\end{equation}

where $\sigma_{v,0}$ corresponds to the `size_linewidth_sigma_0` parameter and the exponent $\gamma$ corresponds to the `size_linewidth_index` parameter. The expected velocity dispersion values for the two kinematic distance values for the input source are determined from the average value of a Monte Carlo sampling of the size-linewidth relationship within the specified errors of  $\sigma_{v,0}$ and $\gamma$, which are given by the `size_linewidth_e_sigma_0` and `size_linewidth_e_index` parameters. The `random_seed` parameter intitializes the random seed for the Monte Carlo sampling and the sample size can be specified with the `sample` parameter.
The `beam` parameter specifies the telescope beam and has to be valid [astropy.units.Quantity](https://docs.astropy.org/en/stable/api/astropy.units.Quantity.html#astropy.units.Quantity) object.

See 3.3 in [Riener et al. 2020b](https://ui.adsabs.harvard.edu/abs/2020A%26A...640A..72R/abstract) for more details about the size-linewidth prior.

```python

```