## Datasets

```python
# Visualization
%pylab inline
from IPython.display import display, Math, Latex
import matplotlib.pyplot as plt

# handling data
import csv
import json
import pandas as pd

# Math
from random import random
import scipy.stats as ss
import numpy as np
import itertools
from collections import Counter
```

    Populating the interactive namespace from numpy and matplotlib

    /usr/local/anaconda3/envs/Datascience/lib/python3.7/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['random']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

We will use the following dataset to define the universe of possible values throughout the notebook to test the paper, it contains the setting's universe values and the people the author's of the paper want to protect. To simplify things without the loss of generality, the authors just use 4 people. The 4 people belong to a high school. The high school would like to release a dataset to be queried; however, this new dataset will not include people that have been in probation, in this case, Terry.

```python
# We define the actual dataset (conforming the universe)
dict_school = {'name': ['Chris', 'Kelly', 'Pat', 'Terry'], 'school_year': [1, 2, 3, 4], 'absence_days': [1, 2, 3, 10]}
```

```python
df_school = pd.DataFrame(dict_school)
df_school
```

      name
      school_year
      absence_days

      0
      Chris
      1
      1

      1
      Kelly
      2
      2

      2
      Pat
      3
      3

      3
      Terry
      4
      10

The attacker's ultimate goal is to find out which student was placed in probation, Terry. However, the attacker will only be able to query this other dataset (Without Terry, because the release dataset only contains students who were not in probation):

```python
# We define the the dataset that we will release
df_school_release = df_school.drop([3], axis=0)
df_school_release
```

      name
      school_year
      absence_days

      0
      Chris
      1
      1

      1
      Kelly
      2
      2

      2
      Pat
      3
      3

To accomplish this, the attacker would perform analytics queries on df_school_release such as the mean. With the results, the adversary will try to improve his/her guess about which dataset is the true one to single out and discover the student who was placed on probation (Terry).

The adversary model adopted in the paper is the worst-case scenario. It will be the one I adopted in this notebook as well: An attacker has infinite computation power. Because DP is supposed to provide privacy given adversaries with arbitrary background knowledge, it is okay to assume that the adversary has full access to all the records (Knows all the universe - df_school). However, there is a dataset made from the universe without an individual (df_school_release), and he does not know who is and who is not in this dataset (This is the only thing he does not know). However, he knows this dataset contains people with a certain quality (The students who have not been on probation). With the initial dataset, the attacker will reconstruct the dataset he does not know, querying the new dataset (df_school_release) without having access to it.

## Functions

### Auxiliary function

```python
# With this funciton, we can make easier to call the mean, median... functions
# REF: https://stackoverflow.com/questions/34794634/how-to-use-a-variable-as-function-name-in-python
# It is not clean to have the var percentile input each function, but it is less verbose than having a function
# For each percentile. We could however limit the maount of percentiles offer to 25 and 75.
class Query_class:
    """
    A class used to represent a query. You instantiate an object that will perform a particlar query on an array

    Attributes
    ----------
    fDic - (dict) containing the possible queries the class can be transformed into
    fActive - (function) it contins the function we created the class to have

    Methods
    -------

    run_query - I will run the query for which we instantiated the class
    The other methods implement the different possible queries

    """

    def __init__(self, fCase):

        # mapping: string --> variable = function name
        fDic = {'mean':self._mean,
                'median':self._median,
                'count': self._count,
                'sum': self._sum,
                'std': self._std,
                'var': self._var,
                'percentile': self._percentile}

        self.fActive = fDic[fCase]

    # Calculate the mean of an array
    def _mean(self, array, percentile):

        return np.mean(array)

    # Calculate the median of an array
    def _median(self, array, percentile):

        return np.median(array)

    # Calculate the number of elements in the array
    def _count(self, array, percentile):

        return len(array)

    # Calculate the sum of an array
    def _sum(self, array, percentile):

        return np.sum(array)

    # Calculate the std of an array
    def _std(self, array, percentile):

        return np.std(array)

    # Calculate the variance of an array
    def _var(self, array, percentile):

        return np.var(array)

    def _percentile(self, array, percentile):

        return np.percentile(array, percentile)

    # It will run the given query
    def run_query(self, array, percentile=50):

        return self.fActive(array, percentile)
```

```python
# Set of checks on the input values
def verify_sensitivity_inputs(universe_cardinality, universe_subset_cardinality, hamming_distance):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset_cardinality - (df) cardinality of the universe subset
        hamming_distance - (int) hamming distance between neighboring datasets

    OUTPUT:
        ValueError - (str) error message due to the value of the inputs

    Description:
    It performs multiple checks to verify the validity of the inputs for the calculation of senstitivity
    """

    # Check on unverse cardinality (1).
    # The cardinality of the subset of the universe cannot be larger than the universe
    if universe_cardinality < universe_subset_cardinality:

        raise ValueError("Your universe dataset cannot be smaller than your release dataset.")

    # Checks on the validity of the chosen hamming_distance (3)
    if hamming_distance >= (universe_subset_cardinality):

        raise ValueError("Hamming distance chosen is larger than the cardinality of the release dataset.")

    if (hamming_distance > np.abs(universe_cardinality - universe_subset_cardinality)):

        raise ValueError("Hamming distance chosen is larger than the cardinality difference between the \
        universe and the release dataset, i.e., \
        there are not enough values in your universe to create such a large neighboring dataset (Re-sampling records).")

    # The hamming distance cannot be 0, then your neighbor dataset is equal to the original dataset
    if hamming_distance == 0:

        raise ValueError("Hamming distance cannot be 0.")
```

```python
# Used by unbounded unbounded_empirical_global_L1_sensitivity_a
def L1_norm_max(release_dataset_query_value, neighbor_datasets, query, percentile):
    """
    INPUT:
        release_dataset_query_value - (float) query value of a particular possible release dataset
        neighbor_datasets - (list) contains the possible neighbors of the specific release dataset
        query - (object) instance of class Query_class
        percentile - (int) percentile value for the percentile query

    OUTPUT:
        L1_norm_maximum - (float) maximum L1 norm calcuated from the differences between the query results
        of the neighbor datasets and the specific release dataset

    Description:
    It claculates the maximum L1 norm between the query results of the neighbor datasets and the specific release dataset
    """

    neighbor_dataset_query_values = []
    for neighbor_dataset in neighbor_datasets:

        neighbor_dataset_query_value = query.run_query(neighbor_dataset, percentile)
        neighbor_dataset_query_values.append(neighbor_dataset_query_value)

    # We select the maximum and minimum values of the queries, as the intermediate values will not
    # yield a larger L1 norm (ultimately, we are interested in the maximum L1 norm)
    neighbor_dataset_query_value_min, neighbor_dataset_query_value_max = \
    min(neighbor_dataset_query_values), max(neighbor_dataset_query_values)
    # We calculate the L1 norm for these two values and pick the maximum
    L1_norm_i = np.abs(release_dataset_query_value - neighbor_dataset_query_value_min)
    L1_norm_ii = np.abs(release_dataset_query_value - neighbor_dataset_query_value_max)
    L1_norm_maximum = max(L1_norm_i, L1_norm_ii)

    return L1_norm_maximum
```

```python
def calculate_unbounded_sensitivities(universe, universe_subset_cardinality, columns, hamming_distance, unbounded_sensitivities):
    """
    INPUT:
        universe - (df or dict) contains all possible values of the dataset
        universe_subset_cardinality - (int) contains the length of the subset chosen for the release dataset
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        hamming_distance - (int) hamming distance between neighboring datasets
        unbounded_sensitivities - (dict) stores sensitivities per hamming distance and query type

    OUTPUT
        unbounded_sensitivities - (dict) stores sensitivities per hamming distance and query type

    Description:
    It calculates the sensitivities for a set of queries given a universe and a release dataset.
    """
    # Calculate the sensitivity of different queries for the unbounded DP
    query_type = 'mean'
    mean_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'median'
    median_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'count'
    count_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'sum'
    sum_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'std'
    std_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'var'
    var_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'percentile'
    percentile = 25
    percentile_25_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)
    percentile = 50
    percentile_50_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)
    percentile = 75
    percentile_75_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)
    percentile = 90
    percentile_90_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)

    print('Unbounded sensitivities for mean', mean_unbounded_global_sensitivities)
    print('Unbounded sensitivities for median', median_unbounded_global_sensitivities)
    print('Unbounded sensitivities for count', count_unbounded_global_sensitivities)
    print('Unbounded sensitivities for sum', sum_unbounded_global_sensitivities)
    print('Unbounded sensitivities for std', std_unbounded_global_sensitivities)
    print('Unbounded sensitivities for var', var_unbounded_global_sensitivities)
    print('Unbounded sensitivities for percentile 25', percentile_25_unbounded_global_sensitivities)
    print('Unbounded sensitivities for percentile 50', percentile_50_unbounded_global_sensitivities)
    print('Unbounded sensitivities for percentile 75', percentile_75_unbounded_global_sensitivities)
    print('Unbounded sensitivities for percentile 90', percentile_90_unbounded_global_sensitivities)

    unbounded_sensitivities = build_sensitivity_dict(unbounded_sensitivities, hamming_distance,\
    mean_unbounded_global_sensitivities, median_unbounded_global_sensitivities, count_unbounded_global_sensitivities, \
    sum_unbounded_global_sensitivities, std_unbounded_global_sensitivities, var_unbounded_global_sensitivities, \
    percentile_25_unbounded_global_sensitivities, percentile_50_unbounded_global_sensitivities, \
    percentile_75_unbounded_global_sensitivities, percentile_90_unbounded_global_sensitivities)

    return unbounded_sensitivities
```

```python
def calculate_bounded_sensitivities(universe, universe_subset_cardinality, columns, hamming_distance, bounded_sensitivities):
    """
    INPUT:
        universe - (df or dict) contains all possible values of the dataset
        universe_subset_cardinality - (int) contains the length of the subset chosen for the release dataset
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        hamming_distance - (int) hamming distance between neighboring datasets
        unbounded_sensitivities - (dict) stores sensitivities per hamming distance and query type

    OUTPUT
        bounded_sensitivities - (dict) stores sensitivities per hamming distance and query type

    Description:
    It calculates the sensitivities for a set of queries given a universe and a release dataset.
    """
    # Calculate the sensitivity of different queries for the unbounded DP
    query_type = 'mean'
    mean_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'median'
    median_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'count'
    count_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'sum'
    sum_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'std'
    std_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'var'
    var_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance)
    query_type = 'percentile'
    percentile = 25
    percentile_25_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)
    percentile = 50
    percentile_50_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)
    percentile = 75
    percentile_75_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)
    percentile = 90
    percentile_90_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns,  query_type, hamming_distance, percentile)

    print('Bounded sensitivities for mean', mean_bounded_global_sensitivities)
    print('Bounded sensitivities for median', median_bounded_global_sensitivities)
    print('Bounded sensitivities for count', count_bounded_global_sensitivities)
    print('Bounded sensitivities for sum', sum_bounded_global_sensitivities)
    print('Bounded sensitivities for std', std_bounded_global_sensitivities)
    print('Bounded sensitivities for var', var_bounded_global_sensitivities)
    print('Bounded sensitivities for percentile 25', percentile_25_bounded_global_sensitivities)
    print('Bounded sensitivities for percentile 50', percentile_50_bounded_global_sensitivities)
    print('Bounded sensitivities for percentile 75', percentile_75_bounded_global_sensitivities)
    print('Bounded sensitivities for percentile 90', percentile_90_bounded_global_sensitivities)

    bounded_sensitivities = build_sensitivity_dict(bounded_sensitivities, hamming_distance,\
    mean_bounded_global_sensitivities, median_bounded_global_sensitivities, count_bounded_global_sensitivities, \
    sum_bounded_global_sensitivities, std_bounded_global_sensitivities, var_bounded_global_sensitivities, \
    percentile_25_bounded_global_sensitivities, percentile_50_bounded_global_sensitivities, \
    percentile_75_bounded_global_sensitivities, percentile_90_bounded_global_sensitivities)

    return bounded_sensitivities
```

```python
# We save the values in a dictionary
def build_sensitivity_dict(unbounded_sensitivities, hamming_distance, mean_sensitivity, median_sensitivity, count_sensitivity, _sum_sensitivity, _std_sensitivity, _var_sensitivity, percentile_25_sensitivity, percentile_50_sensitivity, percentile_75_sensitivity, percentile_90_sensitivity):
    """
    INPUT
        unbounded_sensitivities - (dict) stores sensitivities per hamming distance and query type
        hamming_distance - (int) hamming distance of the neighboring datasets
        mean_sensitivity - (float) sensitivity of the mean query
        median_sensitivity - (float) sensitivity of the media query
        count_sensitivity - (float) sensitivity of the count query
        _sum_sensitivity - (float) sensitivity of the sum query
        _std_sensitivity - (float) sensitivity of the std query
        _var - (float) sensitivity of the var query
        percentile_25_sensitivity - (float) sensitivity of the percentile 25 query
        percentile_50_sensitivity - (float) sensitivity of the percentile 50 query
        percentile_75_sensitivity - (float) sensitivity of the percentile 75query
        percentile_90_sensitivity - (float) sensitivity of the percentile 90 query

    OUTPUT
        unbounded_sensitivities - (dict) stores sensitivities per hamming distance and query type

    """
    unbounded_sensitivities[hamming_distance] = {}
    unbounded_sensitivities[hamming_distance]['mean'] = mean_sensitivity
    unbounded_sensitivities[hamming_distance]['median'] = median_sensitivity
    unbounded_sensitivities[hamming_distance]['count'] = count_sensitivity
    unbounded_sensitivities[hamming_distance]['sum'] = _sum_sensitivity
    unbounded_sensitivities[hamming_distance]['std'] = _std_sensitivity
    unbounded_sensitivities[hamming_distance]['var'] = _var_sensitivity
    unbounded_sensitivities[hamming_distance]['percentile_25'] = percentile_25_sensitivity
    unbounded_sensitivities[hamming_distance]['percentile_50'] = percentile_50_sensitivity
    unbounded_sensitivities[hamming_distance]['percentile_75'] = percentile_75_sensitivity
    unbounded_sensitivities[hamming_distance]['percentile_90'] = percentile_90_sensitivity

    return unbounded_sensitivities
```

### Main Functions

##### Equation in 4.1 after its first paragraph - Definition of sensitivity

```latex
%%latex
\begin{align}
\ell_{1, \mbox{sensitivity}}: \Delta f=\max_{\substack{
{x, y \in \mathbb{N}^{(\mathcal{X})}} \\
\|x-y\|_{1} = h
}} \|f(x)-f(y)\|_{1}
\end{align}
```

\begin{align}
\ell_{1, \mbox{sensitivity}}: \Delta f=\max_{\substack{
{x, y \in \mathbb{N}^{(\mathcal{X})}} \\
\|x-y\|_{1} = h
}} \|f(x)-f(y)\|_{1}
\end{align}

\begin{align}
\ell_{1, \mbox{sensitivity}}: \Delta f=\max_{\substack{
{x, y \in \mathbb{N}^{(\mathcal{X})}} \\
\|x-y\|_{1} = h
}} \|f(x)-f(y)\|_{1}
\end{align}

```python
def unbounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns, query_type, hamming_distance, percentile=50):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset_cardinality - (int) contains the length of the subset chosen for the release dataset
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        query_type - (str) contain the category declaring the type of query to be later on executed
        hamming_distance - (int) hamming distance between neighboring datasets
        percentile - (int) percentile value for the percentile query

    OUTPUT:
        unbounded_global_sensitivity - (float) the unbounded global sensitivity of the input universe

    Description:
    It claculates the global sensitivity of an array based on the knowledge of the entire universe of
    the dataset and query_type.
    """

    # We initialie the type of query for which we would like calculate the sensitivity
    query = Query_class(query_type)

    # We will store the sensitivity of each column of the dataset containing universe in a dictionary
    unbounded_global_sensitivity_per_colum = {}

    for column in columns:

        # Check if the values for the hamming distance and universe sizes comply with the basic constraints
        verify_sensitivity_inputs(len(universe[column]), universe_subset_cardinality, hamming_distance)

        # 1) RELEASE DATASET
        # We calculate all the possible release datasets formed by the combination of values sampled from the universe
        release_datasets = itertools.combinations(universe[column], universe_subset_cardinality)
        release_datasets = list(release_datasets)

        # 2) |NEIGHBORING DATASET| < |RELEASE DATASET| //// cardinalities
        # The neighboring datasets are subsets of a smaller dimension of the possible release datasets (smaller by the hamming_distance)
        # The neighboring release datasets are used to calculate the max sensitivity, stemming from the DP definition
        neighbor_with_less_records_datasets = []
        for release_dataset in release_datasets:

            # These yields the smaller possible neighboring datasets
            neighbor_with_less_records_dataset = itertools.combinations(release_dataset, \
                universe_subset_cardinality - hamming_distance)
            neighbor_with_less_records_dataset = list(neighbor_with_less_records_dataset)
            neighbor_with_less_records_datasets.append(neighbor_with_less_records_dataset)

        # 3) |NEIGHBORING DATASET| > |RELEASE DATASET| //// cardinalities
        # similar process but adding  records
        neighbor_with_more_records_datasets = []
        for release_dataset in release_datasets:

            # We obtain combinations of values from the univsere and these will be appended to the release datasets.
            # The size of each combination is equal to the hamming distance, as the neighboring dataset will be that much larger
            # However, in case your universe is a dataset and not just a range of values, then the  neighboring
            # dataset could contain the same record twice, which is NOT desirable (1 person appearing twice)
            # Therefore, the values must be sampled from the symmetric difference between the release dataset and the universe dataset
            # REF: https://www.geeksforgeeks.org/python-difference-of-two-lists-including-duplicates/
            symmetric_difference = list((Counter(universe[column]) - Counter(release_dataset)).elements())
            neighbor_possible_value_combinations = itertools.combinations(symmetric_difference, hamming_distance)
            neighbor_possible_value_combinations = list(neighbor_possible_value_combinations)

            temp_neighbor_with_more_records_datasets = []
            for neighbor_possible_value_combination in neighbor_possible_value_combinations:

                # We create neighboring datasets by concatenating the neighbor_possible_value_combination with the release dataset
                neighbor_with_more_records_dataset = list(release_dataset + neighbor_possible_value_combination)
                temp_neighbor_with_more_records_datasets.append(neighbor_with_more_records_dataset)

            # We append in this manner to cluster the neighboring datasets with their respective release dataset
            neighbor_with_more_records_datasets.append(temp_neighbor_with_more_records_datasets)

        # 4) For each possible release datase, there is a set of neighboring datasets
        # We will iterate through each possible release dataset and calculate the L1 norm with
        # each of its repspective neighboring datasets
        L1_norms = []

        for i, release_dataset in enumerate(release_datasets):

            release_dataset_query_value = query.run_query(release_dataset, percentile)

            L1_norm =  L1_norm_max(release_dataset_query_value, neighbor_with_less_records_datasets[i], query, percentile)
            L1_norms.append(L1_norm)
            L1_norm =  L1_norm_max(release_dataset_query_value, neighbor_with_more_records_datasets[i], query, percentile)
            L1_norms.append(L1_norm)

        # We pick the maximum out of all the maximum L1_norms calculated from each possible release dataset
        unbounded_global_sensitivity_per_colum[column] = max(L1_norms)

    return unbounded_global_sensitivity_per_colum
```

##### You can find this definition after equation (5) of 5.1

```latex
%%latex
\begin{align}
\Delta v=\max_{\substack{
{1 \leq i, j \leq n} \\ \\
i \neq j
}} \|f(w_i)-f(w_j)\|_{1}
\end{align}
```

\begin{align}
\Delta v=\max_{\substack{
{1 \leq i, j \leq n} \\ \\
i \neq j
}} \|f(w_i)-f(w_j)\|_{1}
\end{align}

\begin{align}
\Delta v=\max_{\substack{
{1 \leq i, j \leq n} \\ \\
i \neq j
}} \|f(w_i)-f(w_j)\|_{1}
\end{align}

```python
def bounded_empirical_global_L1_sensitivity(universe, universe_subset_cardinality, columns, query_type, hamming_distance, percentile=50):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset_cardinality - (int) contains the length of the subset chosen for the release dataset
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        query_type - (str) contain the category declaring the type of query to be later on executed
        hamming_distance - (int) hamming distance between neighboring datasets
        percentile - (int) percentile value for the percentile query

    OUTPUT:
        bounded_global_sensitivity - (float) the bounded global sensitivity of the input universe

    Description:
    It claculates the global sensitivity of an array based on the knowledge of the entire universe of
    the dataset and query_type.
    """

    # We initialie the type of query for which we would like calculate the sensitivity
    query = Query_class(query_type)

    # We will store the sensitivity of each column of the dataset containing universe in a dictionary
    bounded_global_sensitivity_per_column = {}

    for column in columns:

            # Check if the values for the hamming distance and universe sizes comply with the basic constraints
        verify_sensitivity_inputs(len(universe[column]), universe_subset_cardinality, hamming_distance)

        # We calculate all the possible release datasets
        # First we obtain the combinations within the release dataset. The size of this combinations is not the original size
        # but the original size minus the hamming_distance
        release_i_datasets = itertools.combinations(universe[column], universe_subset_cardinality - hamming_distance)
        release_i_datasets = list(release_i_datasets)

        # it will contain sets of neighboring datasets. The L1 norm will be calculated between these sets. The maximum will be chosen
        # The datasets from different groups do not necesarilly need to be neighbors, thus we separate them in groups
        neighbor_datasets = []
        for release_i_dataset in release_i_datasets:

            # second we calculate the combinations of the items in the universe that are not in the release dataset
            # the size of a combination is equal to the hamming distance
            symmetric_difference = list((Counter(universe[column]) - Counter(release_i_dataset)).elements())
            release_ii_datasets = itertools.combinations(symmetric_difference, hamming_distance)
            release_ii_datasets = list(release_ii_datasets)

            # We create neighboring datasets by concatenating i with ii
            temp_neighbors = []
            for release_ii_dataset in release_ii_datasets:

                temp_neighbor = list(release_i_dataset + release_ii_dataset)
                temp_neighbors.append(temp_neighbor)

            neighbor_datasets.append(temp_neighbors)

        # We calculate the L1_norm for the different combinations with the aim to find the max
        # We can loop in this manner because we are obtaining the absolute values
        L1_norms = []
        for m in range(0, len(neighbor_datasets)):

            for i in range(0,  len(neighbor_datasets[m])-1):

                for j in range(i+1, len(neighbor_datasets[m])):

                    L1_norm = np.abs(query.run_query(neighbor_datasets[m][i], percentile) - query.run_query(neighbor_datasets[m][j], percentile))
                    L1_norms.append(L1_norm)

        bounded_global_sensitivity_per_column[column] = max(L1_norms)

    return bounded_global_sensitivity_per_column
```

```python
def prior_belief(universe, universe_subset, columns):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset - (df) contains the values of the dataset to be released
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from

    OUTPUT:
        prior_per_column - (dict) maps each column of all the possible release datasets to the prior knowledge of an adversary

    Description:
    It calculates the prior knowledge of an adversary assuming uniform distribution
    """
    # Initialize the dictionary to store all the posteriors with the column names as keys
    prior_per_column = {}

    for column in columns:

        # We calculate all the possible release datasets
        release_datasets = itertools.combinations(universe[column], len(universe_subset[column]))
        release_datasets = list(release_datasets)
        number_possible_release_datasets = len(release_datasets)
        # We assume a uniform prior
        prior_per_column[column] = [1 / number_possible_release_datasets] * number_possible_release_datasets

    return prior_per_column
```

#### Definition 3, also in equation 2 of 5.1 but with another form / right before the triangle inequality

```python
def posterior_belief(universe, universe_subset, columns, query_type, query_result, sensitivity, epsilon):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset - (df) contains the values of the dataset to be released
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        query_type - (str) contain the category declaring the type of query to be later on executed
        query_result - (float) the result of the query the attacker received
        sensitivity - (dict) it maps the columns to their sensitivities, they can be based on bounded or unbounded DP
        epsilon - (float) it is the parameter that tunes noise, based on DP

    OUTPUT:
        posterior_per_column - (dict) maps each column of all the possible release datasets to the posterior knowledge of an adversary

    Description:
    It calculates the posterior knowledge of an adversary
    """

    # Initialize the dictionary to store all the posteriors with the column names as keys
    posterior_per_column = {}

    # We initialie the type of query for which we would like calculate the sensitivity
    query = Query_class(query_type)

    for column in columns:

        # We calculate all the possible release datasets
        release_datasets = itertools.combinations(universe[column], len(universe_subset[column]))
        release_datasets = list(release_datasets)

        # According to the definiton of DP, the sacle factor of a Laplacian distribution:
        scale_parameter = sensitivity[column]/epsilon

        posterior_probability = []
        for release_dataset in release_datasets:
            probability = (1 / (2 * scale_parameter)) * np.exp(- np.abs(query_result - np.mean(release_dataset)) / scale_parameter)
            posterior_probability.append(probability)
            print(probability)
        posterior_per_column[column] = posterior_probability / np.sum(posterior_probability)

    return posterior_per_column
```

#### Result from 4.1 probability ratio of 3.2933

```python
scale_parameter = (17/12)
```

Using the cumulative distribution functions. REF: https://en.wikipedia.org/wiki/Laplace_distribution

But we want a P(x > value) and not P(x < value), the latter is given by the REF

```python
# probability that output is greater than 1.1677 > 0=mu
0.5*(np.exp(- 1.1677 / scale_parameter))
```

    0.21927996116513163

```python
# prob that he output is greater than -0.832 < 0=mu
1 - 0.5*(np.exp(- 0.832 / scale_parameter))
```

    0.7220853698398951

```python
0.7221/0.2192
```

    3.294251824817518

```python
0.5 + 0.5 * np.sign(1.1677)*(1 - np.exp(- 1.1677 / scale_parameter))
```

    0.7807200388348684

##### Definition 4

```python
def confidence(priors, posteriors, columns):
    """
    INPUT:
        priors - (dict) maps each column of all the possivle release datasets to the prior knowledge of an adversary
        posteriors - (dict) maps each column of all the possible release datasets to the posterior knowledge of an adversary
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from

    OUTPUT:
        confidence_per_column - (dict) maps each column to the confidence of the adversary

    Description:
    It calculates the confidence of an adversary after seeing the prior
    """

    confidence_per_column = {}

    for column in columns:

        confidence_per_column[column] = np.max(posteriors[column] - priors[column])

    return confidence_per_column
```

##### Definition 5

```python
def risk_disclosure(posteriors, columns):
    """
    INPUT:
        posteriors - (dict) maps each column of all the possible release datasets to the posterior knowledge of an adversary
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from

    OUTPUT:
        posterior_max_per_column - (dict) maps each column to the risk of disclosure

    Description:
    It calculates the risk of disclosure per column, which is the max posterior
    """
    posterior_max_per_column = {}

    for column in columns:

        posterior_max_per_column[column] = np.max(posteriors[column])

    return posterior_max_per_column
```

#### 5.1 equation 4-5

#### This  definition is used for equation. (4-5) of point 5.1

```python
def upper_bound_posterior(universe, columns, bounded_sensitivity, unbounded_sensitivity, epsilon):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        bounded_sensitivity - (dict) it maps the columns to their bounded sensitivities
        unbounded_sensitivity - (dict) it maps the columns to their unbounded sensitivities
        epsilon - (float) it is the parameter that tunes noise, based on DP

    OUTPUT:
        upper_bound_per_column - (dict) dictionary that maps the value that bounds the posterior (also the risk) per column

    Description:
    It calculates the upper bound of the posterior
    """

    upper_bound_posterior_per_column = {}

    for column in columns:

        upper_bound_posterior = 1 / (1 + (universe.shape[0] - 1) * \
                                np.exp(-epsilon * bounded_sensitivity[column] / unbounded_sensitivity[column]))

        upper_bound_posterior_per_column[column] = upper_bound_posterior

    return upper_bound_posterior_per_column
```

##### inequality 3 of 5.1

```python
def tighter_upper_bound_posterior(universe, universe_subset, columns, query_type, sensitivity, epsilon, percentile=50):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset - (df) contains the values of the dataset to be released
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        query_type - (str) contain the category declaring the type of query to be later on executed
        sensitivity - (dict) it maps the columns to their sensitivities, they can be based on bounded or unbounded DP
        epsilon - (float) it is the parameter that tunes noise, based on DP

    OUTPUT:
        tighter_upper_bound_posteriorper_column - (dict) maps each column of all the possible release datasets to
        the tighter upper bound of the posterior knowledge of an adversary

    Description:
    It calculates a tighter bound of the posterior knowledge of an adversary
    """
    # Initialize the dictionary to store all the posteriors with the column names as keys
    tighter_upper_bound_posterior_per_column = {}

    # We initialie the type of query for which we would like calculate the sensitivity
    query = Query_class(query_type)

    for column in columns:

        # We calculate all the possible release datasets
        release_datasets = itertools.combinations(universe[column], len(universe_subset[column]))
        release_datasets = list(release_datasets)

        # We calculate the L1_norm for the different combinations of the query result from different data releass
        # We have to complete all loops becuase we need to calculate different values of the posterior
        # Then we select the max after calculating all posteriors
        posterior_probability = []
        for i in range(0, len(release_datasets) - 1):

            L1_norms = []
            for j in range(0,  len(release_datasets)):

                if release_datasets[i] == release_datasets[j]:

                    continue

                else:

                    L1_norms.append(np.abs(query.run_query(release_datasets[i], percentile) - query.run_query(release_datasets[j], percentile)))

            denominator_posterior = 1
            for L1_norm in L1_norms:

                 denominator_posterior += np.exp(-epsilon * L1_norm / sensitivity[column])

            beta = 1 / denominator_posterior
            posterior_probability.append(beta)

        tighter_upper_bound_posterior_per_column[column] =  max(posterior_probability)

    return tighter_upper_bound_posterior_per_column
```

#### 5.2 - inequality 7

```python
def upper_bound_epsilon(universe, columns, bounded_sensitivity, unbounded_sensitivity, risk):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        bounded_sensitivity - (dict) it maps the columns to their bounded sensitivities
        unbounded_sensitivity - (dict) it maps the columns to their unbounded sensitivities
        risk - (float) it is a parameter that sets the privacy requirement. It is the probability that the attacker
        succeeds in his/her attack

    OUTPUT:
        epsilon_upper_bound_per_column - (dict) dictionary that maps the upper bound of epsilon per column

    Description:
    It calculates epsilon given a risk willing to take
    """

    epsilon_upper_bound_per_column = {}

    for column in columns:

        epsilon_upper_bound =  unbounded_sensitivity[column] / bounded_sensitivity[column] * \
                            np.log(((universe.shape[0] - 1) * risk) / (1 - risk))
        epsilon_upper_bound_per_column[column] = epsilon_upper_bound

    return epsilon_upper_bound_per_column
```

#### Binary search - 5.2

```python
def binary_search_epsilon(universe, universe_subset, columns, query_type, bounded_global_sensitivities, unbounded_global_sensitivities, privacy_requirement, posterior_bound_type='tight'):
    """
    INPUT:
        universe - (df) contains all possible values of the dataset
        universe_subset - (df) contains the values of the dataset to be released
        columns - (array) contains the names of the columns we would like to obtain the sensitivity from
        query_type - (str) contain the category declaring the type of query to be later on executed
        bounded_global_sensitivities - (dict) it maps the columns to their bounded sensitivities
        unbounded_global_sensitivities - (dict) it maps the columns to their unbounded sensitivities
        privacy_requirement - (float) the highest probability admisable of disclosure
        posterior_bound_type - (str) the type of bound we use to calculate the new values to decide upon which epsilon to take next

    OUTPUT:
        optimal_epsilon - (dict) dictionary that maps the optimal epsilons per column

    Description:
    It performs binary search to find the optimal epsilon
    """

    optimal_epsilon = {}
    for column in columns:

        max_risk = 0.9999
        epsilon_upper_bound = upper_bound_epsilon(universe, columns, bounded_global_sensitivities, unbounded_global_sensitivities, max_risk)
        epsilon_f = epsilon_upper_bound
        epsilon_s = 0

        for i in range(0,25):

            epsilon = (epsilon_f[column] + epsilon_s)/2

            # Check which type of bound
            if posterior_bound_type != 'upper':

                posterior_upper_bound_new = tighter_upper_bound_posterior(universe, universe_subset, columns, query_type, unbounded_global_sensitivities, epsilon)

            else:

                posterior_upper_bound_new = upper_bound_posterior(universe, columns, bounded_global_sensitivities, unbounded_global_sensitivities, epsilon)

            if posterior_upper_bound_new[column] < privacy_requirement:

                epsilon_s = epsilon

            elif posterior_upper_bound_new[column] > privacy_requirement:

                epsilon_f[column] = epsilon

        optimal_epsilon[column] = epsilon

    return optimal_epsilon
```

## MAIN

### True results for different query types - just a warm up

```python
# Finding true values of different queries
mean_year = df_school['school_year'].mean()
mean_absence_days = df_school['absence_days'].mean()

median_year = df_school['school_year'].median()
median_absence_days = df_school['absence_days'].median()

count_year = df_school['school_year'].count()
count_absence_days = df_school['absence_days'].count()

sum_year = df_school['school_year'].sum()
sum_absence_days = df_school['absence_days'].sum()

std_year = df_school['school_year'].std()
std_absence_days = df_school['absence_days'].std()

var_year = df_school['school_year'].var()
var_absence_days = df_school['absence_days'].var()

var_year = df_school['school_year'].var()
var_absence_days = df_school['absence_days'].var()

percentile_25_year = np.percentile(df_school['school_year'], 25)
percentile_25_absence_days = np.percentile(df_school['absence_days'], 25)
percentile_50_year = np.percentile(df_school['school_year'], 50)
percentile_50_absence_days = np.percentile(df_school['absence_days'], 50)
percentile_75_year = np.percentile(df_school['school_year'], 75)
percentile_75_absence_days = np.percentile(df_school['absence_days'], 75)

print('School year: mean =', mean_year, 'Absence days: mean =', mean_absence_days)
print('School year: median =', median_year, 'Absence days: median =', median_absence_days)
print('School year: count =', count_year, 'Absence days: count =', count_absence_days)
print('School year: sum =', sum_year, 'Absence days: sum =', sum_absence_days)
print('School year: std =', std_year, 'Absence days: std =', std_absence_days)
print('School year: var =', var_year, 'Absence days: var =', var_absence_days)
print('School year: 25th percentile =', percentile_25_year, 'Absence days: 25th percentile =', percentile_25_absence_days)
print('School year: 50th percentile =', percentile_50_year, 'Absence days: 50th percentile =', percentile_50_absence_days)
print('School year: 75th percentile =', percentile_75_year, 'Absence days: 75th percentile =', percentile_75_absence_days)

```

    School year: mean = 2.5 Absence days: mean = 4.0
    School year: median = 2.5 Absence days: median = 2.5
    School year: count = 4 Absence days: count = 4
    School year: sum = 10 Absence days: sum = 16
    School year: std = 1.2909944487358056 Absence days: std = 4.08248290463863
    School year: var = 1.6666666666666667 Absence days: var = 16.666666666666668
    School year: 25th percentile = 1.75 Absence days: 25th percentile = 1.75
    School year: 50th percentile = 2.5 Absence days: 50th percentile = 2.5
    School year: 75th percentile = 3.25 Absence days: 75th percentile = 4.75

## All the cross checks of the paper are done with the Mean, as the paper utilizes the mean as the use case

### Unbounded global sensitivity for different query types - 4.1 - 2.8333 of point 4.1

```python
# Calculate the sensitivity of different queries for the unbounded DP
columns = ['school_year', 'absence_days']
hamming_distance = 1
unbounded_sensitivities = {}
unbounded_sensitivities = calculate_unbounded_sensitivities(df_school, df_school_release.shape[0], columns, hamming_distance, unbounded_sensitivities)
;
```

    Unbounded sensitivities for mean {'school_year': 0.8333333333333335, 'absence_days': 2.833333333333333}
    Unbounded sensitivities for median {'school_year': 1.0, 'absence_days': 4.0}
    Unbounded sensitivities for count {'school_year': 1, 'absence_days': 1}
    Unbounded sensitivities for sum {'school_year': 4, 'absence_days': 10}
    Unbounded sensitivities for std {'school_year': 0.747219128924647, 'absence_days': 3.5276819911981905}
    Unbounded sensitivities for var {'school_year': 1.3055555555555554, 'absence_days': 15.972222222222221}
    Unbounded sensitivities for percentile 25 {'school_year': 1.25, 'absence_days': 2.75}
    Unbounded sensitivities for percentile 50 {'school_year': 1.0, 'absence_days': 4.0}
    Unbounded sensitivities for percentile 75 {'school_year': 1.25, 'absence_days': 4.25}
    Unbounded sensitivities for percentile 90 {'school_year': 1.7000000000000002, 'absence_days': 6.5}

    ''

Notice the obvious, the sensitivity for the median is the same as the sensitivity for the 50th percentile.

#### 4.3 & 4.4

```python
# Calculating prior knowledge. We assume a uniform prior
priors = prior_belief(df_school, df_school_release, columns)
priors
```

    {'school_year': [0.25, 0.25, 0.25, 0.25],
     'absence_days': [0.25, 0.25, 0.25, 0.25]}

###### Posterior 0.618 from the paper replicated - there is a typo on the expression below Table 3, the numerator should be "0.3062"

```python
# Let us calculate the posteriors
query_result = 2.20131
epsilon = 2
query_type = 'mean'
mean_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(df_school, df_school_release.shape[0], columns,  query_type, hamming_distance)
posteriors = posterior_belief(df_school, df_school_release, columns, query_type, query_result, mean_unbounded_global_sensitivities, epsilon)
posteriors
```

    {'school_year': array([0.33898835, 0.4003158 , 0.17987348, 0.08082237]),
     'absence_days': array([0.61802372, 0.15816999, 0.12500781, 0.09879847])}

```python
# Let us calculate the risk of disclosure
risk_disclosure(posteriors, columns)
```

    {'school_year': 0.40031580079087553, 'absence_days': 0.6180237199612966}

```python
# Let us calculate the confidence of the attacker
confidence_adversary = confidence(posteriors, priors, columns)
confidence_adversary
```

    {'school_year': 0.16917763372208006, 'absence_days': 0.15120152863171882}

### Bounded global sensitivity for different query types - prep for 5.1 (right above equation 8, delta_v = 3 for absence days and 1 for school year, for the mean query)

```python
# Calculate the sensitivity of different queries for the unbounded DP
columns = ['school_year', 'absence_days']
hamming_distance = 1
bounded_sensitivities = {}
calculate_bounded_sensitivities(df_school, df_school_release.shape[0], columns, hamming_distance, bounded_sensitivities)
;
```

    Bounded sensitivities for mean {'school_year': 1.0, 'absence_days': 3.0}
    Bounded sensitivities for median {'school_year': 1.0, 'absence_days': 1.0}
    Bounded sensitivities for count {'school_year': 0, 'absence_days': 0}
    Bounded sensitivities for sum {'school_year': 3, 'absence_days': 9}
    Bounded sensitivities for std {'school_year': 0.430722547996921, 'absence_days': 3.2111854102704642}
    Bounded sensitivities for var {'school_year': 0.8888888888888887, 'absence_days': 15.555555555555555}
    Bounded sensitivities for percentile 25 {'school_year': 1.0, 'absence_days': 1.0}
    Bounded sensitivities for percentile 50 {'school_year': 1.0, 'absence_days': 1.0}
    Bounded sensitivities for percentile 75 {'school_year': 1.0, 'absence_days': 4.0}
    Bounded sensitivities for percentile 90 {'school_year': 0.9999999999999996, 'absence_days': 5.799999999999999}

    ''

##### These calculations are not in the paper, but it is interesting to see what values we would get if we assume a bounded DP definition from the beginning.

```python
# Calculating prior knowledge. We assume a uniform prior
priors = prior_belief(df_school, df_school_release, columns)
priors
```

    {'school_year': [0.25, 0.25, 0.25, 0.25],
     'absence_days': [0.25, 0.25, 0.25, 0.25]}

```python
# Let us calculate the posteriors
query_result = 2.20131
epsilon = 2
query_type = 'mean'
mean_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(df_school, df_school_release.shape[0], columns,  query_type, hamming_distance)
posteriors = posterior_belief(df_school, df_school_release, columns, query_type, query_result, mean_bounded_global_sensitivities, epsilon)
posteriors
```

    {'school_year': array([0.32882419, 0.37769861, 0.19391693, 0.09956027]),
     'absence_days': array([0.59733148, 0.16489847, 0.13204038, 0.10572967])}

```python
# Let us calculate the risk of disclosure
risk_disclosure(posteriors, columns)
```

    {'school_year': 0.3776986100000907, 'absence_days': 0.5973314804471918}

```python
# Let us calculate the confidence of the attacker
confidence_adversary = confidence(posteriors, priors, columns)
confidence_adversary
```

    {'school_year': 0.15043972733368804, 'absence_days': 0.14427033183087148}

### Calculate upper bounds for posterior - 5.1

```python
# Experiment with an epsilon of 2
epsilon = 2
posterior_upper_bound = upper_bound_posterior(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, epsilon)
print('For epsilon {} the posterio is {}'.format(epsilon, posterior_upper_bound))
```

    For epsilon 2 the posterio is {'school_year': 0.7860684399476446, 'absence_days': 0.7347845417564229}

```python
# Epsilon of 0 should provie the highest privacy, but also zero utility, hence, the adversary has not updated his/her prior
# It has not learned anaything new. But if this individual querying is not malicious, then the utility of the query is 0
epsilon = 0
posterior_upper_bound = upper_bound_posterior(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, epsilon)
print('For epsilon {} the posterio is {}'.format(epsilon, posterior_upper_bound))
```

    For epsilon 0 the posterio is {'school_year': 0.25, 'absence_days': 0.25}

### Calculate upper bounds for epsilon given risk willing to take - 5.2 (9) - 0.3829

```python
# Let us calculate the upper bound with epsilon with a risk of 0.33 (there is a chance of 1/3 of letting
# the adversary know the true value)
risk = 1/3
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, risk)
epsilon_upper_bound
```

    {'school_year': 0.3378875900901369, 'absence_days': 0.38293926876882173}

### Calcualte a tighter risk bound - with 5.1 equation 3 (result 12) - 0.3292

```python
epsilon = 0.5
query_type = 'mean'
posterior_tighter_upper_bound = tighter_upper_bound_posterior(df_school, df_school_release, columns, query_type, mean_unbounded_global_sensitivities, epsilon)
print('For epsilon {} the posterior is {}'.format(epsilon, posterior_tighter_upper_bound))
```

    For epsilon 0.5 the posterior is {'school_year': 0.3291788293012836, 'absence_days': 0.3476971459619019}

### Plotting Fig 2

If you are wondering why are we using these bounds and not plotting the real value of the posterior:

This is because, in order to plot the real value of the posterior, you already needed to have a query result. For someone to output the query result, he/she had to already decide on a value of epsilon, and that is actually what they are aiming for - hence the name of the paper.

The purpose of finding these upper and tighter bounds of the posterior, is to find an optimal value for epsilon, i.e. by trying out with many values of epsilon and without knowing which query result you will get, you can plot the posterior upper and tight bounds and decide which epsilon to pick based on the risk you are willing to take.

###### Side note: In the paper they refer to the acceptable risk as the greek letters delta and as rho. Delta is never placed in an equation, that might somewhat confusing if you are reading the paper.  They use 1/3 as this value.

First we get the values for the bounds for plotting:

```python
precision = 0.01
limit_x = 5
epsilons = np.linspace(0, 5, num=int(limit_x/precision))

# Setting parameters
columns = ['school_year', 'absence_days']
query_type = 'mean'
# Initialize dicts with correspondng keys: https://stackoverflow.com/questions/11509721/how-do-i-initialize-a-dictionary-of-empty-lists-in-python
posterior_upper_bound = {k: [] for k in columns}
posterior_tighter_upper_bound = {k: [] for k in columns}

# Obtaining the values for the bounds
for epsilon in epsilons:

    temp_posterior_upper_bound = upper_bound_posterior(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, epsilon)
    temp_posterior_tighter_upper_bound = tighter_upper_bound_posterior(df_school, df_school_release, columns, query_type, mean_unbounded_global_sensitivities, epsilon)

    for column in columns:

        posterior_upper_bound[column].append(temp_posterior_upper_bound[column])
        posterior_tighter_upper_bound[column].append(temp_posterior_tighter_upper_bound[column])

```

```python
plt.figure(figsize=(15, 7))

risk = 1/3
# Calculate upper bound
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, risk)

for index, column in enumerate(columns):

    # Start the plot
    plot_index = int(str(1) + str(len(columns)) + str(index+1))
    plt.subplot(plot_index)

    # plot the upper bounds
    upper_bound, = plt.plot(epsilons, posterior_upper_bound[column], 'r',  label="Risk upper bound")
    tighter_bound, = plt.plot(epsilons, posterior_tighter_upper_bound[column], 'b',  label="Risk tighter bound")

    # Legends
    legend = plt.legend(handles=[upper_bound, tighter_bound], loc='lower right')
    ax = plt.gca().add_artist(legend)

    # axis labels and titles
    plt.xlabel('Epsilon')
    plt.ylabel('Risk disclosure probability')
    plt.ylim(0.2,1)
    plt.xlim(0,5)
    plt.title('{}) Domain {} = {}'.format(index+1, column, df_school[column].values))

plt.suptitle('Upper bounds of the risk disclosure (posterior probability) by varying domains')
plt.show()
```

Let us zoom in and check some values for risk and epsilon:

```python
plt.figure(figsize=(15, 7))

risk = 1/3
# Calculate upper bound
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, risk)

for index, column in enumerate(columns):

    # Start the plot
    plot_index = int(str(1) + str(len(columns)) + str(index+1))
    plt.subplot(plot_index)

    # plot the upper bounds
    upper_bound, = plt.plot(epsilons, posterior_upper_bound[column], 'r',  label="Risk upper bound")
    tighter_bound, = plt.plot(epsilons, posterior_tighter_upper_bound[column], 'b',  label="Risk tighter bound")

    # Plot the risk
    privacy_requirement, = plt.plot(epsilons, np.full(shape=len(epsilons), fill_value=risk), color='black',  label="Privacy requirement")

    # Plot the epsilon upper bound
    y_axis_points = np.linspace(0,1,2)
    plt.plot(np.full(shape=len(y_axis_points), fill_value=epsilon_upper_bound[column]), y_axis_points, 'r--')
    plt.plot(np.full(shape=len(y_axis_points), fill_value=0.5), y_axis_points, 'r--')

    # Legends
    legend = plt.legend(handles=[upper_bound, tighter_bound, privacy_requirement], loc='lower right')

    # axis labels and titles
    plt.xlabel('Epsilon')
    plt.ylabel('Risk disclosure probability')
    plt.title('{}) Domain {} = {}'.format(index+1, column, df_school[column].values))

    # Zoom
#     plt.ylim(0, 0.4)
#     plt.xlim(0, 0.6)

    # Additonal info
    print('Epsilon upper bound for risk {} in universe {} = {}'.format(round(risk, 2), column, epsilon_upper_bound[column]))

plt.suptitle('Upper bounds of the risk disclosure (posterior probability) by varying domains')
plt.show()
```

More specifically: (Looking at plot 1), left) The authors want us to notice that even though we have an epsilon bound on 0.3379 (left red dotted vertical line), we can find a value of epsilon (e.g., 0.5, right red dotted line) that still fulfills the privacy requirement (<1/3). (But note than an epsilon of 0.5 on the absence days would make the risk go above the set threshold of 1/3)
You cannot free epsilon on one side of the inequality from the tighter bound formula due to arithmetic constraints. Thus, you cannot find epsilon directly on the tighter bound curve formula, unlike with the upper bound curve. Therefore, they propose to use binary search starting at the max of the domain; in this case, it is with an epsilon of 5, which would approximately yield a 100% probability of the adversary being successful. To make it more optimal, if you already have visualized the curve, you can choose your start and end of the binary search with high precision.

### Binary Search - 5.2

We are going to perform binary search with the upper bound (not with the tight one9 to show that the binary search converges into the expected values:

```python
privacy_requirement = 1/3
query_type = 'mean'
posterior_bound_type = 'upper'
optimal_epsilons = binary_search_epsilon(df_school, df_school_release, columns, query_type, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, privacy_requirement, posterior_bound_type)
optimal_epsilons
```

    {'school_year': 0.3378877996541445, 'absence_days': 0.3829395062746971}

These are the sub-optimal values (as they are calculated with the upper bound) for epsilon to comply with a privacy requirement (disclosure probability) of 1/3

We thus show that our binary search works. See below the exact calculations with the upper bound (they are equal).

```python
risk = 1/3
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, risk)
epsilon_upper_bound

```

    {'school_year': 0.3378875900901369, 'absence_days': 0.38293926876882173}

```python
privacy_requirement = 1/3
query_type = 'mean'
posterior_bound_type = 'tight'
columns = ['school_year', 'absence_days']
optimal_epsilons = binary_search_epsilon(df_school, df_school_release, columns, query_type, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, privacy_requirement, posterior_bound_type)
print('Optimal epsilons:')
optimal_epsilons
```

    Optimal epsilons:

    {'school_year': 0.525149770057615, 'absence_days': 0.43171996782769506}

```python
tight_upper = {}
for column in columns:

    epsilon = optimal_epsilons[column]
    tight_upper[column] = tighter_upper_bound_posterior(df_school, df_school_release, columns, query_type, mean_unbounded_global_sensitivities, epsilon)
print('Posterior with optimal epsilons. Very close to the threhold of 1/3. So OK.')
print('You can also see, of course, how the optimal epsilon for one attribute does not yield a tight upper bound of 1/3 on the other')
tight_upper
```

    Posterior with optimal epsilons. Very close to the threhold of 1/3. So OK.
    You can also see, of course, how the optimal epsilon for one attribute does not yield a tight upper bound of 1/3 on the other

    {'school_year': {'school_year': 0.33333334702817974,
      'absence_days': 0.3530576956680186},
     'absence_days': {'school_year': 0.31796230321506674,
      'absence_days': 0.3333333017312349}}

This is computationally taxing due ti how the algorithm is written. We could save part of the maximum posterior operations so we do not need to run them again in every iteration. However, the purpose of this notebook is to get a deeper undersranding of the intricaces of the paper. Nonetheless, for these datasets sizes used for demonstrations, the algorithm runs smoothly.

Let us plot the output to verify that indeed these optimal epsilons correspond to the tighter upper bound curves.

```python
plt.figure(figsize=(15, 7))

risk = 1/3
# Calculate upper bound
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, mean_bounded_global_sensitivities, mean_unbounded_global_sensitivities, risk)

for index, column in enumerate(columns):

    # Start the plot
    plot_index = int(str(1) + str(len(columns)) + str(index+1))
    plt.subplot(plot_index)

    # plot the upper bounds
    upper_bound, = plt.plot(epsilons, posterior_upper_bound[column], 'r',  label="Risk upper bound")
    tighter_bound, = plt.plot(epsilons, posterior_tighter_upper_bound[column], 'b',  label="Risk tighter bound")

    # Plot the risk
    privacy_requirement, = plt.plot(epsilons, np.full(shape=len(epsilons), fill_value=risk), color='black',  label="Privacy requirement")

    # Plot the epsilon upper bound
    y_axis_points = np.linspace(0,1,2)
    plt.plot(np.full(shape=len(y_axis_points), fill_value=epsilon_upper_bound[column]), y_axis_points, 'r--')
    plt.plot(np.full(shape=len(y_axis_points), fill_value=optimal_epsilons[column]), y_axis_points, 'b--')

    # Legends
    legend = plt.legend(handles=[upper_bound, tighter_bound, privacy_requirement], loc='lower left')

    # axis labels and titles
    plt.xlabel('Epsilon')
    plt.ylabel('Risk disclosure probability')
    plt.title('{}) Domain {} = {}'.format(index+1, column, df_school[column].values))

    # Zoom
    plt.ylim(0, 0.4)
    plt.xlim(0, 0.6)

    # Additonal info
    print('Plot:', index, column)
    print('Epsilon upper bound for risk {} in universe {} = {} in dash red'.format(round(risk, 2), column, epsilon_upper_bound[column]))
    print('Epsilon tight upper bound for risk {} in universe {} = {} in dash blue'.format(round(risk, 2), column, optimal_epsilons[column]))
    print('\n')

plt.suptitle('Upper bounds of the risk disclosure (posterior probability) by varying domains')
plt.show()
```

##### The optimal epsilons (in dash blue) fit pretty well, they maximize utility while preserving privacy.

### MEDIAN - the last part of the paper (6), exemplifies this process with the median

```python
query_type = 'median'
median_unbounded_global_sensitivities = unbounded_empirical_global_L1_sensitivity(df_school, df_school_release.shape[0], columns,  query_type, hamming_distance)
median_bounded_global_sensitivities = bounded_empirical_global_L1_sensitivity(df_school, df_school_release.shape[0], columns,  query_type, hamming_distance)

print('unbounded sensitivity', median_unbounded_global_sensitivities)
print('bounded sensitivity', median_bounded_global_sensitivities)
```

    unbounded sensitivity {'school_year': 1.0, 'absence_days': 4.0}
    bounded sensitivity {'school_year': 1.0, 'absence_days': 1.0}

```python
# Calculating prior knowledge. We assume a uniform prior
priors = prior_belief(df_school, df_school_release, columns)
priors
```

    {'school_year': [0.25, 0.25, 0.25, 0.25],
     'absence_days': [0.25, 0.25, 0.25, 0.25]}

```python
# Let us calculate the posteriors
query_result = 2.20131
epsilon = 2
query_type = 'median'
posteriors = posterior_belief(df_school, df_school_release, columns, query_type, query_result, mean_unbounded_global_sensitivities, epsilon)
posteriors
```

    {'school_year': array([0.33898835, 0.4003158 , 0.17987348, 0.08082237]),
     'absence_days': array([0.61802372, 0.15816999, 0.12500781, 0.09879847])}

```python
# Let us calculate the risk of disclosure
risk_disclosure(posteriors, columns)
```

    {'school_year': 0.40031580079087553, 'absence_days': 0.6180237199612966}

```python
# Let us calculate the confidence of the attacker
confidence_adversary = confidence(posteriors, priors, columns)
confidence_adversary
```

    {'school_year': 0.16917763372208006, 'absence_days': 0.15120152863171882}

##### Calculate the upper bounds of the posterior

```python
epsilon = 2
posterior_upper_bound = upper_bound_posterior(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, epsilon)
print('For epsilon {} the posterio is {}'.format(epsilon, posterior_upper_bound))
```

    For epsilon 2 the posterio is {'school_year': 0.7112345942275939, 'absence_days': 0.35466124439244334}

```python
epsilon = 0
posterior_upper_bound = upper_bound_posterior(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, epsilon)
print('For epsilon {} the posterio is {}'.format(epsilon, posterior_upper_bound))
```

    For epsilon 0 the posterio is {'school_year': 0.25, 'absence_days': 0.25}

##### Calculate upper bounds for epsilon given risk willing to take - 6 result 18 = 1.6219

```python
# Let us calculate the upper bound with epsilon with a risk of 0.33 (there is a chance of 1/3 of letting
# the adversary know the true value)
risk = 1/3
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, risk)
epsilon_upper_bound
```

    {'school_year': 0.4054651081081642, 'absence_days': 1.6218604324326569}

##### Calcualte a tighter risk bound with a given epsilon

```python
epsilon = 0.5
query_type = 'median'
posterior_tighter_upper_bound = tighter_upper_bound_posterior(df_school, df_school_release, columns, query_type, median_unbounded_global_sensitivities, epsilon)
print('For epsilon {} the posterio is {}'.format(epsilon, posterior_tighter_upper_bound))
```

    For epsilon 0.5 the posterio is {'school_year': 0.31122966560092735, 'absence_days': 0.26560468668687814}

##### Plotting

```python
precision = 0.01
limit_x = 5
epsilons = np.linspace(0, 5, num=int(limit_x/precision))

# Setting parameters
columns = ['school_year', 'absence_days']
query_type = 'median'
# Initialize dicts with correspondng keys: https://stackoverflow.com/questions/11509721/how-do-i-initialize-a-dictionary-of-empty-lists-in-python
posterior_upper_bound = {k: [] for k in columns}
posterior_tighter_upper_bound = {k: [] for k in columns}

# Obtaining the values for the bounds
for epsilon in epsilons:

    temp_posterior_upper_bound = upper_bound_posterior(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, epsilon)
    temp_posterior_tighter_upper_bound = tighter_upper_bound_posterior(df_school, df_school_release, columns, query_type, median_unbounded_global_sensitivities, epsilon)

    for column in columns:

        posterior_upper_bound[column].append(temp_posterior_upper_bound[column])
        posterior_tighter_upper_bound[column].append(temp_posterior_tighter_upper_bound[column])
```

```python
plt.figure(figsize=(15, 7))

risk = 1/3
# Calculate upper bound
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, risk)

for index, column in enumerate(columns):

    # Start the plot
    plot_index = int(str(1) + str(len(columns)) + str(index+1))
    plt.subplot(plot_index)

    # plot the upper bounds
    upper_bound, = plt.plot(epsilons, posterior_upper_bound[column], 'r',  label="Risk upper bound")
    tighter_bound, = plt.plot(epsilons, posterior_tighter_upper_bound[column], 'b',  label="Risk tighter bound")

    # Legends
    legend = plt.legend(handles=[upper_bound, tighter_bound], loc='lower right')
    ax = plt.gca().add_artist(legend)

    # axis labels and titles
    plt.xlabel('Epsilon')
    plt.ylabel('Risk disclosure probability')
    plt.ylim(0.2,1)
    plt.xlim(0,5)
    plt.title('{}) Domain {} = {}'.format(index+1, column, df_school[column].values))

plt.suptitle('Upper bounds of the risk disclosure (posterior probability) by varying domains')
plt.show()
```

```python
plt.figure(figsize=(15, 7))

risk = 1/3
# Calculate upper bound
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, risk)

for index, column in enumerate(columns):

    # Start the plot
    plot_index = int(str(1) + str(len(columns)) + str(index+1))
    plt.subplot(plot_index)

    # plot the upper bounds
    upper_bound, = plt.plot(epsilons, posterior_upper_bound[column], 'r',  label="Risk upper bound")
    tighter_bound, = plt.plot(epsilons, posterior_tighter_upper_bound[column], 'b',  label="Risk tighter bound")

    # Plot the risk
    privacy_requirement, = plt.plot(epsilons, np.full(shape=len(epsilons), fill_value=risk), color='black',  label="Privacy requirement")

    # Plot the epsilon upper bound
    y_axis_points = np.linspace(0,1,2)
    plt.plot(np.full(shape=len(y_axis_points), fill_value=epsilon_upper_bound[column]), y_axis_points, 'r--')

    # Legends
    legend = plt.legend(handles=[upper_bound, tighter_bound, privacy_requirement], loc='lower right')

    # axis labels and titles
    plt.xlabel('Epsilon')
    plt.ylabel('Risk disclosure probability')
    plt.title('{}) Domain {} = {}'.format(index+1, column, df_school[column].values))

    # Zoom
#     plt.ylim(0, 0.4)
#     plt.xlim(0, 0.6)

    # Additonal info
    print('Epsilon upper bound for risk {} in universe {} = {}'.format(round(risk, 2), column, epsilon_upper_bound[column]))

plt.suptitle('Upper bounds of the risk disclosure (posterior probability) by varying domains')
plt.show()
```

### Binary Search - 5.2

We are going to perform binary search with the upper bound (not with the tight one9 to show that the binary search converges into the expected values:

```python
privacy_requirement = 1/3
query_type = 'median'
posterior_bound_type = 'upper'
optimal_epsilons = binary_search_epsilon(df_school, df_school_release, columns, query_type, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, privacy_requirement, posterior_bound_type)
optimal_epsilons
```

    {'school_year': 0.40546535958497326, 'absence_days': 1.621861438339893}

These are the sub-optimal values (as they are calculated with the upper bound) for epsilon to comply with a privacy requirement (disclosure probability) of 1/3

We thus show that our binary search works. See below the exact calculations with the upper bound (they are equal).

```python
risk = 1/3
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, risk)
epsilon_upper_bound

```

    {'school_year': 0.4054651081081642, 'absence_days': 1.6218604324326569}

##### Result 6 after equation 21 - 2.773

```python
privacy_requirement = 1/3
query_type = 'median'
posterior_bound_type = 'tight'
columns = ['school_year', 'absence_days']
optimal_epsilons = binary_search_epsilon(df_school, df_school_release, columns, query_type, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, privacy_requirement, posterior_bound_type)
print('Optimal epsilons:')
optimal_epsilons
```

    Optimal epsilons:

    {'school_year': 0.693147280402229, 'absence_days': 2.772589121608916}

```python
query_type = 'median'
tight_upper = {}
for column in columns:

    epsilon = optimal_epsilons[column]
    tight_upper[column] = tighter_upper_bound_posterior(df_school, df_school_release, columns, query_type, median_unbounded_global_sensitivities, epsilon)

print('Posterior with optimal epsilons. Very close to the threhold of 1/3. So OK.')
print('You can also see how the optimal epsilon for one attribute does not yield a tight upper bound of 1/3 on the other')
tight_upper
```

    Posterior with optimal epsilons. Very close to the threhold of 1/3. So OK.
    You can also see how the optimal epsilon for one attribute does not yield a tight upper bound of 1/3 on the other

    {'school_year': {'school_year': 0.3333333444269202,
      'absence_days': 0.27160681152823796},
     'absence_days': {'school_year': 0.47058824634931673,
      'absence_days': 0.3333333444269202}}

This is computationally taxing due ti how the algorithm is written. We could save part of the maximum posterior operations so we do not need to run them again in every iteration. However, the purpose of this notebook is to get a deeper undersranding of the intricaces of the paper. Nonetheless, for these datasets sizes used for demonstrations, the algorithm runs smoothly.

Let us plot the output to verify that indeed these optimal epsilons correspond to the tighter upper bound curves.

```python
plt.figure(figsize=(15, 7))

risk = 1/3
# Calculate upper bound
epsilon_upper_bound = upper_bound_epsilon(df_school, columns, median_bounded_global_sensitivities, median_unbounded_global_sensitivities, risk)

for index, column in enumerate(columns):

    # Start the plot
    plot_index = int(str(1) + str(len(columns)) + str(index+1))
    plt.subplot(plot_index)

    # plot the upper bounds
    upper_bound, = plt.plot(epsilons, posterior_upper_bound[column], 'r',  label="Risk upper bound")
    tighter_bound, = plt.plot(epsilons, posterior_tighter_upper_bound[column], 'b',  label="Risk tighter bound")
    # Plot the risk
    privacy_requirement, = plt.plot(epsilons, np.full(shape=len(epsilons), fill_value=risk), color='black',  label="Privacy requirement")

    # Plot the epsilon upper bound
    y_axis_points = np.linspace(0,1,2)
    plt.plot(np.full(shape=len(y_axis_points), fill_value=epsilon_upper_bound[column]), y_axis_points, 'r--')
    plt.plot(np.full(shape=len(y_axis_points), fill_value=optimal_epsilons[column]), y_axis_points, 'b--')

    # Legends
    legend = plt.legend(handles=[upper_bound, tighter_bound, privacy_requirement], loc='lower right')

    # axis labels and titles
    plt.xlabel('Epsilon')
    plt.ylabel('Risk disclosure probability')
    plt.title('{}) Domain {} = {}'.format(index+1, column, df_school[column].values))

    # Additonal info
    print('Plot:', index, column)
    print('Epsilon upper bound for risk {} in universe {} = {} in dash red'.format(round(risk, 2), column, epsilon_upper_bound[column]))
    print('Epsilon tight upper bound for risk {} in universe {} = {} in dash blue'.format(round(risk, 2), column, optimal_epsilons[column]))
    print('\n')

plt.suptitle('Upper bounds of the risk disclosure (posterior probability) by varying domains')
plt.show()
```

The optimal epsilons (in dash blue) fit pretty well, they maximize utility while preserving privacy.


# 10 Ordinary Differential Equations (ODEs)

[ODE](http://mathworld.wolfram.com/OrdinaryDifferentialEquation.html)s describe many phenomena in physics. They describe the changes of a **dependent variable** $y(t)$ as a function of a **single independent variable** (e.g. $t$ or $x$).

An ODE of **order** $n$

$$
F(t, y^{(0)}, y^{(1)}, ..., y^{(n)}) = 0
$$

contains derivatives $y^{(k)}(t) \equiv y^{(k)} \equiv \frac{d^{k}y(t)}{dt^{k}}$ up to the $n$-th derivative (and $y^{(0)} \equiv y$).

### Initial and boundary conditions

* $n$ **initial conditions** are needed to *uniquely determine* the solution of a $n$-th order ODE, e.g, initial position and velocities.

* **Boundary conditions** (values of solution on domain boundries) can additionaly restrict solutions but the resulting *eigenvalue problems* are more difficult, e.g, wavefunction goes towards 0 for $\pm\infty$.

### Linear ODEs
A **linear** ODE contains no higher powers than 1 of any of the $y^{(k)}$.

*Superposition principle*: Linear combinations of solutions are also solutions.

#### Example: First order linear ODE

\begin{align}
\frac{dy}{dt} &= f(t)y + g(t)\\
y^{(1)} &= f(t)y + g(t)\\
% y^{(1)} - f(t)y - g(t) &= 0
\end{align}

##### Radioactive decay
$$
\frac{dN}{dt} = -k N
$$

### Non-linear ODEs
**Non-linear** ODEs can contain any powers in the dependent variable and its derivatives.

No superposition of solutions. Often impossible to solve analytically.

#### Example: Second order (general) ODE
\begin{gather}
\frac{d^2 y}{dt^2} + \lambda(t) \frac{dy}{dt} = f\left(t, y, \frac{dy}{dt}\right)\\
\end{gather}

##### Newton's equations of motion
$$
m\frac{d^2 x}{dt^2} = F(x) + F_\text{ext}(x, t) \quad \text{with}
\quad F(x) = -\frac{dU}{dx}
$$
(Force is often derived from a potential energy $U(x)$ and may contain non-linear terms such as $x^{-2}$ or $x^3$.)

## Partial differential equations (PDEs)
* more than one independent variable (e.g. $x$ and $t$)
* partial derivatives
* much more difficult than ODEs

#### Example: Schrödinger equation (Quantum Mechanics)
$$
i\hbar \frac{\partial\psi(\mathbf{x}, t)}{\partial t} = -\frac{\hbar^2}{2m}
   \left(\frac{\partial^2 \psi}{\partial x^2} +
         \frac{\partial^2 \psi}{\partial y^2} +
         \frac{\partial^2 \psi}{\partial z^2}
   \right) + V(\mathbf{x})\, \psi(\mathbf{x}, t)
$$

## Harmonic and anharmonic oscillator
* particle with mass $m$ connected to a spring
* spring described by a harmonic potential or anharmonic ones in the displacements from equilibrium $x$

\begin{align}
U_1(x) &= \frac{1}{2} k x^2, \quad k=1\\
U_2(x) &= \frac{1}{2} k x^2 \left(1 - \frac{2}{3}\alpha x\right), \quad k=1,\ \alpha=\frac{1}{2}\\
U_3(x) &= \frac{1}{p} k x^p, \quad k=1,\ p=6
\end{align}

1. What do these potentials look like? Sketch or plot.
2. Calculate the forces.

#### Potentials

```python
import numpy as np

def U1(x, k=1):
    return 0.5 * k * x*x

def U2(x, k=1, alpha=0.5):
    return 0.5 * k * x*x * (1 - (2/3)*alpha*x)

def U3(x, k=1, p=6):
    return (k/p) * np.power(x, p)
```

```python
import matplotlib
import matplotlib.pyplot as plt
matplotlib.style.use('seaborn-talk')
%matplotlib inline
```

```python
X = np.linspace(-3, 3, 100)
ax = plt.subplot(1,1,1)
ax.plot(X, U1(X), label=r"$U_1$")
ax.plot(X, U2(X), label=r"$U_2$")
ax.plot(X, U3(X), label=r"$U_3$")
ax.set_ylim(-0.5, 10)
ax.legend(loc="upper center");
```

#### Forces

\begin{align}
F_1(x) &= -kx\\
F_2(x) &= -kx(1 + \alpha x)\\
F_3(x) &= -k x^{p-1}
\end{align}

## ODE Algorithms
Basic idea:

1. Start with initial conditions, $y_0 \equiv y(t=0)$
2. Use $\frac{dy}{dt} = f(t, y)$ (the RHS!) to advance solution a small step $h$ forward in time: $y(t=h) \equiv y_1$
3. Repeat with $y_1$ to obtain $y_2 \equiv y(t=2h)$... and for all future values of $t$.

Possible issues
* small differences: subtractive cancelation and round-off error accumulation
* extrapolation: numerical "solution" can deviate wildly from exact
* possibly need adaptive $h$

### Euler's rule
Simple: forward difference

\begin{align}
f(t, y) = \frac{dy(t)}{dt} &\approx \frac{y(t_{n+1}) - y(t_n)}{h}\\
y_{n+1} &\approx y_n + h f(t_n, y_n) \quad \text{with} \quad y_n := y(t_n)
\end{align}

Error will be $\mathcal{O}(h^2)$ (bad!).

Also: what if we have a second order ODE ?!?! We only used $dy/dt$.

### Convert 2nd order ODE to 2 coupled _First Order_ ODEs

### Convert 2nd order ODE to 2 coupled 1st order ODEs
The 2nd order ODE is
$$
\frac{d^2 y}{dt^2} = f(t, y)
$$

Introduce "dummy" dependent variables $y_i$ with $y_0 \equiv y$ and

\begin{alignat}{1}
\frac{dy}{dt} &= \frac{dy_0}{dt} &=   y_1\\
\frac{d^2y}{dt^2} &= \frac{dy_1}{dt} &= {} f(t, y_0).
\end{alignat}

The first equation defines the velocity $y_1 = v$ and the second one is the original ODE.

### $n$-th oder ODE to $n$ coupled 1st order ODEs

The $n$-th order ODE is
$$
\frac{d^n y}{dt^n} = f(t, y, \frac{d y}{dt}, \frac{d^2 y}{dt^2}, \dots, \frac{d^{n-1} y}{dt^{n-1}})
$$

Introduce "dummy" dependent variables $y^{(i)}$ with $y^{(0)} \equiv y$ and

\begin{align}
\frac{dy^{(0)}}{dt} &=  y^{(1)}\\
\frac{dy^{(1)}}{dt} &=  y^{(2)}\\
\dots & \\
\frac{dy^{(n-1)}}{dt} &=  f(t, y^{(0)}, y^{(1)}, y^{(2)}, \dots, y^{(n-1)}).
\end{align}

### General standard (dynamic) form of ODEs
1 ODE of *any order* $n$ $\rightarrow$ $n$ coupled simultaneous first-order ODEs in $n$ unknowns  $y^{(0)}, \dots, y^{(n-1)}$:

\begin{align}
\frac{dy^{(0)}}{dt} &= f^{(0)}(t, y^{(0)}, \dots, y^{(n-1)})\\
\frac{dy^{(1)}}{dt} &= f^{(1)}(t, y^{(0)}, \dots, y^{(n-1)})\\
\vdots &  \\
\frac{dy^{(n-1)}}{dt} &= f^{(n-1)}(t, y^{(0)}, \dots, y^{(n-1)})\\
\end{align}

In $n$-dimensional vector notation:

\begin{align}
\frac{d\mathbf{y}(t)}{dt} &= \mathbf{f}(t, \mathbf{y})\\
\mathbf{y} &= \left(\begin{array}{c}
  y^{(0)}(t) \\
  y^{(1)}(t) \\
  \vdots \\
  y^{(n-1)}(t)
  \end{array}\right),
\quad
\mathbf{f} = \left(\begin{array}{c}
  f^{(0)}(t, \mathbf{y}) \\
  f^{(1)}(t, \mathbf{y}) \\
  \vdots \\
  f^{(n-1)}(t, \mathbf{y})
  \end{array}\right)
\end{align}

#### Example: Convert Newton's EOMs to standard form

$$
\frac{d^2 x}{dt^2} = m^{-1} F\Big(t, x, \frac{dx}{dt}\Big)
$$

RHS may *not contain any explicit derivatives* but components of $\mathbf{y}$ can represent derivatives.
* position $x$ as first dependent variable $y^{(0)}$ (as usual).
* velocity $dx/dt$ as second dependent variable $y^{(1)}$

\begin{align}
y^{(0)}(t) &:= x(t)\\
y^{(1)}(t) &:= \frac{dx}{dt} = \frac{dy^{(0)}}{dt}
\end{align}

One 2nd order ODE

$$
\frac{d^2 x}{dt^2} = m^{-1} F\Big(t, x, \frac{dx}{dt}\Big)
$$

to two simultaneous 1st order ODEs:

\begin{align}
\frac{dy^{(0)}}{dt} &= y^{(1)}(t)\\
\frac{dy^{(1)}}{dt} &= m^{-1} F\Big(t, y^{(0)}, y^{(1)}\Big)
\end{align}

\begin{align}
\frac{d\mathbf{y}(t)}{dt} &= \mathbf{f}(t, \mathbf{y})\\
\mathbf{y} &= \left(\begin{array}{c}
     y^{(0)} \\
     y^{(1)}
  \end{array}\right) =
  \left(\begin{array}{c}
     x(t) \\
     \frac{dx}{dt}
  \end{array}\right),\\
\mathbf{f} &= \left(\begin{array}{c}
    y^{(1)}(t) \\
    m^{-1} F\Big(t, y^{(0)}, y^{(1)}\Big)
  \end{array}\right) =
  \left(\begin{array}{c}
    \frac{dx}{dt} \\
    m^{-1} F\Big(t, x(t), \frac{dx}{dt}\Big)
  \end{array}\right)
\end{align}

#### Example:  1D harmonic oscillator in standard form
With $F_1 = -k x$:
$$
\frac{d^2 x}{dt^2} = -m^{-1}k x
$$
convert to

\begin{align}
\frac{dy^{(0)}}{dt} &= y^{(1)}(t) \\
\frac{dy^{(1)}}{dt} &= -m^{-1}k y^{(0)}
\end{align}

Force (or derivative) function $\mathbf{f}$ and initial conditions:

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}(0) &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=  -m^{-1} k y^{(0)},
    &\quad y^{(1)}(0) &= v_0.
\end{alignat}

### Euler's rule (standard form)
Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{\Delta t}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + \Delta t \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

## Problem: Numerically integrate the 1D harmonic oscillator with Euler

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}(0) &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}(0) &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

### Explicit implementation:
* Note how in `f_harmonic` we are constructing the force vector of the standard ODE representation
* `y` is the vector of dependents in the standard representation
* We pre-allocate the array for `y` and then assign to individual elements with the
  ```python
  y[:] = ...
  ```
  notation, which has higher performance than creating the array anew every time.

```python
import numpy as np

def F1(x, k=1):
    """Harmonic force"""
    return -k*x

def f_harmonic(t, y, k=1, m=1):
    """Force vector in standard ODE form (n=2)"""
    return np.array([y[1], F1(y[0], k=k)/m])

t_max = 100
h = 0.01
Nsteps = int(t_max/h)
t_range = h * np.arange(Nsteps)
x = np.empty_like(t_range)

y = np.zeros(2)

# initial conditions
x0, v0 = 0.0, 1.0
y[:] = x0, v0

for i, t in enumerate(t_range):
    # store position that corresponds to time t_i
    x[i] = y[0]
    # Euler integrator
    y[:] = y + h * f_harmonic(t, y)
```

Plot the position $x(t)$ (which is $y_0$) against time:

```python
plt.plot(t_range, x)
plt.plot(t_range, np.sin(t_range), color="red")
```

### Modular solution with functions
We can make the Euler integrator a function, which makes the code more readable and modular and we can make the whole integration a function, too. This will allow us to easily run the integration with different initial values or `h` steps.

```python
import numpy as np

def F1(x, k=1):
    """Harmonic force"""
    return -k*x

def f_harmonic(t, y, k=1, m=1):
    """Force vector in standard ODE form (n=2)"""
    return np.array([y[1], F1(y[0], k=k)/m])

def euler(y, f, t, h):
    """Euler integrator.

    Returns new y at t+h.
    """
    return y + h * f(t, y)

def integrate(x0=0, v0=1, t_max=100, h=0.001):
    """Integrate the harmonic oscillator with force F1.

    Note that the spring constant k and particle mass m are currently
    pre-defined.

    Arguments
    ---------
    x0 : float
       initial position
    v0 : float
       initial velocity
    t_max : float
       time to integrate out to
    h : float, default 0.001
       integration time step

    Returns
    -------
    Tuple ``(t, x)`` with times and positions.
    """

    Nsteps = t_max/h
    t_range = h * np.arange(Nsteps)
    x = np.empty_like(t_range)
    y = np.zeros(2)
    # initial conditions
    y[:] = x0, v0

    for i, t in enumerate(t_range):
        # store position that corresponds to time t_i
        x[i] = y[0]
        # Euler integrator
        y[:] = euler(y, f_harmonic, t, h)

    return t_range, x
```

Plot the position as a function of time, $x(t)$.

```python
t, x = integrate(t_max=300, h=0.001)
plt.plot(t, x)
plt.plot(t, np.sin(t), color="red")
```

Analytical solution:
$$
\omega = \sqrt{\frac{k}{m}}
$$
and
$$
x(t) = A \sin(\omega t)
$$
here with $A=1$ and $\omega = 1$:

```python
plt.plot(t, x)
plt.plot(t, np.sin(t), color="red")
```

Note the increase in amplitude. Explore if smaller $h$ fixes this obvious problem.

```python
t, x = integrate(h=0.001)
plt.plot(t, x)
plt.plot(t, np.sin(t), color="red")
```

Smaller $h$ improves the integration (but Euler is still a bad algorithm... just run out for longer, i.e., higher `t_max`.)

### Note on Euler's global error

How does the global error after time $t$ behave, i.e., what is its dependency on step size $h$?

* Per-step (**local**) error is $\mathcal{O}(h^2)$.
* For time $t = N h$ we have $N \propto 1/h$ steps.

We assume that we linearly accumulate local error for the **global error** for $N$ steps:

$$
\mathcal{O}(h^2) \times h^{-1} \propto \mathcal{O}(h)
$$

Thus, halving $h$ should halve the global error.

## Optional: Semi-implicit Euler

We can apply a simple trick to turn standard (forward) Euler into a _much_ better algorithm but this only works nicely when we solve Hamilton's equations of motion (Newton's equation without velocity-dependent potentials) and it also does not work nicely with the standard form.

In the following, $x$ is the position, $v = \dot{x} = \frac{dx}{dt}$ the velocity, and $f = F/m$ is the acceleration (but keeping the suggestive letter "f" for the force). Note that the force only depends on positions, $f(x) = F(x)/m$.

### Standard Euler for Hamiltonian dynamics

\begin{align}
x_{n+1} &= x_n + h v_n\\
v_{n+1} &= v_n + h f(x_n)
\end{align}

This is the same algorithm that we discussed in a more general form above.

### Semi-implicit Euler for Hamiltonian dynamics

The trick is to use the _updated_ positions $x_{n+1}$ for the velocity $v_{n+1}$ update, which gives the [semi-implicit Euler](https://en.wikipedia.org/wiki/Semi-implicit_Euler_method) method:

\begin{align}
x_{n+1} &= x_n + h v_n\\
v_{n+1} &= v_n + h f(x_{n+1})
\end{align}

This algorithm _does_ conserve energy over the long term because it is a symplectic algorithm (it maintains the deeper mathematical structures and invariants of Hamiltonian dynamics) although its error is still the same as for standard Euler.

```python

```


## Question 1.a
### Assumption:
The dataset for each of the class for the MRI images can be considered as a Gaussian distribution mainly because of small variations in the class distribution. In addition, assuming that each image has a size of 256x256 pixels.

### Prior Information:
In total there are 6 different classes including the CN tower images. The different classes of images are Head images, Neck images, Spine images, abdomen images, pelvis images and CN tower images in the dataset.

### Approach 1:
1. Detecting the Outliers in Preprocessing step: considering the CN tower images as an outlier in the complete human body part images we can try to detect these outliers by using different outliers detecting techniques such as Z-score, IQR and Isolation Forest.

### Approach 2:
If Approach 1 fails, we then use Soft Clustering method to cluster different classes in the dataset as Hard Clustering methods don't have uncertainity measures or a probability that tells us how much a data point is associated with a specific cluster. In our case we use Guassian Mixture Models (GMMs) for this problem. As the image dataset is a 256x256 we first flatten the matrix to a vector in the preprocessing step. In the second step we perform dimensionality reduction using Principle Component Analysis (PCA) to reduce the dimensions of the dataset. The main reason this step is required is due the compexity of GMM of $\mathcal{O}(NKD^{3})$ as D = 256x256 the runtime complexity will be higher so to make it effecient we perform PCA by projecting the data on top principle components (can be choosen by considering the variance for the top Eigenvalues and Eigenvectors) to reduce the dimesion of the dataset and then pass PCA transformed dataset to the GMMs.

### Gaussian Mixture Models (GMMs):
A Gaussian Mixture Model consists of several Gaussians which is given by $ k \quad \epsilon \quad \{1,2,..,K\}$, where K is the number of clusters in the dataset. From the prior information the total number of clusters is 6 so in our case K = 6.

For each of the Gaussian cluster k, the mixture has some important parameters as given below:

1. Mean ($\mu$): The mean ($\mu$) defines the center of the Cluster.
2. Covariance ($\Sigma$): The covariance defines the width of the distribution.
3. Mixing Probability ($\Pi$): It defines how big or small the Gaussian Function is.

Maximization algorithm can be used to obtain the optimal values of these parameters so that each Gaussian fits the data points belonging to each of the cluster. In general the Gaussian density function is given by

\begin{equation}
    \tag{1}
    N(x| \mu, \Sigma) = \frac{1}{(2\pi)^{\frac{D}{2}} |\Sigma|^{\frac{1}{2}}} exp \left(\frac{-1}{2} (x-\mu)^{T} \Sigma ^{-1} (x-\mu)\right)
\end{equation}

In equation 1, x is the datapoint, D is the number of dimensions of each data point. Here $\mu$ is the mean and $\Sigma$ is the covariance and N is the number of datapoints. As there are several Gaussians we need to find the optimal parameters for the whole mixture which can be modelled by considering that we want to know the probability that a given data point $x_{n}$ comes form Gaussian $k$ which can be expressed as below:

\begin{equation}
    \tag{2}
    p(z_{nk} = 1 | x_{n})

\end{equation}

This effectively tells us for a given data point x, what is the probability it came from Gaussian $k$. In equation 2 $z$ is the latent variable which takes the value of 1 when x comes from Gaussian $k$ else it takes 0 as the value. This variable is usefull in determining the Gaussian mixture parameters by calculating its probabiltiy of occurrence. Now let $\pi_{k} = p(z_{k} = 1) $ be the overall probability that a given point comes from Gaussian $k$ and for k different Gaussian let $z = \{ z_{1},...,z_{K} \}$. Now

\begin{equation}
    \tag{3}
    p(x_{n}, z) = \Pi_{k=1}^{K} N(x_{n} | \mu_{k}, \Sigma_{k})^{z_{k}}

\end{equation}

solving this further after applying Bayes rule will yeild us

\begin{equation}
    \tag{4}
    p(z_{k} = 1 | x_{n}) = \frac{\pi_{k} N(x_{n} | \mu_{k},\Sigma_{k})}{ \Sigma_{j=1}^{K} \pi_{j} N(x_{n} | \mu_{j}, \Sigma_{j})} = \gamma(z_{nk})

\end{equation}

using Expectation Maximization(EM) algorithm we can obatin the optimal parameters $\mu_{k}^{*}$ and $\Sigma_{k}^{*}$ as below:

\begin{equation}
    \tag{5}
    \mu_{k}^{*} = \frac{\Sigma_{n=1}^{N} \gamma(z_{nk}) x_{n}}{\Sigma_{n=1}{N}         \gamma(z_{nk})}

\end{equation}

\begin{equation}
    \tag{6}

    \Sigma_{k}^{*} = \frac{\Sigma_{n=1}^{N} \gamma(z_{nk})(x_{n}-\mu_{k})(x_{n} -      \mu_{k})^{T}}{\Sigma_{n=1}^{N} \gamma(z_{nk})}

\end{equation}

using EM algorithm we can converge to the likelihood value of the above two parameter which will effectively determine the cluster class for the data points and then we can identify the CN tower image from the rest of the MRI images.

## Question 1.c
1. The computational complexity for the Guassian Mixture Model is $\mathcal{O}(NKD^{3})$ where N is the total number of samples, K is the number of clusters and D is the dimension of the datapoint which will effectively reduce to $\mathcal{O}(N)$ for the case when $N >> K$ and $N>>D$. In addition, GMMs can be implemented to perform parallel computation so we can use GPUs to perform the computation as the size of the dataset is very large. This will reduce the computation time thereby giving results quickly.

2. We can use the above method for differentiating the MRI vs non-MRI images. If we have access to the labels for each of the dataset we can use different AI/ML models to train the network to identify the image otherwise we have to use unsupervised learning algorithms to differentiate between the two classes.