<p align="center">

</p>

## Interactive Spurious Correlations Demonstration

### Too Few Samples May Result in Spurious Correlations

* in class I bring in 3 red balls, 2 green balls and my cowboy hat, yes I have one, recall I was a farmhand in Alberta, Canada

* then I have students volunteer, one holds the hat, one draws balls with replacement and one records the results on the board

* through multiple bootstrap sample sets we demonstrate the use of bootstrap to calculate uncertainty in the proportion from the sample itself through sampling with replacement

* with this workflow we all provide an interactive plot demonstration with matplotlib and ipywidget packages to demonstrate this virtually

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Source of Spurious Correlations

Let's explore the source of spurious correlations:

* too few sample data

* this issue can be exagerated when sampling from skewed distributions with possibility for extreme values

What's the issue?

* anomalously large absolute correlations between independent features

We 'data mine' relationships that don't exist!  Great examples are available at the [Spurious Correlations](https://www.tylervigen.com/spurious-correlations) website.

#### The Correlation Coefficient

Pearson’s Product‐Moment Correlation Coefficient
* Provides a measure of the degree of linear relationship.
* We refer to it as the 'correlation coefficient'

Let's review the sample variance of variable $x$. Of course, I'm truncating our notation as $x$ is a set of samples a locations in our modeling space, $x(\bf{u_\alpha}), \, \forall \, \alpha = 0, 1, \dots, n - 1$.

\begin{equation}
\sigma^2_{x}  = \frac{\sum_{i=1}^{n} (x_i - \overline{x})^2}{(n-1)}
\end{equation}

We can expand the the squared term and replace on of them with $y$, another variable in addition to $x$.

\begin{equation}
C_{xy}  = \frac{\sum_{i=1}^{n} (x_i - \overline{x})(y_i - \overline{y})}{(n-1)}
\end{equation}

We now have a measure that represents the manner in which variables $x$ and $y$ co-vary or vary together.  We can standardized the covariance by the product of the standard deviations of $x$ and $y$ to calculate the correlation coefficent.

\begin{equation}
\rho_{xy}  = \frac{\sum_{i=1}^{n} (x_i - \overline{x})(y_i - \overline{y})}{(n-1)\sigma_x \sigma_y}, \, -1.0 \le \rho_{xy} \le 1.0
\end{equation}

In summary we can state that the correlation coefficient is related to the covariance as:

\begin{equation}
\rho_{xy}  = \frac{C_{xy}}{\sigma_x \sigma_y}
\end{equation}

#### Objective

Provide an example and demonstration for:

1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets
2. provide an intuitive hands-on example to explore spurious correlations

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

#### Load the Required Libraries

The following code loads the required libraries.

```python
%matplotlib inline
from ipywidgets import interactive                        # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
import matplotlib.pyplot as plt                           # plotting
from matplotlib.colors import ListedColormap
import numpy as np                                        # working with arrays
import pandas as pd                                       # working with DataFrames
import seaborn as sns                                     # for matrix scatter plots
from scipy.stats import triang                            # parametric distributions
from scipy.stats import binom
from scipy.stats import norm
from scipy.stats import uniform
from scipy.stats import triang
from scipy.stats import lognorm
from scipy import stats                                   # statistical calculations
import random                                             # random drawing / bootstrap realizations of the data
from matplotlib.gridspec import GridSpec                  # control of subplots
import seaborn as sns                                     # for matrix scatter plots
```

#### Make a Synthetic Dataset

This is an interactive method to:

* select a parametric distribution

* select the distribution parameters

* select the number of samples

* select the number of features

Then we will view the lower triangular correlation matrix

* we will color the correlations that are large (in absolute value $\gt 0.8$)

```python
bins = np.linspace(-1,1,100)                              # set histogram bins

# interactive calculation of the random sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                                      Spurious Correlation Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
dist = widgets.Dropdown(
    options=['Triangular', 'Uniform', 'Gaussian', 'LogNorm'],
    value='Gaussian',
    description='Dataset Distribution:',
    disabled=False,
    layout=Layout(width='200px', height='30px')
)
a = widgets.FloatSlider(min=0.0, max = 100.0, value = 0.5, description = 'Sample: Mean/Mode',orientation='vertical',layout=Layout(width='170px', height='200px'))
a.style.handle_color = 'blue'
d = widgets.FloatSlider(min=0.01, max = 30.0, value = 5.0, step = 1.0, description = 'Sample: St.Dev.',orientation='vertical',layout=Layout(width='110px', height='200px'))
d.style.handle_color = 'green'
b = widgets.FloatSlider(min = 0, max = 100.0, value = 0.5, description = 'Sample: Min.',orientation='vertical',layout=Layout(width='110px', height='200px'))
b.style.handle_color = 'red'
c = widgets.IntSlider(min = 0, max = 100, value = 100, description = 'Sample: Max.',orientation='vertical',layout=Layout(width='110px', height='200px'))
c.style.handle_color = 'orange'
n = widgets.IntSlider(min = 2, max = 1000, value = 4, description = 'Number Samples',orientation='vertical',layout=Layout(width='110px', height='200px'))
n.style.handle_color = 'gray'
m = widgets.IntSlider(min = 2, max = 20, value = 10, description = 'Number Features',orientation='vertical',layout=Layout(width='110px', height='200px'))
m.style.handle_color = 'gray'

uia = widgets.HBox([dist,a,d,b,c,n,m],kwargs = {'justify_content':'center'})                      # basic widget formatting
#uib = widgets.HBox([n, m],kwargs = {'justify_content':'center'})                      # basic widget formatting
ui2 = widgets.VBox([l,uia],)

def f_make(dist,a, b, c, d, n, m):                        # function to take parameters, make sample and plot
    dataset = make_data(dist,a, b, c, d, n, m)
    df = pd.DataFrame(data = dataset)
    corr = df.corr()

# build a mask to remove the upper triangle
    mask = np.triu(np.ones_like(corr, dtype=np.bool))
    corr_values = corr.values
    corr_values2 = corr_values[mask != True]

# make a custom colormap
    my_colormap = plt.cm.get_cmap('RdBu_r', 256)
    newcolors = my_colormap(np.linspace(0, 1, 256))
    white = np.array([256/256, 256/256, 256/256, 1])
    newcolors[26:230, :] = white                          # mask all correlations less than abs(0.8)
    newcmp = ListedColormap(newcolors)

# Draw the heatmap with the mask and correct aspect ratio
    fig, (ax1, ax2) = plt.subplots(2, 1)
    sns.set(font_scale = 0.8)
    sns.heatmap(corr, ax = ax1, annot = True, mask=mask, cmap=newcmp, vmin = -1.0, vmax=1.0, center=0,
        square=True, linewidths=.5, linecolor = 'white', linewidth = 1, cbar_kws={'shrink': .5, 'label': 'Correlation Coefficents'})
    ax1.set_xlabel('Random Independent Features'); ax1.set_ylabel('Random Independent Features')
    ax1.set_title('Lower Triangular Correlation Matrix Heat Map')

    ax2.hist(corr_values2, alpha=0.2,color="red",edgecolor="black", bins = bins)
    ax2.set_title('Lower Triangular Correlation Coefficent Distribution'); ax2.set_xlabel('Correlation Coefficent'); ax2.set_ylabel('Frequency')
    ax2.set_facecolor('white'); ax2.grid(True);

    plt.subplots_adjust(left=0.0, bottom=0.0, right=1.2, top=3.2, wspace=0.2, hspace=0.2)
    plt.show()

def make_data(dist,a, b, c, d, n, m):                     # function to check parameters and make sample
    if dist == 'Uniform':
        if b >= c:
            print('Invalid uniform distribution parameters')
            return None
        dataset = uniform.rvs(size=[n,m], loc = b, scale = c, random_state = 73073).tolist()
        return dataset
    elif dist == 'Triangular':
        interval = c - b
        if b >= a or a >= c or interval <= 0:
            print('Invalid triangular distribution parameters')
            return None
        dataset = triang.rvs(size=[n,m], loc = b, c = (a-b)/interval, scale = interval, random_state = 73073).tolist()
        return dataset
    elif dist == 'Gaussian':
        dataset = norm.rvs(size=[n,m], loc = a, scale = d, random_state = 73073).tolist()
        return dataset
    elif dist == 'LogNorm':
        dataset = lognorm.rvs(size=[n,m], loc = a, scale = np.exp(a), s = d, random_state = 73073).tolist()
        return dataset

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'dist': dist,'a': a, 'd': d, 'b': b, 'c': c, 'n': n, 'm': m})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

### Spurious Correlations Demonstration

* spurious correlations due to a combination of too few samples and skewed distribution

* interactive plot demonstration with ipywidget, matplotlib packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Let's simulate bootstrap, resampling with replacement from a hat with $n_{red}$ and $n_{green}$ balls

* **$n_{red}$**: number of red balls in the sample (placed in the hat)

* **$n_{green}$**: number of green balls in the sample (placed in the hat)

* **$L$**: number of bootstrap realizations

```python
display(ui2, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value='                                      Spurious Correlation Demonstration, Michael P…

    Output()

#### Observations

Some observations:

* spurious correlations due to a combination of too few samples and skewed distribution

* interactive plot demonstration with ipywidget, matplotlib packages

#### Comments

This was a simple demonstration of interactive plots in Jupyter Notebook Python with the ipywidgets and matplotlib packages.

I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Vertex Cover with qubovert

We'll first solve a Vertex Cover problem with the `PCBO` class, and then solve it using the `qubovert.problems` library.

**Contents**

1. Solving Vertex Cover with `qubovert.PCBO`
2. Solving Vertex Cover with the `qubovert.problems` library

*qubovert* must be pip installed.

The goal of the VertexCover problem is to find the smallest number of verticies that can be colored such that every edge of the graph is incident to a colored vertex. Let's begin by making a random graph (seed random for consistent results).

```python
import random
random.seed(0)

n, p = 10, 0.5

vertices = set(range(n))
edges = set((i, j) for i in range(n) for j in range(i+1, n) if random.random() <= p)
print(edges)
```

    {(4, 7), (2, 6), (6, 9), (6, 8), (4, 5), (5, 6), (7, 9), (8, 9), (0, 6), (1, 5), (1, 8), (3, 6), (0, 4), (0, 9), (0, 3), (0, 8), (3, 4), (5, 8), (3, 5)}

## Solving Vertex Cover with `qubovert.PCBO`

See

    Andrew Lucas. Ising formulations of many np problems. Frontiers in Physics, 2:5, 2014.

for a good review on converting common NP problems to QUBOs.

Create a Polynomial Constrained Boolean Optimization (``PCBO``) object.

```python
from qubovert import PCBO

H = PCBO()
```

Let each vertex be a boolean variable that is 1 if it is colored, otherwise 0. Then we want to minimize the number of colored vertices:

```python
for i in vertices:
    H[(i,)] += 1

# another equivalent way to create H is the following

# from qubovert import boolean_var
# for i in verticies:
#     H += boolean_var(i)
```

Now we want to enforce the constraint that every edge is adjacent to at least one colored vertex. The `PCBO.add_constraint_OR(*variables, lam)` method makes it favorable for the OR of the variables to be 1, by penalizing those bit assignments that violate the OR clause with a positive penalty `lam`. For now, we will define a symbol `lam` that we can tune easily later to find the right value.

```python
# !pip install sympy
from sympy import Symbol

lam = Symbol("lam")

for i, j in edges:
    H.add_constraint_OR(i, j, lam=lam)
```

```python
print(H.constraints, "\n")
print("Equality constraints:\n")
for x in H.constraints.get("eq", []):
    print(x, "\n")
```

    {'eq': [{(4,): -1, (4, 7): 1, (7,): -1, (): 1}, {(2,): -1, (2, 6): 1, (6,): -1, (): 1}, {(6,): -1, (6, 9): 1, (9,): -1, (): 1}, {(6,): -1, (6, 8): 1, (8,): -1, (): 1}, {(4,): -1, (4, 5): 1, (5,): -1, (): 1}, {(5,): -1, (5, 6): 1, (6,): -1, (): 1}, {(7,): -1, (7, 9): 1, (9,): -1, (): 1}, {(8,): -1, (8, 9): 1, (9,): -1, (): 1}, {(0,): -1, (0, 6): 1, (6,): -1, (): 1}, {(1,): -1, (1, 5): 1, (5,): -1, (): 1}, {(1,): -1, (1, 8): 1, (8,): -1, (): 1}, {(3,): -1, (3, 6): 1, (6,): -1, (): 1}, {(0,): -1, (0, 4): 1, (4,): -1, (): 1}, {(0,): -1, (0, 9): 1, (9,): -1, (): 1}, {(0,): -1, (0, 3): 1, (3,): -1, (): 1}, {(0,): -1, (0, 8): 1, (8,): -1, (): 1}, {(3,): -1, (3, 4): 1, (4,): -1, (): 1}, {(5,): -1, (5, 8): 1, (8,): -1, (): 1}, {(3,): -1, (3, 5): 1, (5,): -1, (): 1}]}

    Equality constraints:

    {(4,): -1, (4, 7): 1, (7,): -1, (): 1}

    {(2,): -1, (2, 6): 1, (6,): -1, (): 1}

    {(6,): -1, (6, 9): 1, (9,): -1, (): 1}

    {(6,): -1, (6, 8): 1, (8,): -1, (): 1}

    {(4,): -1, (4, 5): 1, (5,): -1, (): 1}

    {(5,): -1, (5, 6): 1, (6,): -1, (): 1}

    {(7,): -1, (7, 9): 1, (9,): -1, (): 1}

    {(8,): -1, (8, 9): 1, (9,): -1, (): 1}

    {(0,): -1, (0, 6): 1, (6,): -1, (): 1}

    {(1,): -1, (1, 5): 1, (5,): -1, (): 1}

    {(1,): -1, (1, 8): 1, (8,): -1, (): 1}

    {(3,): -1, (3, 6): 1, (6,): -1, (): 1}

    {(0,): -1, (0, 4): 1, (4,): -1, (): 1}

    {(0,): -1, (0, 9): 1, (9,): -1, (): 1}

    {(0,): -1, (0, 3): 1, (3,): -1, (): 1}

    {(0,): -1, (0, 8): 1, (8,): -1, (): 1}

    {(3,): -1, (3, 4): 1, (4,): -1, (): 1}

    {(5,): -1, (5, 8): 1, (8,): -1, (): 1}

    {(3,): -1, (3, 5): 1, (5,): -1, (): 1}

Notice that the constraints are automatically added to the objective function. The `'eq'` key of the constraints dictionary indicates that the quantity equals zero. Other possible keys are `'lt'`, `'le'`, `'gt'`, `'ge'`, and `'ne'`. See the docstrings for `PCBO.add_constraint_eq_zero`, `PCBO.add_constraint_lt_zero`, `PCBO.add_constraint_le_zero`, `PCBO.add_constraint_gt_zero`, `PCBO.add_constraint_ge_zero`, and `PCBO.add_constraint_ne_zero` for info.

Here is the final PUBO formulation of the problem. Finding the solution is equivalent to minimizing it as `lam` goes to infinity.

```python
print(H, "\n")
print("Number of variables:", H.num_binary_variables)
print("degree:", H.degree)
```

    {(0,): 1 - 5*lam, (1,): 1 - 2*lam, (2,): 1 - lam, (3,): 1 - 4*lam, (4,): 1 - 4*lam, (5,): 1 - 5*lam, (6,): 1 - 6*lam, (7,): 1 - 2*lam, (8,): 1 - 5*lam, (9,): 1 - 4*lam, (4, 7): lam, (): 19*lam, (2, 6): lam, (6, 9): lam, (6, 8): lam, (4, 5): lam, (5, 6): lam, (7, 9): lam, (8, 9): lam, (0, 6): lam, (1, 5): lam, (1, 8): lam, (3, 6): lam, (0, 4): lam, (0, 9): lam, (0, 3): lam, (0, 8): lam, (3, 4): lam, (5, 8): lam, (3, 5): lam}

    Number of variables: 10
    degree: 2

Here we see the full PUBO representing the PCBO. Let's solve it bruteforce to make sure everything is working well. `H.solve_bruteforce` will ensure that all the constraints are satisfied.

```python
H_solutions = H.solve_bruteforce(all_solutions=True)
for sol in H_solutions:
    print(sol)
```

    {0: 0, 1: 0, 2: 0, 3: 1, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1}
    {0: 0, 1: 1, 2: 0, 3: 1, 4: 1, 5: 0, 6: 1, 7: 0, 8: 1, 9: 1}
    {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1}
    {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0}
    {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0}
    {0: 1, 1: 1, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 0, 9: 1}

We see there are 6 possible solutions to this Vertex Cover problem.

Now let's solve this problem with a generic QUBO solver. Since the degree of this PCBO is 2, it is already in fact a QUBO. However, let's still do the following steps, since they work in general even when the degree is larger than 2.

```python
Q = H.to_qubo()
print(Q)
```

    {(0,): 1 - 5*lam, (1,): 1 - 2*lam, (2,): 1 - lam, (3,): 1 - 4*lam, (4,): 1 - 4*lam, (5,): 1 - 5*lam, (6,): 1 - 6*lam, (7,): 1 - 2*lam, (8,): 1 - 5*lam, (9,): 1 - 4*lam, (4, 7): lam, (2, 6): lam, (6, 9): lam, (6, 8): lam, (4, 5): lam, (5, 6): lam, (7, 9): lam, (8, 9): lam, (0, 6): lam, (1, 5): lam, (1, 8): lam, (3, 6): lam, (0, 4): lam, (0, 9): lam, (0, 3): lam, (0, 8): lam, (3, 4): lam, (5, 8): lam, (3, 5): lam, (): 19*lam}

For testing purposes, let's solve this with bruteforce to see what the proper value of `lam` should be to enforce the constraints. Notice how we remap the QUBO solution to the PCBO solution with `H.convert_solution(x)`. Also note that although the `H.solve_bruteforce` method ensured that the solutions satisfied all the constraints, the `Q.solve_bruteforce` method will not! This is because the `Q` is a QUBO object which is unconstrained, whereas the `H` is a PCBO object which is allowed to be constrained.

```python
for l in (1, 2, 3):
    Q_temp = Q.subs({lam: l})
    solutions = Q_temp.solve_bruteforce(all_solutions=True)
    solutions = [H.convert_solution(x) for x in solutions]
    print('lam', l)
    for s in solutions:
        print("\t", s, "is", "valid" if H.is_solution_valid(s) else "invalid")
    print()
```

    lam 1
    	 {0: 0, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is invalid
    	 {0: 0, 1: 0, 2: 0, 3: 1, 4: 1, 5: 0, 6: 1, 7: 0, 8: 1, 9: 1} is invalid
    	 {0: 0, 1: 0, 2: 0, 3: 1, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 0, 1: 1, 2: 0, 3: 1, 4: 1, 5: 0, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is invalid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 0, 9: 1} is invalid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 0} is invalid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is valid
    	 {0: 1, 1: 1, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 0, 9: 1} is valid

    lam 2
    	 {0: 0, 1: 0, 2: 0, 3: 1, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 0, 1: 1, 2: 0, 3: 1, 4: 1, 5: 0, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is valid
    	 {0: 1, 1: 1, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 0, 9: 1} is valid

    lam 3
    	 {0: 0, 1: 0, 2: 0, 3: 1, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 0, 1: 1, 2: 0, 3: 1, 4: 1, 5: 0, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 1, 9: 1} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is valid
    	 {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0} is valid
    	 {0: 1, 1: 1, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1, 7: 0, 8: 0, 9: 1} is valid

We see that `lam = 2` is sufficient to enforce the constraints, and we get back the same results as from the PCBO solve. So let's update our QUBO.

```python
Q_good = Q.subs({lam: 2})
```

Now let's solve the QUBO with D'Wave's simulated annealer.

```python
#!pip install dwave-neal
from neal import SimulatedAnnealingSampler

sampler = SimulatedAnnealingSampler()
```

Note that their software package takes in a specific form for QUBOs, namely, the keys of the dictionary must be two element tuples. This form can be accessed from `Q` with `Q.Q` and from `Q_good` with `Q_good.Q`.

```python
qubo_sample = sampler.sample_qubo(Q_good.Q, num_reads=500)
print("objective function:", qubo_sample.first.energy + Q_good.offset, "\n")

qubo_solution = qubo_sample.first.sample
print("qubo solution:", qubo_solution, "\n")

solution = H.convert_solution(qubo_solution)
print("pcbo solution:", solution)
print("objective function:", H.value(solution), "\n")

print("The solution is", "valid" if H.is_solution_valid(solution) else "invalid")
```

    objective function: 6.0

    qubo solution: {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0}

    pcbo solution: {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0}
    objective function: 6

    The solution is valid

Notice that `H.is_solution_valid` checks if all the constraints are satisfied. Notice also that the objective function is equal to the number of colred edges.

Now we'll solve an QUSO formulation of our problem.

```python
L = H.to_quso().subs({lam: 2})
# note that we cannot do H.subs({lam: 2}).to_quso()!! This is because H.subs({lam: 2})
# creates a new PCBO object, and it's mapping from variables labels to integers may be
# different than H's mapping. For example, try H.mapping == H.subs({lam: 2}).mapping a
# few times. They will often be different.
print(L)
```

    {(0,): 2.0, (): 14.5, (1,): 0.5, (3,): 1.5, (4,): 1.5, (5,): 2.0, (6,): 2.5, (7,): 0.5, (8,): 2.0, (9,): 1.5, (4, 7): 0.5, (2, 6): 0.5, (6, 9): 0.5, (6, 8): 0.5, (4, 5): 0.5, (5, 6): 0.5, (7, 9): 0.5, (8, 9): 0.5, (0, 6): 0.5, (1, 5): 0.5, (1, 8): 0.5, (3, 6): 0.5, (0, 4): 0.5, (0, 9): 0.5, (0, 3): 0.5, (0, 8): 0.5, (3, 4): 0.5, (5, 8): 0.5, (3, 5): 0.5}

Similar to their QUBO solver, D'Wave's QUSO solver accepts a specific form for QUSO models, namely a linear term dictionary and a quadratic term dictionary. These can be accessed with `L.h` and `L.J`.

```python
quso_sample = sampler.sample_ising(L.h, L.J, num_reads=500)
print("objective function:", quso_sample.first.energy + L.offset, "\n")

quso_solution = quso_sample.first.sample
print("quso solution:", quso_solution, "\n")

solution = H.convert_solution(quso_solution)
print("pcbo solution:", solution)
print("objective function:", H.value(solution), "\n")

print("The solution is", "valid" if H.is_solution_valid(solution) else "invalid")
```

    objective function: 6.0

    quso solution: {0: 1, 1: -1, 2: 1, 3: -1, 4: -1, 5: 1, 6: -1, 7: 1, 8: -1, 9: -1}

    pcbo solution: {0: 0, 1: 1, 2: 0, 3: 1, 4: 1, 5: 0, 6: 1, 7: 0, 8: 1, 9: 1}
    objective function: 6

    The solution is valid

We see that the solution is again valid.

## Solving Vertex Cover with the `qubovert.problems` library

Here we will use the ``qubovert.problems`` library to abstract away all the thinking involved in formulating the Vertex Cover problem. Begin by importing the `VertexCover` class.

```python
from qubovert.problems import VertexCover

problem = VertexCover(edges)
```

Let's solve it bruteforce to make sure everything is working.

```python
print(problem.solve_bruteforce(all_solutions=True))
```

    [{3, 4, 5, 6, 8, 9}, {1, 3, 4, 6, 8, 9}, {0, 4, 5, 6, 8, 9}, {0, 4, 5, 6, 7, 8}, {0, 3, 5, 6, 7, 8}, {0, 1, 4, 5, 6, 9}]

Notice how the format of each solution is a set of vertices, indicating which vertices to color. Again we find 6 valid solutions as we did with the PCBO method above.

Now let's solve the QUBO with D'Wave's simulated annealer. The `problem.to_qubo` method takes in as an argument some lagrange multipliers for the QUBO formulation, but if left blank they are automatically formed to enforce the constraints.

```python
Q = problem.to_qubo()

print("Number of QUBO variables:", Q.num_binary_variables, "\n")

qubo_solution = qubo_sample.first.sample
print("qubo solution:", qubo_solution)

qubo_sample = sampler.sample_qubo(Q.Q, num_reads=500)
print("qubo objective function:", qubo_sample.first.energy + Q.offset, "\n")

solution = problem.convert_solution(qubo_solution)
print("problem solution (which vertices to cover):", solution, "\n")

print("The solution is", "valid" if problem.is_solution_valid(solution) else "invalid")
```

    Number of QUBO variables: 10

    qubo solution: {0: 1, 1: 0, 2: 0, 3: 1, 4: 0, 5: 1, 6: 1, 7: 1, 8: 1, 9: 0}
    qubo objective function: 6.0

    problem solution (which vertices to cover): {0, 3, 5, 6, 7, 8}

    The solution is valid

Note that the QUBO solution that maps integers 0 through 9 to values 0 or 1 may have a different mapping. Ie the QUBO label 0 may not actually correspond to vertex 0, this is why it is crucial to use the `problem.convert_solution` method!

Now let's solve the QUSO.

```python
L = problem.to_quso()

print("Number of QUSO variables:", L.num_binary_variables, "\n")

quso_solution = quso_sample.first.sample
print("quso solution:", quso_solution)

quso_sample = sampler.sample_ising(L.h, L.J, num_reads=500)
print("quso objective function:", quso_sample.first.energy + L.offset, "\n")

solution = problem.convert_solution(quso_solution)
print("problem solution:", solution, "\n")

print("The solution is", "valid" if problem.is_solution_valid(solution) else "invalid")
```

    Number of QUSO variables: 10

    quso solution: {0: 1, 1: -1, 2: 1, 3: -1, 4: -1, 5: 1, 6: -1, 7: 1, 8: -1, 9: -1}
    quso objective function: 6.0

    problem solution: {1, 3, 4, 6, 8, 9}

    The solution is valid

Thus we have solved the Vertex Cover problem!


# Hauptkomponentenanalyse vs.
# Denoising Variational Autoencoders

## _Intuition, Formalismus und Beispiele_

jupyter nbconvert PCAvsDVAE.ipynb --to slides --post serve

jupyter-nbextension install rise --py --sys-prefix
jupyter-nbextension enable rise --py --sys-prefix

# Eine intuitive Perspektive ...

#### "... realistische, hochdimensionale Daten konzentrieren sich in der Nähe einer nichtlinearen, niedrigdimensionalen Mannigfaltigkeit ..." [Lei et al., 2018]

#### Aber wie lernt man die Mannigfaltigkeit und die Wahrscheinlichkeitsverteilung darauf?

# Evaluating PCA and DVAE through examples

The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.

```python
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
import tensorflow as tf
from keras.layers import Input, Dense, Lambda
from keras.models import Model
from keras import backend as K
from keras import metrics
from keras.datasets import mnist
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Conv2DTranspose,Reshape
from sklearn.decomposition import PCA
import os
%matplotlib inline
os.environ["PATH"] += os.pathsep + 'C:/Program Files (x86)/Graphviz2.38/bin/'

# analytical PCA of the training set
def analytical_pca(y):
    pca = PCA(0.7)
    pca.fit(y)
    loadings = pca.components_
    components = pca.transform(y)
    filtered = pca.inverse_transform(components)
    return filtered

# training params for the example
num_train = 50000
n_images = 6
batch_size = 256
original_dim = 784
latent_dim = 8
epochs = 10
epsilon_std = 1.0
noise_factor = 0.5

# train the VAE on MNIST digits
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# prepare data for PCA
shape_x_train = x_train.shape
pcaInput = np.reshape(x_train,[shape_x_train[0],shape_x_train[1]*shape_x_train[2]]).astype('float32')/255
# prepare data for DVAE
train_num=50000
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), 28,28,1))
x_test = x_test.reshape((len(x_test), 28,28,1))
noise_train = x_train + noise_factor * np.random.randn(*x_train.shape)
noise_test = x_test + noise_factor * np.random.randn(*x_test.shape)
# Clip the images to be between 0 and 1
noise_train = np.clip(noise_train, 0., 1.)
noise_test = np.clip(noise_test, 0., 1.)

# display
showidx=np.random.randint(0,num_train,n_images)
# precalculate PCA
pcaOutput = analytical_pca(pcaInput)
# display input, noisy input and PCA on noisy input
for i,idx in enumerate (showidx):
    figure[0: 28,i *28: (i + 1) * 28] = np.reshape(x_train[idx], [28, 28])
    figure[28: 56,i *28: (i + 1) * 28] = np.reshape(noise_train[idx], [28, 28])
    figure[28 * 2: 28 * 3,i *28: (i + 1) * 28] = np.reshape(pcaOutput[idx], [28, 28])
plt.figure(figsize=(28*3, 28*n_images))
plt.imshow(figure, cmap='Greys_r')
plt.show()

#encoder part
x_noise = Input(shape=(28,28,1))
conv_1 = Conv2D(64,(3, 3), padding='valid',activation='relu')(x_noise)
conv_2 = Conv2D(64,(3, 3), padding='valid',activation='relu')(conv_1)
pool_1 = MaxPooling2D((2, 2))(conv_2)
conv_3 = Conv2D(32,(3, 3), padding='valid',activation='relu')(pool_1)
pool_2 = MaxPooling2D((2, 2))(conv_3)
h=Flatten()(pool_2)
z_mean = Dense(latent_dim)(h)
z_log_var = Dense(latent_dim)(h)

#reparameterization trick
def sampling(args):
    z_mean, z_log_var = args
    epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0.,
                              stddev=epsilon_std)
    return z_mean + K.exp(z_log_var / 2) * epsilon

# note that "output_shape" isn't necessary with the TensorFlow backend
z = Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])

#decoder part
# we instantiate these layers separately so as to reuse them later
z=Reshape([1,1,latent_dim])(z)
conv_0T = Conv2DTranspose(128,(1, 1), padding='valid',activation='relu')(z)#1*1
conv_1T = Conv2DTranspose(64,(3, 3), padding='valid',activation='relu')(conv_0T)#3*3
conv_2T = Conv2DTranspose(64,(3, 3), padding='valid',activation='relu')(conv_1T)#5*5
conv_3T = Conv2DTranspose(48,(3, 3), strides=(2, 2),padding='same',activation='relu')(conv_2T)#10*10
conv_4T = Conv2DTranspose(48,(3, 3), padding='valid',activation='relu')(conv_3T)#12*12
conv_5T = Conv2DTranspose(32,(3, 3), strides=(2, 2),padding='same',activation='relu')(conv_4T)#24*24
conv_6T = Conv2DTranspose(16,(3, 3), padding='valid',activation='relu')(conv_5T)#26*26
x_out = Conv2DTranspose(1,(3, 3), padding='valid',activation='sigmoid')(conv_6T)#28*28

# instantiate VAE model
vae = Model(x_noise, x_out)
vae.summary()

# Compute VAE loss
def VAE_loss(x_origin,x_out):
    x_origin=K.flatten(x_origin)
    x_out=K.flatten(x_out)
    xent_loss = original_dim * metrics.binary_crossentropy(x_origin, x_out)
    kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
    vae_loss = K.mean(xent_loss + kl_loss)
    return vae_loss
vae.compile(optimizer='adam', loss=VAE_loss)

vae.fit(noise_train,x_train,
        shuffle=True,
        epochs=epochs,
        batch_size=batch_size,
        validation_data=(noise_test, x_test))

digit_size = 28
figure = np.zeros((digit_size * 4, digit_size * n_images))

num_test=10000
showidx=np.random.randint(0,num_test,n_images)
x_out=vae.predict(x_test[showidx])
# Display
for i,idx in enumerate (showidx):
    figure[0: 28,i *28: (i + 1) * 28] = np.reshape(x_test[idx], [28, 28])
    figure[28: 28 * 2,i *28: (i + 1) * 28] = np.reshape(noise_test[idx], [28, 28])
    figure[28 * 2: 28 * 3,i *28: (i + 1) * 28] = np.reshape(x_out[i], [28, 28])
    figure[28 * 3: 28 * 4,i *28: (i + 1) * 28] = np.reshape(pcaOutput[idx], [28, 28])

plt.figure(figsize=(28 * 4, 28*n_images))
plt.imshow(figure, cmap='Greys_r')
plt.savefig('result_keras_VAE.png')
plt.show()
```

```python
# https://github.com/dojoteef/dvae
```

# Hauptkomponentenanalyse
# (Principal Component Analysis, PCA)
* __unsupervised__ learning
* __linear transformation__
    * "encode" a set of observations to a new coordinate system in which the values of the first coordinate (component) have the largest possible variance [Friedman et al., 2017]
    * the resulting coordinates (components) are uncorrelated with the preceeding coordinates
* practically computing
    * __eigendecomposition of the covariance matrix__
    * __singular value decomposition__ of the observations
* used for __dimensionality reduction__
* __reconstructions of the observations__("decoding") from the leading __principal components__ have the __least total squared error__

## Grundlegende Mathematik der PCA

### Lineare Transformation

* Let $\{y_i\}^N_{i=1}$ be a set of $N$ observations vectors, each of size $n$, with $n\leq N$.

* A __linear transformation__ on a finite dimensional vector can be expressed as a __matrix multiplication__:

$$ \begin{align} x_i = W y_i \end{align} $$

where $y_i \in R^{n}, x_i \in R^{m}$ and $W \in R^{nxm}$.

* Each $j-th$ element in $x_i$ is the __inner product__ between $y_i$ and the $j-th$ column in $W$, denoted as $w_j$. Let $Y \in R^{nxN}$ be a matrix obtained by horizontally concatenating $\{y_i\}^N_{i=1}$,

$$ Y = \begin{bmatrix} | ... | \\ y_1 ... y_N \\ | ... | \end{bmatrix} $$

* Given the __linear transformation__, it is clear that:

$$ X = W^TY,  X_0 = W^TY_0, $$

where $Y_0$ is the matrix of centered (i.e. subtract the mean from each each observation).

### Maximale Varianzkomponenten, Kovarianz und Dekorrelation
* In particular, when $W^T$ represents the __transformation applying Principal Component Analysis__, we denote $W = P$. Each column of $P$, denoted $\{p_j\}^n_{j=1}$ is a __loading vector__, whereas each transformed vector $\{x_i\}^N_{i=1}$ is a __principal component__.

* The first loading vector is the unit vector with which the inner products of the observations have the __greatest variance__:

$$ \max w_1^T Y_0Y_0^Tw_1, w_1^Tw_1 = 1$$

* The solution of the previous equation is the first eigenvector of the __sample covariance matrix__ $Y_0Y_0^T$ corresponding to the largest eigenvalue.

* MAtrix $P$ can be calculated by diagonalizing the covariance matrix:

$$ Y_0Y_0^T = P \Lambda P^{-1} = P \Lambda P^T $$

$\Lambda = Y_0Y_0^T $ is a diagnoal matrix whose diagonal elements $\{\lambda_i\}^N_{i=1}$ are sorted descendingly. The inverse transformation is $ Y = PX $. Due to the fact that the covariance matrix of $X$ is diagonal PCA is a __decorrelation transformation__.

### Dimensionsreduzierung, Komprimierung, Skalierung
PCA is used for __dimensionality reduction__ due to its capability to __reduce the number of variables__ through a linear transformation. This is done by keeping the first $m$ principal components $(m < n)$ and applying:

$$ X_m = P_m^TY$$

Keeping only the $m$ principal components, PCA __loses information__ (i.e. __lossy compression__), but the __loss is minimized__ by __maximizing the components variances__.

Many __iterative algorithms__ can be used for finding the $m$ largest eigenvalues of $Y_0Y_0^T$
* QR algorithm
* Jacobi algorithm
* power method

For __large datasets__ such algorithms are __prohibitive__.

### Minimaler quadratischer Gesamtrekonstruktionsfehler

The transformation matrix $P_m$ can be also computed as a solution of:

$$ \min_{W \in R^{nxm}} \| Y_0 - WW^TY_0 \|_F^2, W^TW = I_{mxm}$$

where $F$ is the Frobenius norm.

This shows that $P_m$ __compresses each centered vector__ of length $n$ into a vector of length $m$ where ($ m < n$) such that it __minimizes__ the sum of total __squared reconstruction errors__ (i.e. __inverse transformation__).

### Singularwert-Zerlegung (Singular Value Decomposition, SVD)

The matrix $Y_0 \in R^{nxN}$ can be __factorized__ as $Y_0 = U \Sigma V^T$ where $U \in R^{nxn}$ and $V \in R^{NxN}$ are __orthogonal matrices__ and $\Sigma \in R^{nxN}$ has non-zero elements only on the diagonal (i.e. __singular values__).

The SVD of $Y_0$ is equivalent to the __eigendecomposition__ of $Y_0T_0^T$.

A (non-zero) vector v of dimension N is an __eigenvector__ of a square N × N matrix A if it satisfies the __linear equation__

$$Av =\lambda v$$

where $λ$ is a scalar, termed the __eigenvalue corresponding to v__.

# Autoencoders
* unsupervised neural network
* minimize the error of reconstructions of observations [Goodfellow et al., 2016]
* basically responsible to learn the identity function
* training with backpropagation and then separate to implement the encoding / deconding

A typical __autoencoder pipeline__ looks as

## Grundlegende Mathematik der Autoencoders

For each input vector $x$ of dimension $d$ out of the entire dataset of size $n$, the network tries to reconstruct $x'$, by:
* first encoding the input (i.e. applying linear / nonlinear transformation $g_\phi(.)$)
* obtaining a latent, compressed code in the bottleneck layer, $z$, and
* decoding the compressed input at the output using linear / nonlinear transformation $f_\theta(.)$

The parameters $(\theta, \phi)$ are learned together to output a reconstructed data sample similar to the input, $x \approx f_\theta(g_\phi(x))$, in other words the identity function.

There are multiple metrics to quantify the difference, cross-entropy when activation function is sigmoid, or the simple Mean Squared Error (MSE):

$$ \frac{1}{n} \sum_{i=1}^{n}(x^{i} - f_\theta(g_\phi(x^{i}))^2$$

# PCA vs. Autoencoders

*  an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function is closely related to PCA - its weights span the principal subspace [Plaut, 2018]
* in autoencoders the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder
* in autoencoders local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen [Rolinek et al, 2019]
* the difference is that, opposite from PCA, in Autoencoders the coordinates of the output of the bottleneck are correlated and are not sorted in the descending order of variance!

# Denoising Variational Autoencoders (DVAE)

The operation process is __different__ from the basic autoencoder due to the fact that there is __noise__ injected in the input (with a certain probability distribution) and there __latent space__ needs to recover this probability to __reconstruct__ the original input [Im, Bengio et al., 2017, Kingma et al., 2017].

For each corrupted input vector $\tilde x$ of a clean vector $x$ of dimension $d$, the network tries to reconstruct $x'$, by:
* first encoding the input, representing the mapping as the probability of estimating $z$ given the input, knowing the parameters of the noise ($\mu$ and $\sigma$) to allow tractability of the posterior calculation
* obtaining a latent, compressed code in the bottleneck layer, $z$, sampled from $q_\phi(z|x)$
* decoding the compressed input at the output given the observation model $p_\theta(x|z)$

## Grundlegende Mathematik der DVAE

The loss function to recover the original input (__not the corrupted one__), $\tilde{x}^{i} = M(\tilde{x}^{i} | x^{i})$, if we use the simple Mean Squared Error (MSE):

$$ \frac{1}{n} \sum_{i=1}^{n}(x^{i} - f_\theta(g_\phi({\tilde{x}}^{i}))^2$$

The expectation term in the loss function invokes generating samples from $z \backsim q_\phi(z|x)$. Sampling is a stochastic process and therefore we cannot backpropagate the gradient.

The estimated posterior $q_\phi(z|x)$ should be very close to the real one $p_\theta(z|x)$. We can use Kullback-Leibler divergence to quantify the distance between these two distributions. KL divergence DKL(X∥Y) measures how much information is lost if the distribution Y is used to represent X.

Yet, Variational Bayesian methods, the loss function is known as the variational lower bound, or evidence lower bound (ELBO). The “lower bound” part in the name comes from the fact that KL divergence is always non-negative and thus the loss is the lower bound of $log p_\theta(x)$.

$$ log p_\theta(x) - D_{KL}(q_\phi(z|x) || p_\theta(z|x)) \leq log p_\theta(x) $$

Therefore by minimizing the loss, we are maximizing the lower bound of the probability of generating real data samples.

https://github.com/dojoteef/dvae

https://github.com/block98k/Denoise-VAE

# Overall comparison PCA vs. DVAE

### Manifold learning

| __PCA__ | __DVAE__ |
|-----|------|
| linear encoding/decoding, withoug noise robustness |nonlinear, probabilistic encoding/decoding with (input / hidden layer) noise robustness and nonlinear activation functions|
| decorrelated coordinates of the latent space (whitening transformation) | correlated outputs of bottleneck (decoding input) |
| coordinates of the latent space are in descending order of variance | coordinates are not sorted |
| columns of transformation matrix are orthonormal | columns of transformation matrix not necessarily orthonormal |
| robust on moderate noise with known distributions | robust to various types (masking noise, Gaussian noise, salt-and-pepper noise) and quantities of stochastic injected noise (denoising important for generalization performance) |
| basic algorithm (without regularization) low robustness | points in low-dimensional manifold robust to noise in the high-dimensional observation space |

# Overall comparison PCA vs. DVAE

### Training

| __PCA__ | __DVAE__ |
|-----|------|
| map input to a fixed vector | map input to a distribution |
| iterative methods: QR decomposition, Jacobi algorithm, SVD | backpropagation  |
| inefficient on large datasets due to covariance calculation | efficient on large datasets due to strong manifold learning |
| based on correlation/covariance matrix, which can be - at least in theory - very sensitive to outliers | can sample directly from the input space and describe the input noise properties ("reparametrization trick") |

# References and further reading
[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016.

[Friedman et al., 2017] Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, 2017.

[Plaut, 2018] Plaut, E., 2018. From principal subspaces to principal components with linear autoencoders. arXiv preprint arXiv:1804.10253.

[Im, Bengio et al., 2017] Im, D.I.J., Ahn, S., Memisevic, R. and Bengio, Y., 2017, February. Denoising criterion for variational auto-encoding framework. In Thirty-First AAAI Conference on Artificial Intelligence.

[Rolinek et al, 2019] Rolinek, M., Zietlow, D. and Martius, G., 2019. Variational Autoencoders Pursue PCA Directions (by Accident). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 12406-12415).

[Lei et al., 2018] Lei, N., Luo, Z., Yau, S.T. and Gu, D.X., 2018. Geometric understanding of deep learning. arXiv preprint arXiv:1805.10451.

[Kingma et al., 2013] Kingma, D.P. and Welling, M., 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.


#  CS207 Final Project: Milestone 2 - November 19 2019

---

 Geng Yichen, Jian Yingsi, Meeus Matthieu, Zhang Lihong

# 1 Introduction

Derivatives come up in every aspect of science and engineering. Calculus taught us how to derive analytical expressions of functional derivatives, but in many cases this is either impossible or too much of a hassle. Therefore, numerical methods or algorithmic approaches to compute the derivative are extremely important.
Methods of computing functions' derivatives in computer programs can be classified into 4 types:

(1) Determining derivatives by hands and coding them.

(2) Symbolic differentiation in computer tools, such as Mathematica and Maple.

(3) Numerical methods: using finite differences to approxiate derivatives.

(4) Automatic Differentiation, which is the subject of our project.

Automatic differentiation (AD) is a set of techniques for evaluating functional derivatives efficiently and accurately in computer programs. For any analytic function f(x) to be differentiated at a point $x_0$, AD first rewrites f(x) as a combination of elementary functions, then determining the derivative values of f(x) through combining derivatives of elementary functions by the chain rule. Since the derivative values of all elementary functions are known and accurate, and the procedures of AD have no potential sources of errors except tiny rounding errors due to machine precision, thus the derivative values obtained by AD are accurate. As for other differentiation methods, manual calculating functional derivatives and coding them by hands can be tedious, time-consuming and prone to make mistakes; symbolic differentiation could return rigmarole and unreadable symbolic expressions; and numerical method of finite differences could be ill-conditioned due to truncation and round-off errors, and is also inappropriate to handle functional derivatives with many independent variables.

For example, in the numerical method of finite difference, the derivative is calculated as the following where the limit for h is approached but not put to zero:

$$\frac{df}{dx} \approx \frac{f(x+h) - f(x)}{h}$$

While this approach yields decent results in many cases, it is never completely accurate. For too large values of h, the error originates from the intrinsic error of the finite difference derivative. For too small values of h, the error originates from rounding errors.

Although AD does not show explicit derivative expressions, people usually only want to obtain derivative values at some points rather than symbolic expressions, thus AD is much better than other differentiation methods with respect to determining functional derivative values.

In modern differentiation computation, AD is the most efficient and accurate tool to implement differentiation in computer programs. In this project, we will design an accurate, user-oriented differentiation calculator by implementing AD.

# 2 Background

Automatic differentiation is an algorithmic approach to compute derivatives up until machine precision accuracy. It has two common methods: forward and reverse mode. In this project, only the forward mode will be discussed at first. Forward automatic differentiation leverages the chain rule to split the differentiation of a complex function in elementary functions and makes use of the easy derivatives of elementary functions.  If $h(u(t))$ is the function of which the derivative is required, the chain rule for partial derivatives says:

$$\frac{\partial h}{\partial t} = \frac{\partial h}{\partial u}*\frac{\partial u}{\partial t}$$

Or for $h(u(t), v(t))$:

\begin{equation}
    \frac{\partial h}{\partial t} = \frac{\partial h}{\partial u}*\frac{\partial u}{\partial t} + \frac{\partial h}{\partial v}*\frac{\partial v}{\partial t}
\end{equation}

Hence, the computation of the derivatives of complicated functions that consist of multiple, consequent elementary operations can be split into the multiplication and addition of derivatives of the elementary functions in the following table. Examples of the elementary operations are addition and multiplication but also sine, cosine and log. The complete list of elementary functions that will be incorporated in this package can be found in 'Implementation'. A subset is illustrated in the table below:

| Elementary function        | Example |
| :-------------: |:-------------:|
| Field Operations    | $+ - * / $ |
| Powers     |$x^2, x^6$|
| Roots     |$\sqrt{x}$|
| Trigonometric     |$sin(x)$|
| Inverse Trigonometric     |$asin(x)$|
| Logarithmic     |$log(x)$|
| Exponential    |$e^x$|

The split of the complex function into its elementary operations is commonly visualized in a so-called computational graph. This summarizes the sequence of operations that need to be done for the evaluation and differentiation of the function. An example for the simple function $f(x,y) = xy + exp(xy) $ is given in the image below. Note that this example comes from Lecture 12.

The actual implementation of the forward mode can be better understood using the so-called seed vectors $p_i$ for every variable $x_i$. This means that, every time the derivative of a function is computed, the derivative is 'seeded', or multiplied (dot product) by the corresponding seed. An example might clarify this. Imagine function $f = x*cos(y)$, where x and y are the variables corresponding to seed vectors $p_1 = [1,0]$ and $p_2 = [0,1]$ respectively. The directional derivative $D_p$ of f will then be computed as follows:

$$D_px_3 = \sum_{i}^n \frac{\partial x_3}{x_i} p_i$$

Where $x_1$, $x_2$ and $x_3$ correspond to x, y and $x*cos(y)$. We can then specify the direction p to find the desired derivative. For instance, for $p = [1,0]$, we recover:

$$D_px_3 = \frac{\partial x_3}{\partial x_1} p_1 + \frac{\partial x_3}{\partial x_2}p_2 = \frac{\partial x_3}{\partial x_1} = \frac{\partial f}{\partial x}$$

Or mathematically:

$$D_px_3 = \nabla x_3 \cdot p$$

This means that the forward mode of automatic differentiation is actually computing the dot product of the gradient and the seed vector. Choosing the seed vectors appropriately, either this product or the full gradient can be recovered, dependent on the application.

For functions in higher dimensions h(x): ${\rm I\!R}^m \rightarrow {\rm I\!R}^n$, the entire Jacobian will be computed using the same, simple approach. Recall the definition of the Jacobian J in ${\rm I\!R}^{nxm}$ for function h(x):

$$\mathbf{J}=\left[\begin{array}{cccc}
\frac{\partial h_1}{\partial x_1} & \frac{\partial h_1}{\partial x_2} & .. & \frac{\partial h_1}{\partial x_m} \\
\frac{\partial h_2}{\partial x_1} & \frac{\partial h_2}{\partial x_2} & .. & \frac{\partial h_2}{\partial x_m} \\
.. & .. & .. & .. \\
\frac{\partial h_n}{\partial x_1} & \frac{\partial h_n}{\partial x_2} & .. & \frac{\partial h_n}{\partial x_m}
\end{array}\right]$$

Note that the seed vector approach is still valid for this high-dimensional case. So teh forward mode is actually computing the product of the Jacobian with the seed vector. Again, the entire Jacobian can be recoverd by choosing the seed vectors appropriately and adding the resulting products.

# 3 How to Use AutoDiff

## 3.1 How to install AutoDiff
There are two methods of installing our package: Github and PyPI. The installation details are listed below. After installation, the user should import the class Var, which represents variables, from our package (shown as the pseudocode below) and other dependencies (such as numpy and math) since we import those dependencies to our package modules. With the imported package, the user can define x as an object of Var, and give the initial input of x. Then the user can apply function f on x, which returns f(x) with new values and jacobian.
### 3.1.1 Installation from Github

*   Step 1: Download our package from Github to users' local directory by the following commands in the terminal
```
mkdir AutoDiff
cd AutoDiff
git clone https://github.com/BackPropagators/cs207-FinalProject.git
cd cs207-FinalProject
```

*   Step 2: Create and Activate a virtual environment in conda.
 * If the user don't have conda, she/he should install it. Then the user can create his/her own virtual environment in conda by the following codes, where yourname is the name the user wants to call the newly created environment.
```
conda create -n yourname python=3.6 anaconda
```
 * Activate the user's virtual environment
 ```
 source activate yourname
 ```
 * Install the required dependencies and run the given tests in AutoDiff
 ```
 pip install -r requirements.txt
 pytest
 ```
 If pytest is not installed, then the user can install it by
 ```
 pytest install -U pytest
 ```

 * Use AutoDiff Python package in the terminal (See demo in 3.2)
 ```python
>>> from AutoDiff.ForwardAD import Var
>>> x = Var(1)
>>> f = x + 1
>>> f.get_value()
2
>>> f.get_jacobian()
[1]
...
>>> quit()
```
```
# deactivate virtual environment
conda deactivate
```

### 3.1.2 Installation from PyPI.
We will use PyPI to distribute our package in our futre development of the package.

## 3.2 Basic Demo
Install our package, and import the module and dependencies
```python
pip install AutoDiff
python
>>> from AutoDiff.ForwardAd import Var
>>> import numpy as np
```
instantiate a `Var` class.
```
>>> x = Var(1.0)
```
Define a function.
```python
>>> f = x + 1
```
Evaluate the function value which is expected to be 2.0.
```python
>>> f.get_value()
2.0
```
Evaluate the function jacobian which is exepcted to be [1.0].
```python
>>> f.get_jacobian()
[1.0]
```

Evaluate the value and jacobian of scaler function with one variable.
```python
# basic operation
# Expect the value of f is 3.0, and the derivative of f is [2.0]
>>> x = Var(1)
>>> f = 2*x + 1
>>> f.get_value()
3.0
>>> f.jacobian
[2.0]

# exponential function
# Expect the value of f is around 2.718281828459, and the derivative is around [2.718281828459]
>>> f = Var.exp(x)
>>> f.get_value()
2.718281828459045
>>> f.get_jacobian()
[2.718281828459045]

# logarithmic function
# Expect the value of f is 0.0, and the derivative of f is around [0.43429448190325]
>>> f = Var.log(x, b=10)
>>> f.get_value()
0.0
>>> f.get_jacobian()
[0.43429448190325176]

# trignometric function
# Expect the value of f is 0.49999999999999994, and the derivative is [0.8660254037844]
>>> y = Var(np.pi/6)
>>> f = Var.sin(y)
>>> f.get_value()
0.49999999999999994
>>> f.get_jacobian()
[0.8660254037844387]
```

# 4 Software Organization

**4.1 Project Directory Structure**

                    CS207-FinalProject/
                                       README.md
                                       LICENSE
                                       setup.py
                                       AutoDiff/
                                                _init_.py
                                                ForwardAD.py
                                       docs/
                                                     milestone1.ipynb
                                                     milestone2.ipynb
                                       test/
                                            test_all.py
                                       demo/
                                            presentation.pdf
                                       ...

**4.2 Included Modules and Basic Functionality**

AutoDiff: This module includes implementation for classes AutoDiff and Multifunc. AutoDiff and Multifunc implement the forward mode of auto differentiation. AutoDiff takes care of scalar function inputs (with single and multiple variables) and Multifunction takes care of vector function inputs.

Test: This module includes all the test suites for the package.

**4.3 Test**

The test suites live in *CS207-FinalProject/Test/* folder.  We will use Travis CI to run tests and use CodeCov to automatically checks code coverage.

**4.4 Distributing Our Package**

We will use PyPI to distribute our package. The Python Package Index (PyPI) is a repository of software for the Python programming language.

**4.5 Packaging**

We will follow the Python Packaging User Guide published by the Python Packaging Authority(PyPA), which is the authoritative resource on how to package, publish, and install Python projects using current tools. We will use Setuptools because it is comprehensive, actively-maintained, and stable to  create an package for easy distibution out of our project.

# 5 Implementation

**5.1 Core Data Structures**
The core data structure for storing the jacobian is python dictionary, where the key is the variable (an instance of `Var`, see **Core Classes** below) and the value is its partial derivative.

**5.2 Core Classes**
In the `ForwardAD.py`, we implemented the `Var` class, which defines the variables in a function and calculates the jacobian of scalar function with single and multiple variables (multivariables will be tested in the future; see 5.6 for details). We have also implemented the `MultiFunc` class, which calculates the jacobian of vector functions with single and multiple variables (`MultiFunc` will be tested in the future, so it is not included in the code; see 5.6 for details).

**5.3 Important Attributes**
The `Var` class has two private attributes: `self._val` and `self._jacobian`. `self._val` stores the value of the current function, and `self._jacobian` is a dictionary whose key is a `Var` instance and value is its partial derivative. `self._jacobian` is initialized as {self: 1} in the constructor. When there is only one variable in the function, after each operation, only the value (jacobian) may be updated whereas the key (variable) remains the same in the new dictionary. When there are multiple variables in the function, after each operation, values (jacobian) may be updated and new key (variable) may be added in the new dictionary.

**5.4 External Dependencies**
We will rely on
- Numpy: a Python library supporting large, multi-dimensional arrays and matrices, with a large collection of high-level mathematical functions.
- math: a Python library containing a great many mathematical operations.
- pytest: a framework supporting complext tests and make small tests easy.

**5.5 Elementary Functions**
We defined methods that deal with these elementary functions inside the `Var` class. To use these functions, simply call `Var.function_name`. See demo in 3.2.

**5.6 Multivariable Function and Vector Functions**
To get the jacobian of a multivariable function, we ask to user to supply the order of `Var` in a list. The following code demoes the process.
```python
from AutoDiff.ForwardAD import Var
>>> x = Var(1)
>>> y = Var(2)
>>> f = x + 2*y
>>> f.get_jacobian_([x, y])
[1.0, 2.0]
```

For vector functions, we ask to user to supply the functions in a list. The following code demoes the process.
```python
from AutoDiff.ForwardAD import MultiFunc
>>> x = Var(1)
>>> F = MultiFunc([x+1, 2*x])
>>> F.get_jacobian()
[[1], [2]]
```

**5.6 Methods in Var**

```python
class Var:
    def __init__(self, a, j):
        self.val = a
        self.jacobian = j

    def abs(self):
      pass

    def neg(self):
      pass

    def __add__(self, other):
      pass

    def __radd__(self, other):
      pass

    def __sub__(self, other):
      pass

    def __rsub__(self, other):
      pass

    def __mul__(self, other):
      pass

    def __rmul__(self, other):
      pass

    def __truediv__(self, other):
      pass

    def __rtruediv__(self, other):
      pass

    def __pow__(self, other):
      pass

    def __rpow__(self, other):
      pass

    def exp(self):
      pass

    def log(self):
      pass

    def sqrt(self):
      pass

    def sin(self):
      pass

    def cos(self):
      pass

    def tan(self):
      pass

    def arctan(self):
      pass

    def arcsin(self):
      pass

    def arccos(self):
      pass

    def sinh(self):
      pass

    def cosh(self):
      pass

    def tanh(self):
      pass
```

# 6 Future Features

For the moment, only single functions of single variables have been included and tested in the submitted code. However, this has been thought through as discussed in the implementation and will be included for the final deliverables.

With the forward mode of automatic differentiation up and running, additional features that rely on differentiation up until machine precision can be developed. This section will discuss the mathematical background of root-finding and optimization, which will be implemented as additional features of the packages for the final version.

**6.1 Root-finding**

First, root-finding arises in many applications in any kind of engineering and science. Consider the general problem in which the purpose is to find that x-value that satisfies the following equation for a certain function f:

$$f(x) = 0$$

There are multiple numerical approaches to iteratively find this x-value, including the famous Newton's method. Here, a certain starting value $x^{(0)}$ is chosen, after which the following equation is used to update $x^{(k)}$ until a particular convergence criterium is reached:

$$x^{(k)} = x^{(k)}-{\frac {f(x^{(k)})}{f'(x^{(k)})}}\$$

Note that the derivative of f with respect to x evaluated at $x^{(k)}$ plays a crucial role in this equation. In many cases, analytical expressions for $f'(x)$ are not available and numerical differentiation methods are needed. With the forward mode of the automatic differentiation working, the Newton's method for root-finding can be implemented. It is foreseen that the the user interacts as follows with this module of the package AutoDiff:

```python
>>> from AutoDiff.ForwardAd import Var
>>> from AutoDiff import Newton
>>> def f(x):
    return Var.sin(x)**(Var.sqrt(x))
>>> x_0 = 0.1
>>> root = Newton(f, x_0)
```

In multiple dimensions the iterative expression becomes:

$$\mathbf {x} ^{(k+1)}=\mathbf {x} ^{(k)}-[\mathbf {H} f(\mathbf {x} ^{(k)})]^{-1}\nabla f(\mathbf {x} ^{(k)}) $$

Here, $\mathbf {x}$ represents a vector and $\mathbf {H}$ and $\nabla f$ the Hessian and gradient of f with respect to $\mathbf {x}$ respectively.

**6.2 Optimization**

A second, highly relevant application of numerical differentiation can be in optimization. In many problems, and most notably in Computer Science and Machine Learning in the last years, is it crucial to efficiently minimize a certain function F in terms of multiple variables $\mathbf {x} = (x_1, ..., x_n)$. Or mathematically:

$$\mathbf {x}_{solution} = \underset{\mathbf {x}}{argmin}   F(\mathbf {x})$$

A common numerical method that iteratively computes the solution for this equation is gradient descent. In this algorithm, an initial guess is iteratively updated by the taking a step in the direction that is alligned with the steepest descent - or in the opposite direction of the gradient at that particular value. This leads to the following equation:

$${\displaystyle \mathbf {x} ^{(k+1)}=\mathbf {x} ^{(k)}-\gamma \nabla F(\mathbf {x} ^{(k)})}$$

Here, $\gamma$ is called the 'learning rate' and corresponds to the size of the step taken in the steepest direction. Again, it is crucial to accurately compute the gradient $\nabla F(\mathbf {x} ^{(k)})$ in every iteration and this is where the automatic differentiation package comes in. For the moment, it is foreseen that the user would interact with the package as follows:

```python
>>> from AutoDiff.ForwardAd import Var
>>> from AutoDiff import GradientDescent as GS
>>> def F(x1, x2, x3):
    return Var.sin(x1)**(Var.sqrt(x2)) + x3
>>> x_0 = [0.1, 0, 2]
>>> gamma = 0.1
>>> argmin, min_F = GS(F, x_0, gamma)
```

Note that argmin stands for the array of values for $\mathbf {x}$ that minimizes F and min_F corresponds of the actual value of the function F at that minimum. Also, it is important to know that teh gradient descent algorithm does not guarantee an absolute minimum, but through clever implementations it is able to find a local minimum that satisfies the criteria in practice.

```python

```