# Volume Sampling vs projection DPP for low rank approximation
## Introduction
#### In this notebook we compare the volume sampling and projection DPP for low rank approximation.
We recall the result proved in the article [DRVW]:\\
Let S be a random subset of k columns of X chosen with probability: $$P(S) = \frac{1}{Z_{k}} det(X_{.,S}^{T}X_{.,S})$$ with $$Z_{k} = \sum\limits_{S \subset [N], |S| = k} det(X_{.,S}^{T}X_{.,S})$$
Then
$$\begin{equation}
E(\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2}) \leq (k+1)\| X - \pi_{k}(X) \|_{Fr}^{2}
\end{equation}$$

We can prove that the volume sampling distribution is a mixture of projection DPPs distributions..., in particular one projection DPP distribution stands out for the problem of low rank approximation: ....\\
For the moment, there is no analytical expression for $$\begin{equation}
E(\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2})
\end{equation}$$ under the distribution of projection DPP.\\
However, we can calculate this quantity using simulation on some matrices representing cloud points with some specific geometric constraints.

Let $$X \in R^{n \times m}$$ a matrix representing a cloud of points.
We can write the SVD of $$X = UDV^{T}$$
In this notebook we investigate the influence of some structures enforced to V and D on the expected error expressed above for different algorithms: Volume Sampling, Projection DPP and the deterministic algorithm.
As for the Volume Sampling distribution, we can express the expected approximation error using only the elements of D. We can test this theoretical property in the next Numerical Study below. However, there is no closed formula (for the moment) for the expected approximation error under Projection DPP distribution. We will see in the Numerical Study section, that this value cannot depends only on the elements of D.

#### References
[DRVW] Deshpande, Amit and Rademacher, Luis and Vempala, Santosh and Wang, Grant - Matrix Approximation and Projective Clustering via Volume Sampling 2006

[BoDr] Boutsidis, Christos and Drineas, Petros  - Deterministic and randomized column selection algorithms for matrices 2014

[] INDERJIT S. DHILLON , ROBERT W. HEATH JR., MA ́TYA ́S A. SUSTIK, AND
JOEL A. TROPP - GENERALIZED FINITE ALGORITHMS FOR CONSTRUCTING HERMITIAN MATRICES
WITH PRESCRIBED DIAGONAL AND SPECTRUM 2005

## I- Generating a cloud of points with geometric constraints
In this simulation we will enforce some structure on the matrix V for two values of the matrix D. While the matrix U will be choosen randomly.
We want to investigate the influence of the profile of the norms of the V_k rows: the k-leverage scores. For this purpose we use an algorithm proposed in the article []: this algorithm outputs a ( dxk)  matrix Q with orthonormal columns and a prescribed profile of the norms of the rows. If we consider the Gram matrix H= QQ^{T}, this boils down to enforce the diagonal of H while keeping its spectrum containing k ones and d-k zeros.
The algorithm proceed as following:
* Initialization of the matrix Q by the rectangular identity
* Apply a Givens Rotation (of dimension d) to the matrix Q: this step will enforce the norm of a row every iteration
* Outputs the resulting matrix when all the rows norms are enforced.

```python
import numpy as np
import pandas as pd
from itertools import combinations
from scipy.stats import binom
import scipy.special
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from IPython.display import display, HTML
from FrameBuilder.eigenstepsbuilder import *
from decimal import *
```

```python
u = np.random.uniform(0,1)
```

```python
u
```

    0.3358120949925849

### I-1- Givens Rotations generators
These functions generate a Givens rotation

```python
def t_func(q_i,q_j,q_ij,l_i,l_j):
    # t in section 3.1 Dhillon (2005)
    delta = np.power(q_ij,2)-(q_i-l_i)*(q_j-l_i)
    if delta<0:
        print(delta)
        print("error sqrt")
    t = q_ij - np.sqrt(delta)
    t = t/(q_j-l_i)
    return t

def G_func(i,j,q_i,q_j,q_ij,l_i,l_j,N):
    # Gitens Rotation
    G=np.eye(N)
    t = t_func(q_i,q_j,q_ij,l_i,l_j)
    c = 1/(np.sqrt(np.power(t,2)+1))
    s = t*c
    G[i,i]=c
    G[i,j]=s
    G[j,i]= -s
    G[j,j]= c
    return G
```

The following function is an implementation of the algorithm [] figuring in the article [] to generate an orthogonal matrix with a prescribed profile of leverage scores.
In fact this is a simplification of the algorithm .....

```python
class Data_Set_Generator:
    def __init__(self, N, d, nu, Sigma):
        self.N = N
        self.d = d
        self.nu = nu
        self.Sigma = Sigma
        self.mean = np.zeros(d)

    def multivariate_t_rvs(self):
        x = np.random.chisquare(self.nu, self.N)/self.nu
        z = np.random.multivariate_normal(self.mean,self.Sigma,(self.N,))
        return self.mean + z/np.sqrt(x)[:,None]
```

```python
def generate_orthonormal_matrix_with_leverage_scores_ES(N,d,lv_scores_vector,versions_number,nn_cardinal_list):

    lambda_vector = np.zeros((N))
    lambda_vector[0:d] = np.ones((d))

    #mu_vector = np.linspace(1, 0.1, num=N)
    #sum_mu_vector = np.sum(mu_vector)
    #mu_vector = d/sum_mu_vector*mu_vector
    Q = np.zeros((N,d))
    previous_Q = np.zeros((versions_number+1,N,d))
    #mu_vector = d/N*np.ones((N,1))
    E = np.zeros((N,N)) #(d,N)
    counter = 0
    for j in nn_cardinal_list:
        print("counter")
        print(counter)
        mu_vector = generate_leverage_scores_vector_with_dirichlet(N,d,j)
        print(np.sum(mu_vector))
        print(mu_vector)
        E_test = get_eigensteps_random(mu_vector,lambda_vector,N,d)
        E_ = np.zeros((d,N+1))
        for i in range(d):
            E_[i,1:N+1] = E_test[i,:]
        print(E_test)
        #F_test = get_F(d,N,np.asmatrix(E_),mu_vector)
        #previous_Q[counter,:,:] = np.transpose(F_test)
        #Q = np.transpose(F_test)
        counter = counter +1
    return Q,previous_Q
```

```python
Q,previous_Q = generate_orthonormal_matrix_with_leverage_scores_ES(20,2,[],3,[18,15,10])
```

    counter
    0
    2.0
    [0.32764, 0.32153, 0.16907, 0.16028, 0.1438, 0.13843, 0.12805, 0.10632, 0.098022, 0.090393, 0.088684, 0.064392, 0.0578, 0.033997, 0.031799, 0.026398, 0.007, 0.0066986, 0.0, 0.0]
    [[ 0.32767245  0.3700502   0.52789205  0.54268118  0.60703057  0.66726853
       0.73849729  0.77766843  0.83109472  0.8743445   0.90949306  0.93896209
       0.95815749  0.97402108  0.99351995  0.99471348  0.99999317  1.          1.
       1.        ]
     [ 0.          0.27916541  0.29034942  0.43577911  0.51516418  0.59343683
       0.6502118   0.71739659  0.76176117  0.80930595  0.86296506  0.89786697
       0.93626258  0.95420218  0.96637759  0.99154724  0.99322139  1.          1.
       1.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]]
    counter
    1
    2.0
    [0.38745, 0.32837, 0.19214, 0.19165, 0.16626, 0.15088, 0.14209, 0.1322, 0.092407, 0.08728, 0.077881, 0.017899, 0.0177, 0.014999, 0.0007, 0.0, 0.0, 0.0, 0.0, 0.0]
    [[ 0.38744452  0.47842014  0.53780069  0.65514809  0.65618587  0.78095882
       0.78355939  0.90963448  0.9309549   0.96915322  0.97944337  0.99278379
       0.9997044   0.99991955  1.          1.          1.          1.          1.
       1.        ]
     [ 0.          0.23734303  0.3700988   0.44436081  0.60949878  0.6357621
       0.77509324  0.78128828  0.85228821  0.90150299  0.969024    0.97381304
       0.98453977  0.99915845  1.          1.          1.          1.          1.
       1.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]]
    counter
    2
    2.0
    [0.64502, 0.39551, 0.31641, 0.24158, 0.23596, 0.053192, 0.035004, 0.031708, 0.029205, 0.016296, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    [[ 0.64502709  0.73823689  0.92150203  0.94621262  0.96221474  0.96531366
       0.98223498  0.99445032  1.0001653   1.          1.          1.          1.
       1.          1.          1.          1.          1.          1.          1.        ]
     [ 0.          0.30251081  0.43560629  0.65211699  0.87203159  0.9223876
       0.9404978   0.95969517  0.98350529  1.          1.          1.          1.
       1.          1.          1.          1.          1.          1.          1.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]
     [ 0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.          0.
       0.          0.          0.          0.          0.          0.        ]]

```python
def generate_leverage_scores_vector_with_dirichlet(d,k,nn_cardinal):
    getcontext().prec = 3
    mu_vector = np.float16(np.zeros((d,)))
    mu_vector_2 = np.float16(np.zeros((d,)))
    not_bounded = 1
    while(not_bounded == 1):
        mu_vector[0:nn_cardinal] = (k*np.random.dirichlet([1]*nn_cardinal, 1))[0]
        mu_vector = np.flip(np.sort(mu_vector),axis = 0)
        if max(mu_vector)<=1:
            not_bounded = 0
    for i in range(nn_cardinal):
        mu_vector_2[i] = round(mu_vector[i],4)
    mu_vector_2 = k*mu_vector_2/np.sum(mu_vector_2)
    return list(mu_vector_2)

```

```python
l = generate_leverage_scores_vector_with_dirichlet(10,2,6)
print(l)
print(np.sum(l))
```

    [0.65674, 0.41479, 0.33838, 0.30518, 0.25195, 0.03299, 0.0, 0.0, 0.0, 0.0]
    2.0

```python

```

```python
def generate_orthonormal_matrix_with_leverage_scores(N,d,lv_scores_vector,versions_number,mode):
    #Transforming an idendity matrix to an orthogonal matrix with prescribed lengths
    Q = np.zeros((N,d))
    previous_Q = np.zeros((versions_number+1,N,d))
    versionning_period = (int)(N/versions_number)
    if mode == 'identity':
        for _ in range(0,d):
            Q[_,_] = 1
    if mode == 'spread':
        nu = 1
        Sigma = np.diag(np.ones(d))
        mean = np.zeros(d)
        x = np.random.chisquare(nu, N)/nu
        z = np.random.multivariate_normal(mean,Sigma,(N,))
        dataset = mean + z/np.sqrt(x)[:,None]
        [Q,_,_] = np.linalg.svd(dataset,full_matrices=False)
        print(np.shape(Q))
    I_sorting =  list(reversed(np.argsort(lv_scores_vector)))

    lv_scores_vector = np.asarray(list(reversed(np.sort(lv_scores_vector))))
    initial_lv_scores_vector = np.diag(np.dot(Q,Q.T))
    I_initial_sorting = list(reversed(np.argsort(initial_lv_scores_vector)))
    initial_lv_scores_vector = np.asarray(list(reversed(np.sort(np.diag(np.dot(Q,Q.T))))))
    #initial_lv_scores_vector =

    Q[I_initial_sorting,:] = Q
    print(lv_scores_vector)
    print(initial_lv_scores_vector)
    delta_lv_scores_vector = lv_scores_vector - initial_lv_scores_vector
    print(delta_lv_scores_vector)
    min_index = next((i for i, x in enumerate(delta_lv_scores_vector) if x>0), None)
    i = min_index-1
    j = min_index
    print(i)
    print(j)
    #if mode == 'identity':
    #    i = d-1
    #    j = d
    #if mode == 'spread':
    #    i = d-2
    #    j = d-1
    v_counter =0
    for t in range(N-1):
        #print(i)
        #print(j)
        delta_i = np.abs(lv_scores_vector[i] - np.power(np.linalg.norm(Q[i,:]),2))
        delta_j = np.abs(lv_scores_vector[j] - np.power(np.linalg.norm(Q[j,:]),2))
        q_i = np.power(np.linalg.norm(Q[i,:]),2)
        q_j = np.power(np.linalg.norm(Q[j,:]),2)
        q_ij = np.dot(Q[i,:],Q[j,:].T)
        l_i = lv_scores_vector[i]
        l_j = lv_scores_vector[j]
        G = np.eye(N)
        if t%versionning_period ==0:
            previous_Q[v_counter,:,:] = Q
            v_counter = v_counter +1
        if delta_i <= delta_j:
            l_k = q_i + q_j -l_i
            G = G_func(i,j,q_i,q_j,q_ij,l_i,l_k,N)
            Q = np.dot(G,Q)
            i = i-1
        else:
            l_k = q_i + q_j -l_j
            G = G_func(i,j,q_j,q_i,q_ij,l_j,l_k,N)
            Q = np.dot(G,Q)
            j = j+1
    previous_Q[versions_number,:,:] = Q
    return Q,previous_Q
```

The following function allows to estimate the leverage scores for an orthogonal matrix Q: the function calculates the diagonoal of the matrix $$Q Q^{T}$$

```python
def estimate_leverage_scores_from_orthogonal_matrix(Q):
    [N,_] = np.shape(Q)
    lv_scores_vector = np.zeros((N,1))
    lv_scores_vector = np.diag(np.dot(Q,np.transpose(Q)))
    lv_scores_vector = np.asarray(list(reversed(np.sort(lv_scores_vector))))
    return lv_scores_vector
def estimate_sum_first_k_leverage_scores(Q,k):
    lv_scores_vector = estimate_leverage_scores_from_orthogonal_matrix(Q)
    res = np.sum(lv_scores_vector[0:k])
    return res
```

## I-2- Extending the orthogonal matrices

For the task of low rank approximation, we have seen that only the information contained in the first right k eigenvectors of the matrix X are relevant. In the previous step we build only the first right k eigenvectors but we still need to complete these orthogonal matrices with d-k columns. We proceed as following:
Generate a random vector (Nx1) using independent standard gaussian variables,
Project this vector in the orthogonal of the span of Q
Normalize the obtained vector after the projection
Extend the matrix Q
Note that this procedure is not the unique way to extend the matrix Q to an orthogonal (Nxd) matrix.

```python
def extend_orthogonal_matrix(Q,d_target):
    [N,d] = np.shape(Q)
    Q_target = np.zeros((N,d))
    Q_target = Q
    delta = d_target - d
    for t in range(delta):
        Q_test = np.random.normal(0, 1, N)
        for _ in range(d):
            Q_test = Q_test - np.dot(Q_test,Q[:,_])*Q[:,_]
        Q_test = Q_test/np.linalg.norm(Q_test)
        Q_test = Q_test.reshape(N,1)
        Q_target = np.append(Q_target,Q_test,1)
    return Q_target

#extended_Q = extend_orthogonal_matrix(Q,r)
```

## I-3 - Constructing a dataset for every extended orthogonal matrix
The previous step allow us to build (N x d) orthogonal matrices such that the extracted (N x k) matrix have a prescribed profile of leverage scores.
Now we construct a cloud of point by assigning a covariance matrix D and a matrix V

```python
def contruct_dataset_from_orthogonal_matrix(multi_Q,N,target_d,cov,mean,versions_number):
    multi_X = np.zeros((versions_number+1,N,real_dim))
    for t in range(versions_number+1):
        test_X = np.random.multivariate_normal(mean, cov, N)
        [U,_,_] = np.linalg.svd(test_X, full_matrices=False)
        Q_test = extend_orthogonal_matrix(multi_Q[t,:,:],target_d)
        multi_X[t,:,:] = np.dot(np.dot(Q_test,cov),U.T).T
    return multi_X

```

## II- Volume sampling vs Projection DPP for low rank approximation
These functions allow to quantify the approximation error:
* approximation_error_function_fro calculate the ratio of the approximation error of a subset of columns to the optimal approximatione error given by the first k left eigenvectors of the matrix X
* expected_approximation_error_for_sampling_scheme calculate the expected value of the ratio of the approximatione error under some sampling distribution

```python
def approximation_error_function_fro(Sigma,k,X,X_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    d = list(Sigma.shape)[0] # the dimension of the matrix X
    Sigma = np.multiply(Sigma,Sigma)  # Sigma power 2 -> we are intersted in the approximation error square
    sigma_S_temp = np.linalg.inv(np.dot(X_S.T,X_S))  # just a temporary matrix to construct the projection matrix
    projection_S = np.dot(np.dot(X_S,sigma_S_temp),X_S.T) # the projection matrix P_S
    res_X = X - np.dot(projection_S,X) # The projection of the matrix X in the orthogonal of S
    approximation_error_ratio = np.power(np.linalg.norm(res_X,'fro'),2)/np.sum(Sigma[k:d])
    # Calculate the apparoximation error ratio
    return approximation_error_ratio
```

```python
def approximation_error_function_spectral(Sigma,k,X,X_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    d = list(Sigma.shape)[0] # the dimension of the matrix X
    Sigma = np.multiply(Sigma,Sigma)  # Sigma power 2 -> we are intersted in the approximation error square
    sigma_S_temp = np.linalg.inv(np.dot(X_S.T,X_S))  # just a temporary matrix to construct the projection matrix
    projection_S = np.dot(np.dot(X_S,sigma_S_temp),X_S.T) # the projection matrix P_S
    res_X = X - np.dot(projection_S,X) # The projection of the matrix X in the orthogonal of S
    approximation_error_ratio = np.power(np.linalg.norm(res_X,ord = 2),2)/np.sum(Sigma[k:k+1])
    # Calculate the apparoximation error ratio
    return approximation_error_ratio
```

```python
def upper_bound_error_function_for_projection_DPP(k,X,X_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    _,sigma_S_temp,_ = np.linalg.svd(X_S, full_matrices=False)  # just a temporary matrix to construct the projection matrix
    trunc_product = np.power(np.prod(sigma_S_temp[0:k-1]),2)
    if np.power(np.prod(sigma_S_temp[0:k]),2) == 0:
        trunc_product = 0
    # Calculate the apparoximation error ratio
    return trunc_product
```

```python
def tight_upper_bound_error_function_fro(k,X,X_S,V_k,V_k_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    d = list(Sigma.shape)[0]
    Sigma = np.multiply(Sigma,Sigma)
    if np.linalg.matrix_rank(V_k_S,0.000001) == k:
        temp_T = np.dot(np.linalg.inv(V_k_S),V_k)
        temp_matrix = X - np.dot(X_S,temp_T)

        return np.power(np.linalg.norm(temp_matrix,'fro'),2)/np.sum(Sigma[k:d])
    else:
        return 0
```

```python
def get_the_matrix_sum_T_S(k,d,V_k,V_d_k):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    #Sigma = np.multiply(Sigma,Sigma)
    #matrices_array = [ np.dot(V_d_k[:,list(comb)],np.dot(np.dot(np.linalg.inv(V_k[:,list(comb)]),np.linalg.inv(V_k[:,list(comb)]))),np.transpose(V_d_k[:,list(comb)])) for comb in combinations(range(d),k) if np.linalg.matrix_rank(V_k[:,list(comb)],0.000001) == k]
    T = np.zeros((d-k,d-k))
    for comb in combinations(range(d),k):
        if np.linalg.matrix_rank(V_k[:,list(comb)],0.0000000001) == k:
            V_k_S_inv = np.linalg.inv(V_k[:,list(comb)])
            V_d_k_S = V_d_k[:,list(comb)]
            V_k_S_inv_2 = np.transpose(np.dot(V_k_S_inv,np.transpose(V_k_S_inv)))
            #T = np.dot(np.dot(np.dot(V_d_k_S,np.dot(V_k_S_inv,np.transpose(V_k_S_inv)))),np.transpose(V_d_k_S)) + T
            T = np.power(np.linalg.det(V_k[:,list(comb)]),2)*np.dot(V_d_k_S,np.dot(V_k_S_inv_2,np.transpose(V_d_k_S))) +T
    return T
```

```python
def tight_approximation_error_fro_for_sampling_scheme(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)
    _,Sigma,V = np.linalg.svd(X, full_matrices=False)
    V_k = V[0:k,:]
    ## Estimating the spectrum of X -> needed in approximation_error_function_fro
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    ## Construct the array of weights: the volumes of UU^{T}_{S,S}
    volumes_array_sum = np.sum(volumes_array)
    ## The normalization constant
    volumes_array = volumes_array/volumes_array_sum
    ## The weigths normalized
    approximation_error_array = [tight_upper_bound_error_function_fro(k,X,X[:,list(comb)],V_k,V_k[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple
    expected_value = np.dot(approximation_error_array,volumes_array)
    ## The expected value of the approximatione error is just the dot product of the two arrays above
    return expected_value
```

```python
def expected_approximation_error_fro_for_sampling_scheme(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    ## Estimating the spectrum of X -> needed in approximation_error_function_fro
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    ## Construct the array of weights: the volumes of UU^{T}_{S,S}
    volumes_array_sum = np.sum(volumes_array)
    ## The normalization constant
    volumes_array = volumes_array/volumes_array_sum
    ## The weigths normalized
    approximation_error_array = [approximation_error_function_fro(Sigma,k,X,X[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple
    expected_value = np.dot(approximation_error_array,volumes_array)
    ## The expected value of the approximatione error is just the dot product of the two arrays above
    return expected_value
```

```python
def expected_approximation_error_spectral_for_sampling_scheme(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    ## Estimating the spectrum of X -> needed in approximation_error_function_fro
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    ## Construct the array of weights: the volumes of UU^{T}_{S,S}
    volumes_array_sum = np.sum(volumes_array)
    ## The normalization constant
    volumes_array = volumes_array/volumes_array_sum
    ## The weigths normalized
    approximation_error_array = [approximation_error_function_spectral(Sigma,k,X,X[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple
    expected_value = np.dot(approximation_error_array,volumes_array)
    ## The expected value of the approximatione error is just the dot product of the two arrays above
    return expected_value
```

```python
def expected_upper_bound_for_projection_DPP(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)

    approximation_error_array = [upper_bound_error_function_for_projection_DPP(k,X,U[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple

    ## The expected value of the approximatione error is just the dot product of the two arrays above
    #return expected_value
    return np.sum(approximation_error_array)
```

## III - Numerical analysis

In this section we use the functions developed previously to investigate the influence of two parameters: the spectrum of X and the k-leverage scores.
For this purpose, we assemble these functionalities in a class allowing fast numerical experiments.

```python
class Numrerical_Analysis_DPP:
    def __init__(self,N,real_dim,r,k,versions_number,mean,cov,lv_scores,versions_list):
        self.N = N
        self.real_dim = real_dim
        self.r = r
        self.k = k
        self.versions_number = versions_number
        self.mean = mean
        self.cov = cov
        self.lv_scores = lv_scores
        self.Q = np.zeros((real_dim,k))
        self.multi_Q = np.zeros((self.versions_number+1,real_dim,k))
        self.X = np.zeros((N,real_dim))
        self.multi_X = np.zeros((self.versions_number+1,N,real_dim))
        #[self.Q,self.multi_Q] = generate_orthonormal_matrix_with_leverage_scores(real_dim,k,lv_scores,versions_number,'identity')
        [self.Q,self.multi_Q] = generate_orthonormal_matrix_with_leverage_scores_ES(self.real_dim,self.k,[],self.versions_number+1,versions_list)
        self.multi_X = contruct_dataset_from_orthogonal_matrix(self.multi_Q,self.N,self.real_dim,self.cov,self.mean,self.versions_number)
    def contruct_dataset_from_orthogonal_matrix_4(self,multi_Q,N,target_d,cov,mean,versions_number):
        test_multi_X = np.zeros((self.versions_number+1,N,real_dim))
        for t in range(self.versions_number+1):
            test_X = np.random.multivariate_normal(mean, cov, N)
            [U,_,_] = np.linalg.svd(test_X, full_matrices=False)
            Q_test = extend_orthogonal_matrix(self.multi_Q[t,:,:],target_d)
            test_multi_X[t,:,:] = np.dot(np.dot(Q_test,cov),U.T).T
        return test_multi_X
    def get_effective_kernel_from_orthogonal_matrix(self):
        test_eff_V = np.zeros((self.versions_number+1,self.real_dim,self.k))
        p_eff_list = self.get_p_eff()
        for t in range(self.versions_number+1):
            test_V = self.multi_Q[t,:,:]
            p_eff = p_eff_list[t]
            diag_Q_t = np.diag(np.dot(test_V,test_V.T))
            #diag_Q_t = list(diag_Q_t[::-1].sort())
            print(diag_Q_t)
            permutation_t = list(reversed(np.argsort(diag_Q_t)))
            print(permutation_t)
            for i in range(self.real_dim):
                if i >p_eff-1:
                    test_V[permutation_t[i],:] = 0
            #Q_test = extend_orthogonal_matrix(self.multi_Q[t,:,:],target_d)
            test_eff_V[t,:,:] = test_V
        return test_eff_V
    def get_expected_error_fro_for_volume_sampling(self):
        ## Calculate the expected error ratio for the Volume Sampling distribution for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            res_list[t] = expected_approximation_error_fro_for_sampling_scheme(test_X,test_X,self.k,self.real_dim)
        return res_list
    def get_expected_error_fro_for_effective_kernel_sampling(self):
        ## Calculate the expected error ratio for the Volume Sampling distribution for every dataset
        res_list = np.zeros(self.versions_number+1)
        test_eff_V = self.get_effective_kernel_from_orthogonal_matrix()
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = test_eff_V[t,:,:].T
            res_list[t] = expected_approximation_error_fro_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_tight_upper_bound_error_fro_for_projection_DPP(self):
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            res_list[t] = tight_approximation_error_fro_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_max_diag_sum_T_matrices(self):
        res_list = np.zeros((self.versions_number+1))
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            _,_,test_V = np.linalg.svd(test_X, full_matrices=False)
            test_V_k = test_V[0:self.k,:]
            test_V_d_k = test_V[self.k:self.real_dim,:]
            res_list[t] = 1+np.max(np.diag(get_the_matrix_sum_T_S(self.k,self.real_dim,test_V_k,test_V_d_k)))
        return res_list
    def get_max_spectrum_sum_T_matrices(self):
        res_list = np.zeros((self.versions_number+1))
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            _,_,test_V = np.linalg.svd(test_X, full_matrices=False)
            test_V_k = test_V[0:self.k,:]
            test_V_d_k = test_V[self.k:self.real_dim,:]
            res_list[t] = 1+np.max(np.diag(get_the_matrix_sum_T_S(self.k,self.real_dim,test_V_k,test_V_d_k)))
        return res_list
    def get_expected_error_fro_for_projection_DPP(self):
        ## Calculate the expected error ratio for the Projection DPP distribution for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            res_list[t] = expected_approximation_error_fro_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_expected_error_spectral_for_volume_sampling(self):
        ## Calculate the expected error ratio for the Volume Sampling distribution for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            res_list[t] = expected_approximation_error_spectral_for_sampling_scheme(test_X,test_X,self.k,self.real_dim)
        return res_list
    def get_expected_error_spectral_for_projection_DPP(self):
        ## Calculate the expected error ratio for the Projection DPP distribution for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            res_list[t] = expected_approximation_error_spectral_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_upper_bound_error_for_projection_DPP(self):
        ## Calculate the expected error ratio for the Projection DPP distribution for every dataset
        #res_list = np.zeros(self.versions_number+1)
        res_list = []
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            #res_list[t] = expected_upper_bound_for_projection_DPP(test_X,test_U,self.k,self.real_dim)
            res_list.append( expected_upper_bound_for_projection_DPP(test_X,test_U,self.k,self.real_dim))
        return res_list
    def get_error_fro_for_deterministic_selection(self):
        ## Calculate the error ratio for the k-tuple selected by the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            lv_scores_vector = np.diag(np.dot(np.transpose(test_U),test_U))
            test_I_k = list(np.argsort(lv_scores_vector)[self.real_dim-self.k:self.real_dim])
            _,test_Sigma,_ = np.linalg.svd(test_X, full_matrices=False)
            res_list[t] = approximation_error_function_fro(test_Sigma,self.k,test_X,test_X[:,test_I_k])
            #res_list.append(test_I_k)
        return res_list
    def get_error_spectral_for_deterministic_selection(self):
        ## Calculate the error ratio for the k-tuple selected by the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            lv_scores_vector = np.diag(np.dot(np.transpose(test_U),test_U))
            test_I_k = list(np.argsort(lv_scores_vector)[self.real_dim-self.k:self.real_dim])
            _,test_Sigma,_ = np.linalg.svd(test_X, full_matrices=False)
            res_list[t] = approximation_error_function_spectral(test_Sigma,self.k,test_X,test_X[:,test_I_k])
            #res_list.append(test_I_k)
        return res_list
    def get_p_eff(self):
        ## A function that calculate the p_eff.
        ## It is a measure of the concentration of V_k. This is done for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            diag_Q_t = np.diag(np.dot(self.multi_Q[t,:,:],self.multi_Q[t,:,:].T))
            #diag_Q_t = list(diag_Q_t[::-1].sort())
            diag_Q_t = list(np.sort(diag_Q_t)[::-1])
            p = self.real_dim
            print(diag_Q_t)
            while np.sum(diag_Q_t[0:p-1]) > float(self.k-1.0/2):
                p = p-1
            res_list[t] = p
        return res_list
    def get_sum_k_leverage_scores(self):
        ## A function that calculate the k-sum: the sum of the first k k-leverage scores. It is a measure of the concentration of V_k
        ## This is done for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            res_list[t] = estimate_sum_first_k_leverage_scores(self.multi_Q[t,:,:],self.k)
        return res_list
    def get_deterministic_upper_bound(self):
        ## A function that calculate the theoretical upper bound for the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number+1)
        for t in range(self.versions_number+1):
            res_list[t] = 1/(1+estimate_sum_first_k_leverage_scores(self.multi_Q[t,:,:],self.k)-self.k)
        return res_list
    def get_alpha_sum_k_leverage_scores(self,alpha):
        ## A function that calculate the theoretical upper bound for the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number+1)
        #k_l = self.get_sum_k_leverage_scores()
        for t in range(self.versions_number+1):
            k_l = estimate_leverage_scores_from_orthogonal_matrix(self.multi_Q[t,:,:])[0:k]
            func_k = np.power(np.linspace(1, k, num=k),alpha)
            res_list[t] = np.dot(func_k,k_l)
        return res_list
```

```python

```

### III- 0 Parameters of the simultations

```python
## The dimensions of the design matrix X
N = 100       # The number of observations in the dataset
real_dim = 20 # The dimension of the dataset
## The low rank paramters
k = 2         # The rank of the low rank approximation
## The covariance matrix parameters
r = 6         # Just a parameter to control the number of non trivial singular values in the covariance matrix
mean = np.zeros((real_dim))    # The mean vector useful to generate U (X = UDV^T)
cov_test = 0.01*np.ones((real_dim-r))  # The "trivial" singular values in the covariance matrix (there are real_dim-r)
## The paramters of the matrix V
versions_number = 3 # The number of orthogonal matrices (and therefor datasets) (-1) generated by the algorithm above
lv_scores_vector = k/real_dim*np.ones(real_dim)  # The vector of leverage scores (the last one)
```

```python
l = [1,5,2,10]
ll = list(reversed(np.argsort(l)))
```

```python
ll
```

    [3, 1, 2, 0]

### III-1 The influence of the spectrum

In this subsection we compare the Volume Sampling distribution to the projection DPP distribution and the deterministic algorithm of []  for different profiles of the spectrum with k-leverage scores profile fixed. In other words, if we note $$X = UDV^{T}$$ We keep V_{k} constant and we investigate the effect of D.

#### III-1-1 The case of a non-projection spectrum
We mean by a projection spectrum matrix, a matrix with equal the first k singular values.
We observe that the two distributions are very similar.... \todo{reword}

```python
cov_1 = np.diag(np.concatenate(([100,100,1,1,1,1],cov_test)))
versions_list = [20,19,18,17]
NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,versions_list)

projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
deterministic_selection_res_fro_1 = NAL_1.get_error_fro_for_deterministic_selection()
projection_DPP_res_spectral_1 = NAL_1.get_expected_error_spectral_for_projection_DPP()
volume_sampling_res_spectral_1 = NAL_1.get_expected_error_spectral_for_volume_sampling()
deterministic_selection_res_spectral_1 = NAL_1.get_error_spectral_for_deterministic_selection()
effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#sss = NAL_1.get_effective_kernel_from_orthogonal_matrix()
#p_eff_res_1 = NAL_1.get_p_eff()

upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
deterministic_upper_bound_res_1 = NAL_1.get_deterministic_upper_bound()

expected_upper_bound_res_1 = NAL_1.get_upper_bound_error_for_projection_DPP()
multi_Q_1 = NAL_1.multi_Q[1,:,:].T

```

    counter
    0
    2.0
    [0.35767, 0.2356, 0.22803, 0.1825, 0.11902, 0.11578, 0.11469, 0.10797, 0.084717, 0.081909, 0.073425, 0.069092, 0.064392, 0.033386, 0.028198, 0.027405, 0.027206, 0.0233, 0.015503, 0.010399]
    counter
    1
    2.0
    [0.39722, 0.2666, 0.26074, 0.19592, 0.18884, 0.087891, 0.077515, 0.076111, 0.075623, 0.06897, 0.065918, 0.056885, 0.056793, 0.03479, 0.027206, 0.019608, 0.015404, 0.014297, 0.013702, 0.0]
    counter
    2
    2.0
    [0.30811, 0.2428, 0.21558, 0.19116, 0.1665, 0.11578, 0.1156, 0.1004, 0.087891, 0.080383, 0.078918, 0.071716, 0.061005, 0.051788, 0.051392, 0.0354, 0.018494, 0.0073013, 0.0, 0.0]
    counter
    3
    2.0
    [0.41357, 0.36157, 0.17798, 0.1604, 0.15686, 0.14661, 0.10059, 0.07959, 0.076599, 0.074585, 0.069214, 0.052612, 0.032806, 0.032593, 0.031204, 0.022202, 0.011002, 0.0, 0.0, 0.0]
    [0.35777688026428223, 0.23560344846304779, 0.22806371259223471, 0.18255712899271787, 0.11900794670561936, 0.11568677300530091, 0.11476263689791259, 0.10777084186787721, 0.084861728770511319, 0.081967536928561888, 0.073448336217099869, 0.069097723206169998, 0.064390613100475599, 0.033330586092320313, 0.028234252443261599, 0.027369456022050979, 0.027157949171869269, 0.023251832128326343, 0.015405204729490913, 0.010364199016915527]
    [0.39736771583557129, 0.26664229681671597, 0.26072914064963071, 0.19586754570544651, 0.18898419362053762, 0.087822556525339532, 0.077348376281795264, 0.076264117114580934, 0.075689501901983994, 0.068922982710907174, 0.066088051649501336, 0.056825979254394067, 0.056824325235779481, 0.034826121032323423, 0.027234862252278693, 0.019550907116733816, 0.014896496977231733, 0.014455927041064151, 0.013700896349698524, 0.0]
    [0.30822014808654785, 0.24278500796682609, 0.21565109368410726, 0.19105632200162725, 0.16655734427974439, 0.11591341067637717, 0.11556110747004687, 0.10042337024137872, 0.087865207190519559, 0.080251032112411413, 0.079005482615102485, 0.071770978962218943, 0.061071742280536245, 0.051816396826393572, 0.051440847198793362, 0.035266111773377795, 0.018421231645940662, 0.0070718982061277452, 0.0, 0.0]
    [0.41353440284729004, 0.36173493034944848, 0.17776079952536425, 0.16039998190684215, 0.15696485196624904, 0.14654729194664731, 0.10080900324316089, 0.079611251539356484, 0.076364104972679869, 0.074347356934501729, 0.069224585634353145, 0.052798337801564392, 0.033010513017684769, 0.032357706838145539, 0.031246401365466962, 0.022231815988071765, 0.010967017038168816, 0.0, 0.0, 0.0]
    [ 0.35777688  0.23560345  0.22806371  0.18255713  0.11900795  0.11568677
      0.11476264  0.10777084  0.08486173  0.08196754  0.07344834  0.06909772
      0.06439061  0.03333059  0.02823425  0.02736946  0.02715795  0.02325183
      0.0154052   0.0103642 ]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
    [ 0.39736772  0.2666423   0.26072914  0.19586755  0.18898419  0.08782256
      0.07734838  0.07626412  0.0756895   0.06892298  0.06608805  0.05682598
      0.05682433  0.03482612  0.02723486  0.01955091  0.0148965   0.01445593
      0.0137009   0.        ]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
    [ 0.30822015  0.24278501  0.21565109  0.19105632  0.16655734  0.11591341
      0.11556111  0.10042337  0.08786521  0.08025103  0.07900548  0.07177098
      0.06107174  0.0518164   0.05144085  0.03526611  0.01842123  0.0070719   0.
      0.        ]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
    [ 0.4135344   0.36173493  0.1777608   0.16039998  0.15696485  0.14654729
      0.100809    0.07961125  0.0763641   0.07434736  0.06922459  0.05279834
      0.03301051  0.03235771  0.0312464   0.02223182  0.01096702  0.          0.
      0.        ]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 17, 19]

```python
p_eff_res_1 = NAL_1.get_p_eff()
eff_kernel_upper_bound_1 = 1+ (p_eff_res_1-k)/(real_dim-k)*(k+1)
```

    [0.35777688026428223, 0.23560344846304779, 0.22806371259223471, 0.18255712899271787, 0.11900794670561936, 0.11568677300530091, 0.11476263689791259, 0.10777084186787721, 0.084861728770511319, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    [0.39736771583557129, 0.26664229681671597, 0.26072914064963071, 0.19586754570544651, 0.18898419362053762, 0.087822556525339532, 0.077348376281795264, 0.076264117114580934, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    [0.30822014808654785, 0.24278500796682609, 0.21565109368410726, 0.19105632200162725, 0.16655734427974439, 0.11591341067637717, 0.11556110747004687, 0.10042337024137872, 0.087865207190519559, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    [0.41353440284729004, 0.36173493034944848, 0.17776079952536425, 0.16039998190684215, 0.15696485196624904, 0.14654729194664731, 0.10080900324316089, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

```python
eff_kernel_upper_bound
```

    array([ 1.66666667,  2.        ,  2.        ,  1.83333333])

```python

print(k*(real_dim-k+1))
sum_T_matrices =  NAL_1.get_sum_T_matrices()
```

```python
pd_1 = pd.DataFrame(
    {'k-sum (ratio)': sum_U_res_1/k,
     'p_eff':p_eff_res_1,
     'alpha k-sum': alpha_sum_res_1,
     'Expected Upper Bound for Projection DPP': expected_upper_bound_res_1,
     'Volume Sampling(Fro)': volume_sampling_res_fro_1,
     'Projection DPP(Fro)': projection_DPP_res_fro_1,
     'Effective kernel(Fro)' : effective_kernel_sampling_res_fro_1,
     'Effective kernel upper bound (Fro)':eff_kernel_upper_bound_1,
     'Very sharp approximation of Projection DPP(Fro)': upper_tight_bound_projection_DPP_res_fro_1,
     'Deterministic Algorithm(Fro)': deterministic_selection_res_fro_1,
     'Volume Sampling(Spectral)': volume_sampling_res_spectral_1,
     'Projection DPP(Spectral)': projection_DPP_res_spectral_1,
     'Deterministic Algorithm(Spectral)': deterministic_selection_res_spectral_1,
     'Deterministic Upper Bound': deterministic_upper_bound_res_1
    })
pd_1 = pd_1[['k-sum (ratio)','p_eff', 'alpha k-sum','Expected Upper Bound for Projection DPP','Volume Sampling(Fro)','Projection DPP(Fro)','Effective kernel(Fro)','Effective kernel upper bound (Fro)','Very sharp approximation of Projection DPP(Fro)','Deterministic Algorithm(Fro)','Volume Sampling(Spectral)','Projection DPP(Spectral)','Deterministic Algorithm(Spectral)','Deterministic Upper Bound']]
```

```python
p_eff_res_1[3]
```

    7.0

```python
#'1+Largest eigenvalue of sum_T': sum_T_matrices,
```

```python
pd_1
```

<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>k-sum (ratio)</th>
      <th>p_eff</th>
      <th>alpha k-sum</th>
      <th>Expected Upper Bound for Projection DPP</th>
      <th>Volume Sampling(Fro)</th>
      <th>Projection DPP(Fro)</th>
      <th>Effective kernel(Fro)</th>
      <th>Effective kernel upper bound (Fro)</th>
      <th>Very sharp approximation of Projection DPP(Fro)</th>
      <th>Deterministic Algorithm(Fro)</th>
      <th>Volume Sampling(Spectral)</th>
      <th>Projection DPP(Spectral)</th>
      <th>Deterministic Algorithm(Spectral)</th>
      <th>Deterministic Upper Bound</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.296690</td>
      <td>9.0</td>
      <td>0.828984</td>
      <td>10.237879</td>
      <td>2.998458</td>
      <td>2.930497</td>
      <td>2.171672</td>
      <td>2.166667</td>
      <td>2.202830</td>
      <td>1.377183</td>
      <td>5.555760</td>
      <td>5.382416</td>
      <td>1.747463</td>
      <td>-2.459301</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.332005</td>
      <td>8.0</td>
      <td>0.930652</td>
      <td>9.057958</td>
      <td>2.998448</td>
      <td>2.934671</td>
      <td>1.744742</td>
      <td>2.000000</td>
      <td>1.758297</td>
      <td>1.106016</td>
      <td>6.043712</td>
      <td>5.868467</td>
      <td>1.157805</td>
      <td>-2.976279</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.275503</td>
      <td>9.0</td>
      <td>0.793790</td>
      <td>10.189300</td>
      <td>2.998433</td>
      <td>2.462686</td>
      <td>2.417697</td>
      <td>2.166667</td>
      <td>2.424515</td>
      <td>1.661438</td>
      <td>4.939381</td>
      <td>3.714388</td>
      <td>2.156968</td>
      <td>-2.227197</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.387635</td>
      <td>7.0</td>
      <td>1.137004</td>
      <td>7.608430</td>
      <td>2.998474</td>
      <td>2.680769</td>
      <td>1.723236</td>
      <td>1.833333</td>
      <td>1.724160</td>
      <td>1.181214</td>
      <td>6.461901</td>
      <td>5.506943</td>
      <td>1.239377</td>
      <td>-4.449771</td>
    </tr>
  </tbody>
</table>
</div>

#### Observations:
* The expected error is always smaller under the Projection DPP distribution compared to the Volume Sampling distribution.
* The expected error for the Volume Sampling distribution is constant for a contant D
* However the expected error for the Projection DPP distribution depends on the k-sum
* For X_0 and X_1, the profile of the k-leverage scores are highly concentrated (k-sum > k-1) thus epsilon is smaller than 1, in this regime the determinstic algorithm have the lower approximation error and it performs better than expected (the theoretical bound is 1/(1-epsilon).
* However, for the other datasets, the (k-sum < k-1) thus epsilon >1 and the deterministic algorithm have no guarantee in this regime: we observe that the approximation error for the deterministic algorithm can be very high in this regime.

#### Recall:
We recall here some geometrical properties of the matrices $$X_i$$
$$X_i = UD_{j}V_{i}$$
Where for every i, the first k columns of $$V_{i}$$ are the $$Q_{i}$$ while the other columns are gernerated randomly

```python
previous_Q = NAL_1.multi_Q
lv_0 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[0,:,:])
lv_1 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[1,:,:])
lv_2 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[2,:,:])
lv_3 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[3,:,:])
lv_4 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[4,:,:])
lv_5 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[5,:,:])
index_list = list(range(real_dim))
```

In this example the objective is Q and the initialization is Q_0 (the rectangular identity)
We have with respect to the Schur-order (or the majorization):
$$Q = Q_5 \prec_{S} Q_4 \prec_{S} Q_3 \prec_{S} Q_2 \prec_{S} Q_1 \prec_{S} Q_0 $$

```python
plt.plot(index_list[0:10], lv_0[0:10], 'c--',index_list[0:10], lv_1[0:10], 'k--', index_list[0:10], lv_2[0:10], 'r--', index_list[0:10], lv_3[0:10], 'b--',index_list[0:10], lv_4[0:10], 'g--',index_list[0:10], lv_5[0:10], 'y--')
plt.xlabel('index')
plt.ylabel('leverage score')
cyan_patch = mpatches.Patch(color='cyan', label='Q_0')
black_patch = mpatches.Patch(color='black', label='Q_1')
red_patch = mpatches.Patch(color='red', label='Q_2')
blue_patch = mpatches.Patch(color='blue', label='Q_3')
green_patch = mpatches.Patch(color='green', label='Q_4')
yellow_patch = mpatches.Patch(color='yellow', label='Q = Q_5')
plt.legend(handles=[cyan_patch,black_patch,red_patch,blue_patch,green_patch,yellow_patch])

plt.show()
```

#### III-1-2 The case of a projection spectrum
We mean by a projection spectrum matrix, a matrix with equal the first k singular values.
We observe that the two distributions are very similar.... \todo{reword}

```python
cov_2 = np.diag(np.concatenate(([1000,1000,1000,1,0.1],cov_test)))
NAL_2 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_2,lv_scores_vector)
projection_DPP_res_2 = NAL_2.get_expected_error_for_projection_DPP()
volume_sampling_res_2 = NAL_2.get_expected_error_for_volume_sampling()
deterministic_selection_res_2 = NAL_1.get_error_for_deterministic_selection()
sum_U_res_2 = NAL_2.get_sum_k_leverage_scores()
deterministic_upper_bound_res_2 = NAL_2.get_deterministic_upper_bound()

results = [["Dataset","Using Volume Sampling","Using Projection DPP","k-sum","1/(1-epsilon)","Using Deterministic Algorithm"],["X_0",volume_sampling_res_2[0],projection_DPP_res_2[0],sum_U_res_2[0],deterministic_upper_bound_res_2[0],deterministic_selection_res_2[0]],["X_1",volume_sampling_res_2[1],projection_DPP_res_2[1],sum_U_res_2[1],deterministic_upper_bound_res_2[1],deterministic_selection_res_2[1]],
          ["X_2",volume_sampling_res_2[2],projection_DPP_res_2[2],sum_U_res_2[2],deterministic_upper_bound_res_2[2],deterministic_selection_res_2[2]],["X_3",volume_sampling_res_2[3],projection_DPP_res_2[3],sum_U_res_2[3],deterministic_upper_bound_res_2[3],deterministic_selection_res_2[3]],["X_4",volume_sampling_res_2[4],projection_DPP_res_2[4],sum_U_res_2[4],deterministic_upper_bound_res_2[4],deterministic_selection_res_2[4]],["X_5",volume_sampling_res_2[5],projection_DPP_res_2[5],sum_U_res_2[5],deterministic_upper_bound_res_2[5],deterministic_selection_res_2[5]]]
display(HTML(
    '<center><b>The expected approximation error (divided by the optimal error) according to a sampling scheme for different distribution</b><br><table><tr>{}</tr></table>'.format(
        '</tr><tr>'.join(
            '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in results)
        )
 ))
```

### III-2 The influence of the "spread" of V
In this section we investigate the influence of the "spread" (to be defined formally) of the cloud of points. We can change this "spread" by changing the initialization of the generator of orthogonal matrices: we replace the rectangular identity by "other" orthogonal matrices.
Technically, this boils down to change the generator mode in the constructor call from "nonspread" to "spread".

```python

```

```python
np.power(np.linspace(1, k, num=k),2)
```

```python

```

```python
matrices_array = [ np.zeros((4,4)) for comb in combinations(range(5),4)]

```

```python
matrix_sum = np.sum(matrices_array)
```

```python
matrix_sum
```

    0.0

```python
matrices_array
```

    [array([[ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]]), array([[ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.],
            [ 0.,  0.,  0.,  0.]])]

```python

```


## Summary
This notebook contains:
- torch implementations of a few linear algebra techniques:
    - forward- and back-solving
    - LDLt decomposition
    - QR decomposition via Householder reflections

- initial implementations of secure linear regression and Jonathan Bloom's [DASH](https://github.com/jbloom22/DASH/) that leverage PySyft for secure computation.

These implementations linear regression and DASH are not currently strictly secure, in that a few final steps are performed on the local worker for now. That's because our implementations of LDLt decomposition, QR decomposition, etc. don't quite work for the PySyft `AdditiveSharingTensor` just yet. They definitely do in principle (because they're compositions of operations the SPDZ supports), but there are still a few details to hammer out.

## Contents
[Ordinary least squares regression and LDLt decomposition](#OLSandLDLt)
* [LDLt decomposition, forward/back-solving](#LDLt)
* [Secure linear regression example](#OLS)

[DASH](#dashqr)
* [QR decomposition via Householder transforms](#qr)
* [DASH example](#dash)

```python
import numpy as np
import torch as th
import syft as sy
from scipy import stats
```

    WARNING: Logging before flag parsing goes to stderr.
    W0710 23:13:43.013911 4542494144 secure_random.py:26] Falling back to insecure randomness since the required custom op could not be found for the installed version of TensorFlow. Fix this by compiling custom ops. Missing file was '/Users/andrew/.virtualenvs/pysyft/lib/python3.7/site-packages/tf_encrypted-0.5.6-py3.7-macosx-10.14-x86_64.egg/tf_encrypted/operations/secure_random/secure_random_module_tf_1.14.0.so'
    W0710 23:13:43.023926 4542494144 deprecation_wrapper.py:119] From /Users/andrew/.virtualenvs/pysyft/lib/python3.7/site-packages/tf_encrypted-0.5.6-py3.7-macosx-10.14-x86_64.egg/tf_encrypted/session.py:26: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

```python
sy.create_sandbox(globals())
```

    Setting up Sandbox...
    	- Hooking PyTorch
    	- Creating Virtual Workers:
    		- bob
    		- theo
    		- jason
    		- alice
    		- andy
    		- jon
    	Storing hook and workers as global variables...
    	Loading datasets from SciKit Learn...
    		- Boston Housing Dataset
    		- Diabetes Dataset
    		- Breast Cancer Dataset
    	- Digits Dataset
    		- Iris Dataset
    		- Wine Dataset
    		- Linnerud Dataset
    	Distributing Datasets Amongst Workers...
    	Collecting workers into a VirtualGrid...
    Done!

# Ordinary least squared regression and LDLt decomposition

## LDLt decomposition, forward/back-solving

These are torch implementations of basic linear algebra routines we'll use to perform regression (and also in parts of the next section).
- Forward/back-solving allows us to solve triangular linear systems efficiently and stably.
- LDLt decomposition lets us write symmetric matrics as a product LDL^t where L is lower-triangular and D is diagonal (^t denotes transpose). It performs a role similar to Cholesky decomposition (which is normally available as method of a torch tensor), but doesn't require computing square roots. This makes makes LDLt a better fit for the secure setting.

```python
def _eye(n):
    """th.eye doesn't seem to work after hooking torch, so just adding
    a workaround for now.
    """
    return th.FloatTensor(np.eye(n))

def ldlt_decomposition(x):
    """Decompose the square, symmetric, full-rank matrix X as X = LDL^t, where
        - L is upper triangular
        - D is diagonal.
    """
    n, _ = x.shape
    l, diag = _eye(n), th.zeros(n).float()

    for j in range(n):
        diag[j] = x[j, j] - (th.sum((l[j, :j] ** 2) * diag[:j]))
        for i in range(j + 1, n):
            l[i, j] = (x[i, j] - th.sum(diag[:j] * l[i, :j] * l[j, :j])) / diag[j]

    return l, th.diag(diag), l.transpose(0, 1)

def back_solve(u, y):
    """Solve Ux = y for U a square, upper triangular matrix of full rank"""
    n = u.shape[0]
    x = th.zeros(n)
    for i in range(n - 1, -1, -1):
        x[i] = (y[i] - th.sum(u[i, i+1:] * x[i+1:])) / u[i, i]

    return x.reshape(-1, 1)

def forward_solve(l, y):
    """Solve Lx = y for L a square, lower triangular matrix of full rank."""
    n = l.shape[0]
    x = th.zeros(n)
    for i in range(0, n):
        x[i] = (y[i] - th.sum(l[i, :i] * x[:i])) / l[i, i]

    return x.reshape(-1, 1)

def invert_triangular(t, upper=True):
    """
    Invert by repeated forward/back-solving.
    TODO: -Could be made more efficient with vectorized implementation of forward/backsolve
          -detection and validation around triangularity/squareness
    """
    solve = back_solve if upper else forward_solve
    t_inv = th.zeros_like(t)
    n = t.shape[0]
    for i in range(n):
        e = th.zeros(n, 1)
        e[i] = 1.
        t_inv[:, [i]] = solve(t, e)
    return t_inv

def solve_symmetric(a, y):
    """Solve the linear system Ax = y where A is a symmetric matrix of full rank."""
    l, d, lt = ldlt_decomposition(a)

    # TODO: more efficient to just extract diagonal of d as 1D vector and scale?
    x_ = forward_solve(l.mm(d), y)
    return back_solve(lt, x_)

```

```python
"""
Basic tests for LDLt decomposition.
"""

def _assert_small(x, failure_msg=None, threshold=1E-5):
    norm = x.norm()
    assert norm < threshold, failure_msg

def test_ldlt_case(a):
    l, d, lt = ldlt_decomposition(a)
    _assert_small(l - lt.transpose(0, 1))
    _assert_small(l.mm(d).mm(lt) - a, 'Decomposition is inaccurate.')
    _assert_small(l - th.tril(l), 'L is not lower triangular.')
    _assert_small(th.triu(th.tril(d)) - d, 'D is not diagonal.')
    print(f'PASSED for {a}')

def test_solve_symmetric_case(a, x):
    y = a.mm(x)
    _assert_small(solve_symmetric(a, y) - x)
    print(f'PASSED for {a}, {x}')

a = th.tensor([[1, 2, 3],
               [2, 1, 2],
               [3, 2, 1]]).float()

x = th.tensor([1, 2, 3]).float().reshape(-1, 1)

test_ldlt_case(a)
test_solve_symmetric_case(a, x)

```

    PASSED for tensor([[1., 2., 3.],
            [2., 1., 2.],
            [3., 2., 1.]])
    PASSED for tensor([[1., 2., 3.],
            [2., 1., 2.],
            [3., 2., 1.]]), tensor([[1.],
            [2.],
            [3.]])

## Secure linear regression example

#### Problem
We're solving
$$ \min_\beta \|X \beta - y\|_2 $$
in the situation where the data $(X, y)$ is horizontally partitioned (each worker $w$ owns chunks $X_w, y_w$ of the rows of $X$ and $y$).

#### Goals
We want to do this
* securely
* without network overhead or MPC-related costs that scale with the number of rows of $X$.

#### Plan

1. (**local plaintext compression**): each worker locally computes $X_w^t X_w$ and $X_w^t y_w$ in plain text. This is the only step that depends on the number of rows of X, and it's performed in plaintext.
2. (**secure summing**): securely compute the sums $$\begin{align}X^t X &= \sum_w X^t_w X_w \\ X^t y &= \sum_w X^t_w y_w \end{align}$$ as an AdditiveSharingTensor. Some worker or other party (here the local worker) will have a pointers to those two AdditiveSharingTensors.
3. (**secure solve**): We can then solve $X^tX\beta = X^ty$ for $\beta$ by a sequence of operations on those pointers (specifically, we apply `solve_symmetric` defined above).

#### Example data:
The correct $\beta$ is $[1, 2, -1]$

```python
X = th.tensor(10 * np.random.randn(30000, 3))
y = (X[:, 0] + 2 * X[:, 1] - X[:, 2]).reshape(-1, 1)
```

Split the data into chunks and send a chunk to each worker, storing pointers to chunks in two `MultiPointerTensor`s.

```python
workers = [alice, bob, theo]
crypto_provider = jon
chunk_size = int(X.shape[0] / len(workers))

def _get_chunk_pointers(data, chunk_size, workers):
    return [
        data[(i * chunk_size):((i+1)*chunk_size), :].send(worker)
        for i, worker in enumerate(workers)
    ]

X_ptrs = sy.MultiPointerTensor(
    children=_get_chunk_pointers(X, chunk_size, workers))
y_ptrs = sy.MultiPointerTensor(
    children=_get_chunk_pointers(y, chunk_size, workers))
```

### local compression
This is the only step that depends on the number of rows of $X, y$, and it's performed locally on each worker in plain text. The result is two `MultiPointerTensor`s with pointers to each workers' summand of $X^tX$ (or $X^ty$).

```python
Xt_ptrs = X_ptrs.transpose(0, 1)

XtX_summand_ptrs = Xt_ptrs.mm(X_ptrs)
Xty_summand_ptrs = Xt_ptrs.mm(y_ptrs)
```

### secure sum
We add those summands up in two steps:
- share each summand among all other workers
- move the resulting pointers to one place (here just the local worker) and add 'em up.

```python
def _generate_shared_summand_pointers(
        summand_ptrs,
        workers,
        crypto_provider):

    for worker_id, summand_pointer in summand_ptrs.child.items():
        shared_summand_pointer = summand_pointer.fix_precision().share(
            *workers, crypto_provider=crypto_provider)
        yield shared_summand_pointer.get()
```

```python
XtX_shared = sum(
    _generate_shared_summand_pointers(
        XtX_summand_ptrs, workers, crypto_provider))

Xty_shared = sum(_generate_shared_summand_pointers(
    Xty_summand_ptrs, workers, crypto_provider))
```

### secure solve
The coefficient $\beta$ is the solution to
$$X^t X \beta = X^t y$$

We solve for $\beta$ using `solve_symmetric`. Critically, this is a composition of linear operations that should be supported by `AdditiveSharingTensor`. Unlike the classic Cholesky decomposition, the $LDL^t$ decomposition in step 1 does not involve taking square roots, which would be challenging.

**TODO**: there's still some additional work required to get `solve_symmetric` working for `AdditiveSharingTensor`, so we're performing the final linear solve publicly for now.

```python
beta = solve_symmetric(XtX_shared.get().float_precision(), Xty_shared.get().float_precision())
```

```python
beta
```

    tensor([[ 1.0000],
            [ 2.0000],
            [-1.0000]])

# DASH and QR-decomposition

## QR decomposition

A $m \times n$ real matrix $A$ with $m \geq n$ can be written as $$A = QR$$ for $Q$ orthogonal and $R$ upper triangular. This is helpful in solving systems of equations, among other things. It is also central to the compression idea of [DASH](https://arxiv.org/pdf/1901.09531.pdf).

```python
"""
Full QR decomposition via Householder transforms,
following Numerical Linear Algebra (Trefethen and Bau).
"""

def _apply_householder_transform(a, v):
    return a - 2 * v.mm(v.transpose(0, 1).mm(a))

def _build_householder_matrix(v):
    n = v.shape[0]
    u = v / v.norm()
    return _eye(n) - 2 * u.mm(u.transpose(0, 1))

def _householder_qr_step(a):
    x = a[:, 0].reshape(-1, 1)
    alpha = x.norm()
    u = x.copy()

    # note: can get better stability by multiplying by sign(u[0, 0])
    # (where sign(0) = 1); is this supported in the secure context?
    u[0, 0] += u.norm()

    # is there a simple way of getting around computing the norm twice?
    u /= u.norm()
    a = _apply_householder_transform(a, u)

    return a, u

def _recover_q(householder_vectors):
    """
    Build the matrix Q from the Householder transforms.
    """
    n = len(householder_vectors)

    def _apply_transforms(x):
        """Trefethen and Bau, Algorithm 10.3"""
        for k in range(n-1, -1, -1):
            x[k:, :] = _apply_householder_transform(
                x[k:, :],
                householder_vectors[k])
        return x

    m = householder_vectors[0].shape[0]
    n = len(householder_vectors)
    q = th.zeros(m, m)

    # Determine q by evaluating it on a basis
    for i in range(m):
        e = th.zeros(m, 1)
        e[i] = 1.
        q[:, [i]] = _apply_transforms(e)

    return q

def qr(a, return_q=True):
    """
    Args:
        a: shape (m, n), m >= n
        return_q: bool, whether to reconstruct q
    Returns:
        orthogonal q of shape (m, m) (None if return_q is False)
        upper-triangular of shape (m, n)
    """
    m, n = a.shape
    assert m >= n, \
        f"Passed a of shape {a.shape}, must have a.shape[0] >= a.shape[1]"

    r = a.copy()
    householder_unit_normal_vectors = []

    for k in range(n):
        r[k:, k:], u = _householder_qr_step(r[k:, k:])
        householder_unit_normal_vectors.append(u)
    if return_q:
        q = _recover_q(householder_unit_normal_vectors)
    else:
        q = None
    return q, r

```

```python
"""
Basic tests for QR decomposition
"""

def _test_qr_case(a):

    q, r = qr(a)

    # actually have QR = A
    _assert_small(q.mm(r) - a, "QR = A failed")

    # Q is orthogonal
    m, _ = a.shape
    _assert_small(
        q.mm(q.transpose(0, 1)) - _eye(m),
        "QQ^t = I failed"
    )

    # R is upper triangular
    lower_triangular_entries = th.tensor([
        r[i, j].item() for i in range(r.shape[0])
             for j in range(i)])

    _assert_small(
        lower_triangular_entries,
        "R is not upper triangular"
    )

    print(f"PASSED for \n{a}\n")

def test_qr():
    _test_qr_case(
        th.tensor([[1, 0, 1],
                   [1, 1, 0],
                   [0, 1, 1]]).float()
    )

    _test_qr_case(
        th.tensor([[1, 0, 1],
                   [1, 1, 0],
                   [0, 1, 1],
                   [1, 1, 1],]).float()
    )

test_qr()
```

    PASSED for
    tensor([[1., 0., 1.],
            [1., 1., 0.],
            [0., 1., 1.]])

    PASSED for
    tensor([[1., 0., 1.],
            [1., 1., 0.],
            [0., 1., 1.],
            [1., 1., 1.]])

## DASH implementation

We follow https://github.com/jbloom22/DASH/.

The overall structure is roughly analogous to the linear regression example above.

- There's a local compression step that's performed separately on each worker in plaintext.
- We leverage PySyft's SMCP features to perform secure summation.
- For now, the last few steps are performed by a single player (the local worker).
    - Again, this could be performed securely, but there are still a few hitches with getting our torch implementation of QR decomposition to work for an `AdditiveSharingTensor`.

```python
def _generate_worker_data_pointers(
        n, m, k, worker,
        beta_correct, gamma_correct, epsilon=0.01
):
    """
    Return pointers to worker-level data.
    Args:
        n: number of rows
        m: number of transient
        k: number of covariates
        beta_correct: coefficients for transient features (tensor of shape (m, 1))
        gamma_correct: coefficients for covariates (tensor of shape (k, 1))
        epsilon: scale of noise added to response
    Return:
        y, X, C: pointers to response, transients, and covariates
    """
    X = th.randn(n, m).send(worker)
    C = th.randn(n, k).send(worker)

    y = (X.mm(beta_correct.copy().send(worker)).reshape(-1, 1) +
         C.mm(gamma_correct.copy().send(worker)).reshape(-1, 1))

    y += (epsilon * th.randn(n, 1)).send(worker)

    return y, X, C

def _dot(x):
    return (x * x).sum(dim=0).reshape(-1, 1)

def _secure_sum(worker_level_pointers, workers, crypto_provider):
    """
    Securely add up an interable of pointers to (same-sized) tensors.
    Args:
        worker_level_pointers: iterable of pointer tensors
        workers: list of workers
        crypto_provider: worker
    Returns:
        AdditiveSharingTensor shared among workers
    """
    return sum([
        p.fix_precision(precision_fractional=10).share(*workers, crypto_provider=crypto_provider).get()
        for p in worker_level_pointers
    ])
```

```python
def dash_example_secure(
        workers, crypto_provider,
        n_samples_by_worker, m, k,
        beta_correct, gamma_correct,
        epsilon=0.01
):
    """
    Args:
        workers: list of workers
        crypto_provider: worker
        n_samples_by_worker: dict mapping worker ids to ints (number of rows of data)
        m: number of transients
        k: number of covariates
        beta_correct: coefficient for transient features
        gamma_correct: coefficient for covariates
        epsilon: scale of noise added to response
    Returns:
        beta, sigma, tstat, pval: coefficient of transients and accompanying statistics
    """
    # Generate each worker's data
    worker_data_pointers = {
        p: _generate_worker_data_pointers(
            n, m, k, workers[p],
            beta_correct, gamma_correct,
            epsilon=epsilon)
        for p, n in n_samples_by_worker.items()
    }

    # to be populated with pointers to results of local, worker-level computations
    Ctys, CtXs, yys, Xys, XXs, Rs = {}, {}, {}, {}, {}, {}

    def _sum(pointers):
        return _secure_sum(pointers, list(players.values()), crypto_provider)

    # worker-level compression step
    for p, (y, X, C) in worker_data_pointers.items():

        # perform worker-level compression step
        yys[p] = y.norm()
        Xys[p] = X.transpose(0, 1).mm(y)
        XXs[p] = _dot(X)

        Ctys[p] = C.transpose(0, 1).mm(y)
        CtXs[p] = C.transpose(0, 1).mm(X)
        _, R_full = qr(C, return_q=False)
        Rs[p] = R_full[:k, :]

    # Perform secure sum
    # - We're returning result to the local worker and computing there for the rest
    #   of the way, but should be possible to compute via SMPC (on a pointers to AdditiveSharingTensors)
    # - still afew minor-looking issues with implementing invert_triangular/qr for
    #   AdditiveSharingTensor
    yy = _sum(yys.values()).get().float_precision()
    Xy = _sum(Xys.values()).get().float_precision()
    XX = _sum(XXs.values()).get().float_precision()

    Cty = _sum(Ctys.values()).get().float_precision()
    CtX = _sum(CtXs.values()).get().float_precision()

    # Rest is done publicly on the local worker for now
    _, R_public = qr(
        th.cat([R.get() for R in Rs.values()], dim=0),
        return_q=False)

    invR_public = invert_triangular(R_public[:k, :])

    Qty = invR_public.transpose(0, 1).mm(Cty)
    QtX = invR_public.transpose(0, 1).mm(CtX)

    QtXQty = QtX.transpose(0, 1).mm(Qty)
    QtyQty = _dot(Qty)
    QtXQtX = _dot(QtX)

    yyq = yy - QtyQty
    Xyq = Xy - QtXQty
    XXq = XX - QtXQtX

    d = sum(n_samples_by_worker.values()) - k - 1
    beta = Xyq / XXq
    sigma = ((yyq / XXq - (beta ** 2)) / d).abs() ** 0.5
    tstat = beta / sigma
    pval = 2 * stats.t.cdf(-abs(tstat), d)

    return beta, sigma, tstat, pval
```

```python
players = {
    worker.id: worker
    for worker in [alice, bob, theo]
}

# de
n_samples_by_player = {
    alice.id: 100000,
    bob.id: 200000,
    theo.id: 100000
}

crypto_provider = jon

m = 100
k = 3
d = sum(n_samples_by_player.values()) - k - 1

beta_correct = th.ones(m, 1)
gamma_correct = th.ones(k, 1)

dash_example_secure(
    players, crypto_provider,
    n_samples_by_player, m, k,
    beta_correct, gamma_correct)
```

    (tensor([[1.0198],
             [0.9758],
             [0.9643],
             [1.0004],
             [1.0150],
             [0.9944],
             [0.9961],
             [1.0318],
             [1.0002],
             [0.9830],
             [0.9790],
             [0.9926],
             [1.0008],
             [0.9697],
             [0.9954],
             [0.9995],
             [0.9960],
             [0.9767],
             [1.0059],
             [0.9838],
             [0.9911],
             [1.0179],
             [1.0080],
             [0.9829],
             [0.9937],
             [0.9819],
             [1.0188],
             [0.9811],
             [0.9971],
             [0.9866],
             [1.0117],
             [0.9953],
             [0.9966],
             [0.9952],
             [0.9957],
             [0.9860],
             [1.0206],
             [0.9928],
             [0.9925],
             [1.0149],
             [0.9587],
             [0.9851],
             [1.0102],
             [1.0127],
             [1.0143],
             [1.0050],
             [0.9926],
             [0.9646],
             [0.9966],
             [0.9906],
             [1.0212],
             [0.9948],
             [1.0253],
             [0.9936],
             [0.9834],
             [0.9770],
             [0.9885],
             [0.9890],
             [0.9954],
             [0.9900],
             [0.9795],
             [0.9657],
             [0.9836],
             [1.0042],
             [0.9957],
             [0.9929],
             [1.0127],
             [0.9869],
             [0.9969],
             [1.0172],
             [1.0030],
             [0.9844],
             [1.0121],
             [1.0071],
             [0.9954],
             [0.9936],
             [0.9954],
             [1.0070],
             [0.9928],
             [0.9900],
             [0.9970],
             [0.9992],
             [0.9851],
             [0.9942],
             [0.9710],
             [0.9799],
             [0.9675],
             [1.0246],
             [1.0085],
             [0.9906],
             [0.9984],
             [1.0182],
             [0.9805],
             [0.9905],
             [1.0034],
             [0.9965],
             [0.9983],
             [0.9973],
             [0.9872],
             [0.9937]]), tensor([[0.0032],
             [0.0032],
             [0.0031],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0031],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0031],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0031],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0031],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032],
             [0.0032]]), tensor([[319.5643],
             [309.3729],
             [306.5470],
             [315.1175],
             [318.3896],
             [313.9088],
             [314.0636],
             [322.6956],
             [315.1400],
             [311.2341],
             [310.3611],
             [313.5351],
             [315.4006],
             [307.5811],
             [313.6877],
             [314.7977],
             [314.0446],
             [309.6685],
             [316.4051],
             [311.1368],
             [313.2263],
             [319.4759],
             [316.2847],
             [310.5904],
             [313.5548],
             [311.4909],
             [319.7083],
             [310.5916],
             [314.3991],
             [311.7694],
             [317.7445],
             [314.2973],
             [313.8512],
             [313.7931],
             [313.8863],
             [311.7593],
             [320.1660],
             [313.1064],
             [312.8649],
             [318.5549],
             [305.3120],
             [311.6055],
             [317.7577],
             [318.0811],
             [318.6992],
             [316.0238],
             [313.5295],
             [306.4926],
             [314.0880],
             [312.5821],
             [319.9753],
             [313.6790],
             [321.0503],
             [313.4674],
             [310.9974],
             [310.1643],
             [312.6641],
             [312.5327],
             [313.7359],
             [312.9331],
             [310.3143],
             [307.3026],
             [311.3515],
             [316.4857],
             [314.6939],
             [313.4534],
             [318.1405],
             [312.0404],
             [314.7274],
             [318.8090],
             [315.5815],
             [311.3936],
             [317.9680],
             [317.0735],
             [314.1143],
             [313.6470],
             [313.5154],
             [316.9404],
             [313.7480],
             [312.7427],
             [314.7064],
             [314.5428],
             [311.3454],
             [313.8516],
             [308.4349],
             [310.3224],
             [307.0361],
             [320.6432],
             [317.1022],
             [312.7644],
             [314.5857],
             [319.2417],
             [310.1894],
             [312.5483],
             [315.9466],
             [314.3875],
             [314.4272],
             [314.6446],
             [312.2668],
             [313.8906]]), array([[0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.],
            [0.]]))

```python

```