# Volume Sampling vs projection DPP for low rank approximation
## Introduction
#### In this notebook we compare the volume sampling and projection DPP for low rank approximation.
We recall the result proved in the article [DRVW]:\\
Let S be a random subset of k columns of X chosen with probability: $$P(S) = \frac{1}{Z_{k}} det(X_{.,S}^{T}X_{.,S})$$ with $$Z_{k} = \sum\limits_{S \subset [N], |S| = k} det(X_{.,S}^{T}X_{.,S})$$
Then
$$\begin{equation}
E(\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2}) \leq (k+1)\| X - \pi_{k}(X) \|_{Fr}^{2}
\end{equation}$$

We can prove that the volume sampling distribution is a mixture of projection DPPs distributions..., in particular one projection DPP distribution stands out for the problem of low rank approximation: ....\\
For the moment, there is no analytical expression for $$\begin{equation}
E(\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2})
\end{equation}$$ under the distribution of projection DPP.\\
However, we can calculate this quantity using simulation on some matrices representing cloud points with some specific geometric constraints.

Let $$X \in R^{n \times m}$$ a matrix representing a cloud of points.
We can write the SVD of $$X = UDV^{T}$$
In this notebook we investigate the influence of some structures enforced to V and D on the expected error expressed above for different algorithms: Volume Sampling, Projection DPP and the deterministic algorithm.
As for the Volume Sampling distribution, we can express the expected approximation error using only the elements of D. We can test this theoretical property in the next Numerical Study below. However, there is no closed formula (for the moment) for the expected approximation error under Projection DPP distribution. We will see in the Numerical Study section, that this value cannot depends only on the elements of D.

#### References
[DRVW] Deshpande, Amit and Rademacher, Luis and Vempala, Santosh and Wang, Grant - Matrix Approximation and Projective Clustering via Volume Sampling 2006

[BoDr] Boutsidis, Christos and Drineas, Petros  - Deterministic and randomized column selection algorithms for matrices 2014

[] INDERJIT S. DHILLON , ROBERT W. HEATH JR., MA ́TYA ́S A. SUSTIK, AND
JOEL A. TROPP - GENERALIZED FINITE ALGORITHMS FOR CONSTRUCTING HERMITIAN MATRICES
WITH PRESCRIBED DIAGONAL AND SPECTRUM 2005

## I- Generating a cloud of points with geometric constraints
In this simulation we will enforce some structure on the matrix V for two values of the matrix D. While the matrix U will be choosen randomly.
We want to investigate the influence of the profile of the norms of the V_k rows: the k-leverage scores. For this purpose we use an algorithm proposed in the article []: this algorithm outputs a ( dxk)  matrix Q with orthonormal columns and a prescribed profile of the norms of the rows. If we consider the Gram matrix H= QQ^{T}, this boils down to enforce the diagonal of H while keeping its spectrum containing k ones and d-k zeros.
The algorithm proceed as following:
* Initialization of the matrix Q by the rectangular identity
* Apply a Givens Rotation (of dimension d) to the matrix Q: this step will enforce the norm of a row every iteration
* Outputs the resulting matrix when all the rows norms are enforced.

```python
import numpy as np
import pandas as pd
from itertools import combinations
from scipy.stats import binom
import scipy.special
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from IPython.display import display, HTML
from FrameBuilder.eigenstepsbuilder import *
from decimal import *
from copy import deepcopy
import matplotlib.lines as mlines
import matplotlib.transforms as mtransforms

from matplotlib import rc
rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
## for Palatino and other serif fonts use:
#rc('font',**{'family':'serif','serif':['Palatino']})
rc('text', usetex=False)
```

The following function is an implementation of the algorithm [] figuring in the article [] to generate an orthogonal matrix with a prescribed profile of leverage scores.
In fact this is a simplification of the algorithm .....

```python
def generate_leverage_scores_vector_with_dirichlet(d,k,nn_cardinal):
    getcontext().prec = 3
    mu_vector = np.float16(np.zeros((d,)))
    mu_vector_2 = np.float16(np.zeros((d,)))
    not_bounded = 1
    while(not_bounded == 1):
        mu_vector[0:nn_cardinal] = (k*np.random.dirichlet([1]*nn_cardinal, 1))[0]
        mu_vector = np.flip(np.sort(mu_vector),axis = 0)
        if max(mu_vector)<=1:
            not_bounded = 0
    for i in range(nn_cardinal):
        mu_vector_2[i] = round(mu_vector[i],4)
    mu_vector_2 = k*mu_vector_2/np.sum(mu_vector_2)
    return list(mu_vector_2)

```

```python
def generate_orthonormal_matrix_with_leverage_scores_ES(N,d,lv_scores_vector,versions_number,nn_cardinal_list):

    lambda_vector = np.zeros((N))
    lambda_vector[0:d] = np.ones((d))

    #mu_vector = np.linspace(1, 0.1, num=N)
    #sum_mu_vector = np.sum(mu_vector)
    #mu_vector = d/sum_mu_vector*mu_vector
    Q = np.zeros((N,d))
    previous_Q = np.zeros((versions_number,N,d))
    #mu_vector = d/N*np.ones((N,1))
    E = np.zeros((N,N)) #(d,N)
    counter = 0
    for j in nn_cardinal_list:
        #print("counter")
        #print(counter)
        mu_vector = generate_leverage_scores_vector_with_dirichlet(N,d,j)
        #print(np.sum(mu_vector))
        #print(mu_vector)
        E_test = get_eigensteps_random(mu_vector,lambda_vector,N,d)
        E_ = np.zeros((d,N+1))
        for i in range(d):
            E_[i,1:N+1] = E_test[i,:]
        F_test = get_F(d,N,np.asmatrix(E_),mu_vector)
        previous_Q[counter,:,:] = np.transpose(F_test)
        Q = np.transpose(F_test)
        counter = counter +1
    return Q,previous_Q
```

```python
list(range(20-2,-1,-1))
```

    [18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

```python

```

```python

```

```python
static_list = list(range(11,21))

```

```python
random_cardinal_list = list(np.random.choice(static_list, 3))
Q,previous_Q = generate_orthonormal_matrix_with_leverage_scores_ES(20,5,[],3,random_cardinal_list)
```

    /Users/ayoubbelhadji1/Desktop/final_git_projects/FrameBuilder/FrameBuilder/eigenstepsbuilder.py:142: RuntimeWarning: invalid value encountered in sqrt
      v_n[v_n_index] = np.sqrt(-np.prod(nom_v_n)/np.prod(denom_v_n))
    /Users/ayoubbelhadji1/Desktop/final_git_projects/FrameBuilder/FrameBuilder/eigenstepsbuilder.py:149: RuntimeWarning: invalid value encountered in sqrt
      w_n[w_n_index] = np.sqrt(np.prod(nom_w_n)/np.prod(denom_w_n))

```python
## The dimensions of the design matrix X
N = 100       # The number of observations in the dataset
real_dim = 20 # The dimension of the dataset
## The low rank paramters
k = 2         # The rank of the low rank approximation
## The covariance matrix parameters
r = 6         # Just a parameter to control the number of non trivial singular values in the covariance matrix
mean = np.zeros((real_dim))    # The mean vector useful to generate U (X = UDV^T)
cov_test = 0.1*np.ones((real_dim-r))  # The "trivial" singular values in the covariance matrix (there are real_dim-r)
cov = np.diag(np.concatenate(([100,100,0.1,0.1,0.1,0.1],cov_test)))

## The paramters of the matrix V
versions_number = 3 # The number of orthogonal matrices (and therefor datasets) (-1) generated by the algorithm above
lv_scores_vector = k/real_dim*np.ones(real_dim)  # The vector of leverage scores (the last one)
```

```python
test_multi_X = contruct_dataset_from_orthogonal_matrix(previous_Q,N,real_dim,cov,mean,versions_number)
```

```python
Q_test = previous_Q[2,:,:]
```

```python
np.linalg.det(np.dot(Q_test.T,Q_test))
```

    1.0003807632948656

```python
plt.scatter(Q[:,0], Q[:,1])
plt.show()
```

```python
np.dot(Q.T,Q)
```

    array([[  1.00029256e+00,  -8.32667268e-17],
           [ -8.32667268e-17,   1.00000000e+00]])

```python
np.diag(np.dot(Q,Q.T))
```

    array([ 0.46064854,  0.28269387,  0.24554381,  0.19393669,  0.11617887,
            0.11233676,  0.109375  ,  0.10926151,  0.07138592,  0.05079787,
            0.03601446,  0.0337448 ,  0.02985011,  0.03009549,  0.02722856,
            0.02348039,  0.0223261 ,  0.02087258,  0.01334667,  0.01108771])

```python
previous_Q
```

    array([[[  6.48925781e-01,   0.00000000e+00],
            [ -3.47042240e-01,  -4.35454105e-01],
            [ -3.23121789e-01,   4.43691155e-01],
            [  1.53004624e-01,  -3.13982786e-01],
            [ -1.10542148e-01,   3.22592745e-01],
            [  2.75868389e-01,   1.53214817e-01],
            [  8.78928411e-02,  -2.96663446e-01],
            [ -2.99785654e-01,  -5.93646751e-02],
            [  7.37933517e-02,  -2.60402694e-01],
            [  7.69203827e-02,   2.40627055e-01],
            [ -7.04148422e-02,   2.41107649e-01],
            [ -1.14155735e-01,  -2.09606680e-01],
            [ -1.75636728e-01,  -1.54592323e-01],
            [ -2.10311781e-01,  -1.02256808e-01],
            [ -1.47601148e-01,   4.00178347e-02],
            [  7.96067776e-02,  -9.61829914e-02],
            [  5.32811215e-02,   1.02859832e-01],
            [  9.41498979e-02,  -5.65151833e-02],
            [ -1.19868404e-02,   7.47707847e-02],
            [ -2.28958798e-02,  -5.92203437e-02]],

           [[  6.07910156e-01,   0.00000000e+00],
            [  1.83184695e-01,   3.75751668e-01],
            [ -1.64724369e-01,  -3.62789232e-01],
            [  3.34624658e-01,  -1.90919564e-01],
            [  2.28256354e-01,  -2.98334404e-01],
            [  4.21940425e-02,  -3.37306359e-01],
            [  4.68150626e-02,  -3.33584770e-01],
            [ -3.01608345e-01,   1.27711655e-01],
            [ -2.80298158e-01,  -1.23425999e-01],
            [ -1.68572009e-01,  -2.49681422e-01],
            [ -1.27788817e-01,  -2.58297658e-01],
            [ -9.60408214e-02,   2.47395975e-01],
            [ -1.82867027e-01,  -1.92021224e-01],
            [ -1.20942764e-01,   2.33850995e-01],
            [ -2.56773095e-01,   4.91682858e-02],
            [  1.50728913e-01,  -1.76135352e-01],
            [ -1.45514353e-01,  -4.75717683e-02],
            [ -1.16302536e-01,  -8.87872287e-02],
            [ -4.18555411e-02,  -1.26099809e-01],
            [ -7.04060671e-02,   6.19035396e-02]],

           [[  6.53320312e-01,   0.00000000e+00],
            [ -3.25702142e-01,  -2.27547380e-01],
            [ -1.86272719e-01,  -3.43140660e-01],
            [ -1.69009147e-03,  -3.87113804e-01],
            [  5.76478247e-02,   3.81826121e-01],
            [  1.02746440e-01,  -3.70467409e-01],
            [ -2.39048794e-01,   2.64159284e-01],
            [ -3.29342049e-01,   1.03455843e-01],
            [  5.31183319e-03,  -3.01176366e-01],
            [  7.67373022e-02,  -2.77856244e-01],
            [  2.65142423e-01,   8.82039332e-02],
            [ -2.11094115e-01,   1.43983631e-01],
            [  2.03754474e-01,   1.19709710e-01],
            [ -1.46534532e-02,   2.13737488e-01],
            [ -1.97702488e-01,   7.63105811e-02],
            [ -1.31006828e-01,   1.25428346e-01],
            [ -6.63695879e-02,  -1.64819417e-01],
            [ -1.52764950e-01,  -1.76014862e-05],
            [ -4.44897999e-02,  -1.08334089e-01],
            [  6.88064017e-02,  -1.43219817e-02]]])

```python
l = generate_leverage_scores_vector_with_dirichlet(10,2,6)
print(l)
print(np.sum(l))
```

    [0.82812, 0.52832, 0.28271, 0.21301, 0.13379, 0.014603, 0.0, 0.0, 0.0, 0.0]
    2.0

```python

```

The following function allows to estimate the leverage scores for an orthogonal matrix Q: the function calculates the diagonoal of the matrix $$Q Q^{T}$$

```python
def estimate_leverage_scores_from_orthogonal_matrix(Q):
    [N,_] = np.shape(Q)
    lv_scores_vector = np.zeros((N,1))
    lv_scores_vector = np.diag(np.dot(Q,np.transpose(Q)))
    lv_scores_vector = np.asarray(list(reversed(np.sort(lv_scores_vector))))
    return lv_scores_vector
def estimate_sum_first_k_leverage_scores(Q,k):
    lv_scores_vector = estimate_leverage_scores_from_orthogonal_matrix(Q)
    res = np.sum(lv_scores_vector[0:k])
    return res
```

```python
def get_p_eff_leverage_scores(Q,k):
    lv_scores_vector = estimate_leverage_scores_from_orthogonal_matrix(Q)
    p = 0
    lv_scores_sum_until_p = 0
    while lv_scores_sum_until_p < k-1/2:
        lv_scores_sum_until_p = lv_scores_sum_until_p+lv_scores_vector[p]
        p = p +1
    return p
```

## I-2- Extending the orthogonal matrices

For the task of low rank approximation, we have seen that only the information contained in the first right k eigenvectors of the matrix X are relevant. In the previous step we build only the first right k eigenvectors but we still need to complete these orthogonal matrices with d-k columns. We proceed as following:
Generate a random vector (Nx1) using independent standard gaussian variables,
Project this vector in the orthogonal of the span of Q
Normalize the obtained vector after the projection
Extend the matrix Q
Note that this procedure is not the unique way to extend the matrix Q to an orthogonal (Nxd) matrix.

```python
def extend_orthogonal_matrix(Q,d_target):
    [N,d] = np.shape(Q)
    Q_target = np.zeros((N,d))
    Q_target = deepcopy(Q)
    delta = d_target - d
    for t in range(delta):
        Q_test = np.random.normal(0, 1, N)
        for _ in range(d):
            Q_test = Q_test - np.dot(Q_test,Q[:,_])*Q[:,_]
        Q_test = Q_test/np.linalg.norm(Q_test)
        Q_test = Q_test.reshape(N,1)
        Q_target = np.append(Q_target,Q_test,1)
    return Q_target

#extended_Q = extend_orthogonal_matrix(Q,r)
```

## I-3 - Constructing a dataset for every extended orthogonal matrix
The previous step allow us to build (N x d) orthogonal matrices such that the extracted (N x k) matrix have a prescribed profile of leverage scores.
Now we construct a cloud of point by assigning a covariance matrix D and a matrix V

```python
def contruct_dataset_from_orthogonal_matrix(multi_Q,N,target_d,cov,mean,versions_number):
    multi_X = np.zeros((versions_number,N,real_dim))
    for t in range(versions_number):
        test_X = np.random.multivariate_normal(mean, cov, N)
        [U,_,_] = np.linalg.svd(test_X, full_matrices=False)
        Q_test = extend_orthogonal_matrix(multi_Q[t,:,:],target_d)
        multi_X[t,:,:] = np.dot(np.dot(Q_test,cov),U.T).T
    return multi_X

```

## II- Volume sampling vs Projection DPP for low rank approximation
These functions allow to quantify the approximation error:
* approximation_error_function_fro calculate the ratio of the approximation error of a subset of columns to the optimal approximatione error given by the first k left eigenvectors of the matrix X
* expected_approximation_error_for_sampling_scheme calculate the expected value of the ratio of the approximatione error under some sampling distribution

```python
def approximation_error_function_fro(Sigma,k,X,X_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    d = list(Sigma.shape)[0] # the dimension of the matrix X
    Sigma = np.multiply(Sigma,Sigma)  # Sigma power 2 -> we are intersted in the approximation error square
    sigma_S_temp = np.linalg.inv(np.dot(X_S.T,X_S))  # just a temporary matrix to construct the projection matrix
    projection_S = np.dot(np.dot(X_S,sigma_S_temp),X_S.T) # the projection matrix P_S
    res_X = X - np.dot(projection_S,X) # The projection of the matrix X in the orthogonal of S
    approximation_error_ratio = np.power(np.linalg.norm(res_X,'fro'),2)/np.sum(Sigma[k:d])
    # Calculate the apparoximation error ratio
    return approximation_error_ratio
```

```python
def approximation_error_function_spectral(Sigma,k,X,X_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    d = list(Sigma.shape)[0] # the dimension of the matrix X
    Sigma = np.multiply(Sigma,Sigma)  # Sigma power 2 -> we are intersted in the approximation error square
    sigma_S_temp = np.linalg.inv(np.dot(X_S.T,X_S))  # just a temporary matrix to construct the projection matrix
    projection_S = np.dot(np.dot(X_S,sigma_S_temp),X_S.T) # the projection matrix P_S
    res_X = X - np.dot(projection_S,X) # The projection of the matrix X in the orthogonal of S
    approximation_error_ratio = np.power(np.linalg.norm(res_X,ord = 2),2)/np.sum(Sigma[k:k+1])
    # Calculate the apparoximation error ratio
    return approximation_error_ratio
```

```python
def upper_bound_error_function_for_projection_DPP(k,X,X_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    _,sigma_S_temp,_ = np.linalg.svd(X_S, full_matrices=False)  # just a temporary matrix to construct the projection matrix
    trunc_product = np.power(np.prod(sigma_S_temp[0:k-1]),2)
    if np.power(np.prod(sigma_S_temp[0:k]),2) == 0:
        trunc_product = 0
    # Calculate the apparoximation error ratio
    return trunc_product
```

```python
def tight_upper_bound_error_function_fro(k,X,X_S,V_k,V_k_S):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    d = list(Sigma.shape)[0]
    Sigma = np.multiply(Sigma,Sigma)
    if np.linalg.matrix_rank(V_k_S,0.000001) == k:
        temp_T = np.dot(np.linalg.inv(V_k_S),V_k)
        temp_matrix = X - np.dot(X_S,temp_T)

        return np.power(np.linalg.norm(temp_matrix,'fro'),2)/np.sum(Sigma[k:d])
    else:
        return 0
```

```python
def get_the_matrix_sum_T_S(k,d,V_k,V_d_k):
    ## Sigma is the spectrum of the matrix X: we need to calculate the optimal approximation error given by the PCA
    ## k is the rank of the approximation
    ## X is the initial matrix
    ## X_S is the subset of columns of the matrix X for witch we calculate the approximation error ratio
    #Sigma = np.multiply(Sigma,Sigma)
    #matrices_array = [ np.dot(V_d_k[:,list(comb)],np.dot(np.dot(np.linalg.inv(V_k[:,list(comb)]),np.linalg.inv(V_k[:,list(comb)]))),np.transpose(V_d_k[:,list(comb)])) for comb in combinations(range(d),k) if np.linalg.matrix_rank(V_k[:,list(comb)],0.000001) == k]
    T = np.zeros((d-k,d-k))
    for comb in combinations(range(d),k):
        if np.linalg.matrix_rank(V_k[:,list(comb)],0.0000000001) == k:
            V_k_S_inv = np.linalg.inv(V_k[:,list(comb)])
            V_d_k_S = V_d_k[:,list(comb)]
            V_k_S_inv_2 = np.transpose(np.dot(V_k_S_inv,np.transpose(V_k_S_inv)))
            #T = np.dot(np.dot(np.dot(V_d_k_S,np.dot(V_k_S_inv,np.transpose(V_k_S_inv)))),np.transpose(V_d_k_S)) + T
            T = np.power(np.linalg.det(V_k[:,list(comb)]),2)*np.dot(V_d_k_S,np.dot(V_k_S_inv_2,np.transpose(V_d_k_S))) +T
    return T
```

```python
def tight_approximation_error_fro_for_sampling_scheme(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)
    _,Sigma,V = np.linalg.svd(X, full_matrices=False)
    V_k = V[0:k,:]
    ## Estimating the spectrum of X -> needed in approximation_error_function_fro
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    ## Construct the array of weights: the volumes of UU^{T}_{S,S}
    volumes_array_sum = np.sum(volumes_array)
    ## The normalization constant
    volumes_array = volumes_array/volumes_array_sum
    ## The weigths normalized
    approximation_error_array = [tight_upper_bound_error_function_fro(k,X,X[:,list(comb)],V_k,V_k[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple
    expected_value = np.dot(approximation_error_array,volumes_array)
    ## The expected value of the approximatione error is just the dot product of the two arrays above
    return expected_value
```

```python
def expected_approximation_error_fro_for_sampling_scheme(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    ## Estimating the spectrum of X -> needed in approximation_error_function_fro
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    ## Construct the array of weights: the volumes of UU^{T}_{S,S}
    volumes_array_sum = np.sum(volumes_array)
    ## The normalization constant
    volumes_array = volumes_array/volumes_array_sum
    ## The weigths normalized
    approximation_error_array = [approximation_error_function_fro(Sigma,k,X,X[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple
    expected_value = np.dot(approximation_error_array,volumes_array)
    ## The expected value of the approximatione error is just the dot product of the two arrays above
    return expected_value
```

```python
def expected_approximation_error_spectral_for_sampling_scheme(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    ## Estimating the spectrum of X -> needed in approximation_error_function_fro
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    ## Construct the array of weights: the volumes of UU^{T}_{S,S}
    volumes_array_sum = np.sum(volumes_array)
    ## The normalization constant
    volumes_array = volumes_array/volumes_array_sum
    ## The weigths normalized
    approximation_error_array = [approximation_error_function_spectral(Sigma,k,X,X[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple
    expected_value = np.dot(approximation_error_array,volumes_array)
    ## The expected value of the approximatione error is just the dot product of the two arrays above
    return expected_value
```

```python
def expected_upper_bound_for_projection_DPP(X,U,k,N):
    ## X is the matrix X :)
    ## U is the matrix used in the sampling: we sample propotional to the volume of UU^{T}_{S,S}:
    ## we are not sampling but we need the weigth to estimate the expected error
    ## k is the rank of the approximation
    ## N is the number of columns (to be changed to avoid confusion with the number of points)

    approximation_error_array = [upper_bound_error_function_for_projection_DPP(k,X,U[:,list(comb)]) for comb in combinations(range(N),k)]
    ## Calculating the approximation error for every k-tuple

    ## The expected value of the approximatione error is just the dot product of the two arrays above
    #return expected_value
    return np.sum(approximation_error_array)
```

## III - Numerical analysis

In this section we use the functions developed previously to investigate the influence of two parameters: the spectrum of X and the k-leverage scores.
For this purpose, we assemble these functionalities in a class allowing fast numerical experiments.

```python
class Numrerical_Analysis_DPP:
    def __init__(self,N,real_dim,r,k,versions_number,mean,cov,lv_scores,cardinal_list):
        self.N = N
        self.real_dim = real_dim
        self.r = r
        self.k = k
        self.versions_number = versions_number
        self.mean = mean
        self.cov = cov
        self.lv_scores = lv_scores
        self.cardinality_list = cardinal_list
        self.Q = np.zeros((real_dim,k))
        self.multi_Q = np.zeros((self.versions_number,real_dim,k))
        self.X = np.zeros((N,real_dim))
        self.multi_X = np.zeros((self.versions_number,N,real_dim))
        #[self.Q,self.multi_Q] = generate_orthonormal_matrix_with_leverage_scores(real_dim,k,lv_scores,versions_number,'identity')
        [self.Q,self.multi_Q] = generate_orthonormal_matrix_with_leverage_scores_ES(self.real_dim,self.k,[],self.versions_number,self.cardinality_list)
        self.multi_X = contruct_dataset_from_orthogonal_matrix(self.multi_Q,self.N,self.real_dim,self.cov,self.mean,self.versions_number)
    def contruct_dataset_from_orthogonal_matrix_4(self,multi_Q,N,target_d,cov,mean,versions_number):
        test_multi_X = np.zeros((self.versions_number,N,real_dim))
        for t in range(self.versions_number):
            test_X = np.random.multivariate_normal(mean, cov, N)
            [U,_,_] = np.linalg.svd(test_X, full_matrices=False)
            Q_test = extend_orthogonal_matrix(self.multi_Q[t,:,:],target_d)
            test_multi_X[t,:,:] = np.dot(np.dot(Q_test,cov),U.T).T
        return test_multi_X
    def get_expected_error_fro_for_volume_sampling(self):
        ## Calculate the expected error ratio for the Volume Sampling distribution for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            res_list[t] = expected_approximation_error_fro_for_sampling_scheme(test_X,test_X,self.k,self.real_dim)
        return res_list
    def get_tight_upper_bound_error_fro_for_projection_DPP(self):
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            res_list[t] = tight_approximation_error_fro_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_max_diag_sum_T_matrices(self):
        res_list = np.zeros((self.versions_number))
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            _,_,test_V = np.linalg.svd(test_X, full_matrices=False)
            test_V_k = test_V[0:self.k,:]
            test_V_d_k = test_V[self.k:self.real_dim,:]
            res_list[t] = 1+np.max(np.diag(get_the_matrix_sum_T_S(self.k,self.real_dim,test_V_k,test_V_d_k)))
        return res_list
    def get_max_spectrum_sum_T_matrices(self):
        res_list = np.zeros((self.versions_number))
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            _,_,test_V = np.linalg.svd(test_X, full_matrices=False)
            test_V_k = test_V[0:self.k,:]
            test_V_d_k = test_V[self.k:self.real_dim,:]
            res_list[t] = 1+np.max(np.diag(get_the_matrix_sum_T_S(self.k,self.real_dim,test_V_k,test_V_d_k)))
        return res_list
    def get_expected_error_fro_for_projection_DPP(self):
        ## Calculate the expected error ratio for the Projection DPP distribution for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            res_list[t] = expected_approximation_error_fro_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_expected_error_spectral_for_volume_sampling(self):
        ## Calculate the expected error ratio for the Volume Sampling distribution for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            res_list[t] = expected_approximation_error_spectral_for_sampling_scheme(test_X,test_X,self.k,self.real_dim)
        return res_list
    def get_p_eff(self):
        ## A function that calculate the p_eff.
        ## It is a measure of the concentration of V_k. This is done for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            diag_Q_t = np.diag(np.dot(self.multi_Q[t,:,:],self.multi_Q[t,:,:].T))
            #diag_Q_t = list(diag_Q_t[::-1].sort())
            diag_Q_t = list(np.sort(diag_Q_t)[::-1])
            p = self.real_dim
            #print(diag_Q_t)
            while np.sum(diag_Q_t[0:p-1]) > float(self.k-1.0/2):
                p = p-1
            res_list[t] = p
        return res_list
    def get_effective_kernel_from_orthogonal_matrix(self):
        test_eff_V = np.zeros((self.versions_number,self.real_dim,self.k))
        p_eff_list = self.get_p_eff()
        for t in range(self.versions_number):
            test_V = deepcopy(self.multi_Q[t,:,:])
            p_eff = p_eff_list[t]
            diag_Q_t = np.diag(np.dot(test_V,test_V.T))
            #diag_Q_t = list(diag_Q_t[::-1].sort())
            #print(diag_Q_t)
            permutation_t = list(reversed(np.argsort(diag_Q_t)))
            #print(permutation_t)
            for i in range(self.real_dim):
                if i >p_eff-1:
                    test_V[permutation_t[i],:] = 0
            #Q_test = extend_orthogonal_matrix(self.multi_Q[t,:,:],target_d)
            test_eff_V[t,:,:] = test_V
        return test_eff_V
    def get_expected_error_fro_for_effective_kernel_sampling(self):
        ## Calculate the expected error ratio for the Volume Sampling distribution for every dataset
        res_list = np.zeros(self.versions_number)
        test_eff_V = self.get_effective_kernel_from_orthogonal_matrix()
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = test_eff_V[t,:,:].T
            res_list[t] = expected_approximation_error_fro_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list

    def get_expected_error_spectral_for_projection_DPP(self):
        ## Calculate the expected error ratio for the Projection DPP distribution for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            res_list[t] = expected_approximation_error_spectral_for_sampling_scheme(test_X,test_U,self.k,self.real_dim)
        return res_list
    def get_upper_bound_error_for_projection_DPP(self):
        ## Calculate the expected error ratio for the Projection DPP distribution for every dataset
        #res_list = np.zeros(self.versions_number+1)
        res_list = []
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            #res_list[t] = expected_upper_bound_for_projection_DPP(test_X,test_U,self.k,self.real_dim)
            res_list.append( expected_upper_bound_for_projection_DPP(test_X,test_U,self.k,self.real_dim))
        return res_list
    def get_error_fro_for_deterministic_selection(self):
        ## Calculate the error ratio for the k-tuple selected by the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            lv_scores_vector = np.diag(np.dot(np.transpose(test_U),test_U))
            test_I_k = list(np.argsort(lv_scores_vector)[self.real_dim-self.k:self.real_dim])
            _,test_Sigma,_ = np.linalg.svd(test_X, full_matrices=False)
            res_list[t] = approximation_error_function_fro(test_Sigma,self.k,test_X,test_X[:,test_I_k])
            #res_list.append(test_I_k)
        return res_list
    def get_error_spectral_for_deterministic_selection(self):
        ## Calculate the error ratio for the k-tuple selected by the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            test_X = self.multi_X[t,:,:]
            test_U = self.multi_Q[t,:,:].T
            lv_scores_vector = np.diag(np.dot(np.transpose(test_U),test_U))
            test_I_k = list(np.argsort(lv_scores_vector)[self.real_dim-self.k:self.real_dim])
            _,test_Sigma,_ = np.linalg.svd(test_X, full_matrices=False)
            res_list[t] = approximation_error_function_spectral(test_Sigma,self.k,test_X,test_X[:,test_I_k])
            #res_list.append(test_I_k)
        return res_list
    def get_sum_k_leverage_scores(self):
        ## A function that calculate the k-sum: the sum of the first k k-leverage scores. It is a measure of the concentration of V_k
        ## This is done for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            res_list[t] = estimate_sum_first_k_leverage_scores(self.multi_Q[t,:,:],self.k)
        return res_list
    def get_p_eff_leverage_scores(self):
        ## A function that calculate the k-sum: the sum of the first k k-leverage scores. It is a measure of the concentration of V_k
        ## This is done for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            res_list[t] = get_p_eff_leverage_scores(self.multi_Q[t,:,:],self.k)
        return res_list
    def get_deterministic_upper_bound(self):
        ## A function that calculate the theoretical upper bound for the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number)
        for t in range(self.versions_number):
            res_list[t] = 1/(1+estimate_sum_first_k_leverage_scores(self.multi_Q[t,:,:],self.k)-self.k)
        return res_list
    def get_alpha_sum_k_leverage_scores(self,alpha):
        ## A function that calculate the theoretical upper bound for the deterministic algorithm for every dataset
        res_list = np.zeros(self.versions_number)
        #k_l = self.get_sum_k_leverage_scores()
        for t in range(self.versions_number):
            k_l = estimate_leverage_scores_from_orthogonal_matrix(self.multi_Q[t,:,:])[0:k]
            func_k = np.power(np.linspace(1, k, num=k),alpha)
            res_list[t] = np.dot(func_k,k_l)
        return res_list
```

### III- 0 Parameters of the simultations

```python
## The dimensions of the design matrix X
N = 100       # The number of observations in the dataset
real_dim = 20 # The dimension of the dataset
## The low rank paramters
k = 4         # The rank of the low rank approximation
## The covariance matrix parameters
r = 6         # Just a parameter to control the number of non trivial singular values in the covariance matrix
mean = np.zeros((real_dim))    # The mean vector useful to generate U (X = UDV^T)
cov_test = 0.1*np.ones((real_dim-r))  # The "trivial" singular values in the covariance matrix (there are real_dim-r)
## The paramters of the matrix V
versions_number = 3 # The number of orthogonal matrices (and therefor datasets) (-1) generated by the algorithm above
lv_scores_vector = k/real_dim*np.ones(real_dim)  # The vector of leverage scores (the last one)
```

### III-1 The influence of the spectrum

In this subsection we compare the Volume Sampling distribution to the projection DPP distribution and the deterministic algorithm of []  for different profiles of the spectrum with k-leverage scores profile fixed. In other words, if we note $$X = UDV^{T}$$ We keep V_{k} constant and we investigate the effect of D.

#### III-1-1 The case of a non-projection spectrum
We mean by a projection spectrum matrix, a matrix with equal the first k singular values.
We observe that the two distributions are very similar.... \todo{reword}

```python
cov_1 = np.diag(np.concatenate(([100,100,100,100,0.1,0.1],cov_test)))
NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,[20,20,20])

projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
deterministic_selection_res_fro_1 = NAL_1.get_error_fro_for_deterministic_selection()
projection_DPP_res_spectral_1 = NAL_1.get_expected_error_spectral_for_projection_DPP()
volume_sampling_res_spectral_1 = NAL_1.get_expected_error_spectral_for_volume_sampling()
deterministic_selection_res_spectral_1 = NAL_1.get_error_spectral_for_deterministic_selection()
effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

p_eff_res_1 = NAL_1.get_p_eff()
eff_kernel_upper_bound_1 = 1+ (p_eff_res_1-k+1)/(real_dim-k)*(k+1)

sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
p_eff_res_2 = NAL_1.get_p_eff_leverage_scores()
deterministic_upper_bound_res_1 = NAL_1.get_deterministic_upper_bound()

expected_upper_bound_res_1 = NAL_1.get_upper_bound_error_for_projection_DPP()
multi_Q_1 = NAL_1.multi_Q[1,:,:].T

```

```python
pd_1 = pd.DataFrame(
    {'k-sum (ratio)': sum_U_res_1/k,
     'alpha k-sum': alpha_sum_res_1,
     'p_eff' :p_eff_res_1,
     'Expected Upper Bound for Projection DPP': expected_upper_bound_res_1,
     'Volume Sampling(Fro)': volume_sampling_res_fro_1,
     'Projection DPP(Fro)': projection_DPP_res_fro_1,
     'Effective kernel(Fro)' : effective_kernel_sampling_res_fro_1,
     'Effective kernel upper bound (Fro)':eff_kernel_upper_bound_1,
     'Very sharp approximation of Projection DPP(Fro)': upper_tight_bound_projection_DPP_res_fro_1,
     'Deterministic Algorithm(Fro)': deterministic_selection_res_fro_1,
     'Volume Sampling(Spectral)': volume_sampling_res_spectral_1,
     'Projection DPP(Spectral)': projection_DPP_res_spectral_1,
     'Deterministic Algorithm(Spectral)': deterministic_selection_res_spectral_1,
     'Deterministic Upper Bound': deterministic_upper_bound_res_1
    })
pd_1 = pd_1[['k-sum (ratio)', 'alpha k-sum','p_eff','Expected Upper Bound for Projection DPP','Volume Sampling(Fro)','Projection DPP(Fro)','Effective kernel(Fro)','Effective kernel upper bound (Fro)','Very sharp approximation of Projection DPP(Fro)','Deterministic Algorithm(Fro)','Volume Sampling(Spectral)','Projection DPP(Spectral)','Deterministic Algorithm(Spectral)','Deterministic Upper Bound']]
```

```python
pd_1
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>k-sum (ratio)</th>
      <th>alpha k-sum</th>
      <th>p_eff</th>
      <th>Expected Upper Bound for Projection DPP</th>
      <th>Volume Sampling(Fro)</th>
      <th>Projection DPP(Fro)</th>
      <th>Effective kernel(Fro)</th>
      <th>Effective kernel upper bound (Fro)</th>
      <th>Very sharp approximation of Projection DPP(Fro)</th>
      <th>Deterministic Algorithm(Fro)</th>
      <th>Volume Sampling(Spectral)</th>
      <th>Projection DPP(Spectral)</th>
      <th>Deterministic Algorithm(Spectral)</th>
      <th>Deterministic Upper Bound</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.485268</td>
      <td>4.533247</td>
      <td>12.0</td>
      <td>57.176416</td>
      <td>4.999822</td>
      <td>4.966467</td>
      <td>3.150284</td>
      <td>3.8125</td>
      <td>5.0</td>
      <td>2.435945</td>
      <td>19.248552</td>
      <td>19.074479</td>
      <td>6.738316</td>
      <td>-0.944350</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.420277</td>
      <td>3.734385</td>
      <td>13.0</td>
      <td>55.889978</td>
      <td>4.999822</td>
      <td>4.952096</td>
      <td>3.492612</td>
      <td>4.1250</td>
      <td>5.0</td>
      <td>2.547184</td>
      <td>21.545843</td>
      <td>21.260952</td>
      <td>8.387673</td>
      <td>-0.758213</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.426711</td>
      <td>3.874272</td>
      <td>14.0</td>
      <td>55.668905</td>
      <td>4.999821</td>
      <td>4.970466</td>
      <td>4.085380</td>
      <td>4.4375</td>
      <td>5.0</td>
      <td>52.170381</td>
      <td>19.489560</td>
      <td>19.328701</td>
      <td>279.860957</td>
      <td>-0.773302</td>
    </tr>
  </tbody>
</table>
</div>

```python
projection_DPP_res_fro_2 = NAL_1.get_expected_error_fro_for_projection_DPP()

```

```python
projection_DPP_res_fro_2
```

    array([ 2.09267515,  1.74627114,  1.85399953])

```python
projection_DPP_res_fro_1
```

    array([ 2.99037541,  2.81961073,  2.6927861 ])

#### Observations:
* The expected error is always smaller under the Projection DPP distribution compared to the Volume Sampling distribution.
* The expected error for the Volume Sampling distribution is constant for a contant D
* However the expected error for the Projection DPP distribution depends on the k-sum
* For X_0 and X_1, the profile of the k-leverage scores are highly concentrated (k-sum > k-1) thus epsilon is smaller than 1, in this regime the determinstic algorithm have the lower approximation error and it performs better than expected (the theoretical bound is 1/(1-epsilon).
* However, for the other datasets, the (k-sum < k-1) thus epsilon >1 and the deterministic algorithm have no guarantee in this regime: we observe that the approximation error for the deterministic algorithm can be very high in this regime.

```python
T = 8 # The number of experiments
versions_number = 3
k = 4
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,100,100,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
def from_p_eff_to_error_bound(list_of_p,k,d):
    list_res = []
    for p in list_of_p:
        new_element = 1+(p-k+1)/(d-k)*k
        list_res.append(new_element)
    return list_res
```

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
border_array = np.linspace(1, 4.0, num=20)
```

```python
real_dim
```

    20

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 6)
plt.ylim(1, 6)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()
```

```python
def from_p_eff_to_error_bound_2(list_of_p,k,d):
    list_res = []
    for p in list_of_p:
        new_element = 1+(p-k+1)/(d-k)*(k+1)
        list_res.append(new_element)
    return list_res
```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.xlim(1, 6)
plt.ylim(1, 6)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python
plt.scatter(cardinal_list,volume_sampling_fro_list )
plt.show()
```

```python
plt.scatter(p_eff_list,projection_dpp_fro_list )
plt.show()
```

```python

```

    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

```python

```

### The case k = 2, projection spectrum

```python
T = 8 # The number of experiments
versions_number = 3
k = 2
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,0.1,0.1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 4)
plt.ylim(1, 4)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
plt.xlabel("Theoretical upper bound for effective kernel", fontsize=12)
plt.ylabel("Expectations of approximation error using projection DPP", fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.xlim(1, 4)
plt.ylim(1, 4)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 2, smooth spectrum

```python
T = 8 # The number of experiments
versions_number = 3
k = 2
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,10,0.1,0.1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 4)
plt.ylim(1, 4)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.xlim(1, 4)
plt.ylim(1, 4)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 2, ultra-smooth spectrum

```python
T = 8 # The number of experiments
versions_number = 3
k = 2
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([0.1,0.1,0.1,0.1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 4)
plt.ylim(1, 4)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 4)
plt.ylim(1, 4)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 3, projection spectrum

```python
T = 8 # The number of experiments
versions_number = 3
k = 3
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,100,0.1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 5)
plt.ylim(1, 5)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 5)
plt.ylim(1, 5)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

### The case k = 3, smooth spectrum

```python
T = 8 # The number of experiments
versions_number = 3
k = 3
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,10,1,0.1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 5)
plt.ylim(1, 5)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 5)
plt.ylim(1, 5)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 3, ultra smooth spectrum

```python
T = 8 # The number of experiments
versions_number = 3
k = 3
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([0.1,0.1,0.1,0.1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 5)
plt.ylim(1, 5)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.xlim(1, 5)
plt.ylim(1, 5)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

### The case k = 4, projection spectrum

```python
T = 4 # The number of experiments
versions_number = 3
k = 4
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,100,100,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 6)
plt.ylim(1, 6)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 6)
plt.ylim(1, 6)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

### The case k = 4, smooth spectrum

```python
T = 4 # The number of experiments
versions_number = 3
k = 4
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([1000,100,10,1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 6)
plt.ylim(1, 6)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 6)
plt.ylim(1, 6)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 4, projection spectrum, full cardinal

```python
T = 3 # The number of experiments
versions_number = 3
k = 4
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,100,100,0.1,0.1],cov_test)))
static_list = list(range(21,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = [20,20,20]
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 6)
plt.ylim(1, 6)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 6)
plt.ylim(1, 6)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 2, projection spectrum, non constant spectrum after k+1

```python
T = 8 # The number of experiments
versions_number = 3
k = 2
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,1,1,0.1,0.1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 4)
plt.ylim(1, 4)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 5)
plt.ylim(1, 5)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```

### The case k = 3, projection spectrum, non constant spectrum after k+1

```python
T = 8 # The number of experiments
versions_number = 3
k = 3
emperical_results = np.zeros((versions_number*T,4))

volume_sampling_fro_list = []
projection_dpp_fro_list = []
p_eff_list = []
cardinal_list = []
cov_1 = np.diag(np.concatenate(([100,100,100,10,5,1],cov_test)))
static_list = list(range(10,21))

```

```python
volume_sampling_fro_list = []
projection_dpp_fro_list = []
effective_kernel_fro_list = []
p_eff_list = []
cardinal_list = []
for t in range(T):
    print("t")
    print(t)
    random_cardinal_list = list(np.random.choice(static_list, 3))
    NAL_1 = Numrerical_Analysis_DPP(N,real_dim,r,k,versions_number,mean,cov_1,lv_scores_vector,random_cardinal_list)

    projection_DPP_res_fro_1 = NAL_1.get_expected_error_fro_for_projection_DPP()
    volume_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_volume_sampling()
    effective_kernel_sampling_res_fro_1 = NAL_1.get_expected_error_fro_for_effective_kernel_sampling()

#    upper_tight_bound_projection_DPP_res_fro_1 = NAL_1.get_tight_upper_bound_error_fro_for_projection_DPP()

#    alpha_sum_res_1 = NAL_1.get_alpha_sum_k_leverage_scores(1)

#    sum_U_res_1 = NAL_1.get_sum_k_leverage_scores()
    p_eff_res_1 = NAL_1.get_p_eff_leverage_scores()

    effective_kernel_fro_list.append(list(effective_kernel_sampling_res_fro_1))

    volume_sampling_fro_list.append(list(volume_sampling_res_fro_1))
    projection_dpp_fro_list.append(list(projection_DPP_res_fro_1))
    p_eff_list.append(list(p_eff_res_1))
    cardinal_list.append(random_cardinal_list)
    print("next")

```

    t
    0
    next
    t
    1
    next
    t
    2
    next
    t
    3
    next
    t
    4
    next
    t
    5
    next
    t
    6
    next
    t
    7
    next

```python
flattened_cardinal_list= [item for items in cardinal_list for item in items]
flattened_p_eff_list= [item for items in p_eff_list for item in items]

```

```python
theoretical_projection_DPP_error_bound_list = from_p_eff_to_error_bound(flattened_cardinal_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.xlim(1, 5)
plt.ylim(1, 5)

plt.scatter(theoretical_projection_DPP_error_bound_list,projection_dpp_fro_list )
plt.scatter(theoretical_projection_DPP_error_bound_list,volume_sampling_fro_list)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for projection DPP', fontsize=12)
plt.ylabel('Expectations of approximation error using projection DPP ', fontsize=12)
plt.show()

```

```python
theoretical_effective_kernel_error_bound_list = from_p_eff_to_error_bound_2(flattened_p_eff_list,k,real_dim)

fig, ax = plt.subplots()

line = mlines.Line2D([0, 1], [0, 1], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)

plt.scatter(theoretical_effective_kernel_error_bound_list,effective_kernel_fro_list)
plt.scatter(theoretical_effective_kernel_error_bound_list,volume_sampling_fro_list)
plt.xlim(1, 5)
plt.ylim(1, 5)
#plt.scatter(border_array,border_array)
plt.xlabel('Theoretical upper bound for effective kernel', fontsize=12)
plt.ylabel('Expectations of approximation error using effective kernel ', fontsize=12)
plt.show()
```

```python

```