```
from discrete_probability import *
```

First we want to call `init_printing` in `sympy` to get SymPy to output nice MathJax LaTeX.

```
import sympy
sympy.init_printing()
```

Now we'll define our simple example with data to use later.

```
E, H, S = variables = map(Variable, 'EHS')
e, e_ = E
h, h_ = H
s, s_ = S
data_header = [H, S, E]
data_samples = [
    [True, False, True],
    [True, False, True],
    [False, True, False],
    [False, False, True],
    [True, False, False],
    [True, False, True],
    [False, False, False],
    [True, False, True],
    [True, False, True],
    [False, False, True],
    [True, False, True],
    [True, True, True],
    [True, False, True],
    [True, True, True],
    [True, False, True],
    [True, False, True]]
data_assignments = data_to_assignments(data_header, data_samples)
```

A quick example of using SymPy with pyDiscreteProbability is enforcing simplified fractions for probabilities. For example, here is the table from the basic example.

```
P = Table(variables)
P.learn_from_complete_data(data_assignments)
P
```

    H     | S     | E     | P(H,S,E)
    ---------------------------------
    True  | True  | True  | 0.125
    True  | True  | False | 0.0
    True  | False | True  | 0.5625
    True  | False | False | 0.0625
    False | True  | True  | 0.0
    False | True  | False | 0.0625
    False | False | True  | 0.125
    False | False | False | 0.0625

By using the SymPy number system we can let SymPy handle and preserve the original fractions instead of relying on floating point numbers.

```
P = Table(variables, number_system=sympy_number_system)
P.learn_from_complete_data(data_assignments)
P
```

    H     | S     | E     | P(H,S,E)
    ---------------------------------
    True  | True  | True  | 1/8
    True  | True  | False | 0
    True  | False | True  | 9/16
    True  | False | False | 1/16
    False | True  | True  | 0
    False | True  | False | 1/16
    False | False | True  | 1/8
    False | False | False | 1/16

The number system is preserved through operations as well.

```
P(E,H)
```

    H     | E     | P(H,E)
    -----------------------
    True  | True  | 11/16
    True  | False | 1/16
    False | True  | 1/8
    False | False | 1/8

```
P(E|S,H)
```

    H     | S     || E     | P(E|H,S)
    ----------------------------------
    True  | True  || True  | 1
    True  | True  || False | 0
    True  | False || True  | 9/10
    True  | False || False | 1/10
    False | True  || True  | 0
    False | True  || False | 1
    False | False || True  | 2/3
    False | False || False | 1/3

We can also use SymPy to store expressions in our probability table. First we'll define the symbols that will form our probability expressions.

```
x, y, z = sympy.symbols('x y z')
```

Now ee'll create a simple joint marginal `Table`. We need to set it to ignore validity or else it will raise an error because the `Table` is not "valid" since the probabilities do not sum to one. We'll also tell it to use the SymPy number system. By default `Table` uses a `float` number system.

```
A, B, C = variables = map(Variable, 'ABC')
a, a_ = A
b, b_ = B
c, c_ = C
P = Table(variables, ignore_validity=True, number_system=sympy_number_system)
```

Now we'll define our probability table some simple expressions as a silly demonstration of using SymPy with the table.

```
P[a , b , c ] = 0.11
P[a , b , c_] = 0.05+z
P[a , b_, c ] = 0.07
P[a , b_, c_] = 0.21
P[a_, b , c ] = 0.32
P[a_, b , c_] = 0.04
P[a_, b_, c ] = sympy.sqrt(x)
P[a_, b_, c_] = y
P
```

    A     | C     | B     | P(A,C,B)
    ---------------------------------
    True  | True  | True  | 0.11
    True  | True  | False | 0.07
    True  | False | True  | z + 0.05
    True  | False | False | 0.21
    False | True  | True  | 0.32
    False | True  | False | sqrt(x)
    False | False | True  | 0.04
    False | False | False | y

Notice that the table output preserves the expressions thanks to SymPy. Now we'll define some constraints so we can demonstrate SymPy's solver. The following two constraints are simply to fully define the system.

```
constraint_one = sympy.Eq(P(a, b), sympy.S('0.15'))
constraint_one
```

```
constraint_two = sympy.Eq(x, sympy.S('2*y'))
constraint_two
```

This constraint below, however, is fundamental. It is an expression of the second fundamental axiom of probability which states that the sum of probabilities of all possible world instantiations must be equal to one.

```
second_axiom = sympy.Eq(P.total_probability(), sympy.S('1'))
second_axiom
```

Now we'll use SymPy's solver to find appropriate values for $x$ and $y$. I will admit that I couldn't get it to behave with the first fundamental axiom of probability which requires that the individual world probabilites be greater than or equal to zero. Instead the example is contrived to return positive solutions.

```
soln = sympy.solve([second_axiom, constraint_one, constraint_two])[0]
soln
```

To demonstrate the validity of the solution we'll redefine the table with it. The `normalize_number_system` method ensures that the number types all match the table's number system. This means it will convert the Python floats to SymPy floats.

```
P[a , b , c ] = 0.11
P[a , b , c_] = 0.05+soln[z]
P[a , b_, c ] = 0.07
P[a , b_, c_] = 0.21
P[a_, b , c ] = 0.32
P[a_, b , c_] = 0.04
P[a_, b_, c ] = sympy.sqrt(soln[x])
P[a_, b_, c_] = soln[y]
P.normalize_number_system()
print(P)
```

    A     | C     | B     | P(A,C,B)
    ---------------------------------
    True  | True  | True  | 0.110000000000000
    True  | True  | False | 0.0700000000000000
    True  | False | True  | 0.0400000000000000
    True  | False | False | 0.210000000000000
    False | True  | True  | 0.320000000000000
    False | True  | False | 0.191637528781298
    False | False | True  | 0.0400000000000000
    False | False | False | 0.0183624712187015

Now we can check that the solution satisfies the second axiom of probability.

```
print(P.total_probability())
print(P.is_valid)
```

    1.00000000000000
    True

* [https://github.com/vietjtnguyen/py-discrete-probability](https://github.com/vietjtnguyen/py-discrete-probability)
* [https://pypi.python.org/pypi/pyDiscreteProbability](https://pypi.python.org/pypi/pyDiscreteProbability)


# Volume Sampling vs projection DPP for low rank approximation
## Introduction
#### In this notebook we compare the volume sampling and projection DPP for low rank approximation.
We recall the result proved in the article [DRVW]:\\
Let S be a random subset of k columns of X chosen with probability: $$P(S) = \frac{1}{Z_{k}} det(X_{.,S}^{T}X_{.,S})$$ with $$Z_{k} = \sum\limits_{S \subset [N], |S| = k} det(X_{.,S}^{T}X_{.,S})$$
Then
$$\begin{equation}
E(\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2}) \leq (k+1)\| X - \pi_{k}(X) \|_{Fr}^{2}
\end{equation}$$

We can prove that the volume sampling distribution is a mixture of projection DPPs distributions..., in particular one projection DPP distribution stands out for the problem of low rank approximation: ....\\
For the moment, there is no analytical expression for $$\begin{equation}
E(\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2})
\end{equation}$$ under the distribution of projection DPP.\\
However, we can calculate this quantity using simulation on some matrices representing cloud points with some specific geometric constraints.

## I- Generating a cloud of points with geometric constraints
Let $$X \in R^{n \times m}$$ a matrix representing a cloud of points.
We can write the SVD of $$X = UDV^{T}$$

#### References
[DRVW] Deshpande, Amit and Rademacher, Luis and Vempala, Santosh and Wang, Grant - Matrix Approximation and Projective Clustering via Volume Sampling 2006

```python
import numpy as np
from itertools import combinations
from scipy.stats import binom
import scipy.special
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from IPython.display import display, HTML
```

### I-1- Givens Rotations generators
The function

```python
def t_func(q_i,q_j,q_ij,l_i,l_j):
    # t in section 3.1 Dhillon (2005)
    delta = np.power(q_ij,2)-(q_i-l_i)*(q_j-l_i)
    if delta<0:
        print(delta)
        print("error sqrt")
    t = q_ij - np.sqrt(delta)
    t = t/(q_j-l_i)
    return t

def G_func(i,j,q_i,q_j,q_ij,l_i,l_j,N):
    # Gitens Rotation
    G=np.eye(N)
    t = t_func(q_i,q_j,q_ij,l_i,l_j)
    c = 1/(np.sqrt(np.power(t,2)+1))
    s = t*c
    G[i,i]=c
    G[i,j]=s
    G[j,i]= -s
    G[j,j]= c
    return G
```

The following function is an implementation of the algorithm [] figuring in the article [] to generate an orthogonal matrix with a prescribed profile of leverage scores.
In fact this is a simplification of the algorithm .....

```python
def generate_orthonormal_matrix_with_leverage_scores(N,d,lv_scores_vector,versions_number):
    #Transforming an idendity matrix to an orthogonal matrix with prescribed lengths
    Q = np.zeros((N,d))
    previous_Q = np.zeros((versions_number,N,d))
    versionning_period = (int)(N/versions_number)
    for _ in range(0,d):
        Q[_,_] = 1

    I_sorting =  list(reversed(np.argsort(lv_scores_vector)))
    lv_scores_vector = np.asarray(list(reversed(np.sort(lv_scores_vector))))

    i = d-1
    j = d
    v_counter =0
    for t in range(N-1):
        delta_i = np.abs(lv_scores_vector[i] - np.power(np.linalg.norm(Q[i,:]),2))
        delta_j = np.abs(lv_scores_vector[j] - np.power(np.linalg.norm(Q[j,:]),2))
        q_i = np.power(np.linalg.norm(Q[i,:]),2)
        q_j = np.power(np.linalg.norm(Q[j,:]),2)
        q_ij = np.dot(Q[i,:],Q[j,:].T)
        l_i = lv_scores_vector[i]
        l_j = lv_scores_vector[j]
        G = np.eye(N)
        if t%versionning_period ==0:
            previous_Q[v_counter,:,:] = Q
            v_counter = v_counter +1
        if delta_i <= delta_j:
            l_k = q_i + q_j -l_i
            G = G_func(i,j,q_i,q_j,q_ij,l_i,l_k,N)
            Q = np.dot(G,Q)
            i = i-1
        else:
            l_k = q_i + q_j -l_j
            G = G_func(i,j,q_j,q_i,q_ij,l_j,l_k,N)
            Q = np.dot(G,Q)
            j = j+1

    return Q,previous_Q
```

The following function allows to estimate the leverage scores for an orthogonal matrix Q: the function calculates the diagonoal of the matrix $$Q Q^{T}$$

```python
def estimate_leverage_scores_from_orthogonal_matrix(Q):
    [N,_] = np.shape(Q)
    lv_scores_vector = np.zeros((N,1))
    lv_scores_vector = np.diag(np.dot(Q,np.transpose(Q)))
    lv_scores_vector = np.asarray(list(reversed(np.sort(lv_scores_vector))))
    return lv_scores_vector
def estimate_sum_first_k_leverage_scores(Q,k):
    lv_scores_vector = estimate_leverage_scores_from_orthogonal_matrix(Q)
    res = np.sum(lv_scores_vector[0:k])
    return res
```

## I-2- Application of the algorithm
We use the previous function (generate_orthonormal_matrix_with_leverage_scores) to generate a cloud of points represented by an orthogonal matrix Q and we keep a track of the evolution of the matrix Q in the algorithm: at the beginning Q is the rectangular identity matrix (the first d rows are the vectors of canonical basis and the other rows are zeros) the algorithm apply a Givens rotation to Q at each step and at the end the resulting matrix is an orthogonal matrix with the presribed leverage scores.
In this example we request a matrix Q with a uniform profil of leverage scores and we keep 5 matrices representing the 'history' of the matrix Q in the algorithm: the most important property of these 5 matrices is that their leverage score profiles are decreasing with respect to the Schur-order or (the majorization).
The number of points is 100 and the dimension of the space is 2.

```python
real_dim = 10
k = 4
r = 5
versions_number = 5
lv_scores_vector = k/real_dim*np.ones(real_dim)
[Q,previous_Q] = generate_orthonormal_matrix_with_leverage_scores(real_dim,k,lv_scores_vector,versions_number)
```

```python
# Note: If we change versions_number we have to change the number of the lv_i
lv_0 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[0,:,:])
lv_1 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[1,:,:])
lv_2 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[2,:,:])
lv_3 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[3,:,:])
lv_4 = estimate_leverage_scores_from_orthogonal_matrix(previous_Q[4,:,:])
lv_5 = estimate_leverage_scores_from_orthogonal_matrix(Q)
index_list = list(range(real_dim))
```

In this example the objective is Q and the initialization is Q_1 (the rectangular identity)
We have with respect to the Schur-order (or the majorization):
$$Q \prec_{S} Q_5 \prec_{S} Q_4 \prec_{S} Q_3 \prec_{S} Q_2 \prec_{S} Q_1 $$

```python
plt.plot(index_list[0:10], lv_0[0:10], 'c--',index_list[0:10], lv_1[0:10], 'k--', index_list[0:10], lv_2[0:10], 'r--', index_list[0:10], lv_3[0:10], 'b--',index_list[0:10], lv_4[0:10], 'g--',index_list[0:10], lv_5[0:10], 'y--')
plt.xlabel('index')
plt.ylabel('leverage score')
cyan_patch = mpatches.Patch(color='cyan', label='Q_1')
black_patch = mpatches.Patch(color='black', label='Q_2')
red_patch = mpatches.Patch(color='red', label='Q_3')
blue_patch = mpatches.Patch(color='blue', label='Q_4')
green_patch = mpatches.Patch(color='green', label='Q_5')
yellow_patch = mpatches.Patch(color='yellow', label='Q')
plt.legend(handles=[cyan_patch,black_patch,red_patch,blue_patch,green_patch,yellow_patch])

plt.show()
```

## I-3- Extending the orthogonal matrices

For the task of low rank approximation, we have seen that only the information contained in the first right k eigenvectors of the matrix X are relevant. In the previous step we build only the first right k eigenvectors but we still need to complete these orthogonal matrices with d-k columns. We proceed as following:
Generate a random vector (Nx1) using independent standard gaussian variables,
Project this vector in the orthogonal of the span of Q
Normalize the obtained vector after the projection
Extend the matrix Q
Note that this procedure is not the unique way to extend the matrix Q to an orthogonal (Nxd) matrix.

```python
def extend_orthogonal_matrix(Q,d_target):
    [N,d] = np.shape(Q)
    Q_target = np.zeros((N,d))
    Q_target = Q
    delta = d_target - d
    for t in range(delta):
        Q_test = np.random.normal(0, 1, N)
        for _ in range(d):
            Q_test = Q_test - np.dot(Q_test,Q[:,_])*Q[:,_]
        Q_test = Q_test/np.linalg.norm(Q_test)
        Q_test = Q_test.reshape(N,1)
        Q_target = np.append(Q_target,Q_test,1)
    return Q_target

#extended_Q = extend_orthogonal_matrix(Q,r)
```

## I-4 - Constructing a dataset for every extended orthogonal matrix
The previous step allow us to build (N x d) orthogonal matrices such that the extracted (N x k) matrix have a prescribed profile of leverage scores.
Now we construct a cloud of point by assigning a covariance matrix D and a matrix V

```python
N = 100
mean = np.zeros((real_dim))
cov_test = 0.1*np.ones((real_dim-r))
cov = np.diag(np.concatenate(([10,10,4,1,0.1],cov_test)))
def contruct_dataset_from_orthogonal_matrix(multi_Q,N,target_d,cov,mean,versions_number):
    multi_X = np.zeros((versions_number,N,real_dim))
    for t in range(versions_number):
        test_X = np.random.multivariate_normal(mean, cov, N)
        [U,_,_] = np.linalg.svd(test_X, full_matrices=False)
        Q_test = extend_orthogonal_matrix(multi_Q[t,:,:],target_d)
        multi_X[t,:,:] = np.dot(np.dot(Q_test,cov),U.T).T
    return multi_X

```

```python
multi_X = contruct_dataset_from_orthogonal_matrix(previous_Q,N,real_dim,cov,mean,versions_number)
```

## II- Volume sampling vs Projection DPP for low rank approximation

```python
def approximation_error_function(V,Sigma,k,X_S):
    d = list(Sigma.shape)[0]
    Sigma = np.multiply(Sigma,Sigma)
    sub_V,_,_ = np.linalg.svd(X_S, full_matrices=False)
    H_hat = np.dot(V.T,sub_V)
    r_vector = np.ones(d) - np.multiply(np.linalg.norm(H_hat,axis = 1),np.linalg.norm(H_hat,axis = 1))
    approximation_error_ratio= np.sqrt(np.sum(np.multiply(Sigma,r_vector))/np.sum(Sigma[k:d]))
    return approximation_error_ratio
```

```python
def approximation_error_function_2(Sigma,k,X,X_S):
    d = list(Sigma.shape)[0]
    Sigma = np.multiply(Sigma,Sigma)
    sigma_S_temp = np.linalg.inv(np.dot(X_S.T,X_S))
    projection_S = np.dot(np.dot(X_S,sigma_S_temp),X_S.T)
    res_X = X - np.dot(projection_S,X)
    approximation_error_ratio = np.power(np.linalg.norm(res_X,'fro'),2)/np.sum(Sigma[k:d])
    return approximation_error_ratio
```

```python
def probability_approximation_error_for_projection_DPP(X,U,k,N,epsilon):
    V,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    volumes_array_sum = np.sum(volumes_array)
    volumes_array = volumes_array/volumes_array_sum
    #approximation_error_array = [approximation_error_function(V,Sigma,k,X[:,list(comb)]) for comb in combinations(range(N),k)]
    approximation_error_array = [approximation_error_function_2(Sigma,k,X,X[:,list(comb)]) for comb in combinations(range(N),k)]
    indices_below_epsilon = [index for index,test in enumerate(combinations(range(N),k)) if approximation_error_array[index] < 1+epsilon]
    probability_below_epsilon = np.sum(volumes_array[indices_below_epsilon])

    return probability_below_epsilon
```

## II-1 Cumulative distribution function of the approximation error for a sampling scheme

```python
k = 4
epsilons_array = np.arange(0, 8, 0.25)
number_of_epsilons = list(epsilons_array.shape)[0]
probability_array_U_0 = np.zeros(number_of_epsilons)
probability_array_X_0 = np.zeros(number_of_epsilons)
probability_array_U_1 = np.zeros(number_of_epsilons)
probability_array_X_1 = np.zeros(number_of_epsilons)
probability_array_U_2 = np.zeros(number_of_epsilons)
probability_array_X_2 = np.zeros(number_of_epsilons)
probability_array_U_3 = np.zeros(number_of_epsilons)
probability_array_X_3 = np.zeros(number_of_epsilons)
probability_array_U_4 = np.zeros(number_of_epsilons)
probability_array_X_4 = np.zeros(number_of_epsilons)
U_0 = previous_Q[0,:,:].T
U_1 = previous_Q[1,:,:].T
U_2 = previous_Q[2,:,:].T
U_3 = previous_Q[3,:,:].T
U_4 = previous_Q[4,:,:].T
X_0 = multi_X[0,:,:]
X_1 = multi_X[1,:,:]
X_2 = multi_X[2,:,:]
X_3 = multi_X[3,:,:]
X_4 = multi_X[4,:,:]

```

```python
for i in range(number_of_epsilons):
    probability_array_X_0[i] = probability_approximation_error_for_projection_DPP(X_0,X_0,k,real_dim,epsilons_array[i])
    probability_array_U_0[i] = probability_approximation_error_for_projection_DPP(X_0,U_0[0:k, :],k,real_dim,epsilons_array[i])
    probability_array_X_1[i] = probability_approximation_error_for_projection_DPP(X_1,X_1,k,real_dim,epsilons_array[i])
    probability_array_U_1[i] = probability_approximation_error_for_projection_DPP(X_1,U_1[0:k, :],k,real_dim,epsilons_array[i])
    probability_array_X_2[i] = probability_approximation_error_for_projection_DPP(X_2,X_2,k,real_dim,epsilons_array[i])
    probability_array_U_2[i] = probability_approximation_error_for_projection_DPP(X_2,U_2[0:k, :],k,real_dim,epsilons_array[i])
    probability_array_X_3[i] = probability_approximation_error_for_projection_DPP(X_3,X_3,k,real_dim,epsilons_array[i])
    probability_array_U_3[i] = probability_approximation_error_for_projection_DPP(X_3,U_3[0:k, :],k,real_dim,epsilons_array[i])
    probability_array_X_4[i] = probability_approximation_error_for_projection_DPP(X_4,X_4,k,real_dim,epsilons_array[i])
    probability_array_U_4[i] = probability_approximation_error_for_projection_DPP(X_4,U_4[0:k, :],k,real_dim,epsilons_array[i])

```

```python
plt.plot(epsilons_array, probability_array_X_0, 'r--', epsilons_array, probability_array_U_0, 'b--',epsilons_array, probability_array_X_1, 'g--',epsilons_array, probability_array_U_1, 'y--')
plt.xlabel('epsilon')
plt.ylabel('probability')

red_patch = mpatches.Patch(color='red', label='Volume Sampling k-DPP for 0')
blue_patch = mpatches.Patch(color='blue', label='(1,2)-projection DPP for 0')
green_patch = mpatches.Patch(color='green', label='Volume Sampling k-DPP for 1')
yellow_patch = mpatches.Patch(color='yellow', label='(1,2)-projection DPP for 1')

plt.legend(handles=[red_patch,blue_patch,green_patch,yellow_patch])

plt.show()
```

```python
plt.plot(epsilons_array, probability_array_X_0, 'r--', epsilons_array, probability_array_U_0, 'b--',epsilons_array, probability_array_X_4, 'g--',epsilons_array, probability_array_U_4, 'y--')
plt.xlabel('epsilon')
plt.ylabel('probability')

red_patch = mpatches.Patch(color='red', label='Volume Sampling k-DPP for 0')
blue_patch = mpatches.Patch(color='blue', label='(1,2)-projection DPP for 0')
green_patch = mpatches.Patch(color='green', label='Volume Sampling k-DPP for 4')
yellow_patch = mpatches.Patch(color='yellow', label='(1,2)-projection DPP for 4')

plt.legend(handles=[red_patch,blue_patch,green_patch,yellow_patch])

plt.show()
```

## II-2 Estimation of the expected value of the approximation error for a sampling scheme

```python
def expected_approximation_error_for_sampling_scheme(X,U,k,N):
    _,Sigma,_ = np.linalg.svd(X, full_matrices=False)
    volumes_array = [np.abs(np.linalg.det(np.dot(U[:,list(comb)].T,U[:,list(comb)]))) for comb in combinations(range(N),k)]
    volumes_array_sum = np.sum(volumes_array)
    volumes_array = volumes_array/volumes_array_sum
    approximation_error_array = [approximation_error_function_2(Sigma,k,X,X[:,list(comb)]) for comb in combinations(range(N),k)]
    expected_value = np.dot(approximation_error_array,volumes_array)
    return expected_value
```

```python
ex_X_0 = expected_approximation_error_for_sampling_scheme(X_0,X_0,k,real_dim)
ex_X_1 = expected_approximation_error_for_sampling_scheme(X_1,X_1,k,real_dim)
ex_X_2 = expected_approximation_error_for_sampling_scheme(X_2,X_2,k,real_dim)
ex_X_3 = expected_approximation_error_for_sampling_scheme(X_3,X_3,k,real_dim)
ex_X_4 = expected_approximation_error_for_sampling_scheme(X_4,X_4,k,real_dim)

ex_U_0 = expected_approximation_error_for_sampling_scheme(X_0,U_0,k,real_dim)
ex_U_1 = expected_approximation_error_for_sampling_scheme(X_1,U_1,k,real_dim)
ex_U_2 = expected_approximation_error_for_sampling_scheme(X_2,U_2,k,real_dim)
ex_U_3 = expected_approximation_error_for_sampling_scheme(X_3,U_3,k,real_dim)
ex_U_4 = expected_approximation_error_for_sampling_scheme(X_4,U_4,k,real_dim)
#ex_U_4 = expected_approximation_error_for_sampling_scheme(X_4,U_4,d,k,N)
```

```python
sum_U_0 = estimate_sum_first_k_leverage_scores(U_0.T,k)
sum_U_1 = estimate_sum_first_k_leverage_scores(U_1.T,k)
sum_U_2 = estimate_sum_first_k_leverage_scores(U_2.T,k)
sum_U_3 = estimate_sum_first_k_leverage_scores(U_3.T,k)
sum_U_4 = estimate_sum_first_k_leverage_scores(U_4.T,k)

determinstic_error_ratio_0 = 1/(1-k+sum_U_0)
determinstic_error_ratio_1 = 1/(1-k+sum_U_1)
determinstic_error_ratio_2 = 1/(1-k+sum_U_2)
determinstic_error_ratio_3 = 1/(1-k+sum_U_3)
determinstic_error_ratio_4 = 1/(1-k+sum_U_4)
```

```python
results = [["Dataset","Using Volume Sampling","Using Projection DPP","k-sum","1/(1-epsilon)"],["X_0",ex_X_0,ex_U_0,sum_U_0,determinstic_error_ratio_0],["X_1",ex_X_1,ex_U_1,sum_U_1,determinstic_error_ratio_1],
          ["X_2",ex_X_2,ex_U_2,sum_U_2,determinstic_error_ratio_2],["X_3",ex_X_3,ex_U_3,sum_U_3,determinstic_error_ratio_3],["X_4",ex_X_4,ex_U_4,sum_U_4,determinstic_error_ratio_4]]
display(HTML(
    '<center><b>The expected approximation error (divided by the optimal error) according to a sampling scheme for different distribution</b><br><table><tr>{}</tr></table>'.format(
        '</tr><tr>'.join(
            '<td>{}</td>'.format('</td><td>'.join(str(_) for _ in row)) for row in results)
        )
 ))
```

The expected approximation error (divided by the optimal error) according to a sampling scheme for different distribution

Dataset	Using Volume Sampling	Using Projection DPP	k-sum	1/(1-epsilon)

X_0	4.79156186365	1.0	4.0	1.0

X_1	4.79620343969	1.38956179844	3.4	2.5

X_2	4.80581330659	1.80196677582	2.8	-5.0

X_3	4.80075285204	1.80745205512	2.4	-1.66666666667

X_4	4.79721520993	2.13084745145	2.0	-1.0

#### Note: The expected approximation error is defined as : $$\sum_{S \subset [d]} P_{S}Err(S)$$
#### With $$Err(S) = \frac{\| X - \pi_{X_{.,S}}(X) \|_{Fr}^{2}}{\| X - X_{k} \|_{Fr}^{2}}$$

## III - Numerical analysis

In this section we use the functions developed previously to investigate the influence of two parameters: the spectrum of X and the k-leverage scores.

### III-1 The influence of the spectrum

In this subsection we compare the Volume Sampling distribution to the projection DPP distribution for different profile of the spectrum with k-leverage scores profile fixed.

#### III-1-1 The case of a projection spectrum
We mean by a projection spectrum matrix, a matrix with equal the first k singular values.
We observe that the two distributions are very similar.... \todo{reword}

```python

```

#### III-1-2 The case of a non-projection spectrum
We mean by a projection spectrum matrix, a matrix with equal the first k singular values.
We observe that the two distributions are very similar.... \todo{reword}

```python

```


```python
from ipywidgets import interactive, interact
import matplotlib.pyplot as plt
import numpy as np
import ipywidgets as widgets
import sympy as sym
import seaborn as sns
import plotly.graph_objects as go
import plotly.express as px
from plotly.offline import init_notebook_mode, iplot
from numba import jit

init_notebook_mode(connected=True)
jit(nopython=True, parallel=True)
sns.set()

```

```python

class plot():

    def __init__(self, preWidgetN):

        self.N = preWidgetN

        n = np.linspace(0, self.N.value, self.N.value+1)

        a = (3**(n+1)+2**n)/(3**n+2.)

        a = (1/np.sqrt(5))*(((1+np.sqrt(5))/2)**n-((1-np.sqrt(5))/2)**n)

        b = (1/np.sqrt(5))*(((1+np.sqrt(5))/2)**(n+1)-((1-np.sqrt(5))/2)**(n+1))

        a = a/b

        self.trace = go.Scatter(x=n, y=a,
                            mode='markers',
                            name=r'$a$',
                            showlegend=True,
                               )

        layout = go.Layout(template='plotly_dark')

        self.fig = go.FigureWidget(data=[self.trace],
                                   layout = layout,
                                   #layout_yaxis_range=[-3 , 3]
                                  )

    def series(self, change):

        n = np.linspace(0, self.N.value, self.N.value+1)

        a = (3**(n+1)+2**n)/(3**n+2.)

        a = (1/np.sqrt(5))*(((1+np.sqrt(5))/2)**n-((1-np.sqrt(5))/2)**n)

        a = (1/np.sqrt(5))*(((1+np.sqrt(5))/2)**n-((1-np.sqrt(5))/2)**n)

        b = (1/np.sqrt(5))*(((1+np.sqrt(5))/2)**(n+1)-((1-np.sqrt(5))/2)**(n+1))

        a = a/b

        with self.fig.batch_update():
            self.fig.data[0].x = n
            self.fig.data[0].y = a
            self.fig.data[0].name = r'$n\in (0, {%s})$' %(self.N.value)

        return

    def show(self):
        self.N.observe(self.series, names='value')
        display(self.N, self.fig)
        return
```

```python
N = widgets.IntSlider(min=0, max=50, step=1, value=0, description='n')

p = plot(N)

p.show()
```

    IntSlider(value=0, description='n', max=50)

    FigureWidget({
        'data': [{'mode': 'markers',
                  'name': '$a$',
                  'showlegend': True,…

```python

```


```python
# Método para resolver las energías y eigenfunciones de un sistema cuántico numéricamente
# Modelado Molecular 2
# By: José Manuel Casillas Martín
import numpy as np
from sympy import *
from sympy import init_printing; init_printing(use_latex = 'mathjax')
import matplotlib.pyplot as plt
```

```python
# Variables utilizadas
var('x l m hbar w k')
```

$$\left ( x, \quad l, \quad m, \quad \hbar, \quad w, \quad k\right )$$

```python
def Metodo_variaciones():
    print('En este problema están definidas como variables la masa, el parámetro l (se define), el parámetro k(se optimiza) y x')
    print('')
    # La energía cinética está definida por: K=(-hbar**2)/(2*m)*diff(fx,x,2)
    print('La energía cinética esta definida como: K=(-hbar**2)/(2*m)*diff(f(x),x,2)');print('')

    # Declarar el potencial
    V=sympify(input('Introduce la función de potencial: '));print('')
    lim_inf_V=sympify(input('¿Cuál es el límite inferior de la función potencial? '))
    lim_sup_V=sympify(input('¿Cuál es el límite superior de la función potencial? '));print('')

    n = int(input('Introduce el número de funciones que vas a utilizar para resolver el problema: '));print('')
    # Lista para ingresar las funciones
    f=[]
    # Matriz con integral de solapamiento
    Sm=[]
    # Matriz con integral de intercambio
    Hm=[]

    print('Ahora vamos definir las constantes del problema');print('')
    mass=input('¿Cuánto es la masa de tu partícula? ')
    large=input('Define el parámetro l: ');print('')

    # Declarar funciones y límites de dichas funciones
    lim_inf=[]
    lim_sup=[]
    for i in range(n):
        f.append((input('Introduce la función %d: ' %(i+1))))
        lim_inf.append(input('¿Cuál es el límite inferior de la función? '))
        lim_sup.append(input('¿Cuál es el límite superior de la función? '));print('')
    f=sympify(f)
    lim_inf=sympify(lim_inf)
    lim_sup=sympify(lim_sup)
    # Para partícula en un pozo de potencial de 0 a l
    # El siguiente ciclo for resuelve las integrales para formar las matrices Sij (Integrales de solapamiento)
    # y Hij(integrale de intercambio)

    # Aproximación de las energías

    li=0
    ls=0
    for i in range(n):
        for j in range(n):
            integrandoT=(f[i])*((-hbar**2)/(2*m)*diff(f[j],x,2))
            integrandoV=(f[i])*V*(f[j])
            integrandoN=(f[i])*f[j]
            # Definir los limites de integracion

            # Límites inferiores
            if lim_inf[i].subs({l:large})<=lim_inf[j].subs({l:large}):
                li=lim_inf[j]
                if li.subs({l:large})>=lim_inf_V.subs({l:large}):
                    liV=li
                else:
                    liV=lim_inf_V
            if lim_inf[i].subs({l:large})>=lim_inf[j].subs({l:large}):
                li=lim_inf[i]
                if li.subs({l:large})>=lim_inf_V.subs({l:large}):
                    liV=li
                else:
                    liV=lim_inf_V
            # Límites superiores
            if lim_sup[i].subs({l:large})>=lim_sup[j].subs({l:large}):
                ls=lim_sup[j]
                if ls.subs({l:large})<=lim_sup_V.subs({l:large}):
                    lsV=ls
                else:
                    lsV=lim_sup_V
            if lim_sup[i].subs({l:large})<=lim_sup[j].subs({l:large}):
                ls=lim_sup[i]
                ls=lim_sup[j]
                if ls.subs({l:large})<=lim_sup_V.subs({l:large}):
                    lsV=ls
                else:
                    lsV=lim_sup_V

            c=Integral(integrandoT,(x,li,ls))
            e=Integral(integrandoV,(x,liV,lsV))
            g=c+e
            d=Integral(integrandoN,(x,li,ls))
            g=g.doit()
            Hm.append(g)
            d=d.doit()
            Sm.append(d)
    Sm=np.reshape(Sm,(n,n))
    Hm=np.reshape(Hm,(n,n))

    # Matriz M: (Hij-Sij)*w
    M=(Hm-Sm*w)
    H=sympify(Matrix(M))
    Hdet=H.det()
    # Resolver el determinante para encontrar las energías
    E=solve(Hdet,w)

    # Ordenar energías
    Eord=solve(Hdet,w)

    energies=np.zeros(n)

    for i in range (n):
        energies[i]=E[i].subs({m: mass, l: large, hbar:1.0545718e-34})
    energies_ord=sorted(energies)
    for i in range(n):
        for j in range(n):
            if energies[i]==energies_ord[j]:
                Eord[i]=E[j]

    # Matriz de constantes para todas las eigenfunciones
    c=zeros(n)
    for i in range(n):
        for j in range(n):
            c[i,j]=Symbol('c %d %d' %(i+1,j+1))

    # Solución a esas constantes
    sol=[]
    for i in range (n):
        a=np.reshape(c[0+n*i:(n)+n*i],(n))
        SE=Matrix(np.dot(M,a.transpose()))
        SE=sympify((SE.subs({w:Eord[i]})))
        sol.append(solve(SE,c[0+n*i:(n+1)+n*i]))

    if n!= 1:
        csol=zeros(n)
        CTS,cts,Cdet=[],[],[]
        for i in range (n):
            for j in range(n):
                csol[i,j]=(sol[i]).get(c[i,j])
                if csol[i,j] is None:
                    csol[i,j]=c[i,j]
                    CTS.append(c[i,j]); cts.append(c[i,j]); Cdet.append(c[i,j])

    # Impresión en pantalla de los resultados
    print('Matriz Hij')
    print(sympify(Matrix(Hm)));print('')
    print('Matriz Sij')
    print(sympify(Matrix(Sm)));print('')
    print('Energías ordenadas')
    print(Eord);print('')

    # Normalizar las funciones de onda y graficar

    graficar=input('Desea graficar las eigenfunciones calculadas: ');print('')
    if graficar=="si":
        if n>1:
            fa=(np.reshape(f,(n)))
            ef=csol*fa
            for i in range(n):
                integrando=ef[i]*ef[i]
                integ=Integral(integrando,(x,lim_inf[i],lim_sup[i]))
                integ=integ.doit()
                cts[i]=solve(integ-1,Cdet[i])
                if abs(cts[i][0])==cts[0][0]:
                    CTS[i]=cts[i][0]
                else:
                    CTS[i]=cts[i][1]
                ef=ef.subs({Cdet[i]:CTS[i]})
            print('Constantes de cada una de las eigenfunciones (cada eigenfunción tiene una constante extra que se debe normalizar)')
            print(csol);print('')
            print('Para graficar se normalizaron las constantes mostradas anteriormente, cuyos resultados fueron:')
            print(CTS);print('')
            for i in range(n):
                plot(ef[i].subs({l:1}),xlim=(0,1),ylim=(-2,2),title='Eigenfunción: %d' %(i+1))

        # Falta automatizar los limites de las funciones para graficar y que te grafique la primer función
        if n==1:
            ct=Symbol('C22')
            ef=ct*f[0]
            integrando=(ef)*(ef)
            integ=Integral(integrando,(x,lim_inf[0],lim_sup[0]))
            integr=integ.doit()
            cte=solve(integr-1,ct)
            if cte[0].subs({l:large})>cte[1].subs({l:large}):
                ctr=cte[0]
            else:
                ctr=cte[1]
            ef=ef.subs({ct:ctr})
            #print('Constantes de cada una de las eigenfunciones (cada eigenfunción tiene una constante extra que se debe normalizar)')
            #print(csol);print('')
            #print('Para graficar se normalizó las constante mostrada anteriormente, cuyo resultado fue:')
            #print(CTS);print('')
            plot(ef.subs({l:1}),xlim=(0,1),ylim=(-1,2))
    return()
```

```python
Metodo_variaciones()
```

```python

```


This notebook is coded in `python`, a programmming language built around the idea of `modules` where most of the code resides.

You can write your modules or use `library` modules, in any case you access _functions_ or _classes_ defined in modules, `import`-ing them from the modules.

That is said to explain why we import three functions  from the `math` module (and a constant too), and some other stuff too...

```
%pylab --no-import-all inline

from math import exp, log, pi, sqrt

import sympy as sy ; sy.init_printing(use_latex=True)
from IPython.display import display

import style ; style.clean()
```

    Populating the interactive namespace from numpy and matplotlib

<style type="text/css">
div.input {
width: 105ex; /* about 80 chars + buffer */
}

div.text_cell {
width: 105ex /* instead of 100%, */
}

div.text_cell_render {
font-family: Cambria, Candara, serif;
font-size: 14pt
line-height: 145%; /* added for some line spacing of text. */
width: 105ex; /* instead of 'inherit' for shorter lines */
}

/* Set the size of the headers */
div.text_cell_render h1 {
font-size: 20pt;
}

div.text_cell_render h2 {
font-size: 18pt;
}

.CodeMirror {
font-family: Consolas, monospace;
width: 105ex;
}

.rendered_html ol {list-style:decimal; margin: 1em 2em;}

</style>

# Damped response for a given $x_0$

What is the analytical expression of the response of a damped system
subject to the (quite common) initial conditions $x(0)=x_0$ and $\dot x(0)=0$?

We start by creating a set of `symbols` and giving them good names...

```
A, B, t, z, wn, wd, x0 = sy.symbols(
    'A, B, t, zeta, omega_n, omega_d, x_0')
display((A, B, t, z, wn, wd, x0))
```

Then we give the name `u` to the symbolic representation of the displacement,
and the name `v` to the first time derivative of `u`.

```
u = sy.exp(-z*wn*t)*(A*sy.cos(wd*t)+B*sy.sin(wd*t))
v = u.diff(t)
display(u)
```

Now,

 - we form an equation substituting $t=0$ in `u` and `v` and subtracting $x_0$
   from the first equation
 - we ask our `sympy` aka `sy` library to solve the system of equations

```
eq = (u.subs(t,0)-x0, v.subs(t,0))
coeff = sy.solve(eq, A,B)
display(coeff)
display(u.subs(coeff).collect(x0))
display(u.subs(coeff).collect(x0).subs(wn,wd/sy.sqrt(1-z*z)))
```

To plot the response we need to get rid of all the symbols in our expression,
except for $t$, so we choose an arbitrary set of values for the free variables...

```
values = {x0:1.0, z:0.05, wd:2*sy.pi, wn:2*sy.pi/sy.sqrt(1-z*z)}
display(values)
```

At the end, we display the analytical expression of $x(t)$ and plot

 1. $x(t)$,

 2. $\dot x(t)$.

```
display(u.subs(coeff).subs(values))
sy.plot(u.subs(coeff).subs(values), (t,0,5.99));
sy.plot(v.subs(coeff).subs(values), (t,0,5.99)  );
```

# Logarithmic decrement

We assume that our system has been deformed by the application of some static load,
so that at time $t=0$ the initial conditions are $x(0)=x_0$ and $\dot x(0)=0$.

At $t=0$ the static force is suddenly released, we observe an oscillatory behaviour
as we have detailed in the preceding section.

and we are able to measure the peak displacement after $n$ cycles, $x_n$.

## Specify data

The following code gives appropriate names to values measured or computed from
expressions, the value of the logarithmic decrement is eventually printed.

```
n = 3
x0 = 8.00
xn = 3.11
dn = log(x0/xn)
print "delta_n =",dn
```

    delta_n = 0.944818815489

The exact expression for $\zeta$ is

$$  \zeta = \sqrt{\frac{\delta_n^2}{(2 n \pi)^2 + \delta_n^2}}
  \text{ with }  \delta_n = \log\frac{x_0}{x_n}.$$

We've already computed $\delta_n$ in the previous code cell
so we can write the numerator and the denominator of the radicand
and eventually compute the _"exact"_ value of $\zeta$.

```
num = dn*dn
den = 4 * n*n * pi*pi + dn*dn
z = sqrt(num/den)
print "zeta = %f"%(z*100,)
```

    zeta = 5.006135

## Approximate expression

The linearized expression for $\zeta$, that we have found using the first
and second terms of the McLaurin series expressing $\exp(a\zeta)=1+a\zeta$, is

$$\zeta \approx \frac{1}{2 n \pi}\,\frac{x_0-x_n}{x_n}.$$

We know that this approximation is good for low damping and few cycles...

```
print "Estimate", 100 * (x0-xn) / xn / ( 2*n* pi )
print "error (z/est.)",z /((x0-xn) / xn / ( 2*n* pi ))
```

    Estimate 8.34156132604
    error (z/est.) 0.600143600147

## Iterative solution

We write, formally,

$$\zeta = \xi\sqrt{1-\zeta^2}, \text{ with }\xi = \frac{1}{2 n \pi} \log\frac{x_0}{x_n}$$

and derive an iterative formulation

$$\zeta_{i+1} = \xi\sqrt{1-\zeta_i^2}, \text{ with } i = 0, 1, 2, \ldots$$

Note that, for $\zeta_0 = 0$, it is $\zeta_1 = \xi$.
In theory, we have to check the convergence of the sequence and stop when the
convergence is reached, but in our cases the convergence is soo fast that we
are safe implementing the solution in a `for` loop

```
csi= dn/2/n/pi
z_i = csi

print "      zeta_0 = 0.00000000000000%"
print "csi = zeta_1 = %16.14f%%" % (csi*100,)
for i in range(2, 6):
    z_i = csi*sqrt(1-z_i*z_i)
    print '      zeta_%d = %16.14f%%'%(i,100*z_i)
```

          zeta_0 = 0.00000000000000%
    csi = zeta_1 = 5.01241949370850%
          zeta_2 = 5.00611884484703%
          zeta_3 = 5.00613468475389%
          zeta_4 = 5.00613464495722%
          zeta_5 = 5.00613464505721%

```

```


Integration by parts is another technique for simplifying integrands. As we saw in previous posts, each differentiation rule has a corresponding integration rule. In the case of integration by parts, the corresponding differentiation rule is the Product Rule. The technique of integration by parts allows us to simplify integrands of the form:
$$ \int f(x) g(x) dx $$

Examples of this form include:
$$ \int x \cos{x} \space dx, \qquad \int e^x \cos{x} \space dx, \qquad \int x^2 e^x \space dx $$

As integration by parts is the product rule applied to integrals, it helps to state the Product Rule again. The Product Rule is defined as:
$$ \frac{d}{dx} \big[ f(x)g(x) \big] = f^{\prime}(x) g(x) + f(x) g^{\prime}(x) $$

When we apply the product rule to indefinite integrals, we can restate the rule as:

$$ \int \frac{d}{dx} \big[f(x)g(x)\big] \space dx = \int \big[f^{\prime} g(x) + f(x) g^{\prime}(x) \big] \space dx $$

Then, rearranging so we get $f(x)g^{\prime}(x) \space dx$ on the left side of the equation:

$$ \int f(x)g^{\prime}(x) \space dx = \int \frac{d}{dx} \big[f(x)g(x)\big] \space dx - \int f^{\prime}(x)g(x) \space dx $$

Which gives us the integration by parts formula! The formula is typically written in differential form:

$$ \int u \space dv = uv - \int v \space du $$

## Examples

The following examples walkthrough several problems that can be solved using integration by parts. We also employ the wonderful [SymPy](https://www.sympy.org/en/index.html) package for symbolic computation to confirm our answers. To use SymPy later to verify our answers, we load the modules we will require and initialize several variables for use with the SymPy library.

```python
from sympy import symbols, limit, diff, sin, cos, log, tan, sqrt, init_printing, plot, integrate
from mpmath import ln, e, pi, cosh, sinh

init_printing()
x = symbols('x')
y = symbols('y')
```

Example 1: Evaluate the integrand $ \int x \sin{\frac{x}{2}} \space dx $

Recalling the differential form of the integration by parts formula, $ \int u \space dv = uv - \int v \space du $, we set $u = x$ and $dv = \sin{\frac{x}{2}}$

Solving for the derivative of $u$, we arrive at $du = 1 \space dx = dx$. Next, we find the antiderivative of $dv$. To find this antiderivative, we employ the Substitution Rule.

$$ u = \frac{1}{2}x, \qquad du = {1}{2} \space dx, \qquad \frac{du}{dx} = 2 $$
$$ y = \sin{u}, \qquad dy = -\cos{u} \space du, \qquad \frac{dy}{du} = -\cos{u} $$

Therefore, $v = -2 \cos{\frac{x}{2}}$

Entering these into the integration by parts formula:

$$ -2x\cos{\frac{x}{2}} - (-2)\int \cos{\frac{x}{2}} $$

Then, solving for the integrand $\int \cos{\frac{x}{2}}$, we employ the Substitution Rule again as before to arrive at $2\sin{\frac{x}{2}}$ (the steps in solving this integrand are the same as before when we solved for $\int \sin{\frac{x}{2}}$). Thus, the integral is evaluated as:

$$ -2x\cos{\frac{x}{2}} + 4\sin{\frac{x}{2}} + C $$

Using SymPy's [`integrate`](https://docs.sympy.org/latest/modules/integrals/integrals.html), we can verify our answer is correct (SymPy does not include the constant of integration $C$).

```python
integrate(x * sin(x / 2), x)
```

Example 2: Evaluate $\int t^2 \cos{t} \space dt$

We start by setting $u = t^2$ and $dv = \cos{t}$. The derivative of $t^2$ is $2t$, thus $du = 2t \space dt$, or $\frac{du}{dt} = 2t$. Integrating $dv = \cos{t}$ gives us $v = \sin{t} \space du$. Entering these into the integration by parts formula:

$$ t^2 \sin{t} - 2\int t \sin{t} $$

Therefore, we must do another round of integration by parts to solve $\int t \sin{t}$.

$$ u = t, \qquad du = dt $$
$$ dv = \sin{t}, \qquad v = -\cos{t} \space du $$

Putting these together into the integration by parts formula with the above:

$$ t^2 \sin{t} - 2 \big(-t \cos{t} + \int \cos{t} \space dt \big) $$

Which gives us the solution:

$$ t^2 \sin{t} + 2t \cos{t} - 2 \sin{t} + C$$

As before, we can verify that our answer is correct by leveraging SymPy.

```python
t = symbols('t')
integrate(t ** 2 * cos(t), t)
```

Example 3: $\int x e^x \space dx$

Here, we set $u = x$ and $dv = e^x$. Therefore, $du = dx$ and $v = e^x \space dx$. Putting these together in the integration by parts formula:

$$ xe^x - \int e^x $$

As the integral of $e^x$ is just $e^x$, our answer is:

$$ xe^x - e^x + C $$

We can again verify our answer is accurate using SymPy.

```python
integrate(x * e ** x, x)
```

## References

[Hass, J. and Weir, M. (n.d.). Thomas' calculus. 13th ed.](https://amzn.to/2SuuPOz)

[Stewart, J. (2007). Essential calculus: Early transcendentals. Belmont, CA: Thomson Higher Education.](https://amzn.to/38dnRV0)