# **Save this file as studentid1_studentid2_lab#.ipynb**
(Your student-id is the number shown on your student card.)

E.g. if you work with 3 people, the notebook should be named:
12301230_3434343_1238938934_lab1.ipynb.

**This will be parsed by a regexp, so please double check your filename.**

Before you turn this problem in, please make sure everything runs correctly. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

**Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your names and email adresses below.**

```python
NAME = "Pascal Esser"
NAME2 = "Jana Leible"
NAME3 = "Tom de Bruijn"
EMAIL = "pascal.esser@student.uva.nl"
EMAIL2 = "jana.leible@web.de"
EMAIL3 = "tommdebruijn@gmail.com"
```

---

# Lab 3: Gaussian Processes and Support Vector Machines

### Machine Learning 1, September 2017

Notes on implementation:

* You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
* Please write your answers right below the questions.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Refer to last week's lab notes, i.e. http://docs.scipy.org/doc/, if you are unsure about what function to use. There are different correct ways to implement each problem!
* use the provided test boxes to check if your answers are correct

```python
%pylab inline
plt.rcParams["figure.figsize"] = [20,10]

import numpy as np
import matplotlib.pyplot as plt
```

    Populating the interactive namespace from numpy and matplotlib

## Part 1: Gaussian Processes

For part 1 we will be refer to Bishop sections 6.4.2 and 6.4.3. You may also want to refer to Rasmussen's Gaussian Process text which is available online at http://www.gaussianprocess.org/gpml/chapters/ and especially to the project found at https://www.automaticstatistician.com/index/ by Ghahramani for some intuition in GP.  To understand Gaussian processes, it is highly recommended understand how marginal, partitioned Gaussian distributions can be converted into conditional Gaussian distributions.  This is covered in Bishop 2.3 and summarized in Eqns 2.94-2.98.
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\ba}{\mathbf{a}}$

### Periodic Data

We will use the same data generating function that we used previously for regression.

```python
def true_mean_function(x):
    return np.cos(2 * pi * (x + 1))

def add_noise(y, sigma):
    return y + sigma * np.random.randn(len(y))

def generate_t(x, sigma):
    return add_noise(true_mean_function(x), sigma)

```

```python
sigma = 0.2
beta = 1.0 / pow(sigma, 2)
N_test = 100

x_test = np.linspace(-1, 1, N_test)
mu_test = np.zeros(N_test)
y_test = true_mean_function(x_test)
t_test = add_noise(y_test, sigma)

plt.plot(x_test, y_test, 'b-', lw=2)
plt.plot(x_test, t_test, 'go')
plt.show()

```

### 1. Sampling from the Gaussian process prior (30 points)
We will implement Gaussian process regression using the kernel function in Bishop Eqn. 6.63.

#### 1.1 k_n_m( xn, xm, thetas ) (5 points)
To start, implement function `k_n_m(xn, xm, thetas)` that takes scalars $x_n$ and $x_m$, and a vector of $4$ thetas, and computes the kernel function Bishop Eqn. 6.63 (10 points).  NB: usually the kernel function will take $D$ by $1$ vectors, but since we are using a univariate problem, this makes things easier.

```python
def k_n_m(xn, xm, thetas):
    k = thetas[0] * np.exp(-thetas[1] * (xn - xm) ** 2 / 2) + thetas[2] + \
        thetas[3] * xn * xm

    return k

```

```python

```

#### 1.2 computeK( X1, X2, thetas ) (10 points)
Eqn 6.60 is the marginal distribution of mean output of $N$ data vectors: $p(\mathbf{y}) = \mathcal{N}(0, \mathbf{K})$.  Notice that the expected mean function is $0$ at all locations, and that the covariance is a $N$ by $N$ kernel matrix $\mathbf{K}$.  Write a function `computeK(x1, x2, thetas)`
that computes the kernel matrix. Use k_n_m as part of an inner loop (of course, there are more efficient ways of computing the kernel function making better use of vectorization, but that is not necessary) (5 points).

```python
def computeK(x1, x2, thetas):

    K = np.zeros((len(x1), len(x2)))
    for i in range(len(x1)):
        for j in range(len(x2)):
            K[i, j] = k_n_m(x1[i], x2[j], thetas)

    return K

```

```python
### Test your function
x1 = [0, 1, 2]
x2 = [1, 2, 3, 4]
thetas = [1, 2, 3, 4]
K = computeK(x1, x2, thetas)

assert K.shape == (len(x1), len(x2)), "the shape of K is incorrect"

```

#### 1.3 Plot function samples (15 points)
Now sample mean functions at the x_test locations for the theta values in Bishop Figure 6.5, make a figure with a 2 by 3 subplot and make sure the title reflects the theta values (make sure everything is legible).  In other words, sample $\by_i \sim \mathcal{N}(0, \mathbf{K}_{\theta})$.  Make use of numpy.random.multivariate_normal().  On your plots include the expected value of $\by$ with a dashed line and fill_between 2 standard deviations of the uncertainty due to $\mathbf{K}$ (the diagonal of $\mathbf{K}$ is the variance of the model uncertainty) (15 points).

```python
#numbers from Bishop page 308
thetas_inp = np.array(
    [(1.0, 4.0, 0.0, 0.0), (9.0, 4.0, 0.0, 0.0), (1.0, 64.0, 0.0, 0.0),
     (1.0, 0.25, 0.0, 0.0), (1.0, 4.0, 10.0, 0.0), (1.0, 4.0, 0.0, 5.0)])

fig = plt.figure()
# run over all thetas and create the prints at the same time
for i in range(thetas_inp.shape[0]):

    K_theta_i = computeK(x_test, x_test, thetas_inp[i])
    # sample 5 times (as it is done in Bishop)
    for _ in range(5):
        y = np.random.multivariate_normal(np.zeros(len(K_theta_i)), K_theta_i)
        # what subplot to write on
        plt.subplot(2, 3, i + 1)
        plt.plot(x_test, y, linewidth=0.7)
    # plot 0 line as reverence
    plt.plot(x_test, [0] * len(x_test), '--', color='k', linewidth=1.)
    # standard deviation
    std = np.sqrt(np.diag(K_theta_i))
    plt.fill_between(x_test, -2 * std, 2 * std, color='b', alpha=0.10)
    # corresponding theta values as title
    plt.title(str(thetas_inp[i]))

plt.show()

```

### 2. Predictive distribution (35 points)
So far we have sampled mean functions from the prior.  We can draw actual data $\bt$ two ways.  The first way is generatively, by first sampling $\by | \mathbf{K}$, then sampling $\bt | \by, \beta$ (Eqns 6.60 followed by 6.59).  The second way is to integrate over $\by$ (the mean draw) and directly sample $\bt | \mathbf{K}, \beta$ using Eqn 6.61.    This is the generative process for $\bt$.  Note that we have not specified a distribution over inputs $\bx$;  this is because Gaussian processes are conditional models.  Because of this we are free to generate locations $\bx$ when playing around with the GP; obviously a dataset will give us input-output pairs.

Once we have data, we are interested in the predictive distribution (note: the prior is the predictive distribution when there is no data).  Consider the joint distribution for $N+1$ targets, given by Eqn 6.64.  Its covariance matrix is composed of block components $C_N$, $\mathbf{k}$, and $c$.  The covariance matrix $C_N$ for $\bt_N$ is $C_N = \mathbf{K}_N + \beta^{-1}\mathbf{I}_N$.  We have just made explicit the size $N$ of the matrix; $N$ is the number of training points.  The kernel vector $\mathbf{k}$ is a $N$ by $1$ vector of kernel function evaluations between the training input data and the test input vector.  The scalar $c$ is a kernel evaluation at the test input.

#### 2.1 gp_predictive_distribution(...) (10 points)
Write a function `gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None)` that computes  Eqns 6.66 and 6.67, except allow for an arbitrary number of test points (not just one) and now the kernel matrix is for training data.  By having C as an optional parameter, we can avoid computing it more than once (for this problem it is unimportant, but for real problems this is an issue).  The function should compute $\mathbf{C}$, $\mathbf{k}$, and return the mean, variance and $\mathbf{C}$.  Do not forget: the computeK function computes $\mathbf{K}$, not $\mathbf{C}$! (10 points)

```python
# write the calcuation for C as a extra function,
# to reuse it in gp_log_likelihood

def computeC(x_train, beta, theta):
    return computeK(x_train, x_train, theta) + np.identity(len(x_train)) / beta

```

```python
def gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None):
    # if C is given no value in the input for the function,
    # call computeC, to compute it
    if C is None:
        C = computeC(x_train, beta, theta)

    # Bishop 6.66
    k = computeK(x_train, x_test, theta)
    C_inv = np.linalg.inv(C)
    test_mean = k.T.dot(C_inv).dot(t_train)

    # Bishop 6.67
    c = 1.0 / beta + computeK(x_test, x_test, theta)
    test_var = c - k.T.dot(C_inv).dot(k)

    return test_mean, test_var, C

```

```python
### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2 * train_x
test_N = 3
test_x = np.linspace(-1, 1, test_N)
theta = [1, 2, 3, 4]
beta = 25
test_mean, test_var, C = gp_predictive_distribution(train_x, train_t, test_x,
                                                    theta, beta, C=None)

assert test_mean.shape == (test_N,), "the shape of mean is incorrect"
assert test_var.shape == (test_N, test_N), "the shape of var is incorrect"
assert C.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, _, C_out = gp_predictive_distribution(train_x, train_t, test_x, theta, beta,
                                         C=C_in)

assert np.allclose(C_in, C_out), "C is not reused!"

```

#### 2.2 gp_log_likelihood(...) (10 points)
To learn the hyperparameters, we would need to compute the log-likelihood of the of the training data.  Implicitly, this is conditioned on the value setting for $\mathbf{\theta}$.  Write a function `gp_log_likelihood(x_train, t_train, theta, C=None, invC=None, beta=None)`, where C and invC can be stored and reused. It should return the log-likelihood, `C` and `invC`  (10 points)

```python
def gp_log_likelihood(x_train, t_train, theta, beta, C=None, invC=None):
    # to store and reuse C and invC, we test if they are given so we dont
    # calculate them a second time. otherwise we calculate them and return
    if C is None:
        C = computeC(x_train, beta, theta)
    if invC is None:
        invC = np.linalg.inv(C)
    # Bishop 6.69
    lp = ((- np.log(np.linalg.det(C))
           - t_train.T.dot(invC).dot(t_train)
           - len(t_train) * np.log(2 * np.pi)) / 2)

    return lp, C, invC
```

```python
### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2 * train_x
theta = [1, 2, 3, 4]
beta = 25
lp, C, invC = gp_log_likelihood(train_x, train_t, theta, beta, C=None,
                                invC=None)

assert lp < 0, "the log-likelihood should smaller than 0"
assert C.shape == (N, N), "the shape of var is incorrect"
assert invC.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, C_out, _ = gp_log_likelihood(train_x, train_t, theta, beta, C=C_in,
                                invC=None)

assert np.allclose(C_in, C_out), "C is not reused!"

invC_in = np.array([[1.26260453, 0.15416407], [0.15416407, 1.26260453]])
_, _, invC_out = gp_log_likelihood(train_x, train_t, theta, beta, C=None,
                                   invC=invC_in)

assert np.allclose(invC_in, invC_out), "invC is not reused!"

```

#### 2.3 Plotting (10 points)
Repeat the 6 plots above, but this time conditioned on the training points.  Use the periodic data generator to create 2 training points where x is sampled uniformly between $-1$ and $1$.  For these plots, feel free to use the provided function "gp_plot".  Make sure you put the parameters in the title and this time also the log-likelihood. Try to understand the two types of uncertainty!  If you do not use `gp_plot(...)`, please add a fill between for the model and target noise. (10 points)

```python
def gp_plot(x_test, y_test, mean_test, var_test, x_train, t_train, theta, beta):
    # x_test:
    # y_test:    the true function at x_test
    # mean_test: predictive mean at x_test
    # var_test:  predictive covariance at x_test
    # t_train:   the training values
    # theta:     the kernel parameters
    # beta:      the precision (known)

    # the reason for the manipulation is to allow plots separating model and data stddevs.
    std_total = np.sqrt(
        np.diag(var_test))  # includes all uncertainty, model and target noise
    std_model = np.sqrt(
        std_total ** 2 - 1.0 / beta)  # remove data noise to get model uncertainty in stddev
    std_combo = std_model + np.sqrt(
        1.0 / beta)  # add stddev (note: not the same as full)

    plt.plot(x_test, y_test, 'b', lw=3)
    plt.plot(x_test, mean_test, 'k--', lw=2)
    plt.fill_between(x_test, mean_test + 2 * std_combo,
                     mean_test - 2 * std_combo, color='k', alpha=0.25)
    plt.fill_between(x_test, mean_test + 2 * std_model,
                     mean_test - 2 * std_model, color='r', alpha=0.25)
    plt.plot(x_train, t_train, 'ro', ms=10)

```

```python
def plotting_gp(n):
    beta = 1.0 / pow(sigma, 2)
    # create n datapoints
    x_train = np.random.uniform(-1, 1, n)

    y_train = true_mean_function(x_train)
    t_train = add_noise(y_train, sigma)
    x_test = np.linspace(-1, 1, N_test)
    y_test = true_mean_function(x_test)

    for i in range(thetas_inp.shape[0]):
        plt.subplot(2, 3, i + 1)
        test_mean, test_var, _ = gp_predictive_distribution(x_train, t_train,
                                                            x_test,
                                                            thetas_inp[i], beta)
        gp_plot(x_test, y_test, test_mean, test_var, x_train, t_train,
                thetas_inp[i], beta)
        plt.title(str(thetas_inp[i]))

    plt.show()

```

```python

plotting_gp(2)

```

#### 2.4 More plotting (5 points)
Repeat the 6 plots above, but this time conditioned a new set of 10 training points. (5 points)

```python
plotting_gp(10)

```

## Part 2: Support Vector Machines (45 points)
As seen in Part 1: Gaussian Processes, one of the significant limitations of many such algorithms is that the kernel function $k(\bx_n , \bx_m)$ must be evaluated for all possible pairs $\bx_n$ and $\bx_m$ of training points, which can be computationally infeasible during training and can lead to excessive computation times when making predictions for new data points.
In Part 2: Support Vector Machines, we shall look at kernel-based algorithms that have sparse solutions, so that predictions for new inputs depend only on the kernel function evaluated at a subset of the training data points.

### 2.1 Generating a linearly separable dataset (15 points)
a) (5 points) First of all, we are going to create our own 2D toy dataset $X$. The dataset will consists of two i.i.d. subsets $X_1$ and $X_2$, each of the subsets will be sampled from a multivariate Gaussian distribution,

\begin{align}
X_1 \sim &\mathcal{N}(\mu_1, \Sigma_1)\\
&\text{ and }\\
X_2 \sim &\mathcal{N}(\mu_2, \Sigma_2).
\end{align}

In the following, $X_1$ will have $N_1=20$ samples and a mean $\mu_1=(1,1)$. $X_2$ will have $N_2=30$ samples and a mean $\mu_2=(3,3)$.

Plot the two subsets in one figure, choose two colors to indicate which sample belongs to which subset. In addition you should choose, $\Sigma_1$ and $\Sigma_2$ in a way that the two subsets become linearly separable. (Hint: Which form has the covariance matrix for a i.i.d. dataset?)

```python
mean_1 = (1, 1)
variance_1 = np.matrix([[0.1, 0], [0, 0.1]])
X1 = np.random.multivariate_normal(mean_1, variance_1, (20,))

mean_2 = (3, 3)
variance_2 = np.matrix([[0.1, 0], [0, 0.1]])
X2 = np.random.multivariate_normal(mean_2, variance_2, (30,))

def plot_data(X1, X2):
    plt.plot([point[0] for point in X1], [point[1] for point in X1], 'go')
    plt.plot([point[0] for point in X2], [point[1] for point in X2], 'ro')

plot_data(X1, X2)
plt.show()

```

b) (10 points) In the next step we will combine the two datasets X_1, X_2 and generate a vector `t` containing the labels. Write a function `create_X_and_t(X1, X2)` it should return the combined data set X and the corresponding target vector t.

```python

def create_X_and_t(X1, X2):

    X = np.concatenate((X1,  X2))

    t = np.concatenate((np.repeat(1, len(X1)), np.repeat(-1, len(X2))))

    return X, t
```

```python
### Test your function
dim = 2
N1_test = 2
N2_test = 3
X1_test = np.arange(4).reshape((N1_test, dim))
X2_test = np.arange(6).reshape((N2_test, dim))
X_test, t_test = create_X_and_t(X1_test, X2_test)

assert X_test.shape == (N1_test + N2_test, dim), "the shape of X is incorrect"
assert t_test.shape == (N1_test + N2_test,), "the shape of t is incorrect"

```

### 2.2 Finding the support vectors (15 points)
Finally we going to use a SVM to obtain the decision boundary for which the margin is maximized. We have to solve the optimization problem

\begin{align}
\arg \min_{\bw, b} \frac{1}{2} \lVert \bw \rVert^2,
\end{align}

subject to the constraints

\begin{align}
t_n(\bw^T \phi(\bx_n) + b) \geq 1, n = 1,...,N.
\end{align}

In order to solve this constrained optimization problem, we introduce Lagrange multipliers $a_n \geq 0$. We obtain the dual
representation of the maximum margin problem in which we maximize

\begin{align}
\sum_{n=1}^N a_n - \frac{1}{2}\sum_{n=1}^N\sum_{m=1}^N a_n a_m t_n t_m k(\bx_n, \bx_m),
\end{align}

with respect to a subject to the constraints

\begin{align}
a_n &\geq 0, n=1,...,N,\\
\sum_{n=1}^N a_n t_n &= 0.
\end{align}

This takes the form of a quadratic programming problem in which we optimize a quadratic function of a subject to a set of inequality constraints.

a) (5 points) In this example we will use a linear kernel $k(\bx, \bx') = \bx^T\bx'$. Write a function `computeK(X)` that computes the kernel matrix $K$ for the 2D dataset $X$.

```python
def computeK(X):
    K = X.dot(X.T)

    return K
```

```python
dim = 2
N_test = 3
X_test = np.arange(6).reshape((N_test, dim))
K_test = computeK(X_test)

assert K_test.shape == (N_test, N_test)

```

Next, we will rewrite the dual representation so that we can make use of computationally efficient vector-matrix multiplication. The objective becomes

\begin{align}
\min_{\ba} \frac{1}{2} \ba^T K' \ba - 1^T\ba,
\end{align}

subject to the constraints

\begin{align}
a_n &\geq 0, n=1,...,N,\\
\bt^T\ba &= 0.
\end{align}

Where
\begin{align}
K'_{nm} = t_n t_m k(\bx_n, \bx_m),
\end{align}
and in the special case of a linear kernel function,
\begin{align}
K'_{nm} = t_n t_m k(\bx_n, \bx_m) = k(t_n \bx_n, t_m \bx_m).
\end{align}

To solve the quadratic programming problem we will use a python module called cvxopt. You first have to install the module in your virtual environment (you have to activate it first), using the following command:

`conda install -c conda-forge cvxopt`

The quadratic programming solver can be called as

`cvxopt.solvers.qp(P, q[, G, h[, A, b[, solver[, initvals]]]])`

This solves the following problem,

\begin{align}
\min_{\bx} \frac{1}{2} \bx^T P \bx + q^T\bx,
\end{align}

subject to the constraints,

\begin{align}
G\bx &\leq h,\\
A\bx &= b.
\end{align}

All we need to do is to map our formulation to the cvxopt interface.

b) (10 points) Write a function `compute_multipliers(X, t)` that solves the quadratic programming problem using the cvxopt module and returns the lagrangian multiplier for every sample in the dataset.

```python
import cvxopt

def compute_multipliers(X, t):
    K = np.matrix(np.float64(computeK(t[:, None] * X)))
    t = np.array(np.double(t))

    P = cvxopt.matrix(K)
    q = cvxopt.matrix(-np.ones((X.shape[0], 1)))
    G = cvxopt.matrix(-np.eye(X.shape[0]))
    h = cvxopt.matrix(np.zeros(X.shape[0]))
    A = cvxopt.matrix(t.reshape(1, -1))
    b = cvxopt.matrix(np.zeros(1))

    sol = cvxopt.solvers.qp(P, q, G, h, A, b)
    a = np.array(sol['x'])

    return a

```

```python
### Test your function
dim = 2
N_test = 3
X_test = np.arange(6).reshape((N_test, dim))
t_test = np.array([-1., 1., 1.])
a_test = compute_multipliers(X_test, t_test)

assert a_test.shape == (N_test, 1)

```

         pcost       dcost       gap    pres   dres
     0: -7.2895e-01 -1.3626e+00  6e+00  2e+00  2e+00
     1: -3.0230e-01 -6.8816e-01  8e-01  1e-01  1e-01
     2: -2.3865e-01 -3.3686e-01  1e-01  1e-16  2e-15
     3: -2.4973e-01 -2.5198e-01  2e-03  6e-17  6e-16
     4: -2.5000e-01 -2.5002e-01  2e-05  1e-16  2e-16
     5: -2.5000e-01 -2.5000e-01  2e-07  8e-17  6e-16
    Optimal solution found.

### 2.3 Plot support vectors (5 points)
Now that we have obtained the lagrangian multipliers $\ba$, we use them to find our support vectors. Repeat the plot from 2.1, this time use a third color to indicate which samples are the support vectors.

```python
X_inp, t_inp = create_X_and_t(X1, X2)

a = compute_multipliers(X_inp, t_inp)

```

         pcost       dcost       gap    pres   dres
     0: -1.7253e+00 -3.0629e+00  9e+01  9e+00  1e+00
     1: -2.7393e-01 -1.5276e+00  6e+00  4e-01  7e-02
     2: -2.3082e-01 -9.3229e-01  7e-01  5e-16  2e-15
     3: -4.0788e-01 -5.6504e-01  2e-01  1e-16  1e-15
     4: -5.1366e-01 -5.7161e-01  6e-02  2e-16  1e-15
     5: -5.4981e-01 -5.5063e-01  8e-04  2e-16  2e-15
     6: -5.5040e-01 -5.5041e-01  8e-06  6e-16  1e-15
     7: -5.5041e-01 -5.5041e-01  8e-08  5e-16  2e-15
    Optimal solution found.

```python
# for support vector, we know that a must be >0, but because of the way we
# calculate the a all values are above zero. So we approximate: values > 1.0e-02
# are used as support vectors.
# by looking at the data we see that 'zero a' are around _e-10 - _e-09 and
# 'non zero a' around _e-01

sv_bool = a > 1.0e-02

def plot_sv(X_inp, sv_bool):
    # plot the support vectors
    for i in range(len(sv_bool)):
        if sv_bool[i]:
            print(X_inp[i])
            plt.plot(X_inp[i][0], X_inp[i][1], 'o', color='b', mfc='none',
                     markersize=15)

plot_data(X1, X2)
plot_sv(X_inp, sv_bool)
plt.show()

```

### 2.4 Plot the decision boundary (10 Points)
The decision boundary is fully specified by a (usually very small) subset of training samples, the support vectors. Make use of

\begin{align}
\bw &= \sum_{n=1}^N a_n t_n \mathbf{\phi}(\bx_n)\\
b &= \frac{1}{N_S}\sum_{n \in S} (t_n - \sum_{m \in S} a_m t_m k(\bx_n, \bx_m)),
\end{align}

where $S$ denotes the set of indices of the support vectors, to calculate the slope and intercept of the decision boundary. Generate a last plot that contains the two subsets, support vectors and decision boundary.

```python
def plot_decision_boundary(X_inp):
    # calculate weights
    w = np.sum(a * t_inp[:, None] * X_inp, axis=0)
    b = t_inp[sv_bool.reshape(-1)] - X_inp[sv_bool.reshape(-1)].dot(w)
    # calculate bias
    bias = b[0]
    # normalize
    norm = np.linalg.norm(w)
    w = w / norm
    bias = bias / norm
    # calculate slope and intercept
    slope = -w[0] / w[1]
    intercept = -bias / w[1]
    # create x values for plot between the minimum and maximum of the the
    # generated data in x direction
    x = np.linspace(np.min(X_inp[:, 0]), np.max(X_inp[:, 0]), 100)
    # calculate the y valued for the decision boundary as:
    y = x * slope + intercept
    # reduce values of the decition boudery to values between the
    # min and max of the dataset in the y direction
    x_,y_ = [],[]
    for i in range(100):
        if all([y[i] > np.min(X_inp[:, 1]) ,  y[i] < np.max(X_inp[:, 1])]):
            y_.append(y[i])
            x_.append(x[i])
    plt.plot(x_, y_, 'k-')

plot_decision_boundary(X_inp)
plot_data(X1, X2)
plot_sv(X_inp, sv_bool)
plt.show()

```

```python

```

```python

```


# hybrid-quantum-classical-pt
- https://qiskit.org/textbook/ch-machine-learning/machine-learning-qiskit-pytorch.html

```python
!pip install qiskit
```

    Collecting qiskit
      Downloading qiskit-0.34.2.tar.gz (13 kB)
    Collecting qiskit-terra==0.19.2
      Downloading qiskit_terra-0.19.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)
         |████████████████████████████████| 6.5 MB 9.5 MB/s
    Collecting qiskit-aer==0.10.3
      Downloading qiskit_aer-0.10.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (18.0 MB)
         |████████████████████████████████| 18.0 MB 92 kB/s
    Collecting qiskit-ibmq-provider==0.18.3
      Downloading qiskit_ibmq_provider-0.18.3-py3-none-any.whl (238 kB)
         |████████████████████████████████| 238 kB 52.1 MB/s
    Collecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 49.1 MB/s
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.21.5)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.4.1)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.2)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.23.0)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.24.3)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)
         |████████████████████████████████| 53 kB 2.0 MB/s
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.11.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
         |████████████████████████████████| 1.6 MB 44.1 MB/s
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (0.3.4)
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 41.2 MB/s
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.5 MB/s
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.7.1)
    Collecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 1.4 MB/s
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Collecting symengine>=0.8
      Downloading symengine-0.9.0-cp37-cp37m-manylinux2010_x86_64.whl (37.5 MB)
         |████████████████████████████████| 37.5 MB 1.3 MB/s
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (5.4.8)
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.0 MB/s
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Collecting cryptography>=1.3
      Downloading cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
         |████████████████████████████████| 3.6 MB 38.4 MB/s
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.21)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.1-py2.py3-none-any.whl (113 kB)
         |████████████████████████████████| 113 kB 46.7 MB/s
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (4.11.1)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.7.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.10.0.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.2->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.34.2-py3-none-any.whl size=11805 sha256=369bcb6efd24524f75b426cba030e7b8f129284b0dd49e696c267c96283f31eb
      Stored in directory: /root/.cache/pip/wheels/62/77/65/cda6eedfdd2a525bd3f479a4386930ae3088a1eb01f8c944ed
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=af87d60ee7b294643d5e026756ed54f31a8da6541110341527c97eca26df9d62
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed cryptography-36.0.1 ntlm-auth-1.5.0 pbr-5.8.1 ply-3.11 python-constraint-1.4.0 qiskit-0.34.2 qiskit-aer-0.10.3 qiskit-ibmq-provider-0.18.3 qiskit-ignis-0.7.0 qiskit-terra-0.19.2 requests-ntlm-1.1.0 retworkx-0.11.0 scipy-1.7.3 stevedore-3.5.0 symengine-0.9.0 tweedledum-1.1.1 websocket-client-1.2.3

```python
import numpy as np
import matplotlib.pyplot as plt

import torch
from torch.autograd import Function
from torchvision import datasets, transforms
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F

import qiskit
from qiskit import transpile, assemble
from qiskit.visualization import *
```

```python
class QuantumCircuit:
    """
    This class provides a simple interface for interaction
    with the quantum circuit
    """

    def __init__(self, n_qubits, backend, shots):
        # --- Circuit definition ---
        self._circuit = qiskit.QuantumCircuit(n_qubits)

        all_qubits = [i for i in range(n_qubits)]
        self.theta = qiskit.circuit.Parameter('theta')

        self._circuit.h(all_qubits)
        self._circuit.barrier()
        self._circuit.ry(self.theta, all_qubits)

        self._circuit.measure_all()
        # ---------------------------

        self.backend = backend
        self.shots = shots

    def run(self, thetas):
        t_qc = transpile(self._circuit,
                         self.backend)
        qobj = assemble(t_qc,
                        shots=self.shots,
                        parameter_binds = [{self.theta: theta} for theta in thetas])
        job = self.backend.run(qobj)
        result = job.result().get_counts()

        counts = np.array(list(result.values()))
        states = np.array(list(result.keys())).astype(float)

        # Compute probabilities for each state
        probabilities = counts / self.shots
        # Get state expectation
        expectation = np.sum(states * probabilities)

        return np.array([expectation])
```

```python
simulator = qiskit.Aer.get_backend('aer_simulator')

circuit = QuantumCircuit(1, simulator, 100)
print('Expected value for rotation pi {}'.format(circuit.run([np.pi])[0]))
circuit._circuit.draw()
```

    Expected value for rotation pi 0.54

        ┌───┐ ░ ┌───────────┐ ░ ┌─┐
     q: ┤ H ├─░─┤ Ry(theta) ├─░─┤M├
        └───┘ ░ └───────────┘ ░ └╥┘
meas: 1/═════════════════════════╩═
                                 0

```python
class HybridFunction(Function):
    """ Hybrid quantum - classical function definition """

    @staticmethod
    def forward(ctx, input, quantum_circuit, shift):
        """ Forward pass computation """
        ctx.shift = shift
        ctx.quantum_circuit = quantum_circuit

        expectation_z = ctx.quantum_circuit.run(input[0].tolist())
        result = torch.tensor([expectation_z])
        ctx.save_for_backward(input, result)

        return result

    @staticmethod
    def backward(ctx, grad_output):
        """ Backward pass computation """
        input, expectation_z = ctx.saved_tensors
        input_list = np.array(input.tolist())

        shift_right = input_list + np.ones(input_list.shape) * ctx.shift
        shift_left = input_list - np.ones(input_list.shape) * ctx.shift

        gradients = []
        for i in range(len(input_list)):
            expectation_right = ctx.quantum_circuit.run(shift_right[i])
            expectation_left  = ctx.quantum_circuit.run(shift_left[i])

            gradient = torch.tensor([expectation_right]) - torch.tensor([expectation_left])
            gradients.append(gradient)
        gradients = np.array([gradients]).T
        return torch.tensor([gradients]).float() * grad_output.float(), None, None

class Hybrid(nn.Module):
    """ Hybrid quantum - classical layer definition """

    def __init__(self, backend, shots, shift):
        super(Hybrid, self).__init__()
        self.quantum_circuit = QuantumCircuit(1, backend, shots)
        self.shift = shift

    def forward(self, input):
        return HybridFunction.apply(input, self.quantum_circuit, self.shift)
```

```python
# Concentrating on the first 100 samples
n_samples = 100

X_train = datasets.MNIST(root='./data', train=True, download=True,
                         transform=transforms.Compose([transforms.ToTensor()]))

# Leaving only labels 0 and 1
idx = np.append(np.where(X_train.targets == 0)[0][:n_samples],
                np.where(X_train.targets == 1)[0][:n_samples])

X_train.data = X_train.data[idx]
X_train.targets = X_train.targets[idx]

train_loader = torch.utils.data.DataLoader(X_train, batch_size=1, shuffle=True)
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz

      0%|          | 0/9912422 [00:00<?, ?it/s]

    Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw

    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz

      0%|          | 0/28881 [00:00<?, ?it/s]

    Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw

    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz

      0%|          | 0/1648877 [00:00<?, ?it/s]

    Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw

    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz

      0%|          | 0/4542 [00:00<?, ?it/s]

    Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw

```python
n_samples_show = 6

data_iter = iter(train_loader)
fig, axes = plt.subplots(nrows=1, ncols=n_samples_show, figsize=(10, 3))

while n_samples_show > 0:
    images, targets = data_iter.__next__()

    axes[n_samples_show - 1].imshow(images[0].numpy().squeeze(), cmap='gray')
    axes[n_samples_show - 1].set_xticks([])
    axes[n_samples_show - 1].set_yticks([])
    axes[n_samples_show - 1].set_title("Labeled: {}".format(targets.item()))

    n_samples_show -= 1
```

```python
n_samples = 50

X_test = datasets.MNIST(root='./data', train=False, download=True,
                        transform=transforms.Compose([transforms.ToTensor()]))

idx = np.append(np.where(X_test.targets == 0)[0][:n_samples],
                np.where(X_test.targets == 1)[0][:n_samples])

X_test.data = X_test.data[idx]
X_test.targets = X_test.targets[idx]

test_loader = torch.utils.data.DataLoader(X_test, batch_size=1, shuffle=True)
```

```python
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
        self.dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(256, 64)
        self.fc2 = nn.Linear(64, 1)
        self.hybrid = Hybrid(qiskit.Aer.get_backend('aer_simulator'), 100, np.pi / 2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2)
        x = self.dropout(x)
        x = x.view(1, -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.hybrid(x)
        return torch.cat((x, 1 - x), -1)
```

```python
model = Net()
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_func = nn.NLLLoss()

epochs = 20
loss_list = []

model.train()
for epoch in range(epochs):
    total_loss = []
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        # Forward pass
        output = model(data)
        # Calculating loss
        loss = loss_func(output, target)
        # Backward pass
        loss.backward()
        # Optimize the weights
        optimizer.step()

        total_loss.append(loss.item())
    loss_list.append(sum(total_loss)/len(total_loss))
    print('Training [{:.0f}%]\tLoss: {:.4f}'.format(
        100. * (epoch + 1) / epochs, loss_list[-1]))
```

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:11: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)
      # This is added back by InteractiveShellApp.init_path()
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:32: FutureWarning: The input object of type 'Tensor' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'Tensor', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.

    Training [5%]	Loss: -0.8022
    Training [10%]	Loss: -0.9218
    Training [15%]	Loss: -0.9390
    Training [20%]	Loss: -0.9505
    Training [25%]	Loss: -0.9575
    Training [30%]	Loss: -0.9532
    Training [35%]	Loss: -0.9617
    Training [40%]	Loss: -0.9725
    Training [45%]	Loss: -0.9731
    Training [50%]	Loss: -0.9764
    Training [55%]	Loss: -0.9748
    Training [60%]	Loss: -0.9813
    Training [65%]	Loss: -0.9805
    Training [70%]	Loss: -0.9785
    Training [75%]	Loss: -0.9850
    Training [80%]	Loss: -0.9877
    Training [85%]	Loss: -0.9883
    Training [90%]	Loss: -0.9871
    Training [95%]	Loss: -0.9873
    Training [100%]	Loss: -0.9889

```python
plt.plot(loss_list)
plt.title('Hybrid NN Training Convergence')
plt.xlabel('Training Iterations')
plt.ylabel('Neg Log Likelihood Loss')
```

```python
model.eval()
with torch.no_grad():

    correct = 0
    for batch_idx, (data, target) in enumerate(test_loader):
        output = model(data)

        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()

        loss = loss_func(output, target)
        total_loss.append(loss.item())

    print('Performance on test data:\n\tLoss: {:.4f}\n\tAccuracy: {:.1f}%'.format(
        sum(total_loss) / len(total_loss),
        correct / len(test_loader) * 100)
        )
```

    Performance on test data:
    	Loss: -0.9781
    	Accuracy: 100.0%

```python
n_samples_show = 6
count = 0
fig, axes = plt.subplots(nrows=1, ncols=n_samples_show, figsize=(10, 3))

model.eval()
with torch.no_grad():
    for batch_idx, (data, target) in enumerate(test_loader):
        if count == n_samples_show:
            break
        output = model(data)

        pred = output.argmax(dim=1, keepdim=True)

        axes[count].imshow(data[0].numpy().squeeze(), cmap='gray')

        axes[count].set_xticks([])
        axes[count].set_yticks([])
        axes[count].set_title('Predicted {}'.format(pred.item()))

        count += 1
```

```python
import qiskit.tools.jupyter
%qiskit_version_table
```

Version Information

Qiskit Software	Version

qiskit-terra	0.19.2

qiskit-aer	0.10.3

qiskit-ignis	0.7.0

qiskit-ibmq-provider	0.18.3

System information

Python version	3.7.12

Python compiler	GCC 7.5.0

Python build	default, Jan 15 2022 18:48:18

OS	Linux

CPUs	1

Memory (Gb)	12.686653137207031

Wed Feb 23 22:43:07 2022 UTC


# Using the hybrid database with BACI price info.

This note book is an illustration of how BACI price data can be used in combinatoin with the $C_\mathrm{u}$ matrix to hybridise ecoinvent 3.5.

```python
import pandas as pd
import numpy as np

import sys
# your path to pylacio
sys.path.append('/home/jakobs/Documents/IndEcol/OASES/pylcaio/src/')
import pylcaio

import gzip
import pickle
import pandas as pd

import matplotlib.pyplot as plt
import os

import time
import seaborn as sns
from matplotlib import rc
import matplotlib.cm as colormaps
import feather
import scipy
from pypardiso import spsolve, factorized

import scipy.stats as stats
sys.path.append(os.path.realpath('../'))
from price_variance_MC import generate_price_vector, do_price_MC

```

    /home/jakobs/anaconda3/envs/pylcaio_paradiso/lib/python3.7/site-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```python
from matplotlib import rc
# sns.set()
rc('font',**{'family':'DejaVu Sans', 'size':18})
rc('text', usetex=True)
```

## First load in the data:

- ecoinvent, exiobase and the Cu matrix are stored in one pickle file:
- The price data is stored as a feather file.

```python
# Define double counting correction strategy and year
double_counting = 'STAM'  # 'STAM' or 'binary'
year = 2012
# read in pickle
analysis_object = pylcaio.Analysis('/home/jakobs/Documents/IndEcol/OASES/pylcaio/src/Databases/ecoinvent3.5_exiobase3_{}/hybrid_system_{}.pickle'.format(year, double_counting))
```

### Read in correct characterisation matrix for EXIOBASE 3.6

Because we use a different version of EXIOBASE than was used in pylcaio, we need to read it in seperately and use it here. We will use the GWP100 indicator which is of the RECIPE 2008 method.

```python
C_io = pd.read_excel('/home/jakobs/Documents/IndEcol/Data/EXIOBASE/exiobase3_6/Characterization_EB36.xlsx', index_col=0, header=0)
C_io = C_io.T
Impact_names_io = C_io.index.values
C_io = scipy.sparse.csr_matrix(C_io)
C_io
```

    <36x1707 sparse matrix of type '<class 'numpy.int64'>'
    	with 1806 stored elements in Compressed Sparse Row format>

```python
%%time
# Calculate the leontief inverse for the LCA technology matrix. (Note the use of the IO convention here,
# hence we first substract the A matrix form a identity matrix of appropriate size)
A_lca = scipy.sparse.eye(analysis_object.A_ff.shape[0])-analysis_object.A_ff
Lp = spsolve(A_lca, np.eye(A_lca.shape[0]))
```

    CPU times: user 5min 14s, sys: 3.01 s, total: 5min 17s
    Wall time: 2min 47s

```python
%%time
# Calculate the Leontief inverse for EXIOBASE
Lio = scipy.sparse.csr_matrix(
            scipy.linalg.solve(np.eye(analysis_object.A_io.shape[0])-analysis_object.A_io.todense(),
            np.eye(analysis_object.A_io.shape[0])))
```

    CPU times: user 1min 22s, sys: 3.59 s, total: 1min 26s
    Wall time: 50.3 s

```python
# PRO contains the important meta data of the ecoinvent processes/activities
PRO = pd.DataFrame.from_dict(analysis_object.PRO_f)
print(PRO.columns)
PRO
```

    Index(['activityId', 'productId', 'activityName', 'ISIC', 'price', 'priceUnit',
           'EcoSpoldCategory', 'geography', 'technologyLevel',
           'macroEconomicScenario', 'productionVolume', 'productName', 'unitName',
           'cpc', 'dry mass [kg]', 'wet mass [kg]', 'activityNameId',
           'activityType', 'startDate', 'endDate', 'io_geography',
           'ProductTypeName', 'priceless_scale_vector'],
          dtype='object')

      activityId
      productId
      activityName
      ISIC
      price
      priceUnit
      EcoSpoldCategory
      geography
      technologyLevel
      macroEconomicScenario
      ...
      cpc
      dry mass [kg]
      wet mass [kg]
      activityNameId
      activityType
      startDate
      endDate
      io_geography
      ProductTypeName
      priceless_scale_vector

      6885fd40-ff73-40a4-8f71-225577ec684e_aeaf5266-3f9c-4074-bd34-eba76a61760c
      6885fd40-ff73-40a4-8f71-225577ec684e
      aeaf5266-3f9c-4074-bd34-eba76a61760c
      barley grain, feed production
      0111:Growing of cereals (except rice), legumin...
      0.14384
      EUR2005
      agricultural means of production/feed
      RoW
      Current
      Business-as-Usual
      ...
      23319: Preparations used in animal feeding n.e.c.
      0.86
      1
      2816d5b0-50fd-42fd-9bc9-aed0bd1913e7
      0
      1996-01-01
      2018-12-31
      RoW(0)
      Cereal grains nec
      0.0

      a96cb241-a4a9-4980-a16a-ba4b6a80175e_aeaf5266-3f9c-4074-bd34-eba76a61760c
      a96cb241-a4a9-4980-a16a-ba4b6a80175e
      aeaf5266-3f9c-4074-bd34-eba76a61760c
      barley grain, feed production
      0111:Growing of cereals (except rice), legumin...
      0.14384
      EUR2005
      agricultural means of production/feed
      CA-QC
      Current
      Business-as-Usual
      ...
      23319: Preparations used in animal feeding n.e.c.
      0.86
      1
      2816d5b0-50fd-42fd-9bc9-aed0bd1913e7
      0
      1996-01-01
      2018-12-31
      CA
      Cereal grains nec
      0.0

      d1c3b8ad-58b6-4ea4-8877-a51b356556bc_281fc4f0-c05d-410a-a784-06e3508e78e6
      d1c3b8ad-58b6-4ea4-8877-a51b356556bc
      281fc4f0-c05d-410a-a784-06e3508e78e6
      barley grain, feed production, Swiss integrate...
      0111:Growing of cereals (except rice), legumin...
      0.14384
      EUR2005
      agricultural means of production/feed
      CH
      Current
      Business-as-Usual
      ...
      23319: Preparations used in animal feeding n.e.c.
      0.85
      1
      938a1c4e-41f2-40ea-9bf1-9f09c2437798
      0
      1996-01-01
      2018-12-31
      CH
      Cereal grains nec
      0.0

      92068396-88c7-45ed-9008-622008a299f3_0d860eb4-1a25-41b4-a821-81f5726d86e5
      92068396-88c7-45ed-9008-622008a299f3
      0d860eb4-1a25-41b4-a821-81f5726d86e5
      barley grain, feed production, organic
      0111:Growing of cereals (except rice), legumin...
      0.18444
      EUR2005
      agricultural means of production/feed
      CH
      Current
      Business-as-Usual
      ...
      23319: Preparations used in animal feeding n.e.c.
      0.85
      1
      6a49a0c6-c114-4471-b6c4-8aa4006e4b0e
      0
      1996-01-01
      2018-12-31
      CH
      Cereal grains nec
      0.0

      35ddb020-9812-4808-bdfb-6845a454a73c_0d860eb4-1a25-41b4-a821-81f5726d86e5
      35ddb020-9812-4808-bdfb-6845a454a73c
      0d860eb4-1a25-41b4-a821-81f5726d86e5
      barley grain, feed production, organic
      0111:Growing of cereals (except rice), legumin...
      0.18444
      EUR2005
      agricultural means of production/feed
      RoW
      Current
      Business-as-Usual
      ...
      23319: Preparations used in animal feeding n.e.c.
      0.85
      1
      6a49a0c6-c114-4471-b6c4-8aa4006e4b0e
      0
      1996-01-01
      2018-12-31
      RoW(1)
      Cereal grains nec
      0.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      31e6ab74-141d-49f4-9e75-5974bd62742b_b3184435-d00c-5713-823d-5ff6741bfbbc
      31e6ab74-141d-49f4-9e75-5974bd62742b
      b3184435-d00c-5713-823d-5ff6741bfbbc
      waste polyethylene terephthalate, for recyclin...
      NaN
      0.00000
      EUR2005
      NaN
      GLO
      Current
      Business-as-Usual
      ...
      39270: Waste, parings and scrap of plastics
      NaN
      NaN
      712ffc38-2281-4fd2-a84d-089678e23bd3
      0
      2018-01-01
      2018-12-31
      GLO
      NaN
      0.0

      b50b032c-8bf0-41a4-bb7a-ce5e12cabf01_d13ae29b-7d6a-55d2-8380-86343a80fed8
      b50b032c-8bf0-41a4-bb7a-ce5e12cabf01
      d13ae29b-7d6a-55d2-8380-86343a80fed8
      waste polyethylene terephthalate, for recyclin...
      NaN
      0.00000
      EUR2005
      NaN
      GLO
      Current
      Business-as-Usual
      ...
      39270: Waste, parings and scrap of plastics
      NaN
      NaN
      27dcd766-bc63-4a7c-9fdf-ffc55e0575b2
      0
      2018-01-01
      2018-12-31
      GLO
      NaN
      0.0

      307b681d-c6bc-44cb-ab3e-a9067a4e115a_ed25f589-c6c8-56a7-a518-b01dd3e8a453
      307b681d-c6bc-44cb-ab3e-a9067a4e115a
      ed25f589-c6c8-56a7-a518-b01dd3e8a453
      waste polyethylene, for recycling, sorted, Rec...
      NaN
      0.00000
      EUR2005
      NaN
      GLO
      Current
      Business-as-Usual
      ...
      39270: Waste, parings and scrap of plastics
      NaN
      NaN
      40de0860-eadc-43ae-bdbf-5e507563379a
      0
      2018-01-01
      2018-12-31
      GLO
      NaN
      0.0

      73c9e29c-885e-490e-95f3-087227443758_101e55f2-2da0-528b-96fd-21687e242c1c
      73c9e29c-885e-490e-95f3-087227443758
      101e55f2-2da0-528b-96fd-21687e242c1c
      waste polyethylene, for recycling, unsorted, R...
      NaN
      0.00000
      EUR2005
      NaN
      GLO
      Current
      Business-as-Usual
      ...
      39270: Waste, parings and scrap of plastics
      NaN
      NaN
      06f4a8b9-d8b2-42d1-94e4-9da5e9420754
      0
      2018-01-01
      2018-12-31
      GLO
      NaN
      0.0

      91ed532a-08f2-4fdd-b5e9-28bb2afd1336_451550a8-d9e6-4396-9f9a-97e752111228
      91ed532a-08f2-4fdd-b5e9-28bb2afd1336
      451550a8-d9e6-4396-9f9a-97e752111228
      waste wood, post-consumer, Recycled Content cu...
      NaN
      0.00000
      EUR2005
      NaN
      GLO
      Current
      Business-as-Usual
      ...
      3928: Sawdust and wood waste and scrap
      1
      1.4
      e413d513-3e07-4f5a-ba52-b3bfdf3dc9a5
      0
      2018-01-01
      2018-12-31
      GLO
      NaN
      0.0

16022 rows × 23 columns

pylcaio estimates scale factors for groups of activities to omit the ecoinvent price information or in case there is none. These factors act effectively as a price (also unitwise of course). From the ecoinvent price info and these scale factors we create a new property 'effective price'

```python
# In case the scale factor is available use that instead of the ecoinvent price.
for index in PRO.index.values:
    if PRO.loc[index, 'priceless_scale_vector'] > 0:
        PRO.loc[index,'effective_price'] = PRO.loc[index, 'priceless_scale_vector']
    else:
        PRO.loc[index,'effective_price'] = PRO.loc[index, 'price']
PRO['effective_price']
```

    6885fd40-ff73-40a4-8f71-225577ec684e_aeaf5266-3f9c-4074-bd34-eba76a61760c    0.14384
    a96cb241-a4a9-4980-a16a-ba4b6a80175e_aeaf5266-3f9c-4074-bd34-eba76a61760c    0.14384
    d1c3b8ad-58b6-4ea4-8877-a51b356556bc_281fc4f0-c05d-410a-a784-06e3508e78e6    0.14384
    92068396-88c7-45ed-9008-622008a299f3_0d860eb4-1a25-41b4-a821-81f5726d86e5    0.18444
    35ddb020-9812-4808-bdfb-6845a454a73c_0d860eb4-1a25-41b4-a821-81f5726d86e5    0.18444
                                                                                ...
    31e6ab74-141d-49f4-9e75-5974bd62742b_b3184435-d00c-5713-823d-5ff6741bfbbc    0.00000
    b50b032c-8bf0-41a4-bb7a-ce5e12cabf01_d13ae29b-7d6a-55d2-8380-86343a80fed8    0.00000
    307b681d-c6bc-44cb-ab3e-a9067a4e115a_ed25f589-c6c8-56a7-a518-b01dd3e8a453    0.00000
    73c9e29c-885e-490e-95f3-087227443758_101e55f2-2da0-528b-96fd-21687e242c1c    0.00000
    91ed532a-08f2-4fdd-b5e9-28bb2afd1336_451550a8-d9e6-4396-9f9a-97e752111228    0.00000
    Name: effective_price, Length: 16022, dtype: float64

## Load in BACI price data

The feather file contains both BACI price data for those activities that have a match, otherwise it contains samples from a lognormal distribution with a mean at the ecoinvent price (inflation corrected to 2012) and a CoV of 1.05 (the median CoV of the processes with a BACI price).

```python
# Read in feather file
price_data = pd.read_feather('/home/jakobs/Documents/IndEcol/OASES/pylcaio/src/Databases/ecoinvent3.5_exiobase3_2012/price_dataframe.ftr')
price_data.set_index('index', inplace=True)
price_data
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      2990
      2991
      2992
      2993
      2994
      2995
      2996
      2997
      2998
      2999

      index

      6885fd40-ff73-40a4-8f71-225577ec684e_aeaf5266-3f9c-4074-bd34-eba76a61760c
      1.879268
      0.681711
      0.633574
      0.297718
      0.275381
      0.495583
      0.098505
      0.402184
      0.309860
      0.094935
      ...
      0.274618
      0.367743
      0.318454
      0.834322
      0.339737
      1.175450
      0.275381
      0.339737
      1.044699
      0.416030

      a96cb241-a4a9-4980-a16a-ba4b6a80175e_aeaf5266-3f9c-4074-bd34-eba76a61760c
      0.624097
      2.389663
      0.683656
      0.624097
      0.624097
      0.624097
      1.208254
      0.624097
      0.624097
      0.624097
      ...
      1.338070
      1.675702
      0.624097
      3.959463
      1.338070
      0.624097
      2.268278
      1.338070
      0.624097
      0.948400

      d1c3b8ad-58b6-4ea4-8877-a51b356556bc_281fc4f0-c05d-410a-a784-06e3508e78e6
      3.675250
      3.675250
      3.396044
      12.247802
      11.122441
      3.675250
      11.122441
      3.675250
      3.675250
      2.680182
      ...
      3.396044
      3.903311
      3.903311
      3.396044
      3.396044
      3.396044
      3.675250
      3.675250
      3.396044
      10.102856

      92068396-88c7-45ed-9008-622008a299f3_0d860eb4-1a25-41b4-a821-81f5726d86e5
      3.675250
      3.903311
      3.675250
      3.396044
      2.079684
      3.675250
      5.507553
      3.675250
      3.903311
      3.675250
      ...
      3.675250
      3.675250
      3.903311
      2.137872
      5.507553
      22.219628
      3.675250
      3.675250
      1.201978
      12.247802

      35ddb020-9812-4808-bdfb-6845a454a73c_0d860eb4-1a25-41b4-a821-81f5726d86e5
      0.331865
      0.358833
      0.464787
      3.561344
      0.297718
      0.301147
      0.904203
      0.309860
      1.206996
      1.889762
      ...
      0.309860
      1.298496
      3.860966
      0.339737
      0.301147
      0.474415
      0.339737
      0.367173
      0.525427
      1.750417

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      31e6ab74-141d-49f4-9e75-5974bd62742b_b3184435-d00c-5713-823d-5ff6741bfbbc
      0.629772
      0.553619
      0.465567
      0.505673
      0.505673
      0.711286
      0.505673
      0.508607
      0.538488
      0.332482
      ...
      0.332482
      0.732628
      0.515347
      0.505673
      1.016215
      0.336765
      0.641501
      0.285595
      0.859997
      0.505673

      b50b032c-8bf0-41a4-bb7a-ce5e12cabf01_d13ae29b-7d6a-55d2-8380-86343a80fed8
      0.414735
      0.538488
      2.753003
      0.427902
      0.240832
      0.476991
      0.336765
      0.240832
      0.538488
      0.472248
      ...
      0.452560
      1.600383
      0.520246
      0.336765
      0.505673
      0.256560
      0.492641
      0.094215
      0.303952
      0.611896

      307b681d-c6bc-44cb-ab3e-a9067a4e115a_ed25f589-c6c8-56a7-a518-b01dd3e8a453
      0.720422
      0.343043
      0.282820
      0.099190
      0.010543
      0.240832
      0.320269
      0.505673
      0.272400
      0.476991
      ...
      0.481484
      0.355664
      0.336765
      0.332482
      0.346147
      0.237820
      0.357863
      0.649262
      0.653808
      0.217355

      73c9e29c-885e-490e-95f3-087227443758_101e55f2-2da0-528b-96fd-21687e242c1c
      0.538488
      0.549331
      0.034887
      0.538488
      0.346147
      0.515347
      0.417434
      0.994987
      0.346147
      0.346147
      ...
      1.661337
      0.336765
      0.538488
      0.482413
      0.538488
      0.350894
      0.505673
      0.242492
      0.417434
      0.477658

      91ed532a-08f2-4fdd-b5e9-28bb2afd1336_451550a8-d9e6-4396-9f9a-97e752111228
      0.161948
      0.061602
      0.057841
      0.019049
      0.165291
      0.125409
      0.146491
      0.058233
      0.043893
      0.146491
      ...
      0.146491
      0.107341
      0.102470
      0.153226
      0.129955
      0.158157
      0.098705
      0.017497
      0.033496
      0.051883

16022 rows × 3000 columns

## Precalculate EXIOBASE multipliers:

\begin{align}
M = C_\mathrm{io}S_\mathrm{io}L_\mathrm{io}
\end{align}

```python
%%time
M_io = C_io.dot(analysis_object.F_io).dot(Lio)
# get the unscaled cut-off matrix and the price data just as an array
Cu_unscaled = analysis_object.A_io_f.copy()
price_data_samples = price_data.to_numpy()

```

    CPU times: user 1.78 s, sys: 59.9 ms, total: 1.84 s
    Wall time: 1.82 s

## Run MC

For 1 impact category in this case only (for speed/memory purposes but possible for all) Might want to write results line by line in that case so not to fill up memory.

```python
%%time
# define number of samples, the number of independent samples drawn for each activity is 3000.
nsamples = 3000

# The MC  only calculates the impacts from the hybrid/IO part.
impacts_io_samples = do_price_MC(M_io[4,:], Lp.todense(), Cu_unscaled, price_data_samples, Nruns=nsamples)

```

    Starting run at Sat Jan 30 10:56:38 2021
    Results shape:  (1, 16022, 10000)
    Run 1
    Run 201
    Run 401
    Run 601
    Run 801
    Run 1001
    Run 1201
    Run 1401
    Run 1601
    Run 1801
    Run 2001
    Run 2201
    Run 2401
    Run 2601
    Run 2801
    Run 3001
    Run 3201
    Run 3401
    Run 3601
    Run 3801
    Run 4001
    Run 4201
    Run 4401
    Run 4601
    Run 4801
    Run 5001
    Run 5201
    Run 5401
    Run 5601
    Run 5801
    Run 6001
    Run 6201
    Run 6401
    Run 6601
    Run 6801
    Run 7001
    Run 7201
    Run 7401
    Run 7601
    Run 7801
    Run 8001
    Run 8201
    Run 8401
    Run 8601
    Run 8801
    Run 9001
    Run 9201
    Run 9401
    Run 9601
    Run 9801
    Finished run at Sat Jan 30 12:27:55 2021
    Finished 10000 runs in 91.0 minutes and 17.405287504196167 seconds
    CPU times: user 2h 13min 36s, sys: 37min 5s, total: 2h 50min 41s
    Wall time: 1h 31min 17s

## Save data to file:

```python
filename = '/home/jakobs/Documents/IndEcol/OASES/pylcaio/src/Databases/ecoinvent3.5_exiobase3_{}/MonteCarlo_results/IO_GWP100_impacts_{}_{}_MC_{}_lhs_runs_cpc21_hs12_mapping_Proxy_data_for_unmapped_processes_median_cov.npy'.format(year, double_counting, year, nsamples)
impacts_io_samples.dump(filename)
```

## Load data

```python
nsamples = 10000
filename = '/home/jakobs/Documents/IndEcol/OASES/pylcaio/src/Databases/ecoinvent3.5_exiobase3_{}/MonteCarlo_results/IO_GWP100_impacts_{}_{}_MC_{}_lhs_runs_cpc21_hs12_mapping_Proxy_data_for_unmapped_processes_median_cov.npy'.format(year, double_counting, year, nsamples)
impacts_io_samples = np.load(filename, allow_pickle=True)
```

## Quick analysis

```python
# LCA impacts per process:
lca_impacts = analysis_object.C_f.dot(analysis_object.F_f.dot(Lp))
```

```python

```

```python
# Categorise processes:
with open('/home/jakobs/Documents/IndEcol/OASES/pylcaio/src/Databases/ecoinvent3.5_exiobase3_{}/list_processes_with_baci_price.txt'.format(year), 'r') as fh:
    procs_with_baci_price = [line.rstrip() for line in fh]

BACI_hybrid_list = list(set(procs_with_baci_price).intersection(analysis_object.hybridized_processes))
hybrid_processes_without_baci_price = set(analysis_object.hybridized_processes).difference(procs_with_baci_price)

# Make boolean masks
hybrid_processes_without_baci_price_boolean_mask = PRO.index.isin(hybrid_processes_without_baci_price)
BACI_hybrid_list_booolean_mask = PRO.index.isin(BACI_hybrid_list)
hybrid_process_boolean_mask = PRO.index.isin(analysis_object.hybridized_processes)

```

```python
color = ['c', 'orange', 'r', 'orange', 'c']
linestyle = [':', '--', '-', '--', ':']

#hybrid_baci, hybrid_non_baci, hybrid all
masklist = [hybrid_process_boolean_mask, BACI_hybrid_list_booolean_mask,
            hybrid_processes_without_baci_price_boolean_mask]
mask_name = ['B', 'C' , 'D']
for j,(mask,name) in enumerate(zip(masklist, mask_name)):
    print(name)
    fig = plt.figure(j)
    ax = fig.add_subplot(111)
    lca_impact = lca_impacts[425,mask]
    dist = np.nan_to_num(np.nanmedian(impacts_io_samples[0,mask, :], axis=1)/lca_impact*100)
    # print(type(dist))
    bin_edges = np.linspace(*np.percentile(dist, [0, 98]), 30)#  np.histogram_bin_edges(dist, bins='doane')
    print(bin_edges.min(), bin_edges.max())
    ax.hist(dist, bins=bin_edges, color='k', lw=1.5, histtype='step', density=True)
    #'''
    for i,x in enumerate(np.percentile(dist, [2.5, 16, 50, 84, 97.5])):
        ax.axvline(x=x, c=color[i], ls=linestyle[i], lw=1.3)
    #ax.axvline(x=dist.mean(), color='r', ls='-.', lw=1.3)
    #ax.axvline(x=impacts_io_price_from_pylcaio[4,mask].sum()/lca_impact*100, color='k', ls='--', lw=1.3)

    ax.text(0.83, 0.9, s='Subset: {} '.format(name),
            horizontalalignment='center', verticalalignment='top',
            transform=ax.transAxes)
    ax.text(0.83, 0.8, s='$N_\mathrm{{proc}}$={}'.format(sum(mask)),
            horizontalalignment='center', verticalalignment='top',
            transform=ax.transAxes)

    ax.set_xlabel(r'Relative increase in Carbon Footprint [\%]')
    ax.set_ylabel(r'Probability density')
    #'''
    ax.set_yscale('log')
    plt.show()
```

```python

```

Price_Uncertainty_HLCA/notebooks/Using hybrid database with BACI price data.ipynb