# COMP 136 Day 24 - Variational Methods for GMMs

## Outline

* [Part 1: Variational Bayes optimization for GMMs](#part1)
* [Part 2: Model Selection with variational Bayes](#part2)
* [Part 3: Comparison to classic EM for GMMs](#part3)

## Key takeaways

* The method of "Variational Bayes" estimates a posterior distribution for all parameters (mixture weights $\pi$, locations $\mu_k$, cluster covariances $\Sigma_k$)
* By optimizing an objective that is based on a full marginal likelihood, the ELBO can be used for model selection ... simplest model that explains the data well will be strongly preferred
* In contrast, classic EM for GMMs also optimizes an ELBO, but this version still conditions on point estimates for $\pi, \mu, \Sigma$, and thus we cannot use this objective alone for model selection (see part 3)

# Setup

```python
import numpy as np
import pandas as pd
import scipy.stats

np.set_printoptions(precision=3, suppress=True)
```

```python
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("notebook", font_scale=1.25)
```

```python
def plot_GMM(pi_K, mu_KD, cov_KDD, color_list=['b', 'r', 'm', 'g', 'c', 'y'], ax_handle=None):
    ''' Visualize gaussian mixture model
    '''
    # Create new figure if one not provided
    if ax_handle is None:
        plt.figure()
        ax_handle = plt.gca()

    # Sort components from large to small
    sort_ids = np.argsort(pi_K)[::-1]
    pi_K = pi_K[sort_ids]
    mu_KD = mu_KD[sort_ids]
    cov_KDD = cov_KDD[sort_ids]

    for i, (freq, mean, covar, color) in enumerate(zip(
            pi_K, mu_KD, cov_KDD, color_list)):
        if freq < 0.005:
            continue

        # Plot an ellipse to show the Gaussian component
        v, w = np.linalg.eigh(covar)
        v = 2. * np.sqrt(2.) * np.sqrt(v)
        u = w[0] / np.linalg.norm(w[0])
        angle = np.arctan(u[1] / u[0])
        angle = 180. * angle / np.pi # convert to degrees
        ell = mpl.patches.Ellipse(mean, v[0], v[1], 180. + angle, color=color)
        ell.set_clip_box(ax_handle.bbox)
        ell.set_alpha(0.5)
        ax_handle.add_artist(ell)

        ax_handle.plot(
            mean[0], mean[1], color=color, marker='+', mew=2.0, label='pi[%d] = %.3f' % (i,freq))
```

# Create dataset

```python
## Number of true clusters
K = 3
```

```python
## Number of dimensions
D = 2
```

```python
## Appearance probabilities
pi_K = np.asarray([0.6, 0.3, 0.1])
```

```python
mu_KD = np.zeros((K, D))
mu_KD[0] = [-1, -1.0]
mu_KD[1] = [+1, +1.0]
mu_KD[2] = [0.2, -0.2]
```

```python
cov_KDD = np.vstack([
    0.3 * np.eye(2)[np.newaxis,:,:],
    0.3 * np.eye(2)[np.newaxis,:,:],
    0.5 * np.asarray([[1.0, -0.8], [-0.8, 1.0]])[np.newaxis,:,:]])
```

```python
## Number of observations
N = 1000

prng = np.random.RandomState(0)
```

```python
truez_N = np.zeros(N)
x_ND = np.zeros((N, D))
for n in range(N):
    k = prng.choice(np.arange(K), p=pi_K)
    truez_N[n] = k

    x_ND[n] = prng.multivariate_normal(mu_KD[k], cov_KDD[k])
```

### Visualize dataset : color each point by its assigned cluster id

```python
for (k, color) in [(0, 'b'), (1, 'r'), (2, 'm')]:
    plt.plot(x_ND[truez_N == k,0], x_ND[truez_N == k,1], color=color, marker='.', linestyle='')
plt.gca().set_aspect('equal');
plt.xlabel('x_1');
plt.ylabel('x_2');
```

```python
plot_GMM(pi_K, mu_KD, cov_KDD);
plt.xlim([-3, 3]);
plt.ylim([-3, 3]);
plt.legend(loc='upper left');
```

# Part 1 : Fit a GMM to data using 'Full Variational Bayes' Optimization Methods

Here, we'll use sklearn's built-in "BayesianGaussianMixture" implementation.

This is like the variational coordinate ascent algorithm described in Bishop PRML textbook Sec. 10.2

The model is:

$$
p( x, z | \pi, \mu, \Sigma ) = \prod_{n=1}^N \text{Cat}(z_n | \pi ) \text{MVNorm}(x_n | \mu_{z_n}, \Sigma_{z_n} )
$$

with a fully conjugate prior on parameters something like this (defer to sklearn documentation for correct description):

$$
p( \pi ) = \text{Dir}( \pi | a_0, \ldots a_0 )
\\
p( \mu, \Sigma ) = \prod_{k=1}^K \text{MVNorm}(\mu_k | m_0, \kappa_0 \Sigma_k ) \text{Wish}( \Sigma_k | n_0, S_0 )
$$

The goal of `sklearn.mixture.BayesianGaussianMixture` is to find an approximate posterior over all unknown random variables. In this way this is like the "EE" algorithm we discussed in lecture, rather than the classic "E/M" algorithm you implemented in CP4.

### Approximate posterior

\begin{align}
q( \pi ) &= \text{Dir}( a_1, \ldots a_K )
\\
q( \mu, \Sigma ) &= \prod_{k=1}^K \text{MVNorm}(\mu_k | m_k, \kappa_k \Sigma_k ) \text{Wish}( \Sigma_k | n_k, S_k )
\\
q( z ) &= \prod_{n=1}^N \text{Cat}( z_n | r_{n1}, \ldots r_{nK} )
\end{align}

The goal of the algorithm is to minimize the KL between this $q$ and the true posterior $p( z, \pi, \mu, \Sigma | x)$.

This is equivalent to maximizing the ELBO.

The algorithm implemented does a coordinate ascent approach, as discussed in lecture

```python
import sklearn.mixture
```

```python
gmm = sklearn.mixture.BayesianGaussianMixture(
    n_components=3,

    # Prior on mixture weights pi
    weight_concentration_prior_type='dirichlet_distribution',
    weight_concentration_prior=0.5,

    # Prior on locations mu
    mean_prior=np.zeros(D),
    mean_precision_prior=1.0,

    # Prior on covariances Sigma
    covariance_type='full',
    covariance_prior=np.eye(D),
    degrees_of_freedom_prior=D + 1.0,

    # Initialization settings
    random_state=42,
    init_params='kmeans',
    n_init=1,

    # Optimization algorithm settings
    max_iter=100,
    tol=0.0001,
    verbose=2,
    verbose_interval=1,)
```

```python
gmm.fit(x_ND)
```

    Initialization 0
      Iteration 1	 time lapse 0.00649s	 ll change inf
      Iteration 2	 time lapse 0.00122s	 ll change 10.26674
      Iteration 3	 time lapse 0.00109s	 ll change 3.78400
      Iteration 4	 time lapse 0.00105s	 ll change 2.05885
      Iteration 5	 time lapse 0.00115s	 ll change 1.39327
      Iteration 6	 time lapse 0.00116s	 ll change 1.07539
      Iteration 7	 time lapse 0.00093s	 ll change 0.90047
      Iteration 8	 time lapse 0.00089s	 ll change 0.79437
      Iteration 9	 time lapse 0.00080s	 ll change 0.72557
      Iteration 10	 time lapse 0.00084s	 ll change 0.67888
      Iteration 11	 time lapse 0.00081s	 ll change 0.64614
      Iteration 12	 time lapse 0.00092s	 ll change 0.62248
      Iteration 13	 time lapse 0.00086s	 ll change 0.60461
      Iteration 14	 time lapse 0.00090s	 ll change 0.59007
      Iteration 15	 time lapse 0.00087s	 ll change 0.57668
      Iteration 16	 time lapse 0.00089s	 ll change 0.56231
      Iteration 17	 time lapse 0.00084s	 ll change 0.54463
      Iteration 18	 time lapse 0.00087s	 ll change 0.52116
      Iteration 19	 time lapse 0.00105s	 ll change 0.48972
      Iteration 20	 time lapse 0.00098s	 ll change 0.44914
      Iteration 21	 time lapse 0.00112s	 ll change 0.39994
      Iteration 22	 time lapse 0.00121s	 ll change 0.34454
      Iteration 23	 time lapse 0.00098s	 ll change 0.28671
      Iteration 24	 time lapse 0.00103s	 ll change 0.23058
      Iteration 25	 time lapse 0.00100s	 ll change 0.17959
      Iteration 26	 time lapse 0.00113s	 ll change 0.13591
      Iteration 27	 time lapse 0.00107s	 ll change 0.10035
      Iteration 28	 time lapse 0.00096s	 ll change 0.07256
      Iteration 29	 time lapse 0.00086s	 ll change 0.05159
      Iteration 30	 time lapse 0.00083s	 ll change 0.03618
      Iteration 31	 time lapse 0.00084s	 ll change 0.02510
      Iteration 32	 time lapse 0.00085s	 ll change 0.01726
      Iteration 33	 time lapse 0.00086s	 ll change 0.01179
      Iteration 34	 time lapse 0.00087s	 ll change 0.00801
      Iteration 35	 time lapse 0.00093s	 ll change 0.00542
      Iteration 36	 time lapse 0.00083s	 ll change 0.00365
      Iteration 37	 time lapse 0.00102s	 ll change 0.00245
      Iteration 38	 time lapse 0.00088s	 ll change 0.00165
      Iteration 39	 time lapse 0.00083s	 ll change 0.00110
      Iteration 40	 time lapse 0.00084s	 ll change 0.00074
      Iteration 41	 time lapse 0.00083s	 ll change 0.00049
      Iteration 42	 time lapse 0.00084s	 ll change 0.00033
      Iteration 43	 time lapse 0.00086s	 ll change 0.00022
      Iteration 44	 time lapse 0.00083s	 ll change 0.00015
      Iteration 45	 time lapse 0.00082s	 ll change 0.00010
    Initialization converged: True	 time lapse 0.04775s	 ll -538.34461

    BayesianGaussianMixture(covariance_prior=array([[1., 0.],
           [0., 1.]]),
                            degrees_of_freedom_prior=3.0, mean_precision_prior=1.0,
                            mean_prior=array([0., 0.]), n_components=3,
                            random_state=42, tol=0.0001, verbose=2,
                            verbose_interval=1, weight_concentration_prior=0.5,
                            weight_concentration_prior_type='dirichlet_distribution')

```python
gmm.lower_bound_
```

    -538.3446085001409

### Inspect learned approximate posterior

$$
q( \pi ) = \text{Dir}( a_1, a_2, \ldots a_K )
$$

```python
a_K = gmm.weight_concentration_
print(a_K)
```

    [259.    634.045 108.455]

Mean of the pi vector

```python
E_pi_K = a_K / np.sum(a_K)
print(E_pi_K)
```

    [0.259 0.633 0.108]

Print 10 samples of pi from q

```python
prng.dirichlet(a_K, size=5)
```

    array([[0.266, 0.635, 0.099],
           [0.259, 0.623, 0.118],
           [0.273, 0.618, 0.109],
           [0.262, 0.62 , 0.118],
           [0.254, 0.634, 0.112]])

## Visualize learned GMM (use expected value of each parameter)

```python
plot_GMM(E_pi_K, gmm.means_, gmm.covariances_)
plt.xlim([-3, 3]);
plt.ylim([-3, 3]);
plt.legend(loc='upper left');
```

# Part 2: Look at multiple runs of 'Bayesian GMM' across several K values

```python
results_list = list()

best_gmm_per_K = dict()

for Kinit in [1, 2, 3, 4, 5, 6, 8, 10]:
    for trial in [101, 102, 103]:
        gmm = sklearn.mixture.BayesianGaussianMixture(
            n_components=Kinit,

            # Prior on mixture weights pi
            weight_concentration_prior_type='dirichlet_distribution',
            weight_concentration_prior=0.5,

            # Prior on locations mu
            mean_prior=np.zeros(D),
            mean_precision_prior=1.0,

            # Prior on covariances Sigma
            covariance_type='full',
            covariance_prior=np.eye(D),
            degrees_of_freedom_prior=D + 1.0,

            # Initialization settings
            random_state=trial,
            init_params='kmeans',
            n_init=1,

            # Optimization algorithm settings
            max_iter=100,
            tol=0.0001,
            verbose=0)
        gmm.fit(x_ND)
        print("completed trial %4d with K %4d : ELBO value % 9.5f" % (trial, Kinit, gmm.lower_bound_))
        results_list.append(dict(K=Kinit, elbo=gmm.lower_bound_))

        try:
            prev_best_elbo = best_gmm_per_K[Kinit].lower_bound_
        except KeyError:
            prev_best_elbo = -np.inf
        if gmm.lower_bound_ > prev_best_elbo:
            best_gmm_per_K[Kinit] = gmm
```

    completed trial  101 with K    1 : ELBO value -744.27089
    completed trial  102 with K    1 : ELBO value -744.27089
    completed trial  103 with K    1 : ELBO value -744.27089
    completed trial  101 with K    2 : ELBO value -570.80791
    completed trial  102 with K    2 : ELBO value -570.80789
    completed trial  103 with K    2 : ELBO value -570.80791
    completed trial  101 with K    3 : ELBO value -538.34455
    completed trial  102 with K    3 : ELBO value -538.34455
    completed trial  103 with K    3 : ELBO value -538.34456
    completed trial  101 with K    4 : ELBO value -549.64003
    completed trial  102 with K    4 : ELBO value -543.48717

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

    completed trial  103 with K    4 : ELBO value -550.57837
    completed trial  101 with K    5 : ELBO value -559.32673

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

    completed trial  102 with K    5 : ELBO value -554.45086
    completed trial  103 with K    5 : ELBO value -555.30253

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

    completed trial  101 with K    6 : ELBO value -563.86519
    completed trial  102 with K    6 : ELBO value -565.66680

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

    completed trial  103 with K    6 : ELBO value -563.81292
    completed trial  101 with K    8 : ELBO value -582.62986

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

    completed trial  102 with K    8 : ELBO value -563.27466
    completed trial  103 with K    8 : ELBO value -567.04445

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

    completed trial  101 with K   10 : ELBO value -571.66865
    completed trial  102 with K   10 : ELBO value -583.95799
    completed trial  103 with K   10 : ELBO value -563.42928

    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '
    /Users/mhughes/miniconda3/envs/spr_2021s_env/lib/python3.9/site-packages/sklearn/mixture/_base.py:265: ConvergenceWarning: Initialization 1 did not converge. Try different init parameters, or increase max_iter, tol or check for degenerate data.
      warnings.warn('Initialization %d did not converge. '

## Demo 2a: Recreate Bishop's figure 10.7 : Plot of ELBO value vs model size (K)

```python
result_df = pd.DataFrame(results_list)
```

```python
plt.plot(result_df['K'], result_df['elbo'], 'r+')
plt.xlabel('number components');
plt.ylabel('ELBO from EE');
plt.title('Training ELBO vs Number of Clusters');
```

## Discussion 2a : Does this figure show what you'd expect?

```python
# TODO discuss
```

## Exercise 2b : Visualize the best GMM with K=2

```python
gmmK2 = best_gmm_per_K[2]

# TODO plot and interpret
plt.xlim([-3, 3]);
plt.ylim([-3, 3]);
plt.legend(loc='upper left');
```

## Exercise 2c: Visualize the best GMM with K=4

```python
# TODO plot and interpret
```

# Part 3: Compare to classic GMM

```python
results_EM_list = list()

for Kinit in [1, 2, 3, 4, 5, 6, 8, 10]:
    for trial in [101, 102, 103]:
        gmm = sklearn.mixture.GaussianMixture(
            n_components=Kinit,

            # Penalty / regularization on covariances Sigma
            covariance_type='full',
            reg_covar=0.000001,

            # Initialization settings
            random_state=trial,
            init_params='kmeans',
            n_init=1,

            # Optimization algorithm settings
            max_iter=10000,
            tol=0.0001,
            verbose=0)
        gmm.fit(x_ND)
        print("completed trial %4d with K %4d : ELBO value % 9.5f" % (trial, Kinit, gmm.lower_bound_))
        results_EM_list.append(dict(K=Kinit, elbo=gmm.lower_bound_))
```

    completed trial  101 with K    1 : ELBO value  -2.56670
    completed trial  102 with K    1 : ELBO value  -2.56670
    completed trial  103 with K    1 : ELBO value  -2.56670
    completed trial  101 with K    2 : ELBO value  -2.37759
    completed trial  102 with K    2 : ELBO value  -2.37756
    completed trial  103 with K    2 : ELBO value  -2.37759
    completed trial  101 with K    3 : ELBO value  -2.32979
    completed trial  102 with K    3 : ELBO value  -2.32979
    completed trial  103 with K    3 : ELBO value  -2.32974
    completed trial  101 with K    4 : ELBO value  -2.32750
    completed trial  102 with K    4 : ELBO value  -2.32342
    completed trial  103 with K    4 : ELBO value  -2.32732
    completed trial  101 with K    5 : ELBO value  -2.32629
    completed trial  102 with K    5 : ELBO value  -2.32051
    completed trial  103 with K    5 : ELBO value  -2.32522
    completed trial  101 with K    6 : ELBO value  -2.32367
    completed trial  102 with K    6 : ELBO value  -2.31492
    completed trial  103 with K    6 : ELBO value  -2.32369
    completed trial  101 with K    8 : ELBO value  -2.31723
    completed trial  102 with K    8 : ELBO value  -2.31294
    completed trial  103 with K    8 : ELBO value  -2.31236
    completed trial  101 with K   10 : ELBO value  -2.30657
    completed trial  102 with K   10 : ELBO value  -2.31183
    completed trial  103 with K   10 : ELBO value  -2.30561

```python
result_EM_df = pd.DataFrame(results_EM_list)
```

```python
plt.plot(result_EM_df['K'], result_EM_df['elbo'], 'r+')
plt.xlabel('number components');
plt.ylabel('ELBO for EM');
plt.title('Training Set ELBO vs Number of Clusters');
```

## Discussion 3a : Does this figure show what you'd expect?

```python
# TODO Discuss
```

```python

```


# Operações Matemáticas - Arrays e Matrizes

## NumPy

### Arrays NumPy - objetos multidimensionais com N dimensões.
### Matrizes NumPy - objetos com 2 dimensões.

A principal vantagem de utilizar matrizes com NumPy, é que este tipo de objeto possui convenientes notações para multiplicação de matrizes. Arrays e Matrizes NumPy possuem o atributo .T para retornar a transposta da matriz, enquanto objetos do tipo matriz possuem adicionalmente os atributos .I (Inversa) e .H (Transposta Conjugada).

Existem diferenças em operações de álgebra linear entre arrays e matrizes.

Muitas funções NumPy retornam arrays e não matrizes como objeto resultante.

```python
import sys
import numpy as np
print(sys.version)
np.__version__
```

    3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]

    '1.20.1'

## Criando Matrizes

```python
mat1 = np.matrix("1,2,3; 4,5,6")
print(mat1)
```

    [[1 2 3]
     [4 5 6]]

```python
type(mat1)
```

    numpy.matrix

```python
mat2 = np.matrix( [ [1,2,3], [4,5,6] ] )
print(mat2)
```

    [[1 2 3]
     [4 5 6]]

```python
mat3 = np.matrix([ [0,10,0,0,0], [0,0,20,0,0], [0,0,0,30,0], [0,0,0,0,40], [0,0,0,0,0] ])
print(mat3)
```

    [[ 0 10  0  0  0]
     [ 0  0 20  0  0]
     [ 0  0  0 30  0]
     [ 0  0  0  0 40]
     [ 0  0  0  0  0]]

```python
type(mat3)
```

    numpy.matrix

```python
mat3[2, 3]
```

    30

## Operações Matemáticas com Arrays e Matrizes

```python
a = np.array([[1,2],[3,4]])
a
```

    array([[1, 2],
           [3, 4]])

```python
type(a)
```

    numpy.ndarray

```python
a * a
```

    array([[ 1,  4],
           [ 9, 16]])

```python
A = np.mat(a)
A
```

    matrix([[1, 2],
            [3, 4]])

```python
type(A)
```

    numpy.matrix

```python
A * A
```

    matrix([[ 7, 10],
            [15, 22]])

## $$  \boxed{ \begin{align} \begin{pmatrix} 1 & 2    \\ 3 &  4   \end{pmatrix} & \begin{pmatrix} 1 & 2    \\ 3 &  4   \end{pmatrix}  = \begin{pmatrix} 7 & 10    \\ 15 &  22   \end{pmatrix} \end{align} }$$

```python
from IPython.display import Image
Image('imagens/Matriz.png')
```

```python
np.dot(a, a)
```

```python
# Convertendo um Array para Matriz
mat5 = np.asmatrix(a)
```

```python
mat5
```

```python
mat5 * mat5
```

```python
# Convertendo uma Matriz para Array
array2 = np.asarray(mat5)
```

```python
array2
```

```python
array2 * array2
```

# Fim

### Obrigado - Data Science Academy - facebook.com/dsacademybr


```python
from pycalphad import Database, Model
from pycalphad.core.utils import make_callable, point_sample
import pycalphad.variables as v
import numpy as np
from sympy import Matrix
```

```python
db_alpt = Database('alpt_2015.tdb')
my_phases_alpt = ['PT2AL', 'PT5AL21', 'PT5AL3', 'LIQUID', 'AL3NI2', 'PT8AL21', 'PT2AL3', 'FCC_L12', 'BCC_B2']
mod = Model(db_alpt, ['AL', 'PT', 'VA'], 'FCC_L12', parameters={'VX45': 0., 'VX46': 0., 'VX47': 0.})
```

```python
func = make_callable(Matrix([mod.ast]).jacobian(mod.variables), mod.variables, mode='numpy')
```

```python
numexpr_func = func = make_callable(Matrix([mod.ast]).jacobian(mod.variables), mod.variables, mode='numexpr')
```

```python
from itertools import chain
points = point_sample([2, 2, 2, 2, 1], pdof=1000)
print(points.shape)
print(mod.variables)
%timeit numexpr_func(*chain([700.], points.T))
%timeit func(*chain([700.], points.T))
```

    (4000, 9)
    [T, FCC_L120AL, FCC_L120PT, FCC_L121AL, FCC_L121PT, FCC_L122AL, FCC_L122PT, FCC_L123AL, FCC_L123PT, FCC_L124VA]
    The slowest run took 91.15 times longer than the fastest. This could mean that an intermediate result is being cached
    1 loops, best of 3: 3.42 ms per loop
    100 loops, best of 3: 4.3 ms per loop

```python

```


# Extremal linkage networks

This notebook contains code accompanying the paper [extremal linkage networks](https://arxiv.org/abs/1904.01817).

We first implement the network dynamics and then rely on [TikZ](https://github.com/pgf-tikz/pgf) for visualization.

## The Model

We define a random network on an infinite set of layers, each consisting of $N \ge 1$ nodes. The node $i \in  \{0, \dots, N - 1\}$ in layer $h \in \mathbb Z$ has a fitness $F_{i, h}$, where we assume the family $\{F_{i, h}\}_{i \in \{0, \dots, N - 1\}, h \in \mathbb Z}$ to be independent and identically distributed (i.i.d.).

Then, the number of nodes on layer $h+1$ that are visible for the $i$th node in layer $h$, which we call the *scope* of $(i,h)$, is given by $\varphi(F_{i, h}) \wedge N$, where
\begin{equation}\label{eqPhiDef}
        \varphi(f) = 1 + 2 \lceil f \rceil.
\end{equation}

Now, $(i, h)$ connects to precisely one visible node $(j, h+1)$ in layer $h+1$, namely the one of maximum fitness.  In other words,
$$F_{j, h+1} = \max_{j':\, d_N(i, j') \le \lceil F_{i, h}\rceil}F_{j', h+1}.$$

Henceforth, we assume the fitnesses to follow a Fréchet distribution with tail index 1. That is,
$$\mathbb P(F \le s) = \exp(-s^{-1}).$$

## Simulation of Network Dynamics

```python
def simulate_network(hrange = 250,
                     layers = 6):
    """Simulation of the network model
    # Arguments
        hrange: horizontal range of the network
        layers: number of layers

    # Result
        fitnesses and selected edge
    """
    #generate fréchet distribution
    fits =  np.array([1/np.log(1/np.random.rand(hrange)) for _ in range(layers)])
    fits_int = 1+ np.array(fits, dtype = np.int)

    #determine possible neighbors
    neighbs = [[(idx + np.arange(-fit, fit + 1)) % hrange
      for  idx, fit in enumerate(layer) ]
     for layer in fits_int]

    #determine selected neighbor
    sel_edge = [[neighb[np.argmax(fit[neighb])]
      for neighb in nb]
     for (fit, nb) in zip(np.roll(fits, -1, 0), neighbs)]

    return fits, sel_edge
```

Now, we simulate the random network model as described above.

```python
import numpy as np

#seed
seed = 56
np.random.seed(seed)

fits, edges = simulate_network()
```

## Visualization

Now, plot the network in tikz.

```python
def plot_synapses(edges,
                  idxs = np.arange(102, 131),
                  layers = 4,
                  x_scale = .15,
                  node_scale = .5):
    """Plot relevant synapses
    # Arguments
        idxs: indexes of layer-0 node
        edges: edges in the linkage graph
        layers: number of layers to be plotted
        x_scale: scaling in x-direction
        node_scale: scaling of nodes

    # Result
        tikz representation of graph
    """
    result = []

    #horizontal range
    hrange = len(edges[0])

    #plot layer by layer
    for layer in range(layers):
        #plot points
        result +=["\\fill ({0:1.2f}, {1:1.2f}) circle ({2:1.1f}pt);\n".format((idx % hrange) * x_scale,
                                                                              layer,
                                                                              node_scale * np.log(fits)[layer, idx])
              for idx in idxs]

        #plot edges
        string = "\\draw[line width = .5pt]"
        string += " ({0:1.2f}, {1:1.2f})--({2:1.2f}, {3:1.2f});\n"
        path_unordered = [string.format(idx  * x_scale,
                                        layer,
                                        edges[layer][idx] * x_scale,
                                        layer + 1) for idx in idxs]
        result += path_unordered

        #update indexes
        idxs = np.unique([edges[layer][idx] for idx in idxs])

    #plot points
    result +=["\\fill ({0:1.2f}, {1:1.2f}) circle ({2:1.1f}pt);\n".format((idx % hrange) * x_scale,
                                                                          layers,
                                                                          node_scale * np.log(fits)[layer + 1, idx])
              for idx in idxs]

    tikz = ''.join(result)

    return '\\begin{tikzpicture}\n' + tikz + '\\end{tikzpicture}\n'
```

Finally, we write to a file.

```python
fname = 'coalesc.tex'

f = open(fname, "w")
f.write(plot_synapses(idxs,
                      edges))
f.close()
```

```python
!pdflatex evolFig.tex

```


# Acoplamento eletroacústico (microfone capacitivo de medição)

O desenho esquemático do microfone simplificado é:

O circuito eletromecânico do sistema é dado na figura:

Vamos considerar uma pressão sonora de entrada e medir a tensão de saída do sensor.

```python
import numpy as np
from scipy import special
from matplotlib import pyplot as plt
plt.rcParams.update({'font.size': 14})

def ac_compliance(vol, rho0 = 1.21, c0 = 343):
    Ca = vol/(rho0*c0**2)
    return Ca

def ac_oo_smalltube(a, l, freq, flanged = False, rho0 = 1.21, c0 = 343, eta = 1.84e-5):
    if flanged:
        ll = l + 0.85*a
    else:
        ll = l + 0.61*a
    # compliance and mass
    Ra = 8*eta*ll/(np.pi*a**4)
    Ma = 4*rho0*ll/(3*np.pi*a**2)
    # impedance
    omega = 2 *np.pi*freq
    Za = Ra + 1j*omega*Ma
    return Ra, Ma, Za

def ac_slit(h, b, l, freq, rho0 = 1.21, c0 = 343, eta = 1.84e-5):
    # compliance and mass
    Ra = 32*eta*l/((h**3)*b)
    Ma = 6*rho0*l/(5*h*b)
    # impedance
    omega = 2 *np.pi*freq
    Za = Ra + 1j*omega*Ma
    return Ra, Ma, Za

def zrad_cylinder(a, freq, rho0 = 1.21, c0 = 343):
    # Equivalent circuit
    Ra1 = (0.504 * rho0 * c0) / (np.pi * a**2)
    Ra2 = (rho0 * c0) / (np.pi * a**2)
    Ca1 = (5.44 * a**3) / (rho0 * c0**2)
    Ma1 = (0.1952 * rho0) / a
    # Equivalent impedance
    omega = 2*np.pi*freq
    Zrad_eq=(-(omega**2)*Ma1*Ra1*Ra2*Ca1+1j*omega*Ma1*(Ra1+Ra2))/\
        (Ra1+Ra2-(omega**2)*Ma1*Ra1*Ca1+1j*omega*(Ra1*Ra2*Ca1+Ma1))

    return Ra1, Ra2, Ca1, Ma1, Zrad_eq
```

# Parâmetros do microfone

```python
# Parametros do diafragma
a = 4.45e-3; # meia polegada
S = np.pi*a**2 # area
E0 = 200 # Polarization voltage
h0 = 20.77e-6 # Backplate-diaphragm distance 19e-6
Tm0 = 3160 # Tensão Mec
rhom = 8900 # Density of diaphragm
tm = 5e-6 # thickness of diaphragm

Mm = 4*rhom*tm*S/3 # Massa do diafragma
Mad = Mm/S**2
Cm = (1/(8*np.pi*Tm0))
Cad = Cm*S**2
Rm = 0.4

Ra1, Ra2, Ca1, Ma1, Zrad_eq = zrad_cylinder(a, 1, rho0 = 1.21, c0 = 343)

print(r'Mm é {} [kg]'.format(Mm))
print(r'Cm é {} [m/N]'.format(Cm))
print(r'Rm é {} [Ns/m]'.format(Rm))

#############################################
print(r'Ra1 é {} [Pa s/m^3]'.format(Ra1))
print(r'Ra2 é {} [Pa s/m^3]'.format(Ra2))
print(r'Ca1 é {} [m^3/Pa]'.format(Ca1))
print(r'Ma1 é {} [kg/m^4]'.format(Ma1))
```

    Mm é 3.691209052347572e-06 [kg]
    Cm é 1.2591372080055011e-05 [m/N]
    Rm é 0.4 [Ns/m]
    Ra1 é 3362328.425182717 [Pa s/m^3]
    Ra2 é 6671286.557902216 [Pa s/m^3]
    Ca1 é 3.3674823043105737e-12 [m^3/Pa]
    Ma1 é 53.07685393258427 [kg/m^4]

# Slit

Até aqui não haviamos falado sobre o slit, dado no esquema

Cuja impedância é dada por uma Massa acústica em série com uma Resistência acústica

\begin{equation}
Z_{A} = \frac{32 \eta l}{h^3 b} + \mathrm{j} \omega \frac{6 \rho_0 l}{5 h b}
\end{equation}

Podemos considerar
- $b$ circunferência do sensor;
- $h$ distância entre carcaça e início do backplate;
- $l$ comprimento do backplate

```python
# Parametros do circuito acústico
# Backplate
ab = 3.61e-3 # raio do backplate
afb = 0.51e-3
Sfb = np.pi*afb**2
lfb = 0.84e-3
Rab_1f, Mab_1f, Zab_1f = ac_oo_smalltube(afb, lfb, 1, flanged = False)
#Mab = (4/3) * 1.21 * lfb/(np.pi*afb**2)
#Rab = 8*1.84e-5*lfb/(np.pi*afb**4);
Rab = Rab_1f/6
Mab = Mab_1f/6

# Slit
hs = a-ab # distancia entre carcaça e backplate
ls = 0.3e-3;
bs = 2*np.pi*a
Ras, Mas, Zas = ac_slit(hs, bs, ls, 1)

# Tubo de eq
aT = 0.04e-3
ST = np.pi*aT**2
lT = 2e-3
Rat, Mat, Zat = ac_oo_smalltube(aT, lT, 1, flanged = True)

# Cavidades
V_A1 = S * h0
Ca1 = ac_compliance(V_A1)

Hmic = 3.35e-2
V_A2 = 1.264e-7;
Ca2 = ac_compliance(V_A2)

print(r'Rab é {} [Pa s/m^3]'.format(Rab))
print(r'Mab é {} [kg/m^4]'.format(Mab))
print('%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%')
print(r'Ras é {} [Pa s/m^3]'.format(Ras))
print(r'Mas é {} [kg/m^4]'.format(Mas))
print('%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%')
print(r'Rat é {} [Pa s/m^3]'.format(Rat))
print(r'Mat é {} [kg/m^4]'.format(Mat))
print('%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%')
print(r'Ca1 é {} [m^3/Pa]'.format(Ca1))
print(r'Ca2 é {} [m^3/Pa]'.format(Ca2))
```

    Rab é 132873.70242205405 [Pa s/m^3]
    Mab é 378.7875407606094 [kg/m^4]
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    Ras é 10658.873757987914 [Pa s/m^3]
    Mas é 18.54678791086934 [kg/m^4]
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    Rat é 37227932738.62523 [Pa s/m^3]
    Mat é 652837.6610686454 [kg/m^4]
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    Ca1 é 9.076800304482719e-15 [m^3/Pa]
    Ca2 é 8.879192336301656e-13 [m^3/Pa]

# Referenciando ao lado acústico

```python
print(r'Mma é {} [kg/m^4] '.format(Mm/S**2))
print(r'Cma é {} [m^3/Pa] '.format(Cm*S**2))
print(r'Rma é {} [Pas/m^3]'.format(Rm/S**2))
Ce0 = 17.9e-12
print(r'Ce0 é {} [m^3/Pa]'.format(Ce0*(((S*E0*Cm)/h0)**2)))
Rout = 10e6
print(r'Rma é {} [Pas/m^3]'.format(Rout*(h0/(S*E0*Cm))**2))
print('Fator de conversão {}'.format((h0/(S*E0*Cm))))
```

    Mma é 953.7374866930061 [kg/m^4]
    Cma é 4.8731844193868746e-14 [m^3/Pa]
    Rma é 103352313.36588088 [Pas/m^3]
    Ce0 é 1.0184176888234955e-15 [m^3/Pa]
    Rma é 175762854440.19122 [Pas/m^3]
    Fator de conversão 132.57558389092284

# Sensibilidade

```python
sim_1_file = 'vout_bfore_corr.txt'
data = np.loadtxt(sim_1_file, skiprows=1)
freq_s = data[:,0]
V_s1 = (data[:,1] + 1j*data[:,2])/(h0/(S*E0*Cm))

###############################################################################
fig = plt.figure(figsize = (6,4.5))
ax = fig.add_axes([0, 0, 1, 1])
ax.semilogx(freq_s, 20*np.log10(np.abs(V_s1)), '-r', linewidth = 2,
            label = "Circuit")
ax.set_xlabel('Frequency [Hz]', labelpad=10)
ax.set_ylabel(r'$20 \mathrm{log}(|\tilde{V}/\tilde{P}|)$ [dB]', labelpad=10)
ax.legend(frameon=True, fontsize=16)
ax.grid(b=None, which='both', axis='both',linestyle = '--')
ax.set_xlim((1,50000))
ax.set_ylim((-59,-38))
ax.set_xticks([1, 10, 100, 1000, 10000, 50000])
ax.set_xticklabels(('1', '10', '100', '1000', '10000', '50000'));
```


# Exercise 4
* Due: 3.2. at 14:00
* Max points: 8

## General rules
* **Submit your answers by using Moodle.** The link is https://moodle.jyu.fi/mod/assign/view.php?id=685529&forceview=1.
* Name your file as *lastname_ex4.xxx*. If you have multiple files, use *lastname_ex4_1.xxx, lastname_ex4_2.xxx* etc.
* **Remember to add comments to your answers and include also some testing part!**
* You will get feedback about your answers about 1-2 week from the due date

## Exercises

For tasks 1-3, we study optimization problem
$$
\begin{align}
\min \qquad & x_1^2+x_2^2 + x_1+2x_2\\
\text{s.t.}\qquad &x_1+x_2=1\\
    &x_1,x_2\in\mathbb R
\end{align}
$$

1. **(2 points)** Solve the problem using the penalty function method. Note that it is not sufficient to use a fixed value for $r$.
2. **(2 points)** Solve the problem (i.e., approximate the optimal solution) using the barrier function method. Note that you need to do something a bit clever.
3. **(2 points)** Solve the problem using projected gradient method. Compare the performance to the penalty function and barrier function methods. (**Lecture7**)
4. **(2 points)** Check the *necessary first order KKT conditions* for the solution that you found.(**Lecture8**)


# Computational Astrophysics
## ODE 02. Boundary Conditions.

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the techniques to solve a system of Ordinary Differential Equations with boundary conditions.

`A. Garcia. Numerical Methods for Physics. (1999). Chapter 3 `

---

## Ordinary Differential Equations with Boundary Conditions

A boundary value problem (BVP) consists of finding a solution of an ODE
in an interval $[a,b]$ that satisfies constraints at both
ends (boundary conditions).

For example, we can have the system

\begin{equation}
y'' = f(x,y,y')\,\,,
\hspace{0.5em} y(a) = A\,\,,
\hspace{0.5em} y(b) = B\,\,,
\hspace{0.5em} \text{and } x \in [a,b]\,\,.
\end{equation}

---
## The Shooting Method

The shooting method solves a BVP  by transforming it into an initial value problem by
making an educated guess on unknown inner boundary conditions. Then, we iterate until a modified guessed inner boundary condition leads to the correct known outer boundary value.

### Example

Given the BVP above, note that the value of $y'(a)$ is unknown.

We can make an **initial guess** $y'(a) = z_0$ to reduce
the second-order problem to two first-order problems.

\begin{equation}
y' = u(x) \hspace{0.5cm},\hspace{0.5cm} y'(a) = z_0
\end{equation}

\begin{equation}
u' = f(x,y,u) \hspace{0.5cm},\hspace{0.5cm} y(a) = A.
\end{equation}

Then we just need to integrate the two ODEs from the initial condition until we arrive to the point $x=b$. Since we have chosen (arbitrarily) $z_0$, we have solved
to obtain a function $y=y(x,z_0)$, but our goal is to find $y$ such that it satisfies the other boundary condition: $y(b,z_0) = B$.

Hence, we can define a new function

\begin{equation}
\Phi(z_0) = y(b,z_0) - B
\end{equation}

and search for a $z_0$ so that $\Phi(z_0) = 0$. This means that we are looking for a root of the function $\Phi(z_0)$.

---
### The Shooting Algorithm

The full shooting algorithm for $y'' = f(x,y,y')$ goes as

1. Guess a starting value $z_0 = y'(a)$, set the iteration counter
$i = 0$.

2. Compute $y = y(x,z_i)$ by integrating the IVP.

3. Compute $\Phi(z_i) = y(b,z_i) - B$. If $z_i$ does not
      give a sufficiently accurate solution of the full problem,
      increment $i$ to $i+1$ and find  a value for
      $z_{i+1}$ using a root finder on $\Phi(z_i) = 0$.
      Then go back to (2).

Note that one typically ends up with the secant method, since the
derivative of $\Phi(z)$ is not known in the general case and one is
stuck with having to numerically compute it. For this, at least two
guesses for $z$ are needed.

---

## Finite Differences Method

BVPs as described above can be solved
by Taylor expanding the ODE itself to linear
order (assuming there are no non-linearities in y and y'):

\begin{equation}
y'' = g(x) - p(x)y' - q(x) y\,\,,
\end{equation}

$g(x)$, $p(x)$, and $q(x)$ are functions of $x$ only and the sign
convention is arbitrary.

We can now discretize $y'$ and $y''$ on an evenly spaced grid with
step size $h$,

\begin{equation}
\begin{aligned}
y'(x_i) &= \frac{y(x_{i+1}) - y(x_{i-1})}{2h}\,\,,\\
y''(x_i) &= \frac{y_{i+1} + y_{i-1} - 2 y_i}{h^2}\,\,,
\end{aligned}
\end{equation}

where $x_i = a + i h$, $(i = 0, ... , n+1)$, and

\begin{equation}
h = \frac{b-a}{n+1}.
\end{equation}

The discrete version of these equations is a system
of $n+2$ linear algebraic equations,

\begin{equation}
\begin{aligned}
y_0 &= A\,\,\\
\left(1-\frac{h}{2}p_i \right)  y_{i-1} - \left(2-h^2q_i \right) y_i +  \left(1+\frac{h}{2}p_i \right) y_{i+1} &= h^2 g_i\,\,\\
y_{n+1} &= B\,\,,
\end{aligned}
\end{equation}

where $p_i = p(x_i)$, $g_i = g(x_i)$, and $q_i = q(x_i)$.

This system is a tri-diagonal matrix of dimension $n \times n$:

\begin{equation}
\begin{pmatrix}
-2+h^2q_1 & 1+ \frac{h}{2}p_1& 0 & \cdots &\cdots&\cdots  &0\\
1-\frac{h}{2} p_2&\ddots&\ddots & 0 &\cdots & \cdots & 0\\
0 & \ddots & \ddots & \ddots & 0 & \cdots & \vdots\\
\vdots & 0 & \ddots & \ddots & \ddots & 0 & \vdots \\
\vdots & \vdots & 0 & \ddots & \ddots & \ddots & 0\\
\vdots & \vdots & \vdots& 0& \ddots & \ddots & 1+\frac{h}{2}p_{n-1}\\
\vdots & 0 & 0 & \cdots & 0 & 1-\frac{h}{2} p_n & -2+h^2 q_n\\
\end{pmatrix}
\begin{pmatrix}
y_1\\\vdots \\\vdots \\\vdots \\\vdots \\\vdots \\ y_n
\end{pmatrix}
= \begin{pmatrix}
h^2 g_1 - A(1-\frac{h}{2}p_1)\\
h^2 g_2\\
\vdots\\
\vdots\\
\vdots\\
h^2 g_{n-1}\\
h^2 g_n - B(1+\frac{h}{2}p_n)\\
\end{pmatrix}
\end{equation}

```python

```


# Axion coupling to neutron electric dipole moment

QCD axion-edm coupling defined as
\begin{equation}
g_{an\gamma}^{\mathrm{QCD}} \approx \frac{2.4 \times 10^{-16}}{f_{a}} e \cdot \mathrm{cm} \approx 5.9 \times 10^{-10}\left(\frac{m_{a}}{\mathrm{eV}}\right) \mathrm{GeV}^{-2}
\end{equation}

```python
from numpy import *
import matplotlib.pyplot as plt
from PlotFuncs import BlackHoleSpins,AxionEDM,MySaveFig,FigSetup

fig,ax = FigSetup(Shape='Rectangular',ylab=r'$|g_{an\gamma}|$ [GeV$^{-2}$]',mathpazo=True,\
                              g_min=1e-22,g_max=1e-2,m_min=1e-20,m_max=1e-2,FrequencyAxis=True,N_Hz=1)

AxionEDM.QCDAxion(ax)
AxionEDM.nEDM(ax)
AxionEDM.SN1987A(ax)
AxionEDM.CASPEr(ax)

BlackHoleSpins(ax,6.4e-16*1e12,[0.7e-13,0.2e-18],fs=23,rotation=90,text_col='k',text_on=True,alpha=1,facecolor='gray')
plt.gcf().text(0.475,0.29,'Stellar',ha='center',alpha=0.75)
plt.gcf().text(0.475,0.26,'BHs',ha='center',alpha=0.75)
plt.gcf().text(0.195,0.15,'SM \n BHs',alpha=0.75,ha='center')

MySaveFig(fig,'AxionEDM')
```

```python
fig,ax = FigSetup(Shape='Rectangular',ylab=r'$|g_{an\gamma}|$ [GeV$^{-2}$]',mathpazo=True,\
                              g_min=1e-22,g_max=1e-2,m_min=1e-20,m_max=1e-2,FrequencyAxis=True,N_Hz=1)

BlackHoleSpins(ax,6.4e-16*1e12,[0.7e-13,0.2e-18],fs=23,rotation=90,text_col='k',text_on=True,alpha=1,facecolor='gray')
AxionEDM.QCDAxion(ax)
AxionEDM.nEDM(ax)
AxionEDM.SN1987A(ax)
AxionEDM.CASPEr(ax,projection=True)

plt.gcf().text(0.475,0.29,'Stellar',ha='center',alpha=0.75)
plt.gcf().text(0.475,0.26,'BHs',ha='center',alpha=0.75)
plt.gcf().text(0.195,0.18,'SM \n BHs',alpha=0.75,ha='center');

MySaveFig(fig,'AxionEDM_with_Projections')
```

```python

```


# ClickModels: DBN

ClickModels is a field of study in Machine Learning that uses Probabilistic Graphical Models to model the interactions between users and a set of ranked items.

One of the main uses of ClickModels is to train models from past observed data to evaluate how good each document probably is for each query, also known in literature as judgments' values.

In order to compute the judgments for each document for each query, we rely on the [work](https://pdfs.semanticscholar.org/0b19/b37da5e438e6355418c726469f6a00473dc3.pdf) developed by Aleksandr et. al. where users interactions with each query result is modeled through a Dynamic Bayesian Network as depicted below

<p align="center">

</p>

$E_r$ is a random variable that tracks whether a given document $u$ was examined at rank $r$ by the customer or not (this would be equivallent to the impression event from GA's dataset).

$A_r$ is an indicator as to whether the customer found that given document attractive or not. When a sku is examined and it's attractive, then we have a Click event,represented by the observed variable $C_r$.

Another observed variable is $P_r$ which represents the purchasing event. $P_r$ and $C_r$ both directly influence $S_ur$ which indicates whether customer is satisfied already or not.

Case not, then it's considered that customers can continue examining through the result dataset with a $\gamma$ probability factor.

Creating the DNB above is done through the following code:

```python
import daft
from matplotlib import rc

rc("font", family="serif", size=12)
rc("text", usetex=True)

pgm = daft.PGM(grid_unit=4.0, node_unit=1.4)

# Start with the plates.
rect_params = {"lw": 2}
edge_params = {
    'linewidth': 1
}
pgm.add_plate(
    [0, 0, 3, 2],
    label=r"\Large $r$",
    rect_params=rect_params,
)

pgm.add_plate(
    [3 + 0.2, 0, 3, 2],
    label=r"\Large $r+1$",
    rect_params=rect_params,
)

pgm.add_node("e_r", r"$E_r$", 0.5, 0.5, scale=1.5, fontsize=24)
pgm.add_node("e_r_1", r"$E_{r+1}$", 3.5 + 0.2, 0.5, scale=1.5, fontsize=24)
pgm.add_edge("e_r", "e_r_1", plot_params=edge_params)

pgm.add_node("c_r", r"$C_r$", 1.5, 1., scale=1.5, fontsize=24, observed=True)
pgm.add_node("c_r_1", r"$C_{r+1}$", 3.5 + 0.2 + 1, 1., scale=1.5, fontsize=24, observed=True)
pgm.add_edge("e_r", "c_r", plot_params=edge_params)
pgm.add_edge("e_r_1", "c_r_1", plot_params=edge_params)

pgm.add_node("a_r", r"$A_u$", 0.5, 1.5, scale=1.5, fontsize=24)
pgm.add_node("a_r_1", r"$A_{ur+1}$", 3.7, 1.5, scale=1.5, fontsize=24)
pgm.add_edge("a_r", "c_r", plot_params=edge_params)
pgm.add_edge("a_r_1", "c_r_1", plot_params=edge_params)

pgm.add_node("p_r", r"$P_{ur}$", 2.3, 1., scale=1.5, fontsize=24, observed=True)
pgm.add_node("p_r_1", r"$P_{ur+1}$", 3.5 + 2, 1., scale=1.5, fontsize=24, observed=True)
pgm.add_edge("c_r", "p_r", plot_params=edge_params)
pgm.add_edge("c_r_1", "p_r_1", plot_params=edge_params)

pgm.add_node("s_r", r"$S_{ur}$", 2., 1.7, scale=1.5, fontsize=24)
pgm.add_node("s_r_1", r"$S_{ur+1}$", 3.7 + 1.5, 1.7, scale=1.5, fontsize=24)
pgm.add_edge("c_r", "s_r", plot_params=edge_params)
pgm.add_edge("c_r_1", "s_r_1", plot_params=edge_params)

pgm.add_edge("p_r", "s_r", plot_params=edge_params)
pgm.add_edge("p_r_1", "s_r_1", plot_params=edge_params)
pgm.add_edge("s_r", "e_r_1", plot_params=edge_params)

# Render and save.
pgm.render()
pgm.savefig("dbn.png", dpi=150)
```

Here are the equations we'll be using for finding the parameters of the DBN.

\begin{align}
P\left(E_r=1\mid E{r-1}=0\right) & = 0 \label{eq:1} \tag{1} \\
P\left(A_u=1\right) & = \alpha_{uq} \label{eq:2} \tag{2} \\
P\left(C_r=1\mid E_r=1, A_u=1\right) & = 1 \label{eq:3} \tag{3} \\
P\left(S_{r}=1\mid C_r=0,P_r=0\right) & = 0 \label{eq:4} \tag{4} \\
P\left(S_{r}=1\mid C_r=1,P_r=0\right) & = \sigma_{uq} \label{eq:5} \tag{5} \\
P\left(S_{r}=1\mid C_r=1,P_r=1\right) & = 1 \label{eq:6} \tag{6} \\
P\left(E_{r}=1\mid S_{r-1}=1\right) & = 0 \label{eq:7} \tag{7} \\
P\left(E_{r}=1\mid E_{r-1}=1,S_{r-1}=0\right) & = \gamma \label{eq:8} \tag{8} \\
P\left(C_r=1\right) = P\left(C_{r}=1\mid E_r=1\right)\cdot P\left(E_r=1\right) & = \alpha_{uq}\epsilon_{ru}\label{eq:9} \tag{9} \\
\end{align}

Each query and each sku carries an attractive factor $\alpha_{uq}$. When the customer interacts  with a sku, there's a $\sigma_{uq}$ chance them'll enjoy it and end their browsing through the query result page.

If they are not satisfied, they continue browsing through with a probability of $\gamma$.

In this model, only clicks and purchases are observed which means all other variables are hidden; in such case we use EM optimization techniques to find values for each parameter that best describes observed data in terms of log-likelihood.

This being said, the log-likelihood is given by:

$$\ell\ell = \sum_{s \in S}log\left(\sum_{\textbf{X}}P(\textbf{X}, \textbf{C}^{(s)}, \textbf{P}^{(s)} \mid \Psi \right)$$

Where $X$ represents the hidden variables, $C$ and $P$ are the observed data clicks and purhcases and finally $\Psi$ represents all variables used to model the data.

Finding the derivative of this equation is intractable thanks to the summation of the hidden variables. We use them the [Expectation-Maximization](https://towardsdatascience.com/inference-using-em-algorithm-d71cccb647bc) algorithm and aim to maximize the following $Q$ function:

$$Q = \sum_{s \in S} \mathbb{E}_{X|C^{(s)}}\left[logP\left(X, C^{(s)}, P^{(s)} \mid \Psi\right)\right]$$

In our case, as all variables are Bernoulli (either 0 or 1), each modeled by a parameter $\theta_c$ which translates the above to:

$$
Q(\theta_c) =\sum_{s \in S} \sum_{c_i \in s} \left(P\left(X_{c_i}^{(s)}=1, Par(X_{c_i}^{(s)}) = p \mid C^{(s)}, P^{(s)}, \Psi\right)log(\theta_c) + P\left(X_{c_i}^{(s)}=0, Par(X_{c_i}^{(s)}) = p \mid C^{(s)}, P^{(s)}, \Psi\right)log(1-\theta_c)\right) + Z
$$

We'll be using this equation in the maximization step, derive it to find new optimum values for each parameter of our model and repeat the process until either we reach convergence (usually set by no increment in loglikelihood metric) or by total amount of desired iterations.

The derivative to find new values is given by:

$$\theta_c^{(t+1)} = \frac{\sum_{s\in S}\sum_{c_i \in s}P\left(P(X_{c_i}^{(s)}=1, Par(X_{c_i}^{(s)})=p \mid C^{(s)}, P^{(s)}, \Psi\right)}{\sum_{s\in S}\sum_{c_i \in s}P\left(Par(X_{c_i}^{(s)})=p \mid C^{(s)}, P^{(s)}, \Psi\right)}\label{eq:10} \tag{10}$$

## Attractiveness $\alpha_{uq}$

We have that:

$$P(A_u = 1) = \alpha_{uq}$$

Given equations 1-9, we can also derive that:

\begin{equation}
\begin{split}
\epsilon_1 & = P(E_1=1) = 1 \\
\epsilon_{r+1} & = P(E_{r+1} =1) \\
 & = P(E_{r+1} = 1 \mid E_r=1) \cdot P(E_r=1) \\
 & = \epsilon_r P\left(E_{r+1}=1 \mid S_r = 0, E_r=1\right) \cdot P(S_r=0 \mid E_r=1) \\
 & = \epsilon_r\gamma P(S_r=0 \mid E_r=1) \\
 & = \epsilon_r\gamma \left(P\left(S_r=0 \mid C_r = 0, P_r = 0, E_r=1 \right)P\left(C_r=0, P_r=0 \mid E_r=1\right) + P\left(S_r=0 \mid C_r = 0, P_r = 1, E_r=1 \right)P\left(C_r=0, P_r=1 \mid E_r=1\right) + P\left(S_r=0 \mid C_r = 1, P_r = 0, E_r=1 \right)P\left(C_r=1, P_r=0 \mid E_r=1\right) + P\left(S_r=0 \mid C_r = 1, P_r = 1, E_r=1 \right)P\left(C_r=1, P_r=1 \mid E_r=1\right)\right) \\
 & = \epsilon_r \gamma \left((1 - \alpha_{uq}) + (1 - \sigma_{uq})(1 - cr_{uq})\alpha_{uq} \right)
\end{split}\label{eq:11} \tag{11}
\end{equation}

Where $cr$ is the conversion rate of document $u$ for query $q$.

Given equation 10, we derive for the attractiveness parameter the following updating rule:

$$\alpha_{uq}^{t+1} = \frac{\sum_{s \in S_{uq}} P(A_u = 1 \mid C, P)}{|S_{uq}|} \label{eq:12} \tag{12}$$

But given the structure of the DBN, we can infer that if $C$ is observed then $A_u$ is independent of $P$ as the former is a parent for the attractiveness variable. We can use this to assert a simplified updating rule:

$$\alpha_{uq}^{t+1} = \frac{\sum_{s \in S_{uq}} P(A_u = 1 \mid C)}{|S_{uq}|} \label{eq:13} \tag{13}$$

Which can be developed as follows:

$$
\begin{equation}
\begin{split}
 P(A_u = 1 \mid C) & =  P(A_u = 1 \mid C_r, C_{>r}) \\
  & = \unicode{x1D7D9}(C_r=1)\cdot P(A_u=1 \mid C_r = 1, C_{>r}) + \unicode{x1D7D9}(C_r=0)\cdot P(A_u=1 \mid C_r = 0, C_{>r}) \\
  & = c_r + (1 - c_r) \cdot \left(\unicode{x1D7D9}(C_{>r}=1) \cdot P(A_u=1|C_r=0, C_{>r}=1) + \unicode{x1D7D9}(C_{>r}=0) \cdot P(A_u=1 \mid C_r=0, C_{>r}=0)\right) \\
  & = c_r + (1 - c_r)(1 - c_{>r}) \cdot \frac{P(C_r=0, C_{>r}=0 \mid A_u=1) \cdot P(A_u=1)}{P(C_r=0, C_{>r} = 0)}
\end{split}\label{eq:14} \tag{14}
\end{equation}
 $$

Where $C_u$ is the click on current rank document and $C_{>r}$ is a random variable that is 1 if there's any click above current r and 0 otherwise.

Now developing the numerator of (14) we have:

$$
\begin{equation}
\begin{split}
 P(C_r=0, C_{>r}=0 \mid A_u=1) & = P(C_r=0, C_{>r}=0 \mid A_u=1, E_r=0) \cdot P(E_r=0) \\
 & = P(E_r=0) = 1 - \epsilon_r
\end{split}\label{eq:14.1} \tag{14.1}
\end{equation}
 $$

The equation above is derived from the fact that an attractive document is only not clicked if it's not examined.

The numerator is already solved, we still need to develop the denominator:

$$
\begin{equation}
\begin{split}
 P\left(C_r=0, C_{>r}=0\right) = P(C{\geq r}=0) = 1 - P(C_{\geq r} = 1)
\end{split}\label{eq:14.2} \tag{14.2}
\end{equation}
 $$

$$
\begin{equation}
\begin{split}
P(C_{\geq r} = 1) = \epsilon_r \cdot X_r
\end{split}\label{eq:14.3} \tag{14.3}
\end{equation}
 $$

$$
\begin{equation}
\begin{split}
X_r & = P(C_{\geq r} \mid E_r=1) \\
 & = P(C_r = 1 \mid E_r=1) + P(C_r=0, C_{\geq r+1} \mid E_r=1) \\
 & = \alpha_{uq} + P(C_{\geq r+1} \mid C_r=0, E_r=1) \cdot P(C_r=0|E_r=1) \\
 & = \alpha_{uq} + P(C_{\geq r+1} \mid E_{r+1}) \cdot P(E_{r+1}=1 \mid C_r=0, E_r=1) \cdot (1 - \alpha_{uq}) \\
 & = \alpha_{uq} + (1 - \alpha_{uq})\gamma X_{r+1}
\end{split}\label{eq:14.4} \tag{14.4}
\end{equation}
 $$

Finally, we have the updating rule for the attractiveness parameter:

$$ \alpha_{uq}^{(t+1)} = \frac{\sum_{s \in S_{uq}}\left(c_r^{(s)} + \left(1 - c_r^{(s)}\right)\left(1 - c_{>r}^{(s)}\right) \cdot \frac{\left(1 - \epsilon_r^{(t)}\right)\alpha_{uq}^{(t)}}{\left(1 - \epsilon_r^{(t)}X_r^{(t)} \right)} \right)}{|S_{uq}|} \label{eq:15} \tag{15}$$

Where $\epsilon_r$ is given by equation (11).

## Satisfaction $\sigma_{uq}$

In our presented DBN model, the satisfaction factor is only defined when:

$$ \sigma_{uq} = P(S_u=1 \mid C_r=1, P_r=0)$$

This means the updating rule for the satisfaction term is given by:

$$\sigma_{uq}^{(t+1)} = \frac{\sum_{s \in S'_{uq}}P(S_u=1 \mid C, P)}{|S'_{uq}|} \label{eq:16} \tag{16} $$

Which can be developed as:

$$
\begin{equation}
\begin{split}
P(S_u=1 \mid C, P) &= P(S_u = 1 \mid C_r=1, P_r=0, C_{>r}=0, P_{>r}=0) \\
&= (1 - c_{>r})\cdot P(S_u=1 \mid C_r=1, P_r=0, C_{>r}=0, P_{>r}=0) \\
&= (1 - c_{>r})\cdot \frac{P(C_{>r}=0, P_{>r}=0 \mid S_u=1, C_r=1, P_r=0) \cdot P(S_u=1 \mid C_r=1, P_r=0)}{P(C_{>r}=0, P_{>r}=0 \mid C_r=1, P_r=0)} \\
&= \frac{(1 - c_r)(1-p_r)\sigma_{uq}}{P(P_{>r}=0 \mid C_{>r}=0, C_r=1, P_r=0) \cdot P(C_{>r}=0 \mid C_r=1, P_r=0)} \\
&= \frac{(1 - c_r)(1-p_r)\sigma_{uq}}{1 - P(C_{\geq r+1}=1 \mid E_{r+1})\cdot P(E_{r+1}\mid C_r=1, P_r=0)} \\
&= \frac{(1 - c_r)(1-p_r)\sigma_{uq}}{(1 - X_{r+1}\cdot (1-\alpha_{uq})\gamma)}
\end{split}\label{eq:17} \tag{17}
\end{equation}
 $$

Given equations (16) and (17), we devire that the updating rule is given by:

$$\sigma_{uq}^{(t+1)} = \frac{\sum_{s \in S^{[1, 0]}}\frac{(1 - c_r^{(t)})(1-p_r^{(t)})\sigma_{uq}^{(t)}}{(1 - X_{r+1}\cdot (1-\alpha_{uq}^{(t)})\gamma^{(t)})}}{|S^{[1, 0]}|} \label{eq:18} \tag{18}$$

Where $S^{[1, 0]}$ is the set of sessions of customers interactions where at rank $r$ there's an observed click and no purchase for document $u$ and query $q$.

## Persistence  $\gamma$

Persistence is defined as:

$$\gamma = P(E_{r+1} = 1 \mid E_r = 1, S_{ur}=0)$$

The sufficient statistics for this parameter is defined as:

$$ESS(z) = \sum_{s \in S} \sum_r P(E_{r+1}=z, E_r=1, S_{ur}=0 \mid C, P) \label{eq:19} \tag{19}$$

There's no closed form for this equation, so we use some techniques in order to able to compute it, like so:

$$
\begin{equation}
\begin{split}
ESS(z) &= \sum_{s \in S}\sum_{r}\frac{P(E_{r+1}=\, E_r=1, S_u=0, C, P)}{P(C, P)} \\
&= \sum_{s \in S}\sum_{r}\frac{P(E_{r+1}=\, E_r=1, S_u=0, C, P)}{\sum_x \sum_y \sum_z P(E_{r+1}=z,E_r=x, S_u=y, C, P)} \\
&= \sum_{s \in S}\sum_{r}\frac{P(E_{r+1}=\, E_r=1, S_u=0, C, P) \cdot \frac{1}{P(C_{<r}, P_{<r})}}{\sum_x \sum_y \sum_z P(E_{r+1}=z,E_r=x, S_u=y, C, P)\cdot \frac{1}{P(C_{<r}, P_{<r})}} \\
&= \sum_{s \in S}\sum_{r}\frac{\phi^{(s)}(1, 0,z)}{\sum_x \sum_y \sum_z \phi^{(s)}(x, y, z)}
\end{split}\label{eq:19} \tag{19}
\end{equation}
 $$

The challege then becomes finding a description for the $\phi$ factor. We derive it as follows:

$$
\begin{equation}
\begin{split}
\phi (x, y, z) &= P(E_r=x, C_r=c_r,P_r=p_r, S_u=y, E_{r+1}=z, C_{\geq {r+1}}, P_{\geq {r+1}} \mid C_{<r}, P_{<r}) \\
&= P(C{>r}, P{>r} \mid E_{r+1}=z, E_r=x, S_u=y, C_r=c_r, P_r=p_r) \cdot P(E_r=x, S_u=y, E_{r+1}=z, C_r=c_r, P_r=p_r \mid C_{<r}, P_{<r}) \\
&= P(C_{>r}, P_{>r} \mid E_{r+1}=z) \cdot P(E_{r+1}=z, S_u=y, C_r=c_r, P_r=p_r \mid E_r=x) \cdot P(E_r=x \mid C_{<r}, P_{<r})
\end{split}\label{eq:20} \tag{20}
\end{equation}
 $$

To solve this equation we'll devide it in three parts: first will be $$P(C_{>r}, P_{>r} \mid E_{r+1}=z)$$, second is $$P(E_{r+1}=z, S_u=y, C_r=c_r, P_r=p_r \mid E_r=x)$$ and finally $$P(E_r=x \mid C_{<r}, P_{<r})$$

Let's first solve $P(E_r=x \mid C_{<r}, P_{<r})$:

$$
\begin{equation}
\begin{split}
P(E_{r+1}=1 \mid C_{<r+1}, P_{<r+1}) &= \epsilon_{r+1} = P(E_{r+1}=1 \mid E_r=1, C_{<r+1}, P_{<r+1})P(E_r=1 \mid C_{<r+1}, P_{<r+1}) \\
&=P(E_{r+1}=1 \mid E_r=1,C_r=1, P_r=0) \cdot P(E_r=1 \mid C_{<r+1}, P_{<r+1})c_r^{(s)}(1-p_r^{(s)}) + P(E_{r+1}=1 \mid E_r=1,C_r=0, P_r=0) \cdot P(E_r=1 \mid C_{<r+1}, P_{<r+1}) \cdot (1 - c_r^{(s)})(1-p_r^{(s)}) + P(E_{r+1}=1 \mid E_r=1,C_r=1, P_r=1) \cdot P(E_r=1 \mid C_{<r+1}, P_{<r+1})\cdot c_r^{(s)}p_r^{(s)}  \\
&= (1 - \sigma_{uq})\gamma c_r^{(s)}(1-p_r^{(s)}) + \gamma P(E_r=1 \mid C_r=0, P_r=0, C_{<r}, P_{<r})(1-c_r^{(s)})(1 -p_r^{(s)})
\end{split}\label{eq:21} \tag{21}
\end{equation}
 $$

To develop the inner probability equation, we have that:

$$
\begin{equation}
\begin{split}
P(E_r=1 \mid C_r=0, P_r=0, C_{<r}, P_{<r}) &= \frac{P(E_r=1, C_r=0, P_r=0 \mid C_{<r}, P_{<r})}{P(C_r=0, P_r=0 \mid C_{<r}, P_{<r})} \\
&= \frac{P(C_r=0, P_r=0 \mid E_r=1, C_{<r}, P_{<r}) \cdot P(E_r=1\mid C_{<r}, P_{<r})}{P(C_r=0, P_r=0\mid C_{<r}, P_{<r})}
\end{split}\label{eq:22} \tag{22}
\end{equation}
 $$

Once again we need to develop the inner probability equation, which leads to:

$$
\begin{equation}
\begin{split}
P(C_r=0, P_r=0 \mid C_{<r}, P_{<r}) &= P(P_r=0 \mid C_r=0) \cdot P(C_r=0 \mid C_{<r}, P_{<r}) \\
&= P(C_r=0 \mid C_{<r}, P_{<r}) = 1 - P(C_r=1 \mid C_{<r}, P_{<r})\\
&= 1- P(C_r=1 \mid E_r=1, C_{<r}, P_{<r}) \cdot P(E_r=1 \mid P_{<r}, C_{<r}) \\
&= 1 - \alpha_{uq}\epsilon_r
\end{split}\label{eq:23} \tag{23}
\end{equation}
 $$

Using 22 and 23 into 21 we finally get:

$$P(E_r =1\mid C_{<r}, P_{<r}) = (1 - \sigma_{uq}) \gamma c_r(1-p_r) + \frac{\gamma (1-\alpha_{uq})\epsilon_{r-1}}{(1 - \alpha_{uq} \epsilon_{r-1})}(1-c_r)(1-p_r) \label{eq:24} \tag{24}$$

Now we need to compute $P(C_{>r}, P_{>r} \mid E_{r+1}=z)$ which is derived as:

$$
\begin{equation}
\begin{split}
P(C_{>r}, P_{>r} \mid E_{r+1}=z) &= P(C_r, P_r\mid C_{r-1}, P_{r-1}, ..., E_l=1) \cdot P(C_{r-1}, P_{r-1}, E_l=1) \\
 &= (1-\alpha \epsilon_{rl})\left((1-c_r)(1-p_r) + (1-w)(\alpha \epsilon_{rl}c_r(1-p_r)) + w \alpha \epsilon_{rl}c_r p_r \right) \cdot P(C_{r-1},P_{r-1} \mid C_{<r-1}, P_{<r-1}, E_l=1)
\end{split}\label{eq:25} \tag{25}
\end{equation}
 $$

Where $w$ is the conversion rate of given sku on a given query.

```python
from IPython.core.display import HTML,display

def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)

css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


# Custom Models in pycalphad

M.E. Trybula, T. Gancarz, W. Gąsior, Density, surface tension and viscosity of liquid binary Al-Zn and ternary Al-Li-Zn alloys, Fluid Phase Equilibria, Volume 421, 2016, Pages 39-48, ISSN 0378-3812, http://dx.doi.org/10.1016/j.fluid.2016.03.013.
(http://www.sciencedirect.com/science/article/pii/S0378381216301418)

Władysław Gąsior, "Viscosity modeling of binary alloys: Comparative studies",
        Calphad, Volume 44, 2014, Pages 119-128, ISSN 0364-5916,
        http://dx.doi.org/10.1016/j.calphad.2013.10.007.

Chenyang Zhou, Cuiping Guo, Changrong Li, Zhenmin Du, "Thermodynamic assessment of the phase equilibria and prediction of glass-forming ability of the Al–Cu–Zr system," Journal of Non-Crystalline Solids, Volume 461, 2017,
Pages 47-60,
https://doi.org/10.1016/j.jnoncrysol.2016.09.031.
(http://www.sciencedirect.com/science/article/pii/S0022309316304161)

## Background

We are going to take a CALPHAD-based property model from the literature and use it to predict the viscosity of Al-Cu-Zr liquids.

For a binary alloy liquid under small undercooling, Gąsior suggested an entropy model of the form
$$\eta = (\sum_i x_i \eta_i ) (1 - 2\frac{S_{ex}}{R})$$

where $\eta_i$ is the viscosity of the element $i$, $x_i$ is the mole fraction, $S_{ex}$ is the excess entropy, and $R$ is the gas constant.

```python
from pycalphad import Database
```

We can calculate the excess entropy of the liquid using the Al-Cu-Zr thermodynamic database from Zhou et al.

We add three new parameters to describe the viscosity (in Pa-s) of the pure elements Al, Cu, and Zr:
```
   $ Viscosity test parameters
   PARAMETER ETA(LIQUID,AL;0) 2.98150E+02  +0.000281*EXP(12300/(8.3145*T));   6.00000E+03
  N REF:0 !
   PARAMETER ETA(LIQUID,CU;0) 2.98150E+02  +0.000657*EXP(21500/(8.3145*T));   6.00000E+03
  N REF:0 !
  PARAMETER ETA(LIQUID,ZR;0) 2.98150E+02  +4.74E-3 - 4.97E-6*(T-2128) ;   6.00000E+03
    N REF:0 !
```

Great! However, if we try to load the database now, we will get an error:

```python
dbf = Database('databases/alcuzr-viscosity.tdb')
```

This is because "ETA" parameters are not supported by default in pycalphad, so we need to tell pycalphad's TDB parser that "ETA" should be on the list of supported parameter types:

```python
import pycalphad.io.tdb_keywords
pycalphad.io.tdb_keywords.TDB_PARAM_TYPES.append('ETA')
```

Now the database will load:

```python
dbf = Database('databases/alcuzr-viscosity.tdb')
```

## Writing the Custom Model

Now that we have our "ETA" parameters in the database, we need to write a `Model` class to tell pycalphad how to compute viscosity. All custom models are subclasses of the pycalphad `Model` class. A rough draft of our custom model is shown below.

```python
# Useful imports for custom models
from tinydb import where
import sympy
from pycalphad import Model

class ViscosityModel(Model):
    def build_phase(self, dbe):
        # build_phase is a special function which will be called by pycalphad
        # dbe is our Database object
        # First build all the default models
        super(ViscosityModel, self).build_phase(dbe)
        # Now construct our custom viscosity property
        self.viscosity = self.build_viscosity(dbe)

    def build_viscosity(self, dbe):
        raise NotImplementedError('What do I write here?!')
```

Now we need to write the implementation of `build_viscosity`. The function needs to do four things:
1. Query the Database for all the "ETA" parameters
2. Compute their weighted sum
3. Compute the excess entropy of the liquid
4. Plug all the values into the Gąsior equation and return the result

```python
from tinydb import where
import sympy
from pycalphad import Model, variables as v

class ViscosityModel(Model):
    def build_phase(self, dbe):
        super(ViscosityModel, self).build_phase(dbe)
        self.viscosity = self.build_viscosity(dbe)

    def build_viscosity(self, dbe):
        if self.phase_name != 'LIQUID':
            raise ValueError('Viscosity is only defined for LIQUID phase')
        phase = dbe.phases[self.phase_name]
        param_search = dbe.search
        # STEP 1
        eta_param_query = (
            (where('phase_name') == phase.name) & \
            (where('parameter_type') == 'ETA') & \
            (where('constituent_array').test(self._array_validity))
        )
        # STEP 2
        eta = self.redlich_kister_sum(phase, param_search, eta_param_query)
        # STEP 3
        excess_energy = self.GM - self.models['ref'] - self.models['idmix']
        #liquid_mod = Model(dbe, self.components, self.phase_name)
        ## we only want the excess contributions to the entropy
        #del liquid_mod.models['ref']
        #del liquid_mod.models['idmix']
        excess_entropy = -excess_energy.diff(v.T)
        ks = 2
        # STEP 4
        result = eta * (1 - ks * excess_entropy / v.R)
        self.eta = eta
        return result
```

## Putting it all together

Now we can create an instance of `ViscosityModel` for the liquid phase using the `Database` object we created earlier.

```python
mod = ViscosityModel(dbf, ['CU', 'ZR'], 'LIQUID')
```

Finally we plot the result.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from pycalphad import calculate

mod = ViscosityModel(dbf, ['CU', 'ZR'], 'LIQUID')

temp = 2100
res = calculate(dbf, ['CU', 'ZR'], 'LIQUID', P=101325, T=temp,
                model=mod, output='viscosity') # NOTICE: model=mod to tell pycalphad about our model

fig = plt.figure(figsize=(6,6))
ax = fig.gca()
ax.scatter(res.X.sel(component='ZR'), 1000 * res.viscosity.values)
ax.set_xlabel('X(ZR)')
ax.set_ylabel('Viscosity (mPa-s)')
ax.set_xlim((0,1))
ax.set_title('Viscosity at {}K'.format(temp));
```

We repeat the calculation for Al-Cu.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from pycalphad import calculate

mod = ViscosityModel(dbf, ['CU', 'AL'], 'LIQUID')

temp = 1300
res = calculate(dbf, ['CU', 'AL'], 'LIQUID', P=101325, T=temp,
                model=mod, output='viscosity')

fig = plt.figure(figsize=(6,6))
ax = fig.gca()
ax.scatter(res.X.sel(component='CU'), 1000 * res.viscosity.values)
ax.set_xlabel('X(CU)')
ax.set_ylabel('Viscosity (mPa-s)')
ax.set_xlim((0,1))
ax.set_title('Viscosity at {}K'.format(temp));
```

```python

```


# Support Vector Machine: Clasificación

_Autor:_ [Luis Carlos Padierna](https://scholar.google.com/citations?user=JhchZkQAAAAJ&hl=es&oi=ao)

_Editado por:_ Edwin Bedolla

**NOTA:** Esta introducción, y por extensión este documento, no será _rigurosa_ en la descripción matemática del modelo y metodología aquí presentados debido a su extensión, complejidad y dificultad; existe mucha literatura al respecto por lo que se remite al lector a todas la referencias al final de este documento.

## Introducción a las Máquinas de Soporte Vectorial

Las Máquinas de Soporte Vectorial, o bien _Support Vector Machines_ en inglés, son un modelo matemático de optimización que permite aprender características de un conjunto de datos y generalizarlas a datos desconocidos, empleando un paradigma de aprendizaje conocido como _máquinas de kernel._

Este modelo fue un parteaguas en el desarrollo de la investigación de aprendizaje automatizado e inteligente cuando surgió por primera vez, creado por Vladimir Vapnik y muchos otros en un intento por crear un clasificador realmente capaz de distinguir entre dos clases particulares dentro de un conjunto de datos con características determinadas.

La gran innovación de esta metodología fue su _rigor matemático_, esto es, el método se describe completamente por sus propiedades matemáticas a partir de un problema de optimización. Empleando muchas de las áreas novedosas de las matemáticas se pudo crear un esquema de _aprendizaje de máquina_ realmente poderoso, robusto y riguroso pero sobre todo _inteligente_ que podía tomar un conjunto de datos y realizar predicciones tanto precisas como exactas.

Con el desarrollo del paradigma de _aprendizaje profundo_ las Máquinas de Soporte Vectorial empezaron a tomar un rumbo diferente en el panorama contemporáneo de la _Inteligencia Computacional_ donde ahora forman parte de redes neuronales para potenciar la clasificación y predicción de datos; esto es, ya no se emplean como métodos individuales sino como parte de una estructura más compleja de _aprendizaje automático._

## Estructura y funcionamiento del método

Las Máquinas de Soporte Vectorial (MSV) realizan la clasificación y predicción de los datos creando **hiperplanos** de separación entre _dos clases_ dentro de un conjunto de datos. Si un nuevo _punto_ o _dato_ cae de un lado del _hiperplano_ pertenece a una clase, en caso contrario se asigna a la clase opuesta. Aquí se presentan dos de las **propiedades** de las Máquinas de Soporte Vectorial:

1. Solamente pueden separar entre dos clases (clasificación binaria)
2. La separación entre clases **siempre** se realiza mediante un _hiperplano._

Estas dos propiedades de las MSV son fundamentales, y definen por completo la metodología y la estructura del algoritmo.
Pero las preguntas obligatorias son: ¿cómo se define el _hiperplano_ de separación? ¿cómo se emplean las características del conjunto de datos para discernir entre ambas clases?

Estas preguntas se responden de forma simple con la formulación matemática de las MSV, que se presenta a continuación.

## Formulación matemática

Las MSV construyen el _hiperplano_ de separación mediante un problema de optimización descrito a continuación.

Dado un conjunto de datos de entrenamiento $x_i \in \mathbb{R}^p,i=1,2,\cdots,n$ separados en dos clases, y un arreglo de etiquetas para cada elemento del conjunto de datos $y \in \{-1,1\}^n$ las MSV resuelven el siguiente problema de optimización

$$
\begin{align}
    \min_{w,b,\zeta} \frac{1}{2} w^{\text{T}} w + C \sum_{i=1}^{n} \zeta_i \\
    \text{sujeto a } y_i (w^{\text{T}} \phi(x_i) + b) \geq 1 - \zeta_i \\
    \zeta_i \geq 0, i = 1, 2, \cdots, n
\end{align}
$$

donde $K(x_i, x_j) = \phi(x_i)^{\text{T}} \phi(x_j)$ corresponde a un _kernel de Mercer_ que permite mapear el conjunto de datos a otro espacio; se hablará más al respecto más adelante en el documento.

Por el momento, es posible que esta formulación matemática no tenga mucho sentido, y en realidad es cierto dado que el problema tal cual está planteado no se puede resolver como tal; esta es la formulación teórica tal cual se definió cuando el método se publicó. Este probla de optimización en realidad es más complejo dado que se deben tomar en cuenta la topología y estructura del espacio donde se está trabajando, esto implica que se deben emplear herramientas de topología y análisis real para comprender verdaderamente el planteamiento. Se remite al lector a las referencias para ahondar en la parte teórica y derivación.

## Interpretación intuitiva de las MSV

Con la formulación matemática se tiene un concepto general de las MSV y lo que realizan como metodología, pero es posible traducirlo a palabras de la siguiente forma:

> Las MSV minimizan la distancia entre los puntos de las clases y el _hiperplano_ que las separa.

Este enunciado resume una parte de la formulación matemática, es así como el _hiperplano_ es formado y entonces realiza la clasificación correspondiente. La segunda parte de la formulación matemática habla de la _tercera propiedad_ fundamental de las MSV, los **vectores soporte:**

3. Los **vectores soporte** corresponden a los puntos más cercanos al _hiperplano_ de separación, formando la _frontera de decisión._

Esta propiedad es la que le proporcionó fama y gloria a las MSV pues implica que no siempre es necesario tener todo el conjunto de datos para realizar una clasificación precisa y exacta, basta con tener los _vectores soporte_ de cada clase para hacerlo. Es decir, si el conjunto de datos consta de 3000 elementos y los vectores soporte son solamente 3 en total, la eficiencia computacional para realizar la clasificación es excelente, no hay pérdida de información y siempre será exacta.

A continuación se presentan ejemplos de las propiedades y metodología general de las MSV.

## Sobre la implementación de las MSV

Las MSV constituyen un problema computacional de implementación importante. Debido a su formulación matemática y estructura metodológica no se puede realizar una implementación simple. En lo que queda de este documento se emplea la implementación de `scikit-learn`, que a su vez emplea la implementación de `libsvm`; al final del documento se colocan las referencias de las documentaciones de cada librería.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
from sklearn.svm import SVC
from sklearn.datasets import make_blobs, make_circles

# Importar los widgets de interacción
from ipywidgets import interact, fixed
```

### Hiperplano de separación

Empleando un pequeño conjunto de datos se observará cómo se construye el _hiperplano_ de separación entre las clases. El siguiente código es interactivo por lo que se puede manipular y corroborar el efecto del parámetro $C$ que corresponde a la _tolerancia_ de aceptar o rechazar un punto de acuerdo a la distancia correspondiente.

```python
def mySVC(myKernel='linear', myC=0.01):
    # Crear un conjunto de puntos
    X = np.array([[-2.0, 2.0], [3.0, -1.0], [-1.0, -1.0],
                  [-2.0, -1.0], [1.0, 1.0], [2.0, 1.0], [-1.5,.75]])
    # Y sus respectivas etiquetas
    y = np.array([1, 2, 1, 1, 2, 2, 1])

    # Instanciar el clasificador y entrenarlo
    clf = SVC(kernel=myKernel, C=myC)
    clf.fit(X, y)

    print("Función de Decisión: {}".format(clf.decision_function(X)))
    print("Predicción: {}".format(clf.predict(X)))
    print("Vectores Soporte: {}".format(len(clf.support_vectors_)))
    print("Multiplicadores: {}".format(np.abs(clf.dual_coef_)))

    # Los puntos y los vectores soporte encontrados
    plt.scatter(X[:,0], X[:,1] , c=y, s=150, edgecolors='none', cmap='winter')
    plt.scatter(clf.support_vectors_[:,0], clf.support_vectors_[:,1] , lw=2, s=200, edgecolors='k',facecolors='none')
    # Para graficar el resultado y las fronteras de decisión
    ax = plt.gca()
    xlim = ax.get_xlim()
    ylim = ax.get_ylim()
    x = np.linspace(xlim[0], xlim[1], 3)
    z = np.linspace(ylim[0], ylim[1], 3)
    Y, X = np.meshgrid(z,x)
    xz = np.vstack([X.ravel(), Y.ravel()]).T
    P =clf.decision_function(xz).reshape(X.shape)
    ax.contour(X, Y, P, colors='r', levels = [-1, 0, 1], alpha=0.5, linestyles = ['--', '-', '--'])
```

```python
interact(mySVC, myKernel=["linear", "rbf", "poly"], myC=(0.01, 3, 0.1), addPoint=[0,0])
```

    interactive(children=(Dropdown(description='myKernel', options=('linear', 'rbf', 'poly'), value='linear'), Flo…

    <function __main__.mySVC(myKernel='linear', myC=0.01)>

Para un valor de $C > 1$ se puede observar que los _vectores soporte_ se encuentran y son únicos; entre mayor sea el valor de $C$ este hecho no cambiará.

Algo importante a observar con esta figura es el _hiperplano_ de separación, el cual es en realidad una _recta_ y esto se debe al espacio del conjunto de datos, pero siempre debe ser un _hiperplano._ Algo más a observar es la _tercera propiedad_ y es que los _vectores soporte_ son los puntos más cercanos al _hiperplano_ de separación; todo punto que cae dentro de esta frontera y después pertenece a una u otra clase.

## Kernels y espacios auxiliares

Esta es la 4 y última **propiedad** de las MSV, correspondiente al espacio del conjunto de datos y la formulación matemática intrínseca de las MSV:

4. Las MSV pueden emplear **kernels de Mercer** para encontrar un espacio donde el conjunto de datos sea _linealmente separable._

Si se regresa a la formulación matemática dentro de este documento se puede leer que se introdujo un _kernel de Mercer_ en la formulación. ¿Porqué? Esta pregunta se puede responder con otra pregunta: ¿qué sucede cuando el espacio del conjunto de datos no permite un _hiperplano_ de separación entre las clases?

Esta pregunta es simple de responder: cuando el conjunto de datos **no** es _linealmente separable_ las MSV no se pueden aplicar. Esto contradice todas las 4 propiedades, o _axiomas_ de las MSV pues implicaría que estas 4 propiedades se pueden modificar según cada conjunto de datos, y esto es _inaceptable._ El modelo de MSV debe ser lo suficientemente general para ser aplicable a cualquier tipo de conjunto de datos, de hecho está demostrado y existe mucha literatura al respecto. Entonces, ¿qué alternativa existe?

Los _espacios auxiliares_ son la única alternativa y la más robusta. Básicamente, se pretende encontrar un espacio donde el conjunto de datos pueda ser separado en dos clases mediante un _hiperplano_ y para esto se necesita de una función o _mapeo_ que pueda _transformar_ los datos de un espacio a otro y es en este nuevo espacio donde se pueden aplicar las MSV.

A continuación se presentan ejemplos de este hecho, y se irán comentando algunas cosas al respecto.

```python
# Cargar funciones auxiliares para graficar
%run plots.py
```

    <Figure size 432x288 with 0 Axes>

```python
def SVC_polinomial(myC=0.01, rs=0):
    # Crear conjuntos de datos artificiales, dos clases de 10
    # puntos cada una
    X, y = make_blobs(n_samples=20, centers=2, random_state=rs, cluster_std=0.60)
    # Graficar los datos
    p = plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap="winter")

    # Crear el clasificador y entrenarlo
    model = SVC(kernel="poly", C=myC, gamma="auto")
    model.fit(X, y)

    # Graficar los vectores soporte y las fronteras de decisión
    plot_svc_decision_function(model, plot_support=True)
```

Cuando se hace la instancia del clasificador se especifica el kernel _polinomial_, esto es, una función que va a mapear el conjunto de datos a otro espacio auxiliar donde en este nuevo espacio los conjuntos son linealmente separables.

```python
interact(mySVM_plot, myC=(0.001, 2.0, 0.02), rs=(0, 5, 1))
```

    interactive(children=(FloatSlider(value=0.01, description='myC', max=2.0, min=0.001, step=0.02), IntSlider(val…

    <function __main__.mySVM_plot(myC=0.01, rs=0)>

Para hacer más explícito el hecho de que el espacio auxiliar es diferente se grafica la frontera de decisión y el _"hiperplano de separación"_; en realidad no es un hiperplano pues se puede ver que tiene una ligera curvatura. En realidad se está graficando el _hiperplano_ en el espacio auxiliar, una proyección de cómo se vería este hiperplano.

Esta observación es muy importante, nótese que los _vectores soporte_ sobre la frontera de decisión siguen siendo los más cercanos pero se alcanza a distinguir la curvatura de la frontera. Siempre se debe preservar el hecho de que el conjunto de datos sea _linealmente separable_, y por lo tanto es crucial emplear un _kernel._

```python
# Crear un conjunto de datos de círculos concéntricos
X, y = make_circles(100, factor=0.3, noise=0.1)

# Crear el clasificador y entrenarlos
model = SVC(kernel="rbf", C=100, gamma="auto").fit(X, y)

# Graficar los datos y la frontera de decisión
plt.scatter(X[:,0], X[:,1], c=y, s=50, cmap="cool")
plot_svc_decision_function(model, plot_support=True)
```

Este es un ejemplo muy interesante que vale la pena estudiar a fondo. Este conjunto de datos corresponde a dos círculos concéntricos, cada círculo pertenece a una clase particular. En este caso **no** se puede utilizar un MSV con el conjunto de datos dado que no son _linealmente separables_; no se puede construir un _hiperplano_ tal que separe correctamente ambas clases.

Es imperativo emplear un _kernel de Mercer_ que ayude a crear un espacio auxiliar donde el conjunto de datos sea _linealmente separable_. Primero, se va a proyectar el conjunto de datos en un espacio donde se presume se pueden separar en las dos clases que son.

```python
# Definir la función que corresponde a un paraboloide invertido
r = np.exp(-(X ** 2).sum(1))

def plot_3D(elev=30, azim=30, X=X, y=y):
    # Graficar en 3D la proyección de la función matemática anterior
    ax = plt.subplot(projection="3d")
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, cmap="autumn")
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel("x")
    ax.set_ylabel("y")
    ax.set_zlabel("r")
```

```python
grafica = interact(plot_3D, elev=20, azim=45, X=fixed(X), y=fixed(y))
```

    interactive(children=(IntSlider(value=20, description='elev', max=60, min=-20), IntSlider(value=45, descriptio…

Claramente en este nuevo espacio ambas clases son _linealmente separables_ porque se puede dibujar un _hiperplano_ en medio de ambos subconjuntos. Pero este es un espacio auxiliar y no corresponde al espacio donde se tienen los datos originalmente. En este caso, la formulación matemáticas de las MSV se mantiene pero se debe emplear una formulación equivalente para tomar en cuenta estas propiedades del espacio.

De cualquier forma, ahora ambas clases son _linealmente separables_ y se puede proceder a crear la clasificación.

```python
# Instanciar un model SVC y entrenarlo
model = SVC(kernel="rbf", C=10, gamma="auto")
model.fit(X, y)

# Mostrar los datos y los vectores soporte así como la frontera de decisión
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap="cool")
plot_svc_decision_function(model, plot_support=True)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1],
            s=100, lw=2, edgecolors="k", facecolors="none")
```

En la figura se pueden ver las _fronteras de decisión_ así como los _vectores soporte._ Nótese que las _fronteras_ también son círculos concéntricos, esto es porque en el espacio auxiliar donde son _hiperplanos_ en realidad se proyectan y resultan ser círculos.

Nótese algo muy importante, el número de _vectores soporte,_ esta observación no se considera una propiedad o _axioma_ de las MSV por el hecho que no siempre se cumple, como ya se había mencionado en la introducción. El número de _vectores soporte_ en este ejemplo corresponde a un número muy pequeño respecto al total dentro de cada clase y la gran ventaja de las MSV es poder generalizar un modelo de clasificación con tan pocos puntos.

## Referencias (con algunos comentarios)

1. Kecman, V. Support Vector Machines, an introduction. [Link](https://www.researchgate.net/profile/Vojislav_Kecman/publication/226743605_Support_Vector_Machines_-_An_Introduction/links/02e7e52971b20615e7000000.pdf)

    Esta introducción pretende ser simple pero a la vez completa en su descripción de las MSV, pero se considera que ya se tiene algo de conocimiento en el tema de clasificación.

2. Deng, N., Tian, Y., & Zhang, C. (2012). Support vector machines: optimization based theory, algorithms, and extensions. Chapman and Hall/CRC.

    Este libro se considera _el libro_ cuando se trata de MSV y toda la teoría detrás. Es demasiado riguroso en las matemáticas, por lo que solamente se recomienda si se ha llevado varios cursos de cálculo, un curso de métodos numéricos y recordar muy bien los conceptos de teoría de conjuntos. Poder entender todo sobre este libro y lo que propone corresponde a dominar por completo las MSV.

3. De Sa, J. M. (2012). Pattern recognition: concepts, methods and applications. Springer Science & Business Media.

    Este libro de texto contiene una introducción a todos los temas de reconocimiento de patrones incluido las MSV. Es un libro simple de entender pero sin perder el rigor matemático. Se necesitan menores requisitos que el libro anterior en cuanto a matemáticas.

4. Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.

    Este es uno de los mejores libros de texto para todos los algoritmos clásicos de _inteligencia computacional_. Tiene un capítulo dedicado a las MSV donde se explica a detalle el algoritmo sin tener tanto rigor matemático (algo semejante a lo que se presenta en este documento), sin embargo se requiere de leer algunas otras secciones del libro para comprender por completo lo que se expone.


## 1D Linear operator with two parameters

In the previous examples we focus on systems with only one parameter. To see if the framework can be applied to a more general case, we use a 1D linear operator on two parameters, and verify the prediction.

Here we set up the linear operator as follows:

\begin{align*}
\mathcal{L}_x^\phi u(x) &= f(x) \\
\mathcal{L}_x^\phi &:= \phi_1 \cdot + \phi_2\frac{d}{dx}\cdot
\end{align*}

A suitable solution can be:

\begin{align*}
u ( x ) & = \sin ( x ) \\
f ( x ) & = \phi _ { 1 } \sin ( x ) + \phi _ { 2 } \cos ( x ) \\
x & \in [ 0,1 ]
\end{align*}

In this example, we assume $\phi_1 = 2$, $\phi_2 = 5$, and estimate $\phi_1$ and $\phi_2$.

#### Simulate data

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
##Initiating f(x) and u(x) with 20 data points
x = np.random.rand(20)
phi1 = 2.0
phi2 = 5.0
y_u = np.sin(x)
y_f = phi1*np.sin(x) + phi2*np.cos(x)
```

```python

plt.plot(x, y_u, 'ro')
plt.show()
```

#### Evaluate kernels
Corresponding kernels are defined as following:

$k _ { u u } \left( x _ { i } , x _ { j } ; \theta \right) = \theta \exp \left( - \frac { 1 } { 2 l } \left( x _ { i } - x _ { j } \right) ^ { 2 } \right)$

$\left. \begin{array} { l } { k _ { f f } \left( x _ { i } , x _ { j } ; \theta , \phi _ { 1 } , \phi _ { 2 } \right) } \\ { = \mathcal { L } _ { x _ { i } } ^ { \phi } \mathcal { L } _ { x _ { j } } ^ { \phi } k _ { u u } \left( x _ { i } , x _ { j } ; \theta \right) } \\ { = \mathcal { L } _ { x _ { i } } ^ { \phi } \left( \phi _ { 1 } k _ { u u } + \phi _ { 2 } \frac { \partial } { \partial x _ { j } } k _ { u u } \right) } \\ { = \phi _ { 1 } ^ { 2 } k _ { u u } + \phi _ { 1 } \phi _ { 2 } \frac { \partial } { \partial x _ { j } } k _ { u u } + \phi _ { 1 } \phi _ { 2 } \frac { \partial } { \partial x _ { i } } k _ { u u } + \phi _ { 2 } ^ { 2 } \frac { \partial } { \partial x _ { i } } \frac { \partial } { \partial x _ { i } } k _ { u u } } \end{array} \right.$

$k_{fu}(x_i,x_j;\theta,\phi_1, \phi_2) \\
= \mathcal{L}_{x_i}^\phi k_{uu}(x_i, x_j; \theta) \\
= \phi_1 k_{uu} + \phi_2 \frac{\partial}{\partial x_i}k_{uu}$

$\left. \begin{array} { l } { k _ { u f } \left( x _ { i } , x _ { j } ; \theta , \phi _ { 1 } , \phi _ { 2 } \right) } \\ { = \mathcal { L } _ { x _ { j } } ^ { \phi } k _ { u u } \left( x _ { i } , x _ { j } ; \theta \right) } \end{array} \right.$

```python
x_i, x_j, theta, phi1, phi2 = sp.symbols('x_i x_j theta phi1 phi2')
kuu_sym = sp.exp(theta*(-1)/(2)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta), kuu_sym, "numpy")
def kuu(x, theta):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta)
    return k
```

```python
kff_sym = phi1**2*kuu_sym + phi1*phi2*sp.diff(kuu_sym, x_j) + phi1*phi2*sp.diff(kuu_sym, x_i) + phi2**2*sp.diff(kuu_sym, x_j, x_i)
kff_fn = sp.lambdify((x_i, x_j, theta, phi1,phi2), kff_sym, "numpy")
def kff(x, theta, phi1, phi2):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, phi1, phi2)
    return k
```

```python
kfu_sym = phi1*kuu_sym + phi2*sp.diff(kuu_sym, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, phi1, phi2), kfu_sym, "numpy")
def kfu(x1, x2, theta, phi1, phi2):
    k = np.zeros((x1.size, x2.size))
    for i in range(x1.size):
        for j in range(x2.size):
            k[i,j] = kfu_fn(x1[i], x2[j], theta, phi1, phi2)
    return k
```

```python
def kuf(x1, x2, theta, phi1, phi2):
    return kfu(x1,x2,theta,phi1, phi2).T
```

#### step 3: define negative log marginal likelihood

$K = \begin{bmatrix}
k_{uu}(X_u, X_u; \theta) + \sigma_u^2I & k_{uf}(X_u, X_f; \theta, \phi) \\
k_{fu}(X_f, X_u; \theta, \phi) & k_{ff}(X_f, X_f; \theta, \phi) + \sigma_f^2I
\end{bmatrix}$

For simplicity, assume $\sigma_u = \sigma_f$.

$\mathcal{NLML} = \frac{1}{2} \left[ log|K| + y^TK^{-1}y + Nlog(2\pi) \right]$

where $y = \begin{bmatrix}
y_u \\
y_f
\end{bmatrix}$

```python
def nlml(params, x1, x2, y1, y2, s):
    params = np.exp(params)
    K = np.block([
        [kuu(x1, params[0]) + s*np.identity(x1.size), kuf(x1, x2, params[0], params[1], params[2])],
        [kfu(x1, x2, params[0], params[1], params[2]), kff(x2, params[0], params[1], params[2]) + s*np.identity(x2.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
nlml((1,2,5), x, x, y_u, y_f, 1e-7)

```

    4451304.977390464

#### Optimize hyperparameters

```python
m = minimize(nlml, np.random.rand(3), args=(x, x, y_u, y_f, 1e-7), method="Nelder-Mead")
m
```

     final_simplex: (array([[-1.38622423,  0.69314495,  1.60943818],
           [-1.38616747,  0.69314524,  1.60943819],
           [-1.38614913,  0.69314664,  1.60943791],
           [-1.38626227,  0.6931484 ,  1.60943784]]), array([-277.7085336 , -277.70853355, -277.70853348, -277.70853343]))
               fun: -277.7085336009113
           message: 'Optimization terminated successfully.'
              nfev: 274
               nit: 152
            status: 0
           success: True
                 x: array([-1.38622423,  0.69314495,  1.60943818])

```python

np.exp(m.x[1])

```

    1.9999955379492513

```python
np.exp(m.x[2])
```

    5.000001322351089

```python
phi = [np.exp(m.x[1]),np.exp(m.x[2])]
```

```python
phi ## Estimated phi1 and phi2 using noiseless data points
```

    [1.9999955379492513, 5.000001322351089]

| Parameter | Value |
|-----------|-------|
| $\phi_1$  |1.9999 |
| $\phi_2$  |5.0000 |

We see that the error rate is less than 0.01% for the hyperparameter estimation. This example shows, that one can use our framework for multiple parameter estimation.

```python
def get_simulated_data(n = 20):
    x = np.random.rand(n)
    phi1 = 2.0
    phi2 = 5.0
    y_u = np.sin(x)
    y_f = phi1*np.sin(x) + phi2*np.cos(x)
    y_f = 0*x
    return(x, x, y_u, y_f)

(x, x, y_u, y_f) = get_simulated_data()
res = np.zeros((5,25))
for k in range(5, 25):
    (x, x, y_u, y_f) = get_simulated_data(k)
    nlml_wp = lambda params: nlml(params, x, x, y_u, y_f, 1e-7)
    m = minimize(nlml_wp, np.random.rand(3), method="Nelder-Mead")
    p1 = np.exp(m.x[1])
    p2 = np.exp(m.x[2])
    res = np.square(p1-2)+np.square(p2-5)

```

```python

```